2024-04-22T14:59:23.756 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T14:59:23.756 DEBUG:teuthology.run:Teuthology command: teuthology --name adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --archive /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344 --owner scheduled_adking@teuthology -- /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344/orig.config.yaml 2024-04-22T14:59:23.823 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T14:59:23.941 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344 branch: wip-adk-testing-2024-04-22-0809 description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: adking@redhat.com first_in_suite: false job_id: '7668344' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/reef ' name: adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-04-22-0809 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-adk-testing-2024-04-22-0809 sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 523 sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-04-22-0809 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa targets: smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHlvGdYD94z82Et7ziTp7r2vpxLWc3ovL8uK8FtNWRt4atHctIMAyMUHfU0wfScGwApJxIoo8QJHVaKlGn1J+wo= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvkyQLaIR3B6CoD65dCiMwm4qB5viW/gh1YcPYtVK1Odr0uwF4gZnu2zK6kH7+4cNp9FhbFGY0bDsTapUACCz8= tasks: - install: branch: reef exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef roleless: true - print: '**** done end installing reef cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_14:50:52 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-22T14:59:23.941 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa; will attempt to use it 2024-04-22T14:59:23.942 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks 2024-04-22T14:59:23.942 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T14:59:23.944 INFO:teuthology.task.internal:Checking packages... 2024-04-22T14:59:23.968 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa' 2024-04-22T14:59:23.968 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T14:59:23.969 INFO:teuthology.packaging:ref: None 2024-04-22T14:59:23.969 INFO:teuthology.packaging:tag: None 2024-04-22T14:59:23.969 INFO:teuthology.packaging:branch: wip-adk-testing-2024-04-22-0809 2024-04-22T14:59:23.969 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T14:59:23.969 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-04-22-0809 2024-04-22T14:59:24.115 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3223.g5883d494 2024-04-22T14:59:24.116 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T14:59:24.122 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T14:59:24.122 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T14:59:24.131 INFO:teuthology.task.internal:Saving configuration 2024-04-22T14:59:24.151 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T14:59:24.156 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T14:59:24.181 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 14:52:18.526873', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHlvGdYD94z82Et7ziTp7r2vpxLWc3ovL8uK8FtNWRt4atHctIMAyMUHfU0wfScGwApJxIoo8QJHVaKlGn1J+wo='} 2024-04-22T14:59:24.201 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 14:52:18.525569', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvkyQLaIR3B6CoD65dCiMwm4qB5viW/gh1YcPYtVK1Odr0uwF4gZnu2zK6kH7+4cNp9FhbFGY0bDsTapUACCz8='} 2024-04-22T14:59:24.201 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T14:59:24.207 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-22T14:59:24.207 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-22T14:59:24.207 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T14:59:24.276 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f00cdf8c160>, signals=[15]) 2024-04-22T14:59:24.276 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T14:59:24.281 INFO:teuthology.task.internal:Opening connections... 2024-04-22T14:59:24.281 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T14:59:24.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T14:59:24.358 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-04-22T14:59:24.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T14:59:24.434 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T14:59:24.440 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-04-22T14:59:24.458 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-04-22T14:59:24.458 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T14:59:24.515 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-04-22T14:59:24.516 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T14:59:24.516 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T14:59:24.516 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T14:59:24.516 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-04-22T14:59:24.538 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-04-22T14:59:24.555 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-04-22T14:59:24.556 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-04-22T14:59:24.613 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-04-22T14:59:24.613 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-04-22T14:59:24.613 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-04-22T14:59:24.613 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-04-22T14:59:24.613 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-04-22T14:59:24.613 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-04-22T14:59:24.613 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T14:59:24.614 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-04-22T14:59:24.614 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-04-22T14:59:24.614 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T14:59:24.614 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-04-22T14:59:24.614 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T14:59:24.614 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T14:59:24.614 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T14:59:24.615 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-04-22T14:59:24.629 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T14:59:24.638 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T14:59:24.656 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T14:59:24.657 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-04-22T14:59:24.659 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-04-22T14:59:24.674 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T14:59:24.679 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T14:59:24.680 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-04-22T14:59:24.716 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-04-22T14:59:24.812 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T14:59:24.867 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T14:59:24.880 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T14:59:24.881 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T14:59:24.881 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T14:59:24.881 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T14:59:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T14:59:24.897 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-04-22T14:59:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T14:59:24.898 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-04-22T14:59:24.953 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-437.el9.x86_64 2024-04-22T14:59:24.954 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-437.el9.x86_64 2024-04-22T14:59:24.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-04-22T14:59:24.955 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-437.el9.x86_64 2024-04-22T14:59:24.955 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-437.el9.x86_64 2024-04-22T14:59:24.955 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-04-22T14:59:27.437 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-22T14:59:32.154 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 1.5 MB/s | 8.0 MB 00:05 2024-04-22T14:59:32.776 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 19 MB 00:04 2024-04-22T14:59:34.057 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-22T14:59:38.449 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.2 MB 00:00 2024-04-22T14:59:39.950 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 96 kB/s | 16 kB 00:00 2024-04-22T14:59:41.007 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-22T14:59:44.617 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 771 kB/s | 6.2 MB 00:08 2024-04-22T14:59:46.090 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-22T14:59:46.367 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-04-22T14:59:47.963 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T14:59:47.963 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T14:59:48.058 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T14:59:48.059 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T14:59:48.059 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T14:59:48.179 DEBUG:teuthology.orchestra.run.smithi196:> echo no | sudo yum reinstall kernel || true 2024-04-22T14:59:48.678 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 02:59:46 PM UTC. 2024-04-22T14:59:48.804 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T14:59:48.821 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T14:59:48.821 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T14:59:48.821 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T14:59:48.821 INFO:teuthology.orchestra.run.smithi196.stdout:Reinstalling: 2024-04-22T14:59:48.821 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T14:59:48.822 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T14:59:48.822 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T14:59:48.822 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T14:59:48.822 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T14:59:48.822 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 255 k 2024-04-22T14:59:48.822 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 0 2024-04-22T14:59:48.822 INFO:teuthology.orchestra.run.smithi196.stderr:Operation aborted. 2024-04-22T14:59:48.878 INFO:teuthology.orchestra.run.smithi196.stdout:Is this ok [y/N]: 2024-04-22T14:59:48.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum reinstall -y kernel || true 2024-04-22T14:59:48.985 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-04-22T14:59:49.371 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 02:59:46 PM UTC. 2024-04-22T14:59:49.486 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T14:59:49.487 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T14:59:49.487 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T14:59:49.487 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T14:59:49.487 INFO:teuthology.orchestra.run.smithi196.stdout:Reinstalling: 2024-04-22T14:59:49.488 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T14:59:49.488 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T14:59:49.488 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T14:59:49.488 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T14:59:49.488 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T14:59:49.488 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 255 k 2024-04-22T14:59:49.488 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 0 2024-04-22T14:59:49.488 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T14:59:50.467 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-437.el9.x86_64.rpm 295 kB/s | 255 kB 00:00 2024-04-22T14:59:50.468 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T14:59:50.468 INFO:teuthology.orchestra.run.smithi196.stdout:Total 260 kB/s | 255 kB 00:00 2024-04-22T14:59:50.487 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T14:59:50.667 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T14:59:50.667 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T14:59:50.672 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T14:59:50.672 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T14:59:50.818 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T14:59:50.892 INFO:teuthology.orchestra.run.smithi196.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T14:59:50.893 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T14:59:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T14:59:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T14:59:52.183 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T14:59:52.183 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T14:59:52.183 INFO:teuthology.orchestra.run.smithi196.stdout:Reinstalled: 2024-04-22T14:59:52.183 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T14:59:52.183 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T14:59:52.183 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T14:59:52.355 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T14:59:52.402 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T14:59:52.402 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T14:59:52.402 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T14:59:52.403 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-22T14:59:52.403 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T14:59:52.435 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-04-22T14:59:52.457 DEBUG:teuthology.parallel:result is None 2024-04-22T14:59:54.018 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-04-22T14:59:55.832 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T14:59:55.832 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T14:59:55.934 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T14:59:55.935 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T14:59:55.935 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T14:59:56.056 DEBUG:teuthology.orchestra.run.smithi096:> echo no | sudo yum reinstall kernel || true 2024-04-22T14:59:56.560 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 02:59:53 PM UTC. 2024-04-22T14:59:56.688 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T14:59:56.697 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T14:59:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T14:59:56.697 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T14:59:56.697 INFO:teuthology.orchestra.run.smithi096.stdout:Reinstalling: 2024-04-22T14:59:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T14:59:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T14:59:56.698 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T14:59:56.698 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T14:59:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T14:59:56.698 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 255 k 2024-04-22T14:59:56.699 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 0 2024-04-22T14:59:56.699 INFO:teuthology.orchestra.run.smithi096.stderr:Operation aborted. 2024-04-22T14:59:56.759 INFO:teuthology.orchestra.run.smithi096.stdout:Is this ok [y/N]: 2024-04-22T14:59:56.760 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum reinstall -y kernel || true 2024-04-22T14:59:57.254 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 02:59:53 PM UTC. 2024-04-22T14:59:57.369 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout:Reinstalling: 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T14:59:57.370 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 255 k 2024-04-22T14:59:57.371 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 0 2024-04-22T14:59:57.371 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T14:59:58.444 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-437.el9.x86_64.rpm 275 kB/s | 255 kB 00:00 2024-04-22T14:59:58.445 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T14:59:58.445 INFO:teuthology.orchestra.run.smithi096.stdout:Total 237 kB/s | 255 kB 00:01 2024-04-22T14:59:58.469 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T14:59:58.646 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T14:59:58.646 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T14:59:58.651 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T14:59:58.651 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T14:59:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T14:59:58.879 INFO:teuthology.orchestra.run.smithi096.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T14:59:58.879 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T14:59:59.538 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T14:59:59.538 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T15:00:00.241 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T15:00:00.242 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:00:00.242 INFO:teuthology.orchestra.run.smithi096.stdout:Reinstalled: 2024-04-22T15:00:00.242 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T15:00:00.242 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:00:00.242 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:00:00.425 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T15:00:00.471 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T15:00:00.472 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T15:00:00.472 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T15:00:00.472 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-22T15:00:00.472 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T15:00:00.500 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-04-22T15:00:00.533 DEBUG:teuthology.parallel:result is None 2024-04-22T15:00:00.534 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T15:00:00.540 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T15:00:00.540 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T15:00:00.576 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T15:00:00.594 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T15:00:00.600 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T15:00:00.607 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T15:00:00.608 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T15:00:00.630 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T15:00:00.726 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T15:00:00.734 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T15:00:00.734 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T15:00:00.738 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T15:00:00.775 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T15:00:00.786 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T15:00:00.790 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T15:00:00.799 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T15:00:00.801 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T15:00:00.808 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T15:00:00.808 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T15:00:00.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T15:00:00.855 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T15:00:00.864 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T15:00:00.864 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T15:00:00.874 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T15:00:00.912 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T15:00:00.968 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T15:00:01.011 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T15:00:01.089 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T15:00:01.131 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:00:01.131 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T15:00:01.193 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T15:00:01.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T15:00:01.285 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T15:00:01.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T15:00:01.410 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:00:01.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T15:00:01.475 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-04-22T15:00:01.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-04-22T15:00:01.506 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T15:00:01.554 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T15:00:02.048 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T15:00:02.059 INFO:teuthology.task.internal:Starting timer... 2024-04-22T15:00:02.059 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T15:00:02.074 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T15:00:02.082 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T15:00:02.082 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-04-22T15:00:02.119 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-04-22T15:00:02.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-04-22T15:00:02.168 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-04-22T15:00:02.170 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T15:00:02.170 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-04-22T15:00:02.195 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-04-22T15:00:02.196 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-04-22T15:00:02.218 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-04-22T15:00:02.218 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-04-22T15:00:02.218 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T15:00:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:00:02.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T15:00:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:00:02.286 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T15:00:02.286 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-04-22T15:00:02.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-04-22T15:00:02.357 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T15:00:02.368 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T15:00:02.387 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T15:00:02.389 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi096.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T15:04:05.964 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-04-22T15:04:05.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-22T15:04:05.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T15:04:06.069 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-04-22T15:04:06.146 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-22T15:04:06.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-22T15:04:06.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T15:04:06.276 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-04-22T15:04:06.356 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-22T15:04:06.357 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T15:04:06.370 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T15:04:06.371 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T15:04:06.371 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T15:04:06.374 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T15:04:06.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T15:04:06.409 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T15:04:06.417 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T15:04:06.426 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T15:04:06.436 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T15:04:06.440 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-04-22T15:04:06.451 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-04-22T15:04:06.458 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-04-22T15:04:06.464 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T15:04:06.470 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-04-22T15:04:06.477 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T15:04:06.481 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T15:04:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T15:04:06.532 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-04-22T15:04:06.539 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T15:04:06.540 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-04-22T15:04:06.540 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:04:06.540 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:04:06.540 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:04:06.540 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:04:06.555 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-04-22T15:04:06.556 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T15:04:06.556 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-04-22T15:04:06.556 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:04:06.556 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:04:06.556 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:04:06.556 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:04:06.557 INFO:teuthology.run_tasks:Running task install... 2024-04-22T15:04:06.567 DEBUG:teuthology.task.install:project ceph 2024-04-22T15:04:06.567 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'}} 2024-04-22T15:04:06.568 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'} 2024-04-22T15:04:06.568 INFO:teuthology.task.install:Using flavor: default 2024-04-22T15:04:06.581 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T15:04:06.581 INFO:teuthology.task.install:extra packages: [] 2024-04-22T15:04:06.582 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa', 'tag': None, 'wait_for_package': False} 2024-04-22T15:04:06.583 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T15:04:06.583 INFO:teuthology.packaging:ref: None 2024-04-22T15:04:06.583 INFO:teuthology.packaging:tag: None 2024-04-22T15:04:06.583 INFO:teuthology.packaging:branch: reef 2024-04-22T15:04:06.583 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:04:06.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-22T15:04:06.585 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa', 'tag': None, 'wait_for_package': False} 2024-04-22T15:04:06.585 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T15:04:06.586 INFO:teuthology.packaging:ref: None 2024-04-22T15:04:06.586 INFO:teuthology.packaging:tag: None 2024-04-22T15:04:06.586 INFO:teuthology.packaging:branch: reef 2024-04-22T15:04:06.586 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:04:06.586 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-22T15:04:06.829 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/flavors/default/ 2024-04-22T15:04:06.829 INFO:teuthology.task.install.rpm:Package version is 18.2.2-0 2024-04-22T15:04:06.881 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/flavors/default/ 2024-04-22T15:04:06.881 INFO:teuthology.task.install.rpm:Package version is 18.2.2-0 2024-04-22T15:04:06.931 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T15:04:06.931 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:04:06.938 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T15:04:06.957 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T15:04:06.957 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:04:06.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T15:04:06.975 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T15:04:06.975 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T15:04:06.975 INFO:teuthology.packaging:ref: None 2024-04-22T15:04:06.975 INFO:teuthology.packaging:tag: None 2024-04-22T15:04:06.976 INFO:teuthology.packaging:branch: reef 2024-04-22T15:04:06.976 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:04:06.976 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T15:04:06.991 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T15:04:06.991 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T15:04:06.991 INFO:teuthology.packaging:ref: None 2024-04-22T15:04:06.991 INFO:teuthology.packaging:tag: None 2024-04-22T15:04:06.991 INFO:teuthology.packaging:branch: reef 2024-04-22T15:04:06.991 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:04:06.991 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T15:04:07.043 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T15:04:07.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T15:04:07.116 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T15:04:07.131 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T15:04:07.181 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-04-22T15:04:07.183 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-04-22T15:04:07.194 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-04-22T15:04:07.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-04-22T15:04:07.408 INFO:teuthology.orchestra.run.smithi096.stdout:48 files removed 2024-04-22T15:04:07.422 INFO:teuthology.orchestra.run.smithi196.stdout:48 files removed 2024-04-22T15:04:07.442 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw 2024-04-22T15:04:07.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-04-22T15:04:07.901 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 393 kB/s | 76 kB 00:00 2024-04-22T15:04:07.936 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 335 kB/s | 76 kB 00:00 2024-04-22T15:04:08.125 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 52 kB/s | 9.4 kB 00:00 2024-04-22T15:04:08.162 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 51 kB/s | 9.4 kB 00:00 2024-04-22T15:04:08.325 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 13 kB/s | 2.2 kB 00:00 2024-04-22T15:04:08.338 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 15 kB/s | 2.2 kB 00:00 2024-04-22T15:04:09.145 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-22T15:04:09.569 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-04-22T15:04:11.507 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-22T15:04:13.553 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-04-22T15:04:16.579 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.2 MB 00:01 2024-04-22T15:04:17.904 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.2 MB 00:00 2024-04-22T15:04:18.130 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-04-22T15:04:19.392 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-22T15:04:19.454 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-22T15:04:21.299 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-22T15:04:24.367 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-04-22T15:04:26.371 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-22T15:04:26.550 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:04:26.552 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:04:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-04-22T15:04:26.552 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:04:26.552 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:04:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:18.2.2-0.el9 ceph 7.7 M 2024-04-22T15:04:26.552 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-04-22T15:04:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:18.2.2-0.el9 ceph 3.3 M 2024-04-22T15:04:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:18.2.2-0.el9 ceph 3.0 M 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:18.2.2-0.el9 ceph 5.2 M 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:18.2.2-0.el9 ceph 18 M 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:18.2.2-0.el9 ceph 25 k 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:18.2.2-0.el9 ceph 658 k 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T15:04:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:18.2.2-0.el9 ceph 475 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:18.2.2-0.el9 ceph 4.5 M 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:18.2.2-0.el9 ceph 45 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:18.2.2-0.el9 ceph 124 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:18.2.2-0.el9 ceph 162 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T15:04:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:18.2.2-0.el9 ceph 320 k 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:18.2.2-0.el9 ceph 298 k 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:18.2.2-0.el9 ceph 99 k 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout:Install 28 Packages 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-04-22T15:04:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:26.556 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 53 M 2024-04-22T15:04:26.556 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:04:27.034 INFO:teuthology.orchestra.run.smithi196.stdout:(1/30): ceph-base-18.2.2-0.el9.x86_64.rpm 31 MB/s | 5.2 MB 00:00 2024-04-22T15:04:27.059 INFO:teuthology.orchestra.run.smithi196.stdout:(2/30): ceph-selinux-18.2.2-0.el9.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-04-22T15:04:27.103 INFO:teuthology.orchestra.run.smithi196.stdout:(3/30): libcephfs2-18.2.2-0.el9.x86_64.rpm 15 MB/s | 658 kB 00:00 2024-04-22T15:04:27.138 INFO:teuthology.orchestra.run.smithi196.stdout:(4/30): libradosstriper1-18.2.2-0.el9.x86_64.rp 14 MB/s | 475 kB 00:00 2024-04-22T15:04:27.227 INFO:teuthology.orchestra.run.smithi196.stdout:(5/30): librgw2-18.2.2-0.el9.x86_64.rpm 49 MB/s | 4.5 MB 00:00 2024-04-22T15:04:27.310 INFO:teuthology.orchestra.run.smithi196.stdout:(6/30): ceph-radosgw-18.2.2-0.el9.x86_64.rpm 17 MB/s | 7.7 MB 00:00 2024-04-22T15:04:27.328 INFO:teuthology.orchestra.run.smithi196.stdout:(7/30): python3-ceph-argparse-18.2.2-0.el9.x86_ 447 kB/s | 45 kB 00:00 2024-04-22T15:04:27.354 INFO:teuthology.orchestra.run.smithi196.stdout:(8/30): python3-ceph-common-18.2.2-0.el9.x86_64 2.9 MB/s | 124 kB 00:00 2024-04-22T15:04:27.378 INFO:teuthology.orchestra.run.smithi196.stdout:(9/30): python3-cephfs-18.2.2-0.el9.x86_64.rpm 3.1 MB/s | 162 kB 00:00 2024-04-22T15:04:27.404 INFO:teuthology.orchestra.run.smithi196.stdout:(10/30): python3-rados-18.2.2-0.el9.x86_64.rpm 6.2 MB/s | 320 kB 00:00 2024-04-22T15:04:27.429 INFO:teuthology.orchestra.run.smithi196.stdout:(11/30): python3-rbd-18.2.2-0.el9.x86_64.rpm 5.8 MB/s | 298 kB 00:00 2024-04-22T15:04:27.453 INFO:teuthology.orchestra.run.smithi196.stdout:(12/30): python3-rgw-18.2.2-0.el9.x86_64.rpm 2.3 MB/s | 99 kB 00:00 2024-04-22T15:04:27.657 INFO:teuthology.orchestra.run.smithi196.stdout:(13/30): ceph-common-18.2.2-0.el9.x86_64.rpm 23 MB/s | 18 MB 00:00 2024-04-22T15:04:27.738 INFO:teuthology.orchestra.run.smithi196.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-04-22T15:04:27.763 INFO:teuthology.orchestra.run.smithi196.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-04-22T15:04:27.789 INFO:teuthology.orchestra.run.smithi196.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 467 kB/s | 160 kB 00:00 2024-04-22T15:04:27.839 INFO:teuthology.orchestra.run.smithi196.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 848 kB/s | 42 kB 00:00 2024-04-22T15:04:27.872 INFO:teuthology.orchestra.run.smithi196.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-22T15:04:27.906 INFO:teuthology.orchestra.run.smithi196.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-04-22T15:04:27.973 INFO:teuthology.orchestra.run.smithi196.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 828 kB/s | 111 kB 00:00 2024-04-22T15:04:28.023 INFO:teuthology.orchestra.run.smithi196.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 515 kB/s | 26 kB 00:00 2024-04-22T15:04:28.057 INFO:teuthology.orchestra.run.smithi196.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-04-22T15:04:28.082 INFO:teuthology.orchestra.run.smithi196.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 828 kB/s | 49 kB 00:00 2024-04-22T15:04:28.107 INFO:teuthology.orchestra.run.smithi196.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-22T15:04:28.174 INFO:teuthology.orchestra.run.smithi196.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-22T15:04:28.324 INFO:teuthology.orchestra.run.smithi196.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-04-22T15:04:28.401 INFO:teuthology.orchestra.run.smithi196.stdout:(27/30): librados2-18.2.2-0.el9.x86_64.rpm 44 MB/s | 3.3 MB 00:00 2024-04-22T15:04:28.476 INFO:teuthology.orchestra.run.smithi196.stdout:(28/30): librbd1-18.2.2-0.el9.x86_64.rpm 40 MB/s | 3.0 MB 00:00 2024-04-22T15:04:28.504 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:04:28.506 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:04:28.506 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-22T15:04:28.506 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:04:28.506 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:18.2.2-0.el9 ceph 7.7 M 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:18.2.2-0.el9 ceph 3.3 M 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:18.2.2-0.el9 ceph 3.0 M 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:18.2.2-0.el9 ceph 5.2 M 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:18.2.2-0.el9 ceph 18 M 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:18.2.2-0.el9 ceph 25 k 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T15:04:28.507 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:18.2.2-0.el9 ceph 658 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:18.2.2-0.el9 ceph 475 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:18.2.2-0.el9 ceph 4.5 M 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:18.2.2-0.el9 ceph 45 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:18.2.2-0.el9 ceph 124 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:18.2.2-0.el9 ceph 162 k 2024-04-22T15:04:28.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:18.2.2-0.el9 ceph 320 k 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:18.2.2-0.el9 ceph 298 k 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:18.2.2-0.el9 ceph 99 k 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout:Install 28 Packages 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 2 Packages 2024-04-22T15:04:28.509 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:28.513 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 53 M 2024-04-22T15:04:28.513 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:04:28.592 INFO:teuthology.orchestra.run.smithi196.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 3.8 MB/s | 1.6 MB 00:00 2024-04-22T15:04:28.767 INFO:teuthology.orchestra.run.smithi196.stdout:(30/30): libarrow-9.0.0-10.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-04-22T15:04:28.768 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:04:28.769 INFO:teuthology.orchestra.run.smithi196.stdout:Total 24 MB/s | 53 MB 00:02 2024-04-22T15:04:28.839 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:04:28.874 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:04:28.874 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:04:29.181 INFO:teuthology.orchestra.run.smithi096.stdout:(1/30): ceph-base-18.2.2-0.el9.x86_64.rpm 28 MB/s | 5.2 MB 00:00 2024-04-22T15:04:29.206 INFO:teuthology.orchestra.run.smithi096.stdout:(2/30): ceph-selinux-18.2.2-0.el9.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-04-22T15:04:29.240 INFO:teuthology.orchestra.run.smithi096.stdout:(3/30): libcephfs2-18.2.2-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-04-22T15:04:29.276 INFO:teuthology.orchestra.run.smithi096.stdout:(4/30): libradosstriper1-18.2.2-0.el9.x86_64.rp 14 MB/s | 475 kB 00:00 2024-04-22T15:04:29.298 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:04:29.298 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:04:29.399 INFO:teuthology.orchestra.run.smithi096.stdout:(5/30): ceph-radosgw-18.2.2-0.el9.x86_64.rpm 19 MB/s | 7.7 MB 00:00 2024-04-22T15:04:29.466 INFO:teuthology.orchestra.run.smithi096.stdout:(6/30): librgw2-18.2.2-0.el9.x86_64.rpm 23 MB/s | 4.5 MB 00:00 2024-04-22T15:04:29.490 INFO:teuthology.orchestra.run.smithi096.stdout:(7/30): python3-ceph-argparse-18.2.2-0.el9.x86_ 490 kB/s | 45 kB 00:00 2024-04-22T15:04:29.521 INFO:teuthology.orchestra.run.smithi096.stdout:(8/30): python3-ceph-common-18.2.2-0.el9.x86_64 4.9 MB/s | 124 kB 00:00 2024-04-22T15:04:29.541 INFO:teuthology.orchestra.run.smithi096.stdout:(9/30): python3-cephfs-18.2.2-0.el9.x86_64.rpm 3.1 MB/s | 162 kB 00:00 2024-04-22T15:04:29.566 INFO:teuthology.orchestra.run.smithi096.stdout:(10/30): python3-rados-18.2.2-0.el9.x86_64.rpm 6.2 MB/s | 320 kB 00:00 2024-04-22T15:04:29.592 INFO:teuthology.orchestra.run.smithi096.stdout:(11/30): python3-rbd-18.2.2-0.el9.x86_64.rpm 5.8 MB/s | 298 kB 00:00 2024-04-22T15:04:29.617 INFO:teuthology.orchestra.run.smithi096.stdout:(12/30): python3-rgw-18.2.2-0.el9.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-04-22T15:04:29.826 INFO:teuthology.orchestra.run.smithi096.stdout:(13/30): ceph-common-18.2.2-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-04-22T15:04:29.867 INFO:teuthology.orchestra.run.smithi096.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 388 kB/s | 107 kB 00:00 2024-04-22T15:04:29.900 INFO:teuthology.orchestra.run.smithi096.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 581 kB/s | 160 kB 00:00 2024-04-22T15:04:29.926 INFO:teuthology.orchestra.run.smithi096.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-04-22T15:04:30.016 INFO:teuthology.orchestra.run.smithi096.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-04-22T15:04:30.035 INFO:teuthology.orchestra.run.smithi096.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 388 kB/s | 42 kB 00:00 2024-04-22T15:04:30.060 INFO:teuthology.orchestra.run.smithi096.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-04-22T15:04:30.227 INFO:teuthology.orchestra.run.smithi096.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 524 kB/s | 111 kB 00:00 2024-04-22T15:04:30.289 INFO:teuthology.orchestra.run.smithi096.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-04-22T15:04:30.311 INFO:teuthology.orchestra.run.smithi096.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-04-22T15:04:30.353 INFO:teuthology.orchestra.run.smithi096.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-04-22T15:04:30.379 INFO:teuthology.orchestra.run.smithi096.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-22T15:04:30.479 INFO:teuthology.orchestra.run.smithi096.stdout:(25/30): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-22T15:04:30.504 INFO:teuthology.orchestra.run.smithi096.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-04-22T15:04:30.580 INFO:teuthology.orchestra.run.smithi096.stdout:(27/30): librados2-18.2.2-0.el9.x86_64.rpm 43 MB/s | 3.3 MB 00:00 2024-04-22T15:04:30.621 INFO:teuthology.orchestra.run.smithi096.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-22T15:04:30.656 INFO:teuthology.orchestra.run.smithi096.stdout:(29/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-04-22T15:04:30.680 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:04:30.714 INFO:teuthology.orchestra.run.smithi096.stdout:(30/30): librbd1-18.2.2-0.el9.x86_64.rpm 22 MB/s | 3.0 MB 00:00 2024-04-22T15:04:30.715 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:04:30.715 INFO:teuthology.orchestra.run.smithi096.stdout:Total 24 MB/s | 53 MB 00:02 2024-04-22T15:04:30.759 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T15:04:30.782 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:04:30.788 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T15:04:30.817 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:04:30.817 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:04:30.838 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T15:04:31.147 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T15:04:31.238 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:04:31.238 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:04:31.246 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:18.2.2-0.el9.x86_64 5/32 2024-04-22T15:04:31.321 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:18.2.2-0.el9.x86_64 5/32 2024-04-22T15:04:31.403 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:18.2.2-0.el9.x86_64 6/32 2024-04-22T15:04:31.439 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:18.2.2-0.el9.x86_64 7/32 2024-04-22T15:04:31.515 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:18.2.2-0.el9.x86_64 7/32 2024-04-22T15:04:31.539 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T15:04:31.573 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:18.2.2-0.el9.x86_64 9/32 2024-04-22T15:04:31.642 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:18.2.2-0.el9.x86_64 10/32 2024-04-22T15:04:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:18.2.2-0.el9.x86_64 11/32 2024-04-22T15:04:31.715 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:18.2.2-0.el9.x86_64 11/32 2024-04-22T15:04:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T15:04:31.856 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T15:04:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T15:04:31.922 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T15:04:31.949 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T15:04:31.984 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T15:04:32.017 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T15:04:32.314 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T15:04:32.355 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:18.2.2-0.el9.x86_64 20/32 2024-04-22T15:04:32.445 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:18.2.2-0.el9.x86_64 20/32 2024-04-22T15:04:32.478 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:18.2.2-0.el9.x86_64 21/32 2024-04-22T15:04:32.521 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T15:04:32.721 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:04:32.806 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T15:04:32.844 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T15:04:32.893 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T15:04:32.919 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T15:04:32.949 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:18.2.2-0.el9.x86_64 24/32 2024-04-22T15:04:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:18.2.2-0.el9.x86_64 24/32 2024-04-22T15:04:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:18.2.2-0.el9.x86_64 25/32 2024-04-22T15:04:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:18.2.2-0.el9.x86_64 26/32 2024-04-22T15:04:33.202 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T15:04:33.309 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:18.2.2-0.el9.x86_64 5/32 2024-04-22T15:04:33.401 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:18.2.2-0.el9.x86_64 5/32 2024-04-22T15:04:33.481 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:18.2.2-0.el9.x86_64 6/32 2024-04-22T15:04:33.519 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:18.2.2-0.el9.x86_64 7/32 2024-04-22T15:04:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:18.2.2-0.el9.x86_64 7/32 2024-04-22T15:04:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T15:04:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:18.2.2-0.el9.x86_64 9/32 2024-04-22T15:04:33.719 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:18.2.2-0.el9.x86_64 10/32 2024-04-22T15:04:33.751 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:18.2.2-0.el9.x86_64 11/32 2024-04-22T15:04:33.819 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:18.2.2-0.el9.x86_64 11/32 2024-04-22T15:04:33.926 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T15:04:33.961 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T15:04:34.001 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T15:04:34.026 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T15:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T15:04:34.089 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T15:04:34.122 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T15:04:34.422 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T15:04:34.468 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:18.2.2-0.el9.x86_64 20/32 2024-04-22T15:04:34.549 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:18.2.2-0.el9.x86_64 20/32 2024-04-22T15:04:34.605 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:18.2.2-0.el9.x86_64 21/32 2024-04-22T15:04:34.642 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T15:04:35.040 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T15:04:35.071 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:18.2.2-0.el9.x86_64 24/32 2024-04-22T15:04:35.133 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:18.2.2-0.el9.x86_64 24/32 2024-04-22T15:04:35.182 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:18.2.2-0.el9.x86_64 27/32 2024-04-22T15:04:35.200 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:18.2.2-0.el9.x86_64 25/32 2024-04-22T15:04:35.223 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:18.2.2-0.el9.x86_64 27/32 2024-04-22T15:04:35.231 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:18.2.2-0.el9.x86_64 26/32 2024-04-22T15:04:35.757 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:18.2.2-0.el9.x86_64 27/32 2024-04-22T15:04:35.809 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:18.2.2-0.el9.x86_64 28/32 2024-04-22T15:04:35.896 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:18.2.2-0.el9.x86_64 28/32 2024-04-22T15:04:35.897 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T15:04:35.897 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T15:04:35.897 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:35.921 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:18.2.2-0.el9.x86_64 29/32 2024-04-22T15:04:37.313 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:18.2.2-0.el9.x86_64 27/32 2024-04-22T15:04:37.361 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:18.2.2-0.el9.x86_64 27/32 2024-04-22T15:04:37.900 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:18.2.2-0.el9.x86_64 27/32 2024-04-22T15:04:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:18.2.2-0.el9.x86_64 28/32 2024-04-22T15:04:38.038 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:18.2.2-0.el9.x86_64 28/32 2024-04-22T15:04:38.038 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T15:04:38.038 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T15:04:38.039 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:38.065 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:18.2.2-0.el9.x86_64 29/32 2024-04-22T15:04:45.557 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:18.2.2-0.el9.x86_64 29/32 2024-04-22T15:04:45.558 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-04-22T15:04:45.558 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-04-22T15:04:45.558 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-04-22T15:04:45.558 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-04-22T15:04:45.558 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-04-22T15:04:45.558 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-04-22T15:04:45.558 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-04-22T15:04:45.558 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:18.2.2-0.el9.x86_64 30/32 2024-04-22T15:04:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-0.el9.x86_64 30/32 2024-04-22T15:04:46.332 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:04:46.332 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T15:04:46.332 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T15:04:46.332 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T15:04:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T15:04:46.410 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T15:04:46.410 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:04:47.439 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:18.2.2-0.el9.x86_64 29/32 2024-04-22T15:04:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-04-22T15:04:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-04-22T15:04:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-04-22T15:04:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-04-22T15:04:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-04-22T15:04:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-04-22T15:04:47.440 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-04-22T15:04:47.440 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:47.674 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:18.2.2-0.el9.x86_64 1/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:18.2.2-0.el9.x86_64 2/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:18.2.2-0.el9.x86_64 3/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:18.2.2-0.el9.x86_64 4/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:18.2.2-0.el9.x86_64 5/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:18.2.2-0.el9.x86_64 6/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:18.2.2-0.el9.x86_64 7/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:18.2.2-0.el9.x86_64 8/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:18.2.2-0.el9.x86_64 9/32 2024-04-22T15:04:47.675 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:18.2.2-0.el9.x86_64 10/32 2024-04-22T15:04:47.676 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:18.2.2-0.el9.x86_64 11/32 2024-04-22T15:04:47.676 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:18.2.2-0.el9.x86_64 12/32 2024-04-22T15:04:47.676 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:18.2.2-0.el9.x86_64 13/32 2024-04-22T15:04:47.676 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T15:04:47.676 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T15:04:47.676 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T15:04:47.676 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T15:04:47.676 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T15:04:47.676 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T15:04:47.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T15:04:47.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T15:04:47.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T15:04:47.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T15:04:47.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T15:04:47.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T15:04:47.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T15:04:47.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T15:04:47.682 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T15:04:47.682 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:18.2.2-0.el9.x86_64 29/32 2024-04-22T15:04:47.682 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T15:04:47.682 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:18.2.2-0.el9.x86_64 31/32 2024-04-22T15:04:48.198 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:18.2.2-0.el9.x86_64 30/32 2024-04-22T15:04:48.222 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-0.el9.x86_64 30/32 2024-04-22T15:04:48.222 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:04:48.222 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T15:04:48.222 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T15:04:48.222 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T15:04:48.222 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:48.224 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T15:04:48.308 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T15:04:48.308 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:04:48.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:04:48.455 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:18.2.2-0.el9.x86_64 librbd1-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T15:04:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.457 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T15:04:48.458 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.458 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.458 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:48.458 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T15:04:48.458 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T15:04:48.458 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:48.458 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:04:48.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-04-22T15:04:49.257 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:25 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:04:49.379 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:04:49.379 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:18.2.2-0.el9 ceph 40 M 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:18.2.2-0.el9 ceph 165 k 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:04:49.382 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:04:49.383 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-04-22T15:04:49.386 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:49.386 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 40 M 2024-04-22T15:04:49.386 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 166 M 2024-04-22T15:04:49.386 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:04:49.482 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:18.2.2-0.el9.x86_64 1/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:18.2.2-0.el9.x86_64 2/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:18.2.2-0.el9.x86_64 3/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:18.2.2-0.el9.x86_64 4/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:18.2.2-0.el9.x86_64 5/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:18.2.2-0.el9.x86_64 6/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:18.2.2-0.el9.x86_64 7/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:18.2.2-0.el9.x86_64 8/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:18.2.2-0.el9.x86_64 9/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:18.2.2-0.el9.x86_64 10/32 2024-04-22T15:04:49.483 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:18.2.2-0.el9.x86_64 11/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:18.2.2-0.el9.x86_64 12/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:18.2.2-0.el9.x86_64 13/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T15:04:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T15:04:49.485 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T15:04:49.485 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T15:04:49.485 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T15:04:49.485 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T15:04:49.485 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:18.2.2-0.el9.x86_64 29/32 2024-04-22T15:04:49.485 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T15:04:49.485 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:18.2.2-0.el9.x86_64 31/32 2024-04-22T15:04:49.769 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): libcephsqlite-18.2.2-0.el9.x86_64.rpm 1.0 MB/s | 165 kB 00:00 2024-04-22T15:04:50.213 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): ceph-test-18.2.2-0.el9.x86_64.rpm 66 MB/s | 40 MB 00:00 2024-04-22T15:04:50.242 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:04:50.242 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:50.242 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-04-22T15:04:50.242 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:18.2.2-0.el9.x86_64 librbd1-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.242 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:04:50.242 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T15:04:50.242 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.242 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.243 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:50.244 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:04:50.482 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 89 kB/s | 64 kB 00:00 2024-04-22T15:04:50.525 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-test 2024-04-22T15:04:50.582 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): socat-1.7.4.1-5.el9.x86_64.rpm 313 kB/s | 305 kB 00:00 2024-04-22T15:04:50.584 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:04:50.585 INFO:teuthology.orchestra.run.smithi196.stdout:Total 33 MB/s | 40 MB 00:01 2024-04-22T15:04:50.614 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:04:50.631 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:04:50.631 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:04:50.956 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:04:50.956 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:04:51.095 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:25 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:04:51.215 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:04:51.216 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:04:51.216 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T15:04:51.216 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:04:51.216 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:04:51.216 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:18.2.2-0.el9 ceph 40 M 2024-04-22T15:04:51.216 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-22T15:04:51.216 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:18.2.2-0.el9 ceph 165 k 2024-04-22T15:04:51.216 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T15:04:51.216 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T15:04:51.217 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:51.217 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:04:51.217 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:04:51.217 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-04-22T15:04:51.217 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:51.217 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 40 M 2024-04-22T15:04:51.218 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 166 M 2024-04-22T15:04:51.218 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:04:51.472 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:04:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T15:04:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T15:04:51.599 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): libcephsqlite-18.2.2-0.el9.x86_64.rpm 1.5 MB/s | 165 kB 00:00 2024-04-22T15:04:51.624 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:18.2.2-0.el9.x86_64 3/4 2024-04-22T15:04:51.700 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 305 kB 00:00 2024-04-22T15:04:51.784 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 345 kB/s | 64 kB 00:00 2024-04-22T15:04:52.244 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): ceph-test-18.2.2-0.el9.x86_64.rpm 53 MB/s | 40 MB 00:00 2024-04-22T15:04:52.246 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:04:52.247 INFO:teuthology.orchestra.run.smithi096.stdout:Total 39 MB/s | 40 MB 00:01 2024-04-22T15:04:52.271 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:04:52.290 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:04:52.290 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:04:52.616 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:04:52.616 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:04:53.141 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:04:53.221 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T15:04:53.268 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T15:04:53.296 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:18.2.2-0.el9.x86_64 3/4 2024-04-22T15:04:55.299 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:18.2.2-0.el9.x86_64 3/4 2024-04-22T15:04:55.370 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:18.2.2-0.el9.x86_64 4/4 2024-04-22T15:04:56.991 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:18.2.2-0.el9.x86_64 3/4 2024-04-22T15:04:57.067 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:18.2.2-0.el9.x86_64 4/4 2024-04-22T15:04:57.582 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:18.2.2-0.el9.x86_64 4/4 2024-04-22T15:04:57.582 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:18.2.2-0.el9.x86_64 1/4 2024-04-22T15:04:57.582 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:18.2.2-0.el9.x86_64 2/4 2024-04-22T15:04:57.582 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T15:04:57.854 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T15:04:57.854 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:57.855 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:04:57.855 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:18.2.2-0.el9.x86_64 libcephsqlite-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:57.855 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T15:04:57.855 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:57.855 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:04:58.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-04-22T15:04:58.643 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:34 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:04:58.763 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:18.2.2-0.el9 ceph 6.4 k 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:18.2.2-0.el9 ceph 2.1 M 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:18.2.2-0.el9 ceph 1.4 M 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:18.2.2-0.el9 ceph-noarch 242 k 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:18.2.2-0.el9 ceph 4.5 M 2024-04-22T15:04:58.766 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:18.2.2-0.el9 ceph 18 M 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T15:04:58.767 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T15:04:58.768 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T15:04:58.769 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T15:04:58.769 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:58.769 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:04:58.769 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:04:58.769 INFO:teuthology.orchestra.run.smithi196.stdout:Install 33 Packages 2024-04-22T15:04:58.769 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:04:58.770 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 30 M 2024-04-22T15:04:58.770 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 110 M 2024-04-22T15:04:58.770 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:04:59.184 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-test-2:18.2.2-0.el9.x86_64 4/4 2024-04-22T15:04:59.184 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:18.2.2-0.el9.x86_64 1/4 2024-04-22T15:04:59.184 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:18.2.2-0.el9.x86_64 2/4 2024-04-22T15:04:59.185 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T15:04:59.447 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T15:04:59.447 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:59.447 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:04:59.448 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:18.2.2-0.el9.x86_64 libcephsqlite-2:18.2.2-0.el9.x86_64 2024-04-22T15:04:59.448 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T15:04:59.448 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:04:59.448 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:04:59.647 INFO:teuthology.orchestra.run.smithi196.stdout:(1/33): ceph-18.2.2-0.el9.x86_64.rpm 57 kB/s | 6.4 kB 00:00 2024-04-22T15:04:59.707 INFO:teuthology.orchestra.run.smithi196.stdout:(2/33): ceph-mgr-18.2.2-0.el9.x86_64.rpm 8.5 MB/s | 1.4 MB 00:00 2024-04-22T15:04:59.715 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph 2024-04-22T15:04:59.760 INFO:teuthology.orchestra.run.smithi196.stdout:(3/33): ceph-mds-18.2.2-0.el9.x86_64.rpm 9.5 MB/s | 2.1 MB 00:00 2024-04-22T15:04:59.831 INFO:teuthology.orchestra.run.smithi196.stdout:(4/33): ceph-mon-18.2.2-0.el9.x86_64.rpm 24 MB/s | 4.5 MB 00:00 2024-04-22T15:04:59.856 INFO:teuthology.orchestra.run.smithi196.stdout:(5/33): ceph-mgr-modules-core-18.2.2-0.el9.noar 2.4 MB/s | 242 kB 00:00 2024-04-22T15:05:00.081 INFO:teuthology.orchestra.run.smithi196.stdout:(6/33): ceph-osd-18.2.2-0.el9.x86_64.rpm 47 MB/s | 18 MB 00:00 2024-04-22T15:05:00.123 INFO:teuthology.orchestra.run.smithi196.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 867 kB/s | 253 kB 00:00 2024-04-22T15:05:00.157 INFO:teuthology.orchestra.run.smithi196.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-22T15:05:00.190 INFO:teuthology.orchestra.run.smithi196.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 3.7 MB/s | 1.2 MB 00:00 2024-04-22T15:05:00.222 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:34 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:00.283 INFO:teuthology.orchestra.run.smithi196.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 851 kB/s | 135 kB 00:00 2024-04-22T15:05:00.308 INFO:teuthology.orchestra.run.smithi196.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 829 kB/s | 125 kB 00:00 2024-04-22T15:05:00.348 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:00.355 INFO:teuthology.orchestra.run.smithi196.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-04-22T15:05:00.355 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:00.355 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-22T15:05:00.355 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:00.355 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:18.2.2-0.el9 ceph 6.4 k 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:18.2.2-0.el9 ceph 2.1 M 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:18.2.2-0.el9 ceph 1.4 M 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:18.2.2-0.el9 ceph-noarch 242 k 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:18.2.2-0.el9 ceph 4.5 M 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:18.2.2-0.el9 ceph 18 M 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T15:05:00.356 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T15:05:00.357 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout:Install 33 Packages 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 30 M 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 110 M 2024-04-22T15:05:00.358 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:00.933 INFO:teuthology.orchestra.run.smithi196.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 56 kB/s | 35 kB 00:00 2024-04-22T15:05:00.992 INFO:teuthology.orchestra.run.smithi196.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 79 kB/s | 51 kB 00:00 2024-04-22T15:05:01.134 INFO:teuthology.orchestra.run.smithi196.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 202 kB/s | 172 kB 00:00 2024-04-22T15:05:01.220 INFO:teuthology.orchestra.run.smithi096.stdout:(1/33): ceph-18.2.2-0.el9.x86_64.rpm 60 kB/s | 6.4 kB 00:00 2024-04-22T15:05:01.259 INFO:teuthology.orchestra.run.smithi196.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-04-22T15:05:01.259 INFO:teuthology.orchestra.run.smithi096.stdout:(2/33): ceph-mgr-18.2.2-0.el9.x86_64.rpm 9.7 MB/s | 1.4 MB 00:00 2024-04-22T15:05:01.310 INFO:teuthology.orchestra.run.smithi096.stdout:(3/33): ceph-mds-18.2.2-0.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-04-22T15:05:01.344 INFO:teuthology.orchestra.run.smithi096.stdout:(4/33): ceph-mgr-modules-core-18.2.2-0.el9.noar 7.1 MB/s | 242 kB 00:00 2024-04-22T15:05:01.359 INFO:teuthology.orchestra.run.smithi196.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-04-22T15:05:01.428 INFO:teuthology.orchestra.run.smithi096.stdout:(5/33): ceph-mon-18.2.2-0.el9.x86_64.rpm 21 MB/s | 4.5 MB 00:00 2024-04-22T15:05:01.437 INFO:teuthology.orchestra.run.smithi196.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 138 kB/s | 11 kB 00:00 2024-04-22T15:05:01.493 INFO:teuthology.orchestra.run.smithi196.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 316 kB/s | 18 kB 00:00 2024-04-22T15:05:01.518 INFO:teuthology.orchestra.run.smithi196.stdout:(20/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 83 kB/s | 43 kB 00:00 2024-04-22T15:05:01.543 INFO:teuthology.orchestra.run.smithi196.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-04-22T15:05:01.586 INFO:teuthology.orchestra.run.smithi196.stdout:(22/33): python3-toml-0.10.2-6.el9.noarch.rpm 64 kB/s | 42 kB 00:00 2024-04-22T15:05:01.610 INFO:teuthology.orchestra.run.smithi196.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-04-22T15:05:01.669 INFO:teuthology.orchestra.run.smithi096.stdout:(6/33): ceph-osd-18.2.2-0.el9.x86_64.rpm 43 MB/s | 18 MB 00:00 2024-04-22T15:05:01.736 INFO:teuthology.orchestra.run.smithi196.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 632 kB/s | 79 kB 00:00 2024-04-22T15:05:01.761 INFO:teuthology.orchestra.run.smithi196.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 264 kB/s | 46 kB 00:00 2024-04-22T15:05:01.803 INFO:teuthology.orchestra.run.smithi196.stdout:(26/33): python3-jaraco-functools-3.5.0-2.el9.n 68 kB/s | 19 kB 00:00 2024-04-22T15:05:01.845 INFO:teuthology.orchestra.run.smithi196.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-04-22T15:05:01.878 INFO:teuthology.orchestra.run.smithi196.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-04-22T15:05:01.904 INFO:teuthology.orchestra.run.smithi196.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 892 kB/s | 90 kB 00:00 2024-04-22T15:05:01.921 INFO:teuthology.orchestra.run.smithi196.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 472 kB/s | 36 kB 00:00 2024-04-22T15:05:01.954 INFO:teuthology.orchestra.run.smithi196.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 599 kB/s | 20 kB 00:00 2024-04-22T15:05:01.988 INFO:teuthology.orchestra.run.smithi196.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-22T15:05:02.055 INFO:teuthology.orchestra.run.smithi196.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-04-22T15:05:02.057 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:02.057 INFO:teuthology.orchestra.run.smithi196.stdout:Total 9.1 MB/s | 30 MB 00:03 2024-04-22T15:05:02.173 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:02.212 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:02.212 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:02.436 INFO:teuthology.orchestra.run.smithi096.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 139 kB/s | 106 kB 00:00 2024-04-22T15:05:02.478 INFO:teuthology.orchestra.run.smithi096.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 223 kB/s | 253 kB 00:01 2024-04-22T15:05:02.499 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:02.505 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:03.212 INFO:teuthology.orchestra.run.smithi096.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 174 kB/s | 135 kB 00:00 2024-04-22T15:05:03.237 INFO:teuthology.orchestra.run.smithi096.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 165 kB/s | 125 kB 00:00 2024-04-22T15:05:03.338 INFO:teuthology.orchestra.run.smithi096.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-22T15:05:03.394 INFO:teuthology.orchestra.run.smithi096.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 695 kB/s | 35 kB 00:00 2024-04-22T15:05:03.446 INFO:teuthology.orchestra.run.smithi096.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 880 kB/s | 51 kB 00:00 2024-04-22T15:05:03.546 INFO:teuthology.orchestra.run.smithi096.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-04-22T15:05:03.680 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:03.741 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T15:05:03.764 INFO:teuthology.orchestra.run.smithi096.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 201 kB/s | 43 kB 00:00 2024-04-22T15:05:03.781 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T15:05:03.814 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T15:05:03.830 INFO:teuthology.orchestra.run.smithi096.stdout:(16/33): python3-cryptography-36.0.1-4.el9.x86_ 523 kB/s | 1.2 MB 00:02 2024-04-22T15:05:03.908 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T15:05:03.980 INFO:teuthology.orchestra.run.smithi096.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 792 kB/s | 172 kB 00:00 2024-04-22T15:05:03.993 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T15:05:04.033 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T15:05:04.055 INFO:teuthology.orchestra.run.smithi096.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-04-22T15:05:04.068 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T15:05:04.098 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T15:05:04.114 INFO:teuthology.orchestra.run.smithi096.stdout:(19/33): python3-urllib3-1.26.5-5.el9.noarch.rp 238 kB/s | 215 kB 00:00 2024-04-22T15:05:04.124 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T15:05:04.158 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T15:05:04.198 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T15:05:04.206 INFO:teuthology.orchestra.run.smithi096.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 117 kB/s | 18 kB 00:00 2024-04-22T15:05:04.241 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T15:05:04.276 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T15:05:04.306 INFO:teuthology.orchestra.run.smithi096.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-04-22T15:05:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T15:05:04.331 INFO:teuthology.orchestra.run.smithi096.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 107 kB/s | 23 kB 00:00 2024-04-22T15:05:04.357 INFO:teuthology.orchestra.run.smithi096.stdout:(23/33): python3-cherrypy-18.6.1-2.el9.noarch.r 681 kB/s | 358 kB 00:00 2024-04-22T15:05:04.382 INFO:teuthology.orchestra.run.smithi096.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-04-22T15:05:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T15:05:04.441 INFO:teuthology.orchestra.run.smithi096.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 943 kB/s | 79 kB 00:00 2024-04-22T15:05:04.457 INFO:teuthology.orchestra.run.smithi096.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 368 kB/s | 46 kB 00:00 2024-04-22T15:05:04.508 INFO:teuthology.orchestra.run.smithi096.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-04-22T15:05:04.525 INFO:teuthology.orchestra.run.smithi096.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-04-22T15:05:04.543 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T15:05:04.583 INFO:teuthology.orchestra.run.smithi096.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 717 kB/s | 90 kB 00:00 2024-04-22T15:05:04.609 INFO:teuthology.orchestra.run.smithi096.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-04-22T15:05:04.622 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T15:05:04.675 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T15:05:04.684 INFO:teuthology.orchestra.run.smithi096.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-04-22T15:05:04.709 INFO:teuthology.orchestra.run.smithi096.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-04-22T15:05:04.713 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T15:05:04.774 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T15:05:04.786 INFO:teuthology.orchestra.run.smithi096.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.1 MB/s | 427 kB 00:00 2024-04-22T15:05:04.787 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:04.787 INFO:teuthology.orchestra.run.smithi096.stdout:Total 6.8 MB/s | 30 MB 00:04 2024-04-22T15:05:04.826 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T15:05:04.902 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:04.941 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:04.941 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:05.015 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T15:05:05.067 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T15:05:05.146 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T15:05:05.225 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:05.225 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:05.234 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T15:05:05.274 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T15:05:05.391 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T15:05:05.551 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-0.el9.noarch 28/33 2024-04-22T15:05:05.585 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:18.2.2-0.el9.x86_64 29/33 2024-04-22T15:05:05.609 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:18.2.2-0.el9.x86_64 29/33 2024-04-22T15:05:05.609 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:05.609 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T15:05:05.609 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T15:05:05.609 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T15:05:05.609 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:06.417 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:06.479 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T15:05:06.503 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T15:05:06.536 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T15:05:06.630 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T15:05:06.716 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T15:05:06.764 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T15:05:06.798 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T15:05:06.828 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T15:05:06.880 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T15:05:06.905 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T15:05:06.945 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T15:05:06.980 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T15:05:07.014 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T15:05:07.052 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T15:05:07.135 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T15:05:07.153 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:18.2.2-0.el9.x86_64 30/33 2024-04-22T15:05:07.180 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:18.2.2-0.el9.x86_64 30/33 2024-04-22T15:05:07.180 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:07.180 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T15:05:07.180 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T15:05:07.180 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T15:05:07.180 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:07.283 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T15:05:07.361 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T15:05:07.414 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T15:05:07.452 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T15:05:07.509 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T15:05:07.573 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T15:05:07.629 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:18.2.2-0.el9.x86_64 31/33 2024-04-22T15:05:07.652 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:18.2.2-0.el9.x86_64 31/33 2024-04-22T15:05:07.652 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:07.653 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T15:05:07.653 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T15:05:07.653 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T15:05:07.653 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:07.769 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T15:05:07.823 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T15:05:07.854 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:18.2.2-0.el9.x86_64 32/33 2024-04-22T15:05:07.878 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:18.2.2-0.el9.x86_64 32/33 2024-04-22T15:05:07.878 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:07.878 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T15:05:07.879 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T15:05:07.879 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T15:05:07.879 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:07.893 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T15:05:07.942 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:18.2.2-0.el9.x86_64 33/33 2024-04-22T15:05:07.982 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T15:05:08.038 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T15:05:08.155 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T15:05:08.312 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-0.el9.noarch 28/33 2024-04-22T15:05:08.340 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:18.2.2-0.el9.x86_64 29/33 2024-04-22T15:05:08.366 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:18.2.2-0.el9.x86_64 29/33 2024-04-22T15:05:08.366 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:08.366 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T15:05:08.366 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T15:05:08.366 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T15:05:08.366 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:09.500 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:18.2.2-0.el9.x86_64 33/33 2024-04-22T15:05:09.500 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:18.2.2-0.el9.x86_64 1/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:18.2.2-0.el9.x86_64 2/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:18.2.2-0.el9.x86_64 3/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:18.2.2-0.el9.x86_64 4/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:18.2.2-0.el9.x86_64 5/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-0.el9.noarch 6/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T15:05:09.501 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T15:05:09.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T15:05:09.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T15:05:09.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T15:05:09.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T15:05:09.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T15:05:09.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T15:05:09.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T15:05:09.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T15:05:09.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T15:05:09.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T15:05:09.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T15:05:09.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T15:05:09.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T15:05:09.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T15:05:09.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T15:05:09.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T15:05:09.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T15:05:09.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T15:05:09.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T15:05:09.866 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:18.2.2-0.el9.x86_64 30/33 2024-04-22T15:05:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:18.2.2-0.el9.x86_64 30/33 2024-04-22T15:05:09.893 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:09.894 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T15:05:09.894 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T15:05:09.894 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T15:05:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:10.333 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:18.2.2-0.el9.x86_64 31/33 2024-04-22T15:05:10.356 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:18.2.2-0.el9.x86_64 31/33 2024-04-22T15:05:10.357 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:10.357 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T15:05:10.357 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T15:05:10.357 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T15:05:10.357 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:18.2.2-0.el9.noarch 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T15:05:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T15:05:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T15:05:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T15:05:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T15:05:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T15:05:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T15:05:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T15:05:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T15:05:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T15:05:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T15:05:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T15:05:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T15:05:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T15:05:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:10.507 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:10.567 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:18.2.2-0.el9.x86_64 32/33 2024-04-22T15:05:10.590 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:18.2.2-0.el9.x86_64 32/33 2024-04-22T15:05:10.590 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:10.590 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T15:05:10.590 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T15:05:10.590 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T15:05:10.590 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:10.654 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:18.2.2-0.el9.x86_64 33/33 2024-04-22T15:05:10.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-04-22T15:05:11.387 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:47 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:11.466 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:11.506 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:11.507 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T15:05:11.507 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:11.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-04-22T15:05:12.080 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:48 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:12.200 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:18.2.2-0.el9 ceph-noarch 221 k 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:12.201 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 221 k 2024-04-22T15:05:12.202 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 213 k 2024-04-22T15:05:12.202 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:12.260 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-18.2.2-0.el9.noarch.rpm 3.7 MB/s | 221 kB 00:00 2024-04-22T15:05:12.261 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:12.261 INFO:teuthology.orchestra.run.smithi196.stdout:Total 3.6 MB/s | 221 kB 00:00 2024-04-22T15:05:12.261 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:12.266 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:12.266 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:12.269 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:12.269 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:12.339 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-2:18.2.2-0.el9.x86_64 33/33 2024-04-22T15:05:12.339 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:18.2.2-0.el9.x86_64 1/33 2024-04-22T15:05:12.339 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:18.2.2-0.el9.x86_64 2/33 2024-04-22T15:05:12.339 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:18.2.2-0.el9.x86_64 3/33 2024-04-22T15:05:12.339 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:18.2.2-0.el9.x86_64 4/33 2024-04-22T15:05:12.339 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:18.2.2-0.el9.x86_64 5/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-0.el9.noarch 6/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T15:05:12.340 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T15:05:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T15:05:12.342 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:18.2.2-0.el9.noarch 1/1 2024-04-22T15:05:12.814 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:18.2.2-0.el9.noarch 1/1 2024-04-22T15:05:13.140 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:18.2.2-0.el9.noarch 1/1 2024-04-22T15:05:13.141 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T15:05:13.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:13.141 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:13.141 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:13.141 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:13.141 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:13.141 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:18.2.2-0.el9.noarch 2024-04-22T15:05:13.141 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T15:05:13.142 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T15:05:13.143 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T15:05:13.144 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T15:05:13.144 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:13.144 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:13.393 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:18.2.2-0.el9.noarch 1/1 2024-04-22T15:05:13.393 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:13.393 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:13.394 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:18.2.2-0.el9.noarch 2024-04-22T15:05:13.394 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:13.394 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:13.460 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-base 2024-04-22T15:05:13.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T15:05:13.982 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:47 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:14.060 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-base-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:14.101 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:14.101 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T15:05:14.102 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:14.136 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:50 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:14.167 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install cephadm 2024-04-22T15:05:14.256 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:14.257 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:14.257 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-04-22T15:05:14.257 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:14.257 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:14.257 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-0.el9 ceph 142 k 2024-04-22T15:05:14.257 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:14.257 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:14.257 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:14.257 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-04-22T15:05:14.258 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:14.258 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 142 k 2024-04-22T15:05:14.258 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 395 k 2024-04-22T15:05:14.258 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:14.332 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-18.2.2-0.el9.x86_64 1.9 MB/s | 142 kB 00:00 2024-04-22T15:05:14.333 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:14.333 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.8 MB/s | 142 kB 00:00 2024-04-22T15:05:14.333 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:14.340 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:14.340 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:14.397 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:14.397 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:14.534 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:14.617 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:14.641 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:14.641 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:14.641 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T15:05:14.641 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:14.670 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:48 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:14.789 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:14.789 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:14.789 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:14.789 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:18.2.2-0.el9 ceph-noarch 221 k 2024-04-22T15:05:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:14.790 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:14.790 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:14.790 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-04-22T15:05:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:14.790 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 221 k 2024-04-22T15:05:14.791 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 213 k 2024-04-22T15:05:14.791 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:14.873 INFO:teuthology.orchestra.run.smithi096.stdout:cephadm-18.2.2-0.el9.noarch.rpm 2.6 MB/s | 221 kB 00:00 2024-04-22T15:05:14.874 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:14.874 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.6 MB/s | 221 kB 00:00 2024-04-22T15:05:14.874 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:14.879 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:14.879 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:14.882 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:14.882 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:14.947 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:18.2.2-0.el9.noarch 1/1 2024-04-22T15:05:15.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:15.460 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:15.460 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:15.460 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:15.460 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:15.460 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:15.483 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:18.2.2-0.el9.noarch 1/1 2024-04-22T15:05:15.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-04-22T15:05:15.782 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:18.2.2-0.el9.noarch 1/1 2024-04-22T15:05:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:18.2.2-0.el9.noarch 1/1 2024-04-22T15:05:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:16.049 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:18.2.2-0.el9.noarch 2024-04-22T15:05:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:16.049 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:16.207 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:52 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:16.237 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T15:05:16.287 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:16.329 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:16.330 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T15:05:16.330 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:16.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T15:05:16.750 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:50 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:16.869 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:16.869 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:16.869 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-04-22T15:05:16.869 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-0.el9 ceph 142 k 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 142 k 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 395 k 2024-04-22T15:05:16.870 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:16.900 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:52 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:16.936 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-immutable-object-cache-18.2.2-0.el9.x86_64 2.1 MB/s | 142 kB 00:00 2024-04-22T15:05:16.937 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:16.937 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.1 MB/s | 142 kB 00:00 2024-04-22T15:05:16.938 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:16.946 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:16.946 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:17.005 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:17.005 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:17.030 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:17.031 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:17.031 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-04-22T15:05:17.031 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:17.031 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:17.031 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:18.2.2-0.el9 ceph-noarch 1.8 M 2024-04-22T15:05:17.031 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-22T15:05:17.031 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:18.2.2-0.el9 ceph-noarch 23 k 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:18.2.2-0.el9 ceph-noarch 15 k 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout:Install 7 Packages 2024-04-22T15:05:17.032 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:17.033 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.1 M 2024-04-22T15:05:17.033 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 8.5 M 2024-04-22T15:05:17.033 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:17.144 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:17.213 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:17.240 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:17.240 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:17.241 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T15:05:17.241 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:17.304 INFO:teuthology.orchestra.run.smithi196.stdout:(1/7): ceph-grafana-dashboards-18.2.2-0.el9.noa 191 kB/s | 23 kB 00:00 2024-04-22T15:05:17.329 INFO:teuthology.orchestra.run.smithi196.stdout:(2/7): ceph-prometheus-alerts-18.2.2-0.el9.noar 100 kB/s | 15 kB 00:00 2024-04-22T15:05:17.378 INFO:teuthology.orchestra.run.smithi196.stdout:(3/7): ceph-mgr-dashboard-18.2.2-0.el9.noarch.r 9.0 MB/s | 1.8 MB 00:00 2024-04-22T15:05:17.504 INFO:teuthology.orchestra.run.smithi196.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 45 kB/s | 9.0 kB 00:00 2024-04-22T15:05:17.604 INFO:teuthology.orchestra.run.smithi196.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 136 kB/s | 31 kB 00:00 2024-04-22T15:05:17.629 INFO:teuthology.orchestra.run.smithi196.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 136 kB/s | 41 kB 00:00 2024-04-22T15:05:17.779 INFO:teuthology.orchestra.run.smithi196.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 684 kB/s | 188 kB 00:00 2024-04-22T15:05:17.781 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:17.781 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.7 MB/s | 2.1 MB 00:00 2024-04-22T15:05:17.782 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:17.790 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:17.790 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:17.841 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:17.843 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:18.093 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:18.093 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:18.094 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:18.094 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:18.094 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:18.094 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:18.142 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:18.214 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-22T15:05:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-22T15:05:18.302 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-22T15:05:18.334 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-22T15:05:18.363 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr 2024-04-22T15:05:18.364 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-0.el9.noarch 5/7 2024-04-22T15:05:18.600 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-0.el9.noarch 6/7 2024-04-22T15:05:18.633 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-0.el9.noarch 7/7 2024-04-22T15:05:18.880 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:52 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:18.959 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-mgr-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:19.002 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:19.003 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T15:05:19.003 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:19.062 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T15:05:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-0.el9.noarch 7/7 2024-04-22T15:05:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-0.el9.noarch 1/7 2024-04-22T15:05:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-0.el9.noarch 2/7 2024-04-22T15:05:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-0.el9.noarch 3/7 2024-04-22T15:05:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-22T15:05:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-22T15:05:19.133 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-22T15:05:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-22T15:05:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:19.384 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:18.2.2-0.el9.noarch 2024-04-22T15:05:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:18.2.2-0.el9.noarch 2024-04-22T15:05:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:18.2.2-0.el9.noarch 2024-04-22T15:05:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-22T15:05:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-22T15:05:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T15:05:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T15:05:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:19.385 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:19.567 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:53 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:19.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T15:05:19.685 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:19.686 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:19.686 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-22T15:05:19.686 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:19.686 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:19.686 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:18.2.2-0.el9 ceph-noarch 1.8 M 2024-04-22T15:05:19.687 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-22T15:05:19.687 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:18.2.2-0.el9 ceph-noarch 23 k 2024-04-22T15:05:19.687 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:18.2.2-0.el9 ceph-noarch 15 k 2024-04-22T15:05:19.687 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-22T15:05:19.687 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T15:05:19.687 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T15:05:19.687 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-04-22T15:05:19.687 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-22T15:05:19.688 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:19.688 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:19.688 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:19.688 INFO:teuthology.orchestra.run.smithi096.stdout:Install 7 Packages 2024-04-22T15:05:19.688 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:19.689 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 2.1 M 2024-04-22T15:05:19.689 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 8.5 M 2024-04-22T15:05:19.689 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:19.949 INFO:teuthology.orchestra.run.smithi096.stdout:(1/7): ceph-grafana-dashboards-18.2.2-0.el9.noa 196 kB/s | 23 kB 00:00 2024-04-22T15:05:19.974 INFO:teuthology.orchestra.run.smithi096.stdout:(2/7): ceph-prometheus-alerts-18.2.2-0.el9.noar 102 kB/s | 15 kB 00:00 2024-04-22T15:05:20.024 INFO:teuthology.orchestra.run.smithi096.stdout:(3/7): ceph-mgr-dashboard-18.2.2-0.el9.noarch.r 9.1 MB/s | 1.8 MB 00:00 2024-04-22T15:05:20.087 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:56 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:20.116 INFO:teuthology.orchestra.run.smithi096.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 54 kB/s | 9.0 kB 00:00 2024-04-22T15:05:20.191 INFO:teuthology.orchestra.run.smithi096.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 184 kB/s | 31 kB 00:00 2024-04-22T15:05:20.207 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:20.208 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-0.el9 ceph-noarch 7.4 M 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T15:05:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T15:05:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T15:05:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T15:05:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T15:05:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:20.210 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:20.210 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:20.210 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-04-22T15:05:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:20.211 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 38 M 2024-04-22T15:05:20.211 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 209 M 2024-04-22T15:05:20.211 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:20.216 INFO:teuthology.orchestra.run.smithi096.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 169 kB/s | 41 kB 00:00 2024-04-22T15:05:20.384 INFO:teuthology.orchestra.run.smithi096.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 706 kB/s | 188 kB 00:00 2024-04-22T15:05:20.385 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:20.385 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.9 MB/s | 2.1 MB 00:00 2024-04-22T15:05:20.386 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:20.393 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:20.394 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:20.443 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:20.443 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:20.726 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:20.777 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-22T15:05:20.785 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.2-0. 45 MB/s | 7.4 MB 00:00 2024-04-22T15:05:20.802 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-22T15:05:20.856 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-22T15:05:20.889 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-22T15:05:20.917 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-0.el9.noarch 5/7 2024-04-22T15:05:20.918 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 384 kB/s | 33 kB 00:00 2024-04-22T15:05:20.969 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 560 kB/s | 195 kB 00:00 2024-04-22T15:05:21.011 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-04-22T15:05:21.103 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 136 kB/s | 18 kB 00:00 2024-04-22T15:05:21.128 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 301 kB/s | 35 kB 00:00 2024-04-22T15:05:21.151 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-0.el9.noarch 6/7 2024-04-22T15:05:21.199 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-0.el9.noarch 7/7 2024-04-22T15:05:21.204 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-04-22T15:05:21.246 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-04-22T15:05:21.337 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-04-22T15:05:21.479 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 5.1 MB 00:00 2024-04-22T15:05:21.714 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-0.el9.noarch 7/7 2024-04-22T15:05:21.714 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-0.el9.noarch 1/7 2024-04-22T15:05:21.714 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-0.el9.noarch 2/7 2024-04-22T15:05:21.714 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-0.el9.noarch 3/7 2024-04-22T15:05:21.714 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-22T15:05:21.714 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-22T15:05:21.715 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:18.2.2-0.el9.noarch 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:18.2.2-0.el9.noarch 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:18.2.2-0.el9.noarch 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T15:05:21.958 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T15:05:21.959 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:21.959 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:22.105 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-04-22T15:05:22.156 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T15:05:22.279 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 4.0 MB/s | 4.7 MB 00:01 2024-04-22T15:05:22.282 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:22.283 INFO:teuthology.orchestra.run.smithi196.stdout:Total 18 MB/s | 38 MB 00:02 2024-04-22T15:05:22.632 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:22.655 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:56 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:22.658 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:22.658 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:22.775 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:22.776 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-22T15:05:22.776 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:22.776 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-0.el9 ceph-noarch 7.4 M 2024-04-22T15:05:22.776 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T15:05:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T15:05:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:22.778 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:22.778 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:22.778 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-04-22T15:05:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:22.778 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 38 M 2024-04-22T15:05:22.779 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 209 M 2024-04-22T15:05:22.779 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:23.062 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:23.062 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:23.339 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.2-0. 44 MB/s | 7.4 MB 00:00 2024-04-22T15:05:23.465 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 262 kB/s | 33 kB 00:00 2024-04-22T15:05:23.706 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-04-22T15:05:23.757 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 366 kB/s | 18 kB 00:00 2024-04-22T15:05:23.807 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-22T15:05:23.811 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:23.892 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T15:05:23.928 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T15:05:24.027 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T15:05:24.070 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T15:05:24.116 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-04-22T15:05:24.208 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): libquadmath-11.4.1-3.el9.x86_64.rpm 189 kB/s | 195 kB 00:01 2024-04-22T15:05:24.233 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-04-22T15:05:24.248 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T15:05:24.278 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T15:05:24.355 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T15:05:24.383 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 250 kB 00:00 2024-04-22T15:05:24.600 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-04-22T15:05:24.998 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T15:05:25.009 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 437 kB/s | 803 kB 00:01 2024-04-22T15:05:25.117 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T15:05:25.467 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-04-22T15:05:25.470 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:25.470 INFO:teuthology.orchestra.run.smithi096.stdout:Total 14 MB/s | 38 MB 00:02 2024-04-22T15:05:25.819 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:25.845 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:25.845 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:26.251 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:26.251 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:26.341 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T15:05:27.000 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:27.063 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T15:05:27.107 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T15:05:27.206 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T15:05:27.241 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T15:05:27.370 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T15:05:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-0.el9.noarc 12/12 2024-04-22T15:05:27.420 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T15:05:27.457 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T15:05:27.534 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T15:05:28.170 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T15:05:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T15:05:29.502 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T15:05:29.905 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-0.el9.noarc 12/12 2024-04-22T15:05:29.905 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-0.el9.noarc 1/12 2024-04-22T15:05:29.905 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T15:05:29.905 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T15:05:29.905 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T15:05:29.905 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T15:05:29.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T15:05:29.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T15:05:29.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T15:05:29.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T15:05:29.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T15:05:29.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T15:05:30.251 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T15:05:30.251 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:30.251 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:30.251 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:18.2.2-0.el9.noarch 2024-04-22T15:05:30.251 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T15:05:30.251 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T15:05:30.251 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T15:05:30.251 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T15:05:30.251 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T15:05:30.252 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T15:05:30.252 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T15:05:30.252 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T15:05:30.252 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T15:05:30.252 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T15:05:30.252 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T15:05:30.252 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:30.252 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:30.541 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T15:05:30.574 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-0.el9.noarc 12/12 2024-04-22T15:05:30.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-04-22T15:05:31.117 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:07 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:31.238 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:31.239 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:31.239 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:18.2.2-0.el9 ceph-noarch 50 k 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T15:05:31.240 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T15:05:31.241 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T15:05:31.241 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T15:05:31.241 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:31.241 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:31.241 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:31.241 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-04-22T15:05:31.241 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:31.241 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.2 M 2024-04-22T15:05:31.242 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-04-22T15:05:31.242 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:31.609 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): ceph-mgr-rook-18.2.2-0.el9.noarch.rpm 759 kB/s | 50 kB 00:00 2024-04-22T15:05:31.643 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 257 kB/s | 26 kB 00:00 2024-04-22T15:05:31.668 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 153 kB/s | 19 kB 00:00 2024-04-22T15:05:31.778 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-04-22T15:05:31.802 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 997 kB/s | 159 kB 00:00 2024-04-22T15:05:31.827 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.7 MB/s | 279 kB 00:00 2024-04-22T15:05:31.894 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 458 kB/s | 54 kB 00:00 2024-04-22T15:05:32.103 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-04-22T15:05:32.130 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 99 kB/s | 32 kB 00:00 2024-04-22T15:05:32.320 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-04-22T15:05:32.420 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 385 kB/s | 203 kB 00:00 2024-04-22T15:05:32.487 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 537 kB/s | 90 kB 00:00 2024-04-22T15:05:32.671 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-04-22T15:05:32.671 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:32.671 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-04-22T15:05:32.732 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:32.747 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:32.747 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:32.831 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:32.831 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:33.263 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-0.el9.noarc 12/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-0.el9.noarc 1/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T15:05:33.264 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T15:05:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:33.559 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T15:05:33.613 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T15:05:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T15:05:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:33.619 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:18.2.2-0.el9.noarch 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T15:05:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:33.621 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:33.669 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T15:05:33.706 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T15:05:33.747 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T15:05:33.842 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T15:05:33.942 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T15:05:33.954 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-rook 2024-04-22T15:05:33.993 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T15:05:34.476 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:08 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:34.596 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:34.597 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:18.2.2-0.el9 ceph-noarch 50 k 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T15:05:34.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T15:05:34.599 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T15:05:34.599 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T15:05:34.599 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T15:05:34.599 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T15:05:34.599 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:34.599 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:34.599 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:34.599 INFO:teuthology.orchestra.run.smithi096.stdout:Install 13 Packages 2024-04-22T15:05:34.599 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:34.600 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 2.2 M 2024-04-22T15:05:34.600 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 26 M 2024-04-22T15:05:34.600 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:34.703 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T15:05:34.748 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T15:05:34.782 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T15:05:34.829 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T15:05:34.852 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:18.2.2-0.el9.noarch 13/13 2024-04-22T15:05:35.104 INFO:teuthology.orchestra.run.smithi096.stdout:(1/13): ceph-mgr-rook-18.2.2-0.el9.noarch.rpm 862 kB/s | 50 kB 00:00 2024-04-22T15:05:35.305 INFO:teuthology.orchestra.run.smithi096.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 74 kB/s | 19 kB 00:00 2024-04-22T15:05:35.330 INFO:teuthology.orchestra.run.smithi096.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 90 kB/s | 26 kB 00:00 2024-04-22T15:05:35.534 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-0.el9.noarch 13/13 2024-04-22T15:05:35.534 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:18.2.2-0.el9.noarch 1/13 2024-04-22T15:05:35.534 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T15:05:35.534 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T15:05:35.534 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T15:05:35.534 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T15:05:35.534 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T15:05:35.535 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T15:05:35.535 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T15:05:35.535 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T15:05:35.535 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T15:05:35.535 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T15:05:35.535 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T15:05:35.605 INFO:teuthology.orchestra.run.smithi096.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 444 kB/s | 222 kB 00:00 2024-04-22T15:05:35.647 INFO:teuthology.orchestra.run.smithi096.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 464 kB/s | 159 kB 00:00 2024-04-22T15:05:35.706 INFO:teuthology.orchestra.run.smithi096.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 536 kB/s | 54 kB 00:00 2024-04-22T15:05:35.748 INFO:teuthology.orchestra.run.smithi096.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 667 kB/s | 279 kB 00:00 2024-04-22T15:05:35.781 INFO:teuthology.orchestra.run.smithi096.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-04-22T15:05:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:18.2.2-0.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T15:05:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T15:05:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T15:05:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T15:05:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:35.844 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:35.857 INFO:teuthology.orchestra.run.smithi096.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 99 kB/s | 15 kB 00:00 2024-04-22T15:05:35.982 INFO:teuthology.orchestra.run.smithi096.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 866 kB/s | 203 kB 00:00 2024-04-22T15:05:36.007 INFO:teuthology.orchestra.run.smithi096.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-04-22T15:05:36.049 INFO:teuthology.orchestra.run.smithi096.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-04-22T15:05:36.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T15:05:36.074 INFO:teuthology.orchestra.run.smithi096.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 975 kB/s | 90 kB 00:00 2024-04-22T15:05:36.075 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:36.075 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-04-22T15:05:36.140 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:36.155 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:36.155 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:36.238 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:36.238 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:36.586 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:12 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:36.705 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:18.2.2-0.el9 ceph-noarch 132 k 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T15:05:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:36.707 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:36.707 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:36.707 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-04-22T15:05:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:36.707 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 7.0 M 2024-04-22T15:05:36.707 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-04-22T15:05:36.708 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:36.824 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:36.997 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T15:05:37.051 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T15:05:37.057 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): ceph-mgr-cephadm-18.2.2-0.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-04-22T15:05:37.106 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T15:05:37.143 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T15:05:37.184 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T15:05:37.278 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T15:05:37.395 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T15:05:37.444 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T15:05:37.474 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 500 kB/s | 248 kB 00:00 2024-04-22T15:05:37.783 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 756 kB/s | 548 kB 00:00 2024-04-22T15:05:37.908 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 408 kB/s | 51 kB 00:00 2024-04-22T15:05:37.950 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 121 kB/s | 58 kB 00:00 2024-04-22T15:05:38.159 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T15:05:38.208 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T15:05:38.242 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T15:05:38.289 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T15:05:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:18.2.2-0.el9.noarch 13/13 2024-04-22T15:05:38.825 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.2 MB/s | 6.0 MB 00:01 2024-04-22T15:05:38.828 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:38.828 INFO:teuthology.orchestra.run.smithi196.stdout:Total 3.3 MB/s | 7.0 MB 00:02 2024-04-22T15:05:38.901 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:38.913 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:38.913 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:39.008 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:39.009 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-0.el9.noarch 13/13 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:18.2.2-0.el9.noarch 1/13 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T15:05:39.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T15:05:39.055 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T15:05:39.055 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T15:05:39.055 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T15:05:39.312 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:39.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T15:05:39.366 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:18.2.2-0.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T15:05:39.367 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T15:05:39.368 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:39.368 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:39.489 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T15:05:39.544 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T15:05:39.614 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T15:05:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T15:05:40.123 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:14 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:40.154 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T15:05:40.215 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T15:05:40.239 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-0.el9.noarch 6/6 2024-04-22T15:05:40.243 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:40.244 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:40.244 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-22T15:05:40.244 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:40.244 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:40.244 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:18.2.2-0.el9 ceph-noarch 132 k 2024-04-22T15:05:40.244 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-22T15:05:40.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T15:05:40.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T15:05:40.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T15:05:40.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T15:05:40.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T15:05:40.245 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:40.245 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:40.245 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:40.245 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2024-04-22T15:05:40.245 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:40.246 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 7.0 M 2024-04-22T15:05:40.246 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 33 M 2024-04-22T15:05:40.246 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:40.660 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): ceph-mgr-cephadm-18.2.2-0.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-04-22T15:05:40.851 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 962 kB/s | 248 kB 00:00 2024-04-22T15:05:40.877 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.5 MB/s | 548 kB 00:00 2024-04-22T15:05:40.927 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-04-22T15:05:40.977 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 460 kB/s | 58 kB 00:00 2024-04-22T15:05:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-0.el9.noarch 6/6 2024-04-22T15:05:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-0.el9.noarch 1/6 2024-04-22T15:05:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T15:05:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T15:05:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T15:05:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T15:05:41.214 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:18.2.2-0.el9.noarch 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:41.215 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:41.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-04-22T15:05:41.602 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.9 MB/s | 6.0 MB 00:01 2024-04-22T15:05:41.602 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:41.603 INFO:teuthology.orchestra.run.smithi096.stdout:Total 5.1 MB/s | 7.0 MB 00:01 2024-04-22T15:05:41.673 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:41.684 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:41.685 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:41.779 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:41.782 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:41.965 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:17 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:42.078 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:42.083 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:18.2.2-0.el9 ceph 840 k 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:42.084 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:42.085 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-04-22T15:05:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:42.085 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 920 k 2024-04-22T15:05:42.085 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.7 M 2024-04-22T15:05:42.085 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:42.246 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T15:05:42.296 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T15:05:42.443 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): ceph-fuse-18.2.2-0.el9.x86_64.rpm 10 MB/s | 840 kB 00:00 2024-04-22T15:05:42.835 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T15:05:42.926 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T15:05:42.995 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T15:05:43.018 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-0.el9.noarch 6/6 2024-04-22T15:05:43.077 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 112 kB/s | 80 kB 00:00 2024-04-22T15:05:43.077 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:43.078 INFO:teuthology.orchestra.run.smithi196.stdout:Total 927 kB/s | 920 kB 00:00 2024-04-22T15:05:43.090 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:43.111 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:43.112 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:43.187 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:43.188 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:43.374 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:43.508 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T15:05:43.574 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:18.2.2-0.el9.x86_64 2/2 2024-04-22T15:05:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-0.el9.noarch 6/6 2024-04-22T15:05:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-0.el9.noarch 1/6 2024-04-22T15:05:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T15:05:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T15:05:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T15:05:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T15:05:43.973 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T15:05:43.973 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:43.973 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:43.973 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:18.2.2-0.el9.noarch 2024-04-22T15:05:43.973 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T15:05:43.973 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T15:05:43.973 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T15:05:43.974 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T15:05:43.974 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T15:05:43.974 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:43.974 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:44.204 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-fuse 2024-04-22T15:05:44.256 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:18.2.2-0.el9.x86_64 2/2 2024-04-22T15:05:44.256 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:18.2.2-0.el9.x86_64 1/2 2024-04-22T15:05:44.507 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T15:05:44.507 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:44.507 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:44.507 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:18.2.2-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T15:05:44.508 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:44.508 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:44.715 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:18 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:44.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-04-22T15:05:44.835 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:44.835 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:44.835 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:44.835 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:44.835 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:44.835 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:18.2.2-0.el9 ceph 840 k 2024-04-22T15:05:44.835 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-22T15:05:44.835 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T15:05:44.836 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:44.836 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:44.836 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:44.836 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2024-04-22T15:05:44.836 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:44.836 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 920 k 2024-04-22T15:05:44.837 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.7 M 2024-04-22T15:05:44.837 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:45.064 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): ceph-fuse-18.2.2-0.el9.x86_64.rpm 11 MB/s | 840 kB 00:00 2024-04-22T15:05:45.089 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 809 kB/s | 80 kB 00:00 2024-04-22T15:05:45.089 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:45.090 INFO:teuthology.orchestra.run.smithi096.stdout:Total 3.5 MB/s | 920 kB 00:00 2024-04-22T15:05:45.101 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:45.122 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:45.122 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:45.199 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:45.200 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:45.246 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:21 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:45.371 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:45.371 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:45.371 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:45.371 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:45.371 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:45.371 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:18.2.2-0.el9 ceph 126 k 2024-04-22T15:05:45.372 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:45.372 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:45.372 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:45.372 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-04-22T15:05:45.372 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:45.372 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 126 k 2024-04-22T15:05:45.372 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 456 k 2024-04-22T15:05:45.372 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:45.380 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:45.432 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-18.2.2-0.el9.x86_64.rpm 2.1 MB/s | 126 kB 00:00 2024-04-22T15:05:45.432 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:45.433 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.0 MB/s | 126 kB 00:00 2024-04-22T15:05:45.433 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:45.440 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:45.440 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:45.500 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:45.501 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:45.512 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T15:05:45.587 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:18.2.2-0.el9.x86_64 2/2 2024-04-22T15:05:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:45.752 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:46.202 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-fuse-2:18.2.2-0.el9.x86_64 2/2 2024-04-22T15:05:46.202 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:18.2.2-0.el9.x86_64 1/2 2024-04-22T15:05:46.351 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:46.351 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:46.351 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:46.351 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:46.351 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:46.351 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:46.455 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T15:05:46.455 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:46.455 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:46.455 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:18.2.2-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T15:05:46.455 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:46.455 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:46.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-04-22T15:05:46.725 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados-devel 2024-04-22T15:05:47.076 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:23 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:47.156 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:47.196 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:47.197 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T15:05:47.197 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:47.244 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:21 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:47.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-04-22T15:05:47.364 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:47.365 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:47.365 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:47.365 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:18.2.2-0.el9 ceph 126 k 2024-04-22T15:05:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:47.366 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:47.366 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:47.366 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-04-22T15:05:47.366 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:47.366 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 126 k 2024-04-22T15:05:47.366 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 456 k 2024-04-22T15:05:47.366 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:47.443 INFO:teuthology.orchestra.run.smithi096.stdout:librados-devel-18.2.2-0.el9.x86_64.rpm 1.6 MB/s | 126 kB 00:00 2024-04-22T15:05:47.444 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:47.444 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.6 MB/s | 126 kB 00:00 2024-04-22T15:05:47.444 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:47.449 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:47.449 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:47.515 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:47.515 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:47.797 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:23 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:47.797 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:47.917 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:18.2.2-0.el9 ceph 31 k 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:47.918 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 31 k 2024-04-22T15:05:47.919 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 139 k 2024-04-22T15:05:47.919 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:47.984 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-18.2.2-0.el9.x86_64.rpm 474 kB/s | 31 kB 00:00 2024-04-22T15:05:47.985 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:47.985 INFO:teuthology.orchestra.run.smithi196.stdout:Total 463 kB/s | 31 kB 00:00 2024-04-22T15:05:47.985 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:47.988 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:47.988 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:47.990 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:47.991 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:48.198 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:48.434 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:48.434 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:48.434 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:48.434 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:48.434 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:48.434 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:48.479 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:48.704 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs2 2024-04-22T15:05:48.711 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:48.711 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:48.712 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:48.712 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:48.712 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:48.712 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:48.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-04-22T15:05:49.239 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:23 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:49.317 INFO:teuthology.orchestra.run.smithi096.stdout:Package libcephfs2-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:49.358 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:49.359 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T15:05:49.359 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:49.388 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:25 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:49.467 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:49.471 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs-devel 2024-04-22T15:05:49.508 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:49.509 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T15:05:49.509 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:49.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-04-22T15:05:49.991 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:23 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:50.075 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:26 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:50.110 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:50.110 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:50.110 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:50.111 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:50.111 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:50.111 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:18.2.2-0.el9 ceph 31 k 2024-04-22T15:05:50.152 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:50.152 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:50.152 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:50.152 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-04-22T15:05:50.152 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:50.152 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 31 k 2024-04-22T15:05:50.152 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 139 k 2024-04-22T15:05:50.152 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:50.153 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:50.182 INFO:teuthology.orchestra.run.smithi096.stdout:libcephfs-devel-18.2.2-0.el9.x86_64.rpm 447 kB/s | 31 kB 00:00 2024-04-22T15:05:50.183 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:50.183 INFO:teuthology.orchestra.run.smithi096.stdout:Total 437 kB/s | 31 kB 00:00 2024-04-22T15:05:50.183 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:50.184 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:50.184 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:50.186 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:50.187 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:50.192 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:50.193 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T15:05:50.194 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:50.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-04-22T15:05:50.271 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:50.701 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:50.772 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:26 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:50.850 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:50.892 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:50.893 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T15:05:50.893 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:50.926 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:50.926 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:50.926 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:50.927 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:50.927 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:50.927 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:50.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-04-22T15:05:51.122 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados2 2024-04-22T15:05:51.465 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:27 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:51.544 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:51.588 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:51.588 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T15:05:51.589 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:51.623 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:25 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:51.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-04-22T15:05:51.701 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:51.742 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:51.743 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T15:05:51.743 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:51.820 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librbd1 2024-04-22T15:05:52.155 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:28 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:52.233 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:52.273 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:52.274 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T15:05:52.274 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:52.323 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:26 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:52.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-04-22T15:05:52.402 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:52.442 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:52.444 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T15:05:52.444 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:52.507 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rados 2024-04-22T15:05:52.847 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:28 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:52.926 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:52.966 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:52.967 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-04-22T15:05:52.967 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:53.017 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:27 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:53.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-04-22T15:05:53.096 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rados-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:53.136 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:53.137 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T15:05:53.137 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:53.204 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rgw 2024-04-22T15:05:53.545 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:29 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:53.661 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:53.662 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:53.662 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:53.662 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:53.662 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:53.662 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:18.2.2-0.el9 ceph 86 k 2024-04-22T15:05:53.663 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:53.663 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:53.663 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:53.663 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-04-22T15:05:53.663 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:53.663 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 86 k 2024-04-22T15:05:53.663 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 227 k 2024-04-22T15:05:53.663 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:53.717 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:27 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:53.728 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-18.2.2-0.el9.x86_64.rpm 1.3 MB/s | 86 kB 00:00 2024-04-22T15:05:53.729 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:53.729 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.3 MB/s | 86 kB 00:00 2024-04-22T15:05:53.729 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:53.736 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:53.736 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:53.796 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rgw-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:53.799 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:53.799 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:53.836 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:53.838 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T15:05:53.838 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:53.912 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-cephfs 2024-04-22T15:05:53.943 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:54.049 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:54.392 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:54.415 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:28 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:54.493 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-cephfs-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:54.533 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:54.534 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T15:05:54.534 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:54.594 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rbd 2024-04-22T15:05:54.615 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:54.615 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:54.616 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:54.616 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:54.616 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:54.616 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:54.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-04-22T15:05:55.107 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:29 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:55.186 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rbd-2:18.2.2-0.el9.x86_64 is already installed. 2024-04-22T15:05:55.228 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:55.229 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-04-22T15:05:55.229 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:55.299 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-fuse 2024-04-22T15:05:55.349 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:31 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:55.471 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:55.472 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:55.472 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:55.472 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:55.472 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:55.472 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:18.2.2-0.el9 ceph 3.0 M 2024-04-22T15:05:55.472 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:55.472 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:55.472 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:55.473 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-04-22T15:05:55.473 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:55.473 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.0 M 2024-04-22T15:05:55.473 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 12 M 2024-04-22T15:05:55.473 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:55.581 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-18.2.2-0.el9.x86_64.rpm 28 MB/s | 3.0 MB 00:00 2024-04-22T15:05:55.582 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:55.583 INFO:teuthology.orchestra.run.smithi196.stdout:Total 27 MB/s | 3.0 MB 00:00 2024-04-22T15:05:55.583 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:55.591 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:55.593 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:55.669 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:55.670 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:55.801 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:29 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:55.918 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:18.2.2-0.el9 ceph 86 k 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:55.919 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 86 k 2024-04-22T15:05:55.920 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 227 k 2024-04-22T15:05:55.920 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:55.983 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-fuse-18.2.2-0.el9.x86_64.rpm 1.3 MB/s | 86 kB 00:00 2024-04-22T15:05:55.983 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:55.984 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.3 MB/s | 86 kB 00:00 2024-04-22T15:05:55.984 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:55.991 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:55.991 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:56.052 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:56.052 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:56.100 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:56.191 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:56.257 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:56.281 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:56.281 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:56.281 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T15:05:56.281 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T15:05:56.282 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T15:05:56.282 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:56.295 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-fuse-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:56.883 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:56.883 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:56.883 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:56.883 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:56.883 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:56.884 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:57.157 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror 2024-04-22T15:05:57.206 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:57.206 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:57.206 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:57.206 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:57.206 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:57.206 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:57.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-04-22T15:05:57.674 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:31 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:05:57.796 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:05:57.797 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:57.797 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:57.797 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:57.797 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:05:57.797 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:18.2.2-0.el9 ceph 3.0 M 2024-04-22T15:05:57.797 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:57.797 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:05:57.797 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:05:57.798 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-04-22T15:05:57.798 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:57.798 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 3.0 M 2024-04-22T15:05:57.798 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 12 M 2024-04-22T15:05:57.798 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:05:57.910 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-mirror-18.2.2-0.el9.x86_64.rpm 27 MB/s | 3.0 MB 00:00 2024-04-22T15:05:57.910 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:57.911 INFO:teuthology.orchestra.run.smithi096.stdout:Total 26 MB/s | 3.0 MB 00:00 2024-04-22T15:05:57.911 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:05:57.920 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:05:57.920 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:05:57.982 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:33 ago on Mon 22 Apr 2024 03:04:24 PM UTC. 2024-04-22T15:05:58.001 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:05:58.001 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:05:58.104 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-22T15:05:58.104 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:58.104 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-22T15:05:58.104 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:18.2.2-0.el9 ceph 171 k 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 171 k 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 494 k 2024-04-22T15:05:58.105 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-22T15:05:58.177 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-18.2.2-0.el9.x86_64.rpm 2.3 MB/s | 171 kB 00:00 2024-04-22T15:05:58.178 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:05:58.178 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.3 MB/s | 171 kB 00:00 2024-04-22T15:05:58.178 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-22T15:05:58.186 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-22T15:05:58.186 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-22T15:05:58.248 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-22T15:05:58.248 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-22T15:05:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-22T15:05:58.439 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:05:58.531 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:58.595 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:58.620 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:58.620 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:05:58.620 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T15:05:58.620 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T15:05:58.620 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T15:05:58.620 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:58.916 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:59.148 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:59.149 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:59.149 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-22T15:05:59.149 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:59.149 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:05:59.149 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-22T15:05:59.380 DEBUG:teuthology.parallel:result is None 2024-04-22T15:05:59.517 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:05:59.518 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:59.518 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:05:59.518 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:18.2.2-0.el9.x86_64 2024-04-22T15:05:59.518 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:05:59.518 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:05:59.763 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-nbd 2024-04-22T15:06:00.277 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:34 ago on Mon 22 Apr 2024 03:04:26 PM UTC. 2024-04-22T15:06:00.400 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T15:06:00.410 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:06:00.410 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-04-22T15:06:00.410 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:06:00.410 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T15:06:00.410 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:18.2.2-0.el9 ceph 171 k 2024-04-22T15:06:00.410 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:06:00.410 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T15:06:00.411 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-22T15:06:00.411 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-04-22T15:06:00.411 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:06:00.411 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 171 k 2024-04-22T15:06:00.411 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 494 k 2024-04-22T15:06:00.411 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T15:06:00.464 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-nbd-18.2.2-0.el9.x86_64.rpm 2.7 MB/s | 171 kB 00:00 2024-04-22T15:06:00.465 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:06:00.465 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.6 MB/s | 171 kB 00:00 2024-04-22T15:06:00.465 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T15:06:00.473 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T15:06:00.473 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T15:06:00.535 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T15:06:00.536 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T15:06:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T15:06:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:06:01.156 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-nbd-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:06:01.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:18.2.2-0.el9.x86_64 1/1 2024-04-22T15:06:01.403 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:06:01.403 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T15:06:01.403 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:18.2.2-0.el9.x86_64 2024-04-22T15:06:01.403 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:06:01.403 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T15:06:01.705 DEBUG:teuthology.parallel:result is None 2024-04-22T15:06:01.705 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T15:06:01.705 INFO:teuthology.packaging:ref: None 2024-04-22T15:06:01.705 INFO:teuthology.packaging:tag: None 2024-04-22T15:06:01.706 INFO:teuthology.packaging:branch: reef 2024-04-22T15:06:01.706 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:06:01.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-22T15:06:01.931 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T15:06:01.953 INFO:teuthology.orchestra.run.smithi096.stdout:18.2.2-0.el9 2024-04-22T15:06:01.954 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-0.el9 2024-04-22T15:06:01.954 INFO:teuthology.task.install:The correct ceph version 18.2.2-0 is installed. 2024-04-22T15:06:01.955 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T15:06:01.956 INFO:teuthology.packaging:ref: None 2024-04-22T15:06:01.956 INFO:teuthology.packaging:tag: None 2024-04-22T15:06:01.956 INFO:teuthology.packaging:branch: reef 2024-04-22T15:06:01.956 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:06:01.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-22T15:06:02.181 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T15:06:02.206 INFO:teuthology.orchestra.run.smithi196.stdout:18.2.2-0.el9 2024-04-22T15:06:02.206 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-0.el9 2024-04-22T15:06:02.206 INFO:teuthology.task.install:The correct ceph version 18.2.2-0 is installed. 2024-04-22T15:06:02.208 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T15:06:02.208 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:06:02.208 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T15:06:02.232 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:06:02.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T15:06:02.274 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T15:06:02.274 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:06:02.275 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T15:06:02.298 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T15:06:02.362 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:06:02.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T15:06:02.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T15:06:02.455 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T15:06:02.455 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:06:02.455 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T15:06:02.478 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T15:06:02.542 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:06:02.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T15:06:02.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T15:06:02.634 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T15:06:02.635 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:06:02.635 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T15:06:02.658 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T15:06:02.722 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:06:02.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T15:06:02.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T15:06:02.813 INFO:teuthology.run_tasks:Running task print... 2024-04-22T15:06:02.822 INFO:teuthology.task.print:**** done install task... 2024-04-22T15:06:02.822 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T15:06:02.949 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'} 2024-04-22T15:06:02.949 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-04-22T15:06:02.950 INFO:tasks.cephadm:Cluster fsid is d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:06:02.950 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T15:06:02.950 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-22T15:06:02.950 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi096': '172.21.15.96', 'mon.smithi196': '172.21.15.196'} 2024-04-22T15:06:02.950 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T15:06:02.950 DEBUG:teuthology.orchestra.run.smithi096:> sudo hostname $(hostname -s) 2024-04-22T15:06:02.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-04-22T15:06:03.011 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-04-22T15:06:03.012 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:06:03.158 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78798/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3223-g5883d494', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3223.g5883d494'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-22-0809/5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 13:59:15.546489', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-04-22-0809', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-04-22-0809/5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-22T15:06:03.395 INFO:tasks.util.chacra:got chacra host chacra.ceph.com, ref reef, sha1 9d98999acebbad2663cc4938e94c9ffd83485cbd from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-04-22T15:06:03.397 INFO:tasks.cephadm:Discovered cachra url: https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm 2024-04-22T15:06:03.397 DEBUG:teuthology.orchestra.run.smithi096:> curl --silent -L https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T15:06:03.704 INFO:teuthology.orchestra.run.smithi096.stdout:-rw-r--r--. 1 ubuntu ubuntu 212578 Apr 22 15:06 /home/ubuntu/cephtest/cephadm 2024-04-22T15:06:03.704 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent -L https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T15:06:04.002 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 212578 Apr 22 15:06 /home/ubuntu/cephtest/cephadm 2024-04-22T15:06:04.003 DEBUG:teuthology.orchestra.run.smithi096:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T15:06:04.020 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T15:06:04.051 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-04-22T15:06:04.051 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-04-22T15:06:04.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-04-22T15:06:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-04-22T15:06:05.564 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-04-22T15:07:57.457 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:07:57.458 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)", 2024-04-22T15:07:57.458 INFO:teuthology.orchestra.run.smithi096.stdout: "image_id": "9900ed1a6b81aaf043a19454fc819f6cb21ac0421c7c720b31d8fae6faebb644", 2024-04-22T15:07:57.458 INFO:teuthology.orchestra.run.smithi096.stdout: "repo_digests": [ 2024-04-22T15:07:57.458 INFO:teuthology.orchestra.run.smithi096.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210" 2024-04-22T15:07:57.458 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-04-22T15:07:57.458 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:08:06.508 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-04-22T15:08:06.508 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)", 2024-04-22T15:08:06.509 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "9900ed1a6b81aaf043a19454fc819f6cb21ac0421c7c720b31d8fae6faebb644", 2024-04-22T15:08:06.509 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-04-22T15:08:06.509 INFO:teuthology.orchestra.run.smithi196.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210" 2024-04-22T15:08:06.509 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-04-22T15:08:06.509 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-04-22T15:08:06.527 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph 2024-04-22T15:08:06.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-04-22T15:08:06.586 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /etc/ceph 2024-04-22T15:08:06.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-04-22T15:08:06.650 INFO:tasks.cephadm:Writing seed config... 2024-04-22T15:08:06.651 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-22T15:08:06.651 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-22T15:08:06.651 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-22T15:08:06.651 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-22T15:08:06.651 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-22T15:08:06.651 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-22T15:08:06.651 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-22T15:08:06.651 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-22T15:08:06.652 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T15:08:06.653 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T15:08:06.654 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T15:08:06.654 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T15:08:06.654 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T15:08:06.654 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-22T15:08:06.654 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-22T15:08:06.655 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:08:06.655 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T15:08:06.685 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d5e86090-00b9-11ef-bc93-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-22T15:08:06.685 DEBUG:teuthology.orchestra.run.smithi096:mon.smithi096> sudo journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096.service 2024-04-22T15:08:06.726 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T15:08:06.727 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid d5e86090-00b9-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.96 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T15:08:06.884 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:08:06.884 INFO:teuthology.orchestra.run.smithi096.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', 'd5e86090-00b9-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.96', '--skip-admin-label'] 2024-04-22T15:08:06.909 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T15:08:06.909 INFO:teuthology.orchestra.run.smithi096.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-22T15:08:06.910 INFO:teuthology.orchestra.run.smithi096.stdout:Verifying podman|docker is present... 2024-04-22T15:08:06.933 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T15:08:06.933 INFO:teuthology.orchestra.run.smithi096.stdout:Verifying lvm2 is present... 2024-04-22T15:08:06.933 INFO:teuthology.orchestra.run.smithi096.stdout:Verifying time synchronization is in place... 2024-04-22T15:08:06.940 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T15:08:06.940 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T15:08:06.946 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T15:08:06.946 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout inactive 2024-04-22T15:08:06.952 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout enabled 2024-04-22T15:08:06.958 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout active 2024-04-22T15:08:06.958 INFO:teuthology.orchestra.run.smithi096.stdout:Unit chronyd.service is enabled and running 2024-04-22T15:08:06.958 INFO:teuthology.orchestra.run.smithi096.stdout:Repeating the final host check... 2024-04-22T15:08:06.982 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T15:08:06.982 INFO:teuthology.orchestra.run.smithi096.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-22T15:08:06.982 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl is present 2024-04-22T15:08:06.982 INFO:teuthology.orchestra.run.smithi096.stdout:lvcreate is present 2024-04-22T15:08:06.988 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T15:08:06.988 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T15:08:06.994 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T15:08:06.994 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout inactive 2024-04-22T15:08:07.000 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout enabled 2024-04-22T15:08:07.006 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout active 2024-04-22T15:08:07.006 INFO:teuthology.orchestra.run.smithi096.stdout:Unit chronyd.service is enabled and running 2024-04-22T15:08:07.006 INFO:teuthology.orchestra.run.smithi096.stdout:Host looks OK 2024-04-22T15:08:07.006 INFO:teuthology.orchestra.run.smithi096.stdout:Cluster fsid: d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:07.006 INFO:teuthology.orchestra.run.smithi096.stdout:Acquiring lock 140138443009184 on /run/cephadm/d5e86090-00b9-11ef-bc93-c7b262605968.lock 2024-04-22T15:08:07.006 INFO:teuthology.orchestra.run.smithi096.stdout:Lock 140138443009184 acquired on /run/cephadm/d5e86090-00b9-11ef-bc93-c7b262605968.lock 2024-04-22T15:08:07.007 INFO:teuthology.orchestra.run.smithi096.stdout:Verifying IP 172.21.15.96 port 3300 ... 2024-04-22T15:08:07.007 INFO:teuthology.orchestra.run.smithi096.stdout:Verifying IP 172.21.15.96 port 6789 ... 2024-04-22T15:08:07.007 INFO:teuthology.orchestra.run.smithi096.stdout:Base mon IP(s) is [172.21.15.96:3300, 172.21.15.96:6789], mon addrv is [v2:172.21.15.96:3300,v1:172.21.15.96:6789] 2024-04-22T15:08:07.010 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.96 metric 100 2024-04-22T15:08:07.010 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.96 metric 100 2024-04-22T15:08:07.013 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T15:08:07.013 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-22T15:08:07.013 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-22T15:08:07.015 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T15:08:07.015 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-22T15:08:07.015 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T15:08:07.015 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-22T15:08:07.015 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout inet6 fe80::811d:49ea:b028:3af9/64 scope link noprefixroute 2024-04-22T15:08:07.015 INFO:teuthology.orchestra.run.smithi096.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T15:08:07.016 INFO:teuthology.orchestra.run.smithi096.stdout:Mon IP `172.21.15.96` is in CIDR network `172.21.0.0/20` 2024-04-22T15:08:07.016 INFO:teuthology.orchestra.run.smithi096.stdout:Mon IP `172.21.15.96` is in CIDR network `172.21.0.0/20` 2024-04-22T15:08:07.016 INFO:teuthology.orchestra.run.smithi096.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-22T15:08:07.016 INFO:teuthology.orchestra.run.smithi096.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T15:08:07.017 INFO:teuthology.orchestra.run.smithi096.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-04-22T15:08:07.606 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stdout 9900ed1a6b81aaf043a19454fc819f6cb21ac0421c7c720b31d8fae6faebb644 2024-04-22T15:08:07.606 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-04-22T15:08:07.606 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stderr Getting image source signatures 2024-04-22T15:08:07.606 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stderr Copying blob sha256:c98db977c11909275f2799ff686e2165262975518f51f9eea9b24a5c8a4ea0d6 2024-04-22T15:08:07.606 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-22T15:08:07.606 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stderr Copying config sha256:9900ed1a6b81aaf043a19454fc819f6cb21ac0421c7c720b31d8fae6faebb644 2024-04-22T15:08:07.606 INFO:teuthology.orchestra.run.smithi096.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-22T15:08:08.519 INFO:teuthology.orchestra.run.smithi096.stdout:ceph: stdout ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable) 2024-04-22T15:08:08.520 INFO:teuthology.orchestra.run.smithi096.stdout:Ceph version: ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable) 2024-04-22T15:08:08.520 INFO:teuthology.orchestra.run.smithi096.stdout:Extracting ceph user uid/gid from container image... 2024-04-22T15:08:09.360 INFO:teuthology.orchestra.run.smithi096.stdout:stat: stdout 167 167 2024-04-22T15:08:09.360 INFO:teuthology.orchestra.run.smithi096.stdout:Creating initial keys... 2024-04-22T15:08:10.235 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph-authtool: stdout AQBZfSZmKcU8KhAAqqSHa7rnJL8QJg7TUuwjlQ== 2024-04-22T15:08:11.143 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph-authtool: stdout AQBafSZmhyU+IxAA1cGyxaPiMB4qzAHQ/AOMAQ== 2024-04-22T15:08:12.086 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph-authtool: stdout AQBbfSZma4FKGhAA5hPYQGmy/8qkGu6Sug7CZQ== 2024-04-22T15:08:12.087 INFO:teuthology.orchestra.run.smithi096.stdout:Creating initial monmap... 2024-04-22T15:08:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T15:08:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-22T15:08:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T15:08:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool for smithi096 [v2:172.21.15.96:3300,v1:172.21.15.96:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T15:08:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:setting min_mon_release = pacific 2024-04-22T15:08:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/monmaptool: set fsid to d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T15:08:13.078 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:08:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:Creating mon... 2024-04-22T15:08:21.201 INFO:teuthology.orchestra.run.smithi096.stdout:create mon.smithi096 on 2024-04-22T15:08:21.696 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-22T15:08:21.946 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T15:08:22.386 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d5e86090-00b9-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-d5e86090-00b9-11ef-bc93-c7b262605968.target. 2024-04-22T15:08:22.386 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d5e86090-00b9-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-d5e86090-00b9-11ef-bc93-c7b262605968.target. 2024-04-22T15:08:22.890 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096 2024-04-22T15:08:22.890 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096.service: Unit ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096.service not loaded. 2024-04-22T15:08:23.048 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d5e86090-00b9-11ef-bc93-c7b262605968.target.wants/ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096.service → /etc/systemd/system/ceph-d5e86090-00b9-11ef-bc93-c7b262605968@.service. 2024-04-22T15:08:23.373 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:23 smithi096 podman[26287]: 2024-04-22 15:08:23.288810515 +0000 UTC m=+0.118755069 container create 05024fbae9d5b6462c1b564f9823e0e25e48ee46af88047f8e72f9e7afa0b6e9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-646b8c7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T15:08:23.549 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T15:08:23.550 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout disabled 2024-04-22T15:08:23.555 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T15:08:23.555 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout inactive 2024-04-22T15:08:23.555 INFO:teuthology.orchestra.run.smithi096.stdout:firewalld.service is not enabled 2024-04-22T15:08:23.555 INFO:teuthology.orchestra.run.smithi096.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T15:08:23.556 INFO:teuthology.orchestra.run.smithi096.stdout:Waiting for mon to start... 2024-04-22T15:08:23.556 INFO:teuthology.orchestra.run.smithi096.stdout:Waiting for mon... 2024-04-22T15:08:23.662 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:23 smithi096 podman[26287]: 2024-04-22 15:08:23.449430802 +0000 UTC m=+0.279375357 container init 05024fbae9d5b6462c1b564f9823e0e25e48ee46af88047f8e72f9e7afa0b6e9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=reef-646b8c7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-04-22T15:08:23.662 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:23 smithi096 podman[26287]: 2024-04-22 15:08:23.452282785 +0000 UTC m=+0.282227340 container start 05024fbae9d5b6462c1b564f9823e0e25e48ee46af88047f8e72f9e7afa0b6e9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=reef-646b8c7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-22T15:08:23.662 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:23 smithi096 ceph-mon[26306]: mkfs d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:23.662 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:23 smithi096 ceph-mon[26306]: mon.smithi096 is new leader, mons smithi096 in quorum (ranks 0) 2024-04-22T15:08:23.662 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:23 smithi096 bash[26287]: 05024fbae9d5b6462c1b564f9823e0e25e48ee46af88047f8e72f9e7afa0b6e9 2024-04-22T15:08:23.662 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:23 smithi096 systemd[1]: Started Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:08:24.593 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:24 smithi096 ceph-mon[26306]: mon.smithi096 is new leader, mons smithi096 in quorum (ranks 0) 2024-04-22T15:08:24.593 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:24 smithi096 ceph-mon[26306]: monmap e1: 1 mons at {smithi096=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T15:08:24.593 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:24 smithi096 ceph-mon[26306]: fsmap 2024-04-22T15:08:24.593 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:24 smithi096 ceph-mon[26306]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T15:08:24.593 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:24 smithi096 ceph-mon[26306]: mgrmap e1: no daemons active 2024-04-22T15:08:24.594 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:24 smithi096 ceph-mon[26306]: from='client.? 172.21.15.96:0/3589270984' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T15:08:24.602 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout cluster: 2024-04-22T15:08:24.602 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout id: d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:24.606 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-22T15:08:24.606 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:24.606 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout services: 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi096 (age 0.567488s) 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout data: 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout pgs: 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.982+0000 7f0f6f5ab700 1 Processor -- start 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.983+0000 7f0f6f5ab700 1 -- start start 2024-04-22T15:08:24.607 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.983+0000 7f0f6f5ab700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f681069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.983+0000 7f0f6f5ab700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68106fa0 con 0x7f0f681065b0 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.983+0000 7f0f6d347700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f681069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.983+0000 7f0f6d347700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f681069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34148/0 (socket says 172.21.15.96:34148) 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.983+0000 7f0f6d347700 1 -- 172.21.15.96:0/3858509937 learned_addr learned my addr 172.21.15.96:0/3858509937 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.983+0000 7f0f6d347700 1 -- 172.21.15.96:0/3858509937 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f681077b0 con 0x7f0f681065b0 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.983+0000 7f0f6d347700 1 --2- 172.21.15.96:0/3858509937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f681069d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0f580098a0 tx=0x7f0f58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9b8a81e8d94bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f5ffff700 1 -- 172.21.15.96:0/3858509937 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f58004180 con 0x7f0f681065b0 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f5ffff700 1 -- 172.21.15.96:0/3858509937 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f580042e0 con 0x7f0f681065b0 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f5ffff700 1 -- 172.21.15.96:0/3858509937 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f580036a0 con 0x7f0f681065b0 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3858509937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 msgr2=0x7f0f681069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f6f5ab700 1 --2- 172.21.15.96:0/3858509937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f681069d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0f580098a0 tx=0x7f0f58009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:24.608 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3858509937 shutdown_connections 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f6f5ab700 1 --2- 172.21.15.96:0/3858509937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f681069d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3858509937 >> 172.21.15.96:0/3858509937 conn(0x7f0f68101bd0 msgr2=0x7f0f68103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3858509937 shutdown_connections 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.984+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3858509937 wait complete. 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.985+0000 7f0f6f5ab700 1 Processor -- start 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.985+0000 7f0f6f5ab700 1 -- start start 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.985+0000 7f0f6f5ab700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f68199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.985+0000 7f0f6f5ab700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68199fc0 con 0x7f0f681065b0 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.985+0000 7f0f6d347700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f68199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.985+0000 7f0f6d347700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f68199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34160/0 (socket says 172.21.15.96:34160) 2024-04-22T15:08:24.609 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.985+0000 7f0f6d347700 1 -- 172.21.15.96:0/3589270984 learned_addr learned my addr 172.21.15.96:0/3589270984 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.985+0000 7f0f6d347700 1 -- 172.21.15.96:0/3589270984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f58009550 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.986+0000 7f0f6d347700 1 --2- 172.21.15.96:0/3589270984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f68199a80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0f58004000 tx=0x7f0f58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.986+0000 7f0f5e7fc700 1 -- 172.21.15.96:0/3589270984 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f58003ce0 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.986+0000 7f0f5e7fc700 1 -- 172.21.15.96:0/3589270984 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f58020070 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.986+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3589270984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6819a1c0 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.986+0000 7f0f5e7fc700 1 -- 172.21.15.96:0/3589270984 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f58011630 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.986+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3589270984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6819a660 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.986+0000 7f0f5e7fc700 1 -- 172.21.15.96:0/3589270984 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f0f58022030 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.986+0000 7f0f5e7fc700 1 -- 172.21.15.96:0/3589270984 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0f5801b450 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.987+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3589270984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f681937d0 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:23.989+0000 7f0f5e7fc700 1 -- 172.21.15.96:0/3589270984 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f0f58025480 con 0x7f0f681065b0 2024-04-22T15:08:24.610 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:24.058+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3589270984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f0f68107d40 con 0x7f0f681065b0 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:24.058+0000 7f0f5e7fc700 1 -- 172.21.15.96:0/3589270984 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f0f58025620 con 0x7f0f681065b0 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:24.059+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3589270984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 msgr2=0x7f0f68199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:24.059+0000 7f0f6f5ab700 1 --2- 172.21.15.96:0/3589270984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f68199a80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0f58004000 tx=0x7f0f58004750 comp rx=0 tx=0).stop 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:24.059+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3589270984 shutdown_connections 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:24.059+0000 7f0f6f5ab700 1 --2- 172.21.15.96:0/3589270984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f681065b0 0x7f0f68199a80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:24.059+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3589270984 >> 172.21.15.96:0/3589270984 conn(0x7f0f68101bd0 msgr2=0x7f0f68190910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:24.059+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3589270984 shutdown_connections 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:24.059+0000 7f0f6f5ab700 1 -- 172.21.15.96:0/3589270984 wait complete. 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:mon is available 2024-04-22T15:08:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:Assimilating anything we can from ceph.conf... 2024-04-22T15:08:25.683 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:25 smithi096 ceph-mon[26306]: from='client.? 172.21.15.96:0/1824071694' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T15:08:25.683 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:25 smithi096 ceph-mon[26306]: from='client.? 172.21.15.96:0/1824071694' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout fsid = d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.96:3300,v1:172.21.15.96:6789] 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T15:08:25.692 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.013+0000 7fa42831c700 1 Processor -- start 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.013+0000 7fa42831c700 1 -- start start 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.013+0000 7fa42831c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa420108000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.013+0000 7fa42831c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4201085d0 con 0x7fa420107be0 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.014+0000 7fa4260b8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa420108000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:25.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.014+0000 7fa4260b8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa420108000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34162/0 (socket says 172.21.15.96:34162) 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.014+0000 7fa4260b8700 1 -- 172.21.15.96:0/4263263878 learned_addr learned my addr 172.21.15.96:0/4263263878 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.014+0000 7fa4260b8700 1 -- 172.21.15.96:0/4263263878 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa420108710 con 0x7fa420107be0 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.014+0000 7fa4260b8700 1 --2- 172.21.15.96:0/4263263878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa420108000 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009e80 tx=0x7fa41c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f804e6fdb3a28ff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.014+0000 7fa4250b6700 1 -- 172.21.15.96:0/4263263878 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa41c00e070 con 0x7fa420107be0 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.014+0000 7fa4250b6700 1 -- 172.21.15.96:0/4263263878 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41c003a20 con 0x7fa420107be0 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.014+0000 7fa4250b6700 1 -- 172.21.15.96:0/4263263878 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa41c003bd0 con 0x7fa420107be0 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.015+0000 7fa42831c700 1 -- 172.21.15.96:0/4263263878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 msgr2=0x7fa420108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.015+0000 7fa42831c700 1 --2- 172.21.15.96:0/4263263878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa420108000 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009e80 tx=0x7fa41c00b330 comp rx=0 tx=0).stop 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.015+0000 7fa42831c700 1 -- 172.21.15.96:0/4263263878 shutdown_connections 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.015+0000 7fa42831c700 1 --2- 172.21.15.96:0/4263263878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa420108000 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:25.694 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.015+0000 7fa42831c700 1 -- 172.21.15.96:0/4263263878 >> 172.21.15.96:0/4263263878 conn(0x7fa4201001c0 msgr2=0x7fa420102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.015+0000 7fa42831c700 1 -- 172.21.15.96:0/4263263878 shutdown_connections 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.015+0000 7fa42831c700 1 -- 172.21.15.96:0/4263263878 wait complete. 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.015+0000 7fa42831c700 1 Processor -- start 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa42831c700 1 -- start start 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa42831c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa4201071b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa42831c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4201076f0 con 0x7fa420107be0 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa4260b8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa4201071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa4260b8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa4201071b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34178/0 (socket says 172.21.15.96:34178) 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa4260b8700 1 -- 172.21.15.96:0/1824071694 learned_addr learned my addr 172.21.15.96:0/1824071694 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa4260b8700 1 -- 172.21.15.96:0/1824071694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c009b30 con 0x7fa420107be0 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa4260b8700 1 --2- 172.21.15.96:0/1824071694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa4201071b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa41c004000 tx=0x7fa41c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:25.695 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa4177fe700 1 -- 172.21.15.96:0/1824071694 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa41c004240 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa4177fe700 1 -- 172.21.15.96:0/1824071694 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41c0043e0 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa42831c700 1 -- 172.21.15.96:0/1824071694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa420105930 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa4177fe700 1 -- 172.21.15.96:0/1824071694 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa41c0046e0 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa42831c700 1 -- 172.21.15.96:0/1824071694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa420105dd0 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.016+0000 7fa4177fe700 1 -- 172.21.15.96:0/1824071694 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fa41c019050 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.017+0000 7fa4177fe700 1 -- 172.21.15.96:0/1824071694 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa41c0257e0 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.017+0000 7fa42831c700 1 -- 172.21.15.96:0/1824071694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa420062510 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.020+0000 7fa4177fe700 1 -- 172.21.15.96:0/1824071694 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fa41c02a060 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.086+0000 7fa42831c700 1 -- 172.21.15.96:0/1824071694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa420104780 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.089+0000 7fa4177fe700 1 -- 172.21.15.96:0/1824071694 <== mon.0 v2:172.21.15.96:3300/0 7 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa41c012bb0 con 0x7fa420107be0 2024-04-22T15:08:25.696 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.089+0000 7fa4177fe700 1 -- 172.21.15.96:0/1824071694 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fa41c020030 con 0x7fa420107be0 2024-04-22T15:08:25.697 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.090+0000 7fa42831c700 1 -- 172.21.15.96:0/1824071694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 msgr2=0x7fa4201071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:25.697 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.090+0000 7fa42831c700 1 --2- 172.21.15.96:0/1824071694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa4201071b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa41c004000 tx=0x7fa41c004ad0 comp rx=0 tx=0).stop 2024-04-22T15:08:25.697 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.090+0000 7fa42831c700 1 -- 172.21.15.96:0/1824071694 shutdown_connections 2024-04-22T15:08:25.697 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.090+0000 7fa42831c700 1 --2- 172.21.15.96:0/1824071694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa420107be0 0x7fa4201071b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:25.697 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.090+0000 7fa42831c700 1 -- 172.21.15.96:0/1824071694 >> 172.21.15.96:0/1824071694 conn(0x7fa4201001c0 msgr2=0x7fa4201909f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:25.697 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.090+0000 7fa42831c700 1 -- 172.21.15.96:0/1824071694 shutdown_connections 2024-04-22T15:08:25.697 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:25.090+0000 7fa42831c700 1 -- 172.21.15.96:0/1824071694 wait complete. 2024-04-22T15:08:25.697 INFO:teuthology.orchestra.run.smithi096.stdout:Generating new minimal ceph.conf... 2024-04-22T15:08:26.757 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.096+0000 7f11795f4700 1 Processor -- start 2024-04-22T15:08:26.757 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.096+0000 7f11795f4700 1 -- start start 2024-04-22T15:08:26.757 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.096+0000 7f11795f4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f1174105090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:26.757 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.096+0000 7f11795f4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174105660 con 0x7f1174104c70 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.096+0000 7f1172ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f1174105090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.096+0000 7f1172ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f1174105090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34194/0 (socket says 172.21.15.96:34194) 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.096+0000 7f1172ffd700 1 -- 172.21.15.96:0/3956564294 learned_addr learned my addr 172.21.15.96:0/3956564294 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.097+0000 7f1172ffd700 1 -- 172.21.15.96:0/3956564294 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1174105e70 con 0x7f1174104c70 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.097+0000 7f1172ffd700 1 --2- 172.21.15.96:0/3956564294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f1174105090 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f116400bb40 tx=0x7f116400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c67f1dee6d37370 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.097+0000 7f1171ffb700 1 -- 172.21.15.96:0/3956564294 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116400d9a0 con 0x7f1174104c70 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.097+0000 7f1171ffb700 1 -- 172.21.15.96:0/3956564294 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1164004030 con 0x7f1174104c70 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.097+0000 7f1171ffb700 1 -- 172.21.15.96:0/3956564294 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1164003cb0 con 0x7f1174104c70 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.098+0000 7f11795f4700 1 -- 172.21.15.96:0/3956564294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 msgr2=0x7f1174105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.098+0000 7f11795f4700 1 --2- 172.21.15.96:0/3956564294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f1174105090 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f116400bb40 tx=0x7f116400be50 comp rx=0 tx=0).stop 2024-04-22T15:08:26.758 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.098+0000 7f11795f4700 1 -- 172.21.15.96:0/3956564294 shutdown_connections 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.098+0000 7f11795f4700 1 --2- 172.21.15.96:0/3956564294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f1174105090 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.098+0000 7f11795f4700 1 -- 172.21.15.96:0/3956564294 >> 172.21.15.96:0/3956564294 conn(0x7f1174100270 msgr2=0x7f1174102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.098+0000 7f11795f4700 1 -- 172.21.15.96:0/3956564294 shutdown_connections 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.098+0000 7f11795f4700 1 -- 172.21.15.96:0/3956564294 wait complete. 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.098+0000 7f11795f4700 1 Processor -- start 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.098+0000 7f11795f4700 1 -- start start 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f11795f4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f11741980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f11795f4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174198610 con 0x7f1174104c70 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f1172ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f11741980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f1172ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f11741980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34206/0 (socket says 172.21.15.96:34206) 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f1172ffd700 1 -- 172.21.15.96:0/3906604199 learned_addr learned my addr 172.21.15.96:0/3906604199 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f1172ffd700 1 -- 172.21.15.96:0/3906604199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116400b7f0 con 0x7f1174104c70 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f1172ffd700 1 --2- 172.21.15.96:0/3906604199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f11741980d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1164003740 tx=0x7f1164004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:26.759 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f115bfff700 1 -- 172.21.15.96:0/3906604199 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1164004640 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f115bfff700 1 -- 172.21.15.96:0/3906604199 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1164017930 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f11795f4700 1 -- 172.21.15.96:0/3906604199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1174198810 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f115bfff700 1 -- 172.21.15.96:0/3906604199 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11640218a0 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f11795f4700 1 -- 172.21.15.96:0/3906604199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1174198d10 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.099+0000 7f115bfff700 1 -- 172.21.15.96:0/3906604199 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f1164015070 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.100+0000 7f115bfff700 1 -- 172.21.15.96:0/3906604199 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f116402a6f0 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.100+0000 7f11795f4700 1 -- 172.21.15.96:0/3906604199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1174191da0 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.102+0000 7f115bfff700 1 -- 172.21.15.96:0/3906604199 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f1164026070 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.169+0000 7f11795f4700 1 -- 172.21.15.96:0/3906604199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f11741065c0 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.169+0000 7f115bfff700 1 -- 172.21.15.96:0/3906604199 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f1164017aa0 con 0x7f1174104c70 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.170+0000 7f11795f4700 1 -- 172.21.15.96:0/3906604199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 msgr2=0x7f11741980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.170+0000 7f11795f4700 1 --2- 172.21.15.96:0/3906604199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f11741980d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1164003740 tx=0x7f1164004250 comp rx=0 tx=0).stop 2024-04-22T15:08:26.760 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.170+0000 7f11795f4700 1 -- 172.21.15.96:0/3906604199 shutdown_connections 2024-04-22T15:08:26.761 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.170+0000 7f11795f4700 1 --2- 172.21.15.96:0/3906604199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174104c70 0x7f11741980d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:26.761 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.170+0000 7f11795f4700 1 -- 172.21.15.96:0/3906604199 >> 172.21.15.96:0/3906604199 conn(0x7f1174100270 msgr2=0x7f117418eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:26.761 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.170+0000 7f11795f4700 1 -- 172.21.15.96:0/3906604199 shutdown_connections 2024-04-22T15:08:26.761 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:26.170+0000 7f11795f4700 1 -- 172.21.15.96:0/3906604199 wait complete. 2024-04-22T15:08:26.761 INFO:teuthology.orchestra.run.smithi096.stdout:Restarting the monitor... 2024-04-22T15:08:26.774 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:26 smithi096 ceph-mon[26306]: from='client.? 172.21.15.96:0/3906604199' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:08:27.209 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:26 smithi096 systemd[1]: Stopping Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:08:27.210 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:27 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[26302]: 2024-04-22T15:08:27.015+0000 7feda42c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi096 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:08:27.210 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:27 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[26302]: 2024-04-22T15:08:27.015+0000 7feda42c0700 -1 mon.smithi096@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T15:08:27.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:27 smithi096 podman[26553]: 2024-04-22 15:08:27.271480434 +0000 UTC m=+0.356903858 container died 05024fbae9d5b6462c1b564f9823e0e25e48ee46af88047f8e72f9e7afa0b6e9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-646b8c7, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-22T15:08:27.786 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:27 smithi096 podman[26553]: 2024-04-22 15:08:27.59888622 +0000 UTC m=+0.684309643 container cleanup 05024fbae9d5b6462c1b564f9823e0e25e48ee46af88047f8e72f9e7afa0b6e9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=reef-646b8c7, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T15:08:27.786 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:27 smithi096 bash[26553]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096 2024-04-22T15:08:28.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:27 smithi096 podman[26568]: 2024-04-22 15:08:27.783176324 +0000 UTC m=+0.512584191 container remove 05024fbae9d5b6462c1b564f9823e0e25e48ee46af88047f8e72f9e7afa0b6e9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, RELEASE=reef-646b8c7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-22T15:08:28.437 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:28 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096.service: Deactivated successfully. 2024-04-22T15:08:28.437 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:28 smithi096 systemd[1]: Stopped Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:08:28.437 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:28 smithi096 systemd[1]: Starting Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:08:28.437 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:28 smithi096 podman[26668]: 2024-04-22 15:08:28.177673049 +0000 UTC m=+0.018032436 image pull 9900ed1a6b81aaf043a19454fc819f6cb21ac0421c7c720b31d8fae6faebb644 quay.ceph.io/ceph-ci/ceph:reef 2024-04-22T15:08:29.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 podman[26668]: 2024-04-22 15:08:29.22349434 +0000 UTC m=+1.063853723 container create 3f2d97a8db3a149277b16be4539bcd1cb47c5da480083f9827a73733e873f619 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=reef-646b8c7, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True) 2024-04-22T15:08:29.852 INFO:teuthology.orchestra.run.smithi096.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-22T15:08:29.958 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 podman[26668]: 2024-04-22 15:08:29.787245444 +0000 UTC m=+1.627604834 container init 3f2d97a8db3a149277b16be4539bcd1cb47c5da480083f9827a73733e873f619 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, RELEASE=reef-646b8c7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-22T15:08:29.958 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 podman[26668]: 2024-04-22 15:08:29.790550707 +0000 UTC m=+1.630910094 container start 3f2d97a8db3a149277b16be4539bcd1cb47c5da480083f9827a73733e873f619 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-646b8c7, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-22T15:08:29.958 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable), process ceph-mon, pid 2 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: pidfile_write: ignore empty --pid-file 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: load: jerasure load: lrc 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: RocksDB version: 7.9.2 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Git sha 0 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Compile date 2024-04-18 11:37:30 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: DB SUMMARY 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: DB Session ID: JO47Z3HO6M6YS4C0MRDU 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: CURRENT file: CURRENT 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-22T15:08:29.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi096/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi096/store.db: 000009.log size: 98487 ; 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.error_if_exists: 0 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.create_if_missing: 0 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.paranoid_checks: 1 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.env: 0x564be8b44820 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.info_log: 0x564bea6593c0 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.statistics: (nil) 2024-04-22T15:08:29.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.use_fsync: 0 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_log_file_size: 0 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.allow_fallocate: 1 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.use_direct_reads: 0 2024-04-22T15:08:29.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.db_log_dir: 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.wal_dir: 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.write_buffer_manager: 0x564be98ce140 2024-04-22T15:08:29.963 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.unordered_write: 0 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T15:08:29.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.row_cache: None 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.wal_filter: None 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.two_write_queues: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.wal_compression: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.atomic_flush: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.log_readahead_size: 0 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T15:08:29.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_background_jobs: 2 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_background_compactions: -1 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_subcompactions: 1 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T15:08:29.966 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_open_files: -1 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_background_flushes: -1 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Compression algorithms supported: 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: kZSTD supported: 0 2024-04-22T15:08:29.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: kXpressCompression supported: 0 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: kZlibCompression supported: 1 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: kSnappyCompression supported: 1 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: kLZ4Compression supported: 1 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: kBZip2Compression supported: 0 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi096/store.db/MANIFEST-000010 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.merge_operator: 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_filter: None 2024-04-22T15:08:29.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564bea6594e0) 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: cache_index_and_filter_blocks: 1 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: pin_top_level_index_and_filter: 1 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: index_type: 0 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: data_block_index_type: 0 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: index_shortening: 1 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: checksum: 4 2024-04-22T15:08:29.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: no_block_cache: 0 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_cache: 0x564be996f1f0 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_cache_name: BinnedLRUCache 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_cache_options: 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: capacity : 536870912 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: num_shard_bits : 4 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: strict_capacity_limit : 0 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: high_pri_pool_ratio: 0.000 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_cache_compressed: (nil) 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: persistent_cache: (nil) 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_size: 4096 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_size_deviation: 10 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_restart_interval: 16 2024-04-22T15:08:29.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: index_block_restart_interval: 1 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: metadata_block_size: 4096 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: partition_filters: 0 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: use_delta_encoding: 1 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: filter_policy: bloomfilter 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: whole_key_filtering: 1 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: verify_compression: 0 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: read_amp_bytes_per_bit: 0 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: format_version: 5 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: enable_index_compression: 1 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_align: 0 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: max_auto_readahead_size: 262144 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: prepopulate_block_cache: 0 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: initial_auto_readahead_size: 8192 2024-04-22T15:08:29.971 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression: NoCompression 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.num_levels: 7 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T15:08:29.972 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T15:08:29.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T15:08:29.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T15:08:29.975 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.table_properties_collectors: 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.inplace_update_support: 0 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.bloom_locality: 0 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.max_successive_merges: 0 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T15:08:29.976 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.ttl: 2592000 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.enable_blob_files: false 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.min_blob_size: 0 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T15:08:29.977 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi096/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 47b06c48-7c12-4243-b0a8-7f79eb4a9fac 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713798509820114, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713798509821082, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93764, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91798, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84362, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713798509, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "47b06c48-7c12-4243-b0a8-7f79eb4a9fac", "db_session_id": "JO47Z3HO6M6YS4C0MRDU", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713798509821168, "job": 1, "event": "recovery_finished"} 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi096/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564be9a0e000 2024-04-22T15:08:29.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: DB pointer 0x564be99fc000 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ** DB Stats ** 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:08:29.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ** Compaction Stats [default] ** 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: L0 2/0 93.44 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Sum 2/0 93.44 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ** Compaction Stats [default] ** 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:08:29.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:08:29.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T15:08:29.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T15:08:29.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T15:08:29.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T15:08:29.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T15:08:29.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Cumulative compaction: 0.00 GB write, 28.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:08:29.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Interval compaction: 0.00 GB write, 28.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:08:29.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T15:08:29.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Block cache BinnedLRUCache@0x564be996f1f0#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: starting mon.smithi096 rank 0 at public addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] at bind addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi096 fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mon.smithi096@-1(???) e1 preinit fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mon.smithi096@-1(???).mds e1 new map 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mon.smithi096@-1(???).mds e1 print_map 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: e1 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: legacy client fscid: -1 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: No filesystems configured 2024-04-22T15:08:29.982 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mon.smithi096@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mon.smithi096@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mon.smithi096@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mon.smithi096@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mon.smithi096@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 bash[26668]: 3f2d97a8db3a149277b16be4539bcd1cb47c5da480083f9827a73733e873f619 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 systemd[1]: Started Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mon.smithi096 is new leader, mons smithi096 in quorum (ranks 0) 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: monmap e1: 1 mons at {smithi096=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: fsmap 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T15:08:29.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:29 smithi096 ceph-mon[26688]: mgrmap e1: no daemons active 2024-04-22T15:08:30.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.324+0000 7f0303280700 1 Processor -- start 2024-04-22T15:08:30.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.324+0000 7f0303280700 1 -- start start 2024-04-22T15:08:30.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.325+0000 7f0303280700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc104ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:30.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.325+0000 7f0303280700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc1052b0 con 0x7f02fc1048c0 2024-04-22T15:08:30.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.325+0000 7f030101c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc104ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:30.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.325+0000 7f030101c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc104ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34220/0 (socket says 172.21.15.96:34220) 2024-04-22T15:08:30.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.325+0000 7f030101c700 1 -- 172.21.15.96:0/1160478151 learned_addr learned my addr 172.21.15.96:0/1160478151 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:30.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.325+0000 7f030101c700 1 -- 172.21.15.96:0/1160478151 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc107850 con 0x7f02fc1048c0 2024-04-22T15:08:30.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.326+0000 7f030101c700 1 --2- 172.21.15.96:0/1160478151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc104ce0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0098a0 tx=0x7f02ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a403a6914765295 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.326+0000 7f02f3fff700 1 -- 172.21.15.96:0/1160478151 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ec004180 con 0x7f02fc1048c0 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.326+0000 7f02f3fff700 1 -- 172.21.15.96:0/1160478151 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f02ec0042e0 con 0x7f02fc1048c0 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.326+0000 7f02f3fff700 1 -- 172.21.15.96:0/1160478151 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ec011a70 con 0x7f02fc1048c0 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.326+0000 7f0303280700 1 -- 172.21.15.96:0/1160478151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 msgr2=0x7f02fc104ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.326+0000 7f0303280700 1 --2- 172.21.15.96:0/1160478151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc104ce0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0098a0 tx=0x7f02ec009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.326+0000 7f0303280700 1 -- 172.21.15.96:0/1160478151 shutdown_connections 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.326+0000 7f0303280700 1 --2- 172.21.15.96:0/1160478151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc104ce0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.326+0000 7f0303280700 1 -- 172.21.15.96:0/1160478151 >> 172.21.15.96:0/1160478151 conn(0x7f02fc1001f0 msgr2=0x7f02fc102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.327+0000 7f0303280700 1 -- 172.21.15.96:0/1160478151 shutdown_connections 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.327+0000 7f0303280700 1 -- 172.21.15.96:0/1160478151 wait complete. 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.327+0000 7f0303280700 1 Processor -- start 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.327+0000 7f0303280700 1 -- start start 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.327+0000 7f0303280700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:30.972 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.327+0000 7f0303280700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc19a8b0 con 0x7f02fc1048c0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.327+0000 7f030101c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.327+0000 7f030101c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc19a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34228/0 (socket says 172.21.15.96:34228) 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.327+0000 7f030101c700 1 -- 172.21.15.96:0/3453533627 learned_addr learned my addr 172.21.15.96:0/3453533627 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.328+0000 7f030101c700 1 -- 172.21.15.96:0/3453533627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec009550 con 0x7f02fc1048c0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.328+0000 7f030101c700 1 --2- 172.21.15.96:0/3453533627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc19a370 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f02ec011710 tx=0x7f02ec0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.328+0000 7f02f27fc700 1 -- 172.21.15.96:0/3453533627 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ec003b10 con 0x7f02fc1048c0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.328+0000 7f02f27fc700 1 -- 172.21.15.96:0/3453533627 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f02ec01b8c0 con 0x7f02fc1048c0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.328+0000 7f0303280700 1 -- 172.21.15.96:0/3453533627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02fc19aab0 con 0x7f02fc1048c0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.328+0000 7f02f27fc700 1 -- 172.21.15.96:0/3453533627 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ec019710 con 0x7f02fc1048c0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.328+0000 7f0303280700 1 -- 172.21.15.96:0/3453533627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02fc19af50 con 0x7f02fc1048c0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.328+0000 7f02f27fc700 1 -- 172.21.15.96:0/3453533627 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f02ec019be0 con 0x7f02fc1048c0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.328+0000 7f02f27fc700 1 -- 172.21.15.96:0/3453533627 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f02ec01a710 con 0x7f02fc1048c0 2024-04-22T15:08:30.973 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.329+0000 7f0303280700 1 -- 172.21.15.96:0/3453533627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02fc062510 con 0x7f02fc1048c0 2024-04-22T15:08:30.974 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.331+0000 7f02f27fc700 1 -- 172.21.15.96:0/3453533627 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f02ec01ba30 con 0x7f02fc1048c0 2024-04-22T15:08:30.974 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.397+0000 7f0303280700 1 -- 172.21.15.96:0/3453533627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f02fc19d840 con 0x7f02fc1048c0 2024-04-22T15:08:30.974 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.399+0000 7f02f27fc700 1 -- 172.21.15.96:0/3453533627 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f02ec0300c0 con 0x7f02fc1048c0 2024-04-22T15:08:30.974 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.400+0000 7f0303280700 1 -- 172.21.15.96:0/3453533627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 msgr2=0x7f02fc19a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:30.974 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.400+0000 7f0303280700 1 --2- 172.21.15.96:0/3453533627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc19a370 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f02ec011710 tx=0x7f02ec0047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:30.974 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.400+0000 7f0303280700 1 -- 172.21.15.96:0/3453533627 shutdown_connections 2024-04-22T15:08:30.974 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.400+0000 7f0303280700 1 --2- 172.21.15.96:0/3453533627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02fc1048c0 0x7f02fc19a370 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:30.974 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.400+0000 7f0303280700 1 -- 172.21.15.96:0/3453533627 >> 172.21.15.96:0/3453533627 conn(0x7f02fc1001f0 msgr2=0x7f02fc100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:30.974 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.400+0000 7f0303280700 1 -- 172.21.15.96:0/3453533627 shutdown_connections 2024-04-22T15:08:30.975 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:30.400+0000 7f0303280700 1 -- 172.21.15.96:0/3453533627 wait complete. 2024-04-22T15:08:30.975 INFO:teuthology.orchestra.run.smithi096.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-22T15:08:30.988 INFO:teuthology.orchestra.run.smithi096.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T15:08:30.988 INFO:teuthology.orchestra.run.smithi096.stdout:Creating mgr... 2024-04-22T15:08:30.989 INFO:teuthology.orchestra.run.smithi096.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-22T15:08:30.989 INFO:teuthology.orchestra.run.smithi096.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-22T15:08:30.990 INFO:teuthology.orchestra.run.smithi096.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-22T15:08:31.240 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mgr.smithi096.hhkvum 2024-04-22T15:08:31.240 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mgr.smithi096.hhkvum.service: Unit ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mgr.smithi096.hhkvum.service not loaded. 2024-04-22T15:08:31.390 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d5e86090-00b9-11ef-bc93-c7b262605968.target.wants/ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mgr.smithi096.hhkvum.service → /etc/systemd/system/ceph-d5e86090-00b9-11ef-bc93-c7b262605968@.service. 2024-04-22T15:08:31.632 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:31 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3453533627' entity='client.admin' 2024-04-22T15:08:31.874 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T15:08:31.875 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout disabled 2024-04-22T15:08:31.883 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T15:08:31.883 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout inactive 2024-04-22T15:08:31.883 INFO:teuthology.orchestra.run.smithi096.stdout:firewalld.service is not enabled 2024-04-22T15:08:31.883 INFO:teuthology.orchestra.run.smithi096.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T15:08:31.891 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T15:08:31.891 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout disabled 2024-04-22T15:08:31.899 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T15:08:31.899 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout inactive 2024-04-22T15:08:31.899 INFO:teuthology.orchestra.run.smithi096.stdout:firewalld.service is not enabled 2024-04-22T15:08:31.899 INFO:teuthology.orchestra.run.smithi096.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-22T15:08:31.899 INFO:teuthology.orchestra.run.smithi096.stdout:Waiting for mgr to start... 2024-04-22T15:08:31.899 INFO:teuthology.orchestra.run.smithi096.stdout:Waiting for mgr... 2024-04-22T15:08:32.597 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:32 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1026535886' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T15:08:33.097 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "fsid": "d5e86090-00b9-11ef-bc93-c7b262605968", 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 0 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T15:08:33.098 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "smithi096" 2024-04-22T15:08:33.099 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:33.103 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-22T15:08:33.103 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T15:08:33.103 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:33.103 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T15:08:33.103 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T15:08:33.104 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:33.105 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T15:08:23.493463+0000", 2024-04-22T15:08:33.106 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.367+0000 7f1a05c35700 1 Processor -- start 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.367+0000 7f1a05c35700 1 -- start start 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.367+0000 7f1a05c35700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a001058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.367+0000 7f1a05c35700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a00105ea0 con 0x7f1a001054b0 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.367+0000 7f19ff7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a001058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.367+0000 7f19ff7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a001058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48202/0 (socket says 172.21.15.96:48202) 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.367+0000 7f19ff7fe700 1 -- 172.21.15.96:0/1264202308 learned_addr learned my addr 172.21.15.96:0/1264202308 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.368+0000 7f19ff7fe700 1 -- 172.21.15.96:0/1264202308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a001066b0 con 0x7f1a001054b0 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.368+0000 7f19ff7fe700 1 --2- 172.21.15.96:0/1264202308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a001058d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f19f00098a0 tx=0x7f19f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b0e0317bdde29a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:33.107 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.368+0000 7f19fe7fc700 1 -- 172.21.15.96:0/1264202308 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19f0004180 con 0x7f1a001054b0 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.368+0000 7f19fe7fc700 1 -- 172.21.15.96:0/1264202308 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f19f00042e0 con 0x7f1a001054b0 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.368+0000 7f19fe7fc700 1 -- 172.21.15.96:0/1264202308 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19f0011a70 con 0x7f1a001054b0 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.369+0000 7f1a05c35700 1 -- 172.21.15.96:0/1264202308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 msgr2=0x7f1a001058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.369+0000 7f1a05c35700 1 --2- 172.21.15.96:0/1264202308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a001058d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f19f00098a0 tx=0x7f19f0009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.369+0000 7f1a05c35700 1 -- 172.21.15.96:0/1264202308 shutdown_connections 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.369+0000 7f1a05c35700 1 --2- 172.21.15.96:0/1264202308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a001058d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.369+0000 7f1a05c35700 1 -- 172.21.15.96:0/1264202308 >> 172.21.15.96:0/1264202308 conn(0x7f1a00100ab0 msgr2=0x7f1a00102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.369+0000 7f1a05c35700 1 -- 172.21.15.96:0/1264202308 shutdown_connections 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.369+0000 7f1a05c35700 1 -- 172.21.15.96:0/1264202308 wait complete. 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.369+0000 7f1a05c35700 1 Processor -- start 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f1a05c35700 1 -- start start 2024-04-22T15:08:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f1a05c35700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a0019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f1a05c35700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0019a870 con 0x7f1a001054b0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f19ff7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a0019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f19ff7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a0019a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48204/0 (socket says 172.21.15.96:48204) 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f19ff7fe700 1 -- 172.21.15.96:0/1026535886 learned_addr learned my addr 172.21.15.96:0/1026535886 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f19ff7fe700 1 -- 172.21.15.96:0/1026535886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f0009550 con 0x7f1a001054b0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f19ff7fe700 1 --2- 172.21.15.96:0/1026535886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a0019a330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f19f0011710 tx=0x7f19f00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f19fcff9700 1 -- 172.21.15.96:0/1026535886 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19f0003b10 con 0x7f1a001054b0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f19fcff9700 1 -- 172.21.15.96:0/1026535886 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f19f001b8c0 con 0x7f1a001054b0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.370+0000 7f1a05c35700 1 -- 172.21.15.96:0/1026535886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0019aa70 con 0x7f1a001054b0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.371+0000 7f19fcff9700 1 -- 172.21.15.96:0/1026535886 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19f0019770 con 0x7f1a001054b0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.371+0000 7f1a05c35700 1 -- 172.21.15.96:0/1026535886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0019af10 con 0x7f1a001054b0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.371+0000 7f19fcff9700 1 -- 172.21.15.96:0/1026535886 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f19f0019d10 con 0x7f1a001054b0 2024-04-22T15:08:33.109 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.371+0000 7f19fcff9700 1 -- 172.21.15.96:0/1026535886 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f19f001a710 con 0x7f1a001054b0 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.372+0000 7f1a05c35700 1 -- 172.21.15.96:0/1026535886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a001944d0 con 0x7f1a001054b0 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.374+0000 7f19fcff9700 1 -- 172.21.15.96:0/1026535886 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f19f0029060 con 0x7f1a001054b0 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.458+0000 7f1a05c35700 1 -- 172.21.15.96:0/1026535886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1a001098e0 con 0x7f1a001054b0 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.458+0000 7f19fcff9700 1 -- 172.21.15.96:0/1026535886 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f19f001f070 con 0x7f1a001054b0 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.460+0000 7f1a05c35700 1 -- 172.21.15.96:0/1026535886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 msgr2=0x7f1a0019a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.460+0000 7f1a05c35700 1 --2- 172.21.15.96:0/1026535886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a0019a330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f19f0011710 tx=0x7f19f00047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.460+0000 7f1a05c35700 1 -- 172.21.15.96:0/1026535886 shutdown_connections 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.460+0000 7f1a05c35700 1 --2- 172.21.15.96:0/1026535886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a001054b0 0x7f1a0019a330 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.460+0000 7f1a05c35700 1 -- 172.21.15.96:0/1026535886 >> 172.21.15.96:0/1026535886 conn(0x7f1a00100ab0 msgr2=0x7f1a00101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.460+0000 7f1a05c35700 1 -- 172.21.15.96:0/1026535886 shutdown_connections 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:32.460+0000 7f1a05c35700 1 -- 172.21.15.96:0/1026535886 wait complete. 2024-04-22T15:08:33.110 INFO:teuthology.orchestra.run.smithi096.stdout:mgr not available, waiting (1/15)... 2024-04-22T15:08:35.953 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:35 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2204645943' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T15:08:36.279 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:36.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:08:36.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "fsid": "d5e86090-00b9-11ef-bc93-c7b262605968", 2024-04-22T15:08:36.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T15:08:36.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T15:08:36.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T15:08:36.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T15:08:36.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:36.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 0 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "smithi096" 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T15:08:36.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T15:08:36.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:36.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T15:08:36.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:36.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T15:08:36.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T15:08:36.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T15:08:36.291 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T15:08:36.291 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T15:08:36.291 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T15:08:36.291 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:36.291 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T15:08:36.291 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T15:08:36.291 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T15:08:36.291 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T15:08:36.291 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:36.292 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T15:08:36.293 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T15:08:23.493463+0000", 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.558+0000 7fc5214c3700 1 Processor -- start 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.558+0000 7fc5214c3700 1 -- start start 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.558+0000 7fc5214c3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c109470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.558+0000 7fc5214c3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c074750 con 0x7fc51c107080 2024-04-22T15:08:36.294 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.558+0000 7fc51affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c109470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.558+0000 7fc51affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c109470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48218/0 (socket says 172.21.15.96:48218) 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.558+0000 7fc51affd700 1 -- 172.21.15.96:0/735413765 learned_addr learned my addr 172.21.15.96:0/735413765 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.559+0000 7fc51affd700 1 -- 172.21.15.96:0/735413765 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c1099b0 con 0x7fc51c107080 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.559+0000 7fc51affd700 1 --2- 172.21.15.96:0/735413765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c109470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0098a0 tx=0x7fc50c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4ab5a1868e3ed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.559+0000 7fc519ffb700 1 -- 172.21.15.96:0/735413765 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc50c004180 con 0x7fc51c107080 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.559+0000 7fc519ffb700 1 -- 172.21.15.96:0/735413765 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc50c0042e0 con 0x7fc51c107080 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.559+0000 7fc519ffb700 1 -- 172.21.15.96:0/735413765 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc50c011a70 con 0x7fc51c107080 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.560+0000 7fc5214c3700 1 -- 172.21.15.96:0/735413765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 msgr2=0x7fc51c109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.560+0000 7fc5214c3700 1 --2- 172.21.15.96:0/735413765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c109470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0098a0 tx=0x7fc50c009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.560+0000 7fc5214c3700 1 -- 172.21.15.96:0/735413765 shutdown_connections 2024-04-22T15:08:36.295 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.560+0000 7fc5214c3700 1 --2- 172.21.15.96:0/735413765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c109470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.560+0000 7fc5214c3700 1 -- 172.21.15.96:0/735413765 >> 172.21.15.96:0/735413765 conn(0x7fc51c100a50 msgr2=0x7fc51c102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.560+0000 7fc5214c3700 1 -- 172.21.15.96:0/735413765 shutdown_connections 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.560+0000 7fc5214c3700 1 -- 172.21.15.96:0/735413765 wait complete. 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.560+0000 7fc5214c3700 1 Processor -- start 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.560+0000 7fc5214c3700 1 -- start start 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc5214c3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc5214c3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c19a820 con 0x7fc51c107080 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc51affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc51affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48230/0 (socket says 172.21.15.96:48230) 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc51affd700 1 -- 172.21.15.96:0/2204645943 learned_addr learned my addr 172.21.15.96:0/2204645943 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc51affd700 1 -- 172.21.15.96:0/2204645943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c009550 con 0x7fc51c107080 2024-04-22T15:08:36.296 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc51affd700 1 --2- 172.21.15.96:0/2204645943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c19a2e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc50c011710 tx=0x7fc50c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc503fff700 1 -- 172.21.15.96:0/2204645943 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc50c003b10 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc503fff700 1 -- 172.21.15.96:0/2204645943 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc50c01b8c0 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc5214c3700 1 -- 172.21.15.96:0/2204645943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51c19aa20 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc503fff700 1 -- 172.21.15.96:0/2204645943 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc50c0197e0 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc5214c3700 1 -- 172.21.15.96:0/2204645943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51c19aec0 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc503fff700 1 -- 172.21.15.96:0/2204645943 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fc50c019d20 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.561+0000 7fc503fff700 1 -- 172.21.15.96:0/2204645943 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc50c01a750 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.562+0000 7fc5214c3700 1 -- 172.21.15.96:0/2204645943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51c10a6d0 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.564+0000 7fc503fff700 1 -- 172.21.15.96:0/2204645943 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fc50c029060 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.637+0000 7fc5214c3700 1 -- 172.21.15.96:0/2204645943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc51c10a890 con 0x7fc51c107080 2024-04-22T15:08:36.297 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.637+0000 7fc503fff700 1 -- 172.21.15.96:0/2204645943 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fc50c01f070 con 0x7fc51c107080 2024-04-22T15:08:36.298 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.638+0000 7fc5214c3700 1 -- 172.21.15.96:0/2204645943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 msgr2=0x7fc51c19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:36.298 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.638+0000 7fc5214c3700 1 --2- 172.21.15.96:0/2204645943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c19a2e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc50c011710 tx=0x7fc50c0047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:36.298 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.638+0000 7fc5214c3700 1 -- 172.21.15.96:0/2204645943 shutdown_connections 2024-04-22T15:08:36.298 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.638+0000 7fc5214c3700 1 --2- 172.21.15.96:0/2204645943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc51c107080 0x7fc51c19a2e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:36.298 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.638+0000 7fc5214c3700 1 -- 172.21.15.96:0/2204645943 >> 172.21.15.96:0/2204645943 conn(0x7fc51c100a50 msgr2=0x7fc51c101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:36.298 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.639+0000 7fc5214c3700 1 -- 172.21.15.96:0/2204645943 shutdown_connections 2024-04-22T15:08:36.298 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:35.639+0000 7fc5214c3700 1 -- 172.21.15.96:0/2204645943 wait complete. 2024-04-22T15:08:36.298 INFO:teuthology.orchestra.run.smithi096.stdout:mgr not available, waiting (2/15)... 2024-04-22T15:08:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: Activating manager daemon smithi096.hhkvum 2024-04-22T15:08:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: mgrmap e2: smithi096.hhkvum(active, starting, since 0.00286689s) 2024-04-22T15:08:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:08:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:08:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:08:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:08:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:08:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: Manager daemon smithi096.hhkvum is now available 2024-04-22T15:08:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:08:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/trash_purge_schedule"}]: dispatch 2024-04-22T15:08:38.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:38.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:38.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:38 smithi096 ceph-mon[26688]: from='mgr.14100 172.21.15.96:0/3508178699' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:39.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:39.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:08:39.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "fsid": "d5e86090-00b9-11ef-bc93-c7b262605968", 2024-04-22T15:08:39.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T15:08:39.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T15:08:39.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T15:08:39.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 0 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "smithi096" 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:39.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T15:08:39.350 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:39.350 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T15:08:39.350 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T15:08:39.350 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T15:08:39.358 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T15:08:39.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T15:08:23.493463+0000", 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.655+0000 7fa1f1947700 1 Processor -- start 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.655+0000 7fa1f1947700 1 -- start start 2024-04-22T15:08:39.360 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.655+0000 7fa1f1947700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec109440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.655+0000 7fa1f1947700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec079ed0 con 0x7fa1ec107050 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.655+0000 7fa1eaffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec109440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.655+0000 7fa1eaffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec109440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48316/0 (socket says 172.21.15.96:48316) 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.655+0000 7fa1eaffd700 1 -- 172.21.15.96:0/1200898945 learned_addr learned my addr 172.21.15.96:0/1200898945 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.656+0000 7fa1eaffd700 1 -- 172.21.15.96:0/1200898945 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec109980 con 0x7fa1ec107050 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.656+0000 7fa1eaffd700 1 --2- 172.21.15.96:0/1200898945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec109440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0098a0 tx=0x7fa1dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f13dc3b4570cf4a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.656+0000 7fa1e9ffb700 1 -- 172.21.15.96:0/1200898945 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1dc004180 con 0x7fa1ec107050 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.656+0000 7fa1e9ffb700 1 -- 172.21.15.96:0/1200898945 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa1dc0042e0 con 0x7fa1ec107050 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.656+0000 7fa1e9ffb700 1 -- 172.21.15.96:0/1200898945 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1dc011a70 con 0x7fa1ec107050 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.657+0000 7fa1f1947700 1 -- 172.21.15.96:0/1200898945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 msgr2=0x7fa1ec109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.657+0000 7fa1f1947700 1 --2- 172.21.15.96:0/1200898945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec109440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0098a0 tx=0x7fa1dc009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.657+0000 7fa1f1947700 1 -- 172.21.15.96:0/1200898945 shutdown_connections 2024-04-22T15:08:39.361 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.657+0000 7fa1f1947700 1 --2- 172.21.15.96:0/1200898945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec109440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.657+0000 7fa1f1947700 1 -- 172.21.15.96:0/1200898945 >> 172.21.15.96:0/1200898945 conn(0x7fa1ec100b00 msgr2=0x7fa1ec102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.657+0000 7fa1f1947700 1 -- 172.21.15.96:0/1200898945 shutdown_connections 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.657+0000 7fa1f1947700 1 -- 172.21.15.96:0/1200898945 wait complete. 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.657+0000 7fa1f1947700 1 Processor -- start 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.657+0000 7fa1f1947700 1 -- start start 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1f1947700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1f1947700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec19a820 con 0x7fa1ec107050 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1eaffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1eaffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48320/0 (socket says 172.21.15.96:48320) 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1eaffd700 1 -- 172.21.15.96:0/3087074958 learned_addr learned my addr 172.21.15.96:0/3087074958 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1eaffd700 1 -- 172.21.15.96:0/3087074958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc009550 con 0x7fa1ec107050 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1eaffd700 1 --2- 172.21.15.96:0/3087074958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec19a2e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc011710 tx=0x7fa1dc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1f0945700 1 -- 172.21.15.96:0/3087074958 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1dc003b10 con 0x7fa1ec107050 2024-04-22T15:08:39.362 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1f0945700 1 -- 172.21.15.96:0/3087074958 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa1dc01b8c0 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1f1947700 1 -- 172.21.15.96:0/3087074958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec19aa20 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1f0945700 1 -- 172.21.15.96:0/3087074958 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1dc0197e0 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.658+0000 7fa1f1947700 1 -- 172.21.15.96:0/3087074958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec19aec0 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.659+0000 7fa1f0945700 1 -- 172.21.15.96:0/3087074958 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 2) v1 ==== 45312+0+0 (secure 0 0 0) 0x7fa1dc0199d0 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.659+0000 7fa1f0945700 1 -- 172.21.15.96:0/3087074958 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa1dc04c640 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.659+0000 7fa1f1947700 1 -- 172.21.15.96:0/3087074958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec10a700 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.662+0000 7fa1f0945700 1 -- 172.21.15.96:0/3087074958 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fa1dc01f070 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.731+0000 7fa1f1947700 1 -- 172.21.15.96:0/3087074958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa1ec10a8f0 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.732+0000 7fa1f0945700 1 -- 172.21.15.96:0/3087074958 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fa1dc01a3d0 con 0x7fa1ec107050 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.732+0000 7fa1f1947700 1 -- 172.21.15.96:0/3087074958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 msgr2=0x7fa1ec19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.732+0000 7fa1f1947700 1 --2- 172.21.15.96:0/3087074958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec19a2e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc011710 tx=0x7fa1dc0047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:39.363 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.732+0000 7fa1f1947700 1 -- 172.21.15.96:0/3087074958 shutdown_connections 2024-04-22T15:08:39.364 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.732+0000 7fa1f1947700 1 --2- 172.21.15.96:0/3087074958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec107050 0x7fa1ec19a2e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:39.364 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.732+0000 7fa1f1947700 1 -- 172.21.15.96:0/3087074958 >> 172.21.15.96:0/3087074958 conn(0x7fa1ec100b00 msgr2=0x7fa1ec1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:39.364 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.732+0000 7fa1f1947700 1 -- 172.21.15.96:0/3087074958 shutdown_connections 2024-04-22T15:08:39.364 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:38.733+0000 7fa1f1947700 1 -- 172.21.15.96:0/3087074958 wait complete. 2024-04-22T15:08:39.364 INFO:teuthology.orchestra.run.smithi096.stdout:mgr not available, waiting (3/15)... 2024-04-22T15:08:39.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:39 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3087074958' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T15:08:39.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:39 smithi096 ceph-mon[26688]: mgrmap e3: smithi096.hhkvum(active, since 1.00659s) 2024-04-22T15:08:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:41 smithi096 ceph-mon[26688]: mgrmap e4: smithi096.hhkvum(active, since 2s) 2024-04-22T15:08:42.617 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:42.618 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:08:42.618 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "fsid": "d5e86090-00b9-11ef-bc93-c7b262605968", 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 0 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "smithi096" 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T15:08:42.619 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:42.620 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T15:08:42.620 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T15:08:42.620 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:42.620 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T15:08:42.620 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:42.620 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T15:08:42.620 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T15:08:42.620 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T15:08:42.628 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:42.629 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:08:42.630 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T15:08:23.493463+0000", 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.782+0000 7fab320b0700 1 Processor -- start 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.782+0000 7fab320b0700 1 -- start start 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab320b0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab320b0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c107480 con 0x7fab2c106a90 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab2b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab2b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54708/0 (socket says 172.21.15.96:54708) 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab2b7fe700 1 -- 172.21.15.96:0/325455920 learned_addr learned my addr 172.21.15.96:0/325455920 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab2b7fe700 1 -- 172.21.15.96:0/325455920 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c1056f0 con 0x7fab2c106a90 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab2b7fe700 1 --2- 172.21.15.96:0/325455920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c106eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fab200098a0 tx=0x7fab20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd10d0bec1ec202 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab2a7fc700 1 -- 172.21.15.96:0/325455920 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab20004180 con 0x7fab2c106a90 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab2a7fc700 1 -- 172.21.15.96:0/325455920 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fab200042e0 con 0x7fab2c106a90 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.783+0000 7fab2a7fc700 1 -- 172.21.15.96:0/325455920 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab20011a70 con 0x7fab2c106a90 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.784+0000 7fab320b0700 1 -- 172.21.15.96:0/325455920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 msgr2=0x7fab2c106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.784+0000 7fab320b0700 1 --2- 172.21.15.96:0/325455920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c106eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fab200098a0 tx=0x7fab20009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.784+0000 7fab320b0700 1 -- 172.21.15.96:0/325455920 shutdown_connections 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.784+0000 7fab320b0700 1 --2- 172.21.15.96:0/325455920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c106eb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.784+0000 7fab320b0700 1 -- 172.21.15.96:0/325455920 >> 172.21.15.96:0/325455920 conn(0x7fab2c1001c0 msgr2=0x7fab2c102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.784+0000 7fab320b0700 1 -- 172.21.15.96:0/325455920 shutdown_connections 2024-04-22T15:08:42.632 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.784+0000 7fab320b0700 1 -- 172.21.15.96:0/325455920 wait complete. 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.785+0000 7fab320b0700 1 Processor -- start 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.785+0000 7fab320b0700 1 -- start start 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.785+0000 7fab320b0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c19a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.785+0000 7fab320b0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c19a980 con 0x7fab2c106a90 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.785+0000 7fab2b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c19a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.785+0000 7fab2b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c19a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54714/0 (socket says 172.21.15.96:54714) 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.785+0000 7fab2b7fe700 1 -- 172.21.15.96:0/1055344467 learned_addr learned my addr 172.21.15.96:0/1055344467 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.785+0000 7fab2b7fe700 1 -- 172.21.15.96:0/1055344467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab20009550 con 0x7fab2c106a90 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.785+0000 7fab2b7fe700 1 --2- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c19a440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fab20003df0 tx=0x7fab200047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.786+0000 7fab28ff9700 1 -- 172.21.15.96:0/1055344467 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab200038b0 con 0x7fab2c106a90 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.786+0000 7fab28ff9700 1 -- 172.21.15.96:0/1055344467 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fab20003a10 con 0x7fab2c106a90 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.786+0000 7fab28ff9700 1 -- 172.21.15.96:0/1055344467 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab20019600 con 0x7fab2c106a90 2024-04-22T15:08:42.633 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.786+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c19ab80 con 0x7fab2c106a90 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.786+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c19b020 con 0x7fab2c106a90 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.786+0000 7fab28ff9700 1 -- 172.21.15.96:0/1055344467 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 4) v1 ==== 45545+0+0 (secure 0 0 0) 0x7fab200197c0 con 0x7fab2c106a90 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.787+0000 7fab28ff9700 1 --2- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fab14038720 0x7fab1403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.787+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2c1944f0 con 0x7fab2c106a90 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.787+0000 7fab28ff9700 1 -- 172.21.15.96:0/1055344467 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fab2004c340 con 0x7fab2c106a90 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.787+0000 7fab2affd700 1 --2- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fab14038720 0x7fab1403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.787+0000 7fab2affd700 1 --2- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fab14038720 0x7fab1403abe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fab1c006fd0 tx=0x7fab1c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:41.792+0000 7fab28ff9700 1 -- 172.21.15.96:0/1055344467 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fab20019a70 con 0x7fab2c106a90 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.061+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fab2c109930 con 0x7fab2c106a90 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.061+0000 7fab28ff9700 1 -- 172.21.15.96:0/1055344467 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fab20010380 con 0x7fab2c106a90 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fab14038720 msgr2=0x7fab1403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 --2- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fab14038720 0x7fab1403abe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fab1c006fd0 tx=0x7fab1c006e40 comp rx=0 tx=0).stop 2024-04-22T15:08:42.634 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 msgr2=0x7fab2c19a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:42.635 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 --2- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c19a440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fab20003df0 tx=0x7fab200047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:42.635 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 shutdown_connections 2024-04-22T15:08:42.635 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 --2- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fab14038720 0x7fab1403abe0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:42.635 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 --2- 172.21.15.96:0/1055344467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c106a90 0x7fab2c19a440 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:42.635 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 >> 172.21.15.96:0/1055344467 conn(0x7fab2c1001c0 msgr2=0x7fab2c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:42.635 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 shutdown_connections 2024-04-22T15:08:42.635 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:42.063+0000 7fab320b0700 1 -- 172.21.15.96:0/1055344467 wait complete. 2024-04-22T15:08:42.635 INFO:teuthology.orchestra.run.smithi096.stdout:mgr is available 2024-04-22T15:08:42.744 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:42 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1055344467' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T15:08:43.802 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:43 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/487443216' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T15:08:43.811 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:43.811 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T15:08:43.811 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout fsid = d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T15:08:43.812 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.047+0000 7f6d24ff9700 1 Processor -- start 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.048+0000 7f6d24ff9700 1 -- start start 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.048+0000 7f6d24ff9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2007a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.048+0000 7f6d24ff9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2007a910 con 0x7f6d2007be90 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.048+0000 7f6d1e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2007a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.048+0000 7f6d1e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2007a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54724/0 (socket says 172.21.15.96:54724) 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.048+0000 7f6d1e59c700 1 -- 172.21.15.96:0/1094208279 learned_addr learned my addr 172.21.15.96:0/1094208279 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.048+0000 7f6d1e59c700 1 -- 172.21.15.96:0/1094208279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2007aa50 con 0x7f6d2007be90 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d1e59c700 1 --2- 172.21.15.96:0/1094208279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2007a340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d100098a0 tx=0x7f6d10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49ab48bff1eab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d1d59a700 1 -- 172.21.15.96:0/1094208279 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d10004180 con 0x7f6d2007be90 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d1d59a700 1 -- 172.21.15.96:0/1094208279 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6d100042e0 con 0x7f6d2007be90 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d1d59a700 1 -- 172.21.15.96:0/1094208279 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d10011a70 con 0x7f6d2007be90 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d24ff9700 1 -- 172.21.15.96:0/1094208279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 msgr2=0x7f6d2007a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:43.813 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d24ff9700 1 --2- 172.21.15.96:0/1094208279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2007a340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d100098a0 tx=0x7f6d10009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d24ff9700 1 -- 172.21.15.96:0/1094208279 shutdown_connections 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d24ff9700 1 --2- 172.21.15.96:0/1094208279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2007a340 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d24ff9700 1 -- 172.21.15.96:0/1094208279 >> 172.21.15.96:0/1094208279 conn(0x7f6d20101bd0 msgr2=0x7f6d20103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d24ff9700 1 -- 172.21.15.96:0/1094208279 shutdown_connections 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.049+0000 7f6d24ff9700 1 -- 172.21.15.96:0/1094208279 wait complete. 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.050+0000 7f6d24ff9700 1 Processor -- start 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.050+0000 7f6d24ff9700 1 -- start start 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.050+0000 7f6d24ff9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.050+0000 7f6d24ff9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2019ca40 con 0x7f6d2007be90 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.050+0000 7f6d1e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.050+0000 7f6d1e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54732/0 (socket says 172.21.15.96:54732) 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.050+0000 7f6d1e59c700 1 -- 172.21.15.96:0/487443216 learned_addr learned my addr 172.21.15.96:0/487443216 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.051+0000 7f6d1e59c700 1 -- 172.21.15.96:0/487443216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10009550 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.051+0000 7f6d1e59c700 1 --2- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2019c500 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6d10011710 tx=0x7f6d100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.051+0000 7f6d0f7fe700 1 -- 172.21.15.96:0/487443216 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d10003b10 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.051+0000 7f6d0f7fe700 1 -- 172.21.15.96:0/487443216 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6d1001b8c0 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.051+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2019cc40 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.051+0000 7f6d0f7fe700 1 -- 172.21.15.96:0/487443216 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d100197e0 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.051+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2019d0e0 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.052+0000 7f6d0f7fe700 1 -- 172.21.15.96:0/487443216 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 4) v1 ==== 45545+0+0 (secure 0 0 0) 0x7f6d100199d0 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.052+0000 7f6d0f7fe700 1 --2- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f6d08038720 0x7f6d0803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.052+0000 7f6d0f7fe700 1 -- 172.21.15.96:0/487443216 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6d1004ca20 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.052+0000 7f6d1dd9b700 1 --2- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f6d08038720 0x7f6d0803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.052+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d20062510 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.052+0000 7f6d1dd9b700 1 --2- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f6d08038720 0x7f6d0803abe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006fd0 tx=0x7f6d14006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.057+0000 7f6d0f7fe700 1 -- 172.21.15.96:0/487443216 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f6d1001f020 con 0x7f6d2007be90 2024-04-22T15:08:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.246+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6d2010c8d0 con 0x7f6d2007be90 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.247+0000 7f6d0f7fe700 1 -- 172.21.15.96:0/487443216 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f6d1001a3d0 con 0x7f6d2007be90 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f6d08038720 msgr2=0x7f6d0803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 --2- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f6d08038720 0x7f6d0803abe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006fd0 tx=0x7f6d14006e40 comp rx=0 tx=0).stop 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 msgr2=0x7f6d2019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 --2- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2019c500 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6d10011710 tx=0x7f6d100047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 shutdown_connections 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 --2- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f6d08038720 0x7f6d0803abe0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 --2- 172.21.15.96:0/487443216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d2007be90 0x7f6d2019c500 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 >> 172.21.15.96:0/487443216 conn(0x7f6d20101bd0 msgr2=0x7f6d20102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 shutdown_connections 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:43.249+0000 7f6d24ff9700 1 -- 172.21.15.96:0/487443216 wait complete. 2024-04-22T15:08:43.816 INFO:teuthology.orchestra.run.smithi096.stdout:Enabling cephadm module... 2024-04-22T15:08:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:44 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3120834890' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T15:08:45.268 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.224+0000 7fe5ee96d700 1 Processor -- start 2024-04-22T15:08:45.268 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.224+0000 7fe5ee96d700 1 -- start start 2024-04-22T15:08:45.268 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.224+0000 7fe5ee96d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8109460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:45.268 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.224+0000 7fe5ee96d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8079e70 con 0x7fe5e8107070 2024-04-22T15:08:45.268 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.224+0000 7fe5e7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8109460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:45.268 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.225+0000 7fe5e7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8109460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54740/0 (socket says 172.21.15.96:54740) 2024-04-22T15:08:45.268 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.225+0000 7fe5e7fff700 1 -- 172.21.15.96:0/2095388696 learned_addr learned my addr 172.21.15.96:0/2095388696 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:45.268 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.225+0000 7fe5e7fff700 1 -- 172.21.15.96:0/2095388696 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e81099a0 con 0x7fe5e8107070 2024-04-22T15:08:45.268 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.225+0000 7fe5e7fff700 1 --2- 172.21.15.96:0/2095388696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8109460 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80098a0 tx=0x7fe5d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7637f58cd2e643 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.225+0000 7fe5e6ffd700 1 -- 172.21.15.96:0/2095388696 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5d8004180 con 0x7fe5e8107070 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.225+0000 7fe5e6ffd700 1 -- 172.21.15.96:0/2095388696 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe5d80042e0 con 0x7fe5e8107070 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.225+0000 7fe5e6ffd700 1 -- 172.21.15.96:0/2095388696 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5d8011a70 con 0x7fe5e8107070 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.226+0000 7fe5ee96d700 1 -- 172.21.15.96:0/2095388696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 msgr2=0x7fe5e8109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.226+0000 7fe5ee96d700 1 --2- 172.21.15.96:0/2095388696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8109460 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80098a0 tx=0x7fe5d8009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.226+0000 7fe5ee96d700 1 -- 172.21.15.96:0/2095388696 shutdown_connections 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.226+0000 7fe5ee96d700 1 --2- 172.21.15.96:0/2095388696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8109460 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.226+0000 7fe5ee96d700 1 -- 172.21.15.96:0/2095388696 >> 172.21.15.96:0/2095388696 conn(0x7fe5e8100aa0 msgr2=0x7fe5e8102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.226+0000 7fe5ee96d700 1 -- 172.21.15.96:0/2095388696 shutdown_connections 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.226+0000 7fe5ee96d700 1 -- 172.21.15.96:0/2095388696 wait complete. 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.226+0000 7fe5ee96d700 1 Processor -- start 2024-04-22T15:08:45.269 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.226+0000 7fe5ee96d700 1 -- start start 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5ee96d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8106850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5ee96d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8106d90 con 0x7fe5e8107070 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5e7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5e7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54742/0 (socket says 172.21.15.96:54742) 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5e7fff700 1 -- 172.21.15.96:0/3120834890 learned_addr learned my addr 172.21.15.96:0/3120834890 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5e7fff700 1 -- 172.21.15.96:0/3120834890 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8009550 con 0x7fe5e8107070 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5e7fff700 1 --2- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8106850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8011a20 tx=0x7fe5d80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5e57fa700 1 -- 172.21.15.96:0/3120834890 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5d8003b10 con 0x7fe5e8107070 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5e57fa700 1 -- 172.21.15.96:0/3120834890 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe5d801b8c0 con 0x7fe5e8107070 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e8105020 con 0x7fe5e8107070 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5e57fa700 1 -- 172.21.15.96:0/3120834890 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5d80237b0 con 0x7fe5e8107070 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.227+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e81054c0 con 0x7fe5e8107070 2024-04-22T15:08:45.270 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.228+0000 7fe5e57fa700 1 -- 172.21.15.96:0/3120834890 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 4) v1 ==== 45545+0+0 (secure 0 0 0) 0x7fe5d80239e0 con 0x7fe5e8107070 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.228+0000 7fe5e57fa700 1 --2- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fe5d00386e0 0x7fe5d003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.228+0000 7fe5e57fa700 1 -- 172.21.15.96:0/3120834890 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe5d804d960 con 0x7fe5e8107070 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.228+0000 7fe5e77fe700 1 --2- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fe5d00386e0 0x7fe5d003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.228+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e8106ed0 con 0x7fe5e8107070 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.229+0000 7fe5e77fe700 1 --2- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fe5d00386e0 0x7fe5d003aba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.233+0000 7fe5e57fa700 1 -- 172.21.15.96:0/3120834890 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fe5e8106ed0 con 0x7fe5e8107070 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.480+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fe5e8106ed0 con 0x7fe5e8107070 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.599+0000 7fe5e57fa700 1 -- 172.21.15.96:0/3120834890 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 5) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fe5d804c080 con 0x7fe5e8107070 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.600+0000 7fe5e57fa700 1 -- 172.21.15.96:0/3120834890 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe5d8016260 con 0x7fe5e8107070 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fe5d00386e0 msgr2=0x7fe5d003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 --2- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fe5d00386e0 0x7fe5d003aba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc006e40 comp rx=0 tx=0).stop 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 msgr2=0x7fe5e8106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:45.271 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 --2- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8106850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8011a20 tx=0x7fe5d80047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:45.274 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 shutdown_connections 2024-04-22T15:08:45.274 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 --2- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fe5d00386e0 0x7fe5d003aba0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:45.274 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 --2- 172.21.15.96:0/3120834890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e8107070 0x7fe5e8106850 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:45.274 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 >> 172.21.15.96:0/3120834890 conn(0x7fe5e8100aa0 msgr2=0x7fe5e81015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:45.274 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 shutdown_connections 2024-04-22T15:08:45.274 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:44.603+0000 7fe5ee96d700 1 -- 172.21.15.96:0/3120834890 wait complete. 2024-04-22T15:08:45.695 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:45 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3120834890' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T15:08:45.695 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:45 smithi096 ceph-mon[26688]: mgrmap e5: smithi096.hhkvum(active, since 6s) 2024-04-22T15:08:46.574 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:08:46.574 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-22T15:08:46.575 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T15:08:46.575 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "active_name": "smithi096.hhkvum", 2024-04-22T15:08:46.575 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T15:08:46.575 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:08:46.575 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.707+0000 7fb1b71a7700 1 Processor -- start 2024-04-22T15:08:46.575 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.707+0000 7fb1b71a7700 1 -- start start 2024-04-22T15:08:46.575 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.707+0000 7fb1b71a7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b0109450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:46.575 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.707+0000 7fb1b71a7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b0079e90 con 0x7fb1b0107060 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.707+0000 7fb1b4f43700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.707+0000 7fb1b4f43700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54764/0 (socket says 172.21.15.96:54764) 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.707+0000 7fb1b4f43700 1 -- 172.21.15.96:0/1955238407 learned_addr learned my addr 172.21.15.96:0/1955238407 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.708+0000 7fb1b4f43700 1 -- 172.21.15.96:0/1955238407 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b0109990 con 0x7fb1b0107060 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.708+0000 7fb1b4f43700 1 --2- 172.21.15.96:0/1955238407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b0109450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40098a0 tx=0x7fb1a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97d44d13870eb1c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.708+0000 7fb1af7fe700 1 -- 172.21.15.96:0/1955238407 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1a4004180 con 0x7fb1b0107060 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.708+0000 7fb1af7fe700 1 -- 172.21.15.96:0/1955238407 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb1a40042e0 con 0x7fb1b0107060 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.708+0000 7fb1af7fe700 1 -- 172.21.15.96:0/1955238407 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1a4011a70 con 0x7fb1b0107060 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.709+0000 7fb1b71a7700 1 -- 172.21.15.96:0/1955238407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 msgr2=0x7fb1b0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.709+0000 7fb1b71a7700 1 --2- 172.21.15.96:0/1955238407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b0109450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40098a0 tx=0x7fb1a4009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.709+0000 7fb1b71a7700 1 -- 172.21.15.96:0/1955238407 shutdown_connections 2024-04-22T15:08:46.576 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.709+0000 7fb1b71a7700 1 --2- 172.21.15.96:0/1955238407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b0109450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.709+0000 7fb1b71a7700 1 -- 172.21.15.96:0/1955238407 >> 172.21.15.96:0/1955238407 conn(0x7fb1b0100ab0 msgr2=0x7fb1b0102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.709+0000 7fb1b71a7700 1 -- 172.21.15.96:0/1955238407 shutdown_connections 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.709+0000 7fb1b71a7700 1 -- 172.21.15.96:0/1955238407 wait complete. 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.710+0000 7fb1b71a7700 1 Processor -- start 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.710+0000 7fb1b71a7700 1 -- start start 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.710+0000 7fb1b71a7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.710+0000 7fb1b71a7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b019a850 con 0x7fb1b0107060 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.710+0000 7fb1b4f43700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.710+0000 7fb1b4f43700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b019a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54774/0 (socket says 172.21.15.96:54774) 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.710+0000 7fb1b4f43700 1 -- 172.21.15.96:0/477863525 learned_addr learned my addr 172.21.15.96:0/477863525 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.710+0000 7fb1b4f43700 1 -- 172.21.15.96:0/477863525 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a4009550 con 0x7fb1b0107060 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.711+0000 7fb1b4f43700 1 --2- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b019a310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4011a20 tx=0x7fb1a40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.711+0000 7fb1adffb700 1 -- 172.21.15.96:0/477863525 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1a4003b10 con 0x7fb1b0107060 2024-04-22T15:08:46.577 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.711+0000 7fb1adffb700 1 -- 172.21.15.96:0/477863525 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb1a401b8c0 con 0x7fb1b0107060 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.711+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b019aa50 con 0x7fb1b0107060 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.711+0000 7fb1adffb700 1 -- 172.21.15.96:0/477863525 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1a40237b0 con 0x7fb1b0107060 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.711+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b019aef0 con 0x7fb1b0107060 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.712+0000 7fb1adffb700 1 -- 172.21.15.96:0/477863525 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fb1a40239e0 con 0x7fb1b0107060 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.712+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b010a6e0 con 0x7fb1b0107060 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.712+0000 7fb1adffb700 1 --2- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fb1a0038730 0x7fb1a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.712+0000 7fb1adffb700 1 -- 172.21.15.96:0/477863525 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb1a404ca20 con 0x7fb1b0107060 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.712+0000 7fb1affff700 1 -- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fb1a0038730 msgr2=0x7fb1a003abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/2684869170 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.712+0000 7fb1affff700 1 --2- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fb1a0038730 0x7fb1a003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.717+0000 7fb1adffb700 1 -- 172.21.15.96:0/477863525 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb1a4023c90 con 0x7fb1b0107060 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.913+0000 7fb1affff700 1 -- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fb1a0038730 msgr2=0x7fb1a003abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/2684869170 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:45.913+0000 7fb1affff700 1 --2- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fb1a0038730 0x7fb1a003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.006+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb1b019d7e0 con 0x7fb1b0107060 2024-04-22T15:08:46.578 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.006+0000 7fb1adffb700 1 -- 172.21.15.96:0/477863525 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fb1a402bb40 con 0x7fb1b0107060 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fb1a0038730 msgr2=0x7fb1a003abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 --2- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fb1a0038730 0x7fb1a003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 msgr2=0x7fb1b019a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 --2- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b019a310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4011a20 tx=0x7fb1a40047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 shutdown_connections 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 --2- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7fb1a0038730 0x7fb1a003abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 --2- 172.21.15.96:0/477863525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1b0107060 0x7fb1b019a310 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 >> 172.21.15.96:0/477863525 conn(0x7fb1b0100ab0 msgr2=0x7fb1b01015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 shutdown_connections 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:46.008+0000 7fb1b71a7700 1 -- 172.21.15.96:0/477863525 wait complete. 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:Waiting for the mgr to restart... 2024-04-22T15:08:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:Waiting for mgr epoch 5... 2024-04-22T15:08:46.819 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:46 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/477863525' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T15:08:51.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: Active manager daemon smithi096.hhkvum restarted 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: Activating manager daemon smithi096.hhkvum 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: mgrmap e6: smithi096.hhkvum(active, starting, since 0.00337107s) 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: Manager daemon smithi096.hhkvum is now available 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:51.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:51.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:08:51.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:08:51.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:08:51.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:51 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/trash_purge_schedule"}]: dispatch 2024-04-22T15:08:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:52 smithi096 ceph-mon[26688]: Found migration_current of "None". Setting to last migration. 2024-04-22T15:08:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:52 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:52.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:52 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:52.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:52 smithi096 ceph-mon[26688]: mgrmap e7: smithi096.hhkvum(active, since 1.00733s) 2024-04-22T15:08:53.065 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:08:53.065 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-22T15:08:53.065 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T15:08:53.065 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:08:53.065 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.016+0000 7f1a22b51700 1 Processor -- start 2024-04-22T15:08:53.065 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.016+0000 7f1a22b51700 1 -- start start 2024-04-22T15:08:53.065 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.016+0000 7f1a22b51700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c105270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:53.065 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.016+0000 7f1a22b51700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c074770 con 0x7f1a1c104e50 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.016+0000 7f1a208ed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.016+0000 7f1a208ed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c105270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54790/0 (socket says 172.21.15.96:54790) 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.016+0000 7f1a208ed700 1 -- 172.21.15.96:0/479450975 learned_addr learned my addr 172.21.15.96:0/479450975 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.017+0000 7f1a208ed700 1 -- 172.21.15.96:0/479450975 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c1057b0 con 0x7f1a1c104e50 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.017+0000 7f1a208ed700 1 --2- 172.21.15.96:0/479450975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c105270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1a100098a0 tx=0x7f1a10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59336eee9aa37e89 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.017+0000 7f1a1b7fe700 1 -- 172.21.15.96:0/479450975 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a10004180 con 0x7f1a1c104e50 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.017+0000 7f1a1b7fe700 1 -- 172.21.15.96:0/479450975 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1a100042e0 con 0x7f1a1c104e50 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.017+0000 7f1a1b7fe700 1 -- 172.21.15.96:0/479450975 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a10011a70 con 0x7f1a1c104e50 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.017+0000 7f1a22b51700 1 -- 172.21.15.96:0/479450975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 msgr2=0x7f1a1c105270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.017+0000 7f1a22b51700 1 --2- 172.21.15.96:0/479450975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c105270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1a100098a0 tx=0x7f1a10009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a22b51700 1 -- 172.21.15.96:0/479450975 shutdown_connections 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a22b51700 1 --2- 172.21.15.96:0/479450975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c105270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a22b51700 1 -- 172.21.15.96:0/479450975 >> 172.21.15.96:0/479450975 conn(0x7f1a1c100a50 msgr2=0x7f1a1c102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:53.066 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a22b51700 1 -- 172.21.15.96:0/479450975 shutdown_connections 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a22b51700 1 -- 172.21.15.96:0/479450975 wait complete. 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a22b51700 1 Processor -- start 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a22b51700 1 -- start start 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a22b51700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a22b51700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c19ca40 con 0x7f1a1c104e50 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a208ed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a208ed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54802/0 (socket says 172.21.15.96:54802) 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.018+0000 7f1a208ed700 1 -- 172.21.15.96:0/1901692547 learned_addr learned my addr 172.21.15.96:0/1901692547 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.019+0000 7f1a208ed700 1 -- 172.21.15.96:0/1901692547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10009550 con 0x7f1a1c104e50 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.019+0000 7f1a208ed700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c19c500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1a10011710 tx=0x7f1a100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.019+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a10003b10 con 0x7f1a1c104e50 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.019+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1a1001b8c0 con 0x7f1a1c104e50 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.019+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a100197e0 con 0x7f1a1c104e50 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.019+0000 7f1a22b51700 1 -- 172.21.15.96:0/1901692547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1c19cc40 con 0x7f1a1c104e50 2024-04-22T15:08:53.067 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.019+0000 7f1a22b51700 1 -- 172.21.15.96:0/1901692547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c19d0e0 con 0x7f1a1c104e50 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.020+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f1a1001ba30 con 0x7f1a1c104e50 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.020+0000 7f1a19ffb700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 0x7f1a0c03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.020+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 --> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a0c03b340 con 0x7f1a0c038770 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.020+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1a1004c8c0 con 0x7f1a1c104e50 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.020+0000 7f1a1bfff700 1 -- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 msgr2=0x7f1a0c03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/2684869170 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.020+0000 7f1a1bfff700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 0x7f1a0c03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.221+0000 7f1a1bfff700 1 -- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 msgr2=0x7f1a0c03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/2684869170 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.221+0000 7f1a1bfff700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 0x7f1a0c03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.622+0000 7f1a1bfff700 1 -- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 msgr2=0x7f1a0c03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/2684869170 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:47.622+0000 7f1a1bfff700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 0x7f1a0c03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:48.423+0000 7f1a1bfff700 1 -- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 msgr2=0x7f1a0c03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/2684869170 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:48.423+0000 7f1a1bfff700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 0x7f1a0c03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T15:08:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:50.024+0000 7f1a1bfff700 1 -- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 msgr2=0x7f1a0c03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/2684869170 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:50.024+0000 7f1a1bfff700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 0x7f1a0c03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:51.465+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mgrmap(e 6) v1 ==== 45323+0+0 (secure 0 0 0) 0x7f1a1001b440 con 0x7f1a1c104e50 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:51.465+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 msgr2=0x7f1a0c03ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:51.465+0000 7f1a19ffb700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 0x7f1a0c03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.468+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 7) v1 ==== 45450+0+0 (secure 0 0 0) 0x7f1a1001ad90 con 0x7f1a1c104e50 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.468+0000 7f1a19ffb700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1a0c03bfb0 0x7f1a0c03e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.468+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a0c03b340 con 0x7f1a0c03bfb0 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.469+0000 7f1a1bfff700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1a0c03bfb0 0x7f1a0c03e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.469+0000 7f1a1bfff700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1a0c03bfb0 0x7f1a0c03e3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1a08003a10 tx=0x7f1a080092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.470+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f1a0c03b340 con 0x7f1a0c03bfb0 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.480+0000 7f1a22b51700 1 -- 172.21.15.96:0/1901692547 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f1a1c1065d0 con 0x7f1a0c03bfb0 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.480+0000 7f1a19ffb700 1 -- 172.21.15.96:0/1901692547 <== mgr.14120 v2:172.21.15.96:6800/3275369520 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f1a1c1065d0 con 0x7f1a0c03bfb0 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 -- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1a0c03bfb0 msgr2=0x7f1a0c03e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1a0c03bfb0 0x7f1a0c03e3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1a08003a10 tx=0x7f1a080092b0 comp rx=0 tx=0).stop 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 -- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 msgr2=0x7f1a1c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c19c500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1a10011710 tx=0x7f1a100047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:53.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 -- 172.21.15.96:0/1901692547 shutdown_connections 2024-04-22T15:08:53.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1a0c03bfb0 0x7f1a0c03e3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:53.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c104e50 0x7f1a1c19c500 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:53.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 --2- 172.21.15.96:0/1901692547 >> [v2:172.21.15.96:6800/2684869170,v1:172.21.15.96:6801/2684869170] conn(0x7f1a0c038770 0x7f1a0c03ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:53.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 -- 172.21.15.96:0/1901692547 >> 172.21.15.96:0/1901692547 conn(0x7f1a1c100a50 msgr2=0x7f1a1c193310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:53.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 -- 172.21.15.96:0/1901692547 shutdown_connections 2024-04-22T15:08:53.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:52.481+0000 7f1a22b51700 1 -- 172.21.15.96:0/1901692547 wait complete. 2024-04-22T15:08:53.070 INFO:teuthology.orchestra.run.smithi096.stdout:mgr epoch 5 is available 2024-04-22T15:08:53.070 INFO:teuthology.orchestra.run.smithi096.stdout:Setting orchestrator backend to cephadm... 2024-04-22T15:08:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:53 smithi096 ceph-mon[26688]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T15:08:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:53 smithi096 ceph-mon[26688]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T15:08:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:53 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:08:54.345 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.548+0000 7f2f7c3c5700 1 Processor -- start 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.548+0000 7f2f7c3c5700 1 -- start start 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.548+0000 7f2f7c3c5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f741069f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.548+0000 7f2f7c3c5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74106fc0 con 0x7f2f741065d0 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.549+0000 7f2f7a161700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f741069f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.549+0000 7f2f7a161700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f741069f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38918/0 (socket says 172.21.15.96:38918) 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.549+0000 7f2f7a161700 1 -- 172.21.15.96:0/2367631518 learned_addr learned my addr 172.21.15.96:0/2367631518 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.549+0000 7f2f7a161700 1 -- 172.21.15.96:0/2367631518 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f741077d0 con 0x7f2f741065d0 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.549+0000 7f2f7a161700 1 --2- 172.21.15.96:0/2367631518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f741069f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009e80 tx=0x7f2f6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd4acd167ec8ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.549+0000 7f2f7915f700 1 -- 172.21.15.96:0/2367631518 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f6c00e070 con 0x7f2f741065d0 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.549+0000 7f2f7915f700 1 -- 172.21.15.96:0/2367631518 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2f6c00b8c0 con 0x7f2f741065d0 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.549+0000 7f2f7915f700 1 -- 172.21.15.96:0/2367631518 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f6c004440 con 0x7f2f741065d0 2024-04-22T15:08:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.550+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/2367631518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 msgr2=0x7f2f741069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.550+0000 7f2f7c3c5700 1 --2- 172.21.15.96:0/2367631518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f741069f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009e80 tx=0x7f2f6c00b330 comp rx=0 tx=0).stop 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.550+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/2367631518 shutdown_connections 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.550+0000 7f2f7c3c5700 1 --2- 172.21.15.96:0/2367631518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f741069f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.550+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/2367631518 >> 172.21.15.96:0/2367631518 conn(0x7f2f74101bf0 msgr2=0x7f2f74104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.550+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/2367631518 shutdown_connections 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.550+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/2367631518 wait complete. 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.550+0000 7f2f7c3c5700 1 Processor -- start 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7c3c5700 1 -- start start 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7c3c5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f7419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7c3c5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7419a840 con 0x7f2f741065d0 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7a161700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f7419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7a161700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f7419a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38920/0 (socket says 172.21.15.96:38920) 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7a161700 1 -- 172.21.15.96:0/1561581891 learned_addr learned my addr 172.21.15.96:0/1561581891 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:54.347 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7a161700 1 -- 172.21.15.96:0/1561581891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c009b30 con 0x7f2f741065d0 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7a161700 1 --2- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f7419a300 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0099e0 tx=0x7f2f6c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f6b7fe700 1 -- 172.21.15.96:0/1561581891 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f6c00e040 con 0x7f2f741065d0 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f6b7fe700 1 -- 172.21.15.96:0/1561581891 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2f6c00bc40 con 0x7f2f741065d0 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7419aa40 con 0x7f2f741065d0 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f6b7fe700 1 -- 172.21.15.96:0/1561581891 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f6c0125e0 con 0x7f2f741065d0 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.551+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7419af40 con 0x7f2f741065d0 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.552+0000 7f2f6b7fe700 1 -- 172.21.15.96:0/1561581891 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f2f6c019040 con 0x7f2f741065d0 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.553+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f74062510 con 0x7f2f741065d0 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.553+0000 7f2f6b7fe700 1 --2- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f2f60038750 0x7f2f6003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:54.348 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.553+0000 7f2f6b7fe700 1 -- 172.21.15.96:0/1561581891 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2f6c04c060 con 0x7f2f741065d0 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.553+0000 7f2f79960700 1 --2- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f2f60038750 0x7f2f6003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.553+0000 7f2f79960700 1 --2- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f2f60038750 0x7f2f6003ac10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f64006fd0 tx=0x7f2f64006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.558+0000 7f2f6b7fe700 1 -- 172.21.15.96:0/1561581891 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2f6c07fc50 con 0x7f2f741065d0 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.766+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f7410ac00 con 0x7f2f60038750 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.773+0000 7f2f6b7fe700 1 -- 172.21.15.96:0/1561581891 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2f7410ac00 con 0x7f2f60038750 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.775+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f2f60038750 msgr2=0x7f2f6003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.775+0000 7f2f7c3c5700 1 --2- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f2f60038750 0x7f2f6003ac10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f64006fd0 tx=0x7f2f64006e40 comp rx=0 tx=0).stop 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.775+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 msgr2=0x7f2f7419a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.775+0000 7f2f7c3c5700 1 --2- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f7419a300 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0099e0 tx=0x7f2f6c003d00 comp rx=0 tx=0).stop 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.775+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 shutdown_connections 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.775+0000 7f2f7c3c5700 1 --2- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f2f60038750 0x7f2f6003ac10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.775+0000 7f2f7c3c5700 1 --2- 172.21.15.96:0/1561581891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f741065d0 0x7f2f7419a300 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.775+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 >> 172.21.15.96:0/1561581891 conn(0x7f2f74101bf0 msgr2=0x7f2f74102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:54.349 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.776+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 shutdown_connections 2024-04-22T15:08:54.350 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:53.776+0000 7f2f7c3c5700 1 -- 172.21.15.96:0/1561581891 wait complete. 2024-04-22T15:08:54.640 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:54 smithi096 ceph-mon[26688]: [22/Apr/2024:15:08:52] ENGINE Bus STARTING 2024-04-22T15:08:54.640 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:54 smithi096 ceph-mon[26688]: [22/Apr/2024:15:08:52] ENGINE Serving on https://172.21.15.96:7150 2024-04-22T15:08:54.641 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:54 smithi096 ceph-mon[26688]: [22/Apr/2024:15:08:53] ENGINE Serving on http://172.21.15.96:8765 2024-04-22T15:08:54.641 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:54 smithi096 ceph-mon[26688]: [22/Apr/2024:15:08:53] ENGINE Bus STARTED 2024-04-22T15:08:54.641 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:54 smithi096 ceph-mon[26688]: mgrmap e8: smithi096.hhkvum(active, since 2s) 2024-04-22T15:08:54.641 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:54 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:54.641 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:54 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:08:55.585 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:55 smithi096 ceph-mon[26688]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:08:55.593 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-22T15:08:55.593 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.743+0000 7f07034f7700 1 Processor -- start 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.743+0000 7f07034f7700 1 -- start start 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.743+0000 7f07034f7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc109440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.743+0000 7f07034f7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc079ed0 con 0x7f06fc107050 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.744+0000 7f0701293700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc109440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.744+0000 7f0701293700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc109440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38934/0 (socket says 172.21.15.96:38934) 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.744+0000 7f0701293700 1 -- 172.21.15.96:0/497749522 learned_addr learned my addr 172.21.15.96:0/497749522 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.744+0000 7f0701293700 1 -- 172.21.15.96:0/497749522 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc109980 con 0x7f06fc107050 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.744+0000 7f0701293700 1 --2- 172.21.15.96:0/497749522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc109440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f06f80098a0 tx=0x7f06f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3056c2625efa4a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.744+0000 7f06f3fff700 1 -- 172.21.15.96:0/497749522 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06f8004180 con 0x7f06fc107050 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.744+0000 7f06f3fff700 1 -- 172.21.15.96:0/497749522 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f06f80042e0 con 0x7f06fc107050 2024-04-22T15:08:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.744+0000 7f06f3fff700 1 -- 172.21.15.96:0/497749522 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06f8011a70 con 0x7f06fc107050 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.745+0000 7f07034f7700 1 -- 172.21.15.96:0/497749522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 msgr2=0x7f06fc109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.745+0000 7f07034f7700 1 --2- 172.21.15.96:0/497749522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc109440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f06f80098a0 tx=0x7f06f8009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.745+0000 7f07034f7700 1 -- 172.21.15.96:0/497749522 shutdown_connections 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.745+0000 7f07034f7700 1 --2- 172.21.15.96:0/497749522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc109440 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.745+0000 7f07034f7700 1 -- 172.21.15.96:0/497749522 >> 172.21.15.96:0/497749522 conn(0x7f06fc100b00 msgr2=0x7f06fc102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.745+0000 7f07034f7700 1 -- 172.21.15.96:0/497749522 shutdown_connections 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.745+0000 7f07034f7700 1 -- 172.21.15.96:0/497749522 wait complete. 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f07034f7700 1 Processor -- start 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f07034f7700 1 -- start start 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f07034f7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc106880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f07034f7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc106dc0 con 0x7f06fc107050 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f0701293700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc106880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f0701293700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc106880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38948/0 (socket says 172.21.15.96:38948) 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f0701293700 1 -- 172.21.15.96:0/831463929 learned_addr learned my addr 172.21.15.96:0/831463929 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f0701293700 1 -- 172.21.15.96:0/831463929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8009550 con 0x7f06fc107050 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f0701293700 1 --2- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc106880 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f06f8011a20 tx=0x7f06f80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f06f27fc700 1 -- 172.21.15.96:0/831463929 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06f8003b10 con 0x7f06fc107050 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.746+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06fc105000 con 0x7f06fc107050 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.747+0000 7f06f27fc700 1 -- 172.21.15.96:0/831463929 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f06f801b8c0 con 0x7f06fc107050 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.747+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06fc1054a0 con 0x7f06fc107050 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.747+0000 7f06f27fc700 1 -- 172.21.15.96:0/831463929 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06f80237b0 con 0x7f06fc107050 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.747+0000 7f06f27fc700 1 -- 172.21.15.96:0/831463929 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f06f8023a30 con 0x7f06fc107050 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.747+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06fc10a700 con 0x7f06fc107050 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.748+0000 7f06f27fc700 1 --2- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f06e80386e0 0x7f06e803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.748+0000 7f06f27fc700 1 -- 172.21.15.96:0/831463929 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f06f804cb70 con 0x7f06fc107050 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.748+0000 7f0700a92700 1 --2- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f06e80386e0 0x7f06e803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.748+0000 7f0700a92700 1 --2- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f06e80386e0 0x7f06e803aba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06ec006fd0 tx=0x7f06ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.752+0000 7f06f27fc700 1 -- 172.21.15.96:0/831463929 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f06f8018350 con 0x7f06fc107050 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.946+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f06fc105c50 con 0x7f06e80386e0 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.947+0000 7f06f27fc700 1 -- 172.21.15.96:0/831463929 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f06fc105c50 con 0x7f06e80386e0 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.949+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f06e80386e0 msgr2=0x7f06e803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.949+0000 7f07034f7700 1 --2- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f06e80386e0 0x7f06e803aba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06ec006fd0 tx=0x7f06ec006e40 comp rx=0 tx=0).stop 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.949+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 msgr2=0x7f06fc106880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.949+0000 7f07034f7700 1 --2- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc106880 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f06f8011a20 tx=0x7f06f80047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.949+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 shutdown_connections 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.950+0000 7f07034f7700 1 --2- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f06e80386e0 0x7f06e803aba0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.950+0000 7f07034f7700 1 --2- 172.21.15.96:0/831463929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06fc107050 0x7f06fc106880 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.950+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 >> 172.21.15.96:0/831463929 conn(0x7f06fc100b00 msgr2=0x7f06fc1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.950+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 shutdown_connections 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:54.950+0000 7f07034f7700 1 -- 172.21.15.96:0/831463929 wait complete. 2024-04-22T15:08:55.597 INFO:teuthology.orchestra.run.smithi096.stdout:Generating ssh key... 2024-04-22T15:08:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:56 smithi096 ceph-mon[26688]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:08:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:56 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:56 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:08:57.017 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.054+0000 7f70a6118700 1 Processor -- start 2024-04-22T15:08:57.017 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.054+0000 7f70a6118700 1 -- start start 2024-04-22T15:08:57.017 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.054+0000 7f70a6118700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a01069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:57.017 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.054+0000 7f70a6118700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a0106fb0 con 0x7f70a01065c0 2024-04-22T15:08:57.017 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.054+0000 7f709f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a01069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:57.017 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.054+0000 7f709f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a01069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38956/0 (socket says 172.21.15.96:38956) 2024-04-22T15:08:57.017 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.054+0000 7f709f7fe700 1 -- 172.21.15.96:0/2161044117 learned_addr learned my addr 172.21.15.96:0/2161044117 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:57.017 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.054+0000 7f709f7fe700 1 -- 172.21.15.96:0/2161044117 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a01077c0 con 0x7f70a01065c0 2024-04-22T15:08:57.017 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.054+0000 7f709f7fe700 1 --2- 172.21.15.96:0/2161044117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a01069e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f70900098a0 tx=0x7f7090009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26873db757ad26b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f709e7fc700 1 -- 172.21.15.96:0/2161044117 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7090004180 con 0x7f70a01065c0 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f709e7fc700 1 -- 172.21.15.96:0/2161044117 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f70900042e0 con 0x7f70a01065c0 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f709e7fc700 1 -- 172.21.15.96:0/2161044117 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7090011a70 con 0x7f70a01065c0 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f70a6118700 1 -- 172.21.15.96:0/2161044117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 msgr2=0x7f70a01069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f70a6118700 1 --2- 172.21.15.96:0/2161044117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a01069e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f70900098a0 tx=0x7f7090009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f70a6118700 1 -- 172.21.15.96:0/2161044117 shutdown_connections 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f70a6118700 1 --2- 172.21.15.96:0/2161044117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a01069e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f70a6118700 1 -- 172.21.15.96:0/2161044117 >> 172.21.15.96:0/2161044117 conn(0x7f70a0101be0 msgr2=0x7f70a0104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f70a6118700 1 -- 172.21.15.96:0/2161044117 shutdown_connections 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.055+0000 7f70a6118700 1 -- 172.21.15.96:0/2161044117 wait complete. 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.056+0000 7f70a6118700 1 Processor -- start 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.056+0000 7f70a6118700 1 -- start start 2024-04-22T15:08:57.018 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.056+0000 7f70a6118700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a007bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.056+0000 7f70a6118700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a007c080 con 0x7f70a01065c0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.056+0000 7f709f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a007bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.056+0000 7f709f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a007bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38962/0 (socket says 172.21.15.96:38962) 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.056+0000 7f709f7fe700 1 -- 172.21.15.96:0/2178039828 learned_addr learned my addr 172.21.15.96:0/2178039828 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.057+0000 7f709f7fe700 1 -- 172.21.15.96:0/2178039828 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7090009550 con 0x7f70a01065c0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.057+0000 7f709f7fe700 1 --2- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a007bb40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7090011710 tx=0x7f70900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.057+0000 7f709cff9700 1 -- 172.21.15.96:0/2178039828 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7090003b10 con 0x7f70a01065c0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.057+0000 7f709cff9700 1 -- 172.21.15.96:0/2178039828 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f709001b8c0 con 0x7f70a01065c0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.057+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a007a2c0 con 0x7f70a01065c0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.057+0000 7f709cff9700 1 -- 172.21.15.96:0/2178039828 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7090019770 con 0x7f70a01065c0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.057+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a007a760 con 0x7f70a01065c0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.058+0000 7f709cff9700 1 -- 172.21.15.96:0/2178039828 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f70900199a0 con 0x7f70a01065c0 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.058+0000 7f709cff9700 1 --2- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f7088038770 0x7f708803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:57.019 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.058+0000 7f709cff9700 1 -- 172.21.15.96:0/2178039828 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f709004c950 con 0x7f70a01065c0 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.058+0000 7f709effd700 1 --2- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f7088038770 0x7f708803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.058+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a0062510 con 0x7f70a01065c0 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.058+0000 7f709effd700 1 --2- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f7088038770 0x7f708803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7094006fd0 tx=0x7f7094006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.063+0000 7f709cff9700 1 -- 172.21.15.96:0/2178039828 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f709001f080 con 0x7f70a01065c0 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.258+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f70a007acb0 con 0x7f7088038770 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.421+0000 7f709cff9700 1 -- 172.21.15.96:0/2178039828 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f70a007acb0 con 0x7f7088038770 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.423+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f7088038770 msgr2=0x7f708803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.423+0000 7f70a6118700 1 --2- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f7088038770 0x7f708803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7094006fd0 tx=0x7f7094006e40 comp rx=0 tx=0).stop 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.423+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 msgr2=0x7f70a007bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.423+0000 7f70a6118700 1 --2- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a007bb40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7090011710 tx=0x7f70900047c0 comp rx=0 tx=0).stop 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.423+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 shutdown_connections 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.423+0000 7f70a6118700 1 --2- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f7088038770 0x7f708803ac30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.423+0000 7f70a6118700 1 --2- 172.21.15.96:0/2178039828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a01065c0 0x7f70a007bb40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:57.020 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.423+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 >> 172.21.15.96:0/2178039828 conn(0x7f70a0101be0 msgr2=0x7f70a0102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:57.021 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.424+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 shutdown_connections 2024-04-22T15:08:57.021 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:56.424+0000 7f70a6118700 1 -- 172.21.15.96:0/2178039828 wait complete. 2024-04-22T15:08:57.637 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:57 smithi096 ceph-mon[26688]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:08:57.637 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:57 smithi096 ceph-mon[26688]: Generating ssh key... 2024-04-22T15:08:58.236 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvjlJwVAi6it8NxTubVve/Wx8HJK6jvdZXPtOALRCKaQTw1kSJWQ5vKPPgZFxKHN9OT5Cll03ZEbvR/abVHHCM68oRnGl773CoR9F1l+10sp/dPDOLAHJXN8N4CiMrR+ZqZ1Ffejom5KcIrZNMkN97P2mxbpCar46QeKi23ZP12rGDjhoIN3eJ6FnVv/vNnYiwF3H+yBGMgNAUgCceN++xJe7EsEdSD/XRsOAtW2tM90qs+sFdAgtAaWejgglxho8mvg5YiDB5ddxXoMGE/2C9Tg5qzjL45zUIcCmNBqHnco5TKFhaSguaGD8OIBBpXY4YOHND0oA3nzPEgnkQ9Idl3ucRVVYxGK2whHduUnK1yLmXiP7WhoERg7f/tb6LwLgRH+rKxNmcJineBy8/l5gAQJ/w+mRY4q/9lYT3RBtS/bsPxTaBSZUFXA8YBOg8ROArEK7ObL+kSnMizvo6uYZvJMaGSfAR87FgBptywx0lpsU7Vq+afhMwq7P8shJdkoU= ceph-d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:08:58.242 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.425+0000 7f36a416b700 1 Processor -- start 2024-04-22T15:08:58.242 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.425+0000 7f36a416b700 1 -- start start 2024-04-22T15:08:58.242 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.425+0000 7f36a416b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c104880 0x7f369c104ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:58.242 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.425+0000 7f36a416b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c105270 con 0x7f369c104880 2024-04-22T15:08:58.242 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.425+0000 7f36a1f07700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c104880 0x7f369c104ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:58.242 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.425+0000 7f36a1f07700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c104880 0x7f369c104ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38970/0 (socket says 172.21.15.96:38970) 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.425+0000 7f36a1f07700 1 -- 172.21.15.96:0/398542595 learned_addr learned my addr 172.21.15.96:0/398542595 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.426+0000 7f36a1f07700 1 -- 172.21.15.96:0/398542595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c1053b0 con 0x7f369c104880 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.426+0000 7f36a1f07700 1 --2- 172.21.15.96:0/398542595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c104880 0x7f369c104ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f36980098a0 tx=0x7f3698009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a166aaa03113625 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.426+0000 7f36a0f05700 1 -- 172.21.15.96:0/398542595 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3698004180 con 0x7f369c104880 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.426+0000 7f36a0f05700 1 -- 172.21.15.96:0/398542595 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f36980042e0 con 0x7f369c104880 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.426+0000 7f36a0f05700 1 -- 172.21.15.96:0/398542595 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3698011a70 con 0x7f369c104880 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.427+0000 7f36a416b700 1 -- 172.21.15.96:0/398542595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c104880 msgr2=0x7f369c104ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.427+0000 7f36a416b700 1 --2- 172.21.15.96:0/398542595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c104880 0x7f369c104ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f36980098a0 tx=0x7f3698009bb0 comp rx=0 tx=0).stop 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.427+0000 7f36a416b700 1 -- 172.21.15.96:0/398542595 shutdown_connections 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.427+0000 7f36a416b700 1 --2- 172.21.15.96:0/398542595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c104880 0x7f369c104ca0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.427+0000 7f36a416b700 1 -- 172.21.15.96:0/398542595 >> 172.21.15.96:0/398542595 conn(0x7f369c1001c0 msgr2=0x7f369c102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:58.243 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.427+0000 7f36a416b700 1 -- 172.21.15.96:0/398542595 shutdown_connections 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.427+0000 7f36a416b700 1 -- 172.21.15.96:0/398542595 wait complete. 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a416b700 1 Processor -- start 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a416b700 1 -- start start 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a416b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c19a580 0x7f369c19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a416b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c19aee0 con 0x7f369c19a580 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a1f07700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c19a580 0x7f369c19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a1f07700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c19a580 0x7f369c19a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38982/0 (socket says 172.21.15.96:38982) 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a1f07700 1 -- 172.21.15.96:0/842262403 learned_addr learned my addr 172.21.15.96:0/842262403 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a1f07700 1 -- 172.21.15.96:0/842262403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3698009550 con 0x7f369c19a580 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a1f07700 1 --2- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c19a580 0x7f369c19a9a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3698011710 tx=0x7f3698004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f3692ffd700 1 -- 172.21.15.96:0/842262403 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3698011ab0 con 0x7f369c19a580 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f3692ffd700 1 -- 172.21.15.96:0/842262403 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3698020070 con 0x7f369c19a580 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c19b0e0 con 0x7f369c19a580 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.428+0000 7f3692ffd700 1 -- 172.21.15.96:0/842262403 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f369801bab0 con 0x7f369c19a580 2024-04-22T15:08:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.429+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c19dc00 con 0x7f369c19a580 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.429+0000 7f3692ffd700 1 -- 172.21.15.96:0/842262403 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f3698022030 con 0x7f369c19a580 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.429+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c108530 con 0x7f369c19a580 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.430+0000 7f3692ffd700 1 --2- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f36880386e0 0x7f368803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.430+0000 7f3692ffd700 1 -- 172.21.15.96:0/842262403 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f369804c5a0 con 0x7f369c19a580 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.430+0000 7f36a1706700 1 --2- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f36880386e0 0x7f368803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.430+0000 7f36a1706700 1 --2- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f36880386e0 0x7f368803aba0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f368c006fd0 tx=0x7f368c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.435+0000 7f3692ffd700 1 -- 172.21.15.96:0/842262403 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f369802d7b0 con 0x7f369c19a580 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.633+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f369c1015e0 con 0x7f36880386e0 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.633+0000 7f3692ffd700 1 -- 172.21.15.96:0/842262403 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f369c1015e0 con 0x7f36880386e0 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.635+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f36880386e0 msgr2=0x7f368803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.635+0000 7f36a416b700 1 --2- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f36880386e0 0x7f368803aba0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f368c006fd0 tx=0x7f368c006e40 comp rx=0 tx=0).stop 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.635+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c19a580 msgr2=0x7f369c19a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.635+0000 7f36a416b700 1 --2- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c19a580 0x7f369c19a9a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3698011710 tx=0x7f3698004730 comp rx=0 tx=0).stop 2024-04-22T15:08:58.245 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.635+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 shutdown_connections 2024-04-22T15:08:58.246 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.635+0000 7f36a416b700 1 --2- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f36880386e0 0x7f368803aba0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:58.246 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.635+0000 7f36a416b700 1 --2- 172.21.15.96:0/842262403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369c19a580 0x7f369c19a9a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:08:58.246 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.635+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 >> 172.21.15.96:0/842262403 conn(0x7f369c1001c0 msgr2=0x7f369c100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:08:58.246 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.636+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 shutdown_connections 2024-04-22T15:08:58.246 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:57.636+0000 7f36a416b700 1 -- 172.21.15.96:0/842262403 wait complete. 2024-04-22T15:08:58.246 INFO:teuthology.orchestra.run.smithi096.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T15:08:58.246 INFO:teuthology.orchestra.run.smithi096.stdout:Adding key to root@localhost authorized_keys... 2024-04-22T15:08:58.246 INFO:teuthology.orchestra.run.smithi096.stdout:Adding host smithi096... 2024-04-22T15:08:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:58 smithi096 ceph-mon[26688]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:08:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:59 smithi096 ceph-mon[26688]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi096", "addr": "172.21.15.96", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:08:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:08:59 smithi096 ceph-mon[26688]: Deploying cephadm binary to smithi096 2024-04-22T15:09:01.783 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Added host 'smithi096' with addr '172.21.15.96' 2024-04-22T15:09:01.783 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.666+0000 7ff13e4fb700 1 Processor -- start 2024-04-22T15:09:01.783 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.666+0000 7ff13e4fb700 1 -- start start 2024-04-22T15:09:01.783 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.666+0000 7ff13e4fb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff1381052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:01.783 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.666+0000 7ff13e4fb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff138079e50 con 0x7ff138104e90 2024-04-22T15:09:01.783 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.666+0000 7ff137fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff1381052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:01.783 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.666+0000 7ff137fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff1381052b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38988/0 (socket says 172.21.15.96:38988) 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.666+0000 7ff137fff700 1 -- 172.21.15.96:0/725268540 learned_addr learned my addr 172.21.15.96:0/725268540 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.667+0000 7ff137fff700 1 -- 172.21.15.96:0/725268540 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1381057f0 con 0x7ff138104e90 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.667+0000 7ff137fff700 1 --2- 172.21.15.96:0/725268540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff1381052b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff1280098a0 tx=0x7ff128009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43ea31c679625f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.667+0000 7ff136ffd700 1 -- 172.21.15.96:0/725268540 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff128004180 con 0x7ff138104e90 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.667+0000 7ff136ffd700 1 -- 172.21.15.96:0/725268540 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff1280042e0 con 0x7ff138104e90 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.667+0000 7ff136ffd700 1 -- 172.21.15.96:0/725268540 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff128011a70 con 0x7ff138104e90 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.667+0000 7ff13e4fb700 1 -- 172.21.15.96:0/725268540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 msgr2=0x7ff1381052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.667+0000 7ff13e4fb700 1 --2- 172.21.15.96:0/725268540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff1381052b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff1280098a0 tx=0x7ff128009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.668+0000 7ff13e4fb700 1 -- 172.21.15.96:0/725268540 shutdown_connections 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.668+0000 7ff13e4fb700 1 --2- 172.21.15.96:0/725268540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff1381052b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.668+0000 7ff13e4fb700 1 -- 172.21.15.96:0/725268540 >> 172.21.15.96:0/725268540 conn(0x7ff138100af0 msgr2=0x7ff138102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.668+0000 7ff13e4fb700 1 -- 172.21.15.96:0/725268540 shutdown_connections 2024-04-22T15:09:01.784 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.668+0000 7ff13e4fb700 1 -- 172.21.15.96:0/725268540 wait complete. 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.668+0000 7ff13e4fb700 1 Processor -- start 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.668+0000 7ff13e4fb700 1 -- start start 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.668+0000 7ff13e4fb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff13819c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff13e4fb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13819c9f0 con 0x7ff138104e90 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff137fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff13819c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff137fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff13819c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38998/0 (socket says 172.21.15.96:38998) 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff137fff700 1 -- 172.21.15.96:0/619695325 learned_addr learned my addr 172.21.15.96:0/619695325 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff137fff700 1 -- 172.21.15.96:0/619695325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff128009550 con 0x7ff138104e90 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff137fff700 1 --2- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff13819c4b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff128011710 tx=0x7ff1280047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff1357fa700 1 -- 172.21.15.96:0/619695325 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff128003b10 con 0x7ff138104e90 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff1357fa700 1 -- 172.21.15.96:0/619695325 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff12801b8c0 con 0x7ff138104e90 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13819cbf0 con 0x7ff138104e90 2024-04-22T15:09:01.785 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff1357fa700 1 -- 172.21.15.96:0/619695325 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1280197e0 con 0x7ff138104e90 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.669+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13819d090 con 0x7ff138104e90 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.670+0000 7ff1357fa700 1 -- 172.21.15.96:0/619695325 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7ff128019a40 con 0x7ff138104e90 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.670+0000 7ff1357fa700 1 --2- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7ff120038720 0x7ff12003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.670+0000 7ff1357fa700 1 -- 172.21.15.96:0/619695325 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff12804c950 con 0x7ff138104e90 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.670+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1381961c0 con 0x7ff138104e90 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.670+0000 7ff1377fe700 1 --2- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7ff120038720 0x7ff12003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.671+0000 7ff1377fe700 1 --2- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7ff120038720 0x7ff12003abe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff12c006fd0 tx=0x7ff12c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.675+0000 7ff1357fa700 1 -- 172.21.15.96:0/619695325 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff12801f080 con 0x7ff138104e90 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:08:58.872+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi096", "addr": "172.21.15.96", "target": ["mon-mgr", ""]}) v1 -- 0x7ff13802ceb0 con 0x7ff120038720 2024-04-22T15:09:01.786 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.183+0000 7ff1357fa700 1 -- 172.21.15.96:0/619695325 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7ff13802ceb0 con 0x7ff120038720 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.185+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7ff120038720 msgr2=0x7ff12003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.185+0000 7ff13e4fb700 1 --2- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7ff120038720 0x7ff12003abe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff12c006fd0 tx=0x7ff12c006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.185+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 msgr2=0x7ff13819c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.185+0000 7ff13e4fb700 1 --2- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff13819c4b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff128011710 tx=0x7ff1280047c0 comp rx=0 tx=0).stop 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.186+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 shutdown_connections 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.186+0000 7ff13e4fb700 1 --2- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7ff120038720 0x7ff12003abe0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.186+0000 7ff13e4fb700 1 --2- 172.21.15.96:0/619695325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff138104e90 0x7ff13819c4b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.186+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 >> 172.21.15.96:0/619695325 conn(0x7ff138100af0 msgr2=0x7ff138193300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.186+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 shutdown_connections 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:01.186+0000 7ff13e4fb700 1 -- 172.21.15.96:0/619695325 wait complete. 2024-04-22T15:09:01.787 INFO:teuthology.orchestra.run.smithi096.stdout:Deploying mon service with default placement... 2024-04-22T15:09:02.428 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:02 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:02.428 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:02 smithi096 ceph-mon[26688]: Added host smithi096 2024-04-22T15:09:02.429 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:02 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:09:03.377 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.216+0000 7faf70104700 1 Processor -- start 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.216+0000 7faf70104700 1 -- start start 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf70104700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf681058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf70104700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf68105ea0 con 0x7faf681054b0 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf6dea0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf681058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf6dea0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf681058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57408/0 (socket says 172.21.15.96:57408) 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf6dea0700 1 -- 172.21.15.96:0/3990111697 learned_addr learned my addr 172.21.15.96:0/3990111697 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf6dea0700 1 -- 172.21.15.96:0/3990111697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf681066b0 con 0x7faf681054b0 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf6dea0700 1 --2- 172.21.15.96:0/3990111697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf681058d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf6000bb40 tx=0x7faf6000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a955a710c7427727 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf6ce9e700 1 -- 172.21.15.96:0/3990111697 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf6000dc00 con 0x7faf681054b0 2024-04-22T15:09:03.378 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf6ce9e700 1 -- 172.21.15.96:0/3990111697 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7faf600044b0 con 0x7faf681054b0 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.217+0000 7faf6ce9e700 1 -- 172.21.15.96:0/3990111697 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf60010470 con 0x7faf681054b0 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.218+0000 7faf70104700 1 -- 172.21.15.96:0/3990111697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 msgr2=0x7faf681058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.218+0000 7faf70104700 1 --2- 172.21.15.96:0/3990111697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf681058d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf6000bb40 tx=0x7faf6000be50 comp rx=0 tx=0).stop 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.218+0000 7faf70104700 1 -- 172.21.15.96:0/3990111697 shutdown_connections 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.218+0000 7faf70104700 1 --2- 172.21.15.96:0/3990111697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf681058d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.218+0000 7faf70104700 1 -- 172.21.15.96:0/3990111697 >> 172.21.15.96:0/3990111697 conn(0x7faf68100a50 msgr2=0x7faf68102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.218+0000 7faf70104700 1 -- 172.21.15.96:0/3990111697 shutdown_connections 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.218+0000 7faf70104700 1 -- 172.21.15.96:0/3990111697 wait complete. 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf70104700 1 Processor -- start 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf70104700 1 -- start start 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf70104700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf68198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf70104700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf681985b0 con 0x7faf681054b0 2024-04-22T15:09:03.379 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf6dea0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf68198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:03.380 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf6dea0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf68198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57422/0 (socket says 172.21.15.96:57422) 2024-04-22T15:09:03.380 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf6dea0700 1 -- 172.21.15.96:0/3252438124 learned_addr learned my addr 172.21.15.96:0/3252438124 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:03.380 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf6dea0700 1 -- 172.21.15.96:0/3252438124 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6000b7f0 con 0x7faf681054b0 2024-04-22T15:09:03.380 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf6dea0700 1 --2- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf68198070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf6000d650 tx=0x7faf60003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:03.380 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.219+0000 7faf5effd700 1 -- 172.21.15.96:0/3252438124 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf6001e070 con 0x7faf681054b0 2024-04-22T15:09:03.380 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.220+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf681987b0 con 0x7faf681054b0 2024-04-22T15:09:03.381 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.220+0000 7faf5effd700 1 -- 172.21.15.96:0/3252438124 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7faf600042e0 con 0x7faf681054b0 2024-04-22T15:09:03.381 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.220+0000 7faf5effd700 1 -- 172.21.15.96:0/3252438124 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf60022b60 con 0x7faf681054b0 2024-04-22T15:09:03.381 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.220+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf68198bd0 con 0x7faf681054b0 2024-04-22T15:09:03.381 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.220+0000 7faf5effd700 1 -- 172.21.15.96:0/3252438124 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7faf60010730 con 0x7faf681054b0 2024-04-22T15:09:03.381 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.221+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf68062510 con 0x7faf681054b0 2024-04-22T15:09:03.382 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.221+0000 7faf5effd700 1 --2- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7faf540386d0 0x7faf5403ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:03.382 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.221+0000 7faf5effd700 1 -- 172.21.15.96:0/3252438124 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faf6004ed90 con 0x7faf681054b0 2024-04-22T15:09:03.382 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.221+0000 7faf6d69f700 1 --2- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7faf540386d0 0x7faf5403ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:03.382 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.221+0000 7faf6d69f700 1 --2- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7faf540386d0 0x7faf5403ab90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf58006fd0 tx=0x7faf58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:03.382 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.226+0000 7faf5effd700 1 -- 172.21.15.96:0/3252438124 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7faf60021590 con 0x7faf681054b0 2024-04-22T15:09:03.382 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.440+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7faf6819b4e0 con 0x7faf540386d0 2024-04-22T15:09:03.383 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.447+0000 7faf5effd700 1 -- 172.21.15.96:0/3252438124 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7faf6819b4e0 con 0x7faf540386d0 2024-04-22T15:09:03.383 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7faf540386d0 msgr2=0x7faf5403ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:03.383 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 --2- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7faf540386d0 0x7faf5403ab90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf58006fd0 tx=0x7faf58006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:03.383 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 msgr2=0x7faf68198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:03.383 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 --2- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf68198070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf6000d650 tx=0x7faf60003fb0 comp rx=0 tx=0).stop 2024-04-22T15:09:03.383 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 shutdown_connections 2024-04-22T15:09:03.384 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 --2- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7faf540386d0 0x7faf5403ab90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:03.384 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 --2- 172.21.15.96:0/3252438124 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf681054b0 0x7faf68198070 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:03.384 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 >> 172.21.15.96:0/3252438124 conn(0x7faf68100a50 msgr2=0x7faf6818e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:03.384 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 shutdown_connections 2024-04-22T15:09:03.384 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:02.449+0000 7faf70104700 1 -- 172.21.15.96:0/3252438124 wait complete. 2024-04-22T15:09:03.384 INFO:teuthology.orchestra.run.smithi096.stdout:Deploying mgr service with default placement... 2024-04-22T15:09:03.705 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:03 smithi096 ceph-mon[26688]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:03.705 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:03 smithi096 ceph-mon[26688]: Saving service mon spec with placement count:5 2024-04-22T15:09:03.705 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:03 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.811+0000 7f8d17177700 1 Processor -- start 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.811+0000 7f8d17177700 1 -- start start 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.811+0000 7f8d17177700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d101069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.811+0000 7f8d17177700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d10106fa0 con 0x7f8d101065b0 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.811+0000 7f8d14f13700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d101069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.811+0000 7f8d14f13700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d101069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57438/0 (socket says 172.21.15.96:57438) 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.811+0000 7f8d14f13700 1 -- 172.21.15.96:0/3612447301 learned_addr learned my addr 172.21.15.96:0/3612447301 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.811+0000 7f8d14f13700 1 -- 172.21.15.96:0/3612447301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d101077b0 con 0x7f8d101065b0 2024-04-22T15:09:04.836 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d14f13700 1 --2- 172.21.15.96:0/3612447301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d101069d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d040098a0 tx=0x7f8d04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de0e6c3391a25e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d0f7fe700 1 -- 172.21.15.96:0/3612447301 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d04004180 con 0x7f8d101065b0 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d0f7fe700 1 -- 172.21.15.96:0/3612447301 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8d040042e0 con 0x7f8d101065b0 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d0f7fe700 1 -- 172.21.15.96:0/3612447301 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d04011a70 con 0x7f8d101065b0 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d17177700 1 -- 172.21.15.96:0/3612447301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 msgr2=0x7f8d101069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d17177700 1 --2- 172.21.15.96:0/3612447301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d101069d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d040098a0 tx=0x7f8d04009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d17177700 1 -- 172.21.15.96:0/3612447301 shutdown_connections 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d17177700 1 --2- 172.21.15.96:0/3612447301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d101069d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d17177700 1 -- 172.21.15.96:0/3612447301 >> 172.21.15.96:0/3612447301 conn(0x7f8d10101bd0 msgr2=0x7f8d10103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d17177700 1 -- 172.21.15.96:0/3612447301 shutdown_connections 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.812+0000 7f8d17177700 1 -- 172.21.15.96:0/3612447301 wait complete. 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.813+0000 7f8d17177700 1 Processor -- start 2024-04-22T15:09:04.837 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.813+0000 7f8d17177700 1 -- start start 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.813+0000 7f8d17177700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d1019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.813+0000 7f8d17177700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1019a860 con 0x7f8d101065b0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.813+0000 7f8d14f13700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d1019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.813+0000 7f8d14f13700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d1019a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57440/0 (socket says 172.21.15.96:57440) 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.813+0000 7f8d14f13700 1 -- 172.21.15.96:0/2686612090 learned_addr learned my addr 172.21.15.96:0/2686612090 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.814+0000 7f8d14f13700 1 -- 172.21.15.96:0/2686612090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d04009550 con 0x7f8d101065b0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.814+0000 7f8d14f13700 1 --2- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d1019a320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8d04011710 tx=0x7f8d040047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.814+0000 7f8d0dffb700 1 -- 172.21.15.96:0/2686612090 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d04003b10 con 0x7f8d101065b0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.814+0000 7f8d0dffb700 1 -- 172.21.15.96:0/2686612090 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8d0401b8c0 con 0x7f8d101065b0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.814+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1019aa60 con 0x7f8d101065b0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.814+0000 7f8d0dffb700 1 -- 172.21.15.96:0/2686612090 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d04019770 con 0x7f8d101065b0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.814+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1019af00 con 0x7f8d101065b0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.815+0000 7f8d0dffb700 1 -- 172.21.15.96:0/2686612090 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f8d040199f0 con 0x7f8d101065b0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.815+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1010a9e0 con 0x7f8d101065b0 2024-04-22T15:09:04.838 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.815+0000 7f8d0dffb700 1 --2- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f8d00038720 0x7f8d0003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.815+0000 7f8d0dffb700 1 -- 172.21.15.96:0/2686612090 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8d0404ca90 con 0x7f8d101065b0 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.815+0000 7f8d0ffff700 1 --2- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f8d00038720 0x7f8d0003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.816+0000 7f8d0ffff700 1 --2- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f8d00038720 0x7f8d0003abe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc006fd0 tx=0x7f8cfc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:03.820+0000 7f8d0dffb700 1 -- 172.21.15.96:0/2686612090 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f8d04018db0 con 0x7f8d101065b0 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.030+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d1019d7f0 con 0x7f8d00038720 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.034+0000 7f8d0dffb700 1 -- 172.21.15.96:0/2686612090 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8d1019d7f0 con 0x7f8d00038720 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f8d00038720 msgr2=0x7f8d0003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 --2- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f8d00038720 0x7f8d0003abe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc006fd0 tx=0x7f8cfc006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 msgr2=0x7f8d1019a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 --2- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d1019a320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8d04011710 tx=0x7f8d040047c0 comp rx=0 tx=0).stop 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 shutdown_connections 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 --2- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f8d00038720 0x7f8d0003abe0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 --2- 172.21.15.96:0/2686612090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d101065b0 0x7f8d1019a320 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 >> 172.21.15.96:0/2686612090 conn(0x7f8d10101bd0 msgr2=0x7f8d10102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:04.839 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 shutdown_connections 2024-04-22T15:09:04.840 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:04.036+0000 7f8d17177700 1 -- 172.21.15.96:0/2686612090 wait complete. 2024-04-22T15:09:04.840 INFO:teuthology.orchestra.run.smithi096.stdout:Deploying crash service with default placement... 2024-04-22T15:09:05.106 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:05 smithi096 ceph-mon[26688]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:05.106 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:05 smithi096 ceph-mon[26688]: Saving service mgr spec with placement count:2 2024-04-22T15:09:05.106 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:05 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:05.106 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:05 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:05.978 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-22T15:09:05.978 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.203+0000 7f489601d700 1 Processor -- start 2024-04-22T15:09:05.978 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.203+0000 7f489601d700 1 -- start start 2024-04-22T15:09:05.978 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.203+0000 7f489601d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890105900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:05.978 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.203+0000 7f489601d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890105ed0 con 0x7f48901054e0 2024-04-22T15:09:05.978 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.203+0000 7f488f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:05.978 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.203+0000 7f488f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890105900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57454/0 (socket says 172.21.15.96:57454) 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.203+0000 7f488f7fe700 1 -- 172.21.15.96:0/3715958736 learned_addr learned my addr 172.21.15.96:0/3715958736 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.203+0000 7f488f7fe700 1 -- 172.21.15.96:0/3715958736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48901066e0 con 0x7f48901054e0 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f488f7fe700 1 --2- 172.21.15.96:0/3715958736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890105900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f488000bb40 tx=0x7f488000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e0327eaaa8608a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f488e7fc700 1 -- 172.21.15.96:0/3715958736 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488000dc00 con 0x7f48901054e0 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f488e7fc700 1 -- 172.21.15.96:0/3715958736 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48800044f0 con 0x7f48901054e0 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f488e7fc700 1 -- 172.21.15.96:0/3715958736 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4880010470 con 0x7f48901054e0 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f489601d700 1 -- 172.21.15.96:0/3715958736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 msgr2=0x7f4890105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f489601d700 1 --2- 172.21.15.96:0/3715958736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890105900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f488000bb40 tx=0x7f488000be50 comp rx=0 tx=0).stop 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f489601d700 1 -- 172.21.15.96:0/3715958736 shutdown_connections 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f489601d700 1 --2- 172.21.15.96:0/3715958736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890105900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f489601d700 1 -- 172.21.15.96:0/3715958736 >> 172.21.15.96:0/3715958736 conn(0x7f4890100ae0 msgr2=0x7f4890102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f489601d700 1 -- 172.21.15.96:0/3715958736 shutdown_connections 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.204+0000 7f489601d700 1 -- 172.21.15.96:0/3715958736 wait complete. 2024-04-22T15:09:05.979 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.205+0000 7f489601d700 1 Processor -- start 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.205+0000 7f489601d700 1 -- start start 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.205+0000 7f489601d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.205+0000 7f489601d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890198680 con 0x7f48901054e0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.206+0000 7f488f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.206+0000 7f488f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57470/0 (socket says 172.21.15.96:57470) 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.206+0000 7f488f7fe700 1 -- 172.21.15.96:0/859538724 learned_addr learned my addr 172.21.15.96:0/859538724 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.206+0000 7f488f7fe700 1 -- 172.21.15.96:0/859538724 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488000b7f0 con 0x7f48901054e0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.206+0000 7f488f7fe700 1 --2- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890198140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f488000b340 tx=0x7f4880004070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.207+0000 7f488cff9700 1 -- 172.21.15.96:0/859538724 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488001e070 con 0x7f48901054e0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.207+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4890198880 con 0x7f48901054e0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.207+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4890198d20 con 0x7f48901054e0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.207+0000 7f488cff9700 1 -- 172.21.15.96:0/859538724 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f488000de20 con 0x7f48901054e0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.207+0000 7f488cff9700 1 -- 172.21.15.96:0/859538724 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4880022a40 con 0x7f48901054e0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.208+0000 7f488cff9700 1 -- 172.21.15.96:0/859538724 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f4880022ba0 con 0x7f48901054e0 2024-04-22T15:09:05.980 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.208+0000 7f488cff9700 1 --2- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f4878038690 0x7f487803ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.208+0000 7f488cff9700 1 -- 172.21.15.96:0/859538724 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f488004fb60 con 0x7f48901054e0 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.208+0000 7f488effd700 1 --2- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f4878038690 0x7f487803ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.208+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487c005320 con 0x7f48901054e0 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.209+0000 7f488effd700 1 --2- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f4878038690 0x7f487803ab50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4884006fd0 tx=0x7f4884006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.213+0000 7f488cff9700 1 -- 172.21.15.96:0/859538724 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4880027070 con 0x7f48901054e0 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.431+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f487c000bf0 con 0x7f4878038690 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.438+0000 7f488cff9700 1 -- 172.21.15.96:0/859538724 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f487c000bf0 con 0x7f4878038690 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f4878038690 msgr2=0x7f487803ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 --2- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f4878038690 0x7f487803ab50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4884006fd0 tx=0x7f4884006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 msgr2=0x7f4890198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 --2- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890198140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f488000b340 tx=0x7f4880004070 comp rx=0 tx=0).stop 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 shutdown_connections 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 --2- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f4878038690 0x7f487803ab50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:05.981 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 --2- 172.21.15.96:0/859538724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48901054e0 0x7f4890198140 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:05.982 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 >> 172.21.15.96:0/859538724 conn(0x7f4890100ae0 msgr2=0x7f489018e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:05.982 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 shutdown_connections 2024-04-22T15:09:05.982 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:05.440+0000 7f489601d700 1 -- 172.21.15.96:0/859538724 wait complete. 2024-04-22T15:09:05.982 INFO:teuthology.orchestra.run.smithi096.stdout:Deploying ceph-exporter service with default placement... 2024-04-22T15:09:06.356 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:06 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:06.356 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:06 smithi096 ceph-mon[26688]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:06.356 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:06 smithi096 ceph-mon[26688]: Saving service crash spec with placement * 2024-04-22T15:09:06.356 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:06 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.352+0000 7fb084a0b700 1 Processor -- start 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.352+0000 7fb084a0b700 1 -- start start 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.352+0000 7fb084a0b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb0801094a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.352+0000 7fb084a0b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb080079e70 con 0x7fb0801070b0 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.352+0000 7fb07e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb0801094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.352+0000 7fb07e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb0801094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57480/0 (socket says 172.21.15.96:57480) 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.352+0000 7fb07e59c700 1 -- 172.21.15.96:0/3792969941 learned_addr learned my addr 172.21.15.96:0/3792969941 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.352+0000 7fb07e59c700 1 -- 172.21.15.96:0/3792969941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0801099e0 con 0x7fb0801070b0 2024-04-22T15:09:07.788 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.353+0000 7fb07e59c700 1 --2- 172.21.15.96:0/3792969941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb0801094a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0700098a0 tx=0x7fb070009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3745aa73a05d1be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.353+0000 7fb07d59a700 1 -- 172.21.15.96:0/3792969941 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb070004180 con 0x7fb0801070b0 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.353+0000 7fb07d59a700 1 -- 172.21.15.96:0/3792969941 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0700042e0 con 0x7fb0801070b0 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.353+0000 7fb07d59a700 1 -- 172.21.15.96:0/3792969941 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb070011b10 con 0x7fb0801070b0 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.354+0000 7fb084a0b700 1 -- 172.21.15.96:0/3792969941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 msgr2=0x7fb0801094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.354+0000 7fb084a0b700 1 --2- 172.21.15.96:0/3792969941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb0801094a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0700098a0 tx=0x7fb070009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.354+0000 7fb084a0b700 1 -- 172.21.15.96:0/3792969941 shutdown_connections 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.354+0000 7fb084a0b700 1 --2- 172.21.15.96:0/3792969941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb0801094a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.354+0000 7fb084a0b700 1 -- 172.21.15.96:0/3792969941 >> 172.21.15.96:0/3792969941 conn(0x7fb080100ac0 msgr2=0x7fb080102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.354+0000 7fb084a0b700 1 -- 172.21.15.96:0/3792969941 shutdown_connections 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.354+0000 7fb084a0b700 1 -- 172.21.15.96:0/3792969941 wait complete. 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb084a0b700 1 Processor -- start 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb084a0b700 1 -- start start 2024-04-22T15:09:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb084a0b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb08019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb084a0b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08019a8b0 con 0x7fb0801070b0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb07e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb08019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb07e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb08019a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57488/0 (socket says 172.21.15.96:57488) 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb07e59c700 1 -- 172.21.15.96:0/1743218213 learned_addr learned my addr 172.21.15.96:0/1743218213 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb07e59c700 1 -- 172.21.15.96:0/1743218213 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb070009550 con 0x7fb0801070b0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb07e59c700 1 --2- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb08019a370 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb070009fb0 tx=0x7fb070011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.355+0000 7fb06f7fe700 1 -- 172.21.15.96:0/1743218213 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0700038b0 con 0x7fb0801070b0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.356+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08019aab0 con 0x7fb0801070b0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.356+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08019af50 con 0x7fb0801070b0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.356+0000 7fb06f7fe700 1 -- 172.21.15.96:0/1743218213 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb070003a10 con 0x7fb0801070b0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.356+0000 7fb06f7fe700 1 -- 172.21.15.96:0/1743218213 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb070019750 con 0x7fb0801070b0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.356+0000 7fb06f7fe700 1 -- 172.21.15.96:0/1743218213 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fb0700198b0 con 0x7fb0801070b0 2024-04-22T15:09:07.790 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.356+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb060005320 con 0x7fb0801070b0 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.357+0000 7fb06f7fe700 1 --2- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb068038730 0x7fb06803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.357+0000 7fb06f7fe700 1 -- 172.21.15.96:0/1743218213 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb07001ac10 con 0x7fb0801070b0 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.357+0000 7fb07dd9b700 1 --2- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb068038730 0x7fb06803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.357+0000 7fb07dd9b700 1 --2- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb068038730 0x7fb06803abf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb074006fd0 tx=0x7fb074006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.362+0000 7fb06f7fe700 1 -- 172.21.15.96:0/1743218213 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb070022bd0 con 0x7fb0801070b0 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.585+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fb060000bf0 con 0x7fb068038730 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.593+0000 7fb06f7fe700 1 -- 172.21.15.96:0/1743218213 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fb060000bf0 con 0x7fb068038730 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb068038730 msgr2=0x7fb06803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 --2- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb068038730 0x7fb06803abf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb074006fd0 tx=0x7fb074006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 msgr2=0x7fb08019a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 --2- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb08019a370 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb070009fb0 tx=0x7fb070011890 comp rx=0 tx=0).stop 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 shutdown_connections 2024-04-22T15:09:07.791 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 --2- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb068038730 0x7fb06803abf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:07.792 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 --2- 172.21.15.96:0/1743218213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0801070b0 0x7fb08019a370 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:07.792 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 >> 172.21.15.96:0/1743218213 conn(0x7fb080100ac0 msgr2=0x7fb0801015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:07.792 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 shutdown_connections 2024-04-22T15:09:07.792 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:06.595+0000 7fb084a0b700 1 -- 172.21.15.96:0/1743218213 wait complete. 2024-04-22T15:09:07.792 INFO:teuthology.orchestra.run.smithi096.stdout:Deploying prometheus service with default placement... 2024-04-22T15:09:07.888 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:07 smithi096 ceph-mon[26688]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:07.889 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:07 smithi096 ceph-mon[26688]: Saving service ceph-exporter spec with placement * 2024-04-22T15:09:07.889 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:07 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:07.889 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:07 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:09.035 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.206+0000 7fc51ce7d700 1 Processor -- start 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.206+0000 7fc51ce7d700 1 -- start start 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.207+0000 7fc51ce7d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc518106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.207+0000 7fc51ce7d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518107490 con 0x7fc518106aa0 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.207+0000 7fc51659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc518106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.207+0000 7fc51659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc518106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57504/0 (socket says 172.21.15.96:57504) 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.207+0000 7fc51659c700 1 -- 172.21.15.96:0/509912419 learned_addr learned my addr 172.21.15.96:0/509912419 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.207+0000 7fc51659c700 1 -- 172.21.15.96:0/509912419 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc518105700 con 0x7fc518106aa0 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.208+0000 7fc51659c700 1 --2- 172.21.15.96:0/509912419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc518106ec0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc5080098a0 tx=0x7fc508009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e211ffdd8f26088e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.208+0000 7fc51559a700 1 -- 172.21.15.96:0/509912419 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc508004180 con 0x7fc518106aa0 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.208+0000 7fc51559a700 1 -- 172.21.15.96:0/509912419 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5080042e0 con 0x7fc518106aa0 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.208+0000 7fc51559a700 1 -- 172.21.15.96:0/509912419 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc508011b10 con 0x7fc518106aa0 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.208+0000 7fc51ce7d700 1 -- 172.21.15.96:0/509912419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 msgr2=0x7fc518106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:09.036 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.208+0000 7fc51ce7d700 1 --2- 172.21.15.96:0/509912419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc518106ec0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc5080098a0 tx=0x7fc508009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.208+0000 7fc51ce7d700 1 -- 172.21.15.96:0/509912419 shutdown_connections 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.208+0000 7fc51ce7d700 1 --2- 172.21.15.96:0/509912419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc518106ec0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.208+0000 7fc51ce7d700 1 -- 172.21.15.96:0/509912419 >> 172.21.15.96:0/509912419 conn(0x7fc518100270 msgr2=0x7fc518102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.209+0000 7fc51ce7d700 1 -- 172.21.15.96:0/509912419 shutdown_connections 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.209+0000 7fc51ce7d700 1 -- 172.21.15.96:0/509912419 wait complete. 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.209+0000 7fc51ce7d700 1 Processor -- start 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.209+0000 7fc51ce7d700 1 -- start start 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.209+0000 7fc51ce7d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc51819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.209+0000 7fc51ce7d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51819a8e0 con 0x7fc518106aa0 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc51659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc51819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc51659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc51819a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57506/0 (socket says 172.21.15.96:57506) 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc51659c700 1 -- 172.21.15.96:0/297497651 learned_addr learned my addr 172.21.15.96:0/297497651 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc51659c700 1 -- 172.21.15.96:0/297497651 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc508009550 con 0x7fc518106aa0 2024-04-22T15:09:09.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc51659c700 1 --2- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc51819a3a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc508009fb0 tx=0x7fc508011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc5077fe700 1 -- 172.21.15.96:0/297497651 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5080038b0 con 0x7fc518106aa0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc5077fe700 1 -- 172.21.15.96:0/297497651 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc508003a10 con 0x7fc518106aa0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51819aae0 con 0x7fc518106aa0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc5077fe700 1 -- 172.21.15.96:0/297497651 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc508019750 con 0x7fc518106aa0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.210+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51819af80 con 0x7fc518106aa0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.211+0000 7fc5077fe700 1 -- 172.21.15.96:0/297497651 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fc5080199c0 con 0x7fc518106aa0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.211+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5181945a0 con 0x7fc518106aa0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.212+0000 7fc5077fe700 1 --2- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fc5000386e0 0x7fc50003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.212+0000 7fc5077fe700 1 -- 172.21.15.96:0/297497651 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc50804c8d0 con 0x7fc518106aa0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.212+0000 7fc515d9b700 1 --2- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fc5000386e0 0x7fc50003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.212+0000 7fc515d9b700 1 --2- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fc5000386e0 0x7fc50003aba0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc50c006fd0 tx=0x7fc50c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.216+0000 7fc5077fe700 1 -- 172.21.15.96:0/297497651 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fc508010380 con 0x7fc518106aa0 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.424+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc51802cf40 con 0x7fc5000386e0 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.429+0000 7fc5077fe700 1 -- 172.21.15.96:0/297497651 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fc51802cf40 con 0x7fc5000386e0 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.432+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fc5000386e0 msgr2=0x7fc50003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.432+0000 7fc51ce7d700 1 --2- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fc5000386e0 0x7fc50003aba0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc50c006fd0 tx=0x7fc50c006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.432+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 msgr2=0x7fc51819a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.432+0000 7fc51ce7d700 1 --2- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc51819a3a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc508009fb0 tx=0x7fc508011890 comp rx=0 tx=0).stop 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.432+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 shutdown_connections 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.432+0000 7fc51ce7d700 1 --2- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fc5000386e0 0x7fc50003aba0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.432+0000 7fc51ce7d700 1 --2- 172.21.15.96:0/297497651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc518106aa0 0x7fc51819a3a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.432+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 >> 172.21.15.96:0/297497651 conn(0x7fc518100270 msgr2=0x7fc5181010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.432+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 shutdown_connections 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:08.433+0000 7fc51ce7d700 1 -- 172.21.15.96:0/297497651 wait complete. 2024-04-22T15:09:09.039 INFO:teuthology.orchestra.run.smithi096.stdout:Deploying grafana service with default placement... 2024-04-22T15:09:09.686 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:09 smithi096 ceph-mon[26688]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:09.686 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:09 smithi096 ceph-mon[26688]: Saving service prometheus spec with placement count:1 2024-04-22T15:09:09.687 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:09 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:09.687 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:09 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:10.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-22T15:09:10.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.460+0000 7f1fd2d23700 1 Processor -- start 2024-04-22T15:09:10.596 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.460+0000 7f1fd2d23700 1 -- start start 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.460+0000 7f1fd2d23700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc109450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.460+0000 7f1fd2d23700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc074770 con 0x7f1fcc107060 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.460+0000 7f1fd0abf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.460+0000 7f1fd0abf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57512/0 (socket says 172.21.15.96:57512) 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.460+0000 7f1fd0abf700 1 -- 172.21.15.96:0/3560091032 learned_addr learned my addr 172.21.15.96:0/3560091032 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.461+0000 7f1fd0abf700 1 -- 172.21.15.96:0/3560091032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc109990 con 0x7f1fcc107060 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.461+0000 7f1fd0abf700 1 --2- 172.21.15.96:0/3560091032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc109450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00098a0 tx=0x7f1fc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90296d885f08e260 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.461+0000 7f1fcb7fe700 1 -- 172.21.15.96:0/3560091032 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc0004180 con 0x7f1fcc107060 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.461+0000 7f1fcb7fe700 1 -- 172.21.15.96:0/3560091032 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fc00042e0 con 0x7f1fcc107060 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.461+0000 7f1fcb7fe700 1 -- 172.21.15.96:0/3560091032 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc0011b10 con 0x7f1fcc107060 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.462+0000 7f1fd2d23700 1 -- 172.21.15.96:0/3560091032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 msgr2=0x7f1fcc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.462+0000 7f1fd2d23700 1 --2- 172.21.15.96:0/3560091032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc109450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00098a0 tx=0x7f1fc0009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.462+0000 7f1fd2d23700 1 -- 172.21.15.96:0/3560091032 shutdown_connections 2024-04-22T15:09:10.597 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.462+0000 7f1fd2d23700 1 --2- 172.21.15.96:0/3560091032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc109450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.462+0000 7f1fd2d23700 1 -- 172.21.15.96:0/3560091032 >> 172.21.15.96:0/3560091032 conn(0x7f1fcc100a50 msgr2=0x7f1fcc102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.462+0000 7f1fd2d23700 1 -- 172.21.15.96:0/3560091032 shutdown_connections 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.462+0000 7f1fd2d23700 1 -- 172.21.15.96:0/3560091032 wait complete. 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.463+0000 7f1fd2d23700 1 Processor -- start 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.463+0000 7f1fd2d23700 1 -- start start 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.463+0000 7f1fd2d23700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.463+0000 7f1fd2d23700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc19a850 con 0x7f1fcc107060 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.463+0000 7f1fd0abf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.463+0000 7f1fd0abf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc19a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57514/0 (socket says 172.21.15.96:57514) 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.463+0000 7f1fd0abf700 1 -- 172.21.15.96:0/2648203338 learned_addr learned my addr 172.21.15.96:0/2648203338 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.463+0000 7f1fd0abf700 1 -- 172.21.15.96:0/2648203338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc0009550 con 0x7f1fcc107060 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.464+0000 7f1fd0abf700 1 --2- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc19a310 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00117b0 tx=0x7f1fc0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:10.598 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.464+0000 7f1fc9ffb700 1 -- 172.21.15.96:0/2648203338 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc00038b0 con 0x7f1fcc107060 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.464+0000 7f1fc9ffb700 1 -- 172.21.15.96:0/2648203338 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fc0003a10 con 0x7f1fcc107060 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.464+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc19aa50 con 0x7f1fcc107060 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.464+0000 7f1fc9ffb700 1 -- 172.21.15.96:0/2648203338 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc0024780 con 0x7f1fcc107060 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.464+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc19aef0 con 0x7f1fcc107060 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.465+0000 7f1fc9ffb700 1 -- 172.21.15.96:0/2648203338 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f1fc00248e0 con 0x7f1fcc107060 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.465+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc10a720 con 0x7f1fcc107060 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.465+0000 7f1fc9ffb700 1 --2- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1fbc038720 0x7f1fbc03abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.465+0000 7f1fc9ffb700 1 -- 172.21.15.96:0/2648203338 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1fc004c7f0 con 0x7f1fcc107060 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.465+0000 7f1fcbfff700 1 --2- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1fbc038720 0x7f1fbc03abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.466+0000 7f1fcbfff700 1 --2- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1fbc038720 0x7f1fbc03abe0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8006fd0 tx=0x7f1fb8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.470+0000 7f1fc9ffb700 1 -- 172.21.15.96:0/2648203338 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f1fc0024b90 con 0x7f1fcc107060 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.682+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fcc101ce0 con 0x7f1fbc038720 2024-04-22T15:09:10.599 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.688+0000 7f1fc9ffb700 1 -- 172.21.15.96:0/2648203338 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f1fcc101ce0 con 0x7f1fbc038720 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1fbc038720 msgr2=0x7f1fbc03abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 --2- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1fbc038720 0x7f1fbc03abe0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8006fd0 tx=0x7f1fb8006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 msgr2=0x7f1fcc19a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 --2- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc19a310 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00117b0 tx=0x7f1fc0011890 comp rx=0 tx=0).stop 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 shutdown_connections 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 --2- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f1fbc038720 0x7f1fbc03abe0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 --2- 172.21.15.96:0/2648203338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fcc107060 0x7f1fcc19a310 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 >> 172.21.15.96:0/2648203338 conn(0x7f1fcc100a50 msgr2=0x7f1fcc1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 shutdown_connections 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:09.690+0000 7f1fd2d23700 1 -- 172.21.15.96:0/2648203338 wait complete. 2024-04-22T15:09:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:Deploying node-exporter service with default placement... 2024-04-22T15:09:10.720 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:10 smithi096 ceph-mon[26688]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:10.720 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:10 smithi096 ceph-mon[26688]: Saving service grafana spec with placement count:1 2024-04-22T15:09:10.720 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:10 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:12.037 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.041+0000 7f671ee2a700 1 Processor -- start 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.041+0000 7f671ee2a700 1 -- start start 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.041+0000 7f671ee2a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f6718106a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.041+0000 7f671ee2a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6718106ff0 con 0x7f6718106600 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.041+0000 7f671cbc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f6718106a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.041+0000 7f671cbc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f6718106a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40532/0 (socket says 172.21.15.96:40532) 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.041+0000 7f671cbc6700 1 -- 172.21.15.96:0/805408520 learned_addr learned my addr 172.21.15.96:0/805408520 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f671cbc6700 1 -- 172.21.15.96:0/805408520 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67181077b0 con 0x7f6718106600 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f671cbc6700 1 --2- 172.21.15.96:0/805408520 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f6718106a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f670c0098a0 tx=0x7f670c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5077c7dd0de73605 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f67177fe700 1 -- 172.21.15.96:0/805408520 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670c004180 con 0x7f6718106600 2024-04-22T15:09:12.038 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f67177fe700 1 -- 172.21.15.96:0/805408520 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f670c0042e0 con 0x7f6718106600 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f67177fe700 1 -- 172.21.15.96:0/805408520 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670c011b10 con 0x7f6718106600 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f671ee2a700 1 -- 172.21.15.96:0/805408520 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 msgr2=0x7f6718106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f671ee2a700 1 --2- 172.21.15.96:0/805408520 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f6718106a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f670c0098a0 tx=0x7f670c009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f671ee2a700 1 -- 172.21.15.96:0/805408520 shutdown_connections 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f671ee2a700 1 --2- 172.21.15.96:0/805408520 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f6718106a20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.042+0000 7f671ee2a700 1 -- 172.21.15.96:0/805408520 >> 172.21.15.96:0/805408520 conn(0x7f6718101be0 msgr2=0x7f6718104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.043+0000 7f671ee2a700 1 -- 172.21.15.96:0/805408520 shutdown_connections 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.043+0000 7f671ee2a700 1 -- 172.21.15.96:0/805408520 wait complete. 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.043+0000 7f671ee2a700 1 Processor -- start 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.043+0000 7f671ee2a700 1 -- start start 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.043+0000 7f671ee2a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f671807baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.043+0000 7f671ee2a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671807c030 con 0x7f6718106600 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.043+0000 7f671cbc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f671807baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:12.039 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.043+0000 7f671cbc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f671807baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40542/0 (socket says 172.21.15.96:40542) 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.043+0000 7f671cbc6700 1 -- 172.21.15.96:0/2021998286 learned_addr learned my addr 172.21.15.96:0/2021998286 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.044+0000 7f671cbc6700 1 -- 172.21.15.96:0/2021998286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c009550 con 0x7f6718106600 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.044+0000 7f671cbc6700 1 --2- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f671807baf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f670c0117b0 tx=0x7f670c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.044+0000 7f6715ffb700 1 -- 172.21.15.96:0/2021998286 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670c0038b0 con 0x7f6718106600 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.044+0000 7f6715ffb700 1 -- 172.21.15.96:0/2021998286 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f670c003a10 con 0x7f6718106600 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.044+0000 7f6715ffb700 1 -- 172.21.15.96:0/2021998286 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670c0246e0 con 0x7f6718106600 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.044+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671807a2c0 con 0x7f6718106600 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.044+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671807a760 con 0x7f6718106600 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.045+0000 7f6715ffb700 1 -- 172.21.15.96:0/2021998286 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f670c01ab80 con 0x7f6718106600 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.045+0000 7f6715ffb700 1 --2- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f6708038770 0x7f670803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.045+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6718062510 con 0x7f6718106600 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.045+0000 7f6715ffb700 1 -- 172.21.15.96:0/2021998286 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f670c04c860 con 0x7f6718106600 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.045+0000 7f6717fff700 1 --2- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f6708038770 0x7f670803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.046+0000 7f6717fff700 1 --2- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f6708038770 0x7f670803ac30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6704006fd0 tx=0x7f6704006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.050+0000 7f6715ffb700 1 -- 172.21.15.96:0/2021998286 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f670c02d3f0 con 0x7f6718106600 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.259+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f671807ad10 con 0x7f6708038770 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.263+0000 7f6715ffb700 1 -- 172.21.15.96:0/2021998286 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f671807ad10 con 0x7f6708038770 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.265+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f6708038770 msgr2=0x7f670803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.265+0000 7f671ee2a700 1 --2- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f6708038770 0x7f670803ac30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6704006fd0 tx=0x7f6704006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.266+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 msgr2=0x7f671807baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.266+0000 7f671ee2a700 1 --2- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f671807baf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f670c0117b0 tx=0x7f670c011890 comp rx=0 tx=0).stop 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.266+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 shutdown_connections 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.266+0000 7f671ee2a700 1 --2- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f6708038770 0x7f670803ac30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.266+0000 7f671ee2a700 1 --2- 172.21.15.96:0/2021998286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6718106600 0x7f671807baf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.266+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 >> 172.21.15.96:0/2021998286 conn(0x7f6718101be0 msgr2=0x7f67181027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.266+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 shutdown_connections 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:11.266+0000 7f671ee2a700 1 -- 172.21.15.96:0/2021998286 wait complete. 2024-04-22T15:09:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:Deploying alertmanager service with default placement... 2024-04-22T15:09:12.365 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:12 smithi096 ceph-mon[26688]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:12.366 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:12 smithi096 ceph-mon[26688]: Saving service node-exporter spec with placement * 2024-04-22T15:09:12.366 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:12 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:12.366 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:12 smithi096 ceph-mon[26688]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:13.194 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-22T15:09:13.194 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.461+0000 7fb59dd84700 1 Processor -- start 2024-04-22T15:09:13.194 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb59dd84700 1 -- start start 2024-04-22T15:09:13.194 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb59dd84700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb598105de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:13.194 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb59dd84700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5981063b0 con 0x7fb5981059c0 2024-04-22T15:09:13.194 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb5977fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb598105de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:13.194 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb5977fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb598105de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40554/0 (socket says 172.21.15.96:40554) 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb5977fe700 1 -- 172.21.15.96:0/1265417034 learned_addr learned my addr 172.21.15.96:0/1265417034 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb5977fe700 1 -- 172.21.15.96:0/1265417034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5981066e0 con 0x7fb5981059c0 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb5977fe700 1 --2- 172.21.15.96:0/1265417034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb598105de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb58800bb40 tx=0x7fb58800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808dfed3d274330b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb5967fc700 1 -- 172.21.15.96:0/1265417034 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb58800dc00 con 0x7fb5981059c0 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb5967fc700 1 -- 172.21.15.96:0/1265417034 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5880044f0 con 0x7fb5981059c0 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.462+0000 7fb5967fc700 1 -- 172.21.15.96:0/1265417034 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb588010470 con 0x7fb5981059c0 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.463+0000 7fb59dd84700 1 -- 172.21.15.96:0/1265417034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 msgr2=0x7fb598105de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.463+0000 7fb59dd84700 1 --2- 172.21.15.96:0/1265417034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb598105de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb58800bb40 tx=0x7fb58800be50 comp rx=0 tx=0).stop 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.463+0000 7fb59dd84700 1 -- 172.21.15.96:0/1265417034 shutdown_connections 2024-04-22T15:09:13.195 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.463+0000 7fb59dd84700 1 --2- 172.21.15.96:0/1265417034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb598105de0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.463+0000 7fb59dd84700 1 -- 172.21.15.96:0/1265417034 >> 172.21.15.96:0/1265417034 conn(0x7fb5981001c0 msgr2=0x7fb598102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.463+0000 7fb59dd84700 1 -- 172.21.15.96:0/1265417034 shutdown_connections 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.463+0000 7fb59dd84700 1 -- 172.21.15.96:0/1265417034 wait complete. 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.464+0000 7fb59dd84700 1 Processor -- start 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.464+0000 7fb59dd84700 1 -- start start 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.464+0000 7fb59dd84700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb59819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.464+0000 7fb59dd84700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59819ca70 con 0x7fb5981059c0 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.464+0000 7fb5977fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb59819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.464+0000 7fb5977fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb59819c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40566/0 (socket says 172.21.15.96:40566) 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.464+0000 7fb5977fe700 1 -- 172.21.15.96:0/376760278 learned_addr learned my addr 172.21.15.96:0/376760278 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.464+0000 7fb5977fe700 1 -- 172.21.15.96:0/376760278 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58800b7f0 con 0x7fb5981059c0 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.464+0000 7fb5977fe700 1 --2- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb59819c530 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb588003cb0 tx=0x7fb588004070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:13.196 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.465+0000 7fb594ff9700 1 -- 172.21.15.96:0/376760278 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb58800de20 con 0x7fb5981059c0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.465+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59819cc70 con 0x7fb5981059c0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.465+0000 7fb594ff9700 1 -- 172.21.15.96:0/376760278 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb588010b60 con 0x7fb5981059c0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.465+0000 7fb594ff9700 1 -- 172.21.15.96:0/376760278 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb588021aa0 con 0x7fb5981059c0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.465+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59819d0b0 con 0x7fb5981059c0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.465+0000 7fb594ff9700 1 -- 172.21.15.96:0/376760278 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fb5880193f0 con 0x7fb5981059c0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.466+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb598104520 con 0x7fb5981059c0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.466+0000 7fb594ff9700 1 --2- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb580038720 0x7fb58003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.466+0000 7fb594ff9700 1 -- 172.21.15.96:0/376760278 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb58804c990 con 0x7fb5981059c0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.466+0000 7fb596ffd700 1 --2- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb580038720 0x7fb58003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.466+0000 7fb596ffd700 1 --2- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb580038720 0x7fb58003abe0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb58c006fd0 tx=0x7fb58c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.470+0000 7fb594ff9700 1 -- 172.21.15.96:0/376760278 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb58800e360 con 0x7fb5981059c0 2024-04-22T15:09:13.197 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.678+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb598061320 con 0x7fb580038720 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.683+0000 7fb594ff9700 1 -- 172.21.15.96:0/376760278 <== mgr.14120 v2:172.21.15.96:6800/3275369520 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb598061320 con 0x7fb580038720 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.685+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb580038720 msgr2=0x7fb58003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.685+0000 7fb59dd84700 1 --2- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb580038720 0x7fb58003abe0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb58c006fd0 tx=0x7fb58c006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.685+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 msgr2=0x7fb59819c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.685+0000 7fb59dd84700 1 --2- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb59819c530 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb588003cb0 tx=0x7fb588004070 comp rx=0 tx=0).stop 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.685+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 shutdown_connections 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.685+0000 7fb59dd84700 1 --2- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb580038720 0x7fb58003abe0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.685+0000 7fb59dd84700 1 --2- 172.21.15.96:0/376760278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5981059c0 0x7fb59819c530 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.685+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 >> 172.21.15.96:0/376760278 conn(0x7fb5981001c0 msgr2=0x7fb59818e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.685+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 shutdown_connections 2024-04-22T15:09:13.198 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:12.686+0000 7fb59dd84700 1 -- 172.21.15.96:0/376760278 wait complete. 2024-04-22T15:09:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:13 smithi096 ceph-mon[26688]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:13 smithi096 ceph-mon[26688]: Saving service alertmanager spec with placement count:1 2024-04-22T15:09:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:13 smithi096 ceph-mon[26688]: from='mgr.14120 172.21.15.96:0/2857890932' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:13 smithi096 ceph-mon[26688]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:14.441 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.630+0000 7f6069ebd700 1 Processor -- start 2024-04-22T15:09:14.441 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.630+0000 7f6069ebd700 1 -- start start 2024-04-22T15:09:14.441 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.630+0000 7f6069ebd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f6064109450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.631+0000 7f6069ebd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6064074770 con 0x7f6064107060 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.631+0000 7f60637fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f6064109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.631+0000 7f60637fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f6064109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40582/0 (socket says 172.21.15.96:40582) 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.631+0000 7f60637fe700 1 -- 172.21.15.96:0/2726197398 learned_addr learned my addr 172.21.15.96:0/2726197398 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.631+0000 7f60637fe700 1 -- 172.21.15.96:0/2726197398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064109990 con 0x7f6064107060 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.631+0000 7f60637fe700 1 --2- 172.21.15.96:0/2726197398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f6064109450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6054009e80 tx=0x7f605400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e873865c4f342ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.631+0000 7f60627fc700 1 -- 172.21.15.96:0/2726197398 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f605400e070 con 0x7f6064107060 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.631+0000 7f60627fc700 1 -- 172.21.15.96:0/2726197398 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f605400b8c0 con 0x7f6064107060 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.631+0000 7f60627fc700 1 -- 172.21.15.96:0/2726197398 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60540044d0 con 0x7f6064107060 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.632+0000 7f6069ebd700 1 -- 172.21.15.96:0/2726197398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 msgr2=0x7f6064109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.632+0000 7f6069ebd700 1 --2- 172.21.15.96:0/2726197398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f6064109450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6054009e80 tx=0x7f605400b330 comp rx=0 tx=0).stop 2024-04-22T15:09:14.442 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.632+0000 7f6069ebd700 1 -- 172.21.15.96:0/2726197398 shutdown_connections 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.632+0000 7f6069ebd700 1 --2- 172.21.15.96:0/2726197398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f6064109450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.632+0000 7f6069ebd700 1 -- 172.21.15.96:0/2726197398 >> 172.21.15.96:0/2726197398 conn(0x7f6064100a50 msgr2=0x7f6064102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.632+0000 7f6069ebd700 1 -- 172.21.15.96:0/2726197398 shutdown_connections 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.632+0000 7f6069ebd700 1 -- 172.21.15.96:0/2726197398 wait complete. 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.632+0000 7f6069ebd700 1 Processor -- start 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f6069ebd700 1 -- start start 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f6069ebd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f606419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f6069ebd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606419a810 con 0x7f6064107060 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f60637fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f606419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f60637fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f606419a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40596/0 (socket says 172.21.15.96:40596) 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f60637fe700 1 -- 172.21.15.96:0/3660550637 learned_addr learned my addr 172.21.15.96:0/3660550637 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:14.443 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f60637fe700 1 -- 172.21.15.96:0/3660550637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6054009b30 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f60637fe700 1 --2- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f606419a2d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60540072a0 tx=0x7f605400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f6060ff9700 1 -- 172.21.15.96:0/3660550637 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f605400e040 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.633+0000 7f6060ff9700 1 -- 172.21.15.96:0/3660550637 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6054003dc0 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.634+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606419aa10 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.634+0000 7f6060ff9700 1 -- 172.21.15.96:0/3660550637 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6054012750 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.634+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606419af10 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.634+0000 7f6060ff9700 1 -- 172.21.15.96:0/3660550637 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f6054019040 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.635+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606410a720 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.635+0000 7f6060ff9700 1 --2- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f604c038770 0x7f604c03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.635+0000 7f6060ff9700 1 -- 172.21.15.96:0/3660550637 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f605404c0b0 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.635+0000 7f6062ffd700 1 --2- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f604c038770 0x7f604c03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.635+0000 7f6062ffd700 1 --2- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f604c038770 0x7f604c03ac30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6058006fd0 tx=0x7f6058006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.639+0000 7f6060ff9700 1 -- 172.21.15.96:0/3660550637 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f605401b430 con 0x7f6064107060 2024-04-22T15:09:14.444 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.834+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f606410a990 con 0x7f6064107060 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.838+0000 7f6060ff9700 1 -- 172.21.15.96:0/3660550637 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f605401f070 con 0x7f6064107060 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f604c038770 msgr2=0x7f604c03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 --2- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f604c038770 0x7f604c03ac30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6058006fd0 tx=0x7f6058006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 msgr2=0x7f606419a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 --2- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f606419a2d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60540072a0 tx=0x7f605400be00 comp rx=0 tx=0).stop 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 shutdown_connections 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 --2- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f604c038770 0x7f604c03ac30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 --2- 172.21.15.96:0/3660550637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6064107060 0x7f606419a2d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 >> 172.21.15.96:0/3660550637 conn(0x7f6064100a50 msgr2=0x7f6064101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 shutdown_connections 2024-04-22T15:09:14.445 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:13.840+0000 7f6069ebd700 1 -- 172.21.15.96:0/3660550637 wait complete. 2024-04-22T15:09:15.091 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:14 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3660550637' entity='client.admin' 2024-04-22T15:09:15.665 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.871+0000 7fb18ce6d700 1 Processor -- start 2024-04-22T15:09:15.665 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.871+0000 7fb18ce6d700 1 -- start start 2024-04-22T15:09:15.665 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.871+0000 7fb18ce6d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb1881058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:15.665 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.871+0000 7fb18ce6d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188105e90 con 0x7fb1881054a0 2024-04-22T15:09:15.665 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.872+0000 7fb18659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb1881058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:15.665 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.872+0000 7fb18659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb1881058c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40602/0 (socket says 172.21.15.96:40602) 2024-04-22T15:09:15.665 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.872+0000 7fb18659c700 1 -- 172.21.15.96:0/1672535028 learned_addr learned my addr 172.21.15.96:0/1672535028 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:15.665 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.872+0000 7fb18659c700 1 -- 172.21.15.96:0/1672535028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1881066a0 con 0x7fb1881054a0 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.872+0000 7fb18659c700 1 --2- 172.21.15.96:0/1672535028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb1881058c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1780098a0 tx=0x7fb178009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a1a3e36f993b99 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.872+0000 7fb185d9b700 1 -- 172.21.15.96:0/1672535028 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb178004180 con 0x7fb1881054a0 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.872+0000 7fb185d9b700 1 -- 172.21.15.96:0/1672535028 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1780042e0 con 0x7fb1881054a0 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.872+0000 7fb185d9b700 1 -- 172.21.15.96:0/1672535028 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb178011b10 con 0x7fb1881054a0 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.873+0000 7fb18ce6d700 1 -- 172.21.15.96:0/1672535028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 msgr2=0x7fb1881058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.873+0000 7fb18ce6d700 1 --2- 172.21.15.96:0/1672535028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb1881058c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1780098a0 tx=0x7fb178009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.873+0000 7fb18ce6d700 1 -- 172.21.15.96:0/1672535028 shutdown_connections 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.873+0000 7fb18ce6d700 1 --2- 172.21.15.96:0/1672535028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb1881058c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.873+0000 7fb18ce6d700 1 -- 172.21.15.96:0/1672535028 >> 172.21.15.96:0/1672535028 conn(0x7fb188100aa0 msgr2=0x7fb188102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.873+0000 7fb18ce6d700 1 -- 172.21.15.96:0/1672535028 shutdown_connections 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.873+0000 7fb18ce6d700 1 -- 172.21.15.96:0/1672535028 wait complete. 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.873+0000 7fb18ce6d700 1 Processor -- start 2024-04-22T15:09:15.666 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb18ce6d700 1 -- start start 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb18ce6d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb188190560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb18ce6d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188190aa0 con 0x7fb1881054a0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb18659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb188190560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb18659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb188190560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40606/0 (socket says 172.21.15.96:40606) 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb18659c700 1 -- 172.21.15.96:0/2583870111 learned_addr learned my addr 172.21.15.96:0/2583870111 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb18659c700 1 -- 172.21.15.96:0/2583870111 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb178009550 con 0x7fb1881054a0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb18659c700 1 --2- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb188190560 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb178009fb0 tx=0x7fb178011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb17ffff700 1 -- 172.21.15.96:0/2583870111 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1780038b0 con 0x7fb1881054a0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18818ece0 con 0x7fb1881054a0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb17ffff700 1 -- 172.21.15.96:0/2583870111 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb178003a10 con 0x7fb1881054a0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.874+0000 7fb17ffff700 1 -- 172.21.15.96:0/2583870111 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb178019770 con 0x7fb1881054a0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.875+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18818f180 con 0x7fb1881054a0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.875+0000 7fb17ffff700 1 -- 172.21.15.96:0/2583870111 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fb1780198d0 con 0x7fb1881054a0 2024-04-22T15:09:15.667 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.875+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb188062510 con 0x7fb1881054a0 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.875+0000 7fb17ffff700 1 --2- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb1740386e0 0x7fb17403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.876+0000 7fb17ffff700 1 -- 172.21.15.96:0/2583870111 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb17804c7f0 con 0x7fb1881054a0 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.876+0000 7fb17dd9b700 1 --2- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb1740386e0 0x7fb17403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.876+0000 7fb17dd9b700 1 --2- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb1740386e0 0x7fb17403aba0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb170006fd0 tx=0x7fb170006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:14.880+0000 7fb17ffff700 1 -- 172.21.15.96:0/2583870111 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb178022d70 con 0x7fb1881054a0 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.070+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb188194070 con 0x7fb1881054a0 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.074+0000 7fb17ffff700 1 -- 172.21.15.96:0/2583870111 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb178022310 con 0x7fb1881054a0 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb1740386e0 msgr2=0x7fb17403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 --2- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb1740386e0 0x7fb17403aba0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb170006fd0 tx=0x7fb170006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 msgr2=0x7fb188190560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 --2- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb188190560 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb178009fb0 tx=0x7fb178011890 comp rx=0 tx=0).stop 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 shutdown_connections 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 --2- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7fb1740386e0 0x7fb17403aba0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 --2- 172.21.15.96:0/2583870111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1881054a0 0x7fb188190560 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:15.668 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 >> 172.21.15.96:0/2583870111 conn(0x7fb188100aa0 msgr2=0x7fb188101990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:15.669 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 shutdown_connections 2024-04-22T15:09:15.669 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:15.076+0000 7fb18ce6d700 1 -- 172.21.15.96:0/2583870111 wait complete. 2024-04-22T15:09:15.669 INFO:teuthology.orchestra.run.smithi096.stdout:Enabling the dashboard module... 2024-04-22T15:09:16.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:16 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2583870111' entity='client.admin' 2024-04-22T15:09:16.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:16 smithi096 ceph-mon[26688]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:17.446 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:17 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1276342780' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T15:09:17.772 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.080+0000 7efe7967f700 1 Processor -- start 2024-04-22T15:09:17.772 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.080+0000 7efe7967f700 1 -- start start 2024-04-22T15:09:17.772 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.080+0000 7efe7967f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe74104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:17.772 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.080+0000 7efe7967f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe741052c0 con 0x7efe741048d0 2024-04-22T15:09:17.772 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.080+0000 7efe72ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe74104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:17.772 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.080+0000 7efe72ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe74104cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40618/0 (socket says 172.21.15.96:40618) 2024-04-22T15:09:17.772 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.080+0000 7efe72ffd700 1 -- 172.21.15.96:0/1712288400 learned_addr learned my addr 172.21.15.96:0/1712288400 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:17.772 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.080+0000 7efe72ffd700 1 -- 172.21.15.96:0/1712288400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe74105400 con 0x7efe741048d0 2024-04-22T15:09:17.772 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.080+0000 7efe72ffd700 1 --2- 172.21.15.96:0/1712288400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe74104cf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efe640098a0 tx=0x7efe64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ad35aa66892144 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe71ffb700 1 -- 172.21.15.96:0/1712288400 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe64004180 con 0x7efe741048d0 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe71ffb700 1 -- 172.21.15.96:0/1712288400 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe640042e0 con 0x7efe741048d0 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe71ffb700 1 -- 172.21.15.96:0/1712288400 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe64011b10 con 0x7efe741048d0 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe7967f700 1 -- 172.21.15.96:0/1712288400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 msgr2=0x7efe74104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe7967f700 1 --2- 172.21.15.96:0/1712288400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe74104cf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efe640098a0 tx=0x7efe64009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe7967f700 1 -- 172.21.15.96:0/1712288400 shutdown_connections 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe7967f700 1 --2- 172.21.15.96:0/1712288400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe74104cf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe7967f700 1 -- 172.21.15.96:0/1712288400 >> 172.21.15.96:0/1712288400 conn(0x7efe74100290 msgr2=0x7efe741026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe7967f700 1 -- 172.21.15.96:0/1712288400 shutdown_connections 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.081+0000 7efe7967f700 1 -- 172.21.15.96:0/1712288400 wait complete. 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.082+0000 7efe7967f700 1 Processor -- start 2024-04-22T15:09:17.773 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.082+0000 7efe7967f700 1 -- start start 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.082+0000 7efe7967f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe7419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.082+0000 7efe7967f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7419a8f0 con 0x7efe741048d0 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.082+0000 7efe72ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe7419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.082+0000 7efe72ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe7419a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40622/0 (socket says 172.21.15.96:40622) 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.082+0000 7efe72ffd700 1 -- 172.21.15.96:0/1276342780 learned_addr learned my addr 172.21.15.96:0/1276342780 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.083+0000 7efe72ffd700 1 -- 172.21.15.96:0/1276342780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe64009550 con 0x7efe741048d0 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.083+0000 7efe72ffd700 1 --2- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe7419a3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efe640117b0 tx=0x7efe64011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.083+0000 7efe5bfff700 1 -- 172.21.15.96:0/1276342780 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe640038b0 con 0x7efe741048d0 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.083+0000 7efe5bfff700 1 -- 172.21.15.96:0/1276342780 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe64003a10 con 0x7efe741048d0 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.083+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7419aaf0 con 0x7efe741048d0 2024-04-22T15:09:17.774 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.083+0000 7efe5bfff700 1 -- 172.21.15.96:0/1276342780 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe640246e0 con 0x7efe741048d0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.083+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7419af90 con 0x7efe741048d0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.084+0000 7efe5bfff700 1 -- 172.21.15.96:0/1276342780 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7efe6402d3f0 con 0x7efe741048d0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.084+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe741085c0 con 0x7efe741048d0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.084+0000 7efe5bfff700 1 --2- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7efe5c038770 0x7efe5c03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.084+0000 7efe5bfff700 1 -- 172.21.15.96:0/1276342780 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efe6404c990 con 0x7efe741048d0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.084+0000 7efe727fc700 1 --2- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7efe5c038770 0x7efe5c03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.084+0000 7efe727fc700 1 --2- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7efe5c038770 0x7efe5c03ac30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efe68006fd0 tx=0x7efe68006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.089+0000 7efe5bfff700 1 -- 172.21.15.96:0/1276342780 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7efe64010970 con 0x7efe741048d0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:16.339+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7efe7419d880 con 0x7efe741048d0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.080+0000 7efe5bfff700 1 -- 172.21.15.96:0/1276342780 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 9) v1 ==== 45569+0+0 (secure 0 0 0) 0x7efe6401ab80 con 0x7efe741048d0 2024-04-22T15:09:17.775 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.081+0000 7efe5bfff700 1 -- 172.21.15.96:0/1276342780 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7efe64010970 con 0x7efe741048d0 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7efe5c038770 msgr2=0x7efe5c03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 --2- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7efe5c038770 0x7efe5c03ac30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efe68006fd0 tx=0x7efe68006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 msgr2=0x7efe7419a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 --2- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe7419a3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efe640117b0 tx=0x7efe64011890 comp rx=0 tx=0).stop 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 shutdown_connections 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 --2- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7efe5c038770 0x7efe5c03ac30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 --2- 172.21.15.96:0/1276342780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe741048d0 0x7efe7419a3b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 >> 172.21.15.96:0/1276342780 conn(0x7efe74100290 msgr2=0x7efe74100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 shutdown_connections 2024-04-22T15:09:17.776 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:17.084+0000 7efe7967f700 1 -- 172.21.15.96:0/1276342780 wait complete. 2024-04-22T15:09:18.390 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:18 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1276342780' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T15:09:18.390 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:18 smithi096 ceph-mon[26688]: mgrmap e9: smithi096.hhkvum(active, since 25s) 2024-04-22T15:09:19.068 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "active_name": "smithi096.hhkvum", 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.180+0000 7febeddc6700 1 Processor -- start 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.180+0000 7febeddc6700 1 -- start start 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.180+0000 7febeddc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe8104d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.180+0000 7febeddc6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe81052d0 con 0x7febe81048e0 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.180+0000 7febe77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe8104d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.180+0000 7febe77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe8104d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40650/0 (socket says 172.21.15.96:40650) 2024-04-22T15:09:19.069 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.180+0000 7febe77fe700 1 -- 172.21.15.96:0/518924963 learned_addr learned my addr 172.21.15.96:0/518924963 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.181+0000 7febe77fe700 1 -- 172.21.15.96:0/518924963 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe8107870 con 0x7febe81048e0 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.181+0000 7febe77fe700 1 --2- 172.21.15.96:0/518924963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe8104d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7febdc0098a0 tx=0x7febdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4744f8ece5528b36 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.181+0000 7febe67fc700 1 -- 172.21.15.96:0/518924963 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febdc004180 con 0x7febe81048e0 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.181+0000 7febe67fc700 1 -- 172.21.15.96:0/518924963 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febdc0042e0 con 0x7febe81048e0 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.181+0000 7febe67fc700 1 -- 172.21.15.96:0/518924963 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febdc011b10 con 0x7febe81048e0 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.182+0000 7febeddc6700 1 -- 172.21.15.96:0/518924963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 msgr2=0x7febe8104d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.182+0000 7febeddc6700 1 --2- 172.21.15.96:0/518924963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe8104d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7febdc0098a0 tx=0x7febdc009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.182+0000 7febeddc6700 1 -- 172.21.15.96:0/518924963 shutdown_connections 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.182+0000 7febeddc6700 1 --2- 172.21.15.96:0/518924963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe8104d00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.182+0000 7febeddc6700 1 -- 172.21.15.96:0/518924963 >> 172.21.15.96:0/518924963 conn(0x7febe8100230 msgr2=0x7febe8102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.182+0000 7febeddc6700 1 -- 172.21.15.96:0/518924963 shutdown_connections 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.182+0000 7febeddc6700 1 -- 172.21.15.96:0/518924963 wait complete. 2024-04-22T15:09:19.070 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.182+0000 7febeddc6700 1 Processor -- start 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.182+0000 7febeddc6700 1 -- start start 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febeddc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febeddc6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe819a8f0 con 0x7febe81048e0 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febe77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febe77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe819a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40658/0 (socket says 172.21.15.96:40658) 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febe77fe700 1 -- 172.21.15.96:0/2802573254 learned_addr learned my addr 172.21.15.96:0/2802573254 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febe77fe700 1 -- 172.21.15.96:0/2802573254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc009550 con 0x7febe81048e0 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febe77fe700 1 --2- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe819a3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7febdc0117b0 tx=0x7febdc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febe4ff9700 1 -- 172.21.15.96:0/2802573254 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febdc0038b0 con 0x7febe81048e0 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febe4ff9700 1 -- 172.21.15.96:0/2802573254 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febdc003a10 con 0x7febe81048e0 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe819aaf0 con 0x7febe81048e0 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe819af90 con 0x7febe81048e0 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.183+0000 7febe4ff9700 1 -- 172.21.15.96:0/2802573254 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febdc0246e0 con 0x7febe81048e0 2024-04-22T15:09:19.071 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.184+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd4005320 con 0x7febe81048e0 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.184+0000 7febe4ff9700 1 -- 172.21.15.96:0/2802573254 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) v1 ==== 45569+0+0 (secure 0 0 0) 0x7febdc02d3f0 con 0x7febe81048e0 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.185+0000 7febe4ff9700 1 --2- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7febd0038770 0x7febd003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.185+0000 7febe4ff9700 1 -- 172.21.15.96:0/2802573254 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7febdc04c8b0 con 0x7febe81048e0 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.185+0000 7febe6ffd700 1 -- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7febd0038770 msgr2=0x7febd003ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/3275369520 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.185+0000 7febe6ffd700 1 --2- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7febd0038770 0x7febd003ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.193+0000 7febe4ff9700 1 -- 172.21.15.96:0/2802573254 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7febdc010970 con 0x7febe81048e0 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.386+0000 7febe6ffd700 1 -- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7febd0038770 msgr2=0x7febd003ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/3275369520 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.386+0000 7febe6ffd700 1 --2- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7febd0038770 0x7febd003ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.483+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7febd4005de0 con 0x7febe81048e0 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.483+0000 7febe4ff9700 1 -- 172.21.15.96:0/2802573254 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7febdc010380 con 0x7febe81048e0 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7febd0038770 msgr2=0x7febd003ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 --2- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7febd0038770 0x7febd003ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:19.072 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 msgr2=0x7febe819a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:19.073 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 --2- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe819a3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7febdc0117b0 tx=0x7febdc011890 comp rx=0 tx=0).stop 2024-04-22T15:09:19.073 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 shutdown_connections 2024-04-22T15:09:19.073 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 --2- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7febd0038770 0x7febd003ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:19.073 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 --2- 172.21.15.96:0/2802573254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe81048e0 0x7febe819a3b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:19.073 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 >> 172.21.15.96:0/2802573254 conn(0x7febe8100230 msgr2=0x7febe8100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:19.073 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 shutdown_connections 2024-04-22T15:09:19.073 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:18.485+0000 7febeddc6700 1 -- 172.21.15.96:0/2802573254 wait complete. 2024-04-22T15:09:19.073 INFO:teuthology.orchestra.run.smithi096.stdout:Waiting for the mgr to restart... 2024-04-22T15:09:19.073 INFO:teuthology.orchestra.run.smithi096.stdout:Waiting for mgr epoch 9... 2024-04-22T15:09:19.329 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:19 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2802573254' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T15:09:24.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: Active manager daemon smithi096.hhkvum restarted 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: Activating manager daemon smithi096.hhkvum 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: mgrmap e10: smithi096.hhkvum(active, starting, since 0.00399303s) 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: Manager daemon smithi096.hhkvum is now available 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:09:24.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:23 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/trash_purge_schedule"}]: dispatch 2024-04-22T15:09:25.670 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:09:25.670 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.526+0000 7f922ccb6700 1 Processor -- start 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.526+0000 7f922ccb6700 1 -- start start 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922ccb6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f9228104d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922ccb6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281052d0 con 0x7f92281048e0 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f9228104d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f9228104d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40660/0 (socket says 172.21.15.96:40660) 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922659c700 1 -- 172.21.15.96:0/2525261818 learned_addr learned my addr 172.21.15.96:0/2525261818 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922659c700 1 -- 172.21.15.96:0/2525261818 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9228107870 con 0x7f92281048e0 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922659c700 1 --2- 172.21.15.96:0/2525261818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f9228104d00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f921c0098a0 tx=0x7f921c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14784bc9798e5b65 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:25.671 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922559a700 1 -- 172.21.15.96:0/2525261818 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f921c004180 con 0x7f92281048e0 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922559a700 1 -- 172.21.15.96:0/2525261818 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f921c0042e0 con 0x7f92281048e0 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.527+0000 7f922559a700 1 -- 172.21.15.96:0/2525261818 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f921c011b10 con 0x7f92281048e0 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.528+0000 7f922ccb6700 1 -- 172.21.15.96:0/2525261818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 msgr2=0x7f9228104d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.528+0000 7f922ccb6700 1 --2- 172.21.15.96:0/2525261818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f9228104d00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f921c0098a0 tx=0x7f921c009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.528+0000 7f922ccb6700 1 -- 172.21.15.96:0/2525261818 shutdown_connections 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.528+0000 7f922ccb6700 1 --2- 172.21.15.96:0/2525261818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f9228104d00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.528+0000 7f922ccb6700 1 -- 172.21.15.96:0/2525261818 >> 172.21.15.96:0/2525261818 conn(0x7f9228100270 msgr2=0x7f9228102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.528+0000 7f922ccb6700 1 -- 172.21.15.96:0/2525261818 shutdown_connections 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.528+0000 7f922ccb6700 1 -- 172.21.15.96:0/2525261818 wait complete. 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.529+0000 7f922ccb6700 1 Processor -- start 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.529+0000 7f922ccb6700 1 -- start start 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.529+0000 7f922ccb6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f92281070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.529+0000 7f922ccb6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281075e0 con 0x7f92281048e0 2024-04-22T15:09:25.672 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.529+0000 7f922659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f92281070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.529+0000 7f922659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f92281070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40672/0 (socket says 172.21.15.96:40672) 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.529+0000 7f922659c700 1 -- 172.21.15.96:0/2325882474 learned_addr learned my addr 172.21.15.96:0/2325882474 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.530+0000 7f922659c700 1 -- 172.21.15.96:0/2325882474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c009550 con 0x7f92281048e0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.530+0000 7f922659c700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f92281070a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f921c009fb0 tx=0x7f921c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.530+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f921c0038b0 con 0x7f92281048e0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.530+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f921c003a10 con 0x7f92281048e0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.530+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9228105820 con 0x7f92281048e0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.530+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f921c019770 con 0x7f92281048e0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.530+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9228105cc0 con 0x7f92281048e0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.531+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f921c0198d0 con 0x7f92281048e0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.531+0000 7f92177fe700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 0x7f921003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.531+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 --> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92281086d0 con 0x7f9210038780 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.531+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f921c04c800 con 0x7f92281048e0 2024-04-22T15:09:25.673 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.531+0000 7f9225d9b700 1 -- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 msgr2=0x7f921003ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/3275369520 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.531+0000 7f9225d9b700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 0x7f921003ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.732+0000 7f9225d9b700 1 -- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 msgr2=0x7f921003ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/3275369520 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:19.732+0000 7f9225d9b700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 0x7f921003ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:20.133+0000 7f9225d9b700 1 -- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 msgr2=0x7f921003ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/3275369520 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:20.133+0000 7f9225d9b700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 0x7f921003ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:20.934+0000 7f9225d9b700 1 -- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 msgr2=0x7f921003ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/3275369520 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:20.934+0000 7f9225d9b700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 0x7f921003ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:22.535+0000 7f9225d9b700 1 -- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 msgr2=0x7f921003ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/3275369520 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:22.535+0000 7f9225d9b700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 0x7f921003ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:23.921+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mgrmap(e 10) v1 ==== 45336+0+0 (secure 0 0 0) 0x7f921c02c910 con 0x7f92281048e0 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:23.922+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 msgr2=0x7f921003ac40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:23.922+0000 7f92177fe700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 0x7f921003ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.923+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 11) v1 ==== 45463+0+0 (secure 0 0 0) 0x7f921c00bcc0 con 0x7f92281048e0 2024-04-22T15:09:25.674 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.923+0000 7f92177fe700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f921003bff0 0x7f921003e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.923+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 --> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f921003e920 con 0x7f921003bff0 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.923+0000 7f9225d9b700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f921003bff0 0x7f921003e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.924+0000 7f9225d9b700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f921003bff0 0x7f921003e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9218003a10 tx=0x7f9218009270 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.924+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 <== mgr.14164 v2:172.21.15.96:6800/976363826 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f921003e920 con 0x7f921003bff0 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 --> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9228062510 con 0x7f921003bff0 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f92177fe700 1 -- 172.21.15.96:0/2325882474 <== mgr.14164 v2:172.21.15.96:6800/976363826 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f9228062510 con 0x7f921003bff0 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f921003bff0 msgr2=0x7f921003e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f921003bff0 0x7f921003e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9218003a10 tx=0x7f9218009270 comp rx=0 tx=0).stop 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 msgr2=0x7f92281070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f92281070a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f921c009fb0 tx=0x7f921c011890 comp rx=0 tx=0).stop 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 shutdown_connections 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f921003bff0 0x7f921003e3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92281048e0 0x7f92281070a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:25.675 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 --2- 172.21.15.96:0/2325882474 >> [v2:172.21.15.96:6800/3275369520,v1:172.21.15.96:6801/3275369520] conn(0x7f9210038780 0x7f921003ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:25.676 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 >> 172.21.15.96:0/2325882474 conn(0x7f9228100270 msgr2=0x7f9228100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:25.676 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 shutdown_connections 2024-04-22T15:09:25.676 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:24.931+0000 7f922ccb6700 1 -- 172.21.15.96:0/2325882474 wait complete. 2024-04-22T15:09:25.676 INFO:teuthology.orchestra.run.smithi096.stdout:mgr epoch 9 is available 2024-04-22T15:09:25.676 INFO:teuthology.orchestra.run.smithi096.stdout:Generating a dashboard self-signed certificate... 2024-04-22T15:09:26.172 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:25 smithi096 ceph-mon[26688]: mgrmap e11: smithi096.hhkvum(active, since 1.00578s) 2024-04-22T15:09:26.172 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:25 smithi096 ceph-mon[26688]: [22/Apr/2024:15:09:24] ENGINE Bus STARTING 2024-04-22T15:09:26.172 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:25 smithi096 ceph-mon[26688]: [22/Apr/2024:15:09:25] ENGINE Serving on http://172.21.15.96:8765 2024-04-22T15:09:26.172 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:25 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:26 smithi096 ceph-mon[26688]: [22/Apr/2024:15:09:25] ENGINE Serving on https://172.21.15.96:7150 2024-04-22T15:09:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:26 smithi096 ceph-mon[26688]: [22/Apr/2024:15:09:25] ENGINE Bus STARTED 2024-04-22T15:09:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:26 smithi096 ceph-mon[26688]: mgrmap e12: smithi096.hhkvum(active, since 2s) 2024-04-22T15:09:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:26 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:26 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:26 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.270+0000 7f9c3488a700 1 Processor -- start 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.270+0000 7f9c3488a700 1 -- start start 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.270+0000 7f9c3488a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c30106e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.270+0000 7f9c3488a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30107460 con 0x7f9c30106a70 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.271+0000 7f9c2e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c30106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.271+0000 7f9c2e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c30106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43388/0 (socket says 172.21.15.96:43388) 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.271+0000 7f9c2e59c700 1 -- 172.21.15.96:0/4229137125 learned_addr learned my addr 172.21.15.96:0/4229137125 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.271+0000 7f9c2e59c700 1 -- 172.21.15.96:0/4229137125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c301056d0 con 0x7f9c30106a70 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.271+0000 7f9c2e59c700 1 --2- 172.21.15.96:0/4229137125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c30106e90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c240098a0 tx=0x7f9c24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633cb794c17fbc76 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.271+0000 7f9c2d59a700 1 -- 172.21.15.96:0/4229137125 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c24004180 con 0x7f9c30106a70 2024-04-22T15:09:27.280 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.271+0000 7f9c2d59a700 1 -- 172.21.15.96:0/4229137125 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c240042e0 con 0x7f9c30106a70 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.271+0000 7f9c2d59a700 1 -- 172.21.15.96:0/4229137125 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c24011b10 con 0x7f9c30106a70 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.272+0000 7f9c3488a700 1 -- 172.21.15.96:0/4229137125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 msgr2=0x7f9c30106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.272+0000 7f9c3488a700 1 --2- 172.21.15.96:0/4229137125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c30106e90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c240098a0 tx=0x7f9c24009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.272+0000 7f9c3488a700 1 -- 172.21.15.96:0/4229137125 shutdown_connections 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.272+0000 7f9c3488a700 1 --2- 172.21.15.96:0/4229137125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c30106e90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.272+0000 7f9c3488a700 1 -- 172.21.15.96:0/4229137125 >> 172.21.15.96:0/4229137125 conn(0x7f9c301001c0 msgr2=0x7f9c30102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.272+0000 7f9c3488a700 1 -- 172.21.15.96:0/4229137125 shutdown_connections 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.272+0000 7f9c3488a700 1 -- 172.21.15.96:0/4229137125 wait complete. 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c3488a700 1 Processor -- start 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c3488a700 1 -- start start 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c3488a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c3019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c3488a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3019a910 con 0x7f9c30106a70 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c2e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c3019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:27.281 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c2e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c3019a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43400/0 (socket says 172.21.15.96:43400) 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c2e59c700 1 -- 172.21.15.96:0/3598722068 learned_addr learned my addr 172.21.15.96:0/3598722068 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c2e59c700 1 -- 172.21.15.96:0/3598722068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24009550 con 0x7f9c30106a70 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c2e59c700 1 --2- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c3019a3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009fb0 tx=0x7f9c24011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c1f7fe700 1 -- 172.21.15.96:0/3598722068 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c240038b0 con 0x7f9c30106a70 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3019ab10 con 0x7f9c30106a70 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c1f7fe700 1 -- 172.21.15.96:0/3598722068 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c24003a10 con 0x7f9c30106a70 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c1f7fe700 1 -- 172.21.15.96:0/3598722068 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c24019750 con 0x7f9c30106a70 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.273+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3019afb0 con 0x7f9c30106a70 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.274+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c30194570 con 0x7f9c30106a70 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.275+0000 7f9c1f7fe700 1 -- 172.21.15.96:0/3598722068 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 12) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f9c2401ac10 con 0x7f9c30106a70 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.275+0000 7f9c1f7fe700 1 --2- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9c18038730 0x7f9c1803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:27.282 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.275+0000 7f9c1f7fe700 1 -- 172.21.15.96:0/3598722068 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9c2404c950 con 0x7f9c30106a70 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.275+0000 7f9c2dd9b700 1 --2- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9c18038730 0x7f9c1803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.275+0000 7f9c2dd9b700 1 --2- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9c18038730 0x7f9c1803abf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9c20006fd0 tx=0x7f9c20006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.280+0000 7f9c1f7fe700 1 -- 172.21.15.96:0/3598722068 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9c24022700 con 0x7f9c30106a70 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.498+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 --> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c3002cf40 con 0x7f9c18038730 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.662+0000 7f9c1f7fe700 1 -- 172.21.15.96:0/3598722068 <== mgr.14164 v2:172.21.15.96:6800/976363826 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f9c3002cf40 con 0x7f9c18038730 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.665+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9c18038730 msgr2=0x7f9c1803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.665+0000 7f9c3488a700 1 --2- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9c18038730 0x7f9c1803abf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9c20006fd0 tx=0x7f9c20006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.665+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 msgr2=0x7f9c3019a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.665+0000 7f9c3488a700 1 --2- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c3019a3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009fb0 tx=0x7f9c24011890 comp rx=0 tx=0).stop 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.665+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 shutdown_connections 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.665+0000 7f9c3488a700 1 --2- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9c18038730 0x7f9c1803abf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.665+0000 7f9c3488a700 1 --2- 172.21.15.96:0/3598722068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c30106a70 0x7f9c3019a3d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:27.283 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.665+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 >> 172.21.15.96:0/3598722068 conn(0x7f9c301001c0 msgr2=0x7f9c301010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:27.284 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.666+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 shutdown_connections 2024-04-22T15:09:27.284 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:26.666+0000 7f9c3488a700 1 -- 172.21.15.96:0/3598722068 wait complete. 2024-04-22T15:09:27.284 INFO:teuthology.orchestra.run.smithi096.stdout:Creating initial admin user... 2024-04-22T15:09:28.166 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:27 smithi096 ceph-mon[26688]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:28.926 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$T2IZgJYmeqpeiu3J6taqme..VDyjJQKF6cu7IU8j/mult2xtUDDJa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713798568, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T15:09:28.926 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.710+0000 7f7fa7234700 1 Processor -- start 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.710+0000 7f7fa7234700 1 -- start start 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.710+0000 7f7fa7234700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa01069b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.710+0000 7f7fa7234700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa0106f80 con 0x7f7fa0106590 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.711+0000 7f7fa4fd0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa01069b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.711+0000 7f7fa4fd0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa01069b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43412/0 (socket says 172.21.15.96:43412) 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.711+0000 7f7fa4fd0700 1 -- 172.21.15.96:0/3093161461 learned_addr learned my addr 172.21.15.96:0/3093161461 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.711+0000 7f7fa4fd0700 1 -- 172.21.15.96:0/3093161461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa0107790 con 0x7f7fa0106590 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.711+0000 7f7fa4fd0700 1 --2- 172.21.15.96:0/3093161461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa01069b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009e80 tx=0x7f7f9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f9d90fc84c48df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.711+0000 7f7f977fe700 1 -- 172.21.15.96:0/3093161461 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9c00e070 con 0x7f7fa0106590 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.711+0000 7f7f977fe700 1 -- 172.21.15.96:0/3093161461 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f9c00b8c0 con 0x7f7fa0106590 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.711+0000 7f7f977fe700 1 -- 172.21.15.96:0/3093161461 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9c0044d0 con 0x7f7fa0106590 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.712+0000 7f7fa7234700 1 -- 172.21.15.96:0/3093161461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 msgr2=0x7f7fa01069b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:28.927 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.712+0000 7f7fa7234700 1 --2- 172.21.15.96:0/3093161461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa01069b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009e80 tx=0x7f7f9c00b330 comp rx=0 tx=0).stop 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.712+0000 7f7fa7234700 1 -- 172.21.15.96:0/3093161461 shutdown_connections 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.712+0000 7f7fa7234700 1 --2- 172.21.15.96:0/3093161461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa01069b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.712+0000 7f7fa7234700 1 -- 172.21.15.96:0/3093161461 >> 172.21.15.96:0/3093161461 conn(0x7f7fa0101b70 msgr2=0x7f7fa0103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.712+0000 7f7fa7234700 1 -- 172.21.15.96:0/3093161461 shutdown_connections 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.712+0000 7f7fa7234700 1 -- 172.21.15.96:0/3093161461 wait complete. 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.712+0000 7f7fa7234700 1 Processor -- start 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7fa7234700 1 -- start start 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7fa7234700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa019a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7fa7234700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa019a880 con 0x7f7fa0106590 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7fa4fd0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa019a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7fa4fd0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa019a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43414/0 (socket says 172.21.15.96:43414) 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7fa4fd0700 1 -- 172.21.15.96:0/3125126977 learned_addr learned my addr 172.21.15.96:0/3125126977 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7fa4fd0700 1 -- 172.21.15.96:0/3125126977 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c009b30 con 0x7f7fa0106590 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7fa4fd0700 1 --2- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa019a340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0072a0 tx=0x7f7f9c00bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:28.928 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7f95ffb700 1 -- 172.21.15.96:0/3125126977 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9c00e040 con 0x7f7fa0106590 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa019aa80 con 0x7f7fa0106590 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7f95ffb700 1 -- 172.21.15.96:0/3125126977 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f9c003dc0 con 0x7f7fa0106590 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.713+0000 7f7f95ffb700 1 -- 172.21.15.96:0/3125126977 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9c012600 con 0x7f7fa0106590 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.714+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa019af80 con 0x7f7fa0106590 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.714+0000 7f7f95ffb700 1 -- 172.21.15.96:0/3125126977 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 12) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f7f9c019040 con 0x7f7fa0106590 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.715+0000 7f7f95ffb700 1 --2- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f7f88038710 0x7f7f8803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.715+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa0062510 con 0x7f7fa0106590 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.715+0000 7f7f95ffb700 1 -- 172.21.15.96:0/3125126977 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7f9c04c410 con 0x7f7fa0106590 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.715+0000 7f7f97fff700 1 --2- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f7f88038710 0x7f7f8803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.715+0000 7f7f97fff700 1 --2- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f7f88038710 0x7f7f8803abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f90006fd0 tx=0x7f7f90006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.719+0000 7f7f95ffb700 1 -- 172.21.15.96:0/3125126977 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f7f9c01a9e0 con 0x7f7fa0106590 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:27.939+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 --> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7fa0062330 con 0x7f7f88038710 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.163+0000 7f7f95ffb700 1 -- 172.21.15.96:0/3125126977 <== mgr.14164 v2:172.21.15.96:6800/976363826 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7fa0062330 con 0x7f7f88038710 2024-04-22T15:09:28.929 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f7f88038710 msgr2=0x7f7f8803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 --2- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f7f88038710 0x7f7f8803abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f90006fd0 tx=0x7f7f90006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 msgr2=0x7f7fa019a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 --2- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa019a340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0072a0 tx=0x7f7f9c00bf70 comp rx=0 tx=0).stop 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 shutdown_connections 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 --2- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f7f88038710 0x7f7f8803abd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 --2- 172.21.15.96:0/3125126977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fa0106590 0x7f7fa019a340 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 >> 172.21.15.96:0/3125126977 conn(0x7f7fa0101b70 msgr2=0x7f7fa01026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 shutdown_connections 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:28.165+0000 7f7fa7234700 1 -- 172.21.15.96:0/3125126977 wait complete. 2024-04-22T15:09:28.930 INFO:teuthology.orchestra.run.smithi096.stdout:Fetching dashboard port number... 2024-04-22T15:09:29.187 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:28 smithi096 ceph-mon[26688]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:29.188 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:28 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:30.585 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stdout 8443 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.804+0000 7f0c88221700 1 Processor -- start 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.804+0000 7f0c88221700 1 -- start start 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.805+0000 7f0c88221700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c80106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.805+0000 7f0c88221700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c80106fe0 con 0x7f0c801065f0 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.805+0000 7f0c85fbd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c80106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.805+0000 7f0c85fbd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c80106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43420/0 (socket says 172.21.15.96:43420) 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.805+0000 7f0c85fbd700 1 -- 172.21.15.96:0/4115715297 learned_addr learned my addr 172.21.15.96:0/4115715297 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.805+0000 7f0c85fbd700 1 -- 172.21.15.96:0/4115715297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c801077a0 con 0x7f0c801065f0 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.805+0000 7f0c85fbd700 1 --2- 172.21.15.96:0/4115715297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c80106a10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0098a0 tx=0x7f0c7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90509125719284ea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.805+0000 7f0c84fbb700 1 -- 172.21.15.96:0/4115715297 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c7c004180 con 0x7f0c801065f0 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.806+0000 7f0c84fbb700 1 -- 172.21.15.96:0/4115715297 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c7c0042e0 con 0x7f0c801065f0 2024-04-22T15:09:30.586 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.806+0000 7f0c84fbb700 1 -- 172.21.15.96:0/4115715297 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c7c011b10 con 0x7f0c801065f0 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.806+0000 7f0c88221700 1 -- 172.21.15.96:0/4115715297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 msgr2=0x7f0c80106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.806+0000 7f0c88221700 1 --2- 172.21.15.96:0/4115715297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c80106a10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0098a0 tx=0x7f0c7c009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.806+0000 7f0c88221700 1 -- 172.21.15.96:0/4115715297 shutdown_connections 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.806+0000 7f0c88221700 1 --2- 172.21.15.96:0/4115715297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c80106a10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.806+0000 7f0c88221700 1 -- 172.21.15.96:0/4115715297 >> 172.21.15.96:0/4115715297 conn(0x7f0c80101bd0 msgr2=0x7f0c80103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.806+0000 7f0c88221700 1 -- 172.21.15.96:0/4115715297 shutdown_connections 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.806+0000 7f0c88221700 1 -- 172.21.15.96:0/4115715297 wait complete. 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.807+0000 7f0c88221700 1 Processor -- start 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.807+0000 7f0c88221700 1 -- start start 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.807+0000 7f0c88221700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c8007baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.807+0000 7f0c88221700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8007c030 con 0x7f0c801065f0 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.807+0000 7f0c85fbd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c8007baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.807+0000 7f0c85fbd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c8007baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43434/0 (socket says 172.21.15.96:43434) 2024-04-22T15:09:30.587 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.807+0000 7f0c85fbd700 1 -- 172.21.15.96:0/874619240 learned_addr learned my addr 172.21.15.96:0/874619240 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.807+0000 7f0c85fbd700 1 -- 172.21.15.96:0/874619240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c009550 con 0x7f0c801065f0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.807+0000 7f0c85fbd700 1 --2- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c8007baf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0117b0 tx=0x7f0c7c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.808+0000 7f0c76ffd700 1 -- 172.21.15.96:0/874619240 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c7c0038b0 con 0x7f0c801065f0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.808+0000 7f0c76ffd700 1 -- 172.21.15.96:0/874619240 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c7c003a10 con 0x7f0c801065f0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.808+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8007a2c0 con 0x7f0c801065f0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.808+0000 7f0c76ffd700 1 -- 172.21.15.96:0/874619240 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c7c0246e0 con 0x7f0c801065f0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.808+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8007a760 con 0x7f0c801065f0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.808+0000 7f0c76ffd700 1 -- 172.21.15.96:0/874619240 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 12) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f0c7c01ab80 con 0x7f0c801065f0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.809+0000 7f0c76ffd700 1 --2- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f0c6c0387c0 0x7f0c6c03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.809+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8007c170 con 0x7f0c801065f0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.809+0000 7f0c76ffd700 1 -- 172.21.15.96:0/874619240 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0c7c04ca00 con 0x7f0c801065f0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.809+0000 7f0c857bc700 1 --2- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f0c6c0387c0 0x7f0c6c03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:30.588 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.809+0000 7f0c857bc700 1 --2- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f0c6c0387c0 0x7f0c6c03ac80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0c70006fd0 tx=0x7f0c70006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.814+0000 7f0c76ffd700 1 -- 172.21.15.96:0/874619240 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f0c8007c170 con 0x7f0c801065f0 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:29.927+0000 7f0c76ffd700 1 -- 172.21.15.96:0/874619240 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f0c7c02d770 con 0x7f0c801065f0 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.011+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f0c8007c170 con 0x7f0c801065f0 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.012+0000 7f0c76ffd700 1 -- 172.21.15.96:0/874619240 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0c8007c170 con 0x7f0c801065f0 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f0c6c0387c0 msgr2=0x7f0c6c03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 --2- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f0c6c0387c0 0x7f0c6c03ac80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0c70006fd0 tx=0x7f0c70006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 msgr2=0x7f0c8007baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 --2- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c8007baf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0117b0 tx=0x7f0c7c011890 comp rx=0 tx=0).stop 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 shutdown_connections 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 --2- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f0c6c0387c0 0x7f0c6c03ac80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 --2- 172.21.15.96:0/874619240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c801065f0 0x7f0c8007baf0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 >> 172.21.15.96:0/874619240 conn(0x7f0c80101bd0 msgr2=0x7f0c80102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 shutdown_connections 2024-04-22T15:09:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:30.014+0000 7f0c88221700 1 -- 172.21.15.96:0/874619240 wait complete. 2024-04-22T15:09:30.593 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T15:09:30.593 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout disabled 2024-04-22T15:09:30.600 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T15:09:30.600 INFO:teuthology.orchestra.run.smithi096.stdout:systemctl: stdout inactive 2024-04-22T15:09:30.600 INFO:teuthology.orchestra.run.smithi096.stdout:firewalld.service is not enabled 2024-04-22T15:09:30.600 INFO:teuthology.orchestra.run.smithi096.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T15:09:30.601 INFO:teuthology.orchestra.run.smithi096.stdout:Ceph Dashboard is now available at: 2024-04-22T15:09:30.602 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:30.602 INFO:teuthology.orchestra.run.smithi096.stdout: URL: https://smithi096.front.sepia.ceph.com:8443/ 2024-04-22T15:09:30.602 INFO:teuthology.orchestra.run.smithi096.stdout: User: admin 2024-04-22T15:09:30.602 INFO:teuthology.orchestra.run.smithi096.stdout: Password: kg0h2fyjpz 2024-04-22T15:09:30.602 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:30.602 INFO:teuthology.orchestra.run.smithi096.stdout:Saving cluster configuration to /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config directory 2024-04-22T15:09:30.602 INFO:teuthology.orchestra.run.smithi096.stdout:Enabling autotune for osd_memory_target 2024-04-22T15:09:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:30 smithi096 ceph-mon[26688]: mgrmap e13: smithi096.hhkvum(active, since 6s) 2024-04-22T15:09:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:30 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/874619240' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T15:09:31.900 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.087+0000 7f27b495f700 1 Processor -- start 2024-04-22T15:09:31.900 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.087+0000 7f27b495f700 1 -- start start 2024-04-22T15:09:31.900 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27b495f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b0109420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:31.900 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27b495f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b0079e90 con 0x7f27b0107030 2024-04-22T15:09:31.900 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b0109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:31.900 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b0109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37874/0 (socket says 172.21.15.96:37874) 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27ae59c700 1 -- 172.21.15.96:0/3330659866 learned_addr learned my addr 172.21.15.96:0/3330659866 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27ae59c700 1 -- 172.21.15.96:0/3330659866 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b0109960 con 0x7f27b0107030 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27ae59c700 1 --2- 172.21.15.96:0/3330659866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b0109420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f27a000f960 tx=0x7f27a000fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfd50a5cb80af96 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27ad59a700 1 -- 172.21.15.96:0/3330659866 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a0009b30 con 0x7f27b0107030 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27ad59a700 1 -- 172.21.15.96:0/3330659866 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27a00044f0 con 0x7f27b0107030 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.088+0000 7f27ad59a700 1 -- 172.21.15.96:0/3330659866 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a0003d60 con 0x7f27b0107030 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.089+0000 7f27b495f700 1 -- 172.21.15.96:0/3330659866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 msgr2=0x7f27b0109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.089+0000 7f27b495f700 1 --2- 172.21.15.96:0/3330659866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b0109420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f27a000f960 tx=0x7f27a000fc70 comp rx=0 tx=0).stop 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.089+0000 7f27b495f700 1 -- 172.21.15.96:0/3330659866 shutdown_connections 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.089+0000 7f27b495f700 1 --2- 172.21.15.96:0/3330659866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b0109420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.089+0000 7f27b495f700 1 -- 172.21.15.96:0/3330659866 >> 172.21.15.96:0/3330659866 conn(0x7f27b0100a80 msgr2=0x7f27b0102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.089+0000 7f27b495f700 1 -- 172.21.15.96:0/3330659866 shutdown_connections 2024-04-22T15:09:31.901 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.089+0000 7f27b495f700 1 -- 172.21.15.96:0/3330659866 wait complete. 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.090+0000 7f27b495f700 1 Processor -- start 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.090+0000 7f27b495f700 1 -- start start 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.090+0000 7f27b495f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.090+0000 7f27b495f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b0198630 con 0x7f27b0107030 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.090+0000 7f27ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.090+0000 7f27ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b01980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37880/0 (socket says 172.21.15.96:37880) 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.090+0000 7f27ae59c700 1 -- 172.21.15.96:0/2466545837 learned_addr learned my addr 172.21.15.96:0/2466545837 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.090+0000 7f27ae59c700 1 -- 172.21.15.96:0/2466545837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a000f610 con 0x7f27b0107030 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.091+0000 7f27ae59c700 1 --2- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b01980f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27a00036e0 tx=0x7f27a0004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.091+0000 7f279f7fe700 1 -- 172.21.15.96:0/2466545837 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a0017700 con 0x7f27b0107030 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.091+0000 7f279f7fe700 1 -- 172.21.15.96:0/2466545837 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27a0017d20 con 0x7f27b0107030 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.091+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b0198830 con 0x7f27b0107030 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.091+0000 7f279f7fe700 1 -- 172.21.15.96:0/2466545837 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a0022b50 con 0x7f27b0107030 2024-04-22T15:09:31.902 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.091+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b0198cd0 con 0x7f27b0107030 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.092+0000 7f279f7fe700 1 -- 172.21.15.96:0/2466545837 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f27a0017860 con 0x7f27b0107030 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.092+0000 7f279f7fe700 1 --2- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2798038810 0x7f279803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.092+0000 7f279f7fe700 1 -- 172.21.15.96:0/2466545837 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f27a0050220 con 0x7f27b0107030 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.092+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b0191fb0 con 0x7f27b0107030 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.092+0000 7f27add9b700 1 --2- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2798038810 0x7f279803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.092+0000 7f27add9b700 1 --2- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2798038810 0x7f279803acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f27a4006fd0 tx=0x7f27a4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.097+0000 7f279f7fe700 1 -- 172.21.15.96:0/2466545837 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f27a0027030 con 0x7f27b0107030 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.289+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f27b0105a20 con 0x7f27b0107030 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.289+0000 7f279f7fe700 1 -- 172.21.15.96:0/2466545837 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f27a002c440 con 0x7f27b0107030 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2798038810 msgr2=0x7f279803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 --2- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2798038810 0x7f279803acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f27a4006fd0 tx=0x7f27a4006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 msgr2=0x7f27b01980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:31.903 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 --2- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b01980f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27a00036e0 tx=0x7f27a0004250 comp rx=0 tx=0).stop 2024-04-22T15:09:31.904 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 shutdown_connections 2024-04-22T15:09:31.904 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 --2- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2798038810 0x7f279803acd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:31.904 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 --2- 172.21.15.96:0/2466545837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27b0107030 0x7f27b01980f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:31.904 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 >> 172.21.15.96:0/2466545837 conn(0x7f27b0100a80 msgr2=0x7f27b0101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:31.904 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 shutdown_connections 2024-04-22T15:09:31.904 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:31.291+0000 7f27b495f700 1 -- 172.21.15.96:0/2466545837 wait complete. 2024-04-22T15:09:33.173 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.315+0000 7f8493967700 1 Processor -- start 2024-04-22T15:09:33.173 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.315+0000 7f8493967700 1 -- start start 2024-04-22T15:09:33.173 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8493967700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c109440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:33.173 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8493967700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c074770 con 0x7f848c107050 2024-04-22T15:09:33.173 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8491703700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c109440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8491703700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c109440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37886/0 (socket says 172.21.15.96:37886) 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8491703700 1 -- 172.21.15.96:0/753891524 learned_addr learned my addr 172.21.15.96:0/753891524 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8491703700 1 -- 172.21.15.96:0/753891524 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c109980 con 0x7f848c107050 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8491703700 1 --2- 172.21.15.96:0/753891524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c109440 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f847c0098a0 tx=0x7f847c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430dd3fbf57a8639 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8483fff700 1 -- 172.21.15.96:0/753891524 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f847c004180 con 0x7f848c107050 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8483fff700 1 -- 172.21.15.96:0/753891524 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f847c0042e0 con 0x7f848c107050 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.316+0000 7f8483fff700 1 -- 172.21.15.96:0/753891524 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f847c011b10 con 0x7f848c107050 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.317+0000 7f8493967700 1 -- 172.21.15.96:0/753891524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 msgr2=0x7f848c109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.317+0000 7f8493967700 1 --2- 172.21.15.96:0/753891524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c109440 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f847c0098a0 tx=0x7f847c009bb0 comp rx=0 tx=0).stop 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.317+0000 7f8493967700 1 -- 172.21.15.96:0/753891524 shutdown_connections 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.317+0000 7f8493967700 1 --2- 172.21.15.96:0/753891524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c109440 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:33.174 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.317+0000 7f8493967700 1 -- 172.21.15.96:0/753891524 >> 172.21.15.96:0/753891524 conn(0x7f848c100a60 msgr2=0x7f848c102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.317+0000 7f8493967700 1 -- 172.21.15.96:0/753891524 shutdown_connections 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.317+0000 7f8493967700 1 -- 172.21.15.96:0/753891524 wait complete. 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.317+0000 7f8493967700 1 Processor -- start 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8493967700 1 -- start start 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8493967700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c106880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8493967700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c106dc0 con 0x7f848c107050 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8491703700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c106880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8491703700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c106880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37902/0 (socket says 172.21.15.96:37902) 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8491703700 1 -- 172.21.15.96:0/245023632 learned_addr learned my addr 172.21.15.96:0/245023632 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8491703700 1 -- 172.21.15.96:0/245023632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c009550 con 0x7f848c107050 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8491703700 1 --2- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c106880 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f847c009fb0 tx=0x7f847c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f84827fc700 1 -- 172.21.15.96:0/245023632 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f847c0038b0 con 0x7f848c107050 2024-04-22T15:09:33.175 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f84827fc700 1 -- 172.21.15.96:0/245023632 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f847c003a10 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848c105000 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f84827fc700 1 -- 172.21.15.96:0/245023632 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f847c019750 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.318+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848c1054a0 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.319+0000 7f84827fc700 1 -- 172.21.15.96:0/245023632 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f847c01ac10 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.319+0000 7f84827fc700 1 --2- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f84780387d0 0x7f847803ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.319+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848c10a710 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.319+0000 7f84827fc700 1 -- 172.21.15.96:0/245023632 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f847c04ca70 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.319+0000 7f8490f02700 1 --2- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f84780387d0 0x7f847803ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.320+0000 7f8490f02700 1 --2- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f84780387d0 0x7f847803ac90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8488006fd0 tx=0x7f8488006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.324+0000 7f84827fc700 1 -- 172.21.15.96:0/245023632 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f847c022700 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.629+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f848c105990 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.631+0000 7f84827fc700 1 -- 172.21.15.96:0/245023632 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f847c022310 con 0x7f848c107050 2024-04-22T15:09:33.176 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.633+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f84780387d0 msgr2=0x7f847803ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.633+0000 7f8493967700 1 --2- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f84780387d0 0x7f847803ac90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8488006fd0 tx=0x7f8488006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.634+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 msgr2=0x7f848c106880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.634+0000 7f8493967700 1 --2- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c106880 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f847c009fb0 tx=0x7f847c011890 comp rx=0 tx=0).stop 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.634+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 shutdown_connections 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.634+0000 7f8493967700 1 --2- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f84780387d0 0x7f847803ac90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.634+0000 7f8493967700 1 --2- 172.21.15.96:0/245023632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f848c107050 0x7f848c106880 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.634+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 >> 172.21.15.96:0/245023632 conn(0x7f848c100a60 msgr2=0x7f848c1015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.634+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 shutdown_connections 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr 2024-04-22T15:09:32.634+0000 7f8493967700 1 -- 172.21.15.96:0/245023632 wait complete. 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:33.177 INFO:teuthology.orchestra.run.smithi096.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout:Or, if you are only running a single cluster on this host: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: ceph telemetry on 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout:For more information see: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:33.178 INFO:teuthology.orchestra.run.smithi096.stdout:Bootstrap complete. 2024-04-22T15:09:33.201 INFO:tasks.cephadm:Fetching config... 2024-04-22T15:09:33.202 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:09:33.202 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T15:09:33.217 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T15:09:33.217 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:09:33.217 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T15:09:33.273 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T15:09:33.273 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:09:33.273 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/keyring of=/dev/stdout 2024-04-22T15:09:33.337 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T15:09:33.337 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:09:33.337 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T15:09:33.392 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T15:09:33.392 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvjlJwVAi6it8NxTubVve/Wx8HJK6jvdZXPtOALRCKaQTw1kSJWQ5vKPPgZFxKHN9OT5Cll03ZEbvR/abVHHCM68oRnGl773CoR9F1l+10sp/dPDOLAHJXN8N4CiMrR+ZqZ1Ffejom5KcIrZNMkN97P2mxbpCar46QeKi23ZP12rGDjhoIN3eJ6FnVv/vNnYiwF3H+yBGMgNAUgCceN++xJe7EsEdSD/XRsOAtW2tM90qs+sFdAgtAaWejgglxho8mvg5YiDB5ddxXoMGE/2C9Tg5qzjL45zUIcCmNBqHnco5TKFhaSguaGD8OIBBpXY4YOHND0oA3nzPEgnkQ9Idl3ucRVVYxGK2whHduUnK1yLmXiP7WhoERg7f/tb6LwLgRH+rKxNmcJineBy8/l5gAQJ/w+mRY4q/9lYT3RBtS/bsPxTaBSZUFXA8YBOg8ROArEK7ObL+kSnMizvo6uYZvJMaGSfAR87FgBptywx0lpsU7Vq+afhMwq7P8shJdkoU= ceph-d5e86090-00b9-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T15:09:33.463 INFO:teuthology.orchestra.run.smithi096.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvjlJwVAi6it8NxTubVve/Wx8HJK6jvdZXPtOALRCKaQTw1kSJWQ5vKPPgZFxKHN9OT5Cll03ZEbvR/abVHHCM68oRnGl773CoR9F1l+10sp/dPDOLAHJXN8N4CiMrR+ZqZ1Ffejom5KcIrZNMkN97P2mxbpCar46QeKi23ZP12rGDjhoIN3eJ6FnVv/vNnYiwF3H+yBGMgNAUgCceN++xJe7EsEdSD/XRsOAtW2tM90qs+sFdAgtAaWejgglxho8mvg5YiDB5ddxXoMGE/2C9Tg5qzjL45zUIcCmNBqHnco5TKFhaSguaGD8OIBBpXY4YOHND0oA3nzPEgnkQ9Idl3ucRVVYxGK2whHduUnK1yLmXiP7WhoERg7f/tb6LwLgRH+rKxNmcJineBy8/l5gAQJ/w+mRY4q/9lYT3RBtS/bsPxTaBSZUFXA8YBOg8ROArEK7ObL+kSnMizvo6uYZvJMaGSfAR87FgBptywx0lpsU7Vq+afhMwq7P8shJdkoU= ceph-d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:09:33.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvjlJwVAi6it8NxTubVve/Wx8HJK6jvdZXPtOALRCKaQTw1kSJWQ5vKPPgZFxKHN9OT5Cll03ZEbvR/abVHHCM68oRnGl773CoR9F1l+10sp/dPDOLAHJXN8N4CiMrR+ZqZ1Ffejom5KcIrZNMkN97P2mxbpCar46QeKi23ZP12rGDjhoIN3eJ6FnVv/vNnYiwF3H+yBGMgNAUgCceN++xJe7EsEdSD/XRsOAtW2tM90qs+sFdAgtAaWejgglxho8mvg5YiDB5ddxXoMGE/2C9Tg5qzjL45zUIcCmNBqHnco5TKFhaSguaGD8OIBBpXY4YOHND0oA3nzPEgnkQ9Idl3ucRVVYxGK2whHduUnK1yLmXiP7WhoERg7f/tb6LwLgRH+rKxNmcJineBy8/l5gAQJ/w+mRY4q/9lYT3RBtS/bsPxTaBSZUFXA8YBOg8ROArEK7ObL+kSnMizvo6uYZvJMaGSfAR87FgBptywx0lpsU7Vq+afhMwq7P8shJdkoU= ceph-d5e86090-00b9-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T15:09:33.513 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvjlJwVAi6it8NxTubVve/Wx8HJK6jvdZXPtOALRCKaQTw1kSJWQ5vKPPgZFxKHN9OT5Cll03ZEbvR/abVHHCM68oRnGl773CoR9F1l+10sp/dPDOLAHJXN8N4CiMrR+ZqZ1Ffejom5KcIrZNMkN97P2mxbpCar46QeKi23ZP12rGDjhoIN3eJ6FnVv/vNnYiwF3H+yBGMgNAUgCceN++xJe7EsEdSD/XRsOAtW2tM90qs+sFdAgtAaWejgglxho8mvg5YiDB5ddxXoMGE/2C9Tg5qzjL45zUIcCmNBqHnco5TKFhaSguaGD8OIBBpXY4YOHND0oA3nzPEgnkQ9Idl3ucRVVYxGK2whHduUnK1yLmXiP7WhoERg7f/tb6LwLgRH+rKxNmcJineBy8/l5gAQJ/w+mRY4q/9lYT3RBtS/bsPxTaBSZUFXA8YBOg8ROArEK7ObL+kSnMizvo6uYZvJMaGSfAR87FgBptywx0lpsU7Vq+afhMwq7P8shJdkoU= ceph-d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:09:33.526 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T15:09:33.751 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:09:33.874 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:33 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/245023632' entity='client.admin' 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.377+0000 7f29ca4b3700 1 -- 172.21.15.96:0/3558233936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 msgr2=0x7f29c40ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.377+0000 7f29ca4b3700 1 --2- 172.21.15.96:0/3558233936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 0x7f29c40ffb90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009910 tx=0x7f29b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.377+0000 7f29ca4b3700 1 -- 172.21.15.96:0/3558233936 shutdown_connections 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.377+0000 7f29ca4b3700 1 --2- 172.21.15.96:0/3558233936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 0x7f29c40ffb90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.377+0000 7f29ca4b3700 1 -- 172.21.15.96:0/3558233936 >> 172.21.15.96:0/3558233936 conn(0x7f29c40fad10 msgr2=0x7f29c40fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.377+0000 7f29ca4b3700 1 -- 172.21.15.96:0/3558233936 shutdown_connections 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.377+0000 7f29ca4b3700 1 -- 172.21.15.96:0/3558233936 wait complete. 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.377+0000 7f29ca4b3700 1 Processor -- start 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.377+0000 7f29ca4b3700 1 -- start start 2024-04-22T15:09:36.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.378+0000 7f29ca4b3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 0x7f29c4195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:36.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.378+0000 7f29ca4b3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4196230 con 0x7f29c40ff770 2024-04-22T15:09:36.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.378+0000 7f29c3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 0x7f29c4195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:36.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.378+0000 7f29c3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 0x7f29c4195cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37914/0 (socket says 172.21.15.96:37914) 2024-04-22T15:09:36.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.378+0000 7f29c3fff700 1 -- 172.21.15.96:0/1265549424 learned_addr learned my addr 172.21.15.96:0/1265549424 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:36.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.378+0000 7f29c3fff700 1 -- 172.21.15.96:0/1265549424 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b40095f0 con 0x7f29c40ff770 2024-04-22T15:09:36.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.378+0000 7f29c3fff700 1 --2- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 0x7f29c4195cf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f29b4006010 tx=0x7f29b4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:36.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.379+0000 7f29c17fa700 1 -- 172.21.15.96:0/1265549424 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b401d070 con 0x7f29c40ff770 2024-04-22T15:09:36.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.379+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c4196430 con 0x7f29c40ff770 2024-04-22T15:09:36.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.379+0000 7f29c17fa700 1 -- 172.21.15.96:0/1265549424 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29b4005dc0 con 0x7f29c40ff770 2024-04-22T15:09:36.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.379+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c41968d0 con 0x7f29c40ff770 2024-04-22T15:09:36.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.379+0000 7f29c17fa700 1 -- 172.21.15.96:0/1265549424 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b4022e90 con 0x7f29c40ff770 2024-04-22T15:09:36.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.380+0000 7f29c17fa700 1 -- 172.21.15.96:0/1265549424 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f29b400f570 con 0x7f29c40ff770 2024-04-22T15:09:36.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.380+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c418f810 con 0x7f29c40ff770 2024-04-22T15:09:36.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.380+0000 7f29c17fa700 1 --2- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f29ac038760 0x7f29ac03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:36.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.380+0000 7f29c17fa700 1 -- 172.21.15.96:0/1265549424 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f29b404c810 con 0x7f29c40ff770 2024-04-22T15:09:36.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.380+0000 7f29c37fe700 1 --2- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f29ac038760 0x7f29ac03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:36.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.380+0000 7f29c37fe700 1 --2- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f29ac038760 0x7f29ac03ac20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f29b8006fd0 tx=0x7f29b8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:36.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.386+0000 7f29c17fa700 1 -- 172.21.15.96:0/1265549424 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f29b402a950 con 0x7f29c40ff770 2024-04-22T15:09:36.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.620+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f29c4062510 con 0x7f29c40ff770 2024-04-22T15:09:36.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.623+0000 7f29c17fa700 1 -- 172.21.15.96:0/1265549424 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f29b4027020 con 0x7f29c40ff770 2024-04-22T15:09:36.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f29ac038760 msgr2=0x7f29ac03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:36.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 --2- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f29ac038760 0x7f29ac03ac20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f29b8006fd0 tx=0x7f29b8006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:36.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 msgr2=0x7f29c4195cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:36.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 --2- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 0x7f29c4195cf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f29b4006010 tx=0x7f29b4009f80 comp rx=0 tx=0).stop 2024-04-22T15:09:36.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 shutdown_connections 2024-04-22T15:09:36.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 --2- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f29ac038760 0x7f29ac03ac20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:36.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 --2- 172.21.15.96:0/1265549424 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29c40ff770 0x7f29c4195cf0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:36.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 >> 172.21.15.96:0/1265549424 conn(0x7f29c40fad10 msgr2=0x7f29c40684f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:36.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 shutdown_connections 2024-04-22T15:09:36.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:36.625+0000 7f29ca4b3700 1 -- 172.21.15.96:0/1265549424 wait complete. 2024-04-22T15:09:36.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:36 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:36.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:36 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:36.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:36 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:09:36.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:36 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:36.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:36 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:09:36.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:36 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:09:36.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:36 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:09:36.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:36 smithi096 ceph-mon[26688]: Deploying daemon ceph-exporter.smithi096 on smithi096 2024-04-22T15:09:37.336 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T15:09:37.337 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T15:09:37.948 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:37 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1265549424' entity='client.admin' 2024-04-22T15:09:38.503 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:09:40.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.784+0000 7fceb4357700 1 -- 172.21.15.96:0/2073746687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 msgr2=0x7fceac1037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:40.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.784+0000 7fceb4357700 1 --2- 172.21.15.96:0/2073746687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 0x7fceac1037c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00b0a0 tx=0x7fce9c00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:09:40.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.784+0000 7fceb4357700 1 -- 172.21.15.96:0/2073746687 shutdown_connections 2024-04-22T15:09:40.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.784+0000 7fceb4357700 1 --2- 172.21.15.96:0/2073746687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 0x7fceac1037c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:40.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.784+0000 7fceb4357700 1 -- 172.21.15.96:0/2073746687 >> 172.21.15.96:0/2073746687 conn(0x7fceac0fae20 msgr2=0x7fceac0fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:40.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.784+0000 7fceb4357700 1 -- 172.21.15.96:0/2073746687 shutdown_connections 2024-04-22T15:09:40.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.784+0000 7fceb4357700 1 -- 172.21.15.96:0/2073746687 wait complete. 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb4357700 1 Processor -- start 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb4357700 1 -- start start 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb4357700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 0x7fceac197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb4357700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac198500 con 0x7fceac1013d0 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb20f3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 0x7fceac197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb20f3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 0x7fceac197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43166/0 (socket says 172.21.15.96:43166) 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb20f3700 1 -- 172.21.15.96:0/531368831 learned_addr learned my addr 172.21.15.96:0/531368831 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb20f3700 1 -- 172.21.15.96:0/531368831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c009d00 con 0x7fceac1013d0 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb20f3700 1 --2- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 0x7fceac197fc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0062a0 tx=0x7fce9c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceab7fe700 1 -- 172.21.15.96:0/531368831 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce9c00e040 con 0x7fceac1013d0 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceac198700 con 0x7fceac1013d0 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceab7fe700 1 -- 172.21.15.96:0/531368831 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce9c003d70 con 0x7fceac1013d0 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.785+0000 7fceab7fe700 1 -- 172.21.15.96:0/531368831 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce9c01c8d0 con 0x7fceac1013d0 2024-04-22T15:09:40.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.786+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceac198ba0 con 0x7fceac1013d0 2024-04-22T15:09:40.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.786+0000 7fceab7fe700 1 -- 172.21.15.96:0/531368831 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fce9c01cb90 con 0x7fceac1013d0 2024-04-22T15:09:40.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.786+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceac191b70 con 0x7fceac1013d0 2024-04-22T15:09:40.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.786+0000 7fceab7fe700 1 --2- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fce940387c0 0x7fce9403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:40.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.787+0000 7fceab7fe700 1 -- 172.21.15.96:0/531368831 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fce9c04ca80 con 0x7fceac1013d0 2024-04-22T15:09:40.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.787+0000 7fceb18f2700 1 --2- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fce940387c0 0x7fce9403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:40.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.787+0000 7fceb18f2700 1 --2- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fce940387c0 0x7fce9403ac80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcea0006fd0 tx=0x7fcea0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:40.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.791+0000 7fceab7fe700 1 -- 172.21.15.96:0/531368831 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fce9c012640 con 0x7fceac1013d0 2024-04-22T15:09:40.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.994+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 --> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fceac061320 con 0x7fce940387c0 2024-04-22T15:09:41.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:40.998+0000 7fceab7fe700 1 -- 172.21.15.96:0/531368831 <== mgr.14164 v2:172.21.15.96:6800/976363826 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fceac061320 con 0x7fce940387c0 2024-04-22T15:09:41.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.000+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fce940387c0 msgr2=0x7fce9403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:41.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.000+0000 7fceb4357700 1 --2- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fce940387c0 0x7fce9403ac80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcea0006fd0 tx=0x7fcea0006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:41.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.000+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 msgr2=0x7fceac197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:41.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.000+0000 7fceb4357700 1 --2- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 0x7fceac197fc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0062a0 tx=0x7fce9c00bd40 comp rx=0 tx=0).stop 2024-04-22T15:09:41.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.001+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 shutdown_connections 2024-04-22T15:09:41.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.001+0000 7fceb4357700 1 --2- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fce940387c0 0x7fce9403ac80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:41.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.001+0000 7fceb4357700 1 --2- 172.21.15.96:0/531368831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceac1013d0 0x7fceac197fc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:41.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.001+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 >> 172.21.15.96:0/531368831 conn(0x7fceac0fae20 msgr2=0x7fceac0fb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:41.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.001+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 shutdown_connections 2024-04-22T15:09:41.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:41.001+0000 7fceb4357700 1 -- 172.21.15.96:0/531368831 wait complete. 2024-04-22T15:09:41.618 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-04-22T15:09:41.618 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:09:41.618 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-04-22T15:09:41.638 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:09:41.638 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:09:41.695 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-04-22T15:09:41.695 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch host add smithi196 2024-04-22T15:09:42.200 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:42.200 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:42.200 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:42.200 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:42.200 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:42.200 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:42.200 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi096", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:09:42.200 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi096", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T15:09:42.200 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:09:42.201 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:41 smithi096 ceph-mon[26688]: Deploying daemon crash.smithi096 on smithi096 2024-04-22T15:09:42.903 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:09:44.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.866+0000 7fa6c1936700 1 -- 172.21.15.96:0/773883552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 msgr2=0x7fa6bc100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:44.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.866+0000 7fa6c1936700 1 --2- 172.21.15.96:0/773883552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 0x7fa6bc100c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009910 tx=0x7fa6a4009c20 comp rx=0 tx=0).stop 2024-04-22T15:09:44.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.866+0000 7fa6c1936700 1 -- 172.21.15.96:0/773883552 shutdown_connections 2024-04-22T15:09:44.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.866+0000 7fa6c1936700 1 --2- 172.21.15.96:0/773883552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 0x7fa6bc100c90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.866+0000 7fa6c1936700 1 -- 172.21.15.96:0/773883552 >> 172.21.15.96:0/773883552 conn(0x7fa6bc0fbe30 msgr2=0x7fa6bc0fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.866+0000 7fa6c1936700 1 -- 172.21.15.96:0/773883552 shutdown_connections 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.866+0000 7fa6c1936700 1 -- 172.21.15.96:0/773883552 wait complete. 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.867+0000 7fa6c1936700 1 Processor -- start 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.867+0000 7fa6c1936700 1 -- start start 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.867+0000 7fa6c1936700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 0x7fa6bc195ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.867+0000 7fa6c1936700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc196220 con 0x7fa6bc100870 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.867+0000 7fa6baffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 0x7fa6bc195ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.867+0000 7fa6baffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 0x7fa6bc195ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43186/0 (socket says 172.21.15.96:43186) 2024-04-22T15:09:44.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.867+0000 7fa6baffd700 1 -- 172.21.15.96:0/250750549 learned_addr learned my addr 172.21.15.96:0/250750549 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.867+0000 7fa6baffd700 1 -- 172.21.15.96:0/250750549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a40095f0 con 0x7fa6bc100870 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.867+0000 7fa6baffd700 1 --2- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 0x7fa6bc195ce0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4006010 tx=0x7fa6a4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.868+0000 7fa6c0934700 1 -- 172.21.15.96:0/250750549 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a401d070 con 0x7fa6bc100870 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.868+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc196420 con 0x7fa6bc100870 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.868+0000 7fa6c0934700 1 -- 172.21.15.96:0/250750549 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6a4005dc0 con 0x7fa6bc100870 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.868+0000 7fa6c0934700 1 -- 172.21.15.96:0/250750549 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a4022e90 con 0x7fa6bc100870 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.868+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc1968c0 con 0x7fa6bc100870 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.868+0000 7fa6c0934700 1 -- 172.21.15.96:0/250750549 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fa6a400f4f0 con 0x7fa6bc100870 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.869+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6bc18f6a0 con 0x7fa6bc100870 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.869+0000 7fa6c0934700 1 --2- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa6a8038760 0x7fa6a803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.869+0000 7fa6c0934700 1 -- 172.21.15.96:0/250750549 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa6a404ca90 con 0x7fa6bc100870 2024-04-22T15:09:44.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.869+0000 7fa6ba7fc700 1 --2- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa6a8038760 0x7fa6a803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:44.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.869+0000 7fa6ba7fc700 1 --2- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa6a8038760 0x7fa6a803ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac006fd0 tx=0x7fa6ac006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:44.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:44.874+0000 7fa6c0934700 1 -- 172.21.15.96:0/250750549 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa6a402a950 con 0x7fa6bc100870 2024-04-22T15:09:45.066 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:44 smithi096 ceph-mon[26688]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:45.066 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:44 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:45.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:45.083+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 --> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6bc02cdb0 con 0x7fa6a8038760 2024-04-22T15:09:46.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:46 smithi096 ceph-mon[26688]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:09:46.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:46 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:46.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:46 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:46.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:46 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:46.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:46 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:46.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:46 smithi096 ceph-mon[26688]: Deploying daemon node-exporter.smithi096 on smithi096 2024-04-22T15:09:47.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:47 smithi096 ceph-mon[26688]: Deploying cephadm binary to smithi196 2024-04-22T15:09:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:47 smithi096 ceph-mon[26688]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:47.519 INFO:teuthology.orchestra.run.smithi096.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-04-22T15:09:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.514+0000 7fa6c0934700 1 -- 172.21.15.96:0/250750549 <== mgr.14164 v2:172.21.15.96:6800/976363826 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa6bc02cdb0 con 0x7fa6a8038760 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa6a8038760 msgr2=0x7fa6a803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 --2- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa6a8038760 0x7fa6a803ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac006fd0 tx=0x7fa6ac006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 msgr2=0x7fa6bc195ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 --2- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 0x7fa6bc195ce0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4006010 tx=0x7fa6a4009f80 comp rx=0 tx=0).stop 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 shutdown_connections 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 --2- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa6a8038760 0x7fa6a803ac20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 --2- 172.21.15.96:0/250750549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6bc100870 0x7fa6bc195ce0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 >> 172.21.15.96:0/250750549 conn(0x7fa6bc0fbe30 msgr2=0x7fa6bc0fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 shutdown_connections 2024-04-22T15:09:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:47.517+0000 7fa6c1936700 1 -- 172.21.15.96:0/250750549 wait complete. 2024-04-22T15:09:48.329 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T15:09:48.534 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:09:48.655 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:48 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:48.655 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:48 smithi096 ceph-mon[26688]: Added host smithi196 2024-04-22T15:09:48.656 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:48 smithi096 ceph-mon[26688]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:50.230 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:49 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:51.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.002+0000 7f184d32a700 1 -- 172.21.15.96:0/3645776963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 msgr2=0x7f1848100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.002+0000 7f184d32a700 1 --2- 172.21.15.96:0/3645776963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 0x7f1848100cb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1838009910 tx=0x7f1838009c20 comp rx=0 tx=0).stop 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.002+0000 7f184d32a700 1 -- 172.21.15.96:0/3645776963 shutdown_connections 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.002+0000 7f184d32a700 1 --2- 172.21.15.96:0/3645776963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 0x7f1848100cb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.002+0000 7f184d32a700 1 -- 172.21.15.96:0/3645776963 >> 172.21.15.96:0/3645776963 conn(0x7f18480fbe30 msgr2=0x7f18480fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.002+0000 7f184d32a700 1 -- 172.21.15.96:0/3645776963 shutdown_connections 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.002+0000 7f184d32a700 1 -- 172.21.15.96:0/3645776963 wait complete. 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.002+0000 7f184d32a700 1 Processor -- start 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.003+0000 7f184d32a700 1 -- start start 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.003+0000 7f184d32a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 0x7f1848074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:51.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.003+0000 7f184d32a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1848075090 con 0x7f1848100890 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.003+0000 7f1846ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 0x7f1848074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.003+0000 7f1846ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 0x7f1848074b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34636/0 (socket says 172.21.15.96:34636) 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.003+0000 7f1846ffd700 1 -- 172.21.15.96:0/2876977320 learned_addr learned my addr 172.21.15.96:0/2876977320 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.003+0000 7f1846ffd700 1 -- 172.21.15.96:0/2876977320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18380095f0 con 0x7f1848100890 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.003+0000 7f1846ffd700 1 --2- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 0x7f1848074b50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1838006010 tx=0x7f18380050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.003+0000 7f1844ff9700 1 -- 172.21.15.96:0/2876977320 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f183801d070 con 0x7f1848100890 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.004+0000 7f1844ff9700 1 -- 172.21.15.96:0/2876977320 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f183800be20 con 0x7f1848100890 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.004+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18480732d0 con 0x7f1848100890 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.004+0000 7f1844ff9700 1 -- 172.21.15.96:0/2876977320 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1838022de0 con 0x7f1848100890 2024-04-22T15:09:51.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.004+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1848073770 con 0x7f1848100890 2024-04-22T15:09:51.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.004+0000 7f1844ff9700 1 -- 172.21.15.96:0/2876977320 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f183800f460 con 0x7f1848100890 2024-04-22T15:09:51.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.005+0000 7f1844ff9700 1 --2- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f1834038820 0x7f183403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:51.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.005+0000 7f1844ff9700 1 -- 172.21.15.96:0/2876977320 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f183804c840 con 0x7f1848100890 2024-04-22T15:09:51.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.005+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1848106ff0 con 0x7f1848100890 2024-04-22T15:09:51.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.005+0000 7f183e7fc700 1 --2- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f1834038820 0x7f183403ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:51.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.005+0000 7f183e7fc700 1 --2- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f1834038820 0x7f183403ace0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1830006fd0 tx=0x7f1830006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:51.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.010+0000 7f1844ff9700 1 -- 172.21.15.96:0/2876977320 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f1838027070 con 0x7f1848100890 2024-04-22T15:09:51.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.216+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 --> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f18480739b0 con 0x7f1834038820 2024-04-22T15:09:51.220 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:50 smithi096 ceph-mon[26688]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:51.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.217+0000 7f1844ff9700 1 -- 172.21.15.96:0/2876977320 <== mgr.14164 v2:172.21.15.96:6800/976363826 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f18480739b0 con 0x7f1834038820 2024-04-22T15:09:51.221 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:09:51.221 INFO:teuthology.orchestra.run.smithi096.stdout:[{"addr": "172.21.15.96", "hostname": "smithi096", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-04-22T15:09:51.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f1834038820 msgr2=0x7f183403ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:51.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 --2- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f1834038820 0x7f183403ace0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1830006fd0 tx=0x7f1830006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:51.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 msgr2=0x7f1848074b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:51.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 --2- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 0x7f1848074b50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1838006010 tx=0x7f18380050b0 comp rx=0 tx=0).stop 2024-04-22T15:09:51.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 shutdown_connections 2024-04-22T15:09:51.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 --2- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f1834038820 0x7f183403ace0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:51.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 --2- 172.21.15.96:0/2876977320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1848100890 0x7f1848074b50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:51.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 >> 172.21.15.96:0/2876977320 conn(0x7f18480fbe30 msgr2=0x7f18480fc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:51.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 shutdown_connections 2024-04-22T15:09:51.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:51.219+0000 7f184d32a700 1 -- 172.21.15.96:0/2876977320 wait complete. 2024-04-22T15:09:52.074 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T15:09:52.074 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T15:09:52.289 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:09:52.414 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:52 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:52.415 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:52 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:52.415 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:52 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:52.415 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:52 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:52.415 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:52 smithi096 ceph-mon[26688]: Deploying daemon alertmanager.smithi096 on smithi096 2024-04-22T15:09:52.415 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:52 smithi096 ceph-mon[26688]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:09:52.415 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:52 smithi096 ceph-mon[26688]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:54.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.288+0000 7f292f2c0700 1 -- 172.21.15.96:0/155151874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 msgr2=0x7f29280ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:54.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.288+0000 7f292f2c0700 1 --2- 172.21.15.96:0/155151874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 0x7f29280ffb90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2918009910 tx=0x7f2918009c20 comp rx=0 tx=0).stop 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.288+0000 7f292f2c0700 1 -- 172.21.15.96:0/155151874 shutdown_connections 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.288+0000 7f292f2c0700 1 --2- 172.21.15.96:0/155151874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 0x7f29280ffb90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.288+0000 7f292f2c0700 1 -- 172.21.15.96:0/155151874 >> 172.21.15.96:0/155151874 conn(0x7f29280fad10 msgr2=0x7f29280fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.288+0000 7f292f2c0700 1 -- 172.21.15.96:0/155151874 shutdown_connections 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.288+0000 7f292f2c0700 1 -- 172.21.15.96:0/155151874 wait complete. 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.288+0000 7f292f2c0700 1 Processor -- start 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.288+0000 7f292f2c0700 1 -- start start 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f292f2c0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 0x7f2928068ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f292f2c0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2928069000 con 0x7f29280ff770 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f292d05c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 0x7f2928068ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f292d05c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 0x7f2928068ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34654/0 (socket says 172.21.15.96:34654) 2024-04-22T15:09:54.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f292d05c700 1 -- 172.21.15.96:0/1626791306 learned_addr learned my addr 172.21.15.96:0/1626791306 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f292d05c700 1 -- 172.21.15.96:0/1626791306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29180095f0 con 0x7f29280ff770 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f292d05c700 1 --2- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 0x7f2928068ac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2918006010 tx=0x7f2918009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f291e7fc700 1 -- 172.21.15.96:0/1626791306 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f291801d070 con 0x7f29280ff770 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f291e7fc700 1 -- 172.21.15.96:0/1626791306 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2918005dc0 con 0x7f29280ff770 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2928198a00 con 0x7f29280ff770 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f291e7fc700 1 -- 172.21.15.96:0/1626791306 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2918022e90 con 0x7f29280ff770 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.289+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2928198ea0 con 0x7f29280ff770 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.290+0000 7f291e7fc700 1 -- 172.21.15.96:0/1626791306 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f291800b960 con 0x7f29280ff770 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.290+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f292818e5a0 con 0x7f29280ff770 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.290+0000 7f291e7fc700 1 --2- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2914038760 0x7f291403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.290+0000 7f291e7fc700 1 -- 172.21.15.96:0/1626791306 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f291804c800 con 0x7f29280ff770 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.290+0000 7f292c85b700 1 --2- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2914038760 0x7f291403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.291+0000 7f292c85b700 1 --2- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2914038760 0x7f291403ac20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2924006fd0 tx=0x7f2924006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:54.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.295+0000 7f291e7fc700 1 -- 172.21.15.96:0/1626791306 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2918022750 con 0x7f29280ff770 2024-04-22T15:09:54.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.511+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2928062510 con 0x7f29280ff770 2024-04-22T15:09:54.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.986+0000 7f291e7fc700 1 -- 172.21.15.96:0/1626791306 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2928062510 con 0x7f29280ff770 2024-04-22T15:09:54.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.988+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2914038760 msgr2=0x7f291403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:54.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.988+0000 7f292f2c0700 1 --2- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2914038760 0x7f291403ac20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2924006fd0 tx=0x7f2924006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:54.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.988+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 msgr2=0x7f2928068ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:54.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.988+0000 7f292f2c0700 1 --2- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 0x7f2928068ac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2918006010 tx=0x7f2918009f80 comp rx=0 tx=0).stop 2024-04-22T15:09:54.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.989+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 shutdown_connections 2024-04-22T15:09:54.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.989+0000 7f292f2c0700 1 --2- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f2914038760 0x7f291403ac20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:54.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.989+0000 7f292f2c0700 1 --2- 172.21.15.96:0/1626791306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29280ff770 0x7f2928068ac0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:54.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.989+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 >> 172.21.15.96:0/1626791306 conn(0x7f29280fad10 msgr2=0x7f29280fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:54.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.989+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 shutdown_connections 2024-04-22T15:09:54.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:09:54.989+0000 7f292f2c0700 1 -- 172.21.15.96:0/1626791306 wait complete. 2024-04-22T15:09:54.993 INFO:teuthology.orchestra.run.smithi096.stderr:adjusted tunables profile to default 2024-04-22T15:09:55.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:54 smithi096 ceph-mon[26688]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:55.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:54 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:55.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:54 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1626791306' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T15:09:56.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:55 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1626791306' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T15:09:56.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:55 smithi096 ceph-mon[26688]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T15:09:56.763 INFO:tasks.cephadm:Adding mon.smithi096 on smithi096 2024-04-22T15:09:56.763 INFO:tasks.cephadm:Adding mon.smithi196 on smithi196 2024-04-22T15:09:56.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi096:172.21.15.96=smithi096;smithi196:172.21.15.196=smithi196' 2024-04-22T15:09:56.972 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:09:57.020 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:09:57.096 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:56 smithi096 ceph-mon[26688]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:58.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.587+0000 7fb2ef392700 1 -- 172.21.15.196:0/603843445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 msgr2=0x7fb2e80ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:58.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.587+0000 7fb2ef392700 1 --2- 172.21.15.196:0/603843445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 0x7fb2e80ffba0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009910 tx=0x7fb2e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:09:58.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.587+0000 7fb2ef392700 1 -- 172.21.15.196:0/603843445 shutdown_connections 2024-04-22T15:09:58.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.587+0000 7fb2ef392700 1 --2- 172.21.15.196:0/603843445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 0x7fb2e80ffba0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:58.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.587+0000 7fb2ef392700 1 -- 172.21.15.196:0/603843445 >> 172.21.15.196:0/603843445 conn(0x7fb2e80fad40 msgr2=0x7fb2e80fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:58.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.587+0000 7fb2ef392700 1 -- 172.21.15.196:0/603843445 shutdown_connections 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.587+0000 7fb2ef392700 1 -- 172.21.15.196:0/603843445 wait complete. 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.587+0000 7fb2ef392700 1 Processor -- start 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.587+0000 7fb2ef392700 1 -- start start 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2ef392700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 0x7fb2e8193b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2ef392700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e8194060 con 0x7fb2e80ff780 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2ed12e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 0x7fb2e8193b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2ed12e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 0x7fb2e8193b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:36992/0 (socket says 172.21.15.196:36992) 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2ed12e700 1 -- 172.21.15.196:0/2274139030 learned_addr learned my addr 172.21.15.196:0/2274139030 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2ed12e700 1 -- 172.21.15.196:0/2274139030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e40095f0 con 0x7fb2e80ff780 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2ed12e700 1 --2- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 0x7fb2e8193b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4006010 tx=0x7fb2e4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2de7fc700 1 -- 172.21.15.196:0/2274139030 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e401d070 con 0x7fb2e80ff780 2024-04-22T15:09:58.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2de7fc700 1 -- 172.21.15.196:0/2274139030 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2e4005dc0 con 0x7fb2e80ff780 2024-04-22T15:09:58.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e8194260 con 0x7fb2e80ff780 2024-04-22T15:09:58.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.588+0000 7fb2de7fc700 1 -- 172.21.15.196:0/2274139030 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e4022e90 con 0x7fb2e80ff780 2024-04-22T15:09:58.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.589+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e8194700 con 0x7fb2e80ff780 2024-04-22T15:09:58.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.589+0000 7fb2de7fc700 1 -- 172.21.15.196:0/2274139030 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fb2e400b960 con 0x7fb2e80ff780 2024-04-22T15:09:58.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.589+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e818d4a0 con 0x7fb2e80ff780 2024-04-22T15:09:58.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.589+0000 7fb2de7fc700 1 --2- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2d403aa00 0x7fb2d403cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:09:58.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.590+0000 7fb2de7fc700 1 -- 172.21.15.196:0/2274139030 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb2e404c7f0 con 0x7fb2e80ff780 2024-04-22T15:09:58.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.590+0000 7fb2ec92d700 1 --2- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2d403aa00 0x7fb2d403cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:09:58.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.590+0000 7fb2ec92d700 1 --2- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2d403aa00 0x7fb2d403cec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8006fd0 tx=0x7fb2d8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:09:58.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.595+0000 7fb2de7fc700 1 -- 172.21.15.196:0/2274139030 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb2e400bd50 con 0x7fb2e80ff780 2024-04-22T15:09:58.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.797+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 --> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi096:172.21.15.96=smithi096;smithi196:172.21.15.196=smithi196", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2e802cdb0 con 0x7fb2d403aa00 2024-04-22T15:09:58.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.803+0000 7fb2de7fc700 1 -- 172.21.15.196:0/2274139030 <== mgr.14164 v2:172.21.15.96:6800/976363826 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb2e802cdb0 con 0x7fb2d403aa00 2024-04-22T15:09:58.806 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mon update... 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2d403aa00 msgr2=0x7fb2d403cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 --2- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2d403aa00 0x7fb2d403cec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8006fd0 tx=0x7fb2d8006e40 comp rx=0 tx=0).stop 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 msgr2=0x7fb2e8193b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 --2- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 0x7fb2e8193b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4006010 tx=0x7fb2e4009f80 comp rx=0 tx=0).stop 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 shutdown_connections 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 --2- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2d403aa00 0x7fb2d403cec0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 --2- 172.21.15.196:0/2274139030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2e80ff780 0x7fb2e8193b20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 >> 172.21.15.196:0/2274139030 conn(0x7fb2e80fad40 msgr2=0x7fb2e80fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 shutdown_connections 2024-04-22T15:09:58.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:09:58.805+0000 7fb2ef392700 1 -- 172.21.15.196:0/2274139030 wait complete. 2024-04-22T15:09:59.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:58 smithi096 ceph-mon[26688]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:09:59.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:58 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:09:59.332 DEBUG:teuthology.orchestra.run.smithi196:mon.smithi196> sudo journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi196.service 2024-04-22T15:09:59.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:09:59.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:09:59.565 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:09:59.612 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:00.445 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:59 smithi096 ceph-mon[26688]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi096:172.21.15.96=smithi096;smithi196:172.21.15.196=smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:10:00.446 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:09:59 smithi096 ceph-mon[26688]: Saving service mon spec with placement smithi096:172.21.15.96=smithi096;smithi196:172.21.15.196=smithi196;count:2 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.069+0000 7fa857b05700 1 -- 172.21.15.196:0/179206809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 msgr2=0x7fa8500fef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.069+0000 7fa857b05700 1 --2- 172.21.15.196:0/179206809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 0x7fa8500fef60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009910 tx=0x7fa84c009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.069+0000 7fa857b05700 1 -- 172.21.15.196:0/179206809 shutdown_connections 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.069+0000 7fa857b05700 1 --2- 172.21.15.196:0/179206809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 0x7fa8500fef60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.069+0000 7fa857b05700 1 -- 172.21.15.196:0/179206809 >> 172.21.15.196:0/179206809 conn(0x7fa8500fa4a0 msgr2=0x7fa8500fc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.069+0000 7fa857b05700 1 -- 172.21.15.196:0/179206809 shutdown_connections 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.069+0000 7fa857b05700 1 -- 172.21.15.196:0/179206809 wait complete. 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.069+0000 7fa857b05700 1 Processor -- start 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.069+0000 7fa857b05700 1 -- start start 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa857b05700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 0x7fa850197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:01.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa857b05700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa850198460 con 0x7fa8500feb40 2024-04-22T15:10:01.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa8558a1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 0x7fa850197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:01.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa8558a1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 0x7fa850197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:36998/0 (socket says 172.21.15.196:36998) 2024-04-22T15:10:01.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa8558a1700 1 -- 172.21.15.196:0/3980448738 learned_addr learned my addr 172.21.15.196:0/3980448738 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:01.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa8558a1700 1 -- 172.21.15.196:0/3980448738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c0095f0 con 0x7fa8500feb40 2024-04-22T15:10:01.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa8558a1700 1 --2- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 0x7fa850197f20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa84c006010 tx=0x7fa84c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:01.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa846ffd700 1 -- 172.21.15.196:0/3980448738 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa84c01d070 con 0x7fa8500feb40 2024-04-22T15:10:01.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa846ffd700 1 -- 172.21.15.196:0/3980448738 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa84c005dc0 con 0x7fa8500feb40 2024-04-22T15:10:01.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa850198660 con 0x7fa8500feb40 2024-04-22T15:10:01.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.070+0000 7fa846ffd700 1 -- 172.21.15.196:0/3980448738 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa84c022e90 con 0x7fa8500feb40 2024-04-22T15:10:01.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.071+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa850198b00 con 0x7fa8500feb40 2024-04-22T15:10:01.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.071+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa850191ac0 con 0x7fa8500feb40 2024-04-22T15:10:01.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.072+0000 7fa846ffd700 1 -- 172.21.15.196:0/3980448738 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fa84c022470 con 0x7fa8500feb40 2024-04-22T15:10:01.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.072+0000 7fa846ffd700 1 --2- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa83c038760 0x7fa83c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:01.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.072+0000 7fa846ffd700 1 -- 172.21.15.196:0/3980448738 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa84c04c7b0 con 0x7fa8500feb40 2024-04-22T15:10:01.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.072+0000 7fa8550a0700 1 --2- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa83c038760 0x7fa83c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:01.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.072+0000 7fa8550a0700 1 --2- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa83c038760 0x7fa83c03ac20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa840006fd0 tx=0x7fa840006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:01.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.077+0000 7fa846ffd700 1 -- 172.21.15.196:0/3980448738 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa84c00baf0 con 0x7fa8500feb40 2024-04-22T15:10:01.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:01.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:01.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:01.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:01.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:01.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:01.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:01.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:01.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:01.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T15:10:01.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:00 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:01.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.354+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8500ff670 con 0x7fa8500feb40 2024-04-22T15:10:01.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.354+0000 7fa846ffd700 1 -- 172.21.15.196:0/3980448738 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa8500ff670 con 0x7fa8500feb40 2024-04-22T15:10:01.358 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:01.358 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:01.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa83c038760 msgr2=0x7fa83c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:01.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 --2- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa83c038760 0x7fa83c03ac20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa840006fd0 tx=0x7fa840006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:01.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 msgr2=0x7fa850197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:01.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 --2- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 0x7fa850197f20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa84c006010 tx=0x7fa84c009f80 comp rx=0 tx=0).stop 2024-04-22T15:10:01.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 shutdown_connections 2024-04-22T15:10:01.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 --2- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fa83c038760 0x7fa83c03ac20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:01.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 --2- 172.21.15.196:0/3980448738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8500feb40 0x7fa850197f20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:01.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 >> 172.21.15.196:0/3980448738 conn(0x7fa8500fa4a0 msgr2=0x7fa8500fc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:01.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 shutdown_connections 2024-04-22T15:10:01.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:01.357+0000 7fa857b05700 1 -- 172.21.15.196:0/3980448738 wait complete. 2024-04-22T15:10:01.361 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:02.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:01 smithi096 ceph-mon[26688]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-22T15:10:02.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:01 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T15:10:02.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:01 smithi096 ceph-mon[26688]: Deploying daemon grafana.smithi096 on smithi096 2024-04-22T15:10:02.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:01 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/3980448738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:02.966 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:02.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:03.196 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:03.243 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:03.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:02 smithi096 ceph-mon[26688]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:04.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.641+0000 7effeee41700 1 -- 172.21.15.196:0/1024783590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 msgr2=0x7effe80ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:04.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.641+0000 7effeee41700 1 --2- 172.21.15.196:0/1024783590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 0x7effe80ffb90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7effdc009910 tx=0x7effdc009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:04.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.641+0000 7effeee41700 1 -- 172.21.15.196:0/1024783590 shutdown_connections 2024-04-22T15:10:04.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.641+0000 7effeee41700 1 --2- 172.21.15.196:0/1024783590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 0x7effe80ffb90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:04.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.641+0000 7effeee41700 1 -- 172.21.15.196:0/1024783590 >> 172.21.15.196:0/1024783590 conn(0x7effe80fad10 msgr2=0x7effe80fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:04.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.641+0000 7effeee41700 1 -- 172.21.15.196:0/1024783590 shutdown_connections 2024-04-22T15:10:04.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.641+0000 7effeee41700 1 -- 172.21.15.196:0/1024783590 wait complete. 2024-04-22T15:10:04.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.642+0000 7effeee41700 1 Processor -- start 2024-04-22T15:10:04.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.642+0000 7effeee41700 1 -- start start 2024-04-22T15:10:04.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.642+0000 7effeee41700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 0x7effe8068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.642+0000 7effeee41700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe8068fa0 con 0x7effe80ff770 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.642+0000 7effecbdd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 0x7effe8068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.642+0000 7effecbdd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 0x7effe8068a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:37020/0 (socket says 172.21.15.196:37020) 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.642+0000 7effecbdd700 1 -- 172.21.15.196:0/265554221 learned_addr learned my addr 172.21.15.196:0/265554221 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.643+0000 7effecbdd700 1 -- 172.21.15.196:0/265554221 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc0095f0 con 0x7effe80ff770 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.643+0000 7effecbdd700 1 --2- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 0x7effe8068a60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7effdc006010 tx=0x7effdc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.643+0000 7effe5ffb700 1 -- 172.21.15.196:0/265554221 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effdc01d070 con 0x7effe80ff770 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.643+0000 7effe5ffb700 1 -- 172.21.15.196:0/265554221 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effdc005dc0 con 0x7effe80ff770 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.643+0000 7effe5ffb700 1 -- 172.21.15.196:0/265554221 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effdc022e90 con 0x7effe80ff770 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.643+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe8196850 con 0x7effe80ff770 2024-04-22T15:10:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.643+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe819b0b0 con 0x7effe80ff770 2024-04-22T15:10:04.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.644+0000 7effe5ffb700 1 -- 172.21.15.196:0/265554221 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7effdc022470 con 0x7effe80ff770 2024-04-22T15:10:04.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.644+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe818e5a0 con 0x7effe80ff770 2024-04-22T15:10:04.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.644+0000 7effe5ffb700 1 --2- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7effd80387c0 0x7effd803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:04.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.644+0000 7effe5ffb700 1 -- 172.21.15.196:0/265554221 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7effdc04c850 con 0x7effe80ff770 2024-04-22T15:10:04.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.644+0000 7effe7fff700 1 --2- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7effd80387c0 0x7effd803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:04.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.645+0000 7effe7fff700 1 --2- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7effd80387c0 0x7effd803ac80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7effd4006fd0 tx=0x7effd4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:04.652 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.650+0000 7effe5ffb700 1 -- 172.21.15.196:0/265554221 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7effdc00f4b0 con 0x7effe80ff770 2024-04-22T15:10:04.921 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.917+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effe8062510 con 0x7effe80ff770 2024-04-22T15:10:04.921 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.918+0000 7effe5ffb700 1 -- 172.21.15.196:0/265554221 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7effdc027030 con 0x7effe80ff770 2024-04-22T15:10:04.921 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:04.921 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:04.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7effd80387c0 msgr2=0x7effd803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:04.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 --2- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7effd80387c0 0x7effd803ac80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7effd4006fd0 tx=0x7effd4006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:04.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 msgr2=0x7effe8068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:04.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 --2- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 0x7effe8068a60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7effdc006010 tx=0x7effdc009f80 comp rx=0 tx=0).stop 2024-04-22T15:10:04.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 shutdown_connections 2024-04-22T15:10:04.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 --2- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7effd80387c0 0x7effd803ac80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:04.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 --2- 172.21.15.196:0/265554221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe80ff770 0x7effe8068a60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:04.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 >> 172.21.15.196:0/265554221 conn(0x7effe80fad10 msgr2=0x7effe80fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:04.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 shutdown_connections 2024-04-22T15:10:04.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:04.920+0000 7effeee41700 1 -- 172.21.15.196:0/265554221 wait complete. 2024-04-22T15:10:04.925 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:05.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:04 smithi096 ceph-mon[26688]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:05.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:04 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:05.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:04 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/265554221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:06.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:06.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:06.690 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:06.738 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:07.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:06 smithi096 ceph-mon[26688]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:08.109 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.105+0000 7fb84bb25700 1 -- 172.21.15.196:0/3693906386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 msgr2=0x7fb844103750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:08.109 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.105+0000 7fb84bb25700 1 --2- 172.21.15.196:0/3693906386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 0x7fb844103750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb834009910 tx=0x7fb834009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.105+0000 7fb84bb25700 1 -- 172.21.15.196:0/3693906386 shutdown_connections 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.105+0000 7fb84bb25700 1 --2- 172.21.15.196:0/3693906386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 0x7fb844103750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.105+0000 7fb84bb25700 1 -- 172.21.15.196:0/3693906386 >> 172.21.15.196:0/3693906386 conn(0x7fb8440fad10 msgr2=0x7fb8440fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.105+0000 7fb84bb25700 1 -- 172.21.15.196:0/3693906386 shutdown_connections 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.105+0000 7fb84bb25700 1 -- 172.21.15.196:0/3693906386 wait complete. 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.106+0000 7fb84bb25700 1 Processor -- start 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.106+0000 7fb84bb25700 1 -- start start 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.106+0000 7fb84bb25700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 0x7fb844195ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.106+0000 7fb84bb25700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb844196220 con 0x7fb844101360 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.106+0000 7fb8498c1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 0x7fb844195ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.106+0000 7fb8498c1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 0x7fb844195ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:53160/0 (socket says 172.21.15.196:53160) 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.106+0000 7fb8498c1700 1 -- 172.21.15.196:0/4047055156 learned_addr learned my addr 172.21.15.196:0/4047055156 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.107+0000 7fb8498c1700 1 -- 172.21.15.196:0/4047055156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8340095f0 con 0x7fb844101360 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.107+0000 7fb8498c1700 1 --2- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 0x7fb844195ce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb834009ed0 tx=0x7fb834004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.107+0000 7fb83affd700 1 -- 172.21.15.196:0/4047055156 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb83401d070 con 0x7fb844101360 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.107+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb844196420 con 0x7fb844101360 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.107+0000 7fb83affd700 1 -- 172.21.15.196:0/4047055156 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb83400be20 con 0x7fb844101360 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.107+0000 7fb83affd700 1 -- 172.21.15.196:0/4047055156 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb83400f460 con 0x7fb844101360 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.107+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8441968c0 con 0x7fb844101360 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.108+0000 7fb83affd700 1 -- 172.21.15.196:0/4047055156 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fb83400f650 con 0x7fb844101360 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.108+0000 7fb83affd700 1 --2- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb8300387d0 0x7fb83003ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.108+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84418f6a0 con 0x7fb844101360 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.108+0000 7fb83affd700 1 -- 172.21.15.196:0/4047055156 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb83404cbf0 con 0x7fb844101360 2024-04-22T15:10:08.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.108+0000 7fb8490c0700 1 --2- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb8300387d0 0x7fb83003ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:08.113 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.109+0000 7fb8490c0700 1 --2- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb8300387d0 0x7fb83003ac90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb840006fd0 tx=0x7fb840006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:08.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.113+0000 7fb83affd700 1 -- 172.21.15.196:0/4047055156 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb83402c440 con 0x7fb844101360 2024-04-22T15:10:08.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.381+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb844062510 con 0x7fb844101360 2024-04-22T15:10:08.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.382+0000 7fb83affd700 1 -- 172.21.15.196:0/4047055156 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb834020b60 con 0x7fb844101360 2024-04-22T15:10:08.385 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:08.385 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:08.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb8300387d0 msgr2=0x7fb83003ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:08.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 --2- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb8300387d0 0x7fb83003ac90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb840006fd0 tx=0x7fb840006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:08.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 msgr2=0x7fb844195ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:08.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 --2- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 0x7fb844195ce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb834009ed0 tx=0x7fb834004690 comp rx=0 tx=0).stop 2024-04-22T15:10:08.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 shutdown_connections 2024-04-22T15:10:08.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 --2- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb8300387d0 0x7fb83003ac90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:08.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 --2- 172.21.15.196:0/4047055156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb844101360 0x7fb844195ce0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:08.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 >> 172.21.15.196:0/4047055156 conn(0x7fb8440fad10 msgr2=0x7fb8440fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:08.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 shutdown_connections 2024-04-22T15:10:08.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:08.384+0000 7fb84bb25700 1 -- 172.21.15.196:0/4047055156 wait complete. 2024-04-22T15:10:08.388 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:09.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:08 smithi096 ceph-mon[26688]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:09.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:08 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/4047055156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:10.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:10.001 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:10.202 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:10.250 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:10 smithi096 ceph-mon[26688]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:11.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.714+0000 7fd07342d700 1 -- 172.21.15.196:0/711296359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 msgr2=0x7fd06c100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:11.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.714+0000 7fd07342d700 1 --2- 172.21.15.196:0/711296359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 0x7fd06c100cb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd06400b0a0 tx=0x7fd06400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:10:11.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.714+0000 7fd07342d700 1 -- 172.21.15.196:0/711296359 shutdown_connections 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.714+0000 7fd07342d700 1 --2- 172.21.15.196:0/711296359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 0x7fd06c100cb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.714+0000 7fd07342d700 1 -- 172.21.15.196:0/711296359 >> 172.21.15.196:0/711296359 conn(0x7fd06c0fbe30 msgr2=0x7fd06c0fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.714+0000 7fd07342d700 1 -- 172.21.15.196:0/711296359 shutdown_connections 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.714+0000 7fd07342d700 1 -- 172.21.15.196:0/711296359 wait complete. 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.715+0000 7fd07342d700 1 Processor -- start 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.715+0000 7fd07342d700 1 -- start start 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.715+0000 7fd07342d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 0x7fd06c195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.715+0000 7fd07342d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c196280 con 0x7fd06c100890 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.715+0000 7fd0711c9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 0x7fd06c195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.715+0000 7fd0711c9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 0x7fd06c195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:53182/0 (socket says 172.21.15.196:53182) 2024-04-22T15:10:11.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.715+0000 7fd0711c9700 1 -- 172.21.15.196:0/2639497427 learned_addr learned my addr 172.21.15.196:0/2639497427 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:11.721 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.715+0000 7fd0711c9700 1 -- 172.21.15.196:0/2639497427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd064009d00 con 0x7fd06c100890 2024-04-22T15:10:11.721 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.716+0000 7fd0711c9700 1 --2- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 0x7fd06c195d40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd0640062a0 tx=0x7fd06400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:11.721 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.716+0000 7fd06a7fc700 1 -- 172.21.15.196:0/2639497427 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd06400e040 con 0x7fd06c100890 2024-04-22T15:10:11.721 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.716+0000 7fd06a7fc700 1 -- 172.21.15.196:0/2639497427 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd064003d70 con 0x7fd06c100890 2024-04-22T15:10:11.721 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.716+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06c196480 con 0x7fd06c100890 2024-04-22T15:10:11.721 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.716+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06c196920 con 0x7fd06c100890 2024-04-22T15:10:11.721 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.716+0000 7fd06a7fc700 1 -- 172.21.15.196:0/2639497427 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd06401c890 con 0x7fd06c100890 2024-04-22T15:10:11.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.717+0000 7fd06a7fc700 1 -- 172.21.15.196:0/2639497427 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fd06401cb50 con 0x7fd06c100890 2024-04-22T15:10:11.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.717+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06c18f6c0 con 0x7fd06c100890 2024-04-22T15:10:11.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.717+0000 7fd06a7fc700 1 --2- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fd0540387c0 0x7fd05403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:11.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.717+0000 7fd06a7fc700 1 -- 172.21.15.196:0/2639497427 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd06404ca60 con 0x7fd06c100890 2024-04-22T15:10:11.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.717+0000 7fd0709c8700 1 --2- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fd0540387c0 0x7fd05403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:11.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.717+0000 7fd0709c8700 1 --2- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fd0540387c0 0x7fd05403ac80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd060006fd0 tx=0x7fd060006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:11.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.722+0000 7fd06a7fc700 1 -- 172.21.15.196:0/2639497427 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fd06401a3b0 con 0x7fd06c100890 2024-04-22T15:10:11.992 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.989+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd06c062510 con 0x7fd06c100890 2024-04-22T15:10:11.992 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.989+0000 7fd06a7fc700 1 -- 172.21.15.196:0/2639497427 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fd06401a3b0 con 0x7fd06c100890 2024-04-22T15:10:11.993 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:11.993 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:11.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fd0540387c0 msgr2=0x7fd05403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:11.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 --2- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fd0540387c0 0x7fd05403ac80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd060006fd0 tx=0x7fd060006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:11.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 msgr2=0x7fd06c195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:11.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 --2- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 0x7fd06c195d40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd0640062a0 tx=0x7fd06400bd40 comp rx=0 tx=0).stop 2024-04-22T15:10:11.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 shutdown_connections 2024-04-22T15:10:11.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 --2- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fd0540387c0 0x7fd05403ac80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:11.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 --2- 172.21.15.196:0/2639497427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c100890 0x7fd06c195d40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:11.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 >> 172.21.15.196:0/2639497427 conn(0x7fd06c0fbe30 msgr2=0x7fd06c0fc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:11.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 shutdown_connections 2024-04-22T15:10:11.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:11.992+0000 7fd07342d700 1 -- 172.21.15.196:0/2639497427 wait complete. 2024-04-22T15:10:11.996 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:13.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:12 smithi096 ceph-mon[26688]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:13.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:12 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/2639497427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:13.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:13.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:13.703 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:13.751 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:15.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:14 smithi096 ceph-mon[26688]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:15.246 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.242+0000 7f9f55ef1700 1 -- 172.21.15.196:0/2966761241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 msgr2=0x7f9f500ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:15.246 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.242+0000 7f9f55ef1700 1 --2- 172.21.15.196:0/2966761241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 0x7f9f500ffb90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800b0a0 tx=0x7f9f4800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.242+0000 7f9f55ef1700 1 -- 172.21.15.196:0/2966761241 shutdown_connections 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.242+0000 7f9f55ef1700 1 --2- 172.21.15.196:0/2966761241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 0x7f9f500ffb90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.242+0000 7f9f55ef1700 1 -- 172.21.15.196:0/2966761241 >> 172.21.15.196:0/2966761241 conn(0x7f9f500fad10 msgr2=0x7f9f500fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.242+0000 7f9f55ef1700 1 -- 172.21.15.196:0/2966761241 shutdown_connections 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.242+0000 7f9f55ef1700 1 -- 172.21.15.196:0/2966761241 wait complete. 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.243+0000 7f9f55ef1700 1 Processor -- start 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.243+0000 7f9f55ef1700 1 -- start start 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.243+0000 7f9f55ef1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 0x7f9f50195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.243+0000 7f9f55ef1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f50196230 con 0x7f9f500ff770 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.243+0000 7f9f4f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 0x7f9f50195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.243+0000 7f9f4f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 0x7f9f50195cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:53192/0 (socket says 172.21.15.196:53192) 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.243+0000 7f9f4f7fe700 1 -- 172.21.15.196:0/246371468 learned_addr learned my addr 172.21.15.196:0/246371468 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:15.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.243+0000 7f9f4f7fe700 1 -- 172.21.15.196:0/246371468 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f48009d00 con 0x7f9f500ff770 2024-04-22T15:10:15.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.244+0000 7f9f4f7fe700 1 --2- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 0x7f9f50195cf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f480062a0 tx=0x7f9f4800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.244+0000 7f9f4cff9700 1 -- 172.21.15.196:0/246371468 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4800e040 con 0x7f9f500ff770 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.244+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f50196430 con 0x7f9f500ff770 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.244+0000 7f9f4cff9700 1 -- 172.21.15.196:0/246371468 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f48003d70 con 0x7f9f500ff770 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.244+0000 7f9f4cff9700 1 -- 172.21.15.196:0/246371468 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4801c890 con 0x7f9f500ff770 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.244+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f501968d0 con 0x7f9f500ff770 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.245+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5018f820 con 0x7f9f500ff770 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.245+0000 7f9f4cff9700 1 -- 172.21.15.196:0/246371468 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f9f4801cb50 con 0x7f9f500ff770 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.245+0000 7f9f4cff9700 1 --2- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9f380387c0 0x7f9f3803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.245+0000 7f9f4cff9700 1 -- 172.21.15.196:0/246371468 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9f4804ca60 con 0x7f9f500ff770 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.245+0000 7f9f4effd700 1 --2- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9f380387c0 0x7f9f3803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.245+0000 7f9f4effd700 1 --2- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9f380387c0 0x7f9f3803ac80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9f44006fd0 tx=0x7f9f44006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:15.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.250+0000 7f9f4cff9700 1 -- 172.21.15.196:0/246371468 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9f4801a3b0 con 0x7f9f500ff770 2024-04-22T15:10:15.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.521+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f50062510 con 0x7f9f500ff770 2024-04-22T15:10:15.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.522+0000 7f9f4cff9700 1 -- 172.21.15.196:0/246371468 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9f50062510 con 0x7f9f500ff770 2024-04-22T15:10:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:15.525 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:15.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9f380387c0 msgr2=0x7f9f3803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:15.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 --2- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9f380387c0 0x7f9f3803ac80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9f44006fd0 tx=0x7f9f44006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:15.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 msgr2=0x7f9f50195cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:15.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 --2- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 0x7f9f50195cf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f480062a0 tx=0x7f9f4800bd40 comp rx=0 tx=0).stop 2024-04-22T15:10:15.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 shutdown_connections 2024-04-22T15:10:15.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 --2- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9f380387c0 0x7f9f3803ac80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:15.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 --2- 172.21.15.196:0/246371468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f500ff770 0x7f9f50195cf0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:15.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 >> 172.21.15.196:0/246371468 conn(0x7f9f500fad10 msgr2=0x7f9f50068d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:15.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 shutdown_connections 2024-04-22T15:10:15.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:15.524+0000 7f9f55ef1700 1 -- 172.21.15.196:0/246371468 wait complete. 2024-04-22T15:10:15.528 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:16.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:15 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/246371468' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:17.158 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:17.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:17.183 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:16 smithi096 ceph-mon[26688]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:17.357 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:17.406 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:18.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.904+0000 7f5dcb809700 1 -- 172.21.15.196:0/4001711614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4101370 msgr2=0x7f5dc4103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:18.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.904+0000 7f5dcb809700 1 --2- 172.21.15.196:0/4001711614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4101370 0x7f5dc4103760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009910 tx=0x7f5dc0009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:18.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.905+0000 7f5dcb809700 1 -- 172.21.15.196:0/4001711614 shutdown_connections 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.905+0000 7f5dcb809700 1 --2- 172.21.15.196:0/4001711614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4101370 0x7f5dc4103760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.905+0000 7f5dcb809700 1 -- 172.21.15.196:0/4001711614 >> 172.21.15.196:0/4001711614 conn(0x7f5dc40fad00 msgr2=0x7f5dc40fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.905+0000 7f5dcb809700 1 -- 172.21.15.196:0/4001711614 shutdown_connections 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.905+0000 7f5dcb809700 1 -- 172.21.15.196:0/4001711614 wait complete. 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.905+0000 7f5dcb809700 1 Processor -- start 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.905+0000 7f5dcb809700 1 -- start start 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.905+0000 7f5dcb809700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4100b60 0x7f5dc40ff320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.905+0000 7f5dcb809700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc4100f80 con 0x7f5dc4100b60 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dc95a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4100b60 0x7f5dc40ff320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dc95a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4100b60 0x7f5dc40ff320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:41474/0 (socket says 172.21.15.196:41474) 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dc95a5700 1 -- 172.21.15.196:0/2493787876 learned_addr learned my addr 172.21.15.196:0/2493787876 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dc95a5700 1 -- 172.21.15.196:0/2493787876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00095f0 con 0x7f5dc4100b60 2024-04-22T15:10:18.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dc95a5700 1 --2- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4100b60 0x7f5dc40ff320 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00052c0 tx=0x7f5dc0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:18.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dba7fc700 1 -- 172.21.15.196:0/2493787876 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dc001d070 con 0x7f5dc4100b60 2024-04-22T15:10:18.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc4101180 con 0x7f5dc4100b60 2024-04-22T15:10:18.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dba7fc700 1 -- 172.21.15.196:0/2493787876 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5dc0005dc0 con 0x7f5dc4100b60 2024-04-22T15:10:18.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dba7fc700 1 -- 172.21.15.196:0/2493787876 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dc0022d80 con 0x7f5dc4100b60 2024-04-22T15:10:18.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.906+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc40ffab0 con 0x7f5dc4100b60 2024-04-22T15:10:18.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.907+0000 7f5dba7fc700 1 -- 172.21.15.196:0/2493787876 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f5dc000b960 con 0x7f5dc4100b60 2024-04-22T15:10:18.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.907+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc418f690 con 0x7f5dc4100b60 2024-04-22T15:10:18.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.907+0000 7f5dba7fc700 1 --2- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f5db00387d0 0x7f5db003ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:18.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.907+0000 7f5dba7fc700 1 -- 172.21.15.196:0/2493787876 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5dc004c6b0 con 0x7f5dc4100b60 2024-04-22T15:10:18.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.907+0000 7f5dc8da4700 1 --2- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f5db00387d0 0x7f5db003ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:18.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.908+0000 7f5dc8da4700 1 --2- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f5db00387d0 0x7f5db003ac90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5db4006fd0 tx=0x7f5db4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:18.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:18.912+0000 7f5dba7fc700 1 -- 172.21.15.196:0/2493787876 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f5dc0005c00 con 0x7f5dc4100b60 2024-04-22T15:10:19.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.183+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5dc4062510 con 0x7f5dc4100b60 2024-04-22T15:10:19.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.183+0000 7f5dba7fc700 1 -- 172.21.15.196:0/2493787876 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f5dc0027030 con 0x7f5dc4100b60 2024-04-22T15:10:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:19.187 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:19.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f5db00387d0 msgr2=0x7f5db003ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:19.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 --2- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f5db00387d0 0x7f5db003ac90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5db4006fd0 tx=0x7f5db4006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:19.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4100b60 msgr2=0x7f5dc40ff320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:19.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 --2- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4100b60 0x7f5dc40ff320 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00052c0 tx=0x7f5dc0009f80 comp rx=0 tx=0).stop 2024-04-22T15:10:19.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 shutdown_connections 2024-04-22T15:10:19.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 --2- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f5db00387d0 0x7f5db003ac90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:19.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 --2- 172.21.15.196:0/2493787876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dc4100b60 0x7f5dc40ff320 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:19.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 >> 172.21.15.196:0/2493787876 conn(0x7f5dc40fad00 msgr2=0x7f5dc40fb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:19.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 shutdown_connections 2024-04-22T15:10:19.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:19.185+0000 7f5dcb809700 1 -- 172.21.15.196:0/2493787876 wait complete. 2024-04-22T15:10:19.190 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:18 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:18 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:18 smithi096 ceph-mon[26688]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:18 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:18 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:18 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:18 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:19.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:18 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:19.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:18 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:20.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:19 smithi096 ceph-mon[26688]: Deploying daemon prometheus.smithi096 on smithi096 2024-04-22T15:10:20.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:19 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:20.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:19 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/2493787876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:20.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:20.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:21.011 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:21.059 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:21.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:20 smithi096 ceph-mon[26688]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:22.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.507+0000 7f11d882d700 1 -- 172.21.15.196:0/2677472231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 msgr2=0x7f11d40ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:22.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.507+0000 7f11d882d700 1 --2- 172.21.15.196:0/2677472231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 0x7f11d40ffb90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009910 tx=0x7f11c4009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:22.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.507+0000 7f11d882d700 1 -- 172.21.15.196:0/2677472231 shutdown_connections 2024-04-22T15:10:22.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.507+0000 7f11d882d700 1 --2- 172.21.15.196:0/2677472231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 0x7f11d40ffb90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:22.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.507+0000 7f11d882d700 1 -- 172.21.15.196:0/2677472231 >> 172.21.15.196:0/2677472231 conn(0x7f11d40fad10 msgr2=0x7f11d40fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.507+0000 7f11d882d700 1 -- 172.21.15.196:0/2677472231 shutdown_connections 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.507+0000 7f11d882d700 1 -- 172.21.15.196:0/2677472231 wait complete. 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.508+0000 7f11d882d700 1 Processor -- start 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.508+0000 7f11d882d700 1 -- start start 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.508+0000 7f11d882d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 0x7f11d4069310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.508+0000 7f11d882d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4069850 con 0x7f11d40ff770 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.508+0000 7f11d259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 0x7f11d4069310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.508+0000 7f11d259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 0x7f11d4069310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:41486/0 (socket says 172.21.15.196:41486) 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.508+0000 7f11d259c700 1 -- 172.21.15.196:0/1087042194 learned_addr learned my addr 172.21.15.196:0/1087042194 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.508+0000 7f11d259c700 1 -- 172.21.15.196:0/1087042194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40095f0 con 0x7f11d40ff770 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.509+0000 7f11d259c700 1 --2- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 0x7f11d4069310 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f11c4006010 tx=0x7f11c4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.509+0000 7f11cb7fe700 1 -- 172.21.15.196:0/1087042194 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11c401d070 con 0x7f11d40ff770 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.509+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d4198a40 con 0x7f11d40ff770 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.509+0000 7f11cb7fe700 1 -- 172.21.15.196:0/1087042194 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11c4005dc0 con 0x7f11d40ff770 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.509+0000 7f11cb7fe700 1 -- 172.21.15.196:0/1087042194 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11c4022e90 con 0x7f11d40ff770 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.509+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d4198ee0 con 0x7f11d40ff770 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.510+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d418e5a0 con 0x7f11d40ff770 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.510+0000 7f11cb7fe700 1 -- 172.21.15.196:0/1087042194 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f11c4022470 con 0x7f11d40ff770 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.510+0000 7f11cb7fe700 1 --2- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f11c0038760 0x7f11c003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.510+0000 7f11cb7fe700 1 -- 172.21.15.196:0/1087042194 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f11c404c7f0 con 0x7f11d40ff770 2024-04-22T15:10:22.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.510+0000 7f11cbfff700 1 --2- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f11c0038760 0x7f11c003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:22.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.510+0000 7f11cbfff700 1 --2- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f11c0038760 0x7f11c003ac20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f11bc006fd0 tx=0x7f11bc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:22.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.517+0000 7f11cb7fe700 1 -- 172.21.15.196:0/1087042194 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f11c400f4f0 con 0x7f11d40ff770 2024-04-22T15:10:22.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.790+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11d4062510 con 0x7f11d40ff770 2024-04-22T15:10:22.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.791+0000 7f11cb7fe700 1 -- 172.21.15.196:0/1087042194 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f11c4027070 con 0x7f11d40ff770 2024-04-22T15:10:22.794 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:22.794 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:22.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f11c0038760 msgr2=0x7f11c003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:22.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 --2- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f11c0038760 0x7f11c003ac20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f11bc006fd0 tx=0x7f11bc006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:22.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 msgr2=0x7f11d4069310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:22.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 --2- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 0x7f11d4069310 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f11c4006010 tx=0x7f11c4009f80 comp rx=0 tx=0).stop 2024-04-22T15:10:22.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 shutdown_connections 2024-04-22T15:10:22.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 --2- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f11c0038760 0x7f11c003ac20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:22.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 --2- 172.21.15.196:0/1087042194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d40ff770 0x7f11d4069310 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:22.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 >> 172.21.15.196:0/1087042194 conn(0x7f11d40fad10 msgr2=0x7f11d40fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:22.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 shutdown_connections 2024-04-22T15:10:22.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:22.793+0000 7f11d882d700 1 -- 172.21.15.196:0/1087042194 wait complete. 2024-04-22T15:10:22.797 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:22 smithi096 ceph-mon[26688]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:22 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/1087042194' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:24.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:24.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:24.588 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:24.636 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:25.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:24 smithi096 ceph-mon[26688]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:26.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.071+0000 7fb2cdacb700 1 -- 172.21.15.196:0/3794630924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 msgr2=0x7fb2c80ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:26.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.071+0000 7fb2cdacb700 1 --2- 172.21.15.196:0/3794630924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 0x7fb2c80ffb80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009910 tx=0x7fb2b8009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:26.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.071+0000 7fb2cdacb700 1 -- 172.21.15.196:0/3794630924 shutdown_connections 2024-04-22T15:10:26.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.071+0000 7fb2cdacb700 1 --2- 172.21.15.196:0/3794630924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 0x7fb2c80ffb80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:26.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.071+0000 7fb2cdacb700 1 -- 172.21.15.196:0/3794630924 >> 172.21.15.196:0/3794630924 conn(0x7fb2c80fad00 msgr2=0x7fb2c80fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:26.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.071+0000 7fb2cdacb700 1 -- 172.21.15.196:0/3794630924 shutdown_connections 2024-04-22T15:10:26.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.071+0000 7fb2cdacb700 1 -- 172.21.15.196:0/3794630924 wait complete. 2024-04-22T15:10:26.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.071+0000 7fb2cdacb700 1 Processor -- start 2024-04-22T15:10:26.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.072+0000 7fb2cdacb700 1 -- start start 2024-04-22T15:10:26.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.072+0000 7fb2cdacb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 0x7fb2c8195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:26.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.072+0000 7fb2cdacb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8196260 con 0x7fb2c80ff760 2024-04-22T15:10:26.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.072+0000 7fb2c77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 0x7fb2c8195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:26.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.072+0000 7fb2c77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 0x7fb2c8195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:41498/0 (socket says 172.21.15.196:41498) 2024-04-22T15:10:26.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.072+0000 7fb2c77fe700 1 -- 172.21.15.196:0/1922887069 learned_addr learned my addr 172.21.15.196:0/1922887069 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:26.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.072+0000 7fb2c77fe700 1 -- 172.21.15.196:0/1922887069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b80095f0 con 0x7fb2c80ff760 2024-04-22T15:10:26.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.073+0000 7fb2c77fe700 1 --2- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 0x7fb2c8195d20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8006010 tx=0x7fb2b8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:26.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.073+0000 7fb2c4ff9700 1 -- 172.21.15.196:0/1922887069 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2b801d070 con 0x7fb2c80ff760 2024-04-22T15:10:26.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.073+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c8196460 con 0x7fb2c80ff760 2024-04-22T15:10:26.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.073+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c8196900 con 0x7fb2c80ff760 2024-04-22T15:10:26.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.073+0000 7fb2c4ff9700 1 -- 172.21.15.196:0/1922887069 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2b8005dc0 con 0x7fb2c80ff760 2024-04-22T15:10:26.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.073+0000 7fb2c4ff9700 1 -- 172.21.15.196:0/1922887069 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2b8022e70 con 0x7fb2c80ff760 2024-04-22T15:10:26.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.074+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c818f850 con 0x7fb2c80ff760 2024-04-22T15:10:26.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.074+0000 7fb2c4ff9700 1 -- 172.21.15.196:0/1922887069 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fb2b800b960 con 0x7fb2c80ff760 2024-04-22T15:10:26.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.074+0000 7fb2c4ff9700 1 --2- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2b0038770 0x7fb2b003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:26.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.074+0000 7fb2c4ff9700 1 -- 172.21.15.196:0/1922887069 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb2b804c7e0 con 0x7fb2c80ff760 2024-04-22T15:10:26.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.074+0000 7fb2c6ffd700 1 --2- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2b0038770 0x7fb2b003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:26.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.075+0000 7fb2c6ffd700 1 --2- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2b0038770 0x7fb2b003ac30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc006fd0 tx=0x7fb2bc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:26.086 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.082+0000 7fb2c4ff9700 1 -- 172.21.15.196:0/1922887069 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb2b8022470 con 0x7fb2c80ff760 2024-04-22T15:10:26.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.355+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2c8062510 con 0x7fb2c80ff760 2024-04-22T15:10:26.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.355+0000 7fb2c4ff9700 1 -- 172.21.15.196:0/1922887069 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb2b8027030 con 0x7fb2c80ff760 2024-04-22T15:10:26.360 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:26.360 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:26.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2b0038770 msgr2=0x7fb2b003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:26.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 --2- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2b0038770 0x7fb2b003ac30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc006fd0 tx=0x7fb2bc006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:26.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 msgr2=0x7fb2c8195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:26.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 --2- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 0x7fb2c8195d20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8006010 tx=0x7fb2b8009f80 comp rx=0 tx=0).stop 2024-04-22T15:10:26.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 shutdown_connections 2024-04-22T15:10:26.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 --2- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7fb2b0038770 0x7fb2b003ac30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:26.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 --2- 172.21.15.196:0/1922887069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c80ff760 0x7fb2c8195d20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:26.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 >> 172.21.15.196:0/1922887069 conn(0x7fb2c80fad00 msgr2=0x7fb2c80690c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:26.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 shutdown_connections 2024-04-22T15:10:26.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:26.357+0000 7fb2cdacb700 1 -- 172.21.15.196:0/1922887069 wait complete. 2024-04-22T15:10:26.363 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:26 smithi096 ceph-mon[26688]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:26 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/1922887069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:27.945 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:27.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:28.157 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:28.219 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:29.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:28 smithi096 ceph-mon[26688]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:29.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.631+0000 7f845aa0e700 1 -- 172.21.15.196:0/890922560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 msgr2=0x7f84540ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:29.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.631+0000 7f845aa0e700 1 --2- 172.21.15.196:0/890922560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 0x7f84540ffb80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8444009910 tx=0x7f8444009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:29.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.631+0000 7f845aa0e700 1 -- 172.21.15.196:0/890922560 shutdown_connections 2024-04-22T15:10:29.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.631+0000 7f845aa0e700 1 --2- 172.21.15.196:0/890922560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 0x7f84540ffb80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:29.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.631+0000 7f845aa0e700 1 -- 172.21.15.196:0/890922560 >> 172.21.15.196:0/890922560 conn(0x7f84540fad00 msgr2=0x7f84540fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.631+0000 7f845aa0e700 1 -- 172.21.15.196:0/890922560 shutdown_connections 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.631+0000 7f845aa0e700 1 -- 172.21.15.196:0/890922560 wait complete. 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.632+0000 7f845aa0e700 1 Processor -- start 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.632+0000 7f845aa0e700 1 -- start start 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.632+0000 7f845aa0e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 0x7f84540696d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.632+0000 7f845aa0e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8454069c10 con 0x7f84540ff760 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.632+0000 7f8453fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 0x7f84540696d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.632+0000 7f8453fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 0x7f84540696d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:59292/0 (socket says 172.21.15.196:59292) 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.632+0000 7f8453fff700 1 -- 172.21.15.196:0/2953952648 learned_addr learned my addr 172.21.15.196:0/2953952648 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.632+0000 7f8453fff700 1 -- 172.21.15.196:0/2953952648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84440095f0 con 0x7f84540ff760 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.633+0000 7f8453fff700 1 --2- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 0x7f84540696d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8444006010 tx=0x7f8444009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:29.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.633+0000 7f84517fa700 1 -- 172.21.15.196:0/2953952648 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f844401d070 con 0x7f84540ff760 2024-04-22T15:10:29.639 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.633+0000 7f84517fa700 1 -- 172.21.15.196:0/2953952648 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8444005dc0 con 0x7f84540ff760 2024-04-22T15:10:29.639 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.633+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8454198a80 con 0x7f84540ff760 2024-04-22T15:10:29.639 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.633+0000 7f84517fa700 1 -- 172.21.15.196:0/2953952648 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8444022e90 con 0x7f84540ff760 2024-04-22T15:10:29.639 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.633+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8454198ec0 con 0x7f84540ff760 2024-04-22T15:10:29.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.634+0000 7f84517fa700 1 -- 172.21.15.196:0/2953952648 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f844400f4f0 con 0x7f84540ff760 2024-04-22T15:10:29.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.634+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845418e590 con 0x7f84540ff760 2024-04-22T15:10:29.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.634+0000 7f84517fa700 1 --2- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f843c038760 0x7f843c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:29.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.634+0000 7f84517fa700 1 -- 172.21.15.196:0/2953952648 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f844404ca20 con 0x7f84540ff760 2024-04-22T15:10:29.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.634+0000 7f84537fe700 1 --2- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f843c038760 0x7f843c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:29.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.635+0000 7f84537fe700 1 --2- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f843c038760 0x7f843c03ac20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8448006fd0 tx=0x7f8448006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:29.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.639+0000 7f84517fa700 1 -- 172.21.15.196:0/2953952648 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f844402a950 con 0x7f84540ff760 2024-04-22T15:10:29.918 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.913+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8454062510 con 0x7f84540ff760 2024-04-22T15:10:29.918 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.914+0000 7f84517fa700 1 -- 172.21.15.196:0/2953952648 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f8444027020 con 0x7f84540ff760 2024-04-22T15:10:29.918 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:29.918 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:29.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f843c038760 msgr2=0x7f843c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:29.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 --2- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f843c038760 0x7f843c03ac20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8448006fd0 tx=0x7f8448006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:29.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 msgr2=0x7f84540696d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:29.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 --2- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 0x7f84540696d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8444006010 tx=0x7f8444009f80 comp rx=0 tx=0).stop 2024-04-22T15:10:29.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 shutdown_connections 2024-04-22T15:10:29.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 --2- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f843c038760 0x7f843c03ac20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:29.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 --2- 172.21.15.196:0/2953952648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84540ff760 0x7f84540696d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:29.921 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 >> 172.21.15.196:0/2953952648 conn(0x7f84540fad00 msgr2=0x7f84540fb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:29.921 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 shutdown_connections 2024-04-22T15:10:29.921 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:29.916+0000 7f845aa0e700 1 -- 172.21.15.196:0/2953952648 wait complete. 2024-04-22T15:10:29.921 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:30.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:30 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:30.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:30 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:30.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:30 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:30.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:30 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T15:10:30.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:30 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/2953952648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:30.981 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:30 smithi096 ceph-mon[26688]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:31.554 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:31.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:31.756 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:31.803 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:31 smithi096 ceph-mon[26688]: from='mgr.14164 172.21.15.96:0/2457013581' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T15:10:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:31 smithi096 ceph-mon[26688]: mgrmap e14: smithi096.hhkvum(active, since 66s) 2024-04-22T15:10:33.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.277+0000 7f9d205f1700 1 -- 172.21.15.196:0/2302207626 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 msgr2=0x7f9d180ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:33.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.277+0000 7f9d205f1700 1 --2- 172.21.15.196:0/2302207626 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 0x7f9d180ffb90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009910 tx=0x7f9d08009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:33.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.277+0000 7f9d205f1700 1 -- 172.21.15.196:0/2302207626 shutdown_connections 2024-04-22T15:10:33.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.277+0000 7f9d205f1700 1 --2- 172.21.15.196:0/2302207626 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 0x7f9d180ffb90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.277+0000 7f9d205f1700 1 -- 172.21.15.196:0/2302207626 >> 172.21.15.196:0/2302207626 conn(0x7f9d180fad10 msgr2=0x7f9d180fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.277+0000 7f9d205f1700 1 -- 172.21.15.196:0/2302207626 shutdown_connections 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.277+0000 7f9d205f1700 1 -- 172.21.15.196:0/2302207626 wait complete. 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.278+0000 7f9d205f1700 1 Processor -- start 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.278+0000 7f9d205f1700 1 -- start start 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.278+0000 7f9d205f1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 0x7f9d18068ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.278+0000 7f9d205f1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d18069000 con 0x7f9d180ff770 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.278+0000 7f9d1e38d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 0x7f9d18068ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.278+0000 7f9d1e38d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 0x7f9d18068ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:59306/0 (socket says 172.21.15.196:59306) 2024-04-22T15:10:33.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.278+0000 7f9d1e38d700 1 -- 172.21.15.196:0/961083409 learned_addr learned my addr 172.21.15.196:0/961083409 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:33.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.278+0000 7f9d1e38d700 1 -- 172.21.15.196:0/961083409 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d080095f0 con 0x7f9d180ff770 2024-04-22T15:10:33.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.278+0000 7f9d1e38d700 1 --2- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 0x7f9d18068ac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d08006010 tx=0x7f9d080050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:33.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.279+0000 7f9d0f7fe700 1 -- 172.21.15.196:0/961083409 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d0801d070 con 0x7f9d180ff770 2024-04-22T15:10:33.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.279+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d181967f0 con 0x7f9d180ff770 2024-04-22T15:10:33.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.279+0000 7f9d0f7fe700 1 -- 172.21.15.196:0/961083409 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d0800be20 con 0x7f9d180ff770 2024-04-22T15:10:33.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.279+0000 7f9d0f7fe700 1 -- 172.21.15.196:0/961083409 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d08022de0 con 0x7f9d180ff770 2024-04-22T15:10:33.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.279+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1819b0b0 con 0x7f9d180ff770 2024-04-22T15:10:33.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.279+0000 7f9d0f7fe700 1 -- 172.21.15.196:0/961083409 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 14) v1 ==== 45626+0+0 (secure 0 0 0) 0x7f9d080226c0 con 0x7f9d180ff770 2024-04-22T15:10:33.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.279+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1818e5a0 con 0x7f9d180ff770 2024-04-22T15:10:33.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.280+0000 7f9d0f7fe700 1 --2- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9d04038800 0x7f9d0403acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:33.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.280+0000 7f9d0f7fe700 1 -- 172.21.15.196:0/961083409 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9d0804cb30 con 0x7f9d180ff770 2024-04-22T15:10:33.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.280+0000 7f9d1db8c700 1 -- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9d04038800 msgr2=0x7f9d0403acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/976363826 2024-04-22T15:10:33.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.280+0000 7f9d1db8c700 1 --2- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9d04038800 0x7f9d0403acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:10:33.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.285+0000 7f9d0f7fe700 1 -- 172.21.15.196:0/961083409 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9d0802ba00 con 0x7f9d180ff770 2024-04-22T15:10:33.485 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.481+0000 7f9d1db8c700 1 -- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9d04038800 msgr2=0x7f9d0403acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/976363826 2024-04-22T15:10:33.485 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.481+0000 7f9d1db8c700 1 --2- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9d04038800 0x7f9d0403acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:10:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.556+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d18062510 con 0x7f9d180ff770 2024-04-22T15:10:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.556+0000 7f9d0f7fe700 1 -- 172.21.15.196:0/961083409 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9d08027030 con 0x7f9d180ff770 2024-04-22T15:10:33.561 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:33.561 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9d04038800 msgr2=0x7f9d0403acc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:10:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 --2- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9d04038800 0x7f9d0403acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 msgr2=0x7f9d18068ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 --2- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 0x7f9d18068ac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d08006010 tx=0x7f9d080050b0 comp rx=0 tx=0).stop 2024-04-22T15:10:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 shutdown_connections 2024-04-22T15:10:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 --2- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f9d04038800 0x7f9d0403acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 --2- 172.21.15.196:0/961083409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d180ff770 0x7f9d18068ac0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 >> 172.21.15.196:0/961083409 conn(0x7f9d180fad10 msgr2=0x7f9d180fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 shutdown_connections 2024-04-22T15:10:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:33.558+0000 7f9d205f1700 1 -- 172.21.15.196:0/961083409 wait complete. 2024-04-22T15:10:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:33.710 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:33 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/961083409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:35.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:35.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:35.305 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:35.354 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:36.745 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.739+0000 7f16ca08f700 1 -- 172.21.15.196:0/3242187352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 msgr2=0x7f16c4103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:36.745 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.739+0000 7f16ca08f700 1 --2- 172.21.15.196:0/3242187352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 0x7f16c4103740 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009910 tx=0x7f16b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:36.745 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.739+0000 7f16ca08f700 1 -- 172.21.15.196:0/3242187352 shutdown_connections 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.739+0000 7f16ca08f700 1 --2- 172.21.15.196:0/3242187352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 0x7f16c4103740 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.739+0000 7f16ca08f700 1 -- 172.21.15.196:0/3242187352 >> 172.21.15.196:0/3242187352 conn(0x7f16c40fad00 msgr2=0x7f16c40fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.739+0000 7f16ca08f700 1 -- 172.21.15.196:0/3242187352 shutdown_connections 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.739+0000 7f16ca08f700 1 -- 172.21.15.196:0/3242187352 wait complete. 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.740+0000 7f16ca08f700 1 Processor -- start 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.740+0000 7f16ca08f700 1 -- start start 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.740+0000 7f16ca08f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 0x7f16c4195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.740+0000 7f16ca08f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4196210 con 0x7f16c4101350 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.740+0000 7f16c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 0x7f16c4195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.740+0000 7f16c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 0x7f16c4195cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:49054/0 (socket says 172.21.15.196:49054) 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.740+0000 7f16c37fe700 1 -- 172.21.15.196:0/313848512 learned_addr learned my addr 172.21.15.196:0/313848512 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:36.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.740+0000 7f16c37fe700 1 -- 172.21.15.196:0/313848512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b40095f0 con 0x7f16c4101350 2024-04-22T15:10:36.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.741+0000 7f16c37fe700 1 --2- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 0x7f16c4195cd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009ed0 tx=0x7f16b4004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:36.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.741+0000 7f16c0ff9700 1 -- 172.21.15.196:0/313848512 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16b401d070 con 0x7f16c4101350 2024-04-22T15:10:36.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.741+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c4196410 con 0x7f16c4101350 2024-04-22T15:10:36.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.741+0000 7f16c0ff9700 1 -- 172.21.15.196:0/313848512 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16b400be20 con 0x7f16c4101350 2024-04-22T15:10:36.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.741+0000 7f16c0ff9700 1 -- 172.21.15.196:0/313848512 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16b400f460 con 0x7f16c4101350 2024-04-22T15:10:36.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.741+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c41968b0 con 0x7f16c4101350 2024-04-22T15:10:36.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.742+0000 7f16c0ff9700 1 -- 172.21.15.196:0/313848512 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 14) v1 ==== 45626+0+0 (secure 0 0 0) 0x7f16b400b960 con 0x7f16c4101350 2024-04-22T15:10:36.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.742+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c418f690 con 0x7f16c4101350 2024-04-22T15:10:36.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.742+0000 7f16c0ff9700 1 --2- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f16ac038870 0x7f16ac03ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:36.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.742+0000 7f16c0ff9700 1 -- 172.21.15.196:0/313848512 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f16b404cd00 con 0x7f16c4101350 2024-04-22T15:10:36.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.742+0000 7f16c2ffd700 1 -- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f16ac038870 msgr2=0x7f16ac03ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/976363826 2024-04-22T15:10:36.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.742+0000 7f16c2ffd700 1 --2- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f16ac038870 0x7f16ac03ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:10:36.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.748+0000 7f16c0ff9700 1 -- 172.21.15.196:0/313848512 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f16b402bba0 con 0x7f16c4101350 2024-04-22T15:10:36.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.943+0000 7f16c2ffd700 1 -- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f16ac038870 msgr2=0x7f16ac03ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.96:6800/976363826 2024-04-22T15:10:36.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:36.943+0000 7f16c2ffd700 1 --2- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f16ac038870 0x7f16ac03ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:10:37.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.016+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16c4062510 con 0x7f16c4101350 2024-04-22T15:10:37.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.016+0000 7f16c0ff9700 1 -- 172.21.15.196:0/313848512 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f16c4062510 con 0x7f16c4101350 2024-04-22T15:10:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:37.021 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:37.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f16ac038870 msgr2=0x7f16ac03ad30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:10:37.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 --2- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f16ac038870 0x7f16ac03ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:37.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 msgr2=0x7f16c4195cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:37.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 --2- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 0x7f16c4195cd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009ed0 tx=0x7f16b4004690 comp rx=0 tx=0).stop 2024-04-22T15:10:37.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 shutdown_connections 2024-04-22T15:10:37.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 --2- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:6800/976363826,v1:172.21.15.96:6801/976363826] conn(0x7f16ac038870 0x7f16ac03ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:37.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 --2- 172.21.15.196:0/313848512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c4101350 0x7f16c4195cd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:37.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 >> 172.21.15.196:0/313848512 conn(0x7f16c40fad00 msgr2=0x7f16c40fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:37.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 shutdown_connections 2024-04-22T15:10:37.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:37.019+0000 7f16ca08f700 1 -- 172.21.15.196:0/313848512 wait complete. 2024-04-22T15:10:37.024 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:37.382 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:37 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/313848512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: Active manager daemon smithi096.hhkvum restarted 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: Activating manager daemon smithi096.hhkvum 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: mgrmap e15: smithi096.hhkvum(active, starting, since 0.0057994s) 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: Manager daemon smithi096.hhkvum is now available 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:10:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:38.488 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:10:38.488 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:10:38.488 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/trash_purge_schedule"}]: dispatch 2024-04-22T15:10:38.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:38.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:38.813 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:38.861 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:39.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:39 smithi096 ceph-mon[26688]: [22/Apr/2024:15:10:38] ENGINE Bus STARTING 2024-04-22T15:10:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:39 smithi096 ceph-mon[26688]: [22/Apr/2024:15:10:38] ENGINE Serving on http://172.21.15.96:8765 2024-04-22T15:10:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:39 smithi096 ceph-mon[26688]: [22/Apr/2024:15:10:38] ENGINE Serving on https://172.21.15.96:7150 2024-04-22T15:10:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:39 smithi096 ceph-mon[26688]: [22/Apr/2024:15:10:38] ENGINE Bus STARTED 2024-04-22T15:10:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:39 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:39 smithi096 ceph-mon[26688]: mgrmap e16: smithi096.hhkvum(active, since 1.0077s) 2024-04-22T15:10:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:39 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:40.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.412+0000 7f6b3830b700 1 -- 172.21.15.196:0/1548210127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 msgr2=0x7f6b300ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:40.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.412+0000 7f6b3830b700 1 --2- 172.21.15.196:0/1548210127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 0x7f6b300ffb80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009910 tx=0x7f6b2c009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:40.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.412+0000 7f6b3830b700 1 -- 172.21.15.196:0/1548210127 shutdown_connections 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.412+0000 7f6b3830b700 1 --2- 172.21.15.196:0/1548210127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 0x7f6b300ffb80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.412+0000 7f6b3830b700 1 -- 172.21.15.196:0/1548210127 >> 172.21.15.196:0/1548210127 conn(0x7f6b300fad00 msgr2=0x7f6b300fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.412+0000 7f6b3830b700 1 -- 172.21.15.196:0/1548210127 shutdown_connections 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.412+0000 7f6b3830b700 1 -- 172.21.15.196:0/1548210127 wait complete. 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.413+0000 7f6b3830b700 1 Processor -- start 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.413+0000 7f6b3830b700 1 -- start start 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.413+0000 7f6b3830b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 0x7f6b30195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.413+0000 7f6b3830b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b30196260 con 0x7f6b300ff760 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.413+0000 7f6b360a7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 0x7f6b30195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.413+0000 7f6b360a7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 0x7f6b30195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:49064/0 (socket says 172.21.15.196:49064) 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.413+0000 7f6b360a7700 1 -- 172.21.15.196:0/2751775007 learned_addr learned my addr 172.21.15.196:0/2751775007 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:40.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.413+0000 7f6b360a7700 1 -- 172.21.15.196:0/2751775007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c0095f0 con 0x7f6b300ff760 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.413+0000 7f6b360a7700 1 --2- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 0x7f6b30195d20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006010 tx=0x7f6b2c0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.414+0000 7f6b277fe700 1 -- 172.21.15.196:0/2751775007 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b2c01d070 con 0x7f6b300ff760 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.414+0000 7f6b277fe700 1 -- 172.21.15.196:0/2751775007 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b2c00be20 con 0x7f6b300ff760 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.414+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b30196460 con 0x7f6b300ff760 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.414+0000 7f6b277fe700 1 -- 172.21.15.196:0/2751775007 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b2c022de0 con 0x7f6b300ff760 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.414+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b30196900 con 0x7f6b300ff760 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.415+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3018f850 con 0x7f6b300ff760 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.415+0000 7f6b277fe700 1 -- 172.21.15.196:0/2751775007 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) v1 ==== 45477+0+0 (secure 0 0 0) 0x7f6b2c0226c0 con 0x7f6b300ff760 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.415+0000 7f6b277fe700 1 --2- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6b1c038640 0x7f6b1c03ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.415+0000 7f6b277fe700 1 -- 172.21.15.196:0/2751775007 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f6b2c04ca40 con 0x7f6b300ff760 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.415+0000 7f6b358a6700 1 --2- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6b1c038640 0x7f6b1c03ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.415+0000 7f6b358a6700 1 --2- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6b1c038640 0x7f6b1c03ab00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6b20006fd0 tx=0x7f6b20006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:40.502 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.420+0000 7f6b277fe700 1 -- 172.21.15.196:0/2751775007 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f6b2c02ba00 con 0x7f6b300ff760 2024-04-22T15:10:40.584 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.580+0000 7f6b277fe700 1 -- 172.21.15.196:0/2751775007 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f6b2c017d20 con 0x7f6b300ff760 2024-04-22T15:10:40.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.695+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b30105740 con 0x7f6b300ff760 2024-04-22T15:10:40.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.695+0000 7f6b277fe700 1 -- 172.21.15.196:0/2751775007 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6b2c04c5d0 con 0x7f6b300ff760 2024-04-22T15:10:40.698 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:40.699 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6b1c038640 msgr2=0x7f6b1c03ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 --2- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6b1c038640 0x7f6b1c03ab00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6b20006fd0 tx=0x7f6b20006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 msgr2=0x7f6b30195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 --2- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 0x7f6b30195d20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006010 tx=0x7f6b2c0050b0 comp rx=0 tx=0).stop 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 shutdown_connections 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 --2- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6b1c038640 0x7f6b1c03ab00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 --2- 172.21.15.196:0/2751775007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b300ff760 0x7f6b30195d20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 >> 172.21.15.196:0/2751775007 conn(0x7f6b300fad00 msgr2=0x7f6b30069100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 shutdown_connections 2024-04-22T15:10:40.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:40.698+0000 7f6b3830b700 1 -- 172.21.15.196:0/2751775007 wait complete. 2024-04-22T15:10:40.702 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:40 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:41 smithi096 ceph-mon[26688]: mgrmap e17: smithi096.hhkvum(active, since 2s) 2024-04-22T15:10:42.036 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:41 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/2751775007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:43.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:43.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:43.582 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:43.629 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:10:44.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:44 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:44.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:44 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:44.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:44 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:44.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:44 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:44.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:44 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:10:45.007 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.003+0000 7eff82f89700 1 -- 172.21.15.196:0/2142549751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 msgr2=0x7eff7c100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:45.007 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.003+0000 7eff82f89700 1 --2- 172.21.15.196:0/2142549751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 0x7eff7c100c90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7eff70009910 tx=0x7eff70009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.003+0000 7eff82f89700 1 -- 172.21.15.196:0/2142549751 shutdown_connections 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.003+0000 7eff82f89700 1 --2- 172.21.15.196:0/2142549751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 0x7eff7c100c90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.003+0000 7eff82f89700 1 -- 172.21.15.196:0/2142549751 >> 172.21.15.196:0/2142549751 conn(0x7eff7c0fbe30 msgr2=0x7eff7c0fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.003+0000 7eff82f89700 1 -- 172.21.15.196:0/2142549751 shutdown_connections 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.003+0000 7eff82f89700 1 -- 172.21.15.196:0/2142549751 wait complete. 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.004+0000 7eff82f89700 1 Processor -- start 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.004+0000 7eff82f89700 1 -- start start 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.004+0000 7eff82f89700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 0x7eff7c195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.004+0000 7eff82f89700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c196230 con 0x7eff7c100870 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.004+0000 7eff80d25700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 0x7eff7c195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.004+0000 7eff80d25700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 0x7eff7c195cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:49086/0 (socket says 172.21.15.196:49086) 2024-04-22T15:10:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.004+0000 7eff80d25700 1 -- 172.21.15.196:0/1139731158 learned_addr learned my addr 172.21.15.196:0/1139731158 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:45.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.005+0000 7eff80d25700 1 -- 172.21.15.196:0/1139731158 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff700095f0 con 0x7eff7c100870 2024-04-22T15:10:45.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.005+0000 7eff80d25700 1 --2- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 0x7eff7c195cf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff70006010 tx=0x7eff70009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:45.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.005+0000 7eff79ffb700 1 -- 172.21.15.196:0/1139731158 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff7001d070 con 0x7eff7c100870 2024-04-22T15:10:45.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.005+0000 7eff79ffb700 1 -- 172.21.15.196:0/1139731158 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff70005dc0 con 0x7eff7c100870 2024-04-22T15:10:45.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.005+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7c196430 con 0x7eff7c100870 2024-04-22T15:10:45.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.005+0000 7eff79ffb700 1 -- 172.21.15.196:0/1139731158 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff70022e90 con 0x7eff7c100870 2024-04-22T15:10:45.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.005+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7c1968d0 con 0x7eff7c100870 2024-04-22T15:10:45.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.006+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7c18f6a0 con 0x7eff7c100870 2024-04-22T15:10:45.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.006+0000 7eff79ffb700 1 -- 172.21.15.196:0/1139731158 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7eff70022470 con 0x7eff7c100870 2024-04-22T15:10:45.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.006+0000 7eff79ffb700 1 --2- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7eff6c038850 0x7eff6c03ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:45.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.006+0000 7eff79ffb700 1 -- 172.21.15.196:0/1139731158 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7eff7004caa0 con 0x7eff7c100870 2024-04-22T15:10:45.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.007+0000 7eff7bfff700 1 --2- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7eff6c038850 0x7eff6c03ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:45.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.007+0000 7eff7bfff700 1 --2- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7eff6c038850 0x7eff6c03ad10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7eff68006fd0 tx=0x7eff68006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:45.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.012+0000 7eff79ffb700 1 -- 172.21.15.196:0/1139731158 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7eff7000b960 con 0x7eff7c100870 2024-04-22T15:10:45.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.281+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff7c062510 con 0x7eff7c100870 2024-04-22T15:10:45.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.282+0000 7eff79ffb700 1 -- 172.21.15.196:0/1139731158 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7eff7c062510 con 0x7eff7c100870 2024-04-22T15:10:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:45.285 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:45.287 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7eff6c038850 msgr2=0x7eff6c03ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:45.287 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 --2- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7eff6c038850 0x7eff6c03ad10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7eff68006fd0 tx=0x7eff68006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:45.287 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 msgr2=0x7eff7c195cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:45.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 --2- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 0x7eff7c195cf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff70006010 tx=0x7eff70009f80 comp rx=0 tx=0).stop 2024-04-22T15:10:45.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 shutdown_connections 2024-04-22T15:10:45.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 --2- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7eff6c038850 0x7eff6c03ad10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:45.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 --2- 172.21.15.196:0/1139731158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff7c100870 0x7eff7c195cf0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:45.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 >> 172.21.15.196:0/1139731158 conn(0x7eff7c0fbe30 msgr2=0x7eff7c0fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:45.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 shutdown_connections 2024-04-22T15:10:45.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:45.284+0000 7eff82f89700 1 -- 172.21.15.196:0/1139731158 wait complete. 2024-04-22T15:10:45.288 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:46.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:46.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:46.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/1139731158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:46.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:46.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:46.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:10:46.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:10:46.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:10:46.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T15:10:46.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:46 smithi096 ceph-mon[26688]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-22T15:10:46.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:46.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:47.065 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:10:47.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:10:47.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:10:47.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:10:47.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:10:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:48 smithi096 ceph-mon[26688]: Deploying daemon ceph-exporter.smithi196 on smithi196 2024-04-22T15:10:48.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.592+0000 7ff82282d700 1 -- 172.21.15.196:0/3385334217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 msgr2=0x7ff81c0ff4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:48.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.592+0000 7ff82282d700 1 --2- 172.21.15.196:0/3385334217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 0x7ff81c0ff4c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009910 tx=0x7ff80c009c20 comp rx=0 tx=0).stop 2024-04-22T15:10:48.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.592+0000 7ff82282d700 1 -- 172.21.15.196:0/3385334217 shutdown_connections 2024-04-22T15:10:48.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.592+0000 7ff82282d700 1 --2- 172.21.15.196:0/3385334217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 0x7ff81c0ff4c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:48.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.592+0000 7ff82282d700 1 -- 172.21.15.196:0/3385334217 >> 172.21.15.196:0/3385334217 conn(0x7ff81c0fa970 msgr2=0x7ff81c0fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:48.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.592+0000 7ff82282d700 1 -- 172.21.15.196:0/3385334217 shutdown_connections 2024-04-22T15:10:48.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.592+0000 7ff82282d700 1 -- 172.21.15.196:0/3385334217 wait complete. 2024-04-22T15:10:48.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.593+0000 7ff82282d700 1 Processor -- start 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.593+0000 7ff82282d700 1 -- start start 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.593+0000 7ff82282d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 0x7ff81c196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.593+0000 7ff82282d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c101a50 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff81bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 0x7ff81c196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff81bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 0x7ff81c196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:51144/0 (socket says 172.21.15.196:51144) 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff81bfff700 1 -- 172.21.15.196:0/945055367 learned_addr learned my addr 172.21.15.196:0/945055367 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff81bfff700 1 -- 172.21.15.196:0/945055367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0095f0 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff81bfff700 1 --2- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 0x7ff81c196f40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff80c005490 tx=0x7ff80c005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff8197fa700 1 -- 172.21.15.196:0/945055367 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff80c01d070 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81c1974e0 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff8197fa700 1 -- 172.21.15.196:0/945055367 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff80c005700 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff8197fa700 1 -- 172.21.15.196:0/945055367 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff80c003950 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.594+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81c197960 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.595+0000 7ff8197fa700 1 -- 172.21.15.196:0/945055367 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7ff80c003c10 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.596+0000 7ff8197fa700 1 --2- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff8040388c0 0x7ff80403ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:48.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.596+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81c1005c0 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.596+0000 7ff8197fa700 1 -- 172.21.15.196:0/945055367 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff80c04c670 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.596+0000 7ff81b7fe700 1 --2- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff8040388c0 0x7ff80403ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:48.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.596+0000 7ff81b7fe700 1 --2- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff8040388c0 0x7ff80403ad80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff810006fd0 tx=0x7ff810006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:48.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.601+0000 7ff8197fa700 1 -- 172.21.15.196:0/945055367 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff80c026070 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.884+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff81c197ba0 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.885+0000 7ff8197fa700 1 -- 172.21.15.196:0/945055367 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff80c047090 con 0x7ff81c0ff0e0 2024-04-22T15:10:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:48.889 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff8040388c0 msgr2=0x7ff80403ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 --2- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff8040388c0 0x7ff80403ad80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff810006fd0 tx=0x7ff810006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 msgr2=0x7ff81c196f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 --2- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 0x7ff81c196f40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff80c005490 tx=0x7ff80c005570 comp rx=0 tx=0).stop 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 shutdown_connections 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 --2- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff8040388c0 0x7ff80403ad80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 --2- 172.21.15.196:0/945055367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81c0ff0e0 0x7ff81c196f40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 >> 172.21.15.196:0/945055367 conn(0x7ff81c0fa970 msgr2=0x7ff81c106c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 shutdown_connections 2024-04-22T15:10:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:48.888+0000 7ff82282d700 1 -- 172.21.15.196:0/945055367 wait complete. 2024-04-22T15:10:48.892 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:49.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:49 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/945055367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:51.344 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:51.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:51.558 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:10:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:52.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:52.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi196", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:10:52.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi196", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T15:10:52.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:10:52.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:52 smithi096 ceph-mon[26688]: Deploying daemon crash.smithi196 on smithi196 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.555+0000 7f5fc5453700 1 -- 172.21.15.196:0/2591416495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 msgr2=0x7f5fc0106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.555+0000 7f5fc5453700 1 --2- 172.21.15.196:0/2591416495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 0x7f5fc0106be0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000b0a0 tx=0x7f5fb000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.555+0000 7f5fc5453700 1 -- 172.21.15.196:0/2591416495 shutdown_connections 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.555+0000 7f5fc5453700 1 --2- 172.21.15.196:0/2591416495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 0x7f5fc0106be0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.555+0000 7f5fc5453700 1 -- 172.21.15.196:0/2591416495 >> 172.21.15.196:0/2591416495 conn(0x7f5fc0075290 msgr2=0x7f5fc00756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.556+0000 7f5fc5453700 1 -- 172.21.15.196:0/2591416495 shutdown_connections 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.556+0000 7f5fc5453700 1 -- 172.21.15.196:0/2591416495 wait complete. 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.556+0000 7f5fc5453700 1 Processor -- start 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.556+0000 7f5fc5453700 1 -- start start 2024-04-22T15:10:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.556+0000 7f5fc5453700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 0x7f5fc0196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:53.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.556+0000 7f5fc5453700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc01965d0 con 0x7f5fc0106800 2024-04-22T15:10:53.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.557+0000 7f5fbeffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 0x7f5fc0196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:53.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.557+0000 7f5fbeffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 0x7f5fc0196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:51168/0 (socket says 172.21.15.196:51168) 2024-04-22T15:10:53.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.557+0000 7f5fbeffd700 1 -- 172.21.15.196:0/2865473701 learned_addr learned my addr 172.21.15.196:0/2865473701 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:53.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.557+0000 7f5fbeffd700 1 -- 172.21.15.196:0/2865473701 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb0009d00 con 0x7f5fc0106800 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.557+0000 7f5fbeffd700 1 --2- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 0x7f5fc0196090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0009040 tx=0x7f5fb000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.557+0000 7f5fb7fff700 1 -- 172.21.15.196:0/2865473701 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fb000e040 con 0x7f5fc0106800 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.557+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc01967d0 con 0x7f5fc0106800 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.558+0000 7f5fb7fff700 1 -- 172.21.15.196:0/2865473701 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fb0007a60 con 0x7f5fc0106800 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.558+0000 7f5fb7fff700 1 -- 172.21.15.196:0/2865473701 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fb0012430 con 0x7f5fc0106800 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.558+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc0196c50 con 0x7f5fc0106800 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.558+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc0101f40 con 0x7f5fc0106800 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.558+0000 7f5fb7fff700 1 -- 172.21.15.196:0/2865473701 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f5fb0004590 con 0x7f5fc0106800 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.559+0000 7f5fb7fff700 1 --2- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5fac0388b0 0x7f5fac03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.559+0000 7f5fb7fff700 1 -- 172.21.15.196:0/2865473701 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f5fb004c730 con 0x7f5fc0106800 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.559+0000 7f5fbe7fc700 1 --2- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5fac0388b0 0x7f5fac03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:53.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.559+0000 7f5fbe7fc700 1 --2- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5fac0388b0 0x7f5fac03ad70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8006fd0 tx=0x7f5fa8006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:53.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.564+0000 7f5fb7fff700 1 -- 172.21.15.196:0/2865473701 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f5fb001aa00 con 0x7f5fc0106800 2024-04-22T15:10:53.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.840+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fc010abe0 con 0x7f5fc0106800 2024-04-22T15:10:53.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.840+0000 7f5fb7fff700 1 -- 172.21.15.196:0/2865473701 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f5fb001a3b0 con 0x7f5fc0106800 2024-04-22T15:10:53.844 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:53.844 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:53.846 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5fac0388b0 msgr2=0x7f5fac03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:53.846 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 --2- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5fac0388b0 0x7f5fac03ad70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8006fd0 tx=0x7f5fa8006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:53.846 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 msgr2=0x7f5fc0196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:53.846 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 --2- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 0x7f5fc0196090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0009040 tx=0x7f5fb000bd40 comp rx=0 tx=0).stop 2024-04-22T15:10:53.846 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 shutdown_connections 2024-04-22T15:10:53.846 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 --2- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5fac0388b0 0x7f5fac03ad70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:53.846 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 --2- 172.21.15.196:0/2865473701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fc0106800 0x7f5fc0196090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:53.847 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 >> 172.21.15.196:0/2865473701 conn(0x7f5fc0075290 msgr2=0x7f5fc00fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:53.847 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 shutdown_connections 2024-04-22T15:10:53.847 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:53.843+0000 7f5fc5453700 1 -- 172.21.15.196:0/2865473701 wait complete. 2024-04-22T15:10:53.847 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:10:54.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:54 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/2865473701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:55.639 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:55.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:10:55.926 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:10:56.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:56.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:56.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:56.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:56.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:56 smithi096 ceph-mon[26688]: Deploying daemon node-exporter.smithi196 on smithi196 2024-04-22T15:10:57.523 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.518+0000 7f19ac522700 1 -- 172.21.15.196:0/2226737093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 msgr2=0x7f19a4105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.518+0000 7f19ac522700 1 --2- 172.21.15.196:0/2226737093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 0x7f19a4105b40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f199c00b0a0 tx=0x7f199c00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.518+0000 7f19ac522700 1 -- 172.21.15.196:0/2226737093 shutdown_connections 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.518+0000 7f19ac522700 1 --2- 172.21.15.196:0/2226737093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 0x7f19a4105b40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.518+0000 7f19ac522700 1 -- 172.21.15.196:0/2226737093 >> 172.21.15.196:0/2226737093 conn(0x7f19a4075290 msgr2=0x7f19a40756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.518+0000 7f19ac522700 1 -- 172.21.15.196:0/2226737093 shutdown_connections 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.518+0000 7f19ac522700 1 -- 172.21.15.196:0/2226737093 wait complete. 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.519+0000 7f19ac522700 1 Processor -- start 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.519+0000 7f19ac522700 1 -- start start 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.519+0000 7f19ac522700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 0x7f19a4196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.519+0000 7f19ac522700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4069000 con 0x7f19a4105760 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.519+0000 7f19aa2be700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 0x7f19a4196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.519+0000 7f19aa2be700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 0x7f19a4196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:59730/0 (socket says 172.21.15.196:59730) 2024-04-22T15:10:57.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.519+0000 7f19aa2be700 1 -- 172.21.15.196:0/3178918624 learned_addr learned my addr 172.21.15.196:0/3178918624 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:10:57.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.520+0000 7f19aa2be700 1 -- 172.21.15.196:0/3178918624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c009d00 con 0x7f19a4105760 2024-04-22T15:10:57.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.520+0000 7f19aa2be700 1 --2- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 0x7f19a4196fe0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f199c00b830 tx=0x7f199c009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:57.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.520+0000 7f19a37fe700 1 -- 172.21.15.196:0/3178918624 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f199c00e040 con 0x7f19a4105760 2024-04-22T15:10:57.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.520+0000 7f19a37fe700 1 -- 172.21.15.196:0/3178918624 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f199c004040 con 0x7f19a4105760 2024-04-22T15:10:57.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.520+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a41975e0 con 0x7f19a4105760 2024-04-22T15:10:57.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.520+0000 7f19a37fe700 1 -- 172.21.15.196:0/3178918624 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f199c01d440 con 0x7f19a4105760 2024-04-22T15:10:57.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.520+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a4197a60 con 0x7f19a4105760 2024-04-22T15:10:57.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.521+0000 7f19a37fe700 1 -- 172.21.15.196:0/3178918624 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f199c01d720 con 0x7f19a4105760 2024-04-22T15:10:57.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.521+0000 7f19a37fe700 1 --2- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f198c0388c0 0x7f198c03ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:10:57.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.521+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a4100e30 con 0x7f19a4105760 2024-04-22T15:10:57.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.521+0000 7f19a37fe700 1 -- 172.21.15.196:0/3178918624 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f199c04c570 con 0x7f19a4105760 2024-04-22T15:10:57.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.521+0000 7f19a9abd700 1 --2- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f198c0388c0 0x7f198c03ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:10:57.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.522+0000 7f19a9abd700 1 --2- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f198c0388c0 0x7f198c03ad80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1998006fd0 tx=0x7f1998006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:10:57.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.526+0000 7f19a37fe700 1 -- 172.21.15.196:0/3178918624 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f199c012430 con 0x7f19a4105760 2024-04-22T15:10:57.815 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.812+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19a4197c50 con 0x7f19a4105760 2024-04-22T15:10:57.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.813+0000 7f19a37fe700 1 -- 172.21.15.196:0/3178918624 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f199c004650 con 0x7f19a4105760 2024-04-22T15:10:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:10:57.817 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:10:57.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f198c0388c0 msgr2=0x7f198c03ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:57.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 --2- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f198c0388c0 0x7f198c03ad80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1998006fd0 tx=0x7f1998006e40 comp rx=0 tx=0).stop 2024-04-22T15:10:57.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 msgr2=0x7f19a4196fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:10:57.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 --2- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 0x7f19a4196fe0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f199c00b830 tx=0x7f199c009760 comp rx=0 tx=0).stop 2024-04-22T15:10:57.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 shutdown_connections 2024-04-22T15:10:57.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 --2- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f198c0388c0 0x7f198c03ad80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:57.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 --2- 172.21.15.196:0/3178918624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19a4105760 0x7f19a4196fe0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:10:57.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 >> 172.21.15.196:0/3178918624 conn(0x7f19a4075290 msgr2=0x7f19a41011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:10:57.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 shutdown_connections 2024-04-22T15:10:57.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:10:57.815+0000 7f19ac522700 1 -- 172.21.15.196:0/3178918624 wait complete. 2024-04-22T15:10:57.820 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:10:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:58 smithi096 ceph-mon[26688]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:10:58.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:10:58.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:10:58 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/3178918624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:10:59.848 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:10:59.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:11:00.059 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:11:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:00 smithi096 ceph-mon[26688]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:01.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.714+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2725352075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 msgr2=0x7fc918105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:01.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.714+0000 7fc91fb9e700 1 --2- 172.21.15.196:0/2725352075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 0x7fc918105b10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc914009910 tx=0x7fc914009c20 comp rx=0 tx=0).stop 2024-04-22T15:11:01.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.714+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2725352075 shutdown_connections 2024-04-22T15:11:01.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.714+0000 7fc91fb9e700 1 --2- 172.21.15.196:0/2725352075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 0x7fc918105b10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.714+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2725352075 >> 172.21.15.196:0/2725352075 conn(0x7fc9180fb220 msgr2=0x7fc9180fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.715+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2725352075 shutdown_connections 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.715+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2725352075 wait complete. 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.715+0000 7fc91fb9e700 1 Processor -- start 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.715+0000 7fc91fb9e700 1 -- start start 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.715+0000 7fc91fb9e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 0x7fc918194d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.715+0000 7fc91fb9e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc918069ac0 con 0x7fc918105730 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.716+0000 7fc91d93a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 0x7fc918194d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.716+0000 7fc91d93a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 0x7fc918194d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:59744/0 (socket says 172.21.15.196:59744) 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.716+0000 7fc91d93a700 1 -- 172.21.15.196:0/2462740373 learned_addr learned my addr 172.21.15.196:0/2462740373 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.716+0000 7fc91d93a700 1 -- 172.21.15.196:0/2462740373 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9140095f0 con 0x7fc918105730 2024-04-22T15:11:01.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.716+0000 7fc91d93a700 1 --2- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 0x7fc918194d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc914005490 tx=0x7fc914005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:01.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.717+0000 7fc90effd700 1 -- 172.21.15.196:0/2462740373 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc91401d070 con 0x7fc918105730 2024-04-22T15:11:01.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.717+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc918195280 con 0x7fc918105730 2024-04-22T15:11:01.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.717+0000 7fc90effd700 1 -- 172.21.15.196:0/2462740373 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc914005700 con 0x7fc918105730 2024-04-22T15:11:01.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.717+0000 7fc90effd700 1 -- 172.21.15.196:0/2462740373 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9140038e0 con 0x7fc918105730 2024-04-22T15:11:01.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.717+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc918195700 con 0x7fc918105730 2024-04-22T15:11:01.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.717+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc918100e70 con 0x7fc918105730 2024-04-22T15:11:01.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.718+0000 7fc90effd700 1 -- 172.21.15.196:0/2462740373 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7fc914003ba0 con 0x7fc918105730 2024-04-22T15:11:01.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.718+0000 7fc90effd700 1 --2- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc9040388c0 0x7fc90403ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:01.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.718+0000 7fc90effd700 1 -- 172.21.15.196:0/2462740373 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc91404c5e0 con 0x7fc918105730 2024-04-22T15:11:01.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.718+0000 7fc91d139700 1 --2- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc9040388c0 0x7fc90403ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:01.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.718+0000 7fc91d139700 1 --2- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc9040388c0 0x7fc90403ad80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc908006fd0 tx=0x7fc908006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:01.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:01.723+0000 7fc90effd700 1 -- 172.21.15.196:0/2462740373 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fc91402a980 con 0x7fc918105730 2024-04-22T15:11:02.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.019+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc918069300 con 0x7fc918105730 2024-04-22T15:11:02.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.020+0000 7fc90effd700 1 -- 172.21.15.196:0/2462740373 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc914026030 con 0x7fc918105730 2024-04-22T15:11:02.023 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:11:02.023 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:11:02.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc9040388c0 msgr2=0x7fc90403ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:02.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 --2- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc9040388c0 0x7fc90403ad80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc908006fd0 tx=0x7fc908006e40 comp rx=0 tx=0).stop 2024-04-22T15:11:02.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 msgr2=0x7fc918194d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:02.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 --2- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 0x7fc918194d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc914005490 tx=0x7fc914005570 comp rx=0 tx=0).stop 2024-04-22T15:11:02.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 shutdown_connections 2024-04-22T15:11:02.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 --2- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc9040388c0 0x7fc90403ad80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:02.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 --2- 172.21.15.196:0/2462740373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc918105730 0x7fc918194d40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:02.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 >> 172.21.15.196:0/2462740373 conn(0x7fc9180fb220 msgr2=0x7fc9180fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:02.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 shutdown_connections 2024-04-22T15:11:02.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:02.022+0000 7fc91fb9e700 1 -- 172.21.15.196:0/2462740373 wait complete. 2024-04-22T15:11:02.027 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:11:02.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:02 smithi096 ceph-mon[26688]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:02.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:02 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/2462740373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:11:04.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:11:04.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:11:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:11:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T15:11:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:11:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: Deploying daemon mgr.smithi196.gdaegc on smithi196 2024-04-22T15:11:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:04 smithi096 ceph-mon[26688]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:05.043 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:11:06.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.599+0000 7fd918642700 1 -- 172.21.15.196:0/2266012575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 msgr2=0x7fd9100ff4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:06.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.599+0000 7fd918642700 1 --2- 172.21.15.196:0/2266012575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 0x7fd9100ff4b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd90800b0a0 tx=0x7fd90800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:11:06.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.600+0000 7fd918642700 1 -- 172.21.15.196:0/2266012575 shutdown_connections 2024-04-22T15:11:06.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.600+0000 7fd918642700 1 --2- 172.21.15.196:0/2266012575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 0x7fd9100ff4b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.600+0000 7fd918642700 1 -- 172.21.15.196:0/2266012575 >> 172.21.15.196:0/2266012575 conn(0x7fd9100fa940 msgr2=0x7fd9100fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.600+0000 7fd918642700 1 -- 172.21.15.196:0/2266012575 shutdown_connections 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.600+0000 7fd918642700 1 -- 172.21.15.196:0/2266012575 wait complete. 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.600+0000 7fd918642700 1 Processor -- start 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.600+0000 7fd918642700 1 -- start start 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.601+0000 7fd918642700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 0x7fd910196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.601+0000 7fd918642700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd910197530 con 0x7fd9100ff0d0 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.601+0000 7fd9163de700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 0x7fd910196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.601+0000 7fd9163de700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 0x7fd910196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.196:43842/0 (socket says 172.21.15.196:43842) 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.601+0000 7fd9163de700 1 -- 172.21.15.196:0/1163151546 learned_addr learned my addr 172.21.15.196:0/1163151546 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.601+0000 7fd9163de700 1 -- 172.21.15.196:0/1163151546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd908009d00 con 0x7fd9100ff0d0 2024-04-22T15:11:06.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.601+0000 7fd9163de700 1 --2- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 0x7fd910196ff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd9080062a0 tx=0x7fd90800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.602+0000 7fd9077fe700 1 -- 172.21.15.196:0/1163151546 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd90800e040 con 0x7fd9100ff0d0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.602+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd910197730 con 0x7fd9100ff0d0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.602+0000 7fd9077fe700 1 -- 172.21.15.196:0/1163151546 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd908007a40 con 0x7fd9100ff0d0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.602+0000 7fd9077fe700 1 -- 172.21.15.196:0/1163151546 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd908012430 con 0x7fd9100ff0d0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.602+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd910197b50 con 0x7fd9100ff0d0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.603+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9101005b0 con 0x7fd9100ff0d0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.603+0000 7fd9077fe700 1 -- 172.21.15.196:0/1163151546 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7fd908004040 con 0x7fd9100ff0d0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.603+0000 7fd9077fe700 1 --2- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd8fc0388b0 0x7fd8fc03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.603+0000 7fd9077fe700 1 -- 172.21.15.196:0/1163151546 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fd90804c670 con 0x7fd9100ff0d0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.603+0000 7fd915bdd700 1 --2- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd8fc0388b0 0x7fd8fc03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:06.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.604+0000 7fd915bdd700 1 --2- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd8fc0388b0 0x7fd8fc03ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd900006fd0 tx=0x7fd900006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:06.611 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.608+0000 7fd9077fe700 1 -- 172.21.15.196:0/1163151546 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fd908011390 con 0x7fd9100ff0d0 2024-04-22T15:11:06.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.907+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd910062510 con 0x7fd9100ff0d0 2024-04-22T15:11:06.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.908+0000 7fd9077fe700 1 -- 172.21.15.196:0/1163151546 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fd90801b360 con 0x7fd9100ff0d0 2024-04-22T15:11:06.911 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:11:06.911 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:08:12.449340Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:11:06.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd8fc0388b0 msgr2=0x7fd8fc03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:06.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 --2- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd8fc0388b0 0x7fd8fc03ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd900006fd0 tx=0x7fd900006e40 comp rx=0 tx=0).stop 2024-04-22T15:11:06.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 msgr2=0x7fd910196ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:06.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 --2- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 0x7fd910196ff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd9080062a0 tx=0x7fd90800bd40 comp rx=0 tx=0).stop 2024-04-22T15:11:06.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 shutdown_connections 2024-04-22T15:11:06.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 --2- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd8fc0388b0 0x7fd8fc03ad70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:06.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 --2- 172.21.15.196:0/1163151546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9100ff0d0 0x7fd910196ff0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:06.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 >> 172.21.15.196:0/1163151546 conn(0x7fd9100fa940 msgr2=0x7fd910100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:06.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 shutdown_connections 2024-04-22T15:11:06.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:06.910+0000 7fd918642700 1 -- 172.21.15.196:0/1163151546 wait complete. 2024-04-22T15:11:06.914 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-22T15:11:06.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:06 smithi096 ceph-mon[26688]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:07 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/1163151546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:11:07.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:07.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:07.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:07.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:07.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:11:07.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:07.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:11:07.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:08.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:11:08.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:11:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:08 smithi096 ceph-mon[26688]: Deploying daemon mon.smithi196 on smithi196 2024-04-22T15:11:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:08 smithi096 ceph-mon[26688]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:09.435 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:11:10.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:10 smithi196 systemd[1]: Starting Ceph mon.smithi196 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:11:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:10 smithi096 ceph-mon[26688]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:11.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:10 smithi196 podman[34668]: 2024-04-22 15:11:10.620754072 +0000 UTC m=+0.019757894 image pull 9900ed1a6b81aaf043a19454fc819f6cb21ac0421c7c720b31d8fae6faebb644 quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210 2024-04-22T15:11:11.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:10 smithi196 podman[34668]: 2024-04-22 15:11:10.842225159 +0000 UTC m=+0.241228970 container create 79938ec69af6bc46d2ac54dfab45ec5e32f053fd669366f310fe4033067cc879 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, RELEASE=reef-646b8c7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:11:11.486 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 podman[34668]: 2024-04-22 15:11:11.421378203 +0000 UTC m=+0.820382018 container init 79938ec69af6bc46d2ac54dfab45ec5e32f053fd669366f310fe4033067cc879 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, GIT_CLEAN=True, RELEASE=reef-646b8c7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T15:11:11.486 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 podman[34668]: 2024-04-22 15:11:11.424547628 +0000 UTC m=+0.823551440 container start 79938ec69af6bc46d2ac54dfab45ec5e32f053fd669366f310fe4033067cc879 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-646b8c7, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-22T15:11:11.486 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T15:11:11.486 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable), process ceph-mon, pid 2 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: pidfile_write: ignore empty --pid-file 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: load: jerasure load: lrc 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: RocksDB version: 7.9.2 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Git sha 0 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Compile date 2024-04-18 11:37:30 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: DB SUMMARY 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: DB Session ID: BFOL388BC45UP2O4RNL1 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: CURRENT file: CURRENT 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi196/store.db dir, Total Num: 0, files: 2024-04-22T15:11:11.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi196/store.db: 000004.log size: 511 ; 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.error_if_exists: 0 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.create_if_missing: 0 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.paranoid_checks: 1 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.env: 0x55d836765820 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.info_log: 0x55d837f293c0 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.statistics: (nil) 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.use_fsync: 0 2024-04-22T15:11:11.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_log_file_size: 0 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.allow_fallocate: 1 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.use_direct_reads: 0 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T15:11:11.489 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.db_log_dir: 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.wal_dir: 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.write_buffer_manager: 0x55d83719e140 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T15:11:11.490 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.unordered_write: 0 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.row_cache: None 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.wal_filter: None 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T15:11:11.491 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.two_write_queues: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.wal_compression: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.atomic_flush: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.log_readahead_size: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T15:11:11.492 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_background_jobs: 2 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_background_compactions: -1 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_subcompactions: 1 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T15:11:11.493 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_open_files: -1 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_background_flushes: -1 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Compression algorithms supported: 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: kZSTD supported: 0 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: kXpressCompression supported: 0 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: kZlibCompression supported: 1 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: kSnappyCompression supported: 1 2024-04-22T15:11:11.494 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: kLZ4Compression supported: 1 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: kBZip2Compression supported: 0 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi196/store.db/MANIFEST-000005 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.merge_operator: 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_filter: None 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T15:11:11.495 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d837f294e0) 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: index_type: 0 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: data_block_index_type: 0 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: index_shortening: 1 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: checksum: 4 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: no_block_cache: 0 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_cache: 0x55d83723f1f0 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_cache_options: 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: capacity : 536870912 2024-04-22T15:11:11.496 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: num_shard_bits : 4 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: strict_capacity_limit : 0 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_cache_compressed: (nil) 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: persistent_cache: (nil) 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_size: 4096 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_size_deviation: 10 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_restart_interval: 16 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: index_block_restart_interval: 1 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: metadata_block_size: 4096 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: partition_filters: 0 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: use_delta_encoding: 1 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: filter_policy: bloomfilter 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: whole_key_filtering: 1 2024-04-22T15:11:11.497 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: verify_compression: 0 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: format_version: 5 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: enable_index_compression: 1 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_align: 0 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: max_auto_readahead_size: 262144 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: prepopulate_block_cache: 0 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: initial_auto_readahead_size: 8192 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression: NoCompression 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T15:11:11.498 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.num_levels: 7 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:11:11.499 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T15:11:11.500 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T15:11:11.501 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T15:11:11.501 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T15:11:11.501 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T15:11:11.501 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T15:11:11.502 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T15:11:11.502 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T15:11:11.502 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T15:11:11.502 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T15:11:11.502 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T15:11:11.502 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T15:11:11.502 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T15:11:11.503 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.table_properties_collectors: 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.inplace_update_support: 0 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.bloom_locality: 0 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.max_successive_merges: 0 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T15:11:11.504 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.ttl: 2592000 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.enable_blob_files: false 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.min_blob_size: 0 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T15:11:11.505 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi196/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 18ff6f26-d666-4b3f-ad0a-b5ae9b444f56 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713798671453785, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713798671454346, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713798671, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "18ff6f26-d666-4b3f-ad0a-b5ae9b444f56", "db_session_id": "BFOL388BC45UP2O4RNL1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713798671454448, "job": 1, "event": "recovery_finished"} 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi196/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d8372de000 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: DB pointer 0x55d8372cc000 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T15:11:11.506 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ** DB Stats ** 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:11:11.507 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T15:11:11.508 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Block cache BinnedLRUCache@0x55d83723f1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196 does not exist in monmap, will attempt to join an existing cluster 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: using public_addr v2:172.21.15.196:0/0 -> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: starting mon.smithi196 rank -1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi196 fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:11:11.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(???) e0 preinit fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).mds e1 new map 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).mds e1 print_map 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: e1 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: legacy client fscid: -1 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: No filesystems configured 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T15:11:11.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: from='client.? 172.21.15.196:0/1163151546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:11.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:11:11.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:11.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:11:11.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:11.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: Deploying daemon mon.smithi196 on smithi196 2024-04-22T15:11:11.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:11.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:11.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 ceph-mon[34717]: mon.smithi196@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-04-22T15:11:11.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 bash[34668]: 79938ec69af6bc46d2ac54dfab45ec5e32f053fd669366f310fe4033067cc879 2024-04-22T15:11:11.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:11 smithi196 systemd[1]: Started Ceph mon.smithi196 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.530+0000 7f6c2b557700 1 -- 172.21.15.196:0/2570310851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c100035c0 msgr2=0x7f6c10005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.530+0000 7f6c2b557700 1 --2- 172.21.15.196:0/2570310851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c100035c0 0x7f6c10005a50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6c240766b0 tx=0x7f6c1400fe60 comp rx=0 tx=0).stop 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.530+0000 7f6c2b557700 1 -- 172.21.15.196:0/2570310851 shutdown_connections 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.530+0000 7f6c2b557700 1 --2- 172.21.15.196:0/2570310851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c100035c0 0x7f6c10005a50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.530+0000 7f6c2b557700 1 -- 172.21.15.196:0/2570310851 >> 172.21.15.196:0/2570310851 conn(0x7f6c24075290 msgr2=0x7f6c240756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.530+0000 7f6c2b557700 1 -- 172.21.15.196:0/2570310851 shutdown_connections 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.530+0000 7f6c2b557700 1 -- 172.21.15.196:0/2570310851 wait complete. 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.531+0000 7f6c2b557700 1 Processor -- start 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.531+0000 7f6c2b557700 1 -- start start 2024-04-22T15:11:16.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.531+0000 7f6c2b557700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 0x7f6c240fd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.531+0000 7f6c2b557700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c240fdac0 0x7f6c240fdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.531+0000 7f6c2b557700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240ff4d0 con 0x7f6c240fdac0 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.531+0000 7f6c2b557700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c241a23d0 con 0x7f6c100035c0 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c292f3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 0x7f6c240fd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c292f3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 0x7f6c240fd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:54366/0 (socket says 172.21.15.196:54366) 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c292f3700 1 -- 172.21.15.196:0/3626721400 learned_addr learned my addr 172.21.15.196:0/3626721400 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c28af2700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c240fdac0 0x7f6c240fdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c292f3700 1 -- 172.21.15.196:0/3626721400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 msgr2=0x7f6c240fd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c292f3700 1 -- 172.21.15.196:0/3626721400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 msgr2=0x7f6c240fd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c292f3700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 0x7f6c240fd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:11:16.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c292f3700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 0x7f6c240fd580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:11:16.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c28af2700 1 -- 172.21.15.196:0/3626721400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 msgr2=0x7f6c240fd580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:11:16.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c28af2700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 0x7f6c240fd580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:16.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c28af2700 1 -- 172.21.15.196:0/3626721400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1400f8b0 con 0x7f6c240fdac0 2024-04-22T15:11:16.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c28af2700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c240fdac0 0x7f6c240fdf40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000b4d0 tx=0x7f6c2000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:16.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.532+0000 7f6c1a7fc700 1 -- 172.21.15.196:0/3626721400 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c20011800 con 0x7f6c240fdac0 2024-04-22T15:11:16.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.533+0000 7f6c1a7fc700 1 -- 172.21.15.196:0/3626721400 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c20011e20 con 0x7f6c240fdac0 2024-04-22T15:11:16.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.533+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c241a25d0 con 0x7f6c240fdac0 2024-04-22T15:11:16.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.533+0000 7f6c1a7fc700 1 -- 172.21.15.196:0/3626721400 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c2000f2e0 con 0x7f6c240fdac0 2024-04-22T15:11:16.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.533+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c241a2af0 con 0x7f6c240fdac0 2024-04-22T15:11:16.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.534+0000 7f6c1a7fc700 1 -- 172.21.15.196:0/3626721400 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f6c2000f5a0 con 0x7f6c240fdac0 2024-04-22T15:11:16.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.534+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c24100e30 con 0x7f6c240fdac0 2024-04-22T15:11:16.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.534+0000 7f6c1a7fc700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6c08038a30 0x7f6c0803aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:16.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.534+0000 7f6c1a7fc700 1 -- 172.21.15.196:0/3626721400 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f6c2004b0b0 con 0x7f6c240fdac0 2024-04-22T15:11:16.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.534+0000 7f6c292f3700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6c08038a30 0x7f6c0803aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:16.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.535+0000 7f6c292f3700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6c08038a30 0x7f6c0803aef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6c14005e50 tx=0x7f6c14005de0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:16.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.539+0000 7f6c1a7fc700 1 -- 172.21.15.196:0/3626721400 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f6c20017cd0 con 0x7f6c240fdac0 2024-04-22T15:11:16.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.558+0000 7f6c1a7fc700 1 -- 172.21.15.196:0/3626721400 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f6c2001ed10 con 0x7f6c240fdac0 2024-04-22T15:11:16.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.824+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c24107670 con 0x7f6c240fdac0 2024-04-22T15:11:16.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.825+0000 7f6c1a7fc700 1 -- 172.21.15.196:0/3626721400 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1022 (secure 0 0 0) 0x7f6c20015020 con 0x7f6c240fdac0 2024-04-22T15:11:16.829 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:11:16.829 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: mon.smithi096 calling monitor election 2024-04-22T15:11:16.829 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.829 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:16.829 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.829 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: mon.smithi196 calling monitor election 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/1279170994' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: mon.smithi096 is new leader, mons smithi096,smithi196 in quorum (ranks 0,1) 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: monmap e2: 2 mons at {smithi096=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0],smithi196=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: fsmap 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: mgrmap e17: smithi096.hhkvum(active, since 38s) 2024-04-22T15:11:16.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: overall HEALTH_OK 2024-04-22T15:11:16.831 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:11:16.831 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/1279170994' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:11:16.831 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/1279170994' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:11:16.831 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:16.831 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/1279170994' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:11:16.831 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:16.831 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:16.831 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-22T15:11:16.832 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":2,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","modified":"2024-04-22T15:11:11.484887Z","created":"2024-04-22T15:08:12.449340Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi096","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi196","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T15:11:16.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6c08038a30 msgr2=0x7f6c0803aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:16.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6c08038a30 0x7f6c0803aef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6c14005e50 tx=0x7f6c14005de0 comp rx=0 tx=0).stop 2024-04-22T15:11:16.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c240fdac0 msgr2=0x7f6c240fdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:16.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c240fdac0 0x7f6c240fdf40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000b4d0 tx=0x7f6c2000b890 comp rx=0 tx=0).stop 2024-04-22T15:11:16.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 shutdown_connections 2024-04-22T15:11:16.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6c08038a30 0x7f6c0803aef0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:16.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c100035c0 0x7f6c240fd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:16.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 --2- 172.21.15.196:0/3626721400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c240fdac0 0x7f6c240fdf40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:16.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 >> 172.21.15.196:0/3626721400 conn(0x7f6c24075290 msgr2=0x7f6c241031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:16.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 shutdown_connections 2024-04-22T15:11:16.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:11:16.827+0000 7f6c2b557700 1 -- 172.21.15.196:0/3626721400 wait complete. 2024-04-22T15:11:16.833 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 2 2024-04-22T15:11:16.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: mon.smithi096 calling monitor election 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: mon.smithi196 calling monitor election 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/1279170994' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:16.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: mon.smithi096 is new leader, mons smithi096,smithi196 in quorum (ranks 0,1) 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: monmap e2: 2 mons at {smithi096=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0],smithi196=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: fsmap 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: mgrmap e17: smithi096.hhkvum(active, since 38s) 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: overall HEALTH_OK 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/1279170994' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/1279170994' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/1279170994' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:16.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:16.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:17.427 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T15:11:17.427 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T15:11:17.646 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:11:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:17 smithi096 ceph-mon[26688]: mgrmap e18: smithi096.hhkvum(active, since 38s), standbys: smithi196.gdaegc 2024-04-22T15:11:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi196.gdaegc", "id": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:11:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:11:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:17 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/3626721400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:11:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:17 smithi196 ceph-mon[34717]: mgrmap e18: smithi096.hhkvum(active, since 38s), standbys: smithi196.gdaegc 2024-04-22T15:11:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi196.gdaegc", "id": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:11:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:11:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:17 smithi196 ceph-mon[34717]: from='client.? 172.21.15.196:0/3626721400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:11:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:11:18.746 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:18 smithi096 ceph-mon[26688]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:18.746 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:18 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:18.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:18 smithi196 ceph-mon[34717]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:18 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:19.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.269+0000 7f827fa17700 1 -- 172.21.15.96:0/1498841062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82780ff0f0 msgr2=0x7f82780ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:19.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.269+0000 7f827fa17700 1 --2- 172.21.15.96:0/1498841062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82780ff0f0 0x7f82780ff4d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f827000b0a0 tx=0x7f827000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:11:19.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.269+0000 7f827fa17700 1 -- 172.21.15.96:0/1498841062 shutdown_connections 2024-04-22T15:11:19.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.269+0000 7f827fa17700 1 --2- 172.21.15.96:0/1498841062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82780ff0f0 0x7f82780ff4d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:19.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.269+0000 7f827fa17700 1 -- 172.21.15.96:0/1498841062 >> 172.21.15.96:0/1498841062 conn(0x7f82780fa9a0 msgr2=0x7f82780fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.269+0000 7f827fa17700 1 -- 172.21.15.96:0/1498841062 shutdown_connections 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.269+0000 7f827fa17700 1 -- 172.21.15.96:0/1498841062 wait complete. 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827fa17700 1 Processor -- start 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827fa17700 1 -- start start 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827fa17700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82780ff0f0 0x7f82781962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827fa17700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8278196800 0x7f827819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827fa17700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8278196e20 con 0x7f8278196800 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827fa17700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8278196f90 con 0x7f82780ff0f0 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827cfb2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8278196800 0x7f827819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827cfb2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8278196800 0x7f827819abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34136/0 (socket says 172.21.15.96:34136) 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827cfb2700 1 -- 172.21.15.96:0/2277288579 learned_addr learned my addr 172.21.15.96:0/2277288579 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:11:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.270+0000 7f827d7b3700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82780ff0f0 0x7f82781962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.271+0000 7f827cfb2700 1 -- 172.21.15.96:0/2277288579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82780ff0f0 msgr2=0x7f82781962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.271+0000 7f827cfb2700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82780ff0f0 0x7f82781962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.271+0000 7f827cfb2700 1 -- 172.21.15.96:0/2277288579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8270009d00 con 0x7f8278196800 2024-04-22T15:11:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.271+0000 7f827cfb2700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8278196800 0x7f827819abb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f826800b700 tx=0x7f826800bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.271+0000 7f826e7fc700 1 -- 172.21.15.96:0/2277288579 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8268011800 con 0x7f8278196800 2024-04-22T15:11:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.271+0000 7f826e7fc700 1 -- 172.21.15.96:0/2277288579 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8268011e20 con 0x7f8278196800 2024-04-22T15:11:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.271+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827819b1b0 con 0x7f8278196800 2024-04-22T15:11:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.271+0000 7f826e7fc700 1 -- 172.21.15.96:0/2277288579 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f826800f2e0 con 0x7f8278196800 2024-04-22T15:11:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.271+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827819b700 con 0x7f8278196800 2024-04-22T15:11:19.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.272+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827819ba30 con 0x7f8278196800 2024-04-22T15:11:19.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.273+0000 7f826e7fc700 1 -- 172.21.15.96:0/2277288579 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f8268010380 con 0x7f8278196800 2024-04-22T15:11:19.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.273+0000 7f826e7fc700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f826406cdb0 0x7f826406f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:19.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.273+0000 7f827d7b3700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f826406cdb0 0x7f826406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:19.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.273+0000 7f826e7fc700 1 -- 172.21.15.96:0/2277288579 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f826808ad70 con 0x7f8278196800 2024-04-22T15:11:19.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.273+0000 7f827d7b3700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f826406cdb0 0x7f826406f270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8270000f80 tx=0x7f827000b7f0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:19.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.277+0000 7f826e7fc700 1 -- 172.21.15.96:0/2277288579 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f8268056220 con 0x7f8278196800 2024-04-22T15:11:19.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.481+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f82781975d0 con 0x7f8278196800 2024-04-22T15:11:19.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.481+0000 7f826e7fc700 1 -- 172.21.15.96:0/2277288579 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f826808a530 con 0x7f8278196800 2024-04-22T15:11:19.485 INFO:teuthology.orchestra.run.smithi096.stdout:# minimal ceph.conf for d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:11:19.485 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-04-22T15:11:19.485 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:11:19.485 INFO:teuthology.orchestra.run.smithi096.stdout: mon_host = [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] 2024-04-22T15:11:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f826406cdb0 msgr2=0x7f826406f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f826406cdb0 0x7f826406f270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8270000f80 tx=0x7f827000b7f0 comp rx=0 tx=0).stop 2024-04-22T15:11:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8278196800 msgr2=0x7f827819abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8278196800 0x7f827819abb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f826800b700 tx=0x7f826800bac0 comp rx=0 tx=0).stop 2024-04-22T15:11:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 shutdown_connections 2024-04-22T15:11:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f826406cdb0 0x7f826406f270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82780ff0f0 0x7f82781962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 --2- 172.21.15.96:0/2277288579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8278196800 0x7f827819abb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 >> 172.21.15.96:0/2277288579 conn(0x7f82780fa9a0 msgr2=0x7f82781074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.483+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 shutdown_connections 2024-04-22T15:11:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:19.484+0000 7f827fa17700 1 -- 172.21.15.96:0/2277288579 wait complete. 2024-04-22T15:11:20.107 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2277288579' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:20.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: Reconfiguring mon.smithi096 (unknown last config time)... 2024-04-22T15:11:20.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:11:20.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:11:20.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:20.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:20 smithi096 ceph-mon[26688]: Reconfiguring daemon mon.smithi096 on smithi096 2024-04-22T15:11:20.129 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T15:11:20.129 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:11:20.129 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T15:11:20.155 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:11:20.155 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:11:20.220 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:11:20.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T15:11:20.247 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:11:20.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:11:20.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:20.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:11:20.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T15:11:20.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-22T15:11:20.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:11:20.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2277288579' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: Reconfiguring mon.smithi096 (unknown last config time)... 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:20.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:20 smithi196 ceph-mon[34717]: Reconfiguring daemon mon.smithi096 on smithi096 2024-04-22T15:11:20.284 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T15:11:20.284 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:11:20.284 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T15:11:20.299 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T15:11:20.299 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-04-22T15:11:20.354 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T15:11:20.355 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:11:20.355 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T15:11:20.355 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:11:20.355 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:11:20.355 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-22 15:10:29.040386019 +0000 2024-04-22T15:11:20.355 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-22 15:03:40.402789233 +0000 2024-04-22T15:11:20.355 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-22 15:03:40.402789233 +0000 2024-04-22T15:11:20.355 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-22 15:03:40.402789233 +0000 2024-04-22T15:11:20.355 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T15:11:20.416 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-22T15:11:20.417 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-22T15:11:20.417 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.00019581 s, 2.6 MB/s 2024-04-22T15:11:20.418 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T15:11:20.474 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-04-22T15:11:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T15:11:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:11:20.529 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T15:11:20.530 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:11:20.530 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:11:20.530 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-22 15:10:29.040386019 +0000 2024-04-22T15:11:20.530 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-22 15:03:40.788780269 +0000 2024-04-22T15:11:20.530 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-22 15:03:40.788780269 +0000 2024-04-22T15:11:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-22 15:03:40.788780269 +0000 2024-04-22T15:11:20.530 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T15:11:20.593 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-22T15:11:20.593 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-22T15:11:20.594 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000226609 s, 2.3 MB/s 2024-04-22T15:11:20.595 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T15:11:20.650 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-04-22T15:11:20.705 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T15:11:20.705 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:11:20.705 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-22T15:11:20.705 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:11:20.705 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:11:20.705 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-22 15:10:29.041385997 +0000 2024-04-22T15:11:20.705 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-22 15:03:41.198770747 +0000 2024-04-22T15:11:20.705 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-22 15:03:41.198770747 +0000 2024-04-22T15:11:20.705 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-22 15:03:41.198770747 +0000 2024-04-22T15:11:20.706 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T15:11:20.767 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-22T15:11:20.767 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-22T15:11:20.767 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000205001 s, 2.5 MB/s 2024-04-22T15:11:20.768 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T15:11:20.823 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-04-22T15:11:20.878 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T15:11:20.878 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:11:20.878 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T15:11:20.878 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:11:20.879 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:11:20.879 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-22 15:10:29.041385997 +0000 2024-04-22T15:11:20.879 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-22 15:03:41.561762316 +0000 2024-04-22T15:11:20.879 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-22 15:03:41.561762316 +0000 2024-04-22T15:11:20.879 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-22 15:03:41.561762316 +0000 2024-04-22T15:11:20.879 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T15:11:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-22T15:11:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-22T15:11:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000171545 s, 3.0 MB/s 2024-04-22T15:11:20.942 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T15:11:20.997 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:11:20.997 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T15:11:21.016 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T15:11:21.016 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-04-22T15:11:21.073 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T15:11:21.073 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:11:21.073 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-22T15:11:21.073 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:11:21.074 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:11:21.074 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-04-22 15:11:10.301045139 +0000 2024-04-22T15:11:21.074 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-04-22 15:03:40.410878167 +0000 2024-04-22T15:11:21.074 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-04-22 15:03:40.410878167 +0000 2024-04-22T15:11:21.074 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-04-22 15:03:40.410878167 +0000 2024-04-22T15:11:21.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T15:11:21.138 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-04-22T15:11:21.138 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-04-22T15:11:21.139 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000207031 s, 2.5 MB/s 2024-04-22T15:11:21.140 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T15:11:21.197 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-04-22T15:11:21.253 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T15:11:21.254 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:11:21.254 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T15:11:21.254 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:11:21.254 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:11:21.254 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-04-22 15:11:10.301045139 +0000 2024-04-22T15:11:21.254 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-04-22 15:03:40.784869707 +0000 2024-04-22T15:11:21.254 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-04-22 15:03:40.784869707 +0000 2024-04-22T15:11:21.254 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-04-22 15:03:40.784869707 +0000 2024-04-22T15:11:21.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T15:11:21.317 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-04-22T15:11:21.318 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-04-22T15:11:21.318 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000213065 s, 2.4 MB/s 2024-04-22T15:11:21.319 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T15:11:21.375 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-04-22T15:11:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T15:11:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:11:21.430 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-22T15:11:21.430 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:11:21.431 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:11:21.431 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-04-22 15:11:10.301045139 +0000 2024-04-22T15:11:21.431 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-04-22 15:03:41.180860749 +0000 2024-04-22T15:11:21.431 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-04-22 15:03:41.180860749 +0000 2024-04-22T15:11:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-04-22 15:03:41.180860749 +0000 2024-04-22T15:11:21.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T15:11:21.493 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-04-22T15:11:21.494 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-04-22T15:11:21.494 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000239939 s, 2.1 MB/s 2024-04-22T15:11:21.495 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T15:11:21.550 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-04-22T15:11:21.605 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T15:11:21.605 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:11:21.606 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-22T15:11:21.606 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:11:21.606 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:11:21.606 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-04-22 15:11:10.302045118 +0000 2024-04-22T15:11:21.606 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-04-22 15:03:41.560852153 +0000 2024-04-22T15:11:21.606 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-04-22 15:03:41.560852153 +0000 2024-04-22T15:11:21.606 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-04-22 15:03:41.560852153 +0000 2024-04-22T15:11:21.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T15:11:21.668 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-04-22T15:11:21.668 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-04-22T15:11:21.668 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000192561 s, 2.7 MB/s 2024-04-22T15:11:21.669 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T15:11:21.725 INFO:tasks.cephadm:Deploying osd.0 on smithi096 with /dev/vg_nvme/lv_4... 2024-04-22T15:11:21.725 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T15:11:22.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:22.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:22.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:22 smithi096 ceph-mon[26688]: Reconfiguring mgr.smithi096.hhkvum (unknown last config time)... 2024-04-22T15:11:22.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi096.hhkvum", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:11:22.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:11:22.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:22.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:22 smithi096 ceph-mon[26688]: Reconfiguring daemon mgr.smithi096.hhkvum on smithi096 2024-04-22T15:11:22.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:22 smithi096 ceph-mon[26688]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:22.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:22.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:22.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:22 smithi196 ceph-mon[34717]: Reconfiguring mgr.smithi096.hhkvum (unknown last config time)... 2024-04-22T15:11:22.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi096.hhkvum", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:11:22.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:11:22.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:22.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:22 smithi196 ceph-mon[34717]: Reconfiguring daemon mgr.smithi096.hhkvum on smithi096 2024-04-22T15:11:22.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:22 smithi196 ceph-mon[34717]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:22.793 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:11:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:23 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:11:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:23 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:23 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:23 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:11:23.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:23 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:23 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:11:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:23 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:23 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:23 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:11:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:23 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:24.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:24 smithi096 ceph-mon[26688]: Reconfiguring ceph-exporter.smithi096 (monmap changed)... 2024-04-22T15:11:24.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:24 smithi096 ceph-mon[26688]: Reconfiguring daemon ceph-exporter.smithi096 on smithi096 2024-04-22T15:11:24.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:24 smithi096 ceph-mon[26688]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:24 smithi196 ceph-mon[34717]: Reconfiguring ceph-exporter.smithi096 (monmap changed)... 2024-04-22T15:11:24.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:24 smithi196 ceph-mon[34717]: Reconfiguring daemon ceph-exporter.smithi096 on smithi096 2024-04-22T15:11:24.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:24 smithi196 ceph-mon[34717]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T15:11:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T15:11:25.020 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-22T15:11:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-22T15:11:25.020 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114169 s, 918 MB/s 2024-04-22T15:11:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-22T15:11:25.903 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_4 2024-04-22T15:11:26.459 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:26.460 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:26.460 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:26 smithi096 ceph-mon[26688]: Reconfiguring crash.smithi096 (monmap changed)... 2024-04-22T15:11:26.460 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi096", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:11:26.460 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:26.460 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:26 smithi096 ceph-mon[26688]: Reconfiguring daemon crash.smithi096 on smithi096 2024-04-22T15:11:26.460 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:26 smithi096 ceph-mon[26688]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:26.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:26.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:26.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:26 smithi196 ceph-mon[34717]: Reconfiguring crash.smithi096 (monmap changed)... 2024-04-22T15:11:26.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi096", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:11:26.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:26.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:26 smithi196 ceph-mon[34717]: Reconfiguring daemon crash.smithi096 on smithi096 2024-04-22T15:11:26.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:26 smithi196 ceph-mon[34717]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:27.172 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:11:28.319 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:28 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:28.319 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:28 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:28.319 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:28 smithi096 ceph-mon[26688]: Reconfiguring alertmanager.smithi096 (dependencies changed)... 2024-04-22T15:11:28.319 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:28 smithi096 ceph-mon[26688]: Reconfiguring daemon alertmanager.smithi096 on smithi096 2024-04-22T15:11:28.320 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:28 smithi096 ceph-mon[26688]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:28.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:28 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:28.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:28 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:28.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:28 smithi196 ceph-mon[34717]: Reconfiguring alertmanager.smithi096 (dependencies changed)... 2024-04-22T15:11:28.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:28 smithi196 ceph-mon[34717]: Reconfiguring daemon alertmanager.smithi096 on smithi096 2024-04-22T15:11:28.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:28 smithi196 ceph-mon[34717]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:29.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.746+0000 7fa698a4a700 1 -- 172.21.15.96:0/7164878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 msgr2=0x7fa6940fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.746+0000 7fa698a4a700 1 --2- 172.21.15.96:0/7164878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 0x7fa6940fff30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa684009910 tx=0x7fa684009c20 comp rx=0 tx=0).stop 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.747+0000 7fa698a4a700 1 -- 172.21.15.96:0/7164878 shutdown_connections 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.747+0000 7fa698a4a700 1 --2- 172.21.15.96:0/7164878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa694100d10 0x7fa694101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.747+0000 7fa698a4a700 1 --2- 172.21.15.96:0/7164878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 0x7fa6940fff30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.747+0000 7fa698a4a700 1 -- 172.21.15.96:0/7164878 >> 172.21.15.96:0/7164878 conn(0x7fa6940fb0b0 msgr2=0x7fa6940fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.747+0000 7fa698a4a700 1 -- 172.21.15.96:0/7164878 shutdown_connections 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.747+0000 7fa698a4a700 1 -- 172.21.15.96:0/7164878 wait complete. 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.747+0000 7fa698a4a700 1 Processor -- start 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.747+0000 7fa698a4a700 1 -- start start 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa698a4a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 0x7fa694069bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:29.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa698a4a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa694100d10 0x7fa69419b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa698a4a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69419ba10 con 0x7fa6940ffb10 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa698a4a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69419bb50 con 0x7fa694100d10 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa69259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 0x7fa694069bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa69259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 0x7fa694069bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60214/0 (socket says 172.21.15.96:60214) 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa69259c700 1 -- 172.21.15.96:0/2419803728 learned_addr learned my addr 172.21.15.96:0/2419803728 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa691d9b700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa694100d10 0x7fa69419b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa69259c700 1 -- 172.21.15.96:0/2419803728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa694100d10 msgr2=0x7fa69419b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa69259c700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa694100d10 0x7fa69419b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa69259c700 1 -- 172.21.15.96:0/2419803728 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6840095f0 con 0x7fa6940ffb10 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa69259c700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 0x7fa694069bd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa68400bd70 tx=0x7fa68400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa6837fe700 1 -- 172.21.15.96:0/2419803728 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa68401d070 con 0x7fa6940ffb10 2024-04-22T15:11:29.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa6837fe700 1 -- 172.21.15.96:0/2419803728 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa68400f460 con 0x7fa6940ffb10 2024-04-22T15:11:29.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69419bda0 con 0x7fa6940ffb10 2024-04-22T15:11:29.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.748+0000 7fa6837fe700 1 -- 172.21.15.96:0/2419803728 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa684021600 con 0x7fa6940ffb10 2024-04-22T15:11:29.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.749+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69419c2b0 con 0x7fa6940ffb10 2024-04-22T15:11:29.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.749+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69404ec20 con 0x7fa6940ffb10 2024-04-22T15:11:29.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.750+0000 7fa6837fe700 1 -- 172.21.15.96:0/2419803728 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7fa68400fa50 con 0x7fa6940ffb10 2024-04-22T15:11:29.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.750+0000 7fa6837fe700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa67c06cc50 0x7fa67c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:11:29.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.750+0000 7fa6837fe700 1 -- 172.21.15.96:0/2419803728 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa68408c450 con 0x7fa6940ffb10 2024-04-22T15:11:29.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.750+0000 7fa691d9b700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa67c06cc50 0x7fa67c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:11:29.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.751+0000 7fa691d9b700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa67c06cc50 0x7fa67c06f110 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa688006fd0 tx=0x7fa688008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:11:29.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.755+0000 7fa6837fe700 1 -- 172.21.15.96:0/2419803728 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa69404ec20 con 0x7fa6940ffb10 2024-04-22T15:11:29.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:29.964+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa69404ebb0 con 0x7fa67c06cc50 2024-04-22T15:11:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:30 smithi096 ceph-mon[26688]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:30 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:11:30.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:30 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:11:30.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:30 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:30 smithi196 ceph-mon[34717]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:30 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:11:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:30 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:11:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:30 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:31.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:31 smithi096 ceph-mon[26688]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:11:32.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:31 smithi196 ceph-mon[34717]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:11:32.802 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:32 smithi096 ceph-mon[26688]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:32 smithi196 ceph-mon[34717]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:34.931 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:34 smithi096 ceph-mon[26688]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:34.931 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:34.931 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:34 smithi196 ceph-mon[34717]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:35.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:35 smithi096 ceph-mon[26688]: Reconfiguring grafana.smithi096 (dependencies changed)... 2024-04-22T15:11:35.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:35 smithi096 ceph-mon[26688]: Reconfiguring daemon grafana.smithi096 on smithi096 2024-04-22T15:11:35.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:35 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2652853048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f1b7772-06b4-467f-b14c-698a0de7357f"}]: dispatch 2024-04-22T15:11:35.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:35 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2652853048' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f1b7772-06b4-467f-b14c-698a0de7357f"}]': finished 2024-04-22T15:11:35.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:35 smithi096 ceph-mon[26688]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T15:11:35.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:35 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:11:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:35 smithi196 ceph-mon[34717]: Reconfiguring grafana.smithi096 (dependencies changed)... 2024-04-22T15:11:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:35 smithi196 ceph-mon[34717]: Reconfiguring daemon grafana.smithi096 on smithi096 2024-04-22T15:11:36.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:35 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2652853048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f1b7772-06b4-467f-b14c-698a0de7357f"}]: dispatch 2024-04-22T15:11:36.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:35 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2652853048' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f1b7772-06b4-467f-b14c-698a0de7357f"}]': finished 2024-04-22T15:11:36.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:35 smithi196 ceph-mon[34717]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T15:11:36.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:35 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:11:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:36 smithi096 ceph-mon[26688]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:36 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3013047585' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:11:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:36 smithi196 ceph-mon[34717]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:36 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3013047585' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:11:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:11:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:11:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:38 smithi096 ceph-mon[26688]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:38 smithi196 ceph-mon[34717]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:40 smithi096 ceph-mon[26688]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:40 smithi196 ceph-mon[34717]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:42.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:42 smithi096 ceph-mon[26688]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:43.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:42 smithi196 ceph-mon[34717]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:44.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:44 smithi096 ceph-mon[26688]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:44 smithi196 ceph-mon[34717]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:46.117 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:46.117 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:46.117 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:46 smithi096 ceph-mon[26688]: Reconfiguring prometheus.smithi096 (dependencies changed)... 2024-04-22T15:11:46.117 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:46 smithi096 ceph-mon[26688]: Reconfiguring daemon prometheus.smithi096 on smithi096 2024-04-22T15:11:46.117 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:46 smithi096 ceph-mon[26688]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:46.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:46 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:46.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:46 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:46.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:46 smithi196 ceph-mon[34717]: Reconfiguring prometheus.smithi096 (dependencies changed)... 2024-04-22T15:11:46.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:46 smithi196 ceph-mon[34717]: Reconfiguring daemon prometheus.smithi096 on smithi096 2024-04-22T15:11:46.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:46 smithi196 ceph-mon[34717]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:47.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:11:47.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:47.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:47 smithi096 ceph-mon[26688]: Deploying daemon osd.0 on smithi096 2024-04-22T15:11:47.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:47 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:11:47.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:47 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:47.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:47 smithi196 ceph-mon[34717]: Deploying daemon osd.0 on smithi096 2024-04-22T15:11:48.376 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:48 smithi096 ceph-mon[26688]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:48.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:48 smithi196 ceph-mon[34717]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:50.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:50 smithi096 ceph-mon[26688]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:50 smithi196 ceph-mon[34717]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:51.885 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:51 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:51.885 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:51 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:51.885 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:51 smithi096 ceph-mon[26688]: Reconfiguring ceph-exporter.smithi196 (monmap changed)... 2024-04-22T15:11:51.885 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:51 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:11:51.885 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:51 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:51.885 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:51 smithi096 ceph-mon[26688]: Reconfiguring daemon ceph-exporter.smithi196 on smithi196 2024-04-22T15:11:52.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:51 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:52.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:51 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:52.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:51 smithi196 ceph-mon[34717]: Reconfiguring ceph-exporter.smithi196 (monmap changed)... 2024-04-22T15:11:52.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:51 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:11:52.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:51 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:52.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:51 smithi196 ceph-mon[34717]: Reconfiguring daemon ceph-exporter.smithi196 on smithi196 2024-04-22T15:11:52.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:52 smithi096 ceph-mon[26688]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:52.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:52.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:52.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi196", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:11:52.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:52.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:11:53.084 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:52 smithi196 ceph-mon[34717]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:53.085 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:53.085 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:53.085 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi196", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:11:53.085 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:53.085 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:11:54.026 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:53 smithi096 ceph-mon[26688]: Reconfiguring crash.smithi196 (monmap changed)... 2024-04-22T15:11:54.026 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:53 smithi096 ceph-mon[26688]: Reconfiguring daemon crash.smithi196 on smithi196 2024-04-22T15:11:54.026 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:54.026 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:54.205 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:53 smithi196 ceph-mon[34717]: Reconfiguring crash.smithi196 (monmap changed)... 2024-04-22T15:11:54.205 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:53 smithi196 ceph-mon[34717]: Reconfiguring daemon crash.smithi196 on smithi196 2024-04-22T15:11:54.205 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:54.205 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:54.936 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:54 smithi096 ceph-mon[26688]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:54.936 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:54 smithi096 ceph-mon[26688]: Reconfiguring mgr.smithi196.gdaegc (monmap changed)... 2024-04-22T15:11:54.937 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:11:54.937 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:11:54.937 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:54.937 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:54 smithi096 ceph-mon[26688]: Reconfiguring daemon mgr.smithi196.gdaegc on smithi196 2024-04-22T15:11:55.043 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:54 smithi196 ceph-mon[34717]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:55.043 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:54 smithi196 ceph-mon[34717]: Reconfiguring mgr.smithi196.gdaegc (monmap changed)... 2024-04-22T15:11:55.044 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:11:55.044 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:11:55.044 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:55.044 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:54 smithi196 ceph-mon[34717]: Reconfiguring daemon mgr.smithi196.gdaegc on smithi196 2024-04-22T15:11:56.249 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:56.249 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:56.249 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: Reconfiguring mon.smithi196 (monmap changed)... 2024-04-22T15:11:56.250 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:11:56.250 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:11:56.250 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:56.250 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: Reconfiguring daemon mon.smithi196 on smithi196 2024-04-22T15:11:56.250 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:56.250 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:56.250 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:56.431 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:56.431 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:56.431 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: Reconfiguring mon.smithi196 (monmap changed)... 2024-04-22T15:11:56.431 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:11:56.431 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:11:56.431 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:11:56.431 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: Reconfiguring daemon mon.smithi196 on smithi196 2024-04-22T15:11:56.431 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:56.432 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:56.432 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.691 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.691 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.691 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:11:57.691 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi096.front.sepia.ceph.com:9093"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi096.front.sepia.ceph.com:9093"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi096.front.sepia.ceph.com:3000"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi096.front.sepia.ceph.com:3000"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi096.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T15:11:57.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi096.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T15:11:57.693 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.693 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:57 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:11:57.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:11:57.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:11:57.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi096.front.sepia.ceph.com:9093"}]: dispatch 2024-04-22T15:11:57.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi096.front.sepia.ceph.com:9093"}]: dispatch 2024-04-22T15:11:57.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:11:57.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:11:57.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi096.front.sepia.ceph.com:3000"}]: dispatch 2024-04-22T15:11:57.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi096.front.sepia.ceph.com:3000"}]: dispatch 2024-04-22T15:11:57.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:11:57.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:11:57.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi096.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T15:11:57.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi096.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T15:11:57.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:57.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:57 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:11:58.843 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:58 smithi096 ceph-mon[26688]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:58.843 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:58.843 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:11:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:58 smithi196 ceph-mon[34717]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:11:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:11:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:11:59.154 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 0 on host 'smithi096' 2024-04-22T15:11:59.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.150+0000 7fa6837fe700 1 -- 172.21.15.96:0/2419803728 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa69404ebb0 con 0x7fa67c06cc50 2024-04-22T15:11:59.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.152+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa67c06cc50 msgr2=0x7fa67c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:59.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.152+0000 7fa698a4a700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa67c06cc50 0x7fa67c06f110 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa688006fd0 tx=0x7fa688008040 comp rx=0 tx=0).stop 2024-04-22T15:11:59.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.152+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 msgr2=0x7fa694069bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:11:59.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.152+0000 7fa698a4a700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 0x7fa694069bd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa68400bd70 tx=0x7fa68400be50 comp rx=0 tx=0).stop 2024-04-22T15:11:59.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.152+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 shutdown_connections 2024-04-22T15:11:59.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.152+0000 7fa698a4a700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa67c06cc50 0x7fa67c06f110 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:59.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.152+0000 7fa698a4a700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6940ffb10 0x7fa694069bd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:59.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.152+0000 7fa698a4a700 1 --2- 172.21.15.96:0/2419803728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa694100d10 0x7fa69419b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:11:59.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.152+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 >> 172.21.15.96:0/2419803728 conn(0x7fa6940fb0b0 msgr2=0x7fa694104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:11:59.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.153+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 shutdown_connections 2024-04-22T15:11:59.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:11:59.153+0000 7fa698a4a700 1 -- 172.21.15.96:0/2419803728 wait complete. 2024-04-22T15:11:59.795 DEBUG:teuthology.orchestra.run.smithi096:osd.0> sudo journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T15:11:59.797 INFO:tasks.cephadm:Deploying osd.1 on smithi096 with /dev/vg_nvme/lv_3... 2024-04-22T15:11:59.797 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T15:12:00.401 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:00 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:00.401 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:00 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:00.401 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:00 smithi096 ceph-mon[26688]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:12:00.401 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:00 smithi096 ceph-mon[26688]: from='osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T15:12:00.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:00 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:00.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:00 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:00.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:00 smithi196 ceph-mon[34717]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:12:00.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:00 smithi196 ceph-mon[34717]: from='osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T15:12:00.999 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:12:01.234 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:01 smithi096 ceph-mon[26688]: from='osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T15:12:01.234 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:01 smithi096 ceph-mon[26688]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T15:12:01.234 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:01 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:12:01.234 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:01 smithi096 ceph-mon[26688]: from='osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:12:01.234 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:12:01 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[45698]: 2024-04-22T15:12:01.137+0000 7f414ea32700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T15:12:01.234 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:12:01 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[45698]: 2024-04-22T15:12:01.140+0000 7f414b028700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:12:01.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:01 smithi196 ceph-mon[34717]: from='osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T15:12:01.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:01 smithi196 ceph-mon[34717]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T15:12:01.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:01 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:12:01.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:01 smithi196 ceph-mon[34717]: from='osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:12:02.387 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:02 smithi096 ceph-mon[26688]: from='osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T15:12:02.387 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:02 smithi096 ceph-mon[26688]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T15:12:02.387 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:02 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:12:02.387 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:02 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:12:02.388 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:02 smithi096 ceph-mon[26688]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:12:02.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:02 smithi196 ceph-mon[34717]: from='osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T15:12:02.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:02 smithi196 ceph-mon[34717]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T15:12:02.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:02 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:12:02.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:02 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:12:02.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:02 smithi196 ceph-mon[34717]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:12:03.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:03 smithi096 ceph-mon[26688]: purged_snaps scrub starts 2024-04-22T15:12:03.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:03 smithi096 ceph-mon[26688]: purged_snaps scrub ok 2024-04-22T15:12:03.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:03 smithi096 ceph-mon[26688]: osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] boot 2024-04-22T15:12:03.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:03 smithi096 ceph-mon[26688]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T15:12:03.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:12:03.505 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T15:12:03.505 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T15:12:03.506 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-22T15:12:03.506 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-22T15:12:03.506 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013503 s, 777 MB/s 2024-04-22T15:12:03.506 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-22T15:12:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:03 smithi196 ceph-mon[34717]: purged_snaps scrub starts 2024-04-22T15:12:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:03 smithi196 ceph-mon[34717]: purged_snaps scrub ok 2024-04-22T15:12:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:03 smithi196 ceph-mon[34717]: osd.0 [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] boot 2024-04-22T15:12:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:03 smithi196 ceph-mon[34717]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T15:12:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:12:04.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:04 smithi096 ceph-mon[26688]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T15:12:04.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:04 smithi096 ceph-mon[26688]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:04.411 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:04 smithi196 ceph-mon[34717]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T15:12:04.413 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:04 smithi196 ceph-mon[34717]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:04.827 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_3 2024-04-22T15:12:05.055 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:12:05.678 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:05 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:05.679 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:05 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:05 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:05 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:06.602 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:06 smithi196 ceph-mon[34717]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:06.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 -- 172.21.15.96:0/3074929492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 msgr2=0x7f3e80106480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:12:06.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 --2- 172.21.15.96:0/3074929492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 0x7f3e80106480 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009910 tx=0x7f3e6c009c20 comp rx=0 tx=0).stop 2024-04-22T15:12:06.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 -- 172.21.15.96:0/3074929492 shutdown_connections 2024-04-22T15:12:06.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 --2- 172.21.15.96:0/3074929492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 0x7f3e80106480 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:06.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 --2- 172.21.15.96:0/3074929492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e80101760 0x7f3e80103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:06.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 -- 172.21.15.96:0/3074929492 >> 172.21.15.96:0/3074929492 conn(0x7f3e800fb0b0 msgr2=0x7f3e800fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:12:06.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 -- 172.21.15.96:0/3074929492 shutdown_connections 2024-04-22T15:12:06.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 -- 172.21.15.96:0/3074929492 wait complete. 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 Processor -- start 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.613+0000 7f3e86f6e700 1 -- start start 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e86f6e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e80101760 0x7f3e801965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e86f6e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 0x7f3e80196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e86f6e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e80197110 con 0x7f3e80104090 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e86f6e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e80197250 con 0x7f3e80101760 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e7ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 0x7f3e80196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e7ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 0x7f3e80196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47232/0 (socket says 172.21.15.96:47232) 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e7ffff700 1 -- 172.21.15.96:0/2090825790 learned_addr learned my addr 172.21.15.96:0/2090825790 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:12:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e84d0a700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e80101760 0x7f3e801965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:12:06.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e7ffff700 1 -- 172.21.15.96:0/2090825790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e80101760 msgr2=0x7f3e801965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:12:06.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e7ffff700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e80101760 0x7f3e801965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:06.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e7ffff700 1 -- 172.21.15.96:0/2090825790 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c0095f0 con 0x7f3e80104090 2024-04-22T15:12:06.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e7ffff700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 0x7f3e80196af0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c006010 tx=0x7f3e6c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:12:06.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e7dffb700 1 -- 172.21.15.96:0/2090825790 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e6c01d070 con 0x7f3e80104090 2024-04-22T15:12:06.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.614+0000 7f3e7dffb700 1 -- 172.21.15.96:0/2090825790 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e6c00f460 con 0x7f3e80104090 2024-04-22T15:12:06.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.615+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8019bc50 con 0x7f3e80104090 2024-04-22T15:12:06.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.615+0000 7f3e7dffb700 1 -- 172.21.15.96:0/2090825790 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e6c021640 con 0x7f3e80104090 2024-04-22T15:12:06.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.615+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8019c140 con 0x7f3e80104090 2024-04-22T15:12:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.616+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8004ebb0 con 0x7f3e80104090 2024-04-22T15:12:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.616+0000 7f3e7dffb700 1 -- 172.21.15.96:0/2090825790 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f3e6c00fab0 con 0x7f3e80104090 2024-04-22T15:12:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.617+0000 7f3e7dffb700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3e7006cc60 0x7f3e7006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:12:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.617+0000 7f3e7dffb700 1 -- 172.21.15.96:0/2090825790 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f3e6c08c9b0 con 0x7f3e80104090 2024-04-22T15:12:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.617+0000 7f3e84d0a700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3e7006cc60 0x7f3e7006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:12:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.618+0000 7f3e84d0a700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3e7006cc60 0x7f3e7006f120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400bbd0 tx=0x7f3e7400b5b0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:12:06.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.621+0000 7f3e7dffb700 1 -- 172.21.15.96:0/2090825790 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f3e6c05b7b0 con 0x7f3e80104090 2024-04-22T15:12:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:06 smithi096 ceph-mon[26688]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:06.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:06.834+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e80104b90 con 0x7f3e7006cc60 2024-04-22T15:12:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:12:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:12:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:07 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:12:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:07 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:12:07.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:07 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:08.606 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:08 smithi096 ceph-mon[26688]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:12:08.606 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:08 smithi096 ceph-mon[26688]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:08.607 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:12:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:08 smithi196 ceph-mon[34717]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:12:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:08 smithi196 ceph-mon[34717]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:12:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: Detected new or changed devices on smithi096 2024-04-22T15:12:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:12:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: Adjusting osd_memory_target on smithi096 to 11840M 2024-04-22T15:12:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:12:10.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:10.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:10.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:10 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:12:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: Detected new or changed devices on smithi096 2024-04-22T15:12:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:12:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: Adjusting osd_memory_target on smithi096 to 11840M 2024-04-22T15:12:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:12:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:10 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:12:11.909 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:11 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3725230097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59260a36-a2ed-42f9-8f3a-209aae4c4238"}]: dispatch 2024-04-22T15:12:11.909 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:11 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3725230097' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59260a36-a2ed-42f9-8f3a-209aae4c4238"}]': finished 2024-04-22T15:12:11.909 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:11 smithi096 ceph-mon[26688]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T15:12:11.909 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:11 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:12.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:11 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3725230097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59260a36-a2ed-42f9-8f3a-209aae4c4238"}]: dispatch 2024-04-22T15:12:12.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:11 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3725230097' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59260a36-a2ed-42f9-8f3a-209aae4c4238"}]': finished 2024-04-22T15:12:12.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:11 smithi196 ceph-mon[34717]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T15:12:12.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:12 smithi096 ceph-mon[26688]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:12.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:12 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:12.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:12 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:12.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:12 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3383943692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:12:13.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:12 smithi196 ceph-mon[34717]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:13.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:12 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:13.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:12 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:13.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:12 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3383943692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:12:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:14 smithi096 ceph-mon[26688]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:14 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:14.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:14 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:14.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:14 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:14.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:14 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:12:14.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:14 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:14 smithi196 ceph-mon[34717]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:14 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:14 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:15.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:14 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:15.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:14 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:12:15.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:14 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:16.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:16 smithi096 ceph-mon[26688]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:16 smithi196 ceph-mon[34717]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:18.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:18 smithi096 ceph-mon[26688]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:18.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:18 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:12:18.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:18 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:18 smithi196 ceph-mon[34717]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:19.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:18 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:12:19.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:18 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:19.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:19 smithi096 ceph-mon[26688]: Deploying daemon osd.1 on smithi096 2024-04-22T15:12:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:19 smithi196 ceph-mon[34717]: Deploying daemon osd.1 on smithi096 2024-04-22T15:12:20.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:20 smithi096 ceph-mon[26688]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:20 smithi196 ceph-mon[34717]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:22 smithi096 ceph-mon[26688]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:12:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:22 smithi196 ceph-mon[34717]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:12:25.064 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:24 smithi196 ceph-mon[34717]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:25.064 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:24 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:25.064 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:24 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:25.064 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:24 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:12:25.146 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:24 smithi096 ceph-mon[26688]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:25.146 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:24 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:25.146 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:24 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:25.146 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:24 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:12:26.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:26.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:26.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:26 smithi096 ceph-mon[26688]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:27.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:27.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:26 smithi196 ceph-mon[34717]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:27.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.640+0000 7f3e7dffb700 1 -- 172.21.15.96:0/2090825790 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3e80104b90 con 0x7f3e7006cc60 2024-04-22T15:12:27.644 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 1 on host 'smithi096' 2024-04-22T15:12:27.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.642+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3e7006cc60 msgr2=0x7f3e7006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:12:27.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.642+0000 7f3e86f6e700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3e7006cc60 0x7f3e7006f120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400bbd0 tx=0x7f3e7400b5b0 comp rx=0 tx=0).stop 2024-04-22T15:12:27.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.642+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 msgr2=0x7f3e80196af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:12:27.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.642+0000 7f3e86f6e700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 0x7f3e80196af0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c006010 tx=0x7f3e6c00be20 comp rx=0 tx=0).stop 2024-04-22T15:12:27.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.642+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 shutdown_connections 2024-04-22T15:12:27.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.642+0000 7f3e86f6e700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3e7006cc60 0x7f3e7006f120 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:27.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.642+0000 7f3e86f6e700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e80101760 0x7f3e801965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:27.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.642+0000 7f3e86f6e700 1 --2- 172.21.15.96:0/2090825790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e80104090 0x7f3e80196af0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:27.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.643+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 >> 172.21.15.96:0/2090825790 conn(0x7f3e800fb0b0 msgr2=0x7f3e800fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:12:27.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.643+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 shutdown_connections 2024-04-22T15:12:27.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:27.643+0000 7f3e86f6e700 1 -- 172.21.15.96:0/2090825790 wait complete. 2024-04-22T15:12:28.287 DEBUG:teuthology.orchestra.run.smithi096:osd.1> sudo journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T15:12:28.289 INFO:tasks.cephadm:Deploying osd.2 on smithi096 with /dev/vg_nvme/lv_2... 2024-04-22T15:12:28.289 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T15:12:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:28 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:28 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:28 smithi096 ceph-mon[26688]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:28 smithi096 ceph-mon[26688]: from='osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T15:12:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:28 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:29.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:28 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:29.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:28 smithi196 ceph-mon[34717]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:29.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:28 smithi196 ceph-mon[34717]: from='osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T15:12:29.700 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:12:29.940 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:29 smithi096 ceph-mon[26688]: from='osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T15:12:29.940 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:29 smithi096 ceph-mon[26688]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T15:12:29.940 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:29 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:29.940 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:29 smithi096 ceph-mon[26688]: from='osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:12:29.940 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:12:29 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[53642]: 2024-04-22T15:12:29.634+0000 7fd164820700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T15:12:29.941 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:12:29 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[53642]: 2024-04-22T15:12:29.638+0000 7fd160e16700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:12:30.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:29 smithi196 ceph-mon[34717]: from='osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T15:12:30.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:29 smithi196 ceph-mon[34717]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T15:12:30.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:29 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:30.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:29 smithi196 ceph-mon[34717]: from='osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:12:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:30 smithi096 ceph-mon[26688]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:30 smithi096 ceph-mon[26688]: from='osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T15:12:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:30 smithi096 ceph-mon[26688]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T15:12:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:30 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:30 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:30 smithi196 ceph-mon[34717]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:30 smithi196 ceph-mon[34717]: from='osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T15:12:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:30 smithi196 ceph-mon[34717]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T15:12:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:30 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:30 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:31.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:31 smithi096 ceph-mon[26688]: purged_snaps scrub starts 2024-04-22T15:12:31.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:31 smithi096 ceph-mon[26688]: purged_snaps scrub ok 2024-04-22T15:12:31.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:31 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:31.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:31 smithi096 ceph-mon[26688]: osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] boot 2024-04-22T15:12:31.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:31 smithi096 ceph-mon[26688]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T15:12:31.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:31 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:31.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:31 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:31.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:31 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:32.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:31 smithi196 ceph-mon[34717]: purged_snaps scrub starts 2024-04-22T15:12:32.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:31 smithi196 ceph-mon[34717]: purged_snaps scrub ok 2024-04-22T15:12:32.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:31 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:32.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:31 smithi196 ceph-mon[34717]: osd.1 [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] boot 2024-04-22T15:12:32.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:31 smithi196 ceph-mon[34717]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T15:12:32.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:31 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:12:32.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:31 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:32.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:31 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:32.354 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T15:12:32.354 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T15:12:32.354 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-22T15:12:32.355 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-22T15:12:32.355 INFO:teuthology.orchestra.run.smithi096.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150089 s, 699 MB/s 2024-04-22T15:12:32.355 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 2024-04-22T15:12:32.355 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-22T15:12:32.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:32 smithi096 ceph-mon[26688]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:32.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:32 smithi096 ceph-mon[26688]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T15:12:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:32 smithi196 ceph-mon[34717]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:12:33.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:32 smithi196 ceph-mon[34717]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T15:12:33.219 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_2 2024-04-22T15:12:34.252 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:12:34.891 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:34 smithi096 ceph-mon[26688]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:34 smithi196 ceph-mon[34717]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.416+0000 7f487fe82700 1 -- 172.21.15.96:0/387707668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48780feed0 msgr2=0x7f48780ff2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.416+0000 7f487fe82700 1 --2- 172.21.15.96:0/387707668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48780feed0 0x7f48780ff2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4868009910 tx=0x7f4868009c20 comp rx=0 tx=0).stop 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.416+0000 7f487fe82700 1 -- 172.21.15.96:0/387707668 shutdown_connections 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.416+0000 7f487fe82700 1 --2- 172.21.15.96:0/387707668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4878101eb0 0x7f4878102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.416+0000 7f487fe82700 1 --2- 172.21.15.96:0/387707668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48780feed0 0x7f48780ff2f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.416+0000 7f487fe82700 1 -- 172.21.15.96:0/387707668 >> 172.21.15.96:0/387707668 conn(0x7f48780fa830 msgr2=0x7f48780fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.416+0000 7f487fe82700 1 -- 172.21.15.96:0/387707668 shutdown_connections 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.416+0000 7f487fe82700 1 -- 172.21.15.96:0/387707668 wait complete. 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487fe82700 1 Processor -- start 2024-04-22T15:12:36.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487fe82700 1 -- start start 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487fe82700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48780feed0 0x7f4878101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487fe82700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4878101eb0 0x7f48780ffe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487fe82700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48781003a0 con 0x7f4878101eb0 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487fe82700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48781004e0 con 0x7f48780feed0 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487d41d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4878101eb0 0x7f48780ffe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487d41d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4878101eb0 0x7f48780ffe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52408/0 (socket says 172.21.15.96:52408) 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487d41d700 1 -- 172.21.15.96:0/3475913531 learned_addr learned my addr 172.21.15.96:0/3475913531 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487dc1e700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48780feed0 0x7f4878101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487d41d700 1 -- 172.21.15.96:0/3475913531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48780feed0 msgr2=0x7f4878101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487d41d700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48780feed0 0x7f4878101770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.417+0000 7f487d41d700 1 -- 172.21.15.96:0/3475913531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680095f0 con 0x7f4878101eb0 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.418+0000 7f487d41d700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4878101eb0 0x7f48780ffe60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f487400d8d0 tx=0x7f487400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:12:36.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.418+0000 7f486effd700 1 -- 172.21.15.96:0/3475913531 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48740098a0 con 0x7f4878101eb0 2024-04-22T15:12:36.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.418+0000 7f486effd700 1 -- 172.21.15.96:0/3475913531 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4874010460 con 0x7f4878101eb0 2024-04-22T15:12:36.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.418+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48781007c0 con 0x7f4878101eb0 2024-04-22T15:12:36.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.418+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487819e6f0 con 0x7f4878101eb0 2024-04-22T15:12:36.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.418+0000 7f486effd700 1 -- 172.21.15.96:0/3475913531 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487400f5f0 con 0x7f4878101eb0 2024-04-22T15:12:36.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.419+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4878190790 con 0x7f4878101eb0 2024-04-22T15:12:36.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.420+0000 7f486effd700 1 -- 172.21.15.96:0/3475913531 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f487400f750 con 0x7f4878101eb0 2024-04-22T15:12:36.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.420+0000 7f486effd700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f486406cc60 0x7f486406f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:12:36.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.420+0000 7f486effd700 1 -- 172.21.15.96:0/3475913531 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f487408b380 con 0x7f4878101eb0 2024-04-22T15:12:36.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.420+0000 7f487dc1e700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f486406cc60 0x7f486406f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:12:36.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.421+0000 7f487dc1e700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f486406cc60 0x7f486406f120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4868000c00 tx=0x7f4868005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:12:36.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.425+0000 7f486effd700 1 -- 172.21.15.96:0/3475913531 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4874059fa0 con 0x7f4878101eb0 2024-04-22T15:12:36.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:36.641+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f487802ce00 con 0x7f486406cc60 2024-04-22T15:12:36.728 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: Detected new or changed devices on smithi096 2024-04-22T15:12:36.728 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:36.728 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:36.728 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:12:36.728 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:12:36.728 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: Adjusting osd_memory_target on smithi096 to 5920M 2024-04-22T15:12:36.729 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:36.729 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:36.729 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:12:36.729 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:36.729 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:36.729 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:36 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:12:36.891 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: Detected new or changed devices on smithi096 2024-04-22T15:12:36.891 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:36.891 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:36.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:12:36.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:12:36.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: Adjusting osd_memory_target on smithi096 to 5920M 2024-04-22T15:12:36.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:36.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:36.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:12:36.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:36.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:36.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:36 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:12:37.625 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:37 smithi096 ceph-mon[26688]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:12:37.625 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:12:37.625 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:12:37.625 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:37 smithi196 ceph-mon[34717]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:12:37.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:12:37.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:12:37.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:38.636 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:38 smithi096 ceph-mon[26688]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:38.636 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:12:38.636 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:38.637 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:39.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:38 smithi196 ceph-mon[34717]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:39.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:12:39.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:39.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:40 smithi096 ceph-mon[26688]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:40.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:40 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/377076220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "897e21a9-3009-41c6-acf1-e44414fc69c1"}]: dispatch 2024-04-22T15:12:40.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:40 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/377076220' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "897e21a9-3009-41c6-acf1-e44414fc69c1"}]': finished 2024-04-22T15:12:40.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:40 smithi096 ceph-mon[26688]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T15:12:40.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:40 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:12:40.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:40 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/588727263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:12:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:40 smithi196 ceph-mon[34717]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:40 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/377076220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "897e21a9-3009-41c6-acf1-e44414fc69c1"}]: dispatch 2024-04-22T15:12:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:40 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/377076220' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "897e21a9-3009-41c6-acf1-e44414fc69c1"}]': finished 2024-04-22T15:12:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:40 smithi196 ceph-mon[34717]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T15:12:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:40 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:12:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:40 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/588727263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:12:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:42 smithi096 ceph-mon[26688]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:42 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:42 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:42 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:42 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:12:42.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:42 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:42 smithi196 ceph-mon[34717]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:42 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:43.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:42 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:43.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:42 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:43.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:42 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:12:43.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:42 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:44.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:44 smithi096 ceph-mon[26688]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:44 smithi196 ceph-mon[34717]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:46 smithi096 ceph-mon[26688]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:46 smithi196 ceph-mon[34717]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:47.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:12:47.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:47 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:12:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:47 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:12:48.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:48 smithi096 ceph-mon[26688]: Deploying daemon osd.2 on smithi096 2024-04-22T15:12:48.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:48 smithi096 ceph-mon[26688]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:48 smithi196 ceph-mon[34717]: Deploying daemon osd.2 on smithi096 2024-04-22T15:12:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:48 smithi196 ceph-mon[34717]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:50.906 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:50 smithi096 ceph-mon[26688]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:50 smithi196 ceph-mon[34717]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:52.793 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:52 smithi096 ceph-mon[26688]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:52.793 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:12:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:52 smithi196 ceph-mon[34717]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:12:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:12:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:53 smithi096 ceph-mon[26688]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:54.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:12:54.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:54.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:53 smithi196 ceph-mon[34717]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:56.320 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 2 on host 'smithi096' 2024-04-22T15:12:56.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.316+0000 7f486effd700 1 -- 172.21.15.96:0/3475913531 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f487802ce00 con 0x7f486406cc60 2024-04-22T15:12:56.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.318+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f486406cc60 msgr2=0x7f486406f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:12:56.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.318+0000 7f487fe82700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f486406cc60 0x7f486406f120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4868000c00 tx=0x7f4868005960 comp rx=0 tx=0).stop 2024-04-22T15:12:56.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.319+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4878101eb0 msgr2=0x7f48780ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:12:56.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.319+0000 7f487fe82700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4878101eb0 0x7f48780ffe60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f487400d8d0 tx=0x7f487400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:12:56.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.319+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 shutdown_connections 2024-04-22T15:12:56.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.319+0000 7f487fe82700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f486406cc60 0x7f486406f120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:56.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.319+0000 7f487fe82700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48780feed0 0x7f4878101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:56.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.319+0000 7f487fe82700 1 --2- 172.21.15.96:0/3475913531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4878101eb0 0x7f48780ffe60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:12:56.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.319+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 >> 172.21.15.96:0/3475913531 conn(0x7f48780fa830 msgr2=0x7f48781054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:12:56.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.319+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 shutdown_connections 2024-04-22T15:12:56.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:12:56.319+0000 7f487fe82700 1 -- 172.21.15.96:0/3475913531 wait complete. 2024-04-22T15:12:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:56 smithi096 ceph-mon[26688]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:56 smithi196 ceph-mon[34717]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:12:57.022 DEBUG:teuthology.orchestra.run.smithi096:osd.2> sudo journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T15:12:57.025 INFO:tasks.cephadm:Deploying osd.3 on smithi196 with /dev/vg_nvme/lv_4... 2024-04-22T15:12:57.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T15:12:57.236 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:12:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:57 smithi096 ceph-mon[26688]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T15:12:57.998 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:57 smithi196 ceph-mon[34717]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T15:12:58.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:58 smithi096 ceph-mon[26688]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:58.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:58 smithi096 ceph-mon[26688]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T15:12:58.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:58 smithi096 ceph-mon[26688]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T15:12:58.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:12:58.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:58 smithi096 ceph-mon[26688]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:12:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:58 smithi196 ceph-mon[34717]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:12:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:58 smithi196 ceph-mon[34717]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T15:12:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:58 smithi196 ceph-mon[34717]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T15:12:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:12:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:58 smithi196 ceph-mon[34717]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:12:59.105 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T15:12:59.105 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T15:12:59.105 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-04-22T15:12:59.105 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-04-22T15:12:59.105 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124316 s, 843 MB/s 2024-04-22T15:12:59.105 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-04-22T15:12:59.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_4 2024-04-22T15:12:59.790 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:59 smithi096 ceph-mon[26688]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T15:12:59.790 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:59 smithi096 ceph-mon[26688]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T15:12:59.790 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:59 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:12:59.790 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:59 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:12:59.791 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:12:59 smithi096 ceph-mon[26688]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' 2024-04-22T15:12:59.791 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:12:59 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[61334]: 2024-04-22T15:12:59.518+0000 7fe1ecb23700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T15:12:59.791 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:12:59 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[61334]: 2024-04-22T15:12:59.528+0000 7fe1e5111700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:12:59.937 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:13:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:59 smithi196 ceph-mon[34717]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T15:13:00.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:59 smithi196 ceph-mon[34717]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T15:13:00.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:59 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:13:00.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:59 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:13:00.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:12:59 smithi196 ceph-mon[34717]: from='osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985]' entity='osd.2' 2024-04-22T15:13:00.961 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:00 smithi096 ceph-mon[26688]: purged_snaps scrub starts 2024-04-22T15:13:00.961 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:00 smithi096 ceph-mon[26688]: purged_snaps scrub ok 2024-04-22T15:13:00.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:00 smithi096 ceph-mon[26688]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:13:00.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:00 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:13:00.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:00 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:00.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:00 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:00.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:00 smithi096 ceph-mon[26688]: osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] boot 2024-04-22T15:13:00.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:00 smithi096 ceph-mon[26688]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T15:13:00.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:00 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:13:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:00 smithi196 ceph-mon[34717]: purged_snaps scrub starts 2024-04-22T15:13:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:00 smithi196 ceph-mon[34717]: purged_snaps scrub ok 2024-04-22T15:13:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:00 smithi196 ceph-mon[34717]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:13:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:00 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:13:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:00 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:00 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:00 smithi196 ceph-mon[34717]: osd.2 [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] boot 2024-04-22T15:13:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:00 smithi196 ceph-mon[34717]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T15:13:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:00 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:13:01.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.457+0000 7f49953c9700 1 -- 172.21.15.196:0/4196280950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990100d90 msgr2=0x7f49901011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:01.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.457+0000 7f49953c9700 1 --2- 172.21.15.196:0/4196280950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990100d90 0x7f49901011f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4984009910 tx=0x7f4984009c20 comp rx=0 tx=0).stop 2024-04-22T15:13:01.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.457+0000 7f49953c9700 1 -- 172.21.15.196:0/4196280950 shutdown_connections 2024-04-22T15:13:01.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.457+0000 7f49953c9700 1 --2- 172.21.15.196:0/4196280950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990100d90 0x7f49901011f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:01.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.457+0000 7f49953c9700 1 --2- 172.21.15.196:0/4196280950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49900ffb90 0x7f49900fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:01.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.457+0000 7f49953c9700 1 -- 172.21.15.196:0/4196280950 >> 172.21.15.196:0/4196280950 conn(0x7f49900fb110 msgr2=0x7f49900fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.457+0000 7f49953c9700 1 -- 172.21.15.196:0/4196280950 shutdown_connections 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.457+0000 7f49953c9700 1 -- 172.21.15.196:0/4196280950 wait complete. 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f49953c9700 1 Processor -- start 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f49953c9700 1 -- start start 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f49953c9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49900ffb90 0x7f49900690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f49953c9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4990100d90 0x7f499019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f49953c9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49901971d0 con 0x7f4990100d90 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f49953c9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499019dd30 con 0x7f49900ffb90 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f498effd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49900ffb90 0x7f49900690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f498effd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49900ffb90 0x7f49900690a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:41558/0 (socket says 172.21.15.196:41558) 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f498e7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4990100d90 0x7f499019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f498effd700 1 -- 172.21.15.196:0/1957539332 learned_addr learned my addr 172.21.15.196:0/1957539332 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f498effd700 1 -- 172.21.15.196:0/1957539332 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4990100d90 msgr2=0x7f499019d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:01.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f498effd700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4990100d90 0x7f499019d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:01.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.458+0000 7f498effd700 1 -- 172.21.15.196:0/1957539332 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49840095f0 con 0x7f49900ffb90 2024-04-22T15:13:01.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.459+0000 7f498effd700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49900ffb90 0x7f49900690a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f498000c910 tx=0x7f498000cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:13:01.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.459+0000 7f4977fff700 1 -- 172.21.15.196:0/1957539332 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49800041f0 con 0x7f49900ffb90 2024-04-22T15:13:01.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.459+0000 7f4977fff700 1 -- 172.21.15.196:0/1957539332 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f498000ce60 con 0x7f49900ffb90 2024-04-22T15:13:01.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.459+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499019ded0 con 0x7f49900ffb90 2024-04-22T15:13:01.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.459+0000 7f4977fff700 1 -- 172.21.15.196:0/1957539332 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4980003d90 con 0x7f49900ffb90 2024-04-22T15:13:01.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.459+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499019e390 con 0x7f49900ffb90 2024-04-22T15:13:01.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.460+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499018f710 con 0x7f49900ffb90 2024-04-22T15:13:01.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.460+0000 7f4977fff700 1 -- 172.21.15.196:0/1957539332 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f498000f450 con 0x7f49900ffb90 2024-04-22T15:13:01.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.460+0000 7f4977fff700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f497806cc50 0x7f497806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:13:01.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.461+0000 7f4977fff700 1 -- 172.21.15.196:0/1957539332 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2846+0+0 (secure 0 0 0) 0x7f498008b410 con 0x7f49900ffb90 2024-04-22T15:13:01.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.461+0000 7f498e7fc700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f497806cc50 0x7f497806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:13:01.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.461+0000 7f498e7fc700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f497806cc50 0x7f497806f110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4984006010 tx=0x7f4984005180 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:13:01.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.465+0000 7f4977fff700 1 -- 172.21.15.196:0/1957539332 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4980059e00 con 0x7f49900ffb90 2024-04-22T15:13:01.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:01.665+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f499002ceb0 con 0x7f497806cc50 2024-04-22T15:13:02.053 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:01 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:13:02.053 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:01 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:13:02.053 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:01 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:02.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:01 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:13:02.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:01 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:13:02.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:01 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:03.058 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:02 smithi096 ceph-mon[26688]: pgmap v79: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:03.058 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:02 smithi096 ceph-mon[26688]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:13:03.058 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:02 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T15:13:03.058 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:02 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T15:13:03.058 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:02 smithi096 ceph-mon[26688]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T15:13:03.058 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:02 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T15:13:03.201 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:02 smithi196 ceph-mon[34717]: pgmap v79: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:03.201 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:02 smithi196 ceph-mon[34717]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:13:03.201 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:02 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T15:13:03.201 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:02 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T15:13:03.201 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:02 smithi196 ceph-mon[34717]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T15:13:03.201 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:02 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T15:13:04.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T15:13:04.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: Detected new or changed devices on smithi096 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: Adjusting osd_memory_target on smithi096 to 3946M 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:04.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:03 smithi096 ceph-mon[26688]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:04.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T15:13:04.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T15:13:04.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: Detected new or changed devices on smithi096 2024-04-22T15:13:04.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:04.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:04.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:04.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:04.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:04.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: Adjusting osd_memory_target on smithi096 to 3946M 2024-04-22T15:13:04.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:04.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:04.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:04.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:04.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:04.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:03 smithi196 ceph-mon[34717]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:04.526 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66164]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:13:04.526 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66164]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:04.526 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66164]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:04.526 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66164]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:04.820 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66180]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:13:04.820 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66180]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:04.821 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66180]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:04.821 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66180]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:05.080 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66264]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:13:05.080 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66264]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:05.080 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66264]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:05.080 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:13:04 smithi096 sudo[66264]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:05.080 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:04 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/1386612121' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "201c3153-a284-471e-894d-2273001cc29e"}]: dispatch 2024-04-22T15:13:05.080 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:04 smithi096 ceph-mon[26688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "201c3153-a284-471e-894d-2273001cc29e"}]: dispatch 2024-04-22T15:13:05.080 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:04 smithi096 ceph-mon[26688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "201c3153-a284-471e-894d-2273001cc29e"}]': finished 2024-04-22T15:13:05.080 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:04 smithi096 ceph-mon[26688]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T15:13:05.080 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:05.081 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:04 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:05.081 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:04 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/3235418532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:13:05.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:04 smithi196 ceph-mon[34717]: from='client.? 172.21.15.196:0/1386612121' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "201c3153-a284-471e-894d-2273001cc29e"}]: dispatch 2024-04-22T15:13:05.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:04 smithi196 ceph-mon[34717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "201c3153-a284-471e-894d-2273001cc29e"}]: dispatch 2024-04-22T15:13:05.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:04 smithi196 ceph-mon[34717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "201c3153-a284-471e-894d-2273001cc29e"}]': finished 2024-04-22T15:13:05.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:04 smithi196 ceph-mon[34717]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T15:13:05.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:04 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:05.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:04 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:05.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:04 smithi196 ceph-mon[34717]: from='client.? 172.21.15.196:0/3235418532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:13:05.330 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66291]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:13:05.382 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66291]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:05.382 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66291]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:05.382 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66291]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:05.590 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66314]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:13:05.590 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66314]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:05.590 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66314]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:05.590 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66314]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:05.840 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66349]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:13:05.840 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:05.840 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66349]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:05.841 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66349]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:06.098 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66392]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:13:06.098 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66392]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:06.098 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66392]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:06.098 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:05 smithi096 sudo[66392]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:06.454 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 sudo[66397]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:13:06.454 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 sudo[66397]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:06.454 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 sudo[66397]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:06.454 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 sudo[66397]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:06.604 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 sudo[39038]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:13:06.604 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 sudo[39038]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:06.604 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 sudo[39038]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:06.604 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 sudo[39038]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:06.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:06.613+0000 7f4977fff700 1 -- 172.21.15.196:0/1957539332 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4980056740 con 0x7f49900ffb90 2024-04-22T15:13:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 ceph-mon[26688]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T15:13:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 ceph-mon[26688]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 ceph-mon[26688]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T15:13:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:13:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:13:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:13:06.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:06 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:13:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 sudo[39226]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 sudo[39226]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 sudo[39226]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 sudo[39226]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 ceph-mon[34717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 ceph-mon[34717]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 ceph-mon[34717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:13:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:06 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:13:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:07 smithi096 ceph-mon[26688]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T15:13:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:07 smithi096 ceph-mon[26688]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T15:13:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:07 smithi096 ceph-mon[26688]: mgrmap e19: smithi096.hhkvum(active, since 2m), standbys: smithi196.gdaegc 2024-04-22T15:13:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:07 smithi196 ceph-mon[34717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T15:13:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:07 smithi196 ceph-mon[34717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T15:13:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:07 smithi196 ceph-mon[34717]: mgrmap e19: smithi096.hhkvum(active, since 2m), standbys: smithi196.gdaegc 2024-04-22T15:13:08.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:07 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:08 smithi196 ceph-mon[34717]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:13:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:08 smithi096 ceph-mon[26688]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:13:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:08.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:10.899 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:10 smithi196 ceph-mon[34717]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:10 smithi096 ceph-mon[26688]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:11.978 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:13:11.978 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:11 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:11.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:11 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:13:11.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:11 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:12.924 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:12 smithi196 ceph-mon[34717]: Deploying daemon osd.3 on smithi196 2024-04-22T15:13:12.924 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:12 smithi196 ceph-mon[34717]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:12 smithi096 ceph-mon[26688]: Deploying daemon osd.3 on smithi196 2024-04-22T15:13:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:12 smithi096 ceph-mon[26688]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:13:14.917 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:14 smithi196 ceph-mon[34717]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-04-22T15:13:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:14 smithi096 ceph-mon[26688]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-04-22T15:13:16.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:16 smithi096 ceph-mon[26688]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 33 KiB/s wr, 2 op/s 2024-04-22T15:13:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:16 smithi196 ceph-mon[34717]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 33 KiB/s wr, 2 op/s 2024-04-22T15:13:18.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:18.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:18.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:18.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:18.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:18.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:18 smithi196 ceph-mon[34717]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 819 B/s rd, 26 KiB/s wr, 1 op/s 2024-04-22T15:13:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:18 smithi096 ceph-mon[26688]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 819 B/s rd, 26 KiB/s wr, 1 op/s 2024-04-22T15:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 3 on host 'smithi196' 2024-04-22T15:13:19.948 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.944+0000 7f4977fff700 1 -- 172.21.15.196:0/1957539332 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f499002ceb0 con 0x7f497806cc50 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.946+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f497806cc50 msgr2=0x7f497806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.946+0000 7f49953c9700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f497806cc50 0x7f497806f110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4984006010 tx=0x7f4984005180 comp rx=0 tx=0).stop 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.946+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49900ffb90 msgr2=0x7f49900690a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.946+0000 7f49953c9700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49900ffb90 0x7f49900690a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f498000c910 tx=0x7f498000cc20 comp rx=0 tx=0).stop 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.946+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 shutdown_connections 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.946+0000 7f49953c9700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f497806cc50 0x7f497806f110 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.946+0000 7f49953c9700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49900ffb90 0x7f49900690a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.946+0000 7f49953c9700 1 --2- 172.21.15.196:0/1957539332 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4990100d90 0x7f499019d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.946+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 >> 172.21.15.196:0/1957539332 conn(0x7f49900fb110 msgr2=0x7f4990104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:13:19.950 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.947+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 shutdown_connections 2024-04-22T15:13:19.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:19.947+0000 7f49953c9700 1 -- 172.21.15.196:0/1957539332 wait complete. 2024-04-22T15:13:20.576 DEBUG:teuthology.orchestra.run.smithi196:osd.3> sudo journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T15:13:20.579 INFO:tasks.cephadm:Deploying osd.4 on smithi196 with /dev/vg_nvme/lv_3... 2024-04-22T15:13:20.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T15:13:20.907 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:20 smithi196 ceph-mon[34717]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 682 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-22T15:13:20.907 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:20.908 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:20 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:20.908 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:20 smithi196 ceph-mon[34717]: from='osd.3 [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:13:20.908 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:20 smithi196 ceph-mon[34717]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:13:20.908 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:13:20 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[40667]: 2024-04-22T15:13:20.653+0000 7f94057837c0 -1 osd.3 0 log_to_monitors true 2024-04-22T15:13:21.187 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:13:21.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:20 smithi096 ceph-mon[26688]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 682 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-22T15:13:21.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:21.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:20 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:21.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:20 smithi096 ceph-mon[26688]: from='osd.3 [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:13:21.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:20 smithi096 ceph-mon[26688]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:13:22.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:21 smithi096 ceph-mon[26688]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T15:13:22.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:21 smithi096 ceph-mon[26688]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T15:13:22.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:21 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:22.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:21 smithi096 ceph-mon[26688]: from='osd.3 [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:13:22.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:21 smithi096 ceph-mon[26688]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:13:22.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:21 smithi096 ceph-mon[26688]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 819 B/s rd, 26 KiB/s wr, 1 op/s 2024-04-22T15:13:22.255 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:21 smithi196 ceph-mon[34717]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T15:13:22.255 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:21 smithi196 ceph-mon[34717]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T15:13:22.255 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:21 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:22.255 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:21 smithi196 ceph-mon[34717]: from='osd.3 [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:13:22.255 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:21 smithi196 ceph-mon[34717]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:13:22.255 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:21 smithi196 ceph-mon[34717]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 819 B/s rd, 26 KiB/s wr, 1 op/s 2024-04-22T15:13:22.255 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:13:21 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[40667]: 2024-04-22T15:13:21.954+0000 7f93fa5e7700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T15:13:22.255 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:13:21 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[40667]: 2024-04-22T15:13:21.961+0000 7f93f6bdd700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:13:23.160 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T15:13:23.160 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T15:13:23.160 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-04-22T15:13:23.160 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-04-22T15:13:23.160 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123076 s, 852 MB/s 2024-04-22T15:13:23.160 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-04-22T15:13:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-22T15:13:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T15:13:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:23.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:13:23.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: osd.3 [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] boot 2024-04-22T15:13:23.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T15:13:23.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:23.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-22T15:13:23.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T15:13:23.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:23.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:23.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:23.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:23.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:13:23.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: osd.3 [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] boot 2024-04-22T15:13:23.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T15:13:23.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:13:24.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_3 2024-04-22T15:13:24.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:23 smithi096 ceph-mon[26688]: purged_snaps scrub starts 2024-04-22T15:13:24.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:23 smithi096 ceph-mon[26688]: purged_snaps scrub ok 2024-04-22T15:13:24.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:23 smithi096 ceph-mon[26688]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:24.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:23 smithi096 ceph-mon[26688]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T15:13:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:23 smithi196 ceph-mon[34717]: purged_snaps scrub starts 2024-04-22T15:13:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:23 smithi196 ceph-mon[34717]: purged_snaps scrub ok 2024-04-22T15:13:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:23 smithi196 ceph-mon[34717]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:24.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:23 smithi196 ceph-mon[34717]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T15:13:25.140 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:13:26.154 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:25 smithi196 ceph-mon[34717]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T15:13:26.154 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:25 smithi196 ceph-mon[34717]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:26.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:25 smithi096 ceph-mon[26688]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T15:13:26.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:25 smithi096 ceph-mon[26688]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:27.132 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T15:13:27.132 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: Detected new or changed devices on smithi196 2024-04-22T15:13:27.132 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.132 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: Adjusting osd_memory_target on smithi196 to 14912M 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:27.133 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:26 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T15:13:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: Detected new or changed devices on smithi196 2024-04-22T15:13:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: Adjusting osd_memory_target on smithi196 to 14912M 2024-04-22T15:13:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:27.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:26 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:27.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.354+0000 7f9d4b377700 1 -- 172.21.15.196:0/2206007024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 msgr2=0x7f9d44103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:27.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.354+0000 7f9d4b377700 1 --2- 172.21.15.196:0/2206007024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 0x7f9d44103b10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400b0a0 tx=0x7f9d3400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:13:27.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.354+0000 7f9d4b377700 1 -- 172.21.15.196:0/2206007024 shutdown_connections 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.354+0000 7f9d4b377700 1 --2- 172.21.15.196:0/2206007024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d44104050 0x7f9d44106440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.354+0000 7f9d4b377700 1 --2- 172.21.15.196:0/2206007024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 0x7f9d44103b10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.354+0000 7f9d4b377700 1 -- 172.21.15.196:0/2206007024 >> 172.21.15.196:0/2206007024 conn(0x7f9d440fb110 msgr2=0x7f9d440fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.354+0000 7f9d4b377700 1 -- 172.21.15.196:0/2206007024 shutdown_connections 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.354+0000 7f9d4b377700 1 -- 172.21.15.196:0/2206007024 wait complete. 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d4b377700 1 Processor -- start 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d4b377700 1 -- start start 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d4b377700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 0x7f9d44196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d4b377700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d44104050 0x7f9d44196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:13:27.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d4b377700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d441971b0 con 0x7f9d44104050 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d4b377700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d441972f0 con 0x7f9d44101720 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d49113700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 0x7f9d44196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d49113700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 0x7f9d44196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35854/0 (socket says 172.21.15.196:35854) 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d49113700 1 -- 172.21.15.196:0/1538567779 learned_addr learned my addr 172.21.15.196:0/1538567779 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d48912700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d44104050 0x7f9d44196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d49113700 1 -- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d44104050 msgr2=0x7f9d44196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d49113700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d44104050 0x7f9d44196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d49113700 1 -- 172.21.15.196:0/1538567779 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d34009d00 con 0x7f9d44101720 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d49113700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 0x7f9d44196650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9d34006130 tx=0x7f9d34003c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d427fc700 1 -- 172.21.15.196:0/1538567779 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d34021070 con 0x7f9d44101720 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.355+0000 7f9d427fc700 1 -- 172.21.15.196:0/1538567779 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d3400ce60 con 0x7f9d44101720 2024-04-22T15:13:27.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.356+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4419bcf0 con 0x7f9d44101720 2024-04-22T15:13:27.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.356+0000 7f9d427fc700 1 -- 172.21.15.196:0/1538567779 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d3400e040 con 0x7f9d44101720 2024-04-22T15:13:27.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.356+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4419c190 con 0x7f9d44101720 2024-04-22T15:13:27.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.356+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4404ebb0 con 0x7f9d44101720 2024-04-22T15:13:27.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.357+0000 7f9d427fc700 1 -- 172.21.15.196:0/1538567779 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9d34004250 con 0x7f9d44101720 2024-04-22T15:13:27.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.357+0000 7f9d427fc700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9d2c06cce0 0x7f9d2c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:13:27.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.357+0000 7f9d427fc700 1 -- 172.21.15.196:0/1538567779 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f9d3408cf80 con 0x7f9d44101720 2024-04-22T15:13:27.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.358+0000 7f9d48912700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9d2c06cce0 0x7f9d2c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:13:27.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.358+0000 7f9d48912700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9d2c06cce0 0x7f9d2c06f1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d38005950 tx=0x7f9d3800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:13:27.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.362+0000 7f9d427fc700 1 -- 172.21.15.196:0/1538567779 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9d3405b480 con 0x7f9d44101720 2024-04-22T15:13:27.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:27.561+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d44105f30 con 0x7f9d2c06cce0 2024-04-22T15:13:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:27 smithi096 ceph-mon[26688]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T15:13:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:27 smithi096 ceph-mon[26688]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:13:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:27 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:13:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:27 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:13:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:27 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:28.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:27 smithi096 ceph-mon[26688]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T15:13:28.239 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:27 smithi196 ceph-mon[34717]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T15:13:28.239 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:27 smithi196 ceph-mon[34717]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:13:28.239 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:27 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:13:28.239 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:27 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:13:28.240 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:27 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:28.240 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:27 smithi196 ceph-mon[34717]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T15:13:30.952 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:30 smithi196 ceph-mon[34717]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-22T15:13:30.952 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:30 smithi196 ceph-mon[34717]: from='client.? 172.21.15.196:0/3785129851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780ed007-14ef-4c92-a5e9-6c57afcad448"}]: dispatch 2024-04-22T15:13:30.952 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:30 smithi196 ceph-mon[34717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780ed007-14ef-4c92-a5e9-6c57afcad448"}]: dispatch 2024-04-22T15:13:30.953 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:30 smithi196 ceph-mon[34717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "780ed007-14ef-4c92-a5e9-6c57afcad448"}]': finished 2024-04-22T15:13:30.953 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:30 smithi196 ceph-mon[34717]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T15:13:30.953 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:30 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:30 smithi096 ceph-mon[26688]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-22T15:13:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:30 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/3785129851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780ed007-14ef-4c92-a5e9-6c57afcad448"}]: dispatch 2024-04-22T15:13:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:30 smithi096 ceph-mon[26688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "780ed007-14ef-4c92-a5e9-6c57afcad448"}]: dispatch 2024-04-22T15:13:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:30 smithi096 ceph-mon[26688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "780ed007-14ef-4c92-a5e9-6c57afcad448"}]': finished 2024-04-22T15:13:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:30 smithi096 ceph-mon[26688]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T15:13:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:30 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:31 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/4202524897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:13:32.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:31 smithi196 ceph-mon[34717]: from='client.? 172.21.15.196:0/4202524897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:13:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:32 smithi096 ceph-mon[26688]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T15:13:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:32 smithi196 ceph-mon[34717]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T15:13:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:34 smithi096 ceph-mon[26688]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-22T15:13:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:34 smithi196 ceph-mon[34717]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-22T15:13:36.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:36 smithi196 ceph-mon[34717]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:36 smithi096 ceph-mon[26688]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:13:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:13:37.991 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:13:37.991 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:37.992 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:13:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:38 smithi096 ceph-mon[26688]: Deploying daemon osd.4 on smithi196 2024-04-22T15:13:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:38 smithi096 ceph-mon[26688]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:38 smithi196 ceph-mon[34717]: Deploying daemon osd.4 on smithi196 2024-04-22T15:13:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:38 smithi196 ceph-mon[34717]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:40.922 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:40 smithi196 ceph-mon[34717]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:40 smithi096 ceph-mon[26688]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:42 smithi096 ceph-mon[26688]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:42 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:42 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:42 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:42.995 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:42 smithi196 ceph-mon[34717]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:42.995 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:42 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:42.995 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:42 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:42.995 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:42 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:44.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:44 smithi196 ceph-mon[34717]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:44.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:44 smithi096 ceph-mon[26688]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:46.076 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 4 on host 'smithi196' 2024-04-22T15:13:46.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.073+0000 7f9d427fc700 1 -- 172.21.15.196:0/1538567779 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9d44105f30 con 0x7f9d2c06cce0 2024-04-22T15:13:46.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.075+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9d2c06cce0 msgr2=0x7f9d2c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.075+0000 7f9d4b377700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9d2c06cce0 0x7f9d2c06f1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d38005950 tx=0x7f9d3800b410 comp rx=0 tx=0).stop 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.075+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 msgr2=0x7f9d44196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.075+0000 7f9d4b377700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 0x7f9d44196650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9d34006130 tx=0x7f9d34003c30 comp rx=0 tx=0).stop 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.075+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 shutdown_connections 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.075+0000 7f9d4b377700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9d2c06cce0 0x7f9d2c06f1a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.075+0000 7f9d4b377700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9d44101720 0x7f9d44196650 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.075+0000 7f9d4b377700 1 --2- 172.21.15.196:0/1538567779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d44104050 0x7f9d44196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.075+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 >> 172.21.15.196:0/1538567779 conn(0x7f9d440fb110 msgr2=0x7f9d440fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.076+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 shutdown_connections 2024-04-22T15:13:46.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:46.076+0000 7f9d4b377700 1 -- 172.21.15.196:0/1538567779 wait complete. 2024-04-22T15:13:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:46 smithi096 ceph-mon[26688]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:46 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:46 smithi096 ceph-mon[26688]: from='osd.4 [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:13:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:46 smithi096 ceph-mon[26688]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:13:47.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:46 smithi196 ceph-mon[34717]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:47.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:46 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:47.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:46 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:47.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:46 smithi196 ceph-mon[34717]: from='osd.4 [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:13:47.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:46 smithi196 ceph-mon[34717]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:13:47.338 DEBUG:teuthology.orchestra.run.smithi196:osd.4> sudo journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T15:13:47.341 INFO:tasks.cephadm:Deploying osd.5 on smithi196 with /dev/vg_nvme/lv_2... 2024-04-22T15:13:47.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T15:13:47.590 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:13:48.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-mon[34717]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T15:13:48.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-mon[34717]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T15:13:48.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:48.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-mon[34717]: from='osd.4 [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:13:48.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-mon[34717]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:13:48.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:48.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:48.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-mon[34717]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:48.260 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[47277]: 2024-04-22T15:13:48.043+0000 7f39b2938700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T15:13:48.261 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:13:48 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[47277]: 2024-04-22T15:13:48.049+0000 7f39aef2e700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:13:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:48 smithi096 ceph-mon[26688]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T15:13:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:48 smithi096 ceph-mon[26688]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T15:13:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:48.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:48 smithi096 ceph-mon[26688]: from='osd.4 [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:13:48.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:48 smithi096 ceph-mon[26688]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:13:48.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:48.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:48.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:48 smithi096 ceph-mon[26688]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-22T15:13:49.220 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:49 smithi196 ceph-mon[34717]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-22T15:13:49.220 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:49 smithi196 ceph-mon[34717]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T15:13:49.220 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:49 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:49.221 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:49 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:49.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:49 smithi096 ceph-mon[26688]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-22T15:13:49.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:49 smithi096 ceph-mon[26688]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T15:13:49.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:49 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:49.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:49 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:49.666 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T15:13:49.666 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T15:13:49.666 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-04-22T15:13:49.666 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-04-22T15:13:49.666 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141436 s, 741 MB/s 2024-04-22T15:13:49.666 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-04-22T15:13:50.326 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:50 smithi196 ceph-mon[34717]: purged_snaps scrub starts 2024-04-22T15:13:50.326 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:50 smithi196 ceph-mon[34717]: purged_snaps scrub ok 2024-04-22T15:13:50.326 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:50 smithi196 ceph-mon[34717]: osd.4 [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] boot 2024-04-22T15:13:50.326 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:50 smithi196 ceph-mon[34717]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T15:13:50.326 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:50 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:50.326 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:50 smithi196 ceph-mon[34717]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:13:50.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_2 2024-04-22T15:13:50.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:50 smithi096 ceph-mon[26688]: purged_snaps scrub starts 2024-04-22T15:13:50.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:50 smithi096 ceph-mon[26688]: purged_snaps scrub ok 2024-04-22T15:13:50.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:50 smithi096 ceph-mon[26688]: osd.4 [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] boot 2024-04-22T15:13:50.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:50 smithi096 ceph-mon[26688]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T15:13:50.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:50 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:13:50.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:50 smithi096 ceph-mon[26688]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:13:51.255 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:13:51.377 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:51 smithi196 ceph-mon[34717]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T15:13:51.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:51 smithi096 ceph-mon[26688]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T15:13:52.189 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:52 smithi196 ceph-mon[34717]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:13:52.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:52 smithi096 ceph-mon[26688]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:13:53.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.395+0000 7f1176849700 1 -- 172.21.15.196:0/2458355642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 msgr2=0x7f1170102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.395+0000 7f1176849700 1 --2- 172.21.15.196:0/2458355642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 0x7f1170102290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1164009910 tx=0x7f1164009c20 comp rx=0 tx=0).stop 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.395+0000 7f1176849700 1 -- 172.21.15.196:0/2458355642 shutdown_connections 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.395+0000 7f1176849700 1 --2- 172.21.15.196:0/2458355642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 0x7f1170102290 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.395+0000 7f1176849700 1 --2- 172.21.15.196:0/2458355642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1170100c30 0x7f1170101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.395+0000 7f1176849700 1 -- 172.21.15.196:0/2458355642 >> 172.21.15.196:0/2458355642 conn(0x7f11700fc1d0 msgr2=0x7f11700fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.395+0000 7f1176849700 1 -- 172.21.15.196:0/2458355642 shutdown_connections 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.395+0000 7f1176849700 1 -- 172.21.15.196:0/2458355642 wait complete. 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.396+0000 7f1176849700 1 Processor -- start 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.396+0000 7f1176849700 1 -- start start 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.396+0000 7f1176849700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1170100c30 0x7f11701965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.396+0000 7f1176849700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 0x7f1170196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.396+0000 7f1176849700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1170197150 con 0x7f1170100c30 2024-04-22T15:13:53.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.396+0000 7f1176849700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1170197290 con 0x7f1170101e30 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.396+0000 7f116f7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 0x7f1170196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.396+0000 7f116f7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 0x7f1170196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:43436/0 (socket says 172.21.15.196:43436) 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.396+0000 7f116f7fe700 1 -- 172.21.15.196:0/4077094733 learned_addr learned my addr 172.21.15.196:0/4077094733 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f116ffff700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1170100c30 0x7f11701965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f116f7fe700 1 -- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1170100c30 msgr2=0x7f11701965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f116f7fe700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1170100c30 0x7f11701965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f116f7fe700 1 -- 172.21.15.196:0/4077094733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11640095f0 con 0x7f1170101e30 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f116ffff700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1170100c30 0x7f11701965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f116f7fe700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 0x7f1170196b30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f11640092e0 tx=0x7f116400be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f116d7fa700 1 -- 172.21.15.196:0/4077094733 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f116401d070 con 0x7f1170101e30 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117019bc90 con 0x7f1170101e30 2024-04-22T15:13:53.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f116d7fa700 1 -- 172.21.15.196:0/4077094733 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f116400f460 con 0x7f1170101e30 2024-04-22T15:13:53.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f116d7fa700 1 -- 172.21.15.196:0/4077094733 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1164021640 con 0x7f1170101e30 2024-04-22T15:13:53.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.397+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117019c200 con 0x7f1170101e30 2024-04-22T15:13:53.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.398+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11701907b0 con 0x7f1170101e30 2024-04-22T15:13:53.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.398+0000 7f116d7fa700 1 -- 172.21.15.196:0/4077094733 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f116400f5d0 con 0x7f1170101e30 2024-04-22T15:13:53.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.399+0000 7f116d7fa700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f115806cce0 0x7f115806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:13:53.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.399+0000 7f116d7fa700 1 -- 172.21.15.196:0/4077094733 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f116408d0e0 con 0x7f1170101e30 2024-04-22T15:13:53.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.399+0000 7f116ffff700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f115806cce0 0x7f115806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:13:53.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.400+0000 7f116ffff700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f115806cce0 0x7f115806f1a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1170101c90 tx=0x7f1160005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:13:53.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.403+0000 7f116d7fa700 1 -- 172.21.15.196:0/4077094733 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f116405b430 con 0x7f1170101e30 2024-04-22T15:13:53.611 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:13:53.608+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f117002ce00 con 0x7f115806cce0 2024-04-22T15:13:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: Detected new or changed devices on smithi196 2024-04-22T15:13:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: Adjusting osd_memory_target on smithi196 to 7456M 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:53.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:53 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: Detected new or changed devices on smithi196 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: Adjusting osd_memory_target on smithi196 to 7456M 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:13:54.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:53 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:13:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:54 smithi196 ceph-mon[34717]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:13:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:13:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:13:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:54 smithi196 ceph-mon[34717]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:13:54.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:54 smithi096 ceph-mon[26688]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:13:54.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:13:54.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:13:54.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:13:54.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:54 smithi096 ceph-mon[26688]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:13:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:56 smithi096 ceph-mon[26688]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:13:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:56 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/2436489889' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16384d8d-bef4-4f65-9079-e2b4518241da"}]: dispatch 2024-04-22T15:13:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:56 smithi096 ceph-mon[26688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16384d8d-bef4-4f65-9079-e2b4518241da"}]: dispatch 2024-04-22T15:13:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:56 smithi096 ceph-mon[26688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16384d8d-bef4-4f65-9079-e2b4518241da"}]': finished 2024-04-22T15:13:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:56 smithi096 ceph-mon[26688]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T15:13:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:56 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:13:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:56 smithi196 ceph-mon[34717]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:13:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:56 smithi196 ceph-mon[34717]: from='client.? 172.21.15.196:0/2436489889' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16384d8d-bef4-4f65-9079-e2b4518241da"}]: dispatch 2024-04-22T15:13:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:56 smithi196 ceph-mon[34717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16384d8d-bef4-4f65-9079-e2b4518241da"}]: dispatch 2024-04-22T15:13:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:56 smithi196 ceph-mon[34717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16384d8d-bef4-4f65-9079-e2b4518241da"}]': finished 2024-04-22T15:13:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:56 smithi196 ceph-mon[34717]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T15:13:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:56 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:13:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:57 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/2882962364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:13:58.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:57 smithi196 ceph-mon[34717]: from='client.? 172.21.15.196:0/2882962364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:13:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:13:58 smithi096 ceph-mon[26688]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:13:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:13:58 smithi196 ceph-mon[34717]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:00 smithi096 ceph-mon[26688]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:00 smithi196 ceph-mon[34717]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:02.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:02 smithi096 ceph-mon[26688]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:02 smithi196 ceph-mon[34717]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:03.942 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:14:03.942 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:03 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:14:03.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:14:03.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:03 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:14:04.945 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:04 smithi196 ceph-mon[34717]: Deploying daemon osd.5 on smithi196 2024-04-22T15:14:04.945 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:04 smithi196 ceph-mon[34717]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:04 smithi096 ceph-mon[26688]: Deploying daemon osd.5 on smithi196 2024-04-22T15:14:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:04 smithi096 ceph-mon[26688]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:06.837 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:06 smithi196 ceph-mon[34717]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:06.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:06 smithi096 ceph-mon[26688]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:14:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:07 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:14:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:08 smithi096 ceph-mon[26688]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:14:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:08 smithi196 ceph-mon[34717]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:14:10.900 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:10 smithi196 ceph-mon[34717]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:10 smithi096 ceph-mon[26688]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:12.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.370+0000 7f116d7fa700 1 -- 172.21.15.196:0/4077094733 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f117002ce00 con 0x7f115806cce0 2024-04-22T15:14:12.374 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 5 on host 'smithi196' 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f115806cce0 msgr2=0x7f115806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f115806cce0 0x7f115806f1a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1170101c90 tx=0x7f1160005f90 comp rx=0 tx=0).stop 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 msgr2=0x7f1170196b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 0x7f1170196b30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f11640092e0 tx=0x7f116400be20 comp rx=0 tx=0).stop 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 shutdown_connections 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f115806cce0 0x7f115806f1a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1170100c30 0x7f11701965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 --2- 172.21.15.196:0/4077094733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1170101e30 0x7f1170196b30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 >> 172.21.15.196:0/4077094733 conn(0x7f11700fc1d0 msgr2=0x7f1170105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 shutdown_connections 2024-04-22T15:14:12.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:12.373+0000 7f1176849700 1 -- 172.21.15.196:0/4077094733 wait complete. 2024-04-22T15:14:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:12 smithi096 ceph-mon[26688]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:12 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:12 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:12.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:12 smithi096 ceph-mon[26688]: from='osd.5 [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:14:12.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:12 smithi096 ceph-mon[26688]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:14:12.994 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:12 smithi196 ceph-mon[34717]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:12.994 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:12 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:12.994 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:12 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:12.994 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:12 smithi196 ceph-mon[34717]: from='osd.5 [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:14:12.994 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:12 smithi196 ceph-mon[34717]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:14:13.016 DEBUG:teuthology.orchestra.run.smithi196:osd.5> sudo journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T15:14:13.019 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-22T15:14:13.019 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T15:14:13.252 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:14.343 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:14.343 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:14.343 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-mon[34717]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T15:14:14.343 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-mon[34717]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T15:14:14.343 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:14:14.343 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-mon[34717]: from='osd.5 [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:14:14.343 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-mon[34717]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:14:14.343 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-mon[34717]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:14.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:14 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:14.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:14 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:14.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:14 smithi096 ceph-mon[26688]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T15:14:14.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:14 smithi096 ceph-mon[26688]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T15:14:14.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:14 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:14:14.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:14 smithi096 ceph-mon[26688]: from='osd.5 [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:14:14.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:14 smithi096 ceph-mon[26688]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:14:14.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:14 smithi096 ceph-mon[26688]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:14.683 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[54127]: 2024-04-22T15:14:14.338+0000 7f0fbc73b700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T15:14:14.683 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:14:14 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[54127]: 2024-04-22T15:14:14.346+0000 7f0fb8d31700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:14:14.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.849+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/947914022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 msgr2=0x7f2d44101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:14.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.849+0000 7f2d4c3a5700 1 --2- 172.21.15.96:0/947914022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 0x7f2d44101180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009910 tx=0x7f2d40009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:14.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.849+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/947914022 shutdown_connections 2024-04-22T15:14:14.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.849+0000 7f2d4c3a5700 1 --2- 172.21.15.96:0/947914022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 0x7f2d44101180 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:14.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.849+0000 7f2d4c3a5700 1 --2- 172.21.15.96:0/947914022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d440ffb20 0x7f2d440fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:14.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.849+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/947914022 >> 172.21.15.96:0/947914022 conn(0x7f2d440fb0c0 msgr2=0x7f2d440fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:14.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.849+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/947914022 shutdown_connections 2024-04-22T15:14:14.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.849+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/947914022 wait complete. 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d4c3a5700 1 Processor -- start 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d4c3a5700 1 -- start start 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d4c3a5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d440ffb20 0x7f2d44103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d4c3a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 0x7f2d44102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d4c3a5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d44102640 con 0x7f2d44100d20 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d4c3a5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d44102780 con 0x7f2d440ffb20 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d49940700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 0x7f2d44102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d49940700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 0x7f2d44102100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35532/0 (socket says 172.21.15.96:35532) 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d49940700 1 -- 172.21.15.96:0/394343781 learned_addr learned my addr 172.21.15.96:0/394343781 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.850+0000 7f2d4a141700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d440ffb20 0x7f2d44103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.851+0000 7f2d49940700 1 -- 172.21.15.96:0/394343781 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d440ffb20 msgr2=0x7f2d44103a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:14.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.851+0000 7f2d49940700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d440ffb20 0x7f2d44103a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:14.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.851+0000 7f2d49940700 1 -- 172.21.15.96:0/394343781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d400095f0 con 0x7f2d44100d20 2024-04-22T15:14:14.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.851+0000 7f2d49940700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 0x7f2d44102100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d40006010 tx=0x7f2d4000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:14.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.851+0000 7f2d3b7fe700 1 -- 172.21.15.96:0/394343781 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4001d070 con 0x7f2d44100d20 2024-04-22T15:14:14.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.851+0000 7f2d3b7fe700 1 -- 172.21.15.96:0/394343781 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d40005dc0 con 0x7f2d44100d20 2024-04-22T15:14:14.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.851+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d44102a00 con 0x7f2d44100d20 2024-04-22T15:14:14.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.851+0000 7f2d3b7fe700 1 -- 172.21.15.96:0/394343781 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d40021410 con 0x7f2d44100d20 2024-04-22T15:14:14.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.851+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d44102f70 con 0x7f2d44100d20 2024-04-22T15:14:14.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.852+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4404ebb0 con 0x7f2d44100d20 2024-04-22T15:14:14.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.852+0000 7f2d3b7fe700 1 -- 172.21.15.96:0/394343781 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2d4000f460 con 0x7f2d44100d20 2024-04-22T15:14:14.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.853+0000 7f2d3b7fe700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d3006ccf0 0x7f2d3006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:14.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.853+0000 7f2d3b7fe700 1 -- 172.21.15.96:0/394343781 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f2d4008ce80 con 0x7f2d44100d20 2024-04-22T15:14:14.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.853+0000 7f2d4a141700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d3006ccf0 0x7f2d3006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:14.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.853+0000 7f2d4a141700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d3006ccf0 0x7f2d3006f1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d44100b80 tx=0x7f2d3400b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:14.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:14.858+0000 7f2d3b7fe700 1 -- 172.21.15.96:0/394343781 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2d4005b130 con 0x7f2d44100d20 2024-04-22T15:14:15.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.052+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2d44108830 con 0x7f2d44100d20 2024-04-22T15:14:15.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.053+0000 7f2d3b7fe700 1 -- 172.21.15.96:0/394343781 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v37) v1 ==== 74+0+130 (secure 0 0 0) 0x7f2d4005acc0 con 0x7f2d44100d20 2024-04-22T15:14:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:14:15.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d3006ccf0 msgr2=0x7f2d3006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d3006ccf0 0x7f2d3006f1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d44100b80 tx=0x7f2d3400b410 comp rx=0 tx=0).stop 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 msgr2=0x7f2d44102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 0x7f2d44102100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d40006010 tx=0x7f2d4000bd70 comp rx=0 tx=0).stop 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 shutdown_connections 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d3006ccf0 0x7f2d3006f1b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d440ffb20 0x7f2d44103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 --2- 172.21.15.96:0/394343781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44100d20 0x7f2d44102100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 >> 172.21.15.96:0/394343781 conn(0x7f2d440fb0c0 msgr2=0x7f2d44069240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 shutdown_connections 2024-04-22T15:14:15.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:15.055+0000 7f2d4c3a5700 1 -- 172.21.15.96:0/394343781 wait complete. 2024-04-22T15:14:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:15 smithi096 ceph-mon[26688]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-22T15:14:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:15 smithi096 ceph-mon[26688]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T15:14:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:15 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:14:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:15 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:14:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:15 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/394343781' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T15:14:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:15 smithi196 ceph-mon[34717]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-22T15:14:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:15 smithi196 ceph-mon[34717]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T15:14:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:15 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:14:15.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:15 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:14:15.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:15 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/394343781' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T15:14:15.871 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":37,"num_osds":6,"num_up_osds":5,"osd_up_since":1713798829,"num_in_osds":6,"osd_in_since":1713798836,"num_remapped_pgs":0} 2024-04-22T15:14:16.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:16 smithi196 ceph-mon[34717]: purged_snaps scrub starts 2024-04-22T15:14:16.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:16 smithi196 ceph-mon[34717]: purged_snaps scrub ok 2024-04-22T15:14:16.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:16 smithi196 ceph-mon[34717]: osd.5 [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] boot 2024-04-22T15:14:16.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:16 smithi196 ceph-mon[34717]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T15:14:16.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:14:16.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:16 smithi196 ceph-mon[34717]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:16 smithi096 ceph-mon[26688]: purged_snaps scrub starts 2024-04-22T15:14:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:16 smithi096 ceph-mon[26688]: purged_snaps scrub ok 2024-04-22T15:14:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:16 smithi096 ceph-mon[26688]: osd.5 [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] boot 2024-04-22T15:14:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:16 smithi096 ceph-mon[26688]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T15:14:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:14:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:16 smithi096 ceph-mon[26688]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:14:16.872 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T15:14:17.087 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:17.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T15:14:17.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:17.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:14:17.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:17 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:14:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:14:17.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:17 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:14:18.490 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:18 smithi096 ceph-mon[26688]: Detected new or changed devices on smithi196 2024-04-22T15:14:18.490 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:18 smithi096 ceph-mon[26688]: Adjusting osd_memory_target on smithi196 to 4970M 2024-04-22T15:14:18.490 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:18 smithi096 ceph-mon[26688]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:18.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.700+0000 7f2db8678700 1 -- 172.21.15.96:0/441996009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 msgr2=0x7f2db00fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:18.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.700+0000 7f2db8678700 1 --2- 172.21.15.96:0/441996009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 0x7f2db00fffb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009910 tx=0x7f2dac009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:18.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.700+0000 7f2db8678700 1 -- 172.21.15.96:0/441996009 shutdown_connections 2024-04-22T15:14:18.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.700+0000 7f2db8678700 1 --2- 172.21.15.96:0/441996009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db0100d90 0x7f2db01011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.700+0000 7f2db8678700 1 --2- 172.21.15.96:0/441996009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 0x7f2db00fffb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.700+0000 7f2db8678700 1 -- 172.21.15.96:0/441996009 >> 172.21.15.96:0/441996009 conn(0x7f2db00fb110 msgr2=0x7f2db00fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db8678700 1 -- 172.21.15.96:0/441996009 shutdown_connections 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db8678700 1 -- 172.21.15.96:0/441996009 wait complete. 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db8678700 1 Processor -- start 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db8678700 1 -- start start 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db8678700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 0x7f2db0195660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db8678700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db0100d90 0x7f2db0195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db8678700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db01961c0 con 0x7f2db00ffb90 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db8678700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db0196300 con 0x7f2db0100d90 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db6414700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 0x7f2db0195660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db6414700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 0x7f2db0195660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35554/0 (socket says 172.21.15.96:35554) 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.701+0000 7f2db6414700 1 -- 172.21.15.96:0/2075531389 learned_addr learned my addr 172.21.15.96:0/2075531389 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2db5c13700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db0100d90 0x7f2db0195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2db6414700 1 -- 172.21.15.96:0/2075531389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db0100d90 msgr2=0x7f2db0195ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2db6414700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db0100d90 0x7f2db0195ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2db6414700 1 -- 172.21.15.96:0/2075531389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac0095f0 con 0x7f2db00ffb90 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2db5c13700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db0100d90 0x7f2db0195ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2db6414700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 0x7f2db0195660 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2dac005850 tx=0x7f2dac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2da77fe700 1 -- 172.21.15.96:0/2075531389 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dac01d070 con 0x7f2db00ffb90 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2da77fe700 1 -- 172.21.15.96:0/2075531389 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dac00f460 con 0x7f2db00ffb90 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db019ad00 con 0x7f2db00ffb90 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2da77fe700 1 -- 172.21.15.96:0/2075531389 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dac021620 con 0x7f2db00ffb90 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.702+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db019b220 con 0x7f2db00ffb90 2024-04-22T15:14:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.703+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db018f790 con 0x7f2db00ffb90 2024-04-22T15:14:18.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.703+0000 7f2da77fe700 1 -- 172.21.15.96:0/2075531389 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2dac00f5d0 con 0x7f2db00ffb90 2024-04-22T15:14:18.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.704+0000 7f2da77fe700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d9c06cce0 0x7f2d9c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:18.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.704+0000 7f2db5c13700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d9c06cce0 0x7f2d9c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:18.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.704+0000 7f2da77fe700 1 -- 172.21.15.96:0/2075531389 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2dac08d260 con 0x7f2db00ffb90 2024-04-22T15:14:18.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.704+0000 7f2db5c13700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d9c06cce0 0x7f2d9c06f1a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2da0007900 tx=0x7f2da0008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:18.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.708+0000 7f2da77fe700 1 -- 172.21.15.96:0/2075531389 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2dac05b400 con 0x7f2db00ffb90 2024-04-22T15:14:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:18 smithi196 ceph-mon[34717]: Detected new or changed devices on smithi196 2024-04-22T15:14:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:18 smithi196 ceph-mon[34717]: Adjusting osd_memory_target on smithi196 to 4970M 2024-04-22T15:14:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:18 smithi196 ceph-mon[34717]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:18.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.900+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2db002cde0 con 0x7f2db00ffb90 2024-04-22T15:14:18.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.900+0000 7f2da77fe700 1 -- 172.21.15.96:0/2075531389 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f2dac05af90 con 0x7f2db00ffb90 2024-04-22T15:14:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:14:18.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d9c06cce0 msgr2=0x7f2d9c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:18.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d9c06cce0 0x7f2d9c06f1a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2da0007900 tx=0x7f2da0008040 comp rx=0 tx=0).stop 2024-04-22T15:14:18.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 msgr2=0x7f2db0195660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:18.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 0x7f2db0195660 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2dac005850 tx=0x7f2dac00be20 comp rx=0 tx=0).stop 2024-04-22T15:14:18.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 shutdown_connections 2024-04-22T15:14:18.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2d9c06cce0 0x7f2d9c06f1a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:18.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db00ffb90 0x7f2db0195660 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:18.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 --2- 172.21.15.96:0/2075531389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db0100d90 0x7f2db0195ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:18.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 >> 172.21.15.96:0/2075531389 conn(0x7f2db00fb110 msgr2=0x7f2db01043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:18.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 shutdown_connections 2024-04-22T15:14:18.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:18.903+0000 7f2db8678700 1 -- 172.21.15.96:0/2075531389 wait complete. 2024-04-22T15:14:19.572 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:19 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2075531389' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T15:14:19.594 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":39,"num_osds":6,"num_up_osds":6,"osd_up_since":1713798855,"num_in_osds":6,"osd_in_since":1713798836,"num_remapped_pgs":0} 2024-04-22T15:14:19.595 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T15:14:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:19 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2075531389' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T15:14:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:20.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:20 smithi096 ceph-mon[26688]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:20.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:20 smithi196 ceph-mon[34717]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.312+0000 7ffa1774a700 1 -- 172.21.15.96:0/478049105 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100fffe0 msgr2=0x7ffa10103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.312+0000 7ffa1774a700 1 --2- 172.21.15.96:0/478049105 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100fffe0 0x7ffa10103ed0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000aa20 tx=0x7ffa0000ad30 comp rx=0 tx=0).stop 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.312+0000 7ffa1774a700 1 -- 172.21.15.96:0/478049105 shutdown_connections 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.312+0000 7ffa1774a700 1 --2- 172.21.15.96:0/478049105 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100fffe0 0x7ffa10103ed0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.312+0000 7ffa1774a700 1 --2- 172.21.15.96:0/478049105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa100ff630 0x7ffa100ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.312+0000 7ffa1774a700 1 -- 172.21.15.96:0/478049105 >> 172.21.15.96:0/478049105 conn(0x7ffa10074d90 msgr2=0x7ffa100751a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.312+0000 7ffa1774a700 1 -- 172.21.15.96:0/478049105 shutdown_connections 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.312+0000 7ffa1774a700 1 -- 172.21.15.96:0/478049105 wait complete. 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa1774a700 1 Processor -- start 2024-04-22T15:14:21.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa1774a700 1 -- start start 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa1774a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100ff630 0x7ffa10198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa1774a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa100fffe0 0x7ffa101992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa1774a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa101999a0 con 0x7ffa100ff630 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa1774a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1019d670 con 0x7ffa100fffe0 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa154e6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100ff630 0x7ffa10198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa154e6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100ff630 0x7ffa10198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45098/0 (socket says 172.21.15.96:45098) 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa154e6700 1 -- 172.21.15.96:0/1068075972 learned_addr learned my addr 172.21.15.96:0/1068075972 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa14ce5700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa100fffe0 0x7ffa101992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.313+0000 7ffa154e6700 1 -- 172.21.15.96:0/1068075972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa100fffe0 msgr2=0x7ffa101992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.314+0000 7ffa154e6700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa100fffe0 0x7ffa101992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.314+0000 7ffa154e6700 1 -- 172.21.15.96:0/1068075972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0000a700 con 0x7ffa100ff630 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.314+0000 7ffa154e6700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100ff630 0x7ffa10198d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00b4b0 tx=0x7ffa0c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.314+0000 7ffa067fc700 1 -- 172.21.15.96:0/1068075972 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa0c011800 con 0x7ffa100ff630 2024-04-22T15:14:21.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.314+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1019d950 con 0x7ffa100ff630 2024-04-22T15:14:21.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.314+0000 7ffa067fc700 1 -- 172.21.15.96:0/1068075972 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa0c011e20 con 0x7ffa100ff630 2024-04-22T15:14:21.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.314+0000 7ffa067fc700 1 -- 172.21.15.96:0/1068075972 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa0c00f2e0 con 0x7ffa100ff630 2024-04-22T15:14:21.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.314+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1019de70 con 0x7ffa100ff630 2024-04-22T15:14:21.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.315+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa10107800 con 0x7ffa100ff630 2024-04-22T15:14:21.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.315+0000 7ffa067fc700 1 -- 172.21.15.96:0/1068075972 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ffa0c010380 con 0x7ffa100ff630 2024-04-22T15:14:21.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.316+0000 7ffa067fc700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9fc06cce0 0x7ff9fc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:21.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.316+0000 7ffa067fc700 1 -- 172.21.15.96:0/1068075972 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffa0c08bae0 con 0x7ffa100ff630 2024-04-22T15:14:21.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.316+0000 7ffa14ce5700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9fc06cce0 0x7ff9fc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:21.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.316+0000 7ffa14ce5700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9fc06cce0 0x7ff9fc06f1a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000afd0 tx=0x7ffa0000c650 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:21.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.320+0000 7ffa067fc700 1 -- 172.21.15.96:0/1068075972 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ffa0c0561f0 con 0x7ffa100ff630 2024-04-22T15:14:21.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.512+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa101009b0 con 0x7ffa100ff630 2024-04-22T15:14:21.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.513+0000 7ffa067fc700 1 -- 172.21.15.96:0/1068075972 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+11348 (secure 0 0 0) 0x7ffa0c059810 con 0x7ffa100ff630 2024-04-22T15:14:21.517 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:14:21.517 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":39,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","created":"2024-04-22T15:08:23.492994+0000","modified":"2024-04-22T15:14:16.338041+0000","last_up_change":"2024-04-22T15:14:15.336641+0000","last_in_change":"2024-04-22T15:13:56.089739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:13:01.924433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2f1b7772-06b4-467f-b14c-698a0de7357f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6803","nonce":1197486134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6805","nonce":1197486134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6809","nonce":1197486134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6807","nonce":1197486134}]},"public_addr":"172.21.15.96:6803/1197486134","cluster_addr":"172.21.15.96:6805/1197486134","heartbeat_back_addr":"172.21.15.96:6809/1197486134","heartbeat_front_addr":"172.21.15.96:6807/1197486134","state":["exists","up"]},{"osd":1,"uuid":"59260a36-a2ed-42f9-8f3a-209aae4c4238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6811","nonce":3577737028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6813","nonce":3577737028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6817","nonce":3577737028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6815","nonce":3577737028}]},"public_addr":"172.21.15.96:6811/3577737028","cluster_addr":"172.21.15.96:6813/3577737028","heartbeat_back_addr":"172.21.15.96:6817/3577737028","heartbeat_front_addr":"172.21.15.96:6815/3577737028","state":["exists","up"]},{"osd":2,"uuid":"897e21a9-3009-41c6-acf1-e44414fc69c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6819","nonce":1428742985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6821","nonce":1428742985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6825","nonce":1428742985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6823","nonce":1428742985}]},"public_addr":"172.21.15.96:6819/1428742985","cluster_addr":"172.21.15.96:6821/1428742985","heartbeat_back_addr":"172.21.15.96:6825/1428742985","heartbeat_front_addr":"172.21.15.96:6823/1428742985","state":["exists","up"]},{"osd":3,"uuid":"201c3153-a284-471e-894d-2273001cc29e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6801","nonce":1414997674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6803","nonce":1414997674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6807","nonce":1414997674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6805","nonce":1414997674}]},"public_addr":"172.21.15.196:6801/1414997674","cluster_addr":"172.21.15.196:6803/1414997674","heartbeat_back_addr":"172.21.15.196:6807/1414997674","heartbeat_front_addr":"172.21.15.196:6805/1414997674","state":["exists","up"]},{"osd":4,"uuid":"780ed007-14ef-4c92-a5e9-6c57afcad448","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6809","nonce":1827225092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6811","nonce":1827225092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6815","nonce":1827225092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6813","nonce":1827225092}]},"public_addr":"172.21.15.196:6809/1827225092","cluster_addr":"172.21.15.196:6811/1827225092","heartbeat_back_addr":"172.21.15.196:6815/1827225092","heartbeat_front_addr":"172.21.15.196:6813/1827225092","state":["exists","up"]},{"osd":5,"uuid":"16384d8d-bef4-4f65-9079-e2b4518241da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6817","nonce":2750420250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6819","nonce":2750420250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6823","nonce":2750420250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6821","nonce":2750420250}]},"public_addr":"172.21.15.196:6817/2750420250","cluster_addr":"172.21.15.196:6819/2750420250","heartbeat_back_addr":"172.21.15.196:6823/2750420250","heartbeat_front_addr":"172.21.15.196:6821/2750420250","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:00.654872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:28.691804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:57.737056+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:21.679902+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:47.489372+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:14:13.600900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:0/3921802975":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6801/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6800/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3486391579":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/1794679103":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3321679733":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6800/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:0/4224647460":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1237353220":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/838799683":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3667378212":"2024-04-23T15:08:51.463883+0000","172.21.15.96:6800/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1319578885":"2024-04-23T15:09:23.920294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T15:14:21.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.515+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9fc06cce0 msgr2=0x7ff9fc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:21.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.515+0000 7ffa1774a700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9fc06cce0 0x7ff9fc06f1a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000afd0 tx=0x7ffa0000c650 comp rx=0 tx=0).stop 2024-04-22T15:14:21.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.515+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100ff630 msgr2=0x7ffa10198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:21.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.515+0000 7ffa1774a700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100ff630 0x7ffa10198d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00b4b0 tx=0x7ffa0c00b870 comp rx=0 tx=0).stop 2024-04-22T15:14:21.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.516+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 shutdown_connections 2024-04-22T15:14:21.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.516+0000 7ffa1774a700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9fc06cce0 0x7ff9fc06f1a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:21.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.516+0000 7ffa1774a700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa100ff630 0x7ffa10198d80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:21.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.516+0000 7ffa1774a700 1 --2- 172.21.15.96:0/1068075972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa100fffe0 0x7ffa101992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:21.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.516+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 >> 172.21.15.96:0/1068075972 conn(0x7ffa10074d90 msgr2=0x7ffa100fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:21.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.516+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 shutdown_connections 2024-04-22T15:14:21.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:21.516+0000 7ffa1774a700 1 -- 172.21.15.96:0/1068075972 wait complete. 2024-04-22T15:14:21.820 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:21 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1068075972' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:14:21.937 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:21 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1068075972' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:14:22.150 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:13:01.924433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T15:14:22.150 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T15:14:22.355 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:22.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:22 smithi096 ceph-mon[26688]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:23.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:22 smithi196 ceph-mon[34717]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:23.824 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:23 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:14:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:23 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:14:24.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.072+0000 7fc25fbb2700 1 -- 172.21.15.96:0/3498170138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2581026f0 msgr2=0x7fc25810abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:24.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.072+0000 7fc25fbb2700 1 --2- 172.21.15.96:0/3498170138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2581026f0 0x7fc25810abe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc248009940 tx=0x7fc248009c50 comp rx=0 tx=0).stop 2024-04-22T15:14:24.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.072+0000 7fc25fbb2700 1 -- 172.21.15.96:0/3498170138 shutdown_connections 2024-04-22T15:14:24.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.072+0000 7fc25fbb2700 1 --2- 172.21.15.96:0/3498170138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2581026f0 0x7fc25810abe0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:24.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.072+0000 7fc25fbb2700 1 --2- 172.21.15.96:0/3498170138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc258101dd0 0x7fc2581021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.072+0000 7fc25fbb2700 1 -- 172.21.15.96:0/3498170138 >> 172.21.15.96:0/3498170138 conn(0x7fc2580fb5e0 msgr2=0x7fc2580fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.072+0000 7fc25fbb2700 1 -- 172.21.15.96:0/3498170138 shutdown_connections 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.072+0000 7fc25fbb2700 1 -- 172.21.15.96:0/3498170138 wait complete. 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25fbb2700 1 Processor -- start 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25fbb2700 1 -- start start 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25fbb2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc258101dd0 0x7fc25819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25fbb2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2581026f0 0x7fc25819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25fbb2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25819d470 con 0x7fc258101dd0 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25fbb2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc258196a50 con 0x7fc2581026f0 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25d94e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc258101dd0 0x7fc25819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25d94e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc258101dd0 0x7fc25819c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45116/0 (socket says 172.21.15.96:45116) 2024-04-22T15:14:24.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25d94e700 1 -- 172.21.15.96:0/4138837918 learned_addr learned my addr 172.21.15.96:0/4138837918 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25d14d700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2581026f0 0x7fc25819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25d94e700 1 -- 172.21.15.96:0/4138837918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2581026f0 msgr2=0x7fc25819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25d94e700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2581026f0 0x7fc25819ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25d94e700 1 -- 172.21.15.96:0/4138837918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2480095f0 con 0x7fc258101dd0 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.073+0000 7fc25d94e700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc258101dd0 0x7fc25819c910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc25400b4b0 tx=0x7fc25400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.074+0000 7fc24effd700 1 -- 172.21.15.96:0/4138837918 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc254011800 con 0x7fc258101dd0 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.074+0000 7fc24effd700 1 -- 172.21.15.96:0/4138837918 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc254011e20 con 0x7fc258101dd0 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.074+0000 7fc24effd700 1 -- 172.21.15.96:0/4138837918 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25400f2e0 con 0x7fc258101dd0 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.074+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc258196d30 con 0x7fc258101dd0 2024-04-22T15:14:24.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.074+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc258197250 con 0x7fc258101dd0 2024-04-22T15:14:24.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.075+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc258105560 con 0x7fc258101dd0 2024-04-22T15:14:24.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.076+0000 7fc24effd700 1 -- 172.21.15.96:0/4138837918 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc254010380 con 0x7fc258101dd0 2024-04-22T15:14:24.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.076+0000 7fc24effd700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc24406cce0 0x7fc24406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:24.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.076+0000 7fc24effd700 1 -- 172.21.15.96:0/4138837918 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc25408b660 con 0x7fc258101dd0 2024-04-22T15:14:24.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.076+0000 7fc25d14d700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc24406cce0 0x7fc24406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:24.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.076+0000 7fc25d14d700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc24406cce0 0x7fc24406f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc24800b580 tx=0x7fc24800b790 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:24.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.080+0000 7fc24effd700 1 -- 172.21.15.96:0/4138837918 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fc254059930 con 0x7fc258101dd0 2024-04-22T15:14:24.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.268+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc258197f10 con 0x7fc258101dd0 2024-04-22T15:14:24.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.269+0000 7fc24effd700 1 -- 172.21.15.96:0/4138837918 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc2540594c0 con 0x7fc258101dd0 2024-04-22T15:14:24.272 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc24406cce0 msgr2=0x7fc24406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc24406cce0 0x7fc24406f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc24800b580 tx=0x7fc24800b790 comp rx=0 tx=0).stop 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc258101dd0 msgr2=0x7fc25819c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc258101dd0 0x7fc25819c910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc25400b4b0 tx=0x7fc25400b7c0 comp rx=0 tx=0).stop 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 shutdown_connections 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc24406cce0 0x7fc24406f1a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc258101dd0 0x7fc25819c910 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 --2- 172.21.15.96:0/4138837918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2581026f0 0x7fc25819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 >> 172.21.15.96:0/4138837918 conn(0x7fc2580fb5e0 msgr2=0x7fc2581030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 shutdown_connections 2024-04-22T15:14:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:24.271+0000 7fc25fbb2700 1 -- 172.21.15.96:0/4138837918 wait complete. 2024-04-22T15:14:24.853 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:24 smithi096 ceph-mon[26688]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:24.853 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:24 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/4138837918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T15:14:24.876 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T15:14:24.877 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T15:14:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:24 smithi196 ceph-mon[34717]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:24 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/4138837918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T15:14:25.081 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:26.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.669+0000 7f31ffbde700 1 -- 172.21.15.96:0/1499987240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 msgr2=0x7f31f8102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:26.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.669+0000 7f31ffbde700 1 --2- 172.21.15.96:0/1499987240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 0x7f31f8102270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f31e8009910 tx=0x7f31e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:26.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.669+0000 7f31ffbde700 1 -- 172.21.15.96:0/1499987240 shutdown_connections 2024-04-22T15:14:26.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.669+0000 7f31ffbde700 1 --2- 172.21.15.96:0/1499987240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 0x7f31f8102270 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:26.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.669+0000 7f31ffbde700 1 --2- 172.21.15.96:0/1499987240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f31f8100c10 0x7f31f8101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:26.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.669+0000 7f31ffbde700 1 -- 172.21.15.96:0/1499987240 >> 172.21.15.96:0/1499987240 conn(0x7f31f80fc1f0 msgr2=0x7f31f80fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.669+0000 7f31ffbde700 1 -- 172.21.15.96:0/1499987240 shutdown_connections 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.669+0000 7f31ffbde700 1 -- 172.21.15.96:0/1499987240 wait complete. 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31ffbde700 1 Processor -- start 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31ffbde700 1 -- start start 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31ffbde700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f31f8100c10 0x7f31f8196730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31ffbde700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 0x7f31f8196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31ffbde700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8197290 con 0x7f31f8101e10 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31ffbde700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f81973d0 con 0x7f31f8100c10 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31fd179700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 0x7f31f8196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31fd179700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 0x7f31f8196c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45132/0 (socket says 172.21.15.96:45132) 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31fd179700 1 -- 172.21.15.96:0/3080691573 learned_addr learned my addr 172.21.15.96:0/3080691573 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31fd97a700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f31f8100c10 0x7f31f8196730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31fd179700 1 -- 172.21.15.96:0/3080691573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f31f8100c10 msgr2=0x7f31f8196730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31fd179700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f31f8100c10 0x7f31f8196730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31fd179700 1 -- 172.21.15.96:0/3080691573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e80095f0 con 0x7f31f8101e10 2024-04-22T15:14:26.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31fd179700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 0x7f31f8196c70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f31e8006010 tx=0x7f31e800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:26.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31eeffd700 1 -- 172.21.15.96:0/3080691573 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31e801d070 con 0x7f31f8101e10 2024-04-22T15:14:26.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f819bd80 con 0x7f31f8101e10 2024-04-22T15:14:26.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31eeffd700 1 -- 172.21.15.96:0/3080691573 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31e800f460 con 0x7f31f8101e10 2024-04-22T15:14:26.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.670+0000 7f31eeffd700 1 -- 172.21.15.96:0/3080691573 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31e8021640 con 0x7f31f8101e10 2024-04-22T15:14:26.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.671+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f819c2f0 con 0x7f31f8101e10 2024-04-22T15:14:26.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.672+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f8190830 con 0x7f31f8101e10 2024-04-22T15:14:26.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.672+0000 7f31eeffd700 1 -- 172.21.15.96:0/3080691573 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f31e800fad0 con 0x7f31f8101e10 2024-04-22T15:14:26.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.672+0000 7f31eeffd700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f31e406ccf0 0x7f31e406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:26.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.673+0000 7f31eeffd700 1 -- 172.21.15.96:0/3080691573 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f31e808d2a0 con 0x7f31f8101e10 2024-04-22T15:14:26.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.673+0000 7f31fd97a700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f31e406ccf0 0x7f31e406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:26.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.673+0000 7f31fd97a700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f31e406ccf0 0x7f31e406f1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f31f4006fd0 tx=0x7f31f4008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:26.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.677+0000 7f31eeffd700 1 -- 172.21.15.96:0/3080691573 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f31e805b440 con 0x7f31f8101e10 2024-04-22T15:14:26.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.948+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f31f802cde0 con 0x7f31f8101e10 2024-04-22T15:14:26.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.954+0000 7f31eeffd700 1 -- 172.21.15.96:0/3080691573 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f31e805afd0 con 0x7f31f8101e10 2024-04-22T15:14:26.958 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-04-22T15:14:26.958 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQDSfiZmN7qaOBAA6GDMQ/bgl9NvhsThOTEv8g== 2024-04-22T15:14:26.958 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:26 smithi096 ceph-mon[26688]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:26.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f31e406ccf0 msgr2=0x7f31e406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f31e406ccf0 0x7f31e406f1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f31f4006fd0 tx=0x7f31f4008040 comp rx=0 tx=0).stop 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 msgr2=0x7f31f8196c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 0x7f31f8196c70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f31e8006010 tx=0x7f31e800be20 comp rx=0 tx=0).stop 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 shutdown_connections 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f31e406ccf0 0x7f31e406f1b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f31f8100c10 0x7f31f8196730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 --2- 172.21.15.96:0/3080691573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31f8101e10 0x7f31f8196c70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 >> 172.21.15.96:0/3080691573 conn(0x7f31f80fc1f0 msgr2=0x7f31f8105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 shutdown_connections 2024-04-22T15:14:26.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:26.956+0000 7f31ffbde700 1 -- 172.21.15.96:0/3080691573 wait complete. 2024-04-22T15:14:26.976 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:26 smithi196 ceph-mon[34717]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:27.648 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:14:27.648 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T15:14:27.648 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T15:14:27.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T15:14:27.899 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:14:27.910 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:27 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3080691573' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:14:27.910 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:27 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3080691573' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T15:14:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:27 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3080691573' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:14:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:27 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3080691573' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T15:14:28.970 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:28 smithi196 ceph-mon[34717]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:28 smithi096 ceph-mon[26688]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:29.333 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 -- 172.21.15.196:0/574788248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 msgr2=0x7fead41002f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:29.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 --2- 172.21.15.196:0/574788248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 0x7fead41002f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feac8009910 tx=0x7feac8009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:29.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 -- 172.21.15.196:0/574788248 shutdown_connections 2024-04-22T15:14:29.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 --2- 172.21.15.196:0/574788248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 0x7fead41002f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:29.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 --2- 172.21.15.196:0/574788248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead40ff510 0x7fead40ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:29.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 -- 172.21.15.196:0/574788248 >> 172.21.15.196:0/574788248 conn(0x7fead40fb090 msgr2=0x7fead40fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:29.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 -- 172.21.15.196:0/574788248 shutdown_connections 2024-04-22T15:14:29.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 -- 172.21.15.196:0/574788248 wait complete. 2024-04-22T15:14:29.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 Processor -- start 2024-04-22T15:14:29.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.298+0000 7feada405700 1 -- start start 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7feada405700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead40ff510 0x7fead4198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7feada405700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 0x7fead4198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7feada405700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead4199390 con 0x7fead40ff510 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7feada405700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead41994d0 con 0x7fead40ffe70 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead37fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 0x7fead4198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead37fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 0x7fead4198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:37580/0 (socket says 172.21.15.196:37580) 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead37fe700 1 -- 172.21.15.196:0/3319863804 learned_addr learned my addr 172.21.15.196:0/3319863804 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead3fff700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead40ff510 0x7fead4198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead37fe700 1 -- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead40ff510 msgr2=0x7fead4198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead37fe700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead40ff510 0x7fead4198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead37fe700 1 -- 172.21.15.196:0/3319863804 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac80095f0 con 0x7fead40ffe70 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead37fe700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 0x7fead4198d70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feac80092e0 tx=0x7feac800be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead17fa700 1 -- 172.21.15.196:0/3319863804 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feac801d070 con 0x7fead40ffe70 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead419de80 con 0x7fead40ffe70 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead17fa700 1 -- 172.21.15.196:0/3319863804 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feac800f460 con 0x7fead40ffe70 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7fead17fa700 1 -- 172.21.15.196:0/3319863804 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feac8021640 con 0x7fead40ffe70 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.299+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead419e3c0 con 0x7fead40ffe70 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.300+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead4192990 con 0x7fead40ffe70 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.301+0000 7fead17fa700 1 -- 172.21.15.196:0/3319863804 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7feac800f5d0 con 0x7fead40ffe70 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.301+0000 7fead17fa700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7feabc06cce0 0x7feabc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.301+0000 7fead17fa700 1 -- 172.21.15.196:0/3319863804 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7feac808d290 con 0x7fead40ffe70 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.301+0000 7fead3fff700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7feabc06cce0 0x7feabc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.302+0000 7fead3fff700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7feabc06cce0 0x7feabc06f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feac4005950 tx=0x7feac4005cf0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.306+0000 7fead17fa700 1 -- 172.21.15.196:0/3319863804 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7feac805b430 con 0x7fead40ffe70 2024-04-22T15:14:29.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.575+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fead404ebb0 con 0x7fead40ffe70 2024-04-22T15:14:29.584 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.581+0000 7fead17fa700 1 -- 172.21.15.196:0/3319863804 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7feac805afc0 con 0x7fead40ffe70 2024-04-22T15:14:29.585 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-04-22T15:14:29.585 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDVfiZmIZdtIhAAa23ZSw+yUNxrHL39tBg5+Q== 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7feabc06cce0 msgr2=0x7feabc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7feabc06cce0 0x7feabc06f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feac4005950 tx=0x7feac4005cf0 comp rx=0 tx=0).stop 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 msgr2=0x7fead4198d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 0x7fead4198d70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feac80092e0 tx=0x7feac800be20 comp rx=0 tx=0).stop 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 shutdown_connections 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7feabc06cce0 0x7feabc06f1a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead40ff510 0x7fead4198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 --2- 172.21.15.196:0/3319863804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fead40ffe70 0x7fead4198d70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 >> 172.21.15.196:0/3319863804 conn(0x7fead40fb090 msgr2=0x7fead4107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:29.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 shutdown_connections 2024-04-22T15:14:29.588 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:14:29.583+0000 7feada405700 1 -- 172.21.15.196:0/3319863804 wait complete. 2024-04-22T15:14:29.937 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:29 smithi196 ceph-mon[34717]: from='client.? 172.21.15.196:0/3319863804' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:14:29.937 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:29 smithi196 ceph-mon[34717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:14:29.937 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:29 smithi196 ceph-mon[34717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T15:14:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:29 smithi096 ceph-mon[26688]: from='client.? 172.21.15.196:0/3319863804' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:14:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:29 smithi096 ceph-mon[26688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:14:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:29 smithi096 ceph-mon[26688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T15:14:30.244 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:14:30.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-22T15:14:30.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T15:14:30.280 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T15:14:30.280 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T15:14:30.280 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T15:14:30.497 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:30.953 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:30 smithi096 ceph-mon[26688]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:30 smithi196 ceph-mon[34717]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:32.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.206+0000 7f0fd77e6700 1 -- 172.21.15.96:0/4089744850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd00ffaa0 msgr2=0x7f0fd00fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:32.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.206+0000 7f0fd77e6700 1 --2- 172.21.15.96:0/4089744850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd00ffaa0 0x7f0fd00fff20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009910 tx=0x7f0fc0009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:32.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.206+0000 7f0fd77e6700 1 -- 172.21.15.96:0/4089744850 shutdown_connections 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.206+0000 7f0fd77e6700 1 --2- 172.21.15.96:0/4089744850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd00ffaa0 0x7f0fd00fff20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.206+0000 7f0fd77e6700 1 --2- 172.21.15.96:0/4089744850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fd0105aa0 0x7f0fd0105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.206+0000 7f0fd77e6700 1 -- 172.21.15.96:0/4089744850 >> 172.21.15.96:0/4089744850 conn(0x7f0fd0076740 msgr2=0x7f0fd00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.206+0000 7f0fd77e6700 1 -- 172.21.15.96:0/4089744850 shutdown_connections 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.206+0000 7f0fd77e6700 1 -- 172.21.15.96:0/4089744850 wait complete. 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd77e6700 1 Processor -- start 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd77e6700 1 -- start start 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd77e6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fd00ffaa0 0x7f0fd0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd77e6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd0105aa0 0x7f0fd0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd77e6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0198510 con 0x7f0fd0105aa0 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd77e6700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0198650 con 0x7f0fd00ffaa0 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd4d81700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd0105aa0 0x7f0fd0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:32.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd4d81700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd0105aa0 0x7f0fd0197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32786/0 (socket says 172.21.15.96:32786) 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd4d81700 1 -- 172.21.15.96:0/2270548528 learned_addr learned my addr 172.21.15.96:0/2270548528 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd5582700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fd00ffaa0 0x7f0fd0197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd4d81700 1 -- 172.21.15.96:0/2270548528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fd00ffaa0 msgr2=0x7f0fd0197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd4d81700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fd00ffaa0 0x7f0fd0197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.207+0000 7f0fd4d81700 1 -- 172.21.15.96:0/2270548528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc00095f0 con 0x7f0fd0105aa0 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.208+0000 7f0fd4d81700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd0105aa0 0x7f0fd0197f40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0006010 tx=0x7f0fc000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.208+0000 7f0fc67fc700 1 -- 172.21.15.96:0/2270548528 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fc001d070 con 0x7f0fd0105aa0 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.208+0000 7f0fc67fc700 1 -- 172.21.15.96:0/2270548528 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fc0005700 con 0x7f0fd0105aa0 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.208+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd0198870 con 0x7f0fd0105aa0 2024-04-22T15:14:32.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.208+0000 7f0fc67fc700 1 -- 172.21.15.96:0/2270548528 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fc0017400 con 0x7f0fd0105aa0 2024-04-22T15:14:32.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.208+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd01a0dd0 con 0x7f0fd0105aa0 2024-04-22T15:14:32.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.209+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd01080c0 con 0x7f0fd0105aa0 2024-04-22T15:14:32.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.209+0000 7f0fc67fc700 1 -- 172.21.15.96:0/2270548528 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0fc000f460 con 0x7f0fd0105aa0 2024-04-22T15:14:32.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.210+0000 7f0fc67fc700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f0fbc06cce0 0x7f0fbc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:32.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.210+0000 7f0fc67fc700 1 -- 172.21.15.96:0/2270548528 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0fc008d180 con 0x7f0fd0105aa0 2024-04-22T15:14:32.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.210+0000 7f0fd5582700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f0fbc06cce0 0x7f0fbc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:32.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.210+0000 7f0fd5582700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f0fbc06cce0 0x7f0fbc06f1a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800a400 tx=0x7f0fc800d040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:32.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.214+0000 7f0fc67fc700 1 -- 172.21.15.96:0/2270548528 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f0fc0057890 con 0x7f0fd0105aa0 2024-04-22T15:14:32.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.460+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0fd0198870 con 0x7f0fd0105aa0 2024-04-22T15:14:32.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.464+0000 7f0fc67fc700 1 -- 172.21.15.96:0/2270548528 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+174011 (secure 0 0 0) 0x7f0fd0198870 con 0x7f0fd0105aa0 2024-04-22T15:14:32.470 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:14:32.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f0fbc06cce0 msgr2=0x7f0fbc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:32.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f0fbc06cce0 0x7f0fbc06f1a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800a400 tx=0x7f0fc800d040 comp rx=0 tx=0).stop 2024-04-22T15:14:32.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd0105aa0 msgr2=0x7f0fd0197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:32.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd0105aa0 0x7f0fd0197f40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0006010 tx=0x7f0fc000bd70 comp rx=0 tx=0).stop 2024-04-22T15:14:32.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 shutdown_connections 2024-04-22T15:14:32.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f0fbc06cce0 0x7f0fbc06f1a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:32.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fd00ffaa0 0x7f0fd0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:32.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 --2- 172.21.15.96:0/2270548528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fd0105aa0 0x7f0fd0197f40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:32.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 >> 172.21.15.96:0/2270548528 conn(0x7f0fd0076740 msgr2=0x7f0fd00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:32.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 shutdown_connections 2024-04-22T15:14:32.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:32.467+0000 7f0fd77e6700 1 -- 172.21.15.96:0/2270548528 wait complete. 2024-04-22T15:14:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:32 smithi096 ceph-mon[26688]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:32 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2270548528' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T15:14:33.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:32 smithi196 ceph-mon[34717]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:33.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:32 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2270548528' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T15:14:33.178 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":19,"flags":0,"active_gid":14217,"active_name":"smithi096.hhkvum","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2441203700},{"type":"v1","addr":"172.21.15.96:6801","nonce":2441203700}]},"active_addr":"172.21.15.96:6801/2441203700","active_change":"2024-04-22T15:10:37.605445+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14244,"name":"smithi196.gdaegc","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.96:8443/","prometheus":"http://172.21.15.96:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":340578852}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1860450164}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1920801848}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":168995622}]}]} 2024-04-22T15:14:33.182 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T15:14:33.182 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T15:14:33.182 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T15:14:33.386 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:34.852 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:34 smithi096 ceph-mon[26688]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:34 smithi196 ceph-mon[34717]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.080+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/173123545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 msgr2=0x7fa99c105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.080+0000 7fa9a2ea4700 1 --2- 172.21.15.96:0/173123545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 0x7fa99c105f40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa990009910 tx=0x7fa990009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.080+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/173123545 shutdown_connections 2024-04-22T15:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.080+0000 7fa9a2ea4700 1 --2- 172.21.15.96:0/173123545 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa99c0ffad0 0x7fa99c0fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.080+0000 7fa9a2ea4700 1 --2- 172.21.15.96:0/173123545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 0x7fa99c105f40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.080+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/173123545 >> 172.21.15.96:0/173123545 conn(0x7fa99c0fb5e0 msgr2=0x7fa99c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.080+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/173123545 shutdown_connections 2024-04-22T15:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.080+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/173123545 wait complete. 2024-04-22T15:14:35.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa9a2ea4700 1 Processor -- start 2024-04-22T15:14:35.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa9a2ea4700 1 -- start start 2024-04-22T15:14:35.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa9a2ea4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa99c0ffad0 0x7fa99c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:35.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa9a2ea4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 0x7fa99c19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:35.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa9a2ea4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c19d390 con 0x7fa99c105b60 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa9a2ea4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c196970 con 0x7fa99c0ffad0 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa99bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 0x7fa99c19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa99bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 0x7fa99c19cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32798/0 (socket says 172.21.15.96:32798) 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa99bfff700 1 -- 172.21.15.96:0/1297183073 learned_addr learned my addr 172.21.15.96:0/1297183073 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa9a0c40700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa99c0ffad0 0x7fa99c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.081+0000 7fa99bfff700 1 -- 172.21.15.96:0/1297183073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa99c0ffad0 msgr2=0x7fa99c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.082+0000 7fa99bfff700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa99c0ffad0 0x7fa99c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.082+0000 7fa99bfff700 1 -- 172.21.15.96:0/1297183073 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9900095f0 con 0x7fa99c105b60 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.082+0000 7fa99bfff700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 0x7fa99c19cd70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa98800c8a0 tx=0x7fa98800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.082+0000 7fa999ffb700 1 -- 172.21.15.96:0/1297183073 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9880078e0 con 0x7fa99c105b60 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.082+0000 7fa999ffb700 1 -- 172.21.15.96:0/1297183073 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa98800f450 con 0x7fa99c105b60 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.082+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c196c50 con 0x7fa99c105b60 2024-04-22T15:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.082+0000 7fa999ffb700 1 -- 172.21.15.96:0/1297183073 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa98800e5e0 con 0x7fa99c105b60 2024-04-22T15:14:35.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.082+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c1971a0 con 0x7fa99c105b60 2024-04-22T15:14:35.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.083+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c1090e0 con 0x7fa99c105b60 2024-04-22T15:14:35.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.083+0000 7fa999ffb700 1 -- 172.21.15.96:0/1297183073 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa98800f5c0 con 0x7fa99c105b60 2024-04-22T15:14:35.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.084+0000 7fa999ffb700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa98c06cce0 0x7fa98c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:35.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.084+0000 7fa9a0c40700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa98c06cce0 0x7fa98c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:35.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.084+0000 7fa999ffb700 1 -- 172.21.15.96:0/1297183073 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa98808bee0 con 0x7fa99c105b60 2024-04-22T15:14:35.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.084+0000 7fa9a0c40700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa98c06cce0 0x7fa98c06f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa990000c00 tx=0x7fa99001a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:35.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.089+0000 7fa999ffb700 1 -- 172.21.15.96:0/1297183073 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa98805a080 con 0x7fa99c105b60 2024-04-22T15:14:35.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.283+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa99c1011d0 con 0x7fa99c105b60 2024-04-22T15:14:35.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.283+0000 7fa999ffb700 1 -- 172.21.15.96:0/1297183073 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+11348 (secure 0 0 0) 0x7fa988059c10 con 0x7fa99c105b60 2024-04-22T15:14:35.287 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:14:35.288 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":39,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","created":"2024-04-22T15:08:23.492994+0000","modified":"2024-04-22T15:14:16.338041+0000","last_up_change":"2024-04-22T15:14:15.336641+0000","last_in_change":"2024-04-22T15:13:56.089739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:13:01.924433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2f1b7772-06b4-467f-b14c-698a0de7357f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6803","nonce":1197486134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6805","nonce":1197486134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6809","nonce":1197486134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6807","nonce":1197486134}]},"public_addr":"172.21.15.96:6803/1197486134","cluster_addr":"172.21.15.96:6805/1197486134","heartbeat_back_addr":"172.21.15.96:6809/1197486134","heartbeat_front_addr":"172.21.15.96:6807/1197486134","state":["exists","up"]},{"osd":1,"uuid":"59260a36-a2ed-42f9-8f3a-209aae4c4238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6811","nonce":3577737028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6813","nonce":3577737028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6817","nonce":3577737028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6815","nonce":3577737028}]},"public_addr":"172.21.15.96:6811/3577737028","cluster_addr":"172.21.15.96:6813/3577737028","heartbeat_back_addr":"172.21.15.96:6817/3577737028","heartbeat_front_addr":"172.21.15.96:6815/3577737028","state":["exists","up"]},{"osd":2,"uuid":"897e21a9-3009-41c6-acf1-e44414fc69c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6819","nonce":1428742985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6821","nonce":1428742985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6825","nonce":1428742985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6823","nonce":1428742985}]},"public_addr":"172.21.15.96:6819/1428742985","cluster_addr":"172.21.15.96:6821/1428742985","heartbeat_back_addr":"172.21.15.96:6825/1428742985","heartbeat_front_addr":"172.21.15.96:6823/1428742985","state":["exists","up"]},{"osd":3,"uuid":"201c3153-a284-471e-894d-2273001cc29e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6801","nonce":1414997674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6803","nonce":1414997674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6807","nonce":1414997674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6805","nonce":1414997674}]},"public_addr":"172.21.15.196:6801/1414997674","cluster_addr":"172.21.15.196:6803/1414997674","heartbeat_back_addr":"172.21.15.196:6807/1414997674","heartbeat_front_addr":"172.21.15.196:6805/1414997674","state":["exists","up"]},{"osd":4,"uuid":"780ed007-14ef-4c92-a5e9-6c57afcad448","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6809","nonce":1827225092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6811","nonce":1827225092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6815","nonce":1827225092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6813","nonce":1827225092}]},"public_addr":"172.21.15.196:6809/1827225092","cluster_addr":"172.21.15.196:6811/1827225092","heartbeat_back_addr":"172.21.15.196:6815/1827225092","heartbeat_front_addr":"172.21.15.196:6813/1827225092","state":["exists","up"]},{"osd":5,"uuid":"16384d8d-bef4-4f65-9079-e2b4518241da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6817","nonce":2750420250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6819","nonce":2750420250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6823","nonce":2750420250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6821","nonce":2750420250}]},"public_addr":"172.21.15.196:6817/2750420250","cluster_addr":"172.21.15.196:6819/2750420250","heartbeat_back_addr":"172.21.15.196:6823/2750420250","heartbeat_front_addr":"172.21.15.196:6821/2750420250","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:00.654872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:28.691804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:57.737056+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:21.679902+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:47.489372+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:14:13.600900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:0/3921802975":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6801/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6800/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3486391579":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/1794679103":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3321679733":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6800/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:0/4224647460":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1237353220":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/838799683":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3667378212":"2024-04-23T15:08:51.463883+0000","172.21.15.96:6800/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1319578885":"2024-04-23T15:09:23.920294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T15:14:35.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa98c06cce0 msgr2=0x7fa98c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:35.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa98c06cce0 0x7fa98c06f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa990000c00 tx=0x7fa99001a040 comp rx=0 tx=0).stop 2024-04-22T15:14:35.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 msgr2=0x7fa99c19cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:35.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 0x7fa99c19cd70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa98800c8a0 tx=0x7fa98800cbb0 comp rx=0 tx=0).stop 2024-04-22T15:14:35.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 shutdown_connections 2024-04-22T15:14:35.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa98c06cce0 0x7fa98c06f1a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:35.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa99c0ffad0 0x7fa99c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:35.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 --2- 172.21.15.96:0/1297183073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa99c105b60 0x7fa99c19cd70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:35.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 >> 172.21.15.96:0/1297183073 conn(0x7fa99c0fb5e0 msgr2=0x7fa99c069650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:35.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 shutdown_connections 2024-04-22T15:14:35.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:35.286+0000 7fa9a2ea4700 1 -- 172.21.15.96:0/1297183073 wait complete. 2024-04-22T15:14:35.955 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T15:14:35.955 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T15:14:35.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:35 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1297183073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:14:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:35 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1297183073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:14:36.160 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:36.918 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:36 smithi096 ceph-mon[26688]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:37.003 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:36 smithi196 ceph-mon[34717]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.828+0000 7fa6ccf40700 1 -- 172.21.15.96:0/173606361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 msgr2=0x7fa6c8106030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.828+0000 7fa6ccf40700 1 --2- 172.21.15.96:0/173606361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 0x7fa6c8106030 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000b0a0 tx=0x7fa6c000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.828+0000 7fa6ccf40700 1 -- 172.21.15.96:0/173606361 shutdown_connections 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.828+0000 7fa6ccf40700 1 --2- 172.21.15.96:0/173606361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6c81001c0 0x7fa6c8100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.828+0000 7fa6ccf40700 1 --2- 172.21.15.96:0/173606361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 0x7fa6c8106030 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.828+0000 7fa6ccf40700 1 -- 172.21.15.96:0/173606361 >> 172.21.15.96:0/173606361 conn(0x7fa6c8075f30 msgr2=0x7fa6c80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.828+0000 7fa6ccf40700 1 -- 172.21.15.96:0/173606361 shutdown_connections 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.828+0000 7fa6ccf40700 1 -- 172.21.15.96:0/173606361 wait complete. 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6ccf40700 1 Processor -- start 2024-04-22T15:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6ccf40700 1 -- start start 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6ccf40700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6c81001c0 0x7fa6c80fb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6ccf40700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 0x7fa6c80fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6ccf40700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80fbe00 con 0x7fa6c8105c50 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6ccf40700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80fbf70 con 0x7fa6c81001c0 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6c5d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 0x7fa6c80fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6c5d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 0x7fa6c80fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32812/0 (socket says 172.21.15.96:32812) 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6c5d9b700 1 -- 172.21.15.96:0/3827801623 learned_addr learned my addr 172.21.15.96:0/3827801623 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.829+0000 7fa6c659c700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6c81001c0 0x7fa6c80fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.830+0000 7fa6c5d9b700 1 -- 172.21.15.96:0/3827801623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6c81001c0 msgr2=0x7fa6c80fb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.830+0000 7fa6c5d9b700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6c81001c0 0x7fa6c80fb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:37.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.830+0000 7fa6c5d9b700 1 -- 172.21.15.96:0/3827801623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c0009d00 con 0x7fa6c8105c50 2024-04-22T15:14:37.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.830+0000 7fa6c5d9b700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 0x7fa6c80fb8c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b4b0 tx=0x7fa6b000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:37.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.830+0000 7fa6bf7fe700 1 -- 172.21.15.96:0/3827801623 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b0011800 con 0x7fa6c8105c50 2024-04-22T15:14:37.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.830+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c81a2ea0 con 0x7fa6c8105c50 2024-04-22T15:14:37.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.830+0000 7fa6bf7fe700 1 -- 172.21.15.96:0/3827801623 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6b0011e20 con 0x7fa6c8105c50 2024-04-22T15:14:37.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.830+0000 7fa6bf7fe700 1 -- 172.21.15.96:0/3827801623 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b000f2e0 con 0x7fa6c8105c50 2024-04-22T15:14:37.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.830+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c81a3250 con 0x7fa6c8105c50 2024-04-22T15:14:37.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.831+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c81013e0 con 0x7fa6c8105c50 2024-04-22T15:14:37.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.831+0000 7fa6bf7fe700 1 -- 172.21.15.96:0/3827801623 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa6b0010380 con 0x7fa6c8105c50 2024-04-22T15:14:37.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.832+0000 7fa6bf7fe700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa6b406cce0 0x7fa6b406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:37.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.832+0000 7fa6bf7fe700 1 -- 172.21.15.96:0/3827801623 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa6b008ba70 con 0x7fa6c8105c50 2024-04-22T15:14:37.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.832+0000 7fa6c659c700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa6b406cce0 0x7fa6b406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:37.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.832+0000 7fa6c659c700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa6b406cce0 0x7fa6b406f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000f80 tx=0x7fa6c00060c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:37.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:37.836+0000 7fa6bf7fe700 1 -- 172.21.15.96:0/3827801623 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa6b0056180 con 0x7fa6c8105c50 2024-04-22T15:14:37.840 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:14:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:14:38.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.028+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6c80fc130 con 0x7fa6c8105c50 2024-04-22T15:14:38.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.029+0000 7fa6bf7fe700 1 -- 172.21.15.96:0/3827801623 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+11348 (secure 0 0 0) 0x7fa6b00597a0 con 0x7fa6c8105c50 2024-04-22T15:14:38.032 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:14:38.032 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":39,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","created":"2024-04-22T15:08:23.492994+0000","modified":"2024-04-22T15:14:16.338041+0000","last_up_change":"2024-04-22T15:14:15.336641+0000","last_in_change":"2024-04-22T15:13:56.089739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:13:01.924433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2f1b7772-06b4-467f-b14c-698a0de7357f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6803","nonce":1197486134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6805","nonce":1197486134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6809","nonce":1197486134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6807","nonce":1197486134}]},"public_addr":"172.21.15.96:6803/1197486134","cluster_addr":"172.21.15.96:6805/1197486134","heartbeat_back_addr":"172.21.15.96:6809/1197486134","heartbeat_front_addr":"172.21.15.96:6807/1197486134","state":["exists","up"]},{"osd":1,"uuid":"59260a36-a2ed-42f9-8f3a-209aae4c4238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6811","nonce":3577737028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6813","nonce":3577737028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6817","nonce":3577737028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6815","nonce":3577737028}]},"public_addr":"172.21.15.96:6811/3577737028","cluster_addr":"172.21.15.96:6813/3577737028","heartbeat_back_addr":"172.21.15.96:6817/3577737028","heartbeat_front_addr":"172.21.15.96:6815/3577737028","state":["exists","up"]},{"osd":2,"uuid":"897e21a9-3009-41c6-acf1-e44414fc69c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6819","nonce":1428742985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6821","nonce":1428742985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6825","nonce":1428742985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6823","nonce":1428742985}]},"public_addr":"172.21.15.96:6819/1428742985","cluster_addr":"172.21.15.96:6821/1428742985","heartbeat_back_addr":"172.21.15.96:6825/1428742985","heartbeat_front_addr":"172.21.15.96:6823/1428742985","state":["exists","up"]},{"osd":3,"uuid":"201c3153-a284-471e-894d-2273001cc29e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6801","nonce":1414997674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6803","nonce":1414997674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6807","nonce":1414997674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6805","nonce":1414997674}]},"public_addr":"172.21.15.196:6801/1414997674","cluster_addr":"172.21.15.196:6803/1414997674","heartbeat_back_addr":"172.21.15.196:6807/1414997674","heartbeat_front_addr":"172.21.15.196:6805/1414997674","state":["exists","up"]},{"osd":4,"uuid":"780ed007-14ef-4c92-a5e9-6c57afcad448","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6809","nonce":1827225092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6811","nonce":1827225092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6815","nonce":1827225092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6813","nonce":1827225092}]},"public_addr":"172.21.15.196:6809/1827225092","cluster_addr":"172.21.15.196:6811/1827225092","heartbeat_back_addr":"172.21.15.196:6815/1827225092","heartbeat_front_addr":"172.21.15.196:6813/1827225092","state":["exists","up"]},{"osd":5,"uuid":"16384d8d-bef4-4f65-9079-e2b4518241da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6817","nonce":2750420250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6819","nonce":2750420250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6823","nonce":2750420250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6821","nonce":2750420250}]},"public_addr":"172.21.15.196:6817/2750420250","cluster_addr":"172.21.15.196:6819/2750420250","heartbeat_back_addr":"172.21.15.196:6823/2750420250","heartbeat_front_addr":"172.21.15.196:6821/2750420250","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:00.654872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:28.691804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:57.737056+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:21.679902+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:47.489372+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:14:13.600900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:0/3921802975":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6801/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6800/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3486391579":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/1794679103":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3321679733":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6800/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:0/4224647460":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1237353220":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/838799683":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3667378212":"2024-04-23T15:08:51.463883+0000","172.21.15.96:6800/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1319578885":"2024-04-23T15:09:23.920294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa6b406cce0 msgr2=0x7fa6b406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa6b406cce0 0x7fa6b406f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000f80 tx=0x7fa6c00060c0 comp rx=0 tx=0).stop 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 msgr2=0x7fa6c80fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 0x7fa6c80fb8c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b4b0 tx=0x7fa6b000b7c0 comp rx=0 tx=0).stop 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 shutdown_connections 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa6b406cce0 0x7fa6b406f1a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6c81001c0 0x7fa6c80fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 --2- 172.21.15.96:0/3827801623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6c8105c50 0x7fa6c80fb8c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 >> 172.21.15.96:0/3827801623 conn(0x7fa6c8075f30 msgr2=0x7fa6c80ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 shutdown_connections 2024-04-22T15:14:38.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:38.031+0000 7fa6ccf40700 1 -- 172.21.15.96:0/3827801623 wait complete. 2024-04-22T15:14:38.795 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T15:14:38.796 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T15:14:38.796 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T15:14:38.797 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T15:14:38.797 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T15:14:38.798 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T15:14:38.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:38 smithi096 ceph-mon[26688]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:38.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:38 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3827801623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:14:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:38 smithi196 ceph-mon[34717]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:38 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3827801623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:14:39.153 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:39.153 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:39.157 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:40.247 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:40.249 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:40.250 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:40 smithi096 ceph-mon[26688]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:40 smithi196 ceph-mon[34717]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:42 smithi096 ceph-mon[26688]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:42 smithi196 ceph-mon[34717]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:44.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:44 smithi096 ceph-mon[26688]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:44 smithi196 ceph-mon[34717]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:45.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.730+0000 7f16a27b7700 1 -- 172.21.15.96:0/3378794321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c105aa0 msgr2=0x7f169c105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:45.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.730+0000 7f16a27b7700 1 --2- 172.21.15.96:0/3378794321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c105aa0 0x7f169c105e80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1690009910 tx=0x7f1690009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:45.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.730+0000 7f16a27b7700 1 -- 172.21.15.96:0/3378794321 shutdown_connections 2024-04-22T15:14:45.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.730+0000 7f16a27b7700 1 --2- 172.21.15.96:0/3378794321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f169c0ffaa0 0x7f169c0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:45.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.730+0000 7f16a27b7700 1 --2- 172.21.15.96:0/3378794321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c105aa0 0x7f169c105e80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:45.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.730+0000 7f16a27b7700 1 -- 172.21.15.96:0/3378794321 >> 172.21.15.96:0/3378794321 conn(0x7f169c076740 msgr2=0x7f169c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:45.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.730+0000 7f16a27b7700 1 -- 172.21.15.96:0/3378794321 shutdown_connections 2024-04-22T15:14:45.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.730+0000 7f16a27b7700 1 -- 172.21.15.96:0/3378794321 wait complete. 2024-04-22T15:14:45.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.731+0000 7f16a27b7700 1 Processor -- start 2024-04-22T15:14:45.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.731+0000 7f16a27b7700 1 -- start start 2024-04-22T15:14:45.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.731+0000 7f16a27b7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c0ffaa0 0x7f169c0fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.731+0000 7f16a27b7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f169c105aa0 0x7f169c0fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.731+0000 7f16a27b7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0fc6a0 con 0x7f169c0ffaa0 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.731+0000 7f16a27b7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0fc7e0 con 0x7f169c105aa0 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.731+0000 7f169bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c0ffaa0 0x7f169c0fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.731+0000 7f169bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c0ffaa0 0x7f169c0fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40890/0 (socket says 172.21.15.96:40890) 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.731+0000 7f169bfff700 1 -- 172.21.15.96:0/2182533999 learned_addr learned my addr 172.21.15.96:0/2182533999 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f169b7fe700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f169c105aa0 0x7f169c0fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f169bfff700 1 -- 172.21.15.96:0/2182533999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f169c105aa0 msgr2=0x7f169c0fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f169bfff700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f169c105aa0 0x7f169c0fc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f169bfff700 1 -- 172.21.15.96:0/2182533999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900095f0 con 0x7f169c0ffaa0 2024-04-22T15:14:45.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f169bfff700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c0ffaa0 0x7f169c0fbc20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f169000bd70 tx=0x7f169000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f16997fa700 1 -- 172.21.15.96:0/2182533999 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f169001d070 con 0x7f169c0ffaa0 2024-04-22T15:14:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f16997fa700 1 -- 172.21.15.96:0/2182533999 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f169000f460 con 0x7f169c0ffaa0 2024-04-22T15:14:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c0fca60 con 0x7f169c0ffaa0 2024-04-22T15:14:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f16997fa700 1 -- 172.21.15.96:0/2182533999 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1690017630 con 0x7f169c0ffaa0 2024-04-22T15:14:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.732+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c1a3010 con 0x7f169c0ffaa0 2024-04-22T15:14:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.733+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f169c1011e0 con 0x7f169c0ffaa0 2024-04-22T15:14:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.734+0000 7f16997fa700 1 -- 172.21.15.96:0/2182533999 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f169000f5d0 con 0x7f169c0ffaa0 2024-04-22T15:14:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.734+0000 7f16997fa700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f168406cce0 0x7f168406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.734+0000 7f169b7fe700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f168406cce0 0x7f168406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.735+0000 7f169b7fe700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f168406cce0 0x7f168406f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f168c006fd0 tx=0x7f168c008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.735+0000 7f16997fa700 1 -- 172.21.15.96:0/2182533999 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f169005c2f0 con 0x7f169c0ffaa0 2024-04-22T15:14:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.735+0000 7f16997fa700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] conn(0x7f1684072a10 0x7f1684074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.735+0000 7f16997fa700 1 -- 172.21.15.96:0/2182533999 --> [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16840754e0 con 0x7f1684072a10 2024-04-22T15:14:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.735+0000 7f16997fa700 1 -- 172.21.15.96:0/2182533999 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1690005780 con 0x7f169c0ffaa0 2024-04-22T15:14:45.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.735+0000 7f16a0d54700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] conn(0x7f1684072a10 0x7f1684074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:45.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.736+0000 7f16a0d54700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] conn(0x7f1684072a10 0x7f1684074e30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:45.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.736+0000 7f16997fa700 1 -- 172.21.15.96:0/2182533999 <== osd.2 v2:172.21.15.96:6818/1428742985 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f16840754e0 con 0x7f1684072a10 2024-04-22T15:14:45.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.754+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 --> [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f169c10cf60 con 0x7f1684072a10 2024-04-22T15:14:45.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16997fa700 1 -- 172.21.15.96:0/2182533999 <== osd.2 v2:172.21.15.96:6818/1428742985 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f169c10cf60 con 0x7f1684072a10 2024-04-22T15:14:45.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] conn(0x7f1684072a10 msgr2=0x7f1684074e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:45.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] conn(0x7f1684072a10 0x7f1684074e30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:45.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f168406cce0 msgr2=0x7f168406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f168406cce0 0x7f168406f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f168c006fd0 tx=0x7f168c008040 comp rx=0 tx=0).stop 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c0ffaa0 msgr2=0x7f169c0fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c0ffaa0 0x7f169c0fbc20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f169000bd70 tx=0x7f169000be50 comp rx=0 tx=0).stop 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 shutdown_connections 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6818/1428742985,v1:172.21.15.96:6819/1428742985] conn(0x7f1684072a10 0x7f1684074e30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f168406cce0 0x7f168406f1a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f169c0ffaa0 0x7f169c0fbc20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 --2- 172.21.15.96:0/2182533999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f169c105aa0 0x7f169c0fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 >> 172.21.15.96:0/2182533999 conn(0x7f169c076740 msgr2=0x7f169c0fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 shutdown_connections 2024-04-22T15:14:45.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:45.755+0000 7f16a27b7700 1 -- 172.21.15.96:0/2182533999 wait complete. 2024-04-22T15:14:46.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.628+0000 7f8d9f789700 1 -- 172.21.15.96:0/2234281298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 msgr2=0x7f8d9810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.628+0000 7f8d9f789700 1 --2- 172.21.15.96:0/2234281298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 0x7f8d9810ab80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009910 tx=0x7f8d88009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.628+0000 7f8d9f789700 1 -- 172.21.15.96:0/2234281298 shutdown_connections 2024-04-22T15:14:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.628+0000 7f8d9f789700 1 --2- 172.21.15.96:0/2234281298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 0x7f8d9810ab80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.628+0000 7f8d9f789700 1 --2- 172.21.15.96:0/2234281298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d98101ce0 0x7f8d981020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.628+0000 7f8d9f789700 1 -- 172.21.15.96:0/2234281298 >> 172.21.15.96:0/2234281298 conn(0x7f8d980fb5e0 msgr2=0x7f8d980fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.628+0000 7f8d9f789700 1 -- 172.21.15.96:0/2234281298 shutdown_connections 2024-04-22T15:14:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.628+0000 7f8d9f789700 1 -- 172.21.15.96:0/2234281298 wait complete. 2024-04-22T15:14:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.629+0000 7f8d9f789700 1 Processor -- start 2024-04-22T15:14:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.629+0000 7f8d9f789700 1 -- start start 2024-04-22T15:14:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9f789700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d98101ce0 0x7f8d9819d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9f789700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 0x7f8d9819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9f789700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9819dd10 con 0x7f8d98102690 2024-04-22T15:14:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9f789700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d981a19e0 con 0x7f8d98101ce0 2024-04-22T15:14:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9cd24700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 0x7f8d9819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9cd24700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 0x7f8d9819d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40898/0 (socket says 172.21.15.96:40898) 2024-04-22T15:14:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9cd24700 1 -- 172.21.15.96:0/1681074001 learned_addr learned my addr 172.21.15.96:0/1681074001 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9d525700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d98101ce0 0x7f8d9819d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9d525700 1 -- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 msgr2=0x7f8d9819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9d525700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 0x7f8d9819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9d525700 1 -- 172.21.15.96:0/1681074001 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d880095f0 con 0x7f8d98101ce0 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9cd24700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 0x7f8d9819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.630+0000 7f8d9d525700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d98101ce0 0x7f8d9819d0f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400e970 tx=0x7f8d9400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.631+0000 7f8d8e7fc700 1 -- 172.21.15.96:0/1681074001 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d9400c9d0 con 0x7f8d98101ce0 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.631+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d981a1cc0 con 0x7f8d98101ce0 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.631+0000 7f8d8e7fc700 1 -- 172.21.15.96:0/1681074001 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d94010430 con 0x7f8d98101ce0 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.631+0000 7f8d8e7fc700 1 -- 172.21.15.96:0/1681074001 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d94018630 con 0x7f8d98101ce0 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.631+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d981a2210 con 0x7f8d98101ce0 2024-04-22T15:14:46.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.632+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8d9810e0b0 con 0x7f8d98101ce0 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.633+0000 7f8d8e7fc700 1 -- 172.21.15.96:0/1681074001 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8d940105a0 con 0x7f8d98101ce0 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.633+0000 7f8d8e7fc700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8d8406cce0 0x7f8d8406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.633+0000 7f8d8e7fc700 1 -- 172.21.15.96:0/1681074001 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8d94014070 con 0x7f8d98101ce0 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.633+0000 7f8d9cd24700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8d8406cce0 0x7f8d8406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.633+0000 7f8d8e7fc700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] conn(0x7f8d840729c0 0x7f8d84074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.633+0000 7f8d8e7fc700 1 -- 172.21.15.96:0/1681074001 --> [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d84075490 con 0x7f8d840729c0 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.633+0000 7f8d8e7fc700 1 -- 172.21.15.96:0/1681074001 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d9408e660 con 0x7f8d98101ce0 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.634+0000 7f8d9dd26700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] conn(0x7f8d840729c0 0x7f8d84074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.634+0000 7f8d9cd24700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8d8406cce0 0x7f8d8406f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800b760 tx=0x7f8d88005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.634+0000 7f8d9dd26700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] conn(0x7f8d840729c0 0x7f8d84074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:46.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.635+0000 7f8d8e7fc700 1 -- 172.21.15.96:0/1681074001 <== osd.0 v2:172.21.15.96:6802/1197486134 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f8d84075490 con 0x7f8d840729c0 2024-04-22T15:14:46.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.650+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 --> [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8d98069730 con 0x7f8d840729c0 2024-04-22T15:14:46.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d8e7fc700 1 -- 172.21.15.96:0/1681074001 <== osd.0 v2:172.21.15.96:6802/1197486134 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8d98069730 con 0x7f8d840729c0 2024-04-22T15:14:46.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] conn(0x7f8d840729c0 msgr2=0x7f8d84074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:46.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] conn(0x7f8d840729c0 0x7f8d84074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:46.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8d8406cce0 msgr2=0x7f8d8406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:46.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8d8406cce0 0x7f8d8406f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800b760 tx=0x7f8d88005940 comp rx=0 tx=0).stop 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d98101ce0 msgr2=0x7f8d9819d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d98101ce0 0x7f8d9819d0f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400e970 tx=0x7f8d9400ec80 comp rx=0 tx=0).stop 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 shutdown_connections 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6802/1197486134,v1:172.21.15.96:6803/1197486134] conn(0x7f8d840729c0 0x7f8d84074de0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8d8406cce0 0x7f8d8406f1a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d98101ce0 0x7f8d9819d0f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 --2- 172.21.15.96:0/1681074001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d98102690 0x7f8d9819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 >> 172.21.15.96:0/1681074001 conn(0x7f8d980fb5e0 msgr2=0x7f8d980ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 shutdown_connections 2024-04-22T15:14:46.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:46.651+0000 7f8d9f789700 1 -- 172.21.15.96:0/1681074001 wait complete. 2024-04-22T15:14:46.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:46 smithi196 ceph-mon[34717]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:46 smithi096 ceph-mon[26688]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:47.075 INFO:teuthology.orchestra.run.smithi096.stdout:81604378649 2024-04-22T15:14:47.075 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 -- 172.21.15.96:0/912268329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ff3b0 msgr2=0x7f37880ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 --2- 172.21.15.96:0/912268329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ff3b0 0x7f37880ff790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3778009910 tx=0x7f3778009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 -- 172.21.15.96:0/912268329 shutdown_connections 2024-04-22T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 --2- 172.21.15.96:0/912268329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37880ffd60 0x7f37881058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 --2- 172.21.15.96:0/912268329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ff3b0 0x7f37880ff790 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 -- 172.21.15.96:0/912268329 >> 172.21.15.96:0/912268329 conn(0x7f3788075f00 msgr2=0x7f37880fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 -- 172.21.15.96:0/912268329 shutdown_connections 2024-04-22T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 -- 172.21.15.96:0/912268329 wait complete. 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 Processor -- start 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.365+0000 7f378e01c700 1 -- start start 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f378e01c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37880ff3b0 0x7f378819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f378e01c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ffd60 0x7f378819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f378e01c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378819d3d0 con 0x7f37880ffd60 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f378e01c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37881969b0 con 0x7f37880ff3b0 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f377effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ffd60 0x7f378819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f377effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ffd60 0x7f378819cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40914/0 (socket says 172.21.15.96:40914) 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f377effd700 1 -- 172.21.15.96:0/2472094988 learned_addr learned my addr 172.21.15.96:0/2472094988 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f37877fe700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37880ff3b0 0x7f378819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f377effd700 1 -- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37880ff3b0 msgr2=0x7f378819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f377effd700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37880ff3b0 0x7f378819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:47.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.366+0000 7f377effd700 1 -- 172.21.15.96:0/2472094988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37780095f0 con 0x7f37880ffd60 2024-04-22T15:14:47.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.367+0000 7f377effd700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ffd60 0x7f378819cdb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f377000d900 tx=0x7f377000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:47.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.367+0000 7f37857fa700 1 -- 172.21.15.96:0/2472094988 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3770009900 con 0x7f37880ffd60 2024-04-22T15:14:47.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.367+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3788196c90 con 0x7f37880ffd60 2024-04-22T15:14:47.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.367+0000 7f37857fa700 1 -- 172.21.15.96:0/2472094988 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3770010460 con 0x7f37880ffd60 2024-04-22T15:14:47.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.367+0000 7f37857fa700 1 -- 172.21.15.96:0/2472094988 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f377000f5f0 con 0x7f37880ffd60 2024-04-22T15:14:47.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.367+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37881971e0 con 0x7f37880ffd60 2024-04-22T15:14:47.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.368+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f378810e0b0 con 0x7f37880ffd60 2024-04-22T15:14:47.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.368+0000 7f37857fa700 1 -- 172.21.15.96:0/2472094988 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f37700105d0 con 0x7f37880ffd60 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.369+0000 7f37857fa700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f377406cce0 0x7f377406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.369+0000 7f37857fa700 1 -- 172.21.15.96:0/2472094988 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f377008b9a0 con 0x7f37880ffd60 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.369+0000 7f37877fe700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f377406cce0 0x7f377406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.369+0000 7f37857fa700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] conn(0x7f37740729c0 0x7f3774074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.369+0000 7f37857fa700 1 -- 172.21.15.96:0/2472094988 --> [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3774075490 con 0x7f37740729c0 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.369+0000 7f37857fa700 1 -- 172.21.15.96:0/2472094988 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37700902a0 con 0x7f37880ffd60 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.369+0000 7f3787fff700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] conn(0x7f37740729c0 0x7f3774074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.369+0000 7f37877fe700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f377406cce0 0x7f377406f1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3778000c00 tx=0x7f377801a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.370+0000 7f3787fff700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] conn(0x7f37740729c0 0x7f3774074de0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.370+0000 7f37857fa700 1 -- 172.21.15.96:0/2472094988 <== osd.3 v2:172.21.15.196:6800/1414997674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3774075490 con 0x7f37740729c0 2024-04-22T15:14:47.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.386+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 --> [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3788197690 con 0x7f37740729c0 2024-04-22T15:14:47.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.386+0000 7f37857fa700 1 -- 172.21.15.96:0/2472094988 <== osd.3 v2:172.21.15.196:6800/1414997674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3788197690 con 0x7f37740729c0 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] conn(0x7f37740729c0 msgr2=0x7f3774074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] conn(0x7f37740729c0 0x7f3774074de0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f377406cce0 msgr2=0x7f377406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f377406cce0 0x7f377406f1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3778000c00 tx=0x7f377801a040 comp rx=0 tx=0).stop 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ffd60 msgr2=0x7f378819cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ffd60 0x7f378819cdb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f377000d900 tx=0x7f377000dc10 comp rx=0 tx=0).stop 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 shutdown_connections 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:6800/1414997674,v1:172.21.15.196:6801/1414997674] conn(0x7f37740729c0 0x7f3774074de0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f377406cce0 0x7f377406f1a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:47.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37880ff3b0 0x7f378819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:47.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 --2- 172.21.15.96:0/2472094988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37880ffd60 0x7f378819cdb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:47.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 >> 172.21.15.96:0/2472094988 conn(0x7f3788075f00 msgr2=0x7f3788104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:47.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 shutdown_connections 2024-04-22T15:14:47.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:47.387+0000 7f378e01c700 1 -- 172.21.15.96:0/2472094988 wait complete. 2024-04-22T15:14:48.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.607+0000 7f1c783c5700 1 -- 172.21.15.96:0/4077711023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c70105b60 msgr2=0x7f1c70105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:48.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.607+0000 7f1c783c5700 1 --2- 172.21.15.96:0/4077711023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c70105b60 0x7f1c70105f40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009910 tx=0x7f1c6c009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.607+0000 7f1c783c5700 1 -- 172.21.15.96:0/4077711023 shutdown_connections 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.607+0000 7f1c783c5700 1 --2- 172.21.15.96:0/4077711023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c700ffad0 0x7f1c700fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 --2- 172.21.15.96:0/4077711023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c70105b60 0x7f1c70105f40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 -- 172.21.15.96:0/4077711023 >> 172.21.15.96:0/4077711023 conn(0x7f1c700fb600 msgr2=0x7f1c700fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 -- 172.21.15.96:0/4077711023 shutdown_connections 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 -- 172.21.15.96:0/4077711023 wait complete. 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 Processor -- start 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 -- start start 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c700ffad0 0x7f1c7019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c70105b60 0x7f1c7019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:48.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c70196a30 con 0x7f1c700ffad0 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.608+0000 7f1c783c5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c70196ba0 con 0x7f1c70105b60 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c76161700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c700ffad0 0x7f1c7019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c76161700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c700ffad0 0x7f1c7019c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40938/0 (socket says 172.21.15.96:40938) 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c76161700 1 -- 172.21.15.96:0/659945285 learned_addr learned my addr 172.21.15.96:0/659945285 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c76161700 1 -- 172.21.15.96:0/659945285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c70105b60 msgr2=0x7f1c7019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c76161700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c70105b60 0x7f1c7019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c76161700 1 -- 172.21.15.96:0/659945285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c0095f0 con 0x7f1c700ffad0 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c76161700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c700ffad0 0x7f1c7019c980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c006010 tx=0x7f1c6c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c677fe700 1 -- 172.21.15.96:0/659945285 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c6c01d070 con 0x7f1c700ffad0 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c677fe700 1 -- 172.21.15.96:0/659945285 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c6c00f460 con 0x7f1c700ffad0 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c70196e20 con 0x7f1c700ffad0 2024-04-22T15:14:48.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c677fe700 1 -- 172.21.15.96:0/659945285 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c6c017630 con 0x7f1c700ffad0 2024-04-22T15:14:48.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.609+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c70197310 con 0x7f1c700ffad0 2024-04-22T15:14:48.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.610+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c7010d050 con 0x7f1c700ffad0 2024-04-22T15:14:48.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.611+0000 7f1c677fe700 1 -- 172.21.15.96:0/659945285 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1c6c00f5d0 con 0x7f1c700ffad0 2024-04-22T15:14:48.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.611+0000 7f1c677fe700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1c5c06cce0 0x7f1c5c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:48.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.611+0000 7f1c75960700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1c5c06cce0 0x7f1c5c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:48.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.611+0000 7f1c677fe700 1 -- 172.21.15.96:0/659945285 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1c6c08d770 con 0x7f1c700ffad0 2024-04-22T15:14:48.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.611+0000 7f1c677fe700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] conn(0x7f1c5c0729c0 0x7f1c5c074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:48.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.611+0000 7f1c677fe700 1 -- 172.21.15.96:0/659945285 --> [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c5c075490 con 0x7f1c5c0729c0 2024-04-22T15:14:48.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.611+0000 7f1c677fe700 1 -- 172.21.15.96:0/659945285 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c6c08dbd0 con 0x7f1c700ffad0 2024-04-22T15:14:48.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.611+0000 7f1c76962700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] conn(0x7f1c5c0729c0 0x7f1c5c074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:48.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.612+0000 7f1c75960700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1c5c06cce0 0x7f1c5c06f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c701981a0 tx=0x7f1c60006d20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:48.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.612+0000 7f1c76962700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] conn(0x7f1c5c0729c0 0x7f1c5c074de0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:48.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.612+0000 7f1c677fe700 1 -- 172.21.15.96:0/659945285 <== osd.1 v2:172.21.15.96:6810/3577737028 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1c5c075490 con 0x7f1c5c0729c0 2024-04-22T15:14:48.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.628+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 --> [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1c70197770 con 0x7f1c5c0729c0 2024-04-22T15:14:48.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.628+0000 7f1c677fe700 1 -- 172.21.15.96:0/659945285 <== osd.1 v2:172.21.15.96:6810/3577737028 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1c70197770 con 0x7f1c5c0729c0 2024-04-22T15:14:48.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.628+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] conn(0x7f1c5c0729c0 msgr2=0x7f1c5c074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.628+0000 7f1c783c5700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] conn(0x7f1c5c0729c0 0x7f1c5c074de0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1c5c06cce0 msgr2=0x7f1c5c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1c5c06cce0 0x7f1c5c06f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c701981a0 tx=0x7f1c60006d20 comp rx=0 tx=0).stop 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c700ffad0 msgr2=0x7f1c7019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c700ffad0 0x7f1c7019c980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c006010 tx=0x7f1c6c00be20 comp rx=0 tx=0).stop 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 shutdown_connections 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1c5c06cce0 0x7f1c5c06f1a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:6810/3577737028,v1:172.21.15.96:6811/3577737028] conn(0x7f1c5c0729c0 0x7f1c5c074de0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c700ffad0 0x7f1c7019c980 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 --2- 172.21.15.96:0/659945285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c70105b60 0x7f1c7019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 >> 172.21.15.96:0/659945285 conn(0x7f1c700fb600 msgr2=0x7f1c700696b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 shutdown_connections 2024-04-22T15:14:48.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:48.629+0000 7f1c783c5700 1 -- 172.21.15.96:0/659945285 wait complete. 2024-04-22T15:14:48.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:48 smithi096 ceph-mon[26688]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:48 smithi196 ceph-mon[34717]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:49.355 INFO:teuthology.orchestra.run.smithi096.stdout:38654705700 2024-04-22T15:14:49.355 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T15:14:50.170 INFO:teuthology.orchestra.run.smithi096.stdout:60129542174 2024-04-22T15:14:50.170 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T15:14:50.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.394+0000 7f8dca055700 1 -- 172.21.15.96:0/1250635851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 msgr2=0x7f8dc41020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:50.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.394+0000 7f8dca055700 1 --2- 172.21.15.96:0/1250635851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 0x7f8dc41020c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009910 tx=0x7f8db8009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:50.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.394+0000 7f8dca055700 1 -- 172.21.15.96:0/1250635851 shutdown_connections 2024-04-22T15:14:50.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.394+0000 7f8dca055700 1 --2- 172.21.15.96:0/1250635851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc4102690 0x7f8dc410ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:50.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.394+0000 7f8dca055700 1 --2- 172.21.15.96:0/1250635851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 0x7f8dc41020c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:50.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.394+0000 7f8dca055700 1 -- 172.21.15.96:0/1250635851 >> 172.21.15.96:0/1250635851 conn(0x7f8dc40fb5e0 msgr2=0x7f8dc40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:50.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.394+0000 7f8dca055700 1 -- 172.21.15.96:0/1250635851 shutdown_connections 2024-04-22T15:14:50.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.394+0000 7f8dca055700 1 -- 172.21.15.96:0/1250635851 wait complete. 2024-04-22T15:14:50.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.395+0000 7f8dca055700 1 Processor -- start 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.395+0000 7f8dca055700 1 -- start start 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.395+0000 7f8dca055700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 0x7f8dc419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.395+0000 7f8dca055700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc4102690 0x7f8dc419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.395+0000 7f8dca055700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc419d490 con 0x7f8dc4101ce0 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.395+0000 7f8dca055700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc4196a70 con 0x7f8dc4102690 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.395+0000 7f8dc37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 0x7f8dc419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.395+0000 7f8dc37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 0x7f8dc419c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40960/0 (socket says 172.21.15.96:40960) 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.395+0000 7f8dc37fe700 1 -- 172.21.15.96:0/3409047297 learned_addr learned my addr 172.21.15.96:0/3409047297 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dc2ffd700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc4102690 0x7f8dc419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dc37fe700 1 -- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc4102690 msgr2=0x7f8dc419ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dc37fe700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc4102690 0x7f8dc419ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dc37fe700 1 -- 172.21.15.96:0/3409047297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db80095f0 con 0x7f8dc4101ce0 2024-04-22T15:14:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dc2ffd700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc4102690 0x7f8dc419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:14:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dc37fe700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 0x7f8dc419c930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8db800bd70 tx=0x7f8db800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dc0ff9700 1 -- 172.21.15.96:0/3409047297 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8db801d070 con 0x7f8dc4101ce0 2024-04-22T15:14:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc4196cf0 con 0x7f8dc4101ce0 2024-04-22T15:14:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc41971e0 con 0x7f8dc4101ce0 2024-04-22T15:14:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dc0ff9700 1 -- 172.21.15.96:0/3409047297 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8db800f460 con 0x7f8dc4101ce0 2024-04-22T15:14:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.396+0000 7f8dc0ff9700 1 -- 172.21.15.96:0/3409047297 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8db8017630 con 0x7f8dc4101ce0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.397+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8db0000fc0 con 0x7f8dc4101ce0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.398+0000 7f8dc0ff9700 1 -- 172.21.15.96:0/3409047297 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8db800fab0 con 0x7f8dc4101ce0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.398+0000 7f8dc0ff9700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8dac06cce0 0x7f8dac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.399+0000 7f8dc2ffd700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8dac06cce0 0x7f8dac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.399+0000 7f8dc0ff9700 1 -- 172.21.15.96:0/3409047297 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8db808d6b0 con 0x7f8dc4101ce0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.399+0000 7f8dc0ff9700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] conn(0x7f8dac0729c0 0x7f8dac074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.399+0000 7f8dc0ff9700 1 -- 172.21.15.96:0/3409047297 --> [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8dac075490 con 0x7f8dac0729c0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.399+0000 7f8dc0ff9700 1 -- 172.21.15.96:0/3409047297 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8db808db10 con 0x7f8dc4101ce0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.399+0000 7f8dc2ffd700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8dac06cce0 0x7f8dac06f1a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4197c30 tx=0x7f8db4008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.399+0000 7f8dc3fff700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] conn(0x7f8dac0729c0 0x7f8dac074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.400+0000 7f8dc3fff700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] conn(0x7f8dac0729c0 0x7f8dac074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.401+0000 7f8dc0ff9700 1 -- 172.21.15.96:0/3409047297 <== osd.5 v2:172.21.15.196:6816/2750420250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f8dac075490 con 0x7f8dac0729c0 2024-04-22T15:14:50.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.422+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 --> [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8db0002cb0 con 0x7f8dac0729c0 2024-04-22T15:14:50.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.422+0000 7f8dc0ff9700 1 -- 172.21.15.96:0/3409047297 <== osd.5 v2:172.21.15.196:6816/2750420250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8db0002cb0 con 0x7f8dac0729c0 2024-04-22T15:14:50.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] conn(0x7f8dac0729c0 msgr2=0x7f8dac074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:50.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] conn(0x7f8dac0729c0 0x7f8dac074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:50.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8dac06cce0 msgr2=0x7f8dac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:50.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8dac06cce0 0x7f8dac06f1a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4197c30 tx=0x7f8db4008040 comp rx=0 tx=0).stop 2024-04-22T15:14:50.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 msgr2=0x7f8dc419c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:50.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 0x7f8dc419c930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8db800bd70 tx=0x7f8db800be50 comp rx=0 tx=0).stop 2024-04-22T15:14:50.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 shutdown_connections 2024-04-22T15:14:50.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:6816/2750420250,v1:172.21.15.196:6817/2750420250] conn(0x7f8dac0729c0 0x7f8dac074de0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:50.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8dac06cce0 0x7f8dac06f1a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:50.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dc4101ce0 0x7f8dc419c930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:50.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 --2- 172.21.15.96:0/3409047297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc4102690 0x7f8dc419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:50.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 >> 172.21.15.96:0/3409047297 conn(0x7f8dc40fb5e0 msgr2=0x7f8dc4103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:50.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 shutdown_connections 2024-04-22T15:14:50.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:50.423+0000 7f8dca055700 1 -- 172.21.15.96:0/3409047297 wait complete. 2024-04-22T15:14:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:50 smithi196 ceph-mon[34717]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:51.028 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:50 smithi096 ceph-mon[26688]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:51.050 INFO:teuthology.orchestra.run.smithi096.stdout:163208757258 2024-04-22T15:14:51.050 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T15:14:51.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.308+0000 7f6af16a0700 1 -- 172.21.15.96:0/3448442060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec102690 msgr2=0x7f6aec10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:51.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.308+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3448442060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec102690 0x7f6aec10ab80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009910 tx=0x7f6ae0009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:51.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.308+0000 7f6af16a0700 1 -- 172.21.15.96:0/3448442060 shutdown_connections 2024-04-22T15:14:51.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.308+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3448442060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec102690 0x7f6aec10ab80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:51.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.308+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3448442060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aec101ce0 0x7f6aec1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:51.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.308+0000 7f6af16a0700 1 -- 172.21.15.96:0/3448442060 >> 172.21.15.96:0/3448442060 conn(0x7f6aec0fb5e0 msgr2=0x7f6aec0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:51.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.308+0000 7f6af16a0700 1 -- 172.21.15.96:0/3448442060 shutdown_connections 2024-04-22T15:14:51.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.308+0000 7f6af16a0700 1 -- 172.21.15.96:0/3448442060 wait complete. 2024-04-22T15:14:51.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.309+0000 7f6af16a0700 1 Processor -- start 2024-04-22T15:14:51.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.309+0000 7f6af16a0700 1 -- start start 2024-04-22T15:14:51.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.309+0000 7f6af16a0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec101ce0 0x7f6aec19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:51.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.309+0000 7f6af16a0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aec102690 0x7f6aec19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:51.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.309+0000 7f6af16a0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec19d4b0 con 0x7f6aec101ce0 2024-04-22T15:14:51.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.309+0000 7f6af16a0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec196a90 con 0x7f6aec102690 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6aeaffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec101ce0 0x7f6aec19c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6aeaffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec101ce0 0x7f6aec19c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51980/0 (socket says 172.21.15.96:51980) 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6aeaffd700 1 -- 172.21.15.96:0/3757968059 learned_addr learned my addr 172.21.15.96:0/3757968059 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6aea7fc700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aec102690 0x7f6aec19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6aeaffd700 1 -- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aec102690 msgr2=0x7f6aec19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6aeaffd700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aec102690 0x7f6aec19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6aeaffd700 1 -- 172.21.15.96:0/3757968059 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae00095f0 con 0x7f6aec101ce0 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6aeaffd700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec101ce0 0x7f6aec19c950 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00b4b0 tx=0x7f6adc00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6ad3fff700 1 -- 172.21.15.96:0/3757968059 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6adc011800 con 0x7f6aec101ce0 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec196d70 con 0x7f6aec101ce0 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6ad3fff700 1 -- 172.21.15.96:0/3757968059 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6adc011e20 con 0x7f6aec101ce0 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6ad3fff700 1 -- 172.21.15.96:0/3757968059 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6adc00f2e0 con 0x7f6aec101ce0 2024-04-22T15:14:51.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.310+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec197290 con 0x7f6aec101ce0 2024-04-22T15:14:51.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.311+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6aec10e130 con 0x7f6aec101ce0 2024-04-22T15:14:51.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.312+0000 7f6ad3fff700 1 -- 172.21.15.96:0/3757968059 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6adc010380 con 0x7f6aec101ce0 2024-04-22T15:14:51.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.313+0000 7f6ad3fff700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6ad406cce0 0x7f6ad406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.313+0000 7f6ad3fff700 1 -- 172.21.15.96:0/3757968059 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6adc08bfa0 con 0x7f6aec101ce0 2024-04-22T15:14:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.313+0000 7f6aea7fc700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6ad406cce0 0x7f6ad406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.313+0000 7f6ad3fff700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] conn(0x7f6ad40729c0 0x7f6ad4074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.313+0000 7f6ad3fff700 1 -- 172.21.15.96:0/3757968059 --> [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ad4075490 con 0x7f6ad40729c0 2024-04-22T15:14:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.313+0000 7f6ad3fff700 1 -- 172.21.15.96:0/3757968059 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6adc08c400 con 0x7f6aec101ce0 2024-04-22T15:14:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.313+0000 7f6aea7fc700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6ad406cce0 0x7f6ad406f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000b5c0 tx=0x7f6ae001a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.313+0000 7f6aeb7fe700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] conn(0x7f6ad40729c0 0x7f6ad4074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.314+0000 7f6aeb7fe700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] conn(0x7f6ad40729c0 0x7f6ad4074de0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:51.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.314+0000 7f6ad3fff700 1 -- 172.21.15.96:0/3757968059 <== osd.4 v2:172.21.15.196:6808/1827225092 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6ad4075490 con 0x7f6ad40729c0 2024-04-22T15:14:51.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.341+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 --> [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6aec1977a0 con 0x7f6ad40729c0 2024-04-22T15:14:51.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.341+0000 7f6ad3fff700 1 -- 172.21.15.96:0/3757968059 <== osd.4 v2:172.21.15.196:6808/1827225092 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6aec1977a0 con 0x7f6ad40729c0 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] conn(0x7f6ad40729c0 msgr2=0x7f6ad4074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] conn(0x7f6ad40729c0 0x7f6ad4074de0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6ad406cce0 msgr2=0x7f6ad406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6ad406cce0 0x7f6ad406f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000b5c0 tx=0x7f6ae001a040 comp rx=0 tx=0).stop 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec101ce0 msgr2=0x7f6aec19c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec101ce0 0x7f6aec19c950 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00b4b0 tx=0x7f6adc00b7c0 comp rx=0 tx=0).stop 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 shutdown_connections 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:6808/1827225092,v1:172.21.15.196:6809/1827225092] conn(0x7f6ad40729c0 0x7f6ad4074de0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f6ad406cce0 0x7f6ad406f1a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec101ce0 0x7f6aec19c950 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 --2- 172.21.15.96:0/3757968059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aec102690 0x7f6aec19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 >> 172.21.15.96:0/3757968059 conn(0x7f6aec0fb5e0 msgr2=0x7f6aec103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:51.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.342+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 shutdown_connections 2024-04-22T15:14:51.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:51.343+0000 7f6af16a0700 1 -- 172.21.15.96:0/3757968059 wait complete. 2024-04-22T15:14:51.822 INFO:teuthology.orchestra.run.smithi096.stdout:107374182418 2024-04-22T15:14:51.822 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T15:14:52.507 INFO:teuthology.orchestra.run.smithi096.stdout:141733920783 2024-04-22T15:14:52.507 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T15:14:52.533 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:52.534 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:52.534 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:52.582 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:52 smithi196 ceph-mon[34717]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:14:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:52 smithi096 ceph-mon[26688]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:14:54.520 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:14:54.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:54 smithi096 ceph-mon[26688]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:55.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 -- 172.21.15.96:0/1473596915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 msgr2=0x7fe6a81008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:55.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 --2- 172.21.15.96:0/1473596915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 0x7fe6a81008c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009910 tx=0x7fe6a4009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:55.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 -- 172.21.15.96:0/1473596915 shutdown_connections 2024-04-22T15:14:55.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 --2- 172.21.15.96:0/1473596915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6a8100e90 0x7fe6a81049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:55.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 --2- 172.21.15.96:0/1473596915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 0x7fe6a81008c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:55.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 -- 172.21.15.96:0/1473596915 >> 172.21.15.96:0/1473596915 conn(0x7fe6a8075f00 msgr2=0x7fe6a80fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:55.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 -- 172.21.15.96:0/1473596915 shutdown_connections 2024-04-22T15:14:55.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 -- 172.21.15.96:0/1473596915 wait complete. 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 Processor -- start 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.002+0000 7fe6af824700 1 -- start start 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6af824700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 0x7fe6a8195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6af824700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6a8100e90 0x7fe6a8196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6af824700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a8196aa0 con 0x7fe6a81004e0 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6af824700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a819b6b0 con 0x7fe6a8100e90 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6ad5c0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 0x7fe6a8195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6ad5c0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 0x7fe6a8195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51996/0 (socket says 172.21.15.96:51996) 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6ad5c0700 1 -- 172.21.15.96:0/3161389258 learned_addr learned my addr 172.21.15.96:0/3161389258 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6acdbf700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6a8100e90 0x7fe6a8196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6ad5c0700 1 -- 172.21.15.96:0/3161389258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6a8100e90 msgr2=0x7fe6a8196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:55.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6ad5c0700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6a8100e90 0x7fe6a8196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:55.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6ad5c0700 1 -- 172.21.15.96:0/3161389258 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a40095f0 con 0x7fe6a81004e0 2024-04-22T15:14:55.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6ad5c0700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 0x7fe6a8195ed0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe6a400bd70 tx=0x7fe6a400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:55.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe69e7fc700 1 -- 172.21.15.96:0/3161389258 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6a401d070 con 0x7fe6a81004e0 2024-04-22T15:14:55.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe69e7fc700 1 -- 172.21.15.96:0/3161389258 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6a400f460 con 0x7fe6a81004e0 2024-04-22T15:14:55.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.003+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a819b930 con 0x7fe6a81004e0 2024-04-22T15:14:55.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.004+0000 7fe69e7fc700 1 -- 172.21.15.96:0/3161389258 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6a4017630 con 0x7fe6a81004e0 2024-04-22T15:14:55.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.004+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a819be20 con 0x7fe6a81004e0 2024-04-22T15:14:55.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:54 smithi196 ceph-mon[34717]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:55.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.005+0000 7fe69e7fc700 1 -- 172.21.15.96:0/3161389258 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe6a400f5d0 con 0x7fe6a81004e0 2024-04-22T15:14:55.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.005+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a8108360 con 0x7fe6a81004e0 2024-04-22T15:14:55.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.006+0000 7fe69e7fc700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fe69406cce0 0x7fe69406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:55.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.006+0000 7fe69e7fc700 1 -- 172.21.15.96:0/3161389258 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe6a408d4f0 con 0x7fe6a81004e0 2024-04-22T15:14:55.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.006+0000 7fe6acdbf700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fe69406cce0 0x7fe69406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:55.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.006+0000 7fe6acdbf700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fe69406cce0 0x7fe69406f1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe698006fd0 tx=0x7fe698008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:55.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.010+0000 7fe69e7fc700 1 -- 172.21.15.96:0/3161389258 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fe6a4057c00 con 0x7fe6a81004e0 2024-04-22T15:14:55.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.209+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe6a8196fc0 con 0x7fe6a81004e0 2024-04-22T15:14:55.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.209+0000 7fe69e7fc700 1 -- 172.21.15.96:0/3161389258 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe6a405b220 con 0x7fe6a81004e0 2024-04-22T15:14:55.213 INFO:teuthology.orchestra.run.smithi096.stdout:163208757259 2024-04-22T15:14:55.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.211+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fe69406cce0 msgr2=0x7fe69406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:55.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.211+0000 7fe6af824700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fe69406cce0 0x7fe69406f1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe698006fd0 tx=0x7fe698008040 comp rx=0 tx=0).stop 2024-04-22T15:14:55.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.211+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 msgr2=0x7fe6a8195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:55.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.211+0000 7fe6af824700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 0x7fe6a8195ed0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe6a400bd70 tx=0x7fe6a400be50 comp rx=0 tx=0).stop 2024-04-22T15:14:55.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.212+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 shutdown_connections 2024-04-22T15:14:55.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.212+0000 7fe6af824700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fe69406cce0 0x7fe69406f1a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:55.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.212+0000 7fe6af824700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6a81004e0 0x7fe6a8195ed0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:55.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.212+0000 7fe6af824700 1 --2- 172.21.15.96:0/3161389258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6a8100e90 0x7fe6a8196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:55.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.212+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 >> 172.21.15.96:0/3161389258 conn(0x7fe6a8075f00 msgr2=0x7fe6a8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:55.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.212+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 shutdown_connections 2024-04-22T15:14:55.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:55.212+0000 7fe6af824700 1 -- 172.21.15.96:0/3161389258 wait complete. 2024-04-22T15:14:55.911 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:55 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3161389258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T15:14:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:55 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3161389258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T15:14:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:56 smithi196 ceph-mon[34717]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:56 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/964835353' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T15:14:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.257+0000 7f8eefc07700 1 -- 172.21.15.96:0/4067661188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee8105ab0 msgr2=0x7f8ee8105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.257+0000 7f8eefc07700 1 --2- 172.21.15.96:0/4067661188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee8105ab0 0x7f8ee8105e90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8009910 tx=0x7f8ed8009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.257+0000 7f8eefc07700 1 -- 172.21.15.96:0/4067661188 shutdown_connections 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.257+0000 7f8eefc07700 1 --2- 172.21.15.96:0/4067661188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee80ffab0 0x7f8ee80fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.257+0000 7f8eefc07700 1 --2- 172.21.15.96:0/4067661188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee8105ab0 0x7f8ee8105e90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.257+0000 7f8eefc07700 1 -- 172.21.15.96:0/4067661188 >> 172.21.15.96:0/4067661188 conn(0x7f8ee80fb5e0 msgr2=0x7f8ee80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.257+0000 7f8eefc07700 1 -- 172.21.15.96:0/4067661188 shutdown_connections 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.257+0000 7f8eefc07700 1 -- 172.21.15.96:0/4067661188 wait complete. 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.258+0000 7f8eefc07700 1 Processor -- start 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.258+0000 7f8eefc07700 1 -- start start 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eefc07700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee80ffab0 0x7f8ee81995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eefc07700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee8105ab0 0x7f8ee8199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eefc07700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee81946d0 con 0x7f8ee80ffab0 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eefc07700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee8194840 con 0x7f8ee8105ab0 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eed9a3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee80ffab0 0x7f8ee81995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:57.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eed9a3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee80ffab0 0x7f8ee81995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52018/0 (socket says 172.21.15.96:52018) 2024-04-22T15:14:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eed9a3700 1 -- 172.21.15.96:0/964835353 learned_addr learned my addr 172.21.15.96:0/964835353 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eed1a2700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee8105ab0 0x7f8ee8199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:57.093 INFO:teuthology.orchestra.run.smithi096.stdout:81604378650 2024-04-22T15:14:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eed9a3700 1 -- 172.21.15.96:0/964835353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee8105ab0 msgr2=0x7f8ee8199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eed9a3700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee8105ab0 0x7f8ee8199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eed9a3700 1 -- 172.21.15.96:0/964835353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed80095f0 con 0x7f8ee80ffab0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eed9a3700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee80ffab0 0x7f8ee81995d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8006010 tx=0x7f8ed800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8edeffd700 1 -- 172.21.15.96:0/964835353 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ed801d070 con 0x7f8ee80ffab0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8edeffd700 1 -- 172.21.15.96:0/964835353 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ed800f460 con 0x7f8ee80ffab0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8edeffd700 1 -- 172.21.15.96:0/964835353 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ed8017630 con 0x7f8ee80ffab0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee8194ac0 con 0x7f8ee80ffab0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.259+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee8194fb0 con 0x7f8ee80ffab0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.261+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee81011f0 con 0x7f8ee80ffab0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.261+0000 7f8edeffd700 1 -- 172.21.15.96:0/964835353 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8ed800f5d0 con 0x7f8ee80ffab0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.261+0000 7f8edeffd700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8ed406cce0 0x7f8ed406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.261+0000 7f8edeffd700 1 -- 172.21.15.96:0/964835353 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8ed808d360 con 0x7f8ee80ffab0 2024-04-22T15:14:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.261+0000 7f8eed1a2700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8ed406cce0 0x7f8ed406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.262+0000 7f8eed1a2700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8ed406cce0 0x7f8ed406f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8195eb0 tx=0x7f8ee4005cf0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.266+0000 7f8edeffd700 1 -- 172.21.15.96:0/964835353 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f8ed8057b30 con 0x7f8ee80ffab0 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.466+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8ee810ccb0 con 0x7f8ee80ffab0 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.466+0000 7f8edeffd700 1 -- 172.21.15.96:0/964835353 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ed805b150 con 0x7f8ee80ffab0 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.468+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8ed406cce0 msgr2=0x7f8ed406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.468+0000 7f8eefc07700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8ed406cce0 0x7f8ed406f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8195eb0 tx=0x7f8ee4005cf0 comp rx=0 tx=0).stop 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.468+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee80ffab0 msgr2=0x7f8ee81995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.468+0000 7f8eefc07700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee80ffab0 0x7f8ee81995d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8006010 tx=0x7f8ed800be20 comp rx=0 tx=0).stop 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.468+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 shutdown_connections 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.468+0000 7f8eefc07700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f8ed406cce0 0x7f8ed406f1a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.468+0000 7f8eefc07700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee80ffab0 0x7f8ee81995d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.468+0000 7f8eefc07700 1 --2- 172.21.15.96:0/964835353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee8105ab0 0x7f8ee8199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.468+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 >> 172.21.15.96:0/964835353 conn(0x7f8ee80fb5e0 msgr2=0x7f8ee80fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:57.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.469+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 shutdown_connections 2024-04-22T15:14:57.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:56.469+0000 7f8eefc07700 1 -- 172.21.15.96:0/964835353 wait complete. 2024-04-22T15:14:57.096 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:56 smithi096 ceph-mon[26688]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:57.096 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:56 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/964835353' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T15:14:58.497 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378650 for osd.2 2024-04-22T15:14:58.497 DEBUG:teuthology.parallel:result is None 2024-04-22T15:14:58.880 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757258 got 163208757259 for osd.5 2024-04-22T15:14:58.881 DEBUG:teuthology.parallel:result is None 2024-04-22T15:14:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:14:58 smithi196 ceph-mon[34717]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:59.230 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:14:58 smithi096 ceph-mon[26688]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.740+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3983660103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 msgr2=0x7f4e54105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.740+0000 7f4e59f7e700 1 --2- 172.21.15.96:0/3983660103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 0x7f4e54105f40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009910 tx=0x7f4e44009c20 comp rx=0 tx=0).stop 2024-04-22T15:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.740+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3983660103 shutdown_connections 2024-04-22T15:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.740+0000 7f4e59f7e700 1 --2- 172.21.15.96:0/3983660103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e540ffad0 0x7f4e540fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.740+0000 7f4e59f7e700 1 --2- 172.21.15.96:0/3983660103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 0x7f4e54105f40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.740+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3983660103 >> 172.21.15.96:0/3983660103 conn(0x7f4e540fb600 msgr2=0x7f4e540fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.740+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3983660103 shutdown_connections 2024-04-22T15:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.740+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3983660103 wait complete. 2024-04-22T15:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.741+0000 7f4e59f7e700 1 Processor -- start 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.741+0000 7f4e59f7e700 1 -- start start 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.741+0000 7f4e59f7e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e540ffad0 0x7f4e540fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.741+0000 7f4e59f7e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 0x7f4e540fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.741+0000 7f4e59f7e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540fdd60 con 0x7f4e54105b60 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.741+0000 7f4e59f7e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540fc810 con 0x7f4e540ffad0 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e52ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 0x7f4e540fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e52ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 0x7f4e540fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52040/0 (socket says 172.21.15.96:52040) 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e52ffd700 1 -- 172.21.15.96:0/3431618050 learned_addr learned my addr 172.21.15.96:0/3431618050 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e537fe700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e540ffad0 0x7f4e540fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e52ffd700 1 -- 172.21.15.96:0/3431618050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e540ffad0 msgr2=0x7f4e540fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e52ffd700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e540ffad0 0x7f4e540fbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e52ffd700 1 -- 172.21.15.96:0/3431618050 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e440095f0 con 0x7f4e54105b60 2024-04-22T15:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e52ffd700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 0x7f4e540fc2d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800c930 tx=0x7f4e4800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e50ff9700 1 -- 172.21.15.96:0/3431618050 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e48007ad0 con 0x7f4e54105b60 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e50ff9700 1 -- 172.21.15.96:0/3431618050 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e48007c30 con 0x7f4e54105b60 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e540fcac0 con 0x7f4e54105b60 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e50ff9700 1 -- 172.21.15.96:0/3431618050 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e48018610 con 0x7f4e54105b60 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.742+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e541a2f10 con 0x7f4e54105b60 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.743+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e541090e0 con 0x7f4e54105b60 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.744+0000 7f4e50ff9700 1 -- 172.21.15.96:0/3431618050 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4e4800f450 con 0x7f4e54105b60 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.744+0000 7f4e50ff9700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4e3c06cce0 0x7f4e3c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.744+0000 7f4e50ff9700 1 -- 172.21.15.96:0/3431618050 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4e4808c020 con 0x7f4e54105b60 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.744+0000 7f4e537fe700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4e3c06cce0 0x7f4e3c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.744+0000 7f4e537fe700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4e3c06cce0 0x7f4e3c06f1a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e44005f50 tx=0x7f4e4401a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.749+0000 7f4e50ff9700 1 -- 172.21.15.96:0/3431618050 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4e480567b0 con 0x7f4e54105b60 2024-04-22T15:14:59.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.944+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4e540fd270 con 0x7f4e54105b60 2024-04-22T15:14:59.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.944+0000 7f4e50ff9700 1 -- 172.21.15.96:0/3431618050 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e48059dd0 con 0x7f4e54105b60 2024-04-22T15:14:59.948 INFO:teuthology.orchestra.run.smithi096.stdout:38654705703 2024-04-22T15:14:59.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.946+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4e3c06cce0 msgr2=0x7f4e3c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:59.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.946+0000 7f4e59f7e700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4e3c06cce0 0x7f4e3c06f1a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e44005f50 tx=0x7f4e4401a040 comp rx=0 tx=0).stop 2024-04-22T15:14:59.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.946+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 msgr2=0x7f4e540fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:14:59.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.946+0000 7f4e59f7e700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 0x7f4e540fc2d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800c930 tx=0x7f4e4800ccf0 comp rx=0 tx=0).stop 2024-04-22T15:14:59.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.946+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 shutdown_connections 2024-04-22T15:14:59.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.946+0000 7f4e59f7e700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4e3c06cce0 0x7f4e3c06f1a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:59.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.946+0000 7f4e59f7e700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e540ffad0 0x7f4e540fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:59.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.946+0000 7f4e59f7e700 1 --2- 172.21.15.96:0/3431618050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e54105b60 0x7f4e540fc2d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:14:59.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.946+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 >> 172.21.15.96:0/3431618050 conn(0x7f4e540fb600 msgr2=0x7f4e54069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:14:59.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.947+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 shutdown_connections 2024-04-22T15:14:59.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:14:59.947+0000 7f4e59f7e700 1 -- 172.21.15.96:0/3431618050 wait complete. 2024-04-22T15:15:00.644 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:00.888 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:00 smithi096 ceph-mon[26688]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:00.888 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:00 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3431618050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T15:15:01.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:00 smithi196 ceph-mon[34717]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:01.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:00 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3431618050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T15:15:02.506 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705703 for osd.0 2024-04-22T15:15:02.506 DEBUG:teuthology.parallel:result is None 2024-04-22T15:15:02.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:02 smithi096 ceph-mon[26688]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:02.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:02 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1857721426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T15:15:02.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:02 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3101695501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T15:15:03.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.839+0000 7f64c9790700 1 -- 172.21.15.96:0/4182227337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4105aa0 msgr2=0x7f64c4105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:03.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.839+0000 7f64c9790700 1 --2- 172.21.15.96:0/4182227337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4105aa0 0x7f64c4105e80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009910 tx=0x7f64b8009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:03.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.839+0000 7f64c9790700 1 -- 172.21.15.96:0/4182227337 shutdown_connections 2024-04-22T15:15:03.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.839+0000 7f64c9790700 1 --2- 172.21.15.96:0/4182227337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64c40ffaa0 0x7f64c40fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.839+0000 7f64c9790700 1 --2- 172.21.15.96:0/4182227337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4105aa0 0x7f64c4105e80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.839+0000 7f64c9790700 1 -- 172.21.15.96:0/4182227337 >> 172.21.15.96:0/4182227337 conn(0x7f64c4076740 msgr2=0x7f64c40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:03.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.839+0000 7f64c9790700 1 -- 172.21.15.96:0/4182227337 shutdown_connections 2024-04-22T15:15:03.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.839+0000 7f64c9790700 1 -- 172.21.15.96:0/4182227337 wait complete. 2024-04-22T15:15:03.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c9790700 1 Processor -- start 2024-04-22T15:15:03.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c9790700 1 -- start start 2024-04-22T15:15:03.080 INFO:teuthology.orchestra.run.smithi096.stdout:107374182421 2024-04-22T15:15:03.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c9790700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c40ffaa0 0x7f64c4197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c9790700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64c4105aa0 0x7f64c4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c9790700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c41985b0 con 0x7f64c40ffaa0 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c9790700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c41986f0 con 0x7f64c4105aa0 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c40ffaa0 0x7f64c4197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c40ffaa0 0x7f64c4197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59604/0 (socket says 172.21.15.96:59604) 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c2ffd700 1 -- 172.21.15.96:0/3101695501 learned_addr learned my addr 172.21.15.96:0/3101695501 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.840+0000 7f64c27fc700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64c4105aa0 0x7f64c4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.841+0000 7f64c2ffd700 1 -- 172.21.15.96:0/3101695501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64c4105aa0 msgr2=0x7f64c4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.841+0000 7f64c2ffd700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64c4105aa0 0x7f64c4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.841+0000 7f64c2ffd700 1 -- 172.21.15.96:0/3101695501 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b80095f0 con 0x7f64c40ffaa0 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.841+0000 7f64c2ffd700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c40ffaa0 0x7f64c4197aa0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f64b800bd70 tx=0x7f64b800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.841+0000 7f64abfff700 1 -- 172.21.15.96:0/3101695501 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64b801d070 con 0x7f64c40ffaa0 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.841+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c4198910 con 0x7f64c40ffaa0 2024-04-22T15:15:03.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.841+0000 7f64abfff700 1 -- 172.21.15.96:0/3101695501 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64b8005700 con 0x7f64c40ffaa0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.841+0000 7f64abfff700 1 -- 172.21.15.96:0/3101695501 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64b8017400 con 0x7f64c40ffaa0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.841+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c41a0e70 con 0x7f64c40ffaa0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.842+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c41080a0 con 0x7f64c40ffaa0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.842+0000 7f64abfff700 1 -- 172.21.15.96:0/3101695501 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f64b8003a60 con 0x7f64c40ffaa0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.843+0000 7f64abfff700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f64ac06cce0 0x7f64ac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.843+0000 7f64abfff700 1 -- 172.21.15.96:0/3101695501 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f64b808d270 con 0x7f64c40ffaa0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.843+0000 7f64c27fc700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f64ac06cce0 0x7f64ac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.843+0000 7f64c27fc700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f64ac06cce0 0x7f64ac06f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64c4068dd0 tx=0x7f64b4009190 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.847+0000 7f64abfff700 1 -- 172.21.15.96:0/3101695501 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f64b8057980 con 0x7f64c40ffaa0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.061+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f64c4198910 con 0x7f64c40ffaa0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.061+0000 7f64abfff700 1 -- 172.21.15.96:0/3101695501 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f64c4198910 con 0x7f64c40ffaa0 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f64ac06cce0 msgr2=0x7f64ac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:03.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f64ac06cce0 0x7f64ac06f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64c4068dd0 tx=0x7f64b4009190 comp rx=0 tx=0).stop 2024-04-22T15:15:03.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c40ffaa0 msgr2=0x7f64c4197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:03.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c40ffaa0 0x7f64c4197aa0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f64b800bd70 tx=0x7f64b800bda0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 shutdown_connections 2024-04-22T15:15:03.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f64ac06cce0 0x7f64ac06f1a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c40ffaa0 0x7f64c4197aa0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 --2- 172.21.15.96:0/3101695501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64c4105aa0 0x7f64c4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 >> 172.21.15.96:0/3101695501 conn(0x7f64c4076740 msgr2=0x7f64c40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:03.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 shutdown_connections 2024-04-22T15:15:03.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:02.063+0000 7f64c9790700 1 -- 172.21.15.96:0/3101695501 wait complete. 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.744+0000 7f1a30418700 1 -- 172.21.15.96:0/2350600499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a28105c70 msgr2=0x7f1a28106050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.744+0000 7f1a30418700 1 --2- 172.21.15.96:0/2350600499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a28105c70 0x7f1a28106050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009910 tx=0x7f1a24009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.744+0000 7f1a30418700 1 -- 172.21.15.96:0/2350600499 shutdown_connections 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.744+0000 7f1a30418700 1 --2- 172.21.15.96:0/2350600499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a280ff090 0x7f1a280ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.744+0000 7f1a30418700 1 --2- 172.21.15.96:0/2350600499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a28105c70 0x7f1a28106050 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.744+0000 7f1a30418700 1 -- 172.21.15.96:0/2350600499 >> 172.21.15.96:0/2350600499 conn(0x7f1a28075f00 msgr2=0x7f1a280fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.744+0000 7f1a30418700 1 -- 172.21.15.96:0/2350600499 shutdown_connections 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.744+0000 7f1a30418700 1 -- 172.21.15.96:0/2350600499 wait complete. 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a30418700 1 Processor -- start 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a30418700 1 -- start start 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a30418700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a280ff090 0x7f1a2819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a30418700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a28105c70 0x7f1a2819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a30418700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2819d440 con 0x7f1a280ff090 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a30418700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a28196a20 con 0x7f1a28105c70 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a2e1b4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a280ff090 0x7f1a2819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a2e1b4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a280ff090 0x7f1a2819c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59588/0 (socket says 172.21.15.96:59588) 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a2e1b4700 1 -- 172.21.15.96:0/1857721426 learned_addr learned my addr 172.21.15.96:0/1857721426 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a2d9b3700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a28105c70 0x7f1a2819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a2e1b4700 1 -- 172.21.15.96:0/1857721426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a28105c70 msgr2=0x7f1a2819ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a2e1b4700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a28105c70 0x7f1a2819ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a2e1b4700 1 -- 172.21.15.96:0/1857721426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a240095f0 con 0x7f1a280ff090 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a2e1b4700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a280ff090 0x7f1a2819c8e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1a240092e0 tx=0x7f1a2400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a1f7fe700 1 -- 172.21.15.96:0/1857721426 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a2401d070 con 0x7f1a280ff090 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a1f7fe700 1 -- 172.21.15.96:0/1857721426 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a240056e0 con 0x7f1a280ff090 2024-04-22T15:15:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.745+0000 7f1a1f7fe700 1 -- 172.21.15.96:0/1857721426 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a2400fe80 con 0x7f1a280ff090 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.746+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a28196ca0 con 0x7f1a280ff090 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.746+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a28197190 con 0x7f1a280ff090 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.747+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a28109240 con 0x7f1a280ff090 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.748+0000 7f1a1f7fe700 1 -- 172.21.15.96:0/1857721426 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1a2400f6f0 con 0x7f1a280ff090 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.748+0000 7f1a1f7fe700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1a1406ccf0 0x7f1a1406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.748+0000 7f1a1f7fe700 1 -- 172.21.15.96:0/1857721426 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1a2408d260 con 0x7f1a280ff090 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.748+0000 7f1a2d9b3700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1a1406ccf0 0x7f1a1406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.748+0000 7f1a2d9b3700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1a1406ccf0 0x7f1a1406f1b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1a28197be0 tx=0x7f1a1800b5b0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.753+0000 7f1a1f7fe700 1 -- 172.21.15.96:0/1857721426 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f1a2405b480 con 0x7f1a280ff090 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.959+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1a28197ee0 con 0x7f1a280ff090 2024-04-22T15:15:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.960+0000 7f1a1f7fe700 1 -- 172.21.15.96:0/1857721426 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1a2405b010 con 0x7f1a280ff090 2024-04-22T15:15:03.166 INFO:teuthology.orchestra.run.smithi096.stdout:60129542177 2024-04-22T15:15:03.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1a1406ccf0 msgr2=0x7f1a1406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:03.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1a1406ccf0 0x7f1a1406f1b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1a28197be0 tx=0x7f1a1800b5b0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a280ff090 msgr2=0x7f1a2819c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:03.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a280ff090 0x7f1a2819c8e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1a240092e0 tx=0x7f1a2400be20 comp rx=0 tx=0).stop 2024-04-22T15:15:03.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 shutdown_connections 2024-04-22T15:15:03.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1a1406ccf0 0x7f1a1406f1b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a280ff090 0x7f1a2819c8e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 --2- 172.21.15.96:0/1857721426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a28105c70 0x7f1a2819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:03.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 >> 172.21.15.96:0/1857721426 conn(0x7f1a28075f00 msgr2=0x7f1a280fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:03.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 shutdown_connections 2024-04-22T15:15:03.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:01.962+0000 7f1a30418700 1 -- 172.21.15.96:0/1857721426 wait complete. 2024-04-22T15:15:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:02 smithi196 ceph-mon[34717]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:02 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1857721426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T15:15:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:02 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3101695501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T15:15:03.750 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542177 for osd.1 2024-04-22T15:15:03.750 DEBUG:teuthology.parallel:result is None 2024-04-22T15:15:04.570 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182418 got 107374182421 for osd.3 2024-04-22T15:15:04.570 DEBUG:teuthology.parallel:result is None 2024-04-22T15:15:04.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.776+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/1032943047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 msgr2=0x7fb9f40fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:04.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.776+0000 7fb9f8ff4700 1 --2- 172.21.15.96:0/1032943047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 0x7fb9f40fff20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009910 tx=0x7fb9dc009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:04.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.777+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/1032943047 shutdown_connections 2024-04-22T15:15:04.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.777+0000 7fb9f8ff4700 1 --2- 172.21.15.96:0/1032943047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 0x7fb9f40fff20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:04.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.777+0000 7fb9f8ff4700 1 --2- 172.21.15.96:0/1032943047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9f4105aa0 0x7fb9f4105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:04.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.777+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/1032943047 >> 172.21.15.96:0/1032943047 conn(0x7fb9f4076740 msgr2=0x7fb9f40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:04.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.777+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/1032943047 shutdown_connections 2024-04-22T15:15:04.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.777+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/1032943047 wait complete. 2024-04-22T15:15:04.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f8ff4700 1 Processor -- start 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f8ff4700 1 -- start start 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f8ff4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 0x7fb9f4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f8ff4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9f4105aa0 0x7fb9f4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f8ff4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f4198510 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f8ff4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f4198650 con 0x7fb9f4105aa0 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 0x7fb9f4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 0x7fb9f4197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59608/0 (socket says 172.21.15.96:59608) 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f259c700 1 -- 172.21.15.96:0/2227169157 learned_addr learned my addr 172.21.15.96:0/2227169157 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9e9d9b700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9f4105aa0 0x7fb9f4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f259c700 1 -- 172.21.15.96:0/2227169157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9f4105aa0 msgr2=0x7fb9f4197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f259c700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9f4105aa0 0x7fb9f4197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:04.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9f259c700 1 -- 172.21.15.96:0/2227169157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0095f0 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.778+0000 7fb9e9d9b700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9f4105aa0 0x7fb9f4197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:15:04.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.779+0000 7fb9f259c700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 0x7fb9f4197a00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400b4b0 tx=0x7fb9e400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:04.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.779+0000 7fb9ebfff700 1 -- 172.21.15.96:0/2227169157 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e4011800 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.779+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f41a0a70 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.779+0000 7fb9ebfff700 1 -- 172.21.15.96:0/2227169157 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9e4011e20 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.779+0000 7fb9ebfff700 1 -- 172.21.15.96:0/2227169157 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e400f2e0 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.779+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f41a0e80 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.780+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f41080c0 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.780+0000 7fb9ebfff700 1 -- 172.21.15.96:0/2227169157 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb9e4011960 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.781+0000 7fb9ebfff700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb9e006cce0 0x7fb9e006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:04.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.781+0000 7fb9e9d9b700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb9e006cce0 0x7fb9e006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:04.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.781+0000 7fb9ebfff700 1 -- 172.21.15.96:0/2227169157 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb9e40933b0 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.781+0000 7fb9e9d9b700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb9e006cce0 0x7fb9e006f1a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc006010 tx=0x7fb9dc00b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:04.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.785+0000 7fb9ebfff700 1 -- 172.21.15.96:0/2227169157 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb9e40554d0 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.980+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb9f41011e0 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.984 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:04 smithi096 ceph-mon[26688]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:04.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.980+0000 7fb9ebfff700 1 -- 172.21.15.96:0/2227169157 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb9e4058af0 con 0x7fb9f40ffaa0 2024-04-22T15:15:04.984 INFO:teuthology.orchestra.run.smithi096.stdout:141733920785 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb9e006cce0 msgr2=0x7fb9e006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb9e006cce0 0x7fb9e006f1a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc006010 tx=0x7fb9dc00b540 comp rx=0 tx=0).stop 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 msgr2=0x7fb9f4197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 0x7fb9f4197a00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400b4b0 tx=0x7fb9e400b870 comp rx=0 tx=0).stop 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 shutdown_connections 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb9e006cce0 0x7fb9e006f1a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9f40ffaa0 0x7fb9f4197a00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 --2- 172.21.15.96:0/2227169157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9f4105aa0 0x7fb9f4197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 >> 172.21.15.96:0/2227169157 conn(0x7fb9f4076740 msgr2=0x7fb9f40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 shutdown_connections 2024-04-22T15:15:04.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:04.982+0000 7fb9f8ff4700 1 -- 172.21.15.96:0/2227169157 wait complete. 2024-04-22T15:15:05.062 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:04 smithi196 ceph-mon[34717]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:05.564 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920783 got 141733920785 for osd.4 2024-04-22T15:15:05.565 DEBUG:teuthology.parallel:result is None 2024-04-22T15:15:05.565 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T15:15:05.565 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T15:15:05.772 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:05.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:05 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2227169157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T15:15:06.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:05 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2227169157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T15:15:07.117 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:06 smithi096 ceph-mon[26688]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:07.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:06 smithi196 ceph-mon[34717]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:07.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 -- 172.21.15.96:0/3551669116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 msgr2=0x7f3394104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:07.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 --2- 172.21.15.96:0/3551669116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 0x7f3394104940 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3388009910 tx=0x7f3388009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:07.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 -- 172.21.15.96:0/3551669116 shutdown_connections 2024-04-22T15:15:07.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 --2- 172.21.15.96:0/3551669116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 0x7f3394104940 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:07.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 --2- 172.21.15.96:0/3551669116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33940ffb50 0x7f33940fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:07.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 -- 172.21.15.96:0/3551669116 >> 172.21.15.96:0/3551669116 conn(0x7f3394076740 msgr2=0x7f33940fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:07.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 -- 172.21.15.96:0/3551669116 shutdown_connections 2024-04-22T15:15:07.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 -- 172.21.15.96:0/3551669116 wait complete. 2024-04-22T15:15:07.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 Processor -- start 2024-04-22T15:15:07.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.349+0000 7f339a4b2700 1 -- start start 2024-04-22T15:15:07.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f339a4b2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33940ffb50 0x7f339419d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:07.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f339a4b2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 0x7f339419d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:07.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f339a4b2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339419de90 con 0x7f3394100470 2024-04-22T15:15:07.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f339a4b2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33941a1b10 con 0x7f33940ffb50 2024-04-22T15:15:07.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33937fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 0x7f339419d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:07.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33937fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 0x7f339419d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59618/0 (socket says 172.21.15.96:59618) 2024-04-22T15:15:07.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33937fe700 1 -- 172.21.15.96:0/2987684502 learned_addr learned my addr 172.21.15.96:0/2987684502 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f3393fff700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33940ffb50 0x7f339419d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33937fe700 1 -- 172.21.15.96:0/2987684502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33940ffb50 msgr2=0x7f339419d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33937fe700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33940ffb50 0x7f339419d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33937fe700 1 -- 172.21.15.96:0/2987684502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33880095f0 con 0x7f3394100470 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33937fe700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 0x7f339419d7b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f338800bb40 tx=0x7f338800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33917fa700 1 -- 172.21.15.96:0/2987684502 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f338801d070 con 0x7f3394100470 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33917fa700 1 -- 172.21.15.96:0/2987684502 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3388003d50 con 0x7f3394100470 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33941a1d90 con 0x7f3394100470 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.350+0000 7f33917fa700 1 -- 172.21.15.96:0/2987684502 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3388017400 con 0x7f3394100470 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.351+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33941a2280 con 0x7f3394100470 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.351+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394068ea0 con 0x7f3394100470 2024-04-22T15:15:07.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.352+0000 7f33917fa700 1 -- 172.21.15.96:0/2987684502 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3388003890 con 0x7f3394100470 2024-04-22T15:15:07.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.352+0000 7f33917fa700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f337c06cce0 0x7f337c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:07.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.352+0000 7f33917fa700 1 -- 172.21.15.96:0/2987684502 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f338808d2a0 con 0x7f3394100470 2024-04-22T15:15:07.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.352+0000 7f3393fff700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f337c06cce0 0x7f337c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:07.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.353+0000 7f3393fff700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f337c06cce0 0x7f337c06f1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3384006fd0 tx=0x7f3384008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:07.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.357+0000 7f33917fa700 1 -- 172.21.15.96:0/2987684502 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f3388057a60 con 0x7f3394100470 2024-04-22T15:15:07.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.552+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33941144b0 con 0x7f337c06cce0 2024-04-22T15:15:07.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.553+0000 7f33917fa700 1 -- 172.21.15.96:0/2987684502 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27803 (secure 0 0 0) 0x7f33941144b0 con 0x7f337c06cce0 2024-04-22T15:15:07.557 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:07.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f337c06cce0 msgr2=0x7f337c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:07.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f337c06cce0 0x7f337c06f1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3384006fd0 tx=0x7f3384008040 comp rx=0 tx=0).stop 2024-04-22T15:15:07.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 msgr2=0x7f339419d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:07.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 0x7f339419d7b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f338800bb40 tx=0x7f338800bc20 comp rx=0 tx=0).stop 2024-04-22T15:15:07.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 shutdown_connections 2024-04-22T15:15:07.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f337c06cce0 0x7f337c06f1a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:07.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33940ffb50 0x7f339419d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:07.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 --2- 172.21.15.96:0/2987684502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3394100470 0x7f339419d7b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:07.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 >> 172.21.15.96:0/2987684502 conn(0x7f3394076740 msgr2=0x7f339410e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:07.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 shutdown_connections 2024-04-22T15:15:07.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:07.556+0000 7f339a4b2700 1 -- 172.21.15.96:0/2987684502 wait complete. 2024-04-22T15:15:07.560 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-04-22T15:15:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:15:08.218 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-04-22T15:15:05.657135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":170056,"kb_used_data":3828,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":562300856,"statfs":{"total":575970213888,"available":575796076544,"internally_reserved":0,"allocated":3919872,"data_stored":2588574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002225"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":186,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T15:14:15.339436+0000","last_change":"2024-04-22T15:13:49.039784+0000","last_active":"2024-04-22T15:14:15.339436+0000","last_peered":"2024-04-22T15:14:15.339436+0000","last_clean":"2024-04-22T15:14:15.339436+0000","last_became_active":"2024-04-22T15:13:49.039497+0000","last_became_peered":"2024-04-22T15:13:49.039497+0000","last_unstale":"2024-04-22T15:14:15.339436+0000","last_undegraded":"2024-04-22T15:14:15.339436+0000","last_fullsized":"2024-04-22T15:14:15.339436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T15:13:01.924868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T15:13:01.924868+0000","last_clean_scrub_stamp":"2024-04-22T15:13:01.924868+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:06:54.337594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":38,"seq":163208757261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717784,"statfs":{"total":95995035648,"available":95967010816,"internally_reserved":0,"allocated":356352,"data_stored":136245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]}]},{"osd":4,"up_from":33,"seq":141733920786,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717784,"statfs":{"total":95995035648,"available":95967010816,"internally_reserved":0,"allocated":356352,"data_stored":136245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":11.871,"5min":11.871,"15min":11.871},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Mon Apr 22 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":11.848000000000001,"5min":11.848000000000001,"15min":11.848000000000001},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Apr 22 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.60899999999999999}]},{"osd":3,"last update":"Mon Apr 22 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.47499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":726613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.48899999999999999}]},{"osd":1,"last update":"Mon Apr 22 15:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.372},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.38300000000000001}]},{"osd":2,"last update":"Mon Apr 22 15:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.433},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Apr 22 15:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31468,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713684,"statfs":{"total":95995035648,"available":95962812416,"internally_reserved":0,"allocated":356352,"data_stored":136245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 15:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.246},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.27600000000000002}]},{"osd":3,"last update":"Mon Apr 22 15:14:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.64800000000000002}]},{"osd":4,"last update":"Mon Apr 22 15:14:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.70099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":726613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 15:14:33 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.312,"15min":0.308},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.32200000000000001,"15min":0.317},"min":{"1min":0.26600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Apr 22 15:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.308},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Apr 22 15:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Apr 22 15:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.496},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":726613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:14:33 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.30299999999999999,"15min":0.30099999999999999},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.313,"15min":0.312},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Apr 22 15:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.254},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.28199999999999997}]},{"osd":3,"last update":"Mon Apr 22 15:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":19.57,"5min":19.57,"15min":19.57},"last":0.66100000000000003}]},{"osd":4,"last update":"Mon Apr 22 15:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.63},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T15:15:08.232 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T15:15:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:07 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:15:08.440 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:08.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:08 smithi096 ceph-mon[26688]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:15:08.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:08 smithi096 ceph-mon[26688]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:08 smithi196 ceph-mon[34717]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:15:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:08 smithi196 ceph-mon[34717]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:10.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.045+0000 7f3b3a536700 1 -- 172.21.15.96:0/1556880584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 msgr2=0x7f3b341057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:10.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.045+0000 7f3b3a536700 1 --2- 172.21.15.96:0/1556880584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 0x7f3b341057f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009910 tx=0x7f3b28009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:10.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.045+0000 7f3b3a536700 1 -- 172.21.15.96:0/1556880584 shutdown_connections 2024-04-22T15:15:10.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.045+0000 7f3b3a536700 1 --2- 172.21.15.96:0/1556880584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 0x7f3b341057f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:10.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.045+0000 7f3b3a536700 1 --2- 172.21.15.96:0/1556880584 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b34100df0 0x7f3b341011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:10.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.045+0000 7f3b3a536700 1 -- 172.21.15.96:0/1556880584 >> 172.21.15.96:0/1556880584 conn(0x7f3b340fc6a0 msgr2=0x7f3b340feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:10.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.045+0000 7f3b3a536700 1 -- 172.21.15.96:0/1556880584 shutdown_connections 2024-04-22T15:15:10.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.045+0000 7f3b3a536700 1 -- 172.21.15.96:0/1556880584 wait complete. 2024-04-22T15:15:10.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b3a536700 1 Processor -- start 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b3a536700 1 -- start start 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b3a536700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b34100df0 0x7f3b3419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b3a536700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 0x7f3b3419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b3a536700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3419d450 con 0x7f3b341017a0 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b3a536700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34196a30 con 0x7f3b34100df0 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b337fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 0x7f3b3419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b337fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 0x7f3b3419ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59642/0 (socket says 172.21.15.96:59642) 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b337fe700 1 -- 172.21.15.96:0/3729015743 learned_addr learned my addr 172.21.15.96:0/3729015743 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.046+0000 7f3b33fff700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b34100df0 0x7f3b3419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.047+0000 7f3b337fe700 1 -- 172.21.15.96:0/3729015743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b34100df0 msgr2=0x7f3b3419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.047+0000 7f3b337fe700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b34100df0 0x7f3b3419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.047+0000 7f3b337fe700 1 -- 172.21.15.96:0/3729015743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b280095f0 con 0x7f3b341017a0 2024-04-22T15:15:10.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.047+0000 7f3b337fe700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 0x7f3b3419ce30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800bd70 tx=0x7f3b2800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:10.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.047+0000 7f3b317fa700 1 -- 172.21.15.96:0/3729015743 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b2801d070 con 0x7f3b341017a0 2024-04-22T15:15:10.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.047+0000 7f3b317fa700 1 -- 172.21.15.96:0/3729015743 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b280056c0 con 0x7f3b341017a0 2024-04-22T15:15:10.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.047+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b34196cb0 con 0x7f3b341017a0 2024-04-22T15:15:10.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.047+0000 7f3b317fa700 1 -- 172.21.15.96:0/3729015743 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b28017400 con 0x7f3b341017a0 2024-04-22T15:15:10.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.047+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b341971a0 con 0x7f3b341017a0 2024-04-22T15:15:10.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.048+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b34109140 con 0x7f3b341017a0 2024-04-22T15:15:10.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.049+0000 7f3b317fa700 1 -- 172.21.15.96:0/3729015743 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3b28003a60 con 0x7f3b341017a0 2024-04-22T15:15:10.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.049+0000 7f3b317fa700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3b1c06cce0 0x7f3b1c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:10.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.049+0000 7f3b317fa700 1 -- 172.21.15.96:0/3729015743 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3b2808d2b0 con 0x7f3b341017a0 2024-04-22T15:15:10.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.049+0000 7f3b33fff700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3b1c06cce0 0x7f3b1c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:10.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.050+0000 7f3b33fff700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3b1c06cce0 0x7f3b1c06f1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009a10 tx=0x7f3b24009190 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.054+0000 7f3b317fa700 1 -- 172.21.15.96:0/3729015743 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f3b2805b450 con 0x7f3b341017a0 2024-04-22T15:15:10.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.243+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b34197f60 con 0x7f3b1c06cce0 2024-04-22T15:15:10.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.244+0000 7f3b317fa700 1 -- 172.21.15.96:0/3729015743 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27815 (secure 0 0 0) 0x7f3b34197f60 con 0x7f3b1c06cce0 2024-04-22T15:15:10.248 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:10.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.246+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3b1c06cce0 msgr2=0x7f3b1c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:10.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.246+0000 7f3b3a536700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3b1c06cce0 0x7f3b1c06f1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009a10 tx=0x7f3b24009190 comp rx=0 tx=0).stop 2024-04-22T15:15:10.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.246+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 msgr2=0x7f3b3419ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:10.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.246+0000 7f3b3a536700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 0x7f3b3419ce30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800bd70 tx=0x7f3b2800bda0 comp rx=0 tx=0).stop 2024-04-22T15:15:10.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.247+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 shutdown_connections 2024-04-22T15:15:10.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.247+0000 7f3b3a536700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f3b1c06cce0 0x7f3b1c06f1a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:10.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.247+0000 7f3b3a536700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b34100df0 0x7f3b3419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:10.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.247+0000 7f3b3a536700 1 --2- 172.21.15.96:0/3729015743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b341017a0 0x7f3b3419ce30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:10.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.247+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 >> 172.21.15.96:0/3729015743 conn(0x7f3b340fc6a0 msgr2=0x7f3b340fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:10.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.247+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 shutdown_connections 2024-04-22T15:15:10.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:10.247+0000 7f3b3a536700 1 -- 172.21.15.96:0/3729015743 wait complete. 2024-04-22T15:15:10.251 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-04-22T15:15:10.867 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-04-22T15:15:09.657881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":170056,"kb_used_data":3828,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":562300856,"statfs":{"total":575970213888,"available":575796076544,"internally_reserved":0,"allocated":3919872,"data_stored":2588574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002126"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":186,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T15:14:15.339436+0000","last_change":"2024-04-22T15:13:49.039784+0000","last_active":"2024-04-22T15:14:15.339436+0000","last_peered":"2024-04-22T15:14:15.339436+0000","last_clean":"2024-04-22T15:14:15.339436+0000","last_became_active":"2024-04-22T15:13:49.039497+0000","last_became_peered":"2024-04-22T15:13:49.039497+0000","last_unstale":"2024-04-22T15:14:15.339436+0000","last_undegraded":"2024-04-22T15:14:15.339436+0000","last_fullsized":"2024-04-22T15:14:15.339436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T15:13:01.924868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T15:13:01.924868+0000","last_clean_scrub_stamp":"2024-04-22T15:13:01.924868+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:06:54.337594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":38,"seq":163208757262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717784,"statfs":{"total":95995035648,"available":95967010816,"internally_reserved":0,"allocated":356352,"data_stored":136245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920787,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717784,"statfs":{"total":95995035648,"available":95967010816,"internally_reserved":0,"allocated":356352,"data_stored":136245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":11.871,"5min":11.871,"15min":11.871},"last":0.439},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56999999999999995}]},{"osd":1,"last update":"Mon Apr 22 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":11.848000000000001,"5min":11.848000000000001,"15min":11.848000000000001},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Apr 22 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Apr 22 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182423,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":726613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Apr 22 15:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.36799999999999999}]},{"osd":2,"last update":"Mon Apr 22 15:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Apr 22 15:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.313},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.41799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31468,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713684,"statfs":{"total":95995035648,"available":95962812416,"internally_reserved":0,"allocated":356352,"data_stored":136245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.248},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.28799999999999998}]},{"osd":1,"last update":"Mon Apr 22 15:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Apr 22 15:14:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.68000000000000005}]},{"osd":4,"last update":"Mon Apr 22 15:14:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":726613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 15:14:33 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.312,"15min":0.308},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.32200000000000001,"15min":0.317},"min":{"1min":0.26600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.57899999999999996}]},{"osd":2,"last update":"Mon Apr 22 15:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.37,"5min":0.377,"15min":0.378},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Apr 22 15:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.66800000000000004}]},{"osd":4,"last update":"Mon Apr 22 15:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":726613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:14:33 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.30299999999999999,"15min":0.30099999999999999},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.313,"15min":0.312},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Apr 22 15:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.254},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.28199999999999997}]},{"osd":3,"last update":"Mon Apr 22 15:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":19.57,"5min":19.57,"15min":19.57},"last":0.66100000000000003}]},{"osd":4,"last update":"Mon Apr 22 15:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.63},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T15:15:10.868 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T15:15:10.868 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T15:15:10.868 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T15:15:10.869 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T15:15:11.073 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:11.195 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:10 smithi096 ceph-mon[26688]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:11.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:10 smithi196 ceph-mon[34717]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:12.086 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:11 smithi096 ceph-mon[26688]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:15:12.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:11 smithi196 ceph-mon[34717]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:15:12.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.617+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2521282240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 msgr2=0x7fb8b4105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:12.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.617+0000 7fb8bc6f3700 1 --2- 172.21.15.96:0/2521282240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 0x7fb8b4105810 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4009910 tx=0x7fb8a4009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:12.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.617+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2521282240 shutdown_connections 2024-04-22T15:15:12.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.617+0000 7fb8bc6f3700 1 --2- 172.21.15.96:0/2521282240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 0x7fb8b4105810 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:12.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.617+0000 7fb8bc6f3700 1 --2- 172.21.15.96:0/2521282240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8b4100e10 0x7fb8b41011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:12.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.617+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2521282240 >> 172.21.15.96:0/2521282240 conn(0x7fb8b40fc6e0 msgr2=0x7fb8b40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:12.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.617+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2521282240 shutdown_connections 2024-04-22T15:15:12.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.617+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2521282240 wait complete. 2024-04-22T15:15:12.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8bc6f3700 1 Processor -- start 2024-04-22T15:15:12.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8bc6f3700 1 -- start start 2024-04-22T15:15:12.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8bc6f3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8b4100e10 0x7fb8b419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:12.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8bc6f3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 0x7fb8b419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:12.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8bc6f3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b419d470 con 0x7fb8b41017c0 2024-04-22T15:15:12.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8bc6f3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4196a50 con 0x7fb8b4100e10 2024-04-22T15:15:12.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8b9c8e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 0x7fb8b419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:12.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8b9c8e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 0x7fb8b419ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59182/0 (socket says 172.21.15.96:59182) 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8b9c8e700 1 -- 172.21.15.96:0/2218348732 learned_addr learned my addr 172.21.15.96:0/2218348732 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8ba48f700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8b4100e10 0x7fb8b419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8b9c8e700 1 -- 172.21.15.96:0/2218348732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8b4100e10 msgr2=0x7fb8b419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8b9c8e700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8b4100e10 0x7fb8b419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.618+0000 7fb8b9c8e700 1 -- 172.21.15.96:0/2218348732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a40095f0 con 0x7fb8b41017c0 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.619+0000 7fb8b9c8e700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 0x7fb8b419ce50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400bd70 tx=0x7fb8a400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.619+0000 7fb8ab7fe700 1 -- 172.21.15.96:0/2218348732 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8a401d070 con 0x7fb8b41017c0 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.619+0000 7fb8ab7fe700 1 -- 172.21.15.96:0/2218348732 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8a40056c0 con 0x7fb8b41017c0 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.619+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b4196cd0 con 0x7fb8b41017c0 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.619+0000 7fb8ab7fe700 1 -- 172.21.15.96:0/2218348732 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8a4017400 con 0x7fb8b41017c0 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.619+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b41971c0 con 0x7fb8b41017c0 2024-04-22T15:15:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.620+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b41022f0 con 0x7fb8b41017c0 2024-04-22T15:15:12.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.620+0000 7fb8ab7fe700 1 -- 172.21.15.96:0/2218348732 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb8a4003a60 con 0x7fb8b41017c0 2024-04-22T15:15:12.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.620+0000 7fb8ab7fe700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb8a006cce0 0x7fb8a006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:12.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.621+0000 7fb8ba48f700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb8a006cce0 0x7fb8a006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:12.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.621+0000 7fb8ab7fe700 1 -- 172.21.15.96:0/2218348732 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb8a408d320 con 0x7fb8b41017c0 2024-04-22T15:15:12.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.621+0000 7fb8ba48f700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb8a006cce0 0x7fb8a006f1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009a10 tx=0x7fb8b0009190 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:12.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.625+0000 7fb8ab7fe700 1 -- 172.21.15.96:0/2218348732 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb8a405b4c0 con 0x7fb8b41017c0 2024-04-22T15:15:12.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.864+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb8b404ebb0 con 0x7fb8b41017c0 2024-04-22T15:15:12.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.864+0000 7fb8ab7fe700 1 -- 172.21.15.96:0/2218348732 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb8a405b050 con 0x7fb8b41017c0 2024-04-22T15:15:12.868 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:12.868 INFO:teuthology.orchestra.run.smithi096.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb8a006cce0 msgr2=0x7fb8a006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb8a006cce0 0x7fb8a006f1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009a10 tx=0x7fb8b0009190 comp rx=0 tx=0).stop 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 msgr2=0x7fb8b419ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 0x7fb8b419ce50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400bd70 tx=0x7fb8a400bda0 comp rx=0 tx=0).stop 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 shutdown_connections 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb8a006cce0 0x7fb8a006f1a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8b4100e10 0x7fb8b419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 --2- 172.21.15.96:0/2218348732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b41017c0 0x7fb8b419ce50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 >> 172.21.15.96:0/2218348732 conn(0x7fb8b40fc6e0 msgr2=0x7fb8b40fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:12.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 shutdown_connections 2024-04-22T15:15:12.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:12.866+0000 7fb8bc6f3700 1 -- 172.21.15.96:0/2218348732 wait complete. 2024-04-22T15:15:12.880 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:12 smithi096 ceph-mon[26688]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:13.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:12 smithi196 ceph-mon[34717]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:13.539 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T15:15:13.539 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T15:15:13.540 INFO:teuthology.run_tasks:Running task print... 2024-04-22T15:15:13.550 INFO:teuthology.task.print:**** done end installing reef cephadm ... 2024-04-22T15:15:13.550 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:15:13.570 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:15:13.570 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-22T15:15:13.777 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:14.111 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:13 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2218348732' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T15:15:14.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:13 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2218348732' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T15:15:14.951 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:14 smithi096 ceph-mon[26688]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:15.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:14 smithi196 ceph-mon[34717]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.310+0000 7f1960970700 1 -- 172.21.15.96:0/1784310983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 msgr2=0x7f195c1022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.310+0000 7f1960970700 1 --2- 172.21.15.96:0/1784310983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 0x7f195c1022a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1950009910 tx=0x7f1950009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.310+0000 7f1960970700 1 -- 172.21.15.96:0/1784310983 shutdown_connections 2024-04-22T15:15:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.310+0000 7f1960970700 1 --2- 172.21.15.96:0/1784310983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 0x7f195c1022a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.310+0000 7f1960970700 1 --2- 172.21.15.96:0/1784310983 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f195c100c40 0x7f195c101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.310+0000 7f1960970700 1 -- 172.21.15.96:0/1784310983 >> 172.21.15.96:0/1784310983 conn(0x7f195c0fc200 msgr2=0x7f195c0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.310+0000 7f1960970700 1 -- 172.21.15.96:0/1784310983 shutdown_connections 2024-04-22T15:15:15.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.310+0000 7f1960970700 1 -- 172.21.15.96:0/1784310983 wait complete. 2024-04-22T15:15:15.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.311+0000 7f1960970700 1 Processor -- start 2024-04-22T15:15:15.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.311+0000 7f1960970700 1 -- start start 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.311+0000 7f1960970700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f195c100c40 0x7f195c196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.311+0000 7f1960970700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 0x7f195c196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.311+0000 7f1960970700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c197100 con 0x7f195c101e40 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.311+0000 7f1960970700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c197240 con 0x7f195c100c40 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.311+0000 7f1959d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 0x7f195c196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f1959d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 0x7f195c196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59196/0 (socket says 172.21.15.96:59196) 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f1959d9b700 1 -- 172.21.15.96:0/1347281739 learned_addr learned my addr 172.21.15.96:0/1347281739 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f195a59c700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f195c100c40 0x7f195c196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f1959d9b700 1 -- 172.21.15.96:0/1347281739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f195c100c40 msgr2=0x7f195c196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f1959d9b700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f195c100c40 0x7f195c196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f1959d9b700 1 -- 172.21.15.96:0/1347281739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500095f0 con 0x7f195c101e40 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f1959d9b700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 0x7f195c196b70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1950006010 tx=0x7f195000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:15.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f194b7fe700 1 -- 172.21.15.96:0/1347281739 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f195001d070 con 0x7f195c101e40 2024-04-22T15:15:15.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f194b7fe700 1 -- 172.21.15.96:0/1347281739 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f195000f460 con 0x7f195c101e40 2024-04-22T15:15:15.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c19bc00 con 0x7f195c101e40 2024-04-22T15:15:15.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f194b7fe700 1 -- 172.21.15.96:0/1347281739 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1950021620 con 0x7f195c101e40 2024-04-22T15:15:15.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.312+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c19c140 con 0x7f195c101e40 2024-04-22T15:15:15.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.313+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195c1907c0 con 0x7f195c101e40 2024-04-22T15:15:15.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.314+0000 7f194b7fe700 1 -- 172.21.15.96:0/1347281739 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f195000f5d0 con 0x7f195c101e40 2024-04-22T15:15:15.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.314+0000 7f194b7fe700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f194406cce0 0x7f194406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:15.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.314+0000 7f194b7fe700 1 -- 172.21.15.96:0/1347281739 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f195008d230 con 0x7f195c101e40 2024-04-22T15:15:15.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.314+0000 7f195a59c700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f194406cce0 0x7f194406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:15.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.314+0000 7f195a59c700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f194406cce0 0x7f194406f1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f195c101ca0 tx=0x7f194c00a3b0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.319+0000 7f194b7fe700 1 -- 172.21.15.96:0/1347281739 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f195005b3d0 con 0x7f195c101e40 2024-04-22T15:15:15.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.514+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f195c02cde0 con 0x7f195c101e40 2024-04-22T15:15:15.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.518+0000 7f194b7fe700 1 -- 172.21.15.96:0/1347281739 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v21)=0 v21) v1 ==== 143+0+0 (secure 0 0 0) 0x7f195005af60 con 0x7f195c101e40 2024-04-22T15:15:15.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f194406cce0 msgr2=0x7f194406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:15.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f194406cce0 0x7f194406f1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f195c101ca0 tx=0x7f194c00a3b0 comp rx=0 tx=0).stop 2024-04-22T15:15:15.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 msgr2=0x7f195c196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 0x7f195c196b70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1950006010 tx=0x7f195000bd70 comp rx=0 tx=0).stop 2024-04-22T15:15:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 shutdown_connections 2024-04-22T15:15:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f194406cce0 0x7f194406f1a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f195c100c40 0x7f195c196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 --2- 172.21.15.96:0/1347281739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f195c101e40 0x7f195c196b70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 >> 172.21.15.96:0/1347281739 conn(0x7f195c0fc200 msgr2=0x7f195c105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 shutdown_connections 2024-04-22T15:15:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:15.521+0000 7f1960970700 1 -- 172.21.15.96:0/1347281739 wait complete. 2024-04-22T15:15:16.219 INFO:teuthology.run_tasks:Running task print... 2024-04-22T15:15:16.227 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-22T15:15:16.227 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:15:16.235 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:15:16.235 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T15:15:16.454 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:16.782 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:16 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1347281739' entity='client.admin' 2024-04-22T15:15:16.782 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:15:16.783 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:16 smithi096 ceph-mon[26688]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:16.783 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:16.783 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:15:16.783 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:16 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:16 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1347281739' entity='client.admin' 2024-04-22T15:15:17.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:15:17.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:16 smithi196 ceph-mon[34717]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:17.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:17.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:15:17.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:16 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:18.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 -- 172.21.15.96:0/2566974915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f40ff4a0 msgr2=0x7ff9f41018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:18.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 --2- 172.21.15.96:0/2566974915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f40ff4a0 0x7ff9f41018d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0009910 tx=0x7ff9f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:18.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 -- 172.21.15.96:0/2566974915 shutdown_connections 2024-04-22T15:15:18.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 --2- 172.21.15.96:0/2566974915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9f4101e10 0x7ff9f41042a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:18.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 --2- 172.21.15.96:0/2566974915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f40ff4a0 0x7ff9f41018d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:18.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 -- 172.21.15.96:0/2566974915 >> 172.21.15.96:0/2566974915 conn(0x7ff9f40fb0c0 msgr2=0x7ff9f40fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:18.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 -- 172.21.15.96:0/2566974915 shutdown_connections 2024-04-22T15:15:18.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 -- 172.21.15.96:0/2566974915 wait complete. 2024-04-22T15:15:18.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 Processor -- start 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.095+0000 7ff9fb467700 1 -- start start 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9fb467700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9f40ff4a0 0x7ff9f419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9fb467700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f4101e10 0x7ff9f419d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9fb467700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f419d7a0 con 0x7ff9f4101e10 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9fb467700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f419d8e0 con 0x7ff9f40ff4a0 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9f8a02700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f4101e10 0x7ff9f419d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9f8a02700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f4101e10 0x7ff9f419d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59218/0 (socket says 172.21.15.96:59218) 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9f8a02700 1 -- 172.21.15.96:0/3635092265 learned_addr learned my addr 172.21.15.96:0/3635092265 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9f9203700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9f40ff4a0 0x7ff9f419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9f8a02700 1 -- 172.21.15.96:0/3635092265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9f40ff4a0 msgr2=0x7ff9f419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9f8a02700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9f40ff4a0 0x7ff9f419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9f8a02700 1 -- 172.21.15.96:0/3635092265 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f00095f0 con 0x7ff9f4101e10 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9f8a02700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f4101e10 0x7ff9f419d180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400d8d0 tx=0x7ff9e400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:18.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9ea7fc700 1 -- 172.21.15.96:0/3635092265 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9e40098a0 con 0x7ff9f4101e10 2024-04-22T15:15:18.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9ea7fc700 1 -- 172.21.15.96:0/3635092265 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9e4010460 con 0x7ff9f4101e10 2024-04-22T15:15:18.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9ea7fc700 1 -- 172.21.15.96:0/3635092265 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9e400f5f0 con 0x7ff9f4101e10 2024-04-22T15:15:18.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f41a22f0 con 0x7ff9f4101e10 2024-04-22T15:15:18.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.096+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f41a28c0 con 0x7ff9f4101e10 2024-04-22T15:15:18.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.097+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f404ebb0 con 0x7ff9f4101e10 2024-04-22T15:15:18.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.098+0000 7ff9ea7fc700 1 -- 172.21.15.96:0/3635092265 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff9e4009a00 con 0x7ff9f4101e10 2024-04-22T15:15:18.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.098+0000 7ff9ea7fc700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9e006cce0 0x7ff9e006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:18.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.098+0000 7ff9ea7fc700 1 -- 172.21.15.96:0/3635092265 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff9e408bd30 con 0x7ff9f4101e10 2024-04-22T15:15:18.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.098+0000 7ff9f9203700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9e006cce0 0x7ff9e006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:18.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.099+0000 7ff9f9203700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9e006cce0 0x7ff9e006f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0005090 tx=0x7ff9f0005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:18.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.103+0000 7ff9ea7fc700 1 -- 172.21.15.96:0/3635092265 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff9e4059f50 con 0x7ff9f4101e10 2024-04-22T15:15:18.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.315+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9f4102890 con 0x7ff9e006cce0 2024-04-22T15:15:18.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.316+0000 7ff9ea7fc700 1 -- 172.21.15.96:0/3635092265 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7ff9f4102890 con 0x7ff9e006cce0 2024-04-22T15:15:18.319 INFO:teuthology.orchestra.run.smithi096.stdout:Backend: cephadm 2024-04-22T15:15:18.319 INFO:teuthology.orchestra.run.smithi096.stdout:Available: Yes 2024-04-22T15:15:18.319 INFO:teuthology.orchestra.run.smithi096.stdout:Paused: No 2024-04-22T15:15:18.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9e006cce0 msgr2=0x7ff9e006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:18.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9e006cce0 0x7ff9e006f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0005090 tx=0x7ff9f0005960 comp rx=0 tx=0).stop 2024-04-22T15:15:18.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f4101e10 msgr2=0x7ff9f419d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:18.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f4101e10 0x7ff9f419d180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400d8d0 tx=0x7ff9e400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:15:18.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 shutdown_connections 2024-04-22T15:15:18.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff9e006cce0 0x7ff9e006f1a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:18.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9f40ff4a0 0x7ff9f419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:18.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 --2- 172.21.15.96:0/3635092265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9f4101e10 0x7ff9f419d180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:18.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 >> 172.21.15.96:0/3635092265 conn(0x7ff9f40fb0c0 msgr2=0x7ff9f40fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:18.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 shutdown_connections 2024-04-22T15:15:18.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:18.318+0000 7ff9fb467700 1 -- 172.21.15.96:0/3635092265 wait complete. 2024-04-22T15:15:18.935 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:18 smithi096 ceph-mon[26688]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:18.957 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T15:15:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:18 smithi196 ceph-mon[34717]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:19.163 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:19.974 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:19 smithi096 ceph-mon[26688]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:19 smithi196 ceph-mon[34717]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:20.844 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:20 smithi096 ceph-mon[26688]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:20.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.841+0000 7faaacffc700 1 -- 172.21.15.96:0/1999829561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 msgr2=0x7faaa80fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:20.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.841+0000 7faaacffc700 1 --2- 172.21.15.96:0/1999829561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 0x7faaa80fffb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009910 tx=0x7faa9c009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:20.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.841+0000 7faaacffc700 1 -- 172.21.15.96:0/1999829561 shutdown_connections 2024-04-22T15:15:20.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.841+0000 7faaacffc700 1 --2- 172.21.15.96:0/1999829561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faaa8100d90 0x7faaa81011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:20.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.841+0000 7faaacffc700 1 --2- 172.21.15.96:0/1999829561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 0x7faaa80fffb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:20.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.841+0000 7faaacffc700 1 -- 172.21.15.96:0/1999829561 >> 172.21.15.96:0/1999829561 conn(0x7faaa80fb110 msgr2=0x7faaa80fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:20.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.841+0000 7faaacffc700 1 -- 172.21.15.96:0/1999829561 shutdown_connections 2024-04-22T15:15:20.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.841+0000 7faaacffc700 1 -- 172.21.15.96:0/1999829561 wait complete. 2024-04-22T15:15:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaacffc700 1 Processor -- start 2024-04-22T15:15:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaacffc700 1 -- start start 2024-04-22T15:15:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaacffc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 0x7faaa8196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaacffc700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faaa8100d90 0x7faaa8196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaacffc700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa81972a0 con 0x7faaa80ffb90 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaacffc700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa81973e0 con 0x7faaa8100d90 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaa659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 0x7faaa8196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaa659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 0x7faaa8196740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54972/0 (socket says 172.21.15.96:54972) 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaa659c700 1 -- 172.21.15.96:0/1042215913 learned_addr learned my addr 172.21.15.96:0/1042215913 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.842+0000 7faaa5d9b700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faaa8100d90 0x7faaa8196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.843+0000 7faaa659c700 1 -- 172.21.15.96:0/1042215913 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faaa8100d90 msgr2=0x7faaa8196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.843+0000 7faaa659c700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faaa8100d90 0x7faaa8196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.843+0000 7faaa659c700 1 -- 172.21.15.96:0/1042215913 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c0095f0 con 0x7faaa80ffb90 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.843+0000 7faaa659c700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 0x7faaa8196740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0059d0 tx=0x7faa9c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.843+0000 7faa977fe700 1 -- 172.21.15.96:0/1042215913 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa9c01d070 con 0x7faaa80ffb90 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.843+0000 7faa977fe700 1 -- 172.21.15.96:0/1042215913 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa9c00f460 con 0x7faaa80ffb90 2024-04-22T15:15:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.843+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa819bd90 con 0x7faaa80ffb90 2024-04-22T15:15:20.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.843+0000 7faa977fe700 1 -- 172.21.15.96:0/1042215913 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa9c021640 con 0x7faaa80ffb90 2024-04-22T15:15:20.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.843+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa819c220 con 0x7faaa80ffb90 2024-04-22T15:15:20.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.844+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa804ec20 con 0x7faaa80ffb90 2024-04-22T15:15:20.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.844+0000 7faa977fe700 1 -- 172.21.15.96:0/1042215913 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faa9c00f5d0 con 0x7faaa80ffb90 2024-04-22T15:15:20.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.845+0000 7faa977fe700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7faa9006cce0 0x7faa9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:20.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.845+0000 7faa977fe700 1 -- 172.21.15.96:0/1042215913 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7faa9c08d190 con 0x7faaa80ffb90 2024-04-22T15:15:20.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.845+0000 7faaa5d9b700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7faa9006cce0 0x7faa9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:20.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.845+0000 7faaa5d9b700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7faa9006cce0 0x7faa9006f1a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faa98005950 tx=0x7faa9800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:20.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:20.849+0000 7faa977fe700 1 -- 172.21.15.96:0/1042215913 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7faaa804ec20 con 0x7faaa80ffb90 2024-04-22T15:15:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:20 smithi196 ceph-mon[34717]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:21.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.051+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faaa804ebb0 con 0x7faa9006cce0 2024-04-22T15:15:21.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.061+0000 7faa977fe700 1 -- 172.21.15.96:0/1042215913 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3100 (secure 0 0 0) 0x7faaa804ebb0 con 0x7faa9006cce0 2024-04-22T15:15:21.064 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:15:21.064 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (3m) 2m ago 5m 25.8M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:15:21.064 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (5m) 2m ago 5m 7042k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:15:21.064 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (4m) 67s ago 4m 6958k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:15:21.064 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (5m) 2m ago 5m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:15:21.064 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (4m) 67s ago 4m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:15:21.064 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (3m) 2m ago 5m 78.2M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:9283,8765,8443 running (6m) 2m ago 6m 488M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 12e0357210ef 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (4m) 67s ago 4m 449M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 ea53e28b40cc 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (6m) 2m ago 6m 43.3M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (4m) 67s ago 4m 40.5M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (5m) 2m ago 5m 14.4M - 1.5.0 0da6a335fe13 852e13a0c299 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (4m) 67s ago 4m 15.5M - 1.5.0 0da6a335fe13 4d02817c7cd0 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (3m) 2m ago 3m 37.7M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (2m) 2m ago 2m 38.2M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (2m) 2m ago 2m 43.1M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (2m) 67s ago 2m 41.6M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (98s) 67s ago 98s 38.4M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (72s) 67s ago 72s 11.5M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:15:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (3m) 2m ago 4m 39.9M - 2.43.0 a07b618ecd1d 87457032ed08 2024-04-22T15:15:21.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7faa9006cce0 msgr2=0x7faa9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:21.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7faa9006cce0 0x7faa9006f1a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faa98005950 tx=0x7faa9800b410 comp rx=0 tx=0).stop 2024-04-22T15:15:21.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 msgr2=0x7faaa8196740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:21.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 0x7faaa8196740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0059d0 tx=0x7faa9c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:15:21.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 shutdown_connections 2024-04-22T15:15:21.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7faa9006cce0 0x7faa9006f1a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:21.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaa80ffb90 0x7faaa8196740 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:21.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 --2- 172.21.15.96:0/1042215913 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faaa8100d90 0x7faaa8196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:21.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 >> 172.21.15.96:0/1042215913 conn(0x7faaa80fb110 msgr2=0x7faaa8068670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:21.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 shutdown_connections 2024-04-22T15:15:21.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:21.063+0000 7faaacffc700 1 -- 172.21.15.96:0/1042215913 wait complete. 2024-04-22T15:15:21.686 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T15:15:21.891 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:22.978 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:22 smithi096 ceph-mon[26688]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:22.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:22 smithi096 ceph-mon[26688]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:22.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:15:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:22 smithi196 ceph-mon[34717]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:23.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:22 smithi196 ceph-mon[34717]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:23.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:15:23.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.512+0000 7f265e713700 1 -- 172.21.15.96:0/1060759530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 msgr2=0x7f26580ff320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.512+0000 7f265e713700 1 --2- 172.21.15.96:0/1060759530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 0x7f26580ff320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2648009910 tx=0x7f2648009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 -- 172.21.15.96:0/1060759530 shutdown_connections 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 --2- 172.21.15.96:0/1060759530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2658101ee0 0x7f2658102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 --2- 172.21.15.96:0/1060759530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 0x7f26580ff320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 -- 172.21.15.96:0/1060759530 >> 172.21.15.96:0/1060759530 conn(0x7f26580fa860 msgr2=0x7f26580fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 -- 172.21.15.96:0/1060759530 shutdown_connections 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 -- 172.21.15.96:0/1060759530 wait complete. 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 Processor -- start 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 -- start start 2024-04-22T15:15:23.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 0x7f26581989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2658101ee0 0x7f2658198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2658199510 con 0x7f26580fef00 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.513+0000 7f265e713700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2658199650 con 0x7f2658101ee0 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f2657fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 0x7f26581989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f2657fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 0x7f26581989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55002/0 (socket says 172.21.15.96:55002) 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f2657fff700 1 -- 172.21.15.96:0/1940187231 learned_addr learned my addr 172.21.15.96:0/1940187231 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f26577fe700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2658101ee0 0x7f2658198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f2657fff700 1 -- 172.21.15.96:0/1940187231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2658101ee0 msgr2=0x7f2658198ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f2657fff700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2658101ee0 0x7f2658198ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:23.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f2657fff700 1 -- 172.21.15.96:0/1940187231 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26480095f0 con 0x7f26580fef00 2024-04-22T15:15:23.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f26577fe700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2658101ee0 0x7f2658198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:15:23.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f2657fff700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 0x7f26581989b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f26480059d0 tx=0x7f264800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:23.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f26557fa700 1 -- 172.21.15.96:0/1940187231 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f264801d070 con 0x7f26580fef00 2024-04-22T15:15:23.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265819e000 con 0x7f26580fef00 2024-04-22T15:15:23.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f26557fa700 1 -- 172.21.15.96:0/1940187231 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f264800f460 con 0x7f26580fef00 2024-04-22T15:15:23.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f26557fa700 1 -- 172.21.15.96:0/1940187231 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2648021620 con 0x7f26580fef00 2024-04-22T15:15:23.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.514+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265819e570 con 0x7f26580fef00 2024-04-22T15:15:23.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.515+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26581068b0 con 0x7f26580fef00 2024-04-22T15:15:23.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.516+0000 7f26557fa700 1 -- 172.21.15.96:0/1940187231 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f264800fa70 con 0x7f26580fef00 2024-04-22T15:15:23.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.516+0000 7f26557fa700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f264006cce0 0x7f264006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:23.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.516+0000 7f26557fa700 1 -- 172.21.15.96:0/1940187231 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f264808d2a0 con 0x7f26580fef00 2024-04-22T15:15:23.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.516+0000 7f26577fe700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f264006cce0 0x7f264006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:23.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.516+0000 7f26577fe700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f264006cce0 0x7f264006f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f264c007900 tx=0x7f264c008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:23.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.520+0000 7f26557fa700 1 -- 172.21.15.96:0/1940187231 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f264805b440 con 0x7f26580fef00 2024-04-22T15:15:23.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.725+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f265802cde0 con 0x7f264006cce0 2024-04-22T15:15:23.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.730+0000 7f26557fa700 1 -- 172.21.15.96:0/1940187231 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1300 (secure 0 0 0) 0x7f265802cde0 con 0x7f264006cce0 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter 2/2 2m ago 6m * 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:crash 2/2 2m ago 6m * 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:mgr 2/2 2m ago 6m count:2 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:mon 2/2 2m ago 5m smithi096:172.21.15.96=smithi096;smithi196:172.21.15.196=smithi196;count:2 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:osd 6 2m ago - 2024-04-22T15:15:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-22T15:15:23.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.732+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f264006cce0 msgr2=0x7f264006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:23.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.732+0000 7f265e713700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f264006cce0 0x7f264006f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f264c007900 tx=0x7f264c008040 comp rx=0 tx=0).stop 2024-04-22T15:15:23.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.732+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 msgr2=0x7f26581989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:23.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.732+0000 7f265e713700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 0x7f26581989b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f26480059d0 tx=0x7f264800be20 comp rx=0 tx=0).stop 2024-04-22T15:15:23.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.733+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 shutdown_connections 2024-04-22T15:15:23.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.733+0000 7f265e713700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f264006cce0 0x7f264006f1a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:23.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.733+0000 7f265e713700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26580fef00 0x7f26581989b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:23.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.733+0000 7f265e713700 1 --2- 172.21.15.96:0/1940187231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2658101ee0 0x7f2658198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:23.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.733+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 >> 172.21.15.96:0/1940187231 conn(0x7f26580fa860 msgr2=0x7f26581000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:23.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.733+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 shutdown_connections 2024-04-22T15:15:23.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:23.733+0000 7f265e713700 1 -- 172.21.15.96:0/1940187231 wait complete. 2024-04-22T15:15:24.341 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T15:15:24.544 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:24.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:24 smithi096 ceph-mon[26688]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:24.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:24 smithi096 ceph-mon[26688]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:24 smithi196 ceph-mon[34717]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:25.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:24 smithi196 ceph-mon[34717]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:26.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.277+0000 7f5dece8c700 1 -- 172.21.15.96:0/2011082736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 msgr2=0x7f5de8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.277+0000 7f5dece8c700 1 --2- 172.21.15.96:0/2011082736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 0x7f5de8102310 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009910 tx=0x7f5dd0009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.277+0000 7f5dece8c700 1 -- 172.21.15.96:0/2011082736 shutdown_connections 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.277+0000 7f5dece8c700 1 --2- 172.21.15.96:0/2011082736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 0x7f5de8102310 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.277+0000 7f5dece8c700 1 --2- 172.21.15.96:0/2011082736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5de80feed0 0x7f5de80ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.277+0000 7f5dece8c700 1 -- 172.21.15.96:0/2011082736 >> 172.21.15.96:0/2011082736 conn(0x7f5de80fa830 msgr2=0x7f5de80fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.277+0000 7f5dece8c700 1 -- 172.21.15.96:0/2011082736 shutdown_connections 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.277+0000 7f5dece8c700 1 -- 172.21.15.96:0/2011082736 wait complete. 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5dece8c700 1 Processor -- start 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5dece8c700 1 -- start start 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5dece8c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5de80feed0 0x7f5de8196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5dece8c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 0x7f5de8196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5dece8c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de8197260 con 0x7f5de8101eb0 2024-04-22T15:15:26.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5dece8c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de81973a0 con 0x7f5de80feed0 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5de5d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 0x7f5de8196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5de5d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 0x7f5de8196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55024/0 (socket says 172.21.15.96:55024) 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5de5d9b700 1 -- 172.21.15.96:0/3007534450 learned_addr learned my addr 172.21.15.96:0/3007534450 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5de659c700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5de80feed0 0x7f5de8196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5de5d9b700 1 -- 172.21.15.96:0/3007534450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5de80feed0 msgr2=0x7f5de8196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5de5d9b700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5de80feed0 0x7f5de8196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5de5d9b700 1 -- 172.21.15.96:0/3007534450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd00095f0 con 0x7f5de8101eb0 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.278+0000 7f5de659c700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5de80feed0 0x7f5de8196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.279+0000 7f5de5d9b700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 0x7f5de8196c40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0006010 tx=0x7f5dd000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.279+0000 7f5ddf7fe700 1 -- 172.21.15.96:0/3007534450 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd001d070 con 0x7f5de8101eb0 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.279+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de819bda0 con 0x7f5de8101eb0 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.279+0000 7f5ddf7fe700 1 -- 172.21.15.96:0/3007534450 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5dd000f460 con 0x7f5de8101eb0 2024-04-22T15:15:26.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.279+0000 7f5ddf7fe700 1 -- 172.21.15.96:0/3007534450 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd0021640 con 0x7f5de8101eb0 2024-04-22T15:15:26.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.279+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de819c310 con 0x7f5de8101eb0 2024-04-22T15:15:26.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.280+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de8190830 con 0x7f5de8101eb0 2024-04-22T15:15:26.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.280+0000 7f5ddf7fe700 1 -- 172.21.15.96:0/3007534450 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5dd000fab0 con 0x7f5de8101eb0 2024-04-22T15:15:26.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.281+0000 7f5ddf7fe700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5dd406cce0 0x7f5dd406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:26.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.281+0000 7f5ddf7fe700 1 -- 172.21.15.96:0/3007534450 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f5dd008d2d0 con 0x7f5de8101eb0 2024-04-22T15:15:26.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.281+0000 7f5de659c700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5dd406cce0 0x7f5dd406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:26.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.281+0000 7f5de659c700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5dd406cce0 0x7f5dd406f1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5de80683c0 tx=0x7f5dd8009570 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:26.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.285+0000 7f5ddf7fe700 1 -- 172.21.15.96:0/3007534450 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f5dd005b470 con 0x7f5de8101eb0 2024-04-22T15:15:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.482+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5de802ce50 con 0x7f5dd406cce0 2024-04-22T15:15:26.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.484+0000 7f5ddf7fe700 1 -- 172.21.15.96:0/3007534450 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f5de802ce50 con 0x7f5dd406cce0 2024-04-22T15:15:26.487 INFO:teuthology.orchestra.run.smithi096.stdout:HOST ADDR LABELS STATUS 2024-04-22T15:15:26.488 INFO:teuthology.orchestra.run.smithi096.stdout:smithi096 172.21.15.96 2024-04-22T15:15:26.488 INFO:teuthology.orchestra.run.smithi096.stdout:smithi196 172.21.15.196 2024-04-22T15:15:26.488 INFO:teuthology.orchestra.run.smithi096.stdout:2 hosts in cluster 2024-04-22T15:15:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5dd406cce0 msgr2=0x7f5dd406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5dd406cce0 0x7f5dd406f1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5de80683c0 tx=0x7f5dd8009570 comp rx=0 tx=0).stop 2024-04-22T15:15:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 msgr2=0x7f5de8196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 0x7f5de8196c40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0006010 tx=0x7f5dd000be20 comp rx=0 tx=0).stop 2024-04-22T15:15:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 shutdown_connections 2024-04-22T15:15:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5dd406cce0 0x7f5dd406f1a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5de80feed0 0x7f5de8196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 --2- 172.21.15.96:0/3007534450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8101eb0 0x7f5de8196c40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 >> 172.21.15.96:0/3007534450 conn(0x7f5de80fa830 msgr2=0x7f5de81054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 shutdown_connections 2024-04-22T15:15:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:26.487+0000 7f5dece8c700 1 -- 172.21.15.96:0/3007534450 wait complete. 2024-04-22T15:15:26.814 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:26 smithi096 ceph-mon[26688]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:26 smithi196 ceph-mon[34717]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:27.212 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T15:15:27.429 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:27.797 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:27 smithi096 ceph-mon[26688]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:27 smithi196 ceph-mon[34717]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:28.848 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:28 smithi096 ceph-mon[26688]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:28 smithi196 ceph-mon[34717]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 -- 172.21.15.96:0/2302616398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 msgr2=0x7f96c00fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 --2- 172.21.15.96:0/2302616398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 0x7f96c00fff40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009910 tx=0x7f96b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 -- 172.21.15.96:0/2302616398 shutdown_connections 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 --2- 172.21.15.96:0/2302616398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c0100d20 0x7f96c0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 --2- 172.21.15.96:0/2302616398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 0x7f96c00fff40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 -- 172.21.15.96:0/2302616398 >> 172.21.15.96:0/2302616398 conn(0x7f96c00fb0c0 msgr2=0x7f96c00fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 -- 172.21.15.96:0/2302616398 shutdown_connections 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 -- 172.21.15.96:0/2302616398 wait complete. 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 Processor -- start 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 -- start start 2024-04-22T15:15:29.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 0x7f96c01967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c0100d20 0x7f96c0196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c0197300 con 0x7f96c00ffb20 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.206+0000 7f96c6fd4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c0197440 con 0x7f96c0100d20 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96c4d70700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 0x7f96c01967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96c4d70700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 0x7f96c01967a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55050/0 (socket says 172.21.15.96:55050) 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96c4d70700 1 -- 172.21.15.96:0/4226842739 learned_addr learned my addr 172.21.15.96:0/4226842739 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96bffff700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c0100d20 0x7f96c0196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96c4d70700 1 -- 172.21.15.96:0/4226842739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c0100d20 msgr2=0x7f96c0196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96c4d70700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c0100d20 0x7f96c0196ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96c4d70700 1 -- 172.21.15.96:0/4226842739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b40095f0 con 0x7f96c00ffb20 2024-04-22T15:15:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96c4d70700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 0x7f96c01967a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f96b40059d0 tx=0x7f96b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:29.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96bdffb700 1 -- 172.21.15.96:0/4226842739 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b401d070 con 0x7f96c00ffb20 2024-04-22T15:15:29.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c019bdf0 con 0x7f96c00ffb20 2024-04-22T15:15:29.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96bdffb700 1 -- 172.21.15.96:0/4226842739 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96b400f460 con 0x7f96c00ffb20 2024-04-22T15:15:29.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96bdffb700 1 -- 172.21.15.96:0/4226842739 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b4021620 con 0x7f96c00ffb20 2024-04-22T15:15:29.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.207+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c019c2d0 con 0x7f96c00ffb20 2024-04-22T15:15:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.208+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c004ebb0 con 0x7f96c00ffb20 2024-04-22T15:15:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.209+0000 7f96bdffb700 1 -- 172.21.15.96:0/4226842739 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f96b400fa70 con 0x7f96c00ffb20 2024-04-22T15:15:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.209+0000 7f96bdffb700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f96b006cce0 0x7f96b006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.209+0000 7f96bdffb700 1 -- 172.21.15.96:0/4226842739 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f96b408d2a0 con 0x7f96c00ffb20 2024-04-22T15:15:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.209+0000 7f96bffff700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f96b006cce0 0x7f96b006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.210+0000 7f96bffff700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f96b006cce0 0x7f96b006f1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f96ac006fd0 tx=0x7f96ac008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.214+0000 7f96bdffb700 1 -- 172.21.15.96:0/4226842739 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f96b405b440 con 0x7f96c00ffb20 2024-04-22T15:15:29.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.414+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f96c0069890 con 0x7f96b006cce0 2024-04-22T15:15:29.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.416+0000 7f96bdffb700 1 -- 172.21.15.96:0/4226842739 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f96c0069890 con 0x7f96b006cce0 2024-04-22T15:15:29.419 INFO:teuthology.orchestra.run.smithi096.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T15:15:29.419 INFO:teuthology.orchestra.run.smithi096.stdout:smithi096 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NH400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-22T15:15:29.419 INFO:teuthology.orchestra.run.smithi096.stdout:smithi196 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6246000Z400BGN 372G No 72s ago Has a FileSystem, LVM detected 2024-04-22T15:15:29.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f96b006cce0 msgr2=0x7f96b006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:29.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f96b006cce0 0x7f96b006f1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f96ac006fd0 tx=0x7f96ac008040 comp rx=0 tx=0).stop 2024-04-22T15:15:29.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 msgr2=0x7f96c01967a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:29.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 0x7f96c01967a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f96b40059d0 tx=0x7f96b400be20 comp rx=0 tx=0).stop 2024-04-22T15:15:29.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 shutdown_connections 2024-04-22T15:15:29.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f96b006cce0 0x7f96b006f1a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:29.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c00ffb20 0x7f96c01967a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:29.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 --2- 172.21.15.96:0/4226842739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c0100d20 0x7f96c0196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:29.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 >> 172.21.15.96:0/4226842739 conn(0x7f96c00fb0c0 msgr2=0x7f96c0069180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:29.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 shutdown_connections 2024-04-22T15:15:29.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:29.418+0000 7f96c6fd4700 1 -- 172.21.15.96:0/4226842739 wait complete. 2024-04-22T15:15:30.026 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:15:30.037 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:15:30.037 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-22T15:15:30.240 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:30 smithi096 ceph-mon[26688]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:30 smithi096 ceph-mon[26688]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:30 smithi196 ceph-mon[34717]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:30 smithi196 ceph-mon[34717]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.886+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1611612606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14100a70 msgr2=0x7f9f14100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 --2- 172.21.15.96:0/1611612606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14100a70 0x7f9f14100e90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009910 tx=0x7f9f10009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1611612606 shutdown_connections 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 --2- 172.21.15.96:0/1611612606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f14101c70 0x7f9f141020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 --2- 172.21.15.96:0/1611612606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14100a70 0x7f9f14100e90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1611612606 >> 172.21.15.96:0/1611612606 conn(0x7f9f140fc010 msgr2=0x7f9f140fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1611612606 shutdown_connections 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1611612606 wait complete. 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 Processor -- start 2024-04-22T15:15:31.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 -- start start 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f14100a70 0x7f9f14074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14101c70 0x7f9f140732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f14073810 con 0x7f9f14101c70 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.887+0000 7f9f1ada2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f14073950 con 0x7f9f14100a70 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f1959f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14101c70 0x7f9f140732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f1959f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14101c70 0x7f9f140732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41108/0 (socket says 172.21.15.96:41108) 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f1959f700 1 -- 172.21.15.96:0/1786132222 learned_addr learned my addr 172.21.15.96:0/1786132222 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f19da0700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f14100a70 0x7f9f14074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f1959f700 1 -- 172.21.15.96:0/1786132222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f14100a70 msgr2=0x7f9f14074c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f1959f700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f14100a70 0x7f9f14074c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f1959f700 1 -- 172.21.15.96:0/1786132222 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f100095f0 con 0x7f9f14101c70 2024-04-22T15:15:31.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f1959f700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14101c70 0x7f9f140732d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400b4b0 tx=0x7f9f0400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:31.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f0affd700 1 -- 172.21.15.96:0/1786132222 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f04011800 con 0x7f9f14101c70 2024-04-22T15:15:31.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f14073c30 con 0x7f9f14101c70 2024-04-22T15:15:31.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1419e5d0 con 0x7f9f14101c70 2024-04-22T15:15:31.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f0affd700 1 -- 172.21.15.96:0/1786132222 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f04011e20 con 0x7f9f14101c70 2024-04-22T15:15:31.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.888+0000 7f9f0affd700 1 -- 172.21.15.96:0/1786132222 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f0400f2e0 con 0x7f9f14101c70 2024-04-22T15:15:31.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.889+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef8005320 con 0x7f9f14101c70 2024-04-22T15:15:31.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.890+0000 7f9f0affd700 1 -- 172.21.15.96:0/1786132222 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9f04011960 con 0x7f9f14101c70 2024-04-22T15:15:31.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.890+0000 7f9f0affd700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9f0006cce0 0x7f9f0006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:31.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.890+0000 7f9f0affd700 1 -- 172.21.15.96:0/1786132222 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9f0408b860 con 0x7f9f14101c70 2024-04-22T15:15:31.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.890+0000 7f9f19da0700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9f0006cce0 0x7f9f0006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:31.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.890+0000 7f9f19da0700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9f0006cce0 0x7f9f0006f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9f10000c00 tx=0x7f9f10005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:31.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:31.895+0000 7f9f0affd700 1 -- 172.21.15.96:0/1786132222 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f9f04059a80 con 0x7f9f14101c70 2024-04-22T15:15:32.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:32.115+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ef8000bf0 con 0x7f9f0006cce0 2024-04-22T15:15:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:32 smithi096 ceph-mon[26688]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:32 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-22T15:15:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:32 smithi196 ceph-mon[34717]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:33.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:32 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-22T15:15:33.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.780+0000 7f9f0affd700 1 -- 172.21.15.96:0/1786132222 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9ef8000bf0 con 0x7f9f0006cce0 2024-04-22T15:15:33.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.782+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9f0006cce0 msgr2=0x7f9f0006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:33.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.782+0000 7f9f1ada2700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9f0006cce0 0x7f9f0006f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9f10000c00 tx=0x7f9f10005960 comp rx=0 tx=0).stop 2024-04-22T15:15:33.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.782+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14101c70 msgr2=0x7f9f140732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:33.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.782+0000 7f9f1ada2700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14101c70 0x7f9f140732d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400b4b0 tx=0x7f9f0400b7c0 comp rx=0 tx=0).stop 2024-04-22T15:15:33.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.783+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 shutdown_connections 2024-04-22T15:15:33.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.783+0000 7f9f1ada2700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f9f0006cce0 0x7f9f0006f1a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:33.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.783+0000 7f9f1ada2700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f14100a70 0x7f9f14074c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:33.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.783+0000 7f9f1ada2700 1 --2- 172.21.15.96:0/1786132222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f14101c70 0x7f9f140732d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:33.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.783+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 >> 172.21.15.96:0/1786132222 conn(0x7f9f140fc010 msgr2=0x7f9f141052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:33.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.783+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 shutdown_connections 2024-04-22T15:15:33.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:33.783+0000 7f9f1ada2700 1 -- 172.21.15.96:0/1786132222 wait complete. 2024-04-22T15:15:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:33 smithi196 ceph-mon[34717]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:33 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-22T15:15:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:33 smithi196 ceph-mon[34717]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T15:15:34.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:33 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-22T15:15:34.134 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:33 smithi096 ceph-mon[26688]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:15:34.134 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:33 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-22T15:15:34.134 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:33 smithi096 ceph-mon[26688]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T15:15:34.134 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:33 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-22T15:15:34.134 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[26684]: 2024-04-22T15:15:33.755+0000 7f9ffa08c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:15:34.438 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T15:15:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: pgmap v176: 33 pgs: 12 creating+peering, 20 unknown, 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T15:15:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-22T15:15:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: fsmap cephfs:0 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: Saving service mds.cephfs spec with placement count:4 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.dymqlv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:15:34.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.dymqlv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:15:34.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: pgmap v176: 33 pgs: 12 creating+peering, 20 unknown, 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: fsmap cephfs:0 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: Saving service mds.cephfs spec with placement count:4 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:15:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:35.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:15:35.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:35.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.dymqlv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:15:35.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.dymqlv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:15:35.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:35.713 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:35.935 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:35 smithi096 ceph-mon[26688]: Deploying daemon mds.cephfs.smithi096.dymqlv on smithi096 2024-04-22T15:15:35.935 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:35 smithi096 ceph-mon[26688]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T15:15:36.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:35 smithi196 ceph-mon[34717]: Deploying daemon mds.cephfs.smithi096.dymqlv on smithi096 2024-04-22T15:15:36.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:35 smithi196 ceph-mon[34717]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T15:15:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:36 smithi096 ceph-mon[26688]: pgmap v180: 65 pgs: 15 creating+peering, 46 unknown, 4 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:36.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:36 smithi096 ceph-mon[26688]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T15:15:36.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:36 smithi096 ceph-mon[26688]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T15:15:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:36 smithi196 ceph-mon[34717]: pgmap v180: 65 pgs: 15 creating+peering, 46 unknown, 4 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:36 smithi196 ceph-mon[34717]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T15:15:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:36 smithi196 ceph-mon[34717]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T15:15:37.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/3864135859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 msgr2=0x7fb6e0101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:37.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 --2- 172.21.15.96:0/3864135859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 0x7fb6e0101170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009910 tx=0x7fb6c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:37.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/3864135859 shutdown_connections 2024-04-22T15:15:37.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 --2- 172.21.15.96:0/3864135859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 0x7fb6e0101170 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:37.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 --2- 172.21.15.96:0/3864135859 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6e00ffb10 0x7fb6e00fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:37.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/3864135859 >> 172.21.15.96:0/3864135859 conn(0x7fb6e00fb0b0 msgr2=0x7fb6e00fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:37.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/3864135859 shutdown_connections 2024-04-22T15:15:37.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/3864135859 wait complete. 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 Processor -- start 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.680+0000 7fb6e6a2c700 1 -- start start 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6e6a2c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6e00ffb10 0x7fb6e0069960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6e6a2c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 0x7fb6e019b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6e6a2c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e019b970 con 0x7fb6e0100d10 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6e6a2c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e019bae0 con 0x7fb6e00ffb10 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6d7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 0x7fb6e019b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6d7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 0x7fb6e019b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41130/0 (socket says 172.21.15.96:41130) 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6d7fff700 1 -- 172.21.15.96:0/2980977943 learned_addr learned my addr 172.21.15.96:0/2980977943 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6d7fff700 1 -- 172.21.15.96:0/2980977943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6e00ffb10 msgr2=0x7fb6e0069960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:37.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6dffff700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6e00ffb10 0x7fb6e0069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6d7fff700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6e00ffb10 0x7fb6e0069960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6d7fff700 1 -- 172.21.15.96:0/2980977943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c80095f0 con 0x7fb6e0100d10 2024-04-22T15:15:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6d7fff700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 0x7fb6e019b430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8006010 tx=0x7fb6c800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6ddffb700 1 -- 172.21.15.96:0/2980977943 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6c801d070 con 0x7fb6e0100d10 2024-04-22T15:15:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6ddffb700 1 -- 172.21.15.96:0/2980977943 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6c800f460 con 0x7fb6e0100d10 2024-04-22T15:15:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e019bd60 con 0x7fb6e0100d10 2024-04-22T15:15:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6ddffb700 1 -- 172.21.15.96:0/2980977943 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6c8021640 con 0x7fb6e0100d10 2024-04-22T15:15:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.681+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e019c2d0 con 0x7fb6e0100d10 2024-04-22T15:15:37.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.682+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e018f730 con 0x7fb6e0100d10 2024-04-22T15:15:37.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.683+0000 7fb6ddffb700 1 -- 172.21.15.96:0/2980977943 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb6c800fab0 con 0x7fb6e0100d10 2024-04-22T15:15:37.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.683+0000 7fb6ddffb700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb6cc06cce0 0x7fb6cc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:37.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.684+0000 7fb6dffff700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb6cc06cce0 0x7fb6cc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:37.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.684+0000 7fb6ddffb700 1 -- 172.21.15.96:0/2980977943 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fb6c808d520 con 0x7fb6e0100d10 2024-04-22T15:15:37.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.684+0000 7fb6dffff700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb6cc06cce0 0x7fb6cc06f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0006fd0 tx=0x7fb6d0008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:37.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.688+0000 7fb6ddffb700 1 -- 172.21.15.96:0/2980977943 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fb6c805b400 con 0x7fb6e0100d10 2024-04-22T15:15:37.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.932+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb6e004ebb0 con 0x7fb6e0100d10 2024-04-22T15:15:37.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.932+0000 7fb6ddffb700 1 -- 172.21.15.96:0/2980977943 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1114 (secure 0 0 0) 0x7fb6c805af90 con 0x7fb6e0100d10 2024-04-22T15:15:37.936 INFO:teuthology.orchestra.run.smithi096.stdout:e2 2024-04-22T15:15:37.936 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:15:37.936 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 2 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:33.756861+0000 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:15:37.937 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 0 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 1 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:in 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:15:37.938 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:15:37.939 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data disabled 2024-04-22T15:15:37.939 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:15:37.939 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:15:37.939 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 0 2024-04-22T15:15:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb6cc06cce0 msgr2=0x7fb6cc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb6cc06cce0 0x7fb6cc06f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0006fd0 tx=0x7fb6d0008040 comp rx=0 tx=0).stop 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 msgr2=0x7fb6e019b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 0x7fb6e019b430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8006010 tx=0x7fb6c800be20 comp rx=0 tx=0).stop 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 shutdown_connections 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fb6cc06cce0 0x7fb6cc06f1a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6e00ffb10 0x7fb6e0069960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 --2- 172.21.15.96:0/2980977943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6e0100d10 0x7fb6e019b430 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 >> 172.21.15.96:0/2980977943 conn(0x7fb6e00fb0b0 msgr2=0x7fb6e0104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:37.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 shutdown_connections 2024-04-22T15:15:37.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:37.935+0000 7fb6e6a2c700 1 -- 172.21.15.96:0/2980977943 wait complete. 2024-04-22T15:15:37.941 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 2 2024-04-22T15:15:38.630 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:15:38.641 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:15:38.641 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-22T15:15:38.843 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: pgmap v182: 65 pgs: 15 creating+peering, 12 unknown, 38 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:38.843 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:38.843 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:15:38.843 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2980977943' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:15:38.843 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:38.844 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:38.844 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:38.844 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.hwpcvg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:15:38.844 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.hwpcvg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:15:38.844 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:38 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:38.856 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:38.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: pgmap v182: 65 pgs: 15 creating+peering, 12 unknown, 38 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:15:38.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:38.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:15:38.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2980977943' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:15:38.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:38.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:38.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:38.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.hwpcvg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:15:38.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.hwpcvg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:15:38.980 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:38 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:39.746 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: Deploying daemon mds.cephfs.smithi196.hwpcvg on smithi196 2024-04-22T15:15:39.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.96:6826/887933858,v1:172.21.15.96:6827/887933858] up:boot 2024-04-22T15:15:39.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: daemon mds.cephfs.smithi096.dymqlv assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-22T15:15:39.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:15:39.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T15:15:39.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: Cluster is now healthy 2024-04-22T15:15:39.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: fsmap cephfs:0 1 up:standby 2024-04-22T15:15:39.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:15:39.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: fsmap cephfs:1 {0=cephfs.smithi096.dymqlv=up:creating} 2024-04-22T15:15:39.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:39 smithi196 ceph-mon[34717]: daemon mds.cephfs.smithi096.dymqlv is now active in filesystem cephfs as rank 0 2024-04-22T15:15:39.823 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: Deploying daemon mds.cephfs.smithi196.hwpcvg on smithi196 2024-04-22T15:15:39.823 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.96:6826/887933858,v1:172.21.15.96:6827/887933858] up:boot 2024-04-22T15:15:39.823 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: daemon mds.cephfs.smithi096.dymqlv assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-22T15:15:39.824 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:15:39.824 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T15:15:39.824 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: Cluster is now healthy 2024-04-22T15:15:39.824 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: fsmap cephfs:0 1 up:standby 2024-04-22T15:15:39.824 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:15:39.824 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: fsmap cephfs:1 {0=cephfs.smithi096.dymqlv=up:creating} 2024-04-22T15:15:39.824 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:39 smithi096 ceph-mon[26688]: daemon mds.cephfs.smithi096.dymqlv is now active in filesystem cephfs as rank 0 2024-04-22T15:15:40.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.336+0000 7fd8961cb700 1 -- 172.21.15.96:0/402422509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd890100d20 msgr2=0x7fd890101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:40.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.336+0000 7fd8961cb700 1 --2- 172.21.15.96:0/402422509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd890100d20 0x7fd890101180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd884009910 tx=0x7fd884009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:40.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.336+0000 7fd8961cb700 1 -- 172.21.15.96:0/402422509 shutdown_connections 2024-04-22T15:15:40.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.336+0000 7fd8961cb700 1 --2- 172.21.15.96:0/402422509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd890100d20 0x7fd890101180 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:40.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.336+0000 7fd8961cb700 1 --2- 172.21.15.96:0/402422509 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8900ffb20 0x7fd8900fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:40.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.336+0000 7fd8961cb700 1 -- 172.21.15.96:0/402422509 >> 172.21.15.96:0/402422509 conn(0x7fd8900fb0c0 msgr2=0x7fd8900fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:40.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.336+0000 7fd8961cb700 1 -- 172.21.15.96:0/402422509 shutdown_connections 2024-04-22T15:15:40.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.336+0000 7fd8961cb700 1 -- 172.21.15.96:0/402422509 wait complete. 2024-04-22T15:15:40.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd8961cb700 1 Processor -- start 2024-04-22T15:15:40.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd8961cb700 1 -- start start 2024-04-22T15:15:40.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd8961cb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8900ffb20 0x7fd890069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:40.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd8961cb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd890100d20 0x7fd89019b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:40.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd8961cb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89019ba00 con 0x7fd8900ffb20 2024-04-22T15:15:40.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd8961cb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89019bb70 con 0x7fd890100d20 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd88f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8900ffb20 0x7fd890069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd88f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8900ffb20 0x7fd890069c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41168/0 (socket says 172.21.15.96:41168) 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd88f7fe700 1 -- 172.21.15.96:0/606108811 learned_addr learned my addr 172.21.15.96:0/606108811 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd88effd700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd890100d20 0x7fd89019b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd88f7fe700 1 -- 172.21.15.96:0/606108811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd890100d20 msgr2=0x7fd89019b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd88f7fe700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd890100d20 0x7fd89019b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.337+0000 7fd88f7fe700 1 -- 172.21.15.96:0/606108811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8840095f0 con 0x7fd8900ffb20 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.338+0000 7fd88f7fe700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8900ffb20 0x7fd890069c20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd88000b9c0 tx=0x7fd88000bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.338+0000 7fd88cff9700 1 -- 172.21.15.96:0/606108811 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd88000d8b0 con 0x7fd8900ffb20 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.338+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89019be50 con 0x7fd8900ffb20 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.338+0000 7fd88cff9700 1 -- 172.21.15.96:0/606108811 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd880015420 con 0x7fd8900ffb20 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.338+0000 7fd88cff9700 1 -- 172.21.15.96:0/606108811 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd88000c610 con 0x7fd8900ffb20 2024-04-22T15:15:40.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.338+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89019c390 con 0x7fd8900ffb20 2024-04-22T15:15:40.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.339+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89018f740 con 0x7fd8900ffb20 2024-04-22T15:15:40.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.339+0000 7fd88cff9700 1 -- 172.21.15.96:0/606108811 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd880015590 con 0x7fd8900ffb20 2024-04-22T15:15:40.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.340+0000 7fd88cff9700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd87806cce0 0x7fd87806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:40.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.340+0000 7fd88effd700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd87806cce0 0x7fd87806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:40.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.340+0000 7fd88cff9700 1 -- 172.21.15.96:0/606108811 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fd880011070 con 0x7fd8900ffb20 2024-04-22T15:15:40.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.340+0000 7fd88effd700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd87806cce0 0x7fd87806f1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd884005090 tx=0x7fd884005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:40.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.344+0000 7fd88cff9700 1 -- 172.21.15.96:0/606108811 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fd88004fc50 con 0x7fd8900ffb20 2024-04-22T15:15:40.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.589+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fd89002cde0 con 0x7fd8900ffb20 2024-04-22T15:15:40.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.682+0000 7fd88cff9700 1 -- 172.21.15.96:0/606108811 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd880059b60 con 0x7fd8900ffb20 2024-04-22T15:15:40.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd87806cce0 msgr2=0x7fd87806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd87806cce0 0x7fd87806f1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd884005090 tx=0x7fd884005960 comp rx=0 tx=0).stop 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8900ffb20 msgr2=0x7fd890069c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8900ffb20 0x7fd890069c20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd88000b9c0 tx=0x7fd88000bcd0 comp rx=0 tx=0).stop 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 shutdown_connections 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd87806cce0 0x7fd87806f1a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8900ffb20 0x7fd890069c20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 --2- 172.21.15.96:0/606108811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd890100d20 0x7fd89019b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 >> 172.21.15.96:0/606108811 conn(0x7fd8900fb0c0 msgr2=0x7fd890104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 shutdown_connections 2024-04-22T15:15:40.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:40.685+0000 7fd8961cb700 1 -- 172.21.15.96:0/606108811 wait complete. 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 173 B/s wr, 0 op/s 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.96:6826/887933858,v1:172.21.15.96:6827/887933858] up:active 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: fsmap cephfs:1 {0=cephfs.smithi096.dymqlv=up:active} 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.drkkdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.drkkdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:40 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/606108811' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-22T15:15:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 173 B/s wr, 0 op/s 2024-04-22T15:15:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.96:6826/887933858,v1:172.21.15.96:6827/887933858] up:active 2024-04-22T15:15:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: fsmap cephfs:1 {0=cephfs.smithi096.dymqlv=up:active} 2024-04-22T15:15:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:40.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:40.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.drkkdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:15:40.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.drkkdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:15:40.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:40.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:40 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/606108811' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-22T15:15:41.302 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:15:41.315 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:15:41.316 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-22T15:15:41.750 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: Deploying daemon mds.cephfs.smithi096.drkkdi on smithi096 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] up:boot 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/606108811' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: daemon mds.cephfs.smithi196.hwpcvg assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: fsmap cephfs:1 {0=cephfs.smithi096.dymqlv=up:active} 1 up:standby 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:creating} 2024-04-22T15:15:41.751 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:41 smithi196 ceph-mon[34717]: daemon mds.cephfs.smithi196.hwpcvg is now active in filesystem cephfs as rank 1 2024-04-22T15:15:41.858 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: Deploying daemon mds.cephfs.smithi096.drkkdi on smithi096 2024-04-22T15:15:41.858 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T15:15:41.858 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] up:boot 2024-04-22T15:15:41.858 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/606108811' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-22T15:15:41.858 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: daemon mds.cephfs.smithi196.hwpcvg assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T15:15:41.858 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-22T15:15:41.859 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T15:15:41.859 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: fsmap cephfs:1 {0=cephfs.smithi096.dymqlv=up:active} 1 up:standby 2024-04-22T15:15:41.859 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:15:41.859 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:creating} 2024-04-22T15:15:41.859 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:41 smithi096 ceph-mon[26688]: daemon mds.cephfs.smithi196.hwpcvg is now active in filesystem cephfs as rank 1 2024-04-22T15:15:42.574 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:42.832 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:42 smithi096 ceph-mon[26688]: pgmap v184: 65 pgs: 65 active+clean; 578 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-22T15:15:42.832 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:42 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] up:active 2024-04-22T15:15:42.832 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:42 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2024-04-22T15:15:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:42 smithi196 ceph-mon[34717]: pgmap v184: 65 pgs: 65 active+clean; 578 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-22T15:15:43.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:42 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] up:active 2024-04-22T15:15:43.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:42 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2024-04-22T15:15:44.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.381+0000 7f108fb08700 1 -- 172.21.15.96:0/2660485112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 msgr2=0x7f1088101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:44.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 --2- 172.21.15.96:0/2660485112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 0x7f1088101050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1084009910 tx=0x7f1084009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:44.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 -- 172.21.15.96:0/2660485112 shutdown_connections 2024-04-22T15:15:44.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 --2- 172.21.15.96:0/2660485112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1088101e30 0x7f1088102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:44.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 --2- 172.21.15.96:0/2660485112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 0x7f1088101050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:44.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 -- 172.21.15.96:0/2660485112 >> 172.21.15.96:0/2660485112 conn(0x7f10880fc1d0 msgr2=0x7f10880fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:44.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 -- 172.21.15.96:0/2660485112 shutdown_connections 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 -- 172.21.15.96:0/2660485112 wait complete. 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 Processor -- start 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 -- start start 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 0x7f1088074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1088101e30 0x7f10880732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1088073810 con 0x7f1088100c30 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.382+0000 7f108fb08700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1088073950 con 0x7f1088101e30 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108d8a4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 0x7f1088074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108d8a4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 0x7f1088074c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52186/0 (socket says 172.21.15.96:52186) 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108d8a4700 1 -- 172.21.15.96:0/2631586110 learned_addr learned my addr 172.21.15.96:0/2631586110 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108d0a3700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1088101e30 0x7f10880732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:44.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108d8a4700 1 -- 172.21.15.96:0/2631586110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1088101e30 msgr2=0x7f10880732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:44.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108d8a4700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1088101e30 0x7f10880732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:44.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108d8a4700 1 -- 172.21.15.96:0/2631586110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10840095f0 con 0x7f1088100c30 2024-04-22T15:15:44.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108d0a3700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1088101e30 0x7f10880732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:15:44.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108d8a4700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 0x7f1088074c30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f10840059d0 tx=0x7f108400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:44.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f107effd700 1 -- 172.21.15.96:0/2631586110 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f108401d070 con 0x7f1088100c30 2024-04-22T15:15:44.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f107effd700 1 -- 172.21.15.96:0/2631586110 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1084005dc0 con 0x7f1088100c30 2024-04-22T15:15:44.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1088073bd0 con 0x7f1088100c30 2024-04-22T15:15:44.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f107effd700 1 -- 172.21.15.96:0/2631586110 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1084021410 con 0x7f1088100c30 2024-04-22T15:15:44.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.383+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1088074140 con 0x7f1088100c30 2024-04-22T15:15:44.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.384+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10881907b0 con 0x7f1088100c30 2024-04-22T15:15:44.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.385+0000 7f107effd700 1 -- 172.21.15.96:0/2631586110 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f108400f840 con 0x7f1088100c30 2024-04-22T15:15:44.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.385+0000 7f107effd700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f107406cce0 0x7f107406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:44.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.385+0000 7f108d0a3700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f107406cce0 0x7f107406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:44.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.385+0000 7f107effd700 1 -- 172.21.15.96:0/2631586110 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f108408d2e0 con 0x7f1088100c30 2024-04-22T15:15:44.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.385+0000 7f108d0a3700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f107406cce0 0x7f107406f1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1078005950 tx=0x7f10780058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:44.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.389+0000 7f107effd700 1 -- 172.21.15.96:0/2631586110 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f108405b140 con 0x7f1088100c30 2024-04-22T15:15:44.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.637+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f1088074140 con 0x7f1088100c30 2024-04-22T15:15:44.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.698+0000 7f107effd700 1 -- 172.21.15.96:0/2631586110 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7f1088074140 con 0x7f1088100c30 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f107406cce0 msgr2=0x7f107406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f107406cce0 0x7f107406f1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1078005950 tx=0x7f10780058e0 comp rx=0 tx=0).stop 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 msgr2=0x7f1088074c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 0x7f1088074c30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f10840059d0 tx=0x7f108400be20 comp rx=0 tx=0).stop 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 shutdown_connections 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f107406cce0 0x7f107406f1a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100c30 0x7f1088074c30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 --2- 172.21.15.96:0/2631586110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1088101e30 0x7f10880732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 >> 172.21.15.96:0/2631586110 conn(0x7f10880fc1d0 msgr2=0x7f1088105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 shutdown_connections 2024-04-22T15:15:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:44.700+0000 7f108fb08700 1 -- 172.21.15.96:0/2631586110 wait complete. 2024-04-22T15:15:44.834 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:44 smithi096 ceph-mon[26688]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s wr, 9 op/s 2024-04-22T15:15:44.834 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:44 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2631586110' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T15:15:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:44 smithi196 ceph-mon[34717]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s wr, 9 op/s 2024-04-22T15:15:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:44 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2631586110' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T15:15:45.526 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:15:45.536 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:15:45.536 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-22T15:15:45.743 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:45.803 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2631586110' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T15:15:45.803 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2024-04-22T15:15:45.803 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: Cluster is now healthy 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:boot 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.pbvqlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.pbvqlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:15:45.804 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:45 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:45.847 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2631586110' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T15:15:45.847 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2024-04-22T15:15:45.847 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-22T15:15:45.847 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: Cluster is now healthy 2024-04-22T15:15:45.847 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:boot 2024-04-22T15:15:45.847 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 2024-04-22T15:15:45.847 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:15:45.848 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:45.848 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:45.848 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:45.848 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.pbvqlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:15:45.848 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.pbvqlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:15:45.848 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:45 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:46.918 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:46 smithi196 ceph-mon[34717]: Deploying daemon mds.cephfs.smithi196.pbvqlq on smithi196 2024-04-22T15:15:46.918 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:46 smithi196 ceph-mon[34717]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 8 op/s 2024-04-22T15:15:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:46 smithi096 ceph-mon[26688]: Deploying daemon mds.cephfs.smithi196.pbvqlq on smithi196 2024-04-22T15:15:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:46 smithi096 ceph-mon[26688]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 8 op/s 2024-04-22T15:15:47.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 -- 172.21.15.96:0/3290790477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 msgr2=0x7fa074106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:47.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 --2- 172.21.15.96:0/3290790477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 0x7fa074106400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa05c009910 tx=0x7fa05c009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:47.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 -- 172.21.15.96:0/3290790477 shutdown_connections 2024-04-22T15:15:47.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 --2- 172.21.15.96:0/3290790477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 0x7fa074106400 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 --2- 172.21.15.96:0/3290790477 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0741016e0 0x7fa074103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 -- 172.21.15.96:0/3290790477 >> 172.21.15.96:0/3290790477 conn(0x7fa0740fb0b0 msgr2=0x7fa0740fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 -- 172.21.15.96:0/3290790477 shutdown_connections 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 -- 172.21.15.96:0/3290790477 wait complete. 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 Processor -- start 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.307+0000 7fa0789b9700 1 -- start start 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa0789b9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0741016e0 0x7fa074100ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa0789b9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 0x7fa0740ff690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa0789b9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740ffbd0 con 0x7fa074104010 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa0789b9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740ffd10 con 0x7fa0741016e0 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 0x7fa0740ff690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 0x7fa0740ff690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52236/0 (socket says 172.21.15.96:52236) 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06bfff700 1 -- 172.21.15.96:0/2300871319 learned_addr learned my addr 172.21.15.96:0/2300871319 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa07259c700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0741016e0 0x7fa074100ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06bfff700 1 -- 172.21.15.96:0/2300871319 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0741016e0 msgr2=0x7fa074100ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06bfff700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0741016e0 0x7fa074100ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06bfff700 1 -- 172.21.15.96:0/2300871319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c0095f0 con 0x7fa074104010 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06bfff700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 0x7fa0740ff690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa05c006010 tx=0x7fa05c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06b7fe700 1 -- 172.21.15.96:0/2300871319 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa05c01d070 con 0x7fa074104010 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06b7fe700 1 -- 172.21.15.96:0/2300871319 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa05c00f460 con 0x7fa074104010 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0740fff90 con 0x7fa074104010 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa06b7fe700 1 -- 172.21.15.96:0/2300871319 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa05c021640 con 0x7fa074104010 2024-04-22T15:15:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.308+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa074100500 con 0x7fa074104010 2024-04-22T15:15:47.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.309+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07404ebb0 con 0x7fa074104010 2024-04-22T15:15:47.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.310+0000 7fa06b7fe700 1 -- 172.21.15.96:0/2300871319 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa05c00f5d0 con 0x7fa074104010 2024-04-22T15:15:47.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.310+0000 7fa06b7fe700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa06006cce0 0x7fa06006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:47.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.310+0000 7fa06b7fe700 1 -- 172.21.15.96:0/2300871319 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fa05c08d4b0 con 0x7fa074104010 2024-04-22T15:15:47.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.310+0000 7fa07259c700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa06006cce0 0x7fa06006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:47.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.311+0000 7fa07259c700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa06006cce0 0x7fa06006f1a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa064006fd0 tx=0x7fa064008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:47.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.315+0000 7fa06b7fe700 1 -- 172.21.15.96:0/2300871319 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa05c05b310 con 0x7fa074104010 2024-04-22T15:15:47.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.554+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa074109830 con 0x7fa074104010 2024-04-22T15:15:47.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.709+0000 7fa06b7fe700 1 -- 172.21.15.96:0/2300871319 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v11) v1 ==== 168+0+0 (secure 0 0 0) 0x7fa05c05aea0 con 0x7fa074104010 2024-04-22T15:15:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.711+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa06006cce0 msgr2=0x7fa06006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.711+0000 7fa0789b9700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa06006cce0 0x7fa06006f1a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa064006fd0 tx=0x7fa064008040 comp rx=0 tx=0).stop 2024-04-22T15:15:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.711+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 msgr2=0x7fa0740ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.711+0000 7fa0789b9700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 0x7fa0740ff690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa05c006010 tx=0x7fa05c00be20 comp rx=0 tx=0).stop 2024-04-22T15:15:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.712+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 shutdown_connections 2024-04-22T15:15:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.712+0000 7fa0789b9700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa06006cce0 0x7fa06006f1a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.712+0000 7fa0789b9700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0741016e0 0x7fa074100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:47.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.712+0000 7fa0789b9700 1 --2- 172.21.15.96:0/2300871319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa074104010 0x7fa0740ff690 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:47.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.712+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 >> 172.21.15.96:0/2300871319 conn(0x7fa0740fb0b0 msgr2=0x7fa0740fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:47.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.712+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 shutdown_connections 2024-04-22T15:15:47.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:47.712+0000 7fa0789b9700 1 -- 172.21.15.96:0/2300871319 wait complete. 2024-04-22T15:15:47.716 INFO:teuthology.orchestra.run.smithi096.stderr:inline data enabled 2024-04-22T15:15:47.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:47 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2300871319' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T15:15:47.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:47 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:47 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2300871319' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T15:15:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:47 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:48.481 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:15:48.491 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:15:48.492 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T15:15:48.726 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:48.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 86 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-22T15:15:48.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2300871319' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] up:boot 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:48.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:15:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 86 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-22T15:15:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2300871319' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] up:boot 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:15:49.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:49 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] up:active 2024-04-22T15:15:49.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:49 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:15:50.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:49 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] up:active 2024-04-22T15:15:50.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:49 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:15:50.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 -- 172.21.15.96:0/4006124546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc100d10 msgr2=0x7f58dc101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 --2- 172.21.15.96:0/4006124546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc100d10 0x7f58dc101170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009910 tx=0x7f58d0009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 -- 172.21.15.96:0/4006124546 shutdown_connections 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 --2- 172.21.15.96:0/4006124546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc100d10 0x7f58dc101170 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 --2- 172.21.15.96:0/4006124546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58dc0ffb10 0x7f58dc0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 -- 172.21.15.96:0/4006124546 >> 172.21.15.96:0/4006124546 conn(0x7f58dc0fb0b0 msgr2=0x7f58dc0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 -- 172.21.15.96:0/4006124546 shutdown_connections 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 -- 172.21.15.96:0/4006124546 wait complete. 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 Processor -- start 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 -- start start 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc0ffb10 0x7f58dc103a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58dc100d10 0x7f58dc1020f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:50.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc102630 con 0x7f58dc0ffb10 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.836+0000 7f58e2aea700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc102770 con 0x7f58dc100d10 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58e0886700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc0ffb10 0x7f58dc103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58dbfff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58dc100d10 0x7f58dc1020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58e0886700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc0ffb10 0x7f58dc103a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43984/0 (socket says 172.21.15.96:43984) 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58e0886700 1 -- 172.21.15.96:0/2816081044 learned_addr learned my addr 172.21.15.96:0/2816081044 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58e0886700 1 -- 172.21.15.96:0/2816081044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58dc100d10 msgr2=0x7f58dc1020f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58e0886700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58dc100d10 0x7f58dc1020f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58e0886700 1 -- 172.21.15.96:0/2816081044 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d00095f0 con 0x7f58dc0ffb10 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58e0886700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc0ffb10 0x7f58dc103a00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00d900 tx=0x7f58cc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58d9ffb700 1 -- 172.21.15.96:0/2816081044 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58cc0041f0 con 0x7f58dc0ffb10 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58dc102a50 con 0x7f58dc0ffb10 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58d9ffb700 1 -- 172.21.15.96:0/2816081044 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58cc004350 con 0x7f58dc0ffb10 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58d9ffb700 1 -- 172.21.15.96:0/2816081044 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58cc003cf0 con 0x7f58dc0ffb10 2024-04-22T15:15:50.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.837+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc19e690 con 0x7f58dc0ffb10 2024-04-22T15:15:50.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.838+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc04ebb0 con 0x7f58dc0ffb10 2024-04-22T15:15:50.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.839+0000 7f58d9ffb700 1 -- 172.21.15.96:0/2816081044 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f58cc009730 con 0x7f58dc0ffb10 2024-04-22T15:15:50.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.839+0000 7f58d9ffb700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f58c406ccf0 0x7f58c406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:50.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.839+0000 7f58d9ffb700 1 -- 172.21.15.96:0/2816081044 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f58cc08c120 con 0x7f58dc0ffb10 2024-04-22T15:15:50.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.839+0000 7f58dbfff700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f58c406ccf0 0x7f58c406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:50.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.840+0000 7f58dbfff700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f58c406ccf0 0x7f58c406f1b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f58d0006010 tx=0x7f58d0005180 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:50.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:50.844+0000 7f58d9ffb700 1 -- 172.21.15.96:0/2816081044 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f58cc059f20 con 0x7f58dc0ffb10 2024-04-22T15:15:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-22T15:15:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:standby-replay 2024-04-22T15:15:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: Dropping low affinity active daemon mds.cephfs.smithi096.dymqlv in favor of higher affinity standby. 2024-04-22T15:15:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: Replacing daemon mds.cephfs.smithi096.dymqlv as rank 0 with standby daemon mds.cephfs.smithi096.drkkdi 2024-04-22T15:15:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:15:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:15:50.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: osdmap e45: 6 total, 6 up, 6 in 2024-04-22T15:15:50.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: fsmap cephfs:2/2 {0=cephfs.smithi096.drkkdi=up:replay,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 2024-04-22T15:15:50.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:50.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:50 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:51.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.085+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f58dc19e960 con 0x7f58dc0ffb10 2024-04-22T15:15:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.085+0000 7f58d9ffb700 1 -- 172.21.15.96:0/2816081044 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1870 (secure 0 0 0) 0x7f58cc059ab0 con 0x7f58dc0ffb10 2024-04-22T15:15:51.089 INFO:teuthology.orchestra.run.smithi096.stdout:e16 2024-04-22T15:15:51.089 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:15:51.089 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:15:51.089 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:15:51.089 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:51.089 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:15:51.089 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:15:51.089 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 16 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:50.826763+0000 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:15:51.090 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:15:51.091 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:15:51.092 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:15:51.092 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:resolve seq 3 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:15:51.092 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:15:51.092 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:15:51.092 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:51.092 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:51.092 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:15:51.092 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:51.092 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:15:51.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.087+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f58c406ccf0 msgr2=0x7f58c406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:51.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.087+0000 7f58e2aea700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f58c406ccf0 0x7f58c406f1b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f58d0006010 tx=0x7f58d0005180 comp rx=0 tx=0).stop 2024-04-22T15:15:51.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.087+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc0ffb10 msgr2=0x7f58dc103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:51.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.087+0000 7f58e2aea700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc0ffb10 0x7f58dc103a00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00d900 tx=0x7f58cc00dc10 comp rx=0 tx=0).stop 2024-04-22T15:15:51.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.087+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 shutdown_connections 2024-04-22T15:15:51.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.087+0000 7f58e2aea700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f58c406ccf0 0x7f58c406f1b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:51.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.087+0000 7f58e2aea700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc0ffb10 0x7f58dc103a00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:51.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.087+0000 7f58e2aea700 1 --2- 172.21.15.96:0/2816081044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58dc100d10 0x7f58dc1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.087+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 >> 172.21.15.96:0/2816081044 conn(0x7f58dc0fb0b0 msgr2=0x7f58dc068f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.088+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 shutdown_connections 2024-04-22T15:15:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:51.088+0000 7f58e2aea700 1 -- 172.21.15.96:0/2816081044 wait complete. 2024-04-22T15:15:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 16 2024-04-22T15:15:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-22T15:15:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:standby-replay 2024-04-22T15:15:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: Dropping low affinity active daemon mds.cephfs.smithi096.dymqlv in favor of higher affinity standby. 2024-04-22T15:15:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: Replacing daemon mds.cephfs.smithi096.dymqlv as rank 0 with standby daemon mds.cephfs.smithi096.drkkdi 2024-04-22T15:15:51.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:15:51.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:15:51.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: osdmap e45: 6 total, 6 up, 6 in 2024-04-22T15:15:51.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: fsmap cephfs:2/2 {0=cephfs.smithi096.drkkdi=up:replay,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 2024-04-22T15:15:51.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:51.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:50 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:51.692 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-22T15:15:51.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:51 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] up:boot 2024-04-22T15:15:51.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:51 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:resolve 2024-04-22T15:15:51.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:51 smithi096 ceph-mon[26688]: fsmap cephfs:2/2 {0=cephfs.smithi096.drkkdi=up:resolve,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:51.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:51 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:15:51.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:51 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2816081044' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:15:52.142 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:52.182 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:51 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] up:boot 2024-04-22T15:15:52.182 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:51 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:resolve 2024-04-22T15:15:52.182 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:51 smithi196 ceph-mon[34717]: fsmap cephfs:2/2 {0=cephfs.smithi096.drkkdi=up:resolve,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:52.182 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:51 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:15:52.182 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:51 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2816081044' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:15:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:52 smithi196 ceph-mon[34717]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-22T15:15:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:52 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:reconnect 2024-04-22T15:15:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:52 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] up:standby-replay 2024-04-22T15:15:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:52 smithi196 ceph-mon[34717]: fsmap cephfs:2/2 {0=cephfs.smithi096.drkkdi=up:reconnect,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:15:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:53.111 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:52 smithi096 ceph-mon[26688]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-22T15:15:53.112 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:52 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:reconnect 2024-04-22T15:15:53.112 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:52 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] up:standby-replay 2024-04-22T15:15:53.112 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:52 smithi096 ceph-mon[26688]: fsmap cephfs:2/2 {0=cephfs.smithi096.drkkdi=up:reconnect,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:53.112 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:53.112 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:15:53.112 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:54.170 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:53 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:rejoin 2024-04-22T15:15:54.170 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:53 smithi096 ceph-mon[26688]: fsmap cephfs:2/2 {0=cephfs.smithi096.drkkdi=up:rejoin,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:54.171 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:53 smithi096 ceph-mon[26688]: daemon mds.cephfs.smithi096.drkkdi is now active in filesystem cephfs as rank 0 2024-04-22T15:15:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:53 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:rejoin 2024-04-22T15:15:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:53 smithi196 ceph-mon[34717]: fsmap cephfs:2/2 {0=cephfs.smithi096.drkkdi=up:rejoin,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:53 smithi196 ceph-mon[34717]: daemon mds.cephfs.smithi096.drkkdi is now active in filesystem cephfs as rank 0 2024-04-22T15:15:54.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.710+0000 7fbf108ec700 1 -- 172.21.15.96:0/155455948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 msgr2=0x7fbf0c1011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:54.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.710+0000 7fbf108ec700 1 --2- 172.21.15.96:0/155455948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 0x7fbf0c1011f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009910 tx=0x7fbefc009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:54.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.710+0000 7fbf108ec700 1 -- 172.21.15.96:0/155455948 shutdown_connections 2024-04-22T15:15:54.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.710+0000 7fbf108ec700 1 --2- 172.21.15.96:0/155455948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 0x7fbf0c1011f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:54.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.710+0000 7fbf108ec700 1 --2- 172.21.15.96:0/155455948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf0c0ffb90 0x7fbf0c0fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:54.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.710+0000 7fbf108ec700 1 -- 172.21.15.96:0/155455948 >> 172.21.15.96:0/155455948 conn(0x7fbf0c0fb110 msgr2=0x7fbf0c0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:54.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.710+0000 7fbf108ec700 1 -- 172.21.15.96:0/155455948 shutdown_connections 2024-04-22T15:15:54.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.710+0000 7fbf108ec700 1 -- 172.21.15.96:0/155455948 wait complete. 2024-04-22T15:15:54.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf108ec700 1 Processor -- start 2024-04-22T15:15:54.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf108ec700 1 -- start start 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf108ec700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf0c0ffb90 0x7fbf0c196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf108ec700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 0x7fbf0c196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf108ec700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c1972f0 con 0x7fbf0c100d90 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf108ec700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c197430 con 0x7fbf0c0ffb90 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf09d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 0x7fbf0c196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf09d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 0x7fbf0c196cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44004/0 (socket says 172.21.15.96:44004) 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf09d9b700 1 -- 172.21.15.96:0/4075167459 learned_addr learned my addr 172.21.15.96:0/4075167459 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf0a59c700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf0c0ffb90 0x7fbf0c196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf09d9b700 1 -- 172.21.15.96:0/4075167459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf0c0ffb90 msgr2=0x7fbf0c196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf09d9b700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf0c0ffb90 0x7fbf0c196790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf09d9b700 1 -- 172.21.15.96:0/4075167459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc0095f0 con 0x7fbf0c100d90 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf0a59c700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf0c0ffb90 0x7fbf0c196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.711+0000 7fbf09d9b700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 0x7fbf0c196cd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0092e0 tx=0x7fbefc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:54.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.712+0000 7fbefb7fe700 1 -- 172.21.15.96:0/4075167459 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbefc01d070 con 0x7fbf0c100d90 2024-04-22T15:15:54.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.712+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0c19bde0 con 0x7fbf0c100d90 2024-04-22T15:15:54.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.712+0000 7fbefb7fe700 1 -- 172.21.15.96:0/4075167459 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbefc00f460 con 0x7fbf0c100d90 2024-04-22T15:15:54.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.712+0000 7fbefb7fe700 1 -- 172.21.15.96:0/4075167459 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbefc021640 con 0x7fbf0c100d90 2024-04-22T15:15:54.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.712+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0c19c350 con 0x7fbf0c100d90 2024-04-22T15:15:54.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.713+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0c04ebb0 con 0x7fbf0c100d90 2024-04-22T15:15:54.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.713+0000 7fbefb7fe700 1 -- 172.21.15.96:0/4075167459 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbefc00f5d0 con 0x7fbf0c100d90 2024-04-22T15:15:54.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.714+0000 7fbefb7fe700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fbef406cce0 0x7fbef406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:54.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.714+0000 7fbefb7fe700 1 -- 172.21.15.96:0/4075167459 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fbefc08d630 con 0x7fbf0c100d90 2024-04-22T15:15:54.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.714+0000 7fbf0a59c700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fbef406cce0 0x7fbef406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:54.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.714+0000 7fbf0a59c700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fbef406cce0 0x7fbef406f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c100bf0 tx=0x7fbf00008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:54.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.718+0000 7fbefb7fe700 1 -- 172.21.15.96:0/4075167459 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fbefc05b430 con 0x7fbf0c100d90 2024-04-22T15:15:54.897 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:54 smithi096 ceph-mon[26688]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T15:15:54.897 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:54 smithi096 ceph-mon[26688]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:15:54.898 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:54 smithi096 ceph-mon[26688]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:active 2024-04-22T15:15:54.898 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:54 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:54.898 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:54 smithi096 ceph-mon[26688]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:15:54.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.963+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf0c19c750 con 0x7fbf0c100d90 2024-04-22T15:15:54.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.963+0000 7fbefb7fe700 1 -- 172.21.15.96:0/4075167459 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4859 (secure 0 0 0) 0x7fbefc05afc0 con 0x7fbf0c100d90 2024-04-22T15:15:54.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fbef406cce0 msgr2=0x7fbef406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:54.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fbef406cce0 0x7fbef406f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c100bf0 tx=0x7fbf00008040 comp rx=0 tx=0).stop 2024-04-22T15:15:54.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 msgr2=0x7fbf0c196cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:54.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 0x7fbf0c196cd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0092e0 tx=0x7fbefc00be20 comp rx=0 tx=0).stop 2024-04-22T15:15:54.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 shutdown_connections 2024-04-22T15:15:54.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fbef406cce0 0x7fbef406f1a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:54.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf0c0ffb90 0x7fbf0c196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:54.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 --2- 172.21.15.96:0/4075167459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf0c100d90 0x7fbf0c196cd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:54.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 >> 172.21.15.96:0/4075167459 conn(0x7fbf0c0fb110 msgr2=0x7fbf0c068670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:54.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 shutdown_connections 2024-04-22T15:15:54.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:54.965+0000 7fbf108ec700 1 -- 172.21.15.96:0/4075167459 wait complete. 2024-04-22T15:15:54.970 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:15:54.978 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:15:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:54 smithi196 ceph-mon[34717]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T15:15:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:54 smithi196 ceph-mon[34717]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:15:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:54 smithi196 ceph-mon[34717]: mds.? [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] up:active 2024-04-22T15:15:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:54 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:15:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:54 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:15:55.727 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-22T15:15:55.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:55 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/4075167459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T15:15:55.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:55 smithi096 ceph-mon[26688]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T15:15:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:55 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/4075167459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T15:15:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:55 smithi196 ceph-mon[34717]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T15:15:56.454 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:15:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.978+0000 7fcd4de2f700 1 -- 172.21.15.96:0/3117530054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 msgr2=0x7fcd481063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.978+0000 7fcd4de2f700 1 --2- 172.21.15.96:0/3117530054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 0x7fcd481063d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c009910 tx=0x7fcd3c009c20 comp rx=0 tx=0).stop 2024-04-22T15:15:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.978+0000 7fcd4de2f700 1 -- 172.21.15.96:0/3117530054 shutdown_connections 2024-04-22T15:15:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.978+0000 7fcd4de2f700 1 --2- 172.21.15.96:0/3117530054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 0x7fcd481063d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.978+0000 7fcd4de2f700 1 --2- 172.21.15.96:0/3117530054 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd481016b0 0x7fcd48103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.978+0000 7fcd4de2f700 1 -- 172.21.15.96:0/3117530054 >> 172.21.15.96:0/3117530054 conn(0x7fcd480fb0e0 msgr2=0x7fcd480fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.978+0000 7fcd4de2f700 1 -- 172.21.15.96:0/3117530054 shutdown_connections 2024-04-22T15:15:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.978+0000 7fcd4de2f700 1 -- 172.21.15.96:0/3117530054 wait complete. 2024-04-22T15:15:57.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.979+0000 7fcd4de2f700 1 Processor -- start 2024-04-22T15:15:57.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.979+0000 7fcd4de2f700 1 -- start start 2024-04-22T15:15:57.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.979+0000 7fcd4de2f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd481016b0 0x7fcd481965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:57.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.979+0000 7fcd4de2f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 0x7fcd48196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:57.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.979+0000 7fcd4de2f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481970b0 con 0x7fcd48103fe0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.979+0000 7fcd4de2f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481971f0 con 0x7fcd481016b0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.979+0000 7fcd46ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 0x7fcd48196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.979+0000 7fcd46ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 0x7fcd48196b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44010/0 (socket says 172.21.15.96:44010) 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.979+0000 7fcd46ffd700 1 -- 172.21.15.96:0/2945783646 learned_addr learned my addr 172.21.15.96:0/2945783646 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd477fe700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd481016b0 0x7fcd481965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd46ffd700 1 -- 172.21.15.96:0/2945783646 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd481016b0 msgr2=0x7fcd481965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd46ffd700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd481016b0 0x7fcd481965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd46ffd700 1 -- 172.21.15.96:0/2945783646 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c0095f0 con 0x7fcd48103fe0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd46ffd700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 0x7fcd48196b20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c0092e0 tx=0x7fcd3c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd44ff9700 1 -- 172.21.15.96:0/2945783646 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd3c01d070 con 0x7fcd48103fe0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd44ff9700 1 -- 172.21.15.96:0/2945783646 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd3c00f460 con 0x7fcd48103fe0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4819bc00 con 0x7fcd48103fe0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd44ff9700 1 -- 172.21.15.96:0/2945783646 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd3c021640 con 0x7fcd48103fe0 2024-04-22T15:15:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.980+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4819c140 con 0x7fcd48103fe0 2024-04-22T15:15:57.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.981+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4804ebb0 con 0x7fcd48103fe0 2024-04-22T15:15:57.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.982+0000 7fcd44ff9700 1 -- 172.21.15.96:0/2945783646 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcd3c00f5d0 con 0x7fcd48103fe0 2024-04-22T15:15:57.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.982+0000 7fcd44ff9700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fcd3006cce0 0x7fcd3006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:57.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.982+0000 7fcd44ff9700 1 -- 172.21.15.96:0/2945783646 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcd3c08d630 con 0x7fcd48103fe0 2024-04-22T15:15:57.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.982+0000 7fcd477fe700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fcd3006cce0 0x7fcd3006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:57.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.982+0000 7fcd477fe700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fcd3006cce0 0x7fcd3006f1a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcd38005ea0 tx=0x7fcd38005e30 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:57.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:57.987+0000 7fcd44ff9700 1 -- 172.21.15.96:0/2945783646 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fcd3c05b430 con 0x7fcd48103fe0 2024-04-22T15:15:58.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.283+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fcd48109780 con 0x7fcd48103fe0 2024-04-22T15:15:58.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.284+0000 7fcd44ff9700 1 -- 172.21.15.96:0/2945783646 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v20) v1 ==== 78+0+98 (secure 0 0 0) 0x7fcd3c05afc0 con 0x7fcd48103fe0 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fcd3006cce0 msgr2=0x7fcd3006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fcd3006cce0 0x7fcd3006f1a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcd38005ea0 tx=0x7fcd38005e30 comp rx=0 tx=0).stop 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 msgr2=0x7fcd48196b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 0x7fcd48196b20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c0092e0 tx=0x7fcd3c00be20 comp rx=0 tx=0).stop 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 shutdown_connections 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fcd3006cce0 0x7fcd3006f1a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd481016b0 0x7fcd481965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 --2- 172.21.15.96:0/2945783646 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd48103fe0 0x7fcd48196b20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 >> 172.21.15.96:0/2945783646 conn(0x7fcd480fb0e0 msgr2=0x7fcd480fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 shutdown_connections 2024-04-22T15:15:58.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:58.286+0000 7fcd4de2f700 1 -- 172.21.15.96:0/2945783646 wait complete. 2024-04-22T15:15:58.300 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:15:58.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:58 smithi096 ceph-mon[26688]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T15:15:58.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:58.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:58.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:58.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:15:58.929 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:58.929 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:58 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2945783646' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-22T15:15:58.929 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:15:58.948 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-22T15:15:58.960 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T15:15:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:58 smithi196 ceph-mon[34717]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T15:15:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:15:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:15:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:15:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:58 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2945783646' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-22T15:15:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:15:59.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.075+0000 7fd850496640 1 -- 172.21.15.96:0/527309052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 msgr2=0x7fd848105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.075+0000 7fd850496640 1 --2- 172.21.15.96:0/527309052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 0x7fd848105850 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd8380098b0 tx=0x7fd83802bcd0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.075+0000 7fd850496640 1 -- 172.21.15.96:0/527309052 shutdown_connections 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.075+0000 7fd850496640 1 --2- 172.21.15.96:0/527309052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 0x7fd848105850 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.075+0000 7fd850496640 1 --2- 172.21.15.96:0/527309052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848100b30 0x7fd848102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.075+0000 7fd850496640 1 -- 172.21.15.96:0/527309052 >> 172.21.15.96:0/527309052 conn(0x7fd8480fa4e0 msgr2=0x7fd8480fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.075+0000 7fd850496640 1 -- 172.21.15.96:0/527309052 shutdown_connections 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.075+0000 7fd850496640 1 -- 172.21.15.96:0/527309052 wait complete. 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd850496640 1 Processor -- start 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd850496640 1 -- start start 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd850496640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848100b30 0x7fd8481958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd850496640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 0x7fd848195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd850496640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd848196340 con 0x7fd848103460 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd850496640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8481964b0 con 0x7fd848100b30 2024-04-22T15:15:59.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd84da31640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 0x7fd848195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd84da31640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 0x7fd848195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44040/0 (socket says 172.21.15.96:44040) 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd84da31640 1 -- 172.21.15.96:0/1238591934 learned_addr learned my addr 172.21.15.96:0/1238591934 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.076+0000 7fd84e232640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848100b30 0x7fd8481958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.077+0000 7fd84da31640 1 -- 172.21.15.96:0/1238591934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848100b30 msgr2=0x7fd8481958c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.077+0000 7fd84da31640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848100b30 0x7fd8481958c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.077+0000 7fd84da31640 1 -- 172.21.15.96:0/1238591934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838009560 con 0x7fd848103460 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.077+0000 7fd84da31640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 0x7fd848195e00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd838009880 tx=0x7fd838004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.077+0000 7fd83f7fe640 1 -- 172.21.15.96:0/1238591934 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8380037c0 con 0x7fd848103460 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.077+0000 7fd83f7fe640 1 -- 172.21.15.96:0/1238591934 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd838003de0 con 0x7fd848103460 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.077+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84819af20 con 0x7fd848103460 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.077+0000 7fd83f7fe640 1 -- 172.21.15.96:0/1238591934 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd838034670 con 0x7fd848103460 2024-04-22T15:15:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.077+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84819b470 con 0x7fd848103460 2024-04-22T15:15:59.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.078+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84818fa10 con 0x7fd848103460 2024-04-22T15:15:59.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.079+0000 7fd83f7fe640 1 -- 172.21.15.96:0/1238591934 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd838004dd0 con 0x7fd848103460 2024-04-22T15:15:59.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.079+0000 7fd83f7fe640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd83406cd10 0x7fd83406f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:59.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.079+0000 7fd83f7fe640 1 -- 172.21.15.96:0/1238591934 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd8380ae8b0 con 0x7fd848103460 2024-04-22T15:15:59.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.079+0000 7fd84e232640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd83406cd10 0x7fd83406f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:59.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.080+0000 7fd84e232640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd83406cd10 0x7fd83406f1d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd84000abe0 tx=0x7fd84000a2d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:59.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.086+0000 7fd83f7fe640 1 -- 172.21.15.96:0/1238591934 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fd838078d60 con 0x7fd848103460 2024-04-22T15:15:59.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.276+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd848108c90 con 0x7fd848103460 2024-04-22T15:15:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.277+0000 7fd83f7fe640 1 -- 172.21.15.96:0/1238591934 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+15149 (secure 0 0 0) 0x7fd838041090 con 0x7fd848103460 2024-04-22T15:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":45,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","created":"2024-04-22T15:08:23.492994+0000","modified":"2024-04-22T15:15:49.819322+0000","last_up_change":"2024-04-22T15:14:15.336641+0000","last_in_change":"2024-04-22T15:13:56.089739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:13:01.924433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T15:15:32.350570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T15:15:32.982528+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2f1b7772-06b4-467f-b14c-698a0de7357f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6803","nonce":1197486134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6805","nonce":1197486134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6809","nonce":1197486134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1197486134},{"type":"v1","addr":"172.21.15.96:6807","nonce":1197486134}]},"public_addr":"172.21.15.96:6803/1197486134","cluster_addr":"172.21.15.96:6805/1197486134","heartbeat_back_addr":"172.21.15.96:6809/1197486134","heartbeat_front_addr":"172.21.15.96:6807/1197486134","state":["exists","up"]},{"osd":1,"uuid":"59260a36-a2ed-42f9-8f3a-209aae4c4238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6811","nonce":3577737028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6813","nonce":3577737028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6817","nonce":3577737028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3577737028},{"type":"v1","addr":"172.21.15.96:6815","nonce":3577737028}]},"public_addr":"172.21.15.96:6811/3577737028","cluster_addr":"172.21.15.96:6813/3577737028","heartbeat_back_addr":"172.21.15.96:6817/3577737028","heartbeat_front_addr":"172.21.15.96:6815/3577737028","state":["exists","up"]},{"osd":2,"uuid":"897e21a9-3009-41c6-acf1-e44414fc69c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6819","nonce":1428742985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6821","nonce":1428742985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6825","nonce":1428742985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1428742985},{"type":"v1","addr":"172.21.15.96:6823","nonce":1428742985}]},"public_addr":"172.21.15.96:6819/1428742985","cluster_addr":"172.21.15.96:6821/1428742985","heartbeat_back_addr":"172.21.15.96:6825/1428742985","heartbeat_front_addr":"172.21.15.96:6823/1428742985","state":["exists","up"]},{"osd":3,"uuid":"201c3153-a284-471e-894d-2273001cc29e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6801","nonce":1414997674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6803","nonce":1414997674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6807","nonce":1414997674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1414997674},{"type":"v1","addr":"172.21.15.196:6805","nonce":1414997674}]},"public_addr":"172.21.15.196:6801/1414997674","cluster_addr":"172.21.15.196:6803/1414997674","heartbeat_back_addr":"172.21.15.196:6807/1414997674","heartbeat_front_addr":"172.21.15.196:6805/1414997674","state":["exists","up"]},{"osd":4,"uuid":"780ed007-14ef-4c92-a5e9-6c57afcad448","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6809","nonce":1827225092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6811","nonce":1827225092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6815","nonce":1827225092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":1827225092},{"type":"v1","addr":"172.21.15.196:6813","nonce":1827225092}]},"public_addr":"172.21.15.196:6809/1827225092","cluster_addr":"172.21.15.196:6811/1827225092","heartbeat_back_addr":"172.21.15.196:6815/1827225092","heartbeat_front_addr":"172.21.15.196:6813/1827225092","state":["exists","up"]},{"osd":5,"uuid":"16384d8d-bef4-4f65-9079-e2b4518241da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6817","nonce":2750420250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6819","nonce":2750420250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6823","nonce":2750420250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2750420250},{"type":"v1","addr":"172.21.15.196:6821","nonce":2750420250}]},"public_addr":"172.21.15.196:6817/2750420250","cluster_addr":"172.21.15.196:6819/2750420250","heartbeat_back_addr":"172.21.15.196:6823/2750420250","heartbeat_front_addr":"172.21.15.196:6821/2750420250","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:00.654872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:28.691804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:57.737056+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:21.679902+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:47.489372+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:14:13.600900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6827/887933858":"2024-04-23T15:15:49.819302+0000","172.21.15.96:0/1319578885":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6800/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3667378212":"2024-04-23T15:08:51.463883+0000","172.21.15.96:6801/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1237353220":"2024-04-23T15:08:51.463883+0000","172.21.15.96:6826/887933858":"2024-04-23T15:15:49.819302+0000","172.21.15.96:0/4224647460":"2024-04-23T15:08:51.463883+0000","172.21.15.96:6800/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:0/3321679733":"2024-04-23T15:09:23.920294+0000","172.21.15.96:0/1794679103":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3486391579":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/838799683":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6800/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6801/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3921802975":"2024-04-23T15:10:37.605301+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T15:15:59.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd83406cd10 msgr2=0x7fd83406f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd83406cd10 0x7fd83406f1d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd84000abe0 tx=0x7fd84000a2d0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 msgr2=0x7fd848195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 0x7fd848195e00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd838009880 tx=0x7fd838004270 comp rx=0 tx=0).stop 2024-04-22T15:15:59.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 shutdown_connections 2024-04-22T15:15:59.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd848103460 0x7fd848195e00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fd83406cd10 0x7fd83406f1d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 --2- 172.21.15.96:0/1238591934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848100b30 0x7fd8481958c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 >> 172.21.15.96:0/1238591934 conn(0x7fd8480fa4e0 msgr2=0x7fd8480fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 shutdown_connections 2024-04-22T15:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.279+0000 7fd850496640 1 -- 172.21.15.96:0/1238591934 wait complete. 2024-04-22T15:15:59.293 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:13:01.924433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T15:15:32.350570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T15:15:32.982528+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T15:15:59.294 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T15:15:59.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.393+0000 7fa318e69640 1 -- 172.21.15.96:0/3039017273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 msgr2=0x7fa314100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.393+0000 7fa318e69640 1 --2- 172.21.15.96:0/3039017273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 0x7fa314100610 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa308009880 tx=0x7fa30802bce0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.393+0000 7fa318e69640 1 -- 172.21.15.96:0/3039017273 shutdown_connections 2024-04-22T15:15:59.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.393+0000 7fa318e69640 1 --2- 172.21.15.96:0/3039017273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 0x7fa314100610 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.393+0000 7fa318e69640 1 --2- 172.21.15.96:0/3039017273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3140fefa0 0x7fa3140ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.393+0000 7fa318e69640 1 -- 172.21.15.96:0/3039017273 >> 172.21.15.96:0/3039017273 conn(0x7fa3140fa510 msgr2=0x7fa3140fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.394+0000 7fa318e69640 1 -- 172.21.15.96:0/3039017273 shutdown_connections 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.394+0000 7fa318e69640 1 -- 172.21.15.96:0/3039017273 wait complete. 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.394+0000 7fa318e69640 1 Processor -- start 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.394+0000 7fa318e69640 1 -- start start 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.394+0000 7fa318e69640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3140fefa0 0x7fa31419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.394+0000 7fa318e69640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 0x7fa31419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.394+0000 7fa318e69640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31419ca10 con 0x7fa314100190 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.394+0000 7fa318e69640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31419cb80 con 0x7fa3140fefa0 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa311d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 0x7fa31419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa311d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 0x7fa31419c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44058/0 (socket says 172.21.15.96:44058) 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa311d9b640 1 -- 172.21.15.96:0/3504158800 learned_addr learned my addr 172.21.15.96:0/3504158800 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa31259c640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3140fefa0 0x7fa31419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:59.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa311d9b640 1 -- 172.21.15.96:0/3504158800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3140fefa0 msgr2=0x7fa31419bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa311d9b640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3140fefa0 0x7fa31419bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa311d9b640 1 -- 172.21.15.96:0/3504158800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308009560 con 0x7fa314100190 2024-04-22T15:15:59.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa311d9b640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 0x7fa31419c440 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa308009230 tx=0x7fa308004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:59.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa3037fe640 1 -- 172.21.15.96:0/3504158800 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3080037c0 con 0x7fa314100190 2024-04-22T15:15:59.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3141a15b0 con 0x7fa314100190 2024-04-22T15:15:59.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa3037fe640 1 -- 172.21.15.96:0/3504158800 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa308004aa0 con 0x7fa314100190 2024-04-22T15:15:59.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3141a1ab0 con 0x7fa314100190 2024-04-22T15:15:59.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.395+0000 7fa3037fe640 1 -- 172.21.15.96:0/3504158800 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa308034670 con 0x7fa314100190 2024-04-22T15:15:59.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.397+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa314106bc0 con 0x7fa314100190 2024-04-22T15:15:59.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.397+0000 7fa3037fe640 1 -- 172.21.15.96:0/3504158800 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa308003920 con 0x7fa314100190 2024-04-22T15:15:59.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.397+0000 7fa3037fe640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa2fc06cc50 0x7fa2fc06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:59.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.397+0000 7fa3037fe640 1 -- 172.21.15.96:0/3504158800 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa3080ae6a0 con 0x7fa314100190 2024-04-22T15:15:59.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.397+0000 7fa31259c640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa2fc06cc50 0x7fa2fc06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:59.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.398+0000 7fa31259c640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa2fc06cc50 0x7fa2fc06f110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa304005fd0 tx=0x7fa30400b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:59.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.406+0000 7fa3037fe640 1 -- 172.21.15.96:0/3504158800 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fa308078ac0 con 0x7fa314100190 2024-04-22T15:15:59.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.596+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3141a1de0 con 0x7fa314100190 2024-04-22T15:15:59.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.596+0000 7fa3037fe640 1 -- 172.21.15.96:0/3504158800 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa30807c0e0 con 0x7fa314100190 2024-04-22T15:15:59.600 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-04-22T15:15:59.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.598+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa2fc06cc50 msgr2=0x7fa2fc06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.598+0000 7fa318e69640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa2fc06cc50 0x7fa2fc06f110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa304005fd0 tx=0x7fa30400b410 comp rx=0 tx=0).stop 2024-04-22T15:15:59.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.598+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 msgr2=0x7fa31419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.598+0000 7fa318e69640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 0x7fa31419c440 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa308009230 tx=0x7fa308004270 comp rx=0 tx=0).stop 2024-04-22T15:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.598+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 shutdown_connections 2024-04-22T15:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.598+0000 7fa318e69640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa314100190 0x7fa31419c440 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.598+0000 7fa318e69640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fa2fc06cc50 0x7fa2fc06f110 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.598+0000 7fa318e69640 1 --2- 172.21.15.96:0/3504158800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3140fefa0 0x7fa31419bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.598+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 >> 172.21.15.96:0/3504158800 conn(0x7fa3140fa510 msgr2=0x7fa3141033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.599+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 shutdown_connections 2024-04-22T15:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.599+0000 7fa318e69640 1 -- 172.21.15.96:0/3504158800 wait complete. 2024-04-22T15:15:59.613 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T15:15:59.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.712+0000 7f23712ab640 1 -- 172.21.15.96:0/3843998171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c100070 msgr2=0x7f236c1004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.712+0000 7f23712ab640 1 --2- 172.21.15.96:0/3843998171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c100070 0x7f236c1004f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2354009880 tx=0x7f235402bce0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.712+0000 7f23712ab640 1 -- 172.21.15.96:0/3843998171 shutdown_connections 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.712+0000 7f23712ab640 1 --2- 172.21.15.96:0/3843998171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c100070 0x7f236c1004f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.712+0000 7f23712ab640 1 --2- 172.21.15.96:0/3843998171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f236c0fef20 0x7f236c0ff340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.712+0000 7f23712ab640 1 -- 172.21.15.96:0/3843998171 >> 172.21.15.96:0/3843998171 conn(0x7f236c0fa4b0 msgr2=0x7f236c0fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.712+0000 7f23712ab640 1 -- 172.21.15.96:0/3843998171 shutdown_connections 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.712+0000 7f23712ab640 1 -- 172.21.15.96:0/3843998171 wait complete. 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f23712ab640 1 Processor -- start 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f23712ab640 1 -- start start 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f23712ab640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c0fef20 0x7f236c067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f23712ab640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f236c100070 0x7f236c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:59.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f23712ab640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c19ac20 con 0x7f236c0fef20 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f23712ab640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c19ad60 con 0x7f236c100070 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f236affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c0fef20 0x7f236c067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f236affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c0fef20 0x7f236c067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44076/0 (socket says 172.21.15.96:44076) 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f236affd640 1 -- 172.21.15.96:0/2633536884 learned_addr learned my addr 172.21.15.96:0/2633536884 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f2363fff640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f236c100070 0x7f236c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f236affd640 1 -- 172.21.15.96:0/2633536884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f236c100070 msgr2=0x7f236c19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f236affd640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f236c100070 0x7f236c19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f236affd640 1 -- 172.21.15.96:0/2633536884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2354009560 con 0x7f236c0fef20 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f236affd640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c0fef20 0x7f236c067ec0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f235c00d8d0 tx=0x7f235c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f2368ff9640 1 -- 172.21.15.96:0/2633536884 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f235c009c00 con 0x7f236c0fef20 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f2368ff9640 1 -- 172.21.15.96:0/2633536884 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f235c009d60 con 0x7f236c0fef20 2024-04-22T15:15:59.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c19afb0 con 0x7f236c0fef20 2024-04-22T15:15:59.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f2368ff9640 1 -- 172.21.15.96:0/2633536884 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f235c0049d0 con 0x7f236c0fef20 2024-04-22T15:15:59.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.713+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c19b4e0 con 0x7f236c0fef20 2024-04-22T15:15:59.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.714+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c107a70 con 0x7f236c0fef20 2024-04-22T15:15:59.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.715+0000 7f2368ff9640 1 -- 172.21.15.96:0/2633536884 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f235c004c90 con 0x7f236c0fef20 2024-04-22T15:15:59.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.715+0000 7f2368ff9640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f235806cc50 0x7f235806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:15:59.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.715+0000 7f2363fff640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f235806cc50 0x7f235806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:15:59.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.715+0000 7f2368ff9640 1 -- 172.21.15.96:0/2633536884 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f235c08c090 con 0x7f236c0fef20 2024-04-22T15:15:59.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.716+0000 7f2363fff640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f235806cc50 0x7f235806f110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2354006010 tx=0x7f2354000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:15:59.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.720+0000 7f2368ff9640 1 -- 172.21.15.96:0/2633536884 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f235c056400 con 0x7f236c0fef20 2024-04-22T15:15:59.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.899+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f236c107c50 con 0x7f236c0fef20 2024-04-22T15:15:59.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.899+0000 7f2368ff9640 1 -- 172.21.15.96:0/2633536884 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7f235c059a20 con 0x7f236c0fef20 2024-04-22T15:15:59.903 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-04-22T15:15:59.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.901+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f235806cc50 msgr2=0x7f235806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.901+0000 7f23712ab640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f235806cc50 0x7f235806f110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2354006010 tx=0x7f2354000bc0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.901+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c0fef20 msgr2=0x7f236c067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:15:59.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.901+0000 7f23712ab640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c0fef20 0x7f236c067ec0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f235c00d8d0 tx=0x7f235c00dd90 comp rx=0 tx=0).stop 2024-04-22T15:15:59.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.901+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 shutdown_connections 2024-04-22T15:15:59.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.901+0000 7f23712ab640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f235806cc50 0x7f235806f110 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.901+0000 7f23712ab640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f236c100070 0x7f236c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.901+0000 7f23712ab640 1 --2- 172.21.15.96:0/2633536884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f236c0fef20 0x7f236c067ec0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:15:59.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.901+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 >> 172.21.15.96:0/2633536884 conn(0x7f236c0fa4b0 msgr2=0x7f236c103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:15:59.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.902+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 shutdown_connections 2024-04-22T15:15:59.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:15:59.902+0000 7f23712ab640 1 -- 172.21.15.96:0/2633536884 wait complete. 2024-04-22T15:15:59.916 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:59 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1238591934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:15:59.916 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:15:59 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3504158800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T15:15:59.917 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T15:16:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:59 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1238591934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:16:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:15:59 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3504158800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 -- 172.21.15.96:0/468246034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 msgr2=0x7f2e80100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 --2- 172.21.15.96:0/468246034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 0x7f2e80100600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009c20 tx=0x7f2e7002c1b0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 -- 172.21.15.96:0/468246034 shutdown_connections 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 --2- 172.21.15.96:0/468246034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 0x7f2e80100600 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 --2- 172.21.15.96:0/468246034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e800fef90 0x7f2e800ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 -- 172.21.15.96:0/468246034 >> 172.21.15.96:0/468246034 conn(0x7f2e800fa4e0 msgr2=0x7f2e800fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 -- 172.21.15.96:0/468246034 shutdown_connections 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 -- 172.21.15.96:0/468246034 wait complete. 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 Processor -- start 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 -- start start 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e800fef90 0x7f2e80067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:00.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 0x7f2e8019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e801964f0 con 0x7f2e80100180 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e86fa2640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8019cfc0 con 0x7f2e800fef90 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e7ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 0x7f2e8019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e7ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 0x7f2e8019ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44086/0 (socket says 172.21.15.96:44086) 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.030+0000 7f2e7ffff640 1 -- 172.21.15.96:0/1880993472 learned_addr learned my addr 172.21.15.96:0/1880993472 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e84d3e640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e800fef90 0x7f2e80067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e7ffff640 1 -- 172.21.15.96:0/1880993472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e800fef90 msgr2=0x7f2e80067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e7ffff640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e800fef90 0x7f2e80067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e7ffff640 1 -- 172.21.15.96:0/1880993472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e70009900 con 0x7f2e80100180 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e7ffff640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 0x7f2e8019ca80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2e70000f80 tx=0x7f2e70032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e7dffb640 1 -- 172.21.15.96:0/1880993472 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e70034820 con 0x7f2e80100180 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e7dffb640 1 -- 172.21.15.96:0/1880993472 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e70003e30 con 0x7f2e80100180 2024-04-22T15:16:00.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8019d1f0 con 0x7f2e80100180 2024-04-22T15:16:00.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e7dffb640 1 -- 172.21.15.96:0/1880993472 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e7003f440 con 0x7f2e80100180 2024-04-22T15:16:00.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.031+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8019d710 con 0x7f2e80100180 2024-04-22T15:16:00.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.033+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e80107cc0 con 0x7f2e80100180 2024-04-22T15:16:00.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.033+0000 7f2e7dffb640 1 -- 172.21.15.96:0/1880993472 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2e7003b040 con 0x7f2e80100180 2024-04-22T15:16:00.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.033+0000 7f2e7dffb640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2e6c06cc50 0x7f2e6c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:00.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.034+0000 7f2e7dffb640 1 -- 172.21.15.96:0/1880993472 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2e700aefc0 con 0x7f2e80100180 2024-04-22T15:16:00.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.034+0000 7f2e84d3e640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2e6c06cc50 0x7f2e6c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:00.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.034+0000 7f2e84d3e640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2e6c06cc50 0x7f2e6c06f110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2e800fffe0 tx=0x7f2e7800b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:00.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.038+0000 7f2e7dffb640 1 -- 172.21.15.96:0/1880993472 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2e70079330 con 0x7f2e80100180 2024-04-22T15:16:00.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.215+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f2e8019db30 con 0x7f2e80100180 2024-04-22T15:16:00.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.216+0000 7f2e7dffb640 1 -- 172.21.15.96:0/1880993472 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2e7007c950 con 0x7f2e80100180 2024-04-22T15:16:00.219 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-04-22T15:16:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2e6c06cc50 msgr2=0x7f2e6c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2e6c06cc50 0x7f2e6c06f110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2e800fffe0 tx=0x7f2e7800b390 comp rx=0 tx=0).stop 2024-04-22T15:16:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 msgr2=0x7f2e8019ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 0x7f2e8019ca80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2e70000f80 tx=0x7f2e70032b60 comp rx=0 tx=0).stop 2024-04-22T15:16:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 shutdown_connections 2024-04-22T15:16:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e80100180 0x7f2e8019ca80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2e6c06cc50 0x7f2e6c06f110 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 --2- 172.21.15.96:0/1880993472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e800fef90 0x7f2e80067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 >> 172.21.15.96:0/1880993472 conn(0x7f2e800fa4e0 msgr2=0x7f2e801033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 shutdown_connections 2024-04-22T15:16:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.218+0000 7f2e86fa2640 1 -- 172.21.15.96:0/1880993472 wait complete. 2024-04-22T15:16:00.232 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T15:16:00.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.328+0000 7f2ff3620640 1 -- 172.21.15.96:0/870613450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fec1001a0 msgr2=0x7f2fec100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:00.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.328+0000 7f2ff3620640 1 --2- 172.21.15.96:0/870613450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fec1001a0 0x7f2fec100620 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009c20 tx=0x7f2fe002c1b0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.328+0000 7f2ff3620640 1 -- 172.21.15.96:0/870613450 shutdown_connections 2024-04-22T15:16:00.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.328+0000 7f2ff3620640 1 --2- 172.21.15.96:0/870613450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fec1001a0 0x7f2fec100620 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.328+0000 7f2ff3620640 1 --2- 172.21.15.96:0/870613450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fec0fefb0 0x7f2fec0ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.328+0000 7f2ff3620640 1 -- 172.21.15.96:0/870613450 >> 172.21.15.96:0/870613450 conn(0x7f2fec0fa4e0 msgr2=0x7f2fec0fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:00.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.328+0000 7f2ff3620640 1 -- 172.21.15.96:0/870613450 shutdown_connections 2024-04-22T15:16:00.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.328+0000 7f2ff3620640 1 -- 172.21.15.96:0/870613450 wait complete. 2024-04-22T15:16:00.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff3620640 1 Processor -- start 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff3620640 1 -- start start 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff3620640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fec0fefb0 0x7f2fec067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff3620640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fec1001a0 0x7f2fec19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff3620640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec19abe0 con 0x7f2fec1001a0 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff3620640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec19ad20 con 0x7f2fec0fefb0 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff0bbb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fec1001a0 0x7f2fec19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff13bc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fec0fefb0 0x7f2fec067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff13bc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fec0fefb0 0x7f2fec067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:54916/0 (socket says 172.21.15.96:54916) 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff13bc640 1 -- 172.21.15.96:0/2277405595 learned_addr learned my addr 172.21.15.96:0/2277405595 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff0bbb640 1 -- 172.21.15.96:0/2277405595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fec0fefb0 msgr2=0x7f2fec067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.329+0000 7f2ff0bbb640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fec0fefb0 0x7f2fec067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.330+0000 7f2ff0bbb640 1 -- 172.21.15.96:0/2277405595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0009900 con 0x7f2fec1001a0 2024-04-22T15:16:00.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.330+0000 7f2ff0bbb640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fec1001a0 0x7f2fec19a610 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2fe002c6f0 tx=0x7f2fe0032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:00.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.330+0000 7f2fde7fc640 1 -- 172.21.15.96:0/2277405595 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fe0032ed0 con 0x7f2fec1001a0 2024-04-22T15:16:00.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.330+0000 7f2fde7fc640 1 -- 172.21.15.96:0/2277405595 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fe0003e30 con 0x7f2fec1001a0 2024-04-22T15:16:00.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.330+0000 7f2fde7fc640 1 -- 172.21.15.96:0/2277405595 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fe003f440 con 0x7f2fec1001a0 2024-04-22T15:16:00.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.330+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec19af70 con 0x7f2fec1001a0 2024-04-22T15:16:00.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.330+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec19b470 con 0x7f2fec1001a0 2024-04-22T15:16:00.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.331+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec107d40 con 0x7f2fec1001a0 2024-04-22T15:16:00.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.331+0000 7f2fde7fc640 1 -- 172.21.15.96:0/2277405595 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2fe003b040 con 0x7f2fec1001a0 2024-04-22T15:16:00.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.332+0000 7f2fde7fc640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2fd806cd10 0x7f2fd806f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:00.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.332+0000 7f2fde7fc640 1 -- 172.21.15.96:0/2277405595 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2fe00aef90 con 0x7f2fec1001a0 2024-04-22T15:16:00.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.332+0000 7f2ff13bc640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2fd806cd10 0x7f2fd806f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:00.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.332+0000 7f2ff13bc640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2fd806cd10 0x7f2fd806f1d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4005fd0 tx=0x7f2fe400b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:00.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.336+0000 7f2fde7fc640 1 -- 172.21.15.96:0/2277405595 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2fe0079300 con 0x7f2fec1001a0 2024-04-22T15:16:00.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.551+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fec19b860 con 0x7f2fec1001a0 2024-04-22T15:16:00.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.552+0000 7f2fde7fc640 1 -- 172.21.15.96:0/2277405595 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4859 (secure 0 0 0) 0x7f2fe007c920 con 0x7f2fec1001a0 2024-04-22T15:16:00.555 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:16:00.555 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:15:53.847723+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14494,"mds_1":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14494":{"gid":14494,"name":"cephfs.smithi096.drkkdi","rank":0,"incarnation":15,"state":"up:active","state_seq":6,"addr":"172.21.15.96:6829/3534778299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3534778299},{"type":"v1","addr":"172.21.15.96:6829","nonce":3534778299}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14504":{"gid":14504,"name":"cephfs.smithi096.dymqlv","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6827/231416323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":231416323},{"type":"v1","addr":"172.21.15.96:6827","nonce":231416323}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24257":{"gid":24257,"name":"cephfs.smithi196.hwpcvg","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6825/3248990491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":3248990491},{"type":"v1","addr":"172.21.15.196:6825","nonce":3248990491}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24267":{"gid":24267,"name":"cephfs.smithi196.pbvqlq","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.196:6827/3238575297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3238575297},{"type":"v1","addr":"172.21.15.196:6827","nonce":3238575297}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-22T15:16:00.556 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:16:00.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2fd806cd10 msgr2=0x7f2fd806f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2fd806cd10 0x7f2fd806f1d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4005fd0 tx=0x7f2fe400b410 comp rx=0 tx=0).stop 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fec1001a0 msgr2=0x7f2fec19a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fec1001a0 0x7f2fec19a610 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2fe002c6f0 tx=0x7f2fe0032b60 comp rx=0 tx=0).stop 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 shutdown_connections 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fec1001a0 0x7f2fec19a610 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2fd806cd10 0x7f2fd806f1d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 --2- 172.21.15.96:0/2277405595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fec0fefb0 0x7f2fec067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 >> 172.21.15.96:0/2277405595 conn(0x7f2fec0fa4e0 msgr2=0x7f2fec1033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 shutdown_connections 2024-04-22T15:16:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:00.554+0000 7f2ff3620640 1 -- 172.21.15.96:0/2277405595 wait complete. 2024-04-22T15:16:00.568 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 20, 'max_mds': 2, 'flags': 50} 2024-04-22T15:16:00.568 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-22T15:16:00.596 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-22T15:16:00.596 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-22T15:16:00.596 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-22T15:16:00.596 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T15:16:00.596 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:00.596 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-22T15:16:00.596 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T15:16:00.597 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:00.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:00.597 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-04-22T15:16:00.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:00.613 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link delete ceph-brx 2024-04-22T15:16:00.680 INFO:teuthology.orchestra.run.smithi096.stderr:Cannot find device "ceph-brx" 2024-04-22T15:16:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:16:00.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:00.682 DEBUG:teuthology.orchestra.run.smithi196:> ip netns list 2024-04-22T15:16:00.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:00.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip link delete ceph-brx 2024-04-22T15:16:00.767 INFO:teuthology.orchestra.run.smithi196.stderr:Cannot find device "ceph-brx" 2024-04-22T15:16:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:16:00.769 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-22T15:16:00.769 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T15:16:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:00 smithi096 ceph-mon[26688]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-22T15:16:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:00 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2633536884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T15:16:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:00 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1880993472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T15:16:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:00 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:00 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:00 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2277405595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T15:16:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:00 smithi196 ceph-mon[34717]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-22T15:16:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:00 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2633536884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T15:16:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:00 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1880993472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T15:16:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:00 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:00 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:00 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2277405595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T15:16:01.536 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:16:02.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:02 smithi096 ceph-mon[26688]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T15:16:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:02 smithi196 ceph-mon[34717]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T15:16:03.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.769+0000 7f5a33677700 1 -- 172.21.15.96:0/3064121667 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c1017d0 msgr2=0x7f5a2c105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:03.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.769+0000 7f5a33677700 1 --2- 172.21.15.96:0/3064121667 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c1017d0 0x7f5a2c105820 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009910 tx=0x7f5a28009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:03.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.769+0000 7f5a33677700 1 -- 172.21.15.96:0/3064121667 shutdown_connections 2024-04-22T15:16:03.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.769+0000 7f5a33677700 1 --2- 172.21.15.96:0/3064121667 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c1017d0 0x7f5a2c105820 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:03.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.769+0000 7f5a33677700 1 --2- 172.21.15.96:0/3064121667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a2c100e20 0x7f5a2c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:03.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.769+0000 7f5a33677700 1 -- 172.21.15.96:0/3064121667 >> 172.21.15.96:0/3064121667 conn(0x7f5a2c0fc6d0 msgr2=0x7f5a2c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:03.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a33677700 1 -- 172.21.15.96:0/3064121667 shutdown_connections 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a33677700 1 -- 172.21.15.96:0/3064121667 wait complete. 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a33677700 1 Processor -- start 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a33677700 1 -- start start 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a33677700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c100e20 0x7f5a2c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a33677700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a2c1017d0 0x7f5a2c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a33677700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c19d380 con 0x7f5a2c100e20 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a33677700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c196960 con 0x7f5a2c1017d0 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a31413700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c100e20 0x7f5a2c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a31413700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c100e20 0x7f5a2c19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59796/0 (socket says 172.21.15.96:59796) 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.770+0000 7f5a31413700 1 -- 172.21.15.96:0/1195775261 learned_addr learned my addr 172.21.15.96:0/1195775261 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a30c12700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a2c1017d0 0x7f5a2c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a31413700 1 -- 172.21.15.96:0/1195775261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a2c1017d0 msgr2=0x7f5a2c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:03.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a31413700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a2c1017d0 0x7f5a2c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:03.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a31413700 1 -- 172.21.15.96:0/1195775261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a280095f0 con 0x7f5a2c100e20 2024-04-22T15:16:03.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a31413700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c100e20 0x7f5a2c19c820 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00d8d0 tx=0x7f5a1c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:03.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a227fc700 1 -- 172.21.15.96:0/1195775261 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a1c009940 con 0x7f5a2c100e20 2024-04-22T15:16:03.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a227fc700 1 -- 172.21.15.96:0/1195775261 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a1c010460 con 0x7f5a2c100e20 2024-04-22T15:16:03.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2c196c40 con 0x7f5a2c100e20 2024-04-22T15:16:03.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a227fc700 1 -- 172.21.15.96:0/1195775261 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a1c00f5f0 con 0x7f5a2c100e20 2024-04-22T15:16:03.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.771+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2c197190 con 0x7f5a2c100e20 2024-04-22T15:16:03.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.772+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2c109220 con 0x7f5a2c100e20 2024-04-22T15:16:03.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.772+0000 7f5a227fc700 1 -- 172.21.15.96:0/1195775261 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5a1c0105d0 con 0x7f5a2c100e20 2024-04-22T15:16:03.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.773+0000 7f5a227fc700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a1806cce0 0x7f5a1806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:03.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.773+0000 7f5a30c12700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a1806cce0 0x7f5a1806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:03.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.773+0000 7f5a227fc700 1 -- 172.21.15.96:0/1195775261 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5a1c08c210 con 0x7f5a2c100e20 2024-04-22T15:16:03.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.773+0000 7f5a30c12700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a1806cce0 0x7f5a1806f1a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800b5c0 tx=0x7f5a280052a0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:03.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:03.779+0000 7f5a227fc700 1 -- 172.21.15.96:0/1195775261 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f5a1c05a090 con 0x7f5a2c100e20 2024-04-22T15:16:04.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.025+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5a2c102300 con 0x7f5a2c100e20 2024-04-22T15:16:04.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.026+0000 7f5a227fc700 1 -- 172.21.15.96:0/1195775261 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+83 (secure 0 0 0) 0x7f5a1c059c20 con 0x7f5a2c100e20 2024-04-22T15:16:04.029 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-22T15:16:04.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a1806cce0 msgr2=0x7f5a1806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a1806cce0 0x7f5a1806f1a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800b5c0 tx=0x7f5a280052a0 comp rx=0 tx=0).stop 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c100e20 msgr2=0x7f5a2c19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c100e20 0x7f5a2c19c820 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00d8d0 tx=0x7f5a1c00dc90 comp rx=0 tx=0).stop 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 shutdown_connections 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a1806cce0 0x7f5a1806f1a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a2c100e20 0x7f5a2c19c820 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 --2- 172.21.15.96:0/1195775261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a2c1017d0 0x7f5a2c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 >> 172.21.15.96:0/1195775261 conn(0x7f5a2c0fc6d0 msgr2=0x7f5a2c0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 shutdown_connections 2024-04-22T15:16:04.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04.028+0000 7f5a33677700 1 -- 172.21.15.96:0/1195775261 wait complete. 2024-04-22T15:16:04.758 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T15:16:04.758 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T15:16:04.758 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-04-22T15:16:04.758 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T15:16:04.759 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:04.759 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T15:16:04.759 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T15:16:04.759 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T15:16:04.759 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T15:16:04.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:04.759 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-04-22T15:16:04.775 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T15:16:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T15:16:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T15:16:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42171sec preferred_lft 42171sec 2024-04-22T15:16:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::811d:49ea:b028:3af9/64 scope link noprefixroute 2024-04-22T15:16:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:16:04.777 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T15:16:04.777 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:04.777 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-22T15:16:04.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link add name ceph-brx type bridge 2024-04-22T15:16:04.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip addr flush dev ceph-brx 2024-04-22T15:16:04.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set ceph-brx up 2024-04-22T15:16:04.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T15:16:04.777 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-22T15:16:04.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:04.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:04 smithi096 ceph-mon[26688]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T15:16:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:04 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1195775261' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T15:16:04.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:04.988 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T15:16:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:04 smithi196 ceph-mon[34717]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T15:16:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:04 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1195775261' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T15:16:05.011 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-04-22T15:16:05.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:05.013 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-04-22T15:16:05.112 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-04-22T15:16:05.112 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T15:16:05.113 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T15:16:05.113 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T15:16:05.113 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T15:16:05.113 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:05.113 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-22T15:16:05.113 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T15:16:05.113 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T15:16:05.113 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T15:16:05.114 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-22T15:16:05.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:05.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:05.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:05.341 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-04-22T15:16:05.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:05.356 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list-id 2024-04-22T15:16:05.411 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:05.411 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-22T15:16:05.412 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:16:05.412 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T15:16:05.412 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-22T15:16:05.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:05.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:05.527 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T15:16:05.527 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:05.527 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-22T15:16:05.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T15:16:05.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T15:16:05.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T15:16:05.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T15:16:05.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T15:16:05.528 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-22T15:16:05.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:05.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:05.648 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:05.648 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-22T15:16:05.649 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-04-22T15:16:05.649 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T15:16:05.649 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-22T15:16:05.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:05.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:05.723 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T15:16:05.723 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T15:16:05.723 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:05.737 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T15:16:05.737 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T15:16:05.737 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:05.791 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-22T15:16:05.855 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T15:16:05.910 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-22T15:16:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-22T15:16:05.912 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/892a4a4debc30a7f869ead13114c1732697981d8f079bea517f755df1696bfec/merged 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/16e2a093d81b11cb51634cb04e4a9178b765bcd0b2f65a4895c28286ee4bc478/merged 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/0 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/e128ecbec7dbd68733da7970d44bf570f3136a8687489e18a01ec32913fb3b31/merged 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/9dbf9438df83e2c55d222aae78a9bba082bbacac15d12e17535dba68a2526f10/merged 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/7df1f230563a3e744e59266729a73b698a4908ff8a6db2eedbd2a31634233710/merged 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/5cc9776f7afdde709ab27eb3cf732ecad1f14edec43939a9fca6260a31bc0e6f/merged 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/8a271f1fd27825890cc03ffc5598d0992a1a28ba483b07db8b1c544a47930296/merged 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/7158af2627874ffbbfd6417e7ce66e55636fdbd3270c7fa67c532f444701ca50/merged 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/9c7ab20abbc44a061e98cde206d8fef49232e9b9a0bf5b91d2b361553957f15f/merged 2024-04-22T15:16:05.913 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/73f3fd27957a511a4ffcd7158f7726bfb726697e67b546716d022cb3bb875e06/merged 2024-04-22T15:16:05.914 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/f3de62bba6adc55b80819c1b0e2590a0156cd77ce4a42ebc1d42e068dc33f5d1/merged 2024-04-22T15:16:05.914 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/4a346d009a7186cbee8ad6d92e2154fb1255571afff6ca39e521582fd980ad40/merged 2024-04-22T15:16:05.914 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/2f54e74d50cd31e9c92153ea5080b613164e6f042d03a9372e7206e25f3445e2/merged 2024-04-22T15:16:05.914 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-22T15:16:05.914 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:16:05.914 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:16:05.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:05.914 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-22T15:16:05.965 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T15:16:05.966 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T15:16:06.008 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-22T15:16:06.031 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T15:16:06.087 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-22T15:16:06.087 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-22T15:16:06.087 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-22T15:16:06.087 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-22T15:16:06.087 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-22T15:16:06.087 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-22T15:16:06.087 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-22T15:16:06.087 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-22T15:16:06.087 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T15:16:06.088 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/892a4a4debc30a7f869ead13114c1732697981d8f079bea517f755df1696bfec/merged 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/16e2a093d81b11cb51634cb04e4a9178b765bcd0b2f65a4895c28286ee4bc478/merged 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/0 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/e128ecbec7dbd68733da7970d44bf570f3136a8687489e18a01ec32913fb3b31/merged 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/9dbf9438df83e2c55d222aae78a9bba082bbacac15d12e17535dba68a2526f10/merged 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/7df1f230563a3e744e59266729a73b698a4908ff8a6db2eedbd2a31634233710/merged 2024-04-22T15:16:06.089 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/5cc9776f7afdde709ab27eb3cf732ecad1f14edec43939a9fca6260a31bc0e6f/merged 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/8a271f1fd27825890cc03ffc5598d0992a1a28ba483b07db8b1c544a47930296/merged 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/7158af2627874ffbbfd6417e7ce66e55636fdbd3270c7fa67c532f444701ca50/merged 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/9c7ab20abbc44a061e98cde206d8fef49232e9b9a0bf5b91d2b361553957f15f/merged 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/73f3fd27957a511a4ffcd7158f7726bfb726697e67b546716d022cb3bb875e06/merged 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/f3de62bba6adc55b80819c1b0e2590a0156cd77ce4a42ebc1d42e068dc33f5d1/merged 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/4a346d009a7186cbee8ad6d92e2154fb1255571afff6ca39e521582fd980ad40/merged 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/containers/storage/overlay/2f54e74d50cd31e9c92153ea5080b613164e6f042d03a9372e7206e25f3445e2/merged 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:16:06.090 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:16:06.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:06.091 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-22T15:16:06.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-04-22T15:16:06.094+0000 7f3da2039480 -1 init, newargv = 0x55d9540cfd00 newargc=15 2024-04-22T15:16:06.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[75874]: starting ceph client 2024-04-22T15:16:06.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[75874]: starting fuse 2024-04-22T15:16:06.112 INFO:teuthology.orchestra.run.smithi096.stdout:145 2024-04-22T15:16:06.112 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [145] 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-04-22T15:16:06.113 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-04-22T15:16:06.114 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-04-22T15:16:06.229 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.75874.asok 2024-04-22T15:16:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:06.240 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.75874.asok 2024-04-22T15:16:06.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:06.240 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.75874.asok status 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9d98999acebbad2663cc4938e94c9ffd83485cbd", 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 18.2.2 (9d98999acebbad2663cc4938e94c9ffd83485cbd) reef (stable)", 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "75874", 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:16:06.372 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 14544, 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 14544 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 4034229763 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-22T15:16:06.373 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 4034229763 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.14544 192.168.0.1:0/4034229763", 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/4034229763", 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 20, 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 45, 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-04-22T15:16:06.374 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:16:06.380 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T15:16:06.593 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:16:06.715 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:06 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:06.715 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:06 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:06.715 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:06 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:16:06.715 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:06 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:16:06.715 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:06 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:06.715 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:06 smithi096 ceph-mon[26688]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T15:16:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:06 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:06 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:06 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:16:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:06 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:16:06.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:06 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:06.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:06 smithi196 ceph-mon[34717]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T15:16:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.189+0000 7f1e7b653700 1 -- 172.21.15.96:0/2150033090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e74101d90 msgr2=0x7f1e74102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.189+0000 7f1e7b653700 1 --2- 172.21.15.96:0/2150033090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e74101d90 0x7f1e74102170 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009910 tx=0x7f1e70009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.189+0000 7f1e7b653700 1 -- 172.21.15.96:0/2150033090 shutdown_connections 2024-04-22T15:16:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.189+0000 7f1e7b653700 1 --2- 172.21.15.96:0/2150033090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e741026b0 0x7f1e7410aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.189+0000 7f1e7b653700 1 --2- 172.21.15.96:0/2150033090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e74101d90 0x7f1e74102170 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.189+0000 7f1e7b653700 1 -- 172.21.15.96:0/2150033090 >> 172.21.15.96:0/2150033090 conn(0x7f1e740fb600 msgr2=0x7f1e740fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.189+0000 7f1e7b653700 1 -- 172.21.15.96:0/2150033090 shutdown_connections 2024-04-22T15:16:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.189+0000 7f1e7b653700 1 -- 172.21.15.96:0/2150033090 wait complete. 2024-04-22T15:16:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e7b653700 1 Processor -- start 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e7b653700 1 -- start start 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e7b653700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e74101d90 0x7f1e7419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e7b653700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e741026b0 0x7f1e7419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e7b653700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7419d370 con 0x7f1e741026b0 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e7b653700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e74196950 con 0x7f1e74101d90 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e78bee700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e741026b0 0x7f1e7419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e78bee700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e741026b0 0x7f1e7419cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59812/0 (socket says 172.21.15.96:59812) 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e78bee700 1 -- 172.21.15.96:0/832739033 learned_addr learned my addr 172.21.15.96:0/832739033 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.190+0000 7f1e793ef700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e74101d90 0x7f1e7419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.191+0000 7f1e78bee700 1 -- 172.21.15.96:0/832739033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e74101d90 msgr2=0x7f1e7419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.191+0000 7f1e78bee700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e74101d90 0x7f1e7419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.191+0000 7f1e78bee700 1 -- 172.21.15.96:0/832739033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e700095f0 con 0x7f1e741026b0 2024-04-22T15:16:08.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.191+0000 7f1e78bee700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e741026b0 0x7f1e7419cd50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400d6a0 tx=0x7f1e6400d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:08.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.191+0000 7f1e6a7fc700 1 -- 172.21.15.96:0/832739033 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e640041f0 con 0x7f1e741026b0 2024-04-22T15:16:08.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.191+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e74196c30 con 0x7f1e741026b0 2024-04-22T15:16:08.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.191+0000 7f1e6a7fc700 1 -- 172.21.15.96:0/832739033 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e64007570 con 0x7f1e741026b0 2024-04-22T15:16:08.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.191+0000 7f1e6a7fc700 1 -- 172.21.15.96:0/832739033 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e64003b30 con 0x7f1e741026b0 2024-04-22T15:16:08.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.191+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e74197180 con 0x7f1e741026b0 2024-04-22T15:16:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.192+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e741054b0 con 0x7f1e741026b0 2024-04-22T15:16:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.193+0000 7f1e6a7fc700 1 -- 172.21.15.96:0/832739033 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1e64004350 con 0x7f1e741026b0 2024-04-22T15:16:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.193+0000 7f1e6a7fc700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1e6006cce0 0x7f1e6006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.193+0000 7f1e6a7fc700 1 -- 172.21.15.96:0/832739033 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1e6408c2a0 con 0x7f1e741026b0 2024-04-22T15:16:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.193+0000 7f1e793ef700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1e6006cce0 0x7f1e6006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.194+0000 7f1e793ef700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1e6006cce0 0x7f1e6006f1a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1e70000c00 tx=0x7f1e7001a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:08.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.198+0000 7f1e6a7fc700 1 -- 172.21.15.96:0/832739033 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f1e6405a000 con 0x7f1e741026b0 2024-04-22T15:16:08.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.432+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1e74197e40 con 0x7f1e741026b0 2024-04-22T15:16:08.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.432+0000 7f1e6a7fc700 1 -- 172.21.15.96:0/832739033 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+83 (secure 0 0 0) 0x7f1e64059b90 con 0x7f1e741026b0 2024-04-22T15:16:08.436 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-22T15:16:08.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1e6006cce0 msgr2=0x7f1e6006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:08.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1e6006cce0 0x7f1e6006f1a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1e70000c00 tx=0x7f1e7001a040 comp rx=0 tx=0).stop 2024-04-22T15:16:08.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e741026b0 msgr2=0x7f1e7419cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:08.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e741026b0 0x7f1e7419cd50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400d6a0 tx=0x7f1e6400d9b0 comp rx=0 tx=0).stop 2024-04-22T15:16:08.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 shutdown_connections 2024-04-22T15:16:08.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f1e6006cce0 0x7f1e6006f1a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:08.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e74101d90 0x7f1e7419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:08.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 --2- 172.21.15.96:0/832739033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e741026b0 0x7f1e7419cd50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:08.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 >> 172.21.15.96:0/832739033 conn(0x7f1e740fb600 msgr2=0x7f1e74103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:08.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 shutdown_connections 2024-04-22T15:16:08.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:08.434+0000 7f1e7b653700 1 -- 172.21.15.96:0/832739033 wait complete. 2024-04-22T15:16:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:08 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:16:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:08 smithi096 ceph-mon[26688]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 8.0 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T15:16:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:08 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/832739033' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T15:16:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:08 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:16:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:08 smithi196 ceph-mon[34717]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 8.0 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T15:16:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:08 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/832739033' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T15:16:09.085 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T15:16:09.085 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T15:16:09.085 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi196.front.sepia.ceph.com 2024-04-22T15:16:09.085 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T15:16:09.086 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:09.086 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T15:16:09.086 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T15:16:09.086 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T15:16:09.086 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T15:16:09.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:09.086 DEBUG:teuthology.orchestra.run.smithi196:> ip addr 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout: inet6 ::1/128 scope host 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout: link/ether 0c:c4:7a:d9:38:be brd ff:ff:ff:ff:ff:ff 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout: altname enp7s0f0 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout: link/ether 0c:c4:7a:d9:38:bf brd ff:ff:ff:ff:ff:ff 2024-04-22T15:16:09.106 INFO:teuthology.orchestra.run.smithi196.stdout: altname enp7s0f1 2024-04-22T15:16:09.107 INFO:teuthology.orchestra.run.smithi196.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:16:09.107 INFO:teuthology.orchestra.run.smithi196.stdout: link/ether 0c:c4:7a:88:7f:46 brd ff:ff:ff:ff:ff:ff 2024-04-22T15:16:09.107 INFO:teuthology.orchestra.run.smithi196.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T15:16:09.107 INFO:teuthology.orchestra.run.smithi196.stdout: link/ether 0c:c4:7a:88:7f:47 brd ff:ff:ff:ff:ff:ff 2024-04-22T15:16:09.107 INFO:teuthology.orchestra.run.smithi196.stdout: inet 172.21.15.196/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-22T15:16:09.107 INFO:teuthology.orchestra.run.smithi196.stdout: valid_lft 42032sec preferred_lft 42032sec 2024-04-22T15:16:09.107 INFO:teuthology.orchestra.run.smithi196.stdout: inet6 fe80::9bce:b8e:c505:8c4d/64 scope link noprefixroute 2024-04-22T15:16:09.107 INFO:teuthology.orchestra.run.smithi196.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:16:09.107 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T15:16:09.107 DEBUG:teuthology.orchestra.run.smithi196:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:09.108 DEBUG:teuthology.orchestra.run.smithi196:> set -e 2024-04-22T15:16:09.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip link add name ceph-brx type bridge 2024-04-22T15:16:09.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip addr flush dev ceph-brx 2024-04-22T15:16:09.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip link set ceph-brx up 2024-04-22T15:16:09.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T15:16:09.108 DEBUG:teuthology.orchestra.run.smithi196:> ') 2024-04-22T15:16:09.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:09.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:09.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:09.294 DEBUG:teuthology.orchestra.run.smithi196:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T15:16:09.317 INFO:teuthology.orchestra.run.smithi196.stdout:1 2024-04-22T15:16:09.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:09.318 DEBUG:teuthology.orchestra.run.smithi196:> route 2024-04-22T15:16:09.388 INFO:teuthology.orchestra.run.smithi196.stdout:Kernel IP routing table 2024-04-22T15:16:09.388 INFO:teuthology.orchestra.run.smithi196.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T15:16:09.388 INFO:teuthology.orchestra.run.smithi196.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T15:16:09.388 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T15:16:09.388 INFO:teuthology.orchestra.run.smithi196.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T15:16:09.388 DEBUG:teuthology.orchestra.run.smithi196:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:09.389 DEBUG:teuthology.orchestra.run.smithi196:> set -e 2024-04-22T15:16:09.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T15:16:09.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T15:16:09.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T15:16:09.389 DEBUG:teuthology.orchestra.run.smithi196:> ') 2024-04-22T15:16:09.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:09.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:09.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:09.699 DEBUG:teuthology.orchestra.run.smithi196:> ip netns list 2024-04-22T15:16:09.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:09.714 DEBUG:teuthology.orchestra.run.smithi196:> ip netns list-id 2024-04-22T15:16:09.769 DEBUG:teuthology.orchestra.run.smithi196:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:09.769 DEBUG:teuthology.orchestra.run.smithi196:> set -e 2024-04-22T15:16:09.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:16:09.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-22T15:16:09.770 DEBUG:teuthology.orchestra.run.smithi196:> ') 2024-04-22T15:16:09.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:09.878 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:09.884 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-22T15:16:09.885 DEBUG:teuthology.orchestra.run.smithi196:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:09.885 DEBUG:teuthology.orchestra.run.smithi196:> set -e 2024-04-22T15:16:09.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-22T15:16:09.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T15:16:09.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-22T15:16:09.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-22T15:16:09.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-22T15:16:09.885 DEBUG:teuthology.orchestra.run.smithi196:> ') 2024-04-22T15:16:09.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:09.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:10.006 DEBUG:teuthology.orchestra.run.smithi196:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:16:10.006 DEBUG:teuthology.orchestra.run.smithi196:> set -e 2024-04-22T15:16:10.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip link set brx.0 up 2024-04-22T15:16:10.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T15:16:10.006 DEBUG:teuthology.orchestra.run.smithi196:> ') 2024-04-22T15:16:10.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:16:10.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:10.079 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T15:16:10.080 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T15:16:10.080 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:10.094 INFO:teuthology.orchestra.run.smithi196.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T15:16:10.094 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T15:16:10.094 DEBUG:teuthology.orchestra.run.smithi196:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:10.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo modprobe fuse 2024-04-22T15:16:10.212 DEBUG:teuthology.orchestra.run.smithi196:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T15:16:10.268 INFO:teuthology.orchestra.run.smithi196.stdout:/proc 2024-04-22T15:16:10.268 INFO:teuthology.orchestra.run.smithi196.stdout:/sys 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/dev 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/kernel/security 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/dev/shm 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/dev/pts 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/run 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/cgroup 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/pstore 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/bpf 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/ 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/selinux 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/dev/mqueue 2024-04-22T15:16:10.269 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/kernel/debug 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/dev/hugepages 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/kernel/tracing 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/kernel/config 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/fuse/connections 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/run/user/1000 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/ceph 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/run/user/0 2024-04-22T15:16:10.270 INFO:teuthology.orchestra.run.smithi196.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/963079126c49c176fabd03b719ba7b0ff1f53bcde7cf0580b504b2ff17647062/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/266f27e1db8aca0484bd55afce442b9eb525b2361d87209e65208394fac6708c/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/cfe6df0cc81b6c4d25c80d7bb441faad0c6cb24b12ed3b94f8eddf07ce5d95fc/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/244f11691f11076079fac160865b1a8c658bb844b228fdc3c4a82adf37a1091b/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/a30ad44d251fbac383695de7a01f53a553d8ca27589d21eddcf7a7e6b4d10ad1/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/0aaed5581a5bdce0e46c5cee39eb0f0c30cc7fe300f44c999dcf9f37120764b0/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/0dd6845c51eb3d6bb7a54aacdc9c64eb7650f23c82fb309b0bee5e8030b2885b/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/65c72469d996aa54ea39a592d846cbca6906930d8b52cccdf1b9e52571c41c9d/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/c572a232a1ba6c43b1c9001dbac88b715b8fabc61137c7073bb5b2a38fa9f172/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/9876b797d8beff6d3327b281a24a1437f8d8e4d2d50ce32771c07c0969f4388b/merged 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/run/netns 2024-04-22T15:16:10.271 INFO:teuthology.orchestra.run.smithi196.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:16:10.272 INFO:teuthology.orchestra.run.smithi196.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:16:10.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:10.272 DEBUG:teuthology.orchestra.run.smithi196:> ls /sys/fs/fuse/connections 2024-04-22T15:16:10.323 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T15:16:10.323 DEBUG:teuthology.orchestra.run.smithi196:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T15:16:10.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo modprobe fuse 2024-04-22T15:16:10.388 DEBUG:teuthology.orchestra.run.smithi196:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T15:16:10.444 INFO:teuthology.orchestra.run.smithi196.stdout:/proc 2024-04-22T15:16:10.444 INFO:teuthology.orchestra.run.smithi196.stdout:/sys 2024-04-22T15:16:10.444 INFO:teuthology.orchestra.run.smithi196.stdout:/dev 2024-04-22T15:16:10.444 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/kernel/security 2024-04-22T15:16:10.444 INFO:teuthology.orchestra.run.smithi196.stdout:/dev/shm 2024-04-22T15:16:10.444 INFO:teuthology.orchestra.run.smithi196.stdout:/dev/pts 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/run 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/cgroup 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/pstore 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/bpf 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/ 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/selinux 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/dev/mqueue 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/kernel/debug 2024-04-22T15:16:10.445 INFO:teuthology.orchestra.run.smithi196.stdout:/dev/hugepages 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/kernel/tracing 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/kernel/config 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/sys/fs/fuse/connections 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/run/user/1000 2024-04-22T15:16:10.446 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/ceph 2024-04-22T15:16:10.447 INFO:teuthology.orchestra.run.smithi196.stdout:/run/user/0 2024-04-22T15:16:10.447 INFO:teuthology.orchestra.run.smithi196.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:16:10.447 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay 2024-04-22T15:16:10.447 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/963079126c49c176fabd03b719ba7b0ff1f53bcde7cf0580b504b2ff17647062/merged 2024-04-22T15:16:10.447 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/266f27e1db8aca0484bd55afce442b9eb525b2361d87209e65208394fac6708c/merged 2024-04-22T15:16:10.447 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/cfe6df0cc81b6c4d25c80d7bb441faad0c6cb24b12ed3b94f8eddf07ce5d95fc/merged 2024-04-22T15:16:10.447 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/244f11691f11076079fac160865b1a8c658bb844b228fdc3c4a82adf37a1091b/merged 2024-04-22T15:16:10.447 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/a30ad44d251fbac383695de7a01f53a553d8ca27589d21eddcf7a7e6b4d10ad1/merged 2024-04-22T15:16:10.448 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/0aaed5581a5bdce0e46c5cee39eb0f0c30cc7fe300f44c999dcf9f37120764b0/merged 2024-04-22T15:16:10.448 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/0dd6845c51eb3d6bb7a54aacdc9c64eb7650f23c82fb309b0bee5e8030b2885b/merged 2024-04-22T15:16:10.448 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/65c72469d996aa54ea39a592d846cbca6906930d8b52cccdf1b9e52571c41c9d/merged 2024-04-22T15:16:10.448 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/c572a232a1ba6c43b1c9001dbac88b715b8fabc61137c7073bb5b2a38fa9f172/merged 2024-04-22T15:16:10.448 INFO:teuthology.orchestra.run.smithi196.stdout:/var/lib/containers/storage/overlay/9876b797d8beff6d3327b281a24a1437f8d8e4d2d50ce32771c07c0969f4388b/merged 2024-04-22T15:16:10.448 INFO:teuthology.orchestra.run.smithi196.stdout:/run/netns 2024-04-22T15:16:10.448 INFO:teuthology.orchestra.run.smithi196.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:16:10.448 INFO:teuthology.orchestra.run.smithi196.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:16:10.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:10.448 DEBUG:teuthology.orchestra.run.smithi196:> ls /sys/fs/fuse/connections 2024-04-22T15:16:10.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi196.stderr:2024-04-22T15:16:10.475+0000 7fec3b16e480 -1 init, newargv = 0x5555e6f4ed30 newargc=15 2024-04-22T15:16:10.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi196.stderr:ceph-fuse[60578]: starting ceph client 2024-04-22T15:16:10.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi196.stderr:ceph-fuse[60578]: starting fuse 2024-04-22T15:16:10.493 INFO:teuthology.orchestra.run.smithi196.stdout:123 2024-04-22T15:16:10.494 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [123] 2024-04-22T15:16:10.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo stdin-killer -- python3 -c ' 2024-04-22T15:16:10.494 DEBUG:teuthology.orchestra.run.smithi196:> import glob 2024-04-22T15:16:10.494 DEBUG:teuthology.orchestra.run.smithi196:> import re 2024-04-22T15:16:10.494 DEBUG:teuthology.orchestra.run.smithi196:> import os 2024-04-22T15:16:10.494 DEBUG:teuthology.orchestra.run.smithi196:> import subprocess 2024-04-22T15:16:10.494 DEBUG:teuthology.orchestra.run.smithi196:> 2024-04-22T15:16:10.494 DEBUG:teuthology.orchestra.run.smithi196:> def _find_admin_socket(client_name): 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> files = glob.glob(asok_path) 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> # Given a non-glob path, it better be there 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> if "*" not in asok_path: 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> assert(len(files) == 1) 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> return files[0] 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> for f in files: 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> contents = proc_f.read() 2024-04-22T15:16:10.495 DEBUG:teuthology.orchestra.run.smithi196:> if mountpoint in contents: 2024-04-22T15:16:10.496 DEBUG:teuthology.orchestra.run.smithi196:> return f 2024-04-22T15:16:10.496 DEBUG:teuthology.orchestra.run.smithi196:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T15:16:10.496 DEBUG:teuthology.orchestra.run.smithi196:> 2024-04-22T15:16:10.496 DEBUG:teuthology.orchestra.run.smithi196:> print(_find_admin_socket("client.1")) 2024-04-22T15:16:10.496 DEBUG:teuthology.orchestra.run.smithi196:> ' 2024-04-22T15:16:10.609 INFO:teuthology.orchestra.run.smithi196.stdout:/var/run/ceph/ceph-client.1.60578.asok 2024-04-22T15:16:10.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-04-22T15:16:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:16:10.620 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.60578.asok 2024-04-22T15:16:10.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:10.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.60578.asok status 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: "metadata": { 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_sha1": "9d98999acebbad2663cc4938e94c9ffd83485cbd", 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 18.2.2 (9d98999acebbad2663cc4938e94c9ffd83485cbd) reef (stable)", 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: "entity_id": "1", 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: "hostname": "smithi196", 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: "pid": "60578", 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: "root": "/" 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-04-22T15:16:10.753 INFO:teuthology.orchestra.run.smithi196.stdout: "dentry_count": 0, 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "dentry_pinned_count": 0, 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 24309, 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "inst": { 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "name": { 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client", 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "num": 24309 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "addr": { 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "v1", 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "addr": "192.168.0.1:0", 2024-04-22T15:16:10.754 INFO:teuthology.orchestra.run.smithi196.stdout: "nonce": 310702855 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "addr": { 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "v1", 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "addr": "192.168.0.1:0", 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "nonce": 310702855 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "inst_str": "client.24309 192.168.0.1:0/310702855", 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "addr_str": "192.168.0.1:0/310702855", 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "inode_count": 1, 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_epoch": 20, 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_epoch": 45, 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_epoch_barrier": 0, 2024-04-22T15:16:10.755 INFO:teuthology.orchestra.run.smithi196.stdout: "blocklisted": false, 2024-04-22T15:16:10.756 INFO:teuthology.orchestra.run.smithi196.stdout: "fs_name": "cephfs" 2024-04-22T15:16:10.756 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-04-22T15:16:10.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:10.762 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-04-22T15:16:10.762 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:10.782 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-04-22T15:16:10.782 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:10.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:10.782 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:10.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:10.854 DEBUG:teuthology.orchestra.run.smithi196:> stat --file-system '--printf=%T 2024-04-22T15:16:10.854 DEBUG:teuthology.orchestra.run.smithi196:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:10.869 INFO:teuthology.orchestra.run.smithi196.stdout:fuseblk 2024-04-22T15:16:10.870 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:10.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:16:10.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:10.929 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:10 smithi196 ceph-mon[34717]: pgmap v199: 65 pgs: 65 active+clean; 581 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T15:16:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:10 smithi096 ceph-mon[26688]: pgmap v199: 65 pgs: 65 active+clean; 581 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T15:16:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:12 smithi096 ceph-mon[26688]: pgmap v200: 65 pgs: 65 active+clean; 581 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-22T15:16:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:12 smithi196 ceph-mon[34717]: pgmap v200: 65 pgs: 65 active+clean; 581 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-22T15:16:14.869 INFO:teuthology.run_tasks:Running task print... 2024-04-22T15:16:14.879 INFO:teuthology.task.print:**** done client 2024-04-22T15:16:14.879 INFO:teuthology.run_tasks:Running task parallel... 2024-04-22T15:16:14.900 INFO:teuthology.task.parallel:starting parallel... 2024-04-22T15:16:14.901 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T15:16:14.901 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T15:16:14.902 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:16:14.902 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T15:16:14.902 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T15:16:14.902 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-22T15:16:14.906 INFO:tasks.workunit:Pulling workunits from ref 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:16:14.906 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T15:16:14.906 INFO:tasks.workunit:timeout=3h 2024-04-22T15:16:14.906 INFO:tasks.workunit:cleanup=True 2024-04-22T15:16:14.907 DEBUG:teuthology.orchestra.run.smithi096:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:14.926 INFO:teuthology.orchestra.run.smithi096.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:14.926 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-22T15:16:14.927 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 91h/145d Inode: 1 Links: 2 2024-04-22T15:16:14.927 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:16:14.927 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-22T15:16:14.927 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-22T15:16:14.927 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-22 15:15:38.673082327 +0000 2024-04-22T15:16:14.927 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-22 15:16:10.848725214 +0000 2024-04-22T15:16:14.927 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: - 2024-04-22T15:16:14.927 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T15:16:14.927 DEBUG:teuthology.orchestra.run.smithi096:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T15:16:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:14 smithi096 ceph-mon[26688]: pgmap v201: 65 pgs: 65 active+clean; 582 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T15:16:14.998 DEBUG:teuthology.orchestra.run.smithi196:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:14 smithi196 ceph-mon[34717]: pgmap v201: 65 pgs: 65 active+clean; 582 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T15:16:15.019 INFO:teuthology.orchestra.run.smithi196.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:15.019 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-22T15:16:15.019 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 7bh/123d Inode: 1 Links: 3 2024-04-22T15:16:15.019 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:16:15.019 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-22T15:16:15.020 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-22T15:16:15.020 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-04-22 15:16:14.989252139 +0000 2024-04-22T15:16:15.020 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-04-22 15:16:14.989252139 +0000 2024-04-22T15:16:15.020 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2024-04-22T15:16:15.020 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-22T15:16:15.020 DEBUG:teuthology.orchestra.run.smithi196:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-22T15:16:15.095 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:16:15.096 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:16:15.112 INFO:tasks.workunit.client.0.smithi096.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T15:16:15.129 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:16:15.153 INFO:tasks.workunit.client.1.smithi196.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.700+0000 7f5a291f8700 1 -- 172.21.15.96:0/3662134233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a24101ee0 msgr2=0x7f5a24102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.700+0000 7f5a291f8700 1 --2- 172.21.15.96:0/3662134233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a24101ee0 0x7f5a24102340 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009910 tx=0x7f5a18009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.700+0000 7f5a291f8700 1 -- 172.21.15.96:0/3662134233 shutdown_connections 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.700+0000 7f5a291f8700 1 --2- 172.21.15.96:0/3662134233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a24101ee0 0x7f5a24102340 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.700+0000 7f5a291f8700 1 --2- 172.21.15.96:0/3662134233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a240fef00 0x7f5a240ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.700+0000 7f5a291f8700 1 -- 172.21.15.96:0/3662134233 >> 172.21.15.96:0/3662134233 conn(0x7f5a240fa880 msgr2=0x7f5a240fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.700+0000 7f5a291f8700 1 -- 172.21.15.96:0/3662134233 shutdown_connections 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.700+0000 7f5a291f8700 1 -- 172.21.15.96:0/3662134233 wait complete. 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.701+0000 7f5a291f8700 1 Processor -- start 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.701+0000 7f5a291f8700 1 -- start start 2024-04-22T15:16:16.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.701+0000 7f5a291f8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a240fef00 0x7f5a24196730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.701+0000 7f5a291f8700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a24101ee0 0x7f5a24196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.701+0000 7f5a291f8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a24197290 con 0x7f5a240fef00 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.701+0000 7f5a291f8700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a241973d0 con 0x7f5a24101ee0 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.701+0000 7f5a22d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a240fef00 0x7f5a24196730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a22d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a240fef00 0x7f5a24196730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47956/0 (socket says 172.21.15.96:47956) 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a22d9d700 1 -- 172.21.15.96:0/2104881919 learned_addr learned my addr 172.21.15.96:0/2104881919 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a2259c700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a24101ee0 0x7f5a24196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a22d9d700 1 -- 172.21.15.96:0/2104881919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a24101ee0 msgr2=0x7f5a24196c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a22d9d700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a24101ee0 0x7f5a24196c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a22d9d700 1 -- 172.21.15.96:0/2104881919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180095f0 con 0x7f5a240fef00 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a22d9d700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a240fef00 0x7f5a24196730 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400b480 tx=0x7f5a1400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a0bfff700 1 -- 172.21.15.96:0/2104881919 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a140118c0 con 0x7f5a240fef00 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2419be30 con 0x7f5a240fef00 2024-04-22T15:16:16.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a0bfff700 1 -- 172.21.15.96:0/2104881919 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a14010380 con 0x7f5a240fef00 2024-04-22T15:16:16.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a0bfff700 1 -- 172.21.15.96:0/2104881919 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a1400f4f0 con 0x7f5a240fef00 2024-04-22T15:16:16.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.702+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2419c370 con 0x7f5a240fef00 2024-04-22T15:16:16.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.703+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a24190860 con 0x7f5a240fef00 2024-04-22T15:16:16.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.704+0000 7f5a0bfff700 1 -- 172.21.15.96:0/2104881919 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5a140104f0 con 0x7f5a240fef00 2024-04-22T15:16:16.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.704+0000 7f5a0bfff700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a0c06cce0 0x7f5a0c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:16.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.704+0000 7f5a2259c700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a0c06cce0 0x7f5a0c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:16.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.704+0000 7f5a0bfff700 1 -- 172.21.15.96:0/2104881919 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5a1408bf30 con 0x7f5a240fef00 2024-04-22T15:16:16.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.705+0000 7f5a2259c700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a0c06cce0 0x7f5a0c06f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5a18006010 tx=0x7f5a1800b580 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:16.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.709+0000 7f5a0bfff700 1 -- 172.21.15.96:0/2104881919 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f5a140bfc50 con 0x7f5a240fef00 2024-04-22T15:16:16.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.906+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f5a2404ebb0 con 0x7f5a240fef00 2024-04-22T15:16:16.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.906+0000 7f5a0bfff700 1 -- 172.21.15.96:0/2104881919 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v22)=0 v22) v1 ==== 155+0+0 (secure 0 0 0) 0x7f5a14059d30 con 0x7f5a240fef00 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a0c06cce0 msgr2=0x7f5a0c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a0c06cce0 0x7f5a0c06f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5a18006010 tx=0x7f5a1800b580 comp rx=0 tx=0).stop 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a240fef00 msgr2=0x7f5a24196730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a240fef00 0x7f5a24196730 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400b480 tx=0x7f5a1400b840 comp rx=0 tx=0).stop 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 shutdown_connections 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f5a0c06cce0 0x7f5a0c06f1a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a240fef00 0x7f5a24196730 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 --2- 172.21.15.96:0/2104881919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a24101ee0 0x7f5a24196c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 >> 172.21.15.96:0/2104881919 conn(0x7f5a240fa880 msgr2=0x7f5a241054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:16.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 shutdown_connections 2024-04-22T15:16:16.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:16.908+0000 7f5a291f8700 1 -- 172.21.15.96:0/2104881919 wait complete. 2024-04-22T15:16:16.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:16 smithi096 ceph-mon[26688]: pgmap v202: 65 pgs: 65 active+clean; 582 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T15:16:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:16 smithi196 ceph-mon[34717]: pgmap v202: 65 pgs: 65 active+clean; 582 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T15:16:17.490 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T15:16:17.738 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:16:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:18 smithi196 ceph-mon[34717]: pgmap v203: 65 pgs: 65 active+clean; 582 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T15:16:19.044 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:18 smithi096 ceph-mon[26688]: pgmap v203: 65 pgs: 65 active+clean; 582 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T15:16:19.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.279+0000 7fad34069700 1 -- 172.21.15.96:0/1485795587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 msgr2=0x7fad2c1042a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:19.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.279+0000 7fad34069700 1 --2- 172.21.15.96:0/1485795587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 0x7fad2c1042a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009910 tx=0x7fad1c009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:19.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.279+0000 7fad34069700 1 -- 172.21.15.96:0/1485795587 shutdown_connections 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.279+0000 7fad34069700 1 --2- 172.21.15.96:0/1485795587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 0x7fad2c1042a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.279+0000 7fad34069700 1 --2- 172.21.15.96:0/1485795587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad2c0ff4a0 0x7fad2c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.279+0000 7fad34069700 1 -- 172.21.15.96:0/1485795587 >> 172.21.15.96:0/1485795587 conn(0x7fad2c0fb0e0 msgr2=0x7fad2c0fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.279+0000 7fad34069700 1 -- 172.21.15.96:0/1485795587 shutdown_connections 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.279+0000 7fad34069700 1 -- 172.21.15.96:0/1485795587 wait complete. 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad34069700 1 Processor -- start 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad34069700 1 -- start start 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad34069700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad2c0ff4a0 0x7fad2c19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad34069700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 0x7fad2c19d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad34069700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c19d7a0 con 0x7fad2c101e10 2024-04-22T15:16:19.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad34069700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c1a2100 con 0x7fad2c0ff4a0 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad31604700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 0x7fad2c19d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad31604700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 0x7fad2c19d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47976/0 (socket says 172.21.15.96:47976) 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad31604700 1 -- 172.21.15.96:0/4121955242 learned_addr learned my addr 172.21.15.96:0/4121955242 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad31e05700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad2c0ff4a0 0x7fad2c19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad31604700 1 -- 172.21.15.96:0/4121955242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad2c0ff4a0 msgr2=0x7fad2c19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad31604700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad2c0ff4a0 0x7fad2c19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad31604700 1 -- 172.21.15.96:0/4121955242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0095f0 con 0x7fad2c101e10 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad31e05700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad2c0ff4a0 0x7fad2c19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.280+0000 7fad31604700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 0x7fad2c19d180 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0092e0 tx=0x7fad1c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.281+0000 7fad22ffd700 1 -- 172.21.15.96:0/4121955242 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad1c01d070 con 0x7fad2c101e10 2024-04-22T15:16:19.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.281+0000 7fad22ffd700 1 -- 172.21.15.96:0/4121955242 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad1c00f460 con 0x7fad2c101e10 2024-04-22T15:16:19.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.281+0000 7fad22ffd700 1 -- 172.21.15.96:0/4121955242 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad1c021640 con 0x7fad2c101e10 2024-04-22T15:16:19.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.281+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c1a22a0 con 0x7fad2c101e10 2024-04-22T15:16:19.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.281+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c1a2740 con 0x7fad2c101e10 2024-04-22T15:16:19.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.282+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c04ebb0 con 0x7fad2c101e10 2024-04-22T15:16:19.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.282+0000 7fad22ffd700 1 -- 172.21.15.96:0/4121955242 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fad1c00f5d0 con 0x7fad2c101e10 2024-04-22T15:16:19.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.282+0000 7fad22ffd700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fad1806cce0 0x7fad1806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:19.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.283+0000 7fad22ffd700 1 -- 172.21.15.96:0/4121955242 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fad1c08d630 con 0x7fad2c101e10 2024-04-22T15:16:19.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.283+0000 7fad31e05700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fad1806cce0 0x7fad1806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:19.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.283+0000 7fad31e05700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fad1806cce0 0x7fad1806f1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fad28005fd0 tx=0x7fad28005de0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:19.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.287+0000 7fad22ffd700 1 -- 172.21.15.96:0/4121955242 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fad1c0579a0 con 0x7fad2c101e10 2024-04-22T15:16:19.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.480+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fad2c10ff70 con 0x7fad2c101e10 2024-04-22T15:16:19.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.480+0000 7fad22ffd700 1 -- 172.21.15.96:0/4121955242 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v22)=0 v22) v1 ==== 163+0+0 (secure 0 0 0) 0x7fad1c05afc0 con 0x7fad2c101e10 2024-04-22T15:16:19.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fad1806cce0 msgr2=0x7fad1806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:19.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fad1806cce0 0x7fad1806f1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fad28005fd0 tx=0x7fad28005de0 comp rx=0 tx=0).stop 2024-04-22T15:16:19.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 msgr2=0x7fad2c19d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:19.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 0x7fad2c19d180 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0092e0 tx=0x7fad1c00be20 comp rx=0 tx=0).stop 2024-04-22T15:16:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 shutdown_connections 2024-04-22T15:16:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fad1806cce0 0x7fad1806f1a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad2c0ff4a0 0x7fad2c19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 --2- 172.21.15.96:0/4121955242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad2c101e10 0x7fad2c19d180 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 >> 172.21.15.96:0/4121955242 conn(0x7fad2c0fb0e0 msgr2=0x7fad2c0fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 shutdown_connections 2024-04-22T15:16:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:19.483+0000 7fad34069700 1 -- 172.21.15.96:0/4121955242 wait complete. 2024-04-22T15:16:20.213 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T15:16:20.429 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:16:20.786 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:20 smithi096 ceph-mon[26688]: pgmap v204: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-22T15:16:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:20 smithi196 ceph-mon[34717]: pgmap v204: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-22T15:16:22.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 -- 172.21.15.96:0/526800341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 msgr2=0x7fef340ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:22.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 --2- 172.21.15.96:0/526800341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 0x7fef340ffed0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fef24009910 tx=0x7fef24009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:22.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 -- 172.21.15.96:0/526800341 shutdown_connections 2024-04-22T15:16:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 --2- 172.21.15.96:0/526800341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef34100cb0 0x7fef34101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 --2- 172.21.15.96:0/526800341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 0x7fef340ffed0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 -- 172.21.15.96:0/526800341 >> 172.21.15.96:0/526800341 conn(0x7fef340fb050 msgr2=0x7fef340fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 -- 172.21.15.96:0/526800341 shutdown_connections 2024-04-22T15:16:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 -- 172.21.15.96:0/526800341 wait complete. 2024-04-22T15:16:22.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 Processor -- start 2024-04-22T15:16:22.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.120+0000 7fef38b63700 1 -- start start 2024-04-22T15:16:22.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef38b63700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 0x7fef34069c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:22.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef38b63700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef34100cb0 0x7fef3419d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:22.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef38b63700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef34197100 con 0x7fef340ffab0 2024-04-22T15:16:22.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef38b63700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3419dbd0 con 0x7fef34100cb0 2024-04-22T15:16:22.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef3259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 0x7fef34069c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef3259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 0x7fef34069c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49582/0 (socket says 172.21.15.96:49582) 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef3259c700 1 -- 172.21.15.96:0/224457391 learned_addr learned my addr 172.21.15.96:0/224457391 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef3259c700 1 -- 172.21.15.96:0/224457391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef34100cb0 msgr2=0x7fef3419d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef31d9b700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef34100cb0 0x7fef3419d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef3259c700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef34100cb0 0x7fef3419d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef3259c700 1 -- 172.21.15.96:0/224457391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef240095f0 con 0x7fef340ffab0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef31d9b700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef34100cb0 0x7fef3419d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef3259c700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 0x7fef34069c30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fef2400bd70 tx=0x7fef2400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef237fe700 1 -- 172.21.15.96:0/224457391 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2401d070 con 0x7fef340ffab0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef237fe700 1 -- 172.21.15.96:0/224457391 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef24005dc0 con 0x7fef340ffab0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3419de20 con 0x7fef340ffab0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef237fe700 1 -- 172.21.15.96:0/224457391 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef24021410 con 0x7fef340ffab0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.121+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3419e2b0 con 0x7fef340ffab0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.122+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3404ec20 con 0x7fef340ffab0 2024-04-22T15:16:22.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.123+0000 7fef237fe700 1 -- 172.21.15.96:0/224457391 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fef2400f820 con 0x7fef340ffab0 2024-04-22T15:16:22.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.123+0000 7fef237fe700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fef1c06cce0 0x7fef1c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:22.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.123+0000 7fef31d9b700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fef1c06cce0 0x7fef1c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:22.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.123+0000 7fef237fe700 1 -- 172.21.15.96:0/224457391 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fef2408d530 con 0x7fef340ffab0 2024-04-22T15:16:22.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.124+0000 7fef31d9b700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fef1c06cce0 0x7fef1c06f1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fef28005fd0 tx=0x7fef28005ee0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:22.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.128+0000 7fef237fe700 1 -- 172.21.15.96:0/224457391 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fef3404ec20 con 0x7fef340ffab0 2024-04-22T15:16:22.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.323+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fef3418f630 con 0x7fef340ffab0 2024-04-22T15:16:22.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.324+0000 7fef237fe700 1 -- 172.21.15.96:0/224457391 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7fef24057020 con 0x7fef340ffab0 2024-04-22T15:16:22.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fef1c06cce0 msgr2=0x7fef1c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:22.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fef1c06cce0 0x7fef1c06f1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fef28005fd0 tx=0x7fef28005ee0 comp rx=0 tx=0).stop 2024-04-22T15:16:22.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 msgr2=0x7fef34069c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:22.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 0x7fef34069c30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fef2400bd70 tx=0x7fef2400be50 comp rx=0 tx=0).stop 2024-04-22T15:16:22.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 shutdown_connections 2024-04-22T15:16:22.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fef1c06cce0 0x7fef1c06f1a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:22.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef340ffab0 0x7fef34069c30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:22.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 --2- 172.21.15.96:0/224457391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef34100cb0 0x7fef3419d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:22.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 >> 172.21.15.96:0/224457391 conn(0x7fef340fb050 msgr2=0x7fef341042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:22.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 shutdown_connections 2024-04-22T15:16:22.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:22.326+0000 7fef38b63700 1 -- 172.21.15.96:0/224457391 wait complete. 2024-04-22T15:16:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:22 smithi096 ceph-mon[26688]: pgmap v205: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-22T15:16:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:16:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:22 smithi196 ceph-mon[34717]: pgmap v205: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-22T15:16:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:16:23.026 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-22T15:16:23.237 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:16:24.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.867+0000 7f2b83d62700 1 -- 172.21.15.96:0/2630798708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c0ffb90 msgr2=0x7f2b7c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:24.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.867+0000 7f2b83d62700 1 --2- 172.21.15.96:0/2630798708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c0ffb90 0x7f2b7c0fffb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009910 tx=0x7f2b78009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:24.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.867+0000 7f2b83d62700 1 -- 172.21.15.96:0/2630798708 shutdown_connections 2024-04-22T15:16:24.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.867+0000 7f2b83d62700 1 --2- 172.21.15.96:0/2630798708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b7c100d90 0x7f2b7c1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.867+0000 7f2b83d62700 1 --2- 172.21.15.96:0/2630798708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c0ffb90 0x7f2b7c0fffb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.867+0000 7f2b83d62700 1 -- 172.21.15.96:0/2630798708 >> 172.21.15.96:0/2630798708 conn(0x7f2b7c0fb110 msgr2=0x7f2b7c0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.867+0000 7f2b83d62700 1 -- 172.21.15.96:0/2630798708 shutdown_connections 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.867+0000 7f2b83d62700 1 -- 172.21.15.96:0/2630798708 wait complete. 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b83d62700 1 Processor -- start 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b83d62700 1 -- start start 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b83d62700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b7c0ffb90 0x7f2b7c1966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b83d62700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c100d90 0x7f2b7c196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b83d62700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c197200 con 0x7f2b7c100d90 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b83d62700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c197340 con 0x7f2b7c0ffb90 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b812fd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c100d90 0x7f2b7c196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b812fd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c100d90 0x7f2b7c196be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49594/0 (socket says 172.21.15.96:49594) 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b812fd700 1 -- 172.21.15.96:0/261566034 learned_addr learned my addr 172.21.15.96:0/261566034 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:24.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b81afe700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b7c0ffb90 0x7f2b7c1966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:24.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b812fd700 1 -- 172.21.15.96:0/261566034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b7c0ffb90 msgr2=0x7f2b7c1966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:24.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b812fd700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b7c0ffb90 0x7f2b7c1966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:24.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.868+0000 7f2b812fd700 1 -- 172.21.15.96:0/261566034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b780095f0 con 0x7f2b7c100d90 2024-04-22T15:16:24.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.869+0000 7f2b812fd700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c100d90 0x7f2b7c196be0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00c8a0 tx=0x7f2b6c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:24.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.869+0000 7f2b72ffd700 1 -- 172.21.15.96:0/261566034 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b6c00cea0 con 0x7f2b7c100d90 2024-04-22T15:16:24.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.869+0000 7f2b72ffd700 1 -- 172.21.15.96:0/261566034 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b6c004830 con 0x7f2b7c100d90 2024-04-22T15:16:24.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.869+0000 7f2b72ffd700 1 -- 172.21.15.96:0/261566034 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b6c0055b0 con 0x7f2b7c100d90 2024-04-22T15:16:24.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.869+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7c19bda0 con 0x7f2b7c100d90 2024-04-22T15:16:24.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.869+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7c19c2a0 con 0x7f2b7c100d90 2024-04-22T15:16:24.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.870+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7c190720 con 0x7f2b7c100d90 2024-04-22T15:16:24.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.870+0000 7f2b72ffd700 1 -- 172.21.15.96:0/261566034 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2b6c0049a0 con 0x7f2b7c100d90 2024-04-22T15:16:24.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.871+0000 7f2b72ffd700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2b6806cce0 0x7f2b6806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:24.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.871+0000 7f2b81afe700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2b6806cce0 0x7f2b6806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:24.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.871+0000 7f2b72ffd700 1 -- 172.21.15.96:0/261566034 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2b6c08bf50 con 0x7f2b7c100d90 2024-04-22T15:16:24.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.871+0000 7f2b81afe700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2b6806cce0 0x7f2b6806f1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b78006010 tx=0x7f2b7800b580 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:24.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:24.875+0000 7f2b72ffd700 1 -- 172.21.15.96:0/261566034 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2b6c0bfc50 con 0x7f2b7c100d90 2024-04-22T15:16:24.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:24 smithi096 ceph-mon[26688]: pgmap v206: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-22T15:16:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:24 smithi196 ceph-mon[34717]: pgmap v206: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-22T15:16:25.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:25.088+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b7c02cea0 con 0x7f2b6806cce0 2024-04-22T15:16:26.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:25 smithi096 ceph-mon[26688]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:16:26.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:25 smithi096 ceph-mon[26688]: pgmap v207: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-22T15:16:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:25 smithi196 ceph-mon[34717]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:16:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:25 smithi196 ceph-mon[34717]: pgmap v207: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-22T15:16:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:28 smithi096 ceph-mon[26688]: pgmap v208: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-22T15:16:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:28 smithi196 ceph-mon[34717]: pgmap v208: 65 pgs: 65 active+clean; 587 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-22T15:16:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:30 smithi096 ceph-mon[26688]: pgmap v209: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T15:16:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:30 smithi196 ceph-mon[34717]: pgmap v209: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T15:16:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:32 smithi096 ceph-mon[26688]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-22T15:16:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:32 smithi196 ceph-mon[34717]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-22T15:16:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:34 smithi196 ceph-mon[34717]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T15:16:35.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:34 smithi096 ceph-mon[26688]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T15:16:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:36 smithi196 ceph-mon[34717]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-22T15:16:37.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:36 smithi096 ceph-mon[26688]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-22T15:16:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:16:38.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:16:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:38 smithi196 ceph-mon[34717]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T15:16:39.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:38 smithi096 ceph-mon[26688]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T15:16:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:40 smithi196 ceph-mon[34717]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-22T15:16:41.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:40 smithi096 ceph-mon[26688]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-22T15:16:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:42 smithi196 ceph-mon[34717]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:43.056 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:42 smithi096 ceph-mon[26688]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:44 smithi196 ceph-mon[34717]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:45.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:44 smithi096 ceph-mon[26688]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:46 smithi196 ceph-mon[34717]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:16:47.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:46 smithi096 ceph-mon[26688]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:16:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:48 smithi196 ceph-mon[34717]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:49.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:48 smithi096 ceph-mon[26688]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:50 smithi196 ceph-mon[34717]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:50 smithi096 ceph-mon[26688]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:52 smithi096 ceph-mon[26688]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:16:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:52 smithi196 ceph-mon[34717]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:16:53.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.598+0000 7f2b72ffd700 1 -- 172.21.15.96:0/261566034 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f2b7c02cea0 con 0x7f2b6806cce0 2024-04-22T15:16:53.602 INFO:teuthology.orchestra.run.smithi096.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:16:53.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2b6806cce0 msgr2=0x7f2b6806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:53.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2b6806cce0 0x7f2b6806f1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b78006010 tx=0x7f2b7800b580 comp rx=0 tx=0).stop 2024-04-22T15:16:53.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c100d90 msgr2=0x7f2b7c196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:53.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c100d90 0x7f2b7c196be0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00c8a0 tx=0x7f2b6c00cc60 comp rx=0 tx=0).stop 2024-04-22T15:16:53.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 shutdown_connections 2024-04-22T15:16:53.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2b6806cce0 0x7f2b6806f1a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:53.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b7c0ffb90 0x7f2b7c1966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:53.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 --2- 172.21.15.96:0/261566034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b7c100d90 0x7f2b7c196be0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:53.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 >> 172.21.15.96:0/261566034 conn(0x7f2b7c0fb110 msgr2=0x7f2b7c104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:53.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 shutdown_connections 2024-04-22T15:16:53.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:53.601+0000 7f2b83d62700 1 -- 172.21.15.96:0/261566034 wait complete. 2024-04-22T15:16:54.340 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-22T15:16:54.561 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:16:54.855 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:54 smithi096 ceph-mon[26688]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:16:54.855 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:54.855 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:16:54.856 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:54 smithi096 ceph-mon[26688]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:54.856 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:16:54.856 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:16:54.856 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:54 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:54 smithi196 ceph-mon[34717]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:16:55.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:55.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:16:55.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:54 smithi196 ceph-mon[34717]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:55.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:16:55.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:16:55.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:54 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:55.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:55 smithi096 ceph-mon[26688]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:16:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:55 smithi196 ceph-mon[34717]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:16:56.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.502+0000 7fc047709700 1 -- 172.21.15.96:0/2068567827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 msgr2=0x7fc0400ff2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:56.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.502+0000 7fc047709700 1 --2- 172.21.15.96:0/2068567827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 0x7fc0400ff2f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009910 tx=0x7fc03c009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.502+0000 7fc047709700 1 -- 172.21.15.96:0/2068567827 shutdown_connections 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.502+0000 7fc047709700 1 --2- 172.21.15.96:0/2068567827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040101eb0 0x7fc040102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.502+0000 7fc047709700 1 --2- 172.21.15.96:0/2068567827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 0x7fc0400ff2f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.502+0000 7fc047709700 1 -- 172.21.15.96:0/2068567827 >> 172.21.15.96:0/2068567827 conn(0x7fc0400fa830 msgr2=0x7fc0400fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.502+0000 7fc047709700 1 -- 172.21.15.96:0/2068567827 shutdown_connections 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.502+0000 7fc047709700 1 -- 172.21.15.96:0/2068567827 wait complete. 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc047709700 1 Processor -- start 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc047709700 1 -- start start 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc047709700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 0x7fc040196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc047709700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040101eb0 0x7fc040196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc047709700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0401971f0 con 0x7fc0400feed0 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc047709700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc040197330 con 0x7fc040101eb0 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc0454a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 0x7fc040196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc0454a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 0x7fc040196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60442/0 (socket says 172.21.15.96:60442) 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc0454a5700 1 -- 172.21.15.96:0/1817977046 learned_addr learned my addr 172.21.15.96:0/1817977046 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc044ca4700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040101eb0 0x7fc040196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc0454a5700 1 -- 172.21.15.96:0/1817977046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040101eb0 msgr2=0x7fc040196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc0454a5700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040101eb0 0x7fc040196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.503+0000 7fc0454a5700 1 -- 172.21.15.96:0/1817977046 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c0095f0 con 0x7fc0400feed0 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.504+0000 7fc0454a5700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 0x7fc040196690 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0059d0 tx=0x7fc03c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.504+0000 7fc0367fc700 1 -- 172.21.15.96:0/1817977046 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc03c01d070 con 0x7fc0400feed0 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.504+0000 7fc0367fc700 1 -- 172.21.15.96:0/1817977046 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc03c005dc0 con 0x7fc0400feed0 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.504+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04019bce0 con 0x7fc0400feed0 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.504+0000 7fc0367fc700 1 -- 172.21.15.96:0/1817977046 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc03c021410 con 0x7fc0400feed0 2024-04-22T15:16:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.504+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04019c110 con 0x7fc0400feed0 2024-04-22T15:16:56.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.505+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc040190790 con 0x7fc0400feed0 2024-04-22T15:16:56.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.506+0000 7fc0367fc700 1 -- 172.21.15.96:0/1817977046 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc03c00f460 con 0x7fc0400feed0 2024-04-22T15:16:56.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.506+0000 7fc0367fc700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc02c06cce0 0x7fc02c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:56.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.507+0000 7fc0367fc700 1 -- 172.21.15.96:0/1817977046 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc03c08d4c0 con 0x7fc0400feed0 2024-04-22T15:16:56.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.507+0000 7fc044ca4700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc02c06cce0 0x7fc02c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:56.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.507+0000 7fc044ca4700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc02c06cce0 0x7fc02c06f1a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc030005950 tx=0x7fc03000b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:56.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.515+0000 7fc0367fc700 1 -- 172.21.15.96:0/1817977046 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fc03c05b2c0 con 0x7fc0400feed0 2024-04-22T15:16:56.759 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:56 smithi096 ceph-mon[26688]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:16:56.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.758+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc04002ce80 con 0x7fc02c06cce0 2024-04-22T15:16:56.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.760+0000 7fc0367fc700 1 -- 172.21.15.96:0/1817977046 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fc04002ce80 con 0x7fc02c06cce0 2024-04-22T15:16:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.762+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc02c06cce0 msgr2=0x7fc02c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.762+0000 7fc047709700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc02c06cce0 0x7fc02c06f1a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc030005950 tx=0x7fc03000b410 comp rx=0 tx=0).stop 2024-04-22T15:16:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.762+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 msgr2=0x7fc040196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.762+0000 7fc047709700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 0x7fc040196690 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0059d0 tx=0x7fc03c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:16:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.763+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 shutdown_connections 2024-04-22T15:16:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.763+0000 7fc047709700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc02c06cce0 0x7fc02c06f1a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.763+0000 7fc047709700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400feed0 0x7fc040196690 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:56.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.763+0000 7fc047709700 1 --2- 172.21.15.96:0/1817977046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040101eb0 0x7fc040196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:56.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.763+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 >> 172.21.15.96:0/1817977046 conn(0x7fc0400fa830 msgr2=0x7fc0401054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:56.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.763+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 shutdown_connections 2024-04-22T15:16:56.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.763+0000 7fc047709700 1 -- 172.21.15.96:0/1817977046 wait complete. 2024-04-22T15:16:56.779 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:16:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 -- 172.21.15.96:0/2570161616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 msgr2=0x7efdb8101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 --2- 172.21.15.96:0/2570161616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 0x7efdb8101050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efda8009910 tx=0x7efda8009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 -- 172.21.15.96:0/2570161616 shutdown_connections 2024-04-22T15:16:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 --2- 172.21.15.96:0/2570161616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdb8101e30 0x7efdb8102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 --2- 172.21.15.96:0/2570161616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 0x7efdb8101050 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 -- 172.21.15.96:0/2570161616 >> 172.21.15.96:0/2570161616 conn(0x7efdb80fc1d0 msgr2=0x7efdb80fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 -- 172.21.15.96:0/2570161616 shutdown_connections 2024-04-22T15:16:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 -- 172.21.15.96:0/2570161616 wait complete. 2024-04-22T15:16:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 Processor -- start 2024-04-22T15:16:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.876+0000 7efdbd493700 1 -- start start 2024-04-22T15:16:56.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdbd493700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 0x7efdb8196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:56.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdbd493700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdb8101e30 0x7efdb8196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:56.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdbd493700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81971b0 con 0x7efdb8100c30 2024-04-22T15:16:56.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdbd493700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81972f0 con 0x7efdb8101e30 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdb6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 0x7efdb8196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdb6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 0x7efdb8196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60452/0 (socket says 172.21.15.96:60452) 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdb6ffd700 1 -- 172.21.15.96:0/3568580985 learned_addr learned my addr 172.21.15.96:0/3568580985 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdb67fc700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdb8101e30 0x7efdb8196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdb6ffd700 1 -- 172.21.15.96:0/3568580985 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdb8101e30 msgr2=0x7efdb8196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdb6ffd700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdb8101e30 0x7efdb8196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdb6ffd700 1 -- 172.21.15.96:0/3568580985 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda80095f0 con 0x7efdb8100c30 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdb67fc700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdb8101e30 0x7efdb8196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.877+0000 7efdb6ffd700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 0x7efdb8196650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efda800bd70 tx=0x7efda800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.878+0000 7efd9ffff700 1 -- 172.21.15.96:0/3568580985 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda801d070 con 0x7efdb8100c30 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.878+0000 7efd9ffff700 1 -- 172.21.15.96:0/3568580985 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efda8005dc0 con 0x7efdb8100c30 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.878+0000 7efd9ffff700 1 -- 172.21.15.96:0/3568580985 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda8021410 con 0x7efdb8100c30 2024-04-22T15:16:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.878+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb8073250 con 0x7efdb8100c30 2024-04-22T15:16:56.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.878+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb8073770 con 0x7efdb8100c30 2024-04-22T15:16:56.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.879+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb8075180 con 0x7efdb8100c30 2024-04-22T15:16:56.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.879+0000 7efd9ffff700 1 -- 172.21.15.96:0/3568580985 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efda800f460 con 0x7efdb8100c30 2024-04-22T15:16:56.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.880+0000 7efd9ffff700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7efda006ccf0 0x7efda006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:56.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.880+0000 7efd9ffff700 1 -- 172.21.15.96:0/3568580985 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7efda808d500 con 0x7efdb8100c30 2024-04-22T15:16:56.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.880+0000 7efdb67fc700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7efda006ccf0 0x7efda006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:56.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.881+0000 7efdb67fc700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7efda006ccf0 0x7efda006f1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efdac005950 tx=0x7efdac00b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:56.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:56.885+0000 7efd9ffff700 1 -- 172.21.15.96:0/3568580985 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7efdb8075180 con 0x7efdb8100c30 2024-04-22T15:16:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:56 smithi196 ceph-mon[34717]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:16:57.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.118+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efdb802ce80 con 0x7efda006ccf0 2024-04-22T15:16:57.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.119+0000 7efd9ffff700 1 -- 172.21.15.96:0/3568580985 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7efdb802ce80 con 0x7efda006ccf0 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.121+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7efda006ccf0 msgr2=0x7efda006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.121+0000 7efdbd493700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7efda006ccf0 0x7efda006f1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efdac005950 tx=0x7efdac00b410 comp rx=0 tx=0).stop 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.121+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 msgr2=0x7efdb8196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.121+0000 7efdbd493700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 0x7efdb8196650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efda800bd70 tx=0x7efda800bda0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.121+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 shutdown_connections 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.121+0000 7efdbd493700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7efda006ccf0 0x7efda006f1b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.121+0000 7efdbd493700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdb8100c30 0x7efdb8196650 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.121+0000 7efdbd493700 1 --2- 172.21.15.96:0/3568580985 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdb8101e30 0x7efdb8196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.121+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 >> 172.21.15.96:0/3568580985 conn(0x7efdb80fc1d0 msgr2=0x7efdb8105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:57.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.122+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 shutdown_connections 2024-04-22T15:16:57.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.122+0000 7efdbd493700 1 -- 172.21.15.96:0/3568580985 wait complete. 2024-04-22T15:16:57.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.237+0000 7f4ec1516700 1 -- 172.21.15.96:0/2003135517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ebc0ffb90 msgr2=0x7f4ebc0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.237+0000 7f4ec1516700 1 --2- 172.21.15.96:0/2003135517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ebc0ffb90 0x7f4ebc0fffb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009910 tx=0x7f4eac009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:57.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.237+0000 7f4ec1516700 1 -- 172.21.15.96:0/2003135517 shutdown_connections 2024-04-22T15:16:57.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.237+0000 7f4ec1516700 1 --2- 172.21.15.96:0/2003135517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ebc100d90 0x7f4ebc1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.237+0000 7f4ec1516700 1 --2- 172.21.15.96:0/2003135517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ebc0ffb90 0x7f4ebc0fffb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.237+0000 7f4ec1516700 1 -- 172.21.15.96:0/2003135517 >> 172.21.15.96:0/2003135517 conn(0x7f4ebc0fb110 msgr2=0x7f4ebc0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:57.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.237+0000 7f4ec1516700 1 -- 172.21.15.96:0/2003135517 shutdown_connections 2024-04-22T15:16:57.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.237+0000 7f4ec1516700 1 -- 172.21.15.96:0/2003135517 wait complete. 2024-04-22T15:16:57.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4ec1516700 1 Processor -- start 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4ec1516700 1 -- start start 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4ec1516700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ebc0ffb90 0x7f4ebc196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4ec1516700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ebc100d90 0x7f4ebc196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4ec1516700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc1971e0 con 0x7f4ebc100d90 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4ec1516700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc197320 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4ebaffd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ebc0ffb90 0x7f4ebc196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4ebaffd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ebc0ffb90 0x7f4ebc196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:37242/0 (socket says 172.21.15.96:37242) 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4ebaffd700 1 -- 172.21.15.96:0/3079195644 learned_addr learned my addr 172.21.15.96:0/3079195644 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.238+0000 7f4eba7fc700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ebc100d90 0x7f4ebc196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.239+0000 7f4ebaffd700 1 -- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ebc100d90 msgr2=0x7f4ebc196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.239+0000 7f4ebaffd700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ebc100d90 0x7f4ebc196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.239+0000 7f4ebaffd700 1 -- 172.21.15.96:0/3079195644 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac0095f0 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.239+0000 7f4ebaffd700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ebc0ffb90 0x7f4ebc196680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4eac005f00 tx=0x7f4eac00b5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:57.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.239+0000 7f4ea3fff700 1 -- 172.21.15.96:0/3079195644 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eac01d070 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.239+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc19bd20 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.239+0000 7f4ea3fff700 1 -- 172.21.15.96:0/3079195644 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4eac0052a0 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.239+0000 7f4ea3fff700 1 -- 172.21.15.96:0/3079195644 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eac017400 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.239+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc19c1c0 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.240+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc04ebb0 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.241+0000 7f4ea3fff700 1 -- 172.21.15.96:0/3079195644 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4eac00f820 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.241+0000 7f4ea3fff700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4ea406cce0 0x7f4ea406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:57.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.241+0000 7f4ea3fff700 1 -- 172.21.15.96:0/3079195644 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4eac08d6e0 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.241+0000 7f4eba7fc700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4ea406cce0 0x7f4ea406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:57.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.242+0000 7f4eba7fc700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4ea406cce0 0x7f4ea406f1a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009c60 tx=0x7f4eb0009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:57.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.246+0000 7f4ea3fff700 1 -- 172.21.15.96:0/3079195644 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4eac05b590 con 0x7f4ebc0ffb90 2024-04-22T15:16:57.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.464+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ebc0fe520 con 0x7f4ea406cce0 2024-04-22T15:16:57.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.476+0000 7f4ea3fff700 1 -- 172.21.15.96:0/3079195644 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4ebc0fe520 con 0x7f4ea406cce0 2024-04-22T15:16:57.479 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:16:57.479 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (5m) 52s ago 6m 24.7M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:16:57.479 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (7m) 52s ago 7m 7541k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:16:57.479 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (6m) 57s ago 6m 7474k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:16:57.479 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (7m) 52s ago 7m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:16:57.479 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (6m) 57s ago 6m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:16:57.479 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (5m) 52s ago 6m 84.1M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:16:57.479 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (73s) 52s ago 72s 18.7M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (80s) 52s ago 79s 13.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (77s) 57s ago 77s 16.8M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (70s) 57s ago 69s 14.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:9283,8765,8443 running (8m) 52s ago 8m 498M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 12e0357210ef 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (5m) 57s ago 5m 450M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 ea53e28b40cc 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (8m) 52s ago 8m 44.3M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (5m) 57s ago 5m 35.2M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (7m) 52s ago 7m 14.6M - 1.5.0 0da6a335fe13 852e13a0c299 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (5m) 57s ago 5m 16.2M - 1.5.0 0da6a335fe13 4d02817c7cd0 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (5m) 52s ago 5m 47.5M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (4m) 52s ago 4m 48.9M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (4m) 52s ago 4m 53.4M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (3m) 57s ago 3m 47.2M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:16:57.480 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (3m) 57s ago 3m 43.9M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:16:57.481 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (2m) 57s ago 2m 44.4M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:16:57.481 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (5m) 52s ago 6m 55.2M - 2.43.0 a07b618ecd1d 87457032ed08 2024-04-22T15:16:57.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4ea406cce0 msgr2=0x7f4ea406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4ea406cce0 0x7f4ea406f1a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009c60 tx=0x7f4eb0009260 comp rx=0 tx=0).stop 2024-04-22T15:16:57.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ebc0ffb90 msgr2=0x7f4ebc196680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ebc0ffb90 0x7f4ebc196680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4eac005f00 tx=0x7f4eac00b5e0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 shutdown_connections 2024-04-22T15:16:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4ea406cce0 0x7f4ea406f1a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ebc0ffb90 0x7f4ebc196680 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 --2- 172.21.15.96:0/3079195644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ebc100d90 0x7f4ebc196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 >> 172.21.15.96:0/3079195644 conn(0x7f4ebc0fb110 msgr2=0x7f4ebc0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 shutdown_connections 2024-04-22T15:16:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.478+0000 7f4ec1516700 1 -- 172.21.15.96:0/3079195644 wait complete. 2024-04-22T15:16:57.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.594+0000 7fc84697f700 1 -- 172.21.15.96:0/219620644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8400fecb0 msgr2=0x7fc8400ff110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.594+0000 7fc84697f700 1 --2- 172.21.15.96:0/219620644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8400fecb0 0x7fc8400ff110 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc834009910 tx=0x7fc834009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:57.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.594+0000 7fc84697f700 1 -- 172.21.15.96:0/219620644 shutdown_connections 2024-04-22T15:16:57.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.594+0000 7fc84697f700 1 --2- 172.21.15.96:0/219620644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8400fecb0 0x7fc8400ff110 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.594+0000 7fc84697f700 1 --2- 172.21.15.96:0/219620644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc840102230 0x7fc840102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.594+0000 7fc84697f700 1 -- 172.21.15.96:0/219620644 >> 172.21.15.96:0/219620644 conn(0x7fc8400fa890 msgr2=0x7fc8400fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:57.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.594+0000 7fc84697f700 1 -- 172.21.15.96:0/219620644 shutdown_connections 2024-04-22T15:16:57.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.594+0000 7fc84697f700 1 -- 172.21.15.96:0/219620644 wait complete. 2024-04-22T15:16:57.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.595+0000 7fc84697f700 1 Processor -- start 2024-04-22T15:16:57.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.595+0000 7fc84697f700 1 -- start start 2024-04-22T15:16:57.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.595+0000 7fc84697f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8400fecb0 0x7fc840196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:57.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.595+0000 7fc84697f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc840102230 0x7fc840196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:57.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.595+0000 7fc84697f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc840197180 con 0x7fc8400fecb0 2024-04-22T15:16:57.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.595+0000 7fc84697f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8401972c0 con 0x7fc840102230 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.595+0000 7fc83ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8400fecb0 0x7fc840196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.595+0000 7fc83ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8400fecb0 0x7fc840196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60482/0 (socket says 172.21.15.96:60482) 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83ffff700 1 -- 172.21.15.96:0/1390348574 learned_addr learned my addr 172.21.15.96:0/1390348574 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83f7fe700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc840102230 0x7fc840196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83ffff700 1 -- 172.21.15.96:0/1390348574 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc840102230 msgr2=0x7fc840196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83ffff700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc840102230 0x7fc840196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83ffff700 1 -- 172.21.15.96:0/1390348574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8340095f0 con 0x7fc8400fecb0 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83f7fe700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc840102230 0x7fc840196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83ffff700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8400fecb0 0x7fc840196620 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc83000e8f0 tx=0x7fc83000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:57.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83d7fa700 1 -- 172.21.15.96:0/1390348574 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc830009790 con 0x7fc8400fecb0 2024-04-22T15:16:57.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83d7fa700 1 -- 172.21.15.96:0/1390348574 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc83000c810 con 0x7fc8400fecb0 2024-04-22T15:16:57.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc83d7fa700 1 -- 172.21.15.96:0/1390348574 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc830003d20 con 0x7fc8400fecb0 2024-04-22T15:16:57.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8400ffdc0 con 0x7fc8400fecb0 2024-04-22T15:16:57.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.596+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc840100390 con 0x7fc8400fecb0 2024-04-22T15:16:57.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.597+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84004ebb0 con 0x7fc8400fecb0 2024-04-22T15:16:57.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.598+0000 7fc83d7fa700 1 -- 172.21.15.96:0/1390348574 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc8300041f0 con 0x7fc8400fecb0 2024-04-22T15:16:57.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.599+0000 7fc83d7fa700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc8280710f0 0x7fc8280735b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:57.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.599+0000 7fc83d7fa700 1 -- 172.21.15.96:0/1390348574 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc830014070 con 0x7fc8400fecb0 2024-04-22T15:16:57.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.599+0000 7fc83f7fe700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc8280710f0 0x7fc8280735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:57.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.599+0000 7fc83f7fe700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc8280710f0 0x7fc8280735b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc83400b800 tx=0x7fc834009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:57.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.603+0000 7fc83d7fa700 1 -- 172.21.15.96:0/1390348574 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7fc83005a320 con 0x7fc8400fecb0 2024-04-22T15:16:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:57 smithi096 ceph-mon[26688]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:16:57.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.907+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc84002ce40 con 0x7fc8400fecb0 2024-04-22T15:16:57.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.908+0000 7fc83d7fa700 1 -- 172.21.15.96:0/1390348574 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc830059eb0 con 0x7fc8400fecb0 2024-04-22T15:16:57.911 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:16:57.911 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:16:57.911 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 14 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:16:57.912 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:16:57.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc8280710f0 msgr2=0x7fc8280735b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc8280710f0 0x7fc8280735b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc83400b800 tx=0x7fc834009f70 comp rx=0 tx=0).stop 2024-04-22T15:16:57.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8400fecb0 msgr2=0x7fc840196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8400fecb0 0x7fc840196620 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc83000e8f0 tx=0x7fc83000ec00 comp rx=0 tx=0).stop 2024-04-22T15:16:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 shutdown_connections 2024-04-22T15:16:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7fc8280710f0 0x7fc8280735b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8400fecb0 0x7fc840196620 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 --2- 172.21.15.96:0/1390348574 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc840102230 0x7fc840196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 >> 172.21.15.96:0/1390348574 conn(0x7fc8400fa890 msgr2=0x7fc840105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 shutdown_connections 2024-04-22T15:16:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:57.910+0000 7fc84697f700 1 -- 172.21.15.96:0/1390348574 wait complete. 2024-04-22T15:16:58.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:57 smithi196 ceph-mon[34717]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.027+0000 7f875d556700 1 -- 172.21.15.96:0/3031567148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758101750 msgr2=0x7f8758103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.027+0000 7f875d556700 1 --2- 172.21.15.96:0/3031567148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758101750 0x7f8758103b40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8748009910 tx=0x7f8748009c20 comp rx=0 tx=0).stop 2024-04-22T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.027+0000 7f875d556700 1 -- 172.21.15.96:0/3031567148 shutdown_connections 2024-04-22T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.027+0000 7f875d556700 1 --2- 172.21.15.96:0/3031567148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8758104080 0x7f8758106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.027+0000 7f875d556700 1 --2- 172.21.15.96:0/3031567148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758101750 0x7f8758103b40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.027+0000 7f875d556700 1 -- 172.21.15.96:0/3031567148 >> 172.21.15.96:0/3031567148 conn(0x7f87580fb0e0 msgr2=0x7f87580fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.027+0000 7f875d556700 1 -- 172.21.15.96:0/3031567148 shutdown_connections 2024-04-22T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.027+0000 7f875d556700 1 -- 172.21.15.96:0/3031567148 wait complete. 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.027+0000 7f875d556700 1 Processor -- start 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f875d556700 1 -- start start 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f875d556700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8758101750 0x7f87581965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f875d556700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758104080 0x7f8758196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f875d556700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758197110 con 0x7f8758104080 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f875d556700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758197250 con 0x7f8758101750 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f87567fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758104080 0x7f8758196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f8756ffd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8758101750 0x7f87581965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f87567fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758104080 0x7f8758196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60498/0 (socket says 172.21.15.96:60498) 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f87567fc700 1 -- 172.21.15.96:0/2762390144 learned_addr learned my addr 172.21.15.96:0/2762390144 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f8756ffd700 1 -- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758104080 msgr2=0x7f8758196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f8756ffd700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758104080 0x7f8758196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f8756ffd700 1 -- 172.21.15.96:0/2762390144 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87480095f0 con 0x7f8758101750 2024-04-22T15:16:58.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f87567fc700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758104080 0x7f8758196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:16:58.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.028+0000 7f8756ffd700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8758101750 0x7f87581965b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f87480059d0 tx=0x7f874800bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:58.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.029+0000 7f873ffff700 1 -- 172.21.15.96:0/2762390144 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f874801d070 con 0x7f8758101750 2024-04-22T15:16:58.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.029+0000 7f873ffff700 1 -- 172.21.15.96:0/2762390144 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f874800f460 con 0x7f8758101750 2024-04-22T15:16:58.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.029+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875819bc50 con 0x7f8758101750 2024-04-22T15:16:58.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.029+0000 7f873ffff700 1 -- 172.21.15.96:0/2762390144 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8748021640 con 0x7f8758101750 2024-04-22T15:16:58.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.029+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875819c140 con 0x7f8758101750 2024-04-22T15:16:58.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.030+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875804ebb0 con 0x7f8758101750 2024-04-22T15:16:58.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.030+0000 7f873ffff700 1 -- 172.21.15.96:0/2762390144 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f874800f5d0 con 0x7f8758101750 2024-04-22T15:16:58.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.031+0000 7f873ffff700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f874006cce0 0x7f874006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:16:58.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.031+0000 7f873ffff700 1 -- 172.21.15.96:0/2762390144 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f874808d4a0 con 0x7f8758101750 2024-04-22T15:16:58.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.031+0000 7f87567fc700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f874006cce0 0x7f874006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:16:58.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.031+0000 7f87567fc700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f874006cce0 0x7f874006f1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f874c005950 tx=0x7f874c00b500 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:16:58.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.035+0000 7f873ffff700 1 -- 172.21.15.96:0/2762390144 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f874805b240 con 0x7f8758101750 2024-04-22T15:16:58.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.262+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8758105f10 con 0x7f874006cce0 2024-04-22T15:16:58.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.263+0000 7f873ffff700 1 -- 172.21.15.96:0/2762390144 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f8758105f10 con 0x7f874006cce0 2024-04-22T15:16:58.266 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:16:58.266 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:16:58.266 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:16:58.266 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:16:58.266 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [], 2024-04-22T15:16:58.266 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "0/2 daemons upgraded", 2024-04-22T15:16:58.266 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb image on host smithi196", 2024-04-22T15:16:58.267 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:16:58.267 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:16:58.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f874006cce0 msgr2=0x7f874006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f874006cce0 0x7f874006f1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f874c005950 tx=0x7f874c00b500 comp rx=0 tx=0).stop 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8758101750 msgr2=0x7f87581965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8758101750 0x7f87581965b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f87480059d0 tx=0x7f874800bd70 comp rx=0 tx=0).stop 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 shutdown_connections 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f874006cce0 0x7f874006f1a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8758101750 0x7f87581965b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 --2- 172.21.15.96:0/2762390144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8758104080 0x7f8758196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 >> 172.21.15.96:0/2762390144 conn(0x7f87580fb0e0 msgr2=0x7f87580fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 shutdown_connections 2024-04-22T15:16:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:16:58.265+0000 7f875d556700 1 -- 172.21.15.96:0/2762390144 wait complete. 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: from='client.24327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: Upgrade: Target is version 19.0.0-3223-g5883d494 (unknown) 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb'] 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: Upgrade: Need to upgrade myself (mgr.smithi096.hhkvum) 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:58 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1390348574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:16:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:16:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: from='client.24327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:16:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: Upgrade: Target is version 19.0.0-3223-g5883d494 (unknown) 2024-04-22T15:16:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb'] 2024-04-22T15:16:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:16:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:16:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: Upgrade: Need to upgrade myself (mgr.smithi096.hhkvum) 2024-04-22T15:16:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:16:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:58 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1390348574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:16:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:59 smithi096 ceph-mon[26688]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb on smithi196 2024-04-22T15:16:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:16:59 smithi096 ceph-mon[26688]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:59 smithi196 ceph-mon[34717]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb on smithi196 2024-04-22T15:17:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:16:59 smithi196 ceph-mon[34717]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:00 smithi096 ceph-mon[26688]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:00 smithi196 ceph-mon[34717]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:02.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:02 smithi096 ceph-mon[26688]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:02 smithi196 ceph-mon[34717]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:04 smithi196 ceph-mon[34717]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:05.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:04 smithi096 ceph-mon[26688]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:06 smithi196 ceph-mon[34717]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:17:07.055 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:06 smithi096 ceph-mon[26688]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:17:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:07 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:17:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:07 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:17:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:08 smithi096 ceph-mon[26688]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:08 smithi196 ceph-mon[34717]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:10 smithi196 ceph-mon[34717]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:10 smithi096 ceph-mon[26688]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:12 smithi196 ceph-mon[34717]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:13.057 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:12 smithi096 ceph-mon[26688]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:14 smithi196 ceph-mon[34717]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:15.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:14 smithi096 ceph-mon[26688]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:17.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:16 smithi096 ceph-mon[26688]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:17:17.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:16 smithi196 ceph-mon[34717]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:17:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:18 smithi196 ceph-mon[34717]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:19.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:18 smithi096 ceph-mon[26688]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:21.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:20 smithi096 ceph-mon[26688]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:21.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:20 smithi196 ceph-mon[34717]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:23.056 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:22 smithi096 ceph-mon[26688]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:23.057 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:22 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:17:23.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:22 smithi196 ceph-mon[34717]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:23.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:22 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:17:25.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:24 smithi096 ceph-mon[26688]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:25.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:24 smithi196 ceph-mon[34717]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:27.037 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:26 smithi096 ceph-mon[26688]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:17:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:26 smithi196 ceph-mon[34717]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:17:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.377+0000 7f2cebd5a700 1 -- 172.21.15.96:0/951668989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ce4101e40 msgr2=0x7f2ce41022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.377+0000 7f2cebd5a700 1 --2- 172.21.15.96:0/951668989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ce4101e40 0x7f2ce41022a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009910 tx=0x7f2cd4009c20 comp rx=0 tx=0).stop 2024-04-22T15:17:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.377+0000 7f2cebd5a700 1 -- 172.21.15.96:0/951668989 shutdown_connections 2024-04-22T15:17:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.377+0000 7f2cebd5a700 1 --2- 172.21.15.96:0/951668989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ce4101e40 0x7f2ce41022a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.377+0000 7f2cebd5a700 1 --2- 172.21.15.96:0/951668989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ce4100c40 0x7f2ce4101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.377+0000 7f2cebd5a700 1 -- 172.21.15.96:0/951668989 >> 172.21.15.96:0/951668989 conn(0x7f2ce40fc200 msgr2=0x7f2ce40fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.377+0000 7f2cebd5a700 1 -- 172.21.15.96:0/951668989 shutdown_connections 2024-04-22T15:17:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.377+0000 7f2cebd5a700 1 -- 172.21.15.96:0/951668989 wait complete. 2024-04-22T15:17:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.378+0000 7f2cebd5a700 1 Processor -- start 2024-04-22T15:17:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.378+0000 7f2cebd5a700 1 -- start start 2024-04-22T15:17:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.378+0000 7f2cebd5a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ce4100c40 0x7f2ce4071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.378+0000 7f2cebd5a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ce4101e40 0x7f2ce4072290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.378+0000 7f2cebd5a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce40727d0 con 0x7f2ce4101e40 2024-04-22T15:17:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.378+0000 7f2cebd5a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce4072940 con 0x7f2ce4100c40 2024-04-22T15:17:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.379+0000 7f2ce9af6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ce4100c40 0x7f2ce4071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.379+0000 7f2ce9af6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ce4100c40 0x7f2ce4071d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:43472/0 (socket says 172.21.15.96:43472) 2024-04-22T15:17:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.379+0000 7f2ce9af6700 1 -- 172.21.15.96:0/2793363433 learned_addr learned my addr 172.21.15.96:0/2793363433 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:17:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.379+0000 7f2ce9af6700 1 -- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ce4101e40 msgr2=0x7f2ce4072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.379+0000 7f2ce92f5700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ce4101e40 0x7f2ce4072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.379+0000 7f2ce9af6700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ce4101e40 0x7f2ce4072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.379+0000 7f2ce9af6700 1 -- 172.21.15.96:0/2793363433 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd40095f0 con 0x7f2ce4100c40 2024-04-22T15:17:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.379+0000 7f2ce92f5700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ce4101e40 0x7f2ce4072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:17:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.380+0000 7f2ce9af6700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ce4100c40 0x7f2ce4071d50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000e8f0 tx=0x7f2ce000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.380+0000 7f2cdaffd700 1 -- 172.21.15.96:0/2793363433 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ce0009790 con 0x7f2ce4100c40 2024-04-22T15:17:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.380+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce41a2160 con 0x7f2ce4100c40 2024-04-22T15:17:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.380+0000 7f2cdaffd700 1 -- 172.21.15.96:0/2793363433 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ce00044e0 con 0x7f2ce4100c40 2024-04-22T15:17:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.380+0000 7f2cdaffd700 1 -- 172.21.15.96:0/2793363433 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ce0003e00 con 0x7f2ce4100c40 2024-04-22T15:17:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.380+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce41a26e0 con 0x7f2ce4100c40 2024-04-22T15:17:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.381+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc8005320 con 0x7f2ce4100c40 2024-04-22T15:17:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.382+0000 7f2cdaffd700 1 -- 172.21.15.96:0/2793363433 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2ce0004020 con 0x7f2ce4100c40 2024-04-22T15:17:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.382+0000 7f2cdaffd700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2cd006cce0 0x7f2cd006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.382+0000 7f2cdaffd700 1 -- 172.21.15.96:0/2793363433 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2ce0014070 con 0x7f2ce4100c40 2024-04-22T15:17:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.382+0000 7f2ce92f5700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2cd006cce0 0x7f2cd006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.383+0000 7f2ce92f5700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2cd006cce0 0x7f2cd006f1a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009fd0 tx=0x7f2cd4005fb0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:28.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.391+0000 7f2cdaffd700 1 -- 172.21.15.96:0/2793363433 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f2ce005a410 con 0x7f2ce4100c40 2024-04-22T15:17:28.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.620+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cc8000bf0 con 0x7f2cd006cce0 2024-04-22T15:17:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.622+0000 7f2cdaffd700 1 -- 172.21.15.96:0/2793363433 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f2cc8000bf0 con 0x7f2cd006cce0 2024-04-22T15:17:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.624+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2cd006cce0 msgr2=0x7f2cd006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.624+0000 7f2cebd5a700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2cd006cce0 0x7f2cd006f1a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009fd0 tx=0x7f2cd4005fb0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.624+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ce4100c40 msgr2=0x7f2ce4071d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.624+0000 7f2cebd5a700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ce4100c40 0x7f2ce4071d50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000e8f0 tx=0x7f2ce000ecb0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.625+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 shutdown_connections 2024-04-22T15:17:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.625+0000 7f2cebd5a700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f2cd006cce0 0x7f2cd006f1a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.625+0000 7f2cebd5a700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ce4100c40 0x7f2ce4071d50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.625+0000 7f2cebd5a700 1 --2- 172.21.15.96:0/2793363433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ce4101e40 0x7f2ce4072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.625+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 >> 172.21.15.96:0/2793363433 conn(0x7f2ce40fc200 msgr2=0x7f2ce41052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.625+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 shutdown_connections 2024-04-22T15:17:28.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.625+0000 7f2cebd5a700 1 -- 172.21.15.96:0/2793363433 wait complete. 2024-04-22T15:17:28.644 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:17:28.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.746+0000 7f4a8aa09700 1 -- 172.21.15.96:0/4292711368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 msgr2=0x7f4a840fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:28.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.746+0000 7f4a8aa09700 1 --2- 172.21.15.96:0/4292711368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 0x7f4a840fff30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009910 tx=0x7f4a74009c20 comp rx=0 tx=0).stop 2024-04-22T15:17:28.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.746+0000 7f4a8aa09700 1 -- 172.21.15.96:0/4292711368 shutdown_connections 2024-04-22T15:17:28.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.746+0000 7f4a8aa09700 1 --2- 172.21.15.96:0/4292711368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a84100d10 0x7f4a84101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.746+0000 7f4a8aa09700 1 --2- 172.21.15.96:0/4292711368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 0x7f4a840fff30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.746+0000 7f4a8aa09700 1 -- 172.21.15.96:0/4292711368 >> 172.21.15.96:0/4292711368 conn(0x7f4a840fb0b0 msgr2=0x7f4a840fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:28.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.746+0000 7f4a8aa09700 1 -- 172.21.15.96:0/4292711368 shutdown_connections 2024-04-22T15:17:28.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.746+0000 7f4a8aa09700 1 -- 172.21.15.96:0/4292711368 wait complete. 2024-04-22T15:17:28.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.747+0000 7f4a8aa09700 1 Processor -- start 2024-04-22T15:17:28.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.747+0000 7f4a8aa09700 1 -- start start 2024-04-22T15:17:28.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.748+0000 7f4a8aa09700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 0x7f4a841943c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:28.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.748+0000 7f4a8aa09700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a84100d10 0x7f4a84194900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:28.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.748+0000 7f4a83fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 0x7f4a841943c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:28.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.748+0000 7f4a83fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 0x7f4a841943c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59528/0 (socket says 172.21.15.96:59528) 2024-04-22T15:17:28.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.748+0000 7f4a83fff700 1 -- 172.21.15.96:0/2235757864 learned_addr learned my addr 172.21.15.96:0/2235757864 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:17:28.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.748+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84194f20 con 0x7f4a840ffb10 2024-04-22T15:17:28.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.748+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84195060 con 0x7f4a84100d10 2024-04-22T15:17:28.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.749+0000 7f4a837fe700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a84100d10 0x7f4a84194900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:28.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.749+0000 7f4a837fe700 1 -- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 msgr2=0x7f4a841943c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:28.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.749+0000 7f4a837fe700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 0x7f4a841943c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:28.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.749+0000 7f4a837fe700 1 -- 172.21.15.96:0/2235757864 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a740095f0 con 0x7f4a84100d10 2024-04-22T15:17:28.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.749+0000 7f4a83fff700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 0x7f4a841943c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:17:28.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.749+0000 7f4a837fe700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a84100d10 0x7f4a84194900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800e970 tx=0x7f4a7800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:28.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.749+0000 7f4a817fa700 1 -- 172.21.15.96:0/2235757864 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a7800c9d0 con 0x7f4a84100d10 2024-04-22T15:17:28.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.749+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a84199a70 con 0x7f4a84100d10 2024-04-22T15:17:28.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.750+0000 7f4a817fa700 1 -- 172.21.15.96:0/2235757864 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a78010430 con 0x7f4a84100d10 2024-04-22T15:17:28.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.750+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8419a040 con 0x7f4a84100d10 2024-04-22T15:17:28.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.750+0000 7f4a817fa700 1 -- 172.21.15.96:0/2235757864 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a78018630 con 0x7f4a84100d10 2024-04-22T15:17:28.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.751+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a70005320 con 0x7f4a84100d10 2024-04-22T15:17:28.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.752+0000 7f4a817fa700 1 -- 172.21.15.96:0/2235757864 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4a7800cb30 con 0x7f4a84100d10 2024-04-22T15:17:28.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.752+0000 7f4a817fa700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4a6c06ef50 0x7f4a6c071410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:28.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.752+0000 7f4a817fa700 1 -- 172.21.15.96:0/2235757864 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4a78014070 con 0x7f4a84100d10 2024-04-22T15:17:28.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.753+0000 7f4a83fff700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4a6c06ef50 0x7f4a6c071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:28.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.753+0000 7f4a83fff700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4a6c06ef50 0x7f4a6c071410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4a74006010 tx=0x7f4a7400b770 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:28.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:28.760+0000 7f4a817fa700 1 -- 172.21.15.96:0/2235757864 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4a7805a1b0 con 0x7f4a84100d10 2024-04-22T15:17:28.943 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:28 smithi096 ceph-mon[26688]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:29.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.019+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a70000bf0 con 0x7f4a6c06ef50 2024-04-22T15:17:29.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.021+0000 7f4a817fa700 1 -- 172.21.15.96:0/2235757864 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f4a70000bf0 con 0x7f4a6c06ef50 2024-04-22T15:17:29.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.023+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4a6c06ef50 msgr2=0x7f4a6c071410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:29.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.023+0000 7f4a8aa09700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4a6c06ef50 0x7f4a6c071410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4a74006010 tx=0x7f4a7400b770 comp rx=0 tx=0).stop 2024-04-22T15:17:29.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.023+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a84100d10 msgr2=0x7f4a84194900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:29.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.023+0000 7f4a8aa09700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a84100d10 0x7f4a84194900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800e970 tx=0x7f4a7800ec80 comp rx=0 tx=0).stop 2024-04-22T15:17:29.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.023+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 shutdown_connections 2024-04-22T15:17:29.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.023+0000 7f4a8aa09700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4a6c06ef50 0x7f4a6c071410 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.023+0000 7f4a8aa09700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ffb10 0x7f4a841943c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.023+0000 7f4a8aa09700 1 --2- 172.21.15.96:0/2235757864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a84100d10 0x7f4a84194900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.023+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 >> 172.21.15.96:0/2235757864 conn(0x7f4a840fb0b0 msgr2=0x7f4a84104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:29.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.024+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 shutdown_connections 2024-04-22T15:17:29.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.024+0000 7f4a8aa09700 1 -- 172.21.15.96:0/2235757864 wait complete. 2024-04-22T15:17:29.106 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:28 smithi196 ceph-mon[34717]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.176+0000 7ff950092700 1 -- 172.21.15.96:0/3117391845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9480ffb10 msgr2=0x7ff9480fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.176+0000 7ff950092700 1 --2- 172.21.15.96:0/3117391845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9480ffb10 0x7ff9480fff30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff944009910 tx=0x7ff944009c20 comp rx=0 tx=0).stop 2024-04-22T15:17:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.176+0000 7ff950092700 1 -- 172.21.15.96:0/3117391845 shutdown_connections 2024-04-22T15:17:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.176+0000 7ff950092700 1 --2- 172.21.15.96:0/3117391845 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff948100d10 0x7ff948101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.176+0000 7ff950092700 1 --2- 172.21.15.96:0/3117391845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9480ffb10 0x7ff9480fff30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.176+0000 7ff950092700 1 -- 172.21.15.96:0/3117391845 >> 172.21.15.96:0/3117391845 conn(0x7ff9480fb0b0 msgr2=0x7ff9480fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:29.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.176+0000 7ff950092700 1 -- 172.21.15.96:0/3117391845 shutdown_connections 2024-04-22T15:17:29.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.176+0000 7ff950092700 1 -- 172.21.15.96:0/3117391845 wait complete. 2024-04-22T15:17:29.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.177+0000 7ff950092700 1 Processor -- start 2024-04-22T15:17:29.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.177+0000 7ff950092700 1 -- start start 2024-04-22T15:17:29.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff950092700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9480ffb10 0x7ff9481943e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:29.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff950092700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff948100d10 0x7ff948194920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:29.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff950092700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff948194f40 con 0x7ff9480ffb10 2024-04-22T15:17:29.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff950092700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff948195080 con 0x7ff948100d10 2024-04-22T15:17:29.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff94d62d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff948100d10 0x7ff948194920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:29.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff94d62d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff948100d10 0x7ff948194920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:43496/0 (socket says 172.21.15.96:43496) 2024-04-22T15:17:29.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff94d62d700 1 -- 172.21.15.96:0/1274689433 learned_addr learned my addr 172.21.15.96:0/1274689433 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:17:29.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff94de2e700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9480ffb10 0x7ff9481943e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:29.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff94d62d700 1 -- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9480ffb10 msgr2=0x7ff9481943e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:29.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff94d62d700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9480ffb10 0x7ff9481943e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.178+0000 7ff94d62d700 1 -- 172.21.15.96:0/1274689433 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440095f0 con 0x7ff948100d10 2024-04-22T15:17:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.179+0000 7ff94d62d700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff948100d10 0x7ff948194920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff93800e920 tx=0x7ff93800ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.179+0000 7ff93effd700 1 -- 172.21.15.96:0/1274689433 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff93800c9d0 con 0x7ff948100d10 2024-04-22T15:17:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.179+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff948199ae0 con 0x7ff948100d10 2024-04-22T15:17:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.179+0000 7ff93effd700 1 -- 172.21.15.96:0/1274689433 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff938010430 con 0x7ff948100d10 2024-04-22T15:17:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.179+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94819a0b0 con 0x7ff948100d10 2024-04-22T15:17:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.179+0000 7ff93effd700 1 -- 172.21.15.96:0/1274689433 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff938018630 con 0x7ff948100d10 2024-04-22T15:17:29.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.181+0000 7ff93effd700 1 -- 172.21.15.96:0/1274689433 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff93800cb30 con 0x7ff948100d10 2024-04-22T15:17:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.182+0000 7ff93effd700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff93406cce0 0x7ff93406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.182+0000 7ff93effd700 1 -- 172.21.15.96:0/1274689433 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff938014070 con 0x7ff948100d10 2024-04-22T15:17:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.182+0000 7ff94de2e700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff93406cce0 0x7ff93406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.183+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c005320 con 0x7ff948100d10 2024-04-22T15:17:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.183+0000 7ff94de2e700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff93406cce0 0x7ff93406f1a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff944018040 tx=0x7ff944000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:29.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.188+0000 7ff93effd700 1 -- 172.21.15.96:0/1274689433 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7ff938052c50 con 0x7ff948100d10 2024-04-22T15:17:29.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.610+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff92c000bf0 con 0x7ff93406cce0 2024-04-22T15:17:29.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.630+0000 7ff93effd700 1 -- 172.21.15.96:0/1274689433 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3864 (secure 0 0 0) 0x7ff92c000bf0 con 0x7ff93406cce0 2024-04-22T15:17:29.634 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:17:29.634 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (5m) 84s ago 7m 24.7M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:17:29.634 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (7m) 84s ago 7m 7541k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:17:29.635 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (6m) 89s ago 6m 7474k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:17:29.635 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (7m) 84s ago 7m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:17:29.635 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (6m) 89s ago 6m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:17:29.635 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (5m) 84s ago 7m 84.1M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:17:29.635 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (105s) 84s ago 104s 18.7M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:17:29.635 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (112s) 84s ago 111s 13.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:17:29.635 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (109s) 89s ago 109s 16.8M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (102s) 89s ago 101s 14.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:9283,8765,8443 running (8m) 84s ago 8m 498M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 12e0357210ef 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (6m) 89s ago 6m 450M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 ea53e28b40cc 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (9m) 84s ago 9m 44.3M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (6m) 89s ago 6m 35.2M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (7m) 84s ago 7m 14.6M - 1.5.0 0da6a335fe13 852e13a0c299 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (6m) 89s ago 6m 16.2M - 1.5.0 0da6a335fe13 4d02817c7cd0 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (5m) 84s ago 5m 47.5M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (5m) 84s ago 5m 48.9M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (4m) 84s ago 4m 53.4M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (4m) 89s ago 4m 47.2M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (3m) 89s ago 3m 43.9M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (3m) 89s ago 3m 44.4M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:17:29.636 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (5m) 84s ago 6m 55.2M - 2.43.0 a07b618ecd1d 87457032ed08 2024-04-22T15:17:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.632+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff93406cce0 msgr2=0x7ff93406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.632+0000 7ff950092700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff93406cce0 0x7ff93406f1a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff944018040 tx=0x7ff944000bc0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.633+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff948100d10 msgr2=0x7ff948194920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.633+0000 7ff950092700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff948100d10 0x7ff948194920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff93800e920 tx=0x7ff93800ec30 comp rx=0 tx=0).stop 2024-04-22T15:17:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.633+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 shutdown_connections 2024-04-22T15:17:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.633+0000 7ff950092700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7ff93406cce0 0x7ff93406f1a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.633+0000 7ff950092700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9480ffb10 0x7ff9481943e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.633+0000 7ff950092700 1 --2- 172.21.15.96:0/1274689433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff948100d10 0x7ff948194920 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.633+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 >> 172.21.15.96:0/1274689433 conn(0x7ff9480fb0b0 msgr2=0x7ff948104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.633+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 shutdown_connections 2024-04-22T15:17:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.633+0000 7ff950092700 1 -- 172.21.15.96:0/1274689433 wait complete. 2024-04-22T15:17:29.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.796+0000 7f4c7058b700 1 -- 172.21.15.96:0/4196383259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c680ffb90 msgr2=0x7f4c680fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:29.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.796+0000 7f4c7058b700 1 --2- 172.21.15.96:0/4196383259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c680ffb90 0x7f4c680fffb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4c64009910 tx=0x7f4c64009c20 comp rx=0 tx=0).stop 2024-04-22T15:17:29.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.796+0000 7f4c7058b700 1 -- 172.21.15.96:0/4196383259 shutdown_connections 2024-04-22T15:17:29.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.796+0000 7f4c7058b700 1 --2- 172.21.15.96:0/4196383259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c68100d90 0x7f4c681011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.796+0000 7f4c7058b700 1 --2- 172.21.15.96:0/4196383259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c680ffb90 0x7f4c680fffb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.796+0000 7f4c7058b700 1 -- 172.21.15.96:0/4196383259 >> 172.21.15.96:0/4196383259 conn(0x7f4c680fb110 msgr2=0x7f4c680fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:29.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.796+0000 7f4c7058b700 1 -- 172.21.15.96:0/4196383259 shutdown_connections 2024-04-22T15:17:29.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.797+0000 7f4c7058b700 1 -- 172.21.15.96:0/4196383259 wait complete. 2024-04-22T15:17:29.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.797+0000 7f4c7058b700 1 Processor -- start 2024-04-22T15:17:29.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.797+0000 7f4c7058b700 1 -- start start 2024-04-22T15:17:29.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.798+0000 7f4c7058b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c680ffb90 0x7f4c6819cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:29.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.798+0000 7f4c7058b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c68100d90 0x7f4c6819d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:29.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.798+0000 7f4c7058b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6819d6b0 con 0x7f4c68100d90 2024-04-22T15:17:29.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.798+0000 7f4c7058b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6819d7f0 con 0x7f4c680ffb90 2024-04-22T15:17:29.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.798+0000 7f4c6e327700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c680ffb90 0x7f4c6819cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:29.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.798+0000 7f4c6e327700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c680ffb90 0x7f4c6819cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:43522/0 (socket says 172.21.15.96:43522) 2024-04-22T15:17:29.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.798+0000 7f4c6e327700 1 -- 172.21.15.96:0/817876486 learned_addr learned my addr 172.21.15.96:0/817876486 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:17:29.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.798+0000 7f4c6db26700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c68100d90 0x7f4c6819d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:29.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.798+0000 7f4c6e327700 1 -- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c68100d90 msgr2=0x7f4c6819d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:29.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.799+0000 7f4c6e327700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c68100d90 0x7f4c6819d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:29.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.799+0000 7f4c6e327700 1 -- 172.21.15.96:0/817876486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c640095f0 con 0x7f4c680ffb90 2024-04-22T15:17:29.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.799+0000 7f4c6db26700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c68100d90 0x7f4c6819d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:17:29.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.799+0000 7f4c6e327700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c680ffb90 0x7f4c6819cb50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400f690 tx=0x7f4c6400f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:29.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.799+0000 7f4c5f7fe700 1 -- 172.21.15.96:0/817876486 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c6401d070 con 0x7f4c680ffb90 2024-04-22T15:17:29.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.799+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c681a21a0 con 0x7f4c680ffb90 2024-04-22T15:17:29.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.799+0000 7f4c5f7fe700 1 -- 172.21.15.96:0/817876486 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c64005dc0 con 0x7f4c680ffb90 2024-04-22T15:17:29.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.799+0000 7f4c5f7fe700 1 -- 172.21.15.96:0/817876486 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c640226a0 con 0x7f4c680ffb90 2024-04-22T15:17:29.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.799+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c681a2690 con 0x7f4c680ffb90 2024-04-22T15:17:29.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.801+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6818e620 con 0x7f4c680ffb90 2024-04-22T15:17:29.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.801+0000 7f4c5f7fe700 1 -- 172.21.15.96:0/817876486 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4c64003680 con 0x7f4c680ffb90 2024-04-22T15:17:29.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.802+0000 7f4c5f7fe700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4c5406cce0 0x7f4c5406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:29.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.802+0000 7f4c5f7fe700 1 -- 172.21.15.96:0/817876486 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4c6408d8d0 con 0x7f4c680ffb90 2024-04-22T15:17:29.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.802+0000 7f4c6db26700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4c5406cce0 0x7f4c5406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:29.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.803+0000 7f4c6db26700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4c5406cce0 0x7f4c5406f1a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800b3c0 tx=0x7f4c5800d040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:29.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:29.810+0000 7f4c5f7fe700 1 -- 172.21.15.96:0/817876486 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f4c6405b6f0 con 0x7f4c680ffb90 2024-04-22T15:17:29.884 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:29 smithi196 ceph-mon[34717]: from='client.24339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:29.884 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:29 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:29.884 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:29 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:17:29.884 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:29 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:17:29.884 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:29 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:17:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:29 smithi096 ceph-mon[26688]: from='client.24339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:29 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:29 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:17:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:29 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:17:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:29 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:17:30.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.308+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4c6804ebb0 con 0x7f4c680ffb90 2024-04-22T15:17:30.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.309+0000 7f4c5f7fe700 1 -- 172.21.15.96:0/817876486 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f4c6405b280 con 0x7f4c680ffb90 2024-04-22T15:17:30.313 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:17:30.313 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:17:30.313 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:17:30.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:17:30.313 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:17:30.313 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:17:30.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:17:30.313 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:17:30.314 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:17:30.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:17:30.314 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:17:30.314 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:17:30.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:17:30.314 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:17:30.314 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 14 2024-04-22T15:17:30.314 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:17:30.314 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4c5406cce0 msgr2=0x7f4c5406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4c5406cce0 0x7f4c5406f1a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800b3c0 tx=0x7f4c5800d040 comp rx=0 tx=0).stop 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c680ffb90 msgr2=0x7f4c6819cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c680ffb90 0x7f4c6819cb50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400f690 tx=0x7f4c6400f6c0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 shutdown_connections 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f4c5406cce0 0x7f4c5406f1a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c680ffb90 0x7f4c6819cb50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 --2- 172.21.15.96:0/817876486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c68100d90 0x7f4c6819d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 >> 172.21.15.96:0/817876486 conn(0x7f4c680fb110 msgr2=0x7f4c68104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 shutdown_connections 2024-04-22T15:17:30.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.311+0000 7f4c7058b700 1 -- 172.21.15.96:0/817876486 wait complete. 2024-04-22T15:17:30.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.418+0000 7f7dd0161700 1 -- 172.21.15.96:0/795605662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dc8101ec0 msgr2=0x7f7dc8102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:30.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.418+0000 7f7dd0161700 1 --2- 172.21.15.96:0/795605662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dc8101ec0 0x7f7dc8102320 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009910 tx=0x7f7db8009c20 comp rx=0 tx=0).stop 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.418+0000 7f7dd0161700 1 -- 172.21.15.96:0/795605662 shutdown_connections 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.418+0000 7f7dd0161700 1 --2- 172.21.15.96:0/795605662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dc8101ec0 0x7f7dc8102320 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.418+0000 7f7dd0161700 1 --2- 172.21.15.96:0/795605662 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc80feee0 0x7f7dc80ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.418+0000 7f7dd0161700 1 -- 172.21.15.96:0/795605662 >> 172.21.15.96:0/795605662 conn(0x7f7dc80fa840 msgr2=0x7f7dc80fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.419+0000 7f7dd0161700 1 -- 172.21.15.96:0/795605662 shutdown_connections 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.419+0000 7f7dd0161700 1 -- 172.21.15.96:0/795605662 wait complete. 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.419+0000 7f7dd0161700 1 Processor -- start 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.419+0000 7f7dd0161700 1 -- start start 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.419+0000 7f7dd0161700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc80feee0 0x7f7dc8196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.419+0000 7f7dd0161700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dc8101ec0 0x7f7dc8196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.419+0000 7f7dd0161700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc81971a0 con 0x7f7dc8101ec0 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.419+0000 7f7dd0161700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc81972e0 con 0x7f7dc80feee0 2024-04-22T15:17:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.420+0000 7f7dcdefd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc80feee0 0x7f7dc8196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.420+0000 7f7dcdefd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc80feee0 0x7f7dc8196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:43538/0 (socket says 172.21.15.96:43538) 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.420+0000 7f7dcdefd700 1 -- 172.21.15.96:0/2810812870 learned_addr learned my addr 172.21.15.96:0/2810812870 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.420+0000 7f7dcd6fc700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dc8101ec0 0x7f7dc8196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.420+0000 7f7dcdefd700 1 -- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dc8101ec0 msgr2=0x7f7dc8196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.420+0000 7f7dcdefd700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dc8101ec0 0x7f7dc8196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.420+0000 7f7dcdefd700 1 -- 172.21.15.96:0/2810812870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db80095f0 con 0x7f7dc80feee0 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.420+0000 7f7dcd6fc700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dc8101ec0 0x7f7dc8196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.421+0000 7f7dcdefd700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc80feee0 0x7f7dc8196640 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400d900 tx=0x7f7dc400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.421+0000 7f7dbeffd700 1 -- 172.21.15.96:0/2810812870 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dc400b920 con 0x7f7dc80feee0 2024-04-22T15:17:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.421+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc819bcf0 con 0x7f7dc80feee0 2024-04-22T15:17:30.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.421+0000 7f7dbeffd700 1 -- 172.21.15.96:0/2810812870 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7dc4010460 con 0x7f7dc80feee0 2024-04-22T15:17:30.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.421+0000 7f7dbeffd700 1 -- 172.21.15.96:0/2810812870 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dc400f590 con 0x7f7dc80feee0 2024-04-22T15:17:30.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.421+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc819c1e0 con 0x7f7dc80feee0 2024-04-22T15:17:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.422+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc804ec20 con 0x7f7dc80feee0 2024-04-22T15:17:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.423+0000 7f7dbeffd700 1 -- 172.21.15.96:0/2810812870 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7dc40105d0 con 0x7f7dc80feee0 2024-04-22T15:17:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.423+0000 7f7dbeffd700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f7db406ccf0 0x7f7db406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:17:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.423+0000 7f7dbeffd700 1 -- 172.21.15.96:0/2810812870 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7dc408bf60 con 0x7f7dc80feee0 2024-04-22T15:17:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.423+0000 7f7dcd6fc700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f7db406ccf0 0x7f7db406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:17:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.424+0000 7f7dcd6fc700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f7db406ccf0 0x7f7db406f1b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7db8006010 tx=0x7f7db8005180 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:17:30.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.432+0000 7f7dbeffd700 1 -- 172.21.15.96:0/2810812870 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183351 (secure 0 0 0) 0x7f7dc40bfc50 con 0x7f7dc80feee0 2024-04-22T15:17:30.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.673+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 --> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7dc802cf30 con 0x7f7db406ccf0 2024-04-22T15:17:30.678 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:17:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:17:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:17:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:17:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [], 2024-04-22T15:17:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "0/2 daemons upgraded", 2024-04-22T15:17:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mgr daemons", 2024-04-22T15:17:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:17:30.679 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:17:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.675+0000 7f7dbeffd700 1 -- 172.21.15.96:0/2810812870 <== mgr.14217 v2:172.21.15.96:6800/2441203700 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f7dc802cf30 con 0x7f7db406ccf0 2024-04-22T15:17:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.678+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f7db406ccf0 msgr2=0x7f7db406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.678+0000 7f7dd0161700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f7db406ccf0 0x7f7db406f1b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7db8006010 tx=0x7f7db8005180 comp rx=0 tx=0).stop 2024-04-22T15:17:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.678+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc80feee0 msgr2=0x7f7dc8196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:17:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.678+0000 7f7dd0161700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc80feee0 0x7f7dc8196640 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400d900 tx=0x7f7dc400dc10 comp rx=0 tx=0).stop 2024-04-22T15:17:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.678+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 shutdown_connections 2024-04-22T15:17:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.678+0000 7f7dd0161700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:6800/2441203700,v1:172.21.15.96:6801/2441203700] conn(0x7f7db406ccf0 0x7f7db406f1b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.678+0000 7f7dd0161700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc80feee0 0x7f7dc8196640 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.678+0000 7f7dd0161700 1 --2- 172.21.15.96:0/2810812870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dc8101ec0 0x7f7dc8196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:17:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.678+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 >> 172.21.15.96:0/2810812870 conn(0x7f7dc80fa840 msgr2=0x7f7dc81054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:17:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.679+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 shutdown_connections 2024-04-22T15:17:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:17:30.679+0000 7f7dd0161700 1 -- 172.21.15.96:0/2810812870 wait complete. 2024-04-22T15:17:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:30 smithi096 ceph-mon[26688]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:30 smithi096 ceph-mon[26688]: Upgrade: Updating mgr.smithi196.gdaegc 2024-04-22T15:17:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:30 smithi096 ceph-mon[26688]: Deploying daemon mgr.smithi196.gdaegc on smithi196 2024-04-22T15:17:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:30 smithi096 ceph-mon[26688]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:30 smithi096 ceph-mon[26688]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:30 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/817876486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:17:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:30 smithi196 ceph-mon[34717]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:30 smithi196 ceph-mon[34717]: Upgrade: Updating mgr.smithi196.gdaegc 2024-04-22T15:17:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:30 smithi196 ceph-mon[34717]: Deploying daemon mgr.smithi196.gdaegc on smithi196 2024-04-22T15:17:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:30 smithi196 ceph-mon[34717]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:30 smithi196 ceph-mon[34717]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:31.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:30 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/817876486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:17:31.944 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:31 smithi196 ceph-mon[34717]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:32.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:31 smithi096 ceph-mon[26688]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:17:33.027 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:32 smithi196 ceph-mon[34717]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:33.057 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:32 smithi096 ceph-mon[26688]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:34.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:34.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:34.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:34 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:17:34.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:34 smithi096 ceph-mon[26688]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:34.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:34.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:34.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:34 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:17:34.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:34 smithi196 ceph-mon[34717]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:36 smithi196 ceph-mon[34717]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:17:37.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:36 smithi096 ceph-mon[26688]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:17:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:37 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:17:38.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:37 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr:Note: switching to '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'. 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr: 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr: 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr: 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr: git switch -c 2024-04-22T15:17:38.775 INFO:tasks.workunit.client.0.smithi096.stderr: 2024-04-22T15:17:38.776 INFO:tasks.workunit.client.0.smithi096.stderr:Or undo this operation with: 2024-04-22T15:17:38.776 INFO:tasks.workunit.client.0.smithi096.stderr: 2024-04-22T15:17:38.776 INFO:tasks.workunit.client.0.smithi096.stderr: git switch - 2024-04-22T15:17:38.776 INFO:tasks.workunit.client.0.smithi096.stderr: 2024-04-22T15:17:38.776 INFO:tasks.workunit.client.0.smithi096.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T15:17:38.776 INFO:tasks.workunit.client.0.smithi096.stderr: 2024-04-22T15:17:38.776 INFO:tasks.workunit.client.0.smithi096.stderr:HEAD is now at 5883d494d39 Merge branch 'cheroot-upgrade' of https://github.com/rhcs-dashboard/ceph into wip-adk-testing-2024-04-22-0809 2024-04-22T15:17:38.781 DEBUG:teuthology.orchestra.run.smithi096:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T15:17:38.810 INFO:tasks.workunit.client.0.smithi096.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T15:17:38.812 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T15:17:38.812 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T15:17:38.945 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:38 smithi196 ceph-mon[34717]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:39.129 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:38 smithi096 ceph-mon[26688]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:39.601 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T15:17:39.654 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T15:17:39.694 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T15:17:39.695 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T15:17:39.695 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T15:17:39.736 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T15:17:39.739 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T15:17:39.740 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T15:17:39.796 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-22T15:17:39.797 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-22T15:17:39.797 DEBUG:teuthology.orchestra.run.smithi096:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-22T15:17:39.860 INFO:tasks.workunit.client.0.smithi096.stderr:+ mkdir -p fsstress 2024-04-22T15:17:39.862 INFO:tasks.workunit.client.0.smithi096.stderr:+ pushd fsstress 2024-04-22T15:17:39.863 INFO:tasks.workunit.client.0.smithi096.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T15:17:39.864 INFO:tasks.workunit.client.0.smithi096.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-22T15:17:40.375 INFO:tasks.workunit.client.0.smithi096.stderr:+ tar xzf ltp-full.tgz 2024-04-22T15:17:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:40 smithi196 ceph-mon[34717]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:41.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:40 smithi096 ceph-mon[26688]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:17:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:41 smithi196 ceph-mon[34717]: Standby manager daemon smithi196.gdaegc restarted 2024-04-22T15:17:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:41 smithi196 ceph-mon[34717]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:17:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:41 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:17:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:41 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:17:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:41 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:17:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:41 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:17:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:41 smithi096 ceph-mon[26688]: Standby manager daemon smithi196.gdaegc restarted 2024-04-22T15:17:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:41 smithi096 ceph-mon[26688]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:17:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:41 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:17:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:41 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:17:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:41 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:17:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:41 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:17:43.056 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:42 smithi096 ceph-mon[26688]: pgmap v245: 65 pgs: 65 active+clean; 5.3 MiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 401 KiB/s wr, 11 op/s 2024-04-22T15:17:43.056 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:42 smithi096 ceph-mon[26688]: mgrmap e20: smithi096.hhkvum(active, since 7m), standbys: smithi196.gdaegc 2024-04-22T15:17:43.083 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:42 smithi196 ceph-mon[34717]: pgmap v245: 65 pgs: 65 active+clean; 5.3 MiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 401 KiB/s wr, 11 op/s 2024-04-22T15:17:43.083 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:42 smithi196 ceph-mon[34717]: mgrmap e20: smithi096.hhkvum(active, since 7m), standbys: smithi196.gdaegc 2024-04-22T15:17:45.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:44 smithi096 ceph-mon[26688]: pgmap v246: 65 pgs: 65 active+clean; 15 MiB data, 203 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s rd, 1.2 MiB/s wr, 27 op/s 2024-04-22T15:17:45.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:44 smithi196 ceph-mon[34717]: pgmap v246: 65 pgs: 65 active+clean; 15 MiB data, 203 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s rd, 1.2 MiB/s wr, 27 op/s 2024-04-22T15:17:47.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:46 smithi096 ceph-mon[26688]: pgmap v247: 65 pgs: 65 active+clean; 15 MiB data, 207 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-04-22T15:17:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:46 smithi196 ceph-mon[34717]: pgmap v247: 65 pgs: 65 active+clean; 15 MiB data, 207 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-04-22T15:17:47.325 INFO:tasks.workunit.client.1.smithi196.stderr:Updating files: 86% (10586/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-22T15:17:47.966 INFO:tasks.workunit.client.1.smithi196.stderr:Note: switching to '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'. 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr: 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr: 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr: 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr: git switch -c 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr: 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr:Or undo this operation with: 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr: 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr: git switch - 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr: 2024-04-22T15:17:47.967 INFO:tasks.workunit.client.1.smithi196.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T15:17:47.968 INFO:tasks.workunit.client.1.smithi196.stderr: 2024-04-22T15:17:47.968 INFO:tasks.workunit.client.1.smithi196.stderr:HEAD is now at 5883d494d39 Merge branch 'cheroot-upgrade' of https://github.com/rhcs-dashboard/ceph into wip-adk-testing-2024-04-22-0809 2024-04-22T15:17:47.974 DEBUG:teuthology.orchestra.run.smithi196:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-22T15:17:48.045 INFO:tasks.workunit.client.1.smithi196.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T15:17:48.047 INFO:tasks.workunit.client.1.smithi196.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-22T15:17:48.047 INFO:tasks.workunit.client.1.smithi196.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T15:17:48.752 INFO:tasks.workunit.client.1.smithi196.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T15:17:48.816 INFO:tasks.workunit.client.1.smithi196.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T15:17:48.856 INFO:tasks.workunit.client.1.smithi196.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-22T15:17:48.858 INFO:tasks.workunit.client.1.smithi196.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-22T15:17:48.858 INFO:tasks.workunit.client.1.smithi196.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T15:17:48.897 INFO:tasks.workunit.client.1.smithi196.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-22T15:17:48.900 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-22T15:17:48.900 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-22T15:17:48.956 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-22T15:17:48.957 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-22T15:17:48.957 DEBUG:teuthology.orchestra.run.smithi196:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-22T15:17:49.020 INFO:tasks.workunit.client.1.smithi196.stderr:+ mkdir -p fsstress 2024-04-22T15:17:49.023 INFO:tasks.workunit.client.1.smithi196.stderr:+ pushd fsstress 2024-04-22T15:17:49.024 INFO:tasks.workunit.client.1.smithi196.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T15:17:49.025 INFO:tasks.workunit.client.1.smithi196.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-22T15:17:49.054 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:48 smithi196 ceph-mon[34717]: pgmap v248: 65 pgs: 65 active+clean; 16 MiB data, 222 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-22T15:17:49.054 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:49.054 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:49.054 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:49.054 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:48 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:49.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:48 smithi096 ceph-mon[26688]: pgmap v248: 65 pgs: 65 active+clean; 16 MiB data, 222 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-22T15:17:49.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:49.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:49.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:49.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:48 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:49.723 INFO:tasks.workunit.client.1.smithi196.stderr:+ tar xzf ltp-full.tgz 2024-04-22T15:17:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:50 smithi096 ceph-mon[26688]: pgmap v249: 65 pgs: 65 active+clean; 20 MiB data, 237 MiB used, 536 GiB / 536 GiB avail; 343 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-04-22T15:17:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:50 smithi196 ceph-mon[34717]: pgmap v249: 65 pgs: 65 active+clean; 20 MiB data, 237 MiB used, 536 GiB / 536 GiB avail; 343 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-04-22T15:17:53.038 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: pgmap v250: 65 pgs: 65 active+clean; 41 MiB data, 289 MiB used, 536 GiB / 536 GiB avail; 685 KiB/s rd, 3.4 MiB/s wr, 124 op/s 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr fail", "who": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: Activating manager daemon smithi196.gdaegc 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: osdmap e46: 6 total, 6 up, 6 in 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "mgr fail", "who": "smithi096.hhkvum"}]': finished 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: mgrmap e21: smithi196.gdaegc(active, starting, since 0.00677626s) 2024-04-22T15:17:53.039 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr metadata", "who": "smithi196.gdaegc", "id": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:17:53.040 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:17:53.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:17:53.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:17:53.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: Manager daemon smithi196.gdaegc is now available 2024-04-22T15:17:53.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:17:53.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:17:53.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi196.gdaegc/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:17:53.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi196.gdaegc/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:17:53.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi196.gdaegc/trash_purge_schedule"}]: dispatch 2024-04-22T15:17:53.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:52 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi196.gdaegc/trash_purge_schedule"}]: dispatch 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: pgmap v250: 65 pgs: 65 active+clean; 41 MiB data, 289 MiB used, 536 GiB / 536 GiB avail; 685 KiB/s rd, 3.4 MiB/s wr, 124 op/s 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr fail", "who": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: Activating manager daemon smithi196.gdaegc 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: osdmap e46: 6 total, 6 up, 6 in 2024-04-22T15:17:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.14217 172.21.15.96:0/667447495' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "mgr fail", "who": "smithi096.hhkvum"}]': finished 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: mgrmap e21: smithi196.gdaegc(active, starting, since 0.00677626s) 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr metadata", "who": "smithi196.gdaegc", "id": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:17:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:17:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:17:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: Manager daemon smithi196.gdaegc is now available 2024-04-22T15:17:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:17:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:17:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi196.gdaegc/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:17:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi196.gdaegc/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:17:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi196.gdaegc/trash_purge_schedule"}]: dispatch 2024-04-22T15:17:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:52 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi196.gdaegc/trash_purge_schedule"}]: dispatch 2024-04-22T15:17:54.109 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:53 smithi196 ceph-mon[34717]: Deploying cephadm binary to smithi196 2024-04-22T15:17:54.109 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:53 smithi196 ceph-mon[34717]: Deploying cephadm binary to smithi096 2024-04-22T15:17:54.109 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:53 smithi196 ceph-mon[34717]: mgrmap e22: smithi196.gdaegc(active, since 1.00897s) 2024-04-22T15:17:54.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:53 smithi096 ceph-mon[26688]: Deploying cephadm binary to smithi196 2024-04-22T15:17:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:53 smithi096 ceph-mon[26688]: Deploying cephadm binary to smithi096 2024-04-22T15:17:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:53 smithi096 ceph-mon[26688]: mgrmap e22: smithi196.gdaegc(active, since 1.00897s) 2024-04-22T15:17:55.160 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:54 smithi096 ceph-mon[26688]: pgmap v3: 65 pgs: 65 active+clean; 60 MiB data, 370 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:17:55.160 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:54 smithi096 ceph-mon[26688]: [22/Apr/2024:15:17:53] ENGINE Bus STARTING 2024-04-22T15:17:55.160 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:54 smithi096 ceph-mon[26688]: [22/Apr/2024:15:17:53] ENGINE Serving on https://172.21.15.196:7150 2024-04-22T15:17:55.160 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:54 smithi096 ceph-mon[26688]: [22/Apr/2024:15:17:53] ENGINE Serving on http://172.21.15.196:8765 2024-04-22T15:17:55.160 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:54 smithi096 ceph-mon[26688]: [22/Apr/2024:15:17:53] ENGINE Bus STARTED 2024-04-22T15:17:55.160 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:54 smithi096 ceph-mon[26688]: mgrmap e23: smithi196.gdaegc(active, since 2s) 2024-04-22T15:17:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:54 smithi196 ceph-mon[34717]: pgmap v3: 65 pgs: 65 active+clean; 60 MiB data, 370 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:17:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:54 smithi196 ceph-mon[34717]: [22/Apr/2024:15:17:53] ENGINE Bus STARTING 2024-04-22T15:17:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:54 smithi196 ceph-mon[34717]: [22/Apr/2024:15:17:53] ENGINE Serving on https://172.21.15.196:7150 2024-04-22T15:17:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:54 smithi196 ceph-mon[34717]: [22/Apr/2024:15:17:53] ENGINE Serving on http://172.21.15.196:8765 2024-04-22T15:17:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:54 smithi196 ceph-mon[34717]: [22/Apr/2024:15:17:53] ENGINE Bus STARTED 2024-04-22T15:17:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:54 smithi196 ceph-mon[34717]: mgrmap e23: smithi196.gdaegc(active, since 2s) 2024-04-22T15:17:55.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:55 smithi096 ceph-mon[26688]: pgmap v4: 65 pgs: 65 active+clean; 60 MiB data, 370 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:17:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:55 smithi196 ceph-mon[34717]: pgmap v4: 65 pgs: 65 active+clean; 60 MiB data, 370 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:17:57.643 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:57 smithi096 ceph-mon[26688]: pgmap v5: 65 pgs: 65 active+clean; 60 MiB data, 370 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:17:57.643 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:57 smithi096 ceph-mon[26688]: mgrmap e24: smithi196.gdaegc(active, since 4s) 2024-04-22T15:17:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:57 smithi196 ceph-mon[34717]: pgmap v5: 65 pgs: 65 active+clean; 60 MiB data, 370 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:17:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:57 smithi196 ceph-mon[34717]: mgrmap e24: smithi196.gdaegc(active, since 4s) 2024-04-22T15:17:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:17:59 smithi096 ceph-mon[26688]: pgmap v6: 65 pgs: 65 active+clean; 69 MiB data, 383 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-04-22T15:17:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:17:59 smithi196 ceph-mon[34717]: pgmap v6: 65 pgs: 65 active+clean; 69 MiB data, 383 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-04-22T15:18:00.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.833+0000 7f04fd4b2700 1 -- 172.21.15.96:0/1815864829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8101720 msgr2=0x7f04f8103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:00.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.833+0000 7f04fd4b2700 1 --2- 172.21.15.96:0/1815864829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8101720 0x7f04f8103b10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f04e8009910 tx=0x7f04e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:18:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.833+0000 7f04fd4b2700 1 -- 172.21.15.96:0/1815864829 shutdown_connections 2024-04-22T15:18:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.833+0000 7f04fd4b2700 1 --2- 172.21.15.96:0/1815864829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04f8104050 0x7f04f8106440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.833+0000 7f04fd4b2700 1 --2- 172.21.15.96:0/1815864829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8101720 0x7f04f8103b10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.833+0000 7f04fd4b2700 1 -- 172.21.15.96:0/1815864829 >> 172.21.15.96:0/1815864829 conn(0x7f04f80fb110 msgr2=0x7f04f80fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.833+0000 7f04fd4b2700 1 -- 172.21.15.96:0/1815864829 shutdown_connections 2024-04-22T15:18:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.834+0000 7f04fd4b2700 1 -- 172.21.15.96:0/1815864829 wait complete. 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.834+0000 7f04fd4b2700 1 Processor -- start 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.835+0000 7f04fd4b2700 1 -- start start 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.835+0000 7f04fd4b2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04f8101720 0x7f04f8198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.835+0000 7f04fd4b2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8104050 0x7f04f8198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.835+0000 7f04fd4b2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f81993c0 con 0x7f04f8104050 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.835+0000 7f04fd4b2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f8199500 con 0x7f04f8101720 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.835+0000 7f04f67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8104050 0x7f04f8198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.836+0000 7f04f67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8104050 0x7f04f8198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59346/0 (socket says 172.21.15.96:59346) 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.836+0000 7f04f67fc700 1 -- 172.21.15.96:0/566179000 learned_addr learned my addr 172.21.15.96:0/566179000 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.836+0000 7f04f6ffd700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04f8101720 0x7f04f8198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.836+0000 7f04f67fc700 1 -- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04f8101720 msgr2=0x7f04f8198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:00.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.836+0000 7f04f67fc700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04f8101720 0x7f04f8198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:00.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.836+0000 7f04f67fc700 1 -- 172.21.15.96:0/566179000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e80095f0 con 0x7f04f8104050 2024-04-22T15:18:00.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.836+0000 7f04f6ffd700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04f8101720 0x7f04f8198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:18:00.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.837+0000 7f04f67fc700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8104050 0x7f04f8198da0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00e940 tx=0x7f04ec00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.837+0000 7f04dffff700 1 -- 172.21.15.96:0/566179000 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04ec00cba0 con 0x7f04f8104050 2024-04-22T15:18:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.837+0000 7f04dffff700 1 -- 172.21.15.96:0/566179000 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04ec00ee90 con 0x7f04f8104050 2024-04-22T15:18:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.837+0000 7f04dffff700 1 -- 172.21.15.96:0/566179000 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04ec0055d0 con 0x7f04f8104050 2024-04-22T15:18:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.837+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f819df60 con 0x7f04f8104050 2024-04-22T15:18:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.838+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f819e420 con 0x7f04f8104050 2024-04-22T15:18:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.839+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f804ec20 con 0x7f04f8104050 2024-04-22T15:18:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.839+0000 7f04dffff700 1 -- 172.21.15.96:0/566179000 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 24) v1 ==== 46707+0+0 (secure 0 0 0) 0x7f04ec004750 con 0x7f04f8104050 2024-04-22T15:18:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.839+0000 7f04dffff700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f04e0039960 0x7f04e003be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.839+0000 7f04dffff700 1 -- 172.21.15.96:0/566179000 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f04ec014070 con 0x7f04f8104050 2024-04-22T15:18:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.840+0000 7f04f6ffd700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f04e0039960 0x7f04e003be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:00.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.840+0000 7f04f6ffd700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f04e0039960 0x7f04e003be20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f04e800b800 tx=0x7f04e8009f70 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:00.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:00.849+0000 7f04dffff700 1 -- 172.21.15.96:0/566179000 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f04f804ec20 con 0x7f04f8104050 2024-04-22T15:18:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.180+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f04f804ec20 con 0x7f04e0039960 2024-04-22T15:18:01.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.182+0000 7f04dffff700 1 -- 172.21.15.96:0/566179000 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f04f804ec20 con 0x7f04e0039960 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f04e0039960 msgr2=0x7f04e003be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f04e0039960 0x7f04e003be20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f04e800b800 tx=0x7f04e8009f70 comp rx=0 tx=0).stop 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8104050 msgr2=0x7f04f8198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8104050 0x7f04f8198da0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00e940 tx=0x7f04ec00ec50 comp rx=0 tx=0).stop 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 shutdown_connections 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f04e0039960 0x7f04e003be20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04f8101720 0x7f04f8198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 --2- 172.21.15.96:0/566179000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04f8104050 0x7f04f8198da0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 >> 172.21.15.96:0/566179000 conn(0x7f04f80fb110 msgr2=0x7f04f8100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 shutdown_connections 2024-04-22T15:18:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.185+0000 7f04fd4b2700 1 -- 172.21.15.96:0/566179000 wait complete. 2024-04-22T15:18:01.203 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:18:01.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.315+0000 7fc47dbc2700 1 -- 172.21.15.96:0/620767627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4780ff2d0 msgr2=0x7fc4780ff6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:01.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.315+0000 7fc47dbc2700 1 --2- 172.21.15.96:0/620767627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4780ff2d0 0x7fc4780ff6f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc46000aa20 tx=0x7fc46000ad30 comp rx=0 tx=0).stop 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.315+0000 7fc47dbc2700 1 -- 172.21.15.96:0/620767627 shutdown_connections 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.315+0000 7fc47dbc2700 1 --2- 172.21.15.96:0/620767627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4781004d0 0x7fc478100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.315+0000 7fc47dbc2700 1 --2- 172.21.15.96:0/620767627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4780ff2d0 0x7fc4780ff6f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.315+0000 7fc47dbc2700 1 -- 172.21.15.96:0/620767627 >> 172.21.15.96:0/620767627 conn(0x7fc4780fa890 msgr2=0x7fc4780fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.315+0000 7fc47dbc2700 1 -- 172.21.15.96:0/620767627 shutdown_connections 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.315+0000 7fc47dbc2700 1 -- 172.21.15.96:0/620767627 wait complete. 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.316+0000 7fc47dbc2700 1 Processor -- start 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.316+0000 7fc47dbc2700 1 -- start start 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.316+0000 7fc47dbc2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4780ff2d0 0x7fc478198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.316+0000 7fc47dbc2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4781004d0 0x7fc478198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.316+0000 7fc47dbc2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc478199370 con 0x7fc4781004d0 2024-04-22T15:18:01.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.316+0000 7fc47dbc2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4781994b0 con 0x7fc4780ff2d0 2024-04-22T15:18:01.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc476ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4781004d0 0x7fc478198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:01.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc4777fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4780ff2d0 0x7fc478198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc476ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4781004d0 0x7fc478198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59370/0 (socket says 172.21.15.96:59370) 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc476ffd700 1 -- 172.21.15.96:0/824688567 learned_addr learned my addr 172.21.15.96:0/824688567 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc4777fe700 1 -- 172.21.15.96:0/824688567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4781004d0 msgr2=0x7fc478198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc4777fe700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4781004d0 0x7fc478198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc4777fe700 1 -- 172.21.15.96:0/824688567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46000a700 con 0x7fc4780ff2d0 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc476ffd700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4781004d0 0x7fc478198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc4777fe700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4780ff2d0 0x7fc478198810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc460006010 tx=0x7fc460005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.317+0000 7fc474ff9700 1 -- 172.21.15.96:0/824688567 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc460005000 con 0x7fc4780ff2d0 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.318+0000 7fc474ff9700 1 -- 172.21.15.96:0/824688567 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc460005160 con 0x7fc4780ff2d0 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.318+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47819deb0 con 0x7fc4780ff2d0 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.318+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47819e320 con 0x7fc4780ff2d0 2024-04-22T15:18:01.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.319+0000 7fc474ff9700 1 -- 172.21.15.96:0/824688567 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc460016760 con 0x7fc4780ff2d0 2024-04-22T15:18:01.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.319+0000 7fc474ff9700 1 -- 172.21.15.96:0/824688567 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 24) v1 ==== 46707+0+0 (secure 0 0 0) 0x7fc4600169a0 con 0x7fc4780ff2d0 2024-04-22T15:18:01.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.319+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc458005320 con 0x7fc4780ff2d0 2024-04-22T15:18:01.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.319+0000 7fc474ff9700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fc4640396c0 0x7fc46403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:01.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.319+0000 7fc474ff9700 1 -- 172.21.15.96:0/824688567 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fc460012070 con 0x7fc4780ff2d0 2024-04-22T15:18:01.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.319+0000 7fc476ffd700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fc4640396c0 0x7fc46403bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:01.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.320+0000 7fc476ffd700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fc4640396c0 0x7fc46403bb80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc468011550 tx=0x7fc468009260 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:01.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.324+0000 7fc474ff9700 1 -- 172.21.15.96:0/824688567 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fc46001e360 con 0x7fc4780ff2d0 2024-04-22T15:18:01.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:01 smithi096 ceph-mon[26688]: pgmap v7: 65 pgs: 65 active+clean; 74 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-04-22T15:18:01.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.609+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc458000bf0 con 0x7fc4640396c0 2024-04-22T15:18:01.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.610+0000 7fc474ff9700 1 -- 172.21.15.96:0/824688567 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fc458000bf0 con 0x7fc4640396c0 2024-04-22T15:18:01.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fc4640396c0 msgr2=0x7fc46403bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:01.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fc4640396c0 0x7fc46403bb80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc468011550 tx=0x7fc468009260 comp rx=0 tx=0).stop 2024-04-22T15:18:01.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4780ff2d0 msgr2=0x7fc478198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:01.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4780ff2d0 0x7fc478198810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc460006010 tx=0x7fc460005800 comp rx=0 tx=0).stop 2024-04-22T15:18:01.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 shutdown_connections 2024-04-22T15:18:01.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fc4640396c0 0x7fc46403bb80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4780ff2d0 0x7fc478198810 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 --2- 172.21.15.96:0/824688567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4781004d0 0x7fc478198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 >> 172.21.15.96:0/824688567 conn(0x7fc4780fa890 msgr2=0x7fc478103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:01.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 shutdown_connections 2024-04-22T15:18:01.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.613+0000 7fc47dbc2700 1 -- 172.21.15.96:0/824688567 wait complete. 2024-04-22T15:18:01.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.731+0000 7f41f8e66700 1 -- 172.21.15.96:0/2572549341 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41e40054b0 con 0x7f41f4100d90 2024-04-22T15:18:01.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.732+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2572549341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 msgr2=0x7f41f41011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:01.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.732+0000 7f41fc0cc700 1 --2- 172.21.15.96:0/2572549341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 0x7f41f41011f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009910 tx=0x7f41e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:18:01.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.732+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2572549341 shutdown_connections 2024-04-22T15:18:01.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.732+0000 7f41fc0cc700 1 --2- 172.21.15.96:0/2572549341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 0x7f41f41011f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.732+0000 7f41fc0cc700 1 --2- 172.21.15.96:0/2572549341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f40ffb90 0x7f41f40fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.732+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2572549341 >> 172.21.15.96:0/2572549341 conn(0x7f41f40fb110 msgr2=0x7f41f40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:01.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.733+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2572549341 shutdown_connections 2024-04-22T15:18:01.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.733+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2572549341 wait complete. 2024-04-22T15:18:01.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.733+0000 7f41fc0cc700 1 Processor -- start 2024-04-22T15:18:01.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.733+0000 7f41fc0cc700 1 -- start start 2024-04-22T15:18:01.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.733+0000 7f41fc0cc700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f40ffb90 0x7f41f4194470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:01.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.733+0000 7f41fc0cc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 0x7f41f41949b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:01.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.733+0000 7f41fc0cc700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f4194f40 con 0x7f41f4100d90 2024-04-22T15:18:01.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.733+0000 7f41fc0cc700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f4195080 con 0x7f41f40ffb90 2024-04-22T15:18:01.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41f9667700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 0x7f41f41949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:01.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41f9667700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 0x7f41f41949b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59378/0 (socket says 172.21.15.96:59378) 2024-04-22T15:18:01.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41f9667700 1 -- 172.21.15.96:0/2486173521 learned_addr learned my addr 172.21.15.96:0/2486173521 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:01.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41f9e68700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f40ffb90 0x7f41f4194470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:01.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41f9667700 1 -- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f40ffb90 msgr2=0x7f41f4194470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:01.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41f9667700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f40ffb90 0x7f41f4194470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:01.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41f9667700 1 -- 172.21.15.96:0/2486173521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e40095f0 con 0x7f41f4100d90 2024-04-22T15:18:01.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41f9667700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 0x7f41f41949b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f41e40092e0 tx=0x7f41e400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:01.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41eaffd700 1 -- 172.21.15.96:0/2486173521 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41e401d070 con 0x7f41f4100d90 2024-04-22T15:18:01.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41eaffd700 1 -- 172.21.15.96:0/2486173521 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41e400f460 con 0x7f41f4100d90 2024-04-22T15:18:01.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41eaffd700 1 -- 172.21.15.96:0/2486173521 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41e4021640 con 0x7f41f4100d90 2024-04-22T15:18:01.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.734+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f4199a90 con 0x7f41f4100d90 2024-04-22T15:18:01.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.735+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f4199e90 con 0x7f41f4100d90 2024-04-22T15:18:01.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.736+0000 7f41eaffd700 1 -- 172.21.15.96:0/2486173521 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 24) v1 ==== 46707+0+0 (secure 0 0 0) 0x7f41e400f5d0 con 0x7f41f4100d90 2024-04-22T15:18:01.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.736+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f418e600 con 0x7f41f4100d90 2024-04-22T15:18:01.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.736+0000 7f41eaffd700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f41e004a8d0 0x7f41e004cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:01.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.736+0000 7f41eaffd700 1 -- 172.21.15.96:0/2486173521 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f41e404ee60 con 0x7f41f4100d90 2024-04-22T15:18:01.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.736+0000 7f41f9e68700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f41e004a8d0 0x7f41e004cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:01.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.737+0000 7f41f9e68700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f41e004a8d0 0x7f41e004cd90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f41f4100bf0 tx=0x7f41f000a3b0 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:01.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:01.742+0000 7f41eaffd700 1 -- 172.21.15.96:0/2486173521 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f41e4029910 con 0x7f41f4100d90 2024-04-22T15:18:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:01 smithi196 ceph-mon[34717]: pgmap v7: 65 pgs: 65 active+clean; 74 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-04-22T15:18:02.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.030+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f41f402cef0 con 0x7f41e004a8d0 2024-04-22T15:18:02.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.042+0000 7f41eaffd700 1 -- 172.21.15.96:0/2486173521 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f41f402cef0 con 0x7f41e004a8d0 2024-04-22T15:18:02.045 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:18:02.045 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (6m) 116s ago 8m 24.7M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:18:02.045 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (8m) 116s ago 8m 7541k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:18:02.045 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (7m) 14s ago 7m 7591k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (8m) 116s ago 8m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (7m) 14s ago 7m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (6m) 116s ago 7m 84.1M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (2m) 116s ago 2m 18.7M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (2m) 116s ago 2m 13.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (2m) 14s ago 2m 20.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (2m) 14s ago 2m 19.7M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:9283,8765,8443 running (9m) 116s ago 9m 498M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 12e0357210ef 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (29s) 14s ago 6m 54.6M - 19.0.0-3223-g5883d494 3d983c26e278 8e82f21cd80c 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (9m) 116s ago 9m 44.3M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:18:02.046 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (6m) 14s ago 6m 37.2M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:18:02.047 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (8m) 116s ago 8m 14.6M - 1.5.0 0da6a335fe13 852e13a0c299 2024-04-22T15:18:02.047 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (6m) 14s ago 6m 16.6M - 1.5.0 0da6a335fe13 4d02817c7cd0 2024-04-22T15:18:02.047 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (6m) 116s ago 6m 47.5M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:18:02.047 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (5m) 116s ago 5m 48.9M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:18:02.047 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (5m) 116s ago 5m 53.4M 3946M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:18:02.047 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (4m) 14s ago 4m 48.3M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:18:02.047 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (4m) 14s ago 4m 45.1M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:18:02.047 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (3m) 14s ago 3m 45.3M 4970M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:18:02.047 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (6m) 116s ago 7m 55.2M - 2.43.0 a07b618ecd1d 87457032ed08 2024-04-22T15:18:02.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f41e004a8d0 msgr2=0x7f41e004cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:02.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f41e004a8d0 0x7f41e004cd90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f41f4100bf0 tx=0x7f41f000a3b0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 msgr2=0x7f41f41949b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:02.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 0x7f41f41949b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f41e40092e0 tx=0x7f41e400bd70 comp rx=0 tx=0).stop 2024-04-22T15:18:02.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 shutdown_connections 2024-04-22T15:18:02.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f41e004a8d0 0x7f41e004cd90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f40ffb90 0x7f41f4194470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 --2- 172.21.15.96:0/2486173521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f4100d90 0x7f41f41949b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 >> 172.21.15.96:0/2486173521 conn(0x7f41f40fb110 msgr2=0x7f41f41043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:02.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 shutdown_connections 2024-04-22T15:18:02.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.045+0000 7f41fc0cc700 1 -- 172.21.15.96:0/2486173521 wait complete. 2024-04-22T15:18:02.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.182+0000 7f4b6b56d700 1 -- 172.21.15.96:0/2063557220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b64100d20 msgr2=0x7f4b64101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:02.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.182+0000 7f4b6b56d700 1 --2- 172.21.15.96:0/2063557220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b64100d20 0x7f4b64101180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009910 tx=0x7f4b54009c20 comp rx=0 tx=0).stop 2024-04-22T15:18:02.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.182+0000 7f4b6b56d700 1 -- 172.21.15.96:0/2063557220 shutdown_connections 2024-04-22T15:18:02.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.182+0000 7f4b6b56d700 1 --2- 172.21.15.96:0/2063557220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b64100d20 0x7f4b64101180 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.182+0000 7f4b6b56d700 1 --2- 172.21.15.96:0/2063557220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b640ffb20 0x7f4b640fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.182+0000 7f4b6b56d700 1 -- 172.21.15.96:0/2063557220 >> 172.21.15.96:0/2063557220 conn(0x7f4b640fb0c0 msgr2=0x7f4b640fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:02.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.182+0000 7f4b6b56d700 1 -- 172.21.15.96:0/2063557220 shutdown_connections 2024-04-22T15:18:02.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.182+0000 7f4b6b56d700 1 -- 172.21.15.96:0/2063557220 wait complete. 2024-04-22T15:18:02.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.183+0000 7f4b6b56d700 1 Processor -- start 2024-04-22T15:18:02.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.183+0000 7f4b6b56d700 1 -- start start 2024-04-22T15:18:02.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b6b56d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b640ffb20 0x7f4b641965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b6b56d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b64100d20 0x7f4b64196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b6b56d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b64197140 con 0x7f4b640ffb20 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b6b56d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b64197280 con 0x7f4b64100d20 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b69309700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b640ffb20 0x7f4b641965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b69309700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b640ffb20 0x7f4b641965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59398/0 (socket says 172.21.15.96:59398) 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b69309700 1 -- 172.21.15.96:0/1089874914 learned_addr learned my addr 172.21.15.96:0/1089874914 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b68b08700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b64100d20 0x7f4b64196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b69309700 1 -- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b64100d20 msgr2=0x7f4b64196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b69309700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b64100d20 0x7f4b64196b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.184+0000 7f4b69309700 1 -- 172.21.15.96:0/1089874914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b540095f0 con 0x7f4b640ffb20 2024-04-22T15:18:02.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.185+0000 7f4b69309700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b640ffb20 0x7f4b641965e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000e8f0 tx=0x7f4b6000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:02.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.185+0000 7f4b5a7fc700 1 -- 172.21.15.96:0/1089874914 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b60009790 con 0x7f4b640ffb20 2024-04-22T15:18:02.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.185+0000 7f4b5a7fc700 1 -- 172.21.15.96:0/1089874914 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b6000c810 con 0x7f4b640ffb20 2024-04-22T15:18:02.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.185+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6419bc70 con 0x7f4b640ffb20 2024-04-22T15:18:02.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.185+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6419c1c0 con 0x7f4b640ffb20 2024-04-22T15:18:02.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.185+0000 7f4b5a7fc700 1 -- 172.21.15.96:0/1089874914 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b60003d20 con 0x7f4b640ffb20 2024-04-22T15:18:02.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.187+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b641906b0 con 0x7f4b640ffb20 2024-04-22T15:18:02.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.187+0000 7f4b5a7fc700 1 -- 172.21.15.96:0/1089874914 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 24) v1 ==== 46707+0+0 (secure 0 0 0) 0x7f4b600041f0 con 0x7f4b640ffb20 2024-04-22T15:18:02.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.187+0000 7f4b5a7fc700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f4b500398c0 0x7f4b5003bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:02.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.187+0000 7f4b68b08700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f4b500398c0 0x7f4b5003bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:02.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.188+0000 7f4b68b08700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f4b500398c0 0x7f4b5003bd80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400b6f0 tx=0x7f4b54005fb0 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:02.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.188+0000 7f4b5a7fc700 1 -- 172.21.15.96:0/1089874914 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f4b60014070 con 0x7f4b640ffb20 2024-04-22T15:18:02.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.194+0000 7f4b5a7fc700 1 -- 172.21.15.96:0/1089874914 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f4b6001ebd0 con 0x7f4b640ffb20 2024-04-22T15:18:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.385+0000 7f4b5a7fc700 1 -- 172.21.15.96:0/1089874914 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f4b6004bca0 con 0x7f4b640ffb20 2024-04-22T15:18:02.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:02 smithi096 ceph-mon[26688]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:02.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:02 smithi096 ceph-mon[26688]: Standby manager daemon smithi096.hhkvum started 2024-04-22T15:18:02.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:02 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.96:0/4100685720' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi096.hhkvum/crt"}]: dispatch 2024-04-22T15:18:02.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:02 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.96:0/4100685720' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:18:02.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:02 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.96:0/4100685720' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi096.hhkvum/key"}]: dispatch 2024-04-22T15:18:02.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:02 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.96:0/4100685720' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:18:02.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:02 smithi096 ceph-mon[26688]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:02.469 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:02 smithi096 ceph-mon[26688]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:02.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.600+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4b64108750 con 0x7f4b640ffb20 2024-04-22T15:18:02.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.601+0000 7f4b5a7fc700 1 -- 172.21.15.96:0/1089874914 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+841 (secure 0 0 0) 0x7f4b6005ded0 con 0x7f4b640ffb20 2024-04-22T15:18:02.604 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:18:02.604 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:18:02.604 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:18:02.605 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 13, 2024-04-22T15:18:02.606 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:18:02.606 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:18:02.606 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.603+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f4b500398c0 msgr2=0x7f4b5003bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.603+0000 7f4b6b56d700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f4b500398c0 0x7f4b5003bd80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400b6f0 tx=0x7f4b54005fb0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.603+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b640ffb20 msgr2=0x7f4b641965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.603+0000 7f4b6b56d700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b640ffb20 0x7f4b641965e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000e8f0 tx=0x7f4b6000ec00 comp rx=0 tx=0).stop 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.603+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 shutdown_connections 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.603+0000 7f4b6b56d700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f4b500398c0 0x7f4b5003bd80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.603+0000 7f4b6b56d700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b640ffb20 0x7f4b641965e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.603+0000 7f4b6b56d700 1 --2- 172.21.15.96:0/1089874914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b64100d20 0x7f4b64196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.603+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 >> 172.21.15.96:0/1089874914 conn(0x7f4b640fb0c0 msgr2=0x7f4b64104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.604+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 shutdown_connections 2024-04-22T15:18:02.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.604+0000 7f4b6b56d700 1 -- 172.21.15.96:0/1089874914 wait complete. 2024-04-22T15:18:02.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:02 smithi196 ceph-mon[34717]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:02.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:02 smithi196 ceph-mon[34717]: Standby manager daemon smithi096.hhkvum started 2024-04-22T15:18:02.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:02 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.96:0/4100685720' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi096.hhkvum/crt"}]: dispatch 2024-04-22T15:18:02.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:02 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.96:0/4100685720' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:18:02.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:02 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.96:0/4100685720' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi096.hhkvum/key"}]: dispatch 2024-04-22T15:18:02.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:02 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.96:0/4100685720' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:18:02.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:02 smithi196 ceph-mon[34717]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:02.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:02 smithi196 ceph-mon[34717]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:02.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.757+0000 7f8408697700 1 -- 172.21.15.96:0/2113575056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8400100d90 msgr2=0x7f84001011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:02.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.757+0000 7f8408697700 1 --2- 172.21.15.96:0/2113575056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8400100d90 0x7f84001011f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f83f0009910 tx=0x7f83f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:18:02.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.758+0000 7f8408697700 1 -- 172.21.15.96:0/2113575056 shutdown_connections 2024-04-22T15:18:02.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.758+0000 7f8408697700 1 --2- 172.21.15.96:0/2113575056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8400100d90 0x7f84001011f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.758+0000 7f8408697700 1 --2- 172.21.15.96:0/2113575056 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84000ffb90 0x7f84000fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.758+0000 7f8408697700 1 -- 172.21.15.96:0/2113575056 >> 172.21.15.96:0/2113575056 conn(0x7f84000fb110 msgr2=0x7f84000fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.758+0000 7f8408697700 1 -- 172.21.15.96:0/2113575056 shutdown_connections 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.758+0000 7f8408697700 1 -- 172.21.15.96:0/2113575056 wait complete. 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.758+0000 7f8408697700 1 Processor -- start 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.758+0000 7f8408697700 1 -- start start 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.759+0000 7f8408697700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84000ffb90 0x7f840019ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.759+0000 7f8408697700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8400100d90 0x7f840019d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.759+0000 7f8408697700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840019d7f0 con 0x7f84000ffb90 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.759+0000 7f8408697700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840019d960 con 0x7f8400100d90 2024-04-22T15:18:02.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.759+0000 7f8405c32700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8400100d90 0x7f840019d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:02.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.759+0000 7f8405c32700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8400100d90 0x7f840019d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:43966/0 (socket says 172.21.15.96:43966) 2024-04-22T15:18:02.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.759+0000 7f8405c32700 1 -- 172.21.15.96:0/1944437858 learned_addr learned my addr 172.21.15.96:0/1944437858 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:02.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.759+0000 7f8406433700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84000ffb90 0x7f840019ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:02.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.760+0000 7f8406433700 1 -- 172.21.15.96:0/1944437858 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8400100d90 msgr2=0x7f840019d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:02.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.760+0000 7f8406433700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8400100d90 0x7f840019d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:02.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.760+0000 7f8406433700 1 -- 172.21.15.96:0/1944437858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f00095f0 con 0x7f84000ffb90 2024-04-22T15:18:02.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.760+0000 7f8406433700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84000ffb90 0x7f840019ccb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e940 tx=0x7f83fc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:02.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.760+0000 7f83f77fe700 1 -- 172.21.15.96:0/1944437858 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83fc004200 con 0x7f84000ffb90 2024-04-22T15:18:02.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.760+0000 7f83f77fe700 1 -- 172.21.15.96:0/1944437858 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83fc00cc80 con 0x7f84000ffb90 2024-04-22T15:18:02.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.760+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84001a23a0 con 0x7f84000ffb90 2024-04-22T15:18:02.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.760+0000 7f83f77fe700 1 -- 172.21.15.96:0/1944437858 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83fc003d00 con 0x7f84000ffb90 2024-04-22T15:18:02.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.760+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84001a28f0 con 0x7f84000ffb90 2024-04-22T15:18:02.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.762+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840004ebb0 con 0x7f84000ffb90 2024-04-22T15:18:02.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.762+0000 7f83f77fe700 1 -- 172.21.15.96:0/1944437858 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f83fc00c7c0 con 0x7f84000ffb90 2024-04-22T15:18:02.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.763+0000 7f83f77fe700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f83ec06dc90 0x7f83ec070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:02.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.763+0000 7f83f77fe700 1 -- 172.21.15.96:0/1944437858 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f83fc014070 con 0x7f84000ffb90 2024-04-22T15:18:02.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.763+0000 7f8405c32700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f83ec06dc90 0x7f83ec070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:02.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.763+0000 7f8405c32700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f83ec06dc90 0x7f83ec070150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f840019e170 tx=0x7f83f0005a90 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:02.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:02.768+0000 7f83f77fe700 1 -- 172.21.15.96:0/1944437858 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f83fc056d40 con 0x7f84000ffb90 2024-04-22T15:18:03.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.154+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8400068d40 con 0x7f83ec06dc90 2024-04-22T15:18:03.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.156+0000 7f83f77fe700 1 -- 172.21.15.96:0/1944437858 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f8400068d40 con 0x7f83ec06dc90 2024-04-22T15:18:03.159 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:18:03.159 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:18:03.160 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:18:03.160 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:18:03.160 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [], 2024-04-22T15:18:03.160 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "1/2 daemons upgraded", 2024-04-22T15:18:03.160 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "", 2024-04-22T15:18:03.160 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:18:03.160 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:18:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.159+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f83ec06dc90 msgr2=0x7f83ec070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.159+0000 7f8408697700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f83ec06dc90 0x7f83ec070150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f840019e170 tx=0x7f83f0005a90 comp rx=0 tx=0).stop 2024-04-22T15:18:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.159+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84000ffb90 msgr2=0x7f840019ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.159+0000 7f8408697700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84000ffb90 0x7f840019ccb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e940 tx=0x7f83fc00ec50 comp rx=0 tx=0).stop 2024-04-22T15:18:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.160+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 shutdown_connections 2024-04-22T15:18:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.160+0000 7f8408697700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f83ec06dc90 0x7f83ec070150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.160+0000 7f8408697700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84000ffb90 0x7f840019ccb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.160+0000 7f8408697700 1 --2- 172.21.15.96:0/1944437858 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8400100d90 0x7f840019d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.160+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 >> 172.21.15.96:0/1944437858 conn(0x7f84000fb110 msgr2=0x7f8400068630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.160+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 shutdown_connections 2024-04-22T15:18:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:03.160+0000 7f8408697700 1 -- 172.21.15.96:0/1944437858 wait complete. 2024-04-22T15:18:03.695 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:03 smithi196 ceph-mon[34717]: pgmap v8: 65 pgs: 65 active+clean; 74 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 100 op/s 2024-04-22T15:18:03.695 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:03 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:18:03.695 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:03 smithi196 ceph-mon[34717]: mgrmap e25: smithi196.gdaegc(active, since 10s), standbys: smithi096.hhkvum 2024-04-22T15:18:03.695 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:03 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1089874914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:18:03.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:03 smithi096 ceph-mon[26688]: pgmap v8: 65 pgs: 65 active+clean; 74 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 100 op/s 2024-04-22T15:18:03.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:03 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:18:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:03 smithi096 ceph-mon[26688]: mgrmap e25: smithi196.gdaegc(active, since 10s), standbys: smithi096.hhkvum 2024-04-22T15:18:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:03 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1089874914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:18:04.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:04 smithi196 ceph-mon[34717]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:04.677 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:04 smithi096 ceph-mon[26688]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:05 smithi096 ceph-mon[26688]: pgmap v9: 65 pgs: 65 active+clean; 94 MiB data, 478 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 148 op/s 2024-04-22T15:18:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:05 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:05 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:05 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:05 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:05 smithi196 ceph-mon[34717]: pgmap v9: 65 pgs: 65 active+clean; 94 MiB data, 478 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 148 op/s 2024-04-22T15:18:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:05 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:05 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:05 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:05 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:06 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:06 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:06.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:06 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:06.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:06 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:07.821 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: pgmap v10: 65 pgs: 65 active+clean; 94 MiB data, 478 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-04-22T15:18:07.821 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:07.821 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:07.821 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:07.821 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:07.821 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:07.821 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:07.822 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:07.822 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:07.822 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: Adjusting osd_memory_target on smithi096 to 1216M 2024-04-22T15:18:07.822 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:07.822 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:07 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: pgmap v10: 65 pgs: 65 active+clean; 94 MiB data, 478 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: Adjusting osd_memory_target on smithi096 to 1216M 2024-04-22T15:18:08.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:08.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:07 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:18:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: pgmap v11: 65 pgs: 65 active+clean; 101 MiB data, 508 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 170 op/s 2024-04-22T15:18:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:09.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:09.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:18:09.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:09 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:18:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: pgmap v11: 65 pgs: 65 active+clean; 101 MiB data, 508 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 170 op/s 2024-04-22T15:18:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:10.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:10.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:10.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:10.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:18:10.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:10.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:18:10.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:09 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:18:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:10 smithi196 ceph-mon[34717]: Adjusting osd_memory_target on smithi196 to 2240M 2024-04-22T15:18:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:10 smithi196 ceph-mon[34717]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T15:18:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:10 smithi196 ceph-mon[34717]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-22T15:18:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:10 smithi196 ceph-mon[34717]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:18:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:10 smithi196 ceph-mon[34717]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:18:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:10 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:11.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:10 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:10 smithi096 ceph-mon[26688]: Adjusting osd_memory_target on smithi196 to 2240M 2024-04-22T15:18:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:10 smithi096 ceph-mon[26688]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T15:18:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:10 smithi096 ceph-mon[26688]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-22T15:18:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:10 smithi096 ceph-mon[26688]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:18:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:10 smithi096 ceph-mon[26688]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:18:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:10 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:10 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:12.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:11 smithi096 ceph-mon[26688]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:18:12.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:11 smithi096 ceph-mon[26688]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:18:12.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:11 smithi096 ceph-mon[26688]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:18:12.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:11 smithi096 ceph-mon[26688]: pgmap v12: 65 pgs: 65 active+clean; 116 MiB data, 570 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-04-22T15:18:12.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:11 smithi096 ceph-mon[26688]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:18:12.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:11 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:12.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:11 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:12.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:11 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:12.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:11 smithi196 ceph-mon[34717]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:18:12.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:11 smithi196 ceph-mon[34717]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:18:12.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:11 smithi196 ceph-mon[34717]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:18:12.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:11 smithi196 ceph-mon[34717]: pgmap v12: 65 pgs: 65 active+clean; 116 MiB data, 570 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-04-22T15:18:12.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:11 smithi196 ceph-mon[34717]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:18:12.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:11 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:12.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:11 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:12.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:11 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:13.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:12 smithi096 ceph-mon[26688]: Reconfiguring prometheus.smithi096 (dependencies changed)... 2024-04-22T15:18:13.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:12 smithi096 ceph-mon[26688]: Reconfiguring daemon prometheus.smithi096 on smithi096 2024-04-22T15:18:13.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:12 smithi196 ceph-mon[34717]: Reconfiguring prometheus.smithi096 (dependencies changed)... 2024-04-22T15:18:13.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:12 smithi196 ceph-mon[34717]: Reconfiguring daemon prometheus.smithi096 on smithi096 2024-04-22T15:18:14.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:13 smithi096 ceph-mon[26688]: pgmap v13: 65 pgs: 65 active+clean; 116 MiB data, 570 MiB used, 536 GiB / 536 GiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 168 op/s 2024-04-22T15:18:14.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:13 smithi196 ceph-mon[34717]: pgmap v13: 65 pgs: 65 active+clean; 116 MiB data, 570 MiB used, 536 GiB / 536 GiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 168 op/s 2024-04-22T15:18:16.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:15 smithi096 ceph-mon[26688]: pgmap v14: 65 pgs: 65 active+clean; 134 MiB data, 640 MiB used, 536 GiB / 536 GiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 260 op/s 2024-04-22T15:18:16.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:15 smithi196 ceph-mon[34717]: pgmap v14: 65 pgs: 65 active+clean; 134 MiB data, 640 MiB used, 536 GiB / 536 GiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 260 op/s 2024-04-22T15:18:18.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:17 smithi096 ceph-mon[26688]: pgmap v15: 65 pgs: 65 active+clean; 134 MiB data, 640 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 199 op/s 2024-04-22T15:18:18.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:17 smithi196 ceph-mon[34717]: pgmap v15: 65 pgs: 65 active+clean; 134 MiB data, 640 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 199 op/s 2024-04-22T15:18:19.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:19 smithi096 ceph-mon[26688]: pgmap v16: 65 pgs: 65 active+clean; 135 MiB data, 652 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 231 op/s 2024-04-22T15:18:20.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:19 smithi196 ceph-mon[34717]: pgmap v16: 65 pgs: 65 active+clean; 135 MiB data, 652 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 231 op/s 2024-04-22T15:18:22.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:21 smithi096 ceph-mon[26688]: pgmap v17: 65 pgs: 65 active+clean; 145 MiB data, 671 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 268 op/s 2024-04-22T15:18:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:21 smithi196 ceph-mon[34717]: pgmap v17: 65 pgs: 65 active+clean; 145 MiB data, 671 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 268 op/s 2024-04-22T15:18:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:22 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:18:23.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:22 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:18:24.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:23 smithi096 ceph-mon[26688]: pgmap v18: 65 pgs: 65 active+clean; 145 MiB data, 671 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 195 op/s 2024-04-22T15:18:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:23 smithi196 ceph-mon[34717]: pgmap v18: 65 pgs: 65 active+clean; 145 MiB data, 671 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 195 op/s 2024-04-22T15:18:26.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:25 smithi096 ceph-mon[26688]: pgmap v19: 65 pgs: 65 active+clean; 167 MiB data, 708 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 341 op/s 2024-04-22T15:18:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:25 smithi196 ceph-mon[34717]: pgmap v19: 65 pgs: 65 active+clean; 167 MiB data, 708 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 341 op/s 2024-04-22T15:18:26.811 INFO:tasks.workunit.client.0.smithi096.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-22T15:18:26.817 INFO:tasks.workunit.client.0.smithi096.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T15:18:26.817 INFO:tasks.workunit.client.0.smithi096.stderr:+ make 2024-04-22T15:18:27.506 INFO:tasks.workunit.client.0.smithi096.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-22T15:18:28.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:27 smithi096 ceph-mon[26688]: pgmap v20: 65 pgs: 65 active+clean; 167 MiB data, 708 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-04-22T15:18:28.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:27 smithi196 ceph-mon[34717]: pgmap v20: 65 pgs: 65 active+clean; 167 MiB data, 708 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-04-22T15:18:29.721 INFO:tasks.workunit.client.0.smithi096.stderr:++ readlink -f fsstress 2024-04-22T15:18:29.723 INFO:tasks.workunit.client.0.smithi096.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-22T15:18:29.723 INFO:tasks.workunit.client.0.smithi096.stderr:+ popd 2024-04-22T15:18:29.724 INFO:tasks.workunit.client.0.smithi096.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T15:18:29.724 INFO:tasks.workunit.client.0.smithi096.stderr:+ popd 2024-04-22T15:18:29.725 INFO:tasks.workunit.client.0.smithi096.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-22T15:18:29.725 INFO:tasks.workunit.client.0.smithi096.stderr:++ mktemp -d -p . 2024-04-22T15:18:29.727 INFO:tasks.workunit.client.0.smithi096.stderr:+ T=./tmp.jlIWo8x58o 2024-04-22T15:18:29.728 INFO:tasks.workunit.client.0.smithi096.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.jlIWo8x58o -l 1 -n 1000 -p 10 -v 2024-04-22T15:18:29.731 INFO:tasks.workunit.client.0.smithi096.stdout:seed = 1714249358 2024-04-22T15:18:29.738 INFO:tasks.workunit.client.0.smithi096.stdout:2/0: chown . 53062 1 2024-04-22T15:18:29.738 INFO:tasks.workunit.client.0.smithi096.stdout:2/1: dread - no filename 2024-04-22T15:18:29.738 INFO:tasks.workunit.client.0.smithi096.stdout:2/2: stat - no entries 2024-04-22T15:18:29.738 INFO:tasks.workunit.client.0.smithi096.stdout:0/0: creat f0 x:0 0 0 2024-04-22T15:18:29.739 INFO:tasks.workunit.client.0.smithi096.stdout:1/0: mkdir d0 0 2024-04-22T15:18:29.739 INFO:tasks.workunit.client.0.smithi096.stdout:1/1: truncate - no filename 2024-04-22T15:18:29.739 INFO:tasks.workunit.client.0.smithi096.stdout:1/2: write - no filename 2024-04-22T15:18:29.739 INFO:tasks.workunit.client.0.smithi096.stdout:1/3: dwrite - no filename 2024-04-22T15:18:29.739 INFO:tasks.workunit.client.0.smithi096.stdout:1/4: chown d0 610 1 2024-04-22T15:18:29.740 INFO:tasks.workunit.client.0.smithi096.stdout:3/0: stat - no entries 2024-04-22T15:18:29.741 INFO:tasks.workunit.client.0.smithi096.stdout:2/3: creat f0 x:0 0 0 2024-04-22T15:18:29.743 INFO:tasks.workunit.client.0.smithi096.stdout:4/0: write - no filename 2024-04-22T15:18:29.743 INFO:tasks.workunit.client.0.smithi096.stdout:4/1: link - no file 2024-04-22T15:18:29.743 INFO:tasks.workunit.client.0.smithi096.stdout:1/5: symlink d0/l1 0 2024-04-22T15:18:29.744 INFO:tasks.workunit.client.0.smithi096.stdout:1/6: dread - no filename 2024-04-22T15:18:29.749 INFO:tasks.workunit.client.0.smithi096.stdout:3/1: symlink l0 0 2024-04-22T15:18:29.749 INFO:tasks.workunit.client.0.smithi096.stdout:3/2: dread - no filename 2024-04-22T15:18:29.749 INFO:tasks.workunit.client.0.smithi096.stdout:1/7: unlink d0/l1 0 2024-04-22T15:18:29.750 INFO:tasks.workunit.client.0.smithi096.stdout:1/8: dread - no filename 2024-04-22T15:18:29.752 INFO:tasks.workunit.client.0.smithi096.stdout:6/0: chown . 26 1 2024-04-22T15:18:29.752 INFO:tasks.workunit.client.0.smithi096.stdout:6/1: dread - no filename 2024-04-22T15:18:29.752 INFO:tasks.workunit.client.0.smithi096.stdout:4/2: creat f0 x:0 0 0 2024-04-22T15:18:29.754 INFO:tasks.workunit.client.0.smithi096.stdout:2/4: dwrite f0 [0,4194304] 0 2024-04-22T15:18:29.754 INFO:tasks.workunit.client.0.smithi096.stdout:4/3: chown f0 80 1 2024-04-22T15:18:29.756 INFO:tasks.workunit.client.0.smithi096.stdout:5/0: symlink l0 0 2024-04-22T15:18:29.756 INFO:tasks.workunit.client.0.smithi096.stdout:7/0: unlink - no file 2024-04-22T15:18:29.760 INFO:tasks.workunit.client.0.smithi096.stdout:5/1: creat f1 x:0 0 0 2024-04-22T15:18:29.764 INFO:tasks.workunit.client.0.smithi096.stdout:6/2: creat f0 x:0 0 0 2024-04-22T15:18:29.765 INFO:tasks.workunit.client.0.smithi096.stdout:2/5: link f0 f1 0 2024-04-22T15:18:29.766 INFO:tasks.workunit.client.0.smithi096.stdout:6/3: chown f0 10406321 1 2024-04-22T15:18:29.766 INFO:tasks.workunit.client.0.smithi096.stdout:4/4: dwrite f0 [0,4194304] 0 2024-04-22T15:18:29.768 INFO:tasks.workunit.client.0.smithi096.stdout:6/4: dread - f0 zero size 2024-04-22T15:18:29.768 INFO:tasks.workunit.client.0.smithi096.stdout:3/3: link l0 l1 0 2024-04-22T15:18:29.772 INFO:tasks.workunit.client.0.smithi096.stdout:5/2: creat f2 x:0 0 0 2024-04-22T15:18:29.778 INFO:tasks.workunit.client.0.smithi096.stdout:7/1: mknod c0 0 2024-04-22T15:18:29.780 INFO:tasks.workunit.client.0.smithi096.stdout:9/0: write - no filename 2024-04-22T15:18:29.780 INFO:tasks.workunit.client.0.smithi096.stdout:9/1: fsync - no filename 2024-04-22T15:18:29.781 INFO:tasks.workunit.client.0.smithi096.stdout:9/2: read - no filename 2024-04-22T15:18:29.781 INFO:tasks.workunit.client.0.smithi096.stdout:9/3: dwrite - no filename 2024-04-22T15:18:29.781 INFO:tasks.workunit.client.0.smithi096.stdout:9/4: dwrite - no filename 2024-04-22T15:18:29.781 INFO:tasks.workunit.client.0.smithi096.stdout:9/5: dwrite - no filename 2024-04-22T15:18:29.781 INFO:tasks.workunit.client.0.smithi096.stdout:9/6: truncate - no filename 2024-04-22T15:18:29.781 INFO:tasks.workunit.client.0.smithi096.stdout:9/7: read - no filename 2024-04-22T15:18:29.785 INFO:tasks.workunit.client.0.smithi096.stdout:2/6: mknod c2 0 2024-04-22T15:18:29.788 INFO:tasks.workunit.client.0.smithi096.stdout:6/5: creat f1 x:0 0 0 2024-04-22T15:18:29.789 INFO:tasks.workunit.client.0.smithi096.stdout:3/4: creat f2 x:0 0 0 2024-04-22T15:18:29.790 INFO:tasks.workunit.client.0.smithi096.stdout:5/3: mknod c3 0 2024-04-22T15:18:29.792 INFO:tasks.workunit.client.0.smithi096.stdout:7/2: unlink c0 0 2024-04-22T15:18:29.792 INFO:tasks.workunit.client.0.smithi096.stdout:7/3: fdatasync - no filename 2024-04-22T15:18:29.792 INFO:tasks.workunit.client.0.smithi096.stdout:7/4: dwrite - no filename 2024-04-22T15:18:29.794 INFO:tasks.workunit.client.0.smithi096.stdout:8/0: symlink l0 0 2024-04-22T15:18:29.794 INFO:tasks.workunit.client.0.smithi096.stdout:8/1: fdatasync - no filename 2024-04-22T15:18:29.794 INFO:tasks.workunit.client.0.smithi096.stdout:8/2: write - no filename 2024-04-22T15:18:29.796 INFO:tasks.workunit.client.0.smithi096.stdout:2/7: unlink f1 0 2024-04-22T15:18:29.798 INFO:tasks.workunit.client.0.smithi096.stdout:6/6: creat f2 x:0 0 0 2024-04-22T15:18:29.800 INFO:tasks.workunit.client.0.smithi096.stdout:5/4: mkdir d4 0 2024-04-22T15:18:29.801 INFO:tasks.workunit.client.0.smithi096.stdout:7/5: symlink l1 0 2024-04-22T15:18:29.801 INFO:tasks.workunit.client.0.smithi096.stdout:7/6: dread - no filename 2024-04-22T15:18:29.801 INFO:tasks.workunit.client.0.smithi096.stdout:7/7: stat l1 0 2024-04-22T15:18:29.802 INFO:tasks.workunit.client.0.smithi096.stdout:9/8: mknod c0 0 2024-04-22T15:18:29.802 INFO:tasks.workunit.client.0.smithi096.stdout:9/9: stat c0 0 2024-04-22T15:18:29.802 INFO:tasks.workunit.client.0.smithi096.stdout:9/10: fsync - no filename 2024-04-22T15:18:29.803 INFO:tasks.workunit.client.0.smithi096.stdout:2/8: mkdir d3 0 2024-04-22T15:18:29.807 INFO:tasks.workunit.client.0.smithi096.stdout:6/7: mkdir d3 0 2024-04-22T15:18:29.807 INFO:tasks.workunit.client.0.smithi096.stdout:6/8: read - f0 zero size 2024-04-22T15:18:29.809 INFO:tasks.workunit.client.0.smithi096.stdout:9/11: mkdir d1 0 2024-04-22T15:18:29.815 INFO:tasks.workunit.client.0.smithi096.stdout:5/5: creat d4/f5 x:0 0 0 2024-04-22T15:18:29.818 INFO:tasks.workunit.client.0.smithi096.stdout:8/3: link l0 l1 0 2024-04-22T15:18:29.822 INFO:tasks.workunit.client.0.smithi096.stdout:6/9: creat d3/f4 x:0 0 0 2024-04-22T15:18:29.823 INFO:tasks.workunit.client.0.smithi096.stdout:6/10: dread - f0 zero size 2024-04-22T15:18:29.823 INFO:tasks.workunit.client.0.smithi096.stdout:5/6: creat d4/f6 x:0 0 0 2024-04-22T15:18:29.824 INFO:tasks.workunit.client.0.smithi096.stdout:9/12: mknod d1/c2 0 2024-04-22T15:18:29.825 INFO:tasks.workunit.client.0.smithi096.stdout:8/4: symlink l2 0 2024-04-22T15:18:29.825 INFO:tasks.workunit.client.0.smithi096.stdout:8/5: dread - no filename 2024-04-22T15:18:29.825 INFO:tasks.workunit.client.0.smithi096.stdout:8/6: dread - no filename 2024-04-22T15:18:29.826 INFO:tasks.workunit.client.0.smithi096.stdout:6/11: write d3/f4 [92455,130333] 0 2024-04-22T15:18:29.827 INFO:tasks.workunit.client.0.smithi096.stdout:6/12: stat d3 0 2024-04-22T15:18:29.829 INFO:tasks.workunit.client.0.smithi096.stdout:2/9: dwrite f0 [0,4194304] 0 2024-04-22T15:18:29.830 INFO:tasks.workunit.client.0.smithi096.stdout:8/7: creat f3 x:0 0 0 2024-04-22T15:18:29.830 INFO:tasks.workunit.client.0.smithi096.stdout:6/13: write d3/f4 [473908,102770] 0 2024-04-22T15:18:29.833 INFO:tasks.workunit.client.0.smithi096.stdout:8/8: dread - f3 zero size 2024-04-22T15:18:29.838 INFO:tasks.workunit.client.0.smithi096.stdout:8/9: mkdir d4 0 2024-04-22T15:18:29.838 INFO:tasks.workunit.client.0.smithi096.stdout:5/7: dwrite d4/f6 [0,4194304] 0 2024-04-22T15:18:29.839 INFO:tasks.workunit.client.0.smithi096.stdout:5/8: fdatasync f1 0 2024-04-22T15:18:29.839 INFO:tasks.workunit.client.0.smithi096.stdout:6/14: symlink d3/l5 0 2024-04-22T15:18:29.840 INFO:tasks.workunit.client.0.smithi096.stdout:5/9: dread - f2 zero size 2024-04-22T15:18:29.844 INFO:tasks.workunit.client.0.smithi096.stdout:2/10: link f0 d3/f4 0 2024-04-22T15:18:29.850 INFO:tasks.workunit.client.0.smithi096.stdout:5/10: creat d4/f7 x:0 0 0 2024-04-22T15:18:29.851 INFO:tasks.workunit.client.0.smithi096.stdout:6/15: dwrite f2 [0,4194304] 0 2024-04-22T15:18:29.852 INFO:tasks.workunit.client.0.smithi096.stdout:5/11: dread - f2 zero size 2024-04-22T15:18:29.853 INFO:tasks.workunit.client.0.smithi096.stdout:5/12: write f2 [724413,87420] 0 2024-04-22T15:18:29.854 INFO:tasks.workunit.client.0.smithi096.stdout:8/10: link f3 d4/f5 0 2024-04-22T15:18:29.856 INFO:tasks.workunit.client.0.smithi096.stdout:6/16: creat d3/f6 x:0 0 0 2024-04-22T15:18:29.857 INFO:tasks.workunit.client.0.smithi096.stdout:5/13: write d4/f5 [762725,93604] 0 2024-04-22T15:18:29.859 INFO:tasks.workunit.client.0.smithi096.stdout:6/17: mkdir d3/d7 0 2024-04-22T15:18:29.861 INFO:tasks.workunit.client.0.smithi096.stdout:6/18: stat d3/l5 0 2024-04-22T15:18:29.861 INFO:tasks.workunit.client.0.smithi096.stdout:5/14: dread - d4/f7 zero size 2024-04-22T15:18:29.861 INFO:tasks.workunit.client.0.smithi096.stdout:2/11: dwrite d3/f4 [0,4194304] 0 2024-04-22T15:18:29.862 INFO:tasks.workunit.client.0.smithi096.stdout:8/11: rename l1 to d4/l6 0 2024-04-22T15:18:29.862 INFO:tasks.workunit.client.0.smithi096.stdout:6/19: readlink d3/l5 0 2024-04-22T15:18:29.863 INFO:tasks.workunit.client.0.smithi096.stdout:5/15: chown f1 15 1 2024-04-22T15:18:29.864 INFO:tasks.workunit.client.0.smithi096.stdout:2/12: write d3/f4 [2179111,102231] 0 2024-04-22T15:18:29.864 INFO:tasks.workunit.client.0.smithi096.stdout:6/20: write f1 [812667,101889] 0 2024-04-22T15:18:29.865 INFO:tasks.workunit.client.0.smithi096.stdout:6/21: write d3/f6 [787098,23325] 0 2024-04-22T15:18:29.867 INFO:tasks.workunit.client.0.smithi096.stdout:6/22: read f2 [3754608,102280] 0 2024-04-22T15:18:29.867 INFO:tasks.workunit.client.0.smithi096.stdout:8/12: write d4/f5 [486220,38961] 0 2024-04-22T15:18:29.869 INFO:tasks.workunit.client.0.smithi096.stdout:6/23: truncate d3/f6 1798757 0 2024-04-22T15:18:29.903 INFO:tasks.workunit.client.0.smithi096.stdout:8/13: creat d4/f7 x:0 0 0 2024-04-22T15:18:29.903 INFO:tasks.workunit.client.0.smithi096.stdout:8/14: dread - d4/f7 zero size 2024-04-22T15:18:29.906 INFO:tasks.workunit.client.0.smithi096.stdout:8/15: creat d4/f8 x:0 0 0 2024-04-22T15:18:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:29 smithi096 ceph-mon[26688]: pgmap v21: 65 pgs: 65 active+clean; 190 MiB data, 785 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 4.9 MiB/s wr, 377 op/s 2024-04-22T15:18:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:29 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:29 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:29 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:18:29.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:29 smithi096 ceph-mon[26688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:18:29.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:29 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:18:29.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:29 smithi096 ceph-mon[26688]: Upgrade: Need to upgrade myself (mgr.smithi196.gdaegc) 2024-04-22T15:18:29.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:29 smithi096 ceph-mon[26688]: Upgrade: Need to upgrade myself (mgr.smithi196.gdaegc) 2024-04-22T15:18:30.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:29 smithi196 ceph-mon[34717]: pgmap v21: 65 pgs: 65 active+clean; 190 MiB data, 785 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 4.9 MiB/s wr, 377 op/s 2024-04-22T15:18:30.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:29 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:30.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:29 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:30.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:29 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:18:30.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:29 smithi196 ceph-mon[34717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:18:30.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:29 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:18:30.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:29 smithi196 ceph-mon[34717]: Upgrade: Need to upgrade myself (mgr.smithi196.gdaegc) 2024-04-22T15:18:30.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:29 smithi196 ceph-mon[34717]: Upgrade: Need to upgrade myself (mgr.smithi196.gdaegc) 2024-04-22T15:18:30.542 INFO:tasks.workunit.client.0.smithi096.stdout:0/1: getdents . 0 2024-04-22T15:18:30.542 INFO:tasks.workunit.client.0.smithi096.stdout:0/2: stat f0 0 2024-04-22T15:18:30.542 INFO:tasks.workunit.client.0.smithi096.stdout:0/3: dread - f0 zero size 2024-04-22T15:18:30.554 INFO:tasks.workunit.client.0.smithi096.stdout:4/5: getdents . 0 2024-04-22T15:18:30.555 INFO:tasks.workunit.client.0.smithi096.stdout:4/6: write f0 [1050688,96933] 0 2024-04-22T15:18:30.558 INFO:tasks.workunit.client.0.smithi096.stdout:4/7: fsync f0 0 2024-04-22T15:18:30.571 INFO:tasks.workunit.client.0.smithi096.stdout:4/8: dwrite f0 [4194304,4194304] 0 2024-04-22T15:18:30.572 INFO:tasks.workunit.client.0.smithi096.stdout:4/9: write f0 [2528559,22503] 0 2024-04-22T15:18:30.573 INFO:tasks.workunit.client.0.smithi096.stdout:3/5: fsync f2 0 2024-04-22T15:18:30.579 INFO:tasks.workunit.client.0.smithi096.stdout:6/24: fsync f2 0 2024-04-22T15:18:30.588 INFO:tasks.workunit.client.0.smithi096.stdout:6/25: dwrite f0 [0,4194304] 0 2024-04-22T15:18:30.597 INFO:tasks.workunit.client.0.smithi096.stdout:6/26: dwrite f2 [0,4194304] 0 2024-04-22T15:18:30.601 INFO:tasks.workunit.client.0.smithi096.stdout:9/13: getdents d1 0 2024-04-22T15:18:30.624 INFO:tasks.workunit.client.0.smithi096.stdout:0/4: mknod c1 0 2024-04-22T15:18:30.633 INFO:tasks.workunit.client.0.smithi096.stdout:4/10: rename f0 to f1 0 2024-04-22T15:18:30.636 INFO:tasks.workunit.client.0.smithi096.stdout:3/6: symlink l3 0 2024-04-22T15:18:30.639 INFO:tasks.workunit.client.0.smithi096.stdout:9/14: creat d1/f3 x:0 0 0 2024-04-22T15:18:30.642 INFO:tasks.workunit.client.0.smithi096.stdout:1/9: rmdir d0 0 2024-04-22T15:18:30.642 INFO:tasks.workunit.client.0.smithi096.stdout:1/10: rename - no filename 2024-04-22T15:18:30.647 INFO:tasks.workunit.client.0.smithi096.stdout:3/7: chown l1 828 1 2024-04-22T15:18:30.647 INFO:tasks.workunit.client.0.smithi096.stdout:3/8: stat l0 0 2024-04-22T15:18:30.649 INFO:tasks.workunit.client.0.smithi096.stdout:6/27: mknod d3/d7/c8 0 2024-04-22T15:18:30.652 INFO:tasks.workunit.client.0.smithi096.stdout:1/11: creat f2 x:0 0 0 2024-04-22T15:18:30.653 INFO:tasks.workunit.client.0.smithi096.stdout:6/28: dread f2 [0,4194304] 0 2024-04-22T15:18:30.654 INFO:tasks.workunit.client.0.smithi096.stdout:3/9: mknod c4 0 2024-04-22T15:18:30.655 INFO:tasks.workunit.client.0.smithi096.stdout:3/10: write f2 [498796,74900] 0 2024-04-22T15:18:30.657 INFO:tasks.workunit.client.0.smithi096.stdout:0/5: link c1 c2 0 2024-04-22T15:18:30.660 INFO:tasks.workunit.client.0.smithi096.stdout:1/12: mknod c3 0 2024-04-22T15:18:30.669 INFO:tasks.workunit.client.0.smithi096.stdout:1/13: rename c3 to c4 0 2024-04-22T15:18:30.670 INFO:tasks.workunit.client.0.smithi096.stdout:1/14: write f2 [1025780,10224] 0 2024-04-22T15:18:30.671 INFO:tasks.workunit.client.0.smithi096.stdout:0/6: chown c2 3 1 2024-04-22T15:18:30.677 INFO:tasks.workunit.client.0.smithi096.stdout:1/15: symlink l5 0 2024-04-22T15:18:30.681 INFO:tasks.workunit.client.0.smithi096.stdout:0/7: creat f3 x:0 0 0 2024-04-22T15:18:30.682 INFO:tasks.workunit.client.0.smithi096.stdout:1/16: mkdir d6 0 2024-04-22T15:18:30.683 INFO:tasks.workunit.client.0.smithi096.stdout:1/17: chown f2 252195243 1 2024-04-22T15:18:30.685 INFO:tasks.workunit.client.0.smithi096.stdout:5/16: truncate f2 358121 0 2024-04-22T15:18:30.688 INFO:tasks.workunit.client.0.smithi096.stdout:5/17: rmdir d4 39 2024-04-22T15:18:30.692 INFO:tasks.workunit.client.0.smithi096.stdout:1/18: link f2 d6/f7 0 2024-04-22T15:18:30.693 INFO:tasks.workunit.client.0.smithi096.stdout:8/16: getdents d4 0 2024-04-22T15:18:30.693 INFO:tasks.workunit.client.0.smithi096.stdout:8/17: chown l2 3232 1 2024-04-22T15:18:30.694 INFO:tasks.workunit.client.0.smithi096.stdout:8/18: fsync d4/f8 0 2024-04-22T15:18:30.694 INFO:tasks.workunit.client.0.smithi096.stdout:8/19: dread - d4/f7 zero size 2024-04-22T15:18:30.703 INFO:tasks.workunit.client.0.smithi096.stdout:1/19: creat d6/f8 x:0 0 0 2024-04-22T15:18:30.705 INFO:tasks.workunit.client.0.smithi096.stdout:8/20: dwrite d4/f7 [0,4194304] 0 2024-04-22T15:18:30.706 INFO:tasks.workunit.client.0.smithi096.stdout:5/18: dwrite d4/f5 [0,4194304] 0 2024-04-22T15:18:30.709 INFO:tasks.workunit.client.0.smithi096.stdout:8/21: symlink d4/l9 0 2024-04-22T15:18:30.714 INFO:tasks.workunit.client.0.smithi096.stdout:8/22: readlink l0 0 2024-04-22T15:18:30.715 INFO:tasks.workunit.client.0.smithi096.stdout:1/20: dwrite d6/f8 [0,4194304] 0 2024-04-22T15:18:30.716 INFO:tasks.workunit.client.0.smithi096.stdout:1/21: readlink l5 0 2024-04-22T15:18:30.719 INFO:tasks.workunit.client.0.smithi096.stdout:8/23: creat d4/fa x:0 0 0 2024-04-22T15:18:30.729 INFO:tasks.workunit.client.0.smithi096.stdout:8/24: dwrite d4/f7 [0,4194304] 0 2024-04-22T15:18:30.730 INFO:tasks.workunit.client.0.smithi096.stdout:1/22: dwrite d6/f7 [0,4194304] 0 2024-04-22T15:18:30.734 INFO:tasks.workunit.client.0.smithi096.stdout:8/25: rename l0 to d4/lb 0 2024-04-22T15:18:30.737 INFO:tasks.workunit.client.0.smithi096.stdout:8/26: unlink d4/f5 0 2024-04-22T15:18:30.737 INFO:tasks.workunit.client.0.smithi096.stdout:8/27: dread - d4/fa zero size 2024-04-22T15:18:30.738 INFO:tasks.workunit.client.0.smithi096.stdout:8/28: stat d4 0 2024-04-22T15:18:30.739 INFO:tasks.workunit.client.0.smithi096.stdout:1/23: dwrite f2 [0,4194304] 0 2024-04-22T15:18:30.741 INFO:tasks.workunit.client.0.smithi096.stdout:8/29: creat d4/fc x:0 0 0 2024-04-22T15:18:30.743 INFO:tasks.workunit.client.0.smithi096.stdout:1/24: dread d6/f8 [0,4194304] 0 2024-04-22T15:18:30.745 INFO:tasks.workunit.client.0.smithi096.stdout:8/30: creat d4/fd x:0 0 0 2024-04-22T15:18:30.745 INFO:tasks.workunit.client.0.smithi096.stdout:8/31: stat d4/fd 0 2024-04-22T15:18:30.750 INFO:tasks.workunit.client.0.smithi096.stdout:8/32: rename d4/fc to d4/fe 0 2024-04-22T15:18:30.752 INFO:tasks.workunit.client.0.smithi096.stdout:8/33: truncate d4/fa 777562 0 2024-04-22T15:18:30.753 INFO:tasks.workunit.client.0.smithi096.stdout:1/25: dwrite d6/f7 [0,4194304] 0 2024-04-22T15:18:30.754 INFO:tasks.workunit.client.0.smithi096.stdout:8/34: write d4/fa [1449009,9170] 0 2024-04-22T15:18:30.914 INFO:tasks.workunit.client.0.smithi096.stdout:4/11: fsync f1 0 2024-04-22T15:18:30.916 INFO:tasks.workunit.client.0.smithi096.stdout:4/12: creat f2 x:0 0 0 2024-04-22T15:18:30.921 INFO:tasks.workunit.client.0.smithi096.stdout:4/13: dwrite f1 [4194304,4194304] 0 2024-04-22T15:18:30.924 INFO:tasks.workunit.client.0.smithi096.stdout:4/14: creat f3 x:0 0 0 2024-04-22T15:18:30.925 INFO:tasks.workunit.client.0.smithi096.stdout:4/15: chown f3 60916 1 2024-04-22T15:18:30.925 INFO:tasks.workunit.client.0.smithi096.stdout:4/16: rmdir - no directory 2024-04-22T15:18:30.929 INFO:tasks.workunit.client.0.smithi096.stdout:4/17: symlink l4 0 2024-04-22T15:18:30.935 INFO:tasks.workunit.client.0.smithi096.stdout:4/18: dwrite f1 [0,4194304] 0 2024-04-22T15:18:30.935 INFO:tasks.workunit.client.0.smithi096.stdout:4/19: chown f2 3498 1 2024-04-22T15:18:30.941 INFO:tasks.workunit.client.0.smithi096.stdout:4/20: dwrite f3 [0,4194304] 0 2024-04-22T15:18:30.947 INFO:tasks.workunit.client.0.smithi096.stdout:4/21: dwrite f1 [4194304,4194304] 0 2024-04-22T15:18:30.947 INFO:tasks.workunit.client.0.smithi096.stdout:4/22: chown f1 249 1 2024-04-22T15:18:30.948 INFO:tasks.workunit.client.0.smithi096.stdout:4/23: read - f2 zero size 2024-04-22T15:18:30.949 INFO:tasks.workunit.client.0.smithi096.stdout:4/24: write f1 [8301820,122993] 0 2024-04-22T15:18:30.998 INFO:tasks.workunit.client.0.smithi096.stdout:0/8: getdents . 0 2024-04-22T15:18:31.037 INFO:tasks.workunit.client.0.smithi096.stdout:1/26: getdents d6 0 2024-04-22T15:18:31.041 INFO:tasks.workunit.client.0.smithi096.stdout:5/19: truncate d4/f5 1954438 0 2024-04-22T15:18:31.042 INFO:tasks.workunit.client.0.smithi096.stdout:5/20: chown d4/f5 283787 1 2024-04-22T15:18:31.042 INFO:tasks.workunit.client.0.smithi096.stdout:5/21: rename d4 to d4/d8 22 2024-04-22T15:18:31.284 INFO:tasks.workunit.client.0.smithi096.stdout:1/27: chown c4 3733502 1 2024-04-22T15:18:31.285 INFO:tasks.workunit.client.0.smithi096.stdout:1/28: chown d6 2010597 1 2024-04-22T15:18:31.290 INFO:tasks.workunit.client.0.smithi096.stdout:0/9: rename c1 to c4 0 2024-04-22T15:18:31.290 INFO:tasks.workunit.client.0.smithi096.stdout:1/29: rename l5 to d6/l9 0 2024-04-22T15:18:31.292 INFO:tasks.workunit.client.0.smithi096.stdout:1/30: fdatasync f2 0 2024-04-22T15:18:31.292 INFO:tasks.workunit.client.0.smithi096.stdout:5/22: dwrite f2 [0,4194304] 0 2024-04-22T15:18:31.292 INFO:tasks.workunit.client.0.smithi096.stdout:5/23: readlink l0 0 2024-04-22T15:18:31.293 INFO:tasks.workunit.client.0.smithi096.stdout:5/24: stat d4/f6 0 2024-04-22T15:18:31.293 INFO:tasks.workunit.client.0.smithi096.stdout:0/10: creat f5 x:0 0 0 2024-04-22T15:18:31.294 INFO:tasks.workunit.client.0.smithi096.stdout:0/11: dread - f5 zero size 2024-04-22T15:18:31.296 INFO:tasks.workunit.client.0.smithi096.stdout:5/25: symlink d4/l9 0 2024-04-22T15:18:31.297 INFO:tasks.workunit.client.0.smithi096.stdout:5/26: write d4/f7 [625449,30330] 0 2024-04-22T15:18:31.297 INFO:tasks.workunit.client.0.smithi096.stdout:0/12: mknod c6 0 2024-04-22T15:18:31.297 INFO:tasks.workunit.client.0.smithi096.stdout:1/31: dread d6/f8 [0,4194304] 0 2024-04-22T15:18:31.298 INFO:tasks.workunit.client.0.smithi096.stdout:0/13: dread - f0 zero size 2024-04-22T15:18:31.298 INFO:tasks.workunit.client.0.smithi096.stdout:0/14: rmdir - no directory 2024-04-22T15:18:31.299 INFO:tasks.workunit.client.0.smithi096.stdout:1/32: write d6/f7 [3947413,18427] 0 2024-04-22T15:18:31.300 INFO:tasks.workunit.client.0.smithi096.stdout:5/27: creat d4/fa x:0 0 0 2024-04-22T15:18:31.304 INFO:tasks.workunit.client.0.smithi096.stdout:5/28: mkdir d4/db 0 2024-04-22T15:18:31.306 INFO:tasks.workunit.client.0.smithi096.stdout:0/15: unlink c2 0 2024-04-22T15:18:31.307 INFO:tasks.workunit.client.0.smithi096.stdout:0/16: write f0 [924000,36144] 0 2024-04-22T15:18:31.311 INFO:tasks.workunit.client.0.smithi096.stdout:5/29: rmdir d4/db 0 2024-04-22T15:18:31.315 INFO:tasks.workunit.client.0.smithi096.stdout:0/17: dwrite f5 [0,4194304] 0 2024-04-22T15:18:31.315 INFO:tasks.workunit.client.0.smithi096.stdout:5/30: mkdir d4/dc 0 2024-04-22T15:18:31.320 INFO:tasks.workunit.client.0.smithi096.stdout:0/18: dread f0 [0,4194304] 0 2024-04-22T15:18:31.322 INFO:tasks.workunit.client.0.smithi096.stdout:0/19: mkdir d7 0 2024-04-22T15:18:31.323 INFO:tasks.workunit.client.0.smithi096.stdout:5/31: rmdir d4/dc 0 2024-04-22T15:18:31.325 INFO:tasks.workunit.client.0.smithi096.stdout:5/32: write d4/fa [708597,122159] 0 2024-04-22T15:18:31.328 INFO:tasks.workunit.client.0.smithi096.stdout:5/33: creat d4/fd x:0 0 0 2024-04-22T15:18:31.329 INFO:tasks.workunit.client.0.smithi096.stdout:0/20: link f5 d7/f8 0 2024-04-22T15:18:31.334 INFO:tasks.workunit.client.0.smithi096.stdout:0/21: fdatasync f5 0 2024-04-22T15:18:31.336 INFO:tasks.workunit.client.0.smithi096.stdout:5/34: dwrite f2 [0,4194304] 0 2024-04-22T15:18:31.340 INFO:tasks.workunit.client.0.smithi096.stdout:0/22: mkdir d7/d9 0 2024-04-22T15:18:31.340 INFO:tasks.workunit.client.0.smithi096.stdout:0/23: fsync f3 0 2024-04-22T15:18:31.348 INFO:tasks.workunit.client.0.smithi096.stdout:0/24: dwrite f3 [0,4194304] 0 2024-04-22T15:18:31.605 INFO:tasks.workunit.client.0.smithi096.stdout:1/33: truncate d6/f8 1344115 0 2024-04-22T15:18:31.606 INFO:tasks.workunit.client.0.smithi096.stdout:0/25: truncate d7/f8 3925756 0 2024-04-22T15:18:31.606 INFO:tasks.workunit.client.0.smithi096.stdout:8/35: dwrite f3 [0,4194304] 0 2024-04-22T15:18:31.607 INFO:tasks.workunit.client.0.smithi096.stdout:5/35: truncate d4/fa 91757 0 2024-04-22T15:18:31.609 INFO:tasks.workunit.client.0.smithi096.stdout:5/36: write d4/f6 [4120818,48437] 0 2024-04-22T15:18:31.610 INFO:tasks.workunit.client.0.smithi096.stdout:5/37: chown c3 0 1 2024-04-22T15:18:31.611 INFO:tasks.workunit.client.0.smithi096.stdout:5/38: read d4/f5 [1693529,36470] 0 2024-04-22T15:18:31.615 INFO:tasks.workunit.client.0.smithi096.stdout:5/39: mknod d4/ce 0 2024-04-22T15:18:31.620 INFO:tasks.workunit.client.0.smithi096.stdout:0/26: link c4 d7/d9/ca 0 2024-04-22T15:18:31.625 INFO:tasks.workunit.client.0.smithi096.stdout:0/27: mkdir d7/db 0 2024-04-22T15:18:31.629 INFO:tasks.workunit.client.0.smithi096.stdout:0/28: unlink d7/f8 0 2024-04-22T15:18:31.632 INFO:tasks.workunit.client.0.smithi096.stdout:0/29: mkdir d7/d9/dc 0 2024-04-22T15:18:31.632 INFO:tasks.workunit.client.0.smithi096.stdout:0/30: fsync f3 0 2024-04-22T15:18:31.637 INFO:tasks.workunit.client.0.smithi096.stdout:8/36: truncate f3 1548714 0 2024-04-22T15:18:31.637 INFO:tasks.workunit.client.0.smithi096.stdout:0/31: dwrite f3 [0,4194304] 0 2024-04-22T15:18:31.639 INFO:tasks.workunit.client.0.smithi096.stdout:8/37: symlink d4/lf 0 2024-04-22T15:18:31.642 INFO:tasks.workunit.client.0.smithi096.stdout:8/38: mkdir d4/d10 0 2024-04-22T15:18:31.642 INFO:tasks.workunit.client.0.smithi096.stdout:5/40: rmdir d4 39 2024-04-22T15:18:31.648 INFO:tasks.workunit.client.0.smithi096.stdout:1/34: truncate d6/f8 353431 0 2024-04-22T15:18:31.649 INFO:tasks.workunit.client.0.smithi096.stdout:8/39: dwrite d4/fa [0,4194304] 0 2024-04-22T15:18:31.650 INFO:tasks.workunit.client.0.smithi096.stdout:5/41: creat d4/ff x:0 0 0 2024-04-22T15:18:31.652 INFO:tasks.workunit.client.0.smithi096.stdout:1/35: creat d6/fa x:0 0 0 2024-04-22T15:18:31.655 INFO:tasks.workunit.client.0.smithi096.stdout:8/40: unlink d4/f8 0 2024-04-22T15:18:31.655 INFO:tasks.workunit.client.0.smithi096.stdout:8/41: chown d4/fa 57 1 2024-04-22T15:18:31.660 INFO:tasks.workunit.client.0.smithi096.stdout:1/36: mknod d6/cb 0 2024-04-22T15:18:31.661 INFO:tasks.workunit.client.0.smithi096.stdout:1/37: dread - d6/fa zero size 2024-04-22T15:18:31.663 INFO:tasks.workunit.client.0.smithi096.stdout:8/42: rename l2 to d4/d10/l11 0 2024-04-22T15:18:31.663 INFO:tasks.workunit.client.0.smithi096.stdout:1/38: mkdir d6/dc 0 2024-04-22T15:18:31.664 INFO:tasks.workunit.client.0.smithi096.stdout:1/39: write d6/fa [417008,56729] 0 2024-04-22T15:18:31.746 INFO:tasks.workunit.client.0.smithi096.stdout:8/43: fsync d4/fa 0 2024-04-22T15:18:31.749 INFO:tasks.workunit.client.0.smithi096.stdout:8/44: rename d4/fe to d4/f12 0 2024-04-22T15:18:31.752 INFO:tasks.workunit.client.0.smithi096.stdout:8/45: dread f3 [0,4194304] 0 2024-04-22T15:18:31.756 INFO:tasks.workunit.client.0.smithi096.stdout:8/46: rename d4/l6 to d4/d10/l13 0 2024-04-22T15:18:31.757 INFO:tasks.workunit.client.0.smithi096.stdout:1/40: fsync d6/f8 0 2024-04-22T15:18:31.757 INFO:tasks.workunit.client.0.smithi096.stdout:8/47: write d4/fa [2750589,101346] 0 2024-04-22T15:18:31.761 INFO:tasks.workunit.client.0.smithi096.stdout:1/41: rename d6/fa to d6/fd 0 2024-04-22T15:18:31.761 INFO:tasks.workunit.client.0.smithi096.stdout:5/42: rmdir d4 39 2024-04-22T15:18:31.763 INFO:tasks.workunit.client.0.smithi096.stdout:8/48: mknod d4/c14 0 2024-04-22T15:18:31.767 INFO:tasks.workunit.client.0.smithi096.stdout:1/42: dwrite d6/f8 [0,4194304] 0 2024-04-22T15:18:31.769 INFO:tasks.workunit.client.0.smithi096.stdout:8/49: unlink d4/f7 0 2024-04-22T15:18:31.774 INFO:tasks.workunit.client.0.smithi096.stdout:8/50: mknod d4/c15 0 2024-04-22T15:18:31.776 INFO:tasks.workunit.client.0.smithi096.stdout:1/43: symlink d6/dc/le 0 2024-04-22T15:18:31.778 INFO:tasks.workunit.client.0.smithi096.stdout:8/51: mkdir d4/d16 0 2024-04-22T15:18:31.779 INFO:tasks.workunit.client.0.smithi096.stdout:8/52: write d4/fd [62697,32006] 0 2024-04-22T15:18:31.781 INFO:tasks.workunit.client.0.smithi096.stdout:1/44: dread d6/fd [0,4194304] 0 2024-04-22T15:18:31.783 INFO:tasks.workunit.client.0.smithi096.stdout:1/45: readlink d6/l9 0 2024-04-22T15:18:31.790 INFO:tasks.workunit.client.0.smithi096.stdout:1/46: dwrite d6/f8 [0,4194304] 0 2024-04-22T15:18:31.793 INFO:tasks.workunit.client.0.smithi096.stdout:1/47: write d6/fd [217136,113133] 0 2024-04-22T15:18:31.795 INFO:tasks.workunit.client.0.smithi096.stdout:1/48: write f2 [2667568,29050] 0 2024-04-22T15:18:31.797 INFO:tasks.workunit.client.0.smithi096.stdout:1/49: readlink d6/l9 0 2024-04-22T15:18:31.798 INFO:tasks.workunit.client.0.smithi096.stdout:1/50: write d6/f7 [3716735,82862] 0 2024-04-22T15:18:31.805 INFO:tasks.workunit.client.0.smithi096.stdout:1/51: dwrite f2 [4194304,4194304] 0 2024-04-22T15:18:31.806 INFO:tasks.workunit.client.0.smithi096.stdout:1/52: stat d6/f8 0 2024-04-22T15:18:31.819 INFO:tasks.workunit.client.0.smithi096.stdout:1/53: truncate d6/f8 1919364 0 2024-04-22T15:18:31.820 INFO:tasks.workunit.client.0.smithi096.stdout:1/54: chown d6/f7 6491721 1 2024-04-22T15:18:31.821 INFO:tasks.workunit.client.0.smithi096.stdout:1/55: mkdir d6/dc/df 0 2024-04-22T15:18:31.824 INFO:tasks.workunit.client.0.smithi096.stdout:1/56: mkdir d6/dc/d10 0 2024-04-22T15:18:31.826 INFO:tasks.workunit.client.0.smithi096.stdout:1/57: mkdir d6/dc/d11 0 2024-04-22T15:18:31.830 INFO:tasks.workunit.client.0.smithi096.stdout:1/58: getdents d6 0 2024-04-22T15:18:31.834 INFO:tasks.workunit.client.0.smithi096.stdout:1/59: mkdir d6/dc/df/d12 0 2024-04-22T15:18:31.837 INFO:tasks.workunit.client.0.smithi096.stdout:1/60: unlink d6/f7 0 2024-04-22T15:18:31.844 INFO:tasks.workunit.client.0.smithi096.stdout:1/61: dwrite d6/fd [0,4194304] 0 2024-04-22T15:18:31.846 INFO:tasks.workunit.client.0.smithi096.stdout:1/62: write d6/f8 [1384427,102664] 0 2024-04-22T15:18:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:1/63: truncate f2 9112770 0 2024-04-22T15:18:31.864 INFO:tasks.workunit.client.0.smithi096.stdout:1/64: mknod d6/dc/d10/c13 0 2024-04-22T15:18:31.870 INFO:tasks.workunit.client.0.smithi096.stdout:1/65: dwrite d6/f8 [0,4194304] 0 2024-04-22T15:18:31.872 INFO:tasks.workunit.client.0.smithi096.stdout:1/66: rmdir d6/dc/df 39 2024-04-22T15:18:31.874 INFO:tasks.workunit.client.0.smithi096.stdout:1/67: creat d6/f14 x:0 0 0 2024-04-22T15:18:31.875 INFO:tasks.workunit.client.0.smithi096.stdout:1/68: chown d6/dc/d10/c13 85098 1 2024-04-22T15:18:31.877 INFO:tasks.workunit.client.0.smithi096.stdout:1/69: mknod d6/dc/d10/c15 0 2024-04-22T15:18:31.882 INFO:tasks.workunit.client.0.smithi096.stdout:1/70: creat d6/dc/df/f16 x:0 0 0 2024-04-22T15:18:31.884 INFO:tasks.workunit.client.0.smithi096.stdout:1/71: symlink d6/dc/d10/l17 0 2024-04-22T15:18:31.884 INFO:tasks.workunit.client.0.smithi096.stdout:1/72: stat d6/dc/le 0 2024-04-22T15:18:31.898 INFO:tasks.workunit.client.0.smithi096.stdout:1/73: write f2 [1679473,123761] 0 2024-04-22T15:18:31.900 INFO:tasks.workunit.client.0.smithi096.stdout:1/74: rmdir d6 39 2024-04-22T15:18:31.904 INFO:tasks.workunit.client.0.smithi096.stdout:1/75: creat d6/dc/df/d12/f18 x:0 0 0 2024-04-22T15:18:31.907 INFO:tasks.workunit.client.0.smithi096.stdout:1/76: unlink d6/dc/df/f16 0 2024-04-22T15:18:31.907 INFO:tasks.workunit.client.0.smithi096.stdout:1/77: fdatasync d6/fd 0 2024-04-22T15:18:31.910 INFO:tasks.workunit.client.0.smithi096.stdout:1/78: creat d6/dc/d11/f19 x:0 0 0 2024-04-22T15:18:31.917 INFO:tasks.workunit.client.0.smithi096.stdout:1/79: dwrite d6/f14 [0,4194304] 0 2024-04-22T15:18:31.918 INFO:tasks.workunit.client.0.smithi096.stdout:1/80: chown d6/f14 487 1 2024-04-22T15:18:31.924 INFO:tasks.workunit.client.0.smithi096.stdout:1/81: dwrite d6/fd [0,4194304] 0 2024-04-22T15:18:31.926 INFO:tasks.workunit.client.0.smithi096.stdout:1/82: rmdir d6/dc 39 2024-04-22T15:18:31.931 INFO:tasks.workunit.client.0.smithi096.stdout:1/83: dread d6/f14 [0,4194304] 0 2024-04-22T15:18:31.935 INFO:tasks.workunit.client.0.smithi096.stdout:1/84: creat d6/dc/df/f1a x:0 0 0 2024-04-22T15:18:31.944 INFO:tasks.workunit.client.0.smithi096.stdout:1/85: fsync d6/fd 0 2024-04-22T15:18:31.946 INFO:tasks.workunit.client.0.smithi096.stdout:1/86: write d6/f14 [2639089,3646] 0 2024-04-22T15:18:31.947 INFO:tasks.workunit.client.0.smithi096.stdout:1/87: write d6/dc/df/d12/f18 [985600,95594] 0 2024-04-22T15:18:31.951 INFO:tasks.workunit.client.0.smithi096.stdout:1/88: mkdir d6/d1b 0 2024-04-22T15:18:31.954 INFO:tasks.workunit.client.0.smithi096.stdout:1/89: dread d6/fd [0,4194304] 0 2024-04-22T15:18:32.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:31 smithi096 ceph-mon[26688]: pgmap v22: 65 pgs: 65 active+clean; 190 MiB data, 785 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 4.7 MiB/s wr, 345 op/s 2024-04-22T15:18:32.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:31 smithi196 ceph-mon[34717]: pgmap v22: 65 pgs: 65 active+clean; 190 MiB data, 785 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 4.7 MiB/s wr, 345 op/s 2024-04-22T15:18:32.762 INFO:tasks.workunit.client.0.smithi096.stdout:7/8: sync 2024-04-22T15:18:32.762 INFO:tasks.workunit.client.0.smithi096.stdout:7/9: truncate - no filename 2024-04-22T15:18:32.763 INFO:tasks.workunit.client.0.smithi096.stdout:7/10: symlink l2 0 2024-04-22T15:18:32.764 INFO:tasks.workunit.client.0.smithi096.stdout:7/11: dwrite - no filename 2024-04-22T15:18:32.765 INFO:tasks.workunit.client.0.smithi096.stdout:7/12: creat f3 x:0 0 0 2024-04-22T15:18:32.767 INFO:tasks.workunit.client.0.smithi096.stdout:7/13: creat f4 x:0 0 0 2024-04-22T15:18:32.768 INFO:tasks.workunit.client.0.smithi096.stdout:7/14: creat f5 x:0 0 0 2024-04-22T15:18:32.770 INFO:tasks.workunit.client.0.smithi096.stdout:7/15: symlink l6 0 2024-04-22T15:18:32.773 INFO:tasks.workunit.client.0.smithi096.stdout:7/16: rename f4 to f7 0 2024-04-22T15:18:32.774 INFO:tasks.workunit.client.0.smithi096.stdout:7/17: mkdir d8 0 2024-04-22T15:18:32.776 INFO:tasks.workunit.client.0.smithi096.stdout:7/18: rmdir d8 0 2024-04-22T15:18:32.777 INFO:tasks.workunit.client.0.smithi096.stdout:7/19: chown l2 1660 1 2024-04-22T15:18:32.778 INFO:tasks.workunit.client.0.smithi096.stdout:7/20: mknod c9 0 2024-04-22T15:18:32.783 INFO:tasks.workunit.client.0.smithi096.stdout:7/21: dwrite f5 [0,4194304] 0 2024-04-22T15:18:32.795 INFO:tasks.workunit.client.0.smithi096.stdout:7/22: dwrite f7 [0,4194304] 0 2024-04-22T15:18:32.916 INFO:tasks.workunit.client.0.smithi096.stdout:2/13: sync 2024-04-22T15:18:32.916 INFO:tasks.workunit.client.0.smithi096.stdout:9/15: sync 2024-04-22T15:18:32.918 INFO:tasks.workunit.client.0.smithi096.stdout:9/16: mkdir d1/d4 0 2024-04-22T15:18:32.920 INFO:tasks.workunit.client.0.smithi096.stdout:2/14: symlink d3/l5 0 2024-04-22T15:18:32.927 INFO:tasks.workunit.client.0.smithi096.stdout:9/17: dwrite d1/f3 [0,4194304] 0 2024-04-22T15:18:32.928 INFO:tasks.workunit.client.0.smithi096.stdout:2/15: dread d3/f4 [0,4194304] 0 2024-04-22T15:18:32.933 INFO:tasks.workunit.client.0.smithi096.stdout:9/18: dread d1/f3 [0,4194304] 0 2024-04-22T15:18:32.933 INFO:tasks.workunit.client.0.smithi096.stdout:2/16: mknod d3/c6 0 2024-04-22T15:18:32.937 INFO:tasks.workunit.client.0.smithi096.stdout:2/17: write f0 [1600238,86611] 0 2024-04-22T15:18:32.938 INFO:tasks.workunit.client.0.smithi096.stdout:9/19: creat d1/d4/f5 x:0 0 0 2024-04-22T15:18:32.941 INFO:tasks.workunit.client.0.smithi096.stdout:2/18: mknod d3/c7 0 2024-04-22T15:18:32.943 INFO:tasks.workunit.client.0.smithi096.stdout:9/20: creat d1/f6 x:0 0 0 2024-04-22T15:18:32.943 INFO:tasks.workunit.client.0.smithi096.stdout:9/21: stat c0 0 2024-04-22T15:18:32.944 INFO:tasks.workunit.client.0.smithi096.stdout:2/19: creat d3/f8 x:0 0 0 2024-04-22T15:18:32.945 INFO:tasks.workunit.client.0.smithi096.stdout:2/20: chown d3/c6 128950 1 2024-04-22T15:18:32.946 INFO:tasks.workunit.client.0.smithi096.stdout:2/21: write f0 [1882560,81489] 0 2024-04-22T15:18:32.946 INFO:tasks.workunit.client.0.smithi096.stdout:9/22: rename d1/d4/f5 to d1/d4/f7 0 2024-04-22T15:18:32.948 INFO:tasks.workunit.client.0.smithi096.stdout:9/23: write d1/f6 [64751,75563] 0 2024-04-22T15:18:32.954 INFO:tasks.workunit.client.0.smithi096.stdout:2/22: dwrite d3/f4 [0,4194304] 0 2024-04-22T15:18:32.956 INFO:tasks.workunit.client.0.smithi096.stdout:9/24: fdatasync d1/d4/f7 0 2024-04-22T15:18:32.956 INFO:tasks.workunit.client.0.smithi096.stdout:9/25: stat d1/d4/f7 0 2024-04-22T15:18:32.960 INFO:tasks.workunit.client.0.smithi096.stdout:9/26: symlink d1/l8 0 2024-04-22T15:18:32.961 INFO:tasks.workunit.client.0.smithi096.stdout:9/27: truncate d1/d4/f7 501805 0 2024-04-22T15:18:32.964 INFO:tasks.workunit.client.0.smithi096.stdout:9/28: unlink d1/l8 0 2024-04-22T15:18:32.965 INFO:tasks.workunit.client.0.smithi096.stdout:9/29: write d1/d4/f7 [134684,65436] 0 2024-04-22T15:18:32.972 INFO:tasks.workunit.client.0.smithi096.stdout:2/23: fdatasync d3/f4 0 2024-04-22T15:18:32.973 INFO:tasks.workunit.client.0.smithi096.stdout:9/30: dwrite d1/d4/f7 [0,4194304] 0 2024-04-22T15:18:32.973 INFO:tasks.workunit.client.0.smithi096.stdout:9/31: readlink - no filename 2024-04-22T15:18:33.117 INFO:tasks.workunit.client.0.smithi096.stdout:6/29: sync 2024-04-22T15:18:33.117 INFO:tasks.workunit.client.0.smithi096.stdout:3/11: sync 2024-04-22T15:18:33.118 INFO:tasks.workunit.client.0.smithi096.stdout:6/30: fdatasync d3/f6 0 2024-04-22T15:18:33.119 INFO:tasks.workunit.client.0.smithi096.stdout:6/31: read d3/f4 [51379,100789] 0 2024-04-22T15:18:33.120 INFO:tasks.workunit.client.0.smithi096.stdout:3/12: mknod c5 0 2024-04-22T15:18:33.127 INFO:tasks.workunit.client.0.smithi096.stdout:6/32: dwrite f0 [0,4194304] 0 2024-04-22T15:18:33.129 INFO:tasks.workunit.client.0.smithi096.stdout:3/13: dwrite f2 [0,4194304] 0 2024-04-22T15:18:33.129 INFO:tasks.workunit.client.0.smithi096.stdout:3/14: chown c4 61699 1 2024-04-22T15:18:33.135 INFO:tasks.workunit.client.0.smithi096.stdout:6/33: symlink d3/d7/l9 0 2024-04-22T15:18:33.138 INFO:tasks.workunit.client.0.smithi096.stdout:3/15: symlink l6 0 2024-04-22T15:18:33.139 INFO:tasks.workunit.client.0.smithi096.stdout:3/16: write f2 [2210422,63662] 0 2024-04-22T15:18:33.144 INFO:tasks.workunit.client.0.smithi096.stdout:3/17: mknod c7 0 2024-04-22T15:18:33.148 INFO:tasks.workunit.client.0.smithi096.stdout:3/18: mknod c8 0 2024-04-22T15:18:33.150 INFO:tasks.workunit.client.0.smithi096.stdout:6/34: link d3/d7/c8 d3/d7/ca 0 2024-04-22T15:18:33.153 INFO:tasks.workunit.client.0.smithi096.stdout:3/19: mknod c9 0 2024-04-22T15:18:33.153 INFO:tasks.workunit.client.0.smithi096.stdout:3/20: chown l3 305 1 2024-04-22T15:18:33.155 INFO:tasks.workunit.client.0.smithi096.stdout:6/35: rename d3/d7/l9 to d3/lb 0 2024-04-22T15:18:33.155 INFO:tasks.workunit.client.0.smithi096.stdout:6/36: readlink d3/l5 0 2024-04-22T15:18:33.157 INFO:tasks.workunit.client.0.smithi096.stdout:3/21: unlink l0 0 2024-04-22T15:18:33.159 INFO:tasks.workunit.client.0.smithi096.stdout:3/22: mkdir da 0 2024-04-22T15:18:33.162 INFO:tasks.workunit.client.0.smithi096.stdout:6/37: link d3/f6 d3/d7/fc 0 2024-04-22T15:18:33.165 INFO:tasks.workunit.client.0.smithi096.stdout:3/23: dwrite f2 [0,4194304] 0 2024-04-22T15:18:33.173 INFO:tasks.workunit.client.0.smithi096.stdout:3/24: dread f2 [0,4194304] 0 2024-04-22T15:18:33.174 INFO:tasks.workunit.client.0.smithi096.stdout:6/38: dwrite d3/f4 [0,4194304] 0 2024-04-22T15:18:33.175 INFO:tasks.workunit.client.0.smithi096.stdout:3/25: read f2 [469205,93535] 0 2024-04-22T15:18:33.183 INFO:tasks.workunit.client.0.smithi096.stdout:3/26: mkdir da/db 0 2024-04-22T15:18:33.184 INFO:tasks.workunit.client.0.smithi096.stdout:3/27: stat c5 0 2024-04-22T15:18:33.184 INFO:tasks.workunit.client.0.smithi096.stdout:6/39: dwrite f1 [0,4194304] 0 2024-04-22T15:18:33.189 INFO:tasks.workunit.client.0.smithi096.stdout:3/28: dread f2 [0,4194304] 0 2024-04-22T15:18:33.189 INFO:tasks.workunit.client.0.smithi096.stdout:6/40: dread f1 [0,4194304] 0 2024-04-22T15:18:33.190 INFO:tasks.workunit.client.0.smithi096.stdout:3/29: chown l3 51 1 2024-04-22T15:18:33.190 INFO:tasks.workunit.client.0.smithi096.stdout:3/30: chown da 12 1 2024-04-22T15:18:33.195 INFO:tasks.workunit.client.0.smithi096.stdout:6/41: write f1 [1130725,14987] 0 2024-04-22T15:18:33.203 INFO:tasks.workunit.client.0.smithi096.stdout:3/31: dwrite f2 [0,4194304] 0 2024-04-22T15:18:33.209 INFO:tasks.workunit.client.0.smithi096.stdout:5/43: sync 2024-04-22T15:18:33.209 INFO:tasks.workunit.client.0.smithi096.stdout:4/25: sync 2024-04-22T15:18:33.209 INFO:tasks.workunit.client.0.smithi096.stdout:0/32: sync 2024-04-22T15:18:33.209 INFO:tasks.workunit.client.0.smithi096.stdout:8/53: sync 2024-04-22T15:18:33.211 INFO:tasks.workunit.client.0.smithi096.stdout:4/26: read f1 [478058,78818] 0 2024-04-22T15:18:33.213 INFO:tasks.workunit.client.0.smithi096.stdout:8/54: rmdir d4 39 2024-04-22T15:18:33.213 INFO:tasks.workunit.client.0.smithi096.stdout:3/32: dwrite f2 [0,4194304] 0 2024-04-22T15:18:33.218 INFO:tasks.workunit.client.0.smithi096.stdout:4/27: mkdir d5 0 2024-04-22T15:18:33.220 INFO:tasks.workunit.client.0.smithi096.stdout:4/28: chown f2 147 1 2024-04-22T15:18:33.220 INFO:tasks.workunit.client.0.smithi096.stdout:5/44: creat d4/f10 x:0 0 0 2024-04-22T15:18:33.226 INFO:tasks.workunit.client.0.smithi096.stdout:0/33: dwrite f0 [0,4194304] 0 2024-04-22T15:18:33.228 INFO:tasks.workunit.client.0.smithi096.stdout:5/45: mkdir d4/d11 0 2024-04-22T15:18:33.228 INFO:tasks.workunit.client.0.smithi096.stdout:5/46: stat f2 0 2024-04-22T15:18:33.238 INFO:tasks.workunit.client.0.smithi096.stdout:8/55: rename d4/d10/l13 to d4/d16/l17 0 2024-04-22T15:18:33.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.312+0000 7fad43a73700 1 -- 172.21.15.96:0/1855316562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad3c100d10 msgr2=0x7fad3c101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:33.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.312+0000 7fad43a73700 1 --2- 172.21.15.96:0/1855316562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad3c100d10 0x7fad3c101170 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009910 tx=0x7fad2c009c20 comp rx=0 tx=0).stop 2024-04-22T15:18:33.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.312+0000 7fad43a73700 1 -- 172.21.15.96:0/1855316562 shutdown_connections 2024-04-22T15:18:33.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.312+0000 7fad43a73700 1 --2- 172.21.15.96:0/1855316562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad3c100d10 0x7fad3c101170 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:33.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.312+0000 7fad43a73700 1 --2- 172.21.15.96:0/1855316562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad3c0ffb10 0x7fad3c0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:33.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.312+0000 7fad43a73700 1 -- 172.21.15.96:0/1855316562 >> 172.21.15.96:0/1855316562 conn(0x7fad3c0fb0b0 msgr2=0x7fad3c0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:33.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.312+0000 7fad43a73700 1 -- 172.21.15.96:0/1855316562 shutdown_connections 2024-04-22T15:18:33.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.312+0000 7fad43a73700 1 -- 172.21.15.96:0/1855316562 wait complete. 2024-04-22T15:18:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.312+0000 7fad43a73700 1 Processor -- start 2024-04-22T15:18:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad43a73700 1 -- start start 2024-04-22T15:18:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad43a73700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad3c0ffb10 0x7fad3c069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad43a73700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad3c100d10 0x7fad3c199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad43a73700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c199690 con 0x7fad3c0ffb10 2024-04-22T15:18:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad43a73700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c1997d0 con 0x7fad3c100d10 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad4180f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad3c0ffb10 0x7fad3c069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad4180f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad3c0ffb10 0x7fad3c069930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56470/0 (socket says 172.21.15.96:56470) 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad4100e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad3c100d10 0x7fad3c199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad4180f700 1 -- 172.21.15.96:0/2871132755 learned_addr learned my addr 172.21.15.96:0/2871132755 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad4100e700 1 -- 172.21.15.96:0/2871132755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad3c0ffb10 msgr2=0x7fad3c069930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad4100e700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad3c0ffb10 0x7fad3c069930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad4100e700 1 -- 172.21.15.96:0/2871132755 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c0095f0 con 0x7fad3c100d10 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.313+0000 7fad4100e700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad3c100d10 0x7fad3c199150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fad2c006010 tx=0x7fad2c005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.314+0000 7fad32ffd700 1 -- 172.21.15.96:0/2871132755 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2c01d070 con 0x7fad3c100d10 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.314+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c199a50 con 0x7fad3c100d10 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.314+0000 7fad32ffd700 1 -- 172.21.15.96:0/2871132755 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad2c0052a0 con 0x7fad3c100d10 2024-04-22T15:18:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.314+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c199fc0 con 0x7fad3c100d10 2024-04-22T15:18:33.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.314+0000 7fad32ffd700 1 -- 172.21.15.96:0/2871132755 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2c017400 con 0x7fad3c100d10 2024-04-22T15:18:33.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.315+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad20005320 con 0x7fad3c100d10 2024-04-22T15:18:33.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.315+0000 7fad32ffd700 1 -- 172.21.15.96:0/2871132755 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fad2c017630 con 0x7fad3c100d10 2024-04-22T15:18:33.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.316+0000 7fad32ffd700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fad2806dc50 0x7fad28070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:33.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.316+0000 7fad32ffd700 1 -- 172.21.15.96:0/2871132755 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fad2c08e9d0 con 0x7fad3c100d10 2024-04-22T15:18:33.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.316+0000 7fad4180f700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fad2806dc50 0x7fad28070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:33.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.317+0000 7fad4180f700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fad2806dc50 0x7fad28070110 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fad38005950 tx=0x7fad38009260 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:33.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.321+0000 7fad32ffd700 1 -- 172.21.15.96:0/2871132755 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fad2c05b7b0 con 0x7fad3c100d10 2024-04-22T15:18:33.374 INFO:tasks.workunit.client.0.smithi096.stdout:1/90: sync 2024-04-22T15:18:33.375 INFO:tasks.workunit.client.0.smithi096.stdout:7/23: sync 2024-04-22T15:18:33.375 INFO:tasks.workunit.client.0.smithi096.stdout:2/24: sync 2024-04-22T15:18:33.375 INFO:tasks.workunit.client.0.smithi096.stdout:9/32: sync 2024-04-22T15:18:33.375 INFO:tasks.workunit.client.0.smithi096.stdout:7/24: readlink l2 0 2024-04-22T15:18:33.380 INFO:tasks.workunit.client.0.smithi096.stdout:7/25: creat fa x:0 0 0 2024-04-22T15:18:33.383 INFO:tasks.workunit.client.0.smithi096.stdout:7/26: write f5 [4735783,93821] 0 2024-04-22T15:18:33.384 INFO:tasks.workunit.client.0.smithi096.stdout:1/91: dwrite d6/f14 [0,4194304] 0 2024-04-22T15:18:33.387 INFO:tasks.workunit.client.0.smithi096.stdout:2/25: link d3/f8 d3/f9 0 2024-04-22T15:18:33.387 INFO:tasks.workunit.client.0.smithi096.stdout:9/33: dwrite d1/f3 [4194304,4194304] 0 2024-04-22T15:18:33.390 INFO:tasks.workunit.client.0.smithi096.stdout:1/92: dread - d6/dc/d11/f19 zero size 2024-04-22T15:18:33.390 INFO:tasks.workunit.client.0.smithi096.stdout:9/34: fdatasync d1/d4/f7 0 2024-04-22T15:18:33.396 INFO:tasks.workunit.client.0.smithi096.stdout:9/35: mkdir d1/d4/d9 0 2024-04-22T15:18:33.399 INFO:tasks.workunit.client.0.smithi096.stdout:7/27: dwrite f3 [0,4194304] 0 2024-04-22T15:18:33.401 INFO:tasks.workunit.client.0.smithi096.stdout:0/34: sync 2024-04-22T15:18:33.402 INFO:tasks.workunit.client.0.smithi096.stdout:7/28: write fa [185985,55122] 0 2024-04-22T15:18:33.403 INFO:tasks.workunit.client.0.smithi096.stdout:9/36: unlink d1/c2 0 2024-04-22T15:18:33.404 INFO:tasks.workunit.client.0.smithi096.stdout:9/37: stat d1/d4/f7 0 2024-04-22T15:18:33.405 INFO:tasks.workunit.client.0.smithi096.stdout:2/26: dwrite d3/f9 [0,4194304] 0 2024-04-22T15:18:33.407 INFO:tasks.workunit.client.0.smithi096.stdout:0/35: chown c4 5615985 1 2024-04-22T15:18:33.407 INFO:tasks.workunit.client.0.smithi096.stdout:0/36: readlink - no filename 2024-04-22T15:18:33.409 INFO:tasks.workunit.client.0.smithi096.stdout:7/29: creat fb x:0 0 0 2024-04-22T15:18:33.409 INFO:tasks.workunit.client.0.smithi096.stdout:9/38: symlink d1/la 0 2024-04-22T15:18:33.410 INFO:tasks.workunit.client.0.smithi096.stdout:7/30: chown f7 8117902 1 2024-04-22T15:18:33.410 INFO:tasks.workunit.client.0.smithi096.stdout:9/39: readlink d1/la 0 2024-04-22T15:18:33.411 INFO:tasks.workunit.client.0.smithi096.stdout:9/40: write d1/f3 [5973462,42552] 0 2024-04-22T15:18:33.451 INFO:tasks.workunit.client.0.smithi096.stdout:0/37: dread f0 [0,4194304] 0 2024-04-22T15:18:33.457 INFO:tasks.workunit.client.0.smithi096.stdout:7/31: dwrite fa [0,4194304] 0 2024-04-22T15:18:33.461 INFO:tasks.workunit.client.0.smithi096.stdout:0/38: symlink d7/db/ld 0 2024-04-22T15:18:33.462 INFO:tasks.workunit.client.0.smithi096.stdout:7/32: creat fc x:0 0 0 2024-04-22T15:18:33.465 INFO:tasks.workunit.client.0.smithi096.stdout:7/33: unlink l6 0 2024-04-22T15:18:33.468 INFO:tasks.workunit.client.0.smithi096.stdout:7/34: creat fd x:0 0 0 2024-04-22T15:18:33.468 INFO:tasks.workunit.client.0.smithi096.stdout:7/35: dread - fd zero size 2024-04-22T15:18:33.469 INFO:tasks.workunit.client.0.smithi096.stdout:7/36: write f5 [1042525,17841] 0 2024-04-22T15:18:33.470 INFO:tasks.workunit.client.0.smithi096.stdout:7/37: write fd [6137,49751] 0 2024-04-22T15:18:33.472 INFO:tasks.workunit.client.0.smithi096.stdout:7/38: creat fe x:0 0 0 2024-04-22T15:18:33.546 INFO:tasks.workunit.client.0.smithi096.stdout:6/42: write f2 [1385444,29059] 0 2024-04-22T15:18:33.548 INFO:tasks.workunit.client.0.smithi096.stdout:6/43: mkdir d3/d7/dd 0 2024-04-22T15:18:33.551 INFO:tasks.workunit.client.0.smithi096.stdout:4/29: truncate f1 8208475 0 2024-04-22T15:18:33.552 INFO:tasks.workunit.client.0.smithi096.stdout:5/47: rmdir d4 39 2024-04-22T15:18:33.554 INFO:tasks.workunit.client.0.smithi096.stdout:6/44: dread d3/d7/fc [0,4194304] 0 2024-04-22T15:18:33.554 INFO:tasks.workunit.client.0.smithi096.stdout:3/33: truncate f2 1729744 0 2024-04-22T15:18:33.561 INFO:tasks.workunit.client.0.smithi096.stdout:4/30: creat d5/f6 x:0 0 0 2024-04-22T15:18:33.565 INFO:tasks.workunit.client.0.smithi096.stdout:5/48: mkdir d4/d12 0 2024-04-22T15:18:33.565 INFO:tasks.workunit.client.0.smithi096.stdout:5/49: write d4/ff [49547,5956] 0 2024-04-22T15:18:33.569 INFO:tasks.workunit.client.0.smithi096.stdout:8/56: rename d4/d16/l17 to d4/d10/l18 0 2024-04-22T15:18:33.570 INFO:tasks.workunit.client.0.smithi096.stdout:9/41: rename d1/d4 to d1/d4/db 22 2024-04-22T15:18:33.571 INFO:tasks.workunit.client.0.smithi096.stdout:9/42: readlink d1/la 0 2024-04-22T15:18:33.573 INFO:tasks.workunit.client.0.smithi096.stdout:5/50: dwrite d4/f10 [0,4194304] 0 2024-04-22T15:18:33.574 INFO:tasks.workunit.client.0.smithi096.stdout:5/51: chown d4/ff 0 1 2024-04-22T15:18:33.584 INFO:tasks.workunit.client.0.smithi096.stdout:5/52: creat d4/d12/f13 x:0 0 0 2024-04-22T15:18:33.584 INFO:tasks.workunit.client.0.smithi096.stdout:8/57: dwrite f3 [0,4194304] 0 2024-04-22T15:18:33.588 INFO:tasks.workunit.client.0.smithi096.stdout:5/53: rename l0 to d4/l14 0 2024-04-22T15:18:33.599 INFO:tasks.workunit.client.0.smithi096.stdout:8/58: fdatasync f3 0 2024-04-22T15:18:33.599 INFO:tasks.workunit.client.0.smithi096.stdout:8/59: write d4/fd [65642,95106] 0 2024-04-22T15:18:33.601 INFO:tasks.workunit.client.0.smithi096.stdout:1/93: sync 2024-04-22T15:18:33.602 INFO:tasks.workunit.client.0.smithi096.stdout:1/94: write d6/fd [1505856,93691] 0 2024-04-22T15:18:33.607 INFO:tasks.workunit.client.0.smithi096.stdout:1/95: link c4 d6/d1b/c1c 0 2024-04-22T15:18:33.609 INFO:tasks.workunit.client.0.smithi096.stdout:9/43: rmdir d1/d4 39 2024-04-22T15:18:33.609 INFO:tasks.workunit.client.0.smithi096.stdout:9/44: chown d1/f6 23123986 1 2024-04-22T15:18:33.610 INFO:tasks.workunit.client.0.smithi096.stdout:1/96: creat d6/f1d x:0 0 0 2024-04-22T15:18:33.613 INFO:tasks.workunit.client.0.smithi096.stdout:2/27: write d3/f9 [4834567,16608] 0 2024-04-22T15:18:33.615 INFO:tasks.workunit.client.0.smithi096.stdout:2/28: stat d3/f8 0 2024-04-22T15:18:33.618 INFO:tasks.workunit.client.0.smithi096.stdout:1/97: chown c4 3757984 1 2024-04-22T15:18:33.618 INFO:tasks.workunit.client.0.smithi096.stdout:2/29: chown d3/c7 944135477 1 2024-04-22T15:18:33.619 INFO:tasks.workunit.client.0.smithi096.stdout:9/45: dread d1/f3 [4194304,4194304] 0 2024-04-22T15:18:33.619 INFO:tasks.workunit.client.0.smithi096.stdout:7/39: dwrite f3 [4194304,4194304] 0 2024-04-22T15:18:33.627 INFO:tasks.workunit.client.0.smithi096.stdout:0/39: rename d7/db to d7/de 0 2024-04-22T15:18:33.629 INFO:tasks.workunit.client.0.smithi096.stdout:9/46: mkdir d1/dc 0 2024-04-22T15:18:33.631 INFO:tasks.workunit.client.0.smithi096.stdout:9/47: chown d1/la 3268 1 2024-04-22T15:18:33.633 INFO:tasks.workunit.client.0.smithi096.stdout:7/40: dwrite fe [0,4194304] 0 2024-04-22T15:18:33.633 INFO:tasks.workunit.client.0.smithi096.stdout:2/30: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:18:33.637 INFO:tasks.workunit.client.0.smithi096.stdout:0/40: symlink d7/de/lf 0 2024-04-22T15:18:33.639 INFO:tasks.workunit.client.0.smithi096.stdout:1/98: link d6/dc/d10/c15 d6/dc/df/d12/c1e 0 2024-04-22T15:18:33.640 INFO:tasks.workunit.client.0.smithi096.stdout:1/99: chown d6/f14 0 1 2024-04-22T15:18:33.640 INFO:tasks.workunit.client.0.smithi096.stdout:1/100: chown d6 1974 1 2024-04-22T15:18:33.645 INFO:tasks.workunit.client.0.smithi096.stdout:9/48: rename d1/dc to d1/d4/d9/dd 0 2024-04-22T15:18:33.647 INFO:tasks.workunit.client.0.smithi096.stdout:2/31: link f0 d3/fa 0 2024-04-22T15:18:33.649 INFO:tasks.workunit.client.0.smithi096.stdout:1/101: mknod d6/dc/d11/c1f 0 2024-04-22T15:18:33.654 INFO:tasks.workunit.client.0.smithi096.stdout:9/49: creat d1/d4/fe x:0 0 0 2024-04-22T15:18:33.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.667+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fad20000bf0 con 0x7fad2806dc50 2024-04-22T15:18:33.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.669+0000 7fad32ffd700 1 -- 172.21.15.96:0/2871132755 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fad20000bf0 con 0x7fad2806dc50 2024-04-22T15:18:33.674 INFO:tasks.workunit.client.0.smithi096.stdout:5/54: getdents d4/d12 0 2024-04-22T15:18:33.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.671+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fad2806dc50 msgr2=0x7fad28070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:33.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.671+0000 7fad43a73700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fad2806dc50 0x7fad28070110 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fad38005950 tx=0x7fad38009260 comp rx=0 tx=0).stop 2024-04-22T15:18:33.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.671+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad3c100d10 msgr2=0x7fad3c199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:33.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.671+0000 7fad43a73700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad3c100d10 0x7fad3c199150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fad2c006010 tx=0x7fad2c005f00 comp rx=0 tx=0).stop 2024-04-22T15:18:33.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.671+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 shutdown_connections 2024-04-22T15:18:33.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.671+0000 7fad43a73700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fad2806dc50 0x7fad28070110 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:33.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.671+0000 7fad43a73700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad3c0ffb10 0x7fad3c069930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:33.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.672+0000 7fad43a73700 1 --2- 172.21.15.96:0/2871132755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad3c100d10 0x7fad3c199150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:33.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.672+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 >> 172.21.15.96:0/2871132755 conn(0x7fad3c0fb0b0 msgr2=0x7fad3c1042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:33.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.672+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 shutdown_connections 2024-04-22T15:18:33.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.672+0000 7fad43a73700 1 -- 172.21.15.96:0/2871132755 wait complete. 2024-04-22T15:18:33.677 INFO:tasks.workunit.client.0.smithi096.stdout:9/50: mkdir d1/df 0 2024-04-22T15:18:33.677 INFO:tasks.workunit.client.0.smithi096.stdout:5/55: truncate f1 112639 0 2024-04-22T15:18:33.677 INFO:tasks.workunit.client.0.smithi096.stdout:1/102: creat d6/f20 x:0 0 0 2024-04-22T15:18:33.677 INFO:tasks.workunit.client.0.smithi096.stdout:5/56: readlink d4/l9 0 2024-04-22T15:18:33.680 INFO:tasks.workunit.client.0.smithi096.stdout:5/57: dread d4/ff [0,4194304] 0 2024-04-22T15:18:33.681 INFO:tasks.workunit.client.0.smithi096.stdout:6/45: dwrite d3/f6 [0,4194304] 0 2024-04-22T15:18:33.682 INFO:tasks.workunit.client.0.smithi096.stdout:5/58: chown d4/ff 750110261 1 2024-04-22T15:18:33.685 INFO:tasks.workunit.client.0.smithi096.stdout:1/103: creat d6/dc/df/d12/f21 x:0 0 0 2024-04-22T15:18:33.686 INFO:tasks.workunit.client.0.smithi096.stdout:1/104: write d6/fd [1667781,41474] 0 2024-04-22T15:18:33.687 INFO:tasks.workunit.client.0.smithi096.stdout:1/105: readlink d6/dc/le 0 2024-04-22T15:18:33.688 INFO:tasks.workunit.client.0.smithi096.stdout:1/106: write d6/f1d [766308,52082] 0 2024-04-22T15:18:33.689 INFO:tasks.workunit.client.0.smithi096.stdout:8/60: rmdir d4/d16 0 2024-04-22T15:18:33.691 INFO:tasks.workunit.client.0.smithi096.stdout:9/51: mknod d1/c10 0 2024-04-22T15:18:33.692 INFO:tasks.workunit.client.0.smithi096.stdout:5/59: creat d4/d12/f15 x:0 0 0 2024-04-22T15:18:33.694 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:18:33.701 INFO:tasks.workunit.client.0.smithi096.stdout:9/52: rename d1/f3 to d1/d4/f11 0 2024-04-22T15:18:33.701 INFO:tasks.workunit.client.0.smithi096.stdout:2/32: truncate d3/fa 3363368 0 2024-04-22T15:18:33.705 INFO:tasks.workunit.client.0.smithi096.stdout:6/46: mknod d3/d7/dd/ce 0 2024-04-22T15:18:33.706 INFO:tasks.workunit.client.0.smithi096.stdout:0/41: dwrite f0 [0,4194304] 0 2024-04-22T15:18:33.712 INFO:tasks.workunit.client.0.smithi096.stdout:5/60: rename d4/l14 to d4/l16 0 2024-04-22T15:18:33.712 INFO:tasks.workunit.client.0.smithi096.stdout:8/61: dwrite d4/f12 [0,4194304] 0 2024-04-22T15:18:33.716 INFO:tasks.workunit.client.0.smithi096.stdout:4/31: write f1 [7095950,31732] 0 2024-04-22T15:18:33.716 INFO:tasks.workunit.client.0.smithi096.stdout:5/61: fsync d4/d12/f13 0 2024-04-22T15:18:33.718 INFO:tasks.workunit.client.0.smithi096.stdout:1/107: creat d6/f22 x:0 0 0 2024-04-22T15:18:33.720 INFO:tasks.workunit.client.0.smithi096.stdout:6/47: dread f1 [0,4194304] 0 2024-04-22T15:18:33.721 INFO:tasks.workunit.client.0.smithi096.stdout:9/53: dwrite d1/d4/fe [0,4194304] 0 2024-04-22T15:18:33.727 INFO:tasks.workunit.client.0.smithi096.stdout:9/54: truncate d1/f6 213022 0 2024-04-22T15:18:33.730 INFO:tasks.workunit.client.0.smithi096.stdout:0/42: mkdir d7/d9/dc/d10 0 2024-04-22T15:18:33.732 INFO:tasks.workunit.client.0.smithi096.stdout:4/32: dwrite f3 [4194304,4194304] 0 2024-04-22T15:18:33.735 INFO:tasks.workunit.client.0.smithi096.stdout:8/62: fdatasync d4/f12 0 2024-04-22T15:18:33.736 INFO:tasks.workunit.client.0.smithi096.stdout:5/62: mknod d4/d11/c17 0 2024-04-22T15:18:33.738 INFO:tasks.workunit.client.0.smithi096.stdout:4/33: write d5/f6 [377877,107003] 0 2024-04-22T15:18:33.740 INFO:tasks.workunit.client.0.smithi096.stdout:1/108: dwrite d6/dc/d11/f19 [0,4194304] 0 2024-04-22T15:18:33.742 INFO:tasks.workunit.client.0.smithi096.stdout:1/109: chown d6/f20 329335 1 2024-04-22T15:18:33.743 INFO:tasks.workunit.client.0.smithi096.stdout:6/48: rename d3/d7/fc to d3/d7/ff 0 2024-04-22T15:18:33.744 INFO:tasks.workunit.client.0.smithi096.stdout:9/55: dread d1/d4/f7 [0,4194304] 0 2024-04-22T15:18:33.745 INFO:tasks.workunit.client.0.smithi096.stdout:9/56: chown d1/d4 8 1 2024-04-22T15:18:33.746 INFO:tasks.workunit.client.0.smithi096.stdout:8/63: creat d4/d10/f19 x:0 0 0 2024-04-22T15:18:33.748 INFO:tasks.workunit.client.0.smithi096.stdout:0/43: dwrite f3 [0,4194304] 0 2024-04-22T15:18:33.750 INFO:tasks.workunit.client.0.smithi096.stdout:5/63: creat d4/d11/f18 x:0 0 0 2024-04-22T15:18:33.753 INFO:tasks.workunit.client.0.smithi096.stdout:4/34: dread d5/f6 [0,4194304] 0 2024-04-22T15:18:33.753 INFO:tasks.workunit.client.0.smithi096.stdout:3/34: truncate f2 1532244 0 2024-04-22T15:18:33.754 INFO:tasks.workunit.client.0.smithi096.stdout:5/64: dread f1 [0,4194304] 0 2024-04-22T15:18:33.756 INFO:tasks.workunit.client.0.smithi096.stdout:6/49: symlink d3/d7/l10 0 2024-04-22T15:18:33.761 INFO:tasks.workunit.client.0.smithi096.stdout:1/110: dwrite d6/dc/df/f1a [0,4194304] 0 2024-04-22T15:18:33.764 INFO:tasks.workunit.client.0.smithi096.stdout:0/44: rename c6 to d7/d9/dc/c11 0 2024-04-22T15:18:33.764 INFO:tasks.workunit.client.0.smithi096.stdout:1/111: readlink d6/dc/d10/l17 0 2024-04-22T15:18:33.766 INFO:tasks.workunit.client.0.smithi096.stdout:0/45: chown d7 1 1 2024-04-22T15:18:33.766 INFO:tasks.workunit.client.0.smithi096.stdout:8/64: dwrite d4/fa [0,4194304] 0 2024-04-22T15:18:33.772 INFO:tasks.workunit.client.0.smithi096.stdout:5/65: mkdir d4/d19 0 2024-04-22T15:18:33.773 INFO:tasks.workunit.client.0.smithi096.stdout:8/65: write d4/fa [1383840,5326] 0 2024-04-22T15:18:33.774 INFO:tasks.workunit.client.0.smithi096.stdout:9/57: dwrite d1/d4/f11 [4194304,4194304] 0 2024-04-22T15:18:33.779 INFO:tasks.workunit.client.0.smithi096.stdout:2/33: write f0 [763526,12642] 0 2024-04-22T15:18:33.779 INFO:tasks.workunit.client.0.smithi096.stdout:3/35: mkdir da/db/dc 0 2024-04-22T15:18:33.782 INFO:tasks.workunit.client.0.smithi096.stdout:0/46: dread f0 [0,4194304] 0 2024-04-22T15:18:33.787 INFO:tasks.workunit.client.0.smithi096.stdout:5/66: write f1 [306272,87039] 0 2024-04-22T15:18:33.787 INFO:tasks.workunit.client.0.smithi096.stdout:4/35: dwrite d5/f6 [0,4194304] 0 2024-04-22T15:18:33.793 INFO:tasks.workunit.client.0.smithi096.stdout:2/34: mkdir d3/db 0 2024-04-22T15:18:33.797 INFO:tasks.workunit.client.0.smithi096.stdout:8/66: dwrite d4/d10/f19 [0,4194304] 0 2024-04-22T15:18:33.799 INFO:tasks.workunit.client.0.smithi096.stdout:0/47: dread f3 [0,4194304] 0 2024-04-22T15:18:33.800 INFO:tasks.workunit.client.0.smithi096.stdout:3/36: mknod da/db/cd 0 2024-04-22T15:18:33.802 INFO:tasks.workunit.client.0.smithi096.stdout:0/48: chown d7/d9/ca 30 1 2024-04-22T15:18:33.804 INFO:tasks.workunit.client.0.smithi096.stdout:9/58: fdatasync d1/d4/f11 0 2024-04-22T15:18:33.806 INFO:tasks.workunit.client.0.smithi096.stdout:1/112: creat d6/dc/f23 x:0 0 0 2024-04-22T15:18:33.808 INFO:tasks.workunit.client.0.smithi096.stdout:4/36: mkdir d5/d7 0 2024-04-22T15:18:33.810 INFO:tasks.workunit.client.0.smithi096.stdout:5/67: dwrite d4/fd [0,4194304] 0 2024-04-22T15:18:33.811 INFO:tasks.workunit.client.0.smithi096.stdout:5/68: stat d4/ce 0 2024-04-22T15:18:33.813 INFO:tasks.workunit.client.0.smithi096.stdout:8/67: dwrite d4/fd [0,4194304] 0 2024-04-22T15:18:33.816 INFO:tasks.workunit.client.0.smithi096.stdout:3/37: mkdir da/db/de 0 2024-04-22T15:18:33.819 INFO:tasks.workunit.client.0.smithi096.stdout:8/68: dread d4/fd [0,4194304] 0 2024-04-22T15:18:33.821 INFO:tasks.workunit.client.0.smithi096.stdout:4/37: mknod d5/c8 0 2024-04-22T15:18:33.824 INFO:tasks.workunit.client.0.smithi096.stdout:1/113: rename d6/cb to d6/dc/d11/c24 0 2024-04-22T15:18:33.824 INFO:tasks.workunit.client.0.smithi096.stdout:8/69: dread d4/fd [0,4194304] 0 2024-04-22T15:18:33.830 INFO:tasks.workunit.client.0.smithi096.stdout:5/69: mknod d4/d19/c1a 0 2024-04-22T15:18:33.830 INFO:tasks.workunit.client.0.smithi096.stdout:5/70: stat d4/d12 0 2024-04-22T15:18:33.831 INFO:tasks.workunit.client.0.smithi096.stdout:1/114: creat d6/dc/df/f25 x:0 0 0 2024-04-22T15:18:33.831 INFO:tasks.workunit.client.0.smithi096.stdout:0/49: rmdir d7/d9/dc/d10 0 2024-04-22T15:18:33.832 INFO:tasks.workunit.client.0.smithi096.stdout:8/70: dwrite f3 [0,4194304] 0 2024-04-22T15:18:33.832 INFO:tasks.workunit.client.0.smithi096.stdout:4/38: creat d5/d7/f9 x:0 0 0 2024-04-22T15:18:33.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.855+0000 7fee6c7c8700 1 -- 172.21.15.96:0/3924952357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 msgr2=0x7fee64102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:33.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.855+0000 7fee6c7c8700 1 --2- 172.21.15.96:0/3924952357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 0x7fee64102310 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fee54009910 tx=0x7fee54009c20 comp rx=0 tx=0).stop 2024-04-22T15:18:33.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.856+0000 7fee6c7c8700 1 -- 172.21.15.96:0/3924952357 shutdown_connections 2024-04-22T15:18:33.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.856+0000 7fee6c7c8700 1 --2- 172.21.15.96:0/3924952357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 0x7fee64102310 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:33.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.856+0000 7fee6c7c8700 1 --2- 172.21.15.96:0/3924952357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee640feed0 0x7fee640ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:33.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.856+0000 7fee6c7c8700 1 -- 172.21.15.96:0/3924952357 >> 172.21.15.96:0/3924952357 conn(0x7fee640fa830 msgr2=0x7fee640fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:33.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.856+0000 7fee6c7c8700 1 -- 172.21.15.96:0/3924952357 shutdown_connections 2024-04-22T15:18:33.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.856+0000 7fee6c7c8700 1 -- 172.21.15.96:0/3924952357 wait complete. 2024-04-22T15:18:33.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.856+0000 7fee6c7c8700 1 Processor -- start 2024-04-22T15:18:33.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.856+0000 7fee6c7c8700 1 -- start start 2024-04-22T15:18:33.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee6c7c8700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee640feed0 0x7fee64196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:33.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee6c7c8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 0x7fee64196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:33.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee6c7c8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee641971d0 con 0x7fee64101eb0 2024-04-22T15:18:33.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee6c7c8700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee64197310 con 0x7fee640feed0 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee69d63700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 0x7fee64196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee69d63700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 0x7fee64196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56482/0 (socket says 172.21.15.96:56482) 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee69d63700 1 -- 172.21.15.96:0/523720179 learned_addr learned my addr 172.21.15.96:0/523720179 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee6a564700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee640feed0 0x7fee64196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee69d63700 1 -- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee640feed0 msgr2=0x7fee64196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee69d63700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee640feed0 0x7fee64196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee69d63700 1 -- 172.21.15.96:0/523720179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee540095f0 con 0x7fee64101eb0 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee69d63700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 0x7fee64196bb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fee54006010 tx=0x7fee5400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee5b7fe700 1 -- 172.21.15.96:0/523720179 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee5401d070 con 0x7fee64101eb0 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee5b7fe700 1 -- 172.21.15.96:0/523720179 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee5400f460 con 0x7fee64101eb0 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6419bcc0 con 0x7fee64101eb0 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee5b7fe700 1 -- 172.21.15.96:0/523720179 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee54021620 con 0x7fee64101eb0 2024-04-22T15:18:33.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.857+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6419c230 con 0x7fee64101eb0 2024-04-22T15:18:33.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.858+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee64190790 con 0x7fee64101eb0 2024-04-22T15:18:33.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.859+0000 7fee5b7fe700 1 -- 172.21.15.96:0/523720179 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fee5400f5d0 con 0x7fee64101eb0 2024-04-22T15:18:33.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.859+0000 7fee5b7fe700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fee5006dc50 0x7fee50070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:33.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.859+0000 7fee5b7fe700 1 -- 172.21.15.96:0/523720179 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fee5408ea60 con 0x7fee64101eb0 2024-04-22T15:18:33.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.859+0000 7fee6a564700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fee5006dc50 0x7fee50070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:33.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.860+0000 7fee6a564700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fee5006dc50 0x7fee50070110 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fee640683c0 tx=0x7fee6000b410 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:33.866 INFO:tasks.workunit.client.0.smithi096.stdout:4/39: fdatasync f1 0 2024-04-22T15:18:33.867 INFO:tasks.workunit.client.0.smithi096.stdout:1/115: fdatasync d6/dc/df/f1a 0 2024-04-22T15:18:33.868 INFO:tasks.workunit.client.0.smithi096.stdout:2/35: truncate d3/f8 2838997 0 2024-04-22T15:18:33.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:33.867+0000 7fee5b7fe700 1 -- 172.21.15.96:0/523720179 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fee5405b750 con 0x7fee64101eb0 2024-04-22T15:18:33.872 INFO:tasks.workunit.client.0.smithi096.stdout:1/116: chown d6/dc/df/f1a 41316009 1 2024-04-22T15:18:33.872 INFO:tasks.workunit.client.0.smithi096.stdout:2/36: write d3/fa [3847441,54712] 0 2024-04-22T15:18:33.876 INFO:tasks.workunit.client.0.smithi096.stdout:1/117: readlink d6/dc/d10/l17 0 2024-04-22T15:18:33.876 INFO:tasks.workunit.client.0.smithi096.stdout:4/40: dread f1 [0,4194304] 0 2024-04-22T15:18:33.877 INFO:tasks.workunit.client.0.smithi096.stdout:2/37: chown d3/l5 199797 1 2024-04-22T15:18:33.881 INFO:tasks.workunit.client.0.smithi096.stdout:4/41: write f1 [843953,70724] 0 2024-04-22T15:18:33.881 INFO:tasks.workunit.client.0.smithi096.stdout:0/50: dread f0 [0,4194304] 0 2024-04-22T15:18:33.882 INFO:tasks.workunit.client.0.smithi096.stdout:5/71: dwrite d4/f5 [0,4194304] 0 2024-04-22T15:18:33.886 INFO:tasks.workunit.client.0.smithi096.stdout:2/38: symlink d3/db/lc 0 2024-04-22T15:18:33.891 INFO:tasks.workunit.client.0.smithi096.stdout:1/118: dwrite d6/f14 [0,4194304] 0 2024-04-22T15:18:33.891 INFO:tasks.workunit.client.0.smithi096.stdout:4/42: mknod d5/d7/ca 0 2024-04-22T15:18:33.892 INFO:tasks.workunit.client.0.smithi096.stdout:8/71: truncate d4/fa 28891 0 2024-04-22T15:18:33.892 INFO:tasks.workunit.client.0.smithi096.stdout:5/72: readlink d4/l16 0 2024-04-22T15:18:33.893 INFO:tasks.workunit.client.0.smithi096.stdout:0/51: rmdir d7/d9 39 2024-04-22T15:18:33.893 INFO:tasks.workunit.client.0.smithi096.stdout:3/38: write f2 [1017101,27194] 0 2024-04-22T15:18:33.897 INFO:tasks.workunit.client.0.smithi096.stdout:2/39: creat d3/fd x:0 0 0 2024-04-22T15:18:33.899 INFO:tasks.workunit.client.0.smithi096.stdout:4/43: creat d5/fb x:0 0 0 2024-04-22T15:18:33.901 INFO:tasks.workunit.client.0.smithi096.stdout:5/73: symlink d4/d11/l1b 0 2024-04-22T15:18:33.903 INFO:tasks.workunit.client.0.smithi096.stdout:8/72: symlink d4/d10/l1a 0 2024-04-22T15:18:33.903 INFO:tasks.workunit.client.0.smithi096.stdout:3/39: dwrite f2 [0,4194304] 0 2024-04-22T15:18:33.904 INFO:tasks.workunit.client.0.smithi096.stdout:5/74: fsync d4/d12/f15 0 2024-04-22T15:18:33.905 INFO:tasks.workunit.client.0.smithi096.stdout:8/73: stat d4 0 2024-04-22T15:18:33.906 INFO:tasks.workunit.client.0.smithi096.stdout:2/40: creat d3/db/fe x:0 0 0 2024-04-22T15:18:33.909 INFO:tasks.workunit.client.0.smithi096.stdout:4/44: rename d5/f6 to d5/fc 0 2024-04-22T15:18:33.911 INFO:tasks.workunit.client.0.smithi096.stdout:0/52: fsync f5 0 2024-04-22T15:18:33.911 INFO:tasks.workunit.client.0.smithi096.stdout:3/40: readlink l6 0 2024-04-22T15:18:33.914 INFO:tasks.workunit.client.0.smithi096.stdout:3/41: read f2 [2653874,118963] 0 2024-04-22T15:18:33.914 INFO:tasks.workunit.client.0.smithi096.stdout:8/74: dwrite d4/f12 [0,4194304] 0 2024-04-22T15:18:33.915 INFO:tasks.workunit.client.0.smithi096.stdout:8/75: readlink d4/d10/l1a 0 2024-04-22T15:18:33.915 INFO:tasks.workunit.client.0.smithi096.stdout:2/41: mkdir d3/db/df 0 2024-04-22T15:18:33.917 INFO:tasks.workunit.client.0.smithi096.stdout:8/76: truncate d4/fd 4299948 0 2024-04-22T15:18:33.918 INFO:tasks.workunit.client.0.smithi096.stdout:5/75: link d4/d12/f13 d4/f1c 0 2024-04-22T15:18:33.918 INFO:tasks.workunit.client.0.smithi096.stdout:5/76: fdatasync f1 0 2024-04-22T15:18:33.918 INFO:tasks.workunit.client.0.smithi096.stdout:3/42: fsync f2 0 2024-04-22T15:18:33.919 INFO:tasks.workunit.client.0.smithi096.stdout:1/119: link c4 d6/dc/d11/c26 0 2024-04-22T15:18:33.926 INFO:tasks.workunit.client.0.smithi096.stdout:8/77: symlink d4/d10/l1b 0 2024-04-22T15:18:33.929 INFO:tasks.workunit.client.0.smithi096.stdout:0/53: dwrite f3 [0,4194304] 0 2024-04-22T15:18:33.930 INFO:tasks.workunit.client.0.smithi096.stdout:1/120: mknod d6/dc/df/d12/c27 0 2024-04-22T15:18:33.933 INFO:tasks.workunit.client.0.smithi096.stdout:8/78: dread d4/d10/f19 [0,4194304] 0 2024-04-22T15:18:33.934 INFO:tasks.workunit.client.0.smithi096.stdout:5/77: rename d4/l9 to d4/d19/l1d 0 2024-04-22T15:18:33.936 INFO:tasks.workunit.client.0.smithi096.stdout:2/42: mkdir d3/db/df/d10 0 2024-04-22T15:18:33.941 INFO:tasks.workunit.client.0.smithi096.stdout:3/43: unlink c4 0 2024-04-22T15:18:33.943 INFO:tasks.workunit.client.0.smithi096.stdout:1/121: symlink d6/dc/d10/l28 0 2024-04-22T15:18:33.943 INFO:tasks.workunit.client.0.smithi096.stdout:1/122: stat d6/dc/df/d12/c27 0 2024-04-22T15:18:33.950 INFO:tasks.workunit.client.0.smithi096.stdout:2/43: rename d3/fa to d3/db/df/d10/f11 0 2024-04-22T15:18:33.955 INFO:tasks.workunit.client.0.smithi096.stdout:1/123: mkdir d6/dc/d10/d29 0 2024-04-22T15:18:33.956 INFO:tasks.workunit.client.0.smithi096.stdout:3/44: dread f2 [0,4194304] 0 2024-04-22T15:18:33.968 INFO:tasks.workunit.client.0.smithi096.stdout:5/78: write d4/ff [797417,38498] 0 2024-04-22T15:18:33.971 INFO:tasks.workunit.client.0.smithi096.stdout:2/44: rmdir d3/db 39 2024-04-22T15:18:33.976 INFO:tasks.workunit.client.0.smithi096.stdout:1/124: unlink d6/f20 0 2024-04-22T15:18:33.977 INFO:tasks.workunit.client.0.smithi096.stdout:0/54: truncate f3 1844564 0 2024-04-22T15:18:33.980 INFO:tasks.workunit.client.0.smithi096.stdout:8/79: truncate d4/f12 4125885 0 2024-04-22T15:18:33.991 INFO:tasks.workunit.client.0.smithi096.stdout:0/55: fdatasync f3 0 2024-04-22T15:18:33.994 INFO:tasks.workunit.client.0.smithi096.stdout:0/56: dread f3 [0,4194304] 0 2024-04-22T15:18:33.997 INFO:tasks.workunit.client.0.smithi096.stdout:3/45: unlink f2 0 2024-04-22T15:18:33.997 INFO:tasks.workunit.client.0.smithi096.stdout:3/46: dread - no filename 2024-04-22T15:18:33.998 INFO:tasks.workunit.client.0.smithi096.stdout:0/57: dread f0 [0,4194304] 0 2024-04-22T15:18:34.004 INFO:tasks.workunit.client.0.smithi096.stdout:2/45: truncate d3/f8 970538 0 2024-04-22T15:18:34.004 INFO:tasks.workunit.client.0.smithi096.stdout:2/46: readlink d3/l5 0 2024-04-22T15:18:34.011 INFO:tasks.workunit.client.0.smithi096.stdout:3/47: rename da/db/de to da/db/de/df 22 2024-04-22T15:18:34.012 INFO:tasks.workunit.client.0.smithi096.stdout:0/58: creat d7/de/f12 x:0 0 0 2024-04-22T15:18:34.022 INFO:tasks.workunit.client.0.smithi096.stdout:5/79: dwrite d4/fa [0,4194304] 0 2024-04-22T15:18:34.030 INFO:tasks.workunit.client.0.smithi096.stdout:3/48: mkdir da/db/d10 0 2024-04-22T15:18:34.033 INFO:tasks.workunit.client.0.smithi096.stdout:5/80: dread d4/f7 [0,4194304] 0 2024-04-22T15:18:34.034 INFO:tasks.workunit.client.0.smithi096.stdout:8/80: dwrite d4/f12 [0,4194304] 0 2024-04-22T15:18:34.035 INFO:tasks.workunit.client.0.smithi096.stdout:5/81: chown d4/ce 129913381 1 2024-04-22T15:18:34.038 INFO:tasks.workunit.client.0.smithi096.stdout:1/125: link d6/dc/d10/l17 d6/dc/l2a 0 2024-04-22T15:18:34.045 INFO:tasks.workunit.client.0.smithi096.stdout:0/59: creat d7/d9/dc/f13 x:0 0 0 2024-04-22T15:18:34.046 INFO:tasks.workunit.client.0.smithi096.stdout:0/60: fsync d7/de/f12 0 2024-04-22T15:18:34.050 INFO:tasks.workunit.client.0.smithi096.stdout:8/81: write d4/fd [433814,128903] 0 2024-04-22T15:18:34.052 INFO:tasks.workunit.client.0.smithi096.stdout:8/82: write d4/f12 [1073520,30653] 0 2024-04-22T15:18:34.052 INFO:tasks.workunit.client.0.smithi096.stdout:5/82: mkdir d4/d1e 0 2024-04-22T15:18:34.054 INFO:tasks.workunit.client.0.smithi096.stdout:1/126: symlink d6/dc/d10/l2b 0 2024-04-22T15:18:34.054 INFO:tasks.workunit.client.0.smithi096.stdout:2/47: dwrite d3/f9 [0,4194304] 0 2024-04-22T15:18:34.057 INFO:tasks.workunit.client.0.smithi096.stdout:3/49: mknod da/db/de/c11 0 2024-04-22T15:18:34.057 INFO:tasks.workunit.client.0.smithi096.stdout:3/50: stat da/db 0 2024-04-22T15:18:34.059 INFO:tasks.workunit.client.0.smithi096.stdout:0/61: creat d7/d9/dc/f14 x:0 0 0 2024-04-22T15:18:34.064 INFO:tasks.workunit.client.0.smithi096.stdout:8/83: mkdir d4/d1c 0 2024-04-22T15:18:34.065 INFO:tasks.workunit.client.0.smithi096.stdout:8/84: write f3 [2274171,68631] 0 2024-04-22T15:18:34.067 INFO:tasks.workunit.client.0.smithi096.stdout:5/83: rename d4/d19 to d4/d1f 0 2024-04-22T15:18:34.070 INFO:tasks.workunit.client.0.smithi096.stdout:8/85: dread d4/fd [0,4194304] 0 2024-04-22T15:18:34.070 INFO:tasks.workunit.client.0.smithi096.stdout:2/48: chown d3/db/df/d10/f11 50343 1 2024-04-22T15:18:34.071 INFO:tasks.workunit.client.0.smithi096.stdout:2/49: chown d3/db/df/d10/f11 0 1 2024-04-22T15:18:34.073 INFO:tasks.workunit.client.0.smithi096.stdout:3/51: rmdir da/db 39 2024-04-22T15:18:34.073 INFO:tasks.workunit.client.0.smithi096.stdout:3/52: chown da 0 1 2024-04-22T15:18:34.075 INFO:tasks.workunit.client.0.smithi096.stdout:0/62: unlink f5 0 2024-04-22T15:18:34.078 INFO:tasks.workunit.client.0.smithi096.stdout:0/63: dread f0 [0,4194304] 0 2024-04-22T15:18:34.085 INFO:tasks.workunit.client.0.smithi096.stdout:0/64: dwrite d7/de/f12 [0,4194304] 0 2024-04-22T15:18:34.091 INFO:tasks.workunit.client.0.smithi096.stdout:2/50: rename d3/f9 to d3/db/f12 0 2024-04-22T15:18:34.105 INFO:tasks.workunit.client.0.smithi096.stdout:8/86: mknod d4/d1c/c1d 0 2024-04-22T15:18:34.106 INFO:tasks.workunit.client.0.smithi096.stdout:8/87: write d4/f12 [2815179,47603] 0 2024-04-22T15:18:34.109 INFO:tasks.workunit.client.0.smithi096.stdout:0/65: mkdir d7/d9/d15 0 2024-04-22T15:18:34.113 INFO:tasks.workunit.client.0.smithi096.stdout:2/51: unlink d3/db/fe 0 2024-04-22T15:18:34.115 INFO:tasks.workunit.client.0.smithi096.stdout:3/53: mkdir da/db/d12 0 2024-04-22T15:18:34.115 INFO:tasks.workunit.client.0.smithi096.stdout:3/54: dwrite - no filename 2024-04-22T15:18:34.118 INFO:tasks.workunit.client.0.smithi096.stdout:1/127: truncate d6/f8 2713097 0 2024-04-22T15:18:34.118 INFO:tasks.workunit.client.0.smithi096.stdout:5/84: dwrite d4/f7 [0,4194304] 0 2024-04-22T15:18:34.119 INFO:tasks.workunit.client.0.smithi096.stdout:1/128: chown d6/dc/df/f25 251121 1 2024-04-22T15:18:34.119 INFO:tasks.workunit.client.0.smithi096.stdout:5/85: dread - d4/f1c zero size 2024-04-22T15:18:34.124 INFO:tasks.workunit.client.0.smithi096.stdout:0/66: rmdir d7/d9 39 2024-04-22T15:18:34.128 INFO:tasks.workunit.client.0.smithi096.stdout:2/52: write d3/f4 [2101003,20841] 0 2024-04-22T15:18:34.139 INFO:tasks.workunit.client.0.smithi096.stdout:5/86: symlink d4/d1f/l20 0 2024-04-22T15:18:34.145 INFO:tasks.workunit.client.0.smithi096.stdout:8/88: dwrite d4/fd [0,4194304] 0 2024-04-22T15:18:34.148 INFO:tasks.workunit.client.0.smithi096.stdout:3/55: mkdir da/d13 0 2024-04-22T15:18:34.152 INFO:tasks.workunit.client.0.smithi096.stdout:5/87: creat d4/d1f/f21 x:0 0 0 2024-04-22T15:18:34.154 INFO:tasks.workunit.client.0.smithi096.stdout:0/67: mkdir d7/d9/dc/d16 0 2024-04-22T15:18:34.155 INFO:tasks.workunit.client.0.smithi096.stdout:0/68: write d7/de/f12 [4169592,58428] 0 2024-04-22T15:18:34.157 INFO:tasks.workunit.client.0.smithi096.stdout:0/69: dread f3 [0,4194304] 0 2024-04-22T15:18:34.158 INFO:tasks.workunit.client.0.smithi096.stdout:2/53: mkdir d3/d13 0 2024-04-22T15:18:34.161 INFO:tasks.workunit.client.0.smithi096.stdout:8/89: symlink d4/d1c/l1e 0 2024-04-22T15:18:34.163 INFO:tasks.workunit.client.0.smithi096.stdout:2/54: dread d3/f4 [0,4194304] 0 2024-04-22T15:18:34.164 INFO:tasks.workunit.client.0.smithi096.stdout:1/129: creat d6/f2c x:0 0 0 2024-04-22T15:18:34.164 INFO:tasks.workunit.client.0.smithi096.stdout:1/130: readlink d6/l9 0 2024-04-22T15:18:34.166 INFO:tasks.workunit.client.0.smithi096.stdout:8/90: dread f3 [0,4194304] 0 2024-04-22T15:18:34.172 INFO:tasks.workunit.client.0.smithi096.stdout:2/55: fsync d3/db/f12 0 2024-04-22T15:18:34.174 INFO:tasks.workunit.client.0.smithi096.stdout:5/88: link f1 d4/d11/f22 0 2024-04-22T15:18:34.174 INFO:tasks.workunit.client.0.smithi096.stdout:5/89: chown f2 483078661 1 2024-04-22T15:18:34.176 INFO:tasks.workunit.client.0.smithi096.stdout:1/131: symlink d6/dc/d10/l2d 0 2024-04-22T15:18:34.177 INFO:tasks.workunit.client.0.smithi096.stdout:1/132: write d6/f22 [309389,88863] 0 2024-04-22T15:18:34.182 INFO:tasks.workunit.client.0.smithi096.stdout:0/70: link d7/d9/dc/f13 d7/d9/f17 0 2024-04-22T15:18:34.183 INFO:tasks.workunit.client.0.smithi096.stdout:0/71: chown d7/de/ld 1409334 1 2024-04-22T15:18:34.188 INFO:tasks.workunit.client.0.smithi096.stdout:0/72: dwrite d7/de/f12 [0,4194304] 0 2024-04-22T15:18:34.188 INFO:tasks.workunit.client.0.smithi096.stdout:5/90: rmdir d4/d12 39 2024-04-22T15:18:34.189 INFO:tasks.workunit.client.0.smithi096.stdout:0/73: dread - d7/d9/f17 zero size 2024-04-22T15:18:34.195 INFO:tasks.workunit.client.0.smithi096.stdout:5/91: dwrite d4/fd [0,4194304] 0 2024-04-22T15:18:34.197 INFO:tasks.workunit.client.0.smithi096.stdout:2/56: mknod d3/d13/c14 0 2024-04-22T15:18:34.203 INFO:tasks.workunit.client.0.smithi096.stdout:8/91: write d4/d10/f19 [1996342,4522] 0 2024-04-22T15:18:34.210 INFO:tasks.workunit.client.0.smithi096.stdout:2/57: unlink d3/f4 0 2024-04-22T15:18:34.214 INFO:tasks.workunit.client.0.smithi096.stdout:1/133: creat d6/dc/f2e x:0 0 0 2024-04-22T15:18:34.215 INFO:tasks.workunit.client.0.smithi096.stdout:1/134: write d6/dc/d11/f19 [3647830,98906] 0 2024-04-22T15:18:34.216 INFO:tasks.workunit.client.0.smithi096.stdout:0/74: mkdir d7/d9/dc/d16/d18 0 2024-04-22T15:18:34.218 INFO:tasks.workunit.client.0.smithi096.stdout:5/92: mkdir d4/d12/d23 0 2024-04-22T15:18:34.219 INFO:tasks.workunit.client.0.smithi096.stdout:5/93: fdatasync d4/f5 0 2024-04-22T15:18:34.222 INFO:tasks.workunit.client.0.smithi096.stdout:1/135: dwrite d6/dc/df/d12/f18 [0,4194304] 0 2024-04-22T15:18:34.229 INFO:tasks.workunit.client.0.smithi096.stdout:0/75: rename f3 to d7/de/f19 0 2024-04-22T15:18:34.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:33 smithi096 ceph-mon[26688]: pgmap v23: 65 pgs: 65 active+clean; 190 MiB data, 785 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-04-22T15:18:34.243 INFO:tasks.workunit.client.0.smithi096.stdout:8/92: write d4/fa [361000,6465] 0 2024-04-22T15:18:34.244 INFO:tasks.workunit.client.0.smithi096.stdout:1/136: creat d6/dc/d10/d29/f2f x:0 0 0 2024-04-22T15:18:34.246 INFO:tasks.workunit.client.0.smithi096.stdout:1/137: dread - d6/dc/d10/d29/f2f zero size 2024-04-22T15:18:34.247 INFO:tasks.workunit.client.0.smithi096.stdout:2/58: dread d3/db/df/d10/f11 [0,4194304] 0 2024-04-22T15:18:34.247 INFO:tasks.workunit.client.0.smithi096.stdout:0/76: creat d7/d9/d15/f1a x:0 0 0 2024-04-22T15:18:34.251 INFO:tasks.workunit.client.0.smithi096.stdout:8/93: dwrite d4/f12 [0,4194304] 0 2024-04-22T15:18:34.254 INFO:tasks.workunit.client.0.smithi096.stdout:8/94: chown d4/d10/l1b 161 1 2024-04-22T15:18:34.254 INFO:tasks.workunit.client.0.smithi096.stdout:1/138: dread d6/f14 [0,4194304] 0 2024-04-22T15:18:34.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.251+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fee6402ce50 con 0x7fee5006dc50 2024-04-22T15:18:34.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.252+0000 7fee5b7fe700 1 -- 172.21.15.96:0/523720179 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fee6402ce50 con 0x7fee5006dc50 2024-04-22T15:18:34.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fee5006dc50 msgr2=0x7fee50070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:34.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fee5006dc50 0x7fee50070110 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fee640683c0 tx=0x7fee6000b410 comp rx=0 tx=0).stop 2024-04-22T15:18:34.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 msgr2=0x7fee64196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:34.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 0x7fee64196bb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fee54006010 tx=0x7fee5400bd70 comp rx=0 tx=0).stop 2024-04-22T15:18:34.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 shutdown_connections 2024-04-22T15:18:34.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fee5006dc50 0x7fee50070110 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee640feed0 0x7fee64196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 --2- 172.21.15.96:0/523720179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee64101eb0 0x7fee64196bb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 >> 172.21.15.96:0/523720179 conn(0x7fee640fa830 msgr2=0x7fee641054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:34.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 shutdown_connections 2024-04-22T15:18:34.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.256+0000 7fee6c7c8700 1 -- 172.21.15.96:0/523720179 wait complete. 2024-04-22T15:18:34.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:33 smithi196 ceph-mon[34717]: pgmap v23: 65 pgs: 65 active+clean; 190 MiB data, 785 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-04-22T15:18:34.281 INFO:tasks.workunit.client.0.smithi096.stdout:7/41: sync 2024-04-22T15:18:34.281 INFO:tasks.workunit.client.0.smithi096.stdout:7/42: stat f5 0 2024-04-22T15:18:34.282 INFO:tasks.workunit.client.0.smithi096.stdout:7/43: truncate fb 359820 0 2024-04-22T15:18:34.287 INFO:tasks.workunit.client.0.smithi096.stdout:8/95: symlink d4/l1f 0 2024-04-22T15:18:34.289 INFO:tasks.workunit.client.0.smithi096.stdout:1/139: write d6/f14 [3210785,38049] 0 2024-04-22T15:18:34.290 INFO:tasks.workunit.client.0.smithi096.stdout:1/140: stat d6/dc/d10/l2b 0 2024-04-22T15:18:34.291 INFO:tasks.workunit.client.0.smithi096.stdout:8/96: dread d4/f12 [0,4194304] 0 2024-04-22T15:18:34.292 INFO:tasks.workunit.client.0.smithi096.stdout:7/44: mknod cf 0 2024-04-22T15:18:34.292 INFO:tasks.workunit.client.0.smithi096.stdout:7/45: fdatasync fb 0 2024-04-22T15:18:34.295 INFO:tasks.workunit.client.0.smithi096.stdout:2/59: rename d3/d13/c14 to d3/c15 0 2024-04-22T15:18:34.299 INFO:tasks.workunit.client.0.smithi096.stdout:7/46: dwrite fd [0,4194304] 0 2024-04-22T15:18:34.300 INFO:tasks.workunit.client.0.smithi096.stdout:1/141: creat d6/dc/d10/f30 x:0 0 0 2024-04-22T15:18:34.302 INFO:tasks.workunit.client.0.smithi096.stdout:8/97: mknod d4/d10/c20 0 2024-04-22T15:18:34.309 INFO:tasks.workunit.client.0.smithi096.stdout:7/47: dwrite f3 [0,4194304] 0 2024-04-22T15:18:34.313 INFO:tasks.workunit.client.0.smithi096.stdout:1/142: creat d6/dc/d11/f31 x:0 0 0 2024-04-22T15:18:34.318 INFO:tasks.workunit.client.0.smithi096.stdout:8/98: dwrite d4/fa [0,4194304] 0 2024-04-22T15:18:34.321 INFO:tasks.workunit.client.0.smithi096.stdout:1/143: write d6/f14 [3805412,59599] 0 2024-04-22T15:18:34.321 INFO:tasks.workunit.client.0.smithi096.stdout:0/77: dwrite d7/d9/dc/f13 [0,4194304] 0 2024-04-22T15:18:34.325 INFO:tasks.workunit.client.0.smithi096.stdout:0/78: dread - d7/d9/dc/f14 zero size 2024-04-22T15:18:34.327 INFO:tasks.workunit.client.0.smithi096.stdout:8/99: creat d4/d10/f21 x:0 0 0 2024-04-22T15:18:34.327 INFO:tasks.workunit.client.0.smithi096.stdout:0/79: stat d7/d9/f17 0 2024-04-22T15:18:34.329 INFO:tasks.workunit.client.0.smithi096.stdout:7/48: dwrite fb [0,4194304] 0 2024-04-22T15:18:34.330 INFO:tasks.workunit.client.0.smithi096.stdout:7/49: chown l1 3218 1 2024-04-22T15:18:34.336 INFO:tasks.workunit.client.0.smithi096.stdout:1/144: mkdir d6/dc/df/d12/d32 0 2024-04-22T15:18:34.341 INFO:tasks.workunit.client.0.smithi096.stdout:0/80: symlink d7/de/l1b 0 2024-04-22T15:18:34.342 INFO:tasks.workunit.client.0.smithi096.stdout:8/100: dwrite d4/fd [4194304,4194304] 0 2024-04-22T15:18:34.344 INFO:tasks.workunit.client.0.smithi096.stdout:7/50: creat f10 x:0 0 0 2024-04-22T15:18:34.349 INFO:tasks.workunit.client.0.smithi096.stdout:0/81: write f0 [2000995,98882] 0 2024-04-22T15:18:34.349 INFO:tasks.workunit.client.0.smithi096.stdout:8/101: readlink d4/d10/l11 0 2024-04-22T15:18:34.352 INFO:tasks.workunit.client.0.smithi096.stdout:1/145: dwrite d6/dc/df/f1a [4194304,4194304] 0 2024-04-22T15:18:34.354 INFO:tasks.workunit.client.0.smithi096.stdout:7/51: rename fc to f11 0 2024-04-22T15:18:34.354 INFO:tasks.workunit.client.0.smithi096.stdout:1/146: chown d6/d1b 142143753 1 2024-04-22T15:18:34.356 INFO:tasks.workunit.client.0.smithi096.stdout:1/147: stat d6/dc/f2e 0 2024-04-22T15:18:34.359 INFO:tasks.workunit.client.0.smithi096.stdout:1/148: dread - d6/dc/df/f25 zero size 2024-04-22T15:18:34.360 INFO:tasks.workunit.client.0.smithi096.stdout:2/60: dwrite d3/db/df/d10/f11 [0,4194304] 0 2024-04-22T15:18:34.364 INFO:tasks.workunit.client.0.smithi096.stdout:0/82: dwrite d7/de/f12 [0,4194304] 0 2024-04-22T15:18:34.373 INFO:tasks.workunit.client.0.smithi096.stdout:2/61: symlink d3/db/df/d10/l16 0 2024-04-22T15:18:34.377 INFO:tasks.workunit.client.0.smithi096.stdout:0/83: dwrite d7/d9/dc/f14 [0,4194304] 0 2024-04-22T15:18:34.377 INFO:tasks.workunit.client.0.smithi096.stdout:2/62: truncate d3/fd 477783 0 2024-04-22T15:18:34.379 INFO:tasks.workunit.client.0.smithi096.stdout:7/52: dwrite f11 [0,4194304] 0 2024-04-22T15:18:34.379 INFO:tasks.workunit.client.0.smithi096.stdout:2/63: write d3/db/f12 [4475479,24566] 0 2024-04-22T15:18:34.380 INFO:tasks.workunit.client.0.smithi096.stdout:7/53: chown f7 285525 1 2024-04-22T15:18:34.388 INFO:tasks.workunit.client.0.smithi096.stdout:0/84: creat d7/d9/dc/f1c x:0 0 0 2024-04-22T15:18:34.390 INFO:tasks.workunit.client.0.smithi096.stdout:8/102: link d4/lb d4/d1c/l22 0 2024-04-22T15:18:34.390 INFO:tasks.workunit.client.0.smithi096.stdout:0/85: stat d7/d9/d15/f1a 0 2024-04-22T15:18:34.391 INFO:tasks.workunit.client.0.smithi096.stdout:2/64: dwrite f0 [0,4194304] 0 2024-04-22T15:18:34.391 INFO:tasks.workunit.client.0.smithi096.stdout:7/54: dread f7 [0,4194304] 0 2024-04-22T15:18:34.394 INFO:tasks.workunit.client.0.smithi096.stdout:2/65: chown d3/db 27102 1 2024-04-22T15:18:34.401 INFO:tasks.workunit.client.0.smithi096.stdout:8/103: dread d4/d10/f19 [0,4194304] 0 2024-04-22T15:18:34.405 INFO:tasks.workunit.client.0.smithi096.stdout:0/86: dwrite d7/d9/f17 [0,4194304] 0 2024-04-22T15:18:34.407 INFO:tasks.workunit.client.0.smithi096.stdout:2/66: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:18:34.411 INFO:tasks.workunit.client.0.smithi096.stdout:0/87: write d7/de/f12 [3605009,103290] 0 2024-04-22T15:18:34.411 INFO:tasks.workunit.client.0.smithi096.stdout:1/149: write d6/dc/df/d12/f18 [5044068,53922] 0 2024-04-22T15:18:34.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.430+0000 7f4293d4a700 1 -- 172.21.15.96:0/1947921393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c103fd0 msgr2=0x7f428c1063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:34.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.430+0000 7f4293d4a700 1 --2- 172.21.15.96:0/1947921393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c103fd0 0x7f428c1063c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f427c009910 tx=0x7f427c009c20 comp rx=0 tx=0).stop 2024-04-22T15:18:34.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.430+0000 7f4293d4a700 1 -- 172.21.15.96:0/1947921393 shutdown_connections 2024-04-22T15:18:34.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.430+0000 7f4293d4a700 1 --2- 172.21.15.96:0/1947921393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c103fd0 0x7f428c1063c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.430+0000 7f4293d4a700 1 --2- 172.21.15.96:0/1947921393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c1016a0 0x7f428c103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.430+0000 7f4293d4a700 1 -- 172.21.15.96:0/1947921393 >> 172.21.15.96:0/1947921393 conn(0x7f428c0fb0b0 msgr2=0x7f428c0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:34.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.430+0000 7f4293d4a700 1 -- 172.21.15.96:0/1947921393 shutdown_connections 2024-04-22T15:18:34.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.430+0000 7f4293d4a700 1 -- 172.21.15.96:0/1947921393 wait complete. 2024-04-22T15:18:34.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.431+0000 7f4293d4a700 1 Processor -- start 2024-04-22T15:18:34.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.431+0000 7f4293d4a700 1 -- start start 2024-04-22T15:18:34.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.431+0000 7f4293d4a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c1016a0 0x7f428c1965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:34.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.431+0000 7f4293d4a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c103fd0 0x7f428c196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:34.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.431+0000 7f4293d4a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1970a0 con 0x7f428c1016a0 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.431+0000 7f4293d4a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1971e0 con 0x7f428c103fd0 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f4291ae6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c1016a0 0x7f428c1965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f4291ae6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c1016a0 0x7f428c1965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56504/0 (socket says 172.21.15.96:56504) 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f42912e5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c103fd0 0x7f428c196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f4291ae6700 1 -- 172.21.15.96:0/980765231 learned_addr learned my addr 172.21.15.96:0/980765231 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f4291ae6700 1 -- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c103fd0 msgr2=0x7f428c196b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f4291ae6700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c103fd0 0x7f428c196b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f4291ae6700 1 -- 172.21.15.96:0/980765231 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c0095f0 con 0x7f428c1016a0 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f42912e5700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c103fd0 0x7f428c196b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f4291ae6700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c1016a0 0x7f428c1965d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f428800c930 tx=0x7f428800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f4282ffd700 1 -- 172.21.15.96:0/980765231 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4288007ad0 con 0x7f428c1016a0 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.432+0000 7f4282ffd700 1 -- 172.21.15.96:0/980765231 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f428800ce80 con 0x7f428c1016a0 2024-04-22T15:18:34.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.433+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c19bc50 con 0x7f428c1016a0 2024-04-22T15:18:34.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.433+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c19c170 con 0x7f428c1016a0 2024-04-22T15:18:34.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.433+0000 7f4282ffd700 1 -- 172.21.15.96:0/980765231 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4288007ad0 con 0x7f428c1016a0 2024-04-22T15:18:34.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.433+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c04ebb0 con 0x7f428c1016a0 2024-04-22T15:18:34.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.435+0000 7f4282ffd700 1 -- 172.21.15.96:0/980765231 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f4288007c30 con 0x7f428c1016a0 2024-04-22T15:18:34.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.435+0000 7f4282ffd700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f427806dcb0 0x7f4278070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:34.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.435+0000 7f4282ffd700 1 -- 172.21.15.96:0/980765231 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f428808d540 con 0x7f428c1016a0 2024-04-22T15:18:34.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.435+0000 7f42912e5700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f427806dcb0 0x7f4278070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:34.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.436+0000 7f42912e5700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f427806dcb0 0x7f4278070170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f427c006010 tx=0x7f427c005180 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.442+0000 7f4282ffd700 1 -- 172.21.15.96:0/980765231 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f428805a350 con 0x7f428c1016a0 2024-04-22T15:18:34.446 INFO:tasks.workunit.client.0.smithi096.stdout:7/55: link fd f12 0 2024-04-22T15:18:34.447 INFO:tasks.workunit.client.0.smithi096.stdout:8/104: dread d4/fd [4194304,4194304] 0 2024-04-22T15:18:34.447 INFO:tasks.workunit.client.0.smithi096.stdout:0/88: fsync d7/d9/dc/f1c 0 2024-04-22T15:18:34.447 INFO:tasks.workunit.client.0.smithi096.stdout:1/150: rmdir d6/dc/d11 39 2024-04-22T15:18:34.449 INFO:tasks.workunit.client.0.smithi096.stdout:2/67: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:18:34.450 INFO:tasks.workunit.client.0.smithi096.stdout:0/89: write f0 [5055241,26426] 0 2024-04-22T15:18:34.451 INFO:tasks.workunit.client.0.smithi096.stdout:7/56: mknod c13 0 2024-04-22T15:18:34.451 INFO:tasks.workunit.client.0.smithi096.stdout:6/50: sync 2024-04-22T15:18:34.452 INFO:tasks.workunit.client.0.smithi096.stdout:6/51: write f0 [3875601,245] 0 2024-04-22T15:18:34.455 INFO:tasks.workunit.client.0.smithi096.stdout:2/68: dread f0 [0,4194304] 0 2024-04-22T15:18:34.457 INFO:tasks.workunit.client.0.smithi096.stdout:2/69: dread d3/fd [0,4194304] 0 2024-04-22T15:18:34.460 INFO:tasks.workunit.client.0.smithi096.stdout:7/57: unlink f12 0 2024-04-22T15:18:34.465 INFO:tasks.workunit.client.0.smithi096.stdout:6/52: mkdir d3/d7/dd/d11 0 2024-04-22T15:18:34.467 INFO:tasks.workunit.client.0.smithi096.stdout:1/151: mknod d6/dc/d11/c33 0 2024-04-22T15:18:34.491 INFO:tasks.workunit.client.0.smithi096.stdout:7/58: symlink l14 0 2024-04-22T15:18:34.492 INFO:tasks.workunit.client.0.smithi096.stdout:8/105: write f3 [3402777,46174] 0 2024-04-22T15:18:34.492 INFO:tasks.workunit.client.0.smithi096.stdout:6/53: unlink d3/f4 0 2024-04-22T15:18:34.493 INFO:tasks.workunit.client.0.smithi096.stdout:7/59: write fb [4215375,123934] 0 2024-04-22T15:18:34.493 INFO:tasks.workunit.client.0.smithi096.stdout:6/54: write f2 [3063320,15631] 0 2024-04-22T15:18:34.495 INFO:tasks.workunit.client.0.smithi096.stdout:1/152: rename d6/dc/df to d6/d1b/d34 0 2024-04-22T15:18:34.502 INFO:tasks.workunit.client.0.smithi096.stdout:8/106: chown d4/lb 53189640 1 2024-04-22T15:18:34.503 INFO:tasks.workunit.client.0.smithi096.stdout:7/60: mknod c15 0 2024-04-22T15:18:34.504 INFO:tasks.workunit.client.0.smithi096.stdout:7/61: write fb [3803500,15927] 0 2024-04-22T15:18:34.507 INFO:tasks.workunit.client.0.smithi096.stdout:0/90: getdents d7/d9/dc/d16 0 2024-04-22T15:18:34.509 INFO:tasks.workunit.client.0.smithi096.stdout:8/107: creat d4/d10/f23 x:0 0 0 2024-04-22T15:18:34.512 INFO:tasks.workunit.client.0.smithi096.stdout:6/55: rename d3/d7/c8 to d3/d7/dd/d11/c12 0 2024-04-22T15:18:34.513 INFO:tasks.workunit.client.0.smithi096.stdout:7/62: truncate f7 874191 0 2024-04-22T15:18:34.513 INFO:tasks.workunit.client.0.smithi096.stdout:6/56: read d3/d7/ff [2281333,87409] 0 2024-04-22T15:18:34.516 INFO:tasks.workunit.client.0.smithi096.stdout:0/91: creat d7/d9/dc/d16/f1d x:0 0 0 2024-04-22T15:18:34.517 INFO:tasks.workunit.client.0.smithi096.stdout:8/108: mknod d4/d10/c24 0 2024-04-22T15:18:34.517 INFO:tasks.workunit.client.0.smithi096.stdout:8/109: fdatasync f3 0 2024-04-22T15:18:34.518 INFO:tasks.workunit.client.0.smithi096.stdout:7/63: dread f11 [0,4194304] 0 2024-04-22T15:18:34.523 INFO:tasks.workunit.client.0.smithi096.stdout:6/57: symlink d3/d7/dd/d11/l13 0 2024-04-22T15:18:34.526 INFO:tasks.workunit.client.0.smithi096.stdout:7/64: creat f16 x:0 0 0 2024-04-22T15:18:34.532 INFO:tasks.workunit.client.0.smithi096.stdout:1/153: rmdir d6/dc 39 2024-04-22T15:18:34.534 INFO:tasks.workunit.client.0.smithi096.stdout:7/65: mknod c17 0 2024-04-22T15:18:34.539 INFO:tasks.workunit.client.0.smithi096.stdout:1/154: dread - d6/dc/d11/f31 zero size 2024-04-22T15:18:34.540 INFO:tasks.workunit.client.0.smithi096.stdout:1/155: fdatasync d6/d1b/d34/f25 0 2024-04-22T15:18:34.541 INFO:tasks.workunit.client.0.smithi096.stdout:1/156: write d6/dc/f23 [947413,64992] 0 2024-04-22T15:18:34.542 INFO:tasks.workunit.client.0.smithi096.stdout:1/157: write f2 [7724853,29817] 0 2024-04-22T15:18:34.543 INFO:tasks.workunit.client.0.smithi096.stdout:7/66: creat f18 x:0 0 0 2024-04-22T15:18:34.543 INFO:tasks.workunit.client.0.smithi096.stdout:7/67: rmdir - no directory 2024-04-22T15:18:34.543 INFO:tasks.workunit.client.0.smithi096.stdout:1/158: stat d6 0 2024-04-22T15:18:34.544 INFO:tasks.workunit.client.0.smithi096.stdout:0/92: getdents d7/d9 0 2024-04-22T15:18:34.547 INFO:tasks.workunit.client.0.smithi096.stdout:6/58: link d3/d7/dd/ce d3/d7/dd/c14 0 2024-04-22T15:18:34.550 INFO:tasks.workunit.client.0.smithi096.stdout:7/68: symlink l19 0 2024-04-22T15:18:34.551 INFO:tasks.workunit.client.0.smithi096.stdout:7/69: chown l14 3007 1 2024-04-22T15:18:34.552 INFO:tasks.workunit.client.0.smithi096.stdout:7/70: dread - f18 zero size 2024-04-22T15:18:34.552 INFO:tasks.workunit.client.0.smithi096.stdout:0/93: creat d7/d9/dc/d16/f1e x:0 0 0 2024-04-22T15:18:34.554 INFO:tasks.workunit.client.0.smithi096.stdout:6/59: dwrite f0 [0,4194304] 0 2024-04-22T15:18:34.555 INFO:tasks.workunit.client.0.smithi096.stdout:1/159: getdents d6/d1b/d34/d12/d32 0 2024-04-22T15:18:34.555 INFO:tasks.workunit.client.0.smithi096.stdout:6/60: write f2 [4241489,10385] 0 2024-04-22T15:18:34.560 INFO:tasks.workunit.client.0.smithi096.stdout:0/94: dread f0 [4194304,4194304] 0 2024-04-22T15:18:34.563 INFO:tasks.workunit.client.0.smithi096.stdout:6/61: mknod d3/c15 0 2024-04-22T15:18:34.566 INFO:tasks.workunit.client.0.smithi096.stdout:6/62: dread f0 [0,4194304] 0 2024-04-22T15:18:34.567 INFO:tasks.workunit.client.0.smithi096.stdout:7/71: rename fd to f1a 0 2024-04-22T15:18:34.569 INFO:tasks.workunit.client.0.smithi096.stdout:8/110: truncate d4/fa 2544904 0 2024-04-22T15:18:34.570 INFO:tasks.workunit.client.0.smithi096.stdout:8/111: read d4/fd [615685,64075] 0 2024-04-22T15:18:34.571 INFO:tasks.workunit.client.0.smithi096.stdout:0/95: creat d7/d9/dc/d16/f1f x:0 0 0 2024-04-22T15:18:34.572 INFO:tasks.workunit.client.0.smithi096.stdout:6/63: dread f1 [0,4194304] 0 2024-04-22T15:18:34.572 INFO:tasks.workunit.client.0.smithi096.stdout:0/96: dread - d7/d9/dc/d16/f1e zero size 2024-04-22T15:18:34.573 INFO:tasks.workunit.client.0.smithi096.stdout:1/160: creat d6/d1b/d34/d12/d32/f35 x:0 0 0 2024-04-22T15:18:34.574 INFO:tasks.workunit.client.0.smithi096.stdout:0/97: write d7/d9/dc/f13 [4368764,5381] 0 2024-04-22T15:18:34.577 INFO:tasks.workunit.client.0.smithi096.stdout:8/112: mkdir d4/d10/d25 0 2024-04-22T15:18:34.579 INFO:tasks.workunit.client.0.smithi096.stdout:6/64: creat d3/d7/dd/f16 x:0 0 0 2024-04-22T15:18:34.591 INFO:tasks.workunit.client.0.smithi096.stdout:7/72: dread f7 [0,4194304] 0 2024-04-22T15:18:34.593 INFO:tasks.workunit.client.0.smithi096.stdout:8/113: truncate f3 1380030 0 2024-04-22T15:18:34.593 INFO:tasks.workunit.client.0.smithi096.stdout:8/114: readlink d4/l1f 0 2024-04-22T15:18:34.596 INFO:tasks.workunit.client.0.smithi096.stdout:0/98: creat d7/f20 x:0 0 0 2024-04-22T15:18:34.597 INFO:tasks.workunit.client.0.smithi096.stdout:0/99: readlink d7/de/l1b 0 2024-04-22T15:18:34.598 INFO:tasks.workunit.client.0.smithi096.stdout:7/73: symlink l1b 0 2024-04-22T15:18:34.598 INFO:tasks.workunit.client.0.smithi096.stdout:7/74: fdatasync f16 0 2024-04-22T15:18:34.598 INFO:tasks.workunit.client.0.smithi096.stdout:7/75: dread - f18 zero size 2024-04-22T15:18:34.600 INFO:tasks.workunit.client.0.smithi096.stdout:7/76: write f18 [495717,48346] 0 2024-04-22T15:18:34.600 INFO:tasks.workunit.client.0.smithi096.stdout:1/161: getdents d6/dc/d10/d29 0 2024-04-22T15:18:34.603 INFO:tasks.workunit.client.0.smithi096.stdout:0/100: symlink d7/d9/d15/l21 0 2024-04-22T15:18:34.603 INFO:tasks.workunit.client.0.smithi096.stdout:7/77: dread f7 [0,4194304] 0 2024-04-22T15:18:34.604 INFO:tasks.workunit.client.0.smithi096.stdout:8/115: creat d4/d10/d25/f26 x:0 0 0 2024-04-22T15:18:34.607 INFO:tasks.workunit.client.0.smithi096.stdout:6/65: dwrite d3/d7/ff [0,4194304] 0 2024-04-22T15:18:34.609 INFO:tasks.workunit.client.0.smithi096.stdout:7/78: dread fb [0,4194304] 0 2024-04-22T15:18:34.615 INFO:tasks.workunit.client.0.smithi096.stdout:6/66: read f0 [1862352,123299] 0 2024-04-22T15:18:34.616 INFO:tasks.workunit.client.0.smithi096.stdout:7/79: creat f1c x:0 0 0 2024-04-22T15:18:34.617 INFO:tasks.workunit.client.0.smithi096.stdout:0/101: mknod d7/d9/dc/d16/d18/c22 0 2024-04-22T15:18:34.618 INFO:tasks.workunit.client.0.smithi096.stdout:0/102: chown d7/de 72 1 2024-04-22T15:18:34.622 INFO:tasks.workunit.client.0.smithi096.stdout:6/67: creat d3/d7/f17 x:0 0 0 2024-04-22T15:18:34.623 INFO:tasks.workunit.client.0.smithi096.stdout:1/162: truncate d6/f1d 444626 0 2024-04-22T15:18:34.623 INFO:tasks.workunit.client.0.smithi096.stdout:1/163: readlink d6/dc/le 0 2024-04-22T15:18:34.626 INFO:tasks.workunit.client.0.smithi096.stdout:0/103: creat d7/d9/dc/d16/d18/f23 x:0 0 0 2024-04-22T15:18:34.626 INFO:tasks.workunit.client.0.smithi096.stdout:8/116: link d4/c15 d4/c27 0 2024-04-22T15:18:34.627 INFO:tasks.workunit.client.0.smithi096.stdout:6/68: creat d3/d7/dd/d11/f18 x:0 0 0 2024-04-22T15:18:34.628 INFO:tasks.workunit.client.0.smithi096.stdout:7/80: link f3 f1d 0 2024-04-22T15:18:34.629 INFO:tasks.workunit.client.0.smithi096.stdout:1/164: creat d6/dc/d10/d29/f36 x:0 0 0 2024-04-22T15:18:34.630 INFO:tasks.workunit.client.0.smithi096.stdout:0/104: creat d7/d9/f24 x:0 0 0 2024-04-22T15:18:34.633 INFO:tasks.workunit.client.0.smithi096.stdout:6/69: chown d3/d7/dd/d11/c12 231211 1 2024-04-22T15:18:34.633 INFO:tasks.workunit.client.0.smithi096.stdout:7/81: creat f1e x:0 0 0 2024-04-22T15:18:34.634 INFO:tasks.workunit.client.0.smithi096.stdout:1/165: dread d6/d1b/d34/f1a [4194304,4194304] 0 2024-04-22T15:18:34.636 INFO:tasks.workunit.client.0.smithi096.stdout:1/166: truncate d6/d1b/d34/d12/f18 5327696 0 2024-04-22T15:18:34.637 INFO:tasks.workunit.client.0.smithi096.stdout:8/117: link d4/fd d4/f28 0 2024-04-22T15:18:34.637 INFO:tasks.workunit.client.0.smithi096.stdout:8/118: readlink d4/d10/l18 0 2024-04-22T15:18:34.640 INFO:tasks.workunit.client.0.smithi096.stdout:7/82: fsync f1c 0 2024-04-22T15:18:34.643 INFO:tasks.workunit.client.0.smithi096.stdout:1/167: dwrite d6/f22 [0,4194304] 0 2024-04-22T15:18:34.655 INFO:tasks.workunit.client.0.smithi096.stdout:0/105: dwrite f0 [0,4194304] 0 2024-04-22T15:18:34.657 INFO:tasks.workunit.client.0.smithi096.stdout:0/106: write d7/d9/dc/d16/f1d [386723,62029] 0 2024-04-22T15:18:34.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.727+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f428c105eb0 con 0x7f427806dcb0 2024-04-22T15:18:34.737 INFO:tasks.workunit.client.0.smithi096.stdout:0/107: mknod d7/c25 0 2024-04-22T15:18:34.738 INFO:tasks.workunit.client.0.smithi096.stdout:0/108: read d7/de/f12 [1820678,56647] 0 2024-04-22T15:18:34.740 INFO:tasks.workunit.client.0.smithi096.stdout:0/109: chown d7/d9/dc/f14 366 1 2024-04-22T15:18:34.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.739+0000 7f4282ffd700 1 -- 172.21.15.96:0/980765231 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f428c105eb0 con 0x7f427806dcb0 2024-04-22T15:18:34.742 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:18:34.742 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (7m) 29s ago 8m 24.7M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:18:34.742 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (8m) 29s ago 8m 7838k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:18:34.742 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (7m) 30s ago 7m 7625k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (8m) 29s ago 8m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (7m) 30s ago 7m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (6m) 29s ago 8m 84.4M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (2m) 29s ago 2m 123M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (2m) 29s ago 2m 84.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (2m) 30s ago 2m 21.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (2m) 30s ago 2m 21.4M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:9283,8765,8443 running (10m) 29s ago 10m 357M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 12e0357210ef 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (62s) 30s ago 7m 491M - 19.0.0-3223-g5883d494 3d983c26e278 8e82f21cd80c 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (10m) 29s ago 10m 47.7M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (7m) 30s ago 7m 41.1M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (8m) 29s ago 8m 14.4M - 1.5.0 0da6a335fe13 852e13a0c299 2024-04-22T15:18:34.743 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (7m) 30s ago 7m 16.1M - 1.5.0 0da6a335fe13 4d02817c7cd0 2024-04-22T15:18:34.744 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (6m) 29s ago 6m 75.6M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:18:34.744 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (6m) 29s ago 6m 75.3M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:18:34.744 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (5m) 29s ago 5m 68.2M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:18:34.744 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (5m) 30s ago 5m 74.6M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:18:34.744 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (4m) 30s ago 4m 64.5M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:18:34.744 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (4m) 30s ago 4m 64.0M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:18:34.744 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 starting - - - - 2024-04-22T15:18:34.744 INFO:tasks.workunit.client.0.smithi096.stdout:0/110: mkdir d7/d9/dc/d26 0 2024-04-22T15:18:34.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f427806dcb0 msgr2=0x7f4278070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:34.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f427806dcb0 0x7f4278070170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f427c006010 tx=0x7f427c005180 comp rx=0 tx=0).stop 2024-04-22T15:18:34.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c1016a0 msgr2=0x7f428c1965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:34.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c1016a0 0x7f428c1965d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f428800c930 tx=0x7f428800cc40 comp rx=0 tx=0).stop 2024-04-22T15:18:34.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 shutdown_connections 2024-04-22T15:18:34.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f427806dcb0 0x7f4278070170 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f428c1016a0 0x7f428c1965d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 --2- 172.21.15.96:0/980765231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c103fd0 0x7f428c196b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 >> 172.21.15.96:0/980765231 conn(0x7f428c0fb0b0 msgr2=0x7f428c0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:34.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 shutdown_connections 2024-04-22T15:18:34.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.742+0000 7f4293d4a700 1 -- 172.21.15.96:0/980765231 wait complete. 2024-04-22T15:18:34.746 INFO:tasks.workunit.client.0.smithi096.stdout:0/111: rename d7/c25 to d7/d9/dc/d26/c27 0 2024-04-22T15:18:34.753 INFO:tasks.workunit.client.0.smithi096.stdout:0/112: dwrite d7/de/f12 [0,4194304] 0 2024-04-22T15:18:34.756 INFO:tasks.workunit.client.0.smithi096.stdout:0/113: mknod d7/d9/d15/c28 0 2024-04-22T15:18:34.765 INFO:tasks.workunit.client.0.smithi096.stdout:0/114: truncate d7/d9/dc/f14 3154455 0 2024-04-22T15:18:34.775 INFO:tasks.workunit.client.0.smithi096.stdout:0/115: dread d7/d9/dc/f14 [0,4194304] 0 2024-04-22T15:18:34.781 INFO:tasks.workunit.client.0.smithi096.stdout:0/116: dwrite d7/d9/f17 [4194304,4194304] 0 2024-04-22T15:18:34.784 INFO:tasks.workunit.client.0.smithi096.stdout:0/117: creat d7/d9/dc/f29 x:0 0 0 2024-04-22T15:18:34.789 INFO:tasks.workunit.client.0.smithi096.stdout:0/118: getdents d7/d9/d15 0 2024-04-22T15:18:34.790 INFO:tasks.workunit.client.0.smithi096.stdout:0/119: write d7/d9/f24 [95032,20968] 0 2024-04-22T15:18:34.792 INFO:tasks.workunit.client.0.smithi096.stdout:0/120: mknod d7/de/c2a 0 2024-04-22T15:18:34.793 INFO:tasks.workunit.client.0.smithi096.stdout:0/121: readlink d7/d9/d15/l21 0 2024-04-22T15:18:34.795 INFO:tasks.workunit.client.0.smithi096.stdout:0/122: unlink d7/de/c2a 0 2024-04-22T15:18:34.797 INFO:tasks.workunit.client.0.smithi096.stdout:0/123: creat d7/d9/dc/d16/d18/f2b x:0 0 0 2024-04-22T15:18:34.797 INFO:tasks.workunit.client.0.smithi096.stdout:0/124: dread - d7/d9/dc/f29 zero size 2024-04-22T15:18:34.799 INFO:tasks.workunit.client.0.smithi096.stdout:0/125: symlink d7/l2c 0 2024-04-22T15:18:34.801 INFO:tasks.workunit.client.0.smithi096.stdout:0/126: symlink d7/d9/dc/l2d 0 2024-04-22T15:18:34.801 INFO:tasks.workunit.client.0.smithi096.stdout:0/127: chown d7/de/f19 259 1 2024-04-22T15:18:34.803 INFO:tasks.workunit.client.0.smithi096.stdout:0/128: creat d7/d9/dc/d16/f2e x:0 0 0 2024-04-22T15:18:34.804 INFO:tasks.workunit.client.0.smithi096.stdout:0/129: dread - d7/d9/d15/f1a zero size 2024-04-22T15:18:34.806 INFO:tasks.workunit.client.0.smithi096.stdout:0/130: rename d7/d9/dc/f1c to d7/d9/dc/f2f 0 2024-04-22T15:18:34.814 INFO:tasks.workunit.client.0.smithi096.stdout:0/131: dwrite d7/de/f12 [4194304,4194304] 0 2024-04-22T15:18:34.815 INFO:tasks.workunit.client.0.smithi096.stdout:0/132: fdatasync d7/d9/dc/f2f 0 2024-04-22T15:18:34.817 INFO:tasks.workunit.client.0.smithi096.stdout:0/133: creat d7/d9/dc/d16/d18/f30 x:0 0 0 2024-04-22T15:18:34.819 INFO:tasks.workunit.client.0.smithi096.stdout:0/134: mknod d7/de/c31 0 2024-04-22T15:18:34.821 INFO:tasks.workunit.client.0.smithi096.stdout:0/135: unlink d7/d9/dc/d16/f1d 0 2024-04-22T15:18:34.822 INFO:tasks.workunit.client.0.smithi096.stdout:0/136: read d7/de/f19 [782046,102647] 0 2024-04-22T15:18:34.822 INFO:tasks.workunit.client.0.smithi096.stdout:0/137: readlink d7/de/ld 0 2024-04-22T15:18:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:0/138: truncate d7/d9/dc/f13 6520144 0 2024-04-22T15:18:34.841 INFO:tasks.workunit.client.0.smithi096.stdout:0/139: dread f0 [0,4194304] 0 2024-04-22T15:18:34.844 INFO:tasks.workunit.client.0.smithi096.stdout:0/140: creat d7/f32 x:0 0 0 2024-04-22T15:18:34.845 INFO:tasks.workunit.client.0.smithi096.stdout:0/141: write d7/de/f12 [2993753,78847] 0 2024-04-22T15:18:34.847 INFO:tasks.workunit.client.0.smithi096.stdout:0/142: creat d7/de/f33 x:0 0 0 2024-04-22T15:18:34.848 INFO:tasks.workunit.client.0.smithi096.stdout:0/143: write d7/de/f33 [238123,96411] 0 2024-04-22T15:18:34.850 INFO:tasks.workunit.client.0.smithi096.stdout:0/144: fsync f0 0 2024-04-22T15:18:34.868 INFO:tasks.workunit.client.0.smithi096.stdout:9/59: sync 2024-04-22T15:18:34.873 INFO:tasks.workunit.client.0.smithi096.stdout:9/60: dread d1/d4/f7 [0,4194304] 0 2024-04-22T15:18:34.879 INFO:tasks.workunit.client.0.smithi096.stdout:9/61: dwrite d1/d4/fe [0,4194304] 0 2024-04-22T15:18:34.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.895+0000 7f00c3fd0700 1 -- 172.21.15.96:0/2259643322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00bc0ff4f0 msgr2=0x7f00bc0ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.895+0000 7f00c3fd0700 1 --2- 172.21.15.96:0/2259643322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00bc0ff4f0 0x7f00bc0ff910 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f00ac009910 tx=0x7f00ac009c20 comp rx=0 tx=0).stop 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.895+0000 7f00c3fd0700 1 -- 172.21.15.96:0/2259643322 shutdown_connections 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.895+0000 7f00c3fd0700 1 --2- 172.21.15.96:0/2259643322 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00bc0ffe50 0x7f00bc1002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.895+0000 7f00c3fd0700 1 --2- 172.21.15.96:0/2259643322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00bc0ff4f0 0x7f00bc0ff910 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.895+0000 7f00c3fd0700 1 -- 172.21.15.96:0/2259643322 >> 172.21.15.96:0/2259643322 conn(0x7f00bc0fb0b0 msgr2=0x7f00bc0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.895+0000 7f00c3fd0700 1 -- 172.21.15.96:0/2259643322 shutdown_connections 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.895+0000 7f00c3fd0700 1 -- 172.21.15.96:0/2259643322 wait complete. 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c3fd0700 1 Processor -- start 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c3fd0700 1 -- start start 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c3fd0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00bc0ff4f0 0x7f00bc198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c3fd0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00bc0ffe50 0x7f00bc198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:34.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c3fd0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc199390 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c3fd0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc1994d0 con 0x7f00bc0ffe50 2024-04-22T15:18:34.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c156b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00bc0ffe50 0x7f00bc198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:34.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c1d6c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00bc0ff4f0 0x7f00bc198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:34.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c156b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00bc0ffe50 0x7f00bc198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:44340/0 (socket says 172.21.15.96:44340) 2024-04-22T15:18:34.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.896+0000 7f00c156b700 1 -- 172.21.15.96:0/164127795 learned_addr learned my addr 172.21.15.96:0/164127795 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:34.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.897+0000 7f00c1d6c700 1 -- 172.21.15.96:0/164127795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00bc0ffe50 msgr2=0x7f00bc198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:34.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.897+0000 7f00c1d6c700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00bc0ffe50 0x7f00bc198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:34.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.897+0000 7f00c1d6c700 1 -- 172.21.15.96:0/164127795 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0095f0 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.897+0000 7f00c1d6c700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00bc0ff4f0 0x7f00bc198830 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f00ac006010 tx=0x7f00ac005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:34.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.898+0000 7f00b2ffd700 1 -- 172.21.15.96:0/164127795 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00ac01d070 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.898+0000 7f00b2ffd700 1 -- 172.21.15.96:0/164127795 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00ac0052a0 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.898+0000 7f00b2ffd700 1 -- 172.21.15.96:0/164127795 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00ac017400 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.898+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc19de80 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.898+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00bc19e370 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.899+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00bc192990 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.900+0000 7f00b2ffd700 1 -- 172.21.15.96:0/164127795 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f00ac0037f0 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.900+0000 7f00b2ffd700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f00a806dc40 0x7f00a8070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:34.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.900+0000 7f00b2ffd700 1 -- 172.21.15.96:0/164127795 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f00ac08eab0 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.901+0000 7f00c156b700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f00a806dc40 0x7f00a8070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:34.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.902+0000 7f00c156b700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f00a806dc40 0x7f00a8070100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009d20 tx=0x7f00b8009260 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:34.906+0000 7f00b2ffd700 1 -- 172.21.15.96:0/164127795 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f00ac05b820 con 0x7f00bc0ff4f0 2024-04-22T15:18:34.919 INFO:tasks.workunit.client.0.smithi096.stdout:9/62: creat d1/d4/d9/f12 x:0 0 0 2024-04-22T15:18:34.919 INFO:tasks.workunit.client.0.smithi096.stdout:9/63: stat d1/d4/d9/dd 0 2024-04-22T15:18:34.922 INFO:tasks.workunit.client.0.smithi096.stdout:9/64: symlink d1/l13 0 2024-04-22T15:18:34.926 INFO:tasks.workunit.client.0.smithi096.stdout:9/65: creat d1/df/f14 x:0 0 0 2024-04-22T15:18:34.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:34 smithi096 ceph-mon[26688]: from='client.24405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:35.015 INFO:tasks.workunit.client.0.smithi096.stdout:0/145: dwrite d7/d9/dc/f13 [4194304,4194304] 0 2024-04-22T15:18:35.017 INFO:tasks.workunit.client.0.smithi096.stdout:0/146: dread d7/d9/f24 [0,4194304] 0 2024-04-22T15:18:35.023 INFO:tasks.workunit.client.0.smithi096.stdout:0/147: mkdir d7/de/d34 0 2024-04-22T15:18:35.024 INFO:tasks.workunit.client.0.smithi096.stdout:0/148: stat d7/d9/dc/f13 0 2024-04-22T15:18:35.026 INFO:tasks.workunit.client.0.smithi096.stdout:0/149: mkdir d7/d9/d15/d35 0 2024-04-22T15:18:35.027 INFO:tasks.workunit.client.0.smithi096.stdout:0/150: truncate d7/de/f12 8531779 0 2024-04-22T15:18:35.031 INFO:tasks.workunit.client.0.smithi096.stdout:0/151: rename d7/de/f33 to d7/d9/d15/f36 0 2024-04-22T15:18:35.041 INFO:tasks.workunit.client.0.smithi096.stdout:0/152: creat d7/f37 x:0 0 0 2024-04-22T15:18:35.044 INFO:tasks.workunit.client.0.smithi096.stdout:0/153: rmdir d7/d9/d15 39 2024-04-22T15:18:35.051 INFO:tasks.workunit.client.0.smithi096.stdout:0/154: symlink d7/d9/d15/l38 0 2024-04-22T15:18:35.051 INFO:tasks.workunit.client.0.smithi096.stdout:0/155: stat d7/l2c 0 2024-04-22T15:18:35.052 INFO:tasks.workunit.client.0.smithi096.stdout:0/156: fsync d7/d9/dc/f2f 0 2024-04-22T15:18:35.053 INFO:tasks.workunit.client.0.smithi096.stdout:0/157: chown d7/d9/d15/f1a 29474 1 2024-04-22T15:18:35.055 INFO:tasks.workunit.client.0.smithi096.stdout:4/45: sync 2024-04-22T15:18:35.057 INFO:tasks.workunit.client.0.smithi096.stdout:0/158: dread d7/d9/dc/f14 [0,4194304] 0 2024-04-22T15:18:35.058 INFO:tasks.workunit.client.0.smithi096.stdout:4/46: symlink d5/ld 0 2024-04-22T15:18:35.058 INFO:tasks.workunit.client.0.smithi096.stdout:0/159: write d7/f32 [479737,130819] 0 2024-04-22T15:18:35.063 INFO:tasks.workunit.client.0.smithi096.stdout:0/160: dread d7/de/f12 [0,4194304] 0 2024-04-22T15:18:35.065 INFO:tasks.workunit.client.0.smithi096.stdout:4/47: creat d5/fe x:0 0 0 2024-04-22T15:18:35.070 INFO:tasks.workunit.client.0.smithi096.stdout:0/161: creat d7/d9/dc/d16/d18/f39 x:0 0 0 2024-04-22T15:18:35.072 INFO:tasks.workunit.client.0.smithi096.stdout:4/48: mknod d5/cf 0 2024-04-22T15:18:35.073 INFO:tasks.workunit.client.0.smithi096.stdout:4/49: truncate f2 399256 0 2024-04-22T15:18:35.074 INFO:tasks.workunit.client.0.smithi096.stdout:4/50: dread - d5/fb zero size 2024-04-22T15:18:35.077 INFO:tasks.workunit.client.0.smithi096.stdout:0/162: dwrite d7/d9/dc/d16/d18/f2b [0,4194304] 0 2024-04-22T15:18:35.080 INFO:tasks.workunit.client.0.smithi096.stdout:0/163: truncate d7/d9/dc/d16/f1f 231105 0 2024-04-22T15:18:35.088 INFO:tasks.workunit.client.0.smithi096.stdout:4/51: link l4 d5/d7/l10 0 2024-04-22T15:18:35.089 INFO:tasks.workunit.client.0.smithi096.stdout:0/164: dwrite d7/d9/dc/d16/d18/f23 [0,4194304] 0 2024-04-22T15:18:35.098 INFO:tasks.workunit.client.0.smithi096.stdout:0/165: mkdir d7/de/d3a 0 2024-04-22T15:18:35.100 INFO:tasks.workunit.client.0.smithi096.stdout:4/52: dwrite d5/fc [0,4194304] 0 2024-04-22T15:18:35.107 INFO:tasks.workunit.client.0.smithi096.stdout:0/166: dwrite d7/d9/f17 [4194304,4194304] 0 2024-04-22T15:18:35.115 INFO:tasks.workunit.client.0.smithi096.stdout:3/56: sync 2024-04-22T15:18:35.115 INFO:tasks.workunit.client.0.smithi096.stdout:1/168: sync 2024-04-22T15:18:35.115 INFO:tasks.workunit.client.0.smithi096.stdout:6/70: sync 2024-04-22T15:18:35.115 INFO:tasks.workunit.client.0.smithi096.stdout:5/94: sync 2024-04-22T15:18:35.115 INFO:tasks.workunit.client.0.smithi096.stdout:2/70: sync 2024-04-22T15:18:35.115 INFO:tasks.workunit.client.0.smithi096.stdout:8/119: sync 2024-04-22T15:18:35.115 INFO:tasks.workunit.client.0.smithi096.stdout:7/83: sync 2024-04-22T15:18:35.118 INFO:tasks.workunit.client.0.smithi096.stdout:3/57: chown da/db/cd 6545 1 2024-04-22T15:18:35.118 INFO:tasks.workunit.client.0.smithi096.stdout:1/169: read d6/dc/d11/f19 [1584557,120294] 0 2024-04-22T15:18:35.120 INFO:tasks.workunit.client.0.smithi096.stdout:0/167: link d7/d9/dc/f14 d7/d9/dc/d16/f3b 0 2024-04-22T15:18:35.126 INFO:tasks.workunit.client.0.smithi096.stdout:8/120: symlink d4/l29 0 2024-04-22T15:18:35.137 INFO:tasks.workunit.client.0.smithi096.stdout:1/170: creat d6/d1b/d34/d12/d32/f37 x:0 0 0 2024-04-22T15:18:35.139 INFO:tasks.workunit.client.0.smithi096.stdout:4/53: getdents d5 0 2024-04-22T15:18:35.140 INFO:tasks.workunit.client.0.smithi096.stdout:4/54: write d5/fb [16195,56232] 0 2024-04-22T15:18:35.142 INFO:tasks.workunit.client.0.smithi096.stdout:6/71: link d3/d7/f17 d3/d7/dd/f19 0 2024-04-22T15:18:35.144 INFO:tasks.workunit.client.0.smithi096.stdout:8/121: mkdir d4/d2a 0 2024-04-22T15:18:35.148 INFO:tasks.workunit.client.0.smithi096.stdout:6/72: dwrite d3/d7/ff [0,4194304] 0 2024-04-22T15:18:35.155 INFO:tasks.workunit.client.0.smithi096.stdout:3/58: mknod da/db/d12/c14 0 2024-04-22T15:18:35.157 INFO:tasks.workunit.client.0.smithi096.stdout:1/171: rename d6/d1b/d34/f25 to d6/d1b/f38 0 2024-04-22T15:18:35.160 INFO:tasks.workunit.client.0.smithi096.stdout:4/55: creat d5/d7/f11 x:0 0 0 2024-04-22T15:18:35.163 INFO:tasks.workunit.client.0.smithi096.stdout:8/122: truncate d4/f12 1718205 0 2024-04-22T15:18:35.173 INFO:tasks.workunit.client.0.smithi096.stdout:4/56: mknod d5/d7/c12 0 2024-04-22T15:18:35.174 INFO:tasks.workunit.client.0.smithi096.stdout:4/57: chown d5/ld 36 1 2024-04-22T15:18:35.174 INFO:tasks.workunit.client.0.smithi096.stdout:2/71: write d3/fd [711894,39371] 0 2024-04-22T15:18:35.177 INFO:tasks.workunit.client.0.smithi096.stdout:3/59: creat da/d13/f15 x:0 0 0 2024-04-22T15:18:35.178 INFO:tasks.workunit.client.0.smithi096.stdout:7/84: dwrite f11 [0,4194304] 0 2024-04-22T15:18:35.179 INFO:tasks.workunit.client.0.smithi096.stdout:2/72: dread d3/fd [0,4194304] 0 2024-04-22T15:18:35.180 INFO:tasks.workunit.client.0.smithi096.stdout:9/66: sync 2024-04-22T15:18:35.180 INFO:tasks.workunit.client.0.smithi096.stdout:1/172: unlink d6/f8 0 2024-04-22T15:18:35.181 INFO:tasks.workunit.client.0.smithi096.stdout:1/173: readlink d6/dc/d10/l2b 0 2024-04-22T15:18:35.182 INFO:tasks.workunit.client.0.smithi096.stdout:1/174: chown d6/d1b/d34/d12/c27 472449047 1 2024-04-22T15:18:35.185 INFO:tasks.workunit.client.0.smithi096.stdout:0/168: getdents d7/d9/dc 0 2024-04-22T15:18:35.187 INFO:tasks.workunit.client.0.smithi096.stdout:0/169: write d7/d9/dc/d16/f2e [687047,42693] 0 2024-04-22T15:18:35.187 INFO:tasks.workunit.client.0.smithi096.stdout:1/175: dread d6/fd [0,4194304] 0 2024-04-22T15:18:35.188 INFO:tasks.workunit.client.0.smithi096.stdout:4/58: creat d5/f13 x:0 0 0 2024-04-22T15:18:35.189 INFO:tasks.workunit.client.0.smithi096.stdout:4/59: write d5/fe [996370,56392] 0 2024-04-22T15:18:35.189 INFO:tasks.workunit.client.0.smithi096.stdout:6/73: rmdir d3/d7 39 2024-04-22T15:18:35.190 INFO:tasks.workunit.client.0.smithi096.stdout:4/60: chown d5/c8 14687343 1 2024-04-22T15:18:35.195 INFO:tasks.workunit.client.0.smithi096.stdout:7/85: mknod c1f 0 2024-04-22T15:18:35.197 INFO:tasks.workunit.client.0.smithi096.stdout:5/95: truncate d4/fd 1227703 0 2024-04-22T15:18:35.212 INFO:tasks.workunit.client.0.smithi096.stdout:7/86: rename c13 to c20 0 2024-04-22T15:18:35.213 INFO:tasks.workunit.client.0.smithi096.stdout:7/87: write f10 [16178,11210] 0 2024-04-22T15:18:35.214 INFO:tasks.workunit.client.0.smithi096.stdout:3/60: creat da/db/dc/f16 x:0 0 0 2024-04-22T15:18:35.216 INFO:tasks.workunit.client.0.smithi096.stdout:5/96: mknod d4/d12/c24 0 2024-04-22T15:18:35.218 INFO:tasks.workunit.client.0.smithi096.stdout:2/73: link d3/db/f12 d3/db/f17 0 2024-04-22T15:18:35.221 INFO:tasks.workunit.client.0.smithi096.stdout:0/170: mknod d7/d9/c3c 0 2024-04-22T15:18:35.222 INFO:tasks.workunit.client.0.smithi096.stdout:0/171: fdatasync d7/f32 0 2024-04-22T15:18:35.223 INFO:tasks.workunit.client.0.smithi096.stdout:0/172: write d7/d9/dc/f13 [8789031,129221] 0 2024-04-22T15:18:35.224 INFO:tasks.workunit.client.0.smithi096.stdout:8/123: write d4/fa [1641285,31943] 0 2024-04-22T15:18:35.225 INFO:tasks.workunit.client.0.smithi096.stdout:6/74: creat d3/d7/f1a x:0 0 0 2024-04-22T15:18:35.226 INFO:tasks.workunit.client.0.smithi096.stdout:6/75: chown d3/d7 94962229 1 2024-04-22T15:18:35.229 INFO:tasks.workunit.client.0.smithi096.stdout:7/88: creat f21 x:0 0 0 2024-04-22T15:18:35.229 INFO:tasks.workunit.client.0.smithi096.stdout:7/89: fsync f16 0 2024-04-22T15:18:35.230 INFO:tasks.workunit.client.0.smithi096.stdout:7/90: write f21 [133170,28233] 0 2024-04-22T15:18:35.230 INFO:tasks.workunit.client.0.smithi096.stdout:7/91: chown f16 11317443 1 2024-04-22T15:18:35.231 INFO:tasks.workunit.client.0.smithi096.stdout:7/92: write f16 [361319,114762] 0 2024-04-22T15:18:35.232 INFO:tasks.workunit.client.0.smithi096.stdout:3/61: mkdir da/d13/d17 0 2024-04-22T15:18:35.232 INFO:tasks.workunit.client.0.smithi096.stdout:3/62: chown da/db/dc 289 1 2024-04-22T15:18:35.233 INFO:tasks.workunit.client.0.smithi096.stdout:3/63: dread - da/db/dc/f16 zero size 2024-04-22T15:18:35.234 INFO:tasks.workunit.client.0.smithi096.stdout:3/64: fdatasync da/db/dc/f16 0 2024-04-22T15:18:35.239 INFO:tasks.workunit.client.0.smithi096.stdout:1/176: creat d6/f39 x:0 0 0 2024-04-22T15:18:35.245 INFO:tasks.workunit.client.0.smithi096.stdout:0/173: rename d7/f37 to d7/d9/dc/d16/d18/f3d 0 2024-04-22T15:18:35.247 INFO:tasks.workunit.client.0.smithi096.stdout:6/76: symlink d3/d7/dd/d11/l1b 0 2024-04-22T15:18:35.250 INFO:tasks.workunit.client.0.smithi096.stdout:7/93: fdatasync fb 0 2024-04-22T15:18:35.258 INFO:tasks.workunit.client.0.smithi096.stdout:1/177: symlink d6/d1b/d34/d12/d32/l3a 0 2024-04-22T15:18:35.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:34 smithi196 ceph-mon[34717]: from='client.24405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:35.260 INFO:tasks.workunit.client.0.smithi096.stdout:8/124: symlink d4/d2a/l2b 0 2024-04-22T15:18:35.261 INFO:tasks.workunit.client.0.smithi096.stdout:1/178: dread f2 [4194304,4194304] 0 2024-04-22T15:18:35.264 INFO:tasks.workunit.client.0.smithi096.stdout:0/174: creat d7/d9/dc/d16/d18/f3e x:0 0 0 2024-04-22T15:18:35.265 INFO:tasks.workunit.client.0.smithi096.stdout:0/175: write d7/d9/dc/d16/d18/f2b [1786057,46045] 0 2024-04-22T15:18:35.266 INFO:tasks.workunit.client.0.smithi096.stdout:6/77: creat d3/d7/f1c x:0 0 0 2024-04-22T15:18:35.267 INFO:tasks.workunit.client.0.smithi096.stdout:0/176: chown d7/d9/dc/f2f 9231248 1 2024-04-22T15:18:35.268 INFO:tasks.workunit.client.0.smithi096.stdout:4/61: getdents d5/d7 0 2024-04-22T15:18:35.271 INFO:tasks.workunit.client.0.smithi096.stdout:7/94: unlink f1d 0 2024-04-22T15:18:35.271 INFO:tasks.workunit.client.0.smithi096.stdout:7/95: readlink l14 0 2024-04-22T15:18:35.278 INFO:tasks.workunit.client.0.smithi096.stdout:8/125: creat d4/d2a/f2c x:0 0 0 2024-04-22T15:18:35.279 INFO:tasks.workunit.client.0.smithi096.stdout:5/97: fdatasync d4/fd 0 2024-04-22T15:18:35.282 INFO:tasks.workunit.client.0.smithi096.stdout:1/179: mkdir d6/d1b/d34/d12/d32/d3b 0 2024-04-22T15:18:35.283 INFO:tasks.workunit.client.0.smithi096.stdout:1/180: chown d6/d1b/d34/d12/f21 39 1 2024-04-22T15:18:35.287 INFO:tasks.workunit.client.0.smithi096.stdout:6/78: creat d3/d7/f1d x:0 0 0 2024-04-22T15:18:35.289 INFO:tasks.workunit.client.0.smithi096.stdout:0/177: creat d7/d9/dc/d16/d18/f3f x:0 0 0 2024-04-22T15:18:35.290 INFO:tasks.workunit.client.0.smithi096.stdout:6/79: write d3/d7/dd/d11/f18 [330074,91396] 0 2024-04-22T15:18:35.290 INFO:tasks.workunit.client.0.smithi096.stdout:1/181: dwrite d6/f14 [0,4194304] 0 2024-04-22T15:18:35.292 INFO:tasks.workunit.client.0.smithi096.stdout:4/62: mknod d5/d7/c14 0 2024-04-22T15:18:35.293 INFO:tasks.workunit.client.0.smithi096.stdout:1/182: chown d6/dc/d10/f30 258446920 1 2024-04-22T15:18:35.295 INFO:tasks.workunit.client.0.smithi096.stdout:6/80: dread d3/d7/dd/d11/f18 [0,4194304] 0 2024-04-22T15:18:35.300 INFO:tasks.workunit.client.0.smithi096.stdout:0/178: dwrite d7/d9/dc/d16/d18/f30 [0,4194304] 0 2024-04-22T15:18:35.300 INFO:tasks.workunit.client.0.smithi096.stdout:2/74: getdents d3 0 2024-04-22T15:18:35.307 INFO:tasks.workunit.client.0.smithi096.stdout:5/98: fdatasync d4/d11/f22 0 2024-04-22T15:18:35.308 INFO:tasks.workunit.client.0.smithi096.stdout:5/99: chown f2 674254287 1 2024-04-22T15:18:35.314 INFO:tasks.workunit.client.0.smithi096.stdout:4/63: readlink l4 0 2024-04-22T15:18:35.314 INFO:tasks.workunit.client.0.smithi096.stdout:5/100: dwrite f2 [0,4194304] 0 2024-04-22T15:18:35.320 INFO:tasks.workunit.client.0.smithi096.stdout:7/96: write f1a [3171469,126734] 0 2024-04-22T15:18:35.325 INFO:tasks.workunit.client.0.smithi096.stdout:3/65: link c5 da/db/d10/c18 0 2024-04-22T15:18:35.330 INFO:tasks.workunit.client.0.smithi096.stdout:2/75: creat d3/db/df/d10/f18 x:0 0 0 2024-04-22T15:18:35.332 INFO:tasks.workunit.client.0.smithi096.stdout:8/126: link d4/d10/d25/f26 d4/d1c/f2d 0 2024-04-22T15:18:35.336 INFO:tasks.workunit.client.0.smithi096.stdout:6/81: dread f2 [0,4194304] 0 2024-04-22T15:18:35.339 INFO:tasks.workunit.client.0.smithi096.stdout:4/64: creat d5/d7/f15 x:0 0 0 2024-04-22T15:18:35.339 INFO:tasks.workunit.client.0.smithi096.stdout:4/65: chown d5/fb 3866695 1 2024-04-22T15:18:35.340 INFO:tasks.workunit.client.0.smithi096.stdout:4/66: write d5/fe [965752,82185] 0 2024-04-22T15:18:35.341 INFO:tasks.workunit.client.0.smithi096.stdout:4/67: dread - d5/d7/f9 zero size 2024-04-22T15:18:35.342 INFO:tasks.workunit.client.0.smithi096.stdout:4/68: fdatasync d5/fe 0 2024-04-22T15:18:35.343 INFO:tasks.workunit.client.0.smithi096.stdout:3/66: rmdir da/db/d12 39 2024-04-22T15:18:35.343 INFO:tasks.workunit.client.0.smithi096.stdout:4/69: write d5/d7/f11 [187469,19902] 0 2024-04-22T15:18:35.345 INFO:tasks.workunit.client.0.smithi096.stdout:4/70: write d5/f13 [61127,46881] 0 2024-04-22T15:18:35.346 INFO:tasks.workunit.client.0.smithi096.stdout:2/76: mknod d3/db/df/c19 0 2024-04-22T15:18:35.347 INFO:tasks.workunit.client.0.smithi096.stdout:8/127: creat d4/d10/d25/f2e x:0 0 0 2024-04-22T15:18:35.348 INFO:tasks.workunit.client.0.smithi096.stdout:8/128: chown d4/fa 28252 1 2024-04-22T15:18:35.353 INFO:tasks.workunit.client.0.smithi096.stdout:7/97: link fb f22 0 2024-04-22T15:18:35.354 INFO:tasks.workunit.client.0.smithi096.stdout:7/98: write f21 [141474,124288] 0 2024-04-22T15:18:35.356 INFO:tasks.workunit.client.0.smithi096.stdout:7/99: dread f7 [0,4194304] 0 2024-04-22T15:18:35.356 INFO:tasks.workunit.client.0.smithi096.stdout:7/100: chown f21 8051 1 2024-04-22T15:18:35.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.355+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f00bc10b970 con 0x7f00bc0ff4f0 2024-04-22T15:18:35.359 INFO:tasks.workunit.client.0.smithi096.stdout:0/179: rename d7/de/f19 to d7/d9/f40 0 2024-04-22T15:18:35.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.356+0000 7f00b2ffd700 1 -- 172.21.15.96:0/164127795 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+841 (secure 0 0 0) 0x7f00ac05b3b0 con 0x7f00bc0ff4f0 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:18:35.360 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:18:35.361 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:18:35.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:18:35.361 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:18:35.361 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 13, 2024-04-22T15:18:35.361 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:18:35.361 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:18:35.361 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:18:35.362 INFO:tasks.workunit.client.0.smithi096.stdout:0/180: chown d7/de/ld 15234 1 2024-04-22T15:18:35.362 INFO:tasks.workunit.client.0.smithi096.stdout:3/67: mkdir da/db/de/d19 0 2024-04-22T15:18:35.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.359+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f00a806dc40 msgr2=0x7f00a8070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:35.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.359+0000 7f00c3fd0700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f00a806dc40 0x7f00a8070100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009d20 tx=0x7f00b8009260 comp rx=0 tx=0).stop 2024-04-22T15:18:35.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.359+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00bc0ff4f0 msgr2=0x7f00bc198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:35.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.359+0000 7f00c3fd0700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00bc0ff4f0 0x7f00bc198830 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f00ac006010 tx=0x7f00ac005f00 comp rx=0 tx=0).stop 2024-04-22T15:18:35.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.360+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 shutdown_connections 2024-04-22T15:18:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.360+0000 7f00c3fd0700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f00a806dc40 0x7f00a8070100 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.360+0000 7f00c3fd0700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00bc0ff4f0 0x7f00bc198830 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.360+0000 7f00c3fd0700 1 --2- 172.21.15.96:0/164127795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00bc0ffe50 0x7f00bc198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.360+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 >> 172.21.15.96:0/164127795 conn(0x7f00bc0fb0b0 msgr2=0x7f00bc107f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.360+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 shutdown_connections 2024-04-22T15:18:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.360+0000 7f00c3fd0700 1 -- 172.21.15.96:0/164127795 wait complete. 2024-04-22T15:18:35.364 INFO:tasks.workunit.client.0.smithi096.stdout:7/101: dwrite f5 [0,4194304] 0 2024-04-22T15:18:35.365 INFO:tasks.workunit.client.0.smithi096.stdout:3/68: write da/d13/f15 [868586,51687] 0 2024-04-22T15:18:35.366 INFO:tasks.workunit.client.0.smithi096.stdout:4/71: rename d5/d7/f11 to d5/d7/f16 0 2024-04-22T15:18:35.367 INFO:tasks.workunit.client.0.smithi096.stdout:4/72: chown d5/fb 46 1 2024-04-22T15:18:35.370 INFO:tasks.workunit.client.0.smithi096.stdout:0/181: dwrite d7/f32 [0,4194304] 0 2024-04-22T15:18:35.372 INFO:tasks.workunit.client.0.smithi096.stdout:8/129: symlink d4/l2f 0 2024-04-22T15:18:35.372 INFO:tasks.workunit.client.0.smithi096.stdout:0/182: read d7/d9/dc/d16/d18/f2b [491829,38092] 0 2024-04-22T15:18:35.373 INFO:tasks.workunit.client.0.smithi096.stdout:0/183: readlink d7/de/ld 0 2024-04-22T15:18:35.373 INFO:tasks.workunit.client.0.smithi096.stdout:0/184: dread - d7/f20 zero size 2024-04-22T15:18:35.381 INFO:tasks.workunit.client.0.smithi096.stdout:0/185: dwrite d7/d9/dc/d16/d18/f30 [0,4194304] 0 2024-04-22T15:18:35.385 INFO:tasks.workunit.client.0.smithi096.stdout:6/82: write d3/d7/f17 [889310,90216] 0 2024-04-22T15:18:35.387 INFO:tasks.workunit.client.0.smithi096.stdout:3/69: mknod da/db/dc/c1a 0 2024-04-22T15:18:35.393 INFO:tasks.workunit.client.0.smithi096.stdout:3/70: dwrite da/d13/f15 [0,4194304] 0 2024-04-22T15:18:35.396 INFO:tasks.workunit.client.0.smithi096.stdout:8/130: mknod d4/d2a/c30 0 2024-04-22T15:18:35.401 INFO:tasks.workunit.client.0.smithi096.stdout:1/183: link d6/dc/d11/c26 d6/c3c 0 2024-04-22T15:18:35.407 INFO:tasks.workunit.client.0.smithi096.stdout:6/83: creat d3/d7/dd/d11/f1e x:0 0 0 2024-04-22T15:18:35.412 INFO:tasks.workunit.client.0.smithi096.stdout:8/131: fdatasync d4/fd 0 2024-04-22T15:18:35.461 INFO:tasks.workunit.client.0.smithi096.stdout:0/186: creat d7/d9/d15/d35/f41 x:0 0 0 2024-04-22T15:18:35.462 INFO:tasks.workunit.client.0.smithi096.stdout:0/187: chown d7/d9/d15/c28 1767 1 2024-04-22T15:18:35.463 INFO:tasks.workunit.client.0.smithi096.stdout:2/77: getdents d3/db/df 0 2024-04-22T15:18:35.465 INFO:tasks.workunit.client.0.smithi096.stdout:2/78: truncate d3/db/df/d10/f18 96887 0 2024-04-22T15:18:35.465 INFO:tasks.workunit.client.0.smithi096.stdout:3/71: unlink c8 0 2024-04-22T15:18:35.467 INFO:tasks.workunit.client.0.smithi096.stdout:8/132: rename d4/d2a/f2c to d4/d2a/f31 0 2024-04-22T15:18:35.472 INFO:tasks.workunit.client.0.smithi096.stdout:2/79: creat d3/db/df/f1a x:0 0 0 2024-04-22T15:18:35.475 INFO:tasks.workunit.client.0.smithi096.stdout:0/188: mkdir d7/d42 0 2024-04-22T15:18:35.477 INFO:tasks.workunit.client.0.smithi096.stdout:6/84: rename d3/f6 to d3/f1f 0 2024-04-22T15:18:35.479 INFO:tasks.workunit.client.0.smithi096.stdout:3/72: symlink da/l1b 0 2024-04-22T15:18:35.479 INFO:tasks.workunit.client.0.smithi096.stdout:7/102: write f3 [4410751,119623] 0 2024-04-22T15:18:35.481 INFO:tasks.workunit.client.0.smithi096.stdout:7/103: write f1e [651382,5369] 0 2024-04-22T15:18:35.484 INFO:tasks.workunit.client.0.smithi096.stdout:7/104: fdatasync f1e 0 2024-04-22T15:18:35.484 INFO:tasks.workunit.client.0.smithi096.stdout:4/73: truncate f3 700447 0 2024-04-22T15:18:35.485 INFO:tasks.workunit.client.0.smithi096.stdout:7/105: chown fb 1079 1 2024-04-22T15:18:35.487 INFO:tasks.workunit.client.0.smithi096.stdout:6/85: dwrite d3/d7/dd/f19 [0,4194304] 0 2024-04-22T15:18:35.489 INFO:tasks.workunit.client.0.smithi096.stdout:1/184: write f2 [7198978,5544] 0 2024-04-22T15:18:35.490 INFO:tasks.workunit.client.0.smithi096.stdout:7/106: dread f16 [0,4194304] 0 2024-04-22T15:18:35.496 INFO:tasks.workunit.client.0.smithi096.stdout:4/74: symlink d5/l17 0 2024-04-22T15:18:35.503 INFO:tasks.workunit.client.0.smithi096.stdout:6/86: rmdir d3/d7 39 2024-04-22T15:18:35.505 INFO:tasks.workunit.client.0.smithi096.stdout:3/73: unlink c7 0 2024-04-22T15:18:35.509 INFO:tasks.workunit.client.0.smithi096.stdout:7/107: creat f23 x:0 0 0 2024-04-22T15:18:35.510 INFO:tasks.workunit.client.0.smithi096.stdout:2/80: link d3/db/f12 d3/db/df/d10/f1b 0 2024-04-22T15:18:35.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.509+0000 7f23816d5700 1 -- 172.21.15.96:0/531549310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c074f10 msgr2=0x7f237c073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:35.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.509+0000 7f23816d5700 1 --2- 172.21.15.96:0/531549310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c074f10 0x7f237c073370 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f236c009870 tx=0x7f236c009b80 comp rx=0 tx=0).stop 2024-04-22T15:18:35.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.509+0000 7f23816d5700 1 -- 172.21.15.96:0/531549310 shutdown_connections 2024-04-22T15:18:35.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.509+0000 7f23816d5700 1 --2- 172.21.15.96:0/531549310 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f237c073940 0x7f237c073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:35.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.509+0000 7f23816d5700 1 --2- 172.21.15.96:0/531549310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c074f10 0x7f237c073370 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:35.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.509+0000 7f23816d5700 1 -- 172.21.15.96:0/531549310 >> 172.21.15.96:0/531549310 conn(0x7f237c0fc1d0 msgr2=0x7f237c0fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:35.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.509+0000 7f23816d5700 1 -- 172.21.15.96:0/531549310 shutdown_connections 2024-04-22T15:18:35.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.509+0000 7f23816d5700 1 -- 172.21.15.96:0/531549310 wait complete. 2024-04-22T15:18:35.513 INFO:tasks.workunit.client.0.smithi096.stdout:4/75: creat d5/f18 x:0 0 0 2024-04-22T15:18:35.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.510+0000 7f23816d5700 1 Processor -- start 2024-04-22T15:18:35.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.510+0000 7f23816d5700 1 -- start start 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.510+0000 7f23816d5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c073940 0x7f237c10c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.510+0000 7f23816d5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f237c074f10 0x7f237c10c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.510+0000 7f23816d5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c10cb60 con 0x7f237c073940 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.510+0000 7f23816d5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c10cca0 con 0x7f237c074f10 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.511+0000 7f237affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c073940 0x7f237c10c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.511+0000 7f237affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c073940 0x7f237c10c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56554/0 (socket says 172.21.15.96:56554) 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.511+0000 7f237affd700 1 -- 172.21.15.96:0/143071990 learned_addr learned my addr 172.21.15.96:0/143071990 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.511+0000 7f237a7fc700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f237c074f10 0x7f237c10c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.511+0000 7f237affd700 1 -- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f237c074f10 msgr2=0x7f237c10c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.511+0000 7f237affd700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f237c074f10 0x7f237c10c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.511+0000 7f237affd700 1 -- 172.21.15.96:0/143071990 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c009520 con 0x7f237c073940 2024-04-22T15:18:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.511+0000 7f237affd700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c073940 0x7f237c10c000 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f236c00f670 tx=0x7f236c00f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:35.516 INFO:tasks.workunit.client.0.smithi096.stdout:8/133: dwrite d4/f12 [0,4194304] 0 2024-04-22T15:18:35.516 INFO:tasks.workunit.client.0.smithi096.stdout:4/76: chown l4 11999380 1 2024-04-22T15:18:35.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.512+0000 7f2363fff700 1 -- 172.21.15.96:0/143071990 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f236c01c070 con 0x7f237c073940 2024-04-22T15:18:35.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.512+0000 7f2363fff700 1 -- 172.21.15.96:0/143071990 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f236c00b4e0 con 0x7f237c073940 2024-04-22T15:18:35.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.512+0000 7f2363fff700 1 -- 172.21.15.96:0/143071990 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f236c022d50 con 0x7f237c073940 2024-04-22T15:18:35.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.512+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c1aaac0 con 0x7f237c073940 2024-04-22T15:18:35.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.513+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c1ab030 con 0x7f237c073940 2024-04-22T15:18:35.517 INFO:tasks.workunit.client.0.smithi096.stdout:1/185: dwrite d6/d1b/d34/f1a [4194304,4194304] 0 2024-04-22T15:18:35.517 INFO:tasks.workunit.client.0.smithi096.stdout:0/189: link d7/d9/d15/l38 d7/d9/d15/l43 0 2024-04-22T15:18:35.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.514+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c04ebb0 con 0x7f237c073940 2024-04-22T15:18:35.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.515+0000 7f2363fff700 1 -- 172.21.15.96:0/143071990 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f236c020070 con 0x7f237c073940 2024-04-22T15:18:35.518 INFO:tasks.workunit.client.0.smithi096.stdout:8/134: chown d4/d10/c20 386 1 2024-04-22T15:18:35.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.515+0000 7f2363fff700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f236406dce0 0x7f23640701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:18:35.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.515+0000 7f2363fff700 1 -- 172.21.15.96:0/143071990 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f236c08e5f0 con 0x7f237c073940 2024-04-22T15:18:35.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.516+0000 7f237a7fc700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f236406dce0 0x7f23640701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:18:35.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.517+0000 7f237a7fc700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f236406dce0 0x7f23640701a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f237c074c40 tx=0x7f2370009310 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:18:35.522 INFO:tasks.workunit.client.0.smithi096.stdout:7/108: mknod c24 0 2024-04-22T15:18:35.524 INFO:tasks.workunit.client.0.smithi096.stdout:6/87: dwrite d3/f1f [0,4194304] 0 2024-04-22T15:18:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.523+0000 7f2363fff700 1 -- 172.21.15.96:0/143071990 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f236c093060 con 0x7f237c073940 2024-04-22T15:18:35.527 INFO:tasks.workunit.client.0.smithi096.stdout:2/81: read d3/db/f17 [860048,32336] 0 2024-04-22T15:18:35.530 INFO:tasks.workunit.client.0.smithi096.stdout:4/77: creat d5/f19 x:0 0 0 2024-04-22T15:18:35.534 INFO:tasks.workunit.client.0.smithi096.stdout:8/135: creat d4/f32 x:0 0 0 2024-04-22T15:18:35.535 INFO:tasks.workunit.client.0.smithi096.stdout:8/136: read - d4/d1c/f2d zero size 2024-04-22T15:18:35.536 INFO:tasks.workunit.client.0.smithi096.stdout:3/74: symlink da/l1c 0 2024-04-22T15:18:35.536 INFO:tasks.workunit.client.0.smithi096.stdout:8/137: dread - d4/d10/f21 zero size 2024-04-22T15:18:35.537 INFO:tasks.workunit.client.0.smithi096.stdout:8/138: fsync d4/d10/f23 0 2024-04-22T15:18:35.540 INFO:tasks.workunit.client.0.smithi096.stdout:6/88: write f1 [4686796,45545] 0 2024-04-22T15:18:35.540 INFO:tasks.workunit.client.0.smithi096.stdout:0/190: link d7/d9/f40 d7/de/d3a/f44 0 2024-04-22T15:18:35.541 INFO:tasks.workunit.client.0.smithi096.stdout:4/78: mkdir d5/d7/d1a 0 2024-04-22T15:18:35.543 INFO:tasks.workunit.client.0.smithi096.stdout:1/186: mkdir d6/d1b/d34/d12/d32/d3b/d3d 0 2024-04-22T15:18:35.546 INFO:tasks.workunit.client.0.smithi096.stdout:4/79: dread d5/fc [0,4194304] 0 2024-04-22T15:18:35.546 INFO:tasks.workunit.client.0.smithi096.stdout:1/187: fsync d6/d1b/d34/d12/d32/f37 0 2024-04-22T15:18:35.547 INFO:tasks.workunit.client.0.smithi096.stdout:8/139: unlink d4/l9 0 2024-04-22T15:18:35.548 INFO:tasks.workunit.client.0.smithi096.stdout:4/80: truncate d5/d7/f16 1011574 0 2024-04-22T15:18:35.552 INFO:tasks.workunit.client.0.smithi096.stdout:4/81: dread d5/fc [0,4194304] 0 2024-04-22T15:18:35.554 INFO:tasks.workunit.client.0.smithi096.stdout:1/188: creat d6/dc/d11/f3e x:0 0 0 2024-04-22T15:18:35.558 INFO:tasks.workunit.client.0.smithi096.stdout:8/140: symlink d4/d10/l33 0 2024-04-22T15:18:35.560 INFO:tasks.workunit.client.0.smithi096.stdout:4/82: dwrite d5/f13 [0,4194304] 0 2024-04-22T15:18:35.568 INFO:tasks.workunit.client.0.smithi096.stdout:2/82: getdents d3/db/df 0 2024-04-22T15:18:35.570 INFO:tasks.workunit.client.0.smithi096.stdout:8/141: mkdir d4/d1c/d34 0 2024-04-22T15:18:35.573 INFO:tasks.workunit.client.0.smithi096.stdout:0/191: rename d7/d9/dc/d16/d18/f3e to d7/d9/f45 0 2024-04-22T15:18:35.575 INFO:tasks.workunit.client.0.smithi096.stdout:4/83: creat d5/d7/d1a/f1b x:0 0 0 2024-04-22T15:18:35.576 INFO:tasks.workunit.client.0.smithi096.stdout:3/75: truncate da/d13/f15 3606973 0 2024-04-22T15:18:35.576 INFO:tasks.workunit.client.0.smithi096.stdout:4/84: truncate d5/f19 924918 0 2024-04-22T15:18:35.577 INFO:tasks.workunit.client.0.smithi096.stdout:4/85: dread - d5/d7/f15 zero size 2024-04-22T15:18:35.578 INFO:tasks.workunit.client.0.smithi096.stdout:4/86: write f2 [434579,11357] 0 2024-04-22T15:18:35.579 INFO:tasks.workunit.client.0.smithi096.stdout:4/87: stat f2 0 2024-04-22T15:18:35.586 INFO:tasks.workunit.client.0.smithi096.stdout:1/189: dwrite d6/fd [0,4194304] 0 2024-04-22T15:18:35.588 INFO:tasks.workunit.client.0.smithi096.stdout:4/88: dwrite d5/f18 [0,4194304] 0 2024-04-22T15:18:35.589 INFO:tasks.workunit.client.0.smithi096.stdout:0/192: write d7/de/f12 [1456827,38700] 0 2024-04-22T15:18:35.592 INFO:tasks.workunit.client.0.smithi096.stdout:4/89: truncate d5/d7/d1a/f1b 12927 0 2024-04-22T15:18:35.592 INFO:tasks.workunit.client.0.smithi096.stdout:0/193: dread - d7/d9/d15/d35/f41 zero size 2024-04-22T15:18:35.594 INFO:tasks.workunit.client.0.smithi096.stdout:0/194: chown c4 1468 1 2024-04-22T15:18:35.595 INFO:tasks.workunit.client.0.smithi096.stdout:1/190: mkdir d6/dc/d10/d3f 0 2024-04-22T15:18:35.600 INFO:tasks.workunit.client.0.smithi096.stdout:4/90: dread d5/f13 [0,4194304] 0 2024-04-22T15:18:35.600 INFO:tasks.workunit.client.0.smithi096.stdout:3/76: write da/d13/f15 [2720027,125472] 0 2024-04-22T15:18:35.602 INFO:tasks.workunit.client.0.smithi096.stdout:1/191: rename d6/d1b to d6/dc/d11/d40 0 2024-04-22T15:18:35.604 INFO:tasks.workunit.client.0.smithi096.stdout:4/91: symlink d5/d7/d1a/l1c 0 2024-04-22T15:18:35.606 INFO:tasks.workunit.client.0.smithi096.stdout:0/195: dwrite d7/d9/d15/f1a [0,4194304] 0 2024-04-22T15:18:35.606 INFO:tasks.workunit.client.0.smithi096.stdout:1/192: fsync d6/fd 0 2024-04-22T15:18:35.613 INFO:tasks.workunit.client.0.smithi096.stdout:1/193: creat d6/dc/f41 x:0 0 0 2024-04-22T15:18:35.614 INFO:tasks.workunit.client.0.smithi096.stdout:1/194: read - d6/dc/d11/f3e zero size 2024-04-22T15:18:35.615 INFO:tasks.workunit.client.0.smithi096.stdout:1/195: dread - d6/dc/d11/d40/d34/d12/d32/f37 zero size 2024-04-22T15:18:35.616 INFO:tasks.workunit.client.0.smithi096.stdout:1/196: write d6/dc/d11/d40/d34/d12/f21 [131673,7026] 0 2024-04-22T15:18:35.618 INFO:tasks.workunit.client.0.smithi096.stdout:1/197: truncate d6/dc/d11/d40/f38 110111 0 2024-04-22T15:18:35.622 INFO:tasks.workunit.client.0.smithi096.stdout:1/198: truncate d6/fd 3259333 0 2024-04-22T15:18:35.624 INFO:tasks.workunit.client.0.smithi096.stdout:1/199: chown d6/dc/d10/d29/f36 11 1 2024-04-22T15:18:35.625 INFO:tasks.workunit.client.0.smithi096.stdout:1/200: fdatasync d6/dc/d11/d40/d34/d12/f18 0 2024-04-22T15:18:35.629 INFO:tasks.workunit.client.0.smithi096.stdout:1/201: mknod d6/dc/d11/d40/d34/d12/c42 0 2024-04-22T15:18:35.636 INFO:tasks.workunit.client.0.smithi096.stdout:1/202: write d6/f1d [623805,13927] 0 2024-04-22T15:18:35.637 INFO:tasks.workunit.client.0.smithi096.stdout:1/203: chown d6/dc/d10/d29/f2f 15 1 2024-04-22T15:18:35.804 INFO:tasks.workunit.client.0.smithi096.stdout:9/67: sync 2024-04-22T15:18:35.806 INFO:tasks.workunit.client.0.smithi096.stdout:9/68: symlink d1/df/l15 0 2024-04-22T15:18:35.807 INFO:tasks.workunit.client.0.smithi096.stdout:9/69: mknod d1/df/c16 0 2024-04-22T15:18:35.807 INFO:tasks.workunit.client.0.smithi096.stdout:9/70: chown d1/c10 10917 1 2024-04-22T15:18:35.808 INFO:tasks.workunit.client.0.smithi096.stdout:9/71: write d1/d4/f7 [2996505,82173] 0 2024-04-22T15:18:35.810 INFO:tasks.workunit.client.0.smithi096.stdout:9/72: symlink d1/d4/d9/l17 0 2024-04-22T15:18:35.811 INFO:tasks.workunit.client.0.smithi096.stdout:9/73: fsync d1/d4/fe 0 2024-04-22T15:18:35.812 INFO:tasks.workunit.client.0.smithi096.stdout:9/74: creat d1/d4/d9/f18 x:0 0 0 2024-04-22T15:18:35.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.826+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f237c02ce50 con 0x7f236406dce0 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.828+0000 7f2363fff700 1 -- 172.21.15.96:0/143071990 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f237c02ce50 con 0x7f236406dce0 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [], 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "1/2 daemons upgraded", 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mgr daemons", 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:18:35.831 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:18:35.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f236406dce0 msgr2=0x7f23640701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:35.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f236406dce0 0x7f23640701a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f237c074c40 tx=0x7f2370009310 comp rx=0 tx=0).stop 2024-04-22T15:18:35.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c073940 msgr2=0x7f237c10c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:18:35.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c073940 0x7f237c10c000 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f236c00f670 tx=0x7f236c00f750 comp rx=0 tx=0).stop 2024-04-22T15:18:35.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 shutdown_connections 2024-04-22T15:18:35.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f236406dce0 0x7f23640701a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:35.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f237c073940 0x7f237c10c000 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:35.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 --2- 172.21.15.96:0/143071990 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f237c074f10 0x7f237c10c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:18:35.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 >> 172.21.15.96:0/143071990 conn(0x7f237c0fc1d0 msgr2=0x7f237c102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:18:35.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 shutdown_connections 2024-04-22T15:18:35.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:18:35.830+0000 7f23816d5700 1 -- 172.21.15.96:0/143071990 wait complete. 2024-04-22T15:18:35.889 INFO:tasks.workunit.client.0.smithi096.stdout:9/75: link d1/d4/f11 d1/d4/d9/f19 0 2024-04-22T15:18:35.889 INFO:tasks.workunit.client.0.smithi096.stdout:9/76: fsync d1/d4/f7 0 2024-04-22T15:18:35.890 INFO:tasks.workunit.client.0.smithi096.stdout:9/77: readlink d1/df/l15 0 2024-04-22T15:18:36.009 INFO:tasks.workunit.client.0.smithi096.stdout:5/101: sync 2024-04-22T15:18:36.009 INFO:tasks.workunit.client.0.smithi096.stdout:7/109: sync 2024-04-22T15:18:36.013 INFO:tasks.workunit.client.0.smithi096.stdout:5/102: mkdir d4/d11/d25 0 2024-04-22T15:18:36.022 INFO:tasks.workunit.client.0.smithi096.stdout:7/110: getdents . 0 2024-04-22T15:18:36.022 INFO:tasks.workunit.client.0.smithi096.stdout:7/111: chown l1b 20 1 2024-04-22T15:18:36.023 INFO:tasks.workunit.client.0.smithi096.stdout:7/112: chown c1f 43 1 2024-04-22T15:18:36.023 INFO:tasks.workunit.client.0.smithi096.stdout:7/113: chown l14 15613 1 2024-04-22T15:18:36.023 INFO:tasks.workunit.client.0.smithi096.stdout:7/114: chown f1c 599 1 2024-04-22T15:18:36.023 INFO:tasks.workunit.client.0.smithi096.stdout:5/103: link d4/d1f/f21 d4/f26 0 2024-04-22T15:18:36.029 INFO:tasks.workunit.client.0.smithi096.stdout:5/104: creat d4/d12/d23/f27 x:0 0 0 2024-04-22T15:18:36.033 INFO:tasks.workunit.client.0.smithi096.stdout:7/115: dwrite f7 [0,4194304] 0 2024-04-22T15:18:36.041 INFO:tasks.workunit.client.0.smithi096.stdout:7/116: mknod c25 0 2024-04-22T15:18:36.046 INFO:tasks.workunit.client.0.smithi096.stdout:7/117: symlink l26 0 2024-04-22T15:18:36.048 INFO:tasks.workunit.client.0.smithi096.stdout:5/105: rmdir d4/d1e 0 2024-04-22T15:18:36.051 INFO:tasks.workunit.client.0.smithi096.stdout:7/118: dread f1a [0,4194304] 0 2024-04-22T15:18:36.051 INFO:tasks.workunit.client.0.smithi096.stdout:5/106: mkdir d4/d12/d23/d28 0 2024-04-22T15:18:36.056 INFO:tasks.workunit.client.0.smithi096.stdout:7/119: link f21 f27 0 2024-04-22T15:18:36.062 INFO:tasks.workunit.client.0.smithi096.stdout:7/120: dwrite fa [0,4194304] 0 2024-04-22T15:18:36.065 INFO:tasks.workunit.client.0.smithi096.stdout:7/121: unlink f1e 0 2024-04-22T15:18:36.069 INFO:tasks.workunit.client.0.smithi096.stdout:7/122: rename f16 to f28 0 2024-04-22T15:18:36.071 INFO:tasks.workunit.client.0.smithi096.stdout:7/123: unlink f3 0 2024-04-22T15:18:36.071 INFO:tasks.workunit.client.0.smithi096.stdout:7/124: chown fb 12 1 2024-04-22T15:18:36.078 INFO:tasks.workunit.client.0.smithi096.stdout:7/125: dwrite f1c [0,4194304] 0 2024-04-22T15:18:36.080 INFO:tasks.workunit.client.0.smithi096.stdout:7/126: mkdir d29 0 2024-04-22T15:18:36.084 INFO:tasks.workunit.client.0.smithi096.stdout:7/127: dread f5 [0,4194304] 0 2024-04-22T15:18:36.087 INFO:tasks.workunit.client.0.smithi096.stdout:7/128: read f10 [8036,55107] 0 2024-04-22T15:18:36.090 INFO:tasks.workunit.client.0.smithi096.stdout:7/129: dread f5 [0,4194304] 0 2024-04-22T15:18:36.096 INFO:tasks.workunit.client.0.smithi096.stdout:7/130: rmdir d29 0 2024-04-22T15:18:36.102 INFO:tasks.workunit.client.0.smithi096.stdout:9/78: sync 2024-04-22T15:18:36.102 INFO:tasks.workunit.client.0.smithi096.stdout:4/92: sync 2024-04-22T15:18:36.102 INFO:tasks.workunit.client.0.smithi096.stdout:3/77: sync 2024-04-22T15:18:36.102 INFO:tasks.workunit.client.0.smithi096.stdout:6/89: sync 2024-04-22T15:18:36.102 INFO:tasks.workunit.client.0.smithi096.stdout:1/204: sync 2024-04-22T15:18:36.102 INFO:tasks.workunit.client.0.smithi096.stdout:0/196: sync 2024-04-22T15:18:36.102 INFO:tasks.workunit.client.0.smithi096.stdout:8/142: sync 2024-04-22T15:18:36.102 INFO:tasks.workunit.client.0.smithi096.stdout:2/83: sync 2024-04-22T15:18:36.106 INFO:tasks.workunit.client.0.smithi096.stdout:8/143: write d4/d2a/f31 [586624,46982] 0 2024-04-22T15:18:36.107 INFO:tasks.workunit.client.0.smithi096.stdout:4/93: creat d5/d7/d1a/f1d x:0 0 0 2024-04-22T15:18:36.110 INFO:tasks.workunit.client.0.smithi096.stdout:9/79: creat d1/d4/d9/dd/f1a x:0 0 0 2024-04-22T15:18:36.111 INFO:tasks.workunit.client.0.smithi096.stdout:6/90: mkdir d3/d7/d20 0 2024-04-22T15:18:36.111 INFO:tasks.workunit.client.0.smithi096.stdout:2/84: dread d3/db/f17 [0,4194304] 0 2024-04-22T15:18:36.114 INFO:tasks.workunit.client.0.smithi096.stdout:3/78: dwrite da/db/dc/f16 [0,4194304] 0 2024-04-22T15:18:36.114 INFO:tasks.workunit.client.0.smithi096.stdout:6/91: fdatasync d3/d7/dd/d11/f1e 0 2024-04-22T15:18:36.115 INFO:tasks.workunit.client.0.smithi096.stdout:4/94: mknod d5/c1e 0 2024-04-22T15:18:36.115 INFO:tasks.workunit.client.0.smithi096.stdout:8/144: chown d4/c27 64 1 2024-04-22T15:18:36.117 INFO:tasks.workunit.client.0.smithi096.stdout:6/92: fdatasync d3/d7/dd/f16 0 2024-04-22T15:18:36.119 INFO:tasks.workunit.client.0.smithi096.stdout:2/85: creat d3/d13/f1c x:0 0 0 2024-04-22T15:18:36.119 INFO:tasks.workunit.client.0.smithi096.stdout:4/95: mkdir d5/d7/d1f 0 2024-04-22T15:18:36.120 INFO:tasks.workunit.client.0.smithi096.stdout:3/79: chown da/db/de/c11 0 1 2024-04-22T15:18:36.120 INFO:tasks.workunit.client.0.smithi096.stdout:8/145: chown d4/c27 457798 1 2024-04-22T15:18:36.120 INFO:tasks.workunit.client.0.smithi096.stdout:2/86: chown d3/d13 519 1 2024-04-22T15:18:36.122 INFO:tasks.workunit.client.0.smithi096.stdout:3/80: chown da/db/d10 92303119 1 2024-04-22T15:18:36.123 INFO:tasks.workunit.client.0.smithi096.stdout:9/80: dwrite d1/df/f14 [0,4194304] 0 2024-04-22T15:18:36.124 INFO:tasks.workunit.client.0.smithi096.stdout:6/93: creat d3/d7/f21 x:0 0 0 2024-04-22T15:18:36.124 INFO:tasks.workunit.client.0.smithi096.stdout:9/81: readlink d1/la 0 2024-04-22T15:18:36.125 INFO:tasks.workunit.client.0.smithi096.stdout:6/94: fsync d3/d7/dd/f16 0 2024-04-22T15:18:36.126 INFO:tasks.workunit.client.0.smithi096.stdout:4/96: creat d5/d7/f20 x:0 0 0 2024-04-22T15:18:36.132 INFO:tasks.workunit.client.0.smithi096.stdout:9/82: creat d1/d4/d9/dd/f1b x:0 0 0 2024-04-22T15:18:36.134 INFO:tasks.workunit.client.0.smithi096.stdout:4/97: creat d5/f21 x:0 0 0 2024-04-22T15:18:36.136 INFO:tasks.workunit.client.0.smithi096.stdout:3/81: symlink da/d13/d17/l1d 0 2024-04-22T15:18:36.141 INFO:tasks.workunit.client.0.smithi096.stdout:8/146: link d4/d1c/l1e d4/d2a/l35 0 2024-04-22T15:18:36.142 INFO:tasks.workunit.client.0.smithi096.stdout:4/98: dwrite d5/d7/d1a/f1d [0,4194304] 0 2024-04-22T15:18:36.145 INFO:tasks.workunit.client.0.smithi096.stdout:6/95: creat d3/f22 x:0 0 0 2024-04-22T15:18:36.145 INFO:tasks.workunit.client.0.smithi096.stdout:2/87: getdents d3/db 0 2024-04-22T15:18:36.148 INFO:tasks.workunit.client.0.smithi096.stdout:2/88: read d3/fd [163256,108478] 0 2024-04-22T15:18:36.148 INFO:tasks.workunit.client.0.smithi096.stdout:8/147: creat d4/d2a/f36 x:0 0 0 2024-04-22T15:18:36.150 INFO:tasks.workunit.client.0.smithi096.stdout:4/99: symlink d5/l22 0 2024-04-22T15:18:36.151 INFO:tasks.workunit.client.0.smithi096.stdout:3/82: symlink da/db/d12/l1e 0 2024-04-22T15:18:36.153 INFO:tasks.workunit.client.0.smithi096.stdout:6/96: unlink d3/f1f 0 2024-04-22T15:18:36.154 INFO:tasks.workunit.client.0.smithi096.stdout:2/89: read d3/db/df/d10/f11 [2353006,19415] 0 2024-04-22T15:18:36.156 INFO:tasks.workunit.client.0.smithi096.stdout:6/97: write d3/d7/dd/f19 [616232,125702] 0 2024-04-22T15:18:36.157 INFO:tasks.workunit.client.0.smithi096.stdout:8/148: creat d4/d10/d25/f37 x:0 0 0 2024-04-22T15:18:36.159 INFO:tasks.workunit.client.0.smithi096.stdout:8/149: stat d4/d10/l1a 0 2024-04-22T15:18:36.160 INFO:tasks.workunit.client.0.smithi096.stdout:3/83: dwrite da/d13/f15 [4194304,4194304] 0 2024-04-22T15:18:36.162 INFO:tasks.workunit.client.0.smithi096.stdout:3/84: write da/db/dc/f16 [3020306,111829] 0 2024-04-22T15:18:36.166 INFO:tasks.workunit.client.0.smithi096.stdout:8/150: mkdir d4/d10/d25/d38 0 2024-04-22T15:18:36.168 INFO:tasks.workunit.client.0.smithi096.stdout:8/151: truncate d4/d10/d25/f2e 995883 0 2024-04-22T15:18:36.168 INFO:tasks.workunit.client.0.smithi096.stdout:8/152: stat d4/c14 0 2024-04-22T15:18:36.169 INFO:tasks.workunit.client.0.smithi096.stdout:8/153: fdatasync d4/d10/f21 0 2024-04-22T15:18:36.171 INFO:tasks.workunit.client.0.smithi096.stdout:3/85: dwrite da/d13/f15 [0,4194304] 0 2024-04-22T15:18:36.190 INFO:tasks.workunit.client.0.smithi096.stdout:5/107: write d4/f1c [310608,10203] 0 2024-04-22T15:18:36.193 INFO:tasks.workunit.client.0.smithi096.stdout:5/108: mkdir d4/d1f/d29 0 2024-04-22T15:18:36.199 INFO:tasks.workunit.client.0.smithi096.stdout:5/109: rmdir d4/d12/d23/d28 0 2024-04-22T15:18:36.202 INFO:tasks.workunit.client.0.smithi096.stdout:5/110: rename d4/d12/c24 to d4/d12/c2a 0 2024-04-22T15:18:36.205 INFO:tasks.workunit.client.0.smithi096.stdout:5/111: symlink d4/d12/d23/l2b 0 2024-04-22T15:18:36.209 INFO:tasks.workunit.client.0.smithi096.stdout:7/131: truncate f1a 303044 0 2024-04-22T15:18:36.225 INFO:tasks.workunit.client.0.smithi096.stdout:5/112: dread f1 [0,4194304] 0 2024-04-22T15:18:36.225 INFO:tasks.workunit.client.0.smithi096.stdout:7/132: mknod c2a 0 2024-04-22T15:18:36.228 INFO:tasks.workunit.client.0.smithi096.stdout:5/113: dread f1 [0,4194304] 0 2024-04-22T15:18:36.228 INFO:tasks.workunit.client.0.smithi096.stdout:3/86: rename da/d13 to da/db/dc/d1f 0 2024-04-22T15:18:36.229 INFO:tasks.workunit.client.0.smithi096.stdout:9/83: write d1/d4/d9/f19 [2285524,38527] 0 2024-04-22T15:18:36.232 INFO:tasks.workunit.client.0.smithi096.stdout:2/90: write d3/db/df/d10/f1b [1760244,1080] 0 2024-04-22T15:18:36.233 INFO:tasks.workunit.client.0.smithi096.stdout:1/205: dwrite d6/dc/d11/f19 [0,4194304] 0 2024-04-22T15:18:36.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:35 smithi096 ceph-mon[26688]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:36.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:35 smithi096 ceph-mon[26688]: pgmap v24: 65 pgs: 65 active+clean; 306 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.7 MiB/s rd, 21 MiB/s wr, 385 op/s 2024-04-22T15:18:36.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:35 smithi096 ceph-mon[26688]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:36.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:35 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/164127795' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:18:36.237 INFO:tasks.workunit.client.0.smithi096.stdout:9/84: rmdir d1 39 2024-04-22T15:18:36.240 INFO:tasks.workunit.client.0.smithi096.stdout:2/91: write d3/db/f17 [5329395,57000] 0 2024-04-22T15:18:36.242 INFO:tasks.workunit.client.0.smithi096.stdout:3/87: creat da/db/dc/d1f/f20 x:0 0 0 2024-04-22T15:18:36.244 INFO:tasks.workunit.client.0.smithi096.stdout:1/206: dread - d6/dc/d11/d40/d34/d12/d32/f35 zero size 2024-04-22T15:18:36.250 INFO:tasks.workunit.client.0.smithi096.stdout:1/207: write d6/f22 [2753885,111116] 0 2024-04-22T15:18:36.251 INFO:tasks.workunit.client.0.smithi096.stdout:7/133: dwrite f10 [0,4194304] 0 2024-04-22T15:18:36.253 INFO:tasks.workunit.client.0.smithi096.stdout:6/98: dwrite d3/d7/dd/d11/f18 [0,4194304] 0 2024-04-22T15:18:36.255 INFO:tasks.workunit.client.0.smithi096.stdout:8/154: dwrite f3 [0,4194304] 0 2024-04-22T15:18:36.255 INFO:tasks.workunit.client.0.smithi096.stdout:5/114: getdents d4/d1f/d29 0 2024-04-22T15:18:36.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:35 smithi196 ceph-mon[34717]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:36.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:35 smithi196 ceph-mon[34717]: pgmap v24: 65 pgs: 65 active+clean; 306 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.7 MiB/s rd, 21 MiB/s wr, 385 op/s 2024-04-22T15:18:36.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:35 smithi196 ceph-mon[34717]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:36.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:35 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/164127795' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:18:36.262 INFO:tasks.workunit.client.0.smithi096.stdout:2/92: creat d3/db/df/d10/f1d x:0 0 0 2024-04-22T15:18:36.265 INFO:tasks.workunit.client.0.smithi096.stdout:9/85: dwrite d1/d4/d9/f18 [0,4194304] 0 2024-04-22T15:18:36.268 INFO:tasks.workunit.client.0.smithi096.stdout:3/88: rename da/db/de/c11 to da/db/d12/c21 0 2024-04-22T15:18:36.269 INFO:tasks.workunit.client.0.smithi096.stdout:2/93: dread f0 [0,4194304] 0 2024-04-22T15:18:36.275 INFO:tasks.workunit.client.0.smithi096.stdout:1/208: creat d6/dc/d11/d40/d34/d12/f43 x:0 0 0 2024-04-22T15:18:36.275 INFO:tasks.workunit.client.0.smithi096.stdout:1/209: stat d6/dc/d11 0 2024-04-22T15:18:36.276 INFO:tasks.workunit.client.0.smithi096.stdout:1/210: truncate d6/dc/d11/f3e 1010028 0 2024-04-22T15:18:36.283 INFO:tasks.workunit.client.0.smithi096.stdout:3/89: mkdir da/db/d12/d22 0 2024-04-22T15:18:36.284 INFO:tasks.workunit.client.0.smithi096.stdout:3/90: read da/db/dc/f16 [538574,101104] 0 2024-04-22T15:18:36.285 INFO:tasks.workunit.client.0.smithi096.stdout:5/115: mkdir d4/d2c 0 2024-04-22T15:18:36.285 INFO:tasks.workunit.client.0.smithi096.stdout:3/91: read - da/db/dc/d1f/f20 zero size 2024-04-22T15:18:36.286 INFO:tasks.workunit.client.0.smithi096.stdout:9/86: fdatasync d1/d4/d9/f18 0 2024-04-22T15:18:36.286 INFO:tasks.workunit.client.0.smithi096.stdout:3/92: chown da/db/de/d19 21615296 1 2024-04-22T15:18:36.287 INFO:tasks.workunit.client.0.smithi096.stdout:9/87: write d1/d4/fe [3822353,70208] 0 2024-04-22T15:18:36.289 INFO:tasks.workunit.client.0.smithi096.stdout:8/155: write d4/f28 [6358934,77731] 0 2024-04-22T15:18:36.295 INFO:tasks.workunit.client.0.smithi096.stdout:3/93: mknod da/db/de/c23 0 2024-04-22T15:18:36.296 INFO:tasks.workunit.client.0.smithi096.stdout:9/88: mknod d1/d4/c1c 0 2024-04-22T15:18:36.297 INFO:tasks.workunit.client.0.smithi096.stdout:1/211: dwrite d6/dc/d11/d40/f38 [0,4194304] 0 2024-04-22T15:18:36.300 INFO:tasks.workunit.client.0.smithi096.stdout:2/94: getdents d3/db 0 2024-04-22T15:18:36.303 INFO:tasks.workunit.client.0.smithi096.stdout:8/156: dwrite d4/f32 [0,4194304] 0 2024-04-22T15:18:36.303 INFO:tasks.workunit.client.0.smithi096.stdout:5/116: link f2 d4/d2c/f2d 0 2024-04-22T15:18:36.304 INFO:tasks.workunit.client.0.smithi096.stdout:6/99: creat d3/d7/f23 x:0 0 0 2024-04-22T15:18:36.306 INFO:tasks.workunit.client.0.smithi096.stdout:6/100: stat d3 0 2024-04-22T15:18:36.308 INFO:tasks.workunit.client.0.smithi096.stdout:7/134: dwrite f28 [0,4194304] 0 2024-04-22T15:18:36.310 INFO:tasks.workunit.client.0.smithi096.stdout:5/117: truncate d4/d12/f13 1115642 0 2024-04-22T15:18:36.313 INFO:tasks.workunit.client.0.smithi096.stdout:9/89: rename d1/d4/f7 to d1/d4/d9/dd/f1d 0 2024-04-22T15:18:36.314 INFO:tasks.workunit.client.0.smithi096.stdout:2/95: dwrite d3/db/f17 [0,4194304] 0 2024-04-22T15:18:36.316 INFO:tasks.workunit.client.0.smithi096.stdout:2/96: dread - d3/d13/f1c zero size 2024-04-22T15:18:36.316 INFO:tasks.workunit.client.0.smithi096.stdout:3/94: dread da/db/dc/f16 [0,4194304] 0 2024-04-22T15:18:36.317 INFO:tasks.workunit.client.0.smithi096.stdout:3/95: dread - da/db/dc/d1f/f20 zero size 2024-04-22T15:18:36.318 INFO:tasks.workunit.client.0.smithi096.stdout:1/212: creat d6/dc/d11/d40/d34/f44 x:0 0 0 2024-04-22T15:18:36.318 INFO:tasks.workunit.client.0.smithi096.stdout:9/90: dread d1/f6 [0,4194304] 0 2024-04-22T15:18:36.321 INFO:tasks.workunit.client.0.smithi096.stdout:6/101: unlink d3/l5 0 2024-04-22T15:18:36.321 INFO:tasks.workunit.client.0.smithi096.stdout:6/102: fsync d3/f22 0 2024-04-22T15:18:36.329 INFO:tasks.workunit.client.0.smithi096.stdout:6/103: dwrite f1 [0,4194304] 0 2024-04-22T15:18:36.332 INFO:tasks.workunit.client.0.smithi096.stdout:2/97: mkdir d3/d1e 0 2024-04-22T15:18:36.335 INFO:tasks.workunit.client.0.smithi096.stdout:8/157: link d4/d10/f19 d4/f39 0 2024-04-22T15:18:36.336 INFO:tasks.workunit.client.0.smithi096.stdout:1/213: rmdir d6/dc/d10/d29 39 2024-04-22T15:18:36.340 INFO:tasks.workunit.client.0.smithi096.stdout:9/91: dread d1/d4/d9/f19 [0,4194304] 0 2024-04-22T15:18:36.342 INFO:tasks.workunit.client.0.smithi096.stdout:9/92: read d1/d4/d9/f18 [3484758,104211] 0 2024-04-22T15:18:36.343 INFO:tasks.workunit.client.0.smithi096.stdout:2/98: rename d3/d1e to d3/d1f 0 2024-04-22T15:18:36.344 INFO:tasks.workunit.client.0.smithi096.stdout:8/158: fsync d4/d1c/f2d 0 2024-04-22T15:18:36.346 INFO:tasks.workunit.client.0.smithi096.stdout:2/99: mknod d3/db/df/c20 0 2024-04-22T15:18:36.348 INFO:tasks.workunit.client.0.smithi096.stdout:2/100: truncate d3/db/df/f1a 948407 0 2024-04-22T15:18:36.349 INFO:tasks.workunit.client.0.smithi096.stdout:2/101: dread d3/db/df/d10/f18 [0,4194304] 0 2024-04-22T15:18:36.350 INFO:tasks.workunit.client.0.smithi096.stdout:2/102: chown d3/c6 95 1 2024-04-22T15:18:36.351 INFO:tasks.workunit.client.0.smithi096.stdout:6/104: rmdir d3/d7/d20 0 2024-04-22T15:18:36.351 INFO:tasks.workunit.client.0.smithi096.stdout:2/103: read d3/db/df/d10/f11 [1956471,99703] 0 2024-04-22T15:18:36.353 INFO:tasks.workunit.client.0.smithi096.stdout:2/104: dread - d3/db/df/d10/f1d zero size 2024-04-22T15:18:36.357 INFO:tasks.workunit.client.0.smithi096.stdout:2/105: dread d3/db/df/d10/f1b [4194304,4194304] 0 2024-04-22T15:18:36.357 INFO:tasks.workunit.client.0.smithi096.stdout:6/105: creat d3/f24 x:0 0 0 2024-04-22T15:18:36.363 INFO:tasks.workunit.client.0.smithi096.stdout:6/106: dread - d3/d7/dd/f16 zero size 2024-04-22T15:18:36.367 INFO:tasks.workunit.client.0.smithi096.stdout:7/135: dwrite f21 [0,4194304] 0 2024-04-22T15:18:36.368 INFO:tasks.workunit.client.0.smithi096.stdout:5/118: dwrite f2 [0,4194304] 0 2024-04-22T15:18:36.369 INFO:tasks.workunit.client.0.smithi096.stdout:6/107: creat d3/d7/dd/f25 x:0 0 0 2024-04-22T15:18:36.369 INFO:tasks.workunit.client.0.smithi096.stdout:7/136: chown l2 2 1 2024-04-22T15:18:36.369 INFO:tasks.workunit.client.0.smithi096.stdout:7/137: rmdir - no directory 2024-04-22T15:18:36.371 INFO:tasks.workunit.client.0.smithi096.stdout:6/108: write f1 [4791068,125919] 0 2024-04-22T15:18:36.373 INFO:tasks.workunit.client.0.smithi096.stdout:3/96: dwrite da/db/dc/f16 [4194304,4194304] 0 2024-04-22T15:18:36.374 INFO:tasks.workunit.client.0.smithi096.stdout:9/93: truncate d1/f6 440182 0 2024-04-22T15:18:36.376 INFO:tasks.workunit.client.0.smithi096.stdout:7/138: creat f2b x:0 0 0 2024-04-22T15:18:36.376 INFO:tasks.workunit.client.0.smithi096.stdout:3/97: write da/db/dc/f16 [789692,32778] 0 2024-04-22T15:18:36.377 INFO:tasks.workunit.client.0.smithi096.stdout:8/159: truncate d4/fd 7009333 0 2024-04-22T15:18:36.380 INFO:tasks.workunit.client.0.smithi096.stdout:6/109: write d3/d7/f17 [2779724,10306] 0 2024-04-22T15:18:36.380 INFO:tasks.workunit.client.0.smithi096.stdout:3/98: write da/db/dc/d1f/f20 [581897,18600] 0 2024-04-22T15:18:36.384 INFO:tasks.workunit.client.0.smithi096.stdout:7/139: mknod c2c 0 2024-04-22T15:18:36.386 INFO:tasks.workunit.client.0.smithi096.stdout:8/160: creat d4/d1c/f3a x:0 0 0 2024-04-22T15:18:36.386 INFO:tasks.workunit.client.0.smithi096.stdout:6/110: dread d3/d7/dd/d11/f18 [0,4194304] 0 2024-04-22T15:18:36.387 INFO:tasks.workunit.client.0.smithi096.stdout:3/99: dread da/db/dc/f16 [4194304,4194304] 0 2024-04-22T15:18:36.387 INFO:tasks.workunit.client.0.smithi096.stdout:8/161: fsync d4/fa 0 2024-04-22T15:18:36.388 INFO:tasks.workunit.client.0.smithi096.stdout:2/106: write d3/fd [825922,113846] 0 2024-04-22T15:18:36.389 INFO:tasks.workunit.client.0.smithi096.stdout:8/162: chown d4/d2a/c30 519506504 1 2024-04-22T15:18:36.395 INFO:tasks.workunit.client.0.smithi096.stdout:5/119: dread d4/f5 [0,4194304] 0 2024-04-22T15:18:36.398 INFO:tasks.workunit.client.0.smithi096.stdout:9/94: creat d1/f1e x:0 0 0 2024-04-22T15:18:36.400 INFO:tasks.workunit.client.0.smithi096.stdout:9/95: truncate d1/d4/d9/f12 616248 0 2024-04-22T15:18:36.401 INFO:tasks.workunit.client.0.smithi096.stdout:2/107: creat d3/db/df/d10/f21 x:0 0 0 2024-04-22T15:18:36.405 INFO:tasks.workunit.client.0.smithi096.stdout:9/96: dread d1/d4/d9/f18 [0,4194304] 0 2024-04-22T15:18:36.406 INFO:tasks.workunit.client.0.smithi096.stdout:9/97: fdatasync d1/d4/d9/f12 0 2024-04-22T15:18:36.406 INFO:tasks.workunit.client.0.smithi096.stdout:7/140: unlink f27 0 2024-04-22T15:18:36.413 INFO:tasks.workunit.client.0.smithi096.stdout:7/141: dwrite f7 [0,4194304] 0 2024-04-22T15:18:36.422 INFO:tasks.workunit.client.0.smithi096.stdout:6/111: creat d3/f26 x:0 0 0 2024-04-22T15:18:36.425 INFO:tasks.workunit.client.0.smithi096.stdout:3/100: creat da/f24 x:0 0 0 2024-04-22T15:18:36.426 INFO:tasks.workunit.client.0.smithi096.stdout:8/163: write d4/f28 [6218242,83625] 0 2024-04-22T15:18:36.430 INFO:tasks.workunit.client.0.smithi096.stdout:9/98: link d1/d4/d9/f12 d1/d4/d9/dd/f1f 0 2024-04-22T15:18:36.431 INFO:tasks.workunit.client.0.smithi096.stdout:3/101: symlink da/db/de/l25 0 2024-04-22T15:18:36.432 INFO:tasks.workunit.client.0.smithi096.stdout:5/120: dwrite d4/f26 [0,4194304] 0 2024-04-22T15:18:36.434 INFO:tasks.workunit.client.0.smithi096.stdout:8/164: mknod d4/d1c/c3b 0 2024-04-22T15:18:36.435 INFO:tasks.workunit.client.0.smithi096.stdout:8/165: dread - d4/d10/f21 zero size 2024-04-22T15:18:36.439 INFO:tasks.workunit.client.0.smithi096.stdout:6/112: symlink d3/d7/l27 0 2024-04-22T15:18:36.441 INFO:tasks.workunit.client.0.smithi096.stdout:8/166: truncate d4/d10/d25/f2e 1343119 0 2024-04-22T15:18:36.442 INFO:tasks.workunit.client.0.smithi096.stdout:3/102: fsync da/db/dc/f16 0 2024-04-22T15:18:36.444 INFO:tasks.workunit.client.0.smithi096.stdout:2/108: dwrite d3/db/df/d10/f18 [0,4194304] 0 2024-04-22T15:18:36.446 INFO:tasks.workunit.client.0.smithi096.stdout:5/121: dwrite d4/f10 [0,4194304] 0 2024-04-22T15:18:36.453 INFO:tasks.workunit.client.0.smithi096.stdout:3/103: dwrite da/f24 [0,4194304] 0 2024-04-22T15:18:36.460 INFO:tasks.workunit.client.0.smithi096.stdout:7/142: getdents . 0 2024-04-22T15:18:36.462 INFO:tasks.workunit.client.0.smithi096.stdout:7/143: stat l1b 0 2024-04-22T15:18:36.462 INFO:tasks.workunit.client.0.smithi096.stdout:8/167: creat d4/d2a/f3c x:0 0 0 2024-04-22T15:18:36.466 INFO:tasks.workunit.client.0.smithi096.stdout:6/113: dwrite d3/d7/f1c [0,4194304] 0 2024-04-22T15:18:36.467 INFO:tasks.workunit.client.0.smithi096.stdout:9/99: rename d1/f1e to d1/f20 0 2024-04-22T15:18:36.468 INFO:tasks.workunit.client.0.smithi096.stdout:2/109: getdents d3/d1f 0 2024-04-22T15:18:36.469 INFO:tasks.workunit.client.0.smithi096.stdout:2/110: readlink d3/db/df/d10/l16 0 2024-04-22T15:18:36.470 INFO:tasks.workunit.client.0.smithi096.stdout:7/144: rename c20 to c2d 0 2024-04-22T15:18:36.474 INFO:tasks.workunit.client.0.smithi096.stdout:9/100: creat d1/d4/d9/dd/f21 x:0 0 0 2024-04-22T15:18:36.476 INFO:tasks.workunit.client.0.smithi096.stdout:9/101: dread d1/d4/d9/f12 [0,4194304] 0 2024-04-22T15:18:36.477 INFO:tasks.workunit.client.0.smithi096.stdout:2/111: fdatasync d3/db/f12 0 2024-04-22T15:18:36.479 INFO:tasks.workunit.client.0.smithi096.stdout:2/112: fdatasync d3/db/df/d10/f1b 0 2024-04-22T15:18:36.479 INFO:tasks.workunit.client.0.smithi096.stdout:5/122: getdents d4/d1f 0 2024-04-22T15:18:36.479 INFO:tasks.workunit.client.0.smithi096.stdout:2/113: chown d3/db/df/d10 16131 1 2024-04-22T15:18:36.482 INFO:tasks.workunit.client.0.smithi096.stdout:9/102: write d1/d4/d9/f12 [1101366,121795] 0 2024-04-22T15:18:36.482 INFO:tasks.workunit.client.0.smithi096.stdout:3/104: getdents da/db/dc/d1f 0 2024-04-22T15:18:36.484 INFO:tasks.workunit.client.0.smithi096.stdout:6/114: rename d3/d7 to d3/d28 0 2024-04-22T15:18:36.486 INFO:tasks.workunit.client.0.smithi096.stdout:2/114: dread d3/db/f17 [0,4194304] 0 2024-04-22T15:18:36.486 INFO:tasks.workunit.client.0.smithi096.stdout:5/123: truncate f1 781156 0 2024-04-22T15:18:36.486 INFO:tasks.workunit.client.0.smithi096.stdout:6/115: write d3/d28/f17 [599925,98956] 0 2024-04-22T15:18:36.489 INFO:tasks.workunit.client.0.smithi096.stdout:3/105: dread da/f24 [0,4194304] 0 2024-04-22T15:18:36.490 INFO:tasks.workunit.client.0.smithi096.stdout:9/103: creat d1/d4/f22 x:0 0 0 2024-04-22T15:18:36.493 INFO:tasks.workunit.client.0.smithi096.stdout:5/124: dwrite d4/f1c [0,4194304] 0 2024-04-22T15:18:36.497 INFO:tasks.workunit.client.0.smithi096.stdout:2/115: mknod d3/db/c22 0 2024-04-22T15:18:36.501 INFO:tasks.workunit.client.0.smithi096.stdout:7/145: write f1a [155711,74985] 0 2024-04-22T15:18:36.502 INFO:tasks.workunit.client.0.smithi096.stdout:2/116: dread d3/fd [0,4194304] 0 2024-04-22T15:18:36.502 INFO:tasks.workunit.client.0.smithi096.stdout:9/104: creat d1/d4/d9/dd/f23 x:0 0 0 2024-04-22T15:18:36.505 INFO:tasks.workunit.client.0.smithi096.stdout:9/105: write d1/df/f14 [2824377,52537] 0 2024-04-22T15:18:36.507 INFO:tasks.workunit.client.0.smithi096.stdout:8/168: dwrite d4/d10/d25/f26 [0,4194304] 0 2024-04-22T15:18:36.513 INFO:tasks.workunit.client.0.smithi096.stdout:3/106: unlink da/f24 0 2024-04-22T15:18:36.515 INFO:tasks.workunit.client.0.smithi096.stdout:7/146: creat f2e x:0 0 0 2024-04-22T15:18:36.516 INFO:tasks.workunit.client.0.smithi096.stdout:9/106: dwrite d1/d4/d9/dd/f1b [0,4194304] 0 2024-04-22T15:18:36.517 INFO:tasks.workunit.client.0.smithi096.stdout:5/125: mknod d4/c2e 0 2024-04-22T15:18:36.519 INFO:tasks.workunit.client.0.smithi096.stdout:2/117: rename d3/db/df/c20 to d3/d1f/c23 0 2024-04-22T15:18:36.521 INFO:tasks.workunit.client.0.smithi096.stdout:6/116: creat d3/f29 x:0 0 0 2024-04-22T15:18:36.522 INFO:tasks.workunit.client.0.smithi096.stdout:8/169: rmdir d4/d10/d25 39 2024-04-22T15:18:36.523 INFO:tasks.workunit.client.0.smithi096.stdout:6/117: write d3/d28/dd/f16 [866084,56401] 0 2024-04-22T15:18:36.523 INFO:tasks.workunit.client.0.smithi096.stdout:6/118: stat f2 0 2024-04-22T15:18:36.524 INFO:tasks.workunit.client.0.smithi096.stdout:3/107: readlink l6 0 2024-04-22T15:18:36.524 INFO:tasks.workunit.client.0.smithi096.stdout:8/170: truncate d4/d10/f21 425170 0 2024-04-22T15:18:36.527 INFO:tasks.workunit.client.0.smithi096.stdout:7/147: mknod c2f 0 2024-04-22T15:18:36.527 INFO:tasks.workunit.client.0.smithi096.stdout:5/126: rmdir d4/d1f 39 2024-04-22T15:18:36.529 INFO:tasks.workunit.client.0.smithi096.stdout:5/127: read d4/f5 [999140,113970] 0 2024-04-22T15:18:36.529 INFO:tasks.workunit.client.0.smithi096.stdout:3/108: dread da/db/dc/d1f/f15 [4194304,4194304] 0 2024-04-22T15:18:36.530 INFO:tasks.workunit.client.0.smithi096.stdout:9/107: rename d1/d4/d9/dd/f1f to d1/d4/d9/f24 0 2024-04-22T15:18:36.532 INFO:tasks.workunit.client.0.smithi096.stdout:2/118: mkdir d3/db/df/d24 0 2024-04-22T15:18:36.533 INFO:tasks.workunit.client.0.smithi096.stdout:7/148: dread f7 [0,4194304] 0 2024-04-22T15:18:36.533 INFO:tasks.workunit.client.0.smithi096.stdout:9/108: truncate d1/d4/f22 402825 0 2024-04-22T15:18:36.535 INFO:tasks.workunit.client.0.smithi096.stdout:8/171: fdatasync d4/d10/d25/f37 0 2024-04-22T15:18:36.539 INFO:tasks.workunit.client.0.smithi096.stdout:3/109: dwrite da/db/dc/d1f/f20 [0,4194304] 0 2024-04-22T15:18:36.543 INFO:tasks.workunit.client.0.smithi096.stdout:6/119: symlink d3/l2a 0 2024-04-22T15:18:36.545 INFO:tasks.workunit.client.0.smithi096.stdout:7/149: creat f30 x:0 0 0 2024-04-22T15:18:36.545 INFO:tasks.workunit.client.0.smithi096.stdout:7/150: rmdir - no directory 2024-04-22T15:18:36.545 INFO:tasks.workunit.client.0.smithi096.stdout:7/151: write f30 [43435,106472] 0 2024-04-22T15:18:36.546 INFO:tasks.workunit.client.0.smithi096.stdout:7/152: write f10 [838400,80665] 0 2024-04-22T15:18:36.547 INFO:tasks.workunit.client.0.smithi096.stdout:7/153: readlink l26 0 2024-04-22T15:18:36.552 INFO:tasks.workunit.client.0.smithi096.stdout:5/128: mkdir d4/d2c/d2f 0 2024-04-22T15:18:36.555 INFO:tasks.workunit.client.0.smithi096.stdout:6/120: creat d3/d28/dd/f2b x:0 0 0 2024-04-22T15:18:36.557 INFO:tasks.workunit.client.0.smithi096.stdout:9/109: mkdir d1/d25 0 2024-04-22T15:18:36.558 INFO:tasks.workunit.client.0.smithi096.stdout:7/154: mkdir d31 0 2024-04-22T15:18:36.558 INFO:tasks.workunit.client.0.smithi096.stdout:9/110: read d1/d4/d9/f24 [643686,118000] 0 2024-04-22T15:18:36.561 INFO:tasks.workunit.client.0.smithi096.stdout:2/119: truncate d3/f8 3132813 0 2024-04-22T15:18:36.562 INFO:tasks.workunit.client.0.smithi096.stdout:3/110: mkdir da/db/d12/d22/d26 0 2024-04-22T15:18:36.567 INFO:tasks.workunit.client.0.smithi096.stdout:9/111: symlink d1/d4/d9/dd/l26 0 2024-04-22T15:18:36.573 INFO:tasks.workunit.client.0.smithi096.stdout:7/155: mkdir d31/d32 0 2024-04-22T15:18:36.574 INFO:tasks.workunit.client.0.smithi096.stdout:9/112: write d1/d4/d9/dd/f1d [231734,8369] 0 2024-04-22T15:18:36.578 INFO:tasks.workunit.client.0.smithi096.stdout:7/156: mkdir d31/d32/d33 0 2024-04-22T15:18:36.579 INFO:tasks.workunit.client.0.smithi096.stdout:9/113: dread d1/f6 [0,4194304] 0 2024-04-22T15:18:36.581 INFO:tasks.workunit.client.0.smithi096.stdout:9/114: stat d1/d4/d9/dd/f21 0 2024-04-22T15:18:36.581 INFO:tasks.workunit.client.0.smithi096.stdout:9/115: readlink d1/d4/d9/l17 0 2024-04-22T15:18:36.583 INFO:tasks.workunit.client.0.smithi096.stdout:6/121: write d3/d28/ff [3151498,104739] 0 2024-04-22T15:18:36.622 INFO:tasks.workunit.client.0.smithi096.stdout:7/157: creat d31/d32/d33/f34 x:0 0 0 2024-04-22T15:18:36.622 INFO:tasks.workunit.client.0.smithi096.stdout:5/129: dwrite d4/f1c [4194304,4194304] 0 2024-04-22T15:18:36.624 INFO:tasks.workunit.client.0.smithi096.stdout:9/116: symlink d1/l27 0 2024-04-22T15:18:36.624 INFO:tasks.workunit.client.0.smithi096.stdout:6/122: chown d3/f24 0 1 2024-04-22T15:18:36.630 INFO:tasks.workunit.client.0.smithi096.stdout:2/120: dwrite d3/fd [0,4194304] 0 2024-04-22T15:18:36.631 INFO:tasks.workunit.client.0.smithi096.stdout:5/130: chown d4/d1f/f21 1072320 1 2024-04-22T15:18:36.633 INFO:tasks.workunit.client.0.smithi096.stdout:3/111: dwrite da/db/dc/d1f/f15 [0,4194304] 0 2024-04-22T15:18:36.634 INFO:tasks.workunit.client.0.smithi096.stdout:5/131: stat d4/d11/c17 0 2024-04-22T15:18:36.635 INFO:tasks.workunit.client.0.smithi096.stdout:6/123: mkdir d3/d28/dd/d11/d2c 0 2024-04-22T15:18:36.639 INFO:tasks.workunit.client.0.smithi096.stdout:2/121: symlink d3/db/df/l25 0 2024-04-22T15:18:36.639 INFO:tasks.workunit.client.0.smithi096.stdout:5/132: write d4/f1c [7365427,61500] 0 2024-04-22T15:18:36.642 INFO:tasks.workunit.client.0.smithi096.stdout:7/158: dwrite f1a [0,4194304] 0 2024-04-22T15:18:36.642 INFO:tasks.workunit.client.0.smithi096.stdout:6/124: write d3/d28/dd/d11/f1e [253867,31682] 0 2024-04-22T15:18:36.646 INFO:tasks.workunit.client.0.smithi096.stdout:9/117: dwrite d1/d4/d9/f19 [8388608,4194304] 0 2024-04-22T15:18:36.649 INFO:tasks.workunit.client.0.smithi096.stdout:2/122: mknod d3/db/df/d24/c26 0 2024-04-22T15:18:36.651 INFO:tasks.workunit.client.0.smithi096.stdout:3/112: getdents da/db/de 0 2024-04-22T15:18:36.651 INFO:tasks.workunit.client.0.smithi096.stdout:6/125: dread d3/d28/dd/d11/f1e [0,4194304] 0 2024-04-22T15:18:36.652 INFO:tasks.workunit.client.0.smithi096.stdout:5/133: mkdir d4/d2c/d2f/d30 0 2024-04-22T15:18:36.652 INFO:tasks.workunit.client.0.smithi096.stdout:9/118: fsync d1/d4/d9/f24 0 2024-04-22T15:18:36.656 INFO:tasks.workunit.client.0.smithi096.stdout:5/134: mkdir d4/d2c/d2f/d31 0 2024-04-22T15:18:36.658 INFO:tasks.workunit.client.0.smithi096.stdout:7/159: dwrite f2e [0,4194304] 0 2024-04-22T15:18:36.661 INFO:tasks.workunit.client.0.smithi096.stdout:5/135: write d4/f1c [8543629,96850] 0 2024-04-22T15:18:36.661 INFO:tasks.workunit.client.0.smithi096.stdout:6/126: mknod d3/c2d 0 2024-04-22T15:18:36.661 INFO:tasks.workunit.client.0.smithi096.stdout:3/113: creat da/f27 x:0 0 0 2024-04-22T15:18:36.663 INFO:tasks.workunit.client.0.smithi096.stdout:6/127: chown d3/d28/dd 74090 1 2024-04-22T15:18:36.664 INFO:tasks.workunit.client.0.smithi096.stdout:2/123: dwrite d3/d13/f1c [0,4194304] 0 2024-04-22T15:18:36.664 INFO:tasks.workunit.client.0.smithi096.stdout:5/136: chown d4/fa 309 1 2024-04-22T15:18:36.667 INFO:tasks.workunit.client.0.smithi096.stdout:7/160: rename f30 to d31/d32/d33/f35 0 2024-04-22T15:18:36.667 INFO:tasks.workunit.client.0.smithi096.stdout:5/137: write d4/d2c/f2d [2598196,117030] 0 2024-04-22T15:18:36.669 INFO:tasks.workunit.client.0.smithi096.stdout:5/138: chown d4/d1f/l1d 247044 1 2024-04-22T15:18:36.669 INFO:tasks.workunit.client.0.smithi096.stdout:7/161: truncate f18 1443074 0 2024-04-22T15:18:36.670 INFO:tasks.workunit.client.0.smithi096.stdout:5/139: chown d4/d11 993 1 2024-04-22T15:18:36.672 INFO:tasks.workunit.client.0.smithi096.stdout:9/119: link d1/d4/d9/dd/l26 d1/d25/l28 0 2024-04-22T15:18:36.673 INFO:tasks.workunit.client.0.smithi096.stdout:2/124: dread d3/db/df/d10/f11 [0,4194304] 0 2024-04-22T15:18:36.675 INFO:tasks.workunit.client.0.smithi096.stdout:3/114: rename da/db/dc/d1f/d17 to da/db/dc/d28 0 2024-04-22T15:18:36.677 INFO:tasks.workunit.client.0.smithi096.stdout:7/162: mkdir d31/d32/d36 0 2024-04-22T15:18:36.680 INFO:tasks.workunit.client.0.smithi096.stdout:3/115: chown l1 4688 1 2024-04-22T15:18:36.680 INFO:tasks.workunit.client.0.smithi096.stdout:2/125: dwrite d3/fd [0,4194304] 0 2024-04-22T15:18:36.682 INFO:tasks.workunit.client.0.smithi096.stdout:6/128: rename d3/d28/dd/d11/c12 to d3/d28/dd/c2e 0 2024-04-22T15:18:36.685 INFO:tasks.workunit.client.0.smithi096.stdout:7/163: unlink c2c 0 2024-04-22T15:18:36.686 INFO:tasks.workunit.client.0.smithi096.stdout:9/120: symlink d1/l29 0 2024-04-22T15:18:36.688 INFO:tasks.workunit.client.0.smithi096.stdout:2/126: fdatasync d3/db/df/d10/f11 0 2024-04-22T15:18:36.690 INFO:tasks.workunit.client.0.smithi096.stdout:3/116: creat da/db/dc/d1f/f29 x:0 0 0 2024-04-22T15:18:36.691 INFO:tasks.workunit.client.0.smithi096.stdout:3/117: chown l6 3881057 1 2024-04-22T15:18:36.692 INFO:tasks.workunit.client.0.smithi096.stdout:5/140: rename d4/d1f/l1d to d4/d2c/d2f/d31/l32 0 2024-04-22T15:18:36.693 INFO:tasks.workunit.client.0.smithi096.stdout:7/164: dwrite f1c [0,4194304] 0 2024-04-22T15:18:36.694 INFO:tasks.workunit.client.0.smithi096.stdout:3/118: chown da/db/de/l25 0 1 2024-04-22T15:18:36.695 INFO:tasks.workunit.client.0.smithi096.stdout:5/141: write d4/f26 [2427113,3988] 0 2024-04-22T15:18:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:3/119: stat da/db/de/d19 0 2024-04-22T15:18:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:9/121: dwrite d1/df/f14 [0,4194304] 0 2024-04-22T15:18:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:3/120: read da/db/dc/d1f/f15 [4752398,2155] 0 2024-04-22T15:18:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:9/122: fsync d1/d4/d9/dd/f1d 0 2024-04-22T15:18:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:9/123: write d1/d4/d9/dd/f21 [624390,105716] 0 2024-04-22T15:18:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:5/142: mkdir d4/d1f/d33 0 2024-04-22T15:18:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:9/124: dread d1/d4/d9/dd/f1b [0,4194304] 0 2024-04-22T15:18:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:7/165: creat d31/d32/d36/f37 x:0 0 0 2024-04-22T15:18:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:3/121: creat da/db/de/d19/f2a x:0 0 0 2024-04-22T15:18:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:3/122: dread - da/f27 zero size 2024-04-22T15:18:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:3/123: dread - da/db/dc/d1f/f29 zero size 2024-04-22T15:18:36.718 INFO:tasks.workunit.client.0.smithi096.stdout:6/129: getdents d3/d28 0 2024-04-22T15:18:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:3/124: dread - da/db/de/d19/f2a zero size 2024-04-22T15:18:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:7/166: symlink d31/d32/d33/l38 0 2024-04-22T15:18:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:6/130: mkdir d3/d28/d2f 0 2024-04-22T15:18:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:7/167: creat d31/d32/d33/f39 x:0 0 0 2024-04-22T15:18:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:7/168: chown l26 963792815 1 2024-04-22T15:18:36.724 INFO:tasks.workunit.client.0.smithi096.stdout:7/169: chown l26 332 1 2024-04-22T15:18:36.725 INFO:tasks.workunit.client.0.smithi096.stdout:9/125: dread d1/d4/d9/dd/f21 [0,4194304] 0 2024-04-22T15:18:36.726 INFO:tasks.workunit.client.0.smithi096.stdout:9/126: write d1/d4/f11 [3363445,92156] 0 2024-04-22T15:18:36.727 INFO:tasks.workunit.client.0.smithi096.stdout:6/131: truncate f0 2371244 0 2024-04-22T15:18:36.727 INFO:tasks.workunit.client.0.smithi096.stdout:9/127: write d1/d4/d9/dd/f23 [277535,46840] 0 2024-04-22T15:18:36.728 INFO:tasks.workunit.client.0.smithi096.stdout:7/170: mknod d31/d32/c3a 0 2024-04-22T15:18:36.733 INFO:tasks.workunit.client.0.smithi096.stdout:7/171: mkdir d31/d32/d33/d3b 0 2024-04-22T15:18:36.736 INFO:tasks.workunit.client.0.smithi096.stdout:7/172: creat d31/d32/d36/f3c x:0 0 0 2024-04-22T15:18:36.736 INFO:tasks.workunit.client.0.smithi096.stdout:9/128: link d1/l29 d1/df/l2a 0 2024-04-22T15:18:36.739 INFO:tasks.workunit.client.0.smithi096.stdout:7/173: unlink f22 0 2024-04-22T15:18:36.740 INFO:tasks.workunit.client.0.smithi096.stdout:5/143: truncate d4/f26 325997 0 2024-04-22T15:18:36.741 INFO:tasks.workunit.client.0.smithi096.stdout:7/174: rmdir d31 39 2024-04-22T15:18:36.742 INFO:tasks.workunit.client.0.smithi096.stdout:9/129: dread d1/d4/d9/dd/f1b [0,4194304] 0 2024-04-22T15:18:36.743 INFO:tasks.workunit.client.0.smithi096.stdout:9/130: stat d1/d4/d9/dd/f23 0 2024-04-22T15:18:36.744 INFO:tasks.workunit.client.0.smithi096.stdout:5/144: mknod d4/d1f/c34 0 2024-04-22T15:18:36.746 INFO:tasks.workunit.client.0.smithi096.stdout:3/125: truncate da/db/dc/d1f/f20 547986 0 2024-04-22T15:18:36.822 INFO:tasks.workunit.client.0.smithi096.stdout:6/132: read - d3/d28/f23 zero size 2024-04-22T15:18:36.822 INFO:tasks.workunit.client.0.smithi096.stdout:9/131: creat d1/d4/d9/dd/f2b x:0 0 0 2024-04-22T15:18:36.825 INFO:tasks.workunit.client.0.smithi096.stdout:3/126: mknod da/db/dc/d28/c2b 0 2024-04-22T15:18:36.825 INFO:tasks.workunit.client.0.smithi096.stdout:9/132: write d1/d4/d9/dd/f1a [334157,25265] 0 2024-04-22T15:18:36.826 INFO:tasks.workunit.client.0.smithi096.stdout:5/145: dwrite d4/fd [0,4194304] 0 2024-04-22T15:18:36.827 INFO:tasks.workunit.client.0.smithi096.stdout:3/127: readlink da/l1b 0 2024-04-22T15:18:36.829 INFO:tasks.workunit.client.0.smithi096.stdout:9/133: write d1/d4/fe [4164456,104556] 0 2024-04-22T15:18:36.832 INFO:tasks.workunit.client.0.smithi096.stdout:7/175: dwrite fb [4194304,4194304] 0 2024-04-22T15:18:36.835 INFO:tasks.workunit.client.0.smithi096.stdout:3/128: symlink da/db/d12/d22/l2c 0 2024-04-22T15:18:36.838 INFO:tasks.workunit.client.0.smithi096.stdout:6/133: creat d3/f30 x:0 0 0 2024-04-22T15:18:36.839 INFO:tasks.workunit.client.0.smithi096.stdout:9/134: dwrite d1/d4/d9/f19 [0,4194304] 0 2024-04-22T15:18:36.843 INFO:tasks.workunit.client.0.smithi096.stdout:3/129: unlink da/db/dc/d28/c2b 0 2024-04-22T15:18:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:9/135: creat d1/d4/f2c x:0 0 0 2024-04-22T15:18:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:7/176: creat d31/f3d x:0 0 0 2024-04-22T15:18:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:9/136: dread d1/d4/d9/dd/f21 [0,4194304] 0 2024-04-22T15:18:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:9/137: stat d1/d25 0 2024-04-22T15:18:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:3/130: truncate da/db/dc/d1f/f15 3132146 0 2024-04-22T15:18:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:6/134: creat d3/d28/f31 x:0 0 0 2024-04-22T15:18:36.855 INFO:tasks.workunit.client.0.smithi096.stdout:7/177: creat d31/d32/d33/f3e x:0 0 0 2024-04-22T15:18:36.855 INFO:tasks.workunit.client.0.smithi096.stdout:7/178: readlink l19 0 2024-04-22T15:18:36.855 INFO:tasks.workunit.client.0.smithi096.stdout:7/179: stat fa 0 2024-04-22T15:18:36.859 INFO:tasks.workunit.client.0.smithi096.stdout:6/135: rmdir d3/d28/dd/d11 39 2024-04-22T15:18:36.867 INFO:tasks.workunit.client.0.smithi096.stdout:6/136: dwrite d3/d28/f1a [0,4194304] 0 2024-04-22T15:18:36.870 INFO:tasks.workunit.client.0.smithi096.stdout:6/137: truncate d3/d28/dd/d11/f1e 355480 0 2024-04-22T15:18:36.871 INFO:tasks.workunit.client.0.smithi096.stdout:6/138: write d3/d28/dd/f16 [1926961,68952] 0 2024-04-22T15:18:36.871 INFO:tasks.workunit.client.0.smithi096.stdout:6/139: chown d3/d28 32353157 1 2024-04-22T15:18:36.877 INFO:tasks.workunit.client.0.smithi096.stdout:6/140: creat d3/d28/dd/d11/f32 x:0 0 0 2024-04-22T15:18:36.878 INFO:tasks.workunit.client.0.smithi096.stdout:7/180: rmdir d31 39 2024-04-22T15:18:36.882 INFO:tasks.workunit.client.0.smithi096.stdout:7/181: dread - d31/d32/d33/f3e zero size 2024-04-22T15:18:36.884 INFO:tasks.workunit.client.0.smithi096.stdout:6/141: rename d3/c2d to d3/d28/c33 0 2024-04-22T15:18:36.885 INFO:tasks.workunit.client.0.smithi096.stdout:6/142: write d3/d28/dd/d11/f32 [914894,68993] 0 2024-04-22T15:18:36.886 INFO:tasks.workunit.client.0.smithi096.stdout:6/143: write d3/d28/f31 [974031,2652] 0 2024-04-22T15:18:36.889 INFO:tasks.workunit.client.0.smithi096.stdout:6/144: unlink d3/f30 0 2024-04-22T15:18:36.894 INFO:tasks.workunit.client.0.smithi096.stdout:6/145: symlink d3/d28/dd/d11/d2c/l34 0 2024-04-22T15:18:36.898 INFO:tasks.workunit.client.0.smithi096.stdout:7/182: write f7 [2432827,28228] 0 2024-04-22T15:18:36.900 INFO:tasks.workunit.client.0.smithi096.stdout:7/183: dread d31/d32/d33/f35 [0,4194304] 0 2024-04-22T15:18:36.906 INFO:tasks.workunit.client.0.smithi096.stdout:6/146: truncate d3/d28/ff 648732 0 2024-04-22T15:18:36.907 INFO:tasks.workunit.client.0.smithi096.stdout:7/184: dwrite d31/d32/d33/f39 [0,4194304] 0 2024-04-22T15:18:36.907 INFO:tasks.workunit.client.0.smithi096.stdout:6/147: chown d3/d28/dd/d11 3 1 2024-04-22T15:18:36.909 INFO:tasks.workunit.client.0.smithi096.stdout:6/148: write d3/d28/dd/f25 [881462,65334] 0 2024-04-22T15:18:36.910 INFO:tasks.workunit.client.0.smithi096.stdout:7/185: creat d31/d32/f3f x:0 0 0 2024-04-22T15:18:36.914 INFO:tasks.workunit.client.0.smithi096.stdout:7/186: rename f2e to d31/d32/f40 0 2024-04-22T15:18:36.915 INFO:tasks.workunit.client.0.smithi096.stdout:7/187: write f23 [73307,28832] 0 2024-04-22T15:18:36.917 INFO:tasks.workunit.client.0.smithi096.stdout:6/149: dwrite d3/d28/dd/d11/f32 [0,4194304] 0 2024-04-22T15:18:36.918 INFO:tasks.workunit.client.0.smithi096.stdout:6/150: chown d3/d28/f31 4 1 2024-04-22T15:18:36.921 INFO:tasks.workunit.client.0.smithi096.stdout:7/188: creat d31/d32/d36/f41 x:0 0 0 2024-04-22T15:18:36.924 INFO:tasks.workunit.client.0.smithi096.stdout:7/189: symlink d31/l42 0 2024-04-22T15:18:36.926 INFO:tasks.workunit.client.0.smithi096.stdout:6/151: dwrite d3/d28/dd/f25 [0,4194304] 0 2024-04-22T15:18:36.929 INFO:tasks.workunit.client.0.smithi096.stdout:7/190: mkdir d31/d32/d33/d3b/d43 0 2024-04-22T15:18:36.930 INFO:tasks.workunit.client.0.smithi096.stdout:6/152: symlink d3/d28/dd/d11/d2c/l35 0 2024-04-22T15:18:36.930 INFO:tasks.workunit.client.0.smithi096.stdout:7/191: dread - d31/d32/d33/f34 zero size 2024-04-22T15:18:36.932 INFO:tasks.workunit.client.0.smithi096.stdout:6/153: symlink d3/d28/dd/d11/l36 0 2024-04-22T15:18:36.936 INFO:tasks.workunit.client.0.smithi096.stdout:7/192: dwrite f11 [0,4194304] 0 2024-04-22T15:18:36.937 INFO:tasks.workunit.client.0.smithi096.stdout:6/154: rmdir d3/d28/d2f 0 2024-04-22T15:18:36.939 INFO:tasks.workunit.client.0.smithi096.stdout:7/193: mkdir d31/d32/d33/d44 0 2024-04-22T15:18:36.939 INFO:tasks.workunit.client.0.smithi096.stdout:7/194: fsync d31/f3d 0 2024-04-22T15:18:36.947 INFO:tasks.workunit.client.0.smithi096.stdout:7/195: dread d31/d32/d33/f39 [0,4194304] 0 2024-04-22T15:18:36.948 INFO:tasks.workunit.client.0.smithi096.stdout:6/155: dwrite d3/d28/f1d [0,4194304] 0 2024-04-22T15:18:36.952 INFO:tasks.workunit.client.0.smithi096.stdout:7/196: dread f1a [0,4194304] 0 2024-04-22T15:18:36.957 INFO:tasks.workunit.client.0.smithi096.stdout:6/156: creat d3/d28/f37 x:0 0 0 2024-04-22T15:18:36.957 INFO:tasks.workunit.client.0.smithi096.stdout:6/157: chown f1 3586062 1 2024-04-22T15:18:36.959 INFO:tasks.workunit.client.0.smithi096.stdout:7/197: getdents d31/d32/d36 0 2024-04-22T15:18:36.960 INFO:tasks.workunit.client.0.smithi096.stdout:6/158: rmdir d3/d28/dd/d11/d2c 39 2024-04-22T15:18:36.960 INFO:tasks.workunit.client.0.smithi096.stdout:6/159: chown d3/d28/dd/d11/f1e 0 1 2024-04-22T15:18:36.963 INFO:tasks.workunit.client.0.smithi096.stdout:6/160: creat d3/d28/f38 x:0 0 0 2024-04-22T15:18:36.965 INFO:tasks.workunit.client.0.smithi096.stdout:6/161: truncate d3/d28/f23 537846 0 2024-04-22T15:18:36.965 INFO:tasks.workunit.client.0.smithi096.stdout:6/162: write f1 [2479183,12437] 0 2024-04-22T15:18:37.107 INFO:tasks.workunit.client.0.smithi096.stdout:0/197: sync 2024-04-22T15:18:37.176 INFO:tasks.workunit.client.0.smithi096.stdout:4/100: sync 2024-04-22T15:18:37.178 INFO:tasks.workunit.client.0.smithi096.stdout:4/101: creat d5/d7/d1a/f23 x:0 0 0 2024-04-22T15:18:37.181 INFO:tasks.workunit.client.0.smithi096.stdout:4/102: link d5/f18 d5/d7/f24 0 2024-04-22T15:18:37.183 INFO:tasks.workunit.client.0.smithi096.stdout:4/103: mkdir d5/d7/d1a/d25 0 2024-04-22T15:18:37.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:36 smithi096 ceph-mon[26688]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:36 smithi196 ceph-mon[34717]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:18:37.277 INFO:tasks.workunit.client.0.smithi096.stdout:4/104: dwrite f3 [0,4194304] 0 2024-04-22T15:18:37.328 INFO:tasks.workunit.client.0.smithi096.stdout:4/105: mknod d5/d7/d1a/c26 0 2024-04-22T15:18:37.331 INFO:tasks.workunit.client.0.smithi096.stdout:4/106: mkdir d5/d7/d1a/d25/d27 0 2024-04-22T15:18:37.331 INFO:tasks.workunit.client.0.smithi096.stdout:4/107: fsync f2 0 2024-04-22T15:18:37.337 INFO:tasks.workunit.client.0.smithi096.stdout:4/108: rmdir d5/d7/d1a/d25/d27 0 2024-04-22T15:18:37.337 INFO:tasks.workunit.client.0.smithi096.stdout:4/109: write d5/f19 [1150277,120810] 0 2024-04-22T15:18:37.340 INFO:tasks.workunit.client.0.smithi096.stdout:4/110: mknod d5/d7/d1f/c28 0 2024-04-22T15:18:37.341 INFO:tasks.workunit.client.0.smithi096.stdout:4/111: read - d5/d7/d1a/f23 zero size 2024-04-22T15:18:37.351 INFO:tasks.workunit.client.0.smithi096.stdout:4/112: rmdir d5/d7/d1a 39 2024-04-22T15:18:37.357 INFO:tasks.workunit.client.0.smithi096.stdout:4/113: dwrite d5/d7/f9 [0,4194304] 0 2024-04-22T15:18:37.363 INFO:tasks.workunit.client.0.smithi096.stdout:4/114: dread d5/fb [0,4194304] 0 2024-04-22T15:18:37.370 INFO:tasks.workunit.client.0.smithi096.stdout:4/115: dwrite f3 [0,4194304] 0 2024-04-22T15:18:37.370 INFO:tasks.workunit.client.0.smithi096.stdout:4/116: chown d5/ld 16870088 1 2024-04-22T15:18:37.375 INFO:tasks.workunit.client.0.smithi096.stdout:4/117: rename l4 to d5/d7/d1a/d25/l29 0 2024-04-22T15:18:37.376 INFO:tasks.workunit.client.0.smithi096.stdout:4/118: chown d5/fb 1460717 1 2024-04-22T15:18:37.377 INFO:tasks.workunit.client.0.smithi096.stdout:4/119: mknod d5/c2a 0 2024-04-22T15:18:37.378 INFO:tasks.workunit.client.0.smithi096.stdout:4/120: write d5/d7/f9 [2416527,37342] 0 2024-04-22T15:18:37.382 INFO:tasks.workunit.client.0.smithi096.stdout:4/121: symlink d5/d7/l2b 0 2024-04-22T15:18:37.389 INFO:tasks.workunit.client.0.smithi096.stdout:4/122: rmdir d5 39 2024-04-22T15:18:37.393 INFO:tasks.workunit.client.0.smithi096.stdout:4/123: rmdir d5/d7/d1a 39 2024-04-22T15:18:37.399 INFO:tasks.workunit.client.0.smithi096.stdout:4/124: dwrite d5/fb [0,4194304] 0 2024-04-22T15:18:37.401 INFO:tasks.workunit.client.0.smithi096.stdout:4/125: chown d5/d7/d1a 371 1 2024-04-22T15:18:37.403 INFO:tasks.workunit.client.0.smithi096.stdout:4/126: mkdir d5/d7/d2c 0 2024-04-22T15:18:37.406 INFO:tasks.workunit.client.0.smithi096.stdout:4/127: unlink d5/d7/d1a/f1b 0 2024-04-22T15:18:37.407 INFO:tasks.workunit.client.0.smithi096.stdout:4/128: chown d5/d7/f20 2694 1 2024-04-22T15:18:37.409 INFO:tasks.workunit.client.0.smithi096.stdout:4/129: symlink d5/d7/d2c/l2d 0 2024-04-22T15:18:37.414 INFO:tasks.workunit.client.0.smithi096.stdout:4/130: getdents d5 0 2024-04-22T15:18:37.417 INFO:tasks.workunit.client.0.smithi096.stdout:4/131: rename d5/d7/f15 to d5/d7/d2c/f2e 0 2024-04-22T15:18:37.418 INFO:tasks.workunit.client.0.smithi096.stdout:4/132: stat d5/d7/c12 0 2024-04-22T15:18:37.420 INFO:tasks.workunit.client.0.smithi096.stdout:4/133: rename d5/d7/d1f/c28 to d5/d7/d2c/c2f 0 2024-04-22T15:18:37.421 INFO:tasks.workunit.client.0.smithi096.stdout:4/134: rmdir d5/d7/d1a/d25 39 2024-04-22T15:18:37.424 INFO:tasks.workunit.client.0.smithi096.stdout:4/135: creat d5/d7/d1a/d25/f30 x:0 0 0 2024-04-22T15:18:37.424 INFO:tasks.workunit.client.0.smithi096.stdout:4/136: read - d5/d7/d1a/f23 zero size 2024-04-22T15:18:37.426 INFO:tasks.workunit.client.0.smithi096.stdout:4/137: mknod d5/d7/d1a/d25/c31 0 2024-04-22T15:18:37.427 INFO:tasks.workunit.client.0.smithi096.stdout:4/138: chown d5/d7/c12 5609 1 2024-04-22T15:18:37.429 INFO:tasks.workunit.client.0.smithi096.stdout:4/139: symlink d5/d7/d2c/l32 0 2024-04-22T15:18:37.429 INFO:tasks.workunit.client.0.smithi096.stdout:4/140: chown d5/f21 23709844 1 2024-04-22T15:18:37.430 INFO:tasks.workunit.client.0.smithi096.stdout:4/141: dread - d5/d7/d2c/f2e zero size 2024-04-22T15:18:37.431 INFO:tasks.workunit.client.0.smithi096.stdout:4/142: write d5/d7/d1a/f1d [4215480,122431] 0 2024-04-22T15:18:37.433 INFO:tasks.workunit.client.0.smithi096.stdout:4/143: mknod d5/d7/d1a/d25/c33 0 2024-04-22T15:18:37.472 INFO:tasks.workunit.client.0.smithi096.stdout:1/214: sync 2024-04-22T15:18:37.478 INFO:tasks.workunit.client.0.smithi096.stdout:1/215: dwrite d6/dc/d11/d40/d34/d12/d32/f37 [0,4194304] 0 2024-04-22T15:18:37.480 INFO:tasks.workunit.client.0.smithi096.stdout:1/216: mkdir d6/dc/d11/d45 0 2024-04-22T15:18:37.482 INFO:tasks.workunit.client.0.smithi096.stdout:1/217: truncate d6/dc/d11/d40/d34/d12/f43 118042 0 2024-04-22T15:18:37.485 INFO:tasks.workunit.client.0.smithi096.stdout:1/218: symlink d6/dc/d11/d40/d34/d12/l46 0 2024-04-22T15:18:37.491 INFO:tasks.workunit.client.0.smithi096.stdout:1/219: link d6/dc/d11/f19 d6/dc/d11/d40/f47 0 2024-04-22T15:18:37.493 INFO:tasks.workunit.client.0.smithi096.stdout:1/220: write d6/dc/d10/d29/f2f [637389,63852] 0 2024-04-22T15:18:37.558 INFO:tasks.workunit.client.0.smithi096.stdout:1/221: rename d6/dc/d11/d40/d34/d12/c1e to d6/dc/d11/d40/d34/d12/d32/c48 0 2024-04-22T15:18:37.560 INFO:tasks.workunit.client.0.smithi096.stdout:1/222: getdents d6/dc/d10 0 2024-04-22T15:18:37.562 INFO:tasks.workunit.client.0.smithi096.stdout:1/223: creat d6/dc/d10/d3f/f49 x:0 0 0 2024-04-22T15:18:37.564 INFO:tasks.workunit.client.0.smithi096.stdout:1/224: unlink d6/dc/d10/f30 0 2024-04-22T15:18:37.566 INFO:tasks.workunit.client.0.smithi096.stdout:1/225: mkdir d6/dc/d11/d4a 0 2024-04-22T15:18:37.572 INFO:tasks.workunit.client.0.smithi096.stdout:1/226: dwrite d6/dc/f41 [0,4194304] 0 2024-04-22T15:18:37.577 INFO:tasks.workunit.client.0.smithi096.stdout:1/227: unlink d6/dc/d11/c24 0 2024-04-22T15:18:37.614 INFO:tasks.workunit.client.0.smithi096.stdout:8/172: sync 2024-04-22T15:18:37.614 INFO:tasks.workunit.client.0.smithi096.stdout:5/146: sync 2024-04-22T15:18:37.614 INFO:tasks.workunit.client.0.smithi096.stdout:5/147: stat d4/f5 0 2024-04-22T15:18:37.616 INFO:tasks.workunit.client.0.smithi096.stdout:8/173: getdents d4/d1c/d34 0 2024-04-22T15:18:37.617 INFO:tasks.workunit.client.0.smithi096.stdout:5/148: mkdir d4/d1f/d33/d35 0 2024-04-22T15:18:37.617 INFO:tasks.workunit.client.0.smithi096.stdout:8/174: read d4/f39 [3225257,98662] 0 2024-04-22T15:18:37.618 INFO:tasks.workunit.client.0.smithi096.stdout:8/175: chown d4/d10/c24 788006 1 2024-04-22T15:18:37.622 INFO:tasks.workunit.client.0.smithi096.stdout:8/176: dread d4/d2a/f31 [0,4194304] 0 2024-04-22T15:18:37.623 INFO:tasks.workunit.client.0.smithi096.stdout:8/177: symlink d4/d1c/l3d 0 2024-04-22T15:18:37.626 INFO:tasks.workunit.client.0.smithi096.stdout:8/178: creat d4/d1c/d34/f3e x:0 0 0 2024-04-22T15:18:37.627 INFO:tasks.workunit.client.0.smithi096.stdout:8/179: mknod d4/d1c/c3f 0 2024-04-22T15:18:37.629 INFO:tasks.workunit.client.0.smithi096.stdout:8/180: truncate d4/d2a/f31 711991 0 2024-04-22T15:18:37.630 INFO:tasks.workunit.client.0.smithi096.stdout:8/181: mkdir d4/d1c/d40 0 2024-04-22T15:18:37.637 INFO:tasks.workunit.client.0.smithi096.stdout:8/182: dwrite d4/d1c/d34/f3e [0,4194304] 0 2024-04-22T15:18:37.647 INFO:tasks.workunit.client.0.smithi096.stdout:8/183: dwrite d4/d2a/f31 [0,4194304] 0 2024-04-22T15:18:37.650 INFO:tasks.workunit.client.0.smithi096.stdout:8/184: creat d4/d10/d25/f41 x:0 0 0 2024-04-22T15:18:37.650 INFO:tasks.workunit.client.0.smithi096.stdout:8/185: stat d4/c14 0 2024-04-22T15:18:37.754 INFO:tasks.workunit.client.0.smithi096.stdout:9/138: sync 2024-04-22T15:18:37.754 INFO:tasks.workunit.client.0.smithi096.stdout:2/127: sync 2024-04-22T15:18:37.756 INFO:tasks.workunit.client.0.smithi096.stdout:9/139: creat d1/d4/d9/f2d x:0 0 0 2024-04-22T15:18:37.758 INFO:tasks.workunit.client.0.smithi096.stdout:2/128: dread d3/fd [0,4194304] 0 2024-04-22T15:18:37.759 INFO:tasks.workunit.client.0.smithi096.stdout:2/129: truncate d3/db/df/f1a 1444976 0 2024-04-22T15:18:37.786 INFO:tasks.workunit.client.0.smithi096.stdout:2/130: dwrite d3/db/df/d10/f11 [0,4194304] 0 2024-04-22T15:18:37.786 INFO:tasks.workunit.client.0.smithi096.stdout:2/131: chown d3/d13/f1c 985527620 1 2024-04-22T15:18:37.787 INFO:tasks.workunit.client.0.smithi096.stdout:2/132: chown d3/d13/f1c 1 1 2024-04-22T15:18:37.788 INFO:tasks.workunit.client.0.smithi096.stdout:2/133: rmdir d3/d13 39 2024-04-22T15:18:37.791 INFO:tasks.workunit.client.0.smithi096.stdout:2/134: unlink d3/db/df/d10/f1d 0 2024-04-22T15:18:37.794 INFO:tasks.workunit.client.0.smithi096.stdout:2/135: creat d3/db/df/f27 x:0 0 0 2024-04-22T15:18:37.797 INFO:tasks.workunit.client.0.smithi096.stdout:2/136: symlink d3/d13/l28 0 2024-04-22T15:18:37.798 INFO:tasks.workunit.client.0.smithi096.stdout:2/137: rmdir d3 39 2024-04-22T15:18:37.799 INFO:tasks.workunit.client.0.smithi096.stdout:2/138: read d3/db/df/d10/f11 [3634083,107815] 0 2024-04-22T15:18:37.804 INFO:tasks.workunit.client.0.smithi096.stdout:2/139: write d3/f8 [3219359,49465] 0 2024-04-22T15:18:37.805 INFO:tasks.workunit.client.0.smithi096.stdout:2/140: chown d3/d1f 8 1 2024-04-22T15:18:37.806 INFO:tasks.workunit.client.0.smithi096.stdout:2/141: creat d3/d1f/f29 x:0 0 0 2024-04-22T15:18:37.808 INFO:tasks.workunit.client.0.smithi096.stdout:2/142: creat d3/f2a x:0 0 0 2024-04-22T15:18:37.810 INFO:tasks.workunit.client.0.smithi096.stdout:2/143: symlink d3/db/l2b 0 2024-04-22T15:18:37.812 INFO:tasks.workunit.client.0.smithi096.stdout:2/144: unlink d3/d1f/c23 0 2024-04-22T15:18:37.813 INFO:tasks.workunit.client.0.smithi096.stdout:2/145: symlink d3/d1f/l2c 0 2024-04-22T15:18:37.815 INFO:tasks.workunit.client.0.smithi096.stdout:2/146: mkdir d3/db/d2d 0 2024-04-22T15:18:37.817 INFO:tasks.workunit.client.0.smithi096.stdout:2/147: dread d3/db/df/d10/f11 [0,4194304] 0 2024-04-22T15:18:37.818 INFO:tasks.workunit.client.0.smithi096.stdout:2/148: chown d3/l5 0 1 2024-04-22T15:18:37.819 INFO:tasks.workunit.client.0.smithi096.stdout:2/149: write d3/db/df/d10/f11 [721349,115557] 0 2024-04-22T15:18:37.820 INFO:tasks.workunit.client.0.smithi096.stdout:2/150: chown d3/f2a 298 1 2024-04-22T15:18:37.821 INFO:tasks.workunit.client.0.smithi096.stdout:2/151: write d3/f8 [2731514,81752] 0 2024-04-22T15:18:37.822 INFO:tasks.workunit.client.0.smithi096.stdout:2/152: write d3/f2a [267891,67382] 0 2024-04-22T15:18:37.824 INFO:tasks.workunit.client.0.smithi096.stdout:2/153: creat d3/f2e x:0 0 0 2024-04-22T15:18:37.825 INFO:tasks.workunit.client.0.smithi096.stdout:2/154: mkdir d3/d1f/d2f 0 2024-04-22T15:18:37.834 INFO:tasks.workunit.client.0.smithi096.stdout:2/155: truncate d3/fd 1364193 0 2024-04-22T15:18:37.835 INFO:tasks.workunit.client.0.smithi096.stdout:2/156: write d3/db/f12 [3181870,86589] 0 2024-04-22T15:18:37.836 INFO:tasks.workunit.client.0.smithi096.stdout:2/157: mkdir d3/db/d30 0 2024-04-22T15:18:37.837 INFO:tasks.workunit.client.0.smithi096.stdout:2/158: write f0 [4310670,105324] 0 2024-04-22T15:18:37.839 INFO:tasks.workunit.client.0.smithi096.stdout:2/159: dread d3/f2a [0,4194304] 0 2024-04-22T15:18:37.840 INFO:tasks.workunit.client.0.smithi096.stdout:2/160: write d3/db/df/f1a [18680,28973] 0 2024-04-22T15:18:37.843 INFO:tasks.workunit.client.0.smithi096.stdout:2/161: link d3/d1f/f29 d3/db/f31 0 2024-04-22T15:18:37.843 INFO:tasks.workunit.client.0.smithi096.stdout:2/162: write f0 [802192,61242] 0 2024-04-22T15:18:37.845 INFO:tasks.workunit.client.0.smithi096.stdout:2/163: mknod d3/db/c32 0 2024-04-22T15:18:37.847 INFO:tasks.workunit.client.0.smithi096.stdout:2/164: rename d3/d13/f1c to d3/db/d2d/f33 0 2024-04-22T15:18:37.847 INFO:tasks.workunit.client.0.smithi096.stdout:2/165: chown c2 26 1 2024-04-22T15:18:37.847 INFO:tasks.workunit.client.0.smithi096.stdout:2/166: read d3/f2a [95926,37388] 0 2024-04-22T15:18:37.852 INFO:tasks.workunit.client.0.smithi096.stdout:3/131: sync 2024-04-22T15:18:37.852 INFO:tasks.workunit.client.0.smithi096.stdout:6/163: sync 2024-04-22T15:18:37.852 INFO:tasks.workunit.client.0.smithi096.stdout:7/198: sync 2024-04-22T15:18:37.855 INFO:tasks.workunit.client.0.smithi096.stdout:2/167: getdents d3/d13 0 2024-04-22T15:18:37.857 INFO:tasks.workunit.client.0.smithi096.stdout:2/168: write d3/db/df/d10/f1b [1321119,55798] 0 2024-04-22T15:18:37.861 INFO:tasks.workunit.client.0.smithi096.stdout:2/169: dread d3/db/df/d10/f11 [0,4194304] 0 2024-04-22T15:18:37.862 INFO:tasks.workunit.client.0.smithi096.stdout:7/199: getdents d31/d32 0 2024-04-22T15:18:37.863 INFO:tasks.workunit.client.0.smithi096.stdout:6/164: truncate d3/d28/f1d 1758326 0 2024-04-22T15:18:37.863 INFO:tasks.workunit.client.0.smithi096.stdout:2/170: read - d3/db/df/f27 zero size 2024-04-22T15:18:37.864 INFO:tasks.workunit.client.0.smithi096.stdout:6/165: write d3/f22 [653382,126168] 0 2024-04-22T15:18:37.864 INFO:tasks.workunit.client.0.smithi096.stdout:2/171: write d3/f2e [891715,3888] 0 2024-04-22T15:18:37.866 INFO:tasks.workunit.client.0.smithi096.stdout:7/200: read fb [1179612,114835] 0 2024-04-22T15:18:37.869 INFO:tasks.workunit.client.0.smithi096.stdout:2/172: creat d3/d1f/d2f/f34 x:0 0 0 2024-04-22T15:18:37.872 INFO:tasks.workunit.client.0.smithi096.stdout:6/166: creat d3/d28/dd/d11/d2c/f39 x:0 0 0 2024-04-22T15:18:37.873 INFO:tasks.workunit.client.0.smithi096.stdout:6/167: fsync d3/d28/dd/f16 0 2024-04-22T15:18:37.876 INFO:tasks.workunit.client.0.smithi096.stdout:6/168: chown d3/d28/l10 956 1 2024-04-22T15:18:37.877 INFO:tasks.workunit.client.0.smithi096.stdout:7/201: dwrite d31/d32/d33/f39 [0,4194304] 0 2024-04-22T15:18:37.877 INFO:tasks.workunit.client.0.smithi096.stdout:6/169: stat d3/d28/dd/d11/l36 0 2024-04-22T15:18:37.879 INFO:tasks.workunit.client.0.smithi096.stdout:6/170: chown d3/d28/dd/f2b 862086032 1 2024-04-22T15:18:37.879 INFO:tasks.workunit.client.0.smithi096.stdout:2/173: write d3/db/f31 [809073,114201] 0 2024-04-22T15:18:37.884 INFO:tasks.workunit.client.0.smithi096.stdout:6/171: mkdir d3/d3a 0 2024-04-22T15:18:37.886 INFO:tasks.workunit.client.0.smithi096.stdout:6/172: stat d3/d28/c33 0 2024-04-22T15:18:37.888 INFO:tasks.workunit.client.0.smithi096.stdout:6/173: mkdir d3/d28/dd/d11/d2c/d3b 0 2024-04-22T15:18:37.890 INFO:tasks.workunit.client.0.smithi096.stdout:6/174: mknod d3/d28/dd/c3c 0 2024-04-22T15:18:37.891 INFO:tasks.workunit.client.0.smithi096.stdout:6/175: symlink d3/d3a/l3d 0 2024-04-22T15:18:37.892 INFO:tasks.workunit.client.0.smithi096.stdout:6/176: chown d3/d28/dd/d11/f32 25 1 2024-04-22T15:18:37.893 INFO:tasks.workunit.client.0.smithi096.stdout:7/202: dread fe [0,4194304] 0 2024-04-22T15:18:37.894 INFO:tasks.workunit.client.0.smithi096.stdout:6/177: creat d3/d3a/f3e x:0 0 0 2024-04-22T15:18:37.895 INFO:tasks.workunit.client.0.smithi096.stdout:7/203: fdatasync d31/d32/d33/f35 0 2024-04-22T15:18:37.900 INFO:tasks.workunit.client.0.smithi096.stdout:7/204: dread f18 [0,4194304] 0 2024-04-22T15:18:37.901 INFO:tasks.workunit.client.0.smithi096.stdout:7/205: write d31/d32/d36/f41 [952672,81886] 0 2024-04-22T15:18:37.902 INFO:tasks.workunit.client.0.smithi096.stdout:6/178: rmdir d3/d28/dd/d11/d2c 39 2024-04-22T15:18:37.904 INFO:tasks.workunit.client.0.smithi096.stdout:7/206: creat d31/d32/d36/f45 x:0 0 0 2024-04-22T15:18:37.904 INFO:tasks.workunit.client.0.smithi096.stdout:7/207: fsync f2b 0 2024-04-22T15:18:37.905 INFO:tasks.workunit.client.0.smithi096.stdout:6/179: rename d3 to d3/d28/d3f 22 2024-04-22T15:18:37.906 INFO:tasks.workunit.client.0.smithi096.stdout:7/208: write d31/d32/d33/f35 [351741,23748] 0 2024-04-22T15:18:37.906 INFO:tasks.workunit.client.0.smithi096.stdout:7/209: dread - d31/f3d zero size 2024-04-22T15:18:37.907 INFO:tasks.workunit.client.0.smithi096.stdout:7/210: fdatasync f28 0 2024-04-22T15:18:37.907 INFO:tasks.workunit.client.0.smithi096.stdout:7/211: chown c25 1 1 2024-04-22T15:18:37.914 INFO:tasks.workunit.client.0.smithi096.stdout:6/180: truncate d3/f22 263249 0 2024-04-22T15:18:37.917 INFO:tasks.workunit.client.0.smithi096.stdout:6/181: mkdir d3/d28/dd/d40 0 2024-04-22T15:18:37.918 INFO:tasks.workunit.client.0.smithi096.stdout:7/212: write f21 [2793421,125274] 0 2024-04-22T15:18:37.919 INFO:tasks.workunit.client.0.smithi096.stdout:7/213: chown d31/d32/d33/d44 12116 1 2024-04-22T15:18:37.920 INFO:tasks.workunit.client.0.smithi096.stdout:7/214: truncate d31/d32/d33/f34 148938 0 2024-04-22T15:18:37.921 INFO:tasks.workunit.client.0.smithi096.stdout:7/215: symlink d31/d32/d33/d3b/l46 0 2024-04-22T15:18:37.923 INFO:tasks.workunit.client.0.smithi096.stdout:7/216: creat d31/d32/f47 x:0 0 0 2024-04-22T15:18:37.923 INFO:tasks.workunit.client.0.smithi096.stdout:7/217: chown l26 5 1 2024-04-22T15:18:37.925 INFO:tasks.workunit.client.0.smithi096.stdout:7/218: creat d31/d32/d33/d3b/d43/f48 x:0 0 0 2024-04-22T15:18:37.928 INFO:tasks.workunit.client.0.smithi096.stdout:7/219: link c9 d31/d32/d36/c49 0 2024-04-22T15:18:37.931 INFO:tasks.workunit.client.0.smithi096.stdout:7/220: dread f21 [0,4194304] 0 2024-04-22T15:18:37.932 INFO:tasks.workunit.client.0.smithi096.stdout:7/221: mknod d31/c4a 0 2024-04-22T15:18:37.934 INFO:tasks.workunit.client.0.smithi096.stdout:7/222: symlink d31/d32/d33/d3b/d43/l4b 0 2024-04-22T15:18:37.934 INFO:tasks.workunit.client.0.smithi096.stdout:7/223: readlink l1b 0 2024-04-22T15:18:37.936 INFO:tasks.workunit.client.0.smithi096.stdout:7/224: dread d31/d32/d33/f34 [0,4194304] 0 2024-04-22T15:18:37.939 INFO:tasks.workunit.client.0.smithi096.stdout:7/225: chown c9 581174 1 2024-04-22T15:18:37.943 INFO:tasks.workunit.client.0.smithi096.stdout:7/226: truncate f28 1009293 0 2024-04-22T15:18:37.944 INFO:tasks.workunit.client.0.smithi096.stdout:7/227: write d31/d32/d36/f45 [177213,122766] 0 2024-04-22T15:18:37.946 INFO:tasks.workunit.client.0.smithi096.stdout:7/228: creat d31/d32/f4c x:0 0 0 2024-04-22T15:18:37.947 INFO:tasks.workunit.client.0.smithi096.stdout:7/229: chown d31/d32/d36/f3c 1 1 2024-04-22T15:18:37.948 INFO:tasks.workunit.client.0.smithi096.stdout:7/230: write d31/d32/f47 [842124,27427] 0 2024-04-22T15:18:37.983 INFO:tasks.workunit.client.0.smithi096.stdout:4/144: sync 2024-04-22T15:18:37.983 INFO:tasks.workunit.client.0.smithi096.stdout:0/198: sync 2024-04-22T15:18:37.995 INFO:tasks.workunit.client.0.smithi096.stdout:4/145: dread d5/d7/f9 [0,4194304] 0 2024-04-22T15:18:37.998 INFO:tasks.workunit.client.0.smithi096.stdout:0/199: rmdir d7/d9/dc/d16 39 2024-04-22T15:18:38.001 INFO:tasks.workunit.client.0.smithi096.stdout:7/231: rename f28 to d31/d32/d36/f4d 0 2024-04-22T15:18:38.002 INFO:tasks.workunit.client.0.smithi096.stdout:7/232: truncate f23 1076836 0 2024-04-22T15:18:38.004 INFO:tasks.workunit.client.0.smithi096.stdout:4/146: symlink d5/l34 0 2024-04-22T15:18:38.007 INFO:tasks.workunit.client.0.smithi096.stdout:7/233: write fe [798104,44008] 0 2024-04-22T15:18:38.018 INFO:tasks.workunit.client.0.smithi096.stdout:4/147: creat d5/d7/d1a/f35 x:0 0 0 2024-04-22T15:18:38.019 INFO:tasks.workunit.client.0.smithi096.stdout:7/234: link l1b d31/d32/d33/l4e 0 2024-04-22T15:18:38.023 INFO:tasks.workunit.client.0.smithi096.stdout:7/235: creat d31/d32/f4f x:0 0 0 2024-04-22T15:18:38.023 INFO:tasks.workunit.client.0.smithi096.stdout:7/236: fsync f23 0 2024-04-22T15:18:38.025 INFO:tasks.workunit.client.0.smithi096.stdout:4/148: truncate d5/d7/f24 984396 0 2024-04-22T15:18:38.036 INFO:tasks.workunit.client.0.smithi096.stdout:0/200: truncate d7/d9/d15/f1a 3271907 0 2024-04-22T15:18:38.037 INFO:tasks.workunit.client.0.smithi096.stdout:4/149: creat d5/d7/f36 x:0 0 0 2024-04-22T15:18:38.042 INFO:tasks.workunit.client.0.smithi096.stdout:7/237: getdents d31/d32/d33 0 2024-04-22T15:18:38.043 INFO:tasks.workunit.client.0.smithi096.stdout:7/238: write d31/d32/f3f [784529,3470] 0 2024-04-22T15:18:38.043 INFO:tasks.workunit.client.0.smithi096.stdout:4/150: dwrite f3 [0,4194304] 0 2024-04-22T15:18:38.048 INFO:tasks.workunit.client.0.smithi096.stdout:0/201: mknod d7/d9/dc/d16/d18/c46 0 2024-04-22T15:18:38.051 INFO:tasks.workunit.client.0.smithi096.stdout:7/239: dwrite d31/d32/f4c [0,4194304] 0 2024-04-22T15:18:38.059 INFO:tasks.workunit.client.0.smithi096.stdout:7/240: symlink d31/l50 0 2024-04-22T15:18:38.075 INFO:tasks.workunit.client.0.smithi096.stdout:4/151: fsync d5/d7/f16 0 2024-04-22T15:18:38.081 INFO:tasks.workunit.client.0.smithi096.stdout:5/149: sync 2024-04-22T15:18:38.081 INFO:tasks.workunit.client.0.smithi096.stdout:1/228: sync 2024-04-22T15:18:38.081 INFO:tasks.workunit.client.0.smithi096.stdout:4/152: dwrite d5/d7/f16 [0,4194304] 0 2024-04-22T15:18:38.086 INFO:tasks.workunit.client.0.smithi096.stdout:5/150: mknod d4/d2c/c36 0 2024-04-22T15:18:38.088 INFO:tasks.workunit.client.0.smithi096.stdout:0/202: dread d7/d9/d15/f1a [0,4194304] 0 2024-04-22T15:18:38.098 INFO:tasks.workunit.client.0.smithi096.stdout:1/229: dwrite d6/dc/f23 [0,4194304] 0 2024-04-22T15:18:38.098 INFO:tasks.workunit.client.0.smithi096.stdout:7/241: dwrite d31/d32/d36/f4d [0,4194304] 0 2024-04-22T15:18:38.099 INFO:tasks.workunit.client.0.smithi096.stdout:5/151: unlink d4/d12/c2a 0 2024-04-22T15:18:38.103 INFO:tasks.workunit.client.0.smithi096.stdout:7/242: stat d31/d32/d33/l38 0 2024-04-22T15:18:38.104 INFO:tasks.workunit.client.0.smithi096.stdout:4/153: dwrite d5/d7/f20 [0,4194304] 0 2024-04-22T15:18:38.105 INFO:tasks.workunit.client.0.smithi096.stdout:5/152: write d4/d12/d23/f27 [48867,88931] 0 2024-04-22T15:18:38.106 INFO:tasks.workunit.client.0.smithi096.stdout:4/154: chown d5/l22 84708 1 2024-04-22T15:18:38.107 INFO:tasks.workunit.client.0.smithi096.stdout:0/203: dwrite d7/d9/dc/f29 [0,4194304] 0 2024-04-22T15:18:38.115 INFO:tasks.workunit.client.0.smithi096.stdout:7/243: write f5 [3544175,27828] 0 2024-04-22T15:18:38.117 INFO:tasks.workunit.client.0.smithi096.stdout:5/153: unlink d4/d2c/f2d 0 2024-04-22T15:18:38.120 INFO:tasks.workunit.client.0.smithi096.stdout:7/244: dread f21 [0,4194304] 0 2024-04-22T15:18:38.120 INFO:tasks.workunit.client.0.smithi096.stdout:5/154: write d4/d12/f13 [1467410,117768] 0 2024-04-22T15:18:38.120 INFO:tasks.workunit.client.0.smithi096.stdout:4/155: symlink d5/d7/d1a/d25/l37 0 2024-04-22T15:18:38.121 INFO:tasks.workunit.client.0.smithi096.stdout:5/155: dread - d4/d11/f18 zero size 2024-04-22T15:18:38.126 INFO:tasks.workunit.client.0.smithi096.stdout:0/204: rename d7/d9/dc/d16/f1e to d7/d9/d15/d35/f47 0 2024-04-22T15:18:38.130 INFO:tasks.workunit.client.0.smithi096.stdout:1/230: dwrite d6/dc/d10/d29/f2f [0,4194304] 0 2024-04-22T15:18:38.134 INFO:tasks.workunit.client.0.smithi096.stdout:7/245: creat d31/d32/d33/d3b/f51 x:0 0 0 2024-04-22T15:18:38.140 INFO:tasks.workunit.client.0.smithi096.stdout:7/246: dread f21 [0,4194304] 0 2024-04-22T15:18:38.142 INFO:tasks.workunit.client.0.smithi096.stdout:7/247: chown d31/d32/d33/f35 222470 1 2024-04-22T15:18:38.144 INFO:tasks.workunit.client.0.smithi096.stdout:5/156: rename d4/d1f/c34 to d4/d1f/d33/c37 0 2024-04-22T15:18:38.144 INFO:tasks.workunit.client.0.smithi096.stdout:4/156: dwrite d5/d7/d2c/f2e [0,4194304] 0 2024-04-22T15:18:38.146 INFO:tasks.workunit.client.0.smithi096.stdout:7/248: truncate d31/d32/d36/f37 969072 0 2024-04-22T15:18:38.148 INFO:tasks.workunit.client.0.smithi096.stdout:7/249: dread - d31/f3d zero size 2024-04-22T15:18:38.150 INFO:tasks.workunit.client.0.smithi096.stdout:1/231: rename d6/dc/d11/d40/f47 to d6/dc/d11/d40/d34/d12/d32/d3b/d3d/f4b 0 2024-04-22T15:18:38.151 INFO:tasks.workunit.client.0.smithi096.stdout:1/232: chown d6/dc 0 1 2024-04-22T15:18:38.155 INFO:tasks.workunit.client.0.smithi096.stdout:3/132: sync 2024-04-22T15:18:38.155 INFO:tasks.workunit.client.0.smithi096.stdout:8/186: sync 2024-04-22T15:18:38.156 INFO:tasks.workunit.client.0.smithi096.stdout:6/182: sync 2024-04-22T15:18:38.156 INFO:tasks.workunit.client.0.smithi096.stdout:9/140: sync 2024-04-22T15:18:38.156 INFO:tasks.workunit.client.0.smithi096.stdout:2/174: sync 2024-04-22T15:18:38.157 INFO:tasks.workunit.client.0.smithi096.stdout:2/175: write d3/f8 [3648105,122889] 0 2024-04-22T15:18:38.159 INFO:tasks.workunit.client.0.smithi096.stdout:7/250: mknod d31/c52 0 2024-04-22T15:18:38.162 INFO:tasks.workunit.client.0.smithi096.stdout:5/157: rename d4/d11/f18 to d4/d11/d25/f38 0 2024-04-22T15:18:38.164 INFO:tasks.workunit.client.0.smithi096.stdout:1/233: creat d6/dc/d11/d40/d34/d12/f4c x:0 0 0 2024-04-22T15:18:38.166 INFO:tasks.workunit.client.0.smithi096.stdout:3/133: dwrite da/f27 [0,4194304] 0 2024-04-22T15:18:38.167 INFO:tasks.workunit.client.0.smithi096.stdout:6/183: dwrite d3/d28/f23 [0,4194304] 0 2024-04-22T15:18:38.172 INFO:tasks.workunit.client.0.smithi096.stdout:1/234: write d6/dc/d10/d29/f2f [3273368,100603] 0 2024-04-22T15:18:38.178 INFO:tasks.workunit.client.0.smithi096.stdout:5/158: dwrite d4/ff [0,4194304] 0 2024-04-22T15:18:38.181 INFO:tasks.workunit.client.0.smithi096.stdout:3/134: dwrite da/f27 [0,4194304] 0 2024-04-22T15:18:38.181 INFO:tasks.workunit.client.0.smithi096.stdout:2/176: write d3/f2a [1079893,65008] 0 2024-04-22T15:18:38.186 INFO:tasks.workunit.client.0.smithi096.stdout:5/159: chown d4/l16 164497 1 2024-04-22T15:18:38.187 INFO:tasks.workunit.client.0.smithi096.stdout:2/177: write d3/db/f12 [2814927,85963] 0 2024-04-22T15:18:38.189 INFO:tasks.workunit.client.0.smithi096.stdout:0/205: truncate d7/d9/dc/f29 1934470 0 2024-04-22T15:18:38.195 INFO:tasks.workunit.client.0.smithi096.stdout:4/157: creat d5/d7/d1a/f38 x:0 0 0 2024-04-22T15:18:38.197 INFO:tasks.workunit.client.0.smithi096.stdout:8/187: mknod d4/d10/c42 0 2024-04-22T15:18:38.200 INFO:tasks.workunit.client.0.smithi096.stdout:6/184: read d3/d28/f1d [738279,2938] 0 2024-04-22T15:18:38.201 INFO:tasks.workunit.client.0.smithi096.stdout:9/141: creat d1/d25/f2e x:0 0 0 2024-04-22T15:18:38.203 INFO:tasks.workunit.client.0.smithi096.stdout:1/235: rename d6/dc/d11/d40/d34/f1a to d6/dc/d11/d4a/f4d 0 2024-04-22T15:18:38.205 INFO:tasks.workunit.client.0.smithi096.stdout:3/135: mkdir da/db/de/d19/d2d 0 2024-04-22T15:18:38.208 INFO:tasks.workunit.client.0.smithi096.stdout:2/178: creat d3/db/df/f35 x:0 0 0 2024-04-22T15:18:38.209 INFO:tasks.workunit.client.0.smithi096.stdout:2/179: dread - d3/db/df/f27 zero size 2024-04-22T15:18:38.210 INFO:tasks.workunit.client.0.smithi096.stdout:0/206: unlink f0 0 2024-04-22T15:18:38.212 INFO:tasks.workunit.client.0.smithi096.stdout:7/251: mknod d31/d32/d33/d44/c53 0 2024-04-22T15:18:38.214 INFO:tasks.workunit.client.0.smithi096.stdout:7/252: chown fb 0 1 2024-04-22T15:18:38.214 INFO:tasks.workunit.client.0.smithi096.stdout:5/160: dwrite d4/d11/d25/f38 [0,4194304] 0 2024-04-22T15:18:38.215 INFO:tasks.workunit.client.0.smithi096.stdout:8/188: rename d4/d1c/f2d to d4/d1c/f43 0 2024-04-22T15:18:38.215 INFO:tasks.workunit.client.0.smithi096.stdout:5/161: stat d4/d2c 0 2024-04-22T15:18:38.219 INFO:tasks.workunit.client.0.smithi096.stdout:2/180: dwrite d3/db/df/d10/f21 [0,4194304] 0 2024-04-22T15:18:38.228 INFO:tasks.workunit.client.0.smithi096.stdout:2/181: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:18:38.230 INFO:tasks.workunit.client.0.smithi096.stdout:2/182: truncate d3/db/df/f27 581202 0 2024-04-22T15:18:38.230 INFO:tasks.workunit.client.0.smithi096.stdout:4/158: rename d5/d7/f16 to d5/f39 0 2024-04-22T15:18:38.232 INFO:tasks.workunit.client.0.smithi096.stdout:1/236: mkdir d6/dc/d11/d40/d34/d12/d32/d3b/d3d/d4e 0 2024-04-22T15:18:38.232 INFO:tasks.workunit.client.0.smithi096.stdout:1/237: chown d6/dc/d11/c33 137 1 2024-04-22T15:18:38.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:37 smithi096 ceph-mon[26688]: pgmap v25: 65 pgs: 65 active+clean; 306 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 19 MiB/s wr, 239 op/s 2024-04-22T15:18:38.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:37 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:18:38.243 INFO:tasks.workunit.client.0.smithi096.stdout:0/207: symlink d7/d9/d15/l48 0 2024-04-22T15:18:38.250 INFO:tasks.workunit.client.0.smithi096.stdout:5/162: dread d4/d11/f22 [0,4194304] 0 2024-04-22T15:18:38.257 INFO:tasks.workunit.client.0.smithi096.stdout:2/183: rename d3/db/c22 to d3/db/df/d24/c36 0 2024-04-22T15:18:38.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:37 smithi196 ceph-mon[34717]: pgmap v25: 65 pgs: 65 active+clean; 306 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 19 MiB/s wr, 239 op/s 2024-04-22T15:18:38.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:37 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:18:38.260 INFO:tasks.workunit.client.0.smithi096.stdout:4/159: dread d5/f18 [0,4194304] 0 2024-04-22T15:18:38.261 INFO:tasks.workunit.client.0.smithi096.stdout:3/136: creat da/db/de/d19/d2d/f2e x:0 0 0 2024-04-22T15:18:38.266 INFO:tasks.workunit.client.0.smithi096.stdout:6/185: rmdir d3/d28/dd/d40 0 2024-04-22T15:18:38.267 INFO:tasks.workunit.client.0.smithi096.stdout:4/160: dwrite d5/d7/d1a/d25/f30 [0,4194304] 0 2024-04-22T15:18:38.268 INFO:tasks.workunit.client.0.smithi096.stdout:5/163: chown d4/d1f/f21 12036 1 2024-04-22T15:18:38.271 INFO:tasks.workunit.client.0.smithi096.stdout:9/142: creat d1/f2f x:0 0 0 2024-04-22T15:18:38.274 INFO:tasks.workunit.client.0.smithi096.stdout:2/184: rename d3/db/df/f1a to d3/db/df/d10/f37 0 2024-04-22T15:18:38.288 INFO:tasks.workunit.client.0.smithi096.stdout:9/143: symlink d1/df/l30 0 2024-04-22T15:18:38.288 INFO:tasks.workunit.client.0.smithi096.stdout:9/144: readlink d1/d4/d9/l17 0 2024-04-22T15:18:38.291 INFO:tasks.workunit.client.0.smithi096.stdout:2/185: creat d3/db/df/d24/f38 x:0 0 0 2024-04-22T15:18:38.293 INFO:tasks.workunit.client.0.smithi096.stdout:1/238: creat d6/f4f x:0 0 0 2024-04-22T15:18:38.297 INFO:tasks.workunit.client.0.smithi096.stdout:3/137: truncate da/db/dc/f16 2594565 0 2024-04-22T15:18:38.298 INFO:tasks.workunit.client.0.smithi096.stdout:0/208: link d7/d9/dc/c11 d7/d9/dc/d16/c49 0 2024-04-22T15:18:38.299 INFO:tasks.workunit.client.0.smithi096.stdout:0/209: chown d7/d9/d15/f1a 20 1 2024-04-22T15:18:38.300 INFO:tasks.workunit.client.0.smithi096.stdout:0/210: dread - d7/d9/dc/d16/d18/f3d zero size 2024-04-22T15:18:38.300 INFO:tasks.workunit.client.0.smithi096.stdout:0/211: dread - d7/d9/f45 zero size 2024-04-22T15:18:38.304 INFO:tasks.workunit.client.0.smithi096.stdout:4/161: link d5/d7/d1a/d25/f30 d5/d7/d2c/f3a 0 2024-04-22T15:18:38.305 INFO:tasks.workunit.client.0.smithi096.stdout:4/162: write d5/f19 [485117,122036] 0 2024-04-22T15:18:38.308 INFO:tasks.workunit.client.0.smithi096.stdout:9/145: creat d1/d25/f31 x:0 0 0 2024-04-22T15:18:38.315 INFO:tasks.workunit.client.0.smithi096.stdout:6/186: creat d3/d28/dd/d11/f41 x:0 0 0 2024-04-22T15:18:38.315 INFO:tasks.workunit.client.0.smithi096.stdout:6/187: dread - d3/f24 zero size 2024-04-22T15:18:38.316 INFO:tasks.workunit.client.0.smithi096.stdout:6/188: dread - d3/f29 zero size 2024-04-22T15:18:38.317 INFO:tasks.workunit.client.0.smithi096.stdout:0/212: mknod d7/d9/dc/d16/d18/c4a 0 2024-04-22T15:18:38.318 INFO:tasks.workunit.client.0.smithi096.stdout:0/213: fdatasync d7/d9/dc/d16/d18/f30 0 2024-04-22T15:18:38.319 INFO:tasks.workunit.client.0.smithi096.stdout:0/214: fdatasync d7/f20 0 2024-04-22T15:18:38.323 INFO:tasks.workunit.client.0.smithi096.stdout:4/163: mknod d5/d7/d1a/d25/c3b 0 2024-04-22T15:18:38.324 INFO:tasks.workunit.client.0.smithi096.stdout:5/164: dwrite d4/d11/f22 [0,4194304] 0 2024-04-22T15:18:38.326 INFO:tasks.workunit.client.0.smithi096.stdout:4/164: truncate d5/d7/f36 566924 0 2024-04-22T15:18:38.328 INFO:tasks.workunit.client.0.smithi096.stdout:2/186: creat d3/db/d30/f39 x:0 0 0 2024-04-22T15:18:38.330 INFO:tasks.workunit.client.0.smithi096.stdout:1/239: creat d6/dc/d11/d45/f50 x:0 0 0 2024-04-22T15:18:38.333 INFO:tasks.workunit.client.0.smithi096.stdout:3/138: mknod da/db/c2f 0 2024-04-22T15:18:38.341 INFO:tasks.workunit.client.0.smithi096.stdout:5/165: fdatasync f1 0 2024-04-22T15:18:38.341 INFO:tasks.workunit.client.0.smithi096.stdout:5/166: chown d4/d1f/d29 1013638646 1 2024-04-22T15:18:38.344 INFO:tasks.workunit.client.0.smithi096.stdout:0/215: rename d7/d9/dc/d16/d18/c4a to d7/d9/d15/d35/c4b 0 2024-04-22T15:18:38.351 INFO:tasks.workunit.client.0.smithi096.stdout:4/165: mknod d5/d7/d1a/d25/c3c 0 2024-04-22T15:18:38.352 INFO:tasks.workunit.client.0.smithi096.stdout:4/166: chown d5/d7/d2c/l2d 239 1 2024-04-22T15:18:38.355 INFO:tasks.workunit.client.0.smithi096.stdout:1/240: write d6/dc/d10/d3f/f49 [865151,32656] 0 2024-04-22T15:18:38.356 INFO:tasks.workunit.client.0.smithi096.stdout:1/241: write d6/f14 [3491780,52356] 0 2024-04-22T15:18:38.358 INFO:tasks.workunit.client.0.smithi096.stdout:1/242: stat d6/dc/d11/d40/d34/d12/d32/d3b 0 2024-04-22T15:18:38.358 INFO:tasks.workunit.client.0.smithi096.stdout:1/243: dread - d6/f2c zero size 2024-04-22T15:18:38.359 INFO:tasks.workunit.client.0.smithi096.stdout:1/244: write d6/f4f [887795,89833] 0 2024-04-22T15:18:38.362 INFO:tasks.workunit.client.0.smithi096.stdout:5/167: rmdir d4/d1f/d33 39 2024-04-22T15:18:38.366 INFO:tasks.workunit.client.0.smithi096.stdout:9/146: link d1/d25/f2e d1/f32 0 2024-04-22T15:18:38.367 INFO:tasks.workunit.client.0.smithi096.stdout:9/147: read d1/d4/f22 [116073,105885] 0 2024-04-22T15:18:38.368 INFO:tasks.workunit.client.0.smithi096.stdout:9/148: chown d1/d4/d9/f12 52 1 2024-04-22T15:18:38.369 INFO:tasks.workunit.client.0.smithi096.stdout:4/167: mkdir d5/d7/d2c/d3d 0 2024-04-22T15:18:38.384 INFO:tasks.workunit.client.0.smithi096.stdout:4/168: dread d5/fc [0,4194304] 0 2024-04-22T15:18:38.384 INFO:tasks.workunit.client.0.smithi096.stdout:9/149: write d1/f20 [434663,79147] 0 2024-04-22T15:18:38.385 INFO:tasks.workunit.client.0.smithi096.stdout:4/169: fdatasync d5/d7/d1a/f23 0 2024-04-22T15:18:38.386 INFO:tasks.workunit.client.0.smithi096.stdout:0/216: write d7/d9/dc/f14 [3472358,645] 0 2024-04-22T15:18:38.388 INFO:tasks.workunit.client.0.smithi096.stdout:6/189: dwrite d3/f22 [0,4194304] 0 2024-04-22T15:18:38.389 INFO:tasks.workunit.client.0.smithi096.stdout:1/245: rename d6/dc/d10/l2b to d6/dc/d11/d40/d34/d12/d32/l51 0 2024-04-22T15:18:38.402 INFO:tasks.workunit.client.0.smithi096.stdout:2/187: dwrite d3/fd [0,4194304] 0 2024-04-22T15:18:38.402 INFO:tasks.workunit.client.0.smithi096.stdout:0/217: rename d7/de/d34 to d7/d42/d4c 0 2024-04-22T15:18:38.415 INFO:tasks.workunit.client.0.smithi096.stdout:9/150: write d1/f6 [512989,39453] 0 2024-04-22T15:18:38.417 INFO:tasks.workunit.client.0.smithi096.stdout:2/188: unlink d3/db/df/f27 0 2024-04-22T15:18:38.418 INFO:tasks.workunit.client.0.smithi096.stdout:2/189: read - d3/db/df/d24/f38 zero size 2024-04-22T15:18:38.418 INFO:tasks.workunit.client.0.smithi096.stdout:2/190: fsync d3/f2a 0 2024-04-22T15:18:38.418 INFO:tasks.workunit.client.0.smithi096.stdout:5/168: getdents d4/d2c/d2f/d31 0 2024-04-22T15:18:38.419 INFO:tasks.workunit.client.0.smithi096.stdout:5/169: dread - d4/d12/f15 zero size 2024-04-22T15:18:38.421 INFO:tasks.workunit.client.0.smithi096.stdout:4/170: link d5/f19 d5/d7/f3e 0 2024-04-22T15:18:38.423 INFO:tasks.workunit.client.0.smithi096.stdout:5/170: dread d4/d11/f22 [0,4194304] 0 2024-04-22T15:18:38.424 INFO:tasks.workunit.client.0.smithi096.stdout:5/171: chown d4/ff 2855 1 2024-04-22T15:18:38.424 INFO:tasks.workunit.client.0.smithi096.stdout:1/246: dread - d6/dc/d10/d29/f36 zero size 2024-04-22T15:18:38.425 INFO:tasks.workunit.client.0.smithi096.stdout:5/172: chown d4/c2e 80585 1 2024-04-22T15:18:38.426 INFO:tasks.workunit.client.0.smithi096.stdout:4/171: dread d5/d7/d2c/f2e [0,4194304] 0 2024-04-22T15:18:38.426 INFO:tasks.workunit.client.0.smithi096.stdout:5/173: chown d4/f6 14 1 2024-04-22T15:18:38.427 INFO:tasks.workunit.client.0.smithi096.stdout:5/174: truncate d4/fa 4572126 0 2024-04-22T15:18:38.429 INFO:tasks.workunit.client.0.smithi096.stdout:1/247: dread d6/dc/f41 [0,4194304] 0 2024-04-22T15:18:38.430 INFO:tasks.workunit.client.0.smithi096.stdout:9/151: creat d1/d25/f33 x:0 0 0 2024-04-22T15:18:38.432 INFO:tasks.workunit.client.0.smithi096.stdout:9/152: dread d1/f6 [0,4194304] 0 2024-04-22T15:18:38.434 INFO:tasks.workunit.client.0.smithi096.stdout:2/191: creat d3/db/d30/f3a x:0 0 0 2024-04-22T15:18:38.434 INFO:tasks.workunit.client.0.smithi096.stdout:2/192: stat d3/db 0 2024-04-22T15:18:38.435 INFO:tasks.workunit.client.0.smithi096.stdout:9/153: dread d1/d4/d9/dd/f1a [0,4194304] 0 2024-04-22T15:18:38.442 INFO:tasks.workunit.client.0.smithi096.stdout:0/218: creat d7/f4d x:0 0 0 2024-04-22T15:18:38.443 INFO:tasks.workunit.client.0.smithi096.stdout:0/219: write d7/d9/d15/d35/f41 [365561,46764] 0 2024-04-22T15:18:38.447 INFO:tasks.workunit.client.0.smithi096.stdout:1/248: creat d6/dc/d11/d40/d34/d12/d32/f52 x:0 0 0 2024-04-22T15:18:38.448 INFO:tasks.workunit.client.0.smithi096.stdout:1/249: truncate d6/dc/d11/f3e 1329678 0 2024-04-22T15:18:38.450 INFO:tasks.workunit.client.0.smithi096.stdout:1/250: truncate d6/dc/d11/d40/d34/d12/f43 203947 0 2024-04-22T15:18:38.451 INFO:tasks.workunit.client.0.smithi096.stdout:1/251: chown d6/dc/d11/d40/d34 197673 1 2024-04-22T15:18:38.452 INFO:tasks.workunit.client.0.smithi096.stdout:2/193: creat d3/db/df/d10/f3b x:0 0 0 2024-04-22T15:18:38.455 INFO:tasks.workunit.client.0.smithi096.stdout:9/154: mknod d1/d25/c34 0 2024-04-22T15:18:38.456 INFO:tasks.workunit.client.0.smithi096.stdout:1/252: dwrite f2 [4194304,4194304] 0 2024-04-22T15:18:38.458 INFO:tasks.workunit.client.0.smithi096.stdout:1/253: write d6/dc/d11/d40/d34/f44 [628976,74099] 0 2024-04-22T15:18:38.458 INFO:tasks.workunit.client.0.smithi096.stdout:5/175: write d4/f26 [291704,29900] 0 2024-04-22T15:18:38.460 INFO:tasks.workunit.client.0.smithi096.stdout:5/176: chown d4/d12/f13 1931300462 1 2024-04-22T15:18:38.461 INFO:tasks.workunit.client.0.smithi096.stdout:1/254: write d6/dc/d11/d40/d34/d12/f18 [2597626,2015] 0 2024-04-22T15:18:38.462 INFO:tasks.workunit.client.0.smithi096.stdout:6/190: rename d3/d28/dd/c14 to d3/c42 0 2024-04-22T15:18:38.464 INFO:tasks.workunit.client.0.smithi096.stdout:6/191: write d3/f24 [850070,93402] 0 2024-04-22T15:18:38.464 INFO:tasks.workunit.client.0.smithi096.stdout:0/220: mkdir d7/d9/d15/d35/d4e 0 2024-04-22T15:18:38.466 INFO:tasks.workunit.client.0.smithi096.stdout:4/172: link d5/fc d5/d7/d1a/d25/f3f 0 2024-04-22T15:18:38.468 INFO:tasks.workunit.client.0.smithi096.stdout:1/255: read d6/dc/d11/d40/f38 [12220,103975] 0 2024-04-22T15:18:38.474 INFO:tasks.workunit.client.0.smithi096.stdout:0/221: dwrite d7/d9/f17 [0,4194304] 0 2024-04-22T15:18:38.477 INFO:tasks.workunit.client.0.smithi096.stdout:5/177: mknod d4/d2c/d2f/d30/c39 0 2024-04-22T15:18:38.477 INFO:tasks.workunit.client.0.smithi096.stdout:4/173: dwrite f2 [0,4194304] 0 2024-04-22T15:18:38.483 INFO:tasks.workunit.client.0.smithi096.stdout:2/194: link d3/d13/l28 d3/db/df/l3c 0 2024-04-22T15:18:38.484 INFO:tasks.workunit.client.0.smithi096.stdout:1/256: dwrite d6/dc/d11/d45/f50 [0,4194304] 0 2024-04-22T15:18:38.489 INFO:tasks.workunit.client.0.smithi096.stdout:9/155: creat d1/f35 x:0 0 0 2024-04-22T15:18:38.491 INFO:tasks.workunit.client.0.smithi096.stdout:4/174: unlink d5/f18 0 2024-04-22T15:18:38.491 INFO:tasks.workunit.client.0.smithi096.stdout:4/175: chown d5/c8 26 1 2024-04-22T15:18:38.492 INFO:tasks.workunit.client.0.smithi096.stdout:5/178: creat d4/d2c/d2f/f3a x:0 0 0 2024-04-22T15:18:38.496 INFO:tasks.workunit.client.0.smithi096.stdout:2/195: unlink d3/d1f/f29 0 2024-04-22T15:18:38.497 INFO:tasks.workunit.client.0.smithi096.stdout:2/196: read d3/db/f17 [3600882,124275] 0 2024-04-22T15:18:38.498 INFO:tasks.workunit.client.0.smithi096.stdout:6/192: dread f0 [0,4194304] 0 2024-04-22T15:18:38.498 INFO:tasks.workunit.client.0.smithi096.stdout:2/197: write d3/f2e [1310642,118233] 0 2024-04-22T15:18:38.499 INFO:tasks.workunit.client.0.smithi096.stdout:2/198: dread - d3/db/df/d24/f38 zero size 2024-04-22T15:18:38.500 INFO:tasks.workunit.client.0.smithi096.stdout:1/257: creat d6/dc/d11/d40/f53 x:0 0 0 2024-04-22T15:18:38.500 INFO:tasks.workunit.client.0.smithi096.stdout:2/199: readlink d3/l5 0 2024-04-22T15:18:38.502 INFO:tasks.workunit.client.0.smithi096.stdout:9/156: dread d1/f6 [0,4194304] 0 2024-04-22T15:18:38.503 INFO:tasks.workunit.client.0.smithi096.stdout:9/157: chown d1/d4/f22 0 1 2024-04-22T15:18:38.503 INFO:tasks.workunit.client.0.smithi096.stdout:9/158: readlink d1/la 0 2024-04-22T15:18:38.504 INFO:tasks.workunit.client.0.smithi096.stdout:9/159: write d1/d4/d9/dd/f23 [424420,97325] 0 2024-04-22T15:18:38.511 INFO:tasks.workunit.client.0.smithi096.stdout:9/160: dwrite d1/f20 [0,4194304] 0 2024-04-22T15:18:38.512 INFO:tasks.workunit.client.0.smithi096.stdout:9/161: chown d1/d25/f31 219406436 1 2024-04-22T15:18:38.517 INFO:tasks.workunit.client.0.smithi096.stdout:5/179: link d4/f1c d4/d2c/d2f/d31/f3b 0 2024-04-22T15:18:38.523 INFO:tasks.workunit.client.0.smithi096.stdout:0/222: getdents d7/d9 0 2024-04-22T15:18:38.523 INFO:tasks.workunit.client.0.smithi096.stdout:2/200: rmdir d3 39 2024-04-22T15:18:38.525 INFO:tasks.workunit.client.0.smithi096.stdout:9/162: creat d1/d4/d9/f36 x:0 0 0 2024-04-22T15:18:38.527 INFO:tasks.workunit.client.0.smithi096.stdout:0/223: dread d7/d9/f24 [0,4194304] 0 2024-04-22T15:18:38.531 INFO:tasks.workunit.client.0.smithi096.stdout:1/258: dread d6/dc/d11/d4a/f4d [0,4194304] 0 2024-04-22T15:18:38.531 INFO:tasks.workunit.client.0.smithi096.stdout:4/176: rename d5/d7/f3e to d5/d7/d1a/f40 0 2024-04-22T15:18:38.535 INFO:tasks.workunit.client.0.smithi096.stdout:9/163: unlink d1/f35 0 2024-04-22T15:18:38.550 INFO:tasks.workunit.client.0.smithi096.stdout:9/164: dread d1/d4/d9/dd/f1a [0,4194304] 0 2024-04-22T15:18:38.551 INFO:tasks.workunit.client.0.smithi096.stdout:5/180: creat d4/f3c x:0 0 0 2024-04-22T15:18:38.551 INFO:tasks.workunit.client.0.smithi096.stdout:9/165: truncate d1/f32 800441 0 2024-04-22T15:18:38.552 INFO:tasks.workunit.client.0.smithi096.stdout:0/224: link d7/d9/dc/d16/d18/f30 d7/d42/d4c/f4f 0 2024-04-22T15:18:38.553 INFO:tasks.workunit.client.0.smithi096.stdout:0/225: dread - d7/d9/dc/d16/d18/f3d zero size 2024-04-22T15:18:38.553 INFO:tasks.workunit.client.0.smithi096.stdout:4/177: dread - d5/d7/d1a/f38 zero size 2024-04-22T15:18:38.555 INFO:tasks.workunit.client.0.smithi096.stdout:4/178: read d5/d7/d1a/f1d [2110677,122631] 0 2024-04-22T15:18:38.556 INFO:tasks.workunit.client.0.smithi096.stdout:1/259: rename d6/dc/d11/d40/d34/d12 to d6/dc/d10/d3f/d54 0 2024-04-22T15:18:38.559 INFO:tasks.workunit.client.0.smithi096.stdout:9/166: mknod d1/df/c37 0 2024-04-22T15:18:38.562 INFO:tasks.workunit.client.0.smithi096.stdout:1/260: chown d6/fd 4564 1 2024-04-22T15:18:38.563 INFO:tasks.workunit.client.0.smithi096.stdout:2/201: truncate d3/db/df/d10/f1b 4819857 0 2024-04-22T15:18:38.564 INFO:tasks.workunit.client.0.smithi096.stdout:9/167: truncate d1/f6 781376 0 2024-04-22T15:18:38.567 INFO:tasks.workunit.client.0.smithi096.stdout:5/181: rename d4/d1f/d33 to d4/d11/d3d 0 2024-04-22T15:18:38.568 INFO:tasks.workunit.client.0.smithi096.stdout:1/261: dwrite d6/f22 [0,4194304] 0 2024-04-22T15:18:38.568 INFO:tasks.workunit.client.0.smithi096.stdout:1/262: stat d6/dc/f2e 0 2024-04-22T15:18:38.569 INFO:tasks.workunit.client.0.smithi096.stdout:2/202: mknod d3/d13/c3d 0 2024-04-22T15:18:38.569 INFO:tasks.workunit.client.0.smithi096.stdout:1/263: dread - d6/f2c zero size 2024-04-22T15:18:38.575 INFO:tasks.workunit.client.0.smithi096.stdout:2/203: chown d3/db/f31 692 1 2024-04-22T15:18:38.576 INFO:tasks.workunit.client.0.smithi096.stdout:1/264: mknod d6/dc/d11/d40/c55 0 2024-04-22T15:18:38.578 INFO:tasks.workunit.client.0.smithi096.stdout:0/226: rename d7/d9/d15/l38 to d7/d9/dc/d16/l50 0 2024-04-22T15:18:38.580 INFO:tasks.workunit.client.0.smithi096.stdout:5/182: symlink d4/d11/d3d/d35/l3e 0 2024-04-22T15:18:38.580 INFO:tasks.workunit.client.0.smithi096.stdout:1/265: dread d6/dc/f23 [0,4194304] 0 2024-04-22T15:18:38.581 INFO:tasks.workunit.client.0.smithi096.stdout:5/183: write d4/f5 [666424,38586] 0 2024-04-22T15:18:38.581 INFO:tasks.workunit.client.0.smithi096.stdout:9/168: creat d1/f38 x:0 0 0 2024-04-22T15:18:38.583 INFO:tasks.workunit.client.0.smithi096.stdout:2/204: unlink d3/db/df/c19 0 2024-04-22T15:18:38.585 INFO:tasks.workunit.client.0.smithi096.stdout:0/227: mknod d7/d9/dc/d16/c51 0 2024-04-22T15:18:38.588 INFO:tasks.workunit.client.0.smithi096.stdout:9/169: creat d1/d4/f39 x:0 0 0 2024-04-22T15:18:38.588 INFO:tasks.workunit.client.0.smithi096.stdout:5/184: dwrite d4/fd [0,4194304] 0 2024-04-22T15:18:38.590 INFO:tasks.workunit.client.0.smithi096.stdout:1/266: rename d6/dc/d11/d40/d34 to d6/dc/d10/d3f/d54/d32/d3b/d56 0 2024-04-22T15:18:38.595 INFO:tasks.workunit.client.0.smithi096.stdout:1/267: read d6/f14 [3072884,58894] 0 2024-04-22T15:18:38.598 INFO:tasks.workunit.client.0.smithi096.stdout:4/179: truncate d5/d7/d2c/f3a 2874429 0 2024-04-22T15:18:38.604 INFO:tasks.workunit.client.0.smithi096.stdout:2/205: creat d3/db/d30/f3e x:0 0 0 2024-04-22T15:18:38.605 INFO:tasks.workunit.client.0.smithi096.stdout:2/206: stat d3/db 0 2024-04-22T15:18:38.605 INFO:tasks.workunit.client.0.smithi096.stdout:1/268: readlink d6/dc/d10/l28 0 2024-04-22T15:18:38.606 INFO:tasks.workunit.client.0.smithi096.stdout:1/269: readlink d6/dc/d10/d3f/d54/l46 0 2024-04-22T15:18:38.612 INFO:tasks.workunit.client.0.smithi096.stdout:0/228: write d7/d9/dc/d16/d18/f30 [2665352,12483] 0 2024-04-22T15:18:38.614 INFO:tasks.workunit.client.0.smithi096.stdout:0/229: truncate d7/f32 5040119 0 2024-04-22T15:18:38.614 INFO:tasks.workunit.client.0.smithi096.stdout:1/270: fsync d6/fd 0 2024-04-22T15:18:38.618 INFO:tasks.workunit.client.0.smithi096.stdout:2/207: creat d3/db/d2d/f3f x:0 0 0 2024-04-22T15:18:38.619 INFO:tasks.workunit.client.0.smithi096.stdout:9/170: write d1/f6 [925840,32757] 0 2024-04-22T15:18:38.620 INFO:tasks.workunit.client.0.smithi096.stdout:5/185: truncate d4/d11/d25/f38 607256 0 2024-04-22T15:18:38.622 INFO:tasks.workunit.client.0.smithi096.stdout:1/271: creat d6/dc/d10/d3f/f57 x:0 0 0 2024-04-22T15:18:38.624 INFO:tasks.workunit.client.0.smithi096.stdout:1/272: write d6/dc/d10/d29/f36 [425957,58098] 0 2024-04-22T15:18:38.624 INFO:tasks.workunit.client.0.smithi096.stdout:4/180: link d5/f39 d5/d7/f41 0 2024-04-22T15:18:38.627 INFO:tasks.workunit.client.0.smithi096.stdout:5/186: creat d4/d2c/d2f/f3f x:0 0 0 2024-04-22T15:18:38.628 INFO:tasks.workunit.client.0.smithi096.stdout:0/230: symlink d7/d9/dc/l52 0 2024-04-22T15:18:38.630 INFO:tasks.workunit.client.0.smithi096.stdout:1/273: creat d6/dc/d11/d45/f58 x:0 0 0 2024-04-22T15:18:38.636 INFO:tasks.workunit.client.0.smithi096.stdout:5/187: mknod d4/d11/c40 0 2024-04-22T15:18:38.637 INFO:tasks.workunit.client.0.smithi096.stdout:5/188: fsync d4/d12/f15 0 2024-04-22T15:18:38.637 INFO:tasks.workunit.client.0.smithi096.stdout:0/231: symlink d7/d9/dc/d26/l53 0 2024-04-22T15:18:38.639 INFO:tasks.workunit.client.0.smithi096.stdout:9/171: write d1/f32 [1769723,53846] 0 2024-04-22T15:18:38.641 INFO:tasks.workunit.client.0.smithi096.stdout:1/274: creat d6/dc/d10/d29/f59 x:0 0 0 2024-04-22T15:18:38.645 INFO:tasks.workunit.client.0.smithi096.stdout:5/189: unlink d4/f10 0 2024-04-22T15:18:38.645 INFO:tasks.workunit.client.0.smithi096.stdout:9/172: dwrite d1/d4/d9/f18 [0,4194304] 0 2024-04-22T15:18:38.647 INFO:tasks.workunit.client.0.smithi096.stdout:0/232: creat d7/d9/dc/d16/d18/f54 x:0 0 0 2024-04-22T15:18:38.648 INFO:tasks.workunit.client.0.smithi096.stdout:0/233: dread - d7/d9/dc/d16/d18/f39 zero size 2024-04-22T15:18:38.652 INFO:tasks.workunit.client.0.smithi096.stdout:1/275: mknod d6/dc/d10/d3f/d54/c5a 0 2024-04-22T15:18:38.652 INFO:tasks.workunit.client.0.smithi096.stdout:2/208: write d3/f8 [3428093,78638] 0 2024-04-22T15:18:38.653 INFO:tasks.workunit.client.0.smithi096.stdout:5/190: dwrite d4/d2c/d2f/f3a [0,4194304] 0 2024-04-22T15:18:38.653 INFO:tasks.workunit.client.0.smithi096.stdout:1/276: chown f2 89516696 1 2024-04-22T15:18:38.655 INFO:tasks.workunit.client.0.smithi096.stdout:1/277: dread - d6/dc/d10/d3f/d54/f4c zero size 2024-04-22T15:18:38.657 INFO:tasks.workunit.client.0.smithi096.stdout:9/173: creat d1/df/f3a x:0 0 0 2024-04-22T15:18:38.658 INFO:tasks.workunit.client.0.smithi096.stdout:9/174: fsync d1/d4/d9/dd/f1d 0 2024-04-22T15:18:38.660 INFO:tasks.workunit.client.0.smithi096.stdout:9/175: write d1/d4/f11 [2022127,63457] 0 2024-04-22T15:18:38.663 INFO:tasks.workunit.client.0.smithi096.stdout:0/234: mkdir d7/d42/d4c/d55 0 2024-04-22T15:18:38.665 INFO:tasks.workunit.client.0.smithi096.stdout:4/181: rename d5/d7/d1a/d25/f30 to d5/d7/d1f/f42 0 2024-04-22T15:18:38.671 INFO:tasks.workunit.client.0.smithi096.stdout:9/176: truncate d1/d4/d9/dd/f1a 208202 0 2024-04-22T15:18:38.672 INFO:tasks.workunit.client.0.smithi096.stdout:0/235: fsync d7/d9/dc/d16/d18/f3d 0 2024-04-22T15:18:38.673 INFO:tasks.workunit.client.0.smithi096.stdout:4/182: stat d5/d7/f9 0 2024-04-22T15:18:38.674 INFO:tasks.workunit.client.0.smithi096.stdout:4/183: fsync d5/fe 0 2024-04-22T15:18:38.674 INFO:tasks.workunit.client.0.smithi096.stdout:0/236: write d7/d9/dc/d16/f1f [153473,32902] 0 2024-04-22T15:18:38.676 INFO:tasks.workunit.client.0.smithi096.stdout:2/209: symlink d3/l40 0 2024-04-22T15:18:38.677 INFO:tasks.workunit.client.0.smithi096.stdout:1/278: dread d6/fd [0,4194304] 0 2024-04-22T15:18:38.679 INFO:tasks.workunit.client.0.smithi096.stdout:9/177: creat d1/d4/d9/f3b x:0 0 0 2024-04-22T15:18:38.680 INFO:tasks.workunit.client.0.smithi096.stdout:2/210: read d3/db/f31 [4092,81560] 0 2024-04-22T15:18:38.684 INFO:tasks.workunit.client.0.smithi096.stdout:4/184: mkdir d5/d7/d2c/d43 0 2024-04-22T15:18:38.685 INFO:tasks.workunit.client.0.smithi096.stdout:0/237: dwrite d7/d9/f17 [8388608,4194304] 0 2024-04-22T15:18:38.686 INFO:tasks.workunit.client.0.smithi096.stdout:5/191: dread d4/f5 [0,4194304] 0 2024-04-22T15:18:38.701 INFO:tasks.workunit.client.0.smithi096.stdout:5/192: write f1 [3408415,107435] 0 2024-04-22T15:18:38.702 INFO:tasks.workunit.client.0.smithi096.stdout:4/185: creat d5/d7/d1a/d25/f44 x:0 0 0 2024-04-22T15:18:38.705 INFO:tasks.workunit.client.0.smithi096.stdout:5/193: write d4/f3c [807640,121250] 0 2024-04-22T15:18:38.706 INFO:tasks.workunit.client.0.smithi096.stdout:0/238: creat d7/d9/dc/f56 x:0 0 0 2024-04-22T15:18:38.707 INFO:tasks.workunit.client.0.smithi096.stdout:2/211: dwrite d3/db/df/f35 [0,4194304] 0 2024-04-22T15:18:38.710 INFO:tasks.workunit.client.0.smithi096.stdout:1/279: dwrite d6/dc/d11/f19 [0,4194304] 0 2024-04-22T15:18:38.711 INFO:tasks.workunit.client.0.smithi096.stdout:0/239: write d7/d9/dc/d16/d18/f39 [858115,40725] 0 2024-04-22T15:18:38.712 INFO:tasks.workunit.client.0.smithi096.stdout:5/194: mknod d4/d2c/d2f/d30/c41 0 2024-04-22T15:18:38.713 INFO:tasks.workunit.client.0.smithi096.stdout:4/186: dread d5/fb [0,4194304] 0 2024-04-22T15:18:38.715 INFO:tasks.workunit.client.0.smithi096.stdout:1/280: rmdir d6/dc 39 2024-04-22T15:18:38.715 INFO:tasks.workunit.client.0.smithi096.stdout:1/281: chown d6/f39 62 1 2024-04-22T15:18:38.718 INFO:tasks.workunit.client.0.smithi096.stdout:2/212: rename d3/db/d2d/f3f to d3/d1f/d2f/f41 0 2024-04-22T15:18:38.720 INFO:tasks.workunit.client.0.smithi096.stdout:5/195: rmdir d4/d12 39 2024-04-22T15:18:38.731 INFO:tasks.workunit.client.0.smithi096.stdout:4/187: dwrite d5/fb [4194304,4194304] 0 2024-04-22T15:18:38.733 INFO:tasks.workunit.client.0.smithi096.stdout:0/240: creat d7/d9/dc/f57 x:0 0 0 2024-04-22T15:18:38.734 INFO:tasks.workunit.client.0.smithi096.stdout:5/196: truncate d4/d2c/d2f/d31/f3b 7894238 0 2024-04-22T15:18:38.736 INFO:tasks.workunit.client.0.smithi096.stdout:1/282: link d6/dc/d10/d3f/d54/d32/d3b/d3d/f4b d6/dc/d11/d4a/f5b 0 2024-04-22T15:18:38.738 INFO:tasks.workunit.client.0.smithi096.stdout:4/188: unlink d5/d7/d2c/l32 0 2024-04-22T15:18:38.738 INFO:tasks.workunit.client.0.smithi096.stdout:1/283: read d6/f4f [869384,124473] 0 2024-04-22T15:18:38.740 INFO:tasks.workunit.client.0.smithi096.stdout:1/284: chown d6/dc/d10/d3f/d54/f21 557 1 2024-04-22T15:18:38.742 INFO:tasks.workunit.client.0.smithi096.stdout:1/285: write d6/dc/d10/d3f/f57 [606442,117674] 0 2024-04-22T15:18:38.743 INFO:tasks.workunit.client.0.smithi096.stdout:2/213: rename d3/db/f17 to d3/db/f42 0 2024-04-22T15:18:38.746 INFO:tasks.workunit.client.0.smithi096.stdout:2/214: dread d3/db/f42 [4194304,4194304] 0 2024-04-22T15:18:38.749 INFO:tasks.workunit.client.0.smithi096.stdout:5/197: symlink d4/l42 0 2024-04-22T15:18:38.752 INFO:tasks.workunit.client.0.smithi096.stdout:2/215: dwrite d3/db/df/f35 [0,4194304] 0 2024-04-22T15:18:38.753 INFO:tasks.workunit.client.0.smithi096.stdout:2/216: chown d3/d1f/d2f 372458 1 2024-04-22T15:18:38.753 INFO:tasks.workunit.client.0.smithi096.stdout:1/286: creat d6/dc/d10/d29/f5c x:0 0 0 2024-04-22T15:18:38.755 INFO:tasks.workunit.client.0.smithi096.stdout:1/287: chown d6/dc/d10/d3f/d54/d32/f37 1 1 2024-04-22T15:18:38.758 INFO:tasks.workunit.client.0.smithi096.stdout:4/189: symlink d5/d7/d2c/d43/l45 0 2024-04-22T15:18:38.760 INFO:tasks.workunit.client.0.smithi096.stdout:0/241: dwrite d7/d9/f40 [0,4194304] 0 2024-04-22T15:18:38.760 INFO:tasks.workunit.client.0.smithi096.stdout:4/190: readlink d5/l34 0 2024-04-22T15:18:38.762 INFO:tasks.workunit.client.0.smithi096.stdout:2/217: symlink d3/d1f/d2f/l43 0 2024-04-22T15:18:38.763 INFO:tasks.workunit.client.0.smithi096.stdout:1/288: truncate d6/f14 478740 0 2024-04-22T15:18:38.768 INFO:tasks.workunit.client.0.smithi096.stdout:4/191: truncate d5/d7/d1a/f1d 2838696 0 2024-04-22T15:18:38.769 INFO:tasks.workunit.client.0.smithi096.stdout:2/218: write d3/db/df/d10/f11 [2614254,112383] 0 2024-04-22T15:18:38.770 INFO:tasks.workunit.client.0.smithi096.stdout:2/219: chown d3/d13 15446 1 2024-04-22T15:18:38.771 INFO:tasks.workunit.client.0.smithi096.stdout:5/198: creat d4/f43 x:0 0 0 2024-04-22T15:18:38.772 INFO:tasks.workunit.client.0.smithi096.stdout:1/289: dwrite d6/dc/d10/d3f/d54/d32/f35 [0,4194304] 0 2024-04-22T15:18:38.774 INFO:tasks.workunit.client.0.smithi096.stdout:1/290: stat d6/dc/d11/d40 0 2024-04-22T15:18:38.778 INFO:tasks.workunit.client.0.smithi096.stdout:2/220: unlink d3/db/df/d24/f38 0 2024-04-22T15:18:38.779 INFO:tasks.workunit.client.0.smithi096.stdout:2/221: chown d3/db/df/d10 938791 1 2024-04-22T15:18:38.781 INFO:tasks.workunit.client.0.smithi096.stdout:5/199: dwrite d4/d1f/f21 [0,4194304] 0 2024-04-22T15:18:38.785 INFO:tasks.workunit.client.0.smithi096.stdout:2/222: dread d3/f8 [0,4194304] 0 2024-04-22T15:18:38.787 INFO:tasks.workunit.client.0.smithi096.stdout:5/200: mknod d4/d2c/c44 0 2024-04-22T15:18:38.792 INFO:tasks.workunit.client.0.smithi096.stdout:2/223: creat d3/db/df/d10/f44 x:0 0 0 2024-04-22T15:18:38.794 INFO:tasks.workunit.client.0.smithi096.stdout:2/224: fsync d3/db/df/d10/f11 0 2024-04-22T15:18:38.795 INFO:tasks.workunit.client.0.smithi096.stdout:5/201: dwrite d4/f7 [0,4194304] 0 2024-04-22T15:18:38.797 INFO:tasks.workunit.client.0.smithi096.stdout:2/225: write d3/db/d30/f39 [1008846,88790] 0 2024-04-22T15:18:38.800 INFO:tasks.workunit.client.0.smithi096.stdout:2/226: mknod d3/d13/c45 0 2024-04-22T15:18:38.800 INFO:tasks.workunit.client.0.smithi096.stdout:4/192: write d5/fc [1554386,38078] 0 2024-04-22T15:18:38.802 INFO:tasks.workunit.client.0.smithi096.stdout:2/227: unlink d3/c6 0 2024-04-22T15:18:38.808 INFO:tasks.workunit.client.0.smithi096.stdout:5/202: link d4/d11/d3d/c37 d4/d11/d3d/d35/c45 0 2024-04-22T15:18:38.810 INFO:tasks.workunit.client.0.smithi096.stdout:4/193: mknod d5/d7/d1a/c46 0 2024-04-22T15:18:38.811 INFO:tasks.workunit.client.0.smithi096.stdout:1/291: truncate d6/dc/d10/d3f/d54/f43 125094 0 2024-04-22T15:18:38.812 INFO:tasks.workunit.client.0.smithi096.stdout:5/203: mknod d4/d11/d3d/d35/c46 0 2024-04-22T15:18:38.814 INFO:tasks.workunit.client.0.smithi096.stdout:2/228: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:18:38.818 INFO:tasks.workunit.client.0.smithi096.stdout:4/194: dread d5/d7/f41 [0,4194304] 0 2024-04-22T15:18:38.818 INFO:tasks.workunit.client.0.smithi096.stdout:1/292: unlink d6/dc/d10/d3f/f49 0 2024-04-22T15:18:38.820 INFO:tasks.workunit.client.0.smithi096.stdout:1/293: chown d6/dc/d10/d3f/d54/d32/d3b 5853089 1 2024-04-22T15:18:38.820 INFO:tasks.workunit.client.0.smithi096.stdout:5/204: dwrite d4/ff [0,4194304] 0 2024-04-22T15:18:38.821 INFO:tasks.workunit.client.0.smithi096.stdout:5/205: chown d4/f3c 10953 1 2024-04-22T15:18:38.831 INFO:tasks.workunit.client.0.smithi096.stdout:1/294: symlink d6/dc/l5d 0 2024-04-22T15:18:38.834 INFO:tasks.workunit.client.0.smithi096.stdout:4/195: creat d5/d7/f47 x:0 0 0 2024-04-22T15:18:38.835 INFO:tasks.workunit.client.0.smithi096.stdout:4/196: dread - d5/d7/d1a/f23 zero size 2024-04-22T15:18:38.836 INFO:tasks.workunit.client.0.smithi096.stdout:4/197: chown d5/d7/d1a/d25/f3f 40497345 1 2024-04-22T15:18:38.839 INFO:tasks.workunit.client.0.smithi096.stdout:1/295: creat d6/dc/d10/d3f/d54/d32/d3b/d56/f5e x:0 0 0 2024-04-22T15:18:38.841 INFO:tasks.workunit.client.0.smithi096.stdout:5/206: link d4/d2c/c44 d4/d12/d23/c47 0 2024-04-22T15:18:38.843 INFO:tasks.workunit.client.0.smithi096.stdout:1/296: chown d6/c3c 133862760 1 2024-04-22T15:18:38.846 INFO:tasks.workunit.client.0.smithi096.stdout:4/198: symlink d5/d7/d1a/l48 0 2024-04-22T15:18:38.847 INFO:tasks.workunit.client.0.smithi096.stdout:1/297: unlink d6/dc/le 0 2024-04-22T15:18:38.851 INFO:tasks.workunit.client.0.smithi096.stdout:4/199: dread d5/fe [0,4194304] 0 2024-04-22T15:18:38.859 INFO:tasks.workunit.client.0.smithi096.stdout:4/200: creat d5/d7/d2c/d3d/f49 x:0 0 0 2024-04-22T15:18:38.861 INFO:tasks.workunit.client.0.smithi096.stdout:1/298: write d6/dc/d10/d3f/d54/f43 [1153759,87889] 0 2024-04-22T15:18:38.863 INFO:tasks.workunit.client.0.smithi096.stdout:1/299: read - d6/dc/f2e zero size 2024-04-22T15:18:38.864 INFO:tasks.workunit.client.0.smithi096.stdout:4/201: dread f3 [0,4194304] 0 2024-04-22T15:18:38.866 INFO:tasks.workunit.client.0.smithi096.stdout:1/300: write d6/dc/f41 [3431502,100170] 0 2024-04-22T15:18:38.867 INFO:tasks.workunit.client.0.smithi096.stdout:4/202: write d5/d7/f20 [4943969,5443] 0 2024-04-22T15:18:38.867 INFO:tasks.workunit.client.0.smithi096.stdout:5/207: dwrite d4/d12/f13 [4194304,4194304] 0 2024-04-22T15:18:38.872 INFO:tasks.workunit.client.0.smithi096.stdout:5/208: rmdir d4/d2c/d2f 39 2024-04-22T15:18:38.878 INFO:tasks.workunit.client.0.smithi096.stdout:1/301: creat d6/dc/f5f x:0 0 0 2024-04-22T15:18:38.879 INFO:tasks.workunit.client.0.smithi096.stdout:1/302: read d6/dc/d11/f3e [710715,120010] 0 2024-04-22T15:18:38.880 INFO:tasks.workunit.client.0.smithi096.stdout:5/209: rmdir d4/d2c/d2f 39 2024-04-22T15:18:38.881 INFO:tasks.workunit.client.0.smithi096.stdout:5/210: write d4/fd [3198717,113138] 0 2024-04-22T15:18:38.881 INFO:tasks.workunit.client.0.smithi096.stdout:4/203: write d5/d7/f24 [1422148,6134] 0 2024-04-22T15:18:38.887 INFO:tasks.workunit.client.0.smithi096.stdout:4/204: link d5/f13 d5/d7/d2c/f4a 0 2024-04-22T15:18:38.890 INFO:tasks.workunit.client.0.smithi096.stdout:1/303: link d6/dc/d10/d3f/d54/f21 d6/f60 0 2024-04-22T15:18:38.890 INFO:tasks.workunit.client.0.smithi096.stdout:1/304: stat d6/dc/d10 0 2024-04-22T15:18:38.893 INFO:tasks.workunit.client.0.smithi096.stdout:4/205: rename d5/d7/d1a/c26 to d5/d7/d2c/d43/c4b 0 2024-04-22T15:18:38.893 INFO:tasks.workunit.client.0.smithi096.stdout:1/305: symlink d6/dc/d10/d3f/d54/d32/d3b/d3d/l61 0 2024-04-22T15:18:38.894 INFO:tasks.workunit.client.0.smithi096.stdout:1/306: fsync d6/dc/d10/d29/f2f 0 2024-04-22T15:18:38.895 INFO:tasks.workunit.client.0.smithi096.stdout:1/307: chown d6/dc/d10/d3f/d54 231194752 1 2024-04-22T15:18:38.898 INFO:tasks.workunit.client.0.smithi096.stdout:1/308: mkdir d6/dc/d10/d3f/d62 0 2024-04-22T15:18:38.899 INFO:tasks.workunit.client.0.smithi096.stdout:1/309: write d6/dc/d11/f31 [334394,63595] 0 2024-04-22T15:18:38.901 INFO:tasks.workunit.client.0.smithi096.stdout:1/310: write d6/dc/f2e [780786,104890] 0 2024-04-22T15:18:38.904 INFO:tasks.workunit.client.0.smithi096.stdout:1/311: creat d6/dc/d10/d3f/d54/d32/d3b/d3d/f63 x:0 0 0 2024-04-22T15:18:38.905 INFO:tasks.workunit.client.0.smithi096.stdout:5/211: truncate d4/d2c/d2f/d31/f3b 2140210 0 2024-04-22T15:18:38.907 INFO:tasks.workunit.client.0.smithi096.stdout:1/312: truncate d6/dc/d10/d3f/d54/d32/d3b/d3d/f63 527153 0 2024-04-22T15:18:38.907 INFO:tasks.workunit.client.0.smithi096.stdout:4/206: getdents d5/d7/d2c/d43 0 2024-04-22T15:18:38.907 INFO:tasks.workunit.client.0.smithi096.stdout:4/207: chown d5/d7/d1f 1227 1 2024-04-22T15:18:38.911 INFO:tasks.workunit.client.0.smithi096.stdout:4/208: symlink d5/d7/d2c/l4c 0 2024-04-22T15:18:38.912 INFO:tasks.workunit.client.0.smithi096.stdout:1/313: symlink d6/l64 0 2024-04-22T15:18:38.913 INFO:tasks.workunit.client.0.smithi096.stdout:1/314: write d6/dc/d10/d29/f2f [294923,55234] 0 2024-04-22T15:18:38.913 INFO:tasks.workunit.client.0.smithi096.stdout:5/212: getdents d4/d12 0 2024-04-22T15:18:38.916 INFO:tasks.workunit.client.0.smithi096.stdout:5/213: write d4/f5 [1475698,247] 0 2024-04-22T15:18:38.916 INFO:tasks.workunit.client.0.smithi096.stdout:1/315: mknod d6/dc/d10/d3f/d54/d32/d3b/d56/c65 0 2024-04-22T15:18:38.918 INFO:tasks.workunit.client.0.smithi096.stdout:1/316: chown d6/dc/d10/d3f/d54/c5a 926977547 1 2024-04-22T15:18:38.921 INFO:tasks.workunit.client.0.smithi096.stdout:5/214: dread d4/f5 [0,4194304] 0 2024-04-22T15:18:38.927 INFO:tasks.workunit.client.0.smithi096.stdout:4/209: dread d5/d7/d2c/f4a [0,4194304] 0 2024-04-22T15:18:38.951 INFO:tasks.workunit.client.0.smithi096.stdout:1/317: rename d6/dc/d10/d3f/d54/d32/d3b/d56/f5e to d6/dc/d11/d45/f66 0 2024-04-22T15:18:38.951 INFO:tasks.workunit.client.0.smithi096.stdout:5/215: rmdir d4/d11/d25 39 2024-04-22T15:18:38.957 INFO:tasks.workunit.client.0.smithi096.stdout:5/216: dwrite d4/f3c [0,4194304] 0 2024-04-22T15:18:38.958 INFO:tasks.workunit.client.0.smithi096.stdout:4/210: link f2 d5/d7/d1a/d25/f4d 0 2024-04-22T15:18:38.958 INFO:tasks.workunit.client.0.smithi096.stdout:5/217: chown d4/c2e 99 1 2024-04-22T15:18:38.959 INFO:tasks.workunit.client.0.smithi096.stdout:5/218: read d4/d2c/d2f/f3a [83115,22573] 0 2024-04-22T15:18:38.960 INFO:tasks.workunit.client.0.smithi096.stdout:1/318: creat d6/dc/d10/f67 x:0 0 0 2024-04-22T15:18:38.962 INFO:tasks.workunit.client.0.smithi096.stdout:5/219: stat d4/d2c/c44 0 2024-04-22T15:18:38.970 INFO:tasks.workunit.client.0.smithi096.stdout:1/319: symlink d6/dc/d10/d3f/d62/l68 0 2024-04-22T15:18:38.977 INFO:tasks.workunit.client.0.smithi096.stdout:1/320: dwrite d6/f39 [0,4194304] 0 2024-04-22T15:18:38.978 INFO:tasks.workunit.client.0.smithi096.stdout:4/211: truncate d5/fb 977790 0 2024-04-22T15:18:38.980 INFO:tasks.workunit.client.0.smithi096.stdout:1/321: symlink d6/dc/d10/d3f/d54/d32/d3b/d56/l69 0 2024-04-22T15:18:38.983 INFO:tasks.workunit.client.0.smithi096.stdout:5/220: write f2 [1891004,8113] 0 2024-04-22T15:18:38.985 INFO:tasks.workunit.client.0.smithi096.stdout:1/322: mkdir d6/dc/d11/d45/d6a 0 2024-04-22T15:18:38.989 INFO:tasks.workunit.client.0.smithi096.stdout:1/323: dread d6/dc/d11/d45/f50 [0,4194304] 0 2024-04-22T15:18:38.991 INFO:tasks.workunit.client.0.smithi096.stdout:5/221: rename d4/ff to d4/d11/d25/f48 0 2024-04-22T15:18:38.991 INFO:tasks.workunit.client.0.smithi096.stdout:5/222: readlink d4/l16 0 2024-04-22T15:18:38.994 INFO:tasks.workunit.client.0.smithi096.stdout:4/212: getdents d5/d7/d1a/d25 0 2024-04-22T15:18:38.999 INFO:tasks.workunit.client.0.smithi096.stdout:1/324: mknod d6/dc/d11/c6b 0 2024-04-22T15:18:39.000 INFO:tasks.workunit.client.0.smithi096.stdout:4/213: dwrite d5/fc [4194304,4194304] 0 2024-04-22T15:18:39.009 INFO:tasks.workunit.client.0.smithi096.stdout:5/223: rename d4/l42 to d4/l49 0 2024-04-22T15:18:39.010 INFO:tasks.workunit.client.0.smithi096.stdout:1/325: dread d6/f4f [0,4194304] 0 2024-04-22T15:18:39.014 INFO:tasks.workunit.client.0.smithi096.stdout:1/326: rmdir d6/dc/d10/d3f/d62 39 2024-04-22T15:18:39.021 INFO:tasks.workunit.client.0.smithi096.stdout:1/327: dread d6/f14 [0,4194304] 0 2024-04-22T15:18:39.028 INFO:tasks.workunit.client.0.smithi096.stdout:1/328: rename d6/dc/d10/d3f/d54/f4c to d6/dc/f6c 0 2024-04-22T15:18:39.030 INFO:tasks.workunit.client.0.smithi096.stdout:5/224: dwrite d4/d12/f13 [0,4194304] 0 2024-04-22T15:18:39.031 INFO:tasks.workunit.client.0.smithi096.stdout:1/329: unlink d6/dc/f2e 0 2024-04-22T15:18:39.032 INFO:tasks.workunit.client.0.smithi096.stdout:5/225: write d4/d12/f15 [201579,50623] 0 2024-04-22T15:18:39.032 INFO:tasks.workunit.client.0.smithi096.stdout:1/330: read - d6/dc/d10/f67 zero size 2024-04-22T15:18:39.042 INFO:tasks.workunit.client.0.smithi096.stdout:5/226: dwrite d4/f1c [0,4194304] 0 2024-04-22T15:18:39.044 INFO:tasks.workunit.client.0.smithi096.stdout:1/331: rename d6/dc/d11/f3e to d6/dc/d10/d3f/d54/d32/f6d 0 2024-04-22T15:18:39.046 INFO:tasks.workunit.client.0.smithi096.stdout:1/332: write d6/dc/d11/f31 [779550,99994] 0 2024-04-22T15:18:39.047 INFO:tasks.workunit.client.0.smithi096.stdout:5/227: rmdir d4/d11/d3d 39 2024-04-22T15:18:39.056 INFO:tasks.workunit.client.0.smithi096.stdout:1/333: creat d6/dc/d10/d3f/d54/d32/d3b/d56/f6e x:0 0 0 2024-04-22T15:18:39.059 INFO:tasks.workunit.client.0.smithi096.stdout:1/334: read d6/dc/d10/d3f/d54/f21 [99597,74869] 0 2024-04-22T15:18:39.075 INFO:tasks.workunit.client.0.smithi096.stdout:5/228: dread d4/fa [0,4194304] 0 2024-04-22T15:18:39.077 INFO:tasks.workunit.client.0.smithi096.stdout:5/229: dread d4/d12/d23/f27 [0,4194304] 0 2024-04-22T15:18:39.081 INFO:tasks.workunit.client.0.smithi096.stdout:5/230: mknod d4/d11/d3d/c4a 0 2024-04-22T15:18:39.084 INFO:tasks.workunit.client.0.smithi096.stdout:5/231: symlink d4/d11/d3d/d35/l4b 0 2024-04-22T15:18:39.089 INFO:tasks.workunit.client.0.smithi096.stdout:5/232: dread d4/fa [0,4194304] 0 2024-04-22T15:18:39.096 INFO:tasks.workunit.client.0.smithi096.stdout:1/335: write d6/f14 [907335,119293] 0 2024-04-22T15:18:39.098 INFO:tasks.workunit.client.0.smithi096.stdout:1/336: read d6/dc/d10/d29/f2f [1737685,79360] 0 2024-04-22T15:18:39.099 INFO:tasks.workunit.client.0.smithi096.stdout:5/233: link d4/d11/d3d/d35/c45 d4/d1f/c4c 0 2024-04-22T15:18:39.099 INFO:tasks.workunit.client.0.smithi096.stdout:1/337: dread - d6/dc/d11/d45/f66 zero size 2024-04-22T15:18:39.107 INFO:tasks.workunit.client.0.smithi096.stdout:1/338: rename d6/dc/d11/d4a/f5b to d6/dc/d10/d3f/d54/d32/d3b/f6f 0 2024-04-22T15:18:39.112 INFO:tasks.workunit.client.0.smithi096.stdout:1/339: dwrite d6/dc/d11/f31 [0,4194304] 0 2024-04-22T15:18:39.114 INFO:tasks.workunit.client.0.smithi096.stdout:1/340: dread - d6/dc/d10/d3f/d54/d32/d3b/d56/f6e zero size 2024-04-22T15:18:39.133 INFO:tasks.workunit.client.0.smithi096.stdout:7/253: sync 2024-04-22T15:18:39.137 INFO:tasks.workunit.client.0.smithi096.stdout:7/254: rename d31/d32/d33/d3b to d31/d32/d33/d54 0 2024-04-22T15:18:39.138 INFO:tasks.workunit.client.0.smithi096.stdout:7/255: fdatasync d31/d32/d36/f37 0 2024-04-22T15:18:39.141 INFO:tasks.workunit.client.0.smithi096.stdout:5/234: truncate d4/d11/f22 166490 0 2024-04-22T15:18:39.145 INFO:tasks.workunit.client.0.smithi096.stdout:5/235: mkdir d4/d11/d3d/d4d 0 2024-04-22T15:18:39.147 INFO:tasks.workunit.client.0.smithi096.stdout:5/236: fsync d4/f6 0 2024-04-22T15:18:39.152 INFO:tasks.workunit.client.0.smithi096.stdout:1/341: write d6/dc/f23 [3775089,63049] 0 2024-04-22T15:18:39.154 INFO:tasks.workunit.client.0.smithi096.stdout:1/342: write d6/dc/d11/d45/f66 [892741,6198] 0 2024-04-22T15:18:39.155 INFO:tasks.workunit.client.0.smithi096.stdout:1/343: write d6/f22 [1458504,111117] 0 2024-04-22T15:18:39.158 INFO:tasks.workunit.client.0.smithi096.stdout:1/344: chown d6/dc/d10/d3f/d62/l68 226 1 2024-04-22T15:18:39.164 INFO:tasks.workunit.client.0.smithi096.stdout:1/345: dwrite d6/dc/d10/d3f/d54/d32/d3b/f6f [0,4194304] 0 2024-04-22T15:18:39.166 INFO:tasks.workunit.client.0.smithi096.stdout:1/346: dread - d6/dc/d10/d3f/d54/d32/d3b/d56/f6e zero size 2024-04-22T15:18:39.167 INFO:tasks.workunit.client.0.smithi096.stdout:1/347: dread - d6/dc/d10/d29/f59 zero size 2024-04-22T15:18:39.169 INFO:tasks.workunit.client.0.smithi096.stdout:1/348: creat d6/dc/d10/d3f/d62/f70 x:0 0 0 2024-04-22T15:18:39.170 INFO:tasks.workunit.client.0.smithi096.stdout:7/256: write d31/d32/d33/f34 [277074,36070] 0 2024-04-22T15:18:39.172 INFO:tasks.workunit.client.0.smithi096.stdout:1/349: fdatasync d6/dc/d11/d4a/f4d 0 2024-04-22T15:18:39.177 INFO:tasks.workunit.client.0.smithi096.stdout:7/257: link d31/d32/f40 d31/d32/f55 0 2024-04-22T15:18:39.178 INFO:tasks.workunit.client.0.smithi096.stdout:1/350: mknod d6/dc/c71 0 2024-04-22T15:18:39.179 INFO:tasks.workunit.client.0.smithi096.stdout:1/351: fsync d6/dc/d10/d3f/f57 0 2024-04-22T15:18:39.180 INFO:tasks.workunit.client.0.smithi096.stdout:1/352: chown d6/fd 27927066 1 2024-04-22T15:18:39.181 INFO:tasks.workunit.client.0.smithi096.stdout:1/353: write d6/dc/d11/d45/f58 [563569,53936] 0 2024-04-22T15:18:39.182 INFO:tasks.workunit.client.0.smithi096.stdout:1/354: fdatasync d6/f22 0 2024-04-22T15:18:39.185 INFO:tasks.workunit.client.0.smithi096.stdout:7/258: rename c9 to d31/c56 0 2024-04-22T15:18:39.188 INFO:tasks.workunit.client.0.smithi096.stdout:5/237: write d4/d11/d25/f38 [1020418,123950] 0 2024-04-22T15:18:39.191 INFO:tasks.workunit.client.0.smithi096.stdout:5/238: fdatasync d4/f5 0 2024-04-22T15:18:39.192 INFO:tasks.workunit.client.0.smithi096.stdout:1/355: dwrite d6/dc/d10/d3f/d54/d32/d3b/d56/f44 [0,4194304] 0 2024-04-22T15:18:39.194 INFO:tasks.workunit.client.0.smithi096.stdout:5/239: chown d4/d12/d23/l2b 128617109 1 2024-04-22T15:18:39.195 INFO:tasks.workunit.client.0.smithi096.stdout:7/259: dwrite d31/d32/d36/f45 [0,4194304] 0 2024-04-22T15:18:39.202 INFO:tasks.workunit.client.0.smithi096.stdout:7/260: fsync f18 0 2024-04-22T15:18:39.204 INFO:tasks.workunit.client.0.smithi096.stdout:7/261: truncate d31/d32/d33/d54/d43/f48 477436 0 2024-04-22T15:18:39.206 INFO:tasks.workunit.client.0.smithi096.stdout:5/240: creat d4/d2c/f4e x:0 0 0 2024-04-22T15:18:39.207 INFO:tasks.workunit.client.0.smithi096.stdout:5/241: write f2 [4693022,85667] 0 2024-04-22T15:18:39.208 INFO:tasks.workunit.client.0.smithi096.stdout:5/242: write d4/f26 [3157632,21847] 0 2024-04-22T15:18:39.211 INFO:tasks.workunit.client.0.smithi096.stdout:5/243: mkdir d4/d11/d3d/d4f 0 2024-04-22T15:18:39.214 INFO:tasks.workunit.client.0.smithi096.stdout:5/244: rename d4/d12/d23/f27 to d4/d12/d23/f50 0 2024-04-22T15:18:39.215 INFO:tasks.workunit.client.0.smithi096.stdout:5/245: readlink d4/d1f/l20 0 2024-04-22T15:18:39.216 INFO:tasks.workunit.client.0.smithi096.stdout:5/246: write d4/d12/f13 [4216120,18261] 0 2024-04-22T15:18:39.217 INFO:tasks.workunit.client.0.smithi096.stdout:5/247: stat d4/d2c/d2f/d31/f3b 0 2024-04-22T15:18:39.220 INFO:tasks.workunit.client.0.smithi096.stdout:5/248: creat d4/d2c/d2f/f51 x:0 0 0 2024-04-22T15:18:39.222 INFO:tasks.workunit.client.0.smithi096.stdout:5/249: write d4/d11/d25/f48 [4350516,97234] 0 2024-04-22T15:18:39.225 INFO:tasks.workunit.client.0.smithi096.stdout:7/262: dread d31/d32/f55 [0,4194304] 0 2024-04-22T15:18:39.225 INFO:tasks.workunit.client.0.smithi096.stdout:1/356: truncate d6/dc/d11/d45/f66 792147 0 2024-04-22T15:18:39.225 INFO:tasks.workunit.client.0.smithi096.stdout:5/250: rmdir d4/d2c/d2f/d31 39 2024-04-22T15:18:39.228 INFO:tasks.workunit.client.0.smithi096.stdout:1/357: rmdir d6/dc/d11/d45 39 2024-04-22T15:18:39.229 INFO:tasks.workunit.client.0.smithi096.stdout:1/358: readlink d6/dc/d10/d3f/d54/d32/d3b/d3d/l61 0 2024-04-22T15:18:39.229 INFO:tasks.workunit.client.0.smithi096.stdout:1/359: readlink d6/l9 0 2024-04-22T15:18:39.230 INFO:tasks.workunit.client.0.smithi096.stdout:7/263: creat d31/d32/d33/f57 x:0 0 0 2024-04-22T15:18:39.233 INFO:tasks.workunit.client.0.smithi096.stdout:1/360: creat d6/dc/d10/d3f/d54/d32/d3b/d56/f72 x:0 0 0 2024-04-22T15:18:39.234 INFO:tasks.workunit.client.0.smithi096.stdout:7/264: creat d31/d32/d33/f58 x:0 0 0 2024-04-22T15:18:39.236 INFO:tasks.workunit.client.0.smithi096.stdout:5/251: rename d4/d11/d3d/d35/l4b to d4/d2c/d2f/l52 0 2024-04-22T15:18:39.238 INFO:tasks.workunit.client.0.smithi096.stdout:7/265: readlink l1b 0 2024-04-22T15:18:39.240 INFO:tasks.workunit.client.0.smithi096.stdout:1/361: creat d6/dc/d10/d3f/f73 x:0 0 0 2024-04-22T15:18:39.242 INFO:tasks.workunit.client.0.smithi096.stdout:5/252: dwrite d4/f5 [0,4194304] 0 2024-04-22T15:18:39.246 INFO:tasks.workunit.client.0.smithi096.stdout:5/253: mknod d4/d11/d3d/c53 0 2024-04-22T15:18:39.249 INFO:tasks.workunit.client.0.smithi096.stdout:7/266: dwrite fb [4194304,4194304] 0 2024-04-22T15:18:39.252 INFO:tasks.workunit.client.0.smithi096.stdout:7/267: chown d31/d32/d33/d54/d43 763 1 2024-04-22T15:18:39.252 INFO:tasks.workunit.client.0.smithi096.stdout:1/362: dwrite d6/dc/d10/d3f/d54/d32/f35 [0,4194304] 0 2024-04-22T15:18:39.252 INFO:tasks.workunit.client.0.smithi096.stdout:5/254: dread d4/d11/f22 [0,4194304] 0 2024-04-22T15:18:39.254 INFO:tasks.workunit.client.0.smithi096.stdout:5/255: write d4/fd [5112232,16054] 0 2024-04-22T15:18:39.256 INFO:tasks.workunit.client.0.smithi096.stdout:7/268: creat d31/d32/f59 x:0 0 0 2024-04-22T15:18:39.261 INFO:tasks.workunit.client.0.smithi096.stdout:5/256: unlink d4/fa 0 2024-04-22T15:18:39.264 INFO:tasks.workunit.client.0.smithi096.stdout:5/257: dread d4/d12/f15 [0,4194304] 0 2024-04-22T15:18:39.267 INFO:tasks.workunit.client.0.smithi096.stdout:1/363: mkdir d6/dc/d11/d45/d6a/d74 0 2024-04-22T15:18:39.269 INFO:tasks.workunit.client.0.smithi096.stdout:5/258: stat d4/d1f/c4c 0 2024-04-22T15:18:39.271 INFO:tasks.workunit.client.0.smithi096.stdout:1/364: dread d6/dc/d11/d45/f58 [0,4194304] 0 2024-04-22T15:18:39.275 INFO:tasks.workunit.client.0.smithi096.stdout:7/269: dread fa [0,4194304] 0 2024-04-22T15:18:39.276 INFO:tasks.workunit.client.0.smithi096.stdout:1/365: rename d6/dc/d10/d3f/d54/d32/d3b/d56/c65 to d6/dc/d10/d3f/d54/d32/d3b/d3d/d4e/c75 0 2024-04-22T15:18:39.278 INFO:tasks.workunit.client.0.smithi096.stdout:7/270: rmdir d31/d32/d33 39 2024-04-22T15:18:39.280 INFO:tasks.workunit.client.0.smithi096.stdout:7/271: write d31/d32/d36/f3c [967225,9788] 0 2024-04-22T15:18:39.280 INFO:tasks.workunit.client.0.smithi096.stdout:1/366: creat d6/dc/d10/d3f/d62/f76 x:0 0 0 2024-04-22T15:18:39.280 INFO:tasks.workunit.client.0.smithi096.stdout:7/272: fsync f7 0 2024-04-22T15:18:39.282 INFO:tasks.workunit.client.0.smithi096.stdout:1/367: write d6/dc/d10/d3f/d54/d32/f35 [2467880,59844] 0 2024-04-22T15:18:39.283 INFO:tasks.workunit.client.0.smithi096.stdout:1/368: write d6/f22 [792396,62092] 0 2024-04-22T15:18:39.285 INFO:tasks.workunit.client.0.smithi096.stdout:7/273: creat d31/d32/d33/f5a x:0 0 0 2024-04-22T15:18:39.337 INFO:tasks.workunit.client.0.smithi096.stdout:8/189: sync 2024-04-22T15:18:39.337 INFO:tasks.workunit.client.0.smithi096.stdout:8/190: chown d4/d10/d25/d38 258127987 1 2024-04-22T15:18:39.342 INFO:tasks.workunit.client.0.smithi096.stdout:7/274: dread d31/d32/d33/f39 [0,4194304] 0 2024-04-22T15:18:39.343 INFO:tasks.workunit.client.0.smithi096.stdout:7/275: fdatasync d31/d32/f59 0 2024-04-22T15:18:39.343 INFO:tasks.workunit.client.0.smithi096.stdout:7/276: readlink l2 0 2024-04-22T15:18:39.343 INFO:tasks.workunit.client.0.smithi096.stdout:8/191: rename f3 to d4/d10/d25/f44 0 2024-04-22T15:18:39.344 INFO:tasks.workunit.client.0.smithi096.stdout:7/277: chown l1b 4 1 2024-04-22T15:18:39.344 INFO:tasks.workunit.client.0.smithi096.stdout:8/192: fdatasync d4/fd 0 2024-04-22T15:18:39.345 INFO:tasks.workunit.client.0.smithi096.stdout:8/193: write d4/d2a/f36 [975008,129614] 0 2024-04-22T15:18:39.350 INFO:tasks.workunit.client.0.smithi096.stdout:7/278: rename d31/l42 to d31/d32/d33/l5b 0 2024-04-22T15:18:39.351 INFO:tasks.workunit.client.0.smithi096.stdout:7/279: write f1c [4295335,14833] 0 2024-04-22T15:18:39.351 INFO:tasks.workunit.client.0.smithi096.stdout:8/194: readlink d4/d1c/l1e 0 2024-04-22T15:18:39.352 INFO:tasks.workunit.client.0.smithi096.stdout:7/280: write d31/d32/d33/d54/f51 [912959,47954] 0 2024-04-22T15:18:39.353 INFO:tasks.workunit.client.0.smithi096.stdout:7/281: chown d31/d32/d33/d54/f51 3 1 2024-04-22T15:18:39.354 INFO:tasks.workunit.client.0.smithi096.stdout:7/282: readlink d31/d32/d33/l4e 0 2024-04-22T15:18:39.354 INFO:tasks.workunit.client.0.smithi096.stdout:7/283: chown l26 45 1 2024-04-22T15:18:39.355 INFO:tasks.workunit.client.0.smithi096.stdout:7/284: dread - f2b zero size 2024-04-22T15:18:39.356 INFO:tasks.workunit.client.0.smithi096.stdout:7/285: write d31/d32/d33/f34 [957981,62811] 0 2024-04-22T15:18:39.356 INFO:tasks.workunit.client.0.smithi096.stdout:7/286: chown f1a 13172 1 2024-04-22T15:18:39.357 INFO:tasks.workunit.client.0.smithi096.stdout:7/287: truncate f23 2110834 0 2024-04-22T15:18:39.357 INFO:tasks.workunit.client.0.smithi096.stdout:1/369: link d6/dc/d10/l28 d6/l77 0 2024-04-22T15:18:39.358 INFO:tasks.workunit.client.0.smithi096.stdout:7/288: write d31/f3d [272960,74190] 0 2024-04-22T15:18:39.359 INFO:tasks.workunit.client.0.smithi096.stdout:1/370: fsync d6/dc/d10/d3f/d54/d32/d3b/d3d/f63 0 2024-04-22T15:18:39.361 INFO:tasks.workunit.client.0.smithi096.stdout:8/195: mkdir d4/d10/d25/d38/d45 0 2024-04-22T15:18:39.361 INFO:tasks.workunit.client.0.smithi096.stdout:8/196: fdatasync d4/d2a/f36 0 2024-04-22T15:18:39.362 INFO:tasks.workunit.client.0.smithi096.stdout:7/289: unlink fb 0 2024-04-22T15:18:39.363 INFO:tasks.workunit.client.0.smithi096.stdout:7/290: chown d31/f3d 338099857 1 2024-04-22T15:18:39.366 INFO:tasks.workunit.client.0.smithi096.stdout:1/371: creat d6/dc/d11/d4a/f78 x:0 0 0 2024-04-22T15:18:39.368 INFO:tasks.workunit.client.0.smithi096.stdout:8/197: write d4/d10/d25/f44 [3887587,110354] 0 2024-04-22T15:18:39.368 INFO:tasks.workunit.client.0.smithi096.stdout:7/291: fdatasync f21 0 2024-04-22T15:18:39.370 INFO:tasks.workunit.client.0.smithi096.stdout:1/372: rename d6/dc/f5f to d6/dc/d10/d29/f79 0 2024-04-22T15:18:39.373 INFO:tasks.workunit.client.0.smithi096.stdout:7/292: creat d31/d32/d33/f5c x:0 0 0 2024-04-22T15:18:39.375 INFO:tasks.workunit.client.0.smithi096.stdout:8/198: mkdir d4/d1c/d40/d46 0 2024-04-22T15:18:39.377 INFO:tasks.workunit.client.0.smithi096.stdout:1/373: creat d6/dc/d10/f7a x:0 0 0 2024-04-22T15:18:39.379 INFO:tasks.workunit.client.0.smithi096.stdout:7/293: dwrite f5 [0,4194304] 0 2024-04-22T15:18:39.384 INFO:tasks.workunit.client.0.smithi096.stdout:7/294: getdents d31/d32/d33/d54 0 2024-04-22T15:18:39.385 INFO:tasks.workunit.client.0.smithi096.stdout:7/295: write d31/d32/d36/f41 [844452,93554] 0 2024-04-22T15:18:39.389 INFO:tasks.workunit.client.0.smithi096.stdout:7/296: mkdir d31/d32/d36/d5d 0 2024-04-22T15:18:39.390 INFO:tasks.workunit.client.0.smithi096.stdout:7/297: write d31/d32/d33/f5a [957697,41206] 0 2024-04-22T15:18:39.395 INFO:tasks.workunit.client.0.smithi096.stdout:7/298: unlink d31/d32/d36/c49 0 2024-04-22T15:18:39.395 INFO:tasks.workunit.client.0.smithi096.stdout:7/299: read d31/d32/f40 [2577689,92567] 0 2024-04-22T15:18:39.398 INFO:tasks.workunit.client.0.smithi096.stdout:7/300: mkdir d31/d32/d33/d5e 0 2024-04-22T15:18:39.399 INFO:tasks.workunit.client.0.smithi096.stdout:7/301: write d31/d32/f4c [4278209,129692] 0 2024-04-22T15:18:39.400 INFO:tasks.workunit.client.0.smithi096.stdout:1/374: rmdir d6/dc/d11/d4a 39 2024-04-22T15:18:39.405 INFO:tasks.workunit.client.0.smithi096.stdout:7/302: dread d31/d32/d36/f41 [0,4194304] 0 2024-04-22T15:18:39.406 INFO:tasks.workunit.client.0.smithi096.stdout:7/303: read d31/f3d [316864,13122] 0 2024-04-22T15:18:39.410 INFO:tasks.workunit.client.0.smithi096.stdout:7/304: symlink d31/d32/d33/d54/d43/l5f 0 2024-04-22T15:18:39.417 INFO:tasks.workunit.client.0.smithi096.stdout:7/305: creat d31/d32/f60 x:0 0 0 2024-04-22T15:18:39.418 INFO:tasks.workunit.client.0.smithi096.stdout:1/375: dread d6/dc/d11/d40/f38 [0,4194304] 0 2024-04-22T15:18:39.423 INFO:tasks.workunit.client.0.smithi096.stdout:1/376: rename d6/dc/d11/d45/d6a/d74 to d6/dc/d11/d45/d7b 0 2024-04-22T15:18:39.424 INFO:tasks.workunit.client.0.smithi096.stdout:1/377: chown d6/dc/d10/d3f/d54/d32/d3b 1207466198 1 2024-04-22T15:18:39.434 INFO:tasks.workunit.client.0.smithi096.stdout:7/306: dwrite d31/d32/f40 [4194304,4194304] 0 2024-04-22T15:18:39.439 INFO:tasks.workunit.client.0.smithi096.stdout:7/307: unlink d31/c56 0 2024-04-22T15:18:39.448 INFO:tasks.workunit.client.0.smithi096.stdout:7/308: dwrite fa [0,4194304] 0 2024-04-22T15:18:39.453 INFO:tasks.workunit.client.0.smithi096.stdout:7/309: getdents d31/d32/d33/d5e 0 2024-04-22T15:18:39.455 INFO:tasks.workunit.client.0.smithi096.stdout:7/310: rename l1 to d31/l61 0 2024-04-22T15:18:39.456 INFO:tasks.workunit.client.0.smithi096.stdout:7/311: write d31/d32/f4c [3799141,130543] 0 2024-04-22T15:18:39.459 INFO:tasks.workunit.client.0.smithi096.stdout:7/312: creat d31/d32/d33/d54/f62 x:0 0 0 2024-04-22T15:18:39.460 INFO:tasks.workunit.client.0.smithi096.stdout:7/313: read fa [1295386,46081] 0 2024-04-22T15:18:39.460 INFO:tasks.workunit.client.0.smithi096.stdout:7/314: stat d31/d32/c3a 0 2024-04-22T15:18:39.461 INFO:tasks.workunit.client.0.smithi096.stdout:7/315: write f5 [197368,29774] 0 2024-04-22T15:18:39.477 INFO:tasks.workunit.client.0.smithi096.stdout:7/316: write d31/f3d [602995,38148] 0 2024-04-22T15:18:39.483 INFO:tasks.workunit.client.0.smithi096.stdout:7/317: dwrite d31/d32/d33/f58 [0,4194304] 0 2024-04-22T15:18:39.483 INFO:tasks.workunit.client.0.smithi096.stdout:7/318: chown d31 64 1 2024-04-22T15:18:39.491 INFO:tasks.workunit.client.0.smithi096.stdout:7/319: dwrite d31/d32/f3f [0,4194304] 0 2024-04-22T15:18:39.492 INFO:tasks.workunit.client.0.smithi096.stdout:7/320: chown d31/d32/d33/f3e 1043872 1 2024-04-22T15:18:39.495 INFO:tasks.workunit.client.0.smithi096.stdout:7/321: rename f1a to d31/d32/d33/d5e/f63 0 2024-04-22T15:18:39.498 INFO:tasks.workunit.client.0.smithi096.stdout:7/322: rename d31/f3d to d31/d32/d33/d54/f64 0 2024-04-22T15:18:39.500 INFO:tasks.workunit.client.0.smithi096.stdout:7/323: symlink d31/d32/d33/d54/l65 0 2024-04-22T15:18:39.501 INFO:tasks.workunit.client.0.smithi096.stdout:7/324: rmdir d31/d32/d33/d44 39 2024-04-22T15:18:39.503 INFO:tasks.workunit.client.0.smithi096.stdout:7/325: getdents d31/d32/d36/d5d 0 2024-04-22T15:18:39.506 INFO:tasks.workunit.client.0.smithi096.stdout:3/139: sync 2024-04-22T15:18:39.521 INFO:tasks.workunit.client.0.smithi096.stdout:3/140: dread da/db/dc/f16 [0,4194304] 0 2024-04-22T15:18:39.521 INFO:tasks.workunit.client.0.smithi096.stdout:3/141: fdatasync da/f27 0 2024-04-22T15:18:39.559 INFO:tasks.workunit.client.0.smithi096.stdout:3/142: getdents da/db/dc/d1f 0 2024-04-22T15:18:39.561 INFO:tasks.workunit.client.0.smithi096.stdout:3/143: creat da/db/de/d19/f30 x:0 0 0 2024-04-22T15:18:39.570 INFO:tasks.workunit.client.0.smithi096.stdout:3/144: dread da/db/dc/d1f/f15 [0,4194304] 0 2024-04-22T15:18:39.571 INFO:tasks.workunit.client.0.smithi096.stdout:3/145: truncate da/db/de/d19/f2a 870982 0 2024-04-22T15:18:39.572 INFO:tasks.workunit.client.0.smithi096.stdout:3/146: write da/db/de/d19/f2a [764268,42759] 0 2024-04-22T15:18:39.578 INFO:tasks.workunit.client.0.smithi096.stdout:3/147: dwrite da/db/de/d19/f30 [0,4194304] 0 2024-04-22T15:18:39.579 INFO:tasks.workunit.client.0.smithi096.stdout:3/148: chown da/l1b 826556 1 2024-04-22T15:18:39.648 INFO:tasks.workunit.client.0.smithi096.stdout:3/149: chown da/db/d12/c21 22800581 1 2024-04-22T15:18:39.650 INFO:tasks.workunit.client.0.smithi096.stdout:3/150: creat da/db/d12/d22/f31 x:0 0 0 2024-04-22T15:18:39.652 INFO:tasks.workunit.client.0.smithi096.stdout:3/151: mknod da/db/de/d19/c32 0 2024-04-22T15:18:39.655 INFO:tasks.workunit.client.0.smithi096.stdout:3/152: creat da/db/d10/f33 x:0 0 0 2024-04-22T15:18:39.658 INFO:tasks.workunit.client.0.smithi096.stdout:3/153: unlink da/db/d12/l1e 0 2024-04-22T15:18:39.663 INFO:tasks.workunit.client.0.smithi096.stdout:3/154: rename da/db/dc/d1f/f20 to da/f34 0 2024-04-22T15:18:39.664 INFO:tasks.workunit.client.0.smithi096.stdout:3/155: mknod da/db/d10/c35 0 2024-04-22T15:18:39.668 INFO:tasks.workunit.client.0.smithi096.stdout:3/156: link l3 da/db/l36 0 2024-04-22T15:18:39.669 INFO:tasks.workunit.client.0.smithi096.stdout:3/157: chown da/db/de/d19 5768 1 2024-04-22T15:18:39.669 INFO:tasks.workunit.client.0.smithi096.stdout:3/158: fsync da/db/dc/d1f/f29 0 2024-04-22T15:18:39.671 INFO:tasks.workunit.client.0.smithi096.stdout:3/159: mkdir da/db/de/d19/d2d/d37 0 2024-04-22T15:18:39.673 INFO:tasks.workunit.client.0.smithi096.stdout:3/160: write da/db/de/d19/f2a [1394130,19865] 0 2024-04-22T15:18:39.676 INFO:tasks.workunit.client.0.smithi096.stdout:3/161: dread da/db/dc/f16 [0,4194304] 0 2024-04-22T15:18:39.676 INFO:tasks.workunit.client.0.smithi096.stdout:3/162: chown da/db/de/d19 461558 1 2024-04-22T15:18:39.682 INFO:tasks.workunit.client.0.smithi096.stdout:3/163: dwrite da/db/dc/d1f/f29 [0,4194304] 0 2024-04-22T15:18:39.683 INFO:tasks.workunit.client.0.smithi096.stdout:3/164: fdatasync da/db/de/d19/f30 0 2024-04-22T15:18:39.686 INFO:tasks.workunit.client.0.smithi096.stdout:6/193: sync 2024-04-22T15:18:39.687 INFO:tasks.workunit.client.0.smithi096.stdout:3/165: mkdir da/db/de/d38 0 2024-04-22T15:18:39.694 INFO:tasks.workunit.client.0.smithi096.stdout:3/166: dwrite da/db/d10/f33 [0,4194304] 0 2024-04-22T15:18:39.694 INFO:tasks.workunit.client.0.smithi096.stdout:6/194: mknod d3/d28/dd/d11/c43 0 2024-04-22T15:18:39.698 INFO:tasks.workunit.client.0.smithi096.stdout:6/195: rmdir d3/d3a 39 2024-04-22T15:18:39.700 INFO:tasks.workunit.client.0.smithi096.stdout:3/167: fsync da/db/dc/f16 0 2024-04-22T15:18:39.705 INFO:tasks.workunit.client.0.smithi096.stdout:6/196: mkdir d3/d28/dd/d11/d2c/d44 0 2024-04-22T15:18:39.706 INFO:tasks.workunit.client.0.smithi096.stdout:3/168: dwrite da/db/d10/f33 [0,4194304] 0 2024-04-22T15:18:39.706 INFO:tasks.workunit.client.0.smithi096.stdout:6/197: chown d3/d28/dd/f16 14004692 1 2024-04-22T15:18:39.709 INFO:tasks.workunit.client.0.smithi096.stdout:6/198: write d3/d28/dd/d11/f1e [326972,124674] 0 2024-04-22T15:18:39.715 INFO:tasks.workunit.client.0.smithi096.stdout:6/199: link d3/d28/dd/d11/l36 d3/l45 0 2024-04-22T15:18:39.717 INFO:tasks.workunit.client.0.smithi096.stdout:6/200: mknod d3/d28/dd/c46 0 2024-04-22T15:18:39.721 INFO:tasks.workunit.client.0.smithi096.stdout:6/201: rename d3/d28/dd/c3c to d3/d28/dd/d11/c47 0 2024-04-22T15:18:39.837 INFO:tasks.workunit.client.0.smithi096.stdout:9/178: sync 2024-04-22T15:18:39.842 INFO:tasks.workunit.client.0.smithi096.stdout:9/179: dread d1/f32 [0,4194304] 0 2024-04-22T15:18:39.843 INFO:tasks.workunit.client.0.smithi096.stdout:9/180: write d1/d4/d9/dd/f23 [784505,72211] 0 2024-04-22T15:18:39.849 INFO:tasks.workunit.client.0.smithi096.stdout:9/181: mknod d1/d4/c3c 0 2024-04-22T15:18:39.851 INFO:tasks.workunit.client.0.smithi096.stdout:9/182: truncate d1/d4/d9/dd/f21 1510340 0 2024-04-22T15:18:39.919 INFO:tasks.workunit.client.0.smithi096.stdout:9/183: mknod d1/d4/d9/c3d 0 2024-04-22T15:18:39.920 INFO:tasks.workunit.client.0.smithi096.stdout:9/184: write d1/f38 [984739,14169] 0 2024-04-22T15:18:39.952 INFO:tasks.workunit.client.0.smithi096.stdout:0/242: sync 2024-04-22T15:18:39.952 INFO:tasks.workunit.client.0.smithi096.stdout:8/199: sync 2024-04-22T15:18:39.955 INFO:tasks.workunit.client.0.smithi096.stdout:0/243: rmdir d7/de/d3a 39 2024-04-22T15:18:39.969 INFO:tasks.workunit.client.0.smithi096.stdout:8/200: dread d4/d10/d25/f26 [0,4194304] 0 2024-04-22T15:18:39.970 INFO:tasks.workunit.client.0.smithi096.stdout:8/201: write d4/f12 [3455500,88624] 0 2024-04-22T15:18:39.970 INFO:tasks.workunit.client.0.smithi096.stdout:8/202: stat d4/d10/c24 0 2024-04-22T15:18:39.972 INFO:tasks.workunit.client.0.smithi096.stdout:8/203: dread d4/d10/f21 [0,4194304] 0 2024-04-22T15:18:40.006 INFO:tasks.workunit.client.0.smithi096.stdout:8/204: truncate d4/d10/d25/f26 1173958 0 2024-04-22T15:18:40.007 INFO:tasks.workunit.client.0.smithi096.stdout:8/205: write d4/d10/d25/f2e [2360489,129665] 0 2024-04-22T15:18:40.031 INFO:tasks.workunit.client.0.smithi096.stdout:9/185: mknod d1/c3e 0 2024-04-22T15:18:40.031 INFO:tasks.workunit.client.0.smithi096.stdout:9/186: stat d1/d4/d9/dd/f21 0 2024-04-22T15:18:40.038 INFO:tasks.workunit.client.0.smithi096.stdout:9/187: dwrite d1/f6 [0,4194304] 0 2024-04-22T15:18:40.041 INFO:tasks.workunit.client.0.smithi096.stdout:9/188: truncate d1/d4/f2c 499072 0 2024-04-22T15:18:40.041 INFO:tasks.workunit.client.0.smithi096.stdout:9/189: chown d1/d4/d9/f2d 263 1 2024-04-22T15:18:40.044 INFO:tasks.workunit.client.0.smithi096.stdout:9/190: symlink d1/d4/l3f 0 2024-04-22T15:18:40.051 INFO:tasks.workunit.client.0.smithi096.stdout:9/191: dwrite d1/d4/d9/f2d [0,4194304] 0 2024-04-22T15:18:40.056 INFO:tasks.workunit.client.0.smithi096.stdout:9/192: creat d1/d4/d9/dd/f40 x:0 0 0 2024-04-22T15:18:40.057 INFO:tasks.workunit.client.0.smithi096.stdout:9/193: write d1/d4/d9/f2d [3920876,43372] 0 2024-04-22T15:18:40.058 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:39 smithi096 ceph-mon[26688]: pgmap v26: 65 pgs: 65 active+clean; 653 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s rd, 75 MiB/s wr, 381 op/s 2024-04-22T15:18:40.059 INFO:tasks.workunit.client.0.smithi096.stdout:9/194: dread - d1/d4/d9/f36 zero size 2024-04-22T15:18:40.062 INFO:tasks.workunit.client.0.smithi096.stdout:9/195: creat d1/d4/d9/dd/f41 x:0 0 0 2024-04-22T15:18:40.067 INFO:tasks.workunit.client.0.smithi096.stdout:9/196: rename d1/d4/f39 to d1/d4/f42 0 2024-04-22T15:18:40.068 INFO:tasks.workunit.client.0.smithi096.stdout:9/197: truncate d1/d4/d9/f24 2234785 0 2024-04-22T15:18:40.069 INFO:tasks.workunit.client.0.smithi096.stdout:9/198: write d1/d4/d9/f24 [1463468,12268] 0 2024-04-22T15:18:40.069 INFO:tasks.workunit.client.0.smithi096.stdout:4/214: sync 2024-04-22T15:18:40.070 INFO:tasks.workunit.client.0.smithi096.stdout:5/259: sync 2024-04-22T15:18:40.070 INFO:tasks.workunit.client.0.smithi096.stdout:2/229: sync 2024-04-22T15:18:40.072 INFO:tasks.workunit.client.0.smithi096.stdout:2/230: write d3/d1f/d2f/f34 [576240,126704] 0 2024-04-22T15:18:40.072 INFO:tasks.workunit.client.0.smithi096.stdout:4/215: write d5/d7/f47 [588852,102320] 0 2024-04-22T15:18:40.079 INFO:tasks.workunit.client.0.smithi096.stdout:8/206: write d4/d1c/f43 [2122723,44720] 0 2024-04-22T15:18:40.081 INFO:tasks.workunit.client.0.smithi096.stdout:4/216: creat d5/f4e x:0 0 0 2024-04-22T15:18:40.081 INFO:tasks.workunit.client.0.smithi096.stdout:9/199: dread d1/d4/d9/f12 [0,4194304] 0 2024-04-22T15:18:40.082 INFO:tasks.workunit.client.0.smithi096.stdout:4/217: fdatasync d5/d7/d1a/f23 0 2024-04-22T15:18:40.082 INFO:tasks.workunit.client.0.smithi096.stdout:9/200: fdatasync d1/d4/d9/dd/f41 0 2024-04-22T15:18:40.083 INFO:tasks.workunit.client.0.smithi096.stdout:2/231: mkdir d3/db/d2d/d46 0 2024-04-22T15:18:40.084 INFO:tasks.workunit.client.0.smithi096.stdout:2/232: chown d3/d13 821 1 2024-04-22T15:18:40.089 INFO:tasks.workunit.client.0.smithi096.stdout:8/207: unlink d4/d10/l33 0 2024-04-22T15:18:40.090 INFO:tasks.workunit.client.0.smithi096.stdout:9/201: dwrite d1/d4/d9/f18 [4194304,4194304] 0 2024-04-22T15:18:40.091 INFO:tasks.workunit.client.0.smithi096.stdout:2/233: dread d3/f2a [0,4194304] 0 2024-04-22T15:18:40.092 INFO:tasks.workunit.client.0.smithi096.stdout:2/234: write d3/fd [3582223,77041] 0 2024-04-22T15:18:40.093 INFO:tasks.workunit.client.0.smithi096.stdout:4/218: mkdir d5/d7/d2c/d3d/d4f 0 2024-04-22T15:18:40.100 INFO:tasks.workunit.client.0.smithi096.stdout:9/202: mkdir d1/d4/d9/d43 0 2024-04-22T15:18:40.102 INFO:tasks.workunit.client.0.smithi096.stdout:9/203: write d1/f2f [449219,105711] 0 2024-04-22T15:18:40.105 INFO:tasks.workunit.client.0.smithi096.stdout:4/219: creat d5/d7/d1f/f50 x:0 0 0 2024-04-22T15:18:40.109 INFO:tasks.workunit.client.0.smithi096.stdout:8/208: dread d4/fa [0,4194304] 0 2024-04-22T15:18:40.109 INFO:tasks.workunit.client.0.smithi096.stdout:2/235: link d3/f8 d3/d1f/d2f/f47 0 2024-04-22T15:18:40.113 INFO:tasks.workunit.client.0.smithi096.stdout:7/326: sync 2024-04-22T15:18:40.113 INFO:tasks.workunit.client.0.smithi096.stdout:0/244: sync 2024-04-22T15:18:40.113 INFO:tasks.workunit.client.0.smithi096.stdout:3/169: sync 2024-04-22T15:18:40.113 INFO:tasks.workunit.client.0.smithi096.stdout:6/202: sync 2024-04-22T15:18:40.114 INFO:tasks.workunit.client.0.smithi096.stdout:1/378: sync 2024-04-22T15:18:40.114 INFO:tasks.workunit.client.0.smithi096.stdout:0/245: chown d7/de/d3a 470730 1 2024-04-22T15:18:40.115 INFO:tasks.workunit.client.0.smithi096.stdout:8/209: symlink d4/d1c/d40/l47 0 2024-04-22T15:18:40.117 INFO:tasks.workunit.client.0.smithi096.stdout:9/204: creat d1/d4/d9/d43/f44 x:0 0 0 2024-04-22T15:18:40.118 INFO:tasks.workunit.client.0.smithi096.stdout:3/170: chown da/db/dc/d1f/f15 747249 1 2024-04-22T15:18:40.121 INFO:tasks.workunit.client.0.smithi096.stdout:0/246: symlink d7/d9/d15/d35/l58 0 2024-04-22T15:18:40.123 INFO:tasks.workunit.client.0.smithi096.stdout:8/210: mkdir d4/d1c/d40/d48 0 2024-04-22T15:18:40.123 INFO:tasks.workunit.client.0.smithi096.stdout:3/171: write da/db/d10/f33 [331943,97542] 0 2024-04-22T15:18:40.130 INFO:tasks.workunit.client.0.smithi096.stdout:7/327: dwrite d31/d32/d33/f3e [0,4194304] 0 2024-04-22T15:18:40.130 INFO:tasks.workunit.client.0.smithi096.stdout:3/172: write da/db/dc/d1f/f29 [4775840,90422] 0 2024-04-22T15:18:40.130 INFO:tasks.workunit.client.0.smithi096.stdout:6/203: dwrite d3/d28/dd/d11/f1e [0,4194304] 0 2024-04-22T15:18:40.140 INFO:tasks.workunit.client.0.smithi096.stdout:8/211: creat d4/d1c/d34/f49 x:0 0 0 2024-04-22T15:18:40.145 INFO:tasks.workunit.client.0.smithi096.stdout:3/173: dread da/db/dc/f16 [0,4194304] 0 2024-04-22T15:18:40.145 INFO:tasks.workunit.client.0.smithi096.stdout:8/212: mknod d4/c4a 0 2024-04-22T15:18:40.148 INFO:tasks.workunit.client.0.smithi096.stdout:9/205: rename d1/df/f3a to d1/f45 0 2024-04-22T15:18:40.148 INFO:tasks.workunit.client.0.smithi096.stdout:5/260: dwrite d4/d2c/d2f/f3a [0,4194304] 0 2024-04-22T15:18:40.148 INFO:tasks.workunit.client.0.smithi096.stdout:6/204: truncate d3/d28/f37 1005485 0 2024-04-22T15:18:40.153 INFO:tasks.workunit.client.0.smithi096.stdout:0/247: dwrite d7/d9/d15/f36 [0,4194304] 0 2024-04-22T15:18:40.153 INFO:tasks.workunit.client.0.smithi096.stdout:8/213: truncate d4/d1c/d34/f3e 4873022 0 2024-04-22T15:18:40.155 INFO:tasks.workunit.client.0.smithi096.stdout:6/205: dread - d3/d28/dd/f2b zero size 2024-04-22T15:18:40.158 INFO:tasks.workunit.client.0.smithi096.stdout:0/248: dread - d7/d9/d15/d35/f47 zero size 2024-04-22T15:18:40.158 INFO:tasks.workunit.client.0.smithi096.stdout:7/328: dwrite d31/d32/d33/f5c [0,4194304] 0 2024-04-22T15:18:40.161 INFO:tasks.workunit.client.0.smithi096.stdout:1/379: link d6/dc/d10/d3f/d54/d32/l51 d6/dc/d10/d3f/d62/l7c 0 2024-04-22T15:18:40.165 INFO:tasks.workunit.client.0.smithi096.stdout:3/174: symlink da/db/de/d19/l39 0 2024-04-22T15:18:40.166 INFO:tasks.workunit.client.0.smithi096.stdout:5/261: dread d4/d1f/f21 [0,4194304] 0 2024-04-22T15:18:40.166 INFO:tasks.workunit.client.0.smithi096.stdout:4/220: rmdir d5/d7/d1f 39 2024-04-22T15:18:40.167 INFO:tasks.workunit.client.0.smithi096.stdout:8/214: dwrite d4/fd [0,4194304] 0 2024-04-22T15:18:40.169 INFO:tasks.workunit.client.0.smithi096.stdout:4/221: dread - d5/d7/d2c/d3d/f49 zero size 2024-04-22T15:18:40.169 INFO:tasks.workunit.client.0.smithi096.stdout:3/175: write da/db/d12/d22/f31 [193452,106974] 0 2024-04-22T15:18:40.170 INFO:tasks.workunit.client.0.smithi096.stdout:5/262: chown d4/d12/d23/c47 79330443 1 2024-04-22T15:18:40.173 INFO:tasks.workunit.client.0.smithi096.stdout:0/249: mkdir d7/d9/d59 0 2024-04-22T15:18:40.176 INFO:tasks.workunit.client.0.smithi096.stdout:7/329: rename d31/d32/d33/l4e to d31/d32/d36/l66 0 2024-04-22T15:18:40.185 INFO:tasks.workunit.client.0.smithi096.stdout:6/206: creat d3/d28/dd/d11/d2c/d3b/f48 x:0 0 0 2024-04-22T15:18:40.188 INFO:tasks.workunit.client.0.smithi096.stdout:3/176: creat da/db/d12/d22/f3a x:0 0 0 2024-04-22T15:18:40.192 INFO:tasks.workunit.client.0.smithi096.stdout:1/380: rename d6/dc/d11/d40 to d6/dc/d10/d3f/d62/d7d 0 2024-04-22T15:18:40.196 INFO:tasks.workunit.client.0.smithi096.stdout:4/222: unlink d5/fb 0 2024-04-22T15:18:40.197 INFO:tasks.workunit.client.0.smithi096.stdout:4/223: truncate d5/d7/d1a/f40 1880990 0 2024-04-22T15:18:40.198 INFO:tasks.workunit.client.0.smithi096.stdout:5/263: mknod d4/d11/c54 0 2024-04-22T15:18:40.199 INFO:tasks.workunit.client.0.smithi096.stdout:5/264: chown d4/ce 10914 1 2024-04-22T15:18:40.200 INFO:tasks.workunit.client.0.smithi096.stdout:3/177: unlink l6 0 2024-04-22T15:18:40.202 INFO:tasks.workunit.client.0.smithi096.stdout:8/215: rename d4/f39 to d4/d10/d25/d38/f4b 0 2024-04-22T15:18:40.205 INFO:tasks.workunit.client.0.smithi096.stdout:8/216: read d4/f28 [6941890,81599] 0 2024-04-22T15:18:40.205 INFO:tasks.workunit.client.0.smithi096.stdout:9/206: getdents d1/d4/d9/dd 0 2024-04-22T15:18:40.207 INFO:tasks.workunit.client.0.smithi096.stdout:6/207: dread d3/d28/f23 [0,4194304] 0 2024-04-22T15:18:40.207 INFO:tasks.workunit.client.0.smithi096.stdout:1/381: creat d6/dc/d11/d45/d7b/f7e x:0 0 0 2024-04-22T15:18:40.215 INFO:tasks.workunit.client.0.smithi096.stdout:7/330: rename f5 to d31/d32/d33/d54/d43/f67 0 2024-04-22T15:18:40.216 INFO:tasks.workunit.client.0.smithi096.stdout:3/178: rename da/db/de to da/db/de/d3b 22 2024-04-22T15:18:40.216 INFO:tasks.workunit.client.0.smithi096.stdout:3/179: fdatasync da/f27 0 2024-04-22T15:18:40.219 INFO:tasks.workunit.client.0.smithi096.stdout:8/217: mkdir d4/d10/d25/d4c 0 2024-04-22T15:18:40.224 INFO:tasks.workunit.client.0.smithi096.stdout:7/331: dwrite d31/d32/d36/f45 [0,4194304] 0 2024-04-22T15:18:40.225 INFO:tasks.workunit.client.0.smithi096.stdout:1/382: read d6/dc/d11/d4a/f4d [3521589,50179] 0 2024-04-22T15:18:40.226 INFO:tasks.workunit.client.0.smithi096.stdout:3/180: dwrite da/db/dc/d1f/f29 [0,4194304] 0 2024-04-22T15:18:40.242 INFO:tasks.workunit.client.0.smithi096.stdout:5/265: dwrite f1 [0,4194304] 0 2024-04-22T15:18:40.243 INFO:tasks.workunit.client.0.smithi096.stdout:6/208: unlink d3/d28/dd/d11/l36 0 2024-04-22T15:18:40.243 INFO:tasks.workunit.client.0.smithi096.stdout:5/266: fsync d4/d2c/d2f/f3f 0 2024-04-22T15:18:40.245 INFO:tasks.workunit.client.0.smithi096.stdout:7/332: fdatasync d31/d32/d33/d54/f64 0 2024-04-22T15:18:40.247 INFO:tasks.workunit.client.0.smithi096.stdout:1/383: creat d6/dc/d10/d3f/d54/d32/d3b/d56/f7f x:0 0 0 2024-04-22T15:18:40.248 INFO:tasks.workunit.client.0.smithi096.stdout:3/181: truncate da/db/dc/f16 234350 0 2024-04-22T15:18:40.251 INFO:tasks.workunit.client.0.smithi096.stdout:8/218: mkdir d4/d10/d4d 0 2024-04-22T15:18:40.253 INFO:tasks.workunit.client.0.smithi096.stdout:5/267: mknod d4/d2c/d2f/d30/c55 0 2024-04-22T15:18:40.254 INFO:tasks.workunit.client.0.smithi096.stdout:4/224: dread d5/d7/d1f/f42 [0,4194304] 0 2024-04-22T15:18:40.254 INFO:tasks.workunit.client.0.smithi096.stdout:7/333: mkdir d31/d68 0 2024-04-22T15:18:40.255 INFO:tasks.workunit.client.0.smithi096.stdout:5/268: truncate d4/f43 998362 0 2024-04-22T15:18:40.255 INFO:tasks.workunit.client.0.smithi096.stdout:4/225: chown d5/f19 14804066 1 2024-04-22T15:18:40.257 INFO:tasks.workunit.client.0.smithi096.stdout:8/219: dread d4/d2a/f36 [0,4194304] 0 2024-04-22T15:18:40.258 INFO:tasks.workunit.client.0.smithi096.stdout:7/334: unlink l14 0 2024-04-22T15:18:40.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:39 smithi196 ceph-mon[34717]: pgmap v26: 65 pgs: 65 active+clean; 653 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s rd, 75 MiB/s wr, 381 op/s 2024-04-22T15:18:40.259 INFO:tasks.workunit.client.0.smithi096.stdout:8/220: chown d4/d1c/c3f 8880907 1 2024-04-22T15:18:40.261 INFO:tasks.workunit.client.0.smithi096.stdout:6/209: link d3/d28/f38 d3/f49 0 2024-04-22T15:18:40.263 INFO:tasks.workunit.client.0.smithi096.stdout:6/210: readlink d3/d28/dd/d11/l13 0 2024-04-22T15:18:40.263 INFO:tasks.workunit.client.0.smithi096.stdout:7/335: dread d31/d32/d33/d54/d43/f48 [0,4194304] 0 2024-04-22T15:18:40.264 INFO:tasks.workunit.client.0.smithi096.stdout:1/384: creat d6/dc/f80 x:0 0 0 2024-04-22T15:18:40.265 INFO:tasks.workunit.client.0.smithi096.stdout:5/269: dwrite d4/d2c/d2f/f51 [0,4194304] 0 2024-04-22T15:18:40.267 INFO:tasks.workunit.client.0.smithi096.stdout:1/385: write d6/dc/d10/d29/f36 [639441,93311] 0 2024-04-22T15:18:40.269 INFO:tasks.workunit.client.0.smithi096.stdout:1/386: readlink d6/dc/l5d 0 2024-04-22T15:18:40.270 INFO:tasks.workunit.client.0.smithi096.stdout:0/250: truncate d7/d9/f17 9354436 0 2024-04-22T15:18:40.272 INFO:tasks.workunit.client.0.smithi096.stdout:8/221: rmdir d4/d10/d25/d38 39 2024-04-22T15:18:40.273 INFO:tasks.workunit.client.0.smithi096.stdout:8/222: write d4/d10/d25/f26 [1050628,75519] 0 2024-04-22T15:18:40.274 INFO:tasks.workunit.client.0.smithi096.stdout:4/226: creat d5/d7/d2c/d3d/d4f/f51 x:0 0 0 2024-04-22T15:18:40.275 INFO:tasks.workunit.client.0.smithi096.stdout:6/211: write d3/d28/f38 [710164,89414] 0 2024-04-22T15:18:40.291 INFO:tasks.workunit.client.0.smithi096.stdout:4/227: rmdir d5 39 2024-04-22T15:18:40.294 INFO:tasks.workunit.client.0.smithi096.stdout:7/336: creat d31/d68/f69 x:0 0 0 2024-04-22T15:18:40.294 INFO:tasks.workunit.client.0.smithi096.stdout:5/270: write d4/f6 [596558,21595] 0 2024-04-22T15:18:40.298 INFO:tasks.workunit.client.0.smithi096.stdout:0/251: symlink d7/d9/d15/l5a 0 2024-04-22T15:18:40.301 INFO:tasks.workunit.client.0.smithi096.stdout:7/337: dwrite f10 [0,4194304] 0 2024-04-22T15:18:40.305 INFO:tasks.workunit.client.0.smithi096.stdout:8/223: dread d4/f28 [4194304,4194304] 0 2024-04-22T15:18:40.305 INFO:tasks.workunit.client.0.smithi096.stdout:0/252: stat d7/d9/dc/d16/d18/f30 0 2024-04-22T15:18:40.306 INFO:tasks.workunit.client.0.smithi096.stdout:1/387: link d6/dc/d10/d3f/d62/d7d/f53 d6/dc/f81 0 2024-04-22T15:18:40.307 INFO:tasks.workunit.client.0.smithi096.stdout:1/388: dread - d6/dc/f80 zero size 2024-04-22T15:18:40.312 INFO:tasks.workunit.client.0.smithi096.stdout:7/338: mkdir d31/d32/d33/d6a 0 2024-04-22T15:18:40.313 INFO:tasks.workunit.client.0.smithi096.stdout:1/389: fdatasync d6/dc/d10/d3f/d62/d7d/f53 0 2024-04-22T15:18:40.316 INFO:tasks.workunit.client.0.smithi096.stdout:5/271: rmdir d4/d1f/d29 0 2024-04-22T15:18:40.321 INFO:tasks.workunit.client.0.smithi096.stdout:7/339: rename d31/d32/f40 to d31/d32/d36/d5d/f6b 0 2024-04-22T15:18:40.321 INFO:tasks.workunit.client.0.smithi096.stdout:7/340: chown d31/d32/d33/f5c 77394 1 2024-04-22T15:18:40.323 INFO:tasks.workunit.client.0.smithi096.stdout:7/341: write d31/d32/d36/f45 [4808616,128443] 0 2024-04-22T15:18:40.328 INFO:tasks.workunit.client.0.smithi096.stdout:4/228: link f2 d5/d7/f52 0 2024-04-22T15:18:40.328 INFO:tasks.workunit.client.0.smithi096.stdout:7/342: dwrite f7 [0,4194304] 0 2024-04-22T15:18:40.330 INFO:tasks.workunit.client.0.smithi096.stdout:6/212: getdents d3/d3a 0 2024-04-22T15:18:40.330 INFO:tasks.workunit.client.0.smithi096.stdout:7/343: write f2b [913029,67336] 0 2024-04-22T15:18:40.337 INFO:tasks.workunit.client.0.smithi096.stdout:5/272: creat d4/d11/d3d/d4d/f56 x:0 0 0 2024-04-22T15:18:40.339 INFO:tasks.workunit.client.0.smithi096.stdout:8/224: rmdir d4/d10/d25/d38/d45 0 2024-04-22T15:18:40.342 INFO:tasks.workunit.client.0.smithi096.stdout:0/253: getdents d7/d9/dc/d16 0 2024-04-22T15:18:40.343 INFO:tasks.workunit.client.0.smithi096.stdout:0/254: write d7/d9/d15/d35/f41 [1039442,14719] 0 2024-04-22T15:18:40.345 INFO:tasks.workunit.client.0.smithi096.stdout:0/255: write d7/d42/d4c/f4f [1626175,77838] 0 2024-04-22T15:18:40.346 INFO:tasks.workunit.client.0.smithi096.stdout:4/229: truncate d5/d7/f41 1572661 0 2024-04-22T15:18:40.348 INFO:tasks.workunit.client.0.smithi096.stdout:7/344: readlink d31/d32/d33/l5b 0 2024-04-22T15:18:40.349 INFO:tasks.workunit.client.0.smithi096.stdout:1/390: write d6/dc/d10/d3f/d54/d32/f6d [2101990,70109] 0 2024-04-22T15:18:40.350 INFO:tasks.workunit.client.0.smithi096.stdout:7/345: read - d31/d32/d33/d54/f62 zero size 2024-04-22T15:18:40.351 INFO:tasks.workunit.client.0.smithi096.stdout:5/273: unlink d4/f5 0 2024-04-22T15:18:40.360 INFO:tasks.workunit.client.0.smithi096.stdout:1/391: mkdir d6/dc/d10/d3f/d54/d32/d82 0 2024-04-22T15:18:40.361 INFO:tasks.workunit.client.0.smithi096.stdout:1/392: write d6/dc/d10/d3f/d54/d32/f37 [2962171,92248] 0 2024-04-22T15:18:40.362 INFO:tasks.workunit.client.0.smithi096.stdout:1/393: dread - d6/dc/d10/d3f/f73 zero size 2024-04-22T15:18:40.364 INFO:tasks.workunit.client.0.smithi096.stdout:5/274: mkdir d4/d11/d3d/d35/d57 0 2024-04-22T15:18:40.367 INFO:tasks.workunit.client.0.smithi096.stdout:4/230: rename d5/d7/l2b to d5/d7/d1f/l53 0 2024-04-22T15:18:40.368 INFO:tasks.workunit.client.0.smithi096.stdout:4/231: write d5/d7/d1a/f35 [96983,57259] 0 2024-04-22T15:18:40.370 INFO:tasks.workunit.client.0.smithi096.stdout:8/225: creat d4/d10/f4e x:0 0 0 2024-04-22T15:18:40.370 INFO:tasks.workunit.client.0.smithi096.stdout:6/213: dread d3/d28/f1c [0,4194304] 0 2024-04-22T15:18:40.377 INFO:tasks.workunit.client.0.smithi096.stdout:5/275: unlink d4/d1f/c4c 0 2024-04-22T15:18:40.380 INFO:tasks.workunit.client.0.smithi096.stdout:0/256: rmdir d7/d9/d15/d35/d4e 0 2024-04-22T15:18:40.385 INFO:tasks.workunit.client.0.smithi096.stdout:8/226: creat d4/d1c/d34/f4f x:0 0 0 2024-04-22T15:18:40.387 INFO:tasks.workunit.client.0.smithi096.stdout:7/346: creat d31/d32/f6c x:0 0 0 2024-04-22T15:18:40.389 INFO:tasks.workunit.client.0.smithi096.stdout:8/227: dread d4/fa [0,4194304] 0 2024-04-22T15:18:40.391 INFO:tasks.workunit.client.0.smithi096.stdout:0/257: unlink d7/d42/d4c/f4f 0 2024-04-22T15:18:40.393 INFO:tasks.workunit.client.0.smithi096.stdout:0/258: write d7/d9/dc/f2f [945795,60745] 0 2024-04-22T15:18:40.393 INFO:tasks.workunit.client.0.smithi096.stdout:0/259: chown d7/de 4658239 1 2024-04-22T15:18:40.402 INFO:tasks.workunit.client.0.smithi096.stdout:7/347: dwrite d31/d32/d36/f41 [0,4194304] 0 2024-04-22T15:18:40.409 INFO:tasks.workunit.client.0.smithi096.stdout:0/260: fdatasync d7/d9/f40 0 2024-04-22T15:18:40.411 INFO:tasks.workunit.client.0.smithi096.stdout:4/232: dwrite d5/d7/d2c/f3a [0,4194304] 0 2024-04-22T15:18:40.412 INFO:tasks.workunit.client.0.smithi096.stdout:6/214: creat d3/d28/dd/f4a x:0 0 0 2024-04-22T15:18:40.413 INFO:tasks.workunit.client.0.smithi096.stdout:1/394: link d6/dc/d10/d3f/d54/c42 d6/c83 0 2024-04-22T15:18:40.414 INFO:tasks.workunit.client.0.smithi096.stdout:0/261: chown d7/d9/dc/f57 462 1 2024-04-22T15:18:40.416 INFO:tasks.workunit.client.0.smithi096.stdout:4/233: dread d5/fe [0,4194304] 0 2024-04-22T15:18:40.416 INFO:tasks.workunit.client.0.smithi096.stdout:8/228: dread d4/d10/d25/d38/f4b [0,4194304] 0 2024-04-22T15:18:40.417 INFO:tasks.workunit.client.0.smithi096.stdout:7/348: chown d31/d32/d33/d44 1396 1 2024-04-22T15:18:40.422 INFO:tasks.workunit.client.0.smithi096.stdout:6/215: mkdir d3/d3a/d4b 0 2024-04-22T15:18:40.425 INFO:tasks.workunit.client.0.smithi096.stdout:1/395: creat d6/dc/d10/d3f/d62/d7d/f84 x:0 0 0 2024-04-22T15:18:40.426 INFO:tasks.workunit.client.0.smithi096.stdout:5/276: dwrite d4/d12/d23/f50 [0,4194304] 0 2024-04-22T15:18:40.430 INFO:tasks.workunit.client.0.smithi096.stdout:8/229: creat d4/d10/d25/d38/f50 x:0 0 0 2024-04-22T15:18:40.431 INFO:tasks.workunit.client.0.smithi096.stdout:8/230: chown d4/c27 57 1 2024-04-22T15:18:40.431 INFO:tasks.workunit.client.0.smithi096.stdout:4/234: symlink d5/d7/d2c/d43/l54 0 2024-04-22T15:18:40.432 INFO:tasks.workunit.client.0.smithi096.stdout:8/231: dread - d4/d10/f23 zero size 2024-04-22T15:18:40.434 INFO:tasks.workunit.client.0.smithi096.stdout:7/349: creat d31/d32/d33/d5e/f6d x:0 0 0 2024-04-22T15:18:40.435 INFO:tasks.workunit.client.0.smithi096.stdout:7/350: dread - d31/d68/f69 zero size 2024-04-22T15:18:40.439 INFO:tasks.workunit.client.0.smithi096.stdout:0/262: mkdir d7/de/d5b 0 2024-04-22T15:18:40.440 INFO:tasks.workunit.client.0.smithi096.stdout:4/235: creat d5/d7/d1f/f55 x:0 0 0 2024-04-22T15:18:40.441 INFO:tasks.workunit.client.0.smithi096.stdout:8/232: rmdir d4/d1c/d40 39 2024-04-22T15:18:40.441 INFO:tasks.workunit.client.0.smithi096.stdout:6/216: dwrite d3/d28/f1c [0,4194304] 0 2024-04-22T15:18:40.444 INFO:tasks.workunit.client.0.smithi096.stdout:7/351: creat d31/d68/f6e x:0 0 0 2024-04-22T15:18:40.447 INFO:tasks.workunit.client.0.smithi096.stdout:1/396: rmdir d6/dc/d10/d3f/d62/d7d 39 2024-04-22T15:18:40.449 INFO:tasks.workunit.client.0.smithi096.stdout:0/263: dwrite d7/d9/dc/f14 [0,4194304] 0 2024-04-22T15:18:40.454 INFO:tasks.workunit.client.0.smithi096.stdout:4/236: mkdir d5/d7/d2c/d43/d56 0 2024-04-22T15:18:40.454 INFO:tasks.workunit.client.0.smithi096.stdout:6/217: dwrite d3/d3a/f3e [0,4194304] 0 2024-04-22T15:18:40.455 INFO:tasks.workunit.client.0.smithi096.stdout:8/233: symlink d4/d1c/d34/l51 0 2024-04-22T15:18:40.458 INFO:tasks.workunit.client.0.smithi096.stdout:7/352: write fa [2879247,34347] 0 2024-04-22T15:18:40.463 INFO:tasks.workunit.client.0.smithi096.stdout:5/277: link d4/d2c/d2f/d30/c55 d4/d12/c58 0 2024-04-22T15:18:40.465 INFO:tasks.workunit.client.0.smithi096.stdout:1/397: unlink d6/dc/d10/d3f/d54/d32/d3b/d56/f72 0 2024-04-22T15:18:40.468 INFO:tasks.workunit.client.0.smithi096.stdout:4/237: mknod d5/d7/d2c/d43/c57 0 2024-04-22T15:18:40.468 INFO:tasks.workunit.client.0.smithi096.stdout:8/234: fdatasync d4/d10/f21 0 2024-04-22T15:18:40.468 INFO:tasks.workunit.client.0.smithi096.stdout:6/218: dwrite d3/d28/f31 [0,4194304] 0 2024-04-22T15:18:40.470 INFO:tasks.workunit.client.0.smithi096.stdout:5/278: rmdir d4/d2c/d2f/d30 39 2024-04-22T15:18:40.472 INFO:tasks.workunit.client.0.smithi096.stdout:4/238: fsync d5/d7/f20 0 2024-04-22T15:18:40.475 INFO:tasks.workunit.client.0.smithi096.stdout:8/235: read d4/d1c/f43 [581540,50373] 0 2024-04-22T15:18:40.480 INFO:tasks.workunit.client.0.smithi096.stdout:7/353: dwrite d31/d32/d33/d5e/f6d [0,4194304] 0 2024-04-22T15:18:40.486 INFO:tasks.workunit.client.0.smithi096.stdout:5/279: write d4/d2c/d2f/f3a [645347,122564] 0 2024-04-22T15:18:40.489 INFO:tasks.workunit.client.0.smithi096.stdout:0/264: dwrite d7/d9/d15/d35/f47 [0,4194304] 0 2024-04-22T15:18:40.490 INFO:tasks.workunit.client.0.smithi096.stdout:4/239: truncate d5/d7/d1a/d25/f3f 8925519 0 2024-04-22T15:18:40.491 INFO:tasks.workunit.client.0.smithi096.stdout:2/236: sync 2024-04-22T15:18:40.491 INFO:tasks.workunit.client.0.smithi096.stdout:8/236: unlink d4/d10/c20 0 2024-04-22T15:18:40.492 INFO:tasks.workunit.client.0.smithi096.stdout:1/398: dwrite d6/f2c [0,4194304] 0 2024-04-22T15:18:40.496 INFO:tasks.workunit.client.0.smithi096.stdout:0/265: chown d7/l2c 1488 1 2024-04-22T15:18:40.502 INFO:tasks.workunit.client.0.smithi096.stdout:1/399: dread - d6/dc/d11/d45/d7b/f7e zero size 2024-04-22T15:18:40.503 INFO:tasks.workunit.client.0.smithi096.stdout:7/354: symlink d31/d32/d33/d5e/l6f 0 2024-04-22T15:18:40.504 INFO:tasks.workunit.client.0.smithi096.stdout:6/219: dwrite d3/f24 [0,4194304] 0 2024-04-22T15:18:40.505 INFO:tasks.workunit.client.0.smithi096.stdout:4/240: mknod d5/d7/d2c/d43/c58 0 2024-04-22T15:18:40.507 INFO:tasks.workunit.client.0.smithi096.stdout:2/237: truncate d3/db/df/d10/f37 1738798 0 2024-04-22T15:18:40.507 INFO:tasks.workunit.client.0.smithi096.stdout:5/280: mkdir d4/d11/d3d/d4d/d59 0 2024-04-22T15:18:40.508 INFO:tasks.workunit.client.0.smithi096.stdout:4/241: chown d5/d7/d1a/d25/l37 149 1 2024-04-22T15:18:40.509 INFO:tasks.workunit.client.0.smithi096.stdout:2/238: chown d3/d1f 0 1 2024-04-22T15:18:40.510 INFO:tasks.workunit.client.0.smithi096.stdout:4/242: stat d5/f13 0 2024-04-22T15:18:40.511 INFO:tasks.workunit.client.0.smithi096.stdout:2/239: dread - d3/db/d30/f3a zero size 2024-04-22T15:18:40.515 INFO:tasks.workunit.client.0.smithi096.stdout:7/355: creat d31/d32/d36/f70 x:0 0 0 2024-04-22T15:18:40.519 INFO:tasks.workunit.client.0.smithi096.stdout:8/237: mknod d4/d10/d25/c52 0 2024-04-22T15:18:40.521 INFO:tasks.workunit.client.0.smithi096.stdout:5/281: unlink d4/d11/d3d/d35/c46 0 2024-04-22T15:18:40.524 INFO:tasks.workunit.client.0.smithi096.stdout:4/243: chown f2 2 1 2024-04-22T15:18:40.526 INFO:tasks.workunit.client.0.smithi096.stdout:6/220: symlink d3/d28/dd/d11/l4c 0 2024-04-22T15:18:40.527 INFO:tasks.workunit.client.0.smithi096.stdout:1/400: dwrite d6/dc/d10/d29/f79 [0,4194304] 0 2024-04-22T15:18:40.527 INFO:tasks.workunit.client.0.smithi096.stdout:7/356: fdatasync d31/d32/d33/d5e/f63 0 2024-04-22T15:18:40.529 INFO:tasks.workunit.client.0.smithi096.stdout:2/240: symlink d3/db/l48 0 2024-04-22T15:18:40.533 INFO:tasks.workunit.client.0.smithi096.stdout:4/244: unlink d5/f19 0 2024-04-22T15:18:40.533 INFO:tasks.workunit.client.0.smithi096.stdout:6/221: fsync d3/f26 0 2024-04-22T15:18:40.533 INFO:tasks.workunit.client.0.smithi096.stdout:4/245: chown f2 390 1 2024-04-22T15:18:40.533 INFO:tasks.workunit.client.0.smithi096.stdout:2/241: write d3/db/d30/f39 [591601,11544] 0 2024-04-22T15:18:40.536 INFO:tasks.workunit.client.0.smithi096.stdout:2/242: write d3/db/df/d10/f18 [5072022,55801] 0 2024-04-22T15:18:40.537 INFO:tasks.workunit.client.0.smithi096.stdout:0/266: getdents d7 0 2024-04-22T15:18:40.543 INFO:tasks.workunit.client.0.smithi096.stdout:2/243: write d3/fd [2452835,51488] 0 2024-04-22T15:18:40.543 INFO:tasks.workunit.client.0.smithi096.stdout:0/267: truncate d7/d9/dc/d16/d18/f3f 130536 0 2024-04-22T15:18:40.547 INFO:tasks.workunit.client.0.smithi096.stdout:1/401: dwrite d6/dc/d10/d3f/d54/d32/d3b/d3d/f4b [4194304,4194304] 0 2024-04-22T15:18:40.548 INFO:tasks.workunit.client.0.smithi096.stdout:7/357: rmdir d31/d32/d36 39 2024-04-22T15:18:40.549 INFO:tasks.workunit.client.0.smithi096.stdout:6/222: dwrite d3/f26 [0,4194304] 0 2024-04-22T15:18:40.549 INFO:tasks.workunit.client.0.smithi096.stdout:1/402: chown d6/dc/d11/d45 16 1 2024-04-22T15:18:40.550 INFO:tasks.workunit.client.0.smithi096.stdout:8/238: rename d4/lf to d4/d1c/l53 0 2024-04-22T15:18:40.553 INFO:tasks.workunit.client.0.smithi096.stdout:7/358: dread - d31/d32/f60 zero size 2024-04-22T15:18:40.554 INFO:tasks.workunit.client.0.smithi096.stdout:4/246: fsync d5/d7/d1a/d25/f4d 0 2024-04-22T15:18:40.558 INFO:tasks.workunit.client.0.smithi096.stdout:4/247: dread - d5/d7/d1a/f23 zero size 2024-04-22T15:18:40.562 INFO:tasks.workunit.client.0.smithi096.stdout:4/248: read - d5/d7/d1f/f50 zero size 2024-04-22T15:18:40.562 INFO:tasks.workunit.client.0.smithi096.stdout:2/244: write d3/d1f/d2f/f41 [83922,82407] 0 2024-04-22T15:18:40.565 INFO:tasks.workunit.client.0.smithi096.stdout:5/282: rename d4/d12 to d4/d11/d3d/d4d/d59/d5a 0 2024-04-22T15:18:40.566 INFO:tasks.workunit.client.0.smithi096.stdout:1/403: readlink d6/dc/d10/l28 0 2024-04-22T15:18:40.566 INFO:tasks.workunit.client.0.smithi096.stdout:1/404: fdatasync d6/f1d 0 2024-04-22T15:18:40.567 INFO:tasks.workunit.client.0.smithi096.stdout:8/239: dwrite d4/d2a/f36 [0,4194304] 0 2024-04-22T15:18:40.569 INFO:tasks.workunit.client.0.smithi096.stdout:7/359: rmdir d31/d32/d33/d54/d43 39 2024-04-22T15:18:40.570 INFO:tasks.workunit.client.0.smithi096.stdout:0/268: symlink d7/d9/d59/l5c 0 2024-04-22T15:18:40.572 INFO:tasks.workunit.client.0.smithi096.stdout:4/249: write d5/f13 [3361670,101214] 0 2024-04-22T15:18:40.573 INFO:tasks.workunit.client.0.smithi096.stdout:2/245: fsync d3/f8 0 2024-04-22T15:18:40.575 INFO:tasks.workunit.client.0.smithi096.stdout:6/223: mkdir d3/d28/d4d 0 2024-04-22T15:18:40.581 INFO:tasks.workunit.client.0.smithi096.stdout:8/240: creat d4/d10/d25/d38/f54 x:0 0 0 2024-04-22T15:18:40.582 INFO:tasks.workunit.client.0.smithi096.stdout:8/241: dread - d4/d1c/d34/f49 zero size 2024-04-22T15:18:40.584 INFO:tasks.workunit.client.0.smithi096.stdout:0/269: creat d7/d9/dc/d16/f5d x:0 0 0 2024-04-22T15:18:40.587 INFO:tasks.workunit.client.0.smithi096.stdout:8/242: dread d4/fa [0,4194304] 0 2024-04-22T15:18:40.588 INFO:tasks.workunit.client.0.smithi096.stdout:6/224: creat d3/d28/dd/d11/d2c/f4e x:0 0 0 2024-04-22T15:18:40.588 INFO:tasks.workunit.client.0.smithi096.stdout:5/283: mkdir d4/d2c/d2f/d30/d5b 0 2024-04-22T15:18:40.589 INFO:tasks.workunit.client.0.smithi096.stdout:8/243: chown d4/d1c/c3f 0 1 2024-04-22T15:18:40.591 INFO:tasks.workunit.client.0.smithi096.stdout:7/360: creat d31/d32/d36/f71 x:0 0 0 2024-04-22T15:18:40.592 INFO:tasks.workunit.client.0.smithi096.stdout:2/246: dwrite d3/db/df/d10/f1b [0,4194304] 0 2024-04-22T15:18:40.594 INFO:tasks.workunit.client.0.smithi096.stdout:2/247: chown d3/db/df/d10/f18 478475744 1 2024-04-22T15:18:40.594 INFO:tasks.workunit.client.0.smithi096.stdout:0/270: readlink d7/d9/dc/d16/l50 0 2024-04-22T15:18:40.596 INFO:tasks.workunit.client.0.smithi096.stdout:0/271: fsync d7/d9/dc/f2f 0 2024-04-22T15:18:40.597 INFO:tasks.workunit.client.0.smithi096.stdout:5/284: mkdir d4/d11/d3d/d4d/d59/d5a/d5c 0 2024-04-22T15:18:40.598 INFO:tasks.workunit.client.0.smithi096.stdout:5/285: fdatasync d4/f7 0 2024-04-22T15:18:40.608 INFO:tasks.workunit.client.0.smithi096.stdout:6/225: getdents d3/d3a/d4b 0 2024-04-22T15:18:40.608 INFO:tasks.workunit.client.0.smithi096.stdout:2/248: dwrite d3/f2a [0,4194304] 0 2024-04-22T15:18:40.608 INFO:tasks.workunit.client.0.smithi096.stdout:6/226: truncate d3/d28/dd/d11/d2c/f39 944063 0 2024-04-22T15:18:40.608 INFO:tasks.workunit.client.0.smithi096.stdout:0/272: rmdir d7/d9 39 2024-04-22T15:18:40.609 INFO:tasks.workunit.client.0.smithi096.stdout:4/250: getdents d5/d7/d2c/d43 0 2024-04-22T15:18:40.611 INFO:tasks.workunit.client.0.smithi096.stdout:7/361: creat d31/d32/d33/d6a/f72 x:0 0 0 2024-04-22T15:18:40.615 INFO:tasks.workunit.client.0.smithi096.stdout:5/286: creat d4/d11/d3d/d4d/f5d x:0 0 0 2024-04-22T15:18:40.615 INFO:tasks.workunit.client.0.smithi096.stdout:2/249: read d3/db/df/d10/f18 [4987645,116000] 0 2024-04-22T15:18:40.615 INFO:tasks.workunit.client.0.smithi096.stdout:1/405: link d6/dc/d10/d3f/d62/d7d/f38 d6/dc/d10/d3f/d62/f85 0 2024-04-22T15:18:40.618 INFO:tasks.workunit.client.0.smithi096.stdout:6/227: fsync d3/d28/f21 0 2024-04-22T15:18:40.620 INFO:tasks.workunit.client.0.smithi096.stdout:8/244: dwrite d4/d10/f21 [0,4194304] 0 2024-04-22T15:18:40.622 INFO:tasks.workunit.client.0.smithi096.stdout:8/245: write d4/d10/d25/f37 [511182,43352] 0 2024-04-22T15:18:40.624 INFO:tasks.workunit.client.0.smithi096.stdout:6/228: truncate d3/d28/dd/d11/d2c/f4e 638705 0 2024-04-22T15:18:40.625 INFO:tasks.workunit.client.0.smithi096.stdout:5/287: creat d4/d2c/d2f/d30/f5e x:0 0 0 2024-04-22T15:18:40.626 INFO:tasks.workunit.client.0.smithi096.stdout:0/273: dread d7/d9/dc/d16/d18/f3f [0,4194304] 0 2024-04-22T15:18:40.626 INFO:tasks.workunit.client.0.smithi096.stdout:6/229: stat d3/d28/dd/d11/c43 0 2024-04-22T15:18:40.629 INFO:tasks.workunit.client.0.smithi096.stdout:4/251: creat d5/d7/d2c/d43/d56/f59 x:0 0 0 2024-04-22T15:18:40.631 INFO:tasks.workunit.client.0.smithi096.stdout:1/406: creat d6/dc/d10/d3f/d62/d7d/f86 x:0 0 0 2024-04-22T15:18:40.635 INFO:tasks.workunit.client.0.smithi096.stdout:8/246: unlink d4/d10/f23 0 2024-04-22T15:18:40.636 INFO:tasks.workunit.client.0.smithi096.stdout:1/407: truncate d6/dc/d10/d3f/d62/f76 391038 0 2024-04-22T15:18:40.636 INFO:tasks.workunit.client.0.smithi096.stdout:0/274: dwrite d7/d9/dc/d16/d18/f3d [0,4194304] 0 2024-04-22T15:18:40.638 INFO:tasks.workunit.client.0.smithi096.stdout:8/247: dread - d4/d1c/d34/f4f zero size 2024-04-22T15:18:40.639 INFO:tasks.workunit.client.0.smithi096.stdout:0/275: write d7/d9/d15/f36 [2297402,74406] 0 2024-04-22T15:18:40.639 INFO:tasks.workunit.client.0.smithi096.stdout:1/408: write d6/dc/d10/f67 [1045211,84350] 0 2024-04-22T15:18:40.643 INFO:tasks.workunit.client.0.smithi096.stdout:0/276: write d7/d9/dc/f2f [816056,111817] 0 2024-04-22T15:18:40.644 INFO:tasks.workunit.client.0.smithi096.stdout:2/250: creat d3/f49 x:0 0 0 2024-04-22T15:18:40.646 INFO:tasks.workunit.client.0.smithi096.stdout:7/362: dread d31/d32/d33/d5e/f63 [0,4194304] 0 2024-04-22T15:18:40.650 INFO:tasks.workunit.client.0.smithi096.stdout:2/251: dread d3/f8 [0,4194304] 0 2024-04-22T15:18:40.652 INFO:tasks.workunit.client.0.smithi096.stdout:2/252: write d3/db/d30/f39 [55364,130093] 0 2024-04-22T15:18:40.656 INFO:tasks.workunit.client.0.smithi096.stdout:7/363: dread d31/d32/d33/d54/f64 [0,4194304] 0 2024-04-22T15:18:40.658 INFO:tasks.workunit.client.0.smithi096.stdout:7/364: write fa [4245958,51833] 0 2024-04-22T15:18:40.660 INFO:tasks.workunit.client.0.smithi096.stdout:1/409: mkdir d6/dc/d10/d3f/d62/d7d/d87 0 2024-04-22T15:18:40.660 INFO:tasks.workunit.client.0.smithi096.stdout:4/252: write f2 [3881154,110846] 0 2024-04-22T15:18:40.663 INFO:tasks.workunit.client.0.smithi096.stdout:1/410: dread - d6/dc/d10/d3f/d62/d7d/f53 zero size 2024-04-22T15:18:40.663 INFO:tasks.workunit.client.0.smithi096.stdout:0/277: fsync d7/d9/d15/f1a 0 2024-04-22T15:18:40.666 INFO:tasks.workunit.client.0.smithi096.stdout:7/365: dwrite d31/d32/d33/f5c [0,4194304] 0 2024-04-22T15:18:40.668 INFO:tasks.workunit.client.0.smithi096.stdout:2/253: mkdir d3/d1f/d4a 0 2024-04-22T15:18:40.671 INFO:tasks.workunit.client.0.smithi096.stdout:2/254: write d3/db/d30/f3e [619169,114611] 0 2024-04-22T15:18:40.671 INFO:tasks.workunit.client.0.smithi096.stdout:6/230: truncate f1 2470112 0 2024-04-22T15:18:40.678 INFO:tasks.workunit.client.0.smithi096.stdout:7/366: dwrite d31/d32/d33/f34 [0,4194304] 0 2024-04-22T15:18:40.681 INFO:tasks.workunit.client.0.smithi096.stdout:8/248: dwrite d4/fd [4194304,4194304] 0 2024-04-22T15:18:40.681 INFO:tasks.workunit.client.0.smithi096.stdout:1/411: readlink d6/dc/d10/l17 0 2024-04-22T15:18:40.685 INFO:tasks.workunit.client.0.smithi096.stdout:0/278: chown d7/f4d 56162891 1 2024-04-22T15:18:40.687 INFO:tasks.workunit.client.0.smithi096.stdout:7/367: mknod d31/d32/d36/d5d/c73 0 2024-04-22T15:18:40.699 INFO:tasks.workunit.client.0.smithi096.stdout:8/249: symlink d4/d10/d4d/l55 0 2024-04-22T15:18:40.701 INFO:tasks.workunit.client.0.smithi096.stdout:7/368: symlink d31/d32/d36/l74 0 2024-04-22T15:18:40.703 INFO:tasks.workunit.client.0.smithi096.stdout:2/255: rmdir d3/d1f/d4a 0 2024-04-22T15:18:40.705 INFO:tasks.workunit.client.0.smithi096.stdout:2/256: truncate f0 5096872 0 2024-04-22T15:18:40.707 INFO:tasks.workunit.client.0.smithi096.stdout:6/231: rename d3/f24 to d3/d28/f4f 0 2024-04-22T15:18:40.707 INFO:tasks.workunit.client.0.smithi096.stdout:7/369: dread fa [0,4194304] 0 2024-04-22T15:18:40.711 INFO:tasks.workunit.client.0.smithi096.stdout:0/279: symlink d7/d9/l5e 0 2024-04-22T15:18:40.711 INFO:tasks.workunit.client.0.smithi096.stdout:4/253: dread d5/d7/d1a/d25/f4d [0,4194304] 0 2024-04-22T15:18:40.712 INFO:tasks.workunit.client.0.smithi096.stdout:1/412: dread d6/dc/d10/d3f/d54/d32/f37 [0,4194304] 0 2024-04-22T15:18:40.713 INFO:tasks.workunit.client.0.smithi096.stdout:4/254: chown d5/d7/d1a/d25/l37 59 1 2024-04-22T15:18:40.716 INFO:tasks.workunit.client.0.smithi096.stdout:4/255: read d5/d7/d2c/f4a [2611488,115811] 0 2024-04-22T15:18:40.717 INFO:tasks.workunit.client.0.smithi096.stdout:2/257: mknod d3/db/df/d24/c4b 0 2024-04-22T15:18:40.721 INFO:tasks.workunit.client.0.smithi096.stdout:1/413: dwrite d6/dc/d10/d3f/d54/d32/d3b/f6f [4194304,4194304] 0 2024-04-22T15:18:40.728 INFO:tasks.workunit.client.0.smithi096.stdout:2/258: dwrite d3/f49 [0,4194304] 0 2024-04-22T15:18:40.731 INFO:tasks.workunit.client.0.smithi096.stdout:2/259: stat d3/d13/c45 0 2024-04-22T15:18:40.732 INFO:tasks.workunit.client.0.smithi096.stdout:1/414: dread d6/f1d [0,4194304] 0 2024-04-22T15:18:40.736 INFO:tasks.workunit.client.0.smithi096.stdout:7/370: dwrite d31/d32/d36/d5d/f6b [0,4194304] 0 2024-04-22T15:18:40.737 INFO:tasks.workunit.client.0.smithi096.stdout:7/371: stat d31/d32/d33/l5b 0 2024-04-22T15:18:40.738 INFO:tasks.workunit.client.0.smithi096.stdout:7/372: write f23 [1141574,126982] 0 2024-04-22T15:18:40.742 INFO:tasks.workunit.client.0.smithi096.stdout:0/280: dread - d7/d9/f45 zero size 2024-04-22T15:18:40.743 INFO:tasks.workunit.client.0.smithi096.stdout:2/260: fdatasync d3/db/df/d10/f18 0 2024-04-22T15:18:40.745 INFO:tasks.workunit.client.0.smithi096.stdout:4/256: symlink d5/d7/d1a/l5a 0 2024-04-22T15:18:40.746 INFO:tasks.workunit.client.0.smithi096.stdout:1/415: symlink d6/dc/d10/d3f/d54/d32/d3b/d56/l88 0 2024-04-22T15:18:40.748 INFO:tasks.workunit.client.0.smithi096.stdout:1/416: write d6/dc/d10/d3f/d54/d32/f52 [520712,44933] 0 2024-04-22T15:18:40.749 INFO:tasks.workunit.client.0.smithi096.stdout:8/250: rename d4/fa to d4/d10/f56 0 2024-04-22T15:18:40.754 INFO:tasks.workunit.client.0.smithi096.stdout:4/257: rmdir d5/d7/d2c/d3d 39 2024-04-22T15:18:40.756 INFO:tasks.workunit.client.0.smithi096.stdout:1/417: dwrite d6/dc/d10/d3f/d54/d32/d3b/d3d/f4b [0,4194304] 0 2024-04-22T15:18:40.758 INFO:tasks.workunit.client.0.smithi096.stdout:6/232: symlink d3/d3a/d4b/l50 0 2024-04-22T15:18:40.759 INFO:tasks.workunit.client.0.smithi096.stdout:8/251: mknod d4/d10/d25/d38/c57 0 2024-04-22T15:18:40.763 INFO:tasks.workunit.client.0.smithi096.stdout:2/261: symlink d3/db/d2d/d46/l4c 0 2024-04-22T15:18:40.767 INFO:tasks.workunit.client.0.smithi096.stdout:1/418: creat d6/dc/d10/d3f/d54/d32/d3b/f89 x:0 0 0 2024-04-22T15:18:40.767 INFO:tasks.workunit.client.0.smithi096.stdout:7/373: dread d31/d32/d33/f39 [0,4194304] 0 2024-04-22T15:18:40.769 INFO:tasks.workunit.client.0.smithi096.stdout:4/258: rmdir d5/d7/d2c/d3d/d4f 39 2024-04-22T15:18:40.771 INFO:tasks.workunit.client.0.smithi096.stdout:2/262: truncate d3/db/f31 1160729 0 2024-04-22T15:18:40.772 INFO:tasks.workunit.client.0.smithi096.stdout:8/252: dwrite d4/f28 [8388608,4194304] 0 2024-04-22T15:18:40.772 INFO:tasks.workunit.client.0.smithi096.stdout:6/233: write d3/d28/f17 [2828243,62401] 0 2024-04-22T15:18:40.775 INFO:tasks.workunit.client.0.smithi096.stdout:1/419: rmdir d6/dc/d10 39 2024-04-22T15:18:40.777 INFO:tasks.workunit.client.0.smithi096.stdout:7/374: mkdir d31/d32/d36/d75 0 2024-04-22T15:18:40.778 INFO:tasks.workunit.client.0.smithi096.stdout:4/259: dread d5/d7/d1a/f35 [0,4194304] 0 2024-04-22T15:18:40.781 INFO:tasks.workunit.client.0.smithi096.stdout:2/263: symlink d3/d1f/l4d 0 2024-04-22T15:18:40.788 INFO:tasks.workunit.client.0.smithi096.stdout:4/260: fsync f1 0 2024-04-22T15:18:40.790 INFO:tasks.workunit.client.0.smithi096.stdout:7/375: mknod d31/d32/d33/d44/c76 0 2024-04-22T15:18:40.791 INFO:tasks.workunit.client.0.smithi096.stdout:2/264: unlink d3/f2a 0 2024-04-22T15:18:40.794 INFO:tasks.workunit.client.0.smithi096.stdout:6/234: mkdir d3/d28/dd/d11/d2c/d44/d51 0 2024-04-22T15:18:40.856 INFO:tasks.workunit.client.0.smithi096.stdout:2/265: rename d3/db/df/d10/f44 to d3/d1f/d2f/f4e 0 2024-04-22T15:18:40.856 INFO:tasks.workunit.client.0.smithi096.stdout:8/253: mkdir d4/d1c/d40/d46/d58 0 2024-04-22T15:18:40.858 INFO:tasks.workunit.client.0.smithi096.stdout:1/420: dwrite d6/f4f [0,4194304] 0 2024-04-22T15:18:40.858 INFO:tasks.workunit.client.0.smithi096.stdout:7/376: getdents d31/d32/d36/d75 0 2024-04-22T15:18:40.860 INFO:tasks.workunit.client.0.smithi096.stdout:8/254: write d4/d10/d25/f2e [1472447,3276] 0 2024-04-22T15:18:40.862 INFO:tasks.workunit.client.0.smithi096.stdout:6/235: rename d3/d28/dd/d11/l13 to d3/d3a/l52 0 2024-04-22T15:18:40.863 INFO:tasks.workunit.client.0.smithi096.stdout:2/266: symlink d3/db/d30/l4f 0 2024-04-22T15:18:40.865 INFO:tasks.workunit.client.0.smithi096.stdout:4/261: dwrite d5/d7/f36 [0,4194304] 0 2024-04-22T15:18:40.867 INFO:tasks.workunit.client.0.smithi096.stdout:1/421: fsync d6/dc/d10/d3f/d62/f70 0 2024-04-22T15:18:40.867 INFO:tasks.workunit.client.0.smithi096.stdout:6/236: write d3/d28/dd/f4a [596892,67415] 0 2024-04-22T15:18:40.870 INFO:tasks.workunit.client.0.smithi096.stdout:2/267: symlink d3/db/d2d/d46/l50 0 2024-04-22T15:18:40.874 INFO:tasks.workunit.client.0.smithi096.stdout:4/262: dwrite d5/d7/f24 [0,4194304] 0 2024-04-22T15:18:40.874 INFO:tasks.workunit.client.0.smithi096.stdout:7/377: mknod d31/d32/d33/d54/d43/c77 0 2024-04-22T15:18:40.879 INFO:tasks.workunit.client.0.smithi096.stdout:6/237: creat d3/d28/dd/d11/d2c/d44/f53 x:0 0 0 2024-04-22T15:18:40.882 INFO:tasks.workunit.client.0.smithi096.stdout:2/268: creat d3/d1f/f51 x:0 0 0 2024-04-22T15:18:40.883 INFO:tasks.workunit.client.0.smithi096.stdout:2/269: readlink d3/db/df/l25 0 2024-04-22T15:18:40.883 INFO:tasks.workunit.client.0.smithi096.stdout:4/263: mkdir d5/d5b 0 2024-04-22T15:18:40.884 INFO:tasks.workunit.client.0.smithi096.stdout:4/264: chown d5/c8 196226 1 2024-04-22T15:18:40.889 INFO:tasks.workunit.client.0.smithi096.stdout:6/238: dread - d3/f29 zero size 2024-04-22T15:18:40.892 INFO:tasks.workunit.client.0.smithi096.stdout:8/255: rename d4/d10/l1a to d4/d10/l59 0 2024-04-22T15:18:40.893 INFO:tasks.workunit.client.0.smithi096.stdout:2/270: symlink d3/d1f/l52 0 2024-04-22T15:18:40.897 INFO:tasks.workunit.client.0.smithi096.stdout:8/256: dread d4/d2a/f36 [0,4194304] 0 2024-04-22T15:18:40.900 INFO:tasks.workunit.client.0.smithi096.stdout:2/271: dwrite d3/db/df/f35 [0,4194304] 0 2024-04-22T15:18:40.901 INFO:tasks.workunit.client.0.smithi096.stdout:2/272: truncate d3/db/d2d/f33 4906573 0 2024-04-22T15:18:40.902 INFO:tasks.workunit.client.0.smithi096.stdout:4/265: rename d5/d7/d1a/f35 to d5/d7/d2c/d3d/d4f/f5c 0 2024-04-22T15:18:40.902 INFO:tasks.workunit.client.0.smithi096.stdout:2/273: readlink d3/d1f/l4d 0 2024-04-22T15:18:40.902 INFO:tasks.workunit.client.0.smithi096.stdout:1/422: getdents d6/dc/d11/d4a 0 2024-04-22T15:18:40.903 INFO:tasks.workunit.client.0.smithi096.stdout:1/423: write d6/f4f [4335587,102384] 0 2024-04-22T15:18:40.905 INFO:tasks.workunit.client.0.smithi096.stdout:6/239: mkdir d3/d28/d54 0 2024-04-22T15:18:40.907 INFO:tasks.workunit.client.0.smithi096.stdout:8/257: unlink d4/d10/d25/f41 0 2024-04-22T15:18:40.911 INFO:tasks.workunit.client.0.smithi096.stdout:8/258: dread d4/d10/d25/f37 [0,4194304] 0 2024-04-22T15:18:40.912 INFO:tasks.workunit.client.0.smithi096.stdout:8/259: dread - d4/d10/d25/d38/f54 zero size 2024-04-22T15:18:40.913 INFO:tasks.workunit.client.0.smithi096.stdout:8/260: read d4/f28 [11171029,69038] 0 2024-04-22T15:18:40.917 INFO:tasks.workunit.client.0.smithi096.stdout:4/266: fdatasync d5/d7/d1a/f1d 0 2024-04-22T15:18:40.918 INFO:tasks.workunit.client.0.smithi096.stdout:7/378: rmdir d31/d32/d33/d54 39 2024-04-22T15:18:40.919 INFO:tasks.workunit.client.0.smithi096.stdout:7/379: write d31/d68/f6e [449692,116027] 0 2024-04-22T15:18:40.925 INFO:tasks.workunit.client.0.smithi096.stdout:4/267: dwrite d5/d7/d2c/f3a [0,4194304] 0 2024-04-22T15:18:40.927 INFO:tasks.workunit.client.0.smithi096.stdout:6/240: dread f2 [0,4194304] 0 2024-04-22T15:18:40.928 INFO:tasks.workunit.client.0.smithi096.stdout:6/241: fsync d3/d28/dd/f19 0 2024-04-22T15:18:40.932 INFO:tasks.workunit.client.0.smithi096.stdout:2/274: creat d3/f53 x:0 0 0 2024-04-22T15:18:40.933 INFO:tasks.workunit.client.0.smithi096.stdout:2/275: stat d3/db/d2d/d46/l4c 0 2024-04-22T15:18:40.940 INFO:tasks.workunit.client.0.smithi096.stdout:4/268: dread d5/fc [4194304,4194304] 0 2024-04-22T15:18:40.940 INFO:tasks.workunit.client.0.smithi096.stdout:1/424: rename d6/dc/d10 to d6/dc/d8a 0 2024-04-22T15:18:40.944 INFO:tasks.workunit.client.0.smithi096.stdout:8/261: mknod d4/d10/d25/d4c/c5a 0 2024-04-22T15:18:40.948 INFO:tasks.workunit.client.0.smithi096.stdout:1/425: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d3d/f63 [0,4194304] 0 2024-04-22T15:18:40.949 INFO:tasks.workunit.client.0.smithi096.stdout:6/242: write d3/d28/f4f [3049147,126372] 0 2024-04-22T15:18:40.949 INFO:tasks.workunit.client.0.smithi096.stdout:1/426: chown d6/f1d 29513 1 2024-04-22T15:18:40.950 INFO:tasks.workunit.client.0.smithi096.stdout:7/380: symlink d31/d32/l78 0 2024-04-22T15:18:40.951 INFO:tasks.workunit.client.0.smithi096.stdout:7/381: readlink d31/d32/l78 0 2024-04-22T15:18:40.962 INFO:tasks.workunit.client.0.smithi096.stdout:1/427: unlink d6/dc/d11/f31 0 2024-04-22T15:18:40.963 INFO:tasks.workunit.client.0.smithi096.stdout:8/262: dread d4/f12 [0,4194304] 0 2024-04-22T15:18:40.963 INFO:tasks.workunit.client.0.smithi096.stdout:7/382: symlink d31/d68/l79 0 2024-04-22T15:18:40.967 INFO:tasks.workunit.client.0.smithi096.stdout:4/269: link d5/fc d5/d7/d2c/d3d/d4f/f5d 0 2024-04-22T15:18:40.968 INFO:tasks.workunit.client.0.smithi096.stdout:4/270: chown d5/d7/d2c/l2d 1 1 2024-04-22T15:18:40.969 INFO:tasks.workunit.client.0.smithi096.stdout:7/383: dread d31/d32/d36/f45 [0,4194304] 0 2024-04-22T15:18:40.970 INFO:tasks.workunit.client.0.smithi096.stdout:8/263: dwrite d4/d10/d25/f44 [0,4194304] 0 2024-04-22T15:18:40.970 INFO:tasks.workunit.client.0.smithi096.stdout:7/384: fdatasync d31/d32/f59 0 2024-04-22T15:18:40.972 INFO:tasks.workunit.client.0.smithi096.stdout:1/428: creat d6/dc/d8a/d3f/d54/d32/d3b/d3d/f8b x:0 0 0 2024-04-22T15:18:40.979 INFO:tasks.workunit.client.0.smithi096.stdout:1/429: dwrite d6/dc/d8a/d29/f79 [0,4194304] 0 2024-04-22T15:18:41.015 INFO:tasks.workunit.client.0.smithi096.stdout:6/243: creat d3/d28/dd/f55 x:0 0 0 2024-04-22T15:18:41.016 INFO:tasks.workunit.client.0.smithi096.stdout:9/207: sync 2024-04-22T15:18:41.016 INFO:tasks.workunit.client.0.smithi096.stdout:3/182: sync 2024-04-22T15:18:41.016 INFO:tasks.workunit.client.0.smithi096.stdout:4/271: write d5/d7/d1a/f23 [897468,28886] 0 2024-04-22T15:18:41.017 INFO:tasks.workunit.client.0.smithi096.stdout:9/208: chown d1/d4/d9 7217434 1 2024-04-22T15:18:41.018 INFO:tasks.workunit.client.0.smithi096.stdout:3/183: write da/db/de/d19/f30 [665739,111817] 0 2024-04-22T15:18:41.035 INFO:tasks.workunit.client.0.smithi096.stdout:2/276: rename d3/d13/l28 to d3/l54 0 2024-04-22T15:18:41.036 INFO:tasks.workunit.client.0.smithi096.stdout:2/277: fsync d3/db/df/f35 0 2024-04-22T15:18:41.037 INFO:tasks.workunit.client.0.smithi096.stdout:1/430: stat d6/dc/d8a/d3f/d54/d32/c48 0 2024-04-22T15:18:41.081 INFO:tasks.workunit.client.0.smithi096.stdout:9/209: write d1/d4/f42 [717538,74682] 0 2024-04-22T15:18:41.082 INFO:tasks.workunit.client.0.smithi096.stdout:2/278: unlink d3/f8 0 2024-04-22T15:18:41.084 INFO:tasks.workunit.client.0.smithi096.stdout:3/184: fdatasync da/f34 0 2024-04-22T15:18:41.085 INFO:tasks.workunit.client.0.smithi096.stdout:3/185: readlink da/l1b 0 2024-04-22T15:18:41.086 INFO:tasks.workunit.client.0.smithi096.stdout:7/385: creat d31/d32/d33/d54/f7a x:0 0 0 2024-04-22T15:18:41.087 INFO:tasks.workunit.client.0.smithi096.stdout:2/279: truncate d3/d1f/d2f/f41 338899 0 2024-04-22T15:18:41.088 INFO:tasks.workunit.client.0.smithi096.stdout:6/244: rename d3/d28/ff to d3/d28/dd/d11/d2c/f56 0 2024-04-22T15:18:41.089 INFO:tasks.workunit.client.0.smithi096.stdout:4/272: dwrite d5/d7/d1a/d25/f4d [4194304,4194304] 0 2024-04-22T15:18:41.091 INFO:tasks.workunit.client.0.smithi096.stdout:6/245: chown d3/d28/dd/d11/l1b 118602 1 2024-04-22T15:18:41.091 INFO:tasks.workunit.client.0.smithi096.stdout:9/210: creat d1/d4/d9/dd/f46 x:0 0 0 2024-04-22T15:18:41.096 INFO:tasks.workunit.client.0.smithi096.stdout:3/186: dwrite da/db/de/d19/f30 [0,4194304] 0 2024-04-22T15:18:41.100 INFO:tasks.workunit.client.0.smithi096.stdout:7/386: symlink d31/d32/d36/l7b 0 2024-04-22T15:18:41.101 INFO:tasks.workunit.client.0.smithi096.stdout:7/387: write d31/d32/d33/d54/f7a [298245,26474] 0 2024-04-22T15:18:41.104 INFO:tasks.workunit.client.0.smithi096.stdout:1/431: rename d6/dc/d11 to d6/dc/d8a/d3f/d54/d32/d8c 0 2024-04-22T15:18:41.107 INFO:tasks.workunit.client.0.smithi096.stdout:4/273: fsync d5/f13 0 2024-04-22T15:18:41.111 INFO:tasks.workunit.client.0.smithi096.stdout:6/246: symlink d3/d28/dd/d11/d2c/d44/l57 0 2024-04-22T15:18:41.112 INFO:tasks.workunit.client.0.smithi096.stdout:2/280: dwrite d3/db/d2d/f33 [0,4194304] 0 2024-04-22T15:18:41.116 INFO:tasks.workunit.client.0.smithi096.stdout:4/274: dwrite d5/f4e [0,4194304] 0 2024-04-22T15:18:41.128 INFO:tasks.workunit.client.0.smithi096.stdout:2/281: creat d3/db/df/d10/f55 x:0 0 0 2024-04-22T15:18:41.128 INFO:tasks.workunit.client.0.smithi096.stdout:6/247: dwrite d3/f49 [0,4194304] 0 2024-04-22T15:18:41.133 INFO:tasks.workunit.client.0.smithi096.stdout:7/388: mkdir d31/d32/d7c 0 2024-04-22T15:18:41.135 INFO:tasks.workunit.client.0.smithi096.stdout:4/275: unlink d5/d7/d2c/f4a 0 2024-04-22T15:18:41.145 INFO:tasks.workunit.client.0.smithi096.stdout:7/389: symlink d31/d32/d36/d5d/l7d 0 2024-04-22T15:18:41.147 INFO:tasks.workunit.client.0.smithi096.stdout:4/276: dread d5/d7/d2c/d3d/d4f/f5c [0,4194304] 0 2024-04-22T15:18:41.149 INFO:tasks.workunit.client.0.smithi096.stdout:1/432: rename d6/f22 to d6/dc/d8a/d3f/d62/f8d 0 2024-04-22T15:18:41.151 INFO:tasks.workunit.client.0.smithi096.stdout:3/187: getdents da/db/d12/d22 0 2024-04-22T15:18:41.154 INFO:tasks.workunit.client.0.smithi096.stdout:6/248: symlink d3/d28/d54/l58 0 2024-04-22T15:18:41.156 INFO:tasks.workunit.client.0.smithi096.stdout:6/249: dread - d3/d28/dd/d11/f41 zero size 2024-04-22T15:18:41.157 INFO:tasks.workunit.client.0.smithi096.stdout:2/282: link d3/d1f/d2f/f34 d3/d1f/f56 0 2024-04-22T15:18:41.157 INFO:tasks.workunit.client.0.smithi096.stdout:1/433: dwrite d6/dc/d8a/d3f/d62/d7d/f84 [0,4194304] 0 2024-04-22T15:18:41.159 INFO:tasks.workunit.client.0.smithi096.stdout:3/188: dwrite da/db/de/d19/d2d/f2e [0,4194304] 0 2024-04-22T15:18:41.160 INFO:tasks.workunit.client.0.smithi096.stdout:6/250: dread - d3/d28/dd/d11/d2c/d3b/f48 zero size 2024-04-22T15:18:41.160 INFO:tasks.workunit.client.0.smithi096.stdout:1/434: write d6/dc/f81 [804284,93051] 0 2024-04-22T15:18:41.163 INFO:tasks.workunit.client.0.smithi096.stdout:6/251: write d3/d28/dd/f16 [484779,3453] 0 2024-04-22T15:18:41.169 INFO:tasks.workunit.client.0.smithi096.stdout:4/277: rename d5/d7/d1a/d25/f3f to d5/f5e 0 2024-04-22T15:18:41.176 INFO:tasks.workunit.client.0.smithi096.stdout:9/211: fsync d1/d4/d9/dd/f46 0 2024-04-22T15:18:41.179 INFO:tasks.workunit.client.0.smithi096.stdout:6/252: unlink d3/d28/f31 0 2024-04-22T15:18:41.184 INFO:tasks.workunit.client.0.smithi096.stdout:3/189: getdents da/db/de/d19/d2d/d37 0 2024-04-22T15:18:41.191 INFO:tasks.workunit.client.0.smithi096.stdout:9/212: dwrite d1/f45 [0,4194304] 0 2024-04-22T15:18:41.199 INFO:tasks.workunit.client.0.smithi096.stdout:9/213: dwrite d1/d4/d9/dd/f23 [0,4194304] 0 2024-04-22T15:18:41.200 INFO:tasks.workunit.client.0.smithi096.stdout:6/253: rename d3/c15 to d3/d3a/c59 0 2024-04-22T15:18:41.201 INFO:tasks.workunit.client.0.smithi096.stdout:9/214: truncate d1/d4/d9/dd/f41 256467 0 2024-04-22T15:18:41.211 INFO:tasks.workunit.client.0.smithi096.stdout:4/278: getdents d5/d7/d2c 0 2024-04-22T15:18:41.214 INFO:tasks.workunit.client.0.smithi096.stdout:9/215: creat d1/d4/f47 x:0 0 0 2024-04-22T15:18:41.214 INFO:tasks.workunit.client.0.smithi096.stdout:9/216: dread - d1/d4/d9/f36 zero size 2024-04-22T15:18:41.217 INFO:tasks.workunit.client.0.smithi096.stdout:7/390: rmdir d31 39 2024-04-22T15:18:41.220 INFO:tasks.workunit.client.0.smithi096.stdout:9/217: creat d1/d25/f48 x:0 0 0 2024-04-22T15:18:41.223 INFO:tasks.workunit.client.0.smithi096.stdout:2/283: dwrite d3/db/df/d10/f21 [0,4194304] 0 2024-04-22T15:18:41.225 INFO:tasks.workunit.client.0.smithi096.stdout:7/391: dwrite f21 [0,4194304] 0 2024-04-22T15:18:41.229 INFO:tasks.workunit.client.0.smithi096.stdout:4/279: dread d5/d7/f20 [0,4194304] 0 2024-04-22T15:18:41.230 INFO:tasks.workunit.client.0.smithi096.stdout:9/218: rename d1/d4/f2c to d1/d25/f49 0 2024-04-22T15:18:41.232 INFO:tasks.workunit.client.0.smithi096.stdout:2/284: chown d3/db/c32 9039 1 2024-04-22T15:18:41.235 INFO:tasks.workunit.client.0.smithi096.stdout:7/392: truncate d31/d32/d33/f35 712393 0 2024-04-22T15:18:41.239 INFO:tasks.workunit.client.0.smithi096.stdout:7/393: rmdir d31/d32/d33 39 2024-04-22T15:18:41.240 INFO:tasks.workunit.client.0.smithi096.stdout:7/394: truncate d31/d68/f69 62666 0 2024-04-22T15:18:41.243 INFO:tasks.workunit.client.0.smithi096.stdout:7/395: fsync d31/d32/d33/f58 0 2024-04-22T15:18:41.243 INFO:tasks.workunit.client.0.smithi096.stdout:7/396: chown c2f 400359374 1 2024-04-22T15:18:41.244 INFO:tasks.workunit.client.0.smithi096.stdout:7/397: write d31/d32/d33/f34 [4069334,99610] 0 2024-04-22T15:18:41.250 INFO:tasks.workunit.client.0.smithi096.stdout:9/219: getdents d1/d4/d9/d43 0 2024-04-22T15:18:41.251 INFO:tasks.workunit.client.0.smithi096.stdout:2/285: fsync d3/db/df/d10/f21 0 2024-04-22T15:18:41.254 INFO:tasks.workunit.client.0.smithi096.stdout:1/435: dwrite d6/dc/d8a/d3f/d62/d7d/f38 [0,4194304] 0 2024-04-22T15:18:41.254 INFO:tasks.workunit.client.0.smithi096.stdout:7/398: mknod d31/d32/d7c/c7e 0 2024-04-22T15:18:41.256 INFO:tasks.workunit.client.0.smithi096.stdout:1/436: stat d6/dc/d8a/d29/f59 0 2024-04-22T15:18:41.257 INFO:tasks.workunit.client.0.smithi096.stdout:2/286: dread d3/db/df/f35 [0,4194304] 0 2024-04-22T15:18:41.259 INFO:tasks.workunit.client.0.smithi096.stdout:7/399: rmdir d31/d32/d33/d6a 39 2024-04-22T15:18:41.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:41 smithi196 ceph-mon[34717]: Upgrade: Updating mgr.smithi096.hhkvum 2024-04-22T15:18:41.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:41 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:41.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:41 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi096.hhkvum", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:18:41.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:41 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi096.hhkvum", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:18:41.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:41 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:18:41.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:41 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:18:41.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:41 smithi196 ceph-mon[34717]: Deploying daemon mgr.smithi096.hhkvum on smithi096 2024-04-22T15:18:41.260 INFO:tasks.workunit.client.0.smithi096.stdout:2/287: readlink d3/db/df/d10/l16 0 2024-04-22T15:18:41.260 INFO:tasks.workunit.client.0.smithi096.stdout:1/437: dread d6/dc/d8a/d3f/d62/f76 [0,4194304] 0 2024-04-22T15:18:41.263 INFO:tasks.workunit.client.0.smithi096.stdout:5/288: sync 2024-04-22T15:18:41.265 INFO:tasks.workunit.client.0.smithi096.stdout:5/289: readlink d4/d11/l1b 0 2024-04-22T15:18:41.268 INFO:tasks.workunit.client.0.smithi096.stdout:1/438: dwrite d6/dc/d8a/d3f/d62/f70 [0,4194304] 0 2024-04-22T15:18:41.271 INFO:tasks.workunit.client.0.smithi096.stdout:5/290: truncate d4/d11/d3d/d4d/d59/d5a/f13 4587350 0 2024-04-22T15:18:41.272 INFO:tasks.workunit.client.0.smithi096.stdout:9/220: dread d1/f20 [0,4194304] 0 2024-04-22T15:18:41.273 INFO:tasks.workunit.client.0.smithi096.stdout:2/288: symlink d3/db/d2d/d46/l57 0 2024-04-22T15:18:41.277 INFO:tasks.workunit.client.0.smithi096.stdout:6/254: write d3/d28/f21 [243607,66840] 0 2024-04-22T15:18:41.279 INFO:tasks.workunit.client.0.smithi096.stdout:1/439: dwrite f2 [0,4194304] 0 2024-04-22T15:18:41.279 INFO:tasks.workunit.client.0.smithi096.stdout:9/221: mknod d1/d25/c4a 0 2024-04-22T15:18:41.283 INFO:tasks.workunit.client.0.smithi096.stdout:3/190: dwrite da/db/dc/f16 [0,4194304] 0 2024-04-22T15:18:41.288 INFO:tasks.workunit.client.0.smithi096.stdout:7/400: rename d31/d32/f4f to d31/d32/d33/f7f 0 2024-04-22T15:18:41.293 INFO:tasks.workunit.client.0.smithi096.stdout:5/291: creat d4/d11/d3d/d4d/d59/d5a/d5c/f5f x:0 0 0 2024-04-22T15:18:41.294 INFO:tasks.workunit.client.0.smithi096.stdout:5/292: chown d4/d11/d3d/d4d/d59/d5a/d5c 253174 1 2024-04-22T15:18:41.298 INFO:tasks.workunit.client.0.smithi096.stdout:1/440: truncate d6/dc/d8a/d3f/d54/d32/d8c/d45/f50 713438 0 2024-04-22T15:18:41.299 INFO:tasks.workunit.client.0.smithi096.stdout:1/441: fdatasync d6/dc/d8a/d3f/d54/d32/d8c/d4a/f4d 0 2024-04-22T15:18:41.300 INFO:tasks.workunit.client.0.smithi096.stdout:2/289: truncate d3/d1f/d2f/f47 1850769 0 2024-04-22T15:18:41.301 INFO:tasks.workunit.client.0.smithi096.stdout:2/290: stat d3/d13/c3d 0 2024-04-22T15:18:41.302 INFO:tasks.workunit.client.0.smithi096.stdout:1/442: write d6/dc/d8a/d29/f79 [691528,27598] 0 2024-04-22T15:18:41.303 INFO:tasks.workunit.client.0.smithi096.stdout:6/255: mkdir d3/d5a 0 2024-04-22T15:18:41.305 INFO:tasks.workunit.client.0.smithi096.stdout:4/280: dwrite d5/d7/d1a/f40 [0,4194304] 0 2024-04-22T15:18:41.311 INFO:tasks.workunit.client.0.smithi096.stdout:2/291: mkdir d3/db/df/d24/d58 0 2024-04-22T15:18:41.313 INFO:tasks.workunit.client.0.smithi096.stdout:1/443: truncate d6/fd 3531915 0 2024-04-22T15:18:41.314 INFO:tasks.workunit.client.0.smithi096.stdout:4/281: symlink d5/d7/d1f/l5f 0 2024-04-22T15:18:41.315 INFO:tasks.workunit.client.0.smithi096.stdout:1/444: write d6/dc/d8a/d3f/d54/d32/d8c/f19 [3115411,127867] 0 2024-04-22T15:18:41.317 INFO:tasks.workunit.client.0.smithi096.stdout:1/445: readlink d6/dc/d8a/d3f/d54/d32/d3b/d56/l88 0 2024-04-22T15:18:41.319 INFO:tasks.workunit.client.0.smithi096.stdout:2/292: mkdir d3/db/d2d/d46/d59 0 2024-04-22T15:18:41.319 INFO:tasks.workunit.client.0.smithi096.stdout:9/222: rmdir d1/d25 39 2024-04-22T15:18:41.321 INFO:tasks.workunit.client.0.smithi096.stdout:7/401: dread d31/d32/d36/f4d [0,4194304] 0 2024-04-22T15:18:41.322 INFO:tasks.workunit.client.0.smithi096.stdout:4/282: creat d5/d7/d2c/d3d/d4f/f60 x:0 0 0 2024-04-22T15:18:41.324 INFO:tasks.workunit.client.0.smithi096.stdout:4/283: read f3 [242161,3244] 0 2024-04-22T15:18:41.324 INFO:tasks.workunit.client.0.smithi096.stdout:2/293: dread d3/db/df/d10/f37 [0,4194304] 0 2024-04-22T15:18:41.326 INFO:tasks.workunit.client.0.smithi096.stdout:5/293: rename d4/d2c/d2f/d30/f5e to d4/d1f/f60 0 2024-04-22T15:18:41.326 INFO:tasks.workunit.client.0.smithi096.stdout:3/191: link da/db/l36 da/db/d12/d22/l3c 0 2024-04-22T15:18:41.326 INFO:tasks.workunit.client.0.smithi096.stdout:2/294: truncate d3/db/df/f35 5097555 0 2024-04-22T15:18:41.329 INFO:tasks.workunit.client.0.smithi096.stdout:1/446: mkdir d6/dc/d8a/d3f/d62/d7d/d8e 0 2024-04-22T15:18:41.330 INFO:tasks.workunit.client.0.smithi096.stdout:1/447: write f2 [6509428,45461] 0 2024-04-22T15:18:41.332 INFO:tasks.workunit.client.0.smithi096.stdout:9/223: symlink d1/d4/d9/d43/l4b 0 2024-04-22T15:18:41.343 INFO:tasks.workunit.client.0.smithi096.stdout:7/402: dread f10 [0,4194304] 0 2024-04-22T15:18:41.344 INFO:tasks.workunit.client.0.smithi096.stdout:4/284: mkdir d5/d7/d1a/d61 0 2024-04-22T15:18:41.345 INFO:tasks.workunit.client.0.smithi096.stdout:7/403: read d31/d32/d36/f37 [651350,34000] 0 2024-04-22T15:18:41.347 INFO:tasks.workunit.client.0.smithi096.stdout:5/294: rename d4/l49 to d4/d11/d3d/d4d/d59/l61 0 2024-04-22T15:18:41.351 INFO:tasks.workunit.client.0.smithi096.stdout:3/192: creat da/db/f3d x:0 0 0 2024-04-22T15:18:41.355 INFO:tasks.workunit.client.0.smithi096.stdout:9/224: rename d1/d4/d9/f19 to d1/d25/f4c 0 2024-04-22T15:18:41.358 INFO:tasks.workunit.client.0.smithi096.stdout:4/285: dread d5/d7/d2c/d3d/d4f/f5d [0,4194304] 0 2024-04-22T15:18:41.358 INFO:tasks.workunit.client.0.smithi096.stdout:1/448: creat d6/f8f x:0 0 0 2024-04-22T15:18:41.359 INFO:tasks.workunit.client.0.smithi096.stdout:3/193: dwrite da/db/de/d19/f30 [0,4194304] 0 2024-04-22T15:18:41.365 INFO:tasks.workunit.client.0.smithi096.stdout:5/295: creat d4/d11/f62 x:0 0 0 2024-04-22T15:18:41.365 INFO:tasks.workunit.client.0.smithi096.stdout:2/295: dwrite d3/d1f/f56 [0,4194304] 0 2024-04-22T15:18:41.365 INFO:tasks.workunit.client.0.smithi096.stdout:4/286: readlink d5/d7/d1f/l53 0 2024-04-22T15:18:41.367 INFO:tasks.workunit.client.0.smithi096.stdout:4/287: dread - d5/d7/d1f/f50 zero size 2024-04-22T15:18:41.369 INFO:tasks.workunit.client.0.smithi096.stdout:3/194: rename da/db/dc/d1f to da/db/de/d19/d3e 0 2024-04-22T15:18:41.373 INFO:tasks.workunit.client.0.smithi096.stdout:2/296: dwrite f0 [0,4194304] 0 2024-04-22T15:18:41.375 INFO:tasks.workunit.client.0.smithi096.stdout:2/297: write d3/f49 [3345972,75550] 0 2024-04-22T15:18:41.376 INFO:tasks.workunit.client.0.smithi096.stdout:4/288: creat d5/d7/d1f/f62 x:0 0 0 2024-04-22T15:18:41.378 INFO:tasks.workunit.client.0.smithi096.stdout:5/296: rename d4/d2c/d2f/f3f to d4/d11/d3d/d35/f63 0 2024-04-22T15:18:41.378 INFO:tasks.workunit.client.0.smithi096.stdout:4/289: write d5/d7/f36 [1465534,38734] 0 2024-04-22T15:18:41.380 INFO:tasks.workunit.client.0.smithi096.stdout:3/195: dwrite da/db/de/d19/d3e/f29 [0,4194304] 0 2024-04-22T15:18:41.382 INFO:tasks.workunit.client.0.smithi096.stdout:9/225: rmdir d1/d4 39 2024-04-22T15:18:41.391 INFO:tasks.workunit.client.0.smithi096.stdout:4/290: mkdir d5/d7/d2c/d3d/d4f/d63 0 2024-04-22T15:18:41.393 INFO:tasks.workunit.client.0.smithi096.stdout:9/226: read d1/d4/f42 [50916,17299] 0 2024-04-22T15:18:41.394 INFO:tasks.workunit.client.0.smithi096.stdout:1/449: getdents d6/dc/d8a/d3f/d62/d7d 0 2024-04-22T15:18:41.395 INFO:tasks.workunit.client.0.smithi096.stdout:9/227: write d1/df/f14 [1176565,20166] 0 2024-04-22T15:18:41.395 INFO:tasks.workunit.client.0.smithi096.stdout:5/297: symlink d4/d11/d3d/d35/d57/l64 0 2024-04-22T15:18:41.397 INFO:tasks.workunit.client.0.smithi096.stdout:3/196: mkdir da/d3f 0 2024-04-22T15:18:41.398 INFO:tasks.workunit.client.0.smithi096.stdout:4/291: write d5/d7/d2c/f2e [4578224,125710] 0 2024-04-22T15:18:41.402 INFO:tasks.workunit.client.0.smithi096.stdout:4/292: dread f2 [4194304,4194304] 0 2024-04-22T15:18:41.402 INFO:tasks.workunit.client.0.smithi096.stdout:1/450: symlink d6/dc/d8a/d3f/d62/d7d/l90 0 2024-04-22T15:18:41.403 INFO:tasks.workunit.client.0.smithi096.stdout:5/298: symlink d4/d11/d3d/d35/d57/l65 0 2024-04-22T15:18:41.406 INFO:tasks.workunit.client.0.smithi096.stdout:3/197: creat da/db/de/f40 x:0 0 0 2024-04-22T15:18:41.407 INFO:tasks.workunit.client.0.smithi096.stdout:2/298: write d3/db/f31 [357654,17290] 0 2024-04-22T15:18:41.410 INFO:tasks.workunit.client.0.smithi096.stdout:9/228: symlink d1/l4d 0 2024-04-22T15:18:41.411 INFO:tasks.workunit.client.0.smithi096.stdout:1/451: symlink d6/dc/d8a/d3f/d54/d32/d3b/d3d/l91 0 2024-04-22T15:18:41.412 INFO:tasks.workunit.client.0.smithi096.stdout:5/299: dwrite d4/d11/d3d/d4d/d59/d5a/f13 [0,4194304] 0 2024-04-22T15:18:41.412 INFO:tasks.workunit.client.0.smithi096.stdout:1/452: chown d6/dc/d8a 175205 1 2024-04-22T15:18:41.413 INFO:tasks.workunit.client.0.smithi096.stdout:5/300: fsync d4/f6 0 2024-04-22T15:18:41.415 INFO:tasks.workunit.client.0.smithi096.stdout:3/198: mknod da/db/d12/c41 0 2024-04-22T15:18:41.416 INFO:tasks.workunit.client.0.smithi096.stdout:4/293: creat d5/d7/d1a/d61/f64 x:0 0 0 2024-04-22T15:18:41.417 INFO:tasks.workunit.client.0.smithi096.stdout:3/199: write da/db/de/d19/d2d/f2e [2349114,69247] 0 2024-04-22T15:18:41.420 INFO:tasks.workunit.client.0.smithi096.stdout:5/301: dwrite d4/fd [0,4194304] 0 2024-04-22T15:18:41.421 INFO:tasks.workunit.client.0.smithi096.stdout:3/200: write da/db/de/d19/f30 [3151505,120184] 0 2024-04-22T15:18:41.421 INFO:tasks.workunit.client.0.smithi096.stdout:3/201: readlink da/l1c 0 2024-04-22T15:18:41.422 INFO:tasks.workunit.client.0.smithi096.stdout:4/294: creat d5/d7/d2c/f65 x:0 0 0 2024-04-22T15:18:41.423 INFO:tasks.workunit.client.0.smithi096.stdout:5/302: stat d4/d11/d25/f38 0 2024-04-22T15:18:41.424 INFO:tasks.workunit.client.0.smithi096.stdout:4/295: chown d5/d7/f9 24738381 1 2024-04-22T15:18:41.432 INFO:tasks.workunit.client.0.smithi096.stdout:4/296: unlink d5/d7/f52 0 2024-04-22T15:18:41.433 INFO:tasks.workunit.client.0.smithi096.stdout:3/202: dwrite da/db/de/f40 [0,4194304] 0 2024-04-22T15:18:41.435 INFO:tasks.workunit.client.0.smithi096.stdout:5/303: dwrite d4/d11/d25/f38 [0,4194304] 0 2024-04-22T15:18:41.435 INFO:tasks.workunit.client.0.smithi096.stdout:3/203: write da/db/d12/d22/f3a [650473,81204] 0 2024-04-22T15:18:41.441 INFO:tasks.workunit.client.0.smithi096.stdout:3/204: creat da/db/de/d19/d3e/f42 x:0 0 0 2024-04-22T15:18:41.442 INFO:tasks.workunit.client.0.smithi096.stdout:5/304: mknod d4/d2c/d2f/c66 0 2024-04-22T15:18:41.443 INFO:tasks.workunit.client.0.smithi096.stdout:3/205: readlink da/db/l36 0 2024-04-22T15:18:41.445 INFO:tasks.workunit.client.0.smithi096.stdout:2/299: truncate d3/db/df/d10/f11 1979344 0 2024-04-22T15:18:41.445 INFO:tasks.workunit.client.0.smithi096.stdout:3/206: write da/db/de/d19/f2a [1509955,91940] 0 2024-04-22T15:18:41.446 INFO:tasks.workunit.client.0.smithi096.stdout:5/305: fsync d4/d11/d3d/d4d/d59/d5a/f15 0 2024-04-22T15:18:41.447 INFO:tasks.workunit.client.0.smithi096.stdout:2/300: dread - d3/d1f/f51 zero size 2024-04-22T15:18:41.448 INFO:tasks.workunit.client.0.smithi096.stdout:2/301: fsync d3/d1f/d2f/f41 0 2024-04-22T15:18:41.450 INFO:tasks.workunit.client.0.smithi096.stdout:9/229: truncate d1/d4/d9/dd/f23 1384907 0 2024-04-22T15:18:41.452 INFO:tasks.workunit.client.0.smithi096.stdout:5/306: creat d4/d11/d3d/d4d/d59/d5a/f67 x:0 0 0 2024-04-22T15:18:41.456 INFO:tasks.workunit.client.0.smithi096.stdout:2/302: mkdir d3/d1f/d5a 0 2024-04-22T15:18:41.457 INFO:tasks.workunit.client.0.smithi096.stdout:3/207: truncate da/db/de/d19/d3e/f15 3646054 0 2024-04-22T15:18:41.457 INFO:tasks.workunit.client.0.smithi096.stdout:3/208: chown da/l1c 106 1 2024-04-22T15:18:41.458 INFO:tasks.workunit.client.0.smithi096.stdout:3/209: stat da/db/l36 0 2024-04-22T15:18:41.462 INFO:tasks.workunit.client.0.smithi096.stdout:2/303: fsync d3/db/d30/f3e 0 2024-04-22T15:18:41.463 INFO:tasks.workunit.client.0.smithi096.stdout:9/230: dread d1/f32 [0,4194304] 0 2024-04-22T15:18:41.468 INFO:tasks.workunit.client.0.smithi096.stdout:3/210: truncate da/f27 3848506 0 2024-04-22T15:18:41.472 INFO:tasks.workunit.client.0.smithi096.stdout:9/231: creat d1/d4/f4e x:0 0 0 2024-04-22T15:18:41.473 INFO:tasks.workunit.client.0.smithi096.stdout:3/211: dwrite da/db/de/d19/f2a [0,4194304] 0 2024-04-22T15:18:41.474 INFO:tasks.workunit.client.0.smithi096.stdout:9/232: write d1/d25/f48 [72103,4283] 0 2024-04-22T15:18:41.475 INFO:tasks.workunit.client.0.smithi096.stdout:9/233: stat d1/la 0 2024-04-22T15:18:41.478 INFO:tasks.workunit.client.0.smithi096.stdout:2/304: getdents d3/db/df/d24/d58 0 2024-04-22T15:18:41.478 INFO:tasks.workunit.client.0.smithi096.stdout:5/307: getdents d4/d11/d25 0 2024-04-22T15:18:41.482 INFO:tasks.workunit.client.0.smithi096.stdout:9/234: unlink d1/d4/d9/f2d 0 2024-04-22T15:18:41.484 INFO:tasks.workunit.client.0.smithi096.stdout:9/235: truncate d1/d4/d9/dd/f2b 447120 0 2024-04-22T15:18:41.485 INFO:tasks.workunit.client.0.smithi096.stdout:5/308: dwrite d4/d2c/d2f/f51 [0,4194304] 0 2024-04-22T15:18:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:41 smithi096 ceph-mon[26688]: Upgrade: Updating mgr.smithi096.hhkvum 2024-04-22T15:18:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:41 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:18:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:41 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi096.hhkvum", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:18:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:41 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi096.hhkvum", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:18:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:41 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:18:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:41 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:18:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:41 smithi096 ceph-mon[26688]: Deploying daemon mgr.smithi096.hhkvum on smithi096 2024-04-22T15:18:41.487 INFO:tasks.workunit.client.0.smithi096.stdout:9/236: dread - d1/d25/f33 zero size 2024-04-22T15:18:41.487 INFO:tasks.workunit.client.0.smithi096.stdout:2/305: symlink d3/d1f/d2f/l5b 0 2024-04-22T15:18:41.534 INFO:tasks.workunit.client.0.smithi096.stdout:8/264: sync 2024-04-22T15:18:41.534 INFO:tasks.workunit.client.0.smithi096.stdout:0/281: sync 2024-04-22T15:18:41.534 INFO:tasks.workunit.client.0.smithi096.stdout:5/309: rmdir d4/d11/d3d 39 2024-04-22T15:18:41.535 INFO:tasks.workunit.client.0.smithi096.stdout:9/237: unlink d1/d25/f49 0 2024-04-22T15:18:41.535 INFO:tasks.workunit.client.0.smithi096.stdout:2/306: mkdir d3/db/df/d24/d5c 0 2024-04-22T15:18:41.539 INFO:tasks.workunit.client.0.smithi096.stdout:0/282: chown d7/d9/f40 22 1 2024-04-22T15:18:41.541 INFO:tasks.workunit.client.0.smithi096.stdout:5/310: readlink d4/d11/d3d/d35/d57/l64 0 2024-04-22T15:18:41.541 INFO:tasks.workunit.client.0.smithi096.stdout:2/307: creat d3/d1f/d2f/f5d x:0 0 0 2024-04-22T15:18:41.541 INFO:tasks.workunit.client.0.smithi096.stdout:3/212: dwrite da/f34 [0,4194304] 0 2024-04-22T15:18:41.543 INFO:tasks.workunit.client.0.smithi096.stdout:9/238: truncate d1/d4/f22 18168 0 2024-04-22T15:18:41.546 INFO:tasks.workunit.client.0.smithi096.stdout:3/213: write da/db/de/d19/f30 [3203322,37386] 0 2024-04-22T15:18:41.548 INFO:tasks.workunit.client.0.smithi096.stdout:8/265: dwrite d4/d10/d25/f44 [0,4194304] 0 2024-04-22T15:18:41.551 INFO:tasks.workunit.client.0.smithi096.stdout:2/308: truncate f0 1383760 0 2024-04-22T15:18:41.557 INFO:tasks.workunit.client.0.smithi096.stdout:8/266: dwrite d4/d10/d25/f44 [0,4194304] 0 2024-04-22T15:18:41.559 INFO:tasks.workunit.client.0.smithi096.stdout:8/267: write d4/d10/d25/f44 [582510,45719] 0 2024-04-22T15:18:41.560 INFO:tasks.workunit.client.0.smithi096.stdout:5/311: rename d4/f43 to d4/d2c/d2f/f68 0 2024-04-22T15:18:41.561 INFO:tasks.workunit.client.0.smithi096.stdout:8/268: write d4/d10/f4e [377688,84369] 0 2024-04-22T15:18:41.562 INFO:tasks.workunit.client.0.smithi096.stdout:0/283: getdents d7/d9/d59 0 2024-04-22T15:18:41.563 INFO:tasks.workunit.client.0.smithi096.stdout:3/214: link c5 da/db/de/c43 0 2024-04-22T15:18:41.565 INFO:tasks.workunit.client.0.smithi096.stdout:3/215: write da/db/de/f40 [207712,126921] 0 2024-04-22T15:18:41.568 INFO:tasks.workunit.client.0.smithi096.stdout:9/239: getdents d1/d4 0 2024-04-22T15:18:41.569 INFO:tasks.workunit.client.0.smithi096.stdout:5/312: unlink d4/d11/d3d/d4d/d59/d5a/d23/l2b 0 2024-04-22T15:18:41.574 INFO:tasks.workunit.client.0.smithi096.stdout:8/269: mknod d4/d1c/d40/c5b 0 2024-04-22T15:18:41.576 INFO:tasks.workunit.client.0.smithi096.stdout:8/270: chown d4/d10/l11 2363259 1 2024-04-22T15:18:41.577 INFO:tasks.workunit.client.0.smithi096.stdout:8/271: chown d4/d1c 767 1 2024-04-22T15:18:41.577 INFO:tasks.workunit.client.0.smithi096.stdout:5/313: dwrite d4/f6 [4194304,4194304] 0 2024-04-22T15:18:41.581 INFO:tasks.workunit.client.0.smithi096.stdout:0/284: symlink d7/d9/dc/d26/l5f 0 2024-04-22T15:18:41.585 INFO:tasks.workunit.client.0.smithi096.stdout:8/272: write d4/f12 [3959225,87087] 0 2024-04-22T15:18:41.586 INFO:tasks.workunit.client.0.smithi096.stdout:2/309: dread d3/db/df/d10/f18 [0,4194304] 0 2024-04-22T15:18:41.588 INFO:tasks.workunit.client.0.smithi096.stdout:5/314: write d4/d2c/d2f/d31/f3b [5404448,99244] 0 2024-04-22T15:18:41.598 INFO:tasks.workunit.client.0.smithi096.stdout:9/240: creat d1/f4f x:0 0 0 2024-04-22T15:18:41.599 INFO:tasks.workunit.client.0.smithi096.stdout:9/241: dread - d1/d25/f33 zero size 2024-04-22T15:18:41.601 INFO:tasks.workunit.client.0.smithi096.stdout:0/285: write d7/d9/dc/d16/d18/f2b [1737479,95411] 0 2024-04-22T15:18:41.602 INFO:tasks.workunit.client.0.smithi096.stdout:2/310: mkdir d3/db/df/d24/d58/d5e 0 2024-04-22T15:18:41.606 INFO:tasks.workunit.client.0.smithi096.stdout:9/242: dwrite d1/d4/f47 [0,4194304] 0 2024-04-22T15:18:41.606 INFO:tasks.workunit.client.0.smithi096.stdout:8/273: link d4/d1c/d34/f3e d4/d1c/f5c 0 2024-04-22T15:18:41.610 INFO:tasks.workunit.client.0.smithi096.stdout:0/286: chown d7/f32 90 1 2024-04-22T15:18:41.612 INFO:tasks.workunit.client.0.smithi096.stdout:2/311: mkdir d3/db/df/d24/d58/d5f 0 2024-04-22T15:18:41.616 INFO:tasks.workunit.client.0.smithi096.stdout:0/287: chown d7/d42/d4c/d55 19855 1 2024-04-22T15:18:41.616 INFO:tasks.workunit.client.0.smithi096.stdout:9/243: dwrite d1/d4/d9/f3b [0,4194304] 0 2024-04-22T15:18:41.618 INFO:tasks.workunit.client.0.smithi096.stdout:5/315: truncate f1 3716030 0 2024-04-22T15:18:41.621 INFO:tasks.workunit.client.0.smithi096.stdout:2/312: creat d3/db/df/f60 x:0 0 0 2024-04-22T15:18:41.622 INFO:tasks.workunit.client.0.smithi096.stdout:2/313: dread - d3/f53 zero size 2024-04-22T15:18:41.622 INFO:tasks.workunit.client.0.smithi096.stdout:2/314: readlink d3/l5 0 2024-04-22T15:18:41.625 INFO:tasks.workunit.client.0.smithi096.stdout:8/274: link d4/d1c/d34/f3e d4/d10/d25/d4c/f5d 0 2024-04-22T15:18:41.630 INFO:tasks.workunit.client.0.smithi096.stdout:5/316: write d4/d11/d3d/d35/f63 [177246,23277] 0 2024-04-22T15:18:41.632 INFO:tasks.workunit.client.0.smithi096.stdout:2/315: creat d3/db/df/f61 x:0 0 0 2024-04-22T15:18:41.634 INFO:tasks.workunit.client.0.smithi096.stdout:2/316: write d3/db/df/d10/f3b [559643,72815] 0 2024-04-22T15:18:41.635 INFO:tasks.workunit.client.0.smithi096.stdout:0/288: rename d7/de/lf to d7/d9/dc/d16/l60 0 2024-04-22T15:18:41.636 INFO:tasks.workunit.client.0.smithi096.stdout:9/244: dwrite d1/d25/f4c [0,4194304] 0 2024-04-22T15:18:41.637 INFO:tasks.workunit.client.0.smithi096.stdout:9/245: stat d1/d4/d9/dd/f1d 0 2024-04-22T15:18:41.640 INFO:tasks.workunit.client.0.smithi096.stdout:9/246: stat d1/df/c37 0 2024-04-22T15:18:41.645 INFO:tasks.workunit.client.0.smithi096.stdout:2/317: dread d3/d1f/d2f/f34 [0,4194304] 0 2024-04-22T15:18:41.646 INFO:tasks.workunit.client.0.smithi096.stdout:0/289: dwrite d7/f4d [0,4194304] 0 2024-04-22T15:18:41.647 INFO:tasks.workunit.client.0.smithi096.stdout:8/275: dread d4/d10/f19 [0,4194304] 0 2024-04-22T15:18:41.648 INFO:tasks.workunit.client.0.smithi096.stdout:9/247: creat d1/f50 x:0 0 0 2024-04-22T15:18:41.650 INFO:tasks.workunit.client.0.smithi096.stdout:2/318: mknod d3/db/df/c62 0 2024-04-22T15:18:41.651 INFO:tasks.workunit.client.0.smithi096.stdout:0/290: read d7/d9/dc/f13 [8886437,27712] 0 2024-04-22T15:18:41.652 INFO:tasks.workunit.client.0.smithi096.stdout:8/276: creat d4/d10/d25/d4c/f5e x:0 0 0 2024-04-22T15:18:41.655 INFO:tasks.workunit.client.0.smithi096.stdout:8/277: truncate d4/d1c/d34/f49 687797 0 2024-04-22T15:18:41.655 INFO:tasks.workunit.client.0.smithi096.stdout:9/248: rename d1/d4/d9/c3d to d1/d4/d9/c51 0 2024-04-22T15:18:41.656 INFO:tasks.workunit.client.0.smithi096.stdout:0/291: mkdir d7/de/d3a/d61 0 2024-04-22T15:18:41.657 INFO:tasks.workunit.client.0.smithi096.stdout:0/292: dread - d7/d9/f45 zero size 2024-04-22T15:18:41.659 INFO:tasks.workunit.client.0.smithi096.stdout:9/249: creat d1/d4/d9/f52 x:0 0 0 2024-04-22T15:18:41.660 INFO:tasks.workunit.client.0.smithi096.stdout:0/293: mkdir d7/de/d5b/d62 0 2024-04-22T15:18:41.663 INFO:tasks.workunit.client.0.smithi096.stdout:0/294: getdents d7/d9 0 2024-04-22T15:18:41.665 INFO:tasks.workunit.client.0.smithi096.stdout:0/295: creat d7/de/d5b/d62/f63 x:0 0 0 2024-04-22T15:18:41.666 INFO:tasks.workunit.client.0.smithi096.stdout:0/296: mkdir d7/de/d5b/d64 0 2024-04-22T15:18:41.671 INFO:tasks.workunit.client.0.smithi096.stdout:2/319: dread d3/db/f12 [0,4194304] 0 2024-04-22T15:18:41.672 INFO:tasks.workunit.client.0.smithi096.stdout:2/320: chown d3/d1f/d2f/f34 3279 1 2024-04-22T15:18:41.675 INFO:tasks.workunit.client.0.smithi096.stdout:2/321: symlink d3/db/df/d24/d58/d5e/l63 0 2024-04-22T15:18:41.676 INFO:tasks.workunit.client.0.smithi096.stdout:2/322: dread - d3/db/df/f60 zero size 2024-04-22T15:18:41.679 INFO:tasks.workunit.client.0.smithi096.stdout:2/323: symlink d3/db/d2d/d46/d59/l64 0 2024-04-22T15:18:41.679 INFO:tasks.workunit.client.0.smithi096.stdout:0/297: dread d7/d9/dc/f13 [4194304,4194304] 0 2024-04-22T15:18:41.680 INFO:tasks.workunit.client.0.smithi096.stdout:2/324: write d3/d1f/d2f/f41 [1357529,105773] 0 2024-04-22T15:18:41.680 INFO:tasks.workunit.client.0.smithi096.stdout:0/298: readlink d7/d9/d15/l21 0 2024-04-22T15:18:41.682 INFO:tasks.workunit.client.0.smithi096.stdout:0/299: mkdir d7/d9/dc/d16/d65 0 2024-04-22T15:18:41.683 INFO:tasks.workunit.client.0.smithi096.stdout:2/325: mkdir d3/db/df/d24/d65 0 2024-04-22T15:18:41.683 INFO:tasks.workunit.client.0.smithi096.stdout:2/326: chown d3/d1f 39918061 1 2024-04-22T15:18:41.685 INFO:tasks.workunit.client.0.smithi096.stdout:2/327: creat d3/db/df/d24/d65/f66 x:0 0 0 2024-04-22T15:18:41.686 INFO:tasks.workunit.client.0.smithi096.stdout:2/328: creat d3/d1f/d2f/f67 x:0 0 0 2024-04-22T15:18:41.687 INFO:tasks.workunit.client.0.smithi096.stdout:2/329: stat d3/db 0 2024-04-22T15:18:41.687 INFO:tasks.workunit.client.0.smithi096.stdout:2/330: dread - d3/db/df/f61 zero size 2024-04-22T15:18:41.688 INFO:tasks.workunit.client.0.smithi096.stdout:2/331: mknod d3/d1f/c68 0 2024-04-22T15:18:41.719 INFO:tasks.workunit.client.0.smithi096.stdout:8/278: write d4/d10/d25/f26 [424636,69929] 0 2024-04-22T15:18:41.724 INFO:tasks.workunit.client.0.smithi096.stdout:8/279: dwrite d4/f12 [0,4194304] 0 2024-04-22T15:18:41.728 INFO:tasks.workunit.client.0.smithi096.stdout:8/280: symlink d4/d10/d4d/l5f 0 2024-04-22T15:18:41.733 INFO:tasks.workunit.client.0.smithi096.stdout:0/300: rmdir d7/d9/dc/d16 39 2024-04-22T15:18:41.738 INFO:tasks.workunit.client.0.smithi096.stdout:2/332: rename d3/db/df/d24/d65 to d3/db/d2d/d46/d69 0 2024-04-22T15:18:41.760 INFO:tasks.workunit.client.0.smithi096.stdout:8/281: write d4/d2a/f36 [533626,46506] 0 2024-04-22T15:18:41.761 INFO:tasks.workunit.client.0.smithi096.stdout:2/333: mknod d3/db/d2d/d46/d59/c6a 0 2024-04-22T15:18:41.762 INFO:tasks.workunit.client.0.smithi096.stdout:2/334: chown d3/db/d2d/d46/l50 53 1 2024-04-22T15:18:41.764 INFO:tasks.workunit.client.0.smithi096.stdout:2/335: chown d3/db/df/l3c 909 1 2024-04-22T15:18:41.767 INFO:tasks.workunit.client.0.smithi096.stdout:8/282: dwrite d4/d10/d25/d4c/f5e [0,4194304] 0 2024-04-22T15:18:41.773 INFO:tasks.workunit.client.0.smithi096.stdout:2/336: link d3/db/d30/f3e d3/db/df/d24/d58/d5e/f6b 0 2024-04-22T15:18:41.774 INFO:tasks.workunit.client.0.smithi096.stdout:2/337: dread - d3/db/df/f60 zero size 2024-04-22T15:18:41.774 INFO:tasks.workunit.client.0.smithi096.stdout:0/301: dread d7/d9/dc/f14 [0,4194304] 0 2024-04-22T15:18:41.775 INFO:tasks.workunit.client.0.smithi096.stdout:2/338: fdatasync d3/db/d2d/d46/d69/f66 0 2024-04-22T15:18:41.776 INFO:tasks.workunit.client.0.smithi096.stdout:8/283: creat d4/d1c/d40/d46/d58/f60 x:0 0 0 2024-04-22T15:18:41.776 INFO:tasks.workunit.client.0.smithi096.stdout:0/302: write d7/d9/d15/d35/f47 [2753265,62604] 0 2024-04-22T15:18:41.777 INFO:tasks.workunit.client.0.smithi096.stdout:8/284: readlink d4/d2a/l2b 0 2024-04-22T15:18:41.777 INFO:tasks.workunit.client.0.smithi096.stdout:2/339: read - d3/db/d2d/d46/d69/f66 zero size 2024-04-22T15:18:41.779 INFO:tasks.workunit.client.0.smithi096.stdout:8/285: mknod d4/c61 0 2024-04-22T15:18:41.783 INFO:tasks.workunit.client.0.smithi096.stdout:8/286: write d4/d1c/f5c [731920,96672] 0 2024-04-22T15:18:41.784 INFO:tasks.workunit.client.0.smithi096.stdout:0/303: dwrite d7/d9/f40 [0,4194304] 0 2024-04-22T15:18:41.788 INFO:tasks.workunit.client.0.smithi096.stdout:0/304: mkdir d7/d9/d66 0 2024-04-22T15:18:41.788 INFO:tasks.workunit.client.0.smithi096.stdout:2/340: dread d3/f2e [0,4194304] 0 2024-04-22T15:18:41.789 INFO:tasks.workunit.client.0.smithi096.stdout:2/341: dread - d3/d1f/d2f/f5d zero size 2024-04-22T15:18:41.790 INFO:tasks.workunit.client.0.smithi096.stdout:8/287: mkdir d4/d1c/d62 0 2024-04-22T15:18:41.792 INFO:tasks.workunit.client.0.smithi096.stdout:0/305: mkdir d7/de/d5b/d62/d67 0 2024-04-22T15:18:41.797 INFO:tasks.workunit.client.0.smithi096.stdout:8/288: dwrite d4/d1c/d34/f3e [0,4194304] 0 2024-04-22T15:18:41.799 INFO:tasks.workunit.client.0.smithi096.stdout:8/289: fdatasync d4/d2a/f36 0 2024-04-22T15:18:41.800 INFO:tasks.workunit.client.0.smithi096.stdout:2/342: dwrite d3/db/df/d10/f37 [0,4194304] 0 2024-04-22T15:18:41.805 INFO:tasks.workunit.client.0.smithi096.stdout:8/290: creat d4/d1c/d40/d48/f63 x:0 0 0 2024-04-22T15:18:41.813 INFO:tasks.workunit.client.0.smithi096.stdout:2/343: dwrite d3/d1f/d2f/f4e [0,4194304] 0 2024-04-22T15:18:41.815 INFO:tasks.workunit.client.0.smithi096.stdout:0/306: write d7/d9/dc/f14 [3214166,20743] 0 2024-04-22T15:18:41.819 INFO:tasks.workunit.client.0.smithi096.stdout:2/344: creat d3/db/d2d/d46/d69/f6c x:0 0 0 2024-04-22T15:18:41.823 INFO:tasks.workunit.client.0.smithi096.stdout:0/307: creat d7/d9/dc/d16/d18/f68 x:0 0 0 2024-04-22T15:18:41.825 INFO:tasks.workunit.client.0.smithi096.stdout:8/291: dread d4/d10/d25/f2e [0,4194304] 0 2024-04-22T15:18:41.825 INFO:tasks.workunit.client.0.smithi096.stdout:2/345: dread d3/d1f/d2f/f34 [0,4194304] 0 2024-04-22T15:18:41.825 INFO:tasks.workunit.client.0.smithi096.stdout:0/308: creat d7/d9/dc/d16/d18/f69 x:0 0 0 2024-04-22T15:18:41.826 INFO:tasks.workunit.client.0.smithi096.stdout:0/309: chown d7/de/d5b/d64 6 1 2024-04-22T15:18:41.827 INFO:tasks.workunit.client.0.smithi096.stdout:8/292: symlink d4/l64 0 2024-04-22T15:18:41.827 INFO:tasks.workunit.client.0.smithi096.stdout:0/310: readlink d7/d9/dc/d26/l53 0 2024-04-22T15:18:41.828 INFO:tasks.workunit.client.0.smithi096.stdout:0/311: fsync d7/d9/dc/f2f 0 2024-04-22T15:18:41.828 INFO:tasks.workunit.client.0.smithi096.stdout:0/312: stat d7/de/ld 0 2024-04-22T15:18:41.830 INFO:tasks.workunit.client.0.smithi096.stdout:8/293: mknod d4/d1c/d40/c65 0 2024-04-22T15:18:41.834 INFO:tasks.workunit.client.0.smithi096.stdout:2/346: getdents d3/db/df/d24/d58/d5e 0 2024-04-22T15:18:41.836 INFO:tasks.workunit.client.0.smithi096.stdout:2/347: mknod d3/d1f/c6d 0 2024-04-22T15:18:41.840 INFO:tasks.workunit.client.0.smithi096.stdout:2/348: truncate d3/db/df/d10/f21 3711858 0 2024-04-22T15:18:41.840 INFO:tasks.workunit.client.0.smithi096.stdout:2/349: chown d3/db/df/d10/l16 176 1 2024-04-22T15:18:41.843 INFO:tasks.workunit.client.0.smithi096.stdout:0/313: write d7/d9/f24 [783828,121104] 0 2024-04-22T15:18:41.844 INFO:tasks.workunit.client.0.smithi096.stdout:2/350: mkdir d3/db/df/d24/d58/d5f/d6e 0 2024-04-22T15:18:41.846 INFO:tasks.workunit.client.0.smithi096.stdout:2/351: creat d3/db/d2d/d46/d69/f6f x:0 0 0 2024-04-22T15:18:41.847 INFO:tasks.workunit.client.0.smithi096.stdout:0/314: dread d7/d9/dc/f13 [4194304,4194304] 0 2024-04-22T15:18:41.850 INFO:tasks.workunit.client.0.smithi096.stdout:2/352: creat d3/f70 x:0 0 0 2024-04-22T15:18:41.854 INFO:tasks.workunit.client.0.smithi096.stdout:0/315: dwrite d7/d9/d15/f1a [0,4194304] 0 2024-04-22T15:18:41.856 INFO:tasks.workunit.client.0.smithi096.stdout:2/353: dread d3/db/df/d10/f3b [0,4194304] 0 2024-04-22T15:18:41.860 INFO:tasks.workunit.client.0.smithi096.stdout:2/354: dread d3/d1f/d2f/f47 [0,4194304] 0 2024-04-22T15:18:41.861 INFO:tasks.workunit.client.0.smithi096.stdout:0/316: dread d7/d9/d15/d35/f41 [0,4194304] 0 2024-04-22T15:18:41.862 INFO:tasks.workunit.client.0.smithi096.stdout:0/317: readlink d7/d9/dc/d26/l53 0 2024-04-22T15:18:41.863 INFO:tasks.workunit.client.0.smithi096.stdout:2/355: dread d3/db/df/d10/f1b [0,4194304] 0 2024-04-22T15:18:41.866 INFO:tasks.workunit.client.0.smithi096.stdout:0/318: dread d7/d9/f24 [0,4194304] 0 2024-04-22T15:18:41.868 INFO:tasks.workunit.client.0.smithi096.stdout:2/356: rename d3/db/df/d24/d5c to d3/db/df/d24/d58/d5f/d6e/d71 0 2024-04-22T15:18:41.868 INFO:tasks.workunit.client.0.smithi096.stdout:2/357: dread - d3/db/df/f61 zero size 2024-04-22T15:18:41.869 INFO:tasks.workunit.client.0.smithi096.stdout:0/319: creat d7/d42/d4c/f6a x:0 0 0 2024-04-22T15:18:41.871 INFO:tasks.workunit.client.0.smithi096.stdout:0/320: truncate d7/d9/dc/d16/f1f 1257982 0 2024-04-22T15:18:41.872 INFO:tasks.workunit.client.0.smithi096.stdout:2/358: truncate d3/db/d30/f3e 361807 0 2024-04-22T15:18:41.875 INFO:tasks.workunit.client.0.smithi096.stdout:2/359: symlink d3/l72 0 2024-04-22T15:18:41.878 INFO:tasks.workunit.client.0.smithi096.stdout:2/360: truncate d3/d1f/d2f/f47 1659056 0 2024-04-22T15:18:41.974 INFO:tasks.workunit.client.0.smithi096.stdout:2/361: dwrite d3/f70 [0,4194304] 0 2024-04-22T15:18:41.976 INFO:tasks.workunit.client.0.smithi096.stdout:0/321: dwrite d7/d9/dc/d16/d18/f23 [0,4194304] 0 2024-04-22T15:18:41.987 INFO:tasks.workunit.client.0.smithi096.stdout:7/404: sync 2024-04-22T15:18:41.987 INFO:tasks.workunit.client.0.smithi096.stdout:6/256: sync 2024-04-22T15:18:41.989 INFO:tasks.workunit.client.0.smithi096.stdout:0/322: getdents d7/d42/d4c/d55 0 2024-04-22T15:18:41.992 INFO:tasks.workunit.client.0.smithi096.stdout:2/362: dwrite d3/db/d2d/d46/d69/f66 [0,4194304] 0 2024-04-22T15:18:41.993 INFO:tasks.workunit.client.0.smithi096.stdout:0/323: truncate d7/d9/dc/d16/d18/f54 906477 0 2024-04-22T15:18:41.998 INFO:tasks.workunit.client.0.smithi096.stdout:0/324: dwrite d7/d9/dc/d16/d18/f3d [0,4194304] 0 2024-04-22T15:18:42.078 INFO:tasks.workunit.client.0.smithi096.stdout:6/257: rmdir d3/d28/dd/d11/d2c/d3b 39 2024-04-22T15:18:42.106 INFO:tasks.workunit.client.0.smithi096.stdout:6/258: dread - d3/d28/dd/d11/d2c/d3b/f48 zero size 2024-04-22T15:18:42.107 INFO:tasks.workunit.client.0.smithi096.stdout:6/259: write d3/d28/dd/d11/f1e [3914700,76998] 0 2024-04-22T15:18:42.109 INFO:tasks.workunit.client.0.smithi096.stdout:2/363: dread d3/db/df/d24/d58/d5e/f6b [0,4194304] 0 2024-04-22T15:18:42.113 INFO:tasks.workunit.client.0.smithi096.stdout:7/405: dwrite fa [4194304,4194304] 0 2024-04-22T15:18:42.113 INFO:tasks.workunit.client.0.smithi096.stdout:6/260: write d3/f29 [411719,84898] 0 2024-04-22T15:18:42.115 INFO:tasks.workunit.client.0.smithi096.stdout:6/261: chown d3/d28/dd/d11/d2c/d44/l57 29 1 2024-04-22T15:18:42.117 INFO:tasks.workunit.client.0.smithi096.stdout:2/364: symlink d3/db/d2d/l73 0 2024-04-22T15:18:42.118 INFO:tasks.workunit.client.0.smithi096.stdout:2/365: chown d3/db/d2d/d46/l4c 27761867 1 2024-04-22T15:18:42.119 INFO:tasks.workunit.client.0.smithi096.stdout:6/262: unlink d3/d28/dd/d11/c43 0 2024-04-22T15:18:42.121 INFO:tasks.workunit.client.0.smithi096.stdout:7/406: dwrite d31/d32/f59 [0,4194304] 0 2024-04-22T15:18:42.127 INFO:tasks.workunit.client.0.smithi096.stdout:2/366: dread d3/db/d2d/d46/d69/f66 [0,4194304] 0 2024-04-22T15:18:42.130 INFO:tasks.workunit.client.0.smithi096.stdout:6/263: dwrite d3/d28/dd/d11/d2c/d44/f53 [0,4194304] 0 2024-04-22T15:18:42.131 INFO:tasks.workunit.client.0.smithi096.stdout:7/407: dwrite d31/d32/f55 [0,4194304] 0 2024-04-22T15:18:42.136 INFO:tasks.workunit.client.0.smithi096.stdout:6/264: fsync d3/d3a/f3e 0 2024-04-22T15:18:42.139 INFO:tasks.workunit.client.0.smithi096.stdout:2/367: fdatasync d3/db/f42 0 2024-04-22T15:18:42.140 INFO:tasks.workunit.client.0.smithi096.stdout:2/368: readlink d3/d1f/d2f/l43 0 2024-04-22T15:18:42.141 INFO:tasks.workunit.client.0.smithi096.stdout:2/369: chown d3/db/df/d10/f3b 5 1 2024-04-22T15:18:42.144 INFO:tasks.workunit.client.0.smithi096.stdout:7/408: rename d31/d32/d33/d54/d43/f67 to d31/d32/d33/d44/f80 0 2024-04-22T15:18:42.149 INFO:tasks.workunit.client.0.smithi096.stdout:0/325: truncate d7/d9/dc/d16/d18/f39 282662 0 2024-04-22T15:18:42.154 INFO:tasks.workunit.client.0.smithi096.stdout:6/265: rename d3/d28/dd/d11/d2c/d44/f53 to d3/d3a/d4b/f5b 0 2024-04-22T15:18:42.156 INFO:tasks.workunit.client.0.smithi096.stdout:0/326: mkdir d7/d9/d15/d6b 0 2024-04-22T15:18:42.157 INFO:tasks.workunit.client.0.smithi096.stdout:2/370: mknod d3/c74 0 2024-04-22T15:18:42.163 INFO:tasks.workunit.client.0.smithi096.stdout:0/327: creat d7/d42/d4c/f6c x:0 0 0 2024-04-22T15:18:42.165 INFO:tasks.workunit.client.0.smithi096.stdout:2/371: creat d3/d1f/d2f/f75 x:0 0 0 2024-04-22T15:18:42.166 INFO:tasks.workunit.client.0.smithi096.stdout:2/372: chown d3/db/df/d10/l16 364 1 2024-04-22T15:18:42.166 INFO:tasks.workunit.client.0.smithi096.stdout:2/373: dread - d3/f53 zero size 2024-04-22T15:18:42.171 INFO:tasks.workunit.client.0.smithi096.stdout:0/328: mknod d7/d9/d15/c6d 0 2024-04-22T15:18:42.174 INFO:tasks.workunit.client.0.smithi096.stdout:2/374: creat d3/db/d2d/d46/d59/f76 x:0 0 0 2024-04-22T15:18:42.179 INFO:tasks.workunit.client.0.smithi096.stdout:6/266: rename d3/d3a/d4b/l50 to d3/d28/dd/d11/l5c 0 2024-04-22T15:18:42.181 INFO:tasks.workunit.client.0.smithi096.stdout:2/375: dwrite d3/d1f/d2f/f5d [0,4194304] 0 2024-04-22T15:18:42.182 INFO:tasks.workunit.client.0.smithi096.stdout:6/267: read d3/d28/dd/f19 [822471,125531] 0 2024-04-22T15:18:42.182 INFO:tasks.workunit.client.0.smithi096.stdout:7/409: link d31/l61 d31/d32/d33/d6a/l81 0 2024-04-22T15:18:42.187 INFO:tasks.workunit.client.0.smithi096.stdout:0/329: getdents d7/de/d3a/d61 0 2024-04-22T15:18:42.188 INFO:tasks.workunit.client.0.smithi096.stdout:7/410: dread f18 [0,4194304] 0 2024-04-22T15:18:42.189 INFO:tasks.workunit.client.0.smithi096.stdout:0/330: dread - d7/d9/dc/f56 zero size 2024-04-22T15:18:42.199 INFO:tasks.workunit.client.0.smithi096.stdout:7/411: dwrite d31/d32/d33/d44/f80 [0,4194304] 0 2024-04-22T15:18:42.202 INFO:tasks.workunit.client.0.smithi096.stdout:0/331: mknod d7/d9/dc/d16/d65/c6e 0 2024-04-22T15:18:42.204 INFO:tasks.workunit.client.0.smithi096.stdout:7/412: fdatasync f18 0 2024-04-22T15:18:42.206 INFO:tasks.workunit.client.0.smithi096.stdout:6/268: dread d3/f22 [0,4194304] 0 2024-04-22T15:18:42.212 INFO:tasks.workunit.client.0.smithi096.stdout:6/269: fsync d3/d28/dd/d11/f18 0 2024-04-22T15:18:42.221 INFO:tasks.workunit.client.0.smithi096.stdout:0/332: rmdir d7/d9/d15/d6b 0 2024-04-22T15:18:42.222 INFO:tasks.workunit.client.0.smithi096.stdout:0/333: truncate d7/d9/dc/d16/d18/f54 1316097 0 2024-04-22T15:18:42.223 INFO:tasks.workunit.client.0.smithi096.stdout:0/334: stat d7/d9/d15/f36 0 2024-04-22T15:18:42.223 INFO:tasks.workunit.client.0.smithi096.stdout:6/270: symlink d3/d28/d4d/l5d 0 2024-04-22T15:18:42.223 INFO:tasks.workunit.client.0.smithi096.stdout:0/335: dread - d7/de/d5b/d62/f63 zero size 2024-04-22T15:18:42.225 INFO:tasks.workunit.client.0.smithi096.stdout:6/271: fsync d3/f22 0 2024-04-22T15:18:42.226 INFO:tasks.workunit.client.0.smithi096.stdout:6/272: dread - d3/d28/dd/d11/d2c/d3b/f48 zero size 2024-04-22T15:18:42.226 INFO:tasks.workunit.client.0.smithi096.stdout:7/413: truncate d31/d32/f4c 1736240 0 2024-04-22T15:18:42.227 INFO:tasks.workunit.client.0.smithi096.stdout:7/414: fdatasync d31/d32/d36/f70 0 2024-04-22T15:18:42.228 INFO:tasks.workunit.client.0.smithi096.stdout:0/336: truncate d7/d9/f45 169001 0 2024-04-22T15:18:42.230 INFO:tasks.workunit.client.0.smithi096.stdout:7/415: unlink d31/d32/d33/d54/l65 0 2024-04-22T15:18:42.230 INFO:tasks.workunit.client.0.smithi096.stdout:1/453: sync 2024-04-22T15:18:42.231 INFO:tasks.workunit.client.0.smithi096.stdout:7/416: chown d31/d32/d33/d5e/f63 10667 1 2024-04-22T15:18:42.232 INFO:tasks.workunit.client.0.smithi096.stdout:0/337: mknod d7/d9/dc/d16/c6f 0 2024-04-22T15:18:42.232 INFO:tasks.workunit.client.0.smithi096.stdout:0/338: stat d7/d9/dc/f56 0 2024-04-22T15:18:42.233 INFO:tasks.workunit.client.0.smithi096.stdout:0/339: chown d7/d9/d66 38 1 2024-04-22T15:18:42.234 INFO:tasks.workunit.client.0.smithi096.stdout:0/340: chown d7/d9/dc/d16/d18/f2b 72 1 2024-04-22T15:18:42.234 INFO:tasks.workunit.client.0.smithi096.stdout:6/273: creat d3/d28/dd/d11/d2c/f5e x:0 0 0 2024-04-22T15:18:42.235 INFO:tasks.workunit.client.0.smithi096.stdout:7/417: mknod d31/c82 0 2024-04-22T15:18:42.236 INFO:tasks.workunit.client.0.smithi096.stdout:7/418: truncate f7 4725538 0 2024-04-22T15:18:42.236 INFO:tasks.workunit.client.0.smithi096.stdout:1/454: mknod d6/dc/c92 0 2024-04-22T15:18:42.241 INFO:tasks.workunit.client.0.smithi096.stdout:7/419: dread d31/d32/d33/f5a [0,4194304] 0 2024-04-22T15:18:42.241 INFO:tasks.workunit.client.0.smithi096.stdout:6/274: unlink d3/d3a/c59 0 2024-04-22T15:18:42.242 INFO:tasks.workunit.client.0.smithi096.stdout:0/341: creat d7/de/d5b/d62/d67/f70 x:0 0 0 2024-04-22T15:18:42.245 INFO:tasks.workunit.client.0.smithi096.stdout:7/420: mkdir d31/d32/d33/d6a/d83 0 2024-04-22T15:18:42.246 INFO:tasks.workunit.client.0.smithi096.stdout:6/275: mknod d3/d28/d54/c5f 0 2024-04-22T15:18:42.249 INFO:tasks.workunit.client.0.smithi096.stdout:0/342: dwrite d7/f32 [4194304,4194304] 0 2024-04-22T15:18:42.249 INFO:tasks.workunit.client.0.smithi096.stdout:1/455: truncate d6/dc/d8a/d3f/d54/d32/d8c/d4a/f4d 7288606 0 2024-04-22T15:18:42.250 INFO:tasks.workunit.client.0.smithi096.stdout:1/456: readlink d6/l64 0 2024-04-22T15:18:42.253 INFO:tasks.workunit.client.0.smithi096.stdout:7/421: rename d31/d32/d33/f5a to d31/d32/d36/d5d/f84 0 2024-04-22T15:18:42.254 INFO:tasks.workunit.client.0.smithi096.stdout:7/422: dread - d31/d32/d33/f57 zero size 2024-04-22T15:18:42.254 INFO:tasks.workunit.client.0.smithi096.stdout:6/276: dwrite d3/d28/f17 [0,4194304] 0 2024-04-22T15:18:42.255 INFO:tasks.workunit.client.0.smithi096.stdout:7/423: write d31/d32/d33/f34 [4903715,84508] 0 2024-04-22T15:18:42.256 INFO:tasks.workunit.client.0.smithi096.stdout:0/343: symlink d7/d9/dc/d16/d18/l71 0 2024-04-22T15:18:42.257 INFO:tasks.workunit.client.0.smithi096.stdout:7/424: write d31/d32/d33/f34 [3372232,41031] 0 2024-04-22T15:18:42.258 INFO:tasks.workunit.client.0.smithi096.stdout:0/344: write d7/f4d [5060598,83838] 0 2024-04-22T15:18:42.273 INFO:tasks.workunit.client.0.smithi096.stdout:1/457: getdents d6/dc/d8a/d3f/d62/d7d/d87 0 2024-04-22T15:18:42.279 INFO:tasks.workunit.client.0.smithi096.stdout:7/425: dread d31/d32/d33/f58 [0,4194304] 0 2024-04-22T15:18:42.280 INFO:tasks.workunit.client.0.smithi096.stdout:7/426: write d31/d32/d33/d6a/f72 [110906,32920] 0 2024-04-22T15:18:42.281 INFO:tasks.workunit.client.0.smithi096.stdout:1/458: dwrite d6/dc/d8a/d3f/d54/d32/d8c/f19 [0,4194304] 0 2024-04-22T15:18:42.282 INFO:tasks.workunit.client.0.smithi096.stdout:7/427: fdatasync d31/d32/d33/d54/f62 0 2024-04-22T15:18:42.287 INFO:tasks.workunit.client.0.smithi096.stdout:7/428: dread d31/d32/f55 [0,4194304] 0 2024-04-22T15:18:42.288 INFO:tasks.workunit.client.0.smithi096.stdout:7/429: chown c2a 1990084 1 2024-04-22T15:18:42.295 INFO:tasks.workunit.client.0.smithi096.stdout:7/430: dwrite d31/d32/d33/d54/f64 [0,4194304] 0 2024-04-22T15:18:42.297 INFO:tasks.workunit.client.0.smithi096.stdout:7/431: readlink d31/d32/d36/d5d/l7d 0 2024-04-22T15:18:42.300 INFO:tasks.workunit.client.0.smithi096.stdout:7/432: mknod d31/d32/d7c/c85 0 2024-04-22T15:18:42.313 INFO:tasks.workunit.client.0.smithi096.stdout:7/433: getdents d31/d32/d36 0 2024-04-22T15:18:42.318 INFO:tasks.workunit.client.0.smithi096.stdout:7/434: creat d31/d32/d36/d75/f86 x:0 0 0 2024-04-22T15:18:42.320 INFO:tasks.workunit.client.0.smithi096.stdout:0/345: truncate d7/d9/d15/f36 3130232 0 2024-04-22T15:18:42.322 INFO:tasks.workunit.client.0.smithi096.stdout:7/435: dread d31/d32/d33/f35 [0,4194304] 0 2024-04-22T15:18:42.323 INFO:tasks.workunit.client.0.smithi096.stdout:1/459: dread d6/dc/d8a/d3f/d54/f18 [0,4194304] 0 2024-04-22T15:18:42.326 INFO:tasks.workunit.client.0.smithi096.stdout:6/277: truncate d3/f26 1586628 0 2024-04-22T15:18:42.332 INFO:tasks.workunit.client.0.smithi096.stdout:6/278: dwrite d3/f22 [0,4194304] 0 2024-04-22T15:18:42.335 INFO:tasks.workunit.client.0.smithi096.stdout:7/436: getdents d31/d32/d33/d5e 0 2024-04-22T15:18:42.336 INFO:tasks.workunit.client.0.smithi096.stdout:1/460: dwrite d6/dc/d8a/d3f/d54/d32/d8c/d45/f58 [0,4194304] 0 2024-04-22T15:18:42.338 INFO:tasks.workunit.client.0.smithi096.stdout:1/461: chown d6/dc/f6c 15020 1 2024-04-22T15:18:42.343 INFO:tasks.workunit.client.0.smithi096.stdout:7/437: rename d31/d32/d36/f37 to d31/d32/d36/f87 0 2024-04-22T15:18:42.345 INFO:tasks.workunit.client.0.smithi096.stdout:6/279: truncate d3/d28/dd/d11/d2c/f56 1075884 0 2024-04-22T15:18:42.354 INFO:tasks.workunit.client.0.smithi096.stdout:6/280: link f0 d3/d28/dd/d11/d2c/d44/d51/f60 0 2024-04-22T15:18:42.355 INFO:tasks.workunit.client.0.smithi096.stdout:6/281: write d3/d28/f17 [2729100,40416] 0 2024-04-22T15:18:42.357 INFO:tasks.workunit.client.0.smithi096.stdout:7/438: getdents d31/d32/d33/d44 0 2024-04-22T15:18:42.366 INFO:tasks.workunit.client.0.smithi096.stdout:7/439: creat d31/d32/d33/d6a/d83/f88 x:0 0 0 2024-04-22T15:18:42.368 INFO:tasks.workunit.client.0.smithi096.stdout:6/282: rename d3/d28/f23 to d3/d28/f61 0 2024-04-22T15:18:42.369 INFO:tasks.workunit.client.0.smithi096.stdout:6/283: write d3/d28/f1c [2283590,70332] 0 2024-04-22T15:18:42.375 INFO:tasks.workunit.client.0.smithi096.stdout:7/440: creat d31/d32/d33/d54/f89 x:0 0 0 2024-04-22T15:18:42.377 INFO:tasks.workunit.client.0.smithi096.stdout:6/284: symlink d3/d28/dd/l62 0 2024-04-22T15:18:42.378 INFO:tasks.workunit.client.0.smithi096.stdout:6/285: chown d3/d28/d54/l58 982781 1 2024-04-22T15:18:42.379 INFO:tasks.workunit.client.0.smithi096.stdout:6/286: chown d3/d28/f37 14064 1 2024-04-22T15:18:42.382 INFO:tasks.workunit.client.0.smithi096.stdout:6/287: creat d3/d28/dd/d11/d2c/d3b/f63 x:0 0 0 2024-04-22T15:18:42.394 INFO:tasks.workunit.client.0.smithi096.stdout:7/441: dread f11 [0,4194304] 0 2024-04-22T15:18:42.395 INFO:tasks.workunit.client.0.smithi096.stdout:6/288: dread d3/d3a/f3e [0,4194304] 0 2024-04-22T15:18:42.397 INFO:tasks.workunit.client.0.smithi096.stdout:6/289: read d3/f29 [166680,30535] 0 2024-04-22T15:18:42.398 INFO:tasks.workunit.client.0.smithi096.stdout:7/442: mknod d31/d32/d33/d54/c8a 0 2024-04-22T15:18:42.403 INFO:tasks.workunit.client.0.smithi096.stdout:6/290: dwrite d3/d28/dd/f55 [0,4194304] 0 2024-04-22T15:18:42.406 INFO:tasks.workunit.client.0.smithi096.stdout:6/291: mknod d3/d28/c64 0 2024-04-22T15:18:42.421 INFO:tasks.workunit.client.0.smithi096.stdout:6/292: write d3/d28/dd/d11/f18 [1394149,97137] 0 2024-04-22T15:18:42.422 INFO:tasks.workunit.client.0.smithi096.stdout:7/443: dread d31/d32/d33/d5e/f6d [0,4194304] 0 2024-04-22T15:18:42.423 INFO:tasks.workunit.client.0.smithi096.stdout:6/293: symlink d3/d3a/l65 0 2024-04-22T15:18:42.424 INFO:tasks.workunit.client.0.smithi096.stdout:7/444: truncate d31/d32/d33/d54/f7a 694430 0 2024-04-22T15:18:42.430 INFO:tasks.workunit.client.0.smithi096.stdout:7/445: dwrite d31/d32/d33/d54/f7a [0,4194304] 0 2024-04-22T15:18:42.432 INFO:tasks.workunit.client.0.smithi096.stdout:7/446: truncate d31/d32/d36/d75/f86 644688 0 2024-04-22T15:18:42.434 INFO:tasks.workunit.client.0.smithi096.stdout:7/447: getdents d31/d68 0 2024-04-22T15:18:42.436 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:42 smithi096 ceph-mon[26688]: pgmap v27: 65 pgs: 65 active+clean; 653 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.4 MiB/s rd, 73 MiB/s wr, 253 op/s 2024-04-22T15:18:42.437 INFO:tasks.workunit.client.0.smithi096.stdout:3/216: sync 2024-04-22T15:18:42.437 INFO:tasks.workunit.client.0.smithi096.stdout:4/297: sync 2024-04-22T15:18:42.437 INFO:tasks.workunit.client.0.smithi096.stdout:5/317: sync 2024-04-22T15:18:42.439 INFO:tasks.workunit.client.0.smithi096.stdout:3/217: stat da/db/d12/d22/f3a 0 2024-04-22T15:18:42.442 INFO:tasks.workunit.client.0.smithi096.stdout:7/448: dwrite d31/d32/d33/d6a/f72 [0,4194304] 0 2024-04-22T15:18:42.443 INFO:tasks.workunit.client.0.smithi096.stdout:4/298: symlink d5/d7/d1a/l66 0 2024-04-22T15:18:42.444 INFO:tasks.workunit.client.0.smithi096.stdout:7/449: dread - d31/d32/d33/d54/f62 zero size 2024-04-22T15:18:42.444 INFO:tasks.workunit.client.0.smithi096.stdout:5/318: creat d4/d11/d3d/d4f/f69 x:0 0 0 2024-04-22T15:18:42.446 INFO:tasks.workunit.client.0.smithi096.stdout:3/218: symlink da/db/de/d19/d2d/l44 0 2024-04-22T15:18:42.447 INFO:tasks.workunit.client.0.smithi096.stdout:5/319: dread - d4/d11/d3d/d4d/d59/d5a/d5c/f5f zero size 2024-04-22T15:18:42.451 INFO:tasks.workunit.client.0.smithi096.stdout:4/299: dwrite d5/d7/f20 [0,4194304] 0 2024-04-22T15:18:42.453 INFO:tasks.workunit.client.0.smithi096.stdout:5/320: symlink d4/d11/d3d/d4d/d59/d5a/l6a 0 2024-04-22T15:18:42.455 INFO:tasks.workunit.client.0.smithi096.stdout:7/450: creat d31/d32/d33/d54/f8b x:0 0 0 2024-04-22T15:18:42.458 INFO:tasks.workunit.client.0.smithi096.stdout:3/219: dwrite da/db/d12/d22/f31 [0,4194304] 0 2024-04-22T15:18:42.458 INFO:tasks.workunit.client.0.smithi096.stdout:7/451: write d31/d32/f47 [1073775,77115] 0 2024-04-22T15:18:42.460 INFO:tasks.workunit.client.0.smithi096.stdout:4/300: mkdir d5/d7/d2c/d67 0 2024-04-22T15:18:42.462 INFO:tasks.workunit.client.0.smithi096.stdout:7/452: symlink d31/d32/d36/l8c 0 2024-04-22T15:18:42.464 INFO:tasks.workunit.client.0.smithi096.stdout:5/321: rename d4/d11/d3d/d4d/d59/d5a/d23 to d4/d2c/d2f/d6b 0 2024-04-22T15:18:42.465 INFO:tasks.workunit.client.0.smithi096.stdout:5/322: stat d4/d11/d25 0 2024-04-22T15:18:42.469 INFO:tasks.workunit.client.0.smithi096.stdout:7/453: mknod d31/d32/d33/d54/c8d 0 2024-04-22T15:18:42.478 INFO:tasks.workunit.client.0.smithi096.stdout:3/220: getdents da/db/de/d19/d2d 0 2024-04-22T15:18:42.483 INFO:tasks.workunit.client.0.smithi096.stdout:3/221: dwrite da/db/de/f40 [0,4194304] 0 2024-04-22T15:18:42.493 INFO:tasks.workunit.client.0.smithi096.stdout:3/222: write da/f27 [3639602,61214] 0 2024-04-22T15:18:42.493 INFO:tasks.workunit.client.0.smithi096.stdout:7/454: dwrite d31/d32/d36/f45 [0,4194304] 0 2024-04-22T15:18:42.497 INFO:tasks.workunit.client.0.smithi096.stdout:7/455: mknod d31/d32/d36/d75/c8e 0 2024-04-22T15:18:42.507 INFO:tasks.workunit.client.0.smithi096.stdout:7/456: dwrite d31/d32/f4c [0,4194304] 0 2024-04-22T15:18:42.508 INFO:tasks.workunit.client.0.smithi096.stdout:7/457: stat d31/d32/d33/f58 0 2024-04-22T15:18:42.509 INFO:tasks.workunit.client.0.smithi096.stdout:7/458: truncate f10 4981722 0 2024-04-22T15:18:42.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:42 smithi196 ceph-mon[34717]: pgmap v27: 65 pgs: 65 active+clean; 653 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.4 MiB/s rd, 73 MiB/s wr, 253 op/s 2024-04-22T15:18:42.509 INFO:tasks.workunit.client.0.smithi096.stdout:7/459: chown d31/d32/d33/d44 668110 1 2024-04-22T15:18:42.517 INFO:tasks.workunit.client.0.smithi096.stdout:7/460: dwrite d31/d32/d33/f7f [0,4194304] 0 2024-04-22T15:18:42.518 INFO:tasks.workunit.client.0.smithi096.stdout:7/461: chown d31/d32/d36 1002187 1 2024-04-22T15:18:42.521 INFO:tasks.workunit.client.0.smithi096.stdout:7/462: dread f11 [0,4194304] 0 2024-04-22T15:18:42.523 INFO:tasks.workunit.client.0.smithi096.stdout:7/463: write f10 [4201886,94911] 0 2024-04-22T15:18:42.527 INFO:tasks.workunit.client.0.smithi096.stdout:7/464: rename d31/d32/d33/d6a/d83/f88 to d31/f8f 0 2024-04-22T15:18:42.530 INFO:tasks.workunit.client.0.smithi096.stdout:7/465: unlink d31/l50 0 2024-04-22T15:18:42.538 INFO:tasks.workunit.client.0.smithi096.stdout:7/466: truncate d31/d32/d33/d6a/f72 2162324 0 2024-04-22T15:18:42.539 INFO:tasks.workunit.client.0.smithi096.stdout:7/467: symlink d31/d32/d36/l90 0 2024-04-22T15:18:42.541 INFO:tasks.workunit.client.0.smithi096.stdout:7/468: symlink d31/d32/d36/d75/l91 0 2024-04-22T15:18:42.544 INFO:tasks.workunit.client.0.smithi096.stdout:7/469: link cf d31/d32/d33/d6a/c92 0 2024-04-22T15:18:42.545 INFO:tasks.workunit.client.0.smithi096.stdout:7/470: fsync d31/d32/d33/f34 0 2024-04-22T15:18:42.547 INFO:tasks.workunit.client.0.smithi096.stdout:7/471: mkdir d31/d32/d33/d93 0 2024-04-22T15:18:42.549 INFO:tasks.workunit.client.0.smithi096.stdout:7/472: mknod d31/d32/d33/d54/d43/c94 0 2024-04-22T15:18:42.566 INFO:tasks.workunit.client.0.smithi096.stdout:7/473: dread d31/d32/d33/f3e [0,4194304] 0 2024-04-22T15:18:42.568 INFO:tasks.workunit.client.0.smithi096.stdout:7/474: symlink d31/d32/d33/d93/l95 0 2024-04-22T15:18:42.573 INFO:tasks.workunit.client.0.smithi096.stdout:7/475: dwrite d31/d32/d33/f5c [0,4194304] 0 2024-04-22T15:18:42.578 INFO:tasks.workunit.client.0.smithi096.stdout:0/346: sync 2024-04-22T15:18:42.578 INFO:tasks.workunit.client.0.smithi096.stdout:2/376: sync 2024-04-22T15:18:42.578 INFO:tasks.workunit.client.0.smithi096.stdout:9/250: sync 2024-04-22T15:18:42.578 INFO:tasks.workunit.client.0.smithi096.stdout:8/294: sync 2024-04-22T15:18:42.579 INFO:tasks.workunit.client.0.smithi096.stdout:7/476: dwrite d31/d32/d33/f58 [0,4194304] 0 2024-04-22T15:18:42.581 INFO:tasks.workunit.client.0.smithi096.stdout:0/347: stat d7/d9/dc/d26/c27 0 2024-04-22T15:18:42.583 INFO:tasks.workunit.client.0.smithi096.stdout:9/251: symlink d1/df/l53 0 2024-04-22T15:18:42.585 INFO:tasks.workunit.client.0.smithi096.stdout:8/295: symlink d4/d1c/l66 0 2024-04-22T15:18:42.586 INFO:tasks.workunit.client.0.smithi096.stdout:7/477: creat d31/d32/d33/d6a/d83/f96 x:0 0 0 2024-04-22T15:18:42.586 INFO:tasks.workunit.client.0.smithi096.stdout:7/478: dread - d31/d32/f6c zero size 2024-04-22T15:18:42.657 INFO:tasks.workunit.client.0.smithi096.stdout:7/479: chown f11 63690291 1 2024-04-22T15:18:42.657 INFO:tasks.workunit.client.0.smithi096.stdout:2/377: rename d3/d1f/d2f/f75 to d3/db/df/d24/d58/f77 0 2024-04-22T15:18:42.657 INFO:tasks.workunit.client.0.smithi096.stdout:0/348: symlink d7/de/d5b/d62/l72 0 2024-04-22T15:18:42.658 INFO:tasks.workunit.client.0.smithi096.stdout:9/252: symlink d1/l54 0 2024-04-22T15:18:42.662 INFO:tasks.workunit.client.0.smithi096.stdout:2/378: symlink d3/db/d30/l78 0 2024-04-22T15:18:42.663 INFO:tasks.workunit.client.0.smithi096.stdout:7/480: dwrite d31/d32/d33/f5c [0,4194304] 0 2024-04-22T15:18:42.669 INFO:tasks.workunit.client.0.smithi096.stdout:2/379: creat d3/db/d2d/d46/d59/f79 x:0 0 0 2024-04-22T15:18:42.669 INFO:tasks.workunit.client.0.smithi096.stdout:2/380: chown d3/d13/c45 118239 1 2024-04-22T15:18:42.670 INFO:tasks.workunit.client.0.smithi096.stdout:2/381: fsync d3/db/df/d10/f55 0 2024-04-22T15:18:42.671 INFO:tasks.workunit.client.0.smithi096.stdout:7/481: symlink d31/d32/d36/d5d/l97 0 2024-04-22T15:18:42.672 INFO:tasks.workunit.client.0.smithi096.stdout:7/482: chown l26 267996780 1 2024-04-22T15:18:42.672 INFO:tasks.workunit.client.0.smithi096.stdout:8/296: link d4/c61 d4/d10/c67 0 2024-04-22T15:18:42.673 INFO:tasks.workunit.client.0.smithi096.stdout:7/483: dread - d31/d32/f60 zero size 2024-04-22T15:18:42.674 INFO:tasks.workunit.client.0.smithi096.stdout:7/484: readlink d31/d32/d33/l38 0 2024-04-22T15:18:42.679 INFO:tasks.workunit.client.0.smithi096.stdout:8/297: rename d4/f28 to d4/d1c/d40/d48/f68 0 2024-04-22T15:18:42.680 INFO:tasks.workunit.client.0.smithi096.stdout:7/485: mkdir d31/d68/d98 0 2024-04-22T15:18:42.682 INFO:tasks.workunit.client.0.smithi096.stdout:7/486: truncate d31/d32/f60 212295 0 2024-04-22T15:18:42.682 INFO:tasks.workunit.client.0.smithi096.stdout:8/298: creat d4/d10/d25/d38/f69 x:0 0 0 2024-04-22T15:18:42.686 INFO:tasks.workunit.client.0.smithi096.stdout:8/299: symlink d4/d10/d25/d4c/l6a 0 2024-04-22T15:18:42.688 INFO:tasks.workunit.client.0.smithi096.stdout:7/487: dwrite f2b [0,4194304] 0 2024-04-22T15:18:42.690 INFO:tasks.workunit.client.0.smithi096.stdout:8/300: creat d4/d1c/d62/f6b x:0 0 0 2024-04-22T15:18:42.690 INFO:tasks.workunit.client.0.smithi096.stdout:7/488: write f2b [467192,113599] 0 2024-04-22T15:18:42.694 INFO:tasks.workunit.client.0.smithi096.stdout:8/301: dread - d4/d1c/f3a zero size 2024-04-22T15:18:42.701 INFO:tasks.workunit.client.0.smithi096.stdout:7/489: getdents d31/d32/d33/d6a/d83 0 2024-04-22T15:18:42.702 INFO:tasks.workunit.client.0.smithi096.stdout:7/490: truncate d31/d68/f69 555786 0 2024-04-22T15:18:42.703 INFO:tasks.workunit.client.0.smithi096.stdout:8/302: creat d4/d10/d25/f6c x:0 0 0 2024-04-22T15:18:42.706 INFO:tasks.workunit.client.0.smithi096.stdout:8/303: mknod d4/d10/d25/d4c/c6d 0 2024-04-22T15:18:42.710 INFO:tasks.workunit.client.0.smithi096.stdout:8/304: mknod d4/d1c/d40/d48/c6e 0 2024-04-22T15:18:42.710 INFO:tasks.workunit.client.0.smithi096.stdout:7/491: dwrite f1c [0,4194304] 0 2024-04-22T15:18:42.711 INFO:tasks.workunit.client.0.smithi096.stdout:7/492: stat d31/c4a 0 2024-04-22T15:18:42.711 INFO:tasks.workunit.client.0.smithi096.stdout:7/493: stat c1f 0 2024-04-22T15:18:42.712 INFO:tasks.workunit.client.0.smithi096.stdout:9/253: getdents d1 0 2024-04-22T15:18:42.714 INFO:tasks.workunit.client.0.smithi096.stdout:7/494: dread - d31/d32/d33/d54/f89 zero size 2024-04-22T15:18:42.717 INFO:tasks.workunit.client.0.smithi096.stdout:9/254: mknod d1/d4/d9/d43/c55 0 2024-04-22T15:18:42.717 INFO:tasks.workunit.client.0.smithi096.stdout:2/382: rmdir d3/db/d2d 39 2024-04-22T15:18:42.718 INFO:tasks.workunit.client.0.smithi096.stdout:9/255: readlink d1/l27 0 2024-04-22T15:18:42.719 INFO:tasks.workunit.client.0.smithi096.stdout:1/462: sync 2024-04-22T15:18:42.722 INFO:tasks.workunit.client.0.smithi096.stdout:8/305: dwrite d4/d1c/d40/d48/f63 [0,4194304] 0 2024-04-22T15:18:42.723 INFO:tasks.workunit.client.0.smithi096.stdout:0/349: dwrite d7/de/f12 [4194304,4194304] 0 2024-04-22T15:18:42.724 INFO:tasks.workunit.client.0.smithi096.stdout:1/463: truncate d6/dc/f6c 548165 0 2024-04-22T15:18:42.728 INFO:tasks.workunit.client.0.smithi096.stdout:9/256: symlink d1/l56 0 2024-04-22T15:18:42.732 INFO:tasks.workunit.client.0.smithi096.stdout:2/383: dread - d3/db/d30/f3a zero size 2024-04-22T15:18:42.732 INFO:tasks.workunit.client.0.smithi096.stdout:7/495: link d31/d68/f6e d31/d32/d36/f99 0 2024-04-22T15:18:42.739 INFO:tasks.workunit.client.0.smithi096.stdout:8/306: dwrite d4/d1c/d34/f49 [0,4194304] 0 2024-04-22T15:18:42.739 INFO:tasks.workunit.client.0.smithi096.stdout:1/464: dwrite d6/f4f [0,4194304] 0 2024-04-22T15:18:42.741 INFO:tasks.workunit.client.0.smithi096.stdout:0/350: mknod d7/de/d5b/d62/c73 0 2024-04-22T15:18:42.752 INFO:tasks.workunit.client.0.smithi096.stdout:7/496: creat d31/d32/d36/d75/f9a x:0 0 0 2024-04-22T15:18:42.752 INFO:tasks.workunit.client.0.smithi096.stdout:7/497: rename d31 to d31/d32/d36/d9b 22 2024-04-22T15:18:42.752 INFO:tasks.workunit.client.0.smithi096.stdout:8/307: symlink d4/d1c/d34/l6f 0 2024-04-22T15:18:42.753 INFO:tasks.workunit.client.0.smithi096.stdout:8/308: fsync d4/d1c/f43 0 2024-04-22T15:18:42.754 INFO:tasks.workunit.client.0.smithi096.stdout:8/309: readlink d4/d1c/d34/l6f 0 2024-04-22T15:18:42.754 INFO:tasks.workunit.client.0.smithi096.stdout:1/465: mknod d6/dc/d8a/d3f/d54/d32/d8c/d45/c93 0 2024-04-22T15:18:42.758 INFO:tasks.workunit.client.0.smithi096.stdout:9/257: mkdir d1/d4/d9/d43/d57 0 2024-04-22T15:18:42.762 INFO:tasks.workunit.client.0.smithi096.stdout:1/466: dwrite d6/dc/d8a/d3f/d54/d32/f6d [0,4194304] 0 2024-04-22T15:18:42.766 INFO:tasks.workunit.client.0.smithi096.stdout:8/310: symlink d4/d10/d4d/l70 0 2024-04-22T15:18:42.768 INFO:tasks.workunit.client.0.smithi096.stdout:2/384: dwrite d3/db/df/d24/d58/f77 [0,4194304] 0 2024-04-22T15:18:42.771 INFO:tasks.workunit.client.0.smithi096.stdout:7/498: link d31/f8f d31/d32/d36/d5d/f9c 0 2024-04-22T15:18:42.771 INFO:tasks.workunit.client.0.smithi096.stdout:0/351: link c4 d7/d9/dc/d26/c74 0 2024-04-22T15:18:42.776 INFO:tasks.workunit.client.0.smithi096.stdout:1/467: dwrite d6/dc/d8a/f7a [0,4194304] 0 2024-04-22T15:18:42.776 INFO:tasks.workunit.client.0.smithi096.stdout:9/258: dread d1/d25/f48 [0,4194304] 0 2024-04-22T15:18:42.778 INFO:tasks.workunit.client.0.smithi096.stdout:8/311: mkdir d4/d71 0 2024-04-22T15:18:42.784 INFO:tasks.workunit.client.0.smithi096.stdout:0/352: mknod d7/d9/dc/d16/c75 0 2024-04-22T15:18:42.784 INFO:tasks.workunit.client.0.smithi096.stdout:7/499: dwrite d31/d32/d36/f71 [0,4194304] 0 2024-04-22T15:18:42.798 INFO:tasks.workunit.client.0.smithi096.stdout:2/385: truncate d3/f49 2720215 0 2024-04-22T15:18:42.802 INFO:tasks.workunit.client.0.smithi096.stdout:9/259: rename d1/d4/d9/dd/f1d to d1/df/f58 0 2024-04-22T15:18:42.802 INFO:tasks.workunit.client.0.smithi096.stdout:0/353: dwrite d7/d9/f24 [0,4194304] 0 2024-04-22T15:18:42.802 INFO:tasks.workunit.client.0.smithi096.stdout:9/260: readlink d1/df/l30 0 2024-04-22T15:18:42.803 INFO:tasks.workunit.client.0.smithi096.stdout:0/354: read - d7/d9/dc/f56 zero size 2024-04-22T15:18:42.807 INFO:tasks.workunit.client.0.smithi096.stdout:2/386: unlink d3/l54 0 2024-04-22T15:18:42.810 INFO:tasks.workunit.client.0.smithi096.stdout:9/261: mkdir d1/d4/d9/d59 0 2024-04-22T15:18:42.811 INFO:tasks.workunit.client.0.smithi096.stdout:1/468: dwrite d6/fd [0,4194304] 0 2024-04-22T15:18:42.815 INFO:tasks.workunit.client.0.smithi096.stdout:2/387: creat d3/d1f/f7a x:0 0 0 2024-04-22T15:18:42.816 INFO:tasks.workunit.client.0.smithi096.stdout:8/312: getdents d4/d10 0 2024-04-22T15:18:42.818 INFO:tasks.workunit.client.0.smithi096.stdout:6/294: sync 2024-04-22T15:18:42.819 INFO:tasks.workunit.client.0.smithi096.stdout:9/262: unlink d1/d4/fe 0 2024-04-22T15:18:42.821 INFO:tasks.workunit.client.0.smithi096.stdout:7/500: getdents d31/d32/d36 0 2024-04-22T15:18:42.822 INFO:tasks.workunit.client.0.smithi096.stdout:1/469: mkdir d6/dc/d8a/d3f/d54/d32/d3b/d94 0 2024-04-22T15:18:42.825 INFO:tasks.workunit.client.0.smithi096.stdout:7/501: write d31/d32/d36/f70 [459081,48965] 0 2024-04-22T15:18:42.825 INFO:tasks.workunit.client.0.smithi096.stdout:2/388: dwrite d3/db/df/f35 [0,4194304] 0 2024-04-22T15:18:42.826 INFO:tasks.workunit.client.0.smithi096.stdout:0/355: link d7/d9/dc/d16/d18/f2b d7/de/d3a/f76 0 2024-04-22T15:18:42.830 INFO:tasks.workunit.client.0.smithi096.stdout:6/295: symlink d3/d3a/d4b/l66 0 2024-04-22T15:18:42.832 INFO:tasks.workunit.client.0.smithi096.stdout:9/263: mknod d1/d25/c5a 0 2024-04-22T15:18:42.836 INFO:tasks.workunit.client.0.smithi096.stdout:1/470: creat d6/dc/d8a/d3f/d54/d32/d3b/d56/f95 x:0 0 0 2024-04-22T15:18:42.840 INFO:tasks.workunit.client.0.smithi096.stdout:7/502: mknod d31/d32/d33/d54/d43/c9d 0 2024-04-22T15:18:42.843 INFO:tasks.workunit.client.0.smithi096.stdout:0/356: read d7/d9/dc/d16/f2e [121601,125643] 0 2024-04-22T15:18:42.843 INFO:tasks.workunit.client.0.smithi096.stdout:9/264: chown d1/f6 2228 1 2024-04-22T15:18:42.846 INFO:tasks.workunit.client.0.smithi096.stdout:1/471: rename d6/dc/d8a/d3f/d62/f76 to d6/dc/d8a/d3f/d54/d32/d3b/d94/f96 0 2024-04-22T15:18:42.848 INFO:tasks.workunit.client.0.smithi096.stdout:7/503: dwrite d31/d32/f3f [0,4194304] 0 2024-04-22T15:18:42.849 INFO:tasks.workunit.client.0.smithi096.stdout:8/313: dread d4/d10/f21 [0,4194304] 0 2024-04-22T15:18:42.853 INFO:tasks.workunit.client.0.smithi096.stdout:8/314: fsync d4/d2a/f36 0 2024-04-22T15:18:42.854 INFO:tasks.workunit.client.0.smithi096.stdout:0/357: dwrite d7/d9/dc/d16/d18/f23 [0,4194304] 0 2024-04-22T15:18:42.859 INFO:tasks.workunit.client.0.smithi096.stdout:1/472: rmdir d6/dc/d8a/d3f/d54 39 2024-04-22T15:18:42.859 INFO:tasks.workunit.client.0.smithi096.stdout:1/473: readlink d6/dc/l2a 0 2024-04-22T15:18:42.861 INFO:tasks.workunit.client.0.smithi096.stdout:7/504: mknod d31/d32/d7c/c9e 0 2024-04-22T15:18:42.863 INFO:tasks.workunit.client.0.smithi096.stdout:6/296: dread d3/d28/f1c [0,4194304] 0 2024-04-22T15:18:42.863 INFO:tasks.workunit.client.0.smithi096.stdout:9/265: mkdir d1/d4/d9/d43/d57/d5b 0 2024-04-22T15:18:42.866 INFO:tasks.workunit.client.0.smithi096.stdout:2/389: symlink d3/db/df/d10/l7b 0 2024-04-22T15:18:42.866 INFO:tasks.workunit.client.0.smithi096.stdout:0/358: dread d7/d9/dc/d16/f1f [0,4194304] 0 2024-04-22T15:18:42.867 INFO:tasks.workunit.client.0.smithi096.stdout:1/474: fsync d6/dc/d8a/d3f/d54/d32/d8c/d45/d7b/f7e 0 2024-04-22T15:18:42.868 INFO:tasks.workunit.client.0.smithi096.stdout:7/505: write d31/d32/d33/d5e/f6d [1478517,25327] 0 2024-04-22T15:18:42.868 INFO:tasks.workunit.client.0.smithi096.stdout:9/266: readlink d1/d4/d9/dd/l26 0 2024-04-22T15:18:42.869 INFO:tasks.workunit.client.0.smithi096.stdout:6/297: creat d3/d28/dd/d11/d2c/d44/f67 x:0 0 0 2024-04-22T15:18:42.869 INFO:tasks.workunit.client.0.smithi096.stdout:8/315: creat d4/d1c/f72 x:0 0 0 2024-04-22T15:18:42.873 INFO:tasks.workunit.client.0.smithi096.stdout:7/506: creat d31/d68/f9f x:0 0 0 2024-04-22T15:18:42.874 INFO:tasks.workunit.client.0.smithi096.stdout:1/475: mkdir d6/d97 0 2024-04-22T15:18:42.875 INFO:tasks.workunit.client.0.smithi096.stdout:7/507: fdatasync d31/d32/d33/f7f 0 2024-04-22T15:18:42.875 INFO:tasks.workunit.client.0.smithi096.stdout:8/316: mknod d4/d1c/d40/c73 0 2024-04-22T15:18:42.876 INFO:tasks.workunit.client.0.smithi096.stdout:9/267: dread d1/f2f [0,4194304] 0 2024-04-22T15:18:42.876 INFO:tasks.workunit.client.0.smithi096.stdout:0/359: mknod d7/d9/c77 0 2024-04-22T15:18:42.877 INFO:tasks.workunit.client.0.smithi096.stdout:9/268: chown d1 100041 1 2024-04-22T15:18:42.878 INFO:tasks.workunit.client.0.smithi096.stdout:7/508: fsync d31/d32/d36/d75/f9a 0 2024-04-22T15:18:42.878 INFO:tasks.workunit.client.0.smithi096.stdout:6/298: dwrite d3/d28/dd/d11/f1e [0,4194304] 0 2024-04-22T15:18:42.880 INFO:tasks.workunit.client.0.smithi096.stdout:8/317: rmdir d4/d1c/d40/d46/d58 39 2024-04-22T15:18:42.885 INFO:tasks.workunit.client.0.smithi096.stdout:9/269: read d1/d4/d9/f12 [1395939,58792] 0 2024-04-22T15:18:42.885 INFO:tasks.workunit.client.0.smithi096.stdout:0/360: chown d7/d9/dc/d16/c6f 274642 1 2024-04-22T15:18:42.886 INFO:tasks.workunit.client.0.smithi096.stdout:6/299: truncate d3/d28/dd/d11/d2c/d3b/f48 46817 0 2024-04-22T15:18:42.890 INFO:tasks.workunit.client.0.smithi096.stdout:1/476: dwrite d6/dc/d8a/d3f/d62/d7d/f86 [0,4194304] 0 2024-04-22T15:18:42.890 INFO:tasks.workunit.client.0.smithi096.stdout:9/270: write d1/d4/d9/dd/f40 [727504,117492] 0 2024-04-22T15:18:42.891 INFO:tasks.workunit.client.0.smithi096.stdout:7/509: dread d31/d32/d33/d6a/f72 [0,4194304] 0 2024-04-22T15:18:42.893 INFO:tasks.workunit.client.0.smithi096.stdout:1/477: write d6/dc/d8a/d3f/d54/d32/f6d [2041475,100225] 0 2024-04-22T15:18:42.895 INFO:tasks.workunit.client.0.smithi096.stdout:6/300: mkdir d3/d28/d54/d68 0 2024-04-22T15:18:42.898 INFO:tasks.workunit.client.0.smithi096.stdout:7/510: chown d31/d32/d36/l66 1895020 1 2024-04-22T15:18:42.899 INFO:tasks.workunit.client.0.smithi096.stdout:7/511: read - d31/f8f zero size 2024-04-22T15:18:42.901 INFO:tasks.workunit.client.0.smithi096.stdout:0/361: creat d7/f78 x:0 0 0 2024-04-22T15:18:42.902 INFO:tasks.workunit.client.0.smithi096.stdout:9/271: dwrite d1/d25/f33 [0,4194304] 0 2024-04-22T15:18:42.905 INFO:tasks.workunit.client.0.smithi096.stdout:1/478: rename d6/dc/f80 to d6/dc/d8a/d3f/d54/f98 0 2024-04-22T15:18:42.906 INFO:tasks.workunit.client.0.smithi096.stdout:1/479: dread - d6/dc/d8a/d3f/d54/d32/d3b/f89 zero size 2024-04-22T15:18:42.907 INFO:tasks.workunit.client.0.smithi096.stdout:8/318: write d4/d2a/f3c [318265,78465] 0 2024-04-22T15:18:42.908 INFO:tasks.workunit.client.0.smithi096.stdout:8/319: chown d4/d1c/f3a 13 1 2024-04-22T15:18:42.909 INFO:tasks.workunit.client.0.smithi096.stdout:7/512: symlink d31/d32/d36/d5d/la0 0 2024-04-22T15:18:42.909 INFO:tasks.workunit.client.0.smithi096.stdout:7/513: chown f23 1975 1 2024-04-22T15:18:42.912 INFO:tasks.workunit.client.0.smithi096.stdout:6/301: rename d3 to d3/d28/dd/d69 22 2024-04-22T15:18:42.914 INFO:tasks.workunit.client.0.smithi096.stdout:8/320: creat d4/d10/d4d/f74 x:0 0 0 2024-04-22T15:18:42.916 INFO:tasks.workunit.client.0.smithi096.stdout:0/362: mkdir d7/de/d3a/d61/d79 0 2024-04-22T15:18:42.916 INFO:tasks.workunit.client.0.smithi096.stdout:0/363: chown d7/d9/dc/d16 1557 1 2024-04-22T15:18:42.918 INFO:tasks.workunit.client.0.smithi096.stdout:0/364: dread d7/d9/dc/d16/d18/f54 [0,4194304] 0 2024-04-22T15:18:42.918 INFO:tasks.workunit.client.0.smithi096.stdout:7/514: rename d31/d32/d36/d5d/c73 to d31/d32/d33/d54/d43/ca1 0 2024-04-22T15:18:42.919 INFO:tasks.workunit.client.0.smithi096.stdout:1/480: symlink d6/dc/d8a/l99 0 2024-04-22T15:18:42.921 INFO:tasks.workunit.client.0.smithi096.stdout:1/481: read d6/dc/d8a/d3f/d54/d32/f35 [1968309,24027] 0 2024-04-22T15:18:42.922 INFO:tasks.workunit.client.0.smithi096.stdout:8/321: creat d4/d1c/d40/f75 x:0 0 0 2024-04-22T15:18:42.924 INFO:tasks.workunit.client.0.smithi096.stdout:7/515: dread d31/d32/d33/f34 [4194304,4194304] 0 2024-04-22T15:18:42.924 INFO:tasks.workunit.client.0.smithi096.stdout:0/365: creat d7/d42/f7a x:0 0 0 2024-04-22T15:18:42.925 INFO:tasks.workunit.client.0.smithi096.stdout:7/516: stat d31/d32/d7c 0 2024-04-22T15:18:42.926 INFO:tasks.workunit.client.0.smithi096.stdout:0/366: chown d7/d42/d4c/f6a 14418 1 2024-04-22T15:18:42.926 INFO:tasks.workunit.client.0.smithi096.stdout:6/302: rename d3/d28/c64 to d3/d28/d54/d68/c6a 0 2024-04-22T15:18:42.927 INFO:tasks.workunit.client.0.smithi096.stdout:7/517: read d31/d32/d36/f70 [176566,58793] 0 2024-04-22T15:18:42.928 INFO:tasks.workunit.client.0.smithi096.stdout:7/518: write f1c [5199037,108137] 0 2024-04-22T15:18:42.929 INFO:tasks.workunit.client.0.smithi096.stdout:9/272: write d1/d4/f22 [675826,100130] 0 2024-04-22T15:18:42.931 INFO:tasks.workunit.client.0.smithi096.stdout:1/482: truncate d6/dc/d8a/d3f/d54/f21 836209 0 2024-04-22T15:18:42.931 INFO:tasks.workunit.client.0.smithi096.stdout:9/273: dread d1/d4/d9/dd/f41 [0,4194304] 0 2024-04-22T15:18:42.933 INFO:tasks.workunit.client.0.smithi096.stdout:9/274: write d1/d4/f4e [546174,14262] 0 2024-04-22T15:18:42.933 INFO:tasks.workunit.client.0.smithi096.stdout:1/483: chown d6/dc/d8a/d3f/d54/d32/d3b/d94 0 1 2024-04-22T15:18:42.933 INFO:tasks.workunit.client.0.smithi096.stdout:0/367: chown d7/d9/dc/d16/d18/f39 7 1 2024-04-22T15:18:42.934 INFO:tasks.workunit.client.0.smithi096.stdout:1/484: fsync d6/dc/d8a/f67 0 2024-04-22T15:18:42.934 INFO:tasks.workunit.client.0.smithi096.stdout:6/303: creat d3/d28/d54/f6b x:0 0 0 2024-04-22T15:18:42.941 INFO:tasks.workunit.client.0.smithi096.stdout:9/275: dwrite d1/d4/d9/dd/f41 [0,4194304] 0 2024-04-22T15:18:42.941 INFO:tasks.workunit.client.0.smithi096.stdout:1/485: write d6/dc/d8a/d3f/d54/d32/f37 [433414,14294] 0 2024-04-22T15:18:42.942 INFO:tasks.workunit.client.0.smithi096.stdout:0/368: dread d7/d9/dc/d16/d18/f54 [0,4194304] 0 2024-04-22T15:18:42.943 INFO:tasks.workunit.client.0.smithi096.stdout:0/369: chown d7/d9/dc/d16/f2e 9109664 1 2024-04-22T15:18:42.944 INFO:tasks.workunit.client.0.smithi096.stdout:7/519: mkdir d31/d32/d33/da2 0 2024-04-22T15:18:42.947 INFO:tasks.workunit.client.0.smithi096.stdout:6/304: symlink d3/l6c 0 2024-04-22T15:18:42.953 INFO:tasks.workunit.client.0.smithi096.stdout:8/322: getdents d4/d10/d25 0 2024-04-22T15:18:42.954 INFO:tasks.workunit.client.0.smithi096.stdout:9/276: dread d1/d4/d9/f3b [0,4194304] 0 2024-04-22T15:18:42.958 INFO:tasks.workunit.client.0.smithi096.stdout:9/277: dread d1/d4/d9/dd/f41 [0,4194304] 0 2024-04-22T15:18:42.958 INFO:tasks.workunit.client.0.smithi096.stdout:1/486: mkdir d6/d97/d9a 0 2024-04-22T15:18:42.960 INFO:tasks.workunit.client.0.smithi096.stdout:0/370: mknod d7/de/c7b 0 2024-04-22T15:18:42.960 INFO:tasks.workunit.client.0.smithi096.stdout:9/278: chown d1/d25/c4a 1061 1 2024-04-22T15:18:42.962 INFO:tasks.workunit.client.0.smithi096.stdout:7/520: mknod d31/d32/ca3 0 2024-04-22T15:18:42.964 INFO:tasks.workunit.client.0.smithi096.stdout:6/305: dwrite d3/d28/f61 [0,4194304] 0 2024-04-22T15:18:42.967 INFO:tasks.workunit.client.0.smithi096.stdout:1/487: dwrite d6/fd [0,4194304] 0 2024-04-22T15:18:42.969 INFO:tasks.workunit.client.0.smithi096.stdout:1/488: write d6/f8f [472458,60611] 0 2024-04-22T15:18:42.976 INFO:tasks.workunit.client.0.smithi096.stdout:1/489: write d6/dc/d8a/d3f/d54/d32/d3b/d56/f44 [1476496,968] 0 2024-04-22T15:18:42.977 INFO:tasks.workunit.client.0.smithi096.stdout:6/306: dwrite d3/d28/dd/d11/d2c/d3b/f63 [0,4194304] 0 2024-04-22T15:18:42.980 INFO:tasks.workunit.client.0.smithi096.stdout:0/371: dwrite d7/f20 [0,4194304] 0 2024-04-22T15:18:43.053 INFO:tasks.workunit.client.0.smithi096.stdout:4/301: sync 2024-04-22T15:18:43.053 INFO:tasks.workunit.client.0.smithi096.stdout:5/323: sync 2024-04-22T15:18:43.054 INFO:tasks.workunit.client.0.smithi096.stdout:3/223: sync 2024-04-22T15:18:43.054 INFO:tasks.workunit.client.0.smithi096.stdout:9/279: creat d1/d4/d9/d43/f5c x:0 0 0 2024-04-22T15:18:43.055 INFO:tasks.workunit.client.0.smithi096.stdout:3/224: readlink da/db/dc/d28/l1d 0 2024-04-22T15:18:43.060 INFO:tasks.workunit.client.0.smithi096.stdout:7/521: rename d31/d32/d36/d75/c8e to d31/d32/d7c/ca4 0 2024-04-22T15:18:43.075 INFO:tasks.workunit.client.0.smithi096.stdout:9/280: rename d1/df/c16 to d1/df/c5d 0 2024-04-22T15:18:43.076 INFO:tasks.workunit.client.0.smithi096.stdout:9/281: truncate d1/df/f58 4265761 0 2024-04-22T15:18:43.077 INFO:tasks.workunit.client.0.smithi096.stdout:9/282: write d1/d4/d9/d43/f5c [612219,5905] 0 2024-04-22T15:18:43.079 INFO:tasks.workunit.client.0.smithi096.stdout:3/225: creat da/db/dc/d28/f45 x:0 0 0 2024-04-22T15:18:43.079 INFO:tasks.workunit.client.0.smithi096.stdout:9/283: read d1/d4/f22 [527215,23750] 0 2024-04-22T15:18:43.087 INFO:tasks.workunit.client.0.smithi096.stdout:4/302: creat d5/d7/d2c/d43/f68 x:0 0 0 2024-04-22T15:18:43.090 INFO:tasks.workunit.client.0.smithi096.stdout:3/226: mkdir da/db/d12/d46 0 2024-04-22T15:18:43.091 INFO:tasks.workunit.client.0.smithi096.stdout:8/323: getdents d4/d10/d4d 0 2024-04-22T15:18:43.093 INFO:tasks.workunit.client.0.smithi096.stdout:8/324: chown d4/d10/d25/d38/f50 24 1 2024-04-22T15:18:43.095 INFO:tasks.workunit.client.0.smithi096.stdout:9/284: mkdir d1/d4/d9/d43/d57/d5e 0 2024-04-22T15:18:43.098 INFO:tasks.workunit.client.0.smithi096.stdout:3/227: dwrite da/db/d12/d22/f31 [0,4194304] 0 2024-04-22T15:18:43.101 INFO:tasks.workunit.client.0.smithi096.stdout:3/228: chown da/db/dc/d28/l1d 45320 1 2024-04-22T15:18:43.103 INFO:tasks.workunit.client.0.smithi096.stdout:6/307: rename d3/d28/dd/d11/c47 to d3/d28/d54/c6d 0 2024-04-22T15:18:43.103 INFO:tasks.workunit.client.0.smithi096.stdout:7/522: getdents d31/d32/d33/da2 0 2024-04-22T15:18:43.105 INFO:tasks.workunit.client.0.smithi096.stdout:4/303: write d5/d7/d1a/f38 [957547,16612] 0 2024-04-22T15:18:43.106 INFO:tasks.workunit.client.0.smithi096.stdout:4/304: dread - d5/d7/d1f/f50 zero size 2024-04-22T15:18:43.108 INFO:tasks.workunit.client.0.smithi096.stdout:0/372: truncate d7/d9/dc/d16/f3b 1055155 0 2024-04-22T15:18:43.110 INFO:tasks.workunit.client.0.smithi096.stdout:8/325: mkdir d4/d10/d25/d4c/d76 0 2024-04-22T15:18:43.111 INFO:tasks.workunit.client.0.smithi096.stdout:8/326: chown d4/d1c/d62/f6b 183 1 2024-04-22T15:18:43.112 INFO:tasks.workunit.client.0.smithi096.stdout:8/327: chown d4/d10/d25/c52 122820631 1 2024-04-22T15:18:43.112 INFO:tasks.workunit.client.0.smithi096.stdout:9/285: unlink d1/d4/d9/f36 0 2024-04-22T15:18:43.116 INFO:tasks.workunit.client.0.smithi096.stdout:1/490: rename d6/dc/d8a/d3f/d62/d7d/d87 to d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b 0 2024-04-22T15:18:43.118 INFO:tasks.workunit.client.0.smithi096.stdout:6/308: mkdir d3/d28/d54/d68/d6e 0 2024-04-22T15:18:43.118 INFO:tasks.workunit.client.0.smithi096.stdout:6/309: readlink d3/d28/d4d/l5d 0 2024-04-22T15:18:43.119 INFO:tasks.workunit.client.0.smithi096.stdout:6/310: read - d3/d28/dd/f2b zero size 2024-04-22T15:18:43.121 INFO:tasks.workunit.client.0.smithi096.stdout:6/311: chown d3/d28/dd/d11/d2c/f4e 14 1 2024-04-22T15:18:43.122 INFO:tasks.workunit.client.0.smithi096.stdout:3/229: dread da/db/de/d19/f2a [0,4194304] 0 2024-04-22T15:18:43.123 INFO:tasks.workunit.client.0.smithi096.stdout:4/305: readlink d5/d7/l10 0 2024-04-22T15:18:43.123 INFO:tasks.workunit.client.0.smithi096.stdout:6/312: write d3/d28/f61 [5020055,112381] 0 2024-04-22T15:18:43.125 INFO:tasks.workunit.client.0.smithi096.stdout:3/230: truncate da/db/de/d19/f30 4297340 0 2024-04-22T15:18:43.131 INFO:tasks.workunit.client.0.smithi096.stdout:1/491: rmdir d6/d97 39 2024-04-22T15:18:43.132 INFO:tasks.workunit.client.0.smithi096.stdout:8/328: dwrite d4/d10/d25/f2e [0,4194304] 0 2024-04-22T15:18:43.133 INFO:tasks.workunit.client.0.smithi096.stdout:1/492: chown d6/dc/d8a/d3f/d54/d32/d3b/d56/l88 9517513 1 2024-04-22T15:18:43.133 INFO:tasks.workunit.client.0.smithi096.stdout:0/373: dread d7/de/f12 [0,4194304] 0 2024-04-22T15:18:43.134 INFO:tasks.workunit.client.0.smithi096.stdout:5/324: rename d4/d11/d3d/d35/l3e to d4/d11/d3d/d35/d57/l6c 0 2024-04-22T15:18:43.136 INFO:tasks.workunit.client.0.smithi096.stdout:5/325: write d4/d2c/d2f/d6b/f50 [5168123,1346] 0 2024-04-22T15:18:43.147 INFO:tasks.workunit.client.0.smithi096.stdout:0/374: write d7/d9/f24 [818034,116458] 0 2024-04-22T15:18:43.152 INFO:tasks.workunit.client.0.smithi096.stdout:9/286: rename d1/d4/d9/f18 to d1/d4/d9/d43/f5f 0 2024-04-22T15:18:43.153 INFO:tasks.workunit.client.0.smithi096.stdout:0/375: dwrite d7/d9/f24 [0,4194304] 0 2024-04-22T15:18:43.157 INFO:tasks.workunit.client.0.smithi096.stdout:4/306: symlink d5/d5b/l69 0 2024-04-22T15:18:43.161 INFO:tasks.workunit.client.0.smithi096.stdout:7/523: dwrite d31/d32/d33/d54/f7a [4194304,4194304] 0 2024-04-22T15:18:43.162 INFO:tasks.workunit.client.0.smithi096.stdout:4/307: read d5/d7/d1a/f1d [862537,46998] 0 2024-04-22T15:18:43.166 INFO:tasks.workunit.client.0.smithi096.stdout:6/313: dread d3/d28/dd/d11/f32 [0,4194304] 0 2024-04-22T15:18:43.171 INFO:tasks.workunit.client.0.smithi096.stdout:9/287: readlink d1/l54 0 2024-04-22T15:18:43.174 INFO:tasks.workunit.client.0.smithi096.stdout:0/376: mkdir d7/d9/dc/d16/d65/d7c 0 2024-04-22T15:18:43.175 INFO:tasks.workunit.client.0.smithi096.stdout:3/231: link da/db/de/d19/l39 da/db/d12/d22/d26/l47 0 2024-04-22T15:18:43.187 INFO:tasks.workunit.client.0.smithi096.stdout:1/493: creat d6/dc/d8a/d3f/f9c x:0 0 0 2024-04-22T15:18:43.188 INFO:tasks.workunit.client.0.smithi096.stdout:1/494: read d6/f14 [401390,19864] 0 2024-04-22T15:18:43.192 INFO:tasks.workunit.client.0.smithi096.stdout:0/377: mknod d7/d42/c7d 0 2024-04-22T15:18:43.193 INFO:tasks.workunit.client.0.smithi096.stdout:0/378: fdatasync d7/d42/d4c/f6a 0 2024-04-22T15:18:43.196 INFO:tasks.workunit.client.0.smithi096.stdout:8/329: getdents d4/d2a 0 2024-04-22T15:18:43.197 INFO:tasks.workunit.client.0.smithi096.stdout:8/330: fsync d4/d2a/f36 0 2024-04-22T15:18:43.197 INFO:tasks.workunit.client.0.smithi096.stdout:8/331: readlink d4/d1c/d34/l51 0 2024-04-22T15:18:43.198 INFO:tasks.workunit.client.0.smithi096.stdout:7/524: mknod d31/d32/d33/d54/ca5 0 2024-04-22T15:18:43.201 INFO:tasks.workunit.client.0.smithi096.stdout:4/308: unlink d5/d7/d2c/d3d/d4f/f51 0 2024-04-22T15:18:43.202 INFO:tasks.workunit.client.0.smithi096.stdout:4/309: chown d5/d7/d1f/f62 275 1 2024-04-22T15:18:43.203 INFO:tasks.workunit.client.0.smithi096.stdout:4/310: fsync d5/d7/d2c/d43/f68 0 2024-04-22T15:18:43.205 INFO:tasks.workunit.client.0.smithi096.stdout:6/314: mkdir d3/d28/dd/d11/d2c/d6f 0 2024-04-22T15:18:43.207 INFO:tasks.workunit.client.0.smithi096.stdout:9/288: write d1/d25/f2e [908940,40674] 0 2024-04-22T15:18:43.210 INFO:tasks.workunit.client.0.smithi096.stdout:0/379: unlink d7/d9/dc/d16/c51 0 2024-04-22T15:18:43.211 INFO:tasks.workunit.client.0.smithi096.stdout:3/232: symlink da/db/l48 0 2024-04-22T15:18:43.215 INFO:tasks.workunit.client.0.smithi096.stdout:7/525: creat d31/d32/d33/d6a/d83/fa6 x:0 0 0 2024-04-22T15:18:43.216 INFO:tasks.workunit.client.0.smithi096.stdout:7/526: write d31/d32/d33/d6a/d83/f96 [65325,29999] 0 2024-04-22T15:18:43.220 INFO:tasks.workunit.client.0.smithi096.stdout:1/495: dread d6/dc/d8a/d3f/d54/d32/d8c/d4a/f4d [0,4194304] 0 2024-04-22T15:18:43.224 INFO:tasks.workunit.client.0.smithi096.stdout:7/527: dwrite d31/d32/d33/d54/f62 [0,4194304] 0 2024-04-22T15:18:43.227 INFO:tasks.workunit.client.0.smithi096.stdout:9/289: write d1/f32 [411590,62026] 0 2024-04-22T15:18:43.228 INFO:tasks.workunit.client.0.smithi096.stdout:1/496: dread d6/dc/d8a/d3f/d62/d7d/f86 [0,4194304] 0 2024-04-22T15:18:43.228 INFO:tasks.workunit.client.0.smithi096.stdout:0/380: creat d7/d42/d4c/f7e x:0 0 0 2024-04-22T15:18:43.232 INFO:tasks.workunit.client.0.smithi096.stdout:3/233: rename da/f27 to da/db/dc/f49 0 2024-04-22T15:18:43.233 INFO:tasks.workunit.client.0.smithi096.stdout:3/234: stat da/db/d12/d22/f31 0 2024-04-22T15:18:43.239 INFO:tasks.workunit.client.0.smithi096.stdout:8/332: write d4/d1c/f3a [752502,62799] 0 2024-04-22T15:18:43.240 INFO:tasks.workunit.client.0.smithi096.stdout:8/333: chown d4/d10/d4d/l55 542 1 2024-04-22T15:18:43.241 INFO:tasks.workunit.client.0.smithi096.stdout:8/334: dread - d4/d1c/d34/f4f zero size 2024-04-22T15:18:43.241 INFO:tasks.workunit.client.0.smithi096.stdout:8/335: chown d4/d1c/d40/d46 22688740 1 2024-04-22T15:18:43.242 INFO:tasks.workunit.client.0.smithi096.stdout:8/336: chown d4/d1c/f72 18280826 1 2024-04-22T15:18:43.245 INFO:tasks.workunit.client.0.smithi096.stdout:6/315: write d3/d28/f1d [2197317,115589] 0 2024-04-22T15:18:43.247 INFO:tasks.workunit.client.0.smithi096.stdout:0/381: mknod d7/d9/dc/d16/c7f 0 2024-04-22T15:18:43.251 INFO:tasks.workunit.client.0.smithi096.stdout:1/497: dwrite d6/dc/d8a/d3f/d54/f98 [0,4194304] 0 2024-04-22T15:18:43.252 INFO:tasks.workunit.client.0.smithi096.stdout:9/290: rename d1/d4/c3c to d1/d25/c60 0 2024-04-22T15:18:43.252 INFO:tasks.workunit.client.0.smithi096.stdout:9/291: readlink d1/l56 0 2024-04-22T15:18:43.257 INFO:tasks.workunit.client.0.smithi096.stdout:3/235: read da/db/de/d19/d3e/f15 [3283157,104107] 0 2024-04-22T15:18:43.258 INFO:tasks.workunit.client.0.smithi096.stdout:4/311: creat d5/d7/d2c/d3d/f6a x:0 0 0 2024-04-22T15:18:43.259 INFO:tasks.workunit.client.0.smithi096.stdout:1/498: dwrite d6/dc/d8a/d3f/d54/d32/d8c/d45/f58 [0,4194304] 0 2024-04-22T15:18:43.259 INFO:tasks.workunit.client.0.smithi096.stdout:3/236: dread - da/db/de/d19/d3e/f42 zero size 2024-04-22T15:18:43.261 INFO:tasks.workunit.client.0.smithi096.stdout:4/312: truncate d5/d7/d1f/f55 158548 0 2024-04-22T15:18:43.264 INFO:tasks.workunit.client.0.smithi096.stdout:4/313: truncate d5/d7/d1f/f50 707351 0 2024-04-22T15:18:43.265 INFO:tasks.workunit.client.0.smithi096.stdout:4/314: dread - d5/d7/d1a/d25/f44 zero size 2024-04-22T15:18:43.266 INFO:tasks.workunit.client.0.smithi096.stdout:3/237: dread da/db/de/f40 [0,4194304] 0 2024-04-22T15:18:43.267 INFO:tasks.workunit.client.0.smithi096.stdout:4/315: write d5/d7/d1f/f42 [621231,32156] 0 2024-04-22T15:18:43.269 INFO:tasks.workunit.client.0.smithi096.stdout:3/238: write da/db/de/d19/d2d/f2e [2210293,121997] 0 2024-04-22T15:18:43.277 INFO:tasks.workunit.client.0.smithi096.stdout:8/337: creat d4/d71/f77 x:0 0 0 2024-04-22T15:18:43.279 INFO:tasks.workunit.client.0.smithi096.stdout:0/382: mknod d7/d9/d66/c80 0 2024-04-22T15:18:43.284 INFO:tasks.workunit.client.0.smithi096.stdout:6/316: truncate d3/d28/dd/d11/d2c/d44/d51/f60 3242891 0 2024-04-22T15:18:43.284 INFO:tasks.workunit.client.0.smithi096.stdout:0/383: dwrite d7/d9/dc/f56 [0,4194304] 0 2024-04-22T15:18:43.286 INFO:tasks.workunit.client.0.smithi096.stdout:4/316: mknod d5/d7/d1a/d61/c6b 0 2024-04-22T15:18:43.289 INFO:tasks.workunit.client.0.smithi096.stdout:3/239: symlink da/db/dc/l4a 0 2024-04-22T15:18:43.290 INFO:tasks.workunit.client.0.smithi096.stdout:3/240: write da/db/de/d19/d3e/f29 [4050205,117113] 0 2024-04-22T15:18:43.296 INFO:tasks.workunit.client.0.smithi096.stdout:7/528: getdents d31 0 2024-04-22T15:18:43.300 INFO:tasks.workunit.client.0.smithi096.stdout:6/317: unlink d3/d28/dd/f55 0 2024-04-22T15:18:43.303 INFO:tasks.workunit.client.0.smithi096.stdout:2/390: sync 2024-04-22T15:18:43.304 INFO:tasks.workunit.client.0.smithi096.stdout:4/317: write d5/f5e [1633293,16377] 0 2024-04-22T15:18:43.305 INFO:tasks.workunit.client.0.smithi096.stdout:4/318: write d5/d7/d1a/f38 [826589,86063] 0 2024-04-22T15:18:43.308 INFO:tasks.workunit.client.0.smithi096.stdout:4/319: read d5/d7/d1a/f1d [2098567,105974] 0 2024-04-22T15:18:43.312 INFO:tasks.workunit.client.0.smithi096.stdout:0/384: dread d7/d9/dc/d16/d18/f3d [0,4194304] 0 2024-04-22T15:18:43.313 INFO:tasks.workunit.client.0.smithi096.stdout:0/385: read d7/d9/dc/d16/d18/f3d [4020078,35582] 0 2024-04-22T15:18:43.313 INFO:tasks.workunit.client.0.smithi096.stdout:1/499: rename d6/dc/f6c to d6/f9d 0 2024-04-22T15:18:43.314 INFO:tasks.workunit.client.0.smithi096.stdout:0/386: chown d7/de/c31 154 1 2024-04-22T15:18:43.315 INFO:tasks.workunit.client.0.smithi096.stdout:7/529: mknod d31/d32/d33/d44/ca7 0 2024-04-22T15:18:43.317 INFO:tasks.workunit.client.0.smithi096.stdout:8/338: getdents d4/d10/d4d 0 2024-04-22T15:18:43.317 INFO:tasks.workunit.client.0.smithi096.stdout:3/241: dread da/db/de/d19/d2d/f2e [0,4194304] 0 2024-04-22T15:18:43.318 INFO:tasks.workunit.client.0.smithi096.stdout:8/339: write d4/d10/d25/d4c/f5d [2911935,76387] 0 2024-04-22T15:18:43.329 INFO:tasks.workunit.client.0.smithi096.stdout:9/292: truncate d1/f20 1165837 0 2024-04-22T15:18:43.329 INFO:tasks.workunit.client.0.smithi096.stdout:9/293: dread - d1/d25/f31 zero size 2024-04-22T15:18:43.335 INFO:tasks.workunit.client.0.smithi096.stdout:1/500: dwrite d6/dc/d8a/d3f/d62/f8d [0,4194304] 0 2024-04-22T15:18:43.338 INFO:tasks.workunit.client.0.smithi096.stdout:8/340: mknod d4/c78 0 2024-04-22T15:18:43.339 INFO:tasks.workunit.client.0.smithi096.stdout:2/391: dread d3/db/d2d/f33 [0,4194304] 0 2024-04-22T15:18:43.342 INFO:tasks.workunit.client.0.smithi096.stdout:1/501: dread d6/dc/d8a/f67 [0,4194304] 0 2024-04-22T15:18:43.343 INFO:tasks.workunit.client.0.smithi096.stdout:1/502: write d6/dc/d8a/d3f/d62/f8d [4534223,100661] 0 2024-04-22T15:18:43.347 INFO:tasks.workunit.client.0.smithi096.stdout:9/294: rmdir d1/d4/d9/dd 39 2024-04-22T15:18:43.350 INFO:tasks.workunit.client.0.smithi096.stdout:0/387: write d7/d9/d15/d35/f41 [752035,105654] 0 2024-04-22T15:18:43.353 INFO:tasks.workunit.client.0.smithi096.stdout:3/242: mknod da/d3f/c4b 0 2024-04-22T15:18:43.356 INFO:tasks.workunit.client.0.smithi096.stdout:8/341: creat d4/d10/d25/d38/f79 x:0 0 0 2024-04-22T15:18:43.356 INFO:tasks.workunit.client.0.smithi096.stdout:7/530: write d31/d32/d33/f35 [65913,5670] 0 2024-04-22T15:18:43.358 INFO:tasks.workunit.client.0.smithi096.stdout:6/318: creat d3/d28/dd/f70 x:0 0 0 2024-04-22T15:18:43.363 INFO:tasks.workunit.client.0.smithi096.stdout:7/531: dwrite d31/d32/d36/f3c [0,4194304] 0 2024-04-22T15:18:43.364 INFO:tasks.workunit.client.0.smithi096.stdout:1/503: chown d6/dc/d8a/d3f/d54/f21 43933479 1 2024-04-22T15:18:43.364 INFO:tasks.workunit.client.0.smithi096.stdout:7/532: chown f2b 134 1 2024-04-22T15:18:43.367 INFO:tasks.workunit.client.0.smithi096.stdout:9/295: mknod d1/d4/d9/d43/c61 0 2024-04-22T15:18:43.367 INFO:tasks.workunit.client.0.smithi096.stdout:5/326: sync 2024-04-22T15:18:43.368 INFO:tasks.workunit.client.0.smithi096.stdout:9/296: write d1/d4/d9/d43/f5c [1296405,19785] 0 2024-04-22T15:18:43.369 INFO:tasks.workunit.client.0.smithi096.stdout:2/392: dread d3/f49 [0,4194304] 0 2024-04-22T15:18:43.372 INFO:tasks.workunit.client.0.smithi096.stdout:0/388: creat d7/d9/d66/f81 x:0 0 0 2024-04-22T15:18:43.375 INFO:tasks.workunit.client.0.smithi096.stdout:0/389: chown d7/d9/d66/c80 3 1 2024-04-22T15:18:43.377 INFO:tasks.workunit.client.0.smithi096.stdout:9/297: dread d1/d4/f42 [0,4194304] 0 2024-04-22T15:18:43.378 INFO:tasks.workunit.client.0.smithi096.stdout:8/342: chown d4/d10/c67 970140 1 2024-04-22T15:18:43.379 INFO:tasks.workunit.client.0.smithi096.stdout:8/343: read d4/d10/f21 [2007737,1500] 0 2024-04-22T15:18:43.380 INFO:tasks.workunit.client.0.smithi096.stdout:5/327: dwrite d4/d11/d3d/d4f/f69 [0,4194304] 0 2024-04-22T15:18:43.383 INFO:tasks.workunit.client.0.smithi096.stdout:7/533: mkdir d31/d32/d33/d44/da8 0 2024-04-22T15:18:43.385 INFO:tasks.workunit.client.0.smithi096.stdout:4/320: rename f2 to d5/d7/d1a/f6c 0 2024-04-22T15:18:43.393 INFO:tasks.workunit.client.0.smithi096.stdout:3/243: creat da/db/de/d38/f4c x:0 0 0 2024-04-22T15:18:43.397 INFO:tasks.workunit.client.0.smithi096.stdout:3/244: dread da/db/de/d19/f2a [0,4194304] 0 2024-04-22T15:18:43.398 INFO:tasks.workunit.client.0.smithi096.stdout:0/390: creat d7/d9/d15/f82 x:0 0 0 2024-04-22T15:18:43.399 INFO:tasks.workunit.client.0.smithi096.stdout:3/245: truncate da/db/d12/d22/f3a 755873 0 2024-04-22T15:18:43.400 INFO:tasks.workunit.client.0.smithi096.stdout:3/246: chown da/db/de 0 1 2024-04-22T15:18:43.401 INFO:tasks.workunit.client.0.smithi096.stdout:9/298: mknod d1/df/c62 0 2024-04-22T15:18:43.407 INFO:tasks.workunit.client.0.smithi096.stdout:1/504: dwrite d6/f14 [0,4194304] 0 2024-04-22T15:18:43.408 INFO:tasks.workunit.client.0.smithi096.stdout:1/505: chown f2 7 1 2024-04-22T15:18:43.412 INFO:tasks.workunit.client.0.smithi096.stdout:4/321: mkdir d5/d7/d1a/d61/d6d 0 2024-04-22T15:18:43.414 INFO:tasks.workunit.client.0.smithi096.stdout:0/391: truncate d7/d9/dc/d16/d18/f39 112146 0 2024-04-22T15:18:43.416 INFO:tasks.workunit.client.0.smithi096.stdout:3/247: write da/db/de/d19/f2a [4877451,39981] 0 2024-04-22T15:18:43.417 INFO:tasks.workunit.client.0.smithi096.stdout:3/248: readlink l3 0 2024-04-22T15:18:43.419 INFO:tasks.workunit.client.0.smithi096.stdout:9/299: rename d1/d25/f31 to d1/d4/d9/d43/d57/f63 0 2024-04-22T15:18:43.421 INFO:tasks.workunit.client.0.smithi096.stdout:0/392: dwrite d7/de/d3a/f44 [0,4194304] 0 2024-04-22T15:18:43.422 INFO:tasks.workunit.client.0.smithi096.stdout:8/344: creat d4/d1c/d40/d46/d58/f7a x:0 0 0 2024-04-22T15:18:43.427 INFO:tasks.workunit.client.0.smithi096.stdout:5/328: symlink d4/d11/l6d 0 2024-04-22T15:18:43.429 INFO:tasks.workunit.client.0.smithi096.stdout:6/319: dwrite f0 [0,4194304] 0 2024-04-22T15:18:43.436 INFO:tasks.workunit.client.0.smithi096.stdout:7/534: dwrite f18 [0,4194304] 0 2024-04-22T15:18:43.445 INFO:tasks.workunit.client.0.smithi096.stdout:4/322: rename d5/fe to d5/d7/d2c/d43/f6e 0 2024-04-22T15:18:43.445 INFO:tasks.workunit.client.0.smithi096.stdout:4/323: stat d5/d7/d1a/d25/c3b 0 2024-04-22T15:18:43.451 INFO:tasks.workunit.client.0.smithi096.stdout:5/329: stat d4/d11/d3d/d35/d57/l6c 0 2024-04-22T15:18:43.457 INFO:tasks.workunit.client.0.smithi096.stdout:2/393: getdents d3 0 2024-04-22T15:18:43.460 INFO:tasks.workunit.client.0.smithi096.stdout:3/249: dwrite da/db/de/d19/d3e/f15 [0,4194304] 0 2024-04-22T15:18:43.461 INFO:tasks.workunit.client.0.smithi096.stdout:6/320: dread d3/d3a/d4b/f5b [0,4194304] 0 2024-04-22T15:18:43.462 INFO:tasks.workunit.client.0.smithi096.stdout:6/321: chown d3/d28/dd/d11/f32 0 1 2024-04-22T15:18:43.463 INFO:tasks.workunit.client.0.smithi096.stdout:7/535: rename d31/c52 to d31/d32/d33/d54/d43/ca9 0 2024-04-22T15:18:43.463 INFO:tasks.workunit.client.0.smithi096.stdout:3/250: rename da to da/db/d12/d22/d26/d4d 22 2024-04-22T15:18:43.465 INFO:tasks.workunit.client.0.smithi096.stdout:3/251: chown da/db/c2f 15 1 2024-04-22T15:18:43.466 INFO:tasks.workunit.client.0.smithi096.stdout:3/252: write da/db/d12/d22/f31 [987225,70639] 0 2024-04-22T15:18:43.467 INFO:tasks.workunit.client.0.smithi096.stdout:3/253: fsync da/db/de/d38/f4c 0 2024-04-22T15:18:43.467 INFO:tasks.workunit.client.0.smithi096.stdout:3/254: read - da/db/f3d zero size 2024-04-22T15:18:43.468 INFO:tasks.workunit.client.0.smithi096.stdout:1/506: write d6/dc/d8a/d3f/d54/d32/d8c/d45/f66 [511936,5437] 0 2024-04-22T15:18:43.468 INFO:tasks.workunit.client.0.smithi096.stdout:3/255: readlink da/db/dc/l4a 0 2024-04-22T15:18:43.470 INFO:tasks.workunit.client.0.smithi096.stdout:1/507: fdatasync d6/dc/d8a/d3f/d54/d32/d8c/f19 0 2024-04-22T15:18:43.471 INFO:tasks.workunit.client.0.smithi096.stdout:4/324: dread f1 [4194304,4194304] 0 2024-04-22T15:18:43.474 INFO:tasks.workunit.client.0.smithi096.stdout:5/330: creat d4/d11/d3d/d4d/f6e x:0 0 0 2024-04-22T15:18:43.476 INFO:tasks.workunit.client.0.smithi096.stdout:9/300: truncate d1/d4/f11 6477935 0 2024-04-22T15:18:43.478 INFO:tasks.workunit.client.0.smithi096.stdout:2/394: unlink d3/d1f/l4d 0 2024-04-22T15:18:43.479 INFO:tasks.workunit.client.0.smithi096.stdout:6/322: symlink d3/d28/d54/l71 0 2024-04-22T15:18:43.481 INFO:tasks.workunit.client.0.smithi096.stdout:7/536: rename c2f to d31/d32/d7c/caa 0 2024-04-22T15:18:43.497 INFO:tasks.workunit.client.0.smithi096.stdout:0/393: dwrite d7/d9/d15/f36 [0,4194304] 0 2024-04-22T15:18:43.497 INFO:tasks.workunit.client.0.smithi096.stdout:8/345: creat d4/d1c/d40/d46/f7b x:0 0 0 2024-04-22T15:18:43.499 INFO:tasks.workunit.client.0.smithi096.stdout:3/256: creat da/db/d12/d22/f4e x:0 0 0 2024-04-22T15:18:43.502 INFO:tasks.workunit.client.0.smithi096.stdout:7/537: creat d31/d32/d36/d5d/fab x:0 0 0 2024-04-22T15:18:43.503 INFO:tasks.workunit.client.0.smithi096.stdout:2/395: dread d3/db/d2d/f33 [4194304,4194304] 0 2024-04-22T15:18:43.504 INFO:tasks.workunit.client.0.smithi096.stdout:2/396: stat d3/d1f/d2f/f34 0 2024-04-22T15:18:43.504 INFO:tasks.workunit.client.0.smithi096.stdout:4/325: symlink d5/d7/d2c/l6f 0 2024-04-22T15:18:43.513 INFO:tasks.workunit.client.0.smithi096.stdout:3/257: creat da/db/de/d19/d3e/f4f x:0 0 0 2024-04-22T15:18:43.514 INFO:tasks.workunit.client.0.smithi096.stdout:6/323: symlink d3/l72 0 2024-04-22T15:18:43.515 INFO:tasks.workunit.client.0.smithi096.stdout:9/301: write d1/d4/f22 [125301,38996] 0 2024-04-22T15:18:43.517 INFO:tasks.workunit.client.0.smithi096.stdout:7/538: creat d31/d32/d7c/fac x:0 0 0 2024-04-22T15:18:43.518 INFO:tasks.workunit.client.0.smithi096.stdout:7/539: readlink d31/d32/d36/l74 0 2024-04-22T15:18:43.519 INFO:tasks.workunit.client.0.smithi096.stdout:2/397: creat d3/db/d2d/d46/f7c x:0 0 0 2024-04-22T15:18:43.520 INFO:tasks.workunit.client.0.smithi096.stdout:3/258: dwrite da/db/de/d19/f30 [4194304,4194304] 0 2024-04-22T15:18:43.521 INFO:tasks.workunit.client.0.smithi096.stdout:4/326: mknod d5/d7/d1a/d25/c70 0 2024-04-22T15:18:43.529 INFO:tasks.workunit.client.0.smithi096.stdout:3/259: dread da/db/de/d19/d3e/f15 [0,4194304] 0 2024-04-22T15:18:43.534 INFO:tasks.workunit.client.0.smithi096.stdout:3/260: read da/db/de/d19/d3e/f29 [747708,125272] 0 2024-04-22T15:18:43.536 INFO:tasks.workunit.client.0.smithi096.stdout:9/302: rename d1/d25/c4a to d1/d4/d9/d59/c64 0 2024-04-22T15:18:43.537 INFO:tasks.workunit.client.0.smithi096.stdout:7/540: dread f21 [0,4194304] 0 2024-04-22T15:18:43.539 INFO:tasks.workunit.client.0.smithi096.stdout:5/331: write d4/d11/f22 [2276596,85437] 0 2024-04-22T15:18:43.541 INFO:tasks.workunit.client.0.smithi096.stdout:5/332: read d4/d2c/d2f/f51 [3978991,22130] 0 2024-04-22T15:18:43.542 INFO:tasks.workunit.client.0.smithi096.stdout:8/346: link d4/l29 d4/d10/l7c 0 2024-04-22T15:18:43.544 INFO:tasks.workunit.client.0.smithi096.stdout:4/327: dread d5/f5e [4194304,4194304] 0 2024-04-22T15:18:43.548 INFO:tasks.workunit.client.0.smithi096.stdout:3/261: symlink da/db/dc/l50 0 2024-04-22T15:18:43.550 INFO:tasks.workunit.client.0.smithi096.stdout:9/303: creat d1/df/f65 x:0 0 0 2024-04-22T15:18:43.556 INFO:tasks.workunit.client.0.smithi096.stdout:7/541: dwrite f21 [0,4194304] 0 2024-04-22T15:18:43.559 INFO:tasks.workunit.client.0.smithi096.stdout:4/328: stat d5/d7/d2c/f2e 0 2024-04-22T15:18:43.560 INFO:tasks.workunit.client.0.smithi096.stdout:0/394: dwrite d7/d9/dc/d16/d18/f39 [0,4194304] 0 2024-04-22T15:18:43.561 INFO:tasks.workunit.client.0.smithi096.stdout:4/329: truncate d5/d7/d1a/d61/f64 748128 0 2024-04-22T15:18:43.562 INFO:tasks.workunit.client.0.smithi096.stdout:6/324: rmdir d3/d5a 0 2024-04-22T15:18:43.563 INFO:tasks.workunit.client.0.smithi096.stdout:4/330: dread - d5/d7/d2c/d3d/f6a zero size 2024-04-22T15:18:43.564 INFO:tasks.workunit.client.0.smithi096.stdout:3/262: creat da/db/de/d19/d2d/f51 x:0 0 0 2024-04-22T15:18:43.564 INFO:tasks.workunit.client.0.smithi096.stdout:4/331: write d5/d7/d2c/d3d/d4f/f60 [21499,47127] 0 2024-04-22T15:18:43.566 INFO:tasks.workunit.client.0.smithi096.stdout:3/263: fsync da/db/dc/d28/f45 0 2024-04-22T15:18:43.566 INFO:tasks.workunit.client.0.smithi096.stdout:2/398: write d3/db/df/d10/f1b [858351,4250] 0 2024-04-22T15:18:43.570 INFO:tasks.workunit.client.0.smithi096.stdout:5/333: mknod d4/d2c/d2f/d30/d5b/c6f 0 2024-04-22T15:18:43.574 INFO:tasks.workunit.client.0.smithi096.stdout:0/395: mknod d7/d9/dc/d16/c83 0 2024-04-22T15:18:43.575 INFO:tasks.workunit.client.0.smithi096.stdout:2/399: dwrite d3/d1f/d2f/f67 [0,4194304] 0 2024-04-22T15:18:43.576 INFO:tasks.workunit.client.0.smithi096.stdout:6/325: creat d3/d28/dd/d11/d2c/d3b/f73 x:0 0 0 2024-04-22T15:18:43.577 INFO:tasks.workunit.client.0.smithi096.stdout:6/326: stat d3/d28/dd/d11/d2c/f4e 0 2024-04-22T15:18:43.580 INFO:tasks.workunit.client.0.smithi096.stdout:7/542: mkdir d31/d32/dad 0 2024-04-22T15:18:43.585 INFO:tasks.workunit.client.0.smithi096.stdout:3/264: mkdir da/db/d12/d22/d26/d52 0 2024-04-22T15:18:43.595 INFO:tasks.workunit.client.0.smithi096.stdout:0/396: rename d7/d9/dc/f14 to d7/d9/dc/d16/d65/f84 0 2024-04-22T15:18:43.595 INFO:tasks.workunit.client.0.smithi096.stdout:9/304: dread d1/f45 [0,4194304] 0 2024-04-22T15:18:43.596 INFO:tasks.workunit.client.0.smithi096.stdout:6/327: symlink d3/l74 0 2024-04-22T15:18:43.597 INFO:tasks.workunit.client.0.smithi096.stdout:6/328: write f0 [3021275,84867] 0 2024-04-22T15:18:43.599 INFO:tasks.workunit.client.0.smithi096.stdout:3/265: rmdir da/db/de/d19 39 2024-04-22T15:18:43.601 INFO:tasks.workunit.client.0.smithi096.stdout:0/397: dread d7/d9/d15/d35/f41 [0,4194304] 0 2024-04-22T15:18:43.603 INFO:tasks.workunit.client.0.smithi096.stdout:7/543: link d31/d32/d33/d44/f80 d31/d32/d33/d93/fae 0 2024-04-22T15:18:43.604 INFO:tasks.workunit.client.0.smithi096.stdout:9/305: readlink d1/l4d 0 2024-04-22T15:18:43.605 INFO:tasks.workunit.client.0.smithi096.stdout:8/347: truncate d4/d10/f4e 416941 0 2024-04-22T15:18:43.610 INFO:tasks.workunit.client.0.smithi096.stdout:6/329: rmdir d3/d3a 39 2024-04-22T15:18:43.610 INFO:tasks.workunit.client.0.smithi096.stdout:0/398: dwrite d7/de/d5b/d62/d67/f70 [0,4194304] 0 2024-04-22T15:18:43.612 INFO:tasks.workunit.client.0.smithi096.stdout:6/330: chown d3/d28/dd/d11/d2c/d6f 23 1 2024-04-22T15:18:43.613 INFO:tasks.workunit.client.0.smithi096.stdout:6/331: fsync d3/d28/dd/d11/f41 0 2024-04-22T15:18:43.615 INFO:tasks.workunit.client.0.smithi096.stdout:3/266: truncate da/db/de/d19/d3e/f42 745239 0 2024-04-22T15:18:43.615 INFO:tasks.workunit.client.0.smithi096.stdout:8/348: rmdir d4/d71 39 2024-04-22T15:18:43.616 INFO:tasks.workunit.client.0.smithi096.stdout:8/349: fsync d4/d10/d25/f26 0 2024-04-22T15:18:43.620 INFO:tasks.workunit.client.0.smithi096.stdout:7/544: rename d31/d32/d33/d93/l95 to d31/d32/d33/da2/laf 0 2024-04-22T15:18:43.621 INFO:tasks.workunit.client.0.smithi096.stdout:5/334: truncate d4/fd 5101149 0 2024-04-22T15:18:43.621 INFO:tasks.workunit.client.0.smithi096.stdout:5/335: chown d4/d11/d3d/d4f 1331305 1 2024-04-22T15:18:43.623 INFO:tasks.workunit.client.0.smithi096.stdout:5/336: fsync d4/d11/d3d/d4d/d59/d5a/f67 0 2024-04-22T15:18:43.624 INFO:tasks.workunit.client.0.smithi096.stdout:5/337: write d4/f7 [3193465,36375] 0 2024-04-22T15:18:43.628 INFO:tasks.workunit.client.0.smithi096.stdout:9/306: rename d1/d4/d9/f3b to d1/d4/d9/d43/d57/d5b/f66 0 2024-04-22T15:18:43.633 INFO:tasks.workunit.client.0.smithi096.stdout:6/332: rename d3/d3a/d4b to d3/d3a/d75 0 2024-04-22T15:18:43.633 INFO:tasks.workunit.client.0.smithi096.stdout:6/333: fsync d3/d28/f1d 0 2024-04-22T15:18:43.635 INFO:tasks.workunit.client.0.smithi096.stdout:5/338: creat d4/d11/d3d/d4d/f70 x:0 0 0 2024-04-22T15:18:43.637 INFO:tasks.workunit.client.0.smithi096.stdout:0/399: rename d7/d9/dc/d16/f2e to d7/d9/dc/d16/d65/d7c/f85 0 2024-04-22T15:18:43.642 INFO:tasks.workunit.client.0.smithi096.stdout:8/350: creat d4/d1c/d40/d46/f7d x:0 0 0 2024-04-22T15:18:43.644 INFO:tasks.workunit.client.0.smithi096.stdout:6/334: creat d3/d28/d54/d68/f76 x:0 0 0 2024-04-22T15:18:43.646 INFO:tasks.workunit.client.0.smithi096.stdout:5/339: creat d4/d11/d3d/d4d/d59/d5a/d5c/f71 x:0 0 0 2024-04-22T15:18:43.648 INFO:tasks.workunit.client.0.smithi096.stdout:0/400: rename d7/d9/d15/d35/f41 to d7/d42/f86 0 2024-04-22T15:18:43.649 INFO:tasks.workunit.client.0.smithi096.stdout:8/351: mkdir d4/d2a/d7e 0 2024-04-22T15:18:43.652 INFO:tasks.workunit.client.0.smithi096.stdout:6/335: creat d3/d28/dd/d11/d2c/d44/f77 x:0 0 0 2024-04-22T15:18:43.653 INFO:tasks.workunit.client.0.smithi096.stdout:7/545: truncate d31/d32/d33/d5e/f6d 204111 0 2024-04-22T15:18:43.654 INFO:tasks.workunit.client.0.smithi096.stdout:5/340: symlink d4/d11/d25/l72 0 2024-04-22T15:18:43.656 INFO:tasks.workunit.client.0.smithi096.stdout:0/401: rmdir d7/d9/dc/d16 39 2024-04-22T15:18:43.660 INFO:tasks.workunit.client.0.smithi096.stdout:6/336: unlink d3/d28/d54/f6b 0 2024-04-22T15:18:43.662 INFO:tasks.workunit.client.0.smithi096.stdout:6/337: write d3/d28/dd/f16 [1029538,30188] 0 2024-04-22T15:18:43.665 INFO:tasks.workunit.client.0.smithi096.stdout:0/402: dwrite d7/d9/dc/f56 [0,4194304] 0 2024-04-22T15:18:43.666 INFO:tasks.workunit.client.0.smithi096.stdout:5/341: mknod d4/d11/d3d/d4d/d59/d5a/d5c/c73 0 2024-04-22T15:18:43.671 INFO:tasks.workunit.client.0.smithi096.stdout:6/338: dwrite d3/d28/dd/d11/d2c/f39 [0,4194304] 0 2024-04-22T15:18:43.680 INFO:tasks.workunit.client.0.smithi096.stdout:0/403: chown d7/d9/dc/d16/d18/c22 362774912 1 2024-04-22T15:18:43.683 INFO:tasks.workunit.client.0.smithi096.stdout:8/352: truncate d4/d10/d25/d4c/f5e 3642303 0 2024-04-22T15:18:43.687 INFO:tasks.workunit.client.0.smithi096.stdout:0/404: rmdir d7/d9/d15 39 2024-04-22T15:18:43.689 INFO:tasks.workunit.client.0.smithi096.stdout:6/339: dread d3/d28/f1d [0,4194304] 0 2024-04-22T15:18:43.690 INFO:tasks.workunit.client.0.smithi096.stdout:0/405: symlink d7/d9/dc/d16/d65/l87 0 2024-04-22T15:18:43.691 INFO:tasks.workunit.client.0.smithi096.stdout:6/340: truncate d3/d28/dd/f4a 1553507 0 2024-04-22T15:18:43.691 INFO:tasks.workunit.client.0.smithi096.stdout:7/546: truncate d31/d32/f3f 1738722 0 2024-04-22T15:18:43.691 INFO:tasks.workunit.client.0.smithi096.stdout:0/406: chown d7/d9/dc 0 1 2024-04-22T15:18:43.692 INFO:tasks.workunit.client.0.smithi096.stdout:0/407: readlink d7/de/l1b 0 2024-04-22T15:18:43.697 INFO:tasks.workunit.client.0.smithi096.stdout:0/408: read d7/d9/dc/f29 [565220,34605] 0 2024-04-22T15:18:43.698 INFO:tasks.workunit.client.0.smithi096.stdout:7/547: rename d31/d32/d33/d54/ca5 to d31/d32/d33/d5e/cb0 0 2024-04-22T15:18:43.701 INFO:tasks.workunit.client.0.smithi096.stdout:6/341: dwrite d3/d28/dd/d11/d2c/d3b/f63 [4194304,4194304] 0 2024-04-22T15:18:43.701 INFO:tasks.workunit.client.0.smithi096.stdout:5/342: truncate d4/d2c/d2f/f3a 973204 0 2024-04-22T15:18:43.702 INFO:tasks.workunit.client.0.smithi096.stdout:8/353: dread d4/d10/d25/f44 [0,4194304] 0 2024-04-22T15:18:43.702 INFO:tasks.workunit.client.0.smithi096.stdout:0/409: mknod d7/d9/d59/c88 0 2024-04-22T15:18:43.702 INFO:tasks.workunit.client.0.smithi096.stdout:7/548: write d31/d32/d7c/fac [423309,78640] 0 2024-04-22T15:18:43.705 INFO:tasks.workunit.client.0.smithi096.stdout:5/343: readlink d4/d1f/l20 0 2024-04-22T15:18:43.705 INFO:tasks.workunit.client.0.smithi096.stdout:7/549: read f21 [2978428,26670] 0 2024-04-22T15:18:43.705 INFO:tasks.workunit.client.0.smithi096.stdout:6/342: stat d3/d28/dd/d11/d2c/d44/f67 0 2024-04-22T15:18:43.706 INFO:tasks.workunit.client.0.smithi096.stdout:0/410: dread - d7/d9/dc/d16/f5d zero size 2024-04-22T15:18:43.707 INFO:tasks.workunit.client.0.smithi096.stdout:8/354: truncate d4/d10/d25/f6c 144779 0 2024-04-22T15:18:43.708 INFO:tasks.workunit.client.0.smithi096.stdout:8/355: chown d4/c61 4 1 2024-04-22T15:18:43.709 INFO:tasks.workunit.client.0.smithi096.stdout:5/344: symlink d4/d11/d3d/d4f/l74 0 2024-04-22T15:18:43.711 INFO:tasks.workunit.client.0.smithi096.stdout:5/345: truncate d4/d11/d3d/d4d/f70 266932 0 2024-04-22T15:18:43.711 INFO:tasks.workunit.client.0.smithi096.stdout:5/346: readlink d4/l16 0 2024-04-22T15:18:43.720 INFO:tasks.workunit.client.0.smithi096.stdout:5/347: creat d4/d11/d3d/d4d/d59/d5a/d5c/f75 x:0 0 0 2024-04-22T15:18:43.721 INFO:tasks.workunit.client.0.smithi096.stdout:5/348: write d4/d11/d3d/d4d/f70 [744724,128532] 0 2024-04-22T15:18:43.723 INFO:tasks.workunit.client.0.smithi096.stdout:5/349: readlink d4/d11/d3d/d35/d57/l64 0 2024-04-22T15:18:43.725 INFO:tasks.workunit.client.0.smithi096.stdout:6/343: mkdir d3/d28/dd/d11/d2c/d44/d51/d78 0 2024-04-22T15:18:43.726 INFO:tasks.workunit.client.0.smithi096.stdout:6/344: write d3/d28/dd/d11/d2c/d44/f77 [887898,85024] 0 2024-04-22T15:18:43.727 INFO:tasks.workunit.client.0.smithi096.stdout:7/550: creat d31/d32/d33/fb1 x:0 0 0 2024-04-22T15:18:43.727 INFO:tasks.workunit.client.0.smithi096.stdout:5/350: read d4/d2c/d2f/d6b/f50 [1319276,63143] 0 2024-04-22T15:18:43.728 INFO:tasks.workunit.client.0.smithi096.stdout:6/345: chown d3/l6c 2 1 2024-04-22T15:18:43.734 INFO:tasks.workunit.client.0.smithi096.stdout:7/551: symlink d31/d32/d36/d5d/lb2 0 2024-04-22T15:18:43.735 INFO:tasks.workunit.client.0.smithi096.stdout:5/351: dwrite d4/d11/d3d/d4d/f56 [0,4194304] 0 2024-04-22T15:18:43.736 INFO:tasks.workunit.client.0.smithi096.stdout:7/552: chown d31/d32/d7c/c9e 0 1 2024-04-22T15:18:43.736 INFO:tasks.workunit.client.0.smithi096.stdout:6/346: creat d3/d28/dd/d11/d2c/d3b/f79 x:0 0 0 2024-04-22T15:18:43.742 INFO:tasks.workunit.client.0.smithi096.stdout:8/356: dread d4/f32 [0,4194304] 0 2024-04-22T15:18:43.743 INFO:tasks.workunit.client.0.smithi096.stdout:7/553: symlink d31/d32/d36/d75/lb3 0 2024-04-22T15:18:43.743 INFO:tasks.workunit.client.0.smithi096.stdout:0/411: dwrite d7/d42/f86 [0,4194304] 0 2024-04-22T15:18:43.746 INFO:tasks.workunit.client.0.smithi096.stdout:0/412: dread - d7/d9/dc/f57 zero size 2024-04-22T15:18:43.747 INFO:tasks.workunit.client.0.smithi096.stdout:7/554: truncate d31/d32/d33/d54/f64 4498029 0 2024-04-22T15:18:43.751 INFO:tasks.workunit.client.0.smithi096.stdout:5/352: getdents d4/d11 0 2024-04-22T15:18:43.754 INFO:tasks.workunit.client.0.smithi096.stdout:8/357: mkdir d4/d10/d25/d4c/d76/d7f 0 2024-04-22T15:18:43.760 INFO:tasks.workunit.client.0.smithi096.stdout:5/353: mkdir d4/d76 0 2024-04-22T15:18:43.762 INFO:tasks.workunit.client.0.smithi096.stdout:8/358: symlink d4/d10/d25/d38/l80 0 2024-04-22T15:18:43.762 INFO:tasks.workunit.client.0.smithi096.stdout:0/413: link d7/d9/l5e d7/d42/d4c/d55/l89 0 2024-04-22T15:18:43.763 INFO:tasks.workunit.client.0.smithi096.stdout:7/555: link c24 d31/d32/d33/d44/cb4 0 2024-04-22T15:18:43.766 INFO:tasks.workunit.client.0.smithi096.stdout:6/347: dwrite d3/d28/dd/f25 [0,4194304] 0 2024-04-22T15:18:43.767 INFO:tasks.workunit.client.0.smithi096.stdout:5/354: creat d4/d2c/d2f/d6b/f77 x:0 0 0 2024-04-22T15:18:43.768 INFO:tasks.workunit.client.0.smithi096.stdout:0/414: stat d7/d9/c3c 0 2024-04-22T15:18:43.769 INFO:tasks.workunit.client.0.smithi096.stdout:6/348: chown d3/d28/dd/d11/d2c 1142 1 2024-04-22T15:18:43.769 INFO:tasks.workunit.client.0.smithi096.stdout:0/415: readlink d7/de/ld 0 2024-04-22T15:18:43.771 INFO:tasks.workunit.client.0.smithi096.stdout:8/359: rename d4/d1c/d40/d48/f63 to d4/d10/d25/f81 0 2024-04-22T15:18:43.771 INFO:tasks.workunit.client.0.smithi096.stdout:6/349: read d3/d28/f21 [51350,128976] 0 2024-04-22T15:18:43.772 INFO:tasks.workunit.client.0.smithi096.stdout:8/360: truncate d4/d1c/d62/f6b 26252 0 2024-04-22T15:18:43.773 INFO:tasks.workunit.client.0.smithi096.stdout:5/355: symlink d4/d11/d3d/d4d/l78 0 2024-04-22T15:18:43.776 INFO:tasks.workunit.client.0.smithi096.stdout:0/416: mknod d7/de/d3a/c8a 0 2024-04-22T15:18:43.781 INFO:tasks.workunit.client.0.smithi096.stdout:8/361: dwrite d4/d1c/d40/d46/d58/f60 [0,4194304] 0 2024-04-22T15:18:43.784 INFO:tasks.workunit.client.0.smithi096.stdout:7/556: creat d31/d32/d33/fb5 x:0 0 0 2024-04-22T15:18:43.792 INFO:tasks.workunit.client.0.smithi096.stdout:7/557: unlink d31/d32/d36/d5d/fab 0 2024-04-22T15:18:43.798 INFO:tasks.workunit.client.0.smithi096.stdout:7/558: dwrite d31/d32/d33/d5e/f63 [0,4194304] 0 2024-04-22T15:18:43.842 INFO:tasks.workunit.client.0.smithi096.stdout:7/559: read - d31/d32/f6c zero size 2024-04-22T15:18:43.843 INFO:tasks.workunit.client.0.smithi096.stdout:0/417: write d7/d9/dc/d16/d65/f84 [1953177,50635] 0 2024-04-22T15:18:43.844 INFO:tasks.workunit.client.0.smithi096.stdout:5/356: dread f2 [0,4194304] 0 2024-04-22T15:18:43.846 INFO:tasks.workunit.client.0.smithi096.stdout:7/560: truncate d31/f8f 475894 0 2024-04-22T15:18:43.847 INFO:tasks.workunit.client.0.smithi096.stdout:7/561: chown d31/d32/d33/d54/f51 2386901 1 2024-04-22T15:18:43.848 INFO:tasks.workunit.client.0.smithi096.stdout:0/418: rename d7/d9/dc/d16/d18/f23 to d7/d9/dc/d26/f8b 0 2024-04-22T15:18:43.855 INFO:tasks.workunit.client.0.smithi096.stdout:7/562: dwrite d31/d32/d33/d6a/d83/f96 [0,4194304] 0 2024-04-22T15:18:44.018 INFO:tasks.workunit.client.0.smithi096.stdout:5/357: link d4/f1c d4/d11/d3d/d4d/d59/d5a/f79 0 2024-04-22T15:18:44.020 INFO:tasks.workunit.client.0.smithi096.stdout:7/563: symlink d31/d32/d33/d6a/d83/lb6 0 2024-04-22T15:18:44.020 INFO:tasks.workunit.client.0.smithi096.stdout:7/564: chown d31 48937 1 2024-04-22T15:18:44.021 INFO:tasks.workunit.client.0.smithi096.stdout:7/565: fsync d31/d32/d36/d75/f9a 0 2024-04-22T15:18:44.023 INFO:tasks.workunit.client.0.smithi096.stdout:7/566: symlink d31/d32/d33/d6a/d83/lb7 0 2024-04-22T15:18:44.026 INFO:tasks.workunit.client.0.smithi096.stdout:7/567: symlink d31/d32/d33/d54/d43/lb8 0 2024-04-22T15:18:44.027 INFO:tasks.workunit.client.0.smithi096.stdout:7/568: fdatasync d31/d32/d36/d75/f86 0 2024-04-22T15:18:44.029 INFO:tasks.workunit.client.0.smithi096.stdout:7/569: creat d31/d32/d33/da2/fb9 x:0 0 0 2024-04-22T15:18:44.029 INFO:tasks.workunit.client.0.smithi096.stdout:7/570: dread - d31/d32/d33/da2/fb9 zero size 2024-04-22T15:18:44.035 INFO:tasks.workunit.client.0.smithi096.stdout:5/358: dread d4/d11/d25/f48 [0,4194304] 0 2024-04-22T15:18:44.037 INFO:tasks.workunit.client.0.smithi096.stdout:7/571: dwrite d31/d32/f55 [8388608,4194304] 0 2024-04-22T15:18:44.037 INFO:tasks.workunit.client.0.smithi096.stdout:7/572: fsync f18 0 2024-04-22T15:18:44.038 INFO:tasks.workunit.client.0.smithi096.stdout:5/359: creat d4/d11/f7a x:0 0 0 2024-04-22T15:18:44.042 INFO:tasks.workunit.client.0.smithi096.stdout:7/573: readlink d31/d32/d33/d6a/l81 0 2024-04-22T15:18:44.044 INFO:tasks.workunit.client.0.smithi096.stdout:7/574: write d31/d32/d33/f57 [695869,17486] 0 2024-04-22T15:18:44.045 INFO:tasks.workunit.client.0.smithi096.stdout:0/419: truncate d7/d9/f24 4190431 0 2024-04-22T15:18:44.046 INFO:tasks.workunit.client.0.smithi096.stdout:0/420: chown d7/d9/dc/d16/f3b 14 1 2024-04-22T15:18:44.052 INFO:tasks.workunit.client.0.smithi096.stdout:0/421: fdatasync d7/d9/dc/d16/d18/f68 0 2024-04-22T15:18:44.053 INFO:tasks.workunit.client.0.smithi096.stdout:5/360: creat d4/d2c/f7b x:0 0 0 2024-04-22T15:18:44.053 INFO:tasks.workunit.client.0.smithi096.stdout:0/422: stat d7/de/d5b/d62 0 2024-04-22T15:18:44.055 INFO:tasks.workunit.client.0.smithi096.stdout:0/423: fsync d7/d9/d66/f81 0 2024-04-22T15:18:44.056 INFO:tasks.workunit.client.0.smithi096.stdout:7/575: dwrite d31/d32/d33/d54/f8b [0,4194304] 0 2024-04-22T15:18:44.062 INFO:tasks.workunit.client.0.smithi096.stdout:7/576: mkdir d31/d32/d33/d6a/d83/dba 0 2024-04-22T15:18:44.062 INFO:tasks.workunit.client.0.smithi096.stdout:0/424: dwrite d7/d9/dc/d16/f5d [0,4194304] 0 2024-04-22T15:18:44.068 INFO:tasks.workunit.client.0.smithi096.stdout:7/577: creat d31/d32/d33/d93/fbb x:0 0 0 2024-04-22T15:18:44.070 INFO:tasks.workunit.client.0.smithi096.stdout:0/425: mknod d7/d9/dc/d16/d18/c8c 0 2024-04-22T15:18:44.071 INFO:tasks.workunit.client.0.smithi096.stdout:0/426: chown d7/de/d5b 1045034623 1 2024-04-22T15:18:44.072 INFO:tasks.workunit.client.0.smithi096.stdout:7/578: creat d31/d32/d36/d5d/fbc x:0 0 0 2024-04-22T15:18:44.073 INFO:tasks.workunit.client.0.smithi096.stdout:7/579: fdatasync d31/d32/f59 0 2024-04-22T15:18:44.076 INFO:tasks.workunit.client.0.smithi096.stdout:0/427: dread d7/de/d3a/f44 [0,4194304] 0 2024-04-22T15:18:44.080 INFO:tasks.workunit.client.0.smithi096.stdout:7/580: dwrite d31/d32/d36/f71 [0,4194304] 0 2024-04-22T15:18:44.090 INFO:tasks.workunit.client.0.smithi096.stdout:0/428: dwrite d7/d9/dc/d26/f8b [0,4194304] 0 2024-04-22T15:18:44.101 INFO:tasks.workunit.client.0.smithi096.stdout:0/429: creat d7/d9/dc/f8d x:0 0 0 2024-04-22T15:18:44.104 INFO:tasks.workunit.client.0.smithi096.stdout:0/430: unlink d7/d9/dc/d16/d18/f3f 0 2024-04-22T15:18:44.111 INFO:tasks.workunit.client.0.smithi096.stdout:7/581: write d31/d32/d36/f70 [580753,113930] 0 2024-04-22T15:18:44.113 INFO:tasks.workunit.client.0.smithi096.stdout:0/431: link d7/d9/l5e d7/d9/l8e 0 2024-04-22T15:18:44.123 INFO:tasks.workunit.client.0.smithi096.stdout:7/582: getdents d31/d32/d33/d54/d43 0 2024-04-22T15:18:44.124 INFO:tasks.workunit.client.0.smithi096.stdout:0/432: dwrite d7/d9/f40 [4194304,4194304] 0 2024-04-22T15:18:44.126 INFO:tasks.workunit.client.0.smithi096.stdout:7/583: read d31/d32/d33/f58 [272137,65852] 0 2024-04-22T15:18:44.132 INFO:tasks.workunit.client.0.smithi096.stdout:7/584: rename d31/d32/d33/l38 to d31/d32/dad/lbd 0 2024-04-22T15:18:44.133 INFO:tasks.workunit.client.0.smithi096.stdout:7/585: fdatasync d31/d32/d33/d54/f89 0 2024-04-22T15:18:44.134 INFO:tasks.workunit.client.0.smithi096.stdout:7/586: readlink d31/d32/d33/d6a/l81 0 2024-04-22T15:18:44.134 INFO:tasks.workunit.client.0.smithi096.stdout:0/433: mkdir d7/d8f 0 2024-04-22T15:18:44.135 INFO:tasks.workunit.client.0.smithi096.stdout:7/587: rename d31/d32/d36/d5d to d31/d32/d36/d5d/dbe 22 2024-04-22T15:18:44.141 INFO:tasks.workunit.client.0.smithi096.stdout:7/588: mkdir d31/d32/d33/d44/da8/dbf 0 2024-04-22T15:18:44.143 INFO:tasks.workunit.client.0.smithi096.stdout:7/589: rmdir d31/d32/d33/d54/d43 39 2024-04-22T15:18:44.144 INFO:tasks.workunit.client.0.smithi096.stdout:7/590: write d31/d32/d36/d75/f86 [1004517,22156] 0 2024-04-22T15:18:44.145 INFO:tasks.workunit.client.0.smithi096.stdout:0/434: getdents d7/d9/dc/d16 0 2024-04-22T15:18:44.146 INFO:tasks.workunit.client.0.smithi096.stdout:0/435: stat d7/d42 0 2024-04-22T15:18:44.150 INFO:tasks.workunit.client.0.smithi096.stdout:7/591: mkdir d31/d32/dc0 0 2024-04-22T15:18:44.152 INFO:tasks.workunit.client.0.smithi096.stdout:0/436: rename d7/d9/d15/c6d to d7/d9/d66/c90 0 2024-04-22T15:18:44.153 INFO:tasks.workunit.client.0.smithi096.stdout:7/592: dread f18 [0,4194304] 0 2024-04-22T15:18:44.158 INFO:tasks.workunit.client.0.smithi096.stdout:0/437: mknod d7/d9/c91 0 2024-04-22T15:18:44.159 INFO:tasks.workunit.client.0.smithi096.stdout:7/593: dwrite d31/d32/d36/f3c [0,4194304] 0 2024-04-22T15:18:44.167 INFO:tasks.workunit.client.0.smithi096.stdout:0/438: getdents d7/de/d3a 0 2024-04-22T15:18:44.170 INFO:tasks.workunit.client.0.smithi096.stdout:0/439: creat d7/d9/dc/d16/d18/f92 x:0 0 0 2024-04-22T15:18:44.171 INFO:tasks.workunit.client.0.smithi096.stdout:0/440: chown d7/de 35630383 1 2024-04-22T15:18:44.175 INFO:tasks.workunit.client.0.smithi096.stdout:0/441: write d7/d9/dc/d16/d18/f3d [2896355,108869] 0 2024-04-22T15:18:44.176 INFO:tasks.workunit.client.0.smithi096.stdout:0/442: readlink d7/d9/dc/d26/l5f 0 2024-04-22T15:18:44.182 INFO:tasks.workunit.client.0.smithi096.stdout:0/443: dwrite d7/de/d5b/d62/d67/f70 [0,4194304] 0 2024-04-22T15:18:44.187 INFO:tasks.workunit.client.0.smithi096.stdout:0/444: dread d7/d9/dc/d16/f5d [0,4194304] 0 2024-04-22T15:18:44.191 INFO:tasks.workunit.client.0.smithi096.stdout:0/445: creat d7/d42/f93 x:0 0 0 2024-04-22T15:18:44.195 INFO:tasks.workunit.client.0.smithi096.stdout:0/446: mkdir d7/de/d5b/d64/d94 0 2024-04-22T15:18:44.199 INFO:tasks.workunit.client.0.smithi096.stdout:0/447: unlink d7/d9/dc/d16/d18/f54 0 2024-04-22T15:18:44.212 INFO:tasks.workunit.client.0.smithi096.stdout:0/448: rename d7/d9 to d7/de/d5b/d95 0 2024-04-22T15:18:44.216 INFO:tasks.workunit.client.0.smithi096.stdout:0/449: creat d7/de/d5b/d95/dc/f96 x:0 0 0 2024-04-22T15:18:44.217 INFO:tasks.workunit.client.0.smithi096.stdout:0/450: write d7/de/d5b/d95/dc/d16/d65/f84 [2097628,21895] 0 2024-04-22T15:18:44.218 INFO:tasks.workunit.client.0.smithi096.stdout:0/451: readlink d7/de/d5b/d95/dc/l52 0 2024-04-22T15:18:44.220 INFO:tasks.workunit.client.0.smithi096.stdout:0/452: write d7/de/d5b/d95/dc/f8d [558679,81832] 0 2024-04-22T15:18:44.220 INFO:tasks.workunit.client.0.smithi096.stdout:0/453: stat d7/de/d5b/d95/d15/l43 0 2024-04-22T15:18:44.223 INFO:tasks.workunit.client.0.smithi096.stdout:0/454: getdents d7/d42/d4c/d55 0 2024-04-22T15:18:44.235 INFO:tasks.workunit.client.0.smithi096.stdout:1/508: sync 2024-04-22T15:18:44.237 INFO:tasks.workunit.client.0.smithi096.stdout:1/509: creat d6/dc/d8a/d3f/d54/d32/d3b/f9e x:0 0 0 2024-04-22T15:18:44.244 INFO:tasks.workunit.client.0.smithi096.stdout:1/510: link c4 d6/dc/d8a/d3f/d62/d7d/c9f 0 2024-04-22T15:18:44.247 INFO:tasks.workunit.client.0.smithi096.stdout:1/511: truncate d6/dc/d8a/d3f/d54/f43 2984 0 2024-04-22T15:18:44.249 INFO:tasks.workunit.client.0.smithi096.stdout:1/512: fsync d6/dc/d8a/d3f/d62/d7d/f86 0 2024-04-22T15:18:44.251 INFO:tasks.workunit.client.0.smithi096.stdout:1/513: getdents d6/dc/d8a/d3f/d54/d32/d82 0 2024-04-22T15:18:44.252 INFO:tasks.workunit.client.0.smithi096.stdout:1/514: read - d6/dc/d8a/d29/f59 zero size 2024-04-22T15:18:44.257 INFO:tasks.workunit.client.0.smithi096.stdout:0/455: dwrite d7/de/d5b/d95/dc/d16/d18/f2b [0,4194304] 0 2024-04-22T15:18:44.261 INFO:tasks.workunit.client.0.smithi096.stdout:1/515: dread d6/dc/d8a/d3f/d54/d32/d8c/d45/f50 [0,4194304] 0 2024-04-22T15:18:44.262 INFO:tasks.workunit.client.0.smithi096.stdout:0/456: unlink d7/de/d5b/d95/l8e 0 2024-04-22T15:18:44.263 INFO:tasks.workunit.client.0.smithi096.stdout:1/516: chown d6/dc/f41 88329966 1 2024-04-22T15:18:44.267 INFO:tasks.workunit.client.0.smithi096.stdout:0/457: dread d7/de/d5b/d95/d15/f36 [0,4194304] 0 2024-04-22T15:18:44.269 INFO:tasks.workunit.client.0.smithi096.stdout:0/458: chown d7/de/d5b/d95/dc/d26/f8b 5 1 2024-04-22T15:18:44.269 INFO:tasks.workunit.client.0.smithi096.stdout:1/517: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d3d/f63 [0,4194304] 0 2024-04-22T15:18:44.271 INFO:tasks.workunit.client.0.smithi096.stdout:0/459: dread - d7/de/d5b/d95/dc/d16/d18/f68 zero size 2024-04-22T15:18:44.275 INFO:tasks.workunit.client.0.smithi096.stdout:1/518: rmdir d6/dc/d8a/d3f/d54/d32/d8c/d45/d7b 39 2024-04-22T15:18:44.277 INFO:tasks.workunit.client.0.smithi096.stdout:0/460: rmdir d7/de/d5b/d95/d59 39 2024-04-22T15:18:44.278 INFO:tasks.workunit.client.0.smithi096.stdout:0/461: fdatasync d7/de/d5b/d95/d15/f82 0 2024-04-22T15:18:44.279 INFO:tasks.workunit.client.0.smithi096.stdout:0/462: write d7/de/d5b/d95/dc/d26/f8b [1035612,26954] 0 2024-04-22T15:18:44.280 INFO:tasks.workunit.client.0.smithi096.stdout:1/519: creat d6/dc/d8a/d3f/d62/d7d/d8e/fa0 x:0 0 0 2024-04-22T15:18:44.295 INFO:tasks.workunit.client.0.smithi096.stdout:0/463: dwrite d7/de/d5b/d95/dc/d16/d18/f30 [0,4194304] 0 2024-04-22T15:18:44.299 INFO:tasks.workunit.client.0.smithi096.stdout:0/464: unlink d7/de/d5b/d95/dc/d16/d18/f3d 0 2024-04-22T15:18:44.301 INFO:tasks.workunit.client.0.smithi096.stdout:1/520: dread d6/dc/d8a/d3f/d62/d7d/f38 [0,4194304] 0 2024-04-22T15:18:44.301 INFO:tasks.workunit.client.0.smithi096.stdout:0/465: write d7/de/d5b/d95/dc/d16/d18/f92 [858642,130697] 0 2024-04-22T15:18:44.304 INFO:tasks.workunit.client.0.smithi096.stdout:0/466: mknod d7/de/d5b/d95/d15/c97 0 2024-04-22T15:18:44.305 INFO:tasks.workunit.client.0.smithi096.stdout:0/467: readlink d7/de/d5b/d95/dc/d26/l5f 0 2024-04-22T15:18:44.305 INFO:tasks.workunit.client.0.smithi096.stdout:1/521: creat d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fa1 x:0 0 0 2024-04-22T15:18:44.308 INFO:tasks.workunit.client.0.smithi096.stdout:0/468: getdents d7/de/d3a/d61/d79 0 2024-04-22T15:18:44.311 INFO:tasks.workunit.client.0.smithi096.stdout:0/469: mkdir d7/d42/d4c/d55/d98 0 2024-04-22T15:18:44.312 INFO:tasks.workunit.client.0.smithi096.stdout:1/522: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d56/f44 [0,4194304] 0 2024-04-22T15:18:44.314 INFO:tasks.workunit.client.0.smithi096.stdout:0/470: mknod d7/d42/d4c/d55/d98/c99 0 2024-04-22T15:18:44.316 INFO:tasks.workunit.client.0.smithi096.stdout:1/523: mkdir d6/dc/d8a/d3f/d62/da2 0 2024-04-22T15:18:44.317 INFO:tasks.workunit.client.0.smithi096.stdout:0/471: readlink d7/de/d5b/d95/d59/l5c 0 2024-04-22T15:18:44.320 INFO:tasks.workunit.client.0.smithi096.stdout:0/472: creat d7/de/d5b/d64/f9a x:0 0 0 2024-04-22T15:18:44.323 INFO:tasks.workunit.client.0.smithi096.stdout:1/524: dwrite d6/f8f [0,4194304] 0 2024-04-22T15:18:44.325 INFO:tasks.workunit.client.0.smithi096.stdout:1/525: read - d6/dc/d8a/d3f/d54/d32/d3b/d3d/f8b zero size 2024-04-22T15:18:44.326 INFO:tasks.workunit.client.0.smithi096.stdout:0/473: dread d7/de/d5b/d62/d67/f70 [0,4194304] 0 2024-04-22T15:18:44.328 INFO:tasks.workunit.client.0.smithi096.stdout:1/526: creat d6/dc/d8a/d3f/d54/d32/fa3 x:0 0 0 2024-04-22T15:18:44.332 INFO:tasks.workunit.client.0.smithi096.stdout:1/527: creat d6/dc/d8a/d3f/d54/d32/d82/fa4 x:0 0 0 2024-04-22T15:18:44.334 INFO:tasks.workunit.client.0.smithi096.stdout:0/474: dwrite d7/de/d5b/d95/dc/d16/f3b [0,4194304] 0 2024-04-22T15:18:44.336 INFO:tasks.workunit.client.0.smithi096.stdout:1/528: creat d6/dc/d8a/d3f/d54/fa5 x:0 0 0 2024-04-22T15:18:44.338 INFO:tasks.workunit.client.0.smithi096.stdout:0/475: creat d7/de/d5b/d95/dc/d16/d65/d7c/f9b x:0 0 0 2024-04-22T15:18:44.340 INFO:tasks.workunit.client.0.smithi096.stdout:1/529: creat d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e/fa6 x:0 0 0 2024-04-22T15:18:44.341 INFO:tasks.workunit.client.0.smithi096.stdout:0/476: mkdir d7/d42/d4c/d55/d9c 0 2024-04-22T15:18:44.345 INFO:tasks.workunit.client.0.smithi096.stdout:0/477: symlink d7/de/d5b/d64/d94/l9d 0 2024-04-22T15:18:44.348 INFO:tasks.workunit.client.0.smithi096.stdout:1/530: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d3d/f63 [4194304,4194304] 0 2024-04-22T15:18:44.352 INFO:tasks.workunit.client.0.smithi096.stdout:1/531: unlink d6/dc/d8a/d3f/d62/d7d/l90 0 2024-04-22T15:18:44.353 INFO:tasks.workunit.client.0.smithi096.stdout:0/478: dwrite d7/de/d5b/d95/d15/f1a [0,4194304] 0 2024-04-22T15:18:44.355 INFO:tasks.workunit.client.0.smithi096.stdout:1/532: write d6/dc/d8a/d3f/d54/d32/d8c/d45/f66 [1482966,21002] 0 2024-04-22T15:18:44.358 INFO:tasks.workunit.client.0.smithi096.stdout:0/479: rmdir d7/de 39 2024-04-22T15:18:44.362 INFO:tasks.workunit.client.0.smithi096.stdout:1/533: rmdir d6/d97 39 2024-04-22T15:18:44.363 INFO:tasks.workunit.client.0.smithi096.stdout:0/480: creat d7/f9e x:0 0 0 2024-04-22T15:18:44.365 INFO:tasks.workunit.client.0.smithi096.stdout:0/481: rmdir d7/de/d5b/d95 39 2024-04-22T15:18:44.367 INFO:tasks.workunit.client.0.smithi096.stdout:1/534: link d6/dc/d8a/d3f/d62/d7d/f84 d6/dc/d8a/d3f/d54/d32/d8c/d4a/fa7 0 2024-04-22T15:18:44.370 INFO:tasks.workunit.client.0.smithi096.stdout:1/535: write d6/dc/d8a/d3f/d54/f18 [4843394,124920] 0 2024-04-22T15:18:44.374 INFO:tasks.workunit.client.0.smithi096.stdout:1/536: symlink d6/dc/d8a/d3f/d54/la8 0 2024-04-22T15:18:44.375 INFO:tasks.workunit.client.0.smithi096.stdout:0/482: link d7/de/d5b/d95/d15/d35/c4b d7/de/d5b/d62/c9f 0 2024-04-22T15:18:44.376 INFO:tasks.workunit.client.0.smithi096.stdout:1/537: write d6/dc/d8a/d3f/d54/d32/d8c/d45/f66 [760248,91720] 0 2024-04-22T15:18:44.379 INFO:tasks.workunit.client.0.smithi096.stdout:0/483: rename d7/de/d5b/d95/dc/d16/c83 to d7/de/d5b/d95/dc/d16/ca0 0 2024-04-22T15:18:44.382 INFO:tasks.workunit.client.0.smithi096.stdout:1/538: mkdir d6/dc/d8a/d3f/d54/d32/d8c/d45/da9 0 2024-04-22T15:18:44.383 INFO:tasks.workunit.client.0.smithi096.stdout:0/484: mknod d7/d42/d4c/d55/ca1 0 2024-04-22T15:18:44.397 INFO:tasks.workunit.client.0.smithi096.stdout:1/539: getdents d6/dc/d8a/d3f/d54/d32/d8c/d45 0 2024-04-22T15:18:44.403 INFO:tasks.workunit.client.0.smithi096.stdout:1/540: dwrite d6/dc/d8a/d3f/f9c [0,4194304] 0 2024-04-22T15:18:44.403 INFO:tasks.workunit.client.0.smithi096.stdout:0/485: dwrite d7/de/d5b/d95/f24 [0,4194304] 0 2024-04-22T15:18:44.405 INFO:tasks.workunit.client.0.smithi096.stdout:1/541: chown d6/dc/d8a/d3f/d54/d32/d8c/d4a 2662307 1 2024-04-22T15:18:44.406 INFO:tasks.workunit.client.0.smithi096.stdout:1/542: write d6/f2c [3388535,48260] 0 2024-04-22T15:18:44.408 INFO:tasks.workunit.client.0.smithi096.stdout:0/486: mknod d7/de/d5b/d62/d67/ca2 0 2024-04-22T15:18:44.409 INFO:tasks.workunit.client.0.smithi096.stdout:1/543: chown d6/dc/d8a/d3f/d54/d32/d3b/d94 1642 1 2024-04-22T15:18:44.413 INFO:tasks.workunit.client.0.smithi096.stdout:0/487: mkdir d7/d42/d4c/d55/da3 0 2024-04-22T15:18:44.415 INFO:tasks.workunit.client.0.smithi096.stdout:0/488: chown d7/de/d5b/d95/d15/f1a 77220 1 2024-04-22T15:18:44.417 INFO:tasks.workunit.client.0.smithi096.stdout:1/544: dwrite d6/dc/d8a/d3f/d54/d32/d3b/f9e [0,4194304] 0 2024-04-22T15:18:44.422 INFO:tasks.workunit.client.0.smithi096.stdout:0/489: unlink d7/de/d5b/d95/ca 0 2024-04-22T15:18:44.424 INFO:tasks.workunit.client.0.smithi096.stdout:0/490: readlink d7/de/d5b/d95/dc/d16/d18/l71 0 2024-04-22T15:18:44.424 INFO:tasks.workunit.client.0.smithi096.stdout:1/545: dwrite d6/dc/d8a/f7a [0,4194304] 0 2024-04-22T15:18:44.432 INFO:tasks.workunit.client.0.smithi096.stdout:1/546: mknod d6/dc/d8a/caa 0 2024-04-22T15:18:44.433 INFO:tasks.workunit.client.0.smithi096.stdout:1/547: chown d6/dc/d8a 28 1 2024-04-22T15:18:44.434 INFO:tasks.workunit.client.0.smithi096.stdout:1/548: write d6/dc/d8a/d3f/d54/d32/f6d [4713482,42600] 0 2024-04-22T15:18:44.435 INFO:tasks.workunit.client.0.smithi096.stdout:1/549: readlink d6/dc/d8a/d3f/d54/l46 0 2024-04-22T15:18:44.436 INFO:tasks.workunit.client.0.smithi096.stdout:0/491: getdents d7/de/d5b/d95/dc/d16 0 2024-04-22T15:18:44.437 INFO:tasks.workunit.client.0.smithi096.stdout:1/550: mkdir d6/dc/d8a/d3f/d62/d7d/dab 0 2024-04-22T15:18:44.438 INFO:tasks.workunit.client.0.smithi096.stdout:1/551: stat d6/dc/d8a/d3f/d54/d32 0 2024-04-22T15:18:44.439 INFO:tasks.workunit.client.0.smithi096.stdout:0/492: creat d7/de/d3a/d61/d79/fa4 x:0 0 0 2024-04-22T15:18:44.440 INFO:tasks.workunit.client.0.smithi096.stdout:0/493: chown d7/de/d5b/d95/dc/d16 722695334 1 2024-04-22T15:18:44.440 INFO:tasks.workunit.client.0.smithi096.stdout:0/494: fsync d7/de/d5b/d62/f63 0 2024-04-22T15:18:44.442 INFO:tasks.workunit.client.0.smithi096.stdout:1/552: mknod d6/dc/d8a/d3f/d62/cac 0 2024-04-22T15:18:44.442 INFO:tasks.workunit.client.0.smithi096.stdout:0/495: write d7/d42/d4c/f6a [139382,53832] 0 2024-04-22T15:18:44.443 INFO:tasks.workunit.client.0.smithi096.stdout:0/496: chown d7/de/d5b/d95/dc/d16/ca0 1878957 1 2024-04-22T15:18:44.443 INFO:tasks.workunit.client.0.smithi096.stdout:1/553: dread - d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fa1 zero size 2024-04-22T15:18:44.446 INFO:tasks.workunit.client.0.smithi096.stdout:1/554: creat d6/dc/d8a/d3f/d54/d32/d3b/d94/fad x:0 0 0 2024-04-22T15:18:44.446 INFO:tasks.workunit.client.0.smithi096.stdout:0/497: mknod d7/de/d5b/d95/dc/d16/d65/d7c/ca5 0 2024-04-22T15:18:44.447 INFO:tasks.workunit.client.0.smithi096.stdout:1/555: fdatasync d6/dc/d8a/d3f/d62/f70 0 2024-04-22T15:18:44.449 INFO:tasks.workunit.client.0.smithi096.stdout:1/556: symlink d6/dc/d8a/d3f/d62/d7d/lae 0 2024-04-22T15:18:44.453 INFO:tasks.workunit.client.0.smithi096.stdout:4/332: sync 2024-04-22T15:18:44.455 INFO:tasks.workunit.client.0.smithi096.stdout:4/333: mknod d5/d7/d2c/d43/d56/c71 0 2024-04-22T15:18:44.456 INFO:tasks.workunit.client.0.smithi096.stdout:4/334: write d5/d7/d2c/d3d/d4f/f60 [883499,95622] 0 2024-04-22T15:18:44.456 INFO:tasks.workunit.client.0.smithi096.stdout:4/335: fsync d5/f4e 0 2024-04-22T15:18:44.458 INFO:tasks.workunit.client.0.smithi096.stdout:4/336: write f3 [2139810,41539] 0 2024-04-22T15:18:44.460 INFO:tasks.workunit.client.0.smithi096.stdout:1/557: truncate d6/dc/d8a/d3f/d54/d32/f37 1475249 0 2024-04-22T15:18:44.461 INFO:tasks.workunit.client.0.smithi096.stdout:4/337: symlink d5/d7/d2c/d3d/l72 0 2024-04-22T15:18:44.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:44 smithi096 ceph-mon[26688]: pgmap v28: 65 pgs: 65 active+clean; 653 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.4 MiB/s rd, 73 MiB/s wr, 253 op/s 2024-04-22T15:18:44.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:44 smithi196 ceph-mon[34717]: pgmap v28: 65 pgs: 65 active+clean; 653 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.4 MiB/s rd, 73 MiB/s wr, 253 op/s 2024-04-22T15:18:44.522 INFO:tasks.workunit.client.0.smithi096.stdout:4/338: symlink d5/d7/d2c/d3d/d4f/l73 0 2024-04-22T15:18:44.523 INFO:tasks.workunit.client.0.smithi096.stdout:1/558: truncate d6/dc/d8a/d3f/d54/d32/d3b/d3d/f8b 324451 0 2024-04-22T15:18:44.524 INFO:tasks.workunit.client.0.smithi096.stdout:1/559: chown d6/dc/c71 39 1 2024-04-22T15:18:44.527 INFO:tasks.workunit.client.0.smithi096.stdout:1/560: dread - d6/dc/d8a/d29/f5c zero size 2024-04-22T15:18:44.528 INFO:tasks.workunit.client.0.smithi096.stdout:4/339: dwrite d5/d7/f24 [0,4194304] 0 2024-04-22T15:18:44.536 INFO:tasks.workunit.client.0.smithi096.stdout:1/561: dwrite d6/dc/d8a/d29/f79 [0,4194304] 0 2024-04-22T15:18:44.539 INFO:tasks.workunit.client.0.smithi096.stdout:1/562: mknod d6/dc/d8a/d3f/d54/d32/d3b/d3d/caf 0 2024-04-22T15:18:44.546 INFO:tasks.workunit.client.0.smithi096.stdout:1/563: dwrite d6/dc/d8a/d3f/d54/d32/d3b/f6f [8388608,4194304] 0 2024-04-22T15:18:44.549 INFO:tasks.workunit.client.0.smithi096.stdout:1/564: getdents d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b 0 2024-04-22T15:18:44.646 INFO:tasks.workunit.client.1.smithi196.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-22T15:18:44.650 INFO:tasks.workunit.client.1.smithi196.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T15:18:44.650 INFO:tasks.workunit.client.1.smithi196.stderr:+ make 2024-04-22T15:18:44.746 INFO:tasks.workunit.client.1.smithi196.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-22T15:18:44.839 INFO:tasks.workunit.client.0.smithi096.stdout:2/400: sync 2024-04-22T15:18:44.842 INFO:tasks.workunit.client.0.smithi096.stdout:2/401: symlink d3/db/df/d24/d58/d5f/l7d 0 2024-04-22T15:18:44.842 INFO:tasks.workunit.client.0.smithi096.stdout:2/402: fdatasync d3/db/df/d10/f1b 0 2024-04-22T15:18:44.845 INFO:tasks.workunit.client.0.smithi096.stdout:2/403: rename d3/db/df/d24/d58/d5f/d6e/d71 to d3/db/d2d/d46/d7e 0 2024-04-22T15:18:44.847 INFO:tasks.workunit.client.0.smithi096.stdout:2/404: mkdir d3/db/d2d/d46/d69/d7f 0 2024-04-22T15:18:44.847 INFO:tasks.workunit.client.0.smithi096.stdout:2/405: stat d3/d1f/f7a 0 2024-04-22T15:18:44.848 INFO:tasks.workunit.client.0.smithi096.stdout:2/406: creat d3/db/d2d/d46/f80 x:0 0 0 2024-04-22T15:18:44.849 INFO:tasks.workunit.client.0.smithi096.stdout:2/407: rmdir d3 39 2024-04-22T15:18:44.852 INFO:tasks.workunit.client.0.smithi096.stdout:2/408: getdents d3/d1f/d2f 0 2024-04-22T15:18:44.866 INFO:tasks.workunit.client.0.smithi096.stdout:2/409: dread d3/db/d2d/d46/d69/f66 [0,4194304] 0 2024-04-22T15:18:44.877 INFO:tasks.workunit.client.0.smithi096.stdout:5/361: sync 2024-04-22T15:18:44.877 INFO:tasks.workunit.client.0.smithi096.stdout:3/267: sync 2024-04-22T15:18:44.877 INFO:tasks.workunit.client.0.smithi096.stdout:8/362: sync 2024-04-22T15:18:44.877 INFO:tasks.workunit.client.0.smithi096.stdout:6/350: sync 2024-04-22T15:18:44.877 INFO:tasks.workunit.client.0.smithi096.stdout:9/307: sync 2024-04-22T15:18:44.878 INFO:tasks.workunit.client.0.smithi096.stdout:3/268: stat da/db 0 2024-04-22T15:18:44.878 INFO:tasks.workunit.client.0.smithi096.stdout:9/308: fdatasync d1/f6 0 2024-04-22T15:18:44.880 INFO:tasks.workunit.client.0.smithi096.stdout:5/362: write d4/d1f/f60 [733568,24962] 0 2024-04-22T15:18:44.880 INFO:tasks.workunit.client.0.smithi096.stdout:6/351: read f1 [1167899,58823] 0 2024-04-22T15:18:44.881 INFO:tasks.workunit.client.0.smithi096.stdout:3/269: creat da/db/d12/d22/f53 x:0 0 0 2024-04-22T15:18:44.882 INFO:tasks.workunit.client.0.smithi096.stdout:6/352: chown d3/d28/dd/d11/d2c/d44/d51/d78 33 1 2024-04-22T15:18:44.883 INFO:tasks.workunit.client.0.smithi096.stdout:3/270: write da/db/dc/f16 [3561062,37410] 0 2024-04-22T15:18:44.883 INFO:tasks.workunit.client.0.smithi096.stdout:8/363: rename d4/d1c/c3b to d4/d1c/d40/c82 0 2024-04-22T15:18:44.884 INFO:tasks.workunit.client.0.smithi096.stdout:8/364: fsync d4/d2a/f36 0 2024-04-22T15:18:44.885 INFO:tasks.workunit.client.0.smithi096.stdout:9/309: symlink d1/d4/d9/d43/l67 0 2024-04-22T15:18:44.887 INFO:tasks.workunit.client.0.smithi096.stdout:5/363: unlink d4/d2c/d2f/d30/c39 0 2024-04-22T15:18:44.893 INFO:tasks.workunit.client.0.smithi096.stdout:3/271: symlink da/db/l54 0 2024-04-22T15:18:44.894 INFO:tasks.workunit.client.0.smithi096.stdout:3/272: readlink da/db/dc/d28/l1d 0 2024-04-22T15:18:44.894 INFO:tasks.workunit.client.0.smithi096.stdout:3/273: chown da/db/d10 119586612 1 2024-04-22T15:18:44.894 INFO:tasks.workunit.client.0.smithi096.stdout:8/365: symlink d4/d1c/d40/d46/l83 0 2024-04-22T15:18:44.895 INFO:tasks.workunit.client.0.smithi096.stdout:3/274: write da/db/d12/d22/f3a [1090640,92506] 0 2024-04-22T15:18:44.897 INFO:tasks.workunit.client.0.smithi096.stdout:9/310: read d1/d4/d9/dd/f23 [218148,128314] 0 2024-04-22T15:18:44.900 INFO:tasks.workunit.client.0.smithi096.stdout:8/366: creat d4/d10/d4d/f84 x:0 0 0 2024-04-22T15:18:44.902 INFO:tasks.workunit.client.0.smithi096.stdout:8/367: chown d4/d1c/f72 48 1 2024-04-22T15:18:44.902 INFO:tasks.workunit.client.0.smithi096.stdout:9/311: rename d1/d4/d9/d43/l67 to d1/d4/d9/d43/d57/d5b/l68 0 2024-04-22T15:18:44.905 INFO:tasks.workunit.client.0.smithi096.stdout:5/364: dwrite d4/d11/d3d/d4d/d59/d5a/f67 [0,4194304] 0 2024-04-22T15:18:44.905 INFO:tasks.workunit.client.0.smithi096.stdout:8/368: write d4/d1c/d34/f4f [524195,25782] 0 2024-04-22T15:18:44.906 INFO:tasks.workunit.client.0.smithi096.stdout:3/275: getdents da/db/d12/d22/d26 0 2024-04-22T15:18:44.908 INFO:tasks.workunit.client.0.smithi096.stdout:3/276: fsync da/db/de/d19/f30 0 2024-04-22T15:18:44.917 INFO:tasks.workunit.client.0.smithi096.stdout:5/365: mkdir d4/d76/d7c 0 2024-04-22T15:18:44.917 INFO:tasks.workunit.client.0.smithi096.stdout:3/277: dwrite da/f34 [4194304,4194304] 0 2024-04-22T15:18:44.921 INFO:tasks.workunit.client.0.smithi096.stdout:9/312: dwrite d1/d4/d9/f12 [0,4194304] 0 2024-04-22T15:18:44.961 INFO:tasks.workunit.client.0.smithi096.stdout:9/313: mknod d1/d4/c69 0 2024-04-22T15:18:44.963 INFO:tasks.workunit.client.0.smithi096.stdout:3/278: creat da/f55 x:0 0 0 2024-04-22T15:18:44.964 INFO:tasks.workunit.client.0.smithi096.stdout:3/279: fdatasync da/db/de/d19/d3e/f42 0 2024-04-22T15:18:44.965 INFO:tasks.workunit.client.0.smithi096.stdout:3/280: chown da/db/de/d19/f30 63 1 2024-04-22T15:18:44.966 INFO:tasks.workunit.client.0.smithi096.stdout:9/314: creat d1/d4/d9/d43/f6a x:0 0 0 2024-04-22T15:18:44.967 INFO:tasks.workunit.client.0.smithi096.stdout:9/315: stat d1/d25/c5a 0 2024-04-22T15:18:44.971 INFO:tasks.workunit.client.0.smithi096.stdout:3/281: dwrite da/db/de/d19/d3e/f42 [0,4194304] 0 2024-04-22T15:18:44.981 INFO:tasks.workunit.client.0.smithi096.stdout:9/316: dread d1/d4/f11 [0,4194304] 0 2024-04-22T15:18:44.986 INFO:tasks.workunit.client.0.smithi096.stdout:9/317: link d1/c10 d1/d4/d9/d43/c6b 0 2024-04-22T15:18:44.990 INFO:tasks.workunit.client.0.smithi096.stdout:3/282: rmdir da/db/d12/d46 0 2024-04-22T15:18:44.991 INFO:tasks.workunit.client.0.smithi096.stdout:9/318: dwrite d1/f6 [0,4194304] 0 2024-04-22T15:18:45.003 INFO:tasks.workunit.client.0.smithi096.stdout:5/366: dwrite d4/d2c/d2f/d6b/f50 [0,4194304] 0 2024-04-22T15:18:45.013 INFO:tasks.workunit.client.0.smithi096.stdout:9/319: symlink d1/d4/d9/l6c 0 2024-04-22T15:18:45.014 INFO:tasks.workunit.client.0.smithi096.stdout:5/367: dread d4/d11/d3d/d4d/d59/d5a/f79 [4194304,4194304] 0 2024-04-22T15:18:45.022 INFO:tasks.workunit.client.0.smithi096.stdout:3/283: rmdir da/db/de/d38 39 2024-04-22T15:18:45.023 INFO:tasks.workunit.client.0.smithi096.stdout:3/284: chown da/db/l54 9624 1 2024-04-22T15:18:45.036 INFO:tasks.workunit.client.0.smithi096.stdout:5/368: dread d4/f3c [0,4194304] 0 2024-04-22T15:18:45.041 INFO:tasks.workunit.client.0.smithi096.stdout:5/369: dwrite d4/d11/f7a [0,4194304] 0 2024-04-22T15:18:45.042 INFO:tasks.workunit.client.0.smithi096.stdout:0/498: sync 2024-04-22T15:18:45.042 INFO:tasks.workunit.client.0.smithi096.stdout:4/340: sync 2024-04-22T15:18:45.042 INFO:tasks.workunit.client.0.smithi096.stdout:1/565: sync 2024-04-22T15:18:45.042 INFO:tasks.workunit.client.0.smithi096.stdout:7/594: sync 2024-04-22T15:18:45.044 INFO:tasks.workunit.client.0.smithi096.stdout:5/370: mknod d4/d11/d3d/d4d/d59/d5a/c7d 0 2024-04-22T15:18:45.045 INFO:tasks.workunit.client.0.smithi096.stdout:4/341: write d5/fc [2624508,99012] 0 2024-04-22T15:18:45.045 INFO:tasks.workunit.client.0.smithi096.stdout:1/566: readlink d6/dc/d8a/d3f/d54/d32/d3b/d56/l69 0 2024-04-22T15:18:45.045 INFO:tasks.workunit.client.0.smithi096.stdout:0/499: stat d7/de/d5b/d95/dc/c11 0 2024-04-22T15:18:45.045 INFO:tasks.workunit.client.0.smithi096.stdout:7/595: getdents d31/d32/d33/d6a/d83/dba 0 2024-04-22T15:18:45.047 INFO:tasks.workunit.client.0.smithi096.stdout:1/567: dread - d6/dc/d8a/d3f/d54/d32/d8c/d4a/f78 zero size 2024-04-22T15:18:45.047 INFO:tasks.workunit.client.0.smithi096.stdout:5/371: truncate d4/d2c/d2f/f68 1118679 0 2024-04-22T15:18:45.048 INFO:tasks.workunit.client.0.smithi096.stdout:0/500: creat d7/d42/fa6 x:0 0 0 2024-04-22T15:18:45.049 INFO:tasks.workunit.client.0.smithi096.stdout:7/596: truncate d31/d32/d33/fb1 652170 0 2024-04-22T15:18:45.052 INFO:tasks.workunit.client.0.smithi096.stdout:5/372: write d4/d2c/d2f/f68 [71723,76533] 0 2024-04-22T15:18:45.055 INFO:tasks.workunit.client.0.smithi096.stdout:4/342: dwrite d5/fc [4194304,4194304] 0 2024-04-22T15:18:45.079 INFO:tasks.workunit.client.0.smithi096.stdout:1/568: symlink d6/dc/d8a/d3f/d54/d32/d3b/d94/lb0 0 2024-04-22T15:18:45.082 INFO:tasks.workunit.client.0.smithi096.stdout:7/597: mknod d31/d32/d33/d6a/d83/cc1 0 2024-04-22T15:18:45.083 INFO:tasks.workunit.client.0.smithi096.stdout:0/501: dread d7/de/d5b/d95/dc/d16/d18/f92 [0,4194304] 0 2024-04-22T15:18:45.083 INFO:tasks.workunit.client.0.smithi096.stdout:7/598: stat d31/d32/d33/d5e/f63 0 2024-04-22T15:18:45.097 INFO:tasks.workunit.client.0.smithi096.stdout:4/343: fsync d5/f13 0 2024-04-22T15:18:45.097 INFO:tasks.workunit.client.0.smithi096.stdout:1/569: getdents d6/dc/d8a/d3f/d62/da2 0 2024-04-22T15:18:45.098 INFO:tasks.workunit.client.0.smithi096.stdout:0/502: mkdir d7/da7 0 2024-04-22T15:18:45.103 INFO:tasks.workunit.client.0.smithi096.stdout:5/373: dread d4/d1f/f21 [0,4194304] 0 2024-04-22T15:18:45.104 INFO:tasks.workunit.client.0.smithi096.stdout:5/374: chown d4/d11/d3d/d35/d57/l65 2781 1 2024-04-22T15:18:45.105 INFO:tasks.workunit.client.0.smithi096.stdout:0/503: unlink d7/de/d5b/d95/dc/d16/c75 0 2024-04-22T15:18:45.106 INFO:tasks.workunit.client.0.smithi096.stdout:4/344: symlink d5/d7/d2c/d3d/d4f/d63/l74 0 2024-04-22T15:18:45.107 INFO:tasks.workunit.client.0.smithi096.stdout:4/345: fsync d5/d7/d2c/d43/f68 0 2024-04-22T15:18:45.113 INFO:tasks.workunit.client.0.smithi096.stdout:5/375: creat d4/d11/d25/f7e x:0 0 0 2024-04-22T15:18:45.113 INFO:tasks.workunit.client.0.smithi096.stdout:4/346: dwrite d5/d7/d1f/f55 [0,4194304] 0 2024-04-22T15:18:45.114 INFO:tasks.workunit.client.0.smithi096.stdout:4/347: write d5/d7/d1a/f23 [895321,125864] 0 2024-04-22T15:18:45.115 INFO:tasks.workunit.client.0.smithi096.stdout:0/504: creat d7/da7/fa8 x:0 0 0 2024-04-22T15:18:45.121 INFO:tasks.workunit.client.0.smithi096.stdout:4/348: dwrite d5/d7/d2c/f2e [0,4194304] 0 2024-04-22T15:18:45.127 INFO:tasks.workunit.client.0.smithi096.stdout:5/376: dwrite f2 [0,4194304] 0 2024-04-22T15:18:45.128 INFO:tasks.workunit.client.0.smithi096.stdout:4/349: dread d5/fc [4194304,4194304] 0 2024-04-22T15:18:45.136 INFO:tasks.workunit.client.0.smithi096.stdout:5/377: dwrite d4/d11/d3d/d4d/d59/d5a/d5c/f71 [0,4194304] 0 2024-04-22T15:18:45.137 INFO:tasks.workunit.client.0.smithi096.stdout:5/378: chown d4/d2c/f4e 28724 1 2024-04-22T15:18:45.139 INFO:tasks.workunit.client.0.smithi096.stdout:4/350: rmdir d5/d7/d2c/d43/d56 39 2024-04-22T15:18:45.146 INFO:tasks.workunit.client.0.smithi096.stdout:5/379: dwrite d4/d11/d3d/d4d/d59/d5a/d5c/f71 [0,4194304] 0 2024-04-22T15:18:45.150 INFO:tasks.workunit.client.0.smithi096.stdout:4/351: rmdir d5 39 2024-04-22T15:18:45.154 INFO:tasks.workunit.client.0.smithi096.stdout:5/380: readlink d4/d11/l6d 0 2024-04-22T15:18:45.157 INFO:tasks.workunit.client.0.smithi096.stdout:4/352: chown d5/d7/d2c/d3d/f49 0 1 2024-04-22T15:18:45.158 INFO:tasks.workunit.client.0.smithi096.stdout:4/353: chown d5/d7/d1a/d25/l29 25039523 1 2024-04-22T15:18:45.163 INFO:tasks.workunit.client.0.smithi096.stdout:4/354: write d5/d7/d2c/d43/f6e [459114,124423] 0 2024-04-22T15:18:45.173 INFO:tasks.workunit.client.0.smithi096.stdout:4/355: creat d5/d7/d2c/d3d/f75 x:0 0 0 2024-04-22T15:18:45.173 INFO:tasks.workunit.client.0.smithi096.stdout:0/505: truncate d7/f32 5212641 0 2024-04-22T15:18:45.181 INFO:tasks.workunit.client.0.smithi096.stdout:4/356: dwrite d5/d7/d2c/d3d/f49 [0,4194304] 0 2024-04-22T15:18:45.182 INFO:tasks.workunit.client.0.smithi096.stdout:0/506: dwrite d7/de/d5b/d95/f40 [0,4194304] 0 2024-04-22T15:18:45.190 INFO:tasks.workunit.client.0.smithi096.stdout:4/357: unlink d5/d7/f24 0 2024-04-22T15:18:45.191 INFO:tasks.workunit.client.0.smithi096.stdout:4/358: fdatasync d5/d7/d2c/f65 0 2024-04-22T15:18:45.192 INFO:tasks.workunit.client.0.smithi096.stdout:0/507: rmdir d7/de/d5b/d95 39 2024-04-22T15:18:45.192 INFO:tasks.workunit.client.0.smithi096.stdout:4/359: fsync d5/d7/d2c/d3d/f6a 0 2024-04-22T15:18:45.195 INFO:tasks.workunit.client.0.smithi096.stdout:4/360: read d5/d7/d1f/f55 [4170531,74097] 0 2024-04-22T15:18:45.197 INFO:tasks.workunit.client.0.smithi096.stdout:0/508: rmdir d7/d42/d4c/d55 39 2024-04-22T15:18:45.198 INFO:tasks.workunit.client.0.smithi096.stdout:5/381: dwrite d4/f26 [0,4194304] 0 2024-04-22T15:18:45.205 INFO:tasks.workunit.client.0.smithi096.stdout:4/361: fsync d5/d7/d2c/f3a 0 2024-04-22T15:18:45.208 INFO:tasks.workunit.client.0.smithi096.stdout:0/509: chown d7/de/d5b/d95/dc/d16/l50 0 1 2024-04-22T15:18:45.210 INFO:tasks.workunit.client.0.smithi096.stdout:5/382: rename d4/d11/d25/f48 to d4/d11/d3d/d4d/d59/f7f 0 2024-04-22T15:18:45.214 INFO:tasks.workunit.client.0.smithi096.stdout:2/410: sync 2024-04-22T15:18:45.220 INFO:tasks.workunit.client.0.smithi096.stdout:4/362: symlink d5/d7/d2c/d43/l76 0 2024-04-22T15:18:45.222 INFO:tasks.workunit.client.0.smithi096.stdout:2/411: rename d3/db/df/d10/l7b to d3/db/df/d24/d58/d5e/l81 0 2024-04-22T15:18:45.226 INFO:tasks.workunit.client.0.smithi096.stdout:5/383: mkdir d4/d1f/d80 0 2024-04-22T15:18:45.227 INFO:tasks.workunit.client.0.smithi096.stdout:5/384: write f2 [1748367,82442] 0 2024-04-22T15:18:45.227 INFO:tasks.workunit.client.0.smithi096.stdout:4/363: symlink d5/d5b/l77 0 2024-04-22T15:18:45.228 INFO:tasks.workunit.client.0.smithi096.stdout:2/412: dwrite d3/f70 [0,4194304] 0 2024-04-22T15:18:45.229 INFO:tasks.workunit.client.0.smithi096.stdout:5/385: truncate d4/d1f/f21 5228283 0 2024-04-22T15:18:45.230 INFO:tasks.workunit.client.0.smithi096.stdout:2/413: stat d3/db/d2d/d46/f7c 0 2024-04-22T15:18:45.230 INFO:tasks.workunit.client.0.smithi096.stdout:0/510: mknod d7/d42/d4c/d55/da3/ca9 0 2024-04-22T15:18:45.231 INFO:tasks.workunit.client.0.smithi096.stdout:2/414: readlink d3/l72 0 2024-04-22T15:18:45.231 INFO:tasks.workunit.client.0.smithi096.stdout:2/415: stat d3/d1f/d2f/f34 0 2024-04-22T15:18:45.232 INFO:tasks.workunit.client.0.smithi096.stdout:2/416: read - d3/db/d2d/d46/d69/f6f zero size 2024-04-22T15:18:45.233 INFO:tasks.workunit.client.0.smithi096.stdout:2/417: read - d3/db/d2d/d46/d69/f6f zero size 2024-04-22T15:18:45.238 INFO:tasks.workunit.client.0.smithi096.stdout:4/364: fsync d5/d7/d1f/f55 0 2024-04-22T15:18:45.240 INFO:tasks.workunit.client.0.smithi096.stdout:4/365: chown d5/f5e 247465 1 2024-04-22T15:18:45.241 INFO:tasks.workunit.client.0.smithi096.stdout:0/511: dwrite d7/de/d5b/d95/dc/d16/d65/d7c/f9b [0,4194304] 0 2024-04-22T15:18:45.241 INFO:tasks.workunit.client.0.smithi096.stdout:5/386: chown d4/d11/d3d/c37 891 1 2024-04-22T15:18:45.242 INFO:tasks.workunit.client.0.smithi096.stdout:4/366: write d5/d7/d1f/f62 [253520,105514] 0 2024-04-22T15:18:45.244 INFO:tasks.workunit.client.0.smithi096.stdout:2/418: creat d3/db/d30/f82 x:0 0 0 2024-04-22T15:18:45.249 INFO:tasks.workunit.client.0.smithi096.stdout:0/512: creat d7/d42/faa x:0 0 0 2024-04-22T15:18:45.255 INFO:tasks.workunit.client.0.smithi096.stdout:0/513: creat d7/de/d5b/d62/fab x:0 0 0 2024-04-22T15:18:45.261 INFO:tasks.workunit.client.0.smithi096.stdout:4/367: truncate d5/f4e 1354851 0 2024-04-22T15:18:45.267 INFO:tasks.workunit.client.0.smithi096.stdout:5/387: creat d4/d2c/d2f/f81 x:0 0 0 2024-04-22T15:18:45.272 INFO:tasks.workunit.client.0.smithi096.stdout:2/419: rename d3/db/df/d10/f18 to d3/db/f83 0 2024-04-22T15:18:45.273 INFO:tasks.workunit.client.0.smithi096.stdout:2/420: write d3/db/d30/f3a [450241,16904] 0 2024-04-22T15:18:45.274 INFO:tasks.workunit.client.0.smithi096.stdout:0/514: write d7/de/d5b/d95/f45 [1099689,103630] 0 2024-04-22T15:18:45.279 INFO:tasks.workunit.client.0.smithi096.stdout:4/368: dread d5/f39 [0,4194304] 0 2024-04-22T15:18:45.279 INFO:tasks.workunit.client.0.smithi096.stdout:2/421: mkdir d3/db/d2d/d46/d7e/d84 0 2024-04-22T15:18:45.281 INFO:tasks.workunit.client.0.smithi096.stdout:4/369: chown d5/d7/d2c/d43/c4b 62 1 2024-04-22T15:18:45.282 INFO:tasks.workunit.client.0.smithi096.stdout:0/515: dwrite d7/de/d5b/d95/f45 [0,4194304] 0 2024-04-22T15:18:45.287 INFO:tasks.workunit.client.0.smithi096.stdout:0/516: dread d7/de/d5b/d95/f24 [0,4194304] 0 2024-04-22T15:18:45.293 INFO:tasks.workunit.client.0.smithi096.stdout:5/388: write d4/d2c/d2f/d31/f3b [850751,62303] 0 2024-04-22T15:18:45.293 INFO:tasks.workunit.client.0.smithi096.stdout:2/422: dwrite d3/db/d2d/d46/d69/f66 [4194304,4194304] 0 2024-04-22T15:18:45.295 INFO:tasks.workunit.client.0.smithi096.stdout:5/389: readlink d4/d11/d3d/d4f/l74 0 2024-04-22T15:18:45.299 INFO:tasks.workunit.client.0.smithi096.stdout:0/517: rename d7/de/d5b/d95/d66/c80 to d7/de/d5b/d95/d59/cac 0 2024-04-22T15:18:45.314 INFO:tasks.workunit.client.0.smithi096.stdout:5/390: unlink d4/d2c/d2f/d30/c55 0 2024-04-22T15:18:45.314 INFO:tasks.workunit.client.0.smithi096.stdout:4/370: write d5/d7/d1a/f6c [4410159,128589] 0 2024-04-22T15:18:45.316 INFO:tasks.workunit.client.0.smithi096.stdout:4/371: write d5/d7/d2c/f65 [889834,2864] 0 2024-04-22T15:18:45.316 INFO:tasks.workunit.client.0.smithi096.stdout:5/391: read d4/d1f/f60 [235200,121059] 0 2024-04-22T15:18:45.321 INFO:tasks.workunit.client.0.smithi096.stdout:2/423: link d3/db/d2d/d46/d59/l64 d3/db/d2d/l85 0 2024-04-22T15:18:45.322 INFO:tasks.workunit.client.0.smithi096.stdout:4/372: symlink d5/d5b/l78 0 2024-04-22T15:18:45.325 INFO:tasks.workunit.client.0.smithi096.stdout:0/518: link d7/de/d5b/d95/c91 d7/d42/d4c/d55/da3/cad 0 2024-04-22T15:18:45.328 INFO:tasks.workunit.client.0.smithi096.stdout:2/424: mknod d3/db/d2d/d46/c86 0 2024-04-22T15:18:45.329 INFO:tasks.workunit.client.0.smithi096.stdout:2/425: write d3/d1f/f51 [950179,124270] 0 2024-04-22T15:18:45.330 INFO:tasks.workunit.client.0.smithi096.stdout:2/426: write d3/f53 [655838,76278] 0 2024-04-22T15:18:45.330 INFO:tasks.workunit.client.0.smithi096.stdout:4/373: rename d5/cf to d5/c79 0 2024-04-22T15:18:45.336 INFO:tasks.workunit.client.0.smithi096.stdout:2/427: creat d3/f87 x:0 0 0 2024-04-22T15:18:45.338 INFO:tasks.workunit.client.0.smithi096.stdout:4/374: creat d5/d7/d1f/f7a x:0 0 0 2024-04-22T15:18:45.346 INFO:tasks.workunit.client.0.smithi096.stdout:5/392: dwrite d4/d11/d3d/d4d/d59/d5a/f15 [0,4194304] 0 2024-04-22T15:18:45.346 INFO:tasks.workunit.client.0.smithi096.stdout:5/393: stat d4/d1f/d80 0 2024-04-22T15:18:45.352 INFO:tasks.workunit.client.0.smithi096.stdout:0/519: getdents d7/de/d3a/d61 0 2024-04-22T15:18:45.356 INFO:tasks.workunit.client.0.smithi096.stdout:5/394: rename d4/d76/d7c to d4/d2c/d2f/d31/d82 0 2024-04-22T15:18:45.357 INFO:tasks.workunit.client.0.smithi096.stdout:0/520: mknod d7/d42/cae 0 2024-04-22T15:18:45.358 INFO:tasks.workunit.client.0.smithi096.stdout:0/521: stat d7/de/ld 0 2024-04-22T15:18:45.362 INFO:tasks.workunit.client.0.smithi096.stdout:4/375: write d5/d7/f9 [1563182,53533] 0 2024-04-22T15:18:45.365 INFO:tasks.workunit.client.0.smithi096.stdout:0/522: mknod d7/d42/d4c/d55/caf 0 2024-04-22T15:18:45.367 INFO:tasks.workunit.client.0.smithi096.stdout:2/428: dwrite d3/db/d2d/d46/d59/f76 [0,4194304] 0 2024-04-22T15:18:45.375 INFO:tasks.workunit.client.0.smithi096.stdout:2/429: dwrite d3/db/d30/f3a [0,4194304] 0 2024-04-22T15:18:45.377 INFO:tasks.workunit.client.0.smithi096.stdout:2/430: write d3/db/d30/f82 [302175,82724] 0 2024-04-22T15:18:45.378 INFO:tasks.workunit.client.0.smithi096.stdout:2/431: stat d3/db/df/d24/d58/d5f 0 2024-04-22T15:18:45.381 INFO:tasks.workunit.client.0.smithi096.stdout:4/376: creat d5/d7/f7b x:0 0 0 2024-04-22T15:18:45.383 INFO:tasks.workunit.client.0.smithi096.stdout:2/432: dread d3/db/df/d24/d58/d5e/f6b [0,4194304] 0 2024-04-22T15:18:45.383 INFO:tasks.workunit.client.0.smithi096.stdout:5/395: getdents d4 0 2024-04-22T15:18:45.394 INFO:tasks.workunit.client.0.smithi096.stdout:0/523: rename d7/d42/d4c/d55/ca1 to d7/d42/d4c/cb0 0 2024-04-22T15:18:45.397 INFO:tasks.workunit.client.0.smithi096.stdout:0/524: mknod d7/de/d3a/d61/d79/cb1 0 2024-04-22T15:18:45.397 INFO:tasks.workunit.client.0.smithi096.stdout:2/433: dread d3/db/f42 [0,4194304] 0 2024-04-22T15:18:45.398 INFO:tasks.workunit.client.0.smithi096.stdout:0/525: truncate d7/d42/faa 438076 0 2024-04-22T15:18:45.398 INFO:tasks.workunit.client.0.smithi096.stdout:5/396: creat d4/d11/d3d/d4d/d59/d5a/d5c/f83 x:0 0 0 2024-04-22T15:18:45.400 INFO:tasks.workunit.client.0.smithi096.stdout:0/526: chown d7/de/d5b/d95/dc/d16/d18/f39 3913 1 2024-04-22T15:18:45.401 INFO:tasks.workunit.client.0.smithi096.stdout:0/527: fdatasync d7/de/d5b/d95/f45 0 2024-04-22T15:18:45.402 INFO:tasks.workunit.client.0.smithi096.stdout:0/528: dread - d7/de/d5b/d62/f63 zero size 2024-04-22T15:18:45.403 INFO:tasks.workunit.client.0.smithi096.stdout:4/377: dread d5/d7/f9 [0,4194304] 0 2024-04-22T15:18:45.403 INFO:tasks.workunit.client.0.smithi096.stdout:5/397: write d4/d11/d3d/d4d/d59/f7f [4515409,87812] 0 2024-04-22T15:18:45.404 INFO:tasks.workunit.client.0.smithi096.stdout:2/434: dread d3/d1f/f51 [0,4194304] 0 2024-04-22T15:18:45.405 INFO:tasks.workunit.client.0.smithi096.stdout:0/529: chown d7/de/d5b/d64/d94/l9d 60 1 2024-04-22T15:18:45.406 INFO:tasks.workunit.client.0.smithi096.stdout:4/378: write d5/d7/d2c/d43/f68 [119472,66562] 0 2024-04-22T15:18:45.408 INFO:tasks.workunit.client.0.smithi096.stdout:5/398: fdatasync d4/d11/d3d/d4d/d59/d5a/f79 0 2024-04-22T15:18:45.409 INFO:tasks.workunit.client.0.smithi096.stdout:4/379: chown d5/f4e 578 1 2024-04-22T15:18:45.413 INFO:tasks.workunit.client.0.smithi096.stdout:5/399: mknod d4/d1f/d80/c84 0 2024-04-22T15:18:45.413 INFO:tasks.workunit.client.0.smithi096.stdout:0/530: getdents d7/de/d3a 0 2024-04-22T15:18:45.414 INFO:tasks.workunit.client.0.smithi096.stdout:2/435: dread f0 [0,4194304] 0 2024-04-22T15:18:45.414 INFO:tasks.workunit.client.0.smithi096.stdout:4/380: symlink d5/d7/d2c/d67/l7c 0 2024-04-22T15:18:45.415 INFO:tasks.workunit.client.0.smithi096.stdout:4/381: fdatasync d5/d7/d2c/f65 0 2024-04-22T15:18:45.416 INFO:tasks.workunit.client.0.smithi096.stdout:6/353: sync 2024-04-22T15:18:45.416 INFO:tasks.workunit.client.0.smithi096.stdout:2/436: readlink d3/db/df/l25 0 2024-04-22T15:18:45.418 INFO:tasks.workunit.client.0.smithi096.stdout:6/354: read d3/d28/dd/f25 [2229670,3500] 0 2024-04-22T15:18:45.419 INFO:tasks.workunit.client.0.smithi096.stdout:0/531: rename d7/de/d5b/d95/dc/d16/d18/f68 to d7/d42/fb2 0 2024-04-22T15:18:45.424 INFO:tasks.workunit.client.0.smithi096.stdout:2/437: dwrite d3/db/df/d24/d58/f77 [0,4194304] 0 2024-04-22T15:18:45.427 INFO:tasks.workunit.client.0.smithi096.stdout:2/438: read d3/db/d30/f39 [440479,25252] 0 2024-04-22T15:18:45.428 INFO:tasks.workunit.client.0.smithi096.stdout:2/439: readlink d3/db/df/d24/d58/d5f/l7d 0 2024-04-22T15:18:45.428 INFO:tasks.workunit.client.0.smithi096.stdout:0/532: creat d7/d42/d4c/d55/fb3 x:0 0 0 2024-04-22T15:18:45.431 INFO:tasks.workunit.client.0.smithi096.stdout:0/533: write d7/de/d5b/d95/d15/f82 [898134,7263] 0 2024-04-22T15:18:45.432 INFO:tasks.workunit.client.0.smithi096.stdout:5/400: dread d4/d2c/d2f/f51 [0,4194304] 0 2024-04-22T15:18:45.432 INFO:tasks.workunit.client.0.smithi096.stdout:6/355: mknod d3/d28/dd/d11/d2c/c7a 0 2024-04-22T15:18:45.434 INFO:tasks.workunit.client.0.smithi096.stdout:4/382: dread f3 [0,4194304] 0 2024-04-22T15:18:45.435 INFO:tasks.workunit.client.0.smithi096.stdout:2/440: mknod d3/db/d30/c88 0 2024-04-22T15:18:45.436 INFO:tasks.workunit.client.0.smithi096.stdout:2/441: chown d3/db/df/d24/c4b 11 1 2024-04-22T15:18:45.436 INFO:tasks.workunit.client.0.smithi096.stdout:2/442: stat d3/db/df/d10/f37 0 2024-04-22T15:18:45.437 INFO:tasks.workunit.client.0.smithi096.stdout:6/356: mknod d3/d28/d54/d68/c7b 0 2024-04-22T15:18:45.439 INFO:tasks.workunit.client.0.smithi096.stdout:0/534: creat d7/de/d5b/d95/d15/d35/fb4 x:0 0 0 2024-04-22T15:18:45.441 INFO:tasks.workunit.client.0.smithi096.stdout:5/401: chown d4/d11/d3d/d4d/d59/d5a/f15 2 1 2024-04-22T15:18:45.441 INFO:tasks.workunit.client.1.smithi196.stderr:++ readlink -f fsstress 2024-04-22T15:18:45.442 INFO:tasks.workunit.client.0.smithi096.stdout:4/383: unlink d5/d7/d1a/f1d 0 2024-04-22T15:18:45.443 INFO:tasks.workunit.client.1.smithi196.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-22T15:18:45.443 INFO:tasks.workunit.client.1.smithi196.stderr:+ popd 2024-04-22T15:18:45.444 INFO:tasks.workunit.client.0.smithi096.stdout:6/357: dwrite d3/d28/f4f [0,4194304] 0 2024-04-22T15:18:45.444 INFO:tasks.workunit.client.1.smithi196.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T15:18:45.444 INFO:tasks.workunit.client.1.smithi196.stderr:+ popd 2024-04-22T15:18:45.445 INFO:tasks.workunit.client.1.smithi196.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-22T15:18:45.445 INFO:tasks.workunit.client.1.smithi196.stderr:++ mktemp -d -p . 2024-04-22T15:18:45.445 INFO:tasks.workunit.client.0.smithi096.stdout:0/535: creat d7/de/d5b/d95/dc/d26/fb5 x:0 0 0 2024-04-22T15:18:45.451 INFO:tasks.workunit.client.0.smithi096.stdout:2/443: mknod d3/c89 0 2024-04-22T15:18:45.453 INFO:tasks.workunit.client.1.smithi196.stderr:+ T=./tmp.SOsiBfZkYa 2024-04-22T15:18:45.453 INFO:tasks.workunit.client.1.smithi196.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.SOsiBfZkYa -l 1 -n 1000 -p 10 -v 2024-04-22T15:18:45.453 INFO:tasks.workunit.client.0.smithi096.stdout:0/536: dwrite d7/de/d5b/d95/f40 [4194304,4194304] 0 2024-04-22T15:18:45.458 INFO:tasks.workunit.client.0.smithi096.stdout:6/358: mkdir d3/d28/dd/d11/d2c/d6f/d7c 0 2024-04-22T15:18:45.458 INFO:tasks.workunit.client.0.smithi096.stdout:6/359: chown d3 53911664 1 2024-04-22T15:18:45.460 INFO:tasks.workunit.client.0.smithi096.stdout:3/285: sync 2024-04-22T15:18:45.460 INFO:tasks.workunit.client.0.smithi096.stdout:9/320: sync 2024-04-22T15:18:45.460 INFO:tasks.workunit.client.0.smithi096.stdout:8/369: sync 2024-04-22T15:18:45.460 INFO:tasks.workunit.client.0.smithi096.stdout:1/570: sync 2024-04-22T15:18:45.460 INFO:tasks.workunit.client.0.smithi096.stdout:7/599: sync 2024-04-22T15:18:45.461 INFO:tasks.workunit.client.0.smithi096.stdout:0/537: unlink d7/de/d5b/d95/f40 0 2024-04-22T15:18:45.462 INFO:tasks.workunit.client.1.smithi196.stdout:seed = 1713469282 2024-04-22T15:18:45.463 INFO:tasks.workunit.client.0.smithi096.stdout:7/600: write d31/d32/d33/d6a/d83/f96 [72909,125281] 0 2024-04-22T15:18:45.463 INFO:tasks.workunit.client.0.smithi096.stdout:0/538: write d7/d42/d4c/f6a [156374,3300] 0 2024-04-22T15:18:45.463 INFO:tasks.workunit.client.0.smithi096.stdout:6/360: fsync f2 0 2024-04-22T15:18:45.468 INFO:tasks.workunit.client.1.smithi196.stdout:0/0: rename - no filename 2024-04-22T15:18:45.469 INFO:tasks.workunit.client.0.smithi096.stdout:9/321: rmdir d1/d4/d9 39 2024-04-22T15:18:45.471 INFO:tasks.workunit.client.0.smithi096.stdout:3/286: fdatasync da/db/de/d19/d2d/f2e 0 2024-04-22T15:18:45.473 INFO:tasks.workunit.client.1.smithi196.stdout:2/0: chown . 96069 1 2024-04-22T15:18:45.473 INFO:tasks.workunit.client.1.smithi196.stdout:2/1: fsync - no filename 2024-04-22T15:18:45.473 INFO:tasks.workunit.client.1.smithi196.stdout:2/2: dread - no filename 2024-04-22T15:18:45.473 INFO:tasks.workunit.client.0.smithi096.stdout:2/444: mknod d3/db/df/c8a 0 2024-04-22T15:18:45.474 INFO:tasks.workunit.client.1.smithi196.stdout:0/1: symlink l0 0 2024-04-22T15:18:45.474 INFO:tasks.workunit.client.1.smithi196.stdout:0/2: dwrite - no filename 2024-04-22T15:18:45.474 INFO:tasks.workunit.client.1.smithi196.stdout:0/3: chown l0 39 1 2024-04-22T15:18:45.474 INFO:tasks.workunit.client.0.smithi096.stdout:0/539: creat d7/de/d5b/d64/fb6 x:0 0 0 2024-04-22T15:18:45.478 INFO:tasks.workunit.client.0.smithi096.stdout:1/571: creat d6/dc/d8a/d3f/d62/da2/fb1 x:0 0 0 2024-04-22T15:18:45.480 INFO:tasks.workunit.client.1.smithi196.stdout:1/0: mknod c0 0 2024-04-22T15:18:45.481 INFO:tasks.workunit.client.1.smithi196.stdout:4/0: stat - no entries 2024-04-22T15:18:45.482 INFO:tasks.workunit.client.1.smithi196.stdout:4/1: link - no file 2024-04-22T15:18:45.482 INFO:tasks.workunit.client.1.smithi196.stdout:4/2: write - no filename 2024-04-22T15:18:45.482 INFO:tasks.workunit.client.1.smithi196.stdout:4/3: unlink - no file 2024-04-22T15:18:45.482 INFO:tasks.workunit.client.1.smithi196.stdout:4/4: fsync - no filename 2024-04-22T15:18:45.483 INFO:tasks.workunit.client.1.smithi196.stdout:2/3: creat f0 x:0 0 0 2024-04-22T15:18:45.487 INFO:tasks.workunit.client.1.smithi196.stdout:1/1: mkdir d1 0 2024-04-22T15:18:45.487 INFO:tasks.workunit.client.1.smithi196.stdout:1/2: dwrite - no filename 2024-04-22T15:18:45.491 INFO:tasks.workunit.client.0.smithi096.stdout:0/540: unlink d7/de/d5b/d95/d15/f82 0 2024-04-22T15:18:45.491 INFO:tasks.workunit.client.0.smithi096.stdout:5/402: dwrite d4/fd [0,4194304] 0 2024-04-22T15:18:45.493 INFO:tasks.workunit.client.0.smithi096.stdout:6/361: link d3/d3a/f3e d3/d3a/d75/f7d 0 2024-04-22T15:18:45.496 INFO:tasks.workunit.client.1.smithi196.stdout:0/4: link l0 l1 0 2024-04-22T15:18:45.498 INFO:tasks.workunit.client.0.smithi096.stdout:4/384: dwrite d5/f39 [0,4194304] 0 2024-04-22T15:18:45.506 INFO:tasks.workunit.client.0.smithi096.stdout:1/572: dwrite d6/dc/d8a/d3f/d62/d7d/f38 [4194304,4194304] 0 2024-04-22T15:18:45.506 INFO:tasks.workunit.client.0.smithi096.stdout:3/287: rename da/db/de/d38/f4c to da/db/d12/f56 0 2024-04-22T15:18:45.507 INFO:tasks.workunit.client.0.smithi096.stdout:8/370: dwrite d4/f32 [0,4194304] 0 2024-04-22T15:18:45.509 INFO:tasks.workunit.client.0.smithi096.stdout:2/445: creat d3/db/df/d24/d58/d5e/f8b x:0 0 0 2024-04-22T15:18:45.511 INFO:tasks.workunit.client.1.smithi196.stdout:4/5: mknod c0 0 2024-04-22T15:18:45.511 INFO:tasks.workunit.client.1.smithi196.stdout:4/6: write - no filename 2024-04-22T15:18:45.515 INFO:tasks.workunit.client.0.smithi096.stdout:5/403: mknod d4/d2c/d2f/d31/d82/c85 0 2024-04-22T15:18:45.515 INFO:tasks.workunit.client.1.smithi196.stdout:0/5: mknod c2 0 2024-04-22T15:18:45.516 INFO:tasks.workunit.client.0.smithi096.stdout:6/362: mkdir d3/d28/d54/d7e 0 2024-04-22T15:18:45.517 INFO:tasks.workunit.client.1.smithi196.stdout:1/3: getdents d1 0 2024-04-22T15:18:45.519 INFO:tasks.workunit.client.0.smithi096.stdout:6/363: dread d3/d28/dd/d11/d2c/f4e [0,4194304] 0 2024-04-22T15:18:45.519 INFO:tasks.workunit.client.0.smithi096.stdout:5/404: dread d4/d1f/f21 [0,4194304] 0 2024-04-22T15:18:45.521 INFO:tasks.workunit.client.1.smithi196.stdout:2/4: fsync f0 0 2024-04-22T15:18:45.521 INFO:tasks.workunit.client.0.smithi096.stdout:4/385: write d5/d7/f9 [1838483,38766] 0 2024-04-22T15:18:45.522 INFO:tasks.workunit.client.1.smithi196.stdout:2/5: fdatasync f0 0 2024-04-22T15:18:45.523 INFO:tasks.workunit.client.0.smithi096.stdout:7/601: truncate d31/d32/d36/f3c 2835663 0 2024-04-22T15:18:45.526 INFO:tasks.workunit.client.1.smithi196.stdout:5/0: creat f0 x:0 0 0 2024-04-22T15:18:45.526 INFO:tasks.workunit.client.1.smithi196.stdout:5/1: truncate f0 727934 0 2024-04-22T15:18:45.527 INFO:tasks.workunit.client.1.smithi196.stdout:5/2: write f0 [1356744,3361] 0 2024-04-22T15:18:45.528 INFO:tasks.workunit.client.1.smithi196.stdout:4/7: rename c0 to c1 0 2024-04-22T15:18:45.531 INFO:tasks.workunit.client.1.smithi196.stdout:0/6: mkdir d3 0 2024-04-22T15:18:45.531 INFO:tasks.workunit.client.1.smithi196.stdout:0/7: write - no filename 2024-04-22T15:18:45.531 INFO:tasks.workunit.client.0.smithi096.stdout:9/322: truncate d1/d25/f2e 695551 0 2024-04-22T15:18:45.533 INFO:tasks.workunit.client.1.smithi196.stdout:1/4: rename c0 to d1/c2 0 2024-04-22T15:18:45.535 INFO:tasks.workunit.client.1.smithi196.stdout:8/0: dwrite - no filename 2024-04-22T15:18:45.536 INFO:tasks.workunit.client.0.smithi096.stdout:8/371: creat d4/d10/d4d/f85 x:0 0 0 2024-04-22T15:18:45.537 INFO:tasks.workunit.client.1.smithi196.stdout:6/0: creat f0 x:0 0 0 2024-04-22T15:18:45.539 INFO:tasks.workunit.client.1.smithi196.stdout:2/6: mkdir d1 0 2024-04-22T15:18:45.543 INFO:tasks.workunit.client.1.smithi196.stdout:6/1: dwrite f0 [0,4194304] 0 2024-04-22T15:18:45.545 INFO:tasks.workunit.client.0.smithi096.stdout:6/364: rmdir d3/d28 39 2024-04-22T15:18:45.546 INFO:tasks.workunit.client.1.smithi196.stdout:5/3: mkdir d1 0 2024-04-22T15:18:45.548 INFO:tasks.workunit.client.0.smithi096.stdout:5/405: rmdir d4/d2c/d2f/d30 39 2024-04-22T15:18:45.550 INFO:tasks.workunit.client.1.smithi196.stdout:4/8: mknod c2 0 2024-04-22T15:18:45.550 INFO:tasks.workunit.client.1.smithi196.stdout:4/9: dread - no filename 2024-04-22T15:18:45.555 INFO:tasks.workunit.client.1.smithi196.stdout:1/5: rmdir d1 39 2024-04-22T15:18:45.557 INFO:tasks.workunit.client.1.smithi196.stdout:9/0: rename - no filename 2024-04-22T15:18:45.557 INFO:tasks.workunit.client.1.smithi196.stdout:9/1: chown . 127219 1 2024-04-22T15:18:45.558 INFO:tasks.workunit.client.1.smithi196.stdout:9/2: chown . 155 1 2024-04-22T15:18:45.558 INFO:tasks.workunit.client.1.smithi196.stdout:9/3: fdatasync - no filename 2024-04-22T15:18:45.558 INFO:tasks.workunit.client.1.smithi196.stdout:9/4: rename - no filename 2024-04-22T15:18:45.558 INFO:tasks.workunit.client.1.smithi196.stdout:9/5: fsync - no filename 2024-04-22T15:18:45.558 INFO:tasks.workunit.client.1.smithi196.stdout:9/6: rename - no filename 2024-04-22T15:18:45.558 INFO:tasks.workunit.client.1.smithi196.stdout:9/7: rmdir - no directory 2024-04-22T15:18:45.558 INFO:tasks.workunit.client.0.smithi096.stdout:1/573: mknod d6/dc/d8a/cb2 0 2024-04-22T15:18:45.560 INFO:tasks.workunit.client.0.smithi096.stdout:2/446: rename d3/db/lc to d3/db/df/d24/d58/d5f/d6e/l8c 0 2024-04-22T15:18:45.562 INFO:tasks.workunit.client.1.smithi196.stdout:7/0: mknod c0 0 2024-04-22T15:18:45.562 INFO:tasks.workunit.client.1.smithi196.stdout:7/1: write - no filename 2024-04-22T15:18:45.562 INFO:tasks.workunit.client.1.smithi196.stdout:7/2: dwrite - no filename 2024-04-22T15:18:45.562 INFO:tasks.workunit.client.0.smithi096.stdout:4/386: write d5/f4e [1938121,51815] 0 2024-04-22T15:18:45.565 INFO:tasks.workunit.client.0.smithi096.stdout:6/365: chown d3/d28/dd/d11/f41 5067 1 2024-04-22T15:18:45.565 INFO:tasks.workunit.client.0.smithi096.stdout:5/406: creat d4/f86 x:0 0 0 2024-04-22T15:18:45.566 INFO:tasks.workunit.client.0.smithi096.stdout:6/366: dread - d3/d28/dd/d11/d2c/d44/f67 zero size 2024-04-22T15:18:45.567 INFO:tasks.workunit.client.1.smithi196.stdout:4/10: mknod c3 0 2024-04-22T15:18:45.568 INFO:tasks.workunit.client.0.smithi096.stdout:6/367: read d3/d3a/f3e [226292,115051] 0 2024-04-22T15:18:45.569 INFO:tasks.workunit.client.0.smithi096.stdout:7/602: mknod d31/d32/d33/d44/da8/dbf/cc2 0 2024-04-22T15:18:45.570 INFO:tasks.workunit.client.1.smithi196.stdout:0/8: mknod d3/c4 0 2024-04-22T15:18:45.570 INFO:tasks.workunit.client.1.smithi196.stdout:6/2: write f0 [4443405,50700] 0 2024-04-22T15:18:45.570 INFO:tasks.workunit.client.0.smithi096.stdout:7/603: chown d31/d32/d33/d44/da8/dbf/cc2 1 1 2024-04-22T15:18:45.571 INFO:tasks.workunit.client.0.smithi096.stdout:7/604: chown c2a 12853 1 2024-04-22T15:18:45.571 INFO:tasks.workunit.client.0.smithi096.stdout:9/323: symlink d1/d4/d9/dd/l6d 0 2024-04-22T15:18:45.574 INFO:tasks.workunit.client.0.smithi096.stdout:3/288: creat da/db/de/f57 x:0 0 0 2024-04-22T15:18:45.576 INFO:tasks.workunit.client.1.smithi196.stdout:8/1: creat f0 x:0 0 0 2024-04-22T15:18:45.576 INFO:tasks.workunit.client.1.smithi196.stdout:2/7: mknod d1/c2 0 2024-04-22T15:18:45.578 INFO:tasks.workunit.client.1.smithi196.stdout:7/3: unlink c0 0 2024-04-22T15:18:45.578 INFO:tasks.workunit.client.1.smithi196.stdout:7/4: truncate - no filename 2024-04-22T15:18:45.578 INFO:tasks.workunit.client.1.smithi196.stdout:7/5: write - no filename 2024-04-22T15:18:45.578 INFO:tasks.workunit.client.0.smithi096.stdout:1/574: creat d6/dc/d8a/d3f/d54/d32/d3b/d94/fb3 x:0 0 0 2024-04-22T15:18:45.579 INFO:tasks.workunit.client.1.smithi196.stdout:7/6: chown . 0 1 2024-04-22T15:18:45.579 INFO:tasks.workunit.client.0.smithi096.stdout:1/575: readlink d6/dc/d8a/l99 0 2024-04-22T15:18:45.580 INFO:tasks.workunit.client.1.smithi196.stdout:5/4: mkdir d1/d2 0 2024-04-22T15:18:45.580 INFO:tasks.workunit.client.0.smithi096.stdout:8/372: mknod d4/c86 0 2024-04-22T15:18:45.581 INFO:tasks.workunit.client.1.smithi196.stdout:6/3: dwrite f0 [0,4194304] 0 2024-04-22T15:18:45.581 INFO:tasks.workunit.client.0.smithi096.stdout:6/368: dread d3/d28/f1d [0,4194304] 0 2024-04-22T15:18:45.582 INFO:tasks.workunit.client.0.smithi096.stdout:8/373: read - d4/d1c/d40/d46/d58/f7a zero size 2024-04-22T15:18:45.583 INFO:tasks.workunit.client.1.smithi196.stdout:4/11: creat f4 x:0 0 0 2024-04-22T15:18:45.584 INFO:tasks.workunit.client.0.smithi096.stdout:6/369: truncate d3/d28/dd/d11/d2c/d3b/f73 502134 0 2024-04-22T15:18:45.584 INFO:tasks.workunit.client.1.smithi196.stdout:0/9: creat d3/f5 x:0 0 0 2024-04-22T15:18:45.586 INFO:tasks.workunit.client.0.smithi096.stdout:4/387: mknod d5/d7/d1a/d61/c7d 0 2024-04-22T15:18:45.586 INFO:tasks.workunit.client.1.smithi196.stdout:0/10: dread - d3/f5 zero size 2024-04-22T15:18:45.586 INFO:tasks.workunit.client.1.smithi196.stdout:8/2: dwrite f0 [0,4194304] 0 2024-04-22T15:18:45.588 INFO:tasks.workunit.client.1.smithi196.stdout:1/6: symlink d1/l3 0 2024-04-22T15:18:45.588 INFO:tasks.workunit.client.1.smithi196.stdout:1/7: write - no filename 2024-04-22T15:18:45.589 INFO:tasks.workunit.client.1.smithi196.stdout:1/8: write - no filename 2024-04-22T15:18:45.589 INFO:tasks.workunit.client.1.smithi196.stdout:9/8: creat f0 x:0 0 0 2024-04-22T15:18:45.590 INFO:tasks.workunit.client.1.smithi196.stdout:9/9: stat f0 0 2024-04-22T15:18:45.590 INFO:tasks.workunit.client.1.smithi196.stdout:2/8: creat d1/f3 x:0 0 0 2024-04-22T15:18:45.591 INFO:tasks.workunit.client.1.smithi196.stdout:7/7: mknod c1 0 2024-04-22T15:18:45.591 INFO:tasks.workunit.client.1.smithi196.stdout:7/8: dwrite - no filename 2024-04-22T15:18:45.592 INFO:tasks.workunit.client.1.smithi196.stdout:9/10: dread - f0 zero size 2024-04-22T15:18:45.592 INFO:tasks.workunit.client.0.smithi096.stdout:9/324: readlink d1/l27 0 2024-04-22T15:18:45.594 INFO:tasks.workunit.client.1.smithi196.stdout:2/9: write f0 [897106,41549] 0 2024-04-22T15:18:45.595 INFO:tasks.workunit.client.1.smithi196.stdout:7/9: chown c1 1888 1 2024-04-22T15:18:45.595 INFO:tasks.workunit.client.0.smithi096.stdout:7/605: creat d31/d32/d36/d75/fc3 x:0 0 0 2024-04-22T15:18:45.596 INFO:tasks.workunit.client.1.smithi196.stdout:5/5: mknod d1/c3 0 2024-04-22T15:18:45.598 INFO:tasks.workunit.client.1.smithi196.stdout:0/11: dwrite d3/f5 [0,4194304] 0 2024-04-22T15:18:45.599 INFO:tasks.workunit.client.0.smithi096.stdout:3/289: mknod da/db/de/d19/d2d/c58 0 2024-04-22T15:18:45.600 INFO:tasks.workunit.client.0.smithi096.stdout:3/290: write da/db/dc/f16 [4579686,34632] 0 2024-04-22T15:18:45.603 INFO:tasks.workunit.client.1.smithi196.stdout:4/12: chown c1 360542929 1 2024-04-22T15:18:45.605 INFO:tasks.workunit.client.1.smithi196.stdout:4/13: chown c3 2 1 2024-04-22T15:18:45.607 INFO:tasks.workunit.client.1.smithi196.stdout:1/9: creat d1/f4 x:0 0 0 2024-04-22T15:18:45.607 INFO:tasks.workunit.client.1.smithi196.stdout:5/6: dread f0 [0,4194304] 0 2024-04-22T15:18:45.607 INFO:tasks.workunit.client.1.smithi196.stdout:1/10: rename d1 to d1/d5 22 2024-04-22T15:18:45.608 INFO:tasks.workunit.client.1.smithi196.stdout:5/7: write f0 [2322682,103364] 0 2024-04-22T15:18:45.608 INFO:tasks.workunit.client.0.smithi096.stdout:4/388: creat d5/d7/d2c/d3d/d4f/d63/f7e x:0 0 0 2024-04-22T15:18:45.613 INFO:tasks.workunit.client.0.smithi096.stdout:9/325: mknod d1/d25/c6e 0 2024-04-22T15:18:45.615 INFO:tasks.workunit.client.1.smithi196.stdout:7/10: creat f2 x:0 0 0 2024-04-22T15:18:45.616 INFO:tasks.workunit.client.1.smithi196.stdout:5/8: dread f0 [0,4194304] 0 2024-04-22T15:18:45.616 INFO:tasks.workunit.client.1.smithi196.stdout:7/11: truncate f2 246169 0 2024-04-22T15:18:45.617 INFO:tasks.workunit.client.1.smithi196.stdout:7/12: dread f2 [0,4194304] 0 2024-04-22T15:18:45.619 INFO:tasks.workunit.client.1.smithi196.stdout:5/9: dread f0 [0,4194304] 0 2024-04-22T15:18:45.619 INFO:tasks.workunit.client.1.smithi196.stdout:5/10: truncate f0 2574862 0 2024-04-22T15:18:45.621 INFO:tasks.workunit.client.0.smithi096.stdout:7/606: dread fa [0,4194304] 0 2024-04-22T15:18:45.622 INFO:tasks.workunit.client.0.smithi096.stdout:7/607: dread - d31/d32/d33/d54/f89 zero size 2024-04-22T15:18:45.622 INFO:tasks.workunit.client.0.smithi096.stdout:2/447: dread d3/d1f/f56 [0,4194304] 0 2024-04-22T15:18:45.623 INFO:tasks.workunit.client.1.smithi196.stdout:2/10: rename d1/c2 to d1/c4 0 2024-04-22T15:18:45.624 INFO:tasks.workunit.client.0.smithi096.stdout:3/291: unlink da/db/c2f 0 2024-04-22T15:18:45.628 INFO:tasks.workunit.client.1.smithi196.stdout:0/12: mkdir d3/d6 0 2024-04-22T15:18:45.629 INFO:tasks.workunit.client.0.smithi096.stdout:6/370: symlink d3/d28/dd/d11/d2c/d6f/d7c/l7f 0 2024-04-22T15:18:45.629 INFO:tasks.workunit.client.1.smithi196.stdout:0/13: fdatasync d3/f5 0 2024-04-22T15:18:45.630 INFO:tasks.workunit.client.1.smithi196.stdout:2/11: dwrite d1/f3 [0,4194304] 0 2024-04-22T15:18:45.630 INFO:tasks.workunit.client.1.smithi196.stdout:0/14: chown c2 98572472 1 2024-04-22T15:18:45.631 INFO:tasks.workunit.client.1.smithi196.stdout:0/15: stat d3/d6 0 2024-04-22T15:18:45.632 INFO:tasks.workunit.client.1.smithi196.stdout:2/12: read f0 [56800,51884] 0 2024-04-22T15:18:45.634 INFO:tasks.workunit.client.0.smithi096.stdout:5/407: creat d4/d1f/d80/f87 x:0 0 0 2024-04-22T15:18:45.636 INFO:tasks.workunit.client.1.smithi196.stdout:4/14: creat f5 x:0 0 0 2024-04-22T15:18:45.636 INFO:tasks.workunit.client.1.smithi196.stdout:4/15: read - f4 zero size 2024-04-22T15:18:45.637 INFO:tasks.workunit.client.1.smithi196.stdout:1/11: mknod d1/c6 0 2024-04-22T15:18:45.641 INFO:tasks.workunit.client.0.smithi096.stdout:1/576: rename d6/dc/d8a/d3f/d54/d32/d8c/d45/c93 to d6/cb4 0 2024-04-22T15:18:45.642 INFO:tasks.workunit.client.1.smithi196.stdout:6/4: truncate f0 4187040 0 2024-04-22T15:18:45.642 INFO:tasks.workunit.client.1.smithi196.stdout:6/5: readlink - no filename 2024-04-22T15:18:45.642 INFO:tasks.workunit.client.1.smithi196.stdout:6/6: chown f0 352952577 1 2024-04-22T15:18:45.643 INFO:tasks.workunit.client.1.smithi196.stdout:9/11: link f0 f1 0 2024-04-22T15:18:45.643 INFO:tasks.workunit.client.0.smithi096.stdout:9/326: dread d1/d25/f2e [0,4194304] 0 2024-04-22T15:18:45.643 INFO:tasks.workunit.client.1.smithi196.stdout:9/12: chown f1 88431237 1 2024-04-22T15:18:45.644 INFO:tasks.workunit.client.1.smithi196.stdout:9/13: chown f0 398 1 2024-04-22T15:18:45.645 INFO:tasks.workunit.client.0.smithi096.stdout:2/448: mkdir d3/db/d2d/d46/d7e/d8d 0 2024-04-22T15:18:45.646 INFO:tasks.workunit.client.1.smithi196.stdout:8/3: truncate f0 3767953 0 2024-04-22T15:18:45.647 INFO:tasks.workunit.client.0.smithi096.stdout:3/292: mkdir da/db/dc/d59 0 2024-04-22T15:18:45.648 INFO:tasks.workunit.client.1.smithi196.stdout:5/11: mkdir d1/d4 0 2024-04-22T15:18:45.654 INFO:tasks.workunit.client.0.smithi096.stdout:4/389: dwrite d5/d7/d2c/d3d/d4f/f5c [0,4194304] 0 2024-04-22T15:18:45.657 INFO:tasks.workunit.client.0.smithi096.stdout:4/390: write d5/f39 [1447786,58107] 0 2024-04-22T15:18:45.657 INFO:tasks.workunit.client.0.smithi096.stdout:5/408: write d4/d11/d25/f38 [3037870,7981] 0 2024-04-22T15:18:45.658 INFO:tasks.workunit.client.0.smithi096.stdout:3/293: dwrite da/db/de/d19/d2d/f51 [0,4194304] 0 2024-04-22T15:18:45.661 INFO:tasks.workunit.client.0.smithi096.stdout:7/608: creat d31/d32/dc0/fc4 x:0 0 0 2024-04-22T15:18:45.663 INFO:tasks.workunit.client.1.smithi196.stdout:2/13: creat d1/f5 x:0 0 0 2024-04-22T15:18:45.666 INFO:tasks.workunit.client.1.smithi196.stdout:4/16: mkdir d6 0 2024-04-22T15:18:45.666 INFO:tasks.workunit.client.1.smithi196.stdout:4/17: fdatasync f5 0 2024-04-22T15:18:45.668 INFO:tasks.workunit.client.0.smithi096.stdout:5/409: dwrite d4/d11/f7a [0,4194304] 0 2024-04-22T15:18:45.669 INFO:tasks.workunit.client.0.smithi096.stdout:4/391: dwrite d5/d7/d2c/d3d/d4f/f5c [0,4194304] 0 2024-04-22T15:18:45.669 INFO:tasks.workunit.client.1.smithi196.stdout:6/7: fdatasync f0 0 2024-04-22T15:18:45.670 INFO:tasks.workunit.client.1.smithi196.stdout:6/8: chown f0 252 1 2024-04-22T15:18:45.670 INFO:tasks.workunit.client.0.smithi096.stdout:8/374: write d4/d10/d25/d38/f4b [3052362,32322] 0 2024-04-22T15:18:45.678 INFO:tasks.workunit.client.1.smithi196.stdout:8/4: read f0 [2858239,52397] 0 2024-04-22T15:18:45.681 INFO:tasks.workunit.client.1.smithi196.stdout:0/16: rename l1 to d3/d6/l7 0 2024-04-22T15:18:45.683 INFO:tasks.workunit.client.1.smithi196.stdout:9/14: dwrite f0 [0,4194304] 0 2024-04-22T15:18:45.684 INFO:tasks.workunit.client.0.smithi096.stdout:3/294: mknod da/db/de/d19/d3e/c5a 0 2024-04-22T15:18:45.687 INFO:tasks.workunit.client.1.smithi196.stdout:2/14: mknod d1/c6 0 2024-04-22T15:18:45.688 INFO:tasks.workunit.client.1.smithi196.stdout:5/12: dwrite f0 [0,4194304] 0 2024-04-22T15:18:45.695 INFO:tasks.workunit.client.0.smithi096.stdout:8/375: rmdir d4/d1c/d40/d48 39 2024-04-22T15:18:45.696 INFO:tasks.workunit.client.0.smithi096.stdout:8/376: dread - d4/d10/d25/d38/f54 zero size 2024-04-22T15:18:45.696 INFO:tasks.workunit.client.0.smithi096.stdout:1/577: truncate d6/f14 525197 0 2024-04-22T15:18:45.697 INFO:tasks.workunit.client.0.smithi096.stdout:3/295: dread da/db/de/d19/f2a [4194304,4194304] 0 2024-04-22T15:18:45.697 INFO:tasks.workunit.client.0.smithi096.stdout:8/377: chown d4/d10/d25/d38/f54 3630497 1 2024-04-22T15:18:45.700 INFO:tasks.workunit.client.1.smithi196.stdout:8/5: write f0 [2429362,6951] 0 2024-04-22T15:18:45.700 INFO:tasks.workunit.client.0.smithi096.stdout:3/296: write da/db/de/d19/d3e/f4f [767438,94638] 0 2024-04-22T15:18:45.704 INFO:tasks.workunit.client.1.smithi196.stdout:0/17: symlink d3/d6/l8 0 2024-04-22T15:18:45.705 INFO:tasks.workunit.client.1.smithi196.stdout:8/6: dwrite f0 [0,4194304] 0 2024-04-22T15:18:45.706 INFO:tasks.workunit.client.0.smithi096.stdout:2/449: creat d3/f8e x:0 0 0 2024-04-22T15:18:45.707 INFO:tasks.workunit.client.0.smithi096.stdout:2/450: fdatasync d3/d1f/d2f/f41 0 2024-04-22T15:18:45.709 INFO:tasks.workunit.client.1.smithi196.stdout:2/15: rename f0 to d1/f7 0 2024-04-22T15:18:45.714 INFO:tasks.workunit.client.1.smithi196.stdout:0/18: dread d3/f5 [0,4194304] 0 2024-04-22T15:18:45.714 INFO:tasks.workunit.client.1.smithi196.stdout:4/18: mknod d6/c7 0 2024-04-22T15:18:45.714 INFO:tasks.workunit.client.1.smithi196.stdout:0/19: read d3/f5 [1823193,63081] 0 2024-04-22T15:18:45.714 INFO:tasks.workunit.client.1.smithi196.stdout:8/7: dread f0 [0,4194304] 0 2024-04-22T15:18:45.714 INFO:tasks.workunit.client.1.smithi196.stdout:0/20: stat d3/f5 0 2024-04-22T15:18:45.716 INFO:tasks.workunit.client.0.smithi096.stdout:7/609: dwrite d31/d32/d33/d54/f51 [0,4194304] 0 2024-04-22T15:18:45.717 INFO:tasks.workunit.client.1.smithi196.stdout:2/16: creat d1/f8 x:0 0 0 2024-04-22T15:18:45.717 INFO:tasks.workunit.client.1.smithi196.stdout:2/17: readlink - no filename 2024-04-22T15:18:45.718 INFO:tasks.workunit.client.0.smithi096.stdout:3/297: rmdir da/db/d10 39 2024-04-22T15:18:45.719 INFO:tasks.workunit.client.1.smithi196.stdout:4/19: mknod d6/c8 0 2024-04-22T15:18:45.721 INFO:tasks.workunit.client.1.smithi196.stdout:8/8: dwrite f0 [4194304,4194304] 0 2024-04-22T15:18:45.721 INFO:tasks.workunit.client.1.smithi196.stdout:4/20: write f4 [304473,70783] 0 2024-04-22T15:18:45.722 INFO:tasks.workunit.client.1.smithi196.stdout:0/21: mkdir d3/d9 0 2024-04-22T15:18:45.723 INFO:tasks.workunit.client.1.smithi196.stdout:0/22: write d3/f5 [76406,60065] 0 2024-04-22T15:18:45.724 INFO:tasks.workunit.client.0.smithi096.stdout:2/451: fsync d3/f70 0 2024-04-22T15:18:45.726 INFO:tasks.workunit.client.1.smithi196.stdout:0/23: write d3/f5 [1187843,86904] 0 2024-04-22T15:18:45.728 INFO:tasks.workunit.client.1.smithi196.stdout:2/18: symlink d1/l9 0 2024-04-22T15:18:45.729 INFO:tasks.workunit.client.0.smithi096.stdout:5/410: truncate d4/d11/d3d/d4d/d59/d5a/f13 1025708 0 2024-04-22T15:18:45.730 INFO:tasks.workunit.client.1.smithi196.stdout:6/9: getdents . 0 2024-04-22T15:18:45.732 INFO:tasks.workunit.client.0.smithi096.stdout:1/578: unlink d6/dc/d8a/d3f/d54/f43 0 2024-04-22T15:18:45.735 INFO:tasks.workunit.client.0.smithi096.stdout:4/392: rename d5/d7/d2c/d3d/d4f/f5d to d5/d7/f7f 0 2024-04-22T15:18:45.735 INFO:tasks.workunit.client.1.smithi196.stdout:4/21: dwrite f4 [0,4194304] 0 2024-04-22T15:18:45.736 INFO:tasks.workunit.client.0.smithi096.stdout:1/579: read d6/dc/d8a/d3f/d54/d32/f6d [3537903,7692] 0 2024-04-22T15:18:45.736 INFO:tasks.workunit.client.0.smithi096.stdout:1/580: stat d6/dc/d8a/d3f/d62/f70 0 2024-04-22T15:18:45.737 INFO:tasks.workunit.client.0.smithi096.stdout:1/581: truncate d6/f1d 868511 0 2024-04-22T15:18:45.737 INFO:tasks.workunit.client.1.smithi196.stdout:4/22: write f4 [1507629,109646] 0 2024-04-22T15:18:45.737 INFO:tasks.workunit.client.1.smithi196.stdout:4/23: stat c2 0 2024-04-22T15:18:45.738 INFO:tasks.workunit.client.0.smithi096.stdout:8/378: creat d4/d10/d25/d4c/f87 x:0 0 0 2024-04-22T15:18:45.739 INFO:tasks.workunit.client.0.smithi096.stdout:7/610: symlink d31/d32/d33/d5e/lc5 0 2024-04-22T15:18:45.739 INFO:tasks.workunit.client.1.smithi196.stdout:4/24: write f5 [440440,72373] 0 2024-04-22T15:18:45.739 INFO:tasks.workunit.client.1.smithi196.stdout:0/24: readlink l0 0 2024-04-22T15:18:45.740 INFO:tasks.workunit.client.1.smithi196.stdout:2/19: mknod d1/ca 0 2024-04-22T15:18:45.741 INFO:tasks.workunit.client.1.smithi196.stdout:6/10: creat f1 x:0 0 0 2024-04-22T15:18:45.742 INFO:tasks.workunit.client.1.smithi196.stdout:8/9: dwrite f0 [0,4194304] 0 2024-04-22T15:18:45.742 INFO:tasks.workunit.client.1.smithi196.stdout:6/11: read - f1 zero size 2024-04-22T15:18:45.743 INFO:tasks.workunit.client.1.smithi196.stdout:8/10: write f0 [5940422,35751] 0 2024-04-22T15:18:45.744 INFO:tasks.workunit.client.1.smithi196.stdout:0/25: creat d3/fa x:0 0 0 2024-04-22T15:18:45.747 INFO:tasks.workunit.client.1.smithi196.stdout:8/11: symlink l1 0 2024-04-22T15:18:45.749 INFO:tasks.workunit.client.1.smithi196.stdout:8/12: mkdir d2 0 2024-04-22T15:18:45.749 INFO:tasks.workunit.client.1.smithi196.stdout:2/20: dwrite d1/f3 [0,4194304] 0 2024-04-22T15:18:45.750 INFO:tasks.workunit.client.1.smithi196.stdout:6/12: truncate f0 2747249 0 2024-04-22T15:18:45.751 INFO:tasks.workunit.client.1.smithi196.stdout:2/21: write d1/f8 [327008,128496] 0 2024-04-22T15:18:45.754 INFO:tasks.workunit.client.1.smithi196.stdout:2/22: mknod d1/cb 0 2024-04-22T15:18:45.756 INFO:tasks.workunit.client.1.smithi196.stdout:2/23: creat d1/fc x:0 0 0 2024-04-22T15:18:45.756 INFO:tasks.workunit.client.1.smithi196.stdout:8/13: dread f0 [0,4194304] 0 2024-04-22T15:18:45.756 INFO:tasks.workunit.client.1.smithi196.stdout:8/14: chown d2 22 1 2024-04-22T15:18:45.757 INFO:tasks.workunit.client.1.smithi196.stdout:2/24: creat d1/fd x:0 0 0 2024-04-22T15:18:45.757 INFO:tasks.workunit.client.1.smithi196.stdout:2/25: stat d1/c6 0 2024-04-22T15:18:45.758 INFO:tasks.workunit.client.1.smithi196.stdout:2/26: readlink d1/l9 0 2024-04-22T15:18:45.764 INFO:tasks.workunit.client.1.smithi196.stdout:8/15: dwrite f0 [4194304,4194304] 0 2024-04-22T15:18:45.766 INFO:tasks.workunit.client.1.smithi196.stdout:8/16: write f0 [8714931,24165] 0 2024-04-22T15:18:45.766 INFO:tasks.workunit.client.1.smithi196.stdout:2/27: dwrite d1/f8 [0,4194304] 0 2024-04-22T15:18:45.768 INFO:tasks.workunit.client.1.smithi196.stdout:2/28: rename d1/l9 to d1/le 0 2024-04-22T15:18:45.768 INFO:tasks.workunit.client.1.smithi196.stdout:2/29: stat d1 0 2024-04-22T15:18:45.774 INFO:tasks.workunit.client.1.smithi196.stdout:2/30: dwrite d1/fd [0,4194304] 0 2024-04-22T15:18:45.775 INFO:tasks.workunit.client.1.smithi196.stdout:2/31: creat d1/ff x:0 0 0 2024-04-22T15:18:45.778 INFO:tasks.workunit.client.0.smithi096.stdout:4/393: read d5/d7/d1a/f40 [3374069,108687] 0 2024-04-22T15:18:45.779 INFO:tasks.workunit.client.0.smithi096.stdout:2/452: creat d3/db/df/d24/f8f x:0 0 0 2024-04-22T15:18:45.782 INFO:tasks.workunit.client.0.smithi096.stdout:3/298: write da/db/d10/f33 [1395884,128230] 0 2024-04-22T15:18:45.785 INFO:tasks.workunit.client.0.smithi096.stdout:7/611: fdatasync d31/d32/d33/d93/fae 0 2024-04-22T15:18:45.786 INFO:tasks.workunit.client.1.smithi196.stdout:4/25: read f5 [22581,126015] 0 2024-04-22T15:18:45.787 INFO:tasks.workunit.client.0.smithi096.stdout:7/612: truncate d31/d32/f47 1585370 0 2024-04-22T15:18:45.789 INFO:tasks.workunit.client.1.smithi196.stdout:4/26: mknod d6/c9 0 2024-04-22T15:18:45.790 INFO:tasks.workunit.client.0.smithi096.stdout:4/394: creat d5/d5b/f80 x:0 0 0 2024-04-22T15:18:45.792 INFO:tasks.workunit.client.1.smithi196.stdout:4/27: mkdir d6/da 0 2024-04-22T15:18:45.793 INFO:tasks.workunit.client.0.smithi096.stdout:1/582: unlink d6/cb4 0 2024-04-22T15:18:45.794 INFO:tasks.workunit.client.0.smithi096.stdout:5/411: truncate d4/f26 3828390 0 2024-04-22T15:18:45.795 INFO:tasks.workunit.client.1.smithi196.stdout:4/28: dread f4 [0,4194304] 0 2024-04-22T15:18:45.797 INFO:tasks.workunit.client.0.smithi096.stdout:8/379: rename d4/d1c/l53 to d4/d2a/d7e/l88 0 2024-04-22T15:18:45.799 INFO:tasks.workunit.client.0.smithi096.stdout:2/453: symlink d3/db/d2d/d46/d7e/l90 0 2024-04-22T15:18:45.800 INFO:tasks.workunit.client.0.smithi096.stdout:3/299: write da/db/d12/f56 [690762,97253] 0 2024-04-22T15:18:45.801 INFO:tasks.workunit.client.1.smithi196.stdout:4/29: rename f4 to d6/da/fb 0 2024-04-22T15:18:45.806 INFO:tasks.workunit.client.0.smithi096.stdout:7/613: creat d31/d32/d33/d44/fc6 x:0 0 0 2024-04-22T15:18:45.808 INFO:tasks.workunit.client.1.smithi196.stdout:4/30: mkdir d6/da/dc 0 2024-04-22T15:18:45.811 INFO:tasks.workunit.client.0.smithi096.stdout:5/412: unlink d4/d1f/c1a 0 2024-04-22T15:18:45.815 INFO:tasks.workunit.client.0.smithi096.stdout:4/395: chown d5/d7/f7f 3713 1 2024-04-22T15:18:45.820 INFO:tasks.workunit.client.0.smithi096.stdout:8/380: creat d4/d10/d25/d4c/d76/f89 x:0 0 0 2024-04-22T15:18:45.820 INFO:tasks.workunit.client.0.smithi096.stdout:8/381: readlink d4/d10/l59 0 2024-04-22T15:18:45.823 INFO:tasks.workunit.client.0.smithi096.stdout:3/300: creat da/d3f/f5b x:0 0 0 2024-04-22T15:18:45.827 INFO:tasks.workunit.client.0.smithi096.stdout:4/396: creat d5/d7/d2c/d67/f81 x:0 0 0 2024-04-22T15:18:45.828 INFO:tasks.workunit.client.0.smithi096.stdout:8/382: rmdir d4/d10/d25 39 2024-04-22T15:18:45.828 INFO:tasks.workunit.client.0.smithi096.stdout:4/397: stat d5/d7/d2c/d3d/f6a 0 2024-04-22T15:18:45.832 INFO:tasks.workunit.client.0.smithi096.stdout:7/614: dread d31/d32/f59 [0,4194304] 0 2024-04-22T15:18:45.833 INFO:tasks.workunit.client.0.smithi096.stdout:3/301: dread da/db/d12/d22/f3a [0,4194304] 0 2024-04-22T15:18:45.833 INFO:tasks.workunit.client.1.smithi196.stdout:8/17: fdatasync f0 0 2024-04-22T15:18:45.834 INFO:tasks.workunit.client.1.smithi196.stdout:8/18: write f0 [3122256,108346] 0 2024-04-22T15:18:45.834 INFO:tasks.workunit.client.1.smithi196.stdout:2/32: fsync d1/f8 0 2024-04-22T15:18:45.836 INFO:tasks.workunit.client.1.smithi196.stdout:2/33: write d1/f3 [486845,81895] 0 2024-04-22T15:18:45.838 INFO:tasks.workunit.client.1.smithi196.stdout:8/19: dread f0 [0,4194304] 0 2024-04-22T15:18:45.840 INFO:tasks.workunit.client.1.smithi196.stdout:0/26: truncate d3/f5 3934928 0 2024-04-22T15:18:45.840 INFO:tasks.workunit.client.1.smithi196.stdout:6/13: dread f0 [0,4194304] 0 2024-04-22T15:18:45.841 INFO:tasks.workunit.client.1.smithi196.stdout:6/14: truncate f1 129659 0 2024-04-22T15:18:45.842 INFO:tasks.workunit.client.1.smithi196.stdout:2/34: creat d1/f10 x:0 0 0 2024-04-22T15:18:45.843 INFO:tasks.workunit.client.0.smithi096.stdout:3/302: creat da/db/de/d19/d2d/f5c x:0 0 0 2024-04-22T15:18:45.845 INFO:tasks.workunit.client.0.smithi096.stdout:8/383: chown d4/d10/d25/f2e 3973 1 2024-04-22T15:18:45.846 INFO:tasks.workunit.client.0.smithi096.stdout:4/398: creat d5/d7/d1a/d61/d6d/f82 x:0 0 0 2024-04-22T15:18:45.847 INFO:tasks.workunit.client.0.smithi096.stdout:4/399: write d5/d7/d1f/f7a [818545,16938] 0 2024-04-22T15:18:45.847 INFO:tasks.workunit.client.1.smithi196.stdout:2/35: dwrite d1/fc [0,4194304] 0 2024-04-22T15:18:45.848 INFO:tasks.workunit.client.0.smithi096.stdout:5/413: getdents d4/d11/d25 0 2024-04-22T15:18:45.848 INFO:tasks.workunit.client.1.smithi196.stdout:2/36: write d1/ff [59814,72193] 0 2024-04-22T15:18:45.851 INFO:tasks.workunit.client.0.smithi096.stdout:7/615: mkdir d31/d32/d33/d6a/d83/dba/dc7 0 2024-04-22T15:18:45.853 INFO:tasks.workunit.client.1.smithi196.stdout:6/15: creat f2 x:0 0 0 2024-04-22T15:18:45.854 INFO:tasks.workunit.client.0.smithi096.stdout:3/303: creat da/db/dc/f5d x:0 0 0 2024-04-22T15:18:45.854 INFO:tasks.workunit.client.1.smithi196.stdout:6/16: read f1 [84617,7024] 0 2024-04-22T15:18:45.854 INFO:tasks.workunit.client.1.smithi196.stdout:6/17: read - f2 zero size 2024-04-22T15:18:45.856 INFO:tasks.workunit.client.0.smithi096.stdout:1/583: truncate d6/dc/d8a/d3f/d54/d32/f6d 1790918 0 2024-04-22T15:18:45.857 INFO:tasks.workunit.client.1.smithi196.stdout:2/37: dwrite d1/ff [0,4194304] 0 2024-04-22T15:18:45.857 INFO:tasks.workunit.client.1.smithi196.stdout:8/20: creat d2/f3 x:0 0 0 2024-04-22T15:18:45.858 INFO:tasks.workunit.client.1.smithi196.stdout:8/21: dread - d2/f3 zero size 2024-04-22T15:18:45.859 INFO:tasks.workunit.client.0.smithi096.stdout:7/616: dwrite d31/d32/d36/f45 [0,4194304] 0 2024-04-22T15:18:45.865 INFO:tasks.workunit.client.1.smithi196.stdout:6/18: creat f3 x:0 0 0 2024-04-22T15:18:45.869 INFO:tasks.workunit.client.1.smithi196.stdout:4/31: truncate d6/da/fb 3281612 0 2024-04-22T15:18:45.870 INFO:tasks.workunit.client.1.smithi196.stdout:2/38: creat d1/f11 x:0 0 0 2024-04-22T15:18:45.872 INFO:tasks.workunit.client.1.smithi196.stdout:8/22: fsync f0 0 2024-04-22T15:18:45.872 INFO:tasks.workunit.client.1.smithi196.stdout:8/23: dread - d2/f3 zero size 2024-04-22T15:18:45.874 INFO:tasks.workunit.client.0.smithi096.stdout:4/400: rename d5/d7/d2c/l4c to d5/d7/d2c/d3d/d4f/d63/l83 0 2024-04-22T15:18:45.875 INFO:tasks.workunit.client.0.smithi096.stdout:4/401: write d5/f39 [4604355,15311] 0 2024-04-22T15:18:45.875 INFO:tasks.workunit.client.0.smithi096.stdout:7/617: creat d31/d32/d33/d6a/d83/fc8 x:0 0 0 2024-04-22T15:18:45.877 INFO:tasks.workunit.client.1.smithi196.stdout:6/19: dread f1 [0,4194304] 0 2024-04-22T15:18:45.877 INFO:tasks.workunit.client.0.smithi096.stdout:7/618: write d31/d32/d36/d75/f9a [788982,104347] 0 2024-04-22T15:18:45.881 INFO:tasks.workunit.client.1.smithi196.stdout:2/39: creat d1/f12 x:0 0 0 2024-04-22T15:18:45.882 INFO:tasks.workunit.client.0.smithi096.stdout:3/304: dread da/db/de/d19/d2d/f2e [0,4194304] 0 2024-04-22T15:18:45.883 INFO:tasks.workunit.client.1.smithi196.stdout:8/24: creat d2/f4 x:0 0 0 2024-04-22T15:18:45.885 INFO:tasks.workunit.client.0.smithi096.stdout:2/454: truncate d3/db/df/d24/d58/f77 664161 0 2024-04-22T15:18:45.886 INFO:tasks.workunit.client.0.smithi096.stdout:7/619: dwrite d31/d32/d33/f5c [0,4194304] 0 2024-04-22T15:18:45.887 INFO:tasks.workunit.client.1.smithi196.stdout:2/40: dwrite d1/fc [0,4194304] 0 2024-04-22T15:18:45.891 INFO:tasks.workunit.client.1.smithi196.stdout:8/25: dwrite d2/f4 [0,4194304] 0 2024-04-22T15:18:45.892 INFO:tasks.workunit.client.1.smithi196.stdout:8/26: readlink l1 0 2024-04-22T15:18:45.893 INFO:tasks.workunit.client.0.smithi096.stdout:3/305: dwrite da/db/de/d19/d3e/f4f [0,4194304] 0 2024-04-22T15:18:45.894 INFO:tasks.workunit.client.1.smithi196.stdout:8/27: write d2/f4 [3262057,126251] 0 2024-04-22T15:18:45.895 INFO:tasks.workunit.client.0.smithi096.stdout:8/384: getdents d4/d10/d25/d4c 0 2024-04-22T15:18:45.896 INFO:tasks.workunit.client.1.smithi196.stdout:8/28: readlink l1 0 2024-04-22T15:18:45.896 INFO:tasks.workunit.client.1.smithi196.stdout:2/41: mkdir d1/d13 0 2024-04-22T15:18:45.897 INFO:tasks.workunit.client.0.smithi096.stdout:4/402: rename d5/d7/d1a/d61 to d5/d7/d1f/d84 0 2024-04-22T15:18:45.899 INFO:tasks.workunit.client.0.smithi096.stdout:4/403: readlink d5/d7/d2c/l6f 0 2024-04-22T15:18:45.900 INFO:tasks.workunit.client.1.smithi196.stdout:8/29: write d2/f4 [1225698,92620] 0 2024-04-22T15:18:45.902 INFO:tasks.workunit.client.0.smithi096.stdout:1/584: dread d6/f39 [0,4194304] 0 2024-04-22T15:18:45.903 INFO:tasks.workunit.client.1.smithi196.stdout:0/27: dwrite d3/f5 [0,4194304] 0 2024-04-22T15:18:45.903 INFO:tasks.workunit.client.1.smithi196.stdout:8/30: read f0 [5067708,5132] 0 2024-04-22T15:18:45.905 INFO:tasks.workunit.client.0.smithi096.stdout:8/385: dwrite d4/d10/d25/d38/f4b [4194304,4194304] 0 2024-04-22T15:18:45.909 INFO:tasks.workunit.client.1.smithi196.stdout:2/42: dread d1/ff [0,4194304] 0 2024-04-22T15:18:45.911 INFO:tasks.workunit.client.0.smithi096.stdout:3/306: mknod da/db/d12/d22/c5e 0 2024-04-22T15:18:45.912 INFO:tasks.workunit.client.1.smithi196.stdout:8/31: dread f0 [0,4194304] 0 2024-04-22T15:18:45.914 INFO:tasks.workunit.client.1.smithi196.stdout:8/32: write d2/f3 [421331,115346] 0 2024-04-22T15:18:45.917 INFO:tasks.workunit.client.1.smithi196.stdout:2/43: dread d1/fd [0,4194304] 0 2024-04-22T15:18:45.918 INFO:tasks.workunit.client.0.smithi096.stdout:4/404: dread - d5/f21 zero size 2024-04-22T15:18:45.921 INFO:tasks.workunit.client.1.smithi196.stdout:0/28: mknod d3/d9/cb 0 2024-04-22T15:18:45.921 INFO:tasks.workunit.client.1.smithi196.stdout:6/20: fsync f3 0 2024-04-22T15:18:45.921 INFO:tasks.workunit.client.0.smithi096.stdout:3/307: dwrite da/db/de/d19/d2d/f51 [0,4194304] 0 2024-04-22T15:18:45.922 INFO:tasks.workunit.client.1.smithi196.stdout:6/21: chown f0 27 1 2024-04-22T15:18:45.923 INFO:tasks.workunit.client.0.smithi096.stdout:1/585: creat d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b/fb5 x:0 0 0 2024-04-22T15:18:45.924 INFO:tasks.workunit.client.1.smithi196.stdout:8/33: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:18:45.924 INFO:tasks.workunit.client.1.smithi196.stdout:6/22: write f3 [764570,2281] 0 2024-04-22T15:18:45.926 INFO:tasks.workunit.client.0.smithi096.stdout:8/386: symlink d4/d1c/d34/l8a 0 2024-04-22T15:18:45.927 INFO:tasks.workunit.client.0.smithi096.stdout:8/387: chown d4/d10/d25/d38/f54 27373 1 2024-04-22T15:18:45.929 INFO:tasks.workunit.client.1.smithi196.stdout:8/34: dread d2/f3 [0,4194304] 0 2024-04-22T15:18:45.930 INFO:tasks.workunit.client.1.smithi196.stdout:8/35: stat d2 0 2024-04-22T15:18:45.931 INFO:tasks.workunit.client.1.smithi196.stdout:8/36: write d2/f3 [4125668,27258] 0 2024-04-22T15:18:45.932 INFO:tasks.workunit.client.1.smithi196.stdout:0/29: mkdir d3/d6/dc 0 2024-04-22T15:18:45.936 INFO:tasks.workunit.client.0.smithi096.stdout:3/308: stat da/db/d10/c18 0 2024-04-22T15:18:45.937 INFO:tasks.workunit.client.0.smithi096.stdout:5/414: dwrite d4/d11/d3d/d4d/d59/d5a/f79 [0,4194304] 0 2024-04-22T15:18:45.940 INFO:tasks.workunit.client.0.smithi096.stdout:3/309: read - da/d3f/f5b zero size 2024-04-22T15:18:45.944 INFO:tasks.workunit.client.0.smithi096.stdout:5/415: dread d4/d11/d3d/d4d/d59/d5a/f79 [0,4194304] 0 2024-04-22T15:18:45.946 INFO:tasks.workunit.client.0.smithi096.stdout:5/416: truncate d4/d11/d3d/d4d/d59/d5a/d5c/f5f 10293 0 2024-04-22T15:18:45.947 INFO:tasks.workunit.client.1.smithi196.stdout:2/44: getdents d1/d13 0 2024-04-22T15:18:45.953 INFO:tasks.workunit.client.1.smithi196.stdout:0/30: creat d3/fd x:0 0 0 2024-04-22T15:18:45.953 INFO:tasks.workunit.client.0.smithi096.stdout:3/310: read da/db/dc/f16 [2021779,95281] 0 2024-04-22T15:18:45.954 INFO:tasks.workunit.client.0.smithi096.stdout:2/455: write d3/db/d30/f3e [865206,60910] 0 2024-04-22T15:18:45.963 INFO:tasks.workunit.client.1.smithi196.stdout:0/31: mkdir d3/de 0 2024-04-22T15:18:45.967 INFO:tasks.workunit.client.0.smithi096.stdout:7/620: write d31/d32/d33/d54/d43/f48 [1339996,48374] 0 2024-04-22T15:18:45.968 INFO:tasks.workunit.client.0.smithi096.stdout:7/621: fdatasync d31/d32/d36/f45 0 2024-04-22T15:18:45.969 INFO:tasks.workunit.client.0.smithi096.stdout:4/405: rename d5/d7/d2c/d3d/f6a to d5/d7/d2c/d3d/f85 0 2024-04-22T15:18:45.969 INFO:tasks.workunit.client.0.smithi096.stdout:3/311: rmdir da/db/d10 39 2024-04-22T15:18:45.970 INFO:tasks.workunit.client.0.smithi096.stdout:3/312: chown da/db/l54 798643 1 2024-04-22T15:18:45.974 INFO:tasks.workunit.client.1.smithi196.stdout:6/23: write f1 [357995,116039] 0 2024-04-22T15:18:45.976 INFO:tasks.workunit.client.0.smithi096.stdout:7/622: dwrite d31/d32/d33/d54/f51 [0,4194304] 0 2024-04-22T15:18:45.976 INFO:tasks.workunit.client.1.smithi196.stdout:8/37: write d2/f3 [4407782,72221] 0 2024-04-22T15:18:45.979 INFO:tasks.workunit.client.1.smithi196.stdout:6/24: rename f3 to f4 0 2024-04-22T15:18:45.981 INFO:tasks.workunit.client.1.smithi196.stdout:8/38: mkdir d2/d5 0 2024-04-22T15:18:45.984 INFO:tasks.workunit.client.0.smithi096.stdout:7/623: dwrite d31/d32/f60 [0,4194304] 0 2024-04-22T15:18:45.986 INFO:tasks.workunit.client.1.smithi196.stdout:6/25: fsync f0 0 2024-04-22T15:18:45.988 INFO:tasks.workunit.client.1.smithi196.stdout:2/45: write d1/f7 [1965357,91283] 0 2024-04-22T15:18:45.990 INFO:tasks.workunit.client.0.smithi096.stdout:0/541: sync 2024-04-22T15:18:45.990 INFO:tasks.workunit.client.0.smithi096.stdout:6/371: sync 2024-04-22T15:18:45.990 INFO:tasks.workunit.client.0.smithi096.stdout:9/327: sync 2024-04-22T15:18:45.992 INFO:tasks.workunit.client.0.smithi096.stdout:6/372: truncate d3/d28/f1d 2911358 0 2024-04-22T15:18:45.992 INFO:tasks.workunit.client.0.smithi096.stdout:9/328: read d1/d4/d9/f12 [680849,14056] 0 2024-04-22T15:18:45.993 INFO:tasks.workunit.client.0.smithi096.stdout:0/542: dread - d7/de/d5b/d95/dc/d16/d18/f69 zero size 2024-04-22T15:18:45.994 INFO:tasks.workunit.client.0.smithi096.stdout:9/329: readlink d1/d4/d9/l6c 0 2024-04-22T15:18:45.995 INFO:tasks.workunit.client.0.smithi096.stdout:3/313: write da/db/d12/d22/f3a [2089649,194] 0 2024-04-22T15:18:45.996 INFO:tasks.workunit.client.0.smithi096.stdout:5/417: truncate f1 1276462 0 2024-04-22T15:18:45.997 INFO:tasks.workunit.client.0.smithi096.stdout:3/314: write da/db/de/d19/d2d/f51 [266955,8775] 0 2024-04-22T15:18:45.998 INFO:tasks.workunit.client.0.smithi096.stdout:3/315: truncate da/db/de/f57 553554 0 2024-04-22T15:18:46.000 INFO:tasks.workunit.client.0.smithi096.stdout:1/586: rename d6/dc/d8a/c15 to d6/dc/d8a/d3f/cb6 0 2024-04-22T15:18:46.001 INFO:tasks.workunit.client.1.smithi196.stdout:6/26: dwrite f4 [0,4194304] 0 2024-04-22T15:18:46.002 INFO:tasks.workunit.client.1.smithi196.stdout:8/39: mknod d2/d5/c6 0 2024-04-22T15:18:46.002 INFO:tasks.workunit.client.1.smithi196.stdout:8/40: rename d2 to d2/d5/d7 22 2024-04-22T15:18:46.004 INFO:tasks.workunit.client.1.smithi196.stdout:2/46: mknod d1/d13/c14 0 2024-04-22T15:18:46.005 INFO:tasks.workunit.client.0.smithi096.stdout:8/388: getdents d4/d10 0 2024-04-22T15:18:46.006 INFO:tasks.workunit.client.1.smithi196.stdout:6/27: creat f5 x:0 0 0 2024-04-22T15:18:46.006 INFO:tasks.workunit.client.0.smithi096.stdout:8/389: write d4/d1c/d34/f49 [1911802,5729] 0 2024-04-22T15:18:46.007 INFO:tasks.workunit.client.1.smithi196.stdout:8/41: mkdir d2/d8 0 2024-04-22T15:18:46.010 INFO:tasks.workunit.client.0.smithi096.stdout:7/624: mkdir d31/d32/d33/d5e/dc9 0 2024-04-22T15:18:46.010 INFO:tasks.workunit.client.1.smithi196.stdout:8/42: dread d2/f3 [0,4194304] 0 2024-04-22T15:18:46.017 INFO:tasks.workunit.client.1.smithi196.stdout:8/43: mknod d2/c9 0 2024-04-22T15:18:46.021 INFO:tasks.workunit.client.1.smithi196.stdout:8/44: creat d2/fa x:0 0 0 2024-04-22T15:18:46.023 INFO:tasks.workunit.client.0.smithi096.stdout:0/543: rmdir d7/de/d5b/d95/dc/d16/d18 39 2024-04-22T15:18:46.024 INFO:tasks.workunit.client.0.smithi096.stdout:0/544: truncate d7/d42/fb2 674134 0 2024-04-22T15:18:46.027 INFO:tasks.workunit.client.1.smithi196.stdout:2/47: dwrite d1/ff [0,4194304] 0 2024-04-22T15:18:46.032 INFO:tasks.workunit.client.1.smithi196.stdout:2/48: unlink d1/f8 0 2024-04-22T15:18:46.034 INFO:tasks.workunit.client.1.smithi196.stdout:8/45: getdents d2/d5 0 2024-04-22T15:18:46.035 INFO:tasks.workunit.client.0.smithi096.stdout:6/373: dwrite d3/f29 [0,4194304] 0 2024-04-22T15:18:46.037 INFO:tasks.workunit.client.0.smithi096.stdout:6/374: dread - d3/d28/dd/d11/d2c/d44/f67 zero size 2024-04-22T15:18:46.037 INFO:tasks.workunit.client.1.smithi196.stdout:8/46: dread d2/f4 [0,4194304] 0 2024-04-22T15:18:46.038 INFO:tasks.workunit.client.1.smithi196.stdout:2/49: creat d1/f15 x:0 0 0 2024-04-22T15:18:46.038 INFO:tasks.workunit.client.1.smithi196.stdout:2/50: chown d1/d13 3515 1 2024-04-22T15:18:46.040 INFO:tasks.workunit.client.1.smithi196.stdout:8/47: symlink d2/d5/lb 0 2024-04-22T15:18:46.041 INFO:tasks.workunit.client.1.smithi196.stdout:8/48: write d2/fa [434037,26743] 0 2024-04-22T15:18:46.042 INFO:tasks.workunit.client.1.smithi196.stdout:8/49: truncate d2/fa 1450931 0 2024-04-22T15:18:46.042 INFO:tasks.workunit.client.0.smithi096.stdout:9/330: mkdir d1/d6f 0 2024-04-22T15:18:46.043 INFO:tasks.workunit.client.1.smithi196.stdout:2/51: creat d1/f16 x:0 0 0 2024-04-22T15:18:46.043 INFO:tasks.workunit.client.0.smithi096.stdout:9/331: chown d1/d25/f4c 14727968 1 2024-04-22T15:18:46.045 INFO:tasks.workunit.client.0.smithi096.stdout:9/332: readlink d1/df/l15 0 2024-04-22T15:18:46.045 INFO:tasks.workunit.client.0.smithi096.stdout:2/456: dwrite d3/db/df/d24/d58/f77 [0,4194304] 0 2024-04-22T15:18:46.046 INFO:tasks.workunit.client.1.smithi196.stdout:8/50: mkdir d2/dc 0 2024-04-22T15:18:46.047 INFO:tasks.workunit.client.0.smithi096.stdout:9/333: write d1/d4/f4e [787437,61078] 0 2024-04-22T15:18:46.048 INFO:tasks.workunit.client.1.smithi196.stdout:2/52: dwrite d1/f3 [0,4194304] 0 2024-04-22T15:18:46.049 INFO:tasks.workunit.client.0.smithi096.stdout:3/316: creat da/db/de/d19/d2d/d37/f5f x:0 0 0 2024-04-22T15:18:46.050 INFO:tasks.workunit.client.0.smithi096.stdout:1/587: truncate d6/dc/d8a/d3f/d54/d32/d3b/d94/f96 1162279 0 2024-04-22T15:18:46.053 INFO:tasks.workunit.client.0.smithi096.stdout:4/406: getdents d5/d7/d1a/d25 0 2024-04-22T15:18:46.058 INFO:tasks.workunit.client.0.smithi096.stdout:2/457: unlink d3/db/df/d24/d58/d5e/l63 0 2024-04-22T15:18:46.058 INFO:tasks.workunit.client.0.smithi096.stdout:0/545: creat d7/d42/d4c/d55/d9c/fb7 x:0 0 0 2024-04-22T15:18:46.059 INFO:tasks.workunit.client.0.smithi096.stdout:0/546: chown d7/da7 25 1 2024-04-22T15:18:46.060 INFO:tasks.workunit.client.0.smithi096.stdout:3/317: creat da/db/d12/d22/f60 x:0 0 0 2024-04-22T15:18:46.061 INFO:tasks.workunit.client.0.smithi096.stdout:7/625: rename d31/d32/d33/fb1 to d31/d32/d33/fca 0 2024-04-22T15:18:46.063 INFO:tasks.workunit.client.0.smithi096.stdout:0/547: chown d7/d42/d4c/d55/fb3 2 1 2024-04-22T15:18:46.065 INFO:tasks.workunit.client.0.smithi096.stdout:6/375: dwrite d3/f49 [0,4194304] 0 2024-04-22T15:18:46.077 INFO:tasks.workunit.client.0.smithi096.stdout:9/334: mknod d1/c70 0 2024-04-22T15:18:46.078 INFO:tasks.workunit.client.0.smithi096.stdout:9/335: chown d1/d4/d9/dd/f46 443 1 2024-04-22T15:18:46.080 INFO:tasks.workunit.client.1.smithi196.stdout:2/53: mknod d1/c17 0 2024-04-22T15:18:46.080 INFO:tasks.workunit.client.1.smithi196.stdout:8/51: dread f0 [4194304,4194304] 0 2024-04-22T15:18:46.081 INFO:tasks.workunit.client.1.smithi196.stdout:2/54: truncate d1/f12 868321 0 2024-04-22T15:18:46.083 INFO:tasks.workunit.client.0.smithi096.stdout:7/626: creat d31/d68/fcb x:0 0 0 2024-04-22T15:18:46.086 INFO:tasks.workunit.client.0.smithi096.stdout:1/588: rename d6/c3c to d6/dc/d8a/d3f/d54/d32/cb7 0 2024-04-22T15:18:46.090 INFO:tasks.workunit.client.0.smithi096.stdout:3/318: rmdir da/db/de/d19/d2d/d37 39 2024-04-22T15:18:46.092 INFO:tasks.workunit.client.0.smithi096.stdout:0/548: symlink d7/de/d5b/d62/d67/lb8 0 2024-04-22T15:18:46.094 INFO:tasks.workunit.client.0.smithi096.stdout:4/407: read - d5/d7/d2c/d43/d56/f59 zero size 2024-04-22T15:18:46.097 INFO:tasks.workunit.client.0.smithi096.stdout:9/336: mknod d1/d4/d9/d43/d57/c71 0 2024-04-22T15:18:46.103 INFO:tasks.workunit.client.0.smithi096.stdout:6/376: rename d3/f29 to d3/d28/d54/d68/d6e/f80 0 2024-04-22T15:18:46.104 INFO:tasks.workunit.client.0.smithi096.stdout:6/377: chown f1 657321 1 2024-04-22T15:18:46.109 INFO:tasks.workunit.client.0.smithi096.stdout:2/458: unlink d3/db/df/d24/d58/f77 0 2024-04-22T15:18:46.111 INFO:tasks.workunit.client.0.smithi096.stdout:3/319: read da/db/dc/f16 [3279701,52487] 0 2024-04-22T15:18:46.116 INFO:tasks.workunit.client.0.smithi096.stdout:5/418: getdents d4/d2c/d2f/d30/d5b 0 2024-04-22T15:18:46.117 INFO:tasks.workunit.client.0.smithi096.stdout:5/419: fdatasync d4/d11/d3d/d4d/f6e 0 2024-04-22T15:18:46.118 INFO:tasks.workunit.client.0.smithi096.stdout:9/337: write d1/df/f58 [2911809,77895] 0 2024-04-22T15:18:46.121 INFO:tasks.workunit.client.0.smithi096.stdout:5/420: dread - d4/d11/d3d/d4d/d59/d5a/d5c/f83 zero size 2024-04-22T15:18:46.123 INFO:tasks.workunit.client.0.smithi096.stdout:7/627: rename c2a to d31/d32/d33/d6a/d83/dba/ccc 0 2024-04-22T15:18:46.127 INFO:tasks.workunit.client.0.smithi096.stdout:1/589: symlink d6/dc/d8a/d3f/d62/lb8 0 2024-04-22T15:18:46.130 INFO:tasks.workunit.client.0.smithi096.stdout:3/320: write da/db/d10/f33 [4538501,72740] 0 2024-04-22T15:18:46.133 INFO:tasks.workunit.client.0.smithi096.stdout:9/338: mknod d1/d4/d9/d59/c72 0 2024-04-22T15:18:46.134 INFO:tasks.workunit.client.0.smithi096.stdout:5/421: fdatasync d4/d11/f62 0 2024-04-22T15:18:46.137 INFO:tasks.workunit.client.0.smithi096.stdout:7/628: dread d31/d32/d33/f3e [0,4194304] 0 2024-04-22T15:18:46.138 INFO:tasks.workunit.client.0.smithi096.stdout:1/590: chown d6/dc/d8a/d3f/d54/d32/d8c/c26 2 1 2024-04-22T15:18:46.139 INFO:tasks.workunit.client.0.smithi096.stdout:7/629: read d31/d32/d33/d54/f51 [3783114,102787] 0 2024-04-22T15:18:46.139 INFO:tasks.workunit.client.0.smithi096.stdout:1/591: write d6/dc/d8a/f67 [1330249,27811] 0 2024-04-22T15:18:46.140 INFO:tasks.workunit.client.0.smithi096.stdout:1/592: chown d6/dc/d8a/d29/f79 11725209 1 2024-04-22T15:18:46.141 INFO:tasks.workunit.client.0.smithi096.stdout:0/549: creat d7/de/d3a/d61/d79/fb9 x:0 0 0 2024-04-22T15:18:46.146 INFO:tasks.workunit.client.0.smithi096.stdout:4/408: creat d5/d7/d2c/f86 x:0 0 0 2024-04-22T15:18:46.147 INFO:tasks.workunit.client.0.smithi096.stdout:9/339: readlink d1/l29 0 2024-04-22T15:18:46.149 INFO:tasks.workunit.client.0.smithi096.stdout:5/422: fsync d4/d11/d3d/d4d/f5d 0 2024-04-22T15:18:46.154 INFO:tasks.workunit.client.0.smithi096.stdout:0/550: mkdir d7/da7/dba 0 2024-04-22T15:18:46.155 INFO:tasks.workunit.client.0.smithi096.stdout:0/551: fsync d7/de/d5b/d95/dc/d26/fb5 0 2024-04-22T15:18:46.157 INFO:tasks.workunit.client.0.smithi096.stdout:1/593: creat d6/dc/d8a/d3f/d54/d32/d3b/d56/fb9 x:0 0 0 2024-04-22T15:18:46.157 INFO:tasks.workunit.client.0.smithi096.stdout:6/378: write d3/d28/f21 [1000295,25186] 0 2024-04-22T15:18:46.157 INFO:tasks.workunit.client.0.smithi096.stdout:0/552: read d7/de/d5b/d95/dc/d16/f5d [1397164,71428] 0 2024-04-22T15:18:46.159 INFO:tasks.workunit.client.0.smithi096.stdout:3/321: dread da/db/de/f40 [0,4194304] 0 2024-04-22T15:18:46.159 INFO:tasks.workunit.client.0.smithi096.stdout:0/553: readlink d7/de/d5b/d95/dc/d26/l5f 0 2024-04-22T15:18:46.159 INFO:tasks.workunit.client.0.smithi096.stdout:4/409: symlink d5/l87 0 2024-04-22T15:18:46.162 INFO:tasks.workunit.client.0.smithi096.stdout:9/340: mkdir d1/d4/d9/d43/d57/d5b/d73 0 2024-04-22T15:18:46.164 INFO:tasks.workunit.client.0.smithi096.stdout:5/423: rmdir d4/d2c/d2f/d31 39 2024-04-22T15:18:46.167 INFO:tasks.workunit.client.0.smithi096.stdout:7/630: creat d31/d32/d33/d6a/d83/dba/dc7/fcd x:0 0 0 2024-04-22T15:18:46.168 INFO:tasks.workunit.client.0.smithi096.stdout:7/631: fsync d31/d32/d33/f34 0 2024-04-22T15:18:46.168 INFO:tasks.workunit.client.0.smithi096.stdout:7/632: dread - d31/d32/d36/d75/fc3 zero size 2024-04-22T15:18:46.173 INFO:tasks.workunit.client.0.smithi096.stdout:0/554: mkdir d7/de/d3a/d61/dbb 0 2024-04-22T15:18:46.175 INFO:tasks.workunit.client.0.smithi096.stdout:4/410: mkdir d5/d7/d2c/d3d/d4f/d88 0 2024-04-22T15:18:46.176 INFO:tasks.workunit.client.0.smithi096.stdout:3/322: truncate da/db/de/f40 490296 0 2024-04-22T15:18:46.182 INFO:tasks.workunit.client.0.smithi096.stdout:6/379: mkdir d3/d81 0 2024-04-22T15:18:46.183 INFO:tasks.workunit.client.0.smithi096.stdout:3/323: dwrite da/db/de/d19/d2d/f5c [0,4194304] 0 2024-04-22T15:18:46.185 INFO:tasks.workunit.client.0.smithi096.stdout:7/633: unlink d31/d32/c3a 0 2024-04-22T15:18:46.186 INFO:tasks.workunit.client.0.smithi096.stdout:6/380: read d3/d28/dd/d11/d2c/f39 [1779616,51272] 0 2024-04-22T15:18:46.186 INFO:tasks.workunit.client.0.smithi096.stdout:5/424: dread d4/d2c/d2f/d31/f3b [0,4194304] 0 2024-04-22T15:18:46.187 INFO:tasks.workunit.client.0.smithi096.stdout:6/381: write d3/d28/dd/d11/f18 [3558168,35846] 0 2024-04-22T15:18:46.189 INFO:tasks.workunit.client.0.smithi096.stdout:0/555: creat d7/de/d5b/d95/dc/d26/fbc x:0 0 0 2024-04-22T15:18:46.190 INFO:tasks.workunit.client.0.smithi096.stdout:0/556: readlink d7/de/d5b/d95/d15/d35/l58 0 2024-04-22T15:18:46.193 INFO:tasks.workunit.client.0.smithi096.stdout:4/411: rename d5/d7/d1f/f55 to d5/d7/d2c/d3d/d4f/d63/f89 0 2024-04-22T15:18:46.194 INFO:tasks.workunit.client.0.smithi096.stdout:4/412: dread - d5/d7/d1a/d25/f44 zero size 2024-04-22T15:18:46.194 INFO:tasks.workunit.client.0.smithi096.stdout:4/413: fsync d5/d7/d2c/f65 0 2024-04-22T15:18:46.201 INFO:tasks.workunit.client.0.smithi096.stdout:3/324: creat da/db/d12/d22/f61 x:0 0 0 2024-04-22T15:18:46.203 INFO:tasks.workunit.client.0.smithi096.stdout:1/594: dwrite d6/dc/d8a/d3f/d54/d32/d8c/d4a/fa7 [0,4194304] 0 2024-04-22T15:18:46.205 INFO:tasks.workunit.client.0.smithi096.stdout:5/425: rmdir d4/d1f/d80 39 2024-04-22T15:18:46.211 INFO:tasks.workunit.client.0.smithi096.stdout:6/382: mkdir d3/d28/d4d/d82 0 2024-04-22T15:18:46.214 INFO:tasks.workunit.client.0.smithi096.stdout:4/414: rename d5/l22 to d5/d7/d2c/d67/l8a 0 2024-04-22T15:18:46.221 INFO:tasks.workunit.client.0.smithi096.stdout:4/415: dwrite d5/d7/d1a/d25/f44 [0,4194304] 0 2024-04-22T15:18:46.271 INFO:tasks.workunit.client.1.smithi196.stdout:2/55: readlink d1/le 0 2024-04-22T15:18:46.272 INFO:tasks.workunit.client.1.smithi196.stdout:2/56: write d1/f3 [2869625,90938] 0 2024-04-22T15:18:46.272 INFO:tasks.workunit.client.1.smithi196.stdout:2/57: fsync d1/f15 0 2024-04-22T15:18:46.274 INFO:tasks.workunit.client.1.smithi196.stdout:2/58: creat d1/f18 x:0 0 0 2024-04-22T15:18:46.277 INFO:tasks.workunit.client.1.smithi196.stdout:2/59: dread d1/fd [0,4194304] 0 2024-04-22T15:18:46.280 INFO:tasks.workunit.client.1.smithi196.stdout:2/60: rename d1/f11 to d1/d13/f19 0 2024-04-22T15:18:46.282 INFO:tasks.workunit.client.1.smithi196.stdout:2/61: symlink d1/l1a 0 2024-04-22T15:18:46.293 INFO:tasks.workunit.client.1.smithi196.stdout:8/52: write d2/f3 [2570855,121878] 0 2024-04-22T15:18:46.299 INFO:tasks.workunit.client.1.smithi196.stdout:8/53: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:18:46.299 INFO:tasks.workunit.client.1.smithi196.stdout:8/54: read f0 [8043814,87859] 0 2024-04-22T15:18:46.305 INFO:tasks.workunit.client.1.smithi196.stdout:8/55: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:18:46.310 INFO:tasks.workunit.client.0.smithi096.stdout:1/595: mkdir d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b/dba 0 2024-04-22T15:18:46.310 INFO:tasks.workunit.client.0.smithi096.stdout:5/426: mkdir d4/d11/d3d/d4d/d88 0 2024-04-22T15:18:46.310 INFO:tasks.workunit.client.0.smithi096.stdout:4/416: write d5/d7/d2c/d3d/d4f/f5c [1094041,100796] 0 2024-04-22T15:18:46.311 INFO:tasks.workunit.client.1.smithi196.stdout:8/56: dwrite f0 [0,4194304] 0 2024-04-22T15:18:46.313 INFO:tasks.workunit.client.0.smithi096.stdout:5/427: fdatasync d4/d11/f7a 0 2024-04-22T15:18:46.314 INFO:tasks.workunit.client.0.smithi096.stdout:3/325: mkdir da/db/d12/d22/d26/d52/d62 0 2024-04-22T15:18:46.315 INFO:tasks.workunit.client.0.smithi096.stdout:6/383: dread f2 [0,4194304] 0 2024-04-22T15:18:46.315 INFO:tasks.workunit.client.0.smithi096.stdout:1/596: chown d6/dc/d8a/d3f/d54/f18 63 1 2024-04-22T15:18:46.318 INFO:tasks.workunit.client.0.smithi096.stdout:9/341: dwrite d1/f38 [0,4194304] 0 2024-04-22T15:18:46.319 INFO:tasks.workunit.client.0.smithi096.stdout:7/634: getdents d31/d68 0 2024-04-22T15:18:46.321 INFO:tasks.workunit.client.0.smithi096.stdout:3/326: mkdir da/db/d12/d63 0 2024-04-22T15:18:46.322 INFO:tasks.workunit.client.0.smithi096.stdout:6/384: mkdir d3/d3a/d75/d83 0 2024-04-22T15:18:46.325 INFO:tasks.workunit.client.0.smithi096.stdout:5/428: fdatasync f1 0 2024-04-22T15:18:46.328 INFO:tasks.workunit.client.0.smithi096.stdout:4/417: mkdir d5/d7/d1a/d8b 0 2024-04-22T15:18:46.332 INFO:tasks.workunit.client.0.smithi096.stdout:4/418: write d5/f39 [5565270,77577] 0 2024-04-22T15:18:46.332 INFO:tasks.workunit.client.1.smithi196.stdout:8/57: truncate f0 2073543 0 2024-04-22T15:18:46.334 INFO:tasks.workunit.client.0.smithi096.stdout:7/635: dwrite d31/d32/d33/da2/fb9 [0,4194304] 0 2024-04-22T15:18:46.334 INFO:tasks.workunit.client.0.smithi096.stdout:9/342: dread d1/d4/d9/f24 [0,4194304] 0 2024-04-22T15:18:46.335 INFO:tasks.workunit.client.0.smithi096.stdout:1/597: creat d6/dc/d8a/d3f/d62/fbb x:0 0 0 2024-04-22T15:18:46.336 INFO:tasks.workunit.client.0.smithi096.stdout:5/429: creat d4/d11/d3d/d4d/d59/d5a/d5c/f89 x:0 0 0 2024-04-22T15:18:46.337 INFO:tasks.workunit.client.0.smithi096.stdout:5/430: dread - d4/d11/d3d/d4d/f5d zero size 2024-04-22T15:18:46.339 INFO:tasks.workunit.client.0.smithi096.stdout:6/385: mknod d3/d28/dd/d11/d2c/d6f/c84 0 2024-04-22T15:18:46.340 INFO:tasks.workunit.client.1.smithi196.stdout:8/58: creat d2/dc/fd x:0 0 0 2024-04-22T15:18:46.340 INFO:tasks.workunit.client.0.smithi096.stdout:4/419: symlink d5/d7/d2c/d43/d56/l8c 0 2024-04-22T15:18:46.341 INFO:tasks.workunit.client.0.smithi096.stdout:9/343: chown d1/c3e 48330139 1 2024-04-22T15:18:46.343 INFO:tasks.workunit.client.0.smithi096.stdout:3/327: rename da/db/de/d19/d3e/f15 to da/db/de/d19/d2d/f64 0 2024-04-22T15:18:46.345 INFO:tasks.workunit.client.0.smithi096.stdout:3/328: read da/db/de/d19/d3e/f4f [4118980,48769] 0 2024-04-22T15:18:46.345 INFO:tasks.workunit.client.0.smithi096.stdout:5/431: mkdir d4/d11/d3d/d4d/d59/d8a 0 2024-04-22T15:18:46.348 INFO:tasks.workunit.client.0.smithi096.stdout:7/636: rename d31/c82 to d31/d32/d36/cce 0 2024-04-22T15:18:46.349 INFO:tasks.workunit.client.1.smithi196.stdout:8/59: getdents d2/d5 0 2024-04-22T15:18:46.350 INFO:tasks.workunit.client.1.smithi196.stdout:8/60: dread - d2/dc/fd zero size 2024-04-22T15:18:46.350 INFO:tasks.workunit.client.1.smithi196.stdout:8/61: dread - d2/dc/fd zero size 2024-04-22T15:18:46.353 INFO:tasks.workunit.client.0.smithi096.stdout:4/420: mknod d5/d7/d2c/d3d/c8d 0 2024-04-22T15:18:46.354 INFO:tasks.workunit.client.1.smithi196.stdout:8/62: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:18:46.355 INFO:tasks.workunit.client.0.smithi096.stdout:7/637: chown d31/d32/d33/da2/laf 490941366 1 2024-04-22T15:18:46.357 INFO:tasks.workunit.client.0.smithi096.stdout:3/329: creat da/db/d12/d63/f65 x:0 0 0 2024-04-22T15:18:46.359 INFO:tasks.workunit.client.0.smithi096.stdout:5/432: mkdir d4/d1f/d8b 0 2024-04-22T15:18:46.360 INFO:tasks.workunit.client.1.smithi196.stdout:8/63: dwrite d2/dc/fd [0,4194304] 0 2024-04-22T15:18:46.363 INFO:tasks.workunit.client.0.smithi096.stdout:4/421: dwrite d5/f39 [4194304,4194304] 0 2024-04-22T15:18:46.367 INFO:tasks.workunit.client.0.smithi096.stdout:7/638: dwrite d31/d32/f60 [0,4194304] 0 2024-04-22T15:18:46.370 INFO:tasks.workunit.client.0.smithi096.stdout:1/598: mkdir d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc 0 2024-04-22T15:18:46.370 INFO:tasks.workunit.client.0.smithi096.stdout:7/639: stat d31/d32/d33/d6a/d83/dba/dc7 0 2024-04-22T15:18:46.379 INFO:tasks.workunit.client.0.smithi096.stdout:5/433: dwrite d4/d11/d3d/d4d/d59/d5a/d5c/f89 [0,4194304] 0 2024-04-22T15:18:46.382 INFO:tasks.workunit.client.0.smithi096.stdout:3/330: rename da/db/d12/d22/l2c to da/db/de/d19/d2d/l66 0 2024-04-22T15:18:46.385 INFO:tasks.workunit.client.0.smithi096.stdout:7/640: mkdir d31/d32/d7c/dcf 0 2024-04-22T15:18:46.388 INFO:tasks.workunit.client.0.smithi096.stdout:3/331: mkdir da/db/d12/d67 0 2024-04-22T15:18:46.389 INFO:tasks.workunit.client.0.smithi096.stdout:5/434: dread d4/d11/d3d/d4d/d59/d5a/f79 [0,4194304] 0 2024-04-22T15:18:46.389 INFO:tasks.workunit.client.0.smithi096.stdout:4/422: creat d5/d7/f8e x:0 0 0 2024-04-22T15:18:46.391 INFO:tasks.workunit.client.0.smithi096.stdout:7/641: rename d31/d32/d33/d5e/lc5 to d31/d32/d7c/ld0 0 2024-04-22T15:18:46.393 INFO:tasks.workunit.client.0.smithi096.stdout:3/332: mkdir da/db/d12/d22/d68 0 2024-04-22T15:18:46.393 INFO:tasks.workunit.client.0.smithi096.stdout:6/386: dwrite d3/d28/dd/d11/d2c/f56 [0,4194304] 0 2024-04-22T15:18:46.400 INFO:tasks.workunit.client.0.smithi096.stdout:3/333: mkdir da/db/dc/d28/d69 0 2024-04-22T15:18:46.401 INFO:tasks.workunit.client.0.smithi096.stdout:3/334: chown da/db/d10/f33 0 1 2024-04-22T15:18:46.402 INFO:tasks.workunit.client.0.smithi096.stdout:7/642: dread d31/d32/d33/f57 [0,4194304] 0 2024-04-22T15:18:46.402 INFO:tasks.workunit.client.0.smithi096.stdout:5/435: mknod d4/d11/d3d/d4d/d88/c8c 0 2024-04-22T15:18:46.403 INFO:tasks.workunit.client.0.smithi096.stdout:3/335: write da/db/d10/f33 [3932506,125200] 0 2024-04-22T15:18:46.404 INFO:tasks.workunit.client.0.smithi096.stdout:7/643: dread - d31/d32/d33/d6a/d83/fc8 zero size 2024-04-22T15:18:46.405 INFO:tasks.workunit.client.0.smithi096.stdout:3/336: fsync da/db/d12/d22/f3a 0 2024-04-22T15:18:46.406 INFO:tasks.workunit.client.0.smithi096.stdout:1/599: dread d6/dc/d8a/d3f/d54/f98 [0,4194304] 0 2024-04-22T15:18:46.406 INFO:tasks.workunit.client.0.smithi096.stdout:5/436: creat d4/d2c/d2f/d30/d5b/f8d x:0 0 0 2024-04-22T15:18:46.408 INFO:tasks.workunit.client.0.smithi096.stdout:7/644: dread d31/f8f [0,4194304] 0 2024-04-22T15:18:46.416 INFO:tasks.workunit.client.0.smithi096.stdout:5/437: rename d4/d2c/d2f/d6b/c47 to d4/d11/d3d/d4d/d59/c8e 0 2024-04-22T15:18:46.417 INFO:tasks.workunit.client.0.smithi096.stdout:1/600: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d56/f44 [0,4194304] 0 2024-04-22T15:18:46.417 INFO:tasks.workunit.client.0.smithi096.stdout:3/337: mkdir da/db/d6a 0 2024-04-22T15:18:46.421 INFO:tasks.workunit.client.0.smithi096.stdout:7/645: creat d31/d68/d98/fd1 x:0 0 0 2024-04-22T15:18:46.423 INFO:tasks.workunit.client.0.smithi096.stdout:5/438: creat d4/d2c/d2f/d31/f8f x:0 0 0 2024-04-22T15:18:46.423 INFO:tasks.workunit.client.0.smithi096.stdout:4/423: dwrite d5/f5e [0,4194304] 0 2024-04-22T15:18:46.426 INFO:tasks.workunit.client.0.smithi096.stdout:7/646: mknod d31/d32/d33/d6a/d83/cd2 0 2024-04-22T15:18:46.426 INFO:tasks.workunit.client.0.smithi096.stdout:1/601: mkdir d6/dc/dbd 0 2024-04-22T15:18:46.429 INFO:tasks.workunit.client.0.smithi096.stdout:7/647: creat d31/d32/dad/fd3 x:0 0 0 2024-04-22T15:18:46.429 INFO:tasks.workunit.client.0.smithi096.stdout:5/439: write d4/d2c/d2f/d31/f3b [2104482,2685] 0 2024-04-22T15:18:46.430 INFO:tasks.workunit.client.0.smithi096.stdout:1/602: dread - d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e/fa6 zero size 2024-04-22T15:18:46.434 INFO:tasks.workunit.client.0.smithi096.stdout:7/648: chown d31/d32/d33/d44/da8/dbf/cc2 7882 1 2024-04-22T15:18:46.435 INFO:tasks.workunit.client.0.smithi096.stdout:1/603: chown d6/dc/d8a/l28 425065704 1 2024-04-22T15:18:46.435 INFO:tasks.workunit.client.0.smithi096.stdout:4/424: dwrite d5/d7/d2c/f86 [0,4194304] 0 2024-04-22T15:18:46.437 INFO:tasks.workunit.client.0.smithi096.stdout:7/649: chown d31/d32/d33/d6a/d83/fc8 0 1 2024-04-22T15:18:46.437 INFO:tasks.workunit.client.0.smithi096.stdout:4/425: readlink d5/d7/d1f/l5f 0 2024-04-22T15:18:46.439 INFO:tasks.workunit.client.0.smithi096.stdout:1/604: symlink d6/dc/d8a/d3f/d62/d7d/lbe 0 2024-04-22T15:18:46.441 INFO:tasks.workunit.client.0.smithi096.stdout:4/426: mknod d5/c8f 0 2024-04-22T15:18:46.443 INFO:tasks.workunit.client.0.smithi096.stdout:5/440: dwrite d4/d2c/d2f/d30/d5b/f8d [0,4194304] 0 2024-04-22T15:18:46.452 INFO:tasks.workunit.client.0.smithi096.stdout:5/441: creat d4/d11/d25/f90 x:0 0 0 2024-04-22T15:18:46.453 INFO:tasks.workunit.client.0.smithi096.stdout:4/427: dwrite d5/d7/f7b [0,4194304] 0 2024-04-22T15:18:46.454 INFO:tasks.workunit.client.0.smithi096.stdout:3/338: dread da/db/de/d19/d2d/f64 [0,4194304] 0 2024-04-22T15:18:46.455 INFO:tasks.workunit.client.0.smithi096.stdout:3/339: readlink da/db/dc/l50 0 2024-04-22T15:18:46.457 INFO:tasks.workunit.client.0.smithi096.stdout:5/442: mknod d4/d11/d3d/d35/d57/c91 0 2024-04-22T15:18:46.458 INFO:tasks.workunit.client.0.smithi096.stdout:3/340: chown da/db/de/d19/d2d/f64 21681 1 2024-04-22T15:18:46.458 INFO:tasks.workunit.client.0.smithi096.stdout:3/341: stat da/db/d12 0 2024-04-22T15:18:46.459 INFO:tasks.workunit.client.0.smithi096.stdout:4/428: creat d5/d7/d1a/f90 x:0 0 0 2024-04-22T15:18:46.462 INFO:tasks.workunit.client.0.smithi096.stdout:3/342: symlink da/db/l6b 0 2024-04-22T15:18:46.464 INFO:tasks.workunit.client.0.smithi096.stdout:4/429: creat d5/d7/d2c/d3d/d4f/f91 x:0 0 0 2024-04-22T15:18:46.465 INFO:tasks.workunit.client.0.smithi096.stdout:5/443: dwrite d4/d2c/d2f/d6b/f50 [0,4194304] 0 2024-04-22T15:18:46.466 INFO:tasks.workunit.client.0.smithi096.stdout:1/605: write d6/dc/d8a/d3f/f73 [400659,103546] 0 2024-04-22T15:18:46.467 INFO:tasks.workunit.client.0.smithi096.stdout:4/430: read d5/d7/d1f/f62 [139410,42474] 0 2024-04-22T15:18:46.472 INFO:tasks.workunit.client.0.smithi096.stdout:5/444: creat d4/d11/d3d/d35/d57/f92 x:0 0 0 2024-04-22T15:18:46.475 INFO:tasks.workunit.client.0.smithi096.stdout:3/343: dwrite da/f34 [4194304,4194304] 0 2024-04-22T15:18:46.475 INFO:tasks.workunit.client.0.smithi096.stdout:4/431: truncate d5/d7/f47 1486245 0 2024-04-22T15:18:46.476 INFO:tasks.workunit.client.0.smithi096.stdout:7/650: dwrite d31/d32/d36/f3c [0,4194304] 0 2024-04-22T15:18:46.478 INFO:tasks.workunit.client.0.smithi096.stdout:3/344: write da/db/f3d [627595,63298] 0 2024-04-22T15:18:46.478 INFO:tasks.workunit.client.0.smithi096.stdout:1/606: mknod d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/cbf 0 2024-04-22T15:18:46.482 INFO:tasks.workunit.client.0.smithi096.stdout:4/432: symlink d5/d7/d2c/l92 0 2024-04-22T15:18:46.483 INFO:tasks.workunit.client.0.smithi096.stdout:3/345: getdents da/db/d12/d67 0 2024-04-22T15:18:46.484 INFO:tasks.workunit.client.0.smithi096.stdout:5/445: dread d4/d2c/d2f/d30/d5b/f8d [0,4194304] 0 2024-04-22T15:18:46.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:46 smithi096 ceph-mon[26688]: pgmap v29: 65 pgs: 65 active+clean; 1017 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 25 MiB/s rd, 140 MiB/s wr, 410 op/s 2024-04-22T15:18:46.486 INFO:tasks.workunit.client.0.smithi096.stdout:1/607: link d6/dc/d8a/d3f/d62/f8d d6/dc/d8a/d3f/d54/d32/d82/fc0 0 2024-04-22T15:18:46.488 INFO:tasks.workunit.client.0.smithi096.stdout:3/346: read da/db/d12/d22/f31 [2947612,116848] 0 2024-04-22T15:18:46.489 INFO:tasks.workunit.client.0.smithi096.stdout:4/433: creat d5/d7/d1f/d84/f93 x:0 0 0 2024-04-22T15:18:46.491 INFO:tasks.workunit.client.0.smithi096.stdout:5/446: rmdir d4/d2c/d2f/d31/d82 39 2024-04-22T15:18:46.496 INFO:tasks.workunit.client.0.smithi096.stdout:3/347: creat da/db/d12/d22/d26/d52/f6c x:0 0 0 2024-04-22T15:18:46.498 INFO:tasks.workunit.client.0.smithi096.stdout:5/447: mkdir d4/d11/d3d/d4d/d93 0 2024-04-22T15:18:46.499 INFO:tasks.workunit.client.0.smithi096.stdout:1/608: rmdir d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a 39 2024-04-22T15:18:46.500 INFO:tasks.workunit.client.0.smithi096.stdout:4/434: link d5/d7/f41 d5/d7/d2c/d43/d56/f94 0 2024-04-22T15:18:46.501 INFO:tasks.workunit.client.0.smithi096.stdout:3/348: fdatasync da/db/dc/f49 0 2024-04-22T15:18:46.501 INFO:tasks.workunit.client.0.smithi096.stdout:7/651: write d31/d32/d33/d44/f80 [3532420,66689] 0 2024-04-22T15:18:46.502 INFO:tasks.workunit.client.0.smithi096.stdout:7/652: chown d31/d32/d36/l90 16276369 1 2024-04-22T15:18:46.507 INFO:tasks.workunit.client.0.smithi096.stdout:5/448: fsync d4/d11/f22 0 2024-04-22T15:18:46.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:46 smithi196 ceph-mon[34717]: pgmap v29: 65 pgs: 65 active+clean; 1017 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 25 MiB/s rd, 140 MiB/s wr, 410 op/s 2024-04-22T15:18:46.509 INFO:tasks.workunit.client.0.smithi096.stdout:4/435: mkdir d5/d7/d2c/d95 0 2024-04-22T15:18:46.512 INFO:tasks.workunit.client.0.smithi096.stdout:3/349: truncate da/db/de/d19/f30 7570537 0 2024-04-22T15:18:46.513 INFO:tasks.workunit.client.0.smithi096.stdout:3/350: dread - da/db/d12/d22/f60 zero size 2024-04-22T15:18:46.515 INFO:tasks.workunit.client.0.smithi096.stdout:5/449: dread f1 [0,4194304] 0 2024-04-22T15:18:46.515 INFO:tasks.workunit.client.0.smithi096.stdout:4/436: creat d5/d7/d2c/d43/d56/f96 x:0 0 0 2024-04-22T15:18:46.517 INFO:tasks.workunit.client.0.smithi096.stdout:4/437: chown d5/d7/d2c/d43/c4b 28956 1 2024-04-22T15:18:46.522 INFO:tasks.workunit.client.0.smithi096.stdout:1/609: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d56/f7f [0,4194304] 0 2024-04-22T15:18:46.522 INFO:tasks.workunit.client.0.smithi096.stdout:4/438: write d5/fc [7270145,127280] 0 2024-04-22T15:18:46.524 INFO:tasks.workunit.client.0.smithi096.stdout:7/653: truncate d31/d32/d33/f58 2802697 0 2024-04-22T15:18:46.525 INFO:tasks.workunit.client.0.smithi096.stdout:1/610: dread d6/f9d [0,4194304] 0 2024-04-22T15:18:46.527 INFO:tasks.workunit.client.0.smithi096.stdout:3/351: write da/db/de/d19/d3e/f4f [5005080,124568] 0 2024-04-22T15:18:46.528 INFO:tasks.workunit.client.0.smithi096.stdout:4/439: chown d5/d7/d2c/d3d/d4f/d63/l83 1170641 1 2024-04-22T15:18:46.529 INFO:tasks.workunit.client.0.smithi096.stdout:7/654: symlink d31/ld4 0 2024-04-22T15:18:46.531 INFO:tasks.workunit.client.0.smithi096.stdout:3/352: readlink da/db/d12/d22/d26/l47 0 2024-04-22T15:18:46.531 INFO:tasks.workunit.client.0.smithi096.stdout:7/655: readlink d31/d32/d36/d75/lb3 0 2024-04-22T15:18:46.534 INFO:tasks.workunit.client.0.smithi096.stdout:4/440: creat d5/d7/d1a/f97 x:0 0 0 2024-04-22T15:18:46.535 INFO:tasks.workunit.client.0.smithi096.stdout:7/656: mkdir d31/d32/d33/d93/dd5 0 2024-04-22T15:18:46.535 INFO:tasks.workunit.client.0.smithi096.stdout:5/450: dwrite d4/d2c/f4e [0,4194304] 0 2024-04-22T15:18:46.539 INFO:tasks.workunit.client.0.smithi096.stdout:5/451: dread f1 [0,4194304] 0 2024-04-22T15:18:46.540 INFO:tasks.workunit.client.0.smithi096.stdout:4/441: mkdir d5/d7/d1f/d84/d6d/d98 0 2024-04-22T15:18:46.541 INFO:tasks.workunit.client.0.smithi096.stdout:4/442: chown d5/d7/d2c/d43/d56/f96 7 1 2024-04-22T15:18:46.542 INFO:tasks.workunit.client.0.smithi096.stdout:4/443: dread - d5/d7/d1f/d84/f93 zero size 2024-04-22T15:18:46.543 INFO:tasks.workunit.client.0.smithi096.stdout:7/657: getdents d31/d32/d33/d5e/dc9 0 2024-04-22T15:18:46.544 INFO:tasks.workunit.client.0.smithi096.stdout:4/444: dread - d5/d7/d2c/d43/d56/f59 zero size 2024-04-22T15:18:46.548 INFO:tasks.workunit.client.0.smithi096.stdout:7/658: mknod d31/d32/d33/d6a/d83/cd6 0 2024-04-22T15:18:46.548 INFO:tasks.workunit.client.0.smithi096.stdout:5/452: write d4/d2c/d2f/f3a [1547665,79521] 0 2024-04-22T15:18:46.550 INFO:tasks.workunit.client.0.smithi096.stdout:1/611: dread d6/dc/d8a/d29/f2f [0,4194304] 0 2024-04-22T15:18:46.551 INFO:tasks.workunit.client.0.smithi096.stdout:4/445: symlink d5/d7/d2c/l99 0 2024-04-22T15:18:46.554 INFO:tasks.workunit.client.0.smithi096.stdout:4/446: dread d5/d7/d1f/d84/f64 [0,4194304] 0 2024-04-22T15:18:46.554 INFO:tasks.workunit.client.0.smithi096.stdout:5/453: mknod d4/d11/d3d/d4d/d59/d8a/c94 0 2024-04-22T15:18:46.556 INFO:tasks.workunit.client.0.smithi096.stdout:5/454: write d4/d11/d3d/d35/d57/f92 [114711,26676] 0 2024-04-22T15:18:46.559 INFO:tasks.workunit.client.0.smithi096.stdout:7/659: link d31/l61 d31/d32/ld7 0 2024-04-22T15:18:46.561 INFO:tasks.workunit.client.0.smithi096.stdout:5/455: write d4/d2c/d2f/d31/f3b [2853297,28312] 0 2024-04-22T15:18:46.561 INFO:tasks.workunit.client.0.smithi096.stdout:7/660: write d31/d32/f59 [1527352,128215] 0 2024-04-22T15:18:46.561 INFO:tasks.workunit.client.0.smithi096.stdout:1/612: dwrite d6/dc/d8a/d3f/d54/d32/d8c/d4a/f78 [0,4194304] 0 2024-04-22T15:18:46.563 INFO:tasks.workunit.client.0.smithi096.stdout:7/661: chown d31/d32/d36/l90 15446 1 2024-04-22T15:18:46.564 INFO:tasks.workunit.client.0.smithi096.stdout:1/613: readlink d6/dc/d8a/d3f/d54/la8 0 2024-04-22T15:18:46.565 INFO:tasks.workunit.client.0.smithi096.stdout:5/456: rmdir d4/d11/d3d/d4d/d59 39 2024-04-22T15:18:46.566 INFO:tasks.workunit.client.1.smithi196.stdout:3/0: sync 2024-04-22T15:18:46.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/1: rename - no filename 2024-04-22T15:18:46.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/2: chown . 11696 1 2024-04-22T15:18:46.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/3: dread - no filename 2024-04-22T15:18:46.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/4: truncate - no filename 2024-04-22T15:18:46.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/5: rename - no filename 2024-04-22T15:18:46.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/6: fsync - no filename 2024-04-22T15:18:46.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/7: readlink - no filename 2024-04-22T15:18:46.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/8: rmdir - no directory 2024-04-22T15:18:46.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/9: write - no filename 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:7/13: sync 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:9/15: sync 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:7/14: rmdir - no directory 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:1/12: sync 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:9/16: readlink - no filename 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:0/32: sync 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:2/62: sync 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:4/32: sync 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:6/28: sync 2024-04-22T15:18:46.568 INFO:tasks.workunit.client.1.smithi196.stdout:8/64: sync 2024-04-22T15:18:46.569 INFO:tasks.workunit.client.1.smithi196.stdout:5/13: sync 2024-04-22T15:18:46.569 INFO:tasks.workunit.client.1.smithi196.stdout:7/15: chown f2 5 1 2024-04-22T15:18:46.569 INFO:tasks.workunit.client.1.smithi196.stdout:9/17: write f0 [1321555,104038] 0 2024-04-22T15:18:46.570 INFO:tasks.workunit.client.1.smithi196.stdout:0/33: write d3/fa [96595,90503] 0 2024-04-22T15:18:46.570 INFO:tasks.workunit.client.0.smithi096.stdout:1/614: creat d6/dc/d8a/d3f/d62/d7d/dab/fc1 x:0 0 0 2024-04-22T15:18:46.571 INFO:tasks.workunit.client.1.smithi196.stdout:0/34: truncate d3/fa 1152550 0 2024-04-22T15:18:46.573 INFO:tasks.workunit.client.0.smithi096.stdout:5/457: mknod d4/d1f/d80/c95 0 2024-04-22T15:18:46.574 INFO:tasks.workunit.client.1.smithi196.stdout:6/29: fsync f0 0 2024-04-22T15:18:46.574 INFO:tasks.workunit.client.1.smithi196.stdout:6/30: stat f2 0 2024-04-22T15:18:46.574 INFO:tasks.workunit.client.0.smithi096.stdout:7/662: dwrite d31/d32/d36/d75/fc3 [0,4194304] 0 2024-04-22T15:18:46.575 INFO:tasks.workunit.client.1.smithi196.stdout:2/63: creat d1/f1b x:0 0 0 2024-04-22T15:18:46.576 INFO:tasks.workunit.client.1.smithi196.stdout:2/64: rename d1/d13 to d1/d13/d1c 22 2024-04-22T15:18:46.577 INFO:tasks.workunit.client.1.smithi196.stdout:2/65: write d1/f18 [1028770,10888] 0 2024-04-22T15:18:46.577 INFO:tasks.workunit.client.1.smithi196.stdout:3/10: mkdir d0 0 2024-04-22T15:18:46.577 INFO:tasks.workunit.client.1.smithi196.stdout:3/11: link - no file 2024-04-22T15:18:46.577 INFO:tasks.workunit.client.1.smithi196.stdout:3/12: dread - no filename 2024-04-22T15:18:46.577 INFO:tasks.workunit.client.1.smithi196.stdout:3/13: readlink - no filename 2024-04-22T15:18:46.577 INFO:tasks.workunit.client.1.smithi196.stdout:3/14: dread - no filename 2024-04-22T15:18:46.578 INFO:tasks.workunit.client.1.smithi196.stdout:1/13: creat d1/f7 x:0 0 0 2024-04-22T15:18:46.579 INFO:tasks.workunit.client.0.smithi096.stdout:4/447: dwrite f1 [0,4194304] 0 2024-04-22T15:18:46.579 INFO:tasks.workunit.client.1.smithi196.stdout:1/14: write d1/f7 [900477,123350] 0 2024-04-22T15:18:46.580 INFO:tasks.workunit.client.0.smithi096.stdout:4/448: chown d5/d7/d2c/d3d/f49 12502 1 2024-04-22T15:18:46.582 INFO:tasks.workunit.client.1.smithi196.stdout:7/16: dread f2 [0,4194304] 0 2024-04-22T15:18:46.582 INFO:tasks.workunit.client.0.smithi096.stdout:4/449: write d5/d7/f7f [4611325,124311] 0 2024-04-22T15:18:46.585 INFO:tasks.workunit.client.0.smithi096.stdout:7/663: mknod d31/d32/d33/d44/cd8 0 2024-04-22T15:18:46.586 INFO:tasks.workunit.client.0.smithi096.stdout:7/664: stat d31/d32/d33/d6a/d83/fa6 0 2024-04-22T15:18:46.587 INFO:tasks.workunit.client.0.smithi096.stdout:4/450: dread d5/d7/d1a/d25/f44 [0,4194304] 0 2024-04-22T15:18:46.588 INFO:tasks.workunit.client.1.smithi196.stdout:0/35: creat d3/ff x:0 0 0 2024-04-22T15:18:46.589 INFO:tasks.workunit.client.0.smithi096.stdout:4/451: truncate d5/d7/d2c/d43/f6e 1059929 0 2024-04-22T15:18:46.591 INFO:tasks.workunit.client.0.smithi096.stdout:4/452: chown d5/d7/d1f/d84/d6d 19192 1 2024-04-22T15:18:46.592 INFO:tasks.workunit.client.0.smithi096.stdout:7/665: creat d31/d32/d36/d5d/fd9 x:0 0 0 2024-04-22T15:18:46.594 INFO:tasks.workunit.client.1.smithi196.stdout:2/66: mkdir d1/d1d 0 2024-04-22T15:18:46.595 INFO:tasks.workunit.client.0.smithi096.stdout:7/666: write d31/d32/f55 [8603300,94042] 0 2024-04-22T15:18:46.595 INFO:tasks.workunit.client.0.smithi096.stdout:7/667: fdatasync d31/d32/f47 0 2024-04-22T15:18:46.596 INFO:tasks.workunit.client.1.smithi196.stdout:1/15: creat d1/f8 x:0 0 0 2024-04-22T15:18:46.600 INFO:tasks.workunit.client.0.smithi096.stdout:4/453: rmdir d5/d7/d2c 39 2024-04-22T15:18:46.601 INFO:tasks.workunit.client.0.smithi096.stdout:1/615: dread f2 [0,4194304] 0 2024-04-22T15:18:46.602 INFO:tasks.workunit.client.1.smithi196.stdout:4/33: symlink d6/da/dc/ld 0 2024-04-22T15:18:46.605 INFO:tasks.workunit.client.1.smithi196.stdout:6/31: write f4 [4697497,35945] 0 2024-04-22T15:18:46.606 INFO:tasks.workunit.client.0.smithi096.stdout:7/668: chown d31/d32/d7c/ld0 0 1 2024-04-22T15:18:46.607 INFO:tasks.workunit.client.0.smithi096.stdout:5/458: dread d4/d11/d3d/d4d/d59/d5a/f67 [0,4194304] 0 2024-04-22T15:18:46.608 INFO:tasks.workunit.client.1.smithi196.stdout:9/18: rename f0 to f2 0 2024-04-22T15:18:46.609 INFO:tasks.workunit.client.1.smithi196.stdout:6/32: dread f4 [0,4194304] 0 2024-04-22T15:18:46.610 INFO:tasks.workunit.client.0.smithi096.stdout:4/454: write d5/d7/d2c/d3d/d4f/d63/f7e [69542,32162] 0 2024-04-22T15:18:46.612 INFO:tasks.workunit.client.1.smithi196.stdout:0/36: mknod d3/d6/c10 0 2024-04-22T15:18:46.613 INFO:tasks.workunit.client.1.smithi196.stdout:0/37: stat d3/d9/cb 0 2024-04-22T15:18:46.613 INFO:tasks.workunit.client.0.smithi096.stdout:7/669: symlink d31/d32/d33/d5e/lda 0 2024-04-22T15:18:46.614 INFO:tasks.workunit.client.1.smithi196.stdout:3/15: creat d0/f1 x:0 0 0 2024-04-22T15:18:46.615 INFO:tasks.workunit.client.1.smithi196.stdout:3/16: readlink - no filename 2024-04-22T15:18:46.616 INFO:tasks.workunit.client.0.smithi096.stdout:1/616: mkdir d6/dc/d8a/d3f/d62/dc2 0 2024-04-22T15:18:46.617 INFO:tasks.workunit.client.1.smithi196.stdout:1/16: unlink d1/l3 0 2024-04-22T15:18:46.618 INFO:tasks.workunit.client.1.smithi196.stdout:4/34: creat d6/da/dc/fe x:0 0 0 2024-04-22T15:18:46.621 INFO:tasks.workunit.client.1.smithi196.stdout:9/19: write f2 [864278,15996] 0 2024-04-22T15:18:46.623 INFO:tasks.workunit.client.1.smithi196.stdout:6/33: mknod c6 0 2024-04-22T15:18:46.623 INFO:tasks.workunit.client.1.smithi196.stdout:6/34: fdatasync f1 0 2024-04-22T15:18:46.625 INFO:tasks.workunit.client.0.smithi096.stdout:7/670: symlink d31/d32/d33/d54/ldb 0 2024-04-22T15:18:46.625 INFO:tasks.workunit.client.0.smithi096.stdout:7/671: chown d31/d32/d36/l8c 525693710 1 2024-04-22T15:18:46.626 INFO:tasks.workunit.client.1.smithi196.stdout:9/20: dwrite f2 [0,4194304] 0 2024-04-22T15:18:46.626 INFO:tasks.workunit.client.1.smithi196.stdout:3/17: creat d0/f2 x:0 0 0 2024-04-22T15:18:46.627 INFO:tasks.workunit.client.1.smithi196.stdout:3/18: dread - d0/f1 zero size 2024-04-22T15:18:46.627 INFO:tasks.workunit.client.0.smithi096.stdout:5/459: creat d4/d11/f96 x:0 0 0 2024-04-22T15:18:46.629 INFO:tasks.workunit.client.0.smithi096.stdout:5/460: truncate d4/d11/d3d/d4d/f6e 14624 0 2024-04-22T15:18:46.629 INFO:tasks.workunit.client.0.smithi096.stdout:5/461: dread - d4/d11/d3d/d4d/f5d zero size 2024-04-22T15:18:46.630 INFO:tasks.workunit.client.1.smithi196.stdout:2/67: link d1/f3 d1/d13/f1e 0 2024-04-22T15:18:46.632 INFO:tasks.workunit.client.1.smithi196.stdout:0/38: rmdir d3 39 2024-04-22T15:18:46.632 INFO:tasks.workunit.client.1.smithi196.stdout:2/68: write d1/fc [4290544,58108] 0 2024-04-22T15:18:46.632 INFO:tasks.workunit.client.1.smithi196.stdout:3/19: dwrite d0/f2 [0,4194304] 0 2024-04-22T15:18:46.634 INFO:tasks.workunit.client.1.smithi196.stdout:3/20: truncate d0/f1 667076 0 2024-04-22T15:18:46.635 INFO:tasks.workunit.client.0.smithi096.stdout:7/672: mkdir d31/d32/dc0/ddc 0 2024-04-22T15:18:46.640 INFO:tasks.workunit.client.1.smithi196.stdout:4/35: creat d6/ff x:0 0 0 2024-04-22T15:18:46.640 INFO:tasks.workunit.client.1.smithi196.stdout:1/17: rmdir d1 39 2024-04-22T15:18:46.641 INFO:tasks.workunit.client.1.smithi196.stdout:3/21: dwrite d0/f2 [0,4194304] 0 2024-04-22T15:18:46.642 INFO:tasks.workunit.client.0.smithi096.stdout:7/673: dwrite d31/d32/d33/d93/fae [0,4194304] 0 2024-04-22T15:18:46.646 INFO:tasks.workunit.client.0.smithi096.stdout:4/455: truncate d5/d7/f20 2124578 0 2024-04-22T15:18:46.648 INFO:tasks.workunit.client.0.smithi096.stdout:5/462: truncate d4/f3c 2419181 0 2024-04-22T15:18:46.652 INFO:tasks.workunit.client.0.smithi096.stdout:1/617: getdents d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e 0 2024-04-22T15:18:46.654 INFO:tasks.workunit.client.0.smithi096.stdout:7/674: truncate d31/d32/d36/d5d/f9c 527711 0 2024-04-22T15:18:46.657 INFO:tasks.workunit.client.1.smithi196.stdout:9/21: mknod c3 0 2024-04-22T15:18:46.661 INFO:tasks.workunit.client.0.smithi096.stdout:5/463: rename d4/d2c/d2f/f68 to d4/d11/d3d/d4d/d59/d8a/f97 0 2024-04-22T15:18:46.662 INFO:tasks.workunit.client.0.smithi096.stdout:5/464: chown f2 5502 1 2024-04-22T15:18:46.665 INFO:tasks.workunit.client.1.smithi196.stdout:2/69: mkdir d1/d13/d1f 0 2024-04-22T15:18:46.665 INFO:tasks.workunit.client.1.smithi196.stdout:9/22: dwrite f2 [0,4194304] 0 2024-04-22T15:18:46.667 INFO:tasks.workunit.client.0.smithi096.stdout:4/456: symlink d5/d7/d2c/d43/l9a 0 2024-04-22T15:18:46.668 INFO:tasks.workunit.client.1.smithi196.stdout:2/70: write d1/f5 [55218,51213] 0 2024-04-22T15:18:46.669 INFO:tasks.workunit.client.1.smithi196.stdout:5/14: sync 2024-04-22T15:18:46.672 INFO:tasks.workunit.client.1.smithi196.stdout:0/39: dwrite d3/ff [0,4194304] 0 2024-04-22T15:18:46.672 INFO:tasks.workunit.client.1.smithi196.stdout:7/17: getdents . 0 2024-04-22T15:18:46.672 INFO:tasks.workunit.client.0.smithi096.stdout:7/675: dread d31/d32/d33/d6a/f72 [0,4194304] 0 2024-04-22T15:18:46.675 INFO:tasks.workunit.client.0.smithi096.stdout:1/618: mkdir d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/dc3 0 2024-04-22T15:18:46.679 INFO:tasks.workunit.client.1.smithi196.stdout:3/22: fsync d0/f2 0 2024-04-22T15:18:46.679 INFO:tasks.workunit.client.1.smithi196.stdout:0/40: dwrite d3/f5 [0,4194304] 0 2024-04-22T15:18:46.680 INFO:tasks.workunit.client.0.smithi096.stdout:7/676: dwrite d31/d32/f59 [0,4194304] 0 2024-04-22T15:18:46.680 INFO:tasks.workunit.client.1.smithi196.stdout:3/23: chown d0/f2 753627994 1 2024-04-22T15:18:46.682 INFO:tasks.workunit.client.0.smithi096.stdout:5/465: rename d4/d2c/d2f/d30/d5b to d4/d11/d3d/d4d/d59/d98 0 2024-04-22T15:18:46.683 INFO:tasks.workunit.client.0.smithi096.stdout:5/466: chown d4/d11/f7a 1457490566 1 2024-04-22T15:18:46.690 INFO:tasks.workunit.client.1.smithi196.stdout:1/18: dread d1/f7 [0,4194304] 0 2024-04-22T15:18:46.691 INFO:tasks.workunit.client.0.smithi096.stdout:7/677: creat d31/d32/d33/d44/fdd x:0 0 0 2024-04-22T15:18:46.691 INFO:tasks.workunit.client.1.smithi196.stdout:9/23: creat f4 x:0 0 0 2024-04-22T15:18:46.693 INFO:tasks.workunit.client.0.smithi096.stdout:4/457: creat d5/d7/d1f/d84/d6d/d98/f9b x:0 0 0 2024-04-22T15:18:46.695 INFO:tasks.workunit.client.1.smithi196.stdout:4/36: dwrite d6/da/fb [4194304,4194304] 0 2024-04-22T15:18:46.698 INFO:tasks.workunit.client.1.smithi196.stdout:5/15: unlink d1/c3 0 2024-04-22T15:18:46.703 INFO:tasks.workunit.client.0.smithi096.stdout:8/390: sync 2024-04-22T15:18:46.704 INFO:tasks.workunit.client.0.smithi096.stdout:1/619: symlink d6/dc/d8a/lc4 0 2024-04-22T15:18:46.704 INFO:tasks.workunit.client.0.smithi096.stdout:8/391: write d4/d10/f19 [5748518,60807] 0 2024-04-22T15:18:46.710 INFO:tasks.workunit.client.1.smithi196.stdout:0/41: creat d3/f11 x:0 0 0 2024-04-22T15:18:46.712 INFO:tasks.workunit.client.0.smithi096.stdout:4/458: mkdir d5/d7/d2c/d3d/d4f/d9c 0 2024-04-22T15:18:46.713 INFO:tasks.workunit.client.0.smithi096.stdout:8/392: dwrite d4/d1c/d40/d46/d58/f60 [0,4194304] 0 2024-04-22T15:18:46.713 INFO:tasks.workunit.client.0.smithi096.stdout:4/459: chown d5/d7/d1f/f42 2589229 1 2024-04-22T15:18:46.713 INFO:tasks.workunit.client.1.smithi196.stdout:0/42: dread d3/f5 [0,4194304] 0 2024-04-22T15:18:46.714 INFO:tasks.workunit.client.1.smithi196.stdout:0/43: dread - d3/f11 zero size 2024-04-22T15:18:46.715 INFO:tasks.workunit.client.0.smithi096.stdout:5/467: mkdir d4/d11/d3d/d4d/d99 0 2024-04-22T15:18:46.716 INFO:tasks.workunit.client.1.smithi196.stdout:7/18: write f2 [966288,84316] 0 2024-04-22T15:18:46.719 INFO:tasks.workunit.client.1.smithi196.stdout:9/24: creat f5 x:0 0 0 2024-04-22T15:18:46.721 INFO:tasks.workunit.client.0.smithi096.stdout:8/393: dwrite d4/d1c/f72 [0,4194304] 0 2024-04-22T15:18:46.722 INFO:tasks.workunit.client.1.smithi196.stdout:8/65: sync 2024-04-22T15:18:46.722 INFO:tasks.workunit.client.1.smithi196.stdout:8/66: read f0 [934113,112094] 0 2024-04-22T15:18:46.726 INFO:tasks.workunit.client.0.smithi096.stdout:4/460: write d5/d7/d2c/d43/d56/f59 [174588,54633] 0 2024-04-22T15:18:46.728 INFO:tasks.workunit.client.1.smithi196.stdout:4/37: rmdir d6 39 2024-04-22T15:18:46.729 INFO:tasks.workunit.client.1.smithi196.stdout:7/19: dread f2 [0,4194304] 0 2024-04-22T15:18:46.730 INFO:tasks.workunit.client.0.smithi096.stdout:5/468: fdatasync d4/f1c 0 2024-04-22T15:18:46.730 INFO:tasks.workunit.client.1.smithi196.stdout:7/20: write f2 [774457,68024] 0 2024-04-22T15:18:46.733 INFO:tasks.workunit.client.0.smithi096.stdout:5/469: write d4/f1c [3863260,73274] 0 2024-04-22T15:18:46.734 INFO:tasks.workunit.client.0.smithi096.stdout:8/394: mknod d4/d10/d4d/c8b 0 2024-04-22T15:18:46.735 INFO:tasks.workunit.client.0.smithi096.stdout:7/678: dread f1c [0,4194304] 0 2024-04-22T15:18:46.736 INFO:tasks.workunit.client.0.smithi096.stdout:7/679: readlink d31/d32/d36/l7b 0 2024-04-22T15:18:46.737 INFO:tasks.workunit.client.1.smithi196.stdout:3/24: link d0/f1 d0/f3 0 2024-04-22T15:18:46.738 INFO:tasks.workunit.client.1.smithi196.stdout:0/44: mknod d3/d9/c12 0 2024-04-22T15:18:46.740 INFO:tasks.workunit.client.1.smithi196.stdout:6/35: getdents . 0 2024-04-22T15:18:46.742 INFO:tasks.workunit.client.1.smithi196.stdout:9/25: mknod c6 0 2024-04-22T15:18:46.743 INFO:tasks.workunit.client.1.smithi196.stdout:9/26: write f2 [768707,29715] 0 2024-04-22T15:18:46.743 INFO:tasks.workunit.client.0.smithi096.stdout:8/395: chown d4/d2a/d7e/l88 248 1 2024-04-22T15:18:46.744 INFO:tasks.workunit.client.1.smithi196.stdout:9/27: dread - f5 zero size 2024-04-22T15:18:46.744 INFO:tasks.workunit.client.0.smithi096.stdout:8/396: fsync d4/d10/d25/d4c/f87 0 2024-04-22T15:18:46.745 INFO:tasks.workunit.client.1.smithi196.stdout:1/19: rename d1/c2 to d1/c9 0 2024-04-22T15:18:46.745 INFO:tasks.workunit.client.1.smithi196.stdout:1/20: truncate d1/f8 67936 0 2024-04-22T15:18:46.746 INFO:tasks.workunit.client.0.smithi096.stdout:7/680: rmdir d31/d32/d33/d44/da8/dbf 39 2024-04-22T15:18:46.747 INFO:tasks.workunit.client.0.smithi096.stdout:7/681: read d31/d32/d36/f3c [1137180,37079] 0 2024-04-22T15:18:46.750 INFO:tasks.workunit.client.0.smithi096.stdout:1/620: rmdir d6/d97/d9a 0 2024-04-22T15:18:46.751 INFO:tasks.workunit.client.1.smithi196.stdout:7/21: mknod c3 0 2024-04-22T15:18:46.752 INFO:tasks.workunit.client.0.smithi096.stdout:8/397: fdatasync d4/d10/d25/d38/f50 0 2024-04-22T15:18:46.754 INFO:tasks.workunit.client.0.smithi096.stdout:8/398: write d4/d1c/d34/f3e [5073281,121738] 0 2024-04-22T15:18:46.754 INFO:tasks.workunit.client.0.smithi096.stdout:1/621: dread d6/dc/d8a/d3f/d54/f98 [0,4194304] 0 2024-04-22T15:18:46.756 INFO:tasks.workunit.client.0.smithi096.stdout:4/461: creat d5/d7/d2c/f9d x:0 0 0 2024-04-22T15:18:46.758 INFO:tasks.workunit.client.0.smithi096.stdout:1/622: write d6/dc/d8a/d3f/d54/d32/d3b/d94/fb3 [503134,85494] 0 2024-04-22T15:18:46.759 INFO:tasks.workunit.client.0.smithi096.stdout:1/623: write d6/dc/d8a/d3f/d54/d32/d3b/f89 [525079,67871] 0 2024-04-22T15:18:46.760 INFO:tasks.workunit.client.0.smithi096.stdout:5/470: creat d4/d2c/d2f/d31/f9a x:0 0 0 2024-04-22T15:18:46.764 INFO:tasks.workunit.client.1.smithi196.stdout:5/16: truncate f0 1874560 0 2024-04-22T15:18:46.765 INFO:tasks.workunit.client.1.smithi196.stdout:5/17: stat d1/d4 0 2024-04-22T15:18:46.767 INFO:tasks.workunit.client.1.smithi196.stdout:6/36: rename f0 to f7 0 2024-04-22T15:18:46.769 INFO:tasks.workunit.client.0.smithi096.stdout:8/399: creat d4/d1c/d40/d46/d58/f8c x:0 0 0 2024-04-22T15:18:46.771 INFO:tasks.workunit.client.1.smithi196.stdout:0/45: dwrite d3/f5 [0,4194304] 0 2024-04-22T15:18:46.772 INFO:tasks.workunit.client.1.smithi196.stdout:9/28: creat f7 x:0 0 0 2024-04-22T15:18:46.774 INFO:tasks.workunit.client.0.smithi096.stdout:1/624: chown d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fa1 3814263 1 2024-04-22T15:18:46.775 INFO:tasks.workunit.client.1.smithi196.stdout:2/71: getdents d1 0 2024-04-22T15:18:46.776 INFO:tasks.workunit.client.1.smithi196.stdout:9/29: write f1 [1307907,128720] 0 2024-04-22T15:18:46.776 INFO:tasks.workunit.client.1.smithi196.stdout:1/21: symlink d1/la 0 2024-04-22T15:18:46.778 INFO:tasks.workunit.client.1.smithi196.stdout:8/67: link d2/f3 d2/dc/fe 0 2024-04-22T15:18:46.778 INFO:tasks.workunit.client.1.smithi196.stdout:1/22: chown d1/f8 122720421 1 2024-04-22T15:18:46.779 INFO:tasks.workunit.client.1.smithi196.stdout:3/25: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:18:46.780 INFO:tasks.workunit.client.1.smithi196.stdout:1/23: write d1/f4 [544603,116106] 0 2024-04-22T15:18:46.780 INFO:tasks.workunit.client.1.smithi196.stdout:4/38: rename d6/da/dc to d6/d10 0 2024-04-22T15:18:46.781 INFO:tasks.workunit.client.0.smithi096.stdout:8/400: symlink d4/d10/d4d/l8d 0 2024-04-22T15:18:46.781 INFO:tasks.workunit.client.1.smithi196.stdout:3/26: chown d0/f2 2 1 2024-04-22T15:18:46.782 INFO:tasks.workunit.client.1.smithi196.stdout:4/39: stat d6/da 0 2024-04-22T15:18:46.782 INFO:tasks.workunit.client.0.smithi096.stdout:4/462: mknod d5/d7/d2c/d3d/d4f/d9c/c9e 0 2024-04-22T15:18:46.783 INFO:tasks.workunit.client.1.smithi196.stdout:0/46: dread d3/f5 [0,4194304] 0 2024-04-22T15:18:46.783 INFO:tasks.workunit.client.1.smithi196.stdout:4/40: write d6/da/fb [2464119,1073] 0 2024-04-22T15:18:46.784 INFO:tasks.workunit.client.1.smithi196.stdout:7/22: unlink c1 0 2024-04-22T15:18:46.784 INFO:tasks.workunit.client.0.smithi096.stdout:7/682: getdents d31/d68 0 2024-04-22T15:18:46.785 INFO:tasks.workunit.client.0.smithi096.stdout:8/401: symlink d4/d10/d25/d4c/l8e 0 2024-04-22T15:18:46.787 INFO:tasks.workunit.client.0.smithi096.stdout:4/463: symlink d5/d7/d2c/d3d/d4f/d9c/l9f 0 2024-04-22T15:18:46.788 INFO:tasks.workunit.client.1.smithi196.stdout:6/37: fdatasync f4 0 2024-04-22T15:18:46.789 INFO:tasks.workunit.client.0.smithi096.stdout:4/464: write d5/d7/d2c/d3d/d4f/d63/f7e [1136764,5152] 0 2024-04-22T15:18:46.789 INFO:tasks.workunit.client.0.smithi096.stdout:7/683: mknod d31/d32/dad/cde 0 2024-04-22T15:18:46.791 INFO:tasks.workunit.client.0.smithi096.stdout:4/465: dread - d5/d7/d2c/d3d/f75 zero size 2024-04-22T15:18:46.793 INFO:tasks.workunit.client.0.smithi096.stdout:7/684: truncate d31/d32/d33/d6a/d83/fa6 98804 0 2024-04-22T15:18:46.794 INFO:tasks.workunit.client.0.smithi096.stdout:7/685: fsync d31/d32/d33/d6a/d83/dba/dc7/fcd 0 2024-04-22T15:18:46.794 INFO:tasks.workunit.client.0.smithi096.stdout:4/466: creat d5/d7/d1f/d84/fa0 x:0 0 0 2024-04-22T15:18:46.795 INFO:tasks.workunit.client.1.smithi196.stdout:6/38: dwrite f1 [0,4194304] 0 2024-04-22T15:18:46.795 INFO:tasks.workunit.client.1.smithi196.stdout:6/39: stat c6 0 2024-04-22T15:18:46.797 INFO:tasks.workunit.client.0.smithi096.stdout:7/686: truncate d31/d32/d36/d75/f86 1701162 0 2024-04-22T15:18:46.797 INFO:tasks.workunit.client.0.smithi096.stdout:4/467: creat d5/fa1 x:0 0 0 2024-04-22T15:18:46.798 INFO:tasks.workunit.client.1.smithi196.stdout:8/68: write d2/dc/fe [4810553,47872] 0 2024-04-22T15:18:46.799 INFO:tasks.workunit.client.0.smithi096.stdout:7/687: dread d31/d68/f69 [0,4194304] 0 2024-04-22T15:18:46.799 INFO:tasks.workunit.client.1.smithi196.stdout:8/69: readlink d2/d5/lb 0 2024-04-22T15:18:46.800 INFO:tasks.workunit.client.1.smithi196.stdout:1/24: mkdir d1/db 0 2024-04-22T15:18:46.800 INFO:tasks.workunit.client.1.smithi196.stdout:8/70: write d2/fa [1783361,108748] 0 2024-04-22T15:18:46.800 INFO:tasks.workunit.client.0.smithi096.stdout:7/688: read d31/d32/f60 [2293317,77525] 0 2024-04-22T15:18:46.803 INFO:tasks.workunit.client.1.smithi196.stdout:3/27: creat d0/f4 x:0 0 0 2024-04-22T15:18:46.805 INFO:tasks.workunit.client.1.smithi196.stdout:8/71: dwrite d2/fa [0,4194304] 0 2024-04-22T15:18:46.806 INFO:tasks.workunit.client.1.smithi196.stdout:8/72: truncate d2/fa 4583144 0 2024-04-22T15:18:46.808 INFO:tasks.workunit.client.0.smithi096.stdout:7/689: dwrite d31/d68/f9f [0,4194304] 0 2024-04-22T15:18:46.810 INFO:tasks.workunit.client.1.smithi196.stdout:0/47: symlink d3/d6/l13 0 2024-04-22T15:18:46.811 INFO:tasks.workunit.client.1.smithi196.stdout:4/41: mknod d6/c11 0 2024-04-22T15:18:46.812 INFO:tasks.workunit.client.1.smithi196.stdout:4/42: stat d6/d10 0 2024-04-22T15:18:46.813 INFO:tasks.workunit.client.0.smithi096.stdout:4/468: creat d5/d7/d1a/d8b/fa2 x:0 0 0 2024-04-22T15:18:46.813 INFO:tasks.workunit.client.0.smithi096.stdout:4/469: fdatasync d5/d7/d1f/f7a 0 2024-04-22T15:18:46.814 INFO:tasks.workunit.client.1.smithi196.stdout:7/23: creat f4 x:0 0 0 2024-04-22T15:18:46.817 INFO:tasks.workunit.client.1.smithi196.stdout:5/18: creat d1/d2/f5 x:0 0 0 2024-04-22T15:18:46.821 INFO:tasks.workunit.client.0.smithi096.stdout:4/470: dwrite d5/d7/d1a/f97 [0,4194304] 0 2024-04-22T15:18:46.828 INFO:tasks.workunit.client.1.smithi196.stdout:6/40: mknod c8 0 2024-04-22T15:18:46.830 INFO:tasks.workunit.client.1.smithi196.stdout:2/72: creat d1/d13/d1f/f20 x:0 0 0 2024-04-22T15:18:46.831 INFO:tasks.workunit.client.0.smithi096.stdout:4/471: rename d5/d7/d2c/d67/f81 to d5/d7/d2c/d3d/d4f/d88/fa3 0 2024-04-22T15:18:46.836 INFO:tasks.workunit.client.0.smithi096.stdout:7/690: getdents d31 0 2024-04-22T15:18:46.838 INFO:tasks.workunit.client.1.smithi196.stdout:8/73: symlink d2/d5/lf 0 2024-04-22T15:18:46.840 INFO:tasks.workunit.client.1.smithi196.stdout:0/48: unlink l0 0 2024-04-22T15:18:46.842 INFO:tasks.workunit.client.1.smithi196.stdout:7/24: creat f5 x:0 0 0 2024-04-22T15:18:46.844 INFO:tasks.workunit.client.0.smithi096.stdout:7/691: rename d31/d32/d7c/dcf to d31/d32/d33/d44/da8/dbf/ddf 0 2024-04-22T15:18:46.847 INFO:tasks.workunit.client.1.smithi196.stdout:6/41: creat f9 x:0 0 0 2024-04-22T15:18:46.847 INFO:tasks.workunit.client.1.smithi196.stdout:0/49: dwrite d3/ff [0,4194304] 0 2024-04-22T15:18:46.848 INFO:tasks.workunit.client.1.smithi196.stdout:9/30: link c6 c8 0 2024-04-22T15:18:46.849 INFO:tasks.workunit.client.1.smithi196.stdout:4/43: dwrite f5 [0,4194304] 0 2024-04-22T15:18:46.850 INFO:tasks.workunit.client.1.smithi196.stdout:1/25: mkdir d1/db/dc 0 2024-04-22T15:18:46.850 INFO:tasks.workunit.client.1.smithi196.stdout:4/44: chown d6/ff 869648453 1 2024-04-22T15:18:46.851 INFO:tasks.workunit.client.1.smithi196.stdout:4/45: readlink d6/d10/ld 0 2024-04-22T15:18:46.853 INFO:tasks.workunit.client.0.smithi096.stdout:7/692: getdents d31/d68/d98 0 2024-04-22T15:18:46.853 INFO:tasks.workunit.client.1.smithi196.stdout:0/50: dread d3/ff [0,4194304] 0 2024-04-22T15:18:46.854 INFO:tasks.workunit.client.1.smithi196.stdout:0/51: truncate d3/f11 849356 0 2024-04-22T15:18:46.855 INFO:tasks.workunit.client.1.smithi196.stdout:8/74: fsync f0 0 2024-04-22T15:18:46.856 INFO:tasks.workunit.client.1.smithi196.stdout:0/52: truncate d3/fd 184039 0 2024-04-22T15:18:46.859 INFO:tasks.workunit.client.1.smithi196.stdout:8/75: dread d2/f3 [0,4194304] 0 2024-04-22T15:18:46.859 INFO:tasks.workunit.client.1.smithi196.stdout:5/19: creat d1/d4/f6 x:0 0 0 2024-04-22T15:18:46.860 INFO:tasks.workunit.client.1.smithi196.stdout:5/20: read - d1/d2/f5 zero size 2024-04-22T15:18:46.861 INFO:tasks.workunit.client.0.smithi096.stdout:7/693: symlink d31/d32/d33/d93/dd5/le0 0 2024-04-22T15:18:46.861 INFO:tasks.workunit.client.1.smithi196.stdout:5/21: chown d1/d2/f5 41 1 2024-04-22T15:18:46.863 INFO:tasks.workunit.client.1.smithi196.stdout:2/73: mknod d1/d1d/c21 0 2024-04-22T15:18:46.863 INFO:tasks.workunit.client.1.smithi196.stdout:2/74: chown d1/f7 1 1 2024-04-22T15:18:46.864 INFO:tasks.workunit.client.1.smithi196.stdout:6/42: creat fa x:0 0 0 2024-04-22T15:18:46.866 INFO:tasks.workunit.client.0.smithi096.stdout:7/694: creat d31/d32/d33/d44/da8/dbf/fe1 x:0 0 0 2024-04-22T15:18:46.866 INFO:tasks.workunit.client.0.smithi096.stdout:7/695: stat d31/d32/d33/d5e/lda 0 2024-04-22T15:18:46.867 INFO:tasks.workunit.client.0.smithi096.stdout:7/696: chown fa 306316 1 2024-04-22T15:18:46.869 INFO:tasks.workunit.client.1.smithi196.stdout:9/31: creat f9 x:0 0 0 2024-04-22T15:18:46.869 INFO:tasks.workunit.client.1.smithi196.stdout:9/32: write f1 [3698085,84918] 0 2024-04-22T15:18:46.869 INFO:tasks.workunit.client.1.smithi196.stdout:1/26: mkdir d1/dd 0 2024-04-22T15:18:46.870 INFO:tasks.workunit.client.1.smithi196.stdout:1/27: chown d1/db/dc 161 1 2024-04-22T15:18:46.870 INFO:tasks.workunit.client.1.smithi196.stdout:4/46: rmdir d6/d10 39 2024-04-22T15:18:46.871 INFO:tasks.workunit.client.1.smithi196.stdout:3/28: truncate d0/f3 3736714 0 2024-04-22T15:18:46.876 INFO:tasks.workunit.client.1.smithi196.stdout:1/28: dwrite d1/f8 [0,4194304] 0 2024-04-22T15:18:46.937 INFO:tasks.workunit.client.0.smithi096.stdout:2/459: sync 2024-04-22T15:18:46.938 INFO:tasks.workunit.client.1.smithi196.stdout:8/76: creat d2/dc/f10 x:0 0 0 2024-04-22T15:18:46.938 INFO:tasks.workunit.client.1.smithi196.stdout:5/22: symlink d1/l7 0 2024-04-22T15:18:46.938 INFO:tasks.workunit.client.1.smithi196.stdout:2/75: creat d1/d1d/f22 x:0 0 0 2024-04-22T15:18:46.939 INFO:tasks.workunit.client.1.smithi196.stdout:8/77: dread d2/dc/fd [0,4194304] 0 2024-04-22T15:18:46.940 INFO:tasks.workunit.client.1.smithi196.stdout:6/43: symlink lb 0 2024-04-22T15:18:46.945 INFO:tasks.workunit.client.0.smithi096.stdout:7/697: truncate d31/d32/d36/f3c 3905885 0 2024-04-22T15:18:46.951 INFO:tasks.workunit.client.0.smithi096.stdout:2/460: truncate d3/d1f/d2f/f34 3964683 0 2024-04-22T15:18:46.952 INFO:tasks.workunit.client.0.smithi096.stdout:2/461: write d3/db/d30/f3a [3691961,19546] 0 2024-04-22T15:18:46.952 INFO:tasks.workunit.client.1.smithi196.stdout:4/47: dwrite d6/d10/fe [0,4194304] 0 2024-04-22T15:18:46.953 INFO:tasks.workunit.client.0.smithi096.stdout:2/462: chown d3/db/d2d/d46/d7e/d8d 4660 1 2024-04-22T15:18:46.954 INFO:tasks.workunit.client.0.smithi096.stdout:2/463: write d3/d1f/d2f/f4e [3729546,117592] 0 2024-04-22T15:18:46.955 INFO:tasks.workunit.client.0.smithi096.stdout:7/698: creat d31/d32/d33/d44/da8/fe2 x:0 0 0 2024-04-22T15:18:46.958 INFO:tasks.workunit.client.1.smithi196.stdout:0/53: mkdir d3/de/d14 0 2024-04-22T15:18:46.964 INFO:tasks.workunit.client.0.smithi096.stdout:7/699: dwrite d31/d32/dc0/fc4 [0,4194304] 0 2024-04-22T15:18:46.964 INFO:tasks.workunit.client.1.smithi196.stdout:5/23: dread f0 [0,4194304] 0 2024-04-22T15:18:46.965 INFO:tasks.workunit.client.0.smithi096.stdout:2/464: unlink d3/d1f/d2f/f41 0 2024-04-22T15:18:46.969 INFO:tasks.workunit.client.1.smithi196.stdout:2/76: mkdir d1/d23 0 2024-04-22T15:18:46.969 INFO:tasks.workunit.client.1.smithi196.stdout:1/29: write d1/f7 [606047,41755] 0 2024-04-22T15:18:46.970 INFO:tasks.workunit.client.1.smithi196.stdout:1/30: stat d1/c6 0 2024-04-22T15:18:46.971 INFO:tasks.workunit.client.1.smithi196.stdout:8/78: mknod d2/d5/c11 0 2024-04-22T15:18:46.972 INFO:tasks.workunit.client.0.smithi096.stdout:7/700: dwrite d31/d32/d33/d5e/f63 [0,4194304] 0 2024-04-22T15:18:46.974 INFO:tasks.workunit.client.1.smithi196.stdout:6/44: creat fc x:0 0 0 2024-04-22T15:18:46.978 INFO:tasks.workunit.client.0.smithi096.stdout:2/465: symlink d3/db/d2d/d46/l91 0 2024-04-22T15:18:46.982 INFO:tasks.workunit.client.1.smithi196.stdout:7/25: getdents . 0 2024-04-22T15:18:46.984 INFO:tasks.workunit.client.1.smithi196.stdout:0/54: chown d3/d6 17201 1 2024-04-22T15:18:46.984 INFO:tasks.workunit.client.1.smithi196.stdout:0/55: chown d3/f5 3008 1 2024-04-22T15:18:46.986 INFO:tasks.workunit.client.0.smithi096.stdout:2/466: creat d3/db/d2d/d46/d7e/d8d/f92 x:0 0 0 2024-04-22T15:18:46.988 INFO:tasks.workunit.client.1.smithi196.stdout:2/77: creat d1/d13/d1f/f24 x:0 0 0 2024-04-22T15:18:46.990 INFO:tasks.workunit.client.0.smithi096.stdout:7/701: dread f11 [0,4194304] 0 2024-04-22T15:18:46.990 INFO:tasks.workunit.client.0.smithi096.stdout:2/467: mknod d3/db/d2d/d46/c93 0 2024-04-22T15:18:46.991 INFO:tasks.workunit.client.0.smithi096.stdout:2/468: chown d3/db/d2d 8666 1 2024-04-22T15:18:46.991 INFO:tasks.workunit.client.0.smithi096.stdout:7/702: write d31/d32/f59 [3398393,90953] 0 2024-04-22T15:18:46.992 INFO:tasks.workunit.client.1.smithi196.stdout:9/33: sync 2024-04-22T15:18:46.992 INFO:tasks.workunit.client.0.smithi096.stdout:7/703: chown d31/d32/d33/d54/f89 55071171 1 2024-04-22T15:18:46.993 INFO:tasks.workunit.client.1.smithi196.stdout:7/26: symlink l6 0 2024-04-22T15:18:46.994 INFO:tasks.workunit.client.1.smithi196.stdout:7/27: write f5 [255367,59545] 0 2024-04-22T15:18:46.996 INFO:tasks.workunit.client.0.smithi096.stdout:2/469: symlink d3/db/d2d/d46/l94 0 2024-04-22T15:18:46.997 INFO:tasks.workunit.client.1.smithi196.stdout:5/24: link d1/d2/f5 d1/d2/f8 0 2024-04-22T15:18:46.999 INFO:tasks.workunit.client.0.smithi096.stdout:7/704: creat d31/d32/dad/fe3 x:0 0 0 2024-04-22T15:18:46.999 INFO:tasks.workunit.client.1.smithi196.stdout:7/28: dwrite f4 [0,4194304] 0 2024-04-22T15:18:47.000 INFO:tasks.workunit.client.1.smithi196.stdout:2/78: mkdir d1/d13/d25 0 2024-04-22T15:18:47.002 INFO:tasks.workunit.client.0.smithi096.stdout:2/470: rename d3/db/df/d24/c26 to d3/db/d2d/d46/d7e/d8d/c95 0 2024-04-22T15:18:47.003 INFO:tasks.workunit.client.0.smithi096.stdout:7/705: rename d31/d32 to d31/d32/d33/da2/de4 22 2024-04-22T15:18:47.004 INFO:tasks.workunit.client.1.smithi196.stdout:1/31: mkdir d1/db/dc/de 0 2024-04-22T15:18:47.005 INFO:tasks.workunit.client.1.smithi196.stdout:8/79: symlink d2/d8/l12 0 2024-04-22T15:18:47.008 INFO:tasks.workunit.client.1.smithi196.stdout:7/29: dwrite f5 [0,4194304] 0 2024-04-22T15:18:47.009 INFO:tasks.workunit.client.1.smithi196.stdout:2/79: dwrite d1/ff [4194304,4194304] 0 2024-04-22T15:18:47.010 INFO:tasks.workunit.client.1.smithi196.stdout:9/34: creat fa x:0 0 0 2024-04-22T15:18:47.010 INFO:tasks.workunit.client.1.smithi196.stdout:0/56: write d3/f5 [2646178,74808] 0 2024-04-22T15:18:47.010 INFO:tasks.workunit.client.0.smithi096.stdout:2/471: symlink d3/db/d2d/d46/d7e/d84/l96 0 2024-04-22T15:18:47.011 INFO:tasks.workunit.client.1.smithi196.stdout:2/80: fdatasync d1/d1d/f22 0 2024-04-22T15:18:47.012 INFO:tasks.workunit.client.1.smithi196.stdout:9/35: write f5 [57502,68035] 0 2024-04-22T15:18:47.013 INFO:tasks.workunit.client.1.smithi196.stdout:2/81: write d1/f1b [241261,58947] 0 2024-04-22T15:18:47.013 INFO:tasks.workunit.client.0.smithi096.stdout:0/557: sync 2024-04-22T15:18:47.015 INFO:tasks.workunit.client.1.smithi196.stdout:0/57: write d3/f11 [184128,69337] 0 2024-04-22T15:18:47.016 INFO:tasks.workunit.client.1.smithi196.stdout:2/82: truncate d1/f15 789416 0 2024-04-22T15:18:47.017 INFO:tasks.workunit.client.1.smithi196.stdout:0/58: read d3/f5 [2901699,117978] 0 2024-04-22T15:18:47.018 INFO:tasks.workunit.client.1.smithi196.stdout:2/83: fdatasync d1/f10 0 2024-04-22T15:18:47.020 INFO:tasks.workunit.client.0.smithi096.stdout:0/558: chown d7/de/d5b/d95/dc/d16/d18/c46 25366704 1 2024-04-22T15:18:47.021 INFO:tasks.workunit.client.1.smithi196.stdout:9/36: dread f1 [0,4194304] 0 2024-04-22T15:18:47.021 INFO:tasks.workunit.client.0.smithi096.stdout:0/559: stat d7/de/d5b/d64/f9a 0 2024-04-22T15:18:47.022 INFO:tasks.workunit.client.1.smithi196.stdout:3/29: dwrite d0/f1 [0,4194304] 0 2024-04-22T15:18:47.023 INFO:tasks.workunit.client.1.smithi196.stdout:8/80: symlink d2/l13 0 2024-04-22T15:18:47.023 INFO:tasks.workunit.client.0.smithi096.stdout:7/706: getdents d31/d32/d33/d54 0 2024-04-22T15:18:47.029 INFO:tasks.workunit.client.1.smithi196.stdout:7/30: unlink f5 0 2024-04-22T15:18:47.030 INFO:tasks.workunit.client.1.smithi196.stdout:5/25: write d1/d2/f8 [969694,42984] 0 2024-04-22T15:18:47.032 INFO:tasks.workunit.client.0.smithi096.stdout:7/707: rename d31/d32/d36/l74 to d31/le5 0 2024-04-22T15:18:47.034 INFO:tasks.workunit.client.0.smithi096.stdout:2/472: link d3/db/df/l3c d3/db/d30/l97 0 2024-04-22T15:18:47.036 INFO:tasks.workunit.client.0.smithi096.stdout:0/560: dwrite d7/de/d5b/d95/dc/d16/f5d [0,4194304] 0 2024-04-22T15:18:47.038 INFO:tasks.workunit.client.1.smithi196.stdout:2/84: rename d1/c6 to d1/d13/d1f/c26 0 2024-04-22T15:18:47.038 INFO:tasks.workunit.client.1.smithi196.stdout:2/85: read - d1/d13/f19 zero size 2024-04-22T15:18:47.039 INFO:tasks.workunit.client.1.smithi196.stdout:1/32: unlink d1/c9 0 2024-04-22T15:18:47.039 INFO:tasks.workunit.client.1.smithi196.stdout:2/86: chown d1/le 178831774 1 2024-04-22T15:18:47.041 INFO:tasks.workunit.client.1.smithi196.stdout:2/87: write d1/d13/d1f/f20 [878938,16840] 0 2024-04-22T15:18:47.041 INFO:tasks.workunit.client.1.smithi196.stdout:2/88: stat d1/c4 0 2024-04-22T15:18:47.042 INFO:tasks.workunit.client.1.smithi196.stdout:2/89: chown d1/d13/d1f 142043284 1 2024-04-22T15:18:47.043 INFO:tasks.workunit.client.1.smithi196.stdout:3/30: fsync d0/f1 0 2024-04-22T15:18:47.043 INFO:tasks.workunit.client.1.smithi196.stdout:2/90: stat d1/d23 0 2024-04-22T15:18:47.043 INFO:tasks.workunit.client.1.smithi196.stdout:9/37: mkdir db 0 2024-04-22T15:18:47.044 INFO:tasks.workunit.client.1.smithi196.stdout:2/91: write d1/ff [7257684,50741] 0 2024-04-22T15:18:47.045 INFO:tasks.workunit.client.1.smithi196.stdout:2/92: fsync d1/f16 0 2024-04-22T15:18:47.046 INFO:tasks.workunit.client.1.smithi196.stdout:2/93: write d1/f15 [1572787,103118] 0 2024-04-22T15:18:47.049 INFO:tasks.workunit.client.1.smithi196.stdout:7/31: creat f7 x:0 0 0 2024-04-22T15:18:47.050 INFO:tasks.workunit.client.1.smithi196.stdout:5/26: rmdir d1 39 2024-04-22T15:18:47.055 INFO:tasks.workunit.client.1.smithi196.stdout:7/32: dwrite f4 [0,4194304] 0 2024-04-22T15:18:47.057 INFO:tasks.workunit.client.1.smithi196.stdout:4/48: sync 2024-04-22T15:18:47.057 INFO:tasks.workunit.client.1.smithi196.stdout:9/38: write f2 [2771896,39558] 0 2024-04-22T15:18:47.064 INFO:tasks.workunit.client.0.smithi096.stdout:0/561: write d7/de/d5b/d95/dc/f13 [7317575,29296] 0 2024-04-22T15:18:47.069 INFO:tasks.workunit.client.1.smithi196.stdout:2/94: fsync d1/f3 0 2024-04-22T15:18:47.072 INFO:tasks.workunit.client.1.smithi196.stdout:8/81: dwrite d2/f3 [4194304,4194304] 0 2024-04-22T15:18:47.072 INFO:tasks.workunit.client.0.smithi096.stdout:0/562: dwrite d7/de/d5b/d95/dc/d16/d18/f2b [0,4194304] 0 2024-04-22T15:18:47.074 INFO:tasks.workunit.client.1.smithi196.stdout:2/95: dread d1/d13/d1f/f20 [0,4194304] 0 2024-04-22T15:18:47.075 INFO:tasks.workunit.client.1.smithi196.stdout:2/96: truncate d1/fd 4558970 0 2024-04-22T15:18:47.077 INFO:tasks.workunit.client.1.smithi196.stdout:0/59: creat d3/d6/dc/f15 x:0 0 0 2024-04-22T15:18:47.086 INFO:tasks.workunit.client.1.smithi196.stdout:7/33: mknod c8 0 2024-04-22T15:18:47.088 INFO:tasks.workunit.client.1.smithi196.stdout:4/49: creat d6/da/f12 x:0 0 0 2024-04-22T15:18:47.089 INFO:tasks.workunit.client.1.smithi196.stdout:4/50: chown d6/d10/ld 0 1 2024-04-22T15:18:47.089 INFO:tasks.workunit.client.1.smithi196.stdout:6/45: sync 2024-04-22T15:18:47.091 INFO:tasks.workunit.client.1.smithi196.stdout:9/39: rename f9 to db/fc 0 2024-04-22T15:18:47.094 INFO:tasks.workunit.client.1.smithi196.stdout:4/51: dwrite d6/d10/fe [0,4194304] 0 2024-04-22T15:18:47.098 INFO:tasks.workunit.client.1.smithi196.stdout:8/82: write d2/dc/fd [4511180,117666] 0 2024-04-22T15:18:47.099 INFO:tasks.workunit.client.1.smithi196.stdout:8/83: write d2/dc/f10 [313622,56663] 0 2024-04-22T15:18:47.100 INFO:tasks.workunit.client.1.smithi196.stdout:8/84: rename d2/d5 to d2/d5/d14 22 2024-04-22T15:18:47.100 INFO:tasks.workunit.client.1.smithi196.stdout:5/27: mkdir d1/d9 0 2024-04-22T15:18:47.105 INFO:tasks.workunit.client.1.smithi196.stdout:1/33: truncate d1/f7 469167 0 2024-04-22T15:18:47.176 INFO:tasks.workunit.client.0.smithi096.stdout:0/563: chown d7/de/d5b/d95/d15/d35/c4b 47975 1 2024-04-22T15:18:47.181 INFO:tasks.workunit.client.0.smithi096.stdout:0/564: rename d7/de/d5b/d62/f63 to d7/de/d5b/d95/dc/d16/d65/d7c/fbd 0 2024-04-22T15:18:47.181 INFO:tasks.workunit.client.0.smithi096.stdout:0/565: fdatasync d7/d42/f86 0 2024-04-22T15:18:47.184 INFO:tasks.workunit.client.1.smithi196.stdout:9/40: sync 2024-04-22T15:18:47.184 INFO:tasks.workunit.client.0.smithi096.stdout:0/566: unlink d7/de/d5b/d95/d15/l43 0 2024-04-22T15:18:47.188 INFO:tasks.workunit.client.0.smithi096.stdout:0/567: creat d7/de/d3a/d61/d79/fbe x:0 0 0 2024-04-22T15:18:47.189 INFO:tasks.workunit.client.1.smithi196.stdout:9/41: dwrite f7 [0,4194304] 0 2024-04-22T15:18:47.195 INFO:tasks.workunit.client.0.smithi096.stdout:0/568: creat d7/de/fbf x:0 0 0 2024-04-22T15:18:47.199 INFO:tasks.workunit.client.0.smithi096.stdout:0/569: symlink d7/de/d5b/d62/lc0 0 2024-04-22T15:18:47.200 INFO:tasks.workunit.client.0.smithi096.stdout:9/344: sync 2024-04-22T15:18:47.276 INFO:tasks.workunit.client.1.smithi196.stdout:9/42: sync 2024-04-22T15:18:47.277 INFO:tasks.workunit.client.1.smithi196.stdout:9/43: write f4 [231727,59482] 0 2024-04-22T15:18:47.355 INFO:tasks.workunit.client.0.smithi096.stdout:9/345: symlink d1/d6f/l74 0 2024-04-22T15:18:47.356 INFO:tasks.workunit.client.0.smithi096.stdout:9/346: chown d1/d4/d9/dd/f1a 3050 1 2024-04-22T15:18:47.368 INFO:tasks.workunit.client.0.smithi096.stdout:9/347: dread d1/d25/f4c [0,4194304] 0 2024-04-22T15:18:47.396 INFO:tasks.workunit.client.0.smithi096.stdout:6/387: sync 2024-04-22T15:18:47.397 INFO:tasks.workunit.client.0.smithi096.stdout:6/388: chown d3/d28/dd/d11/d2c/l34 994248 1 2024-04-22T15:18:47.403 INFO:tasks.workunit.client.0.smithi096.stdout:6/389: dwrite d3/d28/dd/d11/f18 [0,4194304] 0 2024-04-22T15:18:47.407 INFO:tasks.workunit.client.0.smithi096.stdout:6/390: chown d3/d28/d54/d68/c6a 121 1 2024-04-22T15:18:47.415 INFO:tasks.workunit.client.0.smithi096.stdout:6/391: dwrite d3/d28/dd/d11/d2c/d44/d51/f60 [4194304,4194304] 0 2024-04-22T15:18:47.419 INFO:tasks.workunit.client.0.smithi096.stdout:6/392: rmdir d3/d28/dd/d11/d2c/d3b 39 2024-04-22T15:18:47.427 INFO:tasks.workunit.client.0.smithi096.stdout:6/393: symlink d3/d28/dd/d11/l85 0 2024-04-22T15:18:47.427 INFO:tasks.workunit.client.0.smithi096.stdout:6/394: chown d3/d3a/f3e 44 1 2024-04-22T15:18:47.431 INFO:tasks.workunit.client.0.smithi096.stdout:6/395: rename d3/d28/f37 to d3/d3a/d75/d83/f86 0 2024-04-22T15:18:47.435 INFO:tasks.workunit.client.0.smithi096.stdout:6/396: creat d3/d28/d4d/f87 x:0 0 0 2024-04-22T15:18:47.442 INFO:tasks.workunit.client.0.smithi096.stdout:0/570: dwrite d7/f32 [0,4194304] 0 2024-04-22T15:18:47.442 INFO:tasks.workunit.client.0.smithi096.stdout:6/397: rmdir d3/d81 0 2024-04-22T15:18:47.442 INFO:tasks.workunit.client.0.smithi096.stdout:0/571: fdatasync d7/f9e 0 2024-04-22T15:18:47.447 INFO:tasks.workunit.client.0.smithi096.stdout:0/572: symlink d7/de/d5b/d62/d67/lc1 0 2024-04-22T15:18:47.448 INFO:tasks.workunit.client.0.smithi096.stdout:9/348: write d1/d4/f42 [198798,15214] 0 2024-04-22T15:18:47.450 INFO:tasks.workunit.client.0.smithi096.stdout:6/398: creat d3/d28/dd/d11/d2c/d44/d51/f88 x:0 0 0 2024-04-22T15:18:47.452 INFO:tasks.workunit.client.0.smithi096.stdout:6/399: rmdir d3 39 2024-04-22T15:18:47.455 INFO:tasks.workunit.client.0.smithi096.stdout:0/573: creat d7/de/d5b/d95/fc2 x:0 0 0 2024-04-22T15:18:47.456 INFO:tasks.workunit.client.0.smithi096.stdout:9/349: creat d1/f75 x:0 0 0 2024-04-22T15:18:47.457 INFO:tasks.workunit.client.0.smithi096.stdout:9/350: stat d1/d4/f42 0 2024-04-22T15:18:47.457 INFO:tasks.workunit.client.0.smithi096.stdout:9/351: write d1/f75 [551701,42381] 0 2024-04-22T15:18:47.459 INFO:tasks.workunit.client.0.smithi096.stdout:0/574: symlink d7/de/d3a/lc3 0 2024-04-22T15:18:47.460 INFO:tasks.workunit.client.0.smithi096.stdout:9/352: rmdir d1/d4/d9 39 2024-04-22T15:18:47.466 INFO:tasks.workunit.client.0.smithi096.stdout:9/353: creat d1/d4/d9/d43/d57/d5b/f76 x:0 0 0 2024-04-22T15:18:47.466 INFO:tasks.workunit.client.0.smithi096.stdout:0/575: dwrite d7/de/d5b/d95/d15/d35/fb4 [0,4194304] 0 2024-04-22T15:18:47.477 INFO:tasks.workunit.client.0.smithi096.stdout:9/354: creat d1/d4/d9/d43/d57/d5b/f77 x:0 0 0 2024-04-22T15:18:47.494 INFO:tasks.workunit.client.0.smithi096.stdout:9/355: dread d1/d4/d9/d43/d57/d5b/f66 [0,4194304] 0 2024-04-22T15:18:47.516 INFO:tasks.workunit.client.0.smithi096.stdout:0/576: rmdir d7 39 2024-04-22T15:18:47.532 INFO:tasks.workunit.client.0.smithi096.stdout:0/577: dread d7/de/d5b/d95/dc/d16/d65/d7c/f9b [0,4194304] 0 2024-04-22T15:18:47.532 INFO:tasks.workunit.client.1.smithi196.stdout:7/34: mknod c9 0 2024-04-22T15:18:47.532 INFO:tasks.workunit.client.1.smithi196.stdout:7/35: stat f2 0 2024-04-22T15:18:47.536 INFO:tasks.workunit.client.1.smithi196.stdout:6/46: mkdir dd 0 2024-04-22T15:18:47.536 INFO:tasks.workunit.client.1.smithi196.stdout:6/47: chown lb 4727 1 2024-04-22T15:18:47.537 INFO:tasks.workunit.client.0.smithi096.stdout:0/578: dwrite d7/f4d [0,4194304] 0 2024-04-22T15:18:47.538 INFO:tasks.workunit.client.1.smithi196.stdout:4/52: rmdir d6 39 2024-04-22T15:18:47.541 INFO:tasks.workunit.client.1.smithi196.stdout:8/85: mknod d2/d5/c15 0 2024-04-22T15:18:47.544 INFO:tasks.workunit.client.1.smithi196.stdout:2/97: mkdir d1/d23/d27 0 2024-04-22T15:18:47.544 INFO:tasks.workunit.client.1.smithi196.stdout:2/98: dread - d1/f16 zero size 2024-04-22T15:18:47.546 INFO:tasks.workunit.client.1.smithi196.stdout:1/34: creat d1/db/ff x:0 0 0 2024-04-22T15:18:47.547 INFO:tasks.workunit.client.0.smithi096.stdout:0/579: rmdir d7/da7/dba 0 2024-04-22T15:18:47.547 INFO:tasks.workunit.client.1.smithi196.stdout:1/35: truncate d1/db/ff 1017194 0 2024-04-22T15:18:47.553 INFO:tasks.workunit.client.0.smithi096.stdout:0/580: rmdir d7/de/d5b/d64 39 2024-04-22T15:18:47.554 INFO:tasks.workunit.client.1.smithi196.stdout:7/36: creat fa x:0 0 0 2024-04-22T15:18:47.554 INFO:tasks.workunit.client.1.smithi196.stdout:7/37: write f7 [431311,109794] 0 2024-04-22T15:18:47.554 INFO:tasks.workunit.client.1.smithi196.stdout:3/31: getdents d0 0 2024-04-22T15:18:47.554 INFO:tasks.workunit.client.0.smithi096.stdout:5/471: sync 2024-04-22T15:18:47.555 INFO:tasks.workunit.client.0.smithi096.stdout:1/625: sync 2024-04-22T15:18:47.555 INFO:tasks.workunit.client.0.smithi096.stdout:3/353: sync 2024-04-22T15:18:47.555 INFO:tasks.workunit.client.0.smithi096.stdout:8/402: sync 2024-04-22T15:18:47.555 INFO:tasks.workunit.client.0.smithi096.stdout:9/356: dwrite d1/f4f [0,4194304] 0 2024-04-22T15:18:47.556 INFO:tasks.workunit.client.0.smithi096.stdout:1/626: fsync d6/dc/d8a/d3f/d62/d7d/f84 0 2024-04-22T15:18:47.557 INFO:tasks.workunit.client.1.smithi196.stdout:4/53: write d6/d10/fe [391182,9603] 0 2024-04-22T15:18:47.558 INFO:tasks.workunit.client.0.smithi096.stdout:1/627: write d6/dc/d8a/d3f/d54/d32/d3b/d56/f44 [1687806,58178] 0 2024-04-22T15:18:47.558 INFO:tasks.workunit.client.0.smithi096.stdout:8/403: mkdir d4/d10/d25/d38/d8f 0 2024-04-22T15:18:47.559 INFO:tasks.workunit.client.1.smithi196.stdout:8/86: rmdir d2/d8 39 2024-04-22T15:18:47.560 INFO:tasks.workunit.client.0.smithi096.stdout:8/404: fsync d4/d10/d25/d4c/d76/f89 0 2024-04-22T15:18:47.563 INFO:tasks.workunit.client.1.smithi196.stdout:5/28: symlink d1/d9/la 0 2024-04-22T15:18:47.563 INFO:tasks.workunit.client.1.smithi196.stdout:4/54: dread d6/d10/fe [0,4194304] 0 2024-04-22T15:18:47.565 INFO:tasks.workunit.client.0.smithi096.stdout:0/581: mknod d7/de/d5b/cc4 0 2024-04-22T15:18:47.567 INFO:tasks.workunit.client.0.smithi096.stdout:9/357: mkdir d1/d4/d9/d43/d57/d5b/d78 0 2024-04-22T15:18:47.569 INFO:tasks.workunit.client.1.smithi196.stdout:9/44: unlink c8 0 2024-04-22T15:18:47.570 INFO:tasks.workunit.client.0.smithi096.stdout:0/582: dread d7/de/d5b/d95/dc/f8d [0,4194304] 0 2024-04-22T15:18:47.572 INFO:tasks.workunit.client.1.smithi196.stdout:7/38: creat fb x:0 0 0 2024-04-22T15:18:47.573 INFO:tasks.workunit.client.0.smithi096.stdout:1/628: mknod d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b/cc5 0 2024-04-22T15:18:47.574 INFO:tasks.workunit.client.0.smithi096.stdout:9/358: dwrite d1/d4/d9/d43/f6a [0,4194304] 0 2024-04-22T15:18:47.574 INFO:tasks.workunit.client.1.smithi196.stdout:3/32: rmdir d0 39 2024-04-22T15:18:47.574 INFO:tasks.workunit.client.1.smithi196.stdout:9/45: dwrite f7 [0,4194304] 0 2024-04-22T15:18:47.575 INFO:tasks.workunit.client.0.smithi096.stdout:1/629: stat d6/dc/d8a/d3f/d54/c5a 0 2024-04-22T15:18:47.575 INFO:tasks.workunit.client.1.smithi196.stdout:9/46: chown f7 8 1 2024-04-22T15:18:47.576 INFO:tasks.workunit.client.1.smithi196.stdout:9/47: chown f5 432043 1 2024-04-22T15:18:47.576 INFO:tasks.workunit.client.0.smithi096.stdout:1/630: stat d6/dc/d8a/d3f/d62/d7d/dab 0 2024-04-22T15:18:47.576 INFO:tasks.workunit.client.1.smithi196.stdout:6/48: creat dd/fe x:0 0 0 2024-04-22T15:18:47.577 INFO:tasks.workunit.client.1.smithi196.stdout:6/49: read f4 [3181602,130678] 0 2024-04-22T15:18:47.577 INFO:tasks.workunit.client.1.smithi196.stdout:6/50: stat f9 0 2024-04-22T15:18:47.578 INFO:tasks.workunit.client.1.smithi196.stdout:6/51: write f5 [1005895,18355] 0 2024-04-22T15:18:47.579 INFO:tasks.workunit.client.0.smithi096.stdout:3/354: mknod da/db/d12/d22/d68/c6d 0 2024-04-22T15:18:47.582 INFO:tasks.workunit.client.1.smithi196.stdout:8/87: mknod d2/dc/c16 0 2024-04-22T15:18:47.585 INFO:tasks.workunit.client.1.smithi196.stdout:5/29: mknod d1/cb 0 2024-04-22T15:18:47.585 INFO:tasks.workunit.client.1.smithi196.stdout:6/52: dwrite f2 [0,4194304] 0 2024-04-22T15:18:47.585 INFO:tasks.workunit.client.1.smithi196.stdout:5/30: readlink d1/l7 0 2024-04-22T15:18:47.586 INFO:tasks.workunit.client.1.smithi196.stdout:6/53: write fc [867119,125201] 0 2024-04-22T15:18:47.589 INFO:tasks.workunit.client.1.smithi196.stdout:2/99: mknod d1/d23/d27/c28 0 2024-04-22T15:18:47.590 INFO:tasks.workunit.client.0.smithi096.stdout:8/405: mknod d4/d10/d25/c90 0 2024-04-22T15:18:47.591 INFO:tasks.workunit.client.1.smithi196.stdout:0/60: getdents d3/d9 0 2024-04-22T15:18:47.592 INFO:tasks.workunit.client.0.smithi096.stdout:1/631: rmdir d6/dc/d8a/d3f/d54/d32/d8c 39 2024-04-22T15:18:47.593 INFO:tasks.workunit.client.1.smithi196.stdout:1/36: mknod d1/db/dc/de/c10 0 2024-04-22T15:18:47.594 INFO:tasks.workunit.client.1.smithi196.stdout:5/31: dwrite d1/d4/f6 [0,4194304] 0 2024-04-22T15:18:47.596 INFO:tasks.workunit.client.0.smithi096.stdout:5/472: creat d4/d2c/d2f/f9b x:0 0 0 2024-04-22T15:18:47.599 INFO:tasks.workunit.client.0.smithi096.stdout:3/355: rename da/db/de/d19/d2d/f64 to da/db/d10/f6e 0 2024-04-22T15:18:47.601 INFO:tasks.workunit.client.1.smithi196.stdout:7/39: rename fb to fc 0 2024-04-22T15:18:47.601 INFO:tasks.workunit.client.1.smithi196.stdout:7/40: read f4 [2015438,32798] 0 2024-04-22T15:18:47.602 INFO:tasks.workunit.client.1.smithi196.stdout:7/41: read f2 [808792,53055] 0 2024-04-22T15:18:47.605 INFO:tasks.workunit.client.0.smithi096.stdout:0/583: symlink d7/d42/d4c/lc5 0 2024-04-22T15:18:47.606 INFO:tasks.workunit.client.0.smithi096.stdout:0/584: chown d7/de/d3a 16720375 1 2024-04-22T15:18:47.607 INFO:tasks.workunit.client.1.smithi196.stdout:9/48: mkdir db/dd 0 2024-04-22T15:18:47.608 INFO:tasks.workunit.client.0.smithi096.stdout:9/359: mkdir d1/d4/d79 0 2024-04-22T15:18:47.608 INFO:tasks.workunit.client.1.smithi196.stdout:5/32: dread d1/d2/f8 [0,4194304] 0 2024-04-22T15:18:47.608 INFO:tasks.workunit.client.0.smithi096.stdout:9/360: readlink d1/d4/l3f 0 2024-04-22T15:18:47.609 INFO:tasks.workunit.client.1.smithi196.stdout:9/49: write f2 [1893358,42948] 0 2024-04-22T15:18:47.609 INFO:tasks.workunit.client.1.smithi196.stdout:5/33: fsync d1/d2/f5 0 2024-04-22T15:18:47.609 INFO:tasks.workunit.client.0.smithi096.stdout:8/406: mknod d4/d10/d25/d4c/d76/c91 0 2024-04-22T15:18:47.610 INFO:tasks.workunit.client.0.smithi096.stdout:9/361: truncate d1/d4/d9/d43/d57/d5b/f77 150168 0 2024-04-22T15:18:47.611 INFO:tasks.workunit.client.1.smithi196.stdout:3/33: dwrite d0/f4 [0,4194304] 0 2024-04-22T15:18:47.614 INFO:tasks.workunit.client.0.smithi096.stdout:9/362: dread d1/d4/d9/dd/f1a [0,4194304] 0 2024-04-22T15:18:47.616 INFO:tasks.workunit.client.1.smithi196.stdout:8/88: rename d2/l13 to d2/d5/l17 0 2024-04-22T15:18:47.618 INFO:tasks.workunit.client.1.smithi196.stdout:9/50: dwrite f1 [4194304,4194304] 0 2024-04-22T15:18:47.618 INFO:tasks.workunit.client.1.smithi196.stdout:4/55: mkdir d6/d13 0 2024-04-22T15:18:47.618 INFO:tasks.workunit.client.0.smithi096.stdout:7/708: sync 2024-04-22T15:18:47.618 INFO:tasks.workunit.client.0.smithi096.stdout:4/472: sync 2024-04-22T15:18:47.618 INFO:tasks.workunit.client.0.smithi096.stdout:6/400: sync 2024-04-22T15:18:47.619 INFO:tasks.workunit.client.0.smithi096.stdout:2/473: sync 2024-04-22T15:18:47.619 INFO:tasks.workunit.client.1.smithi196.stdout:9/51: write fa [648861,26560] 0 2024-04-22T15:18:47.620 INFO:tasks.workunit.client.1.smithi196.stdout:4/56: write d6/da/f12 [299105,13910] 0 2024-04-22T15:18:47.622 INFO:tasks.workunit.client.1.smithi196.stdout:3/34: dwrite d0/f4 [0,4194304] 0 2024-04-22T15:18:47.622 INFO:tasks.workunit.client.1.smithi196.stdout:9/52: dread - db/fc zero size 2024-04-22T15:18:47.622 INFO:tasks.workunit.client.0.smithi096.stdout:0/585: rmdir d7/d42/d4c/d55/d98 39 2024-04-22T15:18:47.623 INFO:tasks.workunit.client.1.smithi196.stdout:4/57: chown d6/da/f12 7558 1 2024-04-22T15:18:47.627 INFO:tasks.workunit.client.0.smithi096.stdout:0/586: dread d7/f4d [4194304,4194304] 0 2024-04-22T15:18:47.627 INFO:tasks.workunit.client.1.smithi196.stdout:4/58: dread d6/d10/fe [0,4194304] 0 2024-04-22T15:18:47.628 INFO:tasks.workunit.client.1.smithi196.stdout:0/61: unlink d3/fa 0 2024-04-22T15:18:47.628 INFO:tasks.workunit.client.1.smithi196.stdout:4/59: chown d6/c11 0 1 2024-04-22T15:18:47.629 INFO:tasks.workunit.client.1.smithi196.stdout:1/37: symlink d1/l11 0 2024-04-22T15:18:47.632 INFO:tasks.workunit.client.1.smithi196.stdout:4/60: dread f5 [0,4194304] 0 2024-04-22T15:18:47.632 INFO:tasks.workunit.client.1.smithi196.stdout:4/61: fsync d6/da/f12 0 2024-04-22T15:18:47.632 INFO:tasks.workunit.client.0.smithi096.stdout:1/632: rename d6/f39 to d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/fc6 0 2024-04-22T15:18:47.633 INFO:tasks.workunit.client.1.smithi196.stdout:4/62: fdatasync d6/ff 0 2024-04-22T15:18:47.634 INFO:tasks.workunit.client.0.smithi096.stdout:1/633: read - d6/dc/d8a/d3f/d54/d32/d3b/d94/fad zero size 2024-04-22T15:18:47.641 INFO:tasks.workunit.client.1.smithi196.stdout:7/42: write f2 [1310199,77610] 0 2024-04-22T15:18:47.641 INFO:tasks.workunit.client.1.smithi196.stdout:7/43: dread - fa zero size 2024-04-22T15:18:47.645 INFO:tasks.workunit.client.1.smithi196.stdout:7/44: dread f2 [0,4194304] 0 2024-04-22T15:18:47.646 INFO:tasks.workunit.client.1.smithi196.stdout:7/45: rmdir - no directory 2024-04-22T15:18:47.648 INFO:tasks.workunit.client.1.smithi196.stdout:5/34: symlink d1/d9/lc 0 2024-04-22T15:18:47.650 INFO:tasks.workunit.client.0.smithi096.stdout:7/709: read d31/d32/d33/f58 [2658557,130803] 0 2024-04-22T15:18:47.653 INFO:tasks.workunit.client.0.smithi096.stdout:8/407: creat d4/d10/d25/d4c/d76/d7f/f92 x:0 0 0 2024-04-22T15:18:47.654 INFO:tasks.workunit.client.1.smithi196.stdout:8/89: creat d2/dc/f18 x:0 0 0 2024-04-22T15:18:47.657 INFO:tasks.workunit.client.1.smithi196.stdout:2/100: link d1/f16 d1/d1d/f29 0 2024-04-22T15:18:47.657 INFO:tasks.workunit.client.1.smithi196.stdout:2/101: chown d1/f15 62465514 1 2024-04-22T15:18:47.662 INFO:tasks.workunit.client.0.smithi096.stdout:3/356: truncate da/db/d12/f56 461195 0 2024-04-22T15:18:47.663 INFO:tasks.workunit.client.0.smithi096.stdout:3/357: stat da/db/l6b 0 2024-04-22T15:18:47.666 INFO:tasks.workunit.client.0.smithi096.stdout:2/474: mkdir d3/db/df/d98 0 2024-04-22T15:18:47.666 INFO:tasks.workunit.client.0.smithi096.stdout:2/475: readlink d3/db/d2d/d46/d7e/l90 0 2024-04-22T15:18:47.667 INFO:tasks.workunit.client.0.smithi096.stdout:2/476: readlink d3/d1f/d2f/l43 0 2024-04-22T15:18:47.669 INFO:tasks.workunit.client.1.smithi196.stdout:1/38: symlink d1/db/dc/de/l12 0 2024-04-22T15:18:47.670 INFO:tasks.workunit.client.0.smithi096.stdout:8/408: truncate d4/d10/f4e 574038 0 2024-04-22T15:18:47.673 INFO:tasks.workunit.client.0.smithi096.stdout:0/587: unlink d7/de/d5b/d95/dc/d26/c74 0 2024-04-22T15:18:47.675 INFO:tasks.workunit.client.1.smithi196.stdout:1/39: dwrite d1/db/ff [0,4194304] 0 2024-04-22T15:18:47.677 INFO:tasks.workunit.client.0.smithi096.stdout:9/363: creat d1/d4/f7a x:0 0 0 2024-04-22T15:18:47.679 INFO:tasks.workunit.client.0.smithi096.stdout:5/473: rmdir d4/d76 0 2024-04-22T15:18:47.680 INFO:tasks.workunit.client.1.smithi196.stdout:7/46: write fc [40196,25340] 0 2024-04-22T15:18:47.681 INFO:tasks.workunit.client.1.smithi196.stdout:7/47: chown f2 0 1 2024-04-22T15:18:47.682 INFO:tasks.workunit.client.0.smithi096.stdout:6/401: link d3/d28/dd/f16 d3/d28/dd/d11/d2c/d6f/f89 0 2024-04-22T15:18:47.685 INFO:tasks.workunit.client.0.smithi096.stdout:6/402: chown d3/d28/dd/f25 1654114 1 2024-04-22T15:18:47.685 INFO:tasks.workunit.client.1.smithi196.stdout:6/54: rename f7 to dd/ff 0 2024-04-22T15:18:47.685 INFO:tasks.workunit.client.1.smithi196.stdout:6/55: stat c8 0 2024-04-22T15:18:47.686 INFO:tasks.workunit.client.1.smithi196.stdout:6/56: dread - f9 zero size 2024-04-22T15:18:47.686 INFO:tasks.workunit.client.1.smithi196.stdout:8/90: creat d2/d5/f19 x:0 0 0 2024-04-22T15:18:47.687 INFO:tasks.workunit.client.0.smithi096.stdout:6/403: write d3/d28/dd/d11/d2c/d44/f77 [174406,71920] 0 2024-04-22T15:18:47.687 INFO:tasks.workunit.client.0.smithi096.stdout:7/710: truncate d31/d32/d36/f3c 2312861 0 2024-04-22T15:18:47.687 INFO:tasks.workunit.client.0.smithi096.stdout:7/711: chown f18 18421602 1 2024-04-22T15:18:47.687 INFO:tasks.workunit.client.1.smithi196.stdout:8/91: truncate d2/f3 9138612 0 2024-04-22T15:18:47.688 INFO:tasks.workunit.client.1.smithi196.stdout:2/102: rmdir d1/d1d 39 2024-04-22T15:18:47.690 INFO:tasks.workunit.client.1.smithi196.stdout:9/53: mkdir db/dd/de 0 2024-04-22T15:18:47.691 INFO:tasks.workunit.client.0.smithi096.stdout:0/588: chown d7/de/d3a/f44 866 1 2024-04-22T15:18:47.691 INFO:tasks.workunit.client.1.smithi196.stdout:8/92: dread d2/fa [0,4194304] 0 2024-04-22T15:18:47.696 INFO:tasks.workunit.client.0.smithi096.stdout:5/474: write d4/d11/f7a [4095265,112209] 0 2024-04-22T15:18:47.697 INFO:tasks.workunit.client.1.smithi196.stdout:0/62: mknod d3/de/d14/c16 0 2024-04-22T15:18:47.698 INFO:tasks.workunit.client.0.smithi096.stdout:5/475: dread d4/d11/d3d/d4d/d59/d5a/d5c/f5f [0,4194304] 0 2024-04-22T15:18:47.705 INFO:tasks.workunit.client.1.smithi196.stdout:3/35: truncate d0/f4 1575103 0 2024-04-22T15:18:47.707 INFO:tasks.workunit.client.0.smithi096.stdout:6/404: rmdir d3/d28/d54/d68/d6e 39 2024-04-22T15:18:47.709 INFO:tasks.workunit.client.0.smithi096.stdout:7/712: truncate d31/d32/d36/f4d 2449825 0 2024-04-22T15:18:47.710 INFO:tasks.workunit.client.0.smithi096.stdout:3/358: write da/db/de/f40 [762307,105702] 0 2024-04-22T15:18:47.711 INFO:tasks.workunit.client.1.smithi196.stdout:7/48: dread f2 [0,4194304] 0 2024-04-22T15:18:47.712 INFO:tasks.workunit.client.1.smithi196.stdout:6/57: creat dd/f10 x:0 0 0 2024-04-22T15:18:47.712 INFO:tasks.workunit.client.1.smithi196.stdout:6/58: truncate dd/fe 232850 0 2024-04-22T15:18:47.713 INFO:tasks.workunit.client.1.smithi196.stdout:6/59: stat dd/fe 0 2024-04-22T15:18:47.713 INFO:tasks.workunit.client.0.smithi096.stdout:1/634: link d6/dc/d8a/d3f/d54/d32/d8c/c26 d6/dc/d8a/cc7 0 2024-04-22T15:18:47.714 INFO:tasks.workunit.client.1.smithi196.stdout:6/60: write fa [567595,113913] 0 2024-04-22T15:18:47.714 INFO:tasks.workunit.client.1.smithi196.stdout:6/61: chown fa 1309 1 2024-04-22T15:18:47.715 INFO:tasks.workunit.client.0.smithi096.stdout:9/364: getdents d1/d4/d9/d43/d57/d5b/d73 0 2024-04-22T15:18:47.715 INFO:tasks.workunit.client.1.smithi196.stdout:9/54: fsync db/fc 0 2024-04-22T15:18:47.717 INFO:tasks.workunit.client.1.smithi196.stdout:9/55: dread f5 [0,4194304] 0 2024-04-22T15:18:47.718 INFO:tasks.workunit.client.0.smithi096.stdout:1/635: dread - d6/dc/d8a/d3f/d54/d32/d82/fa4 zero size 2024-04-22T15:18:47.718 INFO:tasks.workunit.client.1.smithi196.stdout:2/103: dread d1/d13/f1e [0,4194304] 0 2024-04-22T15:18:47.719 INFO:tasks.workunit.client.0.smithi096.stdout:7/713: dwrite d31/d32/d33/d44/f80 [4194304,4194304] 0 2024-04-22T15:18:47.721 INFO:tasks.workunit.client.1.smithi196.stdout:0/63: symlink d3/de/d14/l17 0 2024-04-22T15:18:47.722 INFO:tasks.workunit.client.1.smithi196.stdout:0/64: write d3/f11 [1822600,107183] 0 2024-04-22T15:18:47.723 INFO:tasks.workunit.client.0.smithi096.stdout:5/476: truncate d4/d2c/d2f/f3a 2359843 0 2024-04-22T15:18:47.723 INFO:tasks.workunit.client.0.smithi096.stdout:0/589: dwrite d7/de/d5b/d95/d15/f36 [0,4194304] 0 2024-04-22T15:18:47.725 INFO:tasks.workunit.client.1.smithi196.stdout:4/63: rename d6/da to d6/d14 0 2024-04-22T15:18:47.725 INFO:tasks.workunit.client.1.smithi196.stdout:9/56: dwrite f5 [0,4194304] 0 2024-04-22T15:18:47.728 INFO:tasks.workunit.client.1.smithi196.stdout:3/36: mknod d0/c5 0 2024-04-22T15:18:47.729 INFO:tasks.workunit.client.0.smithi096.stdout:6/405: creat d3/d3a/d75/d83/f8a x:0 0 0 2024-04-22T15:18:47.734 INFO:tasks.workunit.client.1.smithi196.stdout:4/64: dread f5 [0,4194304] 0 2024-04-22T15:18:47.734 INFO:tasks.workunit.client.0.smithi096.stdout:8/409: rename d4/d1c/c3f to d4/d1c/d40/c93 0 2024-04-22T15:18:47.735 INFO:tasks.workunit.client.1.smithi196.stdout:5/35: link d1/d9/lc d1/ld 0 2024-04-22T15:18:47.738 INFO:tasks.workunit.client.1.smithi196.stdout:7/49: creat fd x:0 0 0 2024-04-22T15:18:47.739 INFO:tasks.workunit.client.1.smithi196.stdout:1/40: write d1/f7 [205596,65604] 0 2024-04-22T15:18:47.740 INFO:tasks.workunit.client.1.smithi196.stdout:1/41: chown d1/c6 143706978 1 2024-04-22T15:18:47.741 INFO:tasks.workunit.client.0.smithi096.stdout:9/365: unlink d1/l29 0 2024-04-22T15:18:47.742 INFO:tasks.workunit.client.0.smithi096.stdout:9/366: stat d1/df/l15 0 2024-04-22T15:18:47.747 INFO:tasks.workunit.client.1.smithi196.stdout:8/93: truncate f0 535213 0 2024-04-22T15:18:47.748 INFO:tasks.workunit.client.0.smithi096.stdout:2/477: getdents d3/db/df/d24 0 2024-04-22T15:18:47.750 INFO:tasks.workunit.client.0.smithi096.stdout:9/367: dread d1/d4/f4e [0,4194304] 0 2024-04-22T15:18:47.751 INFO:tasks.workunit.client.1.smithi196.stdout:0/65: creat d3/de/f18 x:0 0 0 2024-04-22T15:18:47.753 INFO:tasks.workunit.client.0.smithi096.stdout:0/590: dwrite d7/d42/fb2 [0,4194304] 0 2024-04-22T15:18:47.755 INFO:tasks.workunit.client.1.smithi196.stdout:9/57: creat db/ff x:0 0 0 2024-04-22T15:18:47.757 INFO:tasks.workunit.client.1.smithi196.stdout:3/37: mkdir d0/d6 0 2024-04-22T15:18:47.758 INFO:tasks.workunit.client.1.smithi196.stdout:3/38: chown d0/f2 6351 1 2024-04-22T15:18:47.762 INFO:tasks.workunit.client.1.smithi196.stdout:5/36: rename d1/d2/f8 to d1/fe 0 2024-04-22T15:18:47.763 INFO:tasks.workunit.client.0.smithi096.stdout:5/477: symlink d4/d2c/l9c 0 2024-04-22T15:18:47.764 INFO:tasks.workunit.client.0.smithi096.stdout:5/478: fdatasync d4/d11/d25/f90 0 2024-04-22T15:18:47.766 INFO:tasks.workunit.client.1.smithi196.stdout:4/65: dwrite f5 [4194304,4194304] 0 2024-04-22T15:18:47.770 INFO:tasks.workunit.client.0.smithi096.stdout:3/359: write da/db/d12/f56 [1017578,117633] 0 2024-04-22T15:18:47.771 INFO:tasks.workunit.client.0.smithi096.stdout:6/406: truncate d3/d3a/d75/d83/f86 984741 0 2024-04-22T15:18:47.773 INFO:tasks.workunit.client.1.smithi196.stdout:8/94: mkdir d2/d5/d1a 0 2024-04-22T15:18:47.774 INFO:tasks.workunit.client.0.smithi096.stdout:0/591: mknod d7/de/d5b/d95/d66/cc6 0 2024-04-22T15:18:47.776 INFO:tasks.workunit.client.1.smithi196.stdout:0/66: truncate d3/f5 1544533 0 2024-04-22T15:18:47.776 INFO:tasks.workunit.client.1.smithi196.stdout:0/67: readlink d3/de/d14/l17 0 2024-04-22T15:18:47.779 INFO:tasks.workunit.client.0.smithi096.stdout:8/410: rename d4/d10/d4d/f85 to d4/d1c/d40/d48/f94 0 2024-04-22T15:18:47.779 INFO:tasks.workunit.client.0.smithi096.stdout:3/360: dwrite da/db/d12/d22/d26/d52/f6c [0,4194304] 0 2024-04-22T15:18:47.779 INFO:tasks.workunit.client.1.smithi196.stdout:7/50: write f4 [1064511,676] 0 2024-04-22T15:18:47.786 INFO:tasks.workunit.client.0.smithi096.stdout:7/714: creat d31/d32/fe6 x:0 0 0 2024-04-22T15:18:47.788 INFO:tasks.workunit.client.1.smithi196.stdout:1/42: getdents d1/dd 0 2024-04-22T15:18:47.789 INFO:tasks.workunit.client.0.smithi096.stdout:9/368: mkdir d1/d4/d7b 0 2024-04-22T15:18:47.790 INFO:tasks.workunit.client.1.smithi196.stdout:4/66: unlink d6/c8 0 2024-04-22T15:18:47.791 INFO:tasks.workunit.client.0.smithi096.stdout:6/407: creat d3/d28/dd/d11/d2c/d44/d51/f8b x:0 0 0 2024-04-22T15:18:47.791 INFO:tasks.workunit.client.0.smithi096.stdout:5/479: dwrite d4/d11/d3d/d4d/d59/d5a/f67 [0,4194304] 0 2024-04-22T15:18:47.792 INFO:tasks.workunit.client.0.smithi096.stdout:6/408: readlink d3/d28/l10 0 2024-04-22T15:18:47.795 INFO:tasks.workunit.client.0.smithi096.stdout:9/369: dread d1/d4/d9/d43/d57/d5b/f66 [0,4194304] 0 2024-04-22T15:18:47.795 INFO:tasks.workunit.client.1.smithi196.stdout:0/68: symlink d3/d6/dc/l19 0 2024-04-22T15:18:47.796 INFO:tasks.workunit.client.1.smithi196.stdout:9/58: creat db/dd/de/f10 x:0 0 0 2024-04-22T15:18:47.797 INFO:tasks.workunit.client.1.smithi196.stdout:9/59: write fa [1145226,79552] 0 2024-04-22T15:18:47.797 INFO:tasks.workunit.client.1.smithi196.stdout:7/51: creat fe x:0 0 0 2024-04-22T15:18:47.798 INFO:tasks.workunit.client.0.smithi096.stdout:3/361: creat da/db/de/d19/f6f x:0 0 0 2024-04-22T15:18:47.799 INFO:tasks.workunit.client.1.smithi196.stdout:3/39: fdatasync d0/f4 0 2024-04-22T15:18:47.800 INFO:tasks.workunit.client.0.smithi096.stdout:0/592: mknod d7/de/d3a/cc7 0 2024-04-22T15:18:47.801 INFO:tasks.workunit.client.1.smithi196.stdout:5/37: truncate d1/d2/f5 97761 0 2024-04-22T15:18:47.804 INFO:tasks.workunit.client.1.smithi196.stdout:0/69: dwrite d3/fd [0,4194304] 0 2024-04-22T15:18:47.806 INFO:tasks.workunit.client.0.smithi096.stdout:2/478: dread d3/db/df/d10/f37 [0,4194304] 0 2024-04-22T15:18:47.806 INFO:tasks.workunit.client.1.smithi196.stdout:6/62: getdents dd 0 2024-04-22T15:18:47.808 INFO:tasks.workunit.client.1.smithi196.stdout:6/63: truncate fc 1206035 0 2024-04-22T15:18:47.808 INFO:tasks.workunit.client.1.smithi196.stdout:3/40: dwrite d0/f2 [0,4194304] 0 2024-04-22T15:18:47.810 INFO:tasks.workunit.client.0.smithi096.stdout:3/362: rename da/db/de/d19/d2d/f5c to da/db/de/d19/d3e/f70 0 2024-04-22T15:18:47.811 INFO:tasks.workunit.client.1.smithi196.stdout:5/38: dwrite d1/d4/f6 [0,4194304] 0 2024-04-22T15:18:47.811 INFO:tasks.workunit.client.1.smithi196.stdout:7/52: creat ff x:0 0 0 2024-04-22T15:18:47.812 INFO:tasks.workunit.client.0.smithi096.stdout:0/593: symlink d7/de/d5b/d95/d66/lc8 0 2024-04-22T15:18:47.812 INFO:tasks.workunit.client.1.smithi196.stdout:6/64: dread fa [0,4194304] 0 2024-04-22T15:18:47.813 INFO:tasks.workunit.client.1.smithi196.stdout:9/60: write f5 [1767684,60639] 0 2024-04-22T15:18:47.815 INFO:tasks.workunit.client.0.smithi096.stdout:7/715: read d31/d32/d36/f71 [754008,82227] 0 2024-04-22T15:18:47.817 INFO:tasks.workunit.client.1.smithi196.stdout:1/43: link d1/f8 d1/db/f13 0 2024-04-22T15:18:47.817 INFO:tasks.workunit.client.1.smithi196.stdout:3/41: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:18:47.817 INFO:tasks.workunit.client.0.smithi096.stdout:7/716: write d31/d32/d33/fb5 [386887,122142] 0 2024-04-22T15:18:47.818 INFO:tasks.workunit.client.1.smithi196.stdout:2/104: sync 2024-04-22T15:18:47.819 INFO:tasks.workunit.client.0.smithi096.stdout:2/479: rmdir d3/db/d2d/d46/d69 39 2024-04-22T15:18:47.819 INFO:tasks.workunit.client.1.smithi196.stdout:1/44: chown d1/db/dc 3 1 2024-04-22T15:18:47.819 INFO:tasks.workunit.client.1.smithi196.stdout:4/67: mkdir d6/d13/d15 0 2024-04-22T15:18:47.819 INFO:tasks.workunit.client.0.smithi096.stdout:8/411: link d4/d1c/l1e d4/d10/d25/d38/l95 0 2024-04-22T15:18:47.822 INFO:tasks.workunit.client.0.smithi096.stdout:0/594: dwrite d7/de/d5b/d95/dc/d16/d18/f39 [0,4194304] 0 2024-04-22T15:18:47.826 INFO:tasks.workunit.client.1.smithi196.stdout:6/65: dwrite f5 [0,4194304] 0 2024-04-22T15:18:47.826 INFO:tasks.workunit.client.1.smithi196.stdout:5/39: creat d1/d2/ff x:0 0 0 2024-04-22T15:18:47.829 INFO:tasks.workunit.client.1.smithi196.stdout:9/61: dwrite db/fc [0,4194304] 0 2024-04-22T15:18:47.832 INFO:tasks.workunit.client.1.smithi196.stdout:5/40: truncate d1/d2/ff 1028746 0 2024-04-22T15:18:47.834 INFO:tasks.workunit.client.1.smithi196.stdout:5/41: chown d1/d9/lc 52 1 2024-04-22T15:18:47.834 INFO:tasks.workunit.client.1.smithi196.stdout:3/42: dwrite d0/f1 [4194304,4194304] 0 2024-04-22T15:18:47.836 INFO:tasks.workunit.client.0.smithi096.stdout:5/480: dread d4/d2c/d2f/f3a [0,4194304] 0 2024-04-22T15:18:47.837 INFO:tasks.workunit.client.1.smithi196.stdout:5/42: chown d1/d9/la 623796 1 2024-04-22T15:18:47.840 INFO:tasks.workunit.client.1.smithi196.stdout:8/95: rmdir d2 39 2024-04-22T15:18:47.847 INFO:tasks.workunit.client.1.smithi196.stdout:4/68: creat d6/d13/f16 x:0 0 0 2024-04-22T15:18:47.847 INFO:tasks.workunit.client.1.smithi196.stdout:4/69: chown d6 36150187 1 2024-04-22T15:18:47.848 INFO:tasks.workunit.client.0.smithi096.stdout:2/480: fdatasync d3/fd 0 2024-04-22T15:18:47.849 INFO:tasks.workunit.client.1.smithi196.stdout:4/70: write d6/d13/f16 [591132,87694] 0 2024-04-22T15:18:47.849 INFO:tasks.workunit.client.1.smithi196.stdout:4/71: fsync d6/ff 0 2024-04-22T15:18:47.849 INFO:tasks.workunit.client.1.smithi196.stdout:4/72: stat d6/c9 0 2024-04-22T15:18:47.850 INFO:tasks.workunit.client.1.smithi196.stdout:1/45: creat d1/db/dc/de/f14 x:0 0 0 2024-04-22T15:18:47.852 INFO:tasks.workunit.client.0.smithi096.stdout:0/595: mkdir d7/de/d5b/d64/dc9 0 2024-04-22T15:18:47.856 INFO:tasks.workunit.client.0.smithi096.stdout:8/412: dread d4/d10/f21 [0,4194304] 0 2024-04-22T15:18:47.858 INFO:tasks.workunit.client.1.smithi196.stdout:7/53: link fe f10 0 2024-04-22T15:18:47.858 INFO:tasks.workunit.client.0.smithi096.stdout:8/413: fsync d4/d10/d25/d38/f54 0 2024-04-22T15:18:47.858 INFO:tasks.workunit.client.1.smithi196.stdout:7/54: write f7 [638323,35877] 0 2024-04-22T15:18:47.860 INFO:tasks.workunit.client.1.smithi196.stdout:6/66: creat dd/f11 x:0 0 0 2024-04-22T15:18:47.861 INFO:tasks.workunit.client.1.smithi196.stdout:6/67: write fc [2079371,15953] 0 2024-04-22T15:18:47.862 INFO:tasks.workunit.client.1.smithi196.stdout:9/62: mkdir db/d11 0 2024-04-22T15:18:47.866 INFO:tasks.workunit.client.1.smithi196.stdout:3/43: mkdir d0/d7 0 2024-04-22T15:18:47.872 INFO:tasks.workunit.client.0.smithi096.stdout:9/370: write d1/d4/d9/dd/f23 [1204032,106729] 0 2024-04-22T15:18:47.873 INFO:tasks.workunit.client.0.smithi096.stdout:5/481: mknod d4/d2c/c9d 0 2024-04-22T15:18:47.873 INFO:tasks.workunit.client.0.smithi096.stdout:9/371: chown d1/d6f/l74 0 1 2024-04-22T15:18:47.880 INFO:tasks.workunit.client.0.smithi096.stdout:7/717: getdents d31/d32/d36/d5d 0 2024-04-22T15:18:47.881 INFO:tasks.workunit.client.1.smithi196.stdout:1/46: mknod d1/db/dc/de/c15 0 2024-04-22T15:18:47.882 INFO:tasks.workunit.client.1.smithi196.stdout:1/47: write d1/db/ff [4538255,107887] 0 2024-04-22T15:18:47.883 INFO:tasks.workunit.client.0.smithi096.stdout:6/409: truncate d3/f22 1802630 0 2024-04-22T15:18:47.884 INFO:tasks.workunit.client.0.smithi096.stdout:0/596: creat d7/d8f/fca x:0 0 0 2024-04-22T15:18:47.885 INFO:tasks.workunit.client.0.smithi096.stdout:0/597: dread - d7/de/d5b/d95/fc2 zero size 2024-04-22T15:18:47.886 INFO:tasks.workunit.client.1.smithi196.stdout:7/55: symlink l11 0 2024-04-22T15:18:47.889 INFO:tasks.workunit.client.1.smithi196.stdout:1/48: dwrite d1/f4 [0,4194304] 0 2024-04-22T15:18:47.891 INFO:tasks.workunit.client.1.smithi196.stdout:6/68: fsync dd/ff 0 2024-04-22T15:18:47.893 INFO:tasks.workunit.client.0.smithi096.stdout:9/372: unlink d1/d25/l28 0 2024-04-22T15:18:47.894 INFO:tasks.workunit.client.0.smithi096.stdout:2/481: dread d3/d1f/d2f/f5d [0,4194304] 0 2024-04-22T15:18:47.895 INFO:tasks.workunit.client.0.smithi096.stdout:2/482: stat d3/db/d2d 0 2024-04-22T15:18:47.896 INFO:tasks.workunit.client.0.smithi096.stdout:7/718: mkdir d31/d32/d33/d6a/de7 0 2024-04-22T15:18:47.898 INFO:tasks.workunit.client.0.smithi096.stdout:4/473: sync 2024-04-22T15:18:47.899 INFO:tasks.workunit.client.1.smithi196.stdout:8/96: mknod d2/dc/c1b 0 2024-04-22T15:18:47.901 INFO:tasks.workunit.client.0.smithi096.stdout:3/363: write da/db/de/d19/f30 [7810226,96517] 0 2024-04-22T15:18:47.904 INFO:tasks.workunit.client.1.smithi196.stdout:0/70: dread d3/f5 [0,4194304] 0 2024-04-22T15:18:47.905 INFO:tasks.workunit.client.1.smithi196.stdout:4/73: symlink d6/l17 0 2024-04-22T15:18:47.905 INFO:tasks.workunit.client.1.smithi196.stdout:4/74: chown d6/d10 19642529 1 2024-04-22T15:18:47.907 INFO:tasks.workunit.client.0.smithi096.stdout:9/373: rmdir d1/d4/d9/d43 39 2024-04-22T15:18:47.909 INFO:tasks.workunit.client.0.smithi096.stdout:3/364: dwrite da/d3f/f5b [0,4194304] 0 2024-04-22T15:18:47.910 INFO:tasks.workunit.client.0.smithi096.stdout:6/410: dread d3/d28/f1c [0,4194304] 0 2024-04-22T15:18:47.910 INFO:tasks.workunit.client.1.smithi196.stdout:4/75: dwrite d6/d14/fb [0,4194304] 0 2024-04-22T15:18:47.913 INFO:tasks.workunit.client.0.smithi096.stdout:7/719: unlink l26 0 2024-04-22T15:18:47.914 INFO:tasks.workunit.client.0.smithi096.stdout:7/720: chown d31/d68/d98/fd1 361666 1 2024-04-22T15:18:47.914 INFO:tasks.workunit.client.1.smithi196.stdout:4/76: dread d6/d14/fb [4194304,4194304] 0 2024-04-22T15:18:47.914 INFO:tasks.workunit.client.1.smithi196.stdout:7/56: symlink l12 0 2024-04-22T15:18:47.915 INFO:tasks.workunit.client.0.smithi096.stdout:9/374: dwrite d1/f38 [0,4194304] 0 2024-04-22T15:18:47.918 INFO:tasks.workunit.client.0.smithi096.stdout:0/598: symlink d7/de/d5b/d95/lcb 0 2024-04-22T15:18:47.922 INFO:tasks.workunit.client.1.smithi196.stdout:4/77: dwrite d6/d14/f12 [0,4194304] 0 2024-04-22T15:18:47.923 INFO:tasks.workunit.client.0.smithi096.stdout:4/474: unlink d5/d7/l10 0 2024-04-22T15:18:47.925 INFO:tasks.workunit.client.1.smithi196.stdout:2/105: sync 2024-04-22T15:18:47.926 INFO:tasks.workunit.client.0.smithi096.stdout:5/482: rmdir d4/d11/d3d/d4d/d93 0 2024-04-22T15:18:47.928 INFO:tasks.workunit.client.1.smithi196.stdout:6/69: creat dd/f12 x:0 0 0 2024-04-22T15:18:47.931 INFO:tasks.workunit.client.1.smithi196.stdout:2/106: write d1/f7 [1937078,49870] 0 2024-04-22T15:18:47.933 INFO:tasks.workunit.client.1.smithi196.stdout:6/70: dread dd/fe [0,4194304] 0 2024-04-22T15:18:47.933 INFO:tasks.workunit.client.1.smithi196.stdout:3/44: mkdir d0/d7/d8 0 2024-04-22T15:18:47.935 INFO:tasks.workunit.client.0.smithi096.stdout:6/411: mknod d3/d3a/d75/c8c 0 2024-04-22T15:18:47.935 INFO:tasks.workunit.client.1.smithi196.stdout:4/78: dwrite d6/d14/fb [0,4194304] 0 2024-04-22T15:18:47.937 INFO:tasks.workunit.client.0.smithi096.stdout:6/412: chown d3/d28/dd/d11/d2c/d6f/f89 2072844 1 2024-04-22T15:18:47.938 INFO:tasks.workunit.client.1.smithi196.stdout:3/45: dread d0/f4 [0,4194304] 0 2024-04-22T15:18:47.938 INFO:tasks.workunit.client.0.smithi096.stdout:7/721: creat d31/d32/d33/d54/d43/fe8 x:0 0 0 2024-04-22T15:18:47.939 INFO:tasks.workunit.client.0.smithi096.stdout:7/722: fdatasync d31/d32/d7c/fac 0 2024-04-22T15:18:47.941 INFO:tasks.workunit.client.1.smithi196.stdout:5/43: dwrite d1/d2/f5 [0,4194304] 0 2024-04-22T15:18:47.942 INFO:tasks.workunit.client.1.smithi196.stdout:0/71: unlink c2 0 2024-04-22T15:18:47.942 INFO:tasks.workunit.client.0.smithi096.stdout:8/414: write d4/d10/f4e [408120,50930] 0 2024-04-22T15:18:47.945 INFO:tasks.workunit.client.0.smithi096.stdout:9/375: fdatasync d1/d4/f4e 0 2024-04-22T15:18:47.949 INFO:tasks.workunit.client.0.smithi096.stdout:4/475: dread d5/d7/f9 [0,4194304] 0 2024-04-22T15:18:47.949 INFO:tasks.workunit.client.0.smithi096.stdout:7/723: dread d31/d32/d33/d54/d43/f48 [0,4194304] 0 2024-04-22T15:18:47.950 INFO:tasks.workunit.client.0.smithi096.stdout:7/724: dread - d31/d32/d36/d5d/fd9 zero size 2024-04-22T15:18:47.953 INFO:tasks.workunit.client.1.smithi196.stdout:7/57: creat f13 x:0 0 0 2024-04-22T15:18:47.953 INFO:tasks.workunit.client.1.smithi196.stdout:7/58: dread - ff zero size 2024-04-22T15:18:47.955 INFO:tasks.workunit.client.0.smithi096.stdout:3/365: mkdir da/d71 0 2024-04-22T15:18:47.965 INFO:tasks.workunit.client.1.smithi196.stdout:9/63: truncate f1 8216103 0 2024-04-22T15:18:47.965 INFO:tasks.workunit.client.1.smithi196.stdout:6/71: mknod dd/c13 0 2024-04-22T15:18:47.967 INFO:tasks.workunit.client.1.smithi196.stdout:8/97: mknod d2/d5/d1a/c1c 0 2024-04-22T15:18:47.967 INFO:tasks.workunit.client.1.smithi196.stdout:8/98: fsync d2/dc/fd 0 2024-04-22T15:18:47.969 INFO:tasks.workunit.client.1.smithi196.stdout:8/99: write d2/dc/f10 [431214,123198] 0 2024-04-22T15:18:47.969 INFO:tasks.workunit.client.1.smithi196.stdout:4/79: mkdir d6/d14/d18 0 2024-04-22T15:18:47.970 INFO:tasks.workunit.client.1.smithi196.stdout:4/80: dread - d6/ff zero size 2024-04-22T15:18:47.970 INFO:tasks.workunit.client.0.smithi096.stdout:0/599: creat d7/de/d5b/d64/dc9/fcc x:0 0 0 2024-04-22T15:18:47.973 INFO:tasks.workunit.client.0.smithi096.stdout:4/476: unlink d5/d7/d1a/d25/f44 0 2024-04-22T15:18:47.974 INFO:tasks.workunit.client.1.smithi196.stdout:3/46: creat d0/d7/f9 x:0 0 0 2024-04-22T15:18:47.976 INFO:tasks.workunit.client.1.smithi196.stdout:8/100: dwrite d2/dc/f18 [0,4194304] 0 2024-04-22T15:18:47.976 INFO:tasks.workunit.client.0.smithi096.stdout:7/725: rmdir d31 39 2024-04-22T15:18:47.977 INFO:tasks.workunit.client.1.smithi196.stdout:8/101: write d2/dc/fe [4888124,50827] 0 2024-04-22T15:18:47.978 INFO:tasks.workunit.client.1.smithi196.stdout:0/72: mknod d3/de/c1a 0 2024-04-22T15:18:47.978 INFO:tasks.workunit.client.1.smithi196.stdout:8/102: stat d2 0 2024-04-22T15:18:47.981 INFO:tasks.workunit.client.0.smithi096.stdout:3/366: write da/db/de/d19/d3e/f70 [3346145,62319] 0 2024-04-22T15:18:47.981 INFO:tasks.workunit.client.0.smithi096.stdout:3/367: readlink da/db/de/l25 0 2024-04-22T15:18:47.982 INFO:tasks.workunit.client.1.smithi196.stdout:7/59: unlink l11 0 2024-04-22T15:18:47.983 INFO:tasks.workunit.client.1.smithi196.stdout:7/60: write fa [745509,49881] 0 2024-04-22T15:18:47.983 INFO:tasks.workunit.client.1.smithi196.stdout:7/61: chown f2 0 1 2024-04-22T15:18:47.985 INFO:tasks.workunit.client.1.smithi196.stdout:7/62: dread f2 [0,4194304] 0 2024-04-22T15:18:47.988 INFO:tasks.workunit.client.0.smithi096.stdout:8/415: mknod d4/d1c/c96 0 2024-04-22T15:18:47.989 INFO:tasks.workunit.client.1.smithi196.stdout:2/107: mkdir d1/d13/d25/d2a 0 2024-04-22T15:18:47.993 INFO:tasks.workunit.client.1.smithi196.stdout:6/72: unlink dd/ff 0 2024-04-22T15:18:47.994 INFO:tasks.workunit.client.0.smithi096.stdout:9/376: dwrite d1/d4/f11 [0,4194304] 0 2024-04-22T15:18:47.994 INFO:tasks.workunit.client.0.smithi096.stdout:0/600: creat d7/de/d5b/d95/dc/d16/d65/fcd x:0 0 0 2024-04-22T15:18:47.997 INFO:tasks.workunit.client.0.smithi096.stdout:4/477: stat d5/d7/f20 0 2024-04-22T15:18:47.998 INFO:tasks.workunit.client.0.smithi096.stdout:4/478: fdatasync d5/d7/d1a/f90 0 2024-04-22T15:18:47.999 INFO:tasks.workunit.client.0.smithi096.stdout:4/479: chown d5/d5b/l78 13599387 1 2024-04-22T15:18:48.000 INFO:tasks.workunit.client.0.smithi096.stdout:4/480: fsync d5/d7/d2c/d3d/f75 0 2024-04-22T15:18:48.001 INFO:tasks.workunit.client.0.smithi096.stdout:4/481: write d5/d7/d2c/f65 [471269,96101] 0 2024-04-22T15:18:48.003 INFO:tasks.workunit.client.0.smithi096.stdout:3/368: creat da/db/d12/d22/d68/f72 x:0 0 0 2024-04-22T15:18:48.005 INFO:tasks.workunit.client.1.smithi196.stdout:5/44: dwrite f0 [0,4194304] 0 2024-04-22T15:18:48.005 INFO:tasks.workunit.client.0.smithi096.stdout:6/413: link d3/d28/dd/d11/l1b d3/d3a/d75/d83/l8d 0 2024-04-22T15:18:48.006 INFO:tasks.workunit.client.1.smithi196.stdout:3/47: creat d0/d7/fa x:0 0 0 2024-04-22T15:18:48.008 INFO:tasks.workunit.client.0.smithi096.stdout:8/416: write d4/d71/f77 [113537,68343] 0 2024-04-22T15:18:48.008 INFO:tasks.workunit.client.1.smithi196.stdout:3/48: write d0/f1 [1546679,84484] 0 2024-04-22T15:18:48.009 INFO:tasks.workunit.client.0.smithi096.stdout:8/417: read - d4/d10/d25/d4c/d76/d7f/f92 zero size 2024-04-22T15:18:48.011 INFO:tasks.workunit.client.1.smithi196.stdout:3/49: write d0/f2 [2961107,52338] 0 2024-04-22T15:18:48.013 INFO:tasks.workunit.client.1.smithi196.stdout:3/50: dread - d0/d7/f9 zero size 2024-04-22T15:18:48.013 INFO:tasks.workunit.client.1.smithi196.stdout:3/51: readlink - no filename 2024-04-22T15:18:48.013 INFO:tasks.workunit.client.1.smithi196.stdout:5/45: dwrite d1/d4/f6 [0,4194304] 0 2024-04-22T15:18:48.016 INFO:tasks.workunit.client.0.smithi096.stdout:5/483: creat d4/d1f/f9e x:0 0 0 2024-04-22T15:18:48.018 INFO:tasks.workunit.client.1.smithi196.stdout:1/49: getdents d1/db/dc/de 0 2024-04-22T15:18:48.018 INFO:tasks.workunit.client.0.smithi096.stdout:4/482: creat d5/d5b/fa4 x:0 0 0 2024-04-22T15:18:48.020 INFO:tasks.workunit.client.0.smithi096.stdout:3/369: creat da/db/de/d19/f73 x:0 0 0 2024-04-22T15:18:48.021 INFO:tasks.workunit.client.1.smithi196.stdout:9/64: symlink db/d11/l12 0 2024-04-22T15:18:48.022 INFO:tasks.workunit.client.1.smithi196.stdout:1/50: dread - d1/db/dc/de/f14 zero size 2024-04-22T15:18:48.024 INFO:tasks.workunit.client.1.smithi196.stdout:3/52: dwrite d0/f2 [0,4194304] 0 2024-04-22T15:18:48.025 INFO:tasks.workunit.client.1.smithi196.stdout:9/65: write f4 [171961,46483] 0 2024-04-22T15:18:48.025 INFO:tasks.workunit.client.1.smithi196.stdout:6/73: creat dd/f14 x:0 0 0 2024-04-22T15:18:48.026 INFO:tasks.workunit.client.0.smithi096.stdout:9/377: symlink d1/d4/d9/d43/l7c 0 2024-04-22T15:18:48.026 INFO:tasks.workunit.client.1.smithi196.stdout:1/51: dread - d1/db/dc/de/f14 zero size 2024-04-22T15:18:48.027 INFO:tasks.workunit.client.0.smithi096.stdout:8/418: rmdir d4/d10/d25/d38 39 2024-04-22T15:18:48.027 INFO:tasks.workunit.client.1.smithi196.stdout:5/46: dwrite d1/d2/f5 [4194304,4194304] 0 2024-04-22T15:18:48.029 INFO:tasks.workunit.client.1.smithi196.stdout:3/53: write d0/f2 [307518,54565] 0 2024-04-22T15:18:48.029 INFO:tasks.workunit.client.1.smithi196.stdout:1/52: read d1/db/ff [719781,13883] 0 2024-04-22T15:18:48.029 INFO:tasks.workunit.client.1.smithi196.stdout:5/47: stat d1/d9 0 2024-04-22T15:18:48.031 INFO:tasks.workunit.client.0.smithi096.stdout:0/601: dwrite d7/de/d5b/d95/dc/f2f [0,4194304] 0 2024-04-22T15:18:48.031 INFO:tasks.workunit.client.0.smithi096.stdout:7/726: mkdir d31/d32/dc0/ddc/de9 0 2024-04-22T15:18:48.031 INFO:tasks.workunit.client.1.smithi196.stdout:5/48: write d1/d2/f5 [2944908,92846] 0 2024-04-22T15:18:48.036 INFO:tasks.workunit.client.1.smithi196.stdout:6/74: dread f4 [0,4194304] 0 2024-04-22T15:18:48.038 INFO:tasks.workunit.client.0.smithi096.stdout:6/414: mknod d3/d28/dd/c8e 0 2024-04-22T15:18:48.039 INFO:tasks.workunit.client.0.smithi096.stdout:8/419: dwrite d4/d1c/d40/d46/d58/f60 [0,4194304] 0 2024-04-22T15:18:48.041 INFO:tasks.workunit.client.0.smithi096.stdout:9/378: creat d1/d25/f7d x:0 0 0 2024-04-22T15:18:48.041 INFO:tasks.workunit.client.1.smithi196.stdout:5/49: dwrite f0 [0,4194304] 0 2024-04-22T15:18:48.042 INFO:tasks.workunit.client.0.smithi096.stdout:0/602: dread d7/d42/fb2 [0,4194304] 0 2024-04-22T15:18:48.057 INFO:tasks.workunit.client.1.smithi196.stdout:4/81: creat d6/f19 x:0 0 0 2024-04-22T15:18:48.058 INFO:tasks.workunit.client.0.smithi096.stdout:6/415: symlink d3/d3a/d75/d83/l8f 0 2024-04-22T15:18:48.059 INFO:tasks.workunit.client.1.smithi196.stdout:9/66: symlink db/d11/l13 0 2024-04-22T15:18:48.059 INFO:tasks.workunit.client.1.smithi196.stdout:9/67: write db/ff [708968,48919] 0 2024-04-22T15:18:48.067 INFO:tasks.workunit.client.0.smithi096.stdout:1/636: sync 2024-04-22T15:18:48.067 INFO:tasks.workunit.client.1.smithi196.stdout:1/53: unlink d1/f7 0 2024-04-22T15:18:48.068 INFO:tasks.workunit.client.1.smithi196.stdout:1/54: stat d1/f8 0 2024-04-22T15:18:48.068 INFO:tasks.workunit.client.0.smithi096.stdout:3/370: dread da/db/dc/f16 [0,4194304] 0 2024-04-22T15:18:48.069 INFO:tasks.workunit.client.0.smithi096.stdout:5/484: creat d4/d2c/f9f x:0 0 0 2024-04-22T15:18:48.072 INFO:tasks.workunit.client.0.smithi096.stdout:3/371: readlink da/db/l54 0 2024-04-22T15:18:48.072 INFO:tasks.workunit.client.0.smithi096.stdout:4/483: creat d5/d7/d2c/d3d/fa5 x:0 0 0 2024-04-22T15:18:48.072 INFO:tasks.workunit.client.1.smithi196.stdout:8/103: creat d2/f1d x:0 0 0 2024-04-22T15:18:48.074 INFO:tasks.workunit.client.1.smithi196.stdout:6/75: creat dd/f15 x:0 0 0 2024-04-22T15:18:48.075 INFO:tasks.workunit.client.0.smithi096.stdout:7/727: truncate d31/d32/d33/f34 2505869 0 2024-04-22T15:18:48.075 INFO:tasks.workunit.client.0.smithi096.stdout:6/416: dwrite d3/d28/dd/d11/d2c/d44/f67 [0,4194304] 0 2024-04-22T15:18:48.076 INFO:tasks.workunit.client.1.smithi196.stdout:5/50: mknod d1/d9/c10 0 2024-04-22T15:18:48.077 INFO:tasks.workunit.client.1.smithi196.stdout:2/108: sync 2024-04-22T15:18:48.079 INFO:tasks.workunit.client.0.smithi096.stdout:8/420: dwrite d4/d10/d25/d38/f69 [0,4194304] 0 2024-04-22T15:18:48.080 INFO:tasks.workunit.client.1.smithi196.stdout:5/51: dread d1/d2/f5 [0,4194304] 0 2024-04-22T15:18:48.082 INFO:tasks.workunit.client.0.smithi096.stdout:0/603: dwrite d7/de/d5b/d95/dc/d16/d65/d7c/fbd [0,4194304] 0 2024-04-22T15:18:48.085 INFO:tasks.workunit.client.1.smithi196.stdout:9/68: creat db/f14 x:0 0 0 2024-04-22T15:18:48.088 INFO:tasks.workunit.client.0.smithi096.stdout:0/604: truncate d7/de/d5b/d62/fab 844322 0 2024-04-22T15:18:48.090 INFO:tasks.workunit.client.0.smithi096.stdout:3/372: dwrite da/db/d12/d22/d26/d52/f6c [4194304,4194304] 0 2024-04-22T15:18:48.091 INFO:tasks.workunit.client.0.smithi096.stdout:0/605: readlink d7/de/d5b/d62/d67/lb8 0 2024-04-22T15:18:48.093 INFO:tasks.workunit.client.1.smithi196.stdout:3/54: mkdir d0/d6/db 0 2024-04-22T15:18:48.094 INFO:tasks.workunit.client.1.smithi196.stdout:9/69: dread - db/f14 zero size 2024-04-22T15:18:48.098 INFO:tasks.workunit.client.1.smithi196.stdout:8/104: mknod d2/d5/d1a/c1e 0 2024-04-22T15:18:48.098 INFO:tasks.workunit.client.1.smithi196.stdout:8/105: fsync d2/f3 0 2024-04-22T15:18:48.102 INFO:tasks.workunit.client.1.smithi196.stdout:8/106: dread d2/dc/f18 [0,4194304] 0 2024-04-22T15:18:48.103 INFO:tasks.workunit.client.1.smithi196.stdout:8/107: write d2/dc/f10 [336814,129536] 0 2024-04-22T15:18:48.104 INFO:tasks.workunit.client.1.smithi196.stdout:8/108: fdatasync d2/f3 0 2024-04-22T15:18:48.105 INFO:tasks.workunit.client.1.smithi196.stdout:0/73: getdents d3/d6 0 2024-04-22T15:18:48.108 INFO:tasks.workunit.client.1.smithi196.stdout:7/63: getdents . 0 2024-04-22T15:18:48.108 INFO:tasks.workunit.client.1.smithi196.stdout:4/82: mknod d6/d13/d15/c1a 0 2024-04-22T15:18:48.110 INFO:tasks.workunit.client.0.smithi096.stdout:2/483: sync 2024-04-22T15:18:48.111 INFO:tasks.workunit.client.1.smithi196.stdout:0/74: dwrite d3/fd [0,4194304] 0 2024-04-22T15:18:48.113 INFO:tasks.workunit.client.1.smithi196.stdout:0/75: truncate d3/de/f18 100735 0 2024-04-22T15:18:48.127 INFO:tasks.workunit.client.1.smithi196.stdout:6/76: write f4 [4596509,39841] 0 2024-04-22T15:18:48.128 INFO:tasks.workunit.client.1.smithi196.stdout:6/77: dread - dd/f15 zero size 2024-04-22T15:18:48.128 INFO:tasks.workunit.client.1.smithi196.stdout:6/78: write fc [2370917,77924] 0 2024-04-22T15:18:48.136 INFO:tasks.workunit.client.0.smithi096.stdout:5/485: write d4/fd [6131067,41177] 0 2024-04-22T15:18:48.137 INFO:tasks.workunit.client.0.smithi096.stdout:5/486: dread - d4/d2c/d2f/d31/f9a zero size 2024-04-22T15:18:48.141 INFO:tasks.workunit.client.0.smithi096.stdout:4/484: creat d5/d7/d1f/fa6 x:0 0 0 2024-04-22T15:18:48.148 INFO:tasks.workunit.client.0.smithi096.stdout:7/728: creat d31/d68/fea x:0 0 0 2024-04-22T15:18:48.154 INFO:tasks.workunit.client.0.smithi096.stdout:9/379: mknod d1/df/c7e 0 2024-04-22T15:18:48.154 INFO:tasks.workunit.client.0.smithi096.stdout:7/729: dwrite d31/d68/d98/fd1 [0,4194304] 0 2024-04-22T15:18:48.157 INFO:tasks.workunit.client.0.smithi096.stdout:8/421: creat d4/d10/f97 x:0 0 0 2024-04-22T15:18:48.159 INFO:tasks.workunit.client.0.smithi096.stdout:0/606: unlink d7/de/d5b/d95/c77 0 2024-04-22T15:18:48.162 INFO:tasks.workunit.client.0.smithi096.stdout:0/607: truncate d7/de/d3a/d61/d79/fa4 569857 0 2024-04-22T15:18:48.172 INFO:tasks.workunit.client.0.smithi096.stdout:1/637: fsync d6/dc/d8a/d3f/d54/d32/d8c/d45/d7b/f7e 0 2024-04-22T15:18:48.179 INFO:tasks.workunit.client.0.smithi096.stdout:9/380: rmdir d1/d4 39 2024-04-22T15:18:48.180 INFO:tasks.workunit.client.0.smithi096.stdout:7/730: dread d31/d32/d33/d54/f64 [0,4194304] 0 2024-04-22T15:18:48.181 INFO:tasks.workunit.client.0.smithi096.stdout:7/731: chown d31/d32/ca3 3694818 1 2024-04-22T15:18:48.184 INFO:tasks.workunit.client.0.smithi096.stdout:8/422: fdatasync d4/d1c/d40/d48/f94 0 2024-04-22T15:18:48.186 INFO:tasks.workunit.client.0.smithi096.stdout:3/373: mknod da/db/d12/d67/c74 0 2024-04-22T15:18:48.188 INFO:tasks.workunit.client.0.smithi096.stdout:4/485: rmdir d5/d7/d1f 39 2024-04-22T15:18:48.189 INFO:tasks.workunit.client.0.smithi096.stdout:2/484: mknod d3/db/df/d98/c99 0 2024-04-22T15:18:48.191 INFO:tasks.workunit.client.0.smithi096.stdout:0/608: mkdir d7/de/d5b/d95/d66/dce 0 2024-04-22T15:18:48.194 INFO:tasks.workunit.client.0.smithi096.stdout:5/487: mkdir d4/d2c/d2f/d31/d82/da0 0 2024-04-22T15:18:48.202 INFO:tasks.workunit.client.0.smithi096.stdout:7/732: creat d31/d32/d33/d93/feb x:0 0 0 2024-04-22T15:18:48.203 INFO:tasks.workunit.client.0.smithi096.stdout:7/733: stat d31/d32/d7c/fac 0 2024-04-22T15:18:48.209 INFO:tasks.workunit.client.0.smithi096.stdout:4/486: dread - d5/d7/d1f/d84/fa0 zero size 2024-04-22T15:18:48.209 INFO:tasks.workunit.client.0.smithi096.stdout:4/487: chown d5/f5e 75120882 1 2024-04-22T15:18:48.210 INFO:tasks.workunit.client.0.smithi096.stdout:4/488: chown d5/d7/d2c/d3d/d4f/d88 0 1 2024-04-22T15:18:48.210 INFO:tasks.workunit.client.0.smithi096.stdout:2/485: readlink d3/db/d2d/d46/d59/l64 0 2024-04-22T15:18:48.211 INFO:tasks.workunit.client.0.smithi096.stdout:4/489: fdatasync d5/d7/d2c/d3d/d4f/f91 0 2024-04-22T15:18:48.212 INFO:tasks.workunit.client.0.smithi096.stdout:0/609: symlink d7/de/d5b/d95/dc/d16/d18/lcf 0 2024-04-22T15:18:48.214 INFO:tasks.workunit.client.0.smithi096.stdout:5/488: dread d4/d11/d3d/d4d/d59/d5a/f79 [0,4194304] 0 2024-04-22T15:18:48.216 INFO:tasks.workunit.client.0.smithi096.stdout:1/638: creat d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b/dba/fc8 x:0 0 0 2024-04-22T15:18:48.219 INFO:tasks.workunit.client.0.smithi096.stdout:9/381: chown d1/d4/d9/c51 432437762 1 2024-04-22T15:18:48.225 INFO:tasks.workunit.client.0.smithi096.stdout:4/490: dread d5/d7/d1a/f6c [4194304,4194304] 0 2024-04-22T15:18:48.234 INFO:tasks.workunit.client.0.smithi096.stdout:6/417: link d3/d3a/l52 d3/d28/d54/d68/d6e/l90 0 2024-04-22T15:18:48.238 INFO:tasks.workunit.client.0.smithi096.stdout:8/423: creat d4/d1c/d40/f98 x:0 0 0 2024-04-22T15:18:48.243 INFO:tasks.workunit.client.0.smithi096.stdout:4/491: mknod d5/d7/d2c/d3d/d4f/d9c/ca7 0 2024-04-22T15:18:48.247 INFO:tasks.workunit.client.0.smithi096.stdout:5/489: creat d4/d11/d3d/d4d/d99/fa1 x:0 0 0 2024-04-22T15:18:48.249 INFO:tasks.workunit.client.0.smithi096.stdout:7/734: write d31/d32/f4c [4762268,118391] 0 2024-04-22T15:18:48.254 INFO:tasks.workunit.client.0.smithi096.stdout:0/610: write d7/f4d [1225881,124700] 0 2024-04-22T15:18:48.255 INFO:tasks.workunit.client.0.smithi096.stdout:3/374: getdents da/db/dc/d28 0 2024-04-22T15:18:48.256 INFO:tasks.workunit.client.0.smithi096.stdout:6/418: dwrite d3/d28/dd/d11/d2c/d3b/f73 [0,4194304] 0 2024-04-22T15:18:48.257 INFO:tasks.workunit.client.0.smithi096.stdout:2/486: link d3/db/d30/l78 d3/d1f/d5a/l9a 0 2024-04-22T15:18:48.258 INFO:tasks.workunit.client.0.smithi096.stdout:1/639: write d6/dc/f23 [5003106,17504] 0 2024-04-22T15:18:48.263 INFO:tasks.workunit.client.0.smithi096.stdout:5/490: rmdir d4/d11/d3d/d4d/d59 39 2024-04-22T15:18:48.269 INFO:tasks.workunit.client.0.smithi096.stdout:9/382: rename d1/d4/d9/d43/d57/d5b/d78 to d1/d4/d7f 0 2024-04-22T15:18:48.281 INFO:tasks.workunit.client.0.smithi096.stdout:2/487: creat d3/db/df/d24/d58/d5f/d6e/f9b x:0 0 0 2024-04-22T15:18:48.281 INFO:tasks.workunit.client.0.smithi096.stdout:4/492: write d5/d7/d1a/f40 [4946091,99617] 0 2024-04-22T15:18:48.282 INFO:tasks.workunit.client.0.smithi096.stdout:2/488: stat d3/db/d2d 0 2024-04-22T15:18:48.283 INFO:tasks.workunit.client.0.smithi096.stdout:2/489: write d3/db/d2d/d46/f7c [100215,129496] 0 2024-04-22T15:18:48.289 INFO:tasks.workunit.client.0.smithi096.stdout:8/424: truncate d4/d71/f77 4327 0 2024-04-22T15:18:48.290 INFO:tasks.workunit.client.0.smithi096.stdout:8/425: fsync d4/d1c/d34/f4f 0 2024-04-22T15:18:48.293 INFO:tasks.workunit.client.0.smithi096.stdout:6/419: write d3/d28/f17 [3190268,36713] 0 2024-04-22T15:18:48.298 INFO:tasks.workunit.client.0.smithi096.stdout:4/493: symlink d5/d7/d1f/d84/d6d/d98/la8 0 2024-04-22T15:18:48.301 INFO:tasks.workunit.client.0.smithi096.stdout:1/640: dwrite d6/dc/f41 [4194304,4194304] 0 2024-04-22T15:18:48.301 INFO:tasks.workunit.client.0.smithi096.stdout:2/490: fdatasync d3/db/d2d/d46/d59/f79 0 2024-04-22T15:18:48.303 INFO:tasks.workunit.client.0.smithi096.stdout:7/735: creat d31/d32/fec x:0 0 0 2024-04-22T15:18:48.309 INFO:tasks.workunit.client.0.smithi096.stdout:2/491: dwrite d3/db/d30/f3e [0,4194304] 0 2024-04-22T15:18:48.310 INFO:tasks.workunit.client.0.smithi096.stdout:8/426: creat d4/d10/f99 x:0 0 0 2024-04-22T15:18:48.312 INFO:tasks.workunit.client.0.smithi096.stdout:9/383: truncate d1/f20 1641152 0 2024-04-22T15:18:48.313 INFO:tasks.workunit.client.0.smithi096.stdout:7/736: dwrite d31/d32/d33/d5e/f63 [0,4194304] 0 2024-04-22T15:18:48.315 INFO:tasks.workunit.client.0.smithi096.stdout:8/427: write d4/d10/d4d/f84 [295887,99625] 0 2024-04-22T15:18:48.318 INFO:tasks.workunit.client.0.smithi096.stdout:0/611: creat d7/de/d5b/d95/dc/fd0 x:0 0 0 2024-04-22T15:18:48.320 INFO:tasks.workunit.client.0.smithi096.stdout:0/612: dread - d7/de/d5b/d64/dc9/fcc zero size 2024-04-22T15:18:48.321 INFO:tasks.workunit.client.0.smithi096.stdout:4/494: mknod d5/d7/d2c/d67/ca9 0 2024-04-22T15:18:48.321 INFO:tasks.workunit.client.0.smithi096.stdout:4/495: chown d5/d7/f7b 0 1 2024-04-22T15:18:48.322 INFO:tasks.workunit.client.0.smithi096.stdout:9/384: dread d1/f32 [0,4194304] 0 2024-04-22T15:18:48.322 INFO:tasks.workunit.client.0.smithi096.stdout:9/385: chown d1/d4/d9/dd/f21 91888681 1 2024-04-22T15:18:48.323 INFO:tasks.workunit.client.0.smithi096.stdout:9/386: write d1/d4/f42 [1460051,18386] 0 2024-04-22T15:18:48.324 INFO:tasks.workunit.client.0.smithi096.stdout:9/387: read d1/d4/d9/f12 [1116774,103475] 0 2024-04-22T15:18:48.335 INFO:tasks.workunit.client.0.smithi096.stdout:9/388: dread d1/d4/f22 [0,4194304] 0 2024-04-22T15:18:48.337 INFO:tasks.workunit.client.0.smithi096.stdout:3/375: link da/db/dc/l50 da/l75 0 2024-04-22T15:18:48.343 INFO:tasks.workunit.client.1.smithi196.stdout:9/70: mkdir db/dd/de/d15 0 2024-04-22T15:18:48.344 INFO:tasks.workunit.client.0.smithi096.stdout:4/496: mknod d5/d7/d1a/d25/caa 0 2024-04-22T15:18:48.348 INFO:tasks.workunit.client.1.smithi196.stdout:9/71: dwrite f5 [0,4194304] 0 2024-04-22T15:18:48.354 INFO:tasks.workunit.client.0.smithi096.stdout:5/491: getdents d4/d1f 0 2024-04-22T15:18:48.357 INFO:tasks.workunit.client.1.smithi196.stdout:9/72: dwrite db/ff [0,4194304] 0 2024-04-22T15:18:48.359 INFO:tasks.workunit.client.1.smithi196.stdout:8/109: symlink d2/d5/d1a/l1f 0 2024-04-22T15:18:48.361 INFO:tasks.workunit.client.0.smithi096.stdout:2/492: write d3/db/df/d10/f11 [1222314,122176] 0 2024-04-22T15:18:48.362 INFO:tasks.workunit.client.0.smithi096.stdout:9/389: dread d1/f6 [0,4194304] 0 2024-04-22T15:18:48.363 INFO:tasks.workunit.client.0.smithi096.stdout:0/613: write d7/f78 [725000,81126] 0 2024-04-22T15:18:48.364 INFO:tasks.workunit.client.0.smithi096.stdout:0/614: chown d7/de/d5b/d95/d15/l5a 469988 1 2024-04-22T15:18:48.365 INFO:tasks.workunit.client.0.smithi096.stdout:6/420: rename d3/d28/dd/d11/d2c/d44/f67 to d3/d28/dd/d11/d2c/f91 0 2024-04-22T15:18:48.370 INFO:tasks.workunit.client.0.smithi096.stdout:6/421: readlink d3/d3a/d75/d83/l8d 0 2024-04-22T15:18:48.370 INFO:tasks.workunit.client.0.smithi096.stdout:0/615: truncate d7/de/d5b/d95/dc/d16/f5d 5072213 0 2024-04-22T15:18:48.370 INFO:tasks.workunit.client.0.smithi096.stdout:0/616: truncate d7/de/d5b/d95/dc/d26/fb5 300427 0 2024-04-22T15:18:48.371 INFO:tasks.workunit.client.1.smithi196.stdout:4/83: creat d6/d13/f1b x:0 0 0 2024-04-22T15:18:48.371 INFO:tasks.workunit.client.1.smithi196.stdout:4/84: readlink d6/d10/ld 0 2024-04-22T15:18:48.371 INFO:tasks.workunit.client.1.smithi196.stdout:4/85: readlink d6/d10/ld 0 2024-04-22T15:18:48.371 INFO:tasks.workunit.client.1.smithi196.stdout:0/76: symlink d3/d6/dc/l1b 0 2024-04-22T15:18:48.371 INFO:tasks.workunit.client.0.smithi096.stdout:0/617: chown d7/de/d5b/d95/dc/d26/c27 172228 1 2024-04-22T15:18:48.372 INFO:tasks.workunit.client.1.smithi196.stdout:4/86: dread f5 [4194304,4194304] 0 2024-04-22T15:18:48.372 INFO:tasks.workunit.client.0.smithi096.stdout:4/497: creat d5/d7/d2c/d67/fab x:0 0 0 2024-04-22T15:18:48.373 INFO:tasks.workunit.client.0.smithi096.stdout:0/618: write d7/d42/d4c/f6a [1150246,35989] 0 2024-04-22T15:18:48.375 INFO:tasks.workunit.client.1.smithi196.stdout:6/79: creat dd/f16 x:0 0 0 2024-04-22T15:18:48.377 INFO:tasks.workunit.client.1.smithi196.stdout:3/55: mkdir d0/d6/db/dc 0 2024-04-22T15:18:48.378 INFO:tasks.workunit.client.1.smithi196.stdout:3/56: dread - d0/d7/fa zero size 2024-04-22T15:18:48.379 INFO:tasks.workunit.client.0.smithi096.stdout:4/498: dwrite d5/fc [4194304,4194304] 0 2024-04-22T15:18:48.381 INFO:tasks.workunit.client.0.smithi096.stdout:5/492: truncate d4/d11/d3d/d4d/d59/d5a/f79 4702638 0 2024-04-22T15:18:48.381 INFO:tasks.workunit.client.1.smithi196.stdout:1/55: rmdir d1/dd 0 2024-04-22T15:18:48.382 INFO:tasks.workunit.client.0.smithi096.stdout:7/737: creat d31/d32/fed x:0 0 0 2024-04-22T15:18:48.388 INFO:tasks.workunit.client.1.smithi196.stdout:2/109: creat d1/f2b x:0 0 0 2024-04-22T15:18:48.389 INFO:tasks.workunit.client.1.smithi196.stdout:9/73: symlink db/dd/de/l16 0 2024-04-22T15:18:48.392 INFO:tasks.workunit.client.1.smithi196.stdout:8/110: creat d2/dc/f20 x:0 0 0 2024-04-22T15:18:48.392 INFO:tasks.workunit.client.1.smithi196.stdout:7/64: getdents . 0 2024-04-22T15:18:48.393 INFO:tasks.workunit.client.0.smithi096.stdout:0/619: creat d7/da7/fd1 x:0 0 0 2024-04-22T15:18:48.394 INFO:tasks.workunit.client.0.smithi096.stdout:0/620: readlink d7/de/d5b/d62/l72 0 2024-04-22T15:18:48.394 INFO:tasks.workunit.client.1.smithi196.stdout:0/77: mkdir d3/de/d1c 0 2024-04-22T15:18:48.395 INFO:tasks.workunit.client.0.smithi096.stdout:1/641: getdents d6/dc 0 2024-04-22T15:18:48.396 INFO:tasks.workunit.client.0.smithi096.stdout:0/621: dread - d7/de/d3a/d61/d79/fbe zero size 2024-04-22T15:18:48.397 INFO:tasks.workunit.client.0.smithi096.stdout:4/499: unlink d5/d7/d1a/d25/f4d 0 2024-04-22T15:18:48.397 INFO:tasks.workunit.client.1.smithi196.stdout:9/74: dwrite f4 [0,4194304] 0 2024-04-22T15:18:48.397 INFO:tasks.workunit.client.1.smithi196.stdout:7/65: dread f7 [0,4194304] 0 2024-04-22T15:18:48.398 INFO:tasks.workunit.client.1.smithi196.stdout:7/66: dread - fd zero size 2024-04-22T15:18:48.398 INFO:tasks.workunit.client.0.smithi096.stdout:5/493: symlink d4/d11/d3d/d4d/d99/la2 0 2024-04-22T15:18:48.399 INFO:tasks.workunit.client.1.smithi196.stdout:3/57: fdatasync d0/f4 0 2024-04-22T15:18:48.400 INFO:tasks.workunit.client.0.smithi096.stdout:7/738: unlink d31/d32/d33/d54/d43/l4b 0 2024-04-22T15:18:48.402 INFO:tasks.workunit.client.0.smithi096.stdout:3/376: link l1 da/d71/l76 0 2024-04-22T15:18:48.403 INFO:tasks.workunit.client.1.smithi196.stdout:1/56: mkdir d1/db/d16 0 2024-04-22T15:18:48.404 INFO:tasks.workunit.client.0.smithi096.stdout:3/377: truncate da/db/d12/d63/f65 746979 0 2024-04-22T15:18:48.404 INFO:tasks.workunit.client.0.smithi096.stdout:2/493: mknod d3/c9c 0 2024-04-22T15:18:48.408 INFO:tasks.workunit.client.1.smithi196.stdout:5/52: getdents d1/d9 0 2024-04-22T15:18:48.409 INFO:tasks.workunit.client.0.smithi096.stdout:6/422: creat d3/d28/dd/d11/d2c/d44/d51/d78/f92 x:0 0 0 2024-04-22T15:18:48.410 INFO:tasks.workunit.client.0.smithi096.stdout:0/622: fdatasync d7/de/f12 0 2024-04-22T15:18:48.411 INFO:tasks.workunit.client.0.smithi096.stdout:0/623: readlink d7/de/d5b/d95/lcb 0 2024-04-22T15:18:48.413 INFO:tasks.workunit.client.0.smithi096.stdout:4/500: rename d5/d7/d1a/d8b/fa2 to d5/d7/d1f/fac 0 2024-04-22T15:18:48.416 INFO:tasks.workunit.client.1.smithi196.stdout:0/78: creat d3/de/f1d x:0 0 0 2024-04-22T15:18:48.417 INFO:tasks.workunit.client.0.smithi096.stdout:5/494: mknod d4/d11/d3d/d4f/ca3 0 2024-04-22T15:18:48.419 INFO:tasks.workunit.client.0.smithi096.stdout:7/739: read d31/d32/d33/d54/d43/f48 [316738,70322] 0 2024-04-22T15:18:48.421 INFO:tasks.workunit.client.1.smithi196.stdout:7/67: mkdir d14 0 2024-04-22T15:18:48.421 INFO:tasks.workunit.client.1.smithi196.stdout:6/80: dwrite f5 [4194304,4194304] 0 2024-04-22T15:18:48.422 INFO:tasks.workunit.client.0.smithi096.stdout:9/390: dread d1/df/f14 [0,4194304] 0 2024-04-22T15:18:48.422 INFO:tasks.workunit.client.0.smithi096.stdout:3/378: creat da/db/d12/f77 x:0 0 0 2024-04-22T15:18:48.422 INFO:tasks.workunit.client.0.smithi096.stdout:2/494: rmdir d3/db/d2d/d46/d59 39 2024-04-22T15:18:48.423 INFO:tasks.workunit.client.0.smithi096.stdout:3/379: dread - da/db/d12/d22/f53 zero size 2024-04-22T15:18:48.424 INFO:tasks.workunit.client.0.smithi096.stdout:3/380: fdatasync da/d3f/f5b 0 2024-04-22T15:18:48.424 INFO:tasks.workunit.client.0.smithi096.stdout:6/423: creat d3/d3a/d75/f93 x:0 0 0 2024-04-22T15:18:48.426 INFO:tasks.workunit.client.0.smithi096.stdout:7/740: dread d31/d32/d36/f70 [0,4194304] 0 2024-04-22T15:18:48.429 INFO:tasks.workunit.client.1.smithi196.stdout:8/111: symlink d2/d8/l21 0 2024-04-22T15:18:48.432 INFO:tasks.workunit.client.0.smithi096.stdout:6/424: dwrite d3/d28/dd/d11/d2c/d3b/f73 [0,4194304] 0 2024-04-22T15:18:48.435 INFO:tasks.workunit.client.1.smithi196.stdout:6/81: creat dd/f17 x:0 0 0 2024-04-22T15:18:48.436 INFO:tasks.workunit.client.1.smithi196.stdout:6/82: write dd/f16 [216106,42977] 0 2024-04-22T15:18:48.439 INFO:tasks.workunit.client.0.smithi096.stdout:2/495: mknod d3/db/df/d24/d58/d5f/d6e/c9d 0 2024-04-22T15:18:48.440 INFO:tasks.workunit.client.1.smithi196.stdout:3/58: symlink d0/d7/d8/ld 0 2024-04-22T15:18:48.440 INFO:tasks.workunit.client.0.smithi096.stdout:6/425: fsync d3/d28/dd/d11/d2c/d44/d51/d78/f92 0 2024-04-22T15:18:48.441 INFO:tasks.workunit.client.0.smithi096.stdout:0/624: mkdir d7/d42/d4c/dd2 0 2024-04-22T15:18:48.441 INFO:tasks.workunit.client.0.smithi096.stdout:6/426: fdatasync d3/d3a/d75/f93 0 2024-04-22T15:18:48.442 INFO:tasks.workunit.client.1.smithi196.stdout:2/110: creat d1/f2c x:0 0 0 2024-04-22T15:18:48.443 INFO:tasks.workunit.client.0.smithi096.stdout:0/625: write d7/de/d5b/d95/dc/d16/d65/f84 [4348182,62242] 0 2024-04-22T15:18:48.445 INFO:tasks.workunit.client.0.smithi096.stdout:7/741: fdatasync f11 0 2024-04-22T15:18:48.448 INFO:tasks.workunit.client.1.smithi196.stdout:2/111: dwrite d1/f2b [0,4194304] 0 2024-04-22T15:18:48.449 INFO:tasks.workunit.client.0.smithi096.stdout:6/427: dwrite d3/d28/d4d/f87 [0,4194304] 0 2024-04-22T15:18:48.450 INFO:tasks.workunit.client.1.smithi196.stdout:8/112: rename d2/d5/d1a/c1e to d2/dc/c22 0 2024-04-22T15:18:48.451 INFO:tasks.workunit.client.1.smithi196.stdout:8/113: fdatasync d2/f1d 0 2024-04-22T15:18:48.451 INFO:tasks.workunit.client.1.smithi196.stdout:8/114: fsync d2/dc/f10 0 2024-04-22T15:18:48.454 INFO:tasks.workunit.client.1.smithi196.stdout:0/79: mknod d3/de/d1c/c1e 0 2024-04-22T15:18:48.454 INFO:tasks.workunit.client.1.smithi196.stdout:5/53: write d1/fe [7035253,71717] 0 2024-04-22T15:18:48.454 INFO:tasks.workunit.client.1.smithi196.stdout:0/80: stat d3 0 2024-04-22T15:18:48.457 INFO:tasks.workunit.client.0.smithi096.stdout:1/642: truncate d6/dc/d8a/d3f/d54/d32/d3b/f9e 1252212 0 2024-04-22T15:18:48.458 INFO:tasks.workunit.client.1.smithi196.stdout:5/54: dread d1/d2/ff [0,4194304] 0 2024-04-22T15:18:48.459 INFO:tasks.workunit.client.1.smithi196.stdout:1/57: dwrite d1/f8 [0,4194304] 0 2024-04-22T15:18:48.459 INFO:tasks.workunit.client.1.smithi196.stdout:7/68: mkdir d14/d15 0 2024-04-22T15:18:48.460 INFO:tasks.workunit.client.1.smithi196.stdout:1/58: chown d1 32122075 1 2024-04-22T15:18:48.460 INFO:tasks.workunit.client.1.smithi196.stdout:7/69: dread - fe zero size 2024-04-22T15:18:48.460 INFO:tasks.workunit.client.1.smithi196.stdout:1/59: readlink d1/la 0 2024-04-22T15:18:48.460 INFO:tasks.workunit.client.1.smithi196.stdout:7/70: stat d14/d15 0 2024-04-22T15:18:48.462 INFO:tasks.workunit.client.1.smithi196.stdout:6/83: creat dd/f18 x:0 0 0 2024-04-22T15:18:48.462 INFO:tasks.workunit.client.1.smithi196.stdout:1/60: write d1/db/dc/de/f14 [699184,16661] 0 2024-04-22T15:18:48.462 INFO:tasks.workunit.client.1.smithi196.stdout:6/84: readlink lb 0 2024-04-22T15:18:48.463 INFO:tasks.workunit.client.0.smithi096.stdout:9/391: symlink d1/l80 0 2024-04-22T15:18:48.464 INFO:tasks.workunit.client.1.smithi196.stdout:3/59: mkdir d0/d7/d8/de 0 2024-04-22T15:18:48.470 INFO:tasks.workunit.client.0.smithi096.stdout:9/392: dwrite d1/d4/f42 [0,4194304] 0 2024-04-22T15:18:48.471 INFO:tasks.workunit.client.0.smithi096.stdout:0/626: creat d7/de/d5b/d64/fd3 x:0 0 0 2024-04-22T15:18:48.472 INFO:tasks.workunit.client.0.smithi096.stdout:0/627: readlink d7/de/d5b/d95/dc/l2d 0 2024-04-22T15:18:48.480 INFO:tasks.workunit.client.1.smithi196.stdout:9/75: link c6 db/c17 0 2024-04-22T15:18:48.480 INFO:tasks.workunit.client.1.smithi196.stdout:9/76: chown f7 1603663 1 2024-04-22T15:18:48.481 INFO:tasks.workunit.client.1.smithi196.stdout:9/77: write db/dd/de/f10 [528010,15432] 0 2024-04-22T15:18:48.482 INFO:tasks.workunit.client.1.smithi196.stdout:0/81: rename d3/de/d1c/c1e to d3/d6/dc/c1f 0 2024-04-22T15:18:48.484 INFO:tasks.workunit.client.0.smithi096.stdout:6/428: mkdir d3/d28/dd/d11/d2c/d6f/d7c/d94 0 2024-04-22T15:18:48.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:48 smithi096 ceph-mon[26688]: pgmap v30: 65 pgs: 65 active+clean; 1017 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 25 MiB/s rd, 123 MiB/s wr, 299 op/s 2024-04-22T15:18:48.486 INFO:tasks.workunit.client.1.smithi196.stdout:4/87: sync 2024-04-22T15:18:48.486 INFO:tasks.workunit.client.1.smithi196.stdout:5/55: creat d1/d9/f11 x:0 0 0 2024-04-22T15:18:48.486 INFO:tasks.workunit.client.0.smithi096.stdout:4/501: link d5/l87 d5/d7/d1f/d84/lad 0 2024-04-22T15:18:48.489 INFO:tasks.workunit.client.0.smithi096.stdout:5/495: rename d4/d2c/d2f/f51 to d4/d11/d3d/d4d/fa4 0 2024-04-22T15:18:48.489 INFO:tasks.workunit.client.1.smithi196.stdout:9/78: dwrite db/f14 [0,4194304] 0 2024-04-22T15:18:48.490 INFO:tasks.workunit.client.0.smithi096.stdout:5/496: read d4/d1f/f60 [281222,101620] 0 2024-04-22T15:18:48.491 INFO:tasks.workunit.client.0.smithi096.stdout:1/643: chown d6/dc/d8a/d3f/d54/d32/f37 23191395 1 2024-04-22T15:18:48.491 INFO:tasks.workunit.client.0.smithi096.stdout:5/497: dread - d4/d2c/d2f/f81 zero size 2024-04-22T15:18:48.493 INFO:tasks.workunit.client.1.smithi196.stdout:6/85: dread fa [0,4194304] 0 2024-04-22T15:18:48.494 INFO:tasks.workunit.client.1.smithi196.stdout:6/86: write dd/f15 [971653,48101] 0 2024-04-22T15:18:48.495 INFO:tasks.workunit.client.1.smithi196.stdout:6/87: truncate dd/f17 1022837 0 2024-04-22T15:18:48.498 INFO:tasks.workunit.client.1.smithi196.stdout:6/88: dread f4 [0,4194304] 0 2024-04-22T15:18:48.500 INFO:tasks.workunit.client.1.smithi196.stdout:2/112: mknod d1/c2d 0 2024-04-22T15:18:48.500 INFO:tasks.workunit.client.1.smithi196.stdout:2/113: write d1/ff [8156386,64781] 0 2024-04-22T15:18:48.503 INFO:tasks.workunit.client.1.smithi196.stdout:8/115: unlink d2/dc/c22 0 2024-04-22T15:18:48.503 INFO:tasks.workunit.client.1.smithi196.stdout:2/114: dread d1/f1b [0,4194304] 0 2024-04-22T15:18:48.503 INFO:tasks.workunit.client.1.smithi196.stdout:8/116: stat d2/d5/f19 0 2024-04-22T15:18:48.504 INFO:tasks.workunit.client.0.smithi096.stdout:0/628: mkdir d7/d42/dd4 0 2024-04-22T15:18:48.504 INFO:tasks.workunit.client.1.smithi196.stdout:2/115: truncate d1/f12 963379 0 2024-04-22T15:18:48.504 INFO:tasks.workunit.client.0.smithi096.stdout:2/496: write d3/db/f83 [1852781,8492] 0 2024-04-22T15:18:48.506 INFO:tasks.workunit.client.0.smithi096.stdout:3/381: rmdir da/db/d12/d22/d26/d52/d62 0 2024-04-22T15:18:48.508 INFO:tasks.workunit.client.1.smithi196.stdout:1/61: rename d1/db/f13 to d1/db/d16/f17 0 2024-04-22T15:18:48.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:48 smithi196 ceph-mon[34717]: pgmap v30: 65 pgs: 65 active+clean; 1017 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 25 MiB/s rd, 123 MiB/s wr, 299 op/s 2024-04-22T15:18:48.510 INFO:tasks.workunit.client.1.smithi196.stdout:0/82: mknod d3/de/d14/c20 0 2024-04-22T15:18:48.511 INFO:tasks.workunit.client.0.smithi096.stdout:0/629: dwrite d7/d42/f93 [0,4194304] 0 2024-04-22T15:18:48.511 INFO:tasks.workunit.client.0.smithi096.stdout:4/502: creat d5/d7/d1f/d84/d6d/fae x:0 0 0 2024-04-22T15:18:48.511 INFO:tasks.workunit.client.1.smithi196.stdout:0/83: write d3/fd [2437298,130604] 0 2024-04-22T15:18:48.512 INFO:tasks.workunit.client.0.smithi096.stdout:5/498: dread d4/d11/d3d/d4d/d59/d5a/d5c/f71 [0,4194304] 0 2024-04-22T15:18:48.513 INFO:tasks.workunit.client.1.smithi196.stdout:5/56: mknod d1/d4/c12 0 2024-04-22T15:18:48.513 INFO:tasks.workunit.client.0.smithi096.stdout:0/630: chown d7/de/d3a/lc3 2600 1 2024-04-22T15:18:48.515 INFO:tasks.workunit.client.1.smithi196.stdout:0/84: dread d3/fd [0,4194304] 0 2024-04-22T15:18:48.515 INFO:tasks.workunit.client.1.smithi196.stdout:4/88: creat d6/d10/f1c x:0 0 0 2024-04-22T15:18:48.516 INFO:tasks.workunit.client.1.smithi196.stdout:4/89: truncate d6/f19 557141 0 2024-04-22T15:18:48.517 INFO:tasks.workunit.client.1.smithi196.stdout:5/57: dread d1/d4/f6 [0,4194304] 0 2024-04-22T15:18:48.518 INFO:tasks.workunit.client.1.smithi196.stdout:5/58: read f0 [1652189,56888] 0 2024-04-22T15:18:48.521 INFO:tasks.workunit.client.1.smithi196.stdout:5/59: dread f0 [0,4194304] 0 2024-04-22T15:18:48.521 INFO:tasks.workunit.client.1.smithi196.stdout:7/71: getdents d14/d15 0 2024-04-22T15:18:48.523 INFO:tasks.workunit.client.0.smithi096.stdout:9/393: creat d1/d4/d7b/f81 x:0 0 0 2024-04-22T15:18:48.524 INFO:tasks.workunit.client.0.smithi096.stdout:2/497: fdatasync d3/db/df/d10/f3b 0 2024-04-22T15:18:48.527 INFO:tasks.workunit.client.0.smithi096.stdout:3/382: creat da/db/de/d38/f78 x:0 0 0 2024-04-22T15:18:48.527 INFO:tasks.workunit.client.0.smithi096.stdout:3/383: chown da/f34 32 1 2024-04-22T15:18:48.528 INFO:tasks.workunit.client.1.smithi196.stdout:3/60: link d0/f1 d0/d7/d8/de/ff 0 2024-04-22T15:18:48.530 INFO:tasks.workunit.client.1.smithi196.stdout:3/61: chown d0/d7/f9 1854 1 2024-04-22T15:18:48.530 INFO:tasks.workunit.client.1.smithi196.stdout:9/79: dwrite f4 [4194304,4194304] 0 2024-04-22T15:18:48.530 INFO:tasks.workunit.client.1.smithi196.stdout:9/80: chown c3 46 1 2024-04-22T15:18:48.534 INFO:tasks.workunit.client.1.smithi196.stdout:3/62: dread d0/f3 [0,4194304] 0 2024-04-22T15:18:48.534 INFO:tasks.workunit.client.1.smithi196.stdout:9/81: dread db/fc [0,4194304] 0 2024-04-22T15:18:48.534 INFO:tasks.workunit.client.1.smithi196.stdout:3/63: chown d0/d7/fa 803 1 2024-04-22T15:18:48.536 INFO:tasks.workunit.client.0.smithi096.stdout:2/498: dread f0 [0,4194304] 0 2024-04-22T15:18:48.538 INFO:tasks.workunit.client.1.smithi196.stdout:2/116: mkdir d1/d23/d27/d2e 0 2024-04-22T15:18:48.539 INFO:tasks.workunit.client.0.smithi096.stdout:1/644: creat d6/dc/dbd/fc9 x:0 0 0 2024-04-22T15:18:48.542 INFO:tasks.workunit.client.1.smithi196.stdout:1/62: creat d1/db/dc/f18 x:0 0 0 2024-04-22T15:18:48.545 INFO:tasks.workunit.client.0.smithi096.stdout:4/503: getdents d5/d7/d2c/d95 0 2024-04-22T15:18:48.545 INFO:tasks.workunit.client.0.smithi096.stdout:4/504: stat d5/c2a 0 2024-04-22T15:18:48.546 INFO:tasks.workunit.client.1.smithi196.stdout:0/85: creat d3/d9/f21 x:0 0 0 2024-04-22T15:18:48.547 INFO:tasks.workunit.client.1.smithi196.stdout:0/86: truncate d3/de/f18 908208 0 2024-04-22T15:18:48.548 INFO:tasks.workunit.client.0.smithi096.stdout:2/499: rmdir d3/db 39 2024-04-22T15:18:48.550 INFO:tasks.workunit.client.1.smithi196.stdout:0/87: dread d3/ff [0,4194304] 0 2024-04-22T15:18:48.551 INFO:tasks.workunit.client.0.smithi096.stdout:0/631: mkdir d7/de/dd5 0 2024-04-22T15:18:48.551 INFO:tasks.workunit.client.0.smithi096.stdout:4/505: dwrite d5/f4e [0,4194304] 0 2024-04-22T15:18:48.555 INFO:tasks.workunit.client.1.smithi196.stdout:7/72: rename l6 to d14/d15/l16 0 2024-04-22T15:18:48.556 INFO:tasks.workunit.client.1.smithi196.stdout:0/88: dwrite d3/de/f1d [0,4194304] 0 2024-04-22T15:18:48.559 INFO:tasks.workunit.client.0.smithi096.stdout:9/394: dwrite d1/d4/d9/f52 [0,4194304] 0 2024-04-22T15:18:48.560 INFO:tasks.workunit.client.0.smithi096.stdout:6/429: link f1 d3/d28/dd/d11/d2c/f95 0 2024-04-22T15:18:48.566 INFO:tasks.workunit.client.0.smithi096.stdout:5/499: rename d4/d1f/f9e to d4/d11/fa5 0 2024-04-22T15:18:48.567 INFO:tasks.workunit.client.0.smithi096.stdout:5/500: chown d4/d11/d3d/d4d 5576618 1 2024-04-22T15:18:48.572 INFO:tasks.workunit.client.1.smithi196.stdout:5/60: dwrite d1/d2/ff [0,4194304] 0 2024-04-22T15:18:48.573 INFO:tasks.workunit.client.0.smithi096.stdout:9/395: dread - d1/d4/d9/d43/f44 zero size 2024-04-22T15:18:48.574 INFO:tasks.workunit.client.1.smithi196.stdout:8/117: truncate d2/dc/fe 6711613 0 2024-04-22T15:18:48.574 INFO:tasks.workunit.client.0.smithi096.stdout:0/632: dread d7/de/d5b/d95/dc/d16/d18/f92 [0,4194304] 0 2024-04-22T15:18:48.574 INFO:tasks.workunit.client.1.smithi196.stdout:2/117: creat d1/d23/f2f x:0 0 0 2024-04-22T15:18:48.577 INFO:tasks.workunit.client.0.smithi096.stdout:3/384: rename da/db/de/d19/d2d/f51 to da/db/dc/d59/f79 0 2024-04-22T15:18:48.581 INFO:tasks.workunit.client.0.smithi096.stdout:4/506: symlink d5/d7/d2c/d95/laf 0 2024-04-22T15:18:48.582 INFO:tasks.workunit.client.1.smithi196.stdout:4/90: getdents d6/d14/d18 0 2024-04-22T15:18:48.582 INFO:tasks.workunit.client.0.smithi096.stdout:4/507: chown d5/d7/d2c/d3d/d4f/f60 55598 1 2024-04-22T15:18:48.583 INFO:tasks.workunit.client.1.smithi196.stdout:4/91: write d6/ff [323771,20933] 0 2024-04-22T15:18:48.584 INFO:tasks.workunit.client.0.smithi096.stdout:9/396: stat d1/df/c5d 0 2024-04-22T15:18:48.584 INFO:tasks.workunit.client.0.smithi096.stdout:4/508: dread - d5/d7/d2c/f9d zero size 2024-04-22T15:18:48.584 INFO:tasks.workunit.client.1.smithi196.stdout:4/92: chown c1 11713072 1 2024-04-22T15:18:48.584 INFO:tasks.workunit.client.1.smithi196.stdout:4/93: chown d6/d10/fe 1717388891 1 2024-04-22T15:18:48.590 INFO:tasks.workunit.client.0.smithi096.stdout:9/397: dwrite d1/d4/d9/f52 [0,4194304] 0 2024-04-22T15:18:48.590 INFO:tasks.workunit.client.0.smithi096.stdout:0/633: creat d7/da7/fd6 x:0 0 0 2024-04-22T15:18:48.592 INFO:tasks.workunit.client.1.smithi196.stdout:7/73: mknod d14/c17 0 2024-04-22T15:18:48.595 INFO:tasks.workunit.client.1.smithi196.stdout:0/89: mknod d3/d6/dc/c22 0 2024-04-22T15:18:48.600 INFO:tasks.workunit.client.1.smithi196.stdout:3/64: dwrite d0/f4 [0,4194304] 0 2024-04-22T15:18:48.601 INFO:tasks.workunit.client.1.smithi196.stdout:1/63: write d1/db/ff [97855,59246] 0 2024-04-22T15:18:48.602 INFO:tasks.workunit.client.1.smithi196.stdout:7/74: dwrite fc [0,4194304] 0 2024-04-22T15:18:48.603 INFO:tasks.workunit.client.1.smithi196.stdout:7/75: write f4 [4959587,40028] 0 2024-04-22T15:18:48.608 INFO:tasks.workunit.client.0.smithi096.stdout:2/500: dwrite d3/d1f/f56 [0,4194304] 0 2024-04-22T15:18:48.608 INFO:tasks.workunit.client.0.smithi096.stdout:4/509: creat d5/d7/d2c/d3d/d4f/fb0 x:0 0 0 2024-04-22T15:18:48.611 INFO:tasks.workunit.client.1.smithi196.stdout:5/61: creat d1/d9/f13 x:0 0 0 2024-04-22T15:18:48.616 INFO:tasks.workunit.client.0.smithi096.stdout:9/398: dread d1/d4/d9/d43/f5f [4194304,4194304] 0 2024-04-22T15:18:48.616 INFO:tasks.workunit.client.0.smithi096.stdout:9/399: readlink d1/l4d 0 2024-04-22T15:18:48.617 INFO:tasks.workunit.client.1.smithi196.stdout:2/118: rename d1/ca to d1/d13/d25/c30 0 2024-04-22T15:18:48.618 INFO:tasks.workunit.client.0.smithi096.stdout:3/385: write da/db/dc/f16 [5545113,45679] 0 2024-04-22T15:18:48.618 INFO:tasks.workunit.client.0.smithi096.stdout:3/386: chown da/db/d12/d22 97 1 2024-04-22T15:18:48.623 INFO:tasks.workunit.client.1.smithi196.stdout:2/119: dread d1/ff [4194304,4194304] 0 2024-04-22T15:18:48.623 INFO:tasks.workunit.client.1.smithi196.stdout:0/90: unlink d3/d9/cb 0 2024-04-22T15:18:48.623 INFO:tasks.workunit.client.0.smithi096.stdout:4/510: unlink d5/d7/d2c/l6f 0 2024-04-22T15:18:48.624 INFO:tasks.workunit.client.1.smithi196.stdout:9/82: write f1 [5722041,76982] 0 2024-04-22T15:18:48.625 INFO:tasks.workunit.client.0.smithi096.stdout:2/501: creat d3/d1f/d2f/f9e x:0 0 0 2024-04-22T15:18:48.626 INFO:tasks.workunit.client.1.smithi196.stdout:6/89: getdents dd 0 2024-04-22T15:18:48.626 INFO:tasks.workunit.client.1.smithi196.stdout:6/90: stat dd/f15 0 2024-04-22T15:18:48.628 INFO:tasks.workunit.client.0.smithi096.stdout:1/645: rename d6/dc/d8a/d3f/d62/da2 to d6/dc/d8a/d3f/d54/d32/d8c/dca 0 2024-04-22T15:18:48.630 INFO:tasks.workunit.client.1.smithi196.stdout:3/65: mknod d0/c10 0 2024-04-22T15:18:48.630 INFO:tasks.workunit.client.1.smithi196.stdout:3/66: dread - d0/d7/f9 zero size 2024-04-22T15:18:48.631 INFO:tasks.workunit.client.0.smithi096.stdout:9/400: mknod d1/d4/d9/d43/c82 0 2024-04-22T15:18:48.632 INFO:tasks.workunit.client.1.smithi196.stdout:1/64: mknod d1/c19 0 2024-04-22T15:18:48.633 INFO:tasks.workunit.client.0.smithi096.stdout:3/387: unlink da/db/d10/c35 0 2024-04-22T15:18:48.634 INFO:tasks.workunit.client.0.smithi096.stdout:0/634: rmdir d7/de/dd5 0 2024-04-22T15:18:48.636 INFO:tasks.workunit.client.1.smithi196.stdout:8/118: dwrite d2/f4 [0,4194304] 0 2024-04-22T15:18:48.637 INFO:tasks.workunit.client.0.smithi096.stdout:3/388: read da/db/dc/d59/f79 [1964325,122186] 0 2024-04-22T15:18:48.640 INFO:tasks.workunit.client.0.smithi096.stdout:5/501: truncate d4/f7 4179144 0 2024-04-22T15:18:48.642 INFO:tasks.workunit.client.1.smithi196.stdout:4/94: rename d6/ff to d6/d13/f1d 0 2024-04-22T15:18:48.643 INFO:tasks.workunit.client.0.smithi096.stdout:5/502: write d4/d11/d3d/d4d/d59/d5a/f67 [356274,21120] 0 2024-04-22T15:18:48.643 INFO:tasks.workunit.client.1.smithi196.stdout:4/95: truncate d6/d10/f1c 613174 0 2024-04-22T15:18:48.646 INFO:tasks.workunit.client.1.smithi196.stdout:4/96: dread d6/d13/f1d [0,4194304] 0 2024-04-22T15:18:48.646 INFO:tasks.workunit.client.0.smithi096.stdout:3/389: dwrite da/db/d12/d22/f53 [0,4194304] 0 2024-04-22T15:18:48.649 INFO:tasks.workunit.client.0.smithi096.stdout:6/430: rename d3/c42 to d3/d28/dd/d11/d2c/d6f/c96 0 2024-04-22T15:18:48.651 INFO:tasks.workunit.client.1.smithi196.stdout:2/120: mknod d1/d13/d25/c31 0 2024-04-22T15:18:48.654 INFO:tasks.workunit.client.1.smithi196.stdout:6/91: symlink dd/l19 0 2024-04-22T15:18:48.656 INFO:tasks.workunit.client.1.smithi196.stdout:6/92: read fc [747748,84922] 0 2024-04-22T15:18:48.657 INFO:tasks.workunit.client.1.smithi196.stdout:6/93: write dd/f16 [376567,16095] 0 2024-04-22T15:18:48.661 INFO:tasks.workunit.client.0.smithi096.stdout:2/502: write d3/db/d2d/f33 [5711615,44241] 0 2024-04-22T15:18:48.662 INFO:tasks.workunit.client.0.smithi096.stdout:2/503: chown d3/db/f83 1921130 1 2024-04-22T15:18:48.664 INFO:tasks.workunit.client.1.smithi196.stdout:1/65: dwrite d1/db/d16/f17 [4194304,4194304] 0 2024-04-22T15:18:48.665 INFO:tasks.workunit.client.1.smithi196.stdout:8/119: creat d2/d5/f23 x:0 0 0 2024-04-22T15:18:48.666 INFO:tasks.workunit.client.1.smithi196.stdout:1/66: chown d1/db 3754283 1 2024-04-22T15:18:48.669 INFO:tasks.workunit.client.0.smithi096.stdout:2/504: dwrite d3/f87 [0,4194304] 0 2024-04-22T15:18:48.670 INFO:tasks.workunit.client.1.smithi196.stdout:0/91: dwrite d3/fd [4194304,4194304] 0 2024-04-22T15:18:48.672 INFO:tasks.workunit.client.1.smithi196.stdout:1/67: dread d1/f8 [4194304,4194304] 0 2024-04-22T15:18:48.674 INFO:tasks.workunit.client.0.smithi096.stdout:9/401: link d1/df/f58 d1/d4/d7f/f83 0 2024-04-22T15:18:48.676 INFO:tasks.workunit.client.1.smithi196.stdout:4/97: mknod d6/d14/c1e 0 2024-04-22T15:18:48.678 INFO:tasks.workunit.client.0.smithi096.stdout:0/635: creat d7/de/fd7 x:0 0 0 2024-04-22T15:18:48.679 INFO:tasks.workunit.client.1.smithi196.stdout:2/121: mknod d1/d13/d25/c32 0 2024-04-22T15:18:48.681 INFO:tasks.workunit.client.1.smithi196.stdout:2/122: truncate d1/d13/d1f/f24 808785 0 2024-04-22T15:18:48.681 INFO:tasks.workunit.client.1.smithi196.stdout:4/98: dread d6/d13/f16 [0,4194304] 0 2024-04-22T15:18:48.685 INFO:tasks.workunit.client.0.smithi096.stdout:3/390: mknod da/d71/c7a 0 2024-04-22T15:18:48.685 INFO:tasks.workunit.client.0.smithi096.stdout:6/431: dwrite d3/d28/dd/f16 [0,4194304] 0 2024-04-22T15:18:48.686 INFO:tasks.workunit.client.0.smithi096.stdout:3/391: write da/db/d12/d22/d68/f72 [7206,113818] 0 2024-04-22T15:18:48.686 INFO:tasks.workunit.client.0.smithi096.stdout:6/432: read d3/d28/dd/d11/d2c/d44/f77 [714820,43997] 0 2024-04-22T15:18:48.686 INFO:tasks.workunit.client.0.smithi096.stdout:4/511: rename d5/d7/d1a/d25/l29 to d5/d7/d2c/d3d/d4f/d63/lb1 0 2024-04-22T15:18:48.688 INFO:tasks.workunit.client.0.smithi096.stdout:2/505: readlink d3/db/l48 0 2024-04-22T15:18:48.690 INFO:tasks.workunit.client.1.smithi196.stdout:6/94: creat dd/f1a x:0 0 0 2024-04-22T15:18:48.693 INFO:tasks.workunit.client.1.smithi196.stdout:6/95: dread dd/f16 [0,4194304] 0 2024-04-22T15:18:48.693 INFO:tasks.workunit.client.0.smithi096.stdout:1/646: getdents d6 0 2024-04-22T15:18:48.694 INFO:tasks.workunit.client.0.smithi096.stdout:9/402: creat d1/d4/d9/d43/d57/d5b/f84 x:0 0 0 2024-04-22T15:18:48.698 INFO:tasks.workunit.client.1.smithi196.stdout:6/96: dwrite f1 [4194304,4194304] 0 2024-04-22T15:18:48.699 INFO:tasks.workunit.client.1.smithi196.stdout:6/97: chown lb 13 1 2024-04-22T15:18:48.700 INFO:tasks.workunit.client.0.smithi096.stdout:0/636: creat d7/de/d5b/d95/dc/d16/d18/fd8 x:0 0 0 2024-04-22T15:18:48.700 INFO:tasks.workunit.client.0.smithi096.stdout:1/647: dwrite d6/dc/d8a/d3f/f9c [0,4194304] 0 2024-04-22T15:18:48.702 INFO:tasks.workunit.client.0.smithi096.stdout:1/648: readlink d6/dc/d8a/d3f/d54/d32/d3b/d94/lb0 0 2024-04-22T15:18:48.703 INFO:tasks.workunit.client.1.smithi196.stdout:9/83: truncate db/dd/de/f10 113561 0 2024-04-22T15:18:48.710 INFO:tasks.workunit.client.0.smithi096.stdout:0/637: dwrite d7/de/d5b/d95/dc/f56 [0,4194304] 0 2024-04-22T15:18:48.714 INFO:tasks.workunit.client.1.smithi196.stdout:0/92: symlink d3/d6/l23 0 2024-04-22T15:18:48.715 INFO:tasks.workunit.client.0.smithi096.stdout:3/392: creat da/db/dc/f7b x:0 0 0 2024-04-22T15:18:48.716 INFO:tasks.workunit.client.1.smithi196.stdout:1/68: mknod d1/db/dc/de/c1a 0 2024-04-22T15:18:48.717 INFO:tasks.workunit.client.1.smithi196.stdout:1/69: chown d1/db/dc 2238526 1 2024-04-22T15:18:48.718 INFO:tasks.workunit.client.1.smithi196.stdout:1/70: stat d1/db/dc/f18 0 2024-04-22T15:18:48.721 INFO:tasks.workunit.client.1.smithi196.stdout:4/99: mknod d6/d14/c1f 0 2024-04-22T15:18:48.723 INFO:tasks.workunit.client.0.smithi096.stdout:9/403: chown d1/d4/d9/dd/l26 119078 1 2024-04-22T15:18:48.727 INFO:tasks.workunit.client.1.smithi196.stdout:7/76: getdents d14/d15 0 2024-04-22T15:18:48.728 INFO:tasks.workunit.client.0.smithi096.stdout:2/506: dread d3/db/f31 [0,4194304] 0 2024-04-22T15:18:48.728 INFO:tasks.workunit.client.1.smithi196.stdout:7/77: write f13 [353573,122797] 0 2024-04-22T15:18:48.728 INFO:tasks.workunit.client.1.smithi196.stdout:7/78: chown f2 13436055 1 2024-04-22T15:18:48.732 INFO:tasks.workunit.client.1.smithi196.stdout:7/79: dread fa [0,4194304] 0 2024-04-22T15:18:48.733 INFO:tasks.workunit.client.0.smithi096.stdout:0/638: rename d7/de/d5b/d62/fab to d7/de/d5b/d64/fd9 0 2024-04-22T15:18:48.734 INFO:tasks.workunit.client.1.smithi196.stdout:5/62: getdents d1 0 2024-04-22T15:18:48.738 INFO:tasks.workunit.client.0.smithi096.stdout:1/649: dread d6/dc/d8a/d3f/d62/f8d [0,4194304] 0 2024-04-22T15:18:48.738 INFO:tasks.workunit.client.1.smithi196.stdout:6/98: creat dd/f1b x:0 0 0 2024-04-22T15:18:48.738 INFO:tasks.workunit.client.1.smithi196.stdout:6/99: chown dd/f1b 22 1 2024-04-22T15:18:48.739 INFO:tasks.workunit.client.0.smithi096.stdout:1/650: chown d6/dc/d8a/cc7 2332580 1 2024-04-22T15:18:48.739 INFO:tasks.workunit.client.0.smithi096.stdout:6/433: symlink d3/d28/l97 0 2024-04-22T15:18:48.741 INFO:tasks.workunit.client.0.smithi096.stdout:0/639: dwrite d7/de/d5b/d95/dc/d16/d18/f69 [0,4194304] 0 2024-04-22T15:18:48.741 INFO:tasks.workunit.client.0.smithi096.stdout:1/651: readlink d6/dc/d8a/d3f/d62/d7d/lbe 0 2024-04-22T15:18:48.743 INFO:tasks.workunit.client.0.smithi096.stdout:1/652: stat d6/dc/d8a/d3f/d62/d7d/c55 0 2024-04-22T15:18:48.746 INFO:tasks.workunit.client.1.smithi196.stdout:0/93: creat d3/de/f24 x:0 0 0 2024-04-22T15:18:48.748 INFO:tasks.workunit.client.0.smithi096.stdout:5/503: link d4/f3c d4/d2c/d2f/fa6 0 2024-04-22T15:18:48.750 INFO:tasks.workunit.client.1.smithi196.stdout:1/71: mknod d1/db/dc/de/c1b 0 2024-04-22T15:18:48.751 INFO:tasks.workunit.client.0.smithi096.stdout:3/393: mknod da/db/dc/d28/d69/c7c 0 2024-04-22T15:18:48.752 INFO:tasks.workunit.client.1.smithi196.stdout:2/123: mknod d1/d1d/c33 0 2024-04-22T15:18:48.753 INFO:tasks.workunit.client.1.smithi196.stdout:4/100: creat d6/d13/f20 x:0 0 0 2024-04-22T15:18:48.753 INFO:tasks.workunit.client.1.smithi196.stdout:3/67: getdents d0/d7 0 2024-04-22T15:18:48.756 INFO:tasks.workunit.client.0.smithi096.stdout:4/512: link d5/d5b/l77 d5/d7/d1a/d8b/lb2 0 2024-04-22T15:18:48.757 INFO:tasks.workunit.client.0.smithi096.stdout:4/513: truncate d5/d7/d1f/d84/f93 338437 0 2024-04-22T15:18:48.759 INFO:tasks.workunit.client.0.smithi096.stdout:9/404: mkdir d1/d85 0 2024-04-22T15:18:48.763 INFO:tasks.workunit.client.1.smithi196.stdout:6/100: creat dd/f1c x:0 0 0 2024-04-22T15:18:48.765 INFO:tasks.workunit.client.0.smithi096.stdout:9/405: dwrite d1/d25/f7d [0,4194304] 0 2024-04-22T15:18:48.766 INFO:tasks.workunit.client.0.smithi096.stdout:3/394: creat da/db/d12/d22/f7d x:0 0 0 2024-04-22T15:18:48.768 INFO:tasks.workunit.client.1.smithi196.stdout:0/94: mkdir d3/d9/d25 0 2024-04-22T15:18:48.770 INFO:tasks.workunit.client.1.smithi196.stdout:1/72: creat d1/db/d16/f1c x:0 0 0 2024-04-22T15:18:48.770 INFO:tasks.workunit.client.0.smithi096.stdout:0/640: mkdir d7/de/d5b/dda 0 2024-04-22T15:18:48.772 INFO:tasks.workunit.client.1.smithi196.stdout:7/80: write f10 [681896,75060] 0 2024-04-22T15:18:48.772 INFO:tasks.workunit.client.1.smithi196.stdout:2/124: write d1/d13/f19 [862849,32623] 0 2024-04-22T15:18:48.773 INFO:tasks.workunit.client.1.smithi196.stdout:4/101: symlink d6/d14/l21 0 2024-04-22T15:18:48.774 INFO:tasks.workunit.client.0.smithi096.stdout:1/653: mkdir d6/dc/d8a/d3f/d62/dc2/dcb 0 2024-04-22T15:18:48.778 INFO:tasks.workunit.client.0.smithi096.stdout:0/641: dwrite d7/f9e [0,4194304] 0 2024-04-22T15:18:48.782 INFO:tasks.workunit.client.1.smithi196.stdout:3/68: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:18:48.782 INFO:tasks.workunit.client.0.smithi096.stdout:5/504: symlink d4/d2c/d2f/d31/d82/da0/la7 0 2024-04-22T15:18:48.782 INFO:tasks.workunit.client.0.smithi096.stdout:3/395: mkdir da/db/dc/d59/d7e 0 2024-04-22T15:18:48.783 INFO:tasks.workunit.client.1.smithi196.stdout:8/120: link d2/d5/c11 d2/c24 0 2024-04-22T15:18:48.783 INFO:tasks.workunit.client.1.smithi196.stdout:8/121: chown d2/d5/lb 475 1 2024-04-22T15:18:48.783 INFO:tasks.workunit.client.1.smithi196.stdout:8/122: dread - d2/f1d zero size 2024-04-22T15:18:48.787 INFO:tasks.workunit.client.1.smithi196.stdout:6/101: mknod dd/c1d 0 2024-04-22T15:18:48.787 INFO:tasks.workunit.client.0.smithi096.stdout:4/514: write d5/f39 [1403216,104964] 0 2024-04-22T15:18:48.787 INFO:tasks.workunit.client.1.smithi196.stdout:6/102: write dd/f12 [807681,20300] 0 2024-04-22T15:18:48.790 INFO:tasks.workunit.client.0.smithi096.stdout:4/515: truncate d5/d7/d1f/fa6 189884 0 2024-04-22T15:18:48.790 INFO:tasks.workunit.client.0.smithi096.stdout:9/406: dread d1/d4/d9/d43/f5f [0,4194304] 0 2024-04-22T15:18:48.794 INFO:tasks.workunit.client.0.smithi096.stdout:1/654: rename d6/dc/d8a/d29/f36 to d6/dc/d8a/d3f/d62/dc2/dcb/fcc 0 2024-04-22T15:18:48.795 INFO:tasks.workunit.client.1.smithi196.stdout:0/95: creat d3/de/f26 x:0 0 0 2024-04-22T15:18:48.795 INFO:tasks.workunit.client.0.smithi096.stdout:1/655: chown d6/dc/d8a/cb2 9 1 2024-04-22T15:18:48.796 INFO:tasks.workunit.client.1.smithi196.stdout:1/73: creat d1/db/d16/f1d x:0 0 0 2024-04-22T15:18:48.796 INFO:tasks.workunit.client.1.smithi196.stdout:1/74: chown d1/db 26 1 2024-04-22T15:18:48.798 INFO:tasks.workunit.client.0.smithi096.stdout:5/505: mknod d4/d2c/d2f/d31/d82/da0/ca8 0 2024-04-22T15:18:48.798 INFO:tasks.workunit.client.1.smithi196.stdout:1/75: dread - d1/db/d16/f1d zero size 2024-04-22T15:18:48.798 INFO:tasks.workunit.client.1.smithi196.stdout:7/81: fdatasync fa 0 2024-04-22T15:18:48.801 INFO:tasks.workunit.client.0.smithi096.stdout:6/434: getdents d3/d28/dd 0 2024-04-22T15:18:48.802 INFO:tasks.workunit.client.0.smithi096.stdout:4/516: creat d5/d7/d2c/d67/fb3 x:0 0 0 2024-04-22T15:18:48.804 INFO:tasks.workunit.client.1.smithi196.stdout:8/123: creat d2/d8/f25 x:0 0 0 2024-04-22T15:18:48.804 INFO:tasks.workunit.client.0.smithi096.stdout:3/396: dread da/db/d10/f33 [0,4194304] 0 2024-04-22T15:18:48.805 INFO:tasks.workunit.client.1.smithi196.stdout:3/69: fsync d0/f1 0 2024-04-22T15:18:48.805 INFO:tasks.workunit.client.0.smithi096.stdout:4/517: fdatasync d5/d7/d2c/d67/fb3 0 2024-04-22T15:18:48.806 INFO:tasks.workunit.client.1.smithi196.stdout:3/70: write d0/d7/d8/de/ff [913713,7446] 0 2024-04-22T15:18:48.806 INFO:tasks.workunit.client.1.smithi196.stdout:6/103: rename c8 to dd/c1e 0 2024-04-22T15:18:48.808 INFO:tasks.workunit.client.0.smithi096.stdout:5/506: dwrite d4/d11/d25/f7e [0,4194304] 0 2024-04-22T15:18:48.808 INFO:tasks.workunit.client.0.smithi096.stdout:0/642: mkdir d7/de/d3a/d61/dbb/ddb 0 2024-04-22T15:18:48.810 INFO:tasks.workunit.client.1.smithi196.stdout:9/84: write db/dd/de/f10 [327136,8238] 0 2024-04-22T15:18:48.811 INFO:tasks.workunit.client.0.smithi096.stdout:1/656: dread d6/dc/d8a/d3f/d54/d32/f37 [0,4194304] 0 2024-04-22T15:18:48.813 INFO:tasks.workunit.client.1.smithi196.stdout:6/104: dwrite dd/f14 [0,4194304] 0 2024-04-22T15:18:48.813 INFO:tasks.workunit.client.1.smithi196.stdout:9/85: write fa [2111048,41173] 0 2024-04-22T15:18:48.813 INFO:tasks.workunit.client.0.smithi096.stdout:9/407: creat d1/d4/d79/f86 x:0 0 0 2024-04-22T15:18:48.813 INFO:tasks.workunit.client.0.smithi096.stdout:5/507: write d4/d11/d3d/d4d/d59/f7f [1637490,96653] 0 2024-04-22T15:18:48.816 INFO:tasks.workunit.client.1.smithi196.stdout:1/76: mkdir d1/db/d16/d1e 0 2024-04-22T15:18:48.818 INFO:tasks.workunit.client.0.smithi096.stdout:6/435: fdatasync d3/d28/f1a 0 2024-04-22T15:18:48.818 INFO:tasks.workunit.client.0.smithi096.stdout:4/518: dwrite d5/d7/d2c/d43/d56/f59 [0,4194304] 0 2024-04-22T15:18:48.819 INFO:tasks.workunit.client.1.smithi196.stdout:7/82: creat d14/d15/f18 x:0 0 0 2024-04-22T15:18:48.820 INFO:tasks.workunit.client.0.smithi096.stdout:4/519: stat d5/d7/d1a/f23 0 2024-04-22T15:18:48.823 INFO:tasks.workunit.client.0.smithi096.stdout:3/397: mknod da/db/d12/c7f 0 2024-04-22T15:18:48.824 INFO:tasks.workunit.client.1.smithi196.stdout:2/125: write d1/d13/f1e [1634765,100030] 0 2024-04-22T15:18:48.828 INFO:tasks.workunit.client.1.smithi196.stdout:3/71: rename d0/f2 to d0/d7/f11 0 2024-04-22T15:18:48.830 INFO:tasks.workunit.client.1.smithi196.stdout:3/72: truncate d0/d7/f11 4990331 0 2024-04-22T15:18:48.837 INFO:tasks.workunit.client.1.smithi196.stdout:9/86: mkdir db/d18 0 2024-04-22T15:18:48.837 INFO:tasks.workunit.client.1.smithi196.stdout:9/87: readlink db/dd/de/l16 0 2024-04-22T15:18:48.838 INFO:tasks.workunit.client.0.smithi096.stdout:9/408: rename d1/df/f14 to d1/d4/d9/d43/d57/d5e/f87 0 2024-04-22T15:18:48.839 INFO:tasks.workunit.client.0.smithi096.stdout:9/409: stat d1/d4/d7b/f81 0 2024-04-22T15:18:48.839 INFO:tasks.workunit.client.1.smithi196.stdout:6/105: mkdir dd/d1f 0 2024-04-22T15:18:48.839 INFO:tasks.workunit.client.1.smithi196.stdout:6/106: fdatasync dd/f1a 0 2024-04-22T15:18:48.840 INFO:tasks.workunit.client.1.smithi196.stdout:6/107: chown dd/f18 49575 1 2024-04-22T15:18:48.840 INFO:tasks.workunit.client.0.smithi096.stdout:6/436: creat d3/d3a/d75/d83/f98 x:0 0 0 2024-04-22T15:18:48.841 INFO:tasks.workunit.client.1.smithi196.stdout:6/108: write dd/f11 [709926,15147] 0 2024-04-22T15:18:48.841 INFO:tasks.workunit.client.1.smithi196.stdout:1/77: symlink d1/db/dc/l1f 0 2024-04-22T15:18:48.841 INFO:tasks.workunit.client.0.smithi096.stdout:6/437: chown f0 6 1 2024-04-22T15:18:48.841 INFO:tasks.workunit.client.0.smithi096.stdout:5/508: dread d4/d11/d3d/d4d/d59/d98/f8d [0,4194304] 0 2024-04-22T15:18:48.841 INFO:tasks.workunit.client.0.smithi096.stdout:6/438: stat d3/d28/f1c 0 2024-04-22T15:18:48.842 INFO:tasks.workunit.client.1.smithi196.stdout:7/83: symlink d14/d15/l19 0 2024-04-22T15:18:48.842 INFO:tasks.workunit.client.0.smithi096.stdout:6/439: dread - d3/d28/dd/f70 zero size 2024-04-22T15:18:48.843 INFO:tasks.workunit.client.0.smithi096.stdout:3/398: creat da/db/d12/d22/d68/f80 x:0 0 0 2024-04-22T15:18:48.844 INFO:tasks.workunit.client.0.smithi096.stdout:3/399: readlink da/db/dc/d28/l1d 0 2024-04-22T15:18:48.844 INFO:tasks.workunit.client.1.smithi196.stdout:4/102: creat d6/f22 x:0 0 0 2024-04-22T15:18:48.845 INFO:tasks.workunit.client.0.smithi096.stdout:0/643: rmdir d7/d42/d4c/d55/d98 39 2024-04-22T15:18:48.845 INFO:tasks.workunit.client.0.smithi096.stdout:0/644: chown d7/de/d5b/d95/dc/d16 366662143 1 2024-04-22T15:18:48.846 INFO:tasks.workunit.client.0.smithi096.stdout:0/645: write d7/da7/fd6 [667641,12558] 0 2024-04-22T15:18:48.847 INFO:tasks.workunit.client.0.smithi096.stdout:1/657: mkdir d6/dc/d8a/d3f/d54/d32/dcd 0 2024-04-22T15:18:48.848 INFO:tasks.workunit.client.1.smithi196.stdout:4/103: write d6/f22 [596936,31734] 0 2024-04-22T15:18:48.849 INFO:tasks.workunit.client.0.smithi096.stdout:1/658: read d6/dc/d8a/d3f/f73 [381808,44121] 0 2024-04-22T15:18:48.851 INFO:tasks.workunit.client.0.smithi096.stdout:1/659: readlink d6/dc/d8a/d3f/d54/d32/d3b/d94/lb0 0 2024-04-22T15:18:48.852 INFO:tasks.workunit.client.1.smithi196.stdout:1/78: dwrite d1/f4 [0,4194304] 0 2024-04-22T15:18:48.852 INFO:tasks.workunit.client.1.smithi196.stdout:3/73: unlink d0/d7/f9 0 2024-04-22T15:18:48.853 INFO:tasks.workunit.client.0.smithi096.stdout:9/410: rmdir d1/d4/d79 39 2024-04-22T15:18:48.853 INFO:tasks.workunit.client.1.smithi196.stdout:6/109: dwrite dd/f11 [0,4194304] 0 2024-04-22T15:18:48.855 INFO:tasks.workunit.client.0.smithi096.stdout:9/411: chown d1/df/c7e 316 1 2024-04-22T15:18:48.855 INFO:tasks.workunit.client.1.smithi196.stdout:9/88: mknod db/d11/c19 0 2024-04-22T15:18:48.856 INFO:tasks.workunit.client.1.smithi196.stdout:7/84: dwrite fe [0,4194304] 0 2024-04-22T15:18:48.857 INFO:tasks.workunit.client.0.smithi096.stdout:4/520: link d5/d7/d1a/f97 d5/d7/d2c/d3d/d4f/d88/fb4 0 2024-04-22T15:18:48.857 INFO:tasks.workunit.client.1.smithi196.stdout:6/110: chown dd/f1b 39792632 1 2024-04-22T15:18:48.858 INFO:tasks.workunit.client.1.smithi196.stdout:7/85: chown d14/d15 4641082 1 2024-04-22T15:18:48.858 INFO:tasks.workunit.client.0.smithi096.stdout:4/521: fsync d5/d7/d1f/f42 0 2024-04-22T15:18:48.859 INFO:tasks.workunit.client.1.smithi196.stdout:2/126: dread d1/ff [4194304,4194304] 0 2024-04-22T15:18:48.860 INFO:tasks.workunit.client.0.smithi096.stdout:1/660: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e/fa6 [0,4194304] 0 2024-04-22T15:18:48.861 INFO:tasks.workunit.client.1.smithi196.stdout:2/127: write d1/fc [4445489,110236] 0 2024-04-22T15:18:48.864 INFO:tasks.workunit.client.0.smithi096.stdout:6/440: creat d3/d28/dd/d11/d2c/d6f/f99 x:0 0 0 2024-04-22T15:18:48.866 INFO:tasks.workunit.client.1.smithi196.stdout:1/79: dread d1/f8 [0,4194304] 0 2024-04-22T15:18:48.867 INFO:tasks.workunit.client.1.smithi196.stdout:9/89: dread f1 [4194304,4194304] 0 2024-04-22T15:18:48.868 INFO:tasks.workunit.client.0.smithi096.stdout:3/400: unlink da/db/de/d19/d3e/f70 0 2024-04-22T15:18:48.869 INFO:tasks.workunit.client.0.smithi096.stdout:3/401: stat da/db/dc 0 2024-04-22T15:18:48.870 INFO:tasks.workunit.client.0.smithi096.stdout:3/402: dread - da/db/de/d38/f78 zero size 2024-04-22T15:18:48.876 INFO:tasks.workunit.client.1.smithi196.stdout:0/96: unlink d3/d6/dc/c1f 0 2024-04-22T15:18:48.876 INFO:tasks.workunit.client.1.smithi196.stdout:0/97: readlink d3/d6/l23 0 2024-04-22T15:18:48.877 INFO:tasks.workunit.client.0.smithi096.stdout:9/412: chown d1/d4/d79/f86 6572 1 2024-04-22T15:18:48.879 INFO:tasks.workunit.client.1.smithi196.stdout:0/98: dread d3/de/f1d [0,4194304] 0 2024-04-22T15:18:48.880 INFO:tasks.workunit.client.1.smithi196.stdout:0/99: readlink d3/d6/dc/l1b 0 2024-04-22T15:18:48.880 INFO:tasks.workunit.client.0.smithi096.stdout:4/522: mknod d5/d7/d1a/d25/cb5 0 2024-04-22T15:18:48.881 INFO:tasks.workunit.client.1.smithi196.stdout:6/111: mkdir dd/d20 0 2024-04-22T15:18:48.881 INFO:tasks.workunit.client.1.smithi196.stdout:6/112: fdatasync dd/f17 0 2024-04-22T15:18:48.882 INFO:tasks.workunit.client.0.smithi096.stdout:4/523: fsync d5/d7/d2c/d43/f68 0 2024-04-22T15:18:48.882 INFO:tasks.workunit.client.0.smithi096.stdout:5/509: mknod d4/ca9 0 2024-04-22T15:18:48.882 INFO:tasks.workunit.client.1.smithi196.stdout:7/86: creat d14/d15/f1a x:0 0 0 2024-04-22T15:18:48.882 INFO:tasks.workunit.client.0.smithi096.stdout:5/510: read - d4/d2c/d2f/d6b/f77 zero size 2024-04-22T15:18:48.883 INFO:tasks.workunit.client.1.smithi196.stdout:7/87: chown d14/d15/f18 215 1 2024-04-22T15:18:48.885 INFO:tasks.workunit.client.1.smithi196.stdout:6/113: dread dd/f14 [0,4194304] 0 2024-04-22T15:18:48.885 INFO:tasks.workunit.client.0.smithi096.stdout:5/511: dread d4/d11/d3d/d35/d57/f92 [0,4194304] 0 2024-04-22T15:18:48.887 INFO:tasks.workunit.client.1.smithi196.stdout:9/90: unlink f7 0 2024-04-22T15:18:48.889 INFO:tasks.workunit.client.1.smithi196.stdout:1/80: symlink d1/db/d16/l20 0 2024-04-22T15:18:48.892 INFO:tasks.workunit.client.1.smithi196.stdout:4/104: write d6/d13/f16 [233897,60342] 0 2024-04-22T15:18:48.893 INFO:tasks.workunit.client.1.smithi196.stdout:4/105: read d6/d13/f1d [120983,62592] 0 2024-04-22T15:18:48.893 INFO:tasks.workunit.client.1.smithi196.stdout:5/63: sync 2024-04-22T15:18:48.895 INFO:tasks.workunit.client.1.smithi196.stdout:0/100: mkdir d3/de/d1c/d27 0 2024-04-22T15:18:48.899 INFO:tasks.workunit.client.1.smithi196.stdout:7/88: symlink d14/l1b 0 2024-04-22T15:18:48.939 INFO:tasks.workunit.client.0.smithi096.stdout:5/512: unlink d4/d2c/d2f/fa6 0 2024-04-22T15:18:48.939 INFO:tasks.workunit.client.0.smithi096.stdout:6/441: write d3/d28/dd/d11/d2c/f5e [756321,104799] 0 2024-04-22T15:18:48.940 INFO:tasks.workunit.client.0.smithi096.stdout:3/403: mknod da/db/de/c81 0 2024-04-22T15:18:48.941 INFO:tasks.workunit.client.0.smithi096.stdout:5/513: fdatasync d4/d11/d3d/d4d/f56 0 2024-04-22T15:18:48.942 INFO:tasks.workunit.client.1.smithi196.stdout:9/91: creat db/dd/f1a x:0 0 0 2024-04-22T15:18:48.942 INFO:tasks.workunit.client.1.smithi196.stdout:9/92: chown db/d18 12 1 2024-04-22T15:18:48.942 INFO:tasks.workunit.client.1.smithi196.stdout:9/93: readlink db/d11/l12 0 2024-04-22T15:18:48.943 INFO:tasks.workunit.client.0.smithi096.stdout:0/646: rmdir d7/de/d3a/d61/dbb/ddb 0 2024-04-22T15:18:48.944 INFO:tasks.workunit.client.0.smithi096.stdout:1/661: dwrite d6/dc/d8a/d29/f5c [0,4194304] 0 2024-04-22T15:18:48.945 INFO:tasks.workunit.client.1.smithi196.stdout:1/81: write d1/db/d16/f17 [5716884,36986] 0 2024-04-22T15:18:48.948 INFO:tasks.workunit.client.0.smithi096.stdout:9/413: rename d1/d4/d9/d59/c64 to d1/d4/d9/d43/d57/d5e/c88 0 2024-04-22T15:18:48.952 INFO:tasks.workunit.client.1.smithi196.stdout:1/82: dwrite d1/db/d16/f1d [0,4194304] 0 2024-04-22T15:18:48.953 INFO:tasks.workunit.client.1.smithi196.stdout:1/83: fsync d1/db/dc/de/f14 0 2024-04-22T15:18:48.954 INFO:tasks.workunit.client.1.smithi196.stdout:1/84: fdatasync d1/db/d16/f1c 0 2024-04-22T15:18:48.957 INFO:tasks.workunit.client.1.smithi196.stdout:5/64: mkdir d1/d14 0 2024-04-22T15:18:48.960 INFO:tasks.workunit.client.1.smithi196.stdout:0/101: rename d3/ff to d3/de/f28 0 2024-04-22T15:18:48.961 INFO:tasks.workunit.client.0.smithi096.stdout:3/404: dread da/db/dc/d59/f79 [0,4194304] 0 2024-04-22T15:18:48.963 INFO:tasks.workunit.client.0.smithi096.stdout:5/514: mknod d4/d2c/d2f/d6b/caa 0 2024-04-22T15:18:48.964 INFO:tasks.workunit.client.0.smithi096.stdout:5/515: truncate d4/d2c/f7b 8420 0 2024-04-22T15:18:48.965 INFO:tasks.workunit.client.0.smithi096.stdout:5/516: chown d4/d11/d3d/d4d/d59/d5a/d5c 46 1 2024-04-22T15:18:48.966 INFO:tasks.workunit.client.0.smithi096.stdout:0/647: creat d7/de/d5b/d95/dc/d16/d65/d7c/fdc x:0 0 0 2024-04-22T15:18:48.966 INFO:tasks.workunit.client.0.smithi096.stdout:5/517: dread - d4/d2c/d2f/d31/f9a zero size 2024-04-22T15:18:48.966 INFO:tasks.workunit.client.1.smithi196.stdout:2/128: link d1/d1d/c33 d1/d1d/c34 0 2024-04-22T15:18:48.967 INFO:tasks.workunit.client.1.smithi196.stdout:2/129: stat d1/d23/d27 0 2024-04-22T15:18:48.967 INFO:tasks.workunit.client.0.smithi096.stdout:0/648: write d7/d8f/fca [877415,78081] 0 2024-04-22T15:18:48.967 INFO:tasks.workunit.client.1.smithi196.stdout:2/130: chown d1/d1d/f22 827 1 2024-04-22T15:18:48.970 INFO:tasks.workunit.client.1.smithi196.stdout:9/94: mkdir db/dd/d1b 0 2024-04-22T15:18:48.971 INFO:tasks.workunit.client.1.smithi196.stdout:9/95: chown db/dd 530020592 1 2024-04-22T15:18:48.971 INFO:tasks.workunit.client.0.smithi096.stdout:1/662: dread d6/dc/d8a/d3f/d54/d32/d3b/d3d/f4b [8388608,4194304] 0 2024-04-22T15:18:48.974 INFO:tasks.workunit.client.0.smithi096.stdout:1/663: write d6/dc/d8a/d3f/d54/d32/d3b/f89 [1175468,88183] 0 2024-04-22T15:18:48.975 INFO:tasks.workunit.client.1.smithi196.stdout:3/74: getdents d0/d7 0 2024-04-22T15:18:48.976 INFO:tasks.workunit.client.0.smithi096.stdout:6/442: mknod d3/d28/d4d/d82/c9a 0 2024-04-22T15:18:48.976 INFO:tasks.workunit.client.0.smithi096.stdout:4/524: dwrite d5/d7/f47 [0,4194304] 0 2024-04-22T15:18:48.978 INFO:tasks.workunit.client.0.smithi096.stdout:9/414: dread d1/f4f [0,4194304] 0 2024-04-22T15:18:48.978 INFO:tasks.workunit.client.0.smithi096.stdout:1/664: dread - d6/dc/d8a/d3f/d54/d32/d3b/d56/fb9 zero size 2024-04-22T15:18:48.978 INFO:tasks.workunit.client.0.smithi096.stdout:6/443: dread - d3/d28/dd/d11/d2c/d3b/f79 zero size 2024-04-22T15:18:48.979 INFO:tasks.workunit.client.0.smithi096.stdout:4/525: stat d5/d7/d1f/d84/f64 0 2024-04-22T15:18:48.980 INFO:tasks.workunit.client.1.smithi196.stdout:4/106: symlink d6/l23 0 2024-04-22T15:18:48.986 INFO:tasks.workunit.client.0.smithi096.stdout:1/665: truncate d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b/fb5 1001205 0 2024-04-22T15:18:48.987 INFO:tasks.workunit.client.1.smithi196.stdout:1/85: symlink d1/db/dc/l21 0 2024-04-22T15:18:48.987 INFO:tasks.workunit.client.1.smithi196.stdout:8/124: sync 2024-04-22T15:18:48.987 INFO:tasks.workunit.client.1.smithi196.stdout:5/65: mknod d1/d9/c15 0 2024-04-22T15:18:48.987 INFO:tasks.workunit.client.1.smithi196.stdout:0/102: mkdir d3/de/d14/d29 0 2024-04-22T15:18:48.987 INFO:tasks.workunit.client.1.smithi196.stdout:0/103: read d3/de/f28 [1455412,53606] 0 2024-04-22T15:18:48.988 INFO:tasks.workunit.client.0.smithi096.stdout:3/405: mknod da/db/d6a/c82 0 2024-04-22T15:18:48.988 INFO:tasks.workunit.client.1.smithi196.stdout:5/66: dread d1/d2/ff [0,4194304] 0 2024-04-22T15:18:48.990 INFO:tasks.workunit.client.0.smithi096.stdout:3/406: dread da/db/d12/d22/d68/f72 [0,4194304] 0 2024-04-22T15:18:48.991 INFO:tasks.workunit.client.1.smithi196.stdout:4/107: unlink d6/l23 0 2024-04-22T15:18:48.991 INFO:tasks.workunit.client.1.smithi196.stdout:5/67: dread d1/d2/f5 [4194304,4194304] 0 2024-04-22T15:18:48.998 INFO:tasks.workunit.client.1.smithi196.stdout:2/131: mkdir d1/d13/d25/d2a/d35 0 2024-04-22T15:18:48.999 INFO:tasks.workunit.client.0.smithi096.stdout:6/444: write f1 [1391675,26145] 0 2024-04-22T15:18:49.000 INFO:tasks.workunit.client.1.smithi196.stdout:9/96: mknod db/dd/de/d15/c1c 0 2024-04-22T15:18:49.001 INFO:tasks.workunit.client.0.smithi096.stdout:4/526: write d5/f21 [502448,61445] 0 2024-04-22T15:18:49.001 INFO:tasks.workunit.client.1.smithi196.stdout:9/97: chown db/d11 140766 1 2024-04-22T15:18:49.001 INFO:tasks.workunit.client.1.smithi196.stdout:9/98: write fa [2494772,71439] 0 2024-04-22T15:18:49.002 INFO:tasks.workunit.client.0.smithi096.stdout:4/527: truncate d5/d5b/fa4 464245 0 2024-04-22T15:18:49.002 INFO:tasks.workunit.client.1.smithi196.stdout:0/104: mknod d3/d6/dc/c2a 0 2024-04-22T15:18:49.003 INFO:tasks.workunit.client.1.smithi196.stdout:3/75: creat d0/d7/d8/de/f12 x:0 0 0 2024-04-22T15:18:49.003 INFO:tasks.workunit.client.1.smithi196.stdout:3/76: readlink d0/d7/d8/ld 0 2024-04-22T15:18:49.006 INFO:tasks.workunit.client.1.smithi196.stdout:4/108: write d6/d10/fe [898642,41046] 0 2024-04-22T15:18:49.007 INFO:tasks.workunit.client.1.smithi196.stdout:1/86: creat d1/db/d16/d1e/f22 x:0 0 0 2024-04-22T15:18:49.008 INFO:tasks.workunit.client.0.smithi096.stdout:6/445: rmdir d3/d28/d54/d68/d6e 39 2024-04-22T15:18:49.008 INFO:tasks.workunit.client.1.smithi196.stdout:1/87: chown d1/db/ff 6 1 2024-04-22T15:18:49.008 INFO:tasks.workunit.client.0.smithi096.stdout:6/446: fsync d3/d28/f21 0 2024-04-22T15:18:49.009 INFO:tasks.workunit.client.0.smithi096.stdout:4/528: creat d5/d7/d2c/d3d/d4f/fb6 x:0 0 0 2024-04-22T15:18:49.009 INFO:tasks.workunit.client.0.smithi096.stdout:6/447: fdatasync d3/d28/dd/f2b 0 2024-04-22T15:18:49.010 INFO:tasks.workunit.client.1.smithi196.stdout:3/77: dwrite d0/f1 [4194304,4194304] 0 2024-04-22T15:18:49.010 INFO:tasks.workunit.client.1.smithi196.stdout:7/89: getdents d14 0 2024-04-22T15:18:49.011 INFO:tasks.workunit.client.1.smithi196.stdout:3/78: write d0/d7/d8/de/f12 [870444,77639] 0 2024-04-22T15:18:49.011 INFO:tasks.workunit.client.0.smithi096.stdout:5/518: link d4/c2e d4/d11/d3d/d4d/d99/cab 0 2024-04-22T15:18:49.012 INFO:tasks.workunit.client.0.smithi096.stdout:5/519: read - d4/d2c/f9f zero size 2024-04-22T15:18:49.013 INFO:tasks.workunit.client.0.smithi096.stdout:0/649: getdents d7/de/d3a/d61/d79 0 2024-04-22T15:18:49.014 INFO:tasks.workunit.client.0.smithi096.stdout:6/448: read d3/d28/dd/d11/f18 [1075795,126731] 0 2024-04-22T15:18:49.014 INFO:tasks.workunit.client.0.smithi096.stdout:1/666: link d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e/c75 d6/dc/d8a/d3f/d54/d32/d3b/d94/cce 0 2024-04-22T15:18:49.015 INFO:tasks.workunit.client.0.smithi096.stdout:0/650: stat d7/de/d5b/d95/dc/d16/d18/c46 0 2024-04-22T15:18:49.016 INFO:tasks.workunit.client.1.smithi196.stdout:3/79: dread d0/f3 [4194304,4194304] 0 2024-04-22T15:18:49.016 INFO:tasks.workunit.client.0.smithi096.stdout:0/651: chown d7/de/d5b/d62/c73 7 1 2024-04-22T15:18:49.018 INFO:tasks.workunit.client.0.smithi096.stdout:4/529: rename d5/d5b/fa4 to d5/d7/d1a/d25/fb7 0 2024-04-22T15:18:49.019 INFO:tasks.workunit.client.1.smithi196.stdout:5/68: creat d1/d14/f16 x:0 0 0 2024-04-22T15:18:49.019 INFO:tasks.workunit.client.1.smithi196.stdout:5/69: dread - d1/d9/f11 zero size 2024-04-22T15:18:49.021 INFO:tasks.workunit.client.1.smithi196.stdout:4/109: creat d6/d13/f24 x:0 0 0 2024-04-22T15:18:49.022 INFO:tasks.workunit.client.0.smithi096.stdout:9/415: link d1/d25/c5a d1/df/c89 0 2024-04-22T15:18:49.024 INFO:tasks.workunit.client.0.smithi096.stdout:9/416: chown d1/d4/d7f/f83 1814 1 2024-04-22T15:18:49.024 INFO:tasks.workunit.client.0.smithi096.stdout:0/652: dwrite d7/de/d5b/d95/d66/f81 [0,4194304] 0 2024-04-22T15:18:49.026 INFO:tasks.workunit.client.1.smithi196.stdout:7/90: mknod d14/d15/c1c 0 2024-04-22T15:18:49.027 INFO:tasks.workunit.client.1.smithi196.stdout:7/91: write ff [436094,46319] 0 2024-04-22T15:18:49.036 INFO:tasks.workunit.client.0.smithi096.stdout:5/520: dread d4/d2c/d2f/d6b/f50 [0,4194304] 0 2024-04-22T15:18:49.037 INFO:tasks.workunit.client.0.smithi096.stdout:5/521: read - d4/d11/d3d/d4d/d59/d5a/d5c/f83 zero size 2024-04-22T15:18:49.039 INFO:tasks.workunit.client.0.smithi096.stdout:6/449: dread d3/d28/f1c [0,4194304] 0 2024-04-22T15:18:49.040 INFO:tasks.workunit.client.1.smithi196.stdout:3/80: rmdir d0 39 2024-04-22T15:18:49.042 INFO:tasks.workunit.client.1.smithi196.stdout:5/70: mkdir d1/d17 0 2024-04-22T15:18:49.043 INFO:tasks.workunit.client.0.smithi096.stdout:0/653: creat d7/de/d3a/d61/fdd x:0 0 0 2024-04-22T15:18:49.043 INFO:tasks.workunit.client.0.smithi096.stdout:0/654: readlink d7/de/l1b 0 2024-04-22T15:18:49.044 INFO:tasks.workunit.client.1.smithi196.stdout:4/110: unlink d6/d13/f20 0 2024-04-22T15:18:49.047 INFO:tasks.workunit.client.1.smithi196.stdout:8/125: getdents d2 0 2024-04-22T15:18:49.048 INFO:tasks.workunit.client.0.smithi096.stdout:5/522: stat d4/d11/c40 0 2024-04-22T15:18:49.048 INFO:tasks.workunit.client.1.smithi196.stdout:4/111: dread d6/f22 [0,4194304] 0 2024-04-22T15:18:49.048 INFO:tasks.workunit.client.1.smithi196.stdout:8/126: read - d2/d8/f25 zero size 2024-04-22T15:18:49.049 INFO:tasks.workunit.client.1.smithi196.stdout:9/99: truncate db/f14 8188 0 2024-04-22T15:18:49.049 INFO:tasks.workunit.client.1.smithi196.stdout:8/127: read - d2/d8/f25 zero size 2024-04-22T15:18:49.049 INFO:tasks.workunit.client.1.smithi196.stdout:6/114: sync 2024-04-22T15:18:49.050 INFO:tasks.workunit.client.1.smithi196.stdout:6/115: dread dd/fe [0,4194304] 0 2024-04-22T15:18:49.051 INFO:tasks.workunit.client.0.smithi096.stdout:6/450: mknod d3/d28/d4d/d82/c9b 0 2024-04-22T15:18:49.053 INFO:tasks.workunit.client.0.smithi096.stdout:4/530: write d5/d7/d2c/d3d/d4f/d63/f89 [3361559,91275] 0 2024-04-22T15:18:49.055 INFO:tasks.workunit.client.0.smithi096.stdout:5/523: mknod d4/d11/d3d/cac 0 2024-04-22T15:18:49.055 INFO:tasks.workunit.client.1.smithi196.stdout:3/81: write d0/d7/fa [622658,26677] 0 2024-04-22T15:18:49.055 INFO:tasks.workunit.client.0.smithi096.stdout:5/524: dread - d4/f86 zero size 2024-04-22T15:18:49.056 INFO:tasks.workunit.client.0.smithi096.stdout:9/417: creat d1/f8a x:0 0 0 2024-04-22T15:18:49.057 INFO:tasks.workunit.client.0.smithi096.stdout:6/451: rename d3 to d3/d28/d54/d68/d6e/d9c 22 2024-04-22T15:18:49.057 INFO:tasks.workunit.client.1.smithi196.stdout:5/71: dread d1/d2/f5 [0,4194304] 0 2024-04-22T15:18:49.059 INFO:tasks.workunit.client.1.smithi196.stdout:9/100: mknod db/dd/c1d 0 2024-04-22T15:18:49.062 INFO:tasks.workunit.client.1.smithi196.stdout:6/116: creat dd/f21 x:0 0 0 2024-04-22T15:18:49.062 INFO:tasks.workunit.client.1.smithi196.stdout:0/105: link d3/d6/c10 d3/de/d1c/d27/c2b 0 2024-04-22T15:18:49.063 INFO:tasks.workunit.client.1.smithi196.stdout:3/82: mkdir d0/d6/db/d13 0 2024-04-22T15:18:49.064 INFO:tasks.workunit.client.1.smithi196.stdout:5/72: symlink d1/d4/l18 0 2024-04-22T15:18:49.064 INFO:tasks.workunit.client.1.smithi196.stdout:7/92: link d14/l1b d14/d15/l1d 0 2024-04-22T15:18:49.067 INFO:tasks.workunit.client.1.smithi196.stdout:9/101: rename f1 to db/dd/de/d15/f1e 0 2024-04-22T15:18:49.069 INFO:tasks.workunit.client.1.smithi196.stdout:0/106: dwrite d3/d6/dc/f15 [0,4194304] 0 2024-04-22T15:18:49.070 INFO:tasks.workunit.client.1.smithi196.stdout:9/102: write db/dd/de/f10 [552253,105310] 0 2024-04-22T15:18:49.070 INFO:tasks.workunit.client.1.smithi196.stdout:7/93: dread f4 [0,4194304] 0 2024-04-22T15:18:49.071 INFO:tasks.workunit.client.1.smithi196.stdout:5/73: fsync f0 0 2024-04-22T15:18:49.076 INFO:tasks.workunit.client.1.smithi196.stdout:6/117: creat dd/d20/f22 x:0 0 0 2024-04-22T15:18:49.077 INFO:tasks.workunit.client.1.smithi196.stdout:6/118: write dd/f10 [983732,123496] 0 2024-04-22T15:18:49.081 INFO:tasks.workunit.client.1.smithi196.stdout:7/94: creat d14/f1e x:0 0 0 2024-04-22T15:18:49.082 INFO:tasks.workunit.client.1.smithi196.stdout:5/74: creat d1/f19 x:0 0 0 2024-04-22T15:18:49.084 INFO:tasks.workunit.client.1.smithi196.stdout:4/112: truncate d6/f19 533939 0 2024-04-22T15:18:49.085 INFO:tasks.workunit.client.1.smithi196.stdout:0/107: mkdir d3/de/d1c/d27/d2c 0 2024-04-22T15:18:49.086 INFO:tasks.workunit.client.1.smithi196.stdout:7/95: mknod d14/d15/c1f 0 2024-04-22T15:18:49.087 INFO:tasks.workunit.client.1.smithi196.stdout:7/96: chown f13 122117 1 2024-04-22T15:18:49.087 INFO:tasks.workunit.client.1.smithi196.stdout:8/128: write d2/dc/fe [6317736,30316] 0 2024-04-22T15:18:49.088 INFO:tasks.workunit.client.1.smithi196.stdout:8/129: truncate d2/dc/f20 213373 0 2024-04-22T15:18:49.089 INFO:tasks.workunit.client.1.smithi196.stdout:8/130: read d2/fa [2213675,75282] 0 2024-04-22T15:18:49.093 INFO:tasks.workunit.client.1.smithi196.stdout:0/108: mknod d3/de/c2d 0 2024-04-22T15:18:49.100 INFO:tasks.workunit.client.1.smithi196.stdout:7/97: mknod d14/c20 0 2024-04-22T15:18:49.100 INFO:tasks.workunit.client.1.smithi196.stdout:5/75: mkdir d1/d17/d1a 0 2024-04-22T15:18:49.101 INFO:tasks.workunit.client.1.smithi196.stdout:8/131: mknod d2/d5/d1a/c26 0 2024-04-22T15:18:49.101 INFO:tasks.workunit.client.1.smithi196.stdout:0/109: rmdir d3/d9 39 2024-04-22T15:18:49.101 INFO:tasks.workunit.client.1.smithi196.stdout:5/76: stat d1/d9/c10 0 2024-04-22T15:18:49.102 INFO:tasks.workunit.client.1.smithi196.stdout:8/132: truncate d2/f1d 217298 0 2024-04-22T15:18:49.102 INFO:tasks.workunit.client.1.smithi196.stdout:5/77: truncate d1/f19 693486 0 2024-04-22T15:18:49.105 INFO:tasks.workunit.client.1.smithi196.stdout:6/119: truncate f2 1954585 0 2024-04-22T15:18:49.109 INFO:tasks.workunit.client.1.smithi196.stdout:4/113: dwrite f5 [0,4194304] 0 2024-04-22T15:18:49.109 INFO:tasks.workunit.client.1.smithi196.stdout:4/114: read - d6/d13/f1b zero size 2024-04-22T15:18:49.117 INFO:tasks.workunit.client.1.smithi196.stdout:2/132: sync 2024-04-22T15:18:49.118 INFO:tasks.workunit.client.1.smithi196.stdout:2/133: dread - d1/f10 zero size 2024-04-22T15:18:49.118 INFO:tasks.workunit.client.1.smithi196.stdout:2/134: dread - d1/d1d/f22 zero size 2024-04-22T15:18:49.119 INFO:tasks.workunit.client.1.smithi196.stdout:2/135: write d1/fc [201531,61110] 0 2024-04-22T15:18:49.141 INFO:tasks.workunit.client.1.smithi196.stdout:9/103: unlink db/f14 0 2024-04-22T15:18:49.142 INFO:tasks.workunit.client.1.smithi196.stdout:8/133: creat d2/d8/f27 x:0 0 0 2024-04-22T15:18:49.143 INFO:tasks.workunit.client.1.smithi196.stdout:8/134: dread - d2/d5/f19 zero size 2024-04-22T15:18:49.144 INFO:tasks.workunit.client.1.smithi196.stdout:6/120: creat dd/f23 x:0 0 0 2024-04-22T15:18:49.144 INFO:tasks.workunit.client.1.smithi196.stdout:4/115: creat d6/d13/f25 x:0 0 0 2024-04-22T15:18:49.145 INFO:tasks.workunit.client.1.smithi196.stdout:1/88: sync 2024-04-22T15:18:49.145 INFO:tasks.workunit.client.1.smithi196.stdout:3/83: sync 2024-04-22T15:18:49.146 INFO:tasks.workunit.client.1.smithi196.stdout:0/110: dwrite d3/de/f28 [0,4194304] 0 2024-04-22T15:18:49.146 INFO:tasks.workunit.client.1.smithi196.stdout:1/89: fdatasync d1/db/d16/d1e/f22 0 2024-04-22T15:18:49.150 INFO:tasks.workunit.client.1.smithi196.stdout:2/136: rename d1/fc to d1/d13/d25/d2a/d35/f36 0 2024-04-22T15:18:49.152 INFO:tasks.workunit.client.1.smithi196.stdout:2/137: read d1/f18 [1024720,64571] 0 2024-04-22T15:18:49.155 INFO:tasks.workunit.client.1.smithi196.stdout:8/135: creat d2/dc/f28 x:0 0 0 2024-04-22T15:18:49.177 INFO:tasks.workunit.client.0.smithi096.stdout:4/531: write f3 [3233226,97054] 0 2024-04-22T15:18:49.177 INFO:tasks.workunit.client.1.smithi196.stdout:6/121: creat dd/d20/f24 x:0 0 0 2024-04-22T15:18:49.178 INFO:tasks.workunit.client.0.smithi096.stdout:0/655: link d7/de/d5b/d95/d66/lc8 d7/de/d5b/d95/d66/dce/lde 0 2024-04-22T15:18:49.178 INFO:tasks.workunit.client.1.smithi196.stdout:6/122: dread fa [0,4194304] 0 2024-04-22T15:18:49.180 INFO:tasks.workunit.client.0.smithi096.stdout:9/418: dwrite d1/d4/f7a [0,4194304] 0 2024-04-22T15:18:49.181 INFO:tasks.workunit.client.0.smithi096.stdout:0/656: fsync d7/de/d5b/d95/d66/f81 0 2024-04-22T15:18:49.182 INFO:tasks.workunit.client.0.smithi096.stdout:4/532: read d5/d7/d2c/d3d/d4f/f60 [976183,128104] 0 2024-04-22T15:18:49.185 INFO:tasks.workunit.client.1.smithi196.stdout:1/90: symlink d1/db/l23 0 2024-04-22T15:18:49.186 INFO:tasks.workunit.client.1.smithi196.stdout:1/91: write d1/db/ff [161685,116651] 0 2024-04-22T15:18:49.191 INFO:tasks.workunit.client.1.smithi196.stdout:9/104: creat db/d18/f1f x:0 0 0 2024-04-22T15:18:49.194 INFO:tasks.workunit.client.0.smithi096.stdout:6/452: dread d3/d28/f61 [0,4194304] 0 2024-04-22T15:18:49.194 INFO:tasks.workunit.client.1.smithi196.stdout:5/78: link d1/d9/c15 d1/d17/c1b 0 2024-04-22T15:18:49.195 INFO:tasks.workunit.client.1.smithi196.stdout:9/105: dread db/fc [0,4194304] 0 2024-04-22T15:18:49.196 INFO:tasks.workunit.client.1.smithi196.stdout:8/136: rename d2/dc/c1b to d2/c29 0 2024-04-22T15:18:49.198 INFO:tasks.workunit.client.1.smithi196.stdout:2/138: dwrite d1/fd [4194304,4194304] 0 2024-04-22T15:18:49.198 INFO:tasks.workunit.client.1.smithi196.stdout:4/116: mkdir d6/d14/d18/d26 0 2024-04-22T15:18:49.199 INFO:tasks.workunit.client.1.smithi196.stdout:2/139: write d1/d13/d1f/f24 [1448178,81537] 0 2024-04-22T15:18:49.204 INFO:tasks.workunit.client.1.smithi196.stdout:4/117: dread d6/d10/fe [0,4194304] 0 2024-04-22T15:18:49.207 INFO:tasks.workunit.client.1.smithi196.stdout:0/111: symlink d3/d9/l2e 0 2024-04-22T15:18:49.207 INFO:tasks.workunit.client.1.smithi196.stdout:0/112: dread - d3/d9/f21 zero size 2024-04-22T15:18:49.210 INFO:tasks.workunit.client.1.smithi196.stdout:5/79: fsync d1/d4/f6 0 2024-04-22T15:18:49.212 INFO:tasks.workunit.client.1.smithi196.stdout:9/106: symlink db/d18/l20 0 2024-04-22T15:18:49.215 INFO:tasks.workunit.client.1.smithi196.stdout:1/92: rename d1/f8 to d1/db/d16/f24 0 2024-04-22T15:18:49.216 INFO:tasks.workunit.client.1.smithi196.stdout:1/93: stat d1/db/l23 0 2024-04-22T15:18:49.220 INFO:tasks.workunit.client.1.smithi196.stdout:1/94: dwrite d1/f4 [0,4194304] 0 2024-04-22T15:18:49.221 INFO:tasks.workunit.client.1.smithi196.stdout:2/140: mknod d1/d13/c37 0 2024-04-22T15:18:49.222 INFO:tasks.workunit.client.1.smithi196.stdout:2/141: fdatasync d1/d13/d1f/f24 0 2024-04-22T15:18:49.224 INFO:tasks.workunit.client.1.smithi196.stdout:4/118: write d6/d13/f1d [1311157,110320] 0 2024-04-22T15:18:49.228 INFO:tasks.workunit.client.1.smithi196.stdout:7/98: write f4 [461514,795] 0 2024-04-22T15:18:49.231 INFO:tasks.workunit.client.1.smithi196.stdout:5/80: mkdir d1/d17/d1c 0 2024-04-22T15:18:49.231 INFO:tasks.workunit.client.1.smithi196.stdout:7/99: dread fe [0,4194304] 0 2024-04-22T15:18:49.231 INFO:tasks.workunit.client.1.smithi196.stdout:5/81: chown d1/l7 0 1 2024-04-22T15:18:49.233 INFO:tasks.workunit.client.1.smithi196.stdout:9/107: symlink db/dd/de/l21 0 2024-04-22T15:18:49.236 INFO:tasks.workunit.client.1.smithi196.stdout:9/108: dread f5 [0,4194304] 0 2024-04-22T15:18:49.237 INFO:tasks.workunit.client.1.smithi196.stdout:9/109: write db/d18/f1f [960615,37060] 0 2024-04-22T15:18:49.244 INFO:tasks.workunit.client.0.smithi096.stdout:9/419: fdatasync d1/d25/f33 0 2024-04-22T15:18:49.246 INFO:tasks.workunit.client.1.smithi196.stdout:2/142: mkdir d1/d13/d25/d2a/d38 0 2024-04-22T15:18:49.246 INFO:tasks.workunit.client.0.smithi096.stdout:9/420: read d1/df/f58 [789162,36840] 0 2024-04-22T15:18:49.247 INFO:tasks.workunit.client.0.smithi096.stdout:0/657: rename d7/de/d5b/d95/d15/l5a to d7/de/d5b/d95/d59/ldf 0 2024-04-22T15:18:49.249 INFO:tasks.workunit.client.0.smithi096.stdout:0/658: chown d7/de/d5b/d95/dc/d16/d18/l71 299619 1 2024-04-22T15:18:49.255 INFO:tasks.workunit.client.1.smithi196.stdout:7/100: symlink d14/d15/l21 0 2024-04-22T15:18:49.255 INFO:tasks.workunit.client.1.smithi196.stdout:7/101: write ff [728432,106535] 0 2024-04-22T15:18:49.258 INFO:tasks.workunit.client.1.smithi196.stdout:7/102: fdatasync ff 0 2024-04-22T15:18:49.258 INFO:tasks.workunit.client.1.smithi196.stdout:5/82: rename d1/d9/f13 to d1/d17/d1a/f1d 0 2024-04-22T15:18:49.258 INFO:tasks.workunit.client.1.smithi196.stdout:5/83: read d1/d2/f5 [1531955,5956] 0 2024-04-22T15:18:49.263 INFO:tasks.workunit.client.0.smithi096.stdout:9/421: read d1/f20 [1244842,5460] 0 2024-04-22T15:18:49.265 INFO:tasks.workunit.client.1.smithi196.stdout:6/123: fdatasync f2 0 2024-04-22T15:18:49.266 INFO:tasks.workunit.client.1.smithi196.stdout:6/124: fsync dd/f12 0 2024-04-22T15:18:49.266 INFO:tasks.workunit.client.1.smithi196.stdout:6/125: fsync dd/f12 0 2024-04-22T15:18:49.266 INFO:tasks.workunit.client.1.smithi196.stdout:6/126: chown dd/fe 8 1 2024-04-22T15:18:49.267 INFO:tasks.workunit.client.1.smithi196.stdout:9/110: mknod db/d18/c22 0 2024-04-22T15:18:49.268 INFO:tasks.workunit.client.1.smithi196.stdout:3/84: truncate d0/f4 953172 0 2024-04-22T15:18:49.269 INFO:tasks.workunit.client.1.smithi196.stdout:3/85: chown d0/d7/d8/ld 258268536 1 2024-04-22T15:18:49.269 INFO:tasks.workunit.client.0.smithi096.stdout:5/525: getdents d4 0 2024-04-22T15:18:49.270 INFO:tasks.workunit.client.1.smithi196.stdout:2/143: mknod d1/d13/c39 0 2024-04-22T15:18:49.272 INFO:tasks.workunit.client.0.smithi096.stdout:9/422: symlink d1/d4/d79/l8b 0 2024-04-22T15:18:49.274 INFO:tasks.workunit.client.0.smithi096.stdout:6/453: rename d3/d28/l27 to d3/d28/dd/l9d 0 2024-04-22T15:18:49.276 INFO:tasks.workunit.client.0.smithi096.stdout:0/659: symlink d7/d42/d4c/d55/le0 0 2024-04-22T15:18:49.277 INFO:tasks.workunit.client.0.smithi096.stdout:0/660: fdatasync d7/de/d5b/d95/dc/d16/f3b 0 2024-04-22T15:18:49.278 INFO:tasks.workunit.client.0.smithi096.stdout:4/533: creat d5/d7/fb8 x:0 0 0 2024-04-22T15:18:49.279 INFO:tasks.workunit.client.0.smithi096.stdout:4/534: dread - d5/d7/d2c/d43/d56/f96 zero size 2024-04-22T15:18:49.282 INFO:tasks.workunit.client.0.smithi096.stdout:9/423: mknod d1/d4/d9/dd/c8c 0 2024-04-22T15:18:49.284 INFO:tasks.workunit.client.1.smithi196.stdout:7/103: creat d14/f22 x:0 0 0 2024-04-22T15:18:49.285 INFO:tasks.workunit.client.1.smithi196.stdout:7/104: readlink d14/d15/l19 0 2024-04-22T15:18:49.285 INFO:tasks.workunit.client.0.smithi096.stdout:0/661: dwrite d7/de/d5b/d95/dc/d16/d65/fcd [0,4194304] 0 2024-04-22T15:18:49.286 INFO:tasks.workunit.client.1.smithi196.stdout:5/84: chown d1/d9/c15 25332443 1 2024-04-22T15:18:49.295 INFO:tasks.workunit.client.1.smithi196.stdout:8/137: dread d2/f4 [0,4194304] 0 2024-04-22T15:18:49.297 INFO:tasks.workunit.client.1.smithi196.stdout:9/111: creat db/d18/f23 x:0 0 0 2024-04-22T15:18:49.301 INFO:tasks.workunit.client.1.smithi196.stdout:3/86: rmdir d0/d6 39 2024-04-22T15:18:49.303 INFO:tasks.workunit.client.0.smithi096.stdout:9/424: rename d1/d4/d9/d43/d57/d5e/f87 to d1/d4/d9/d43/d57/f8d 0 2024-04-22T15:18:49.307 INFO:tasks.workunit.client.1.smithi196.stdout:4/119: truncate d6/d10/fe 1637716 0 2024-04-22T15:18:49.307 INFO:tasks.workunit.client.1.smithi196.stdout:4/120: write f5 [6761659,32400] 0 2024-04-22T15:18:49.308 INFO:tasks.workunit.client.1.smithi196.stdout:7/105: symlink d14/d15/l23 0 2024-04-22T15:18:49.309 INFO:tasks.workunit.client.1.smithi196.stdout:7/106: truncate d14/d15/f1a 773503 0 2024-04-22T15:18:49.310 INFO:tasks.workunit.client.1.smithi196.stdout:5/85: creat d1/d14/f1e x:0 0 0 2024-04-22T15:18:49.311 INFO:tasks.workunit.client.0.smithi096.stdout:9/425: fdatasync d1/d4/f22 0 2024-04-22T15:18:49.313 INFO:tasks.workunit.client.0.smithi096.stdout:0/662: link d7/de/c7b d7/de/d3a/d61/dbb/ce1 0 2024-04-22T15:18:49.313 INFO:tasks.workunit.client.1.smithi196.stdout:4/121: dwrite d6/d13/f1d [0,4194304] 0 2024-04-22T15:18:49.314 INFO:tasks.workunit.client.0.smithi096.stdout:0/663: write d7/d42/d4c/f6a [205295,33214] 0 2024-04-22T15:18:49.314 INFO:tasks.workunit.client.1.smithi196.stdout:4/122: write f5 [5287763,11010] 0 2024-04-22T15:18:49.315 INFO:tasks.workunit.client.1.smithi196.stdout:8/138: write d2/f3 [4372096,535] 0 2024-04-22T15:18:49.315 INFO:tasks.workunit.client.1.smithi196.stdout:0/113: truncate d3/de/f18 289016 0 2024-04-22T15:18:49.316 INFO:tasks.workunit.client.0.smithi096.stdout:4/535: getdents d5/d7/d2c/d43 0 2024-04-22T15:18:49.317 INFO:tasks.workunit.client.0.smithi096.stdout:4/536: dread - d5/d7/d1f/d84/d6d/fae zero size 2024-04-22T15:18:49.320 INFO:tasks.workunit.client.0.smithi096.stdout:9/426: mkdir d1/d4/d9/d43/d57/d5b/d8e 0 2024-04-22T15:18:49.322 INFO:tasks.workunit.client.1.smithi196.stdout:4/123: dwrite f5 [8388608,4194304] 0 2024-04-22T15:18:49.322 INFO:tasks.workunit.client.1.smithi196.stdout:1/95: getdents d1/db/d16/d1e 0 2024-04-22T15:18:49.326 INFO:tasks.workunit.client.0.smithi096.stdout:4/537: mknod d5/d7/d2c/d3d/d4f/d88/cb9 0 2024-04-22T15:18:49.327 INFO:tasks.workunit.client.0.smithi096.stdout:4/538: dread d5/d7/d1a/d25/fb7 [0,4194304] 0 2024-04-22T15:18:49.329 INFO:tasks.workunit.client.0.smithi096.stdout:9/427: mknod d1/d4/d7b/c8f 0 2024-04-22T15:18:49.330 INFO:tasks.workunit.client.1.smithi196.stdout:3/87: dread d0/f3 [0,4194304] 0 2024-04-22T15:18:49.331 INFO:tasks.workunit.client.1.smithi196.stdout:3/88: stat d0/f3 0 2024-04-22T15:18:49.332 INFO:tasks.workunit.client.1.smithi196.stdout:4/124: dwrite d6/d13/f16 [0,4194304] 0 2024-04-22T15:18:49.332 INFO:tasks.workunit.client.0.smithi096.stdout:4/539: dread d5/d7/f7f [4194304,4194304] 0 2024-04-22T15:18:49.333 INFO:tasks.workunit.client.1.smithi196.stdout:4/125: readlink d6/d14/l21 0 2024-04-22T15:18:49.334 INFO:tasks.workunit.client.1.smithi196.stdout:4/126: stat d6/d13/f25 0 2024-04-22T15:18:49.336 INFO:tasks.workunit.client.1.smithi196.stdout:5/86: creat d1/f1f x:0 0 0 2024-04-22T15:18:49.339 INFO:tasks.workunit.client.0.smithi096.stdout:0/664: creat d7/de/d5b/fe2 x:0 0 0 2024-04-22T15:18:49.340 INFO:tasks.workunit.client.0.smithi096.stdout:0/665: chown d7/de/d5b/d62/d67/lc1 9381158 1 2024-04-22T15:18:49.341 INFO:tasks.workunit.client.0.smithi096.stdout:0/666: stat d7/de/d5b/cc4 0 2024-04-22T15:18:49.341 INFO:tasks.workunit.client.0.smithi096.stdout:0/667: fsync d7/d42/f86 0 2024-04-22T15:18:49.342 INFO:tasks.workunit.client.0.smithi096.stdout:4/540: mknod d5/d7/d2c/d3d/d4f/cba 0 2024-04-22T15:18:49.343 INFO:tasks.workunit.client.0.smithi096.stdout:0/668: stat d7/de/d5b/d95/dc/d16/f5d 0 2024-04-22T15:18:49.346 INFO:tasks.workunit.client.0.smithi096.stdout:9/428: mknod d1/df/c90 0 2024-04-22T15:18:49.347 INFO:tasks.workunit.client.1.smithi196.stdout:8/139: read d2/fa [3170620,24624] 0 2024-04-22T15:18:49.348 INFO:tasks.workunit.client.1.smithi196.stdout:2/144: write d1/d13/d1f/f20 [59090,65060] 0 2024-04-22T15:18:49.349 INFO:tasks.workunit.client.0.smithi096.stdout:5/526: dwrite d4/d11/d3d/d4d/fa4 [4194304,4194304] 0 2024-04-22T15:18:49.353 INFO:tasks.workunit.client.0.smithi096.stdout:4/541: dwrite d5/d7/d2c/d43/f68 [0,4194304] 0 2024-04-22T15:18:49.353 INFO:tasks.workunit.client.0.smithi096.stdout:5/527: dread - d4/d11/d3d/d4d/d59/d5a/d5c/f83 zero size 2024-04-22T15:18:49.354 INFO:tasks.workunit.client.0.smithi096.stdout:0/669: rename d7/de/d5b/d95/d59/c88 to d7/de/d5b/d64/ce3 0 2024-04-22T15:18:49.355 INFO:tasks.workunit.client.0.smithi096.stdout:5/528: readlink d4/d2c/l9c 0 2024-04-22T15:18:49.356 INFO:tasks.workunit.client.0.smithi096.stdout:0/670: chown d7/de/d3a/f44 1111 1 2024-04-22T15:18:49.358 INFO:tasks.workunit.client.0.smithi096.stdout:0/671: write d7/de/d5b/d95/dc/f13 [9277664,106881] 0 2024-04-22T15:18:49.360 INFO:tasks.workunit.client.0.smithi096.stdout:0/672: chown d7/de/d5b/d64/fb6 19 1 2024-04-22T15:18:49.368 INFO:tasks.workunit.client.0.smithi096.stdout:4/542: creat d5/d7/d1f/d84/d6d/fbb x:0 0 0 2024-04-22T15:18:49.369 INFO:tasks.workunit.client.1.smithi196.stdout:3/89: fdatasync d0/d7/d8/de/ff 0 2024-04-22T15:18:49.375 INFO:tasks.workunit.client.1.smithi196.stdout:5/87: fdatasync d1/fe 0 2024-04-22T15:18:49.376 INFO:tasks.workunit.client.1.smithi196.stdout:8/140: fsync d2/dc/fe 0 2024-04-22T15:18:49.380 INFO:tasks.workunit.client.1.smithi196.stdout:5/88: dwrite d1/f19 [0,4194304] 0 2024-04-22T15:18:49.382 INFO:tasks.workunit.client.1.smithi196.stdout:0/114: mknod d3/c2f 0 2024-04-22T15:18:49.384 INFO:tasks.workunit.client.1.smithi196.stdout:7/107: write fe [2363486,53338] 0 2024-04-22T15:18:49.385 INFO:tasks.workunit.client.1.smithi196.stdout:7/108: truncate fd 702285 0 2024-04-22T15:18:49.385 INFO:tasks.workunit.client.1.smithi196.stdout:6/127: link dd/c1e dd/d1f/c25 0 2024-04-22T15:18:49.386 INFO:tasks.workunit.client.1.smithi196.stdout:7/109: stat c3 0 2024-04-22T15:18:49.388 INFO:tasks.workunit.client.1.smithi196.stdout:1/96: truncate d1/db/dc/de/f14 376991 0 2024-04-22T15:18:49.390 INFO:tasks.workunit.client.1.smithi196.stdout:3/90: symlink d0/d7/d8/de/l14 0 2024-04-22T15:18:49.395 INFO:tasks.workunit.client.1.smithi196.stdout:5/89: rename f0 to d1/d9/f20 0 2024-04-22T15:18:49.396 INFO:tasks.workunit.client.1.smithi196.stdout:9/112: truncate db/ff 1688940 0 2024-04-22T15:18:49.396 INFO:tasks.workunit.client.1.smithi196.stdout:5/90: dread - d1/f1f zero size 2024-04-22T15:18:49.396 INFO:tasks.workunit.client.1.smithi196.stdout:9/113: chown db/d18/f23 3046147 1 2024-04-22T15:18:49.397 INFO:tasks.workunit.client.0.smithi096.stdout:9/429: write d1/d4/d9/dd/f46 [374473,70904] 0 2024-04-22T15:18:49.397 INFO:tasks.workunit.client.1.smithi196.stdout:6/128: fdatasync fc 0 2024-04-22T15:18:49.397 INFO:tasks.workunit.client.1.smithi196.stdout:5/91: stat d1/d9/c10 0 2024-04-22T15:18:49.398 INFO:tasks.workunit.client.0.smithi096.stdout:9/430: read - d1/d4/d9/d43/d57/f63 zero size 2024-04-22T15:18:49.398 INFO:tasks.workunit.client.0.smithi096.stdout:9/431: stat d1/f20 0 2024-04-22T15:18:49.398 INFO:tasks.workunit.client.1.smithi196.stdout:6/129: chown dd/c13 6599 1 2024-04-22T15:18:49.398 INFO:tasks.workunit.client.1.smithi196.stdout:3/91: stat d0/d6/db/d13 0 2024-04-22T15:18:49.399 INFO:tasks.workunit.client.1.smithi196.stdout:6/130: chown dd/f16 0 1 2024-04-22T15:18:49.399 INFO:tasks.workunit.client.0.smithi096.stdout:9/432: dread d1/d4/d9/d43/d57/d5b/f77 [0,4194304] 0 2024-04-22T15:18:49.401 INFO:tasks.workunit.client.1.smithi196.stdout:6/131: dread dd/f12 [0,4194304] 0 2024-04-22T15:18:49.403 INFO:tasks.workunit.client.1.smithi196.stdout:0/115: mkdir d3/de/d1c/d27/d2c/d30 0 2024-04-22T15:18:49.404 INFO:tasks.workunit.client.0.smithi096.stdout:4/543: rmdir d5 39 2024-04-22T15:18:49.404 INFO:tasks.workunit.client.1.smithi196.stdout:0/116: chown d3/d9/f21 656 1 2024-04-22T15:18:49.406 INFO:tasks.workunit.client.1.smithi196.stdout:9/114: creat db/d18/f24 x:0 0 0 2024-04-22T15:18:49.407 INFO:tasks.workunit.client.1.smithi196.stdout:5/92: creat d1/d17/f21 x:0 0 0 2024-04-22T15:18:49.409 INFO:tasks.workunit.client.0.smithi096.stdout:4/544: dread - d5/d7/d2c/d3d/d4f/f91 zero size 2024-04-22T15:18:49.410 INFO:tasks.workunit.client.1.smithi196.stdout:4/127: dwrite d6/f22 [0,4194304] 0 2024-04-22T15:18:49.412 INFO:tasks.workunit.client.1.smithi196.stdout:7/110: rename d14/d15/l1d to d14/l24 0 2024-04-22T15:18:49.413 INFO:tasks.workunit.client.0.smithi096.stdout:5/529: write d4/f3c [2981697,8960] 0 2024-04-22T15:18:49.413 INFO:tasks.workunit.client.1.smithi196.stdout:7/111: write f13 [366340,41927] 0 2024-04-22T15:18:49.413 INFO:tasks.workunit.client.0.smithi096.stdout:0/673: dwrite d7/de/d5b/d95/dc/d16/f1f [0,4194304] 0 2024-04-22T15:18:49.414 INFO:tasks.workunit.client.1.smithi196.stdout:2/145: write d1/d1d/f29 [977277,101452] 0 2024-04-22T15:18:49.414 INFO:tasks.workunit.client.0.smithi096.stdout:9/433: symlink d1/df/l91 0 2024-04-22T15:18:49.417 INFO:tasks.workunit.client.0.smithi096.stdout:9/434: write d1/f8a [738674,115643] 0 2024-04-22T15:18:49.418 INFO:tasks.workunit.client.1.smithi196.stdout:9/115: creat db/d11/f25 x:0 0 0 2024-04-22T15:18:49.420 INFO:tasks.workunit.client.1.smithi196.stdout:9/116: write db/dd/f1a [632108,111535] 0 2024-04-22T15:18:49.421 INFO:tasks.workunit.client.1.smithi196.stdout:9/117: readlink db/d18/l20 0 2024-04-22T15:18:49.421 INFO:tasks.workunit.client.1.smithi196.stdout:5/93: creat d1/d4/f22 x:0 0 0 2024-04-22T15:18:49.422 INFO:tasks.workunit.client.1.smithi196.stdout:5/94: stat d1/d2/ff 0 2024-04-22T15:18:49.423 INFO:tasks.workunit.client.0.smithi096.stdout:4/545: dwrite d5/d7/d2c/d67/fb3 [0,4194304] 0 2024-04-22T15:18:49.423 INFO:tasks.workunit.client.1.smithi196.stdout:9/118: write db/dd/de/f10 [649084,47948] 0 2024-04-22T15:18:49.423 INFO:tasks.workunit.client.1.smithi196.stdout:1/97: dread d1/db/dc/de/f14 [0,4194304] 0 2024-04-22T15:18:49.424 INFO:tasks.workunit.client.1.smithi196.stdout:5/95: read d1/d4/f6 [1412195,129837] 0 2024-04-22T15:18:49.425 INFO:tasks.workunit.client.1.smithi196.stdout:1/98: write d1/db/ff [2740468,104299] 0 2024-04-22T15:18:49.428 INFO:tasks.workunit.client.1.smithi196.stdout:5/96: dread d1/d2/f5 [0,4194304] 0 2024-04-22T15:18:49.429 INFO:tasks.workunit.client.1.smithi196.stdout:5/97: chown d1/f19 191027691 1 2024-04-22T15:18:49.429 INFO:tasks.workunit.client.1.smithi196.stdout:4/128: creat d6/d14/f27 x:0 0 0 2024-04-22T15:18:49.431 INFO:tasks.workunit.client.1.smithi196.stdout:5/98: truncate d1/d14/f16 190205 0 2024-04-22T15:18:49.431 INFO:tasks.workunit.client.0.smithi096.stdout:0/674: rename d7/de/d5b/d95/dc/d16/d65/fcd to d7/de/d5b/d95/d59/fe4 0 2024-04-22T15:18:49.433 INFO:tasks.workunit.client.0.smithi096.stdout:4/546: fdatasync d5/d7/d1f/f50 0 2024-04-22T15:18:49.436 INFO:tasks.workunit.client.1.smithi196.stdout:4/129: dwrite d6/d13/f1d [0,4194304] 0 2024-04-22T15:18:49.436 INFO:tasks.workunit.client.1.smithi196.stdout:4/130: read - d6/d14/f27 zero size 2024-04-22T15:18:49.438 INFO:tasks.workunit.client.1.smithi196.stdout:7/112: mkdir d14/d25 0 2024-04-22T15:18:49.439 INFO:tasks.workunit.client.1.smithi196.stdout:7/113: dread fd [0,4194304] 0 2024-04-22T15:18:49.441 INFO:tasks.workunit.client.1.smithi196.stdout:6/132: link dd/f1c dd/f26 0 2024-04-22T15:18:49.441 INFO:tasks.workunit.client.1.smithi196.stdout:6/133: read dd/f14 [813130,80776] 0 2024-04-22T15:18:49.449 INFO:tasks.workunit.client.1.smithi196.stdout:9/119: stat f2 0 2024-04-22T15:18:49.450 INFO:tasks.workunit.client.0.smithi096.stdout:9/435: dread d1/d4/d7f/f83 [0,4194304] 0 2024-04-22T15:18:49.451 INFO:tasks.workunit.client.0.smithi096.stdout:9/436: chown d1/d4/d9/dd/f41 1 1 2024-04-22T15:18:49.453 INFO:tasks.workunit.client.1.smithi196.stdout:1/99: mknod d1/db/d16/c25 0 2024-04-22T15:18:49.454 INFO:tasks.workunit.client.0.smithi096.stdout:5/530: link d4/d11/c17 d4/d11/d3d/d4d/d99/cad 0 2024-04-22T15:18:49.459 INFO:tasks.workunit.client.1.smithi196.stdout:1/100: dwrite d1/db/dc/f18 [0,4194304] 0 2024-04-22T15:18:49.459 INFO:tasks.workunit.client.0.smithi096.stdout:9/437: chown d1/c10 1218586407 1 2024-04-22T15:18:49.460 INFO:tasks.workunit.client.1.smithi196.stdout:5/99: truncate d1/fe 1610643 0 2024-04-22T15:18:49.460 INFO:tasks.workunit.client.1.smithi196.stdout:5/100: readlink d1/ld 0 2024-04-22T15:18:49.460 INFO:tasks.workunit.client.0.smithi096.stdout:5/531: dwrite d4/d11/d3d/d4d/f56 [0,4194304] 0 2024-04-22T15:18:49.462 INFO:tasks.workunit.client.1.smithi196.stdout:4/131: creat d6/d14/d18/f28 x:0 0 0 2024-04-22T15:18:49.462 INFO:tasks.workunit.client.1.smithi196.stdout:4/132: chown d6/f22 10073 1 2024-04-22T15:18:49.463 INFO:tasks.workunit.client.0.smithi096.stdout:0/675: getdents d7/de/d3a/d61 0 2024-04-22T15:18:49.470 INFO:tasks.workunit.client.1.smithi196.stdout:7/114: stat d14/l24 0 2024-04-22T15:18:49.473 INFO:tasks.workunit.client.1.smithi196.stdout:6/134: rmdir dd/d20 39 2024-04-22T15:18:49.479 INFO:tasks.workunit.client.0.smithi096.stdout:5/532: dread d4/d11/d3d/d4d/d59/d5a/f79 [0,4194304] 0 2024-04-22T15:18:49.479 INFO:tasks.workunit.client.1.smithi196.stdout:6/135: dwrite dd/f18 [0,4194304] 0 2024-04-22T15:18:49.480 INFO:tasks.workunit.client.1.smithi196.stdout:3/92: rename d0/d7/d8/de to d0/d15 0 2024-04-22T15:18:49.481 INFO:tasks.workunit.client.0.smithi096.stdout:4/547: dwrite d5/d7/d1a/f6c [0,4194304] 0 2024-04-22T15:18:49.482 INFO:tasks.workunit.client.0.smithi096.stdout:0/676: dwrite d7/de/d3a/f76 [0,4194304] 0 2024-04-22T15:18:49.484 INFO:tasks.workunit.client.1.smithi196.stdout:1/101: rmdir d1/db/dc 39 2024-04-22T15:18:49.484 INFO:tasks.workunit.client.0.smithi096.stdout:0/677: chown d7/d42/d4c/d55/d9c 416631650 1 2024-04-22T15:18:49.487 INFO:tasks.workunit.client.1.smithi196.stdout:1/102: dread d1/f4 [0,4194304] 0 2024-04-22T15:18:49.487 INFO:tasks.workunit.client.0.smithi096.stdout:9/438: getdents d1/d4/d79 0 2024-04-22T15:18:49.493 INFO:tasks.workunit.client.1.smithi196.stdout:4/133: mknod d6/d13/c29 0 2024-04-22T15:18:49.494 INFO:tasks.workunit.client.1.smithi196.stdout:4/134: stat d6/d10/f1c 0 2024-04-22T15:18:49.495 INFO:tasks.workunit.client.1.smithi196.stdout:4/135: fsync d6/d13/f24 0 2024-04-22T15:18:49.495 INFO:tasks.workunit.client.0.smithi096.stdout:0/678: creat d7/de/d5b/d62/d67/fe5 x:0 0 0 2024-04-22T15:18:49.497 INFO:tasks.workunit.client.0.smithi096.stdout:9/439: unlink d1/d4/d9/d43/c61 0 2024-04-22T15:18:49.499 INFO:tasks.workunit.client.1.smithi196.stdout:7/115: mkdir d14/d26 0 2024-04-22T15:18:49.502 INFO:tasks.workunit.client.0.smithi096.stdout:9/440: unlink d1/c10 0 2024-04-22T15:18:49.504 INFO:tasks.workunit.client.0.smithi096.stdout:5/533: creat d4/d11/d3d/d4d/fae x:0 0 0 2024-04-22T15:18:49.506 INFO:tasks.workunit.client.1.smithi196.stdout:0/117: rename d3/de/f28 to d3/de/d1c/f31 0 2024-04-22T15:18:49.506 INFO:tasks.workunit.client.1.smithi196.stdout:8/141: sync 2024-04-22T15:18:49.512 INFO:tasks.workunit.client.1.smithi196.stdout:4/136: creat d6/d14/f2a x:0 0 0 2024-04-22T15:18:49.512 INFO:tasks.workunit.client.1.smithi196.stdout:7/116: symlink d14/d15/l27 0 2024-04-22T15:18:49.513 INFO:tasks.workunit.client.1.smithi196.stdout:2/146: getdents d1 0 2024-04-22T15:18:49.514 INFO:tasks.workunit.client.1.smithi196.stdout:3/93: creat d0/d6/db/d13/f16 x:0 0 0 2024-04-22T15:18:49.515 INFO:tasks.workunit.client.1.smithi196.stdout:2/147: chown d1/d13/d1f/f20 0 1 2024-04-22T15:18:49.515 INFO:tasks.workunit.client.1.smithi196.stdout:8/142: dread - d2/d8/f27 zero size 2024-04-22T15:18:49.516 INFO:tasks.workunit.client.1.smithi196.stdout:6/136: dread f1 [0,4194304] 0 2024-04-22T15:18:49.516 INFO:tasks.workunit.client.1.smithi196.stdout:0/118: unlink d3/de/f1d 0 2024-04-22T15:18:49.518 INFO:tasks.workunit.client.1.smithi196.stdout:0/119: chown d3/f5 0 1 2024-04-22T15:18:49.519 INFO:tasks.workunit.client.1.smithi196.stdout:1/103: mkdir d1/db/dc/d26 0 2024-04-22T15:18:49.524 INFO:tasks.workunit.client.0.smithi096.stdout:0/679: dread d7/de/d5b/d95/dc/f2f [0,4194304] 0 2024-04-22T15:18:49.524 INFO:tasks.workunit.client.1.smithi196.stdout:4/137: unlink d6/f22 0 2024-04-22T15:18:49.525 INFO:tasks.workunit.client.1.smithi196.stdout:3/94: creat d0/d15/f17 x:0 0 0 2024-04-22T15:18:49.525 INFO:tasks.workunit.client.1.smithi196.stdout:7/117: dwrite d14/d15/f18 [0,4194304] 0 2024-04-22T15:18:49.527 INFO:tasks.workunit.client.1.smithi196.stdout:0/120: readlink d3/d6/l7 0 2024-04-22T15:18:49.528 INFO:tasks.workunit.client.0.smithi096.stdout:0/680: mknod d7/de/d5b/d95/d15/d35/ce6 0 2024-04-22T15:18:49.528 INFO:tasks.workunit.client.0.smithi096.stdout:0/681: chown d7/d42/fb2 1828363 1 2024-04-22T15:18:49.530 INFO:tasks.workunit.client.1.smithi196.stdout:5/101: dwrite d1/d4/f6 [0,4194304] 0 2024-04-22T15:18:49.532 INFO:tasks.workunit.client.1.smithi196.stdout:8/143: dwrite d2/dc/fe [4194304,4194304] 0 2024-04-22T15:18:49.532 INFO:tasks.workunit.client.1.smithi196.stdout:6/137: mknod dd/d1f/c27 0 2024-04-22T15:18:49.533 INFO:tasks.workunit.client.0.smithi096.stdout:0/682: chown d7/d42/d4c/d55/l89 4548 1 2024-04-22T15:18:49.533 INFO:tasks.workunit.client.0.smithi096.stdout:9/441: write d1/d4/d9/d43/d57/f8d [4723652,36464] 0 2024-04-22T15:18:49.535 INFO:tasks.workunit.client.0.smithi096.stdout:9/442: read d1/d4/d9/f52 [3737667,7698] 0 2024-04-22T15:18:49.538 INFO:tasks.workunit.client.0.smithi096.stdout:5/534: dwrite d4/d11/d3d/d4d/d59/d5a/d5c/f71 [0,4194304] 0 2024-04-22T15:18:49.539 INFO:tasks.workunit.client.1.smithi196.stdout:1/104: dwrite d1/f4 [0,4194304] 0 2024-04-22T15:18:49.539 INFO:tasks.workunit.client.1.smithi196.stdout:7/118: dwrite d14/d15/f18 [0,4194304] 0 2024-04-22T15:18:49.540 INFO:tasks.workunit.client.1.smithi196.stdout:7/119: readlink d14/d15/l27 0 2024-04-22T15:18:49.540 INFO:tasks.workunit.client.1.smithi196.stdout:7/120: stat ff 0 2024-04-22T15:18:49.541 INFO:tasks.workunit.client.0.smithi096.stdout:5/535: truncate d4/d11/d3d/d4d/f56 4762586 0 2024-04-22T15:18:49.542 INFO:tasks.workunit.client.1.smithi196.stdout:3/95: read d0/f4 [800457,48981] 0 2024-04-22T15:18:49.552 INFO:tasks.workunit.client.0.smithi096.stdout:0/683: mknod d7/de/d5b/d95/dc/d16/d65/d7c/ce7 0 2024-04-22T15:18:49.553 INFO:tasks.workunit.client.1.smithi196.stdout:9/120: sync 2024-04-22T15:18:49.554 INFO:tasks.workunit.client.1.smithi196.stdout:1/105: dwrite d1/db/d16/d1e/f22 [0,4194304] 0 2024-04-22T15:18:49.554 INFO:tasks.workunit.client.0.smithi096.stdout:5/536: rmdir d4/d11/d3d/d4d/d59/d98 39 2024-04-22T15:18:49.556 INFO:tasks.workunit.client.1.smithi196.stdout:9/121: fdatasync db/dd/de/f10 0 2024-04-22T15:18:49.556 INFO:tasks.workunit.client.1.smithi196.stdout:1/106: chown d1/db/dc/de/l12 14205709 1 2024-04-22T15:18:49.558 INFO:tasks.workunit.client.1.smithi196.stdout:4/138: rmdir d6 39 2024-04-22T15:18:49.561 INFO:tasks.workunit.client.1.smithi196.stdout:8/144: mknod d2/d8/c2a 0 2024-04-22T15:18:49.561 INFO:tasks.workunit.client.1.smithi196.stdout:8/145: fsync d2/dc/fd 0 2024-04-22T15:18:49.561 INFO:tasks.workunit.client.1.smithi196.stdout:8/146: dread d2/dc/f20 [0,4194304] 0 2024-04-22T15:18:49.562 INFO:tasks.workunit.client.0.smithi096.stdout:0/684: rmdir d7/de/d5b/d95/dc/d16/d18 39 2024-04-22T15:18:49.565 INFO:tasks.workunit.client.1.smithi196.stdout:7/121: dread f7 [0,4194304] 0 2024-04-22T15:18:49.573 INFO:tasks.workunit.client.1.smithi196.stdout:0/121: fdatasync d3/de/f18 0 2024-04-22T15:18:49.574 INFO:tasks.workunit.client.1.smithi196.stdout:9/122: dread db/d18/f1f [0,4194304] 0 2024-04-22T15:18:49.576 INFO:tasks.workunit.client.0.smithi096.stdout:5/537: link d4/d2c/d2f/d31/d82/c85 d4/d11/d3d/d35/caf 0 2024-04-22T15:18:49.576 INFO:tasks.workunit.client.1.smithi196.stdout:6/138: creat dd/d20/f28 x:0 0 0 2024-04-22T15:18:49.577 INFO:tasks.workunit.client.0.smithi096.stdout:9/443: unlink d1/d4/d9/d43/d57/c71 0 2024-04-22T15:18:49.578 INFO:tasks.workunit.client.0.smithi096.stdout:0/685: creat d7/de/fe8 x:0 0 0 2024-04-22T15:18:49.584 INFO:tasks.workunit.client.0.smithi096.stdout:0/686: dwrite d7/da7/fa8 [0,4194304] 0 2024-04-22T15:18:49.585 INFO:tasks.workunit.client.0.smithi096.stdout:8/428: sync 2024-04-22T15:18:49.590 INFO:tasks.workunit.client.0.smithi096.stdout:5/538: mkdir d4/d11/d3d/d4d/d59/db0 0 2024-04-22T15:18:49.590 INFO:tasks.workunit.client.1.smithi196.stdout:8/147: stat d2/c9 0 2024-04-22T15:18:49.594 INFO:tasks.workunit.client.1.smithi196.stdout:3/96: mkdir d0/d6/db/dc/d18 0 2024-04-22T15:18:49.596 INFO:tasks.workunit.client.0.smithi096.stdout:8/429: creat d4/d2a/d7e/f9a x:0 0 0 2024-04-22T15:18:49.597 INFO:tasks.workunit.client.0.smithi096.stdout:8/430: fdatasync d4/d10/d4d/f84 0 2024-04-22T15:18:49.598 INFO:tasks.workunit.client.0.smithi096.stdout:8/431: fdatasync d4/d10/d25/d38/f4b 0 2024-04-22T15:18:49.598 INFO:tasks.workunit.client.1.smithi196.stdout:0/122: creat d3/de/d1c/d27/f32 x:0 0 0 2024-04-22T15:18:49.601 INFO:tasks.workunit.client.0.smithi096.stdout:0/687: unlink d7/de/d5b/d95/d59/cac 0 2024-04-22T15:18:49.602 INFO:tasks.workunit.client.1.smithi196.stdout:6/139: read dd/f16 [104031,73158] 0 2024-04-22T15:18:49.602 INFO:tasks.workunit.client.1.smithi196.stdout:6/140: chown dd/f1c 17502 1 2024-04-22T15:18:49.605 INFO:tasks.workunit.client.1.smithi196.stdout:5/102: link d1/ld d1/d9/l23 0 2024-04-22T15:18:49.606 INFO:tasks.workunit.client.1.smithi196.stdout:5/103: read - d1/f1f zero size 2024-04-22T15:18:49.610 INFO:tasks.workunit.client.0.smithi096.stdout:9/444: link d1/df/f65 d1/d4/d9/dd/f92 0 2024-04-22T15:18:49.612 INFO:tasks.workunit.client.0.smithi096.stdout:9/445: read d1/d4/d9/dd/f23 [132911,128092] 0 2024-04-22T15:18:49.613 INFO:tasks.workunit.client.0.smithi096.stdout:5/539: link d4/d2c/d2f/f9b d4/d11/d3d/d4d/fb1 0 2024-04-22T15:18:49.614 INFO:tasks.workunit.client.1.smithi196.stdout:3/97: creat d0/d7/d8/f19 x:0 0 0 2024-04-22T15:18:49.616 INFO:tasks.workunit.client.0.smithi096.stdout:9/446: dread d1/f4f [0,4194304] 0 2024-04-22T15:18:49.617 INFO:tasks.workunit.client.0.smithi096.stdout:9/447: stat d1/d25/f48 0 2024-04-22T15:18:49.617 INFO:tasks.workunit.client.1.smithi196.stdout:0/123: fsync d3/de/d1c/f31 0 2024-04-22T15:18:49.618 INFO:tasks.workunit.client.0.smithi096.stdout:0/688: dread d7/de/d5b/d95/d15/d35/fb4 [0,4194304] 0 2024-04-22T15:18:49.618 INFO:tasks.workunit.client.0.smithi096.stdout:9/448: chown d1/d4/d9/dd/f21 1161414 1 2024-04-22T15:18:49.621 INFO:tasks.workunit.client.1.smithi196.stdout:1/107: truncate d1/db/d16/d1e/f22 3437400 0 2024-04-22T15:18:49.623 INFO:tasks.workunit.client.1.smithi196.stdout:6/141: creat dd/d20/f29 x:0 0 0 2024-04-22T15:18:49.624 INFO:tasks.workunit.client.1.smithi196.stdout:6/142: truncate dd/d20/f29 155618 0 2024-04-22T15:18:49.628 INFO:tasks.workunit.client.0.smithi096.stdout:0/689: rename d7/de/d5b/d95/dc/f57 to d7/da7/fe9 0 2024-04-22T15:18:49.628 INFO:tasks.workunit.client.1.smithi196.stdout:6/143: dwrite fc [0,4194304] 0 2024-04-22T15:18:49.629 INFO:tasks.workunit.client.0.smithi096.stdout:5/540: mkdir d4/d11/d3d/d4d/d59/db0/db2 0 2024-04-22T15:18:49.632 INFO:tasks.workunit.client.1.smithi196.stdout:8/148: rmdir d2/d5 39 2024-04-22T15:18:49.634 INFO:tasks.workunit.client.0.smithi096.stdout:8/432: getdents d4/d1c/d40/d46/d58 0 2024-04-22T15:18:49.634 INFO:tasks.workunit.client.0.smithi096.stdout:9/449: dread d1/d4/f11 [0,4194304] 0 2024-04-22T15:18:49.635 INFO:tasks.workunit.client.1.smithi196.stdout:2/148: sync 2024-04-22T15:18:49.635 INFO:tasks.workunit.client.1.smithi196.stdout:2/149: fsync d1/f12 0 2024-04-22T15:18:49.635 INFO:tasks.workunit.client.0.smithi096.stdout:8/433: stat d4/c86 0 2024-04-22T15:18:49.635 INFO:tasks.workunit.client.0.smithi096.stdout:9/450: chown d1/c3e 0 1 2024-04-22T15:18:49.636 INFO:tasks.workunit.client.1.smithi196.stdout:3/98: creat d0/d7/f1a x:0 0 0 2024-04-22T15:18:49.636 INFO:tasks.workunit.client.1.smithi196.stdout:3/99: chown d0/d15/f17 56633164 1 2024-04-22T15:18:49.640 INFO:tasks.workunit.client.1.smithi196.stdout:9/123: link db/d11/c19 db/dd/de/c26 0 2024-04-22T15:18:49.642 INFO:tasks.workunit.client.0.smithi096.stdout:5/541: read - d4/d1f/d80/f87 zero size 2024-04-22T15:18:49.646 INFO:tasks.workunit.client.1.smithi196.stdout:4/139: creat d6/f2b x:0 0 0 2024-04-22T15:18:49.649 INFO:tasks.workunit.client.0.smithi096.stdout:8/434: mkdir d4/d1c/d40/d46/d9b 0 2024-04-22T15:18:49.650 INFO:tasks.workunit.client.0.smithi096.stdout:8/435: dread - d4/d10/d25/d4c/d76/d7f/f92 zero size 2024-04-22T15:18:49.653 INFO:tasks.workunit.client.0.smithi096.stdout:9/451: write d1/d4/d9/d43/d57/d5b/f77 [856247,115775] 0 2024-04-22T15:18:49.654 INFO:tasks.workunit.client.0.smithi096.stdout:5/542: rmdir d4/d11/d3d/d4d/d59/d98 39 2024-04-22T15:18:49.655 INFO:tasks.workunit.client.1.smithi196.stdout:7/122: sync 2024-04-22T15:18:49.656 INFO:tasks.workunit.client.1.smithi196.stdout:7/123: write f4 [4947231,76700] 0 2024-04-22T15:18:49.656 INFO:tasks.workunit.client.1.smithi196.stdout:5/104: dwrite d1/d9/f20 [0,4194304] 0 2024-04-22T15:18:49.657 INFO:tasks.workunit.client.0.smithi096.stdout:8/436: mkdir d4/d10/d25/d38/d8f/d9c 0 2024-04-22T15:18:49.657 INFO:tasks.workunit.client.0.smithi096.stdout:0/690: dread d7/de/d5b/d95/dc/d16/d65/d7c/fbd [0,4194304] 0 2024-04-22T15:18:49.657 INFO:tasks.workunit.client.1.smithi196.stdout:7/124: write fa [1386268,11425] 0 2024-04-22T15:18:49.659 INFO:tasks.workunit.client.1.smithi196.stdout:9/124: symlink db/l27 0 2024-04-22T15:18:49.659 INFO:tasks.workunit.client.1.smithi196.stdout:2/150: write d1/f1b [850116,71471] 0 2024-04-22T15:18:49.660 INFO:tasks.workunit.client.0.smithi096.stdout:9/452: mknod d1/d4/d9/d43/d57/d5e/c93 0 2024-04-22T15:18:49.663 INFO:tasks.workunit.client.0.smithi096.stdout:0/691: creat d7/de/d3a/d61/dbb/fea x:0 0 0 2024-04-22T15:18:49.664 INFO:tasks.workunit.client.1.smithi196.stdout:5/105: dwrite d1/d4/f22 [0,4194304] 0 2024-04-22T15:18:49.664 INFO:tasks.workunit.client.1.smithi196.stdout:4/140: mkdir d6/d13/d15/d2c 0 2024-04-22T15:18:49.665 INFO:tasks.workunit.client.1.smithi196.stdout:4/141: fsync d6/d13/f25 0 2024-04-22T15:18:49.666 INFO:tasks.workunit.client.1.smithi196.stdout:8/149: write d2/d5/f19 [428506,47721] 0 2024-04-22T15:18:49.666 INFO:tasks.workunit.client.1.smithi196.stdout:4/142: fsync d6/d13/f1d 0 2024-04-22T15:18:49.666 INFO:tasks.workunit.client.0.smithi096.stdout:9/453: creat d1/d85/f94 x:0 0 0 2024-04-22T15:18:49.668 INFO:tasks.workunit.client.0.smithi096.stdout:9/454: readlink d1/d4/d9/dd/l6d 0 2024-04-22T15:18:49.669 INFO:tasks.workunit.client.1.smithi196.stdout:0/124: creat d3/d6/f33 x:0 0 0 2024-04-22T15:18:49.670 INFO:tasks.workunit.client.1.smithi196.stdout:3/100: creat d0/d6/db/dc/d18/f1b x:0 0 0 2024-04-22T15:18:49.671 INFO:tasks.workunit.client.1.smithi196.stdout:3/101: write d0/d6/db/d13/f16 [504804,13608] 0 2024-04-22T15:18:49.672 INFO:tasks.workunit.client.1.smithi196.stdout:9/125: truncate f5 2661135 0 2024-04-22T15:18:49.674 INFO:tasks.workunit.client.1.smithi196.stdout:2/151: creat d1/d13/d1f/f3a x:0 0 0 2024-04-22T15:18:49.674 INFO:tasks.workunit.client.0.smithi096.stdout:8/437: dwrite d4/d10/d25/f81 [4194304,4194304] 0 2024-04-22T15:18:49.676 INFO:tasks.workunit.client.0.smithi096.stdout:0/692: creat d7/de/d5b/d95/dc/d16/feb x:0 0 0 2024-04-22T15:18:49.677 INFO:tasks.workunit.client.0.smithi096.stdout:8/438: chown d4/d1c/d40/d46 82483 1 2024-04-22T15:18:49.682 INFO:tasks.workunit.client.1.smithi196.stdout:4/143: symlink d6/d10/l2d 0 2024-04-22T15:18:49.683 INFO:tasks.workunit.client.0.smithi096.stdout:5/543: dread d4/d11/d3d/d4f/f69 [0,4194304] 0 2024-04-22T15:18:49.685 INFO:tasks.workunit.client.1.smithi196.stdout:0/125: dread d3/de/f18 [0,4194304] 0 2024-04-22T15:18:49.685 INFO:tasks.workunit.client.1.smithi196.stdout:0/126: dread - d3/d9/f21 zero size 2024-04-22T15:18:49.686 INFO:tasks.workunit.client.1.smithi196.stdout:0/127: write d3/fd [3205717,707] 0 2024-04-22T15:18:49.686 INFO:tasks.workunit.client.1.smithi196.stdout:1/108: getdents d1/db/d16 0 2024-04-22T15:18:49.687 INFO:tasks.workunit.client.0.smithi096.stdout:8/439: fsync d4/d10/d25/f6c 0 2024-04-22T15:18:49.689 INFO:tasks.workunit.client.1.smithi196.stdout:1/109: read d1/db/ff [3556428,44903] 0 2024-04-22T15:18:49.690 INFO:tasks.workunit.client.1.smithi196.stdout:9/126: creat db/f28 x:0 0 0 2024-04-22T15:18:49.691 INFO:tasks.workunit.client.0.smithi096.stdout:5/544: truncate d4/d2c/d2f/f3a 1112317 0 2024-04-22T15:18:49.693 INFO:tasks.workunit.client.0.smithi096.stdout:5/545: truncate d4/d2c/d2f/f9b 243802 0 2024-04-22T15:18:49.694 INFO:tasks.workunit.client.1.smithi196.stdout:2/152: unlink d1/f12 0 2024-04-22T15:18:49.694 INFO:tasks.workunit.client.0.smithi096.stdout:9/455: getdents d1/d4/d9/d43/d57/d5e 0 2024-04-22T15:18:49.696 INFO:tasks.workunit.client.1.smithi196.stdout:6/144: getdents dd 0 2024-04-22T15:18:49.702 INFO:tasks.workunit.client.0.smithi096.stdout:8/440: dread d4/d10/d25/f37 [0,4194304] 0 2024-04-22T15:18:49.703 INFO:tasks.workunit.client.0.smithi096.stdout:5/546: mkdir d4/d11/d3d/d35/db3 0 2024-04-22T15:18:49.705 INFO:tasks.workunit.client.0.smithi096.stdout:8/441: truncate d4/d1c/d40/d46/d58/f7a 138126 0 2024-04-22T15:18:49.710 INFO:tasks.workunit.client.1.smithi196.stdout:3/102: link d0/d15/f17 d0/d6/db/dc/f1c 0 2024-04-22T15:18:49.711 INFO:tasks.workunit.client.0.smithi096.stdout:5/547: dwrite d4/d11/d3d/d4d/d59/d5a/f15 [0,4194304] 0 2024-04-22T15:18:49.711 INFO:tasks.workunit.client.0.smithi096.stdout:0/693: getdents d7/de/d5b/d95/d66 0 2024-04-22T15:18:49.711 INFO:tasks.workunit.client.1.smithi196.stdout:8/150: write d2/dc/f20 [244745,109640] 0 2024-04-22T15:18:49.711 INFO:tasks.workunit.client.1.smithi196.stdout:8/151: stat d2/dc 0 2024-04-22T15:18:49.713 INFO:tasks.workunit.client.1.smithi196.stdout:1/110: rmdir d1/db 39 2024-04-22T15:18:49.714 INFO:tasks.workunit.client.0.smithi096.stdout:0/694: fsync d7/da7/fa8 0 2024-04-22T15:18:49.714 INFO:tasks.workunit.client.0.smithi096.stdout:8/442: dwrite d4/d1c/d34/f4f [0,4194304] 0 2024-04-22T15:18:49.716 INFO:tasks.workunit.client.1.smithi196.stdout:9/127: mkdir db/d11/d29 0 2024-04-22T15:18:49.718 INFO:tasks.workunit.client.0.smithi096.stdout:9/456: mkdir d1/df/d95 0 2024-04-22T15:18:49.725 INFO:tasks.workunit.client.1.smithi196.stdout:6/145: unlink dd/f10 0 2024-04-22T15:18:49.728 INFO:tasks.workunit.client.0.smithi096.stdout:5/548: unlink d4/d11/d3d/d4f/f69 0 2024-04-22T15:18:49.729 INFO:tasks.workunit.client.0.smithi096.stdout:5/549: chown d4/d11/d3d/d4d/d59/d5a 52 1 2024-04-22T15:18:49.730 INFO:tasks.workunit.client.1.smithi196.stdout:4/144: fdatasync d6/d10/fe 0 2024-04-22T15:18:49.730 INFO:tasks.workunit.client.0.smithi096.stdout:0/695: fdatasync d7/f20 0 2024-04-22T15:18:49.731 INFO:tasks.workunit.client.1.smithi196.stdout:4/145: dread d6/d10/fe [0,4194304] 0 2024-04-22T15:18:49.731 INFO:tasks.workunit.client.1.smithi196.stdout:0/128: symlink d3/d6/l34 0 2024-04-22T15:18:49.732 INFO:tasks.workunit.client.0.smithi096.stdout:8/443: mknod d4/d10/c9d 0 2024-04-22T15:18:49.735 INFO:tasks.workunit.client.0.smithi096.stdout:9/457: creat d1/d6f/f96 x:0 0 0 2024-04-22T15:18:49.735 INFO:tasks.workunit.client.1.smithi196.stdout:3/103: symlink d0/d6/db/l1d 0 2024-04-22T15:18:49.736 INFO:tasks.workunit.client.0.smithi096.stdout:5/550: creat d4/d11/d3d/d35/d57/fb4 x:0 0 0 2024-04-22T15:18:49.737 INFO:tasks.workunit.client.0.smithi096.stdout:5/551: write d4/fd [6319358,117744] 0 2024-04-22T15:18:49.740 INFO:tasks.workunit.client.1.smithi196.stdout:1/111: chown d1/db/d16/c25 110 1 2024-04-22T15:18:49.740 INFO:tasks.workunit.client.1.smithi196.stdout:8/152: dread d2/f3 [4194304,4194304] 0 2024-04-22T15:18:49.740 INFO:tasks.workunit.client.1.smithi196.stdout:1/112: chown d1/l11 67505251 1 2024-04-22T15:18:49.741 INFO:tasks.workunit.client.1.smithi196.stdout:1/113: chown d1 13521147 1 2024-04-22T15:18:49.741 INFO:tasks.workunit.client.0.smithi096.stdout:0/696: rename d7/d42/d4c/d55/da3/ca9 to d7/de/d3a/d61/dbb/cec 0 2024-04-22T15:18:49.742 INFO:tasks.workunit.client.0.smithi096.stdout:8/444: truncate d4/d1c/f43 447320 0 2024-04-22T15:18:49.746 INFO:tasks.workunit.client.0.smithi096.stdout:0/697: dread d7/de/d5b/d95/dc/d16/d65/d7c/fbd [0,4194304] 0 2024-04-22T15:18:49.748 INFO:tasks.workunit.client.0.smithi096.stdout:5/552: rmdir d4/d11/d3d/d4d/d88 39 2024-04-22T15:18:49.754 INFO:tasks.workunit.client.0.smithi096.stdout:5/553: mknod d4/d11/d3d/d4d/d59/d8a/cb5 0 2024-04-22T15:18:49.754 INFO:tasks.workunit.client.0.smithi096.stdout:5/554: dread - d4/d11/f96 zero size 2024-04-22T15:18:49.754 INFO:tasks.workunit.client.0.smithi096.stdout:5/555: chown d4/d1f 172984 1 2024-04-22T15:18:49.755 INFO:tasks.workunit.client.0.smithi096.stdout:5/556: fsync f2 0 2024-04-22T15:18:49.755 INFO:tasks.workunit.client.0.smithi096.stdout:5/557: chown d4/d11/d3d 0 1 2024-04-22T15:18:49.760 INFO:tasks.workunit.client.1.smithi196.stdout:7/125: sync 2024-04-22T15:18:49.764 INFO:tasks.workunit.client.0.smithi096.stdout:9/458: dwrite d1/d25/f2e [0,4194304] 0 2024-04-22T15:18:49.765 INFO:tasks.workunit.client.1.smithi196.stdout:0/129: rename d3/de/f26 to d3/de/d1c/d27/d2c/d30/f35 0 2024-04-22T15:18:49.765 INFO:tasks.workunit.client.1.smithi196.stdout:8/153: rmdir d2/dc 39 2024-04-22T15:18:49.766 INFO:tasks.workunit.client.1.smithi196.stdout:9/128: creat db/d11/d29/f2a x:0 0 0 2024-04-22T15:18:49.766 INFO:tasks.workunit.client.1.smithi196.stdout:7/126: fsync d14/d15/f1a 0 2024-04-22T15:18:49.767 INFO:tasks.workunit.client.1.smithi196.stdout:9/129: stat db/dd/de/l21 0 2024-04-22T15:18:49.769 INFO:tasks.workunit.client.1.smithi196.stdout:6/146: truncate dd/d20/f28 457496 0 2024-04-22T15:18:49.769 INFO:tasks.workunit.client.0.smithi096.stdout:9/459: dread d1/d25/f7d [0,4194304] 0 2024-04-22T15:18:49.770 INFO:tasks.workunit.client.0.smithi096.stdout:9/460: chown d1 46690965 1 2024-04-22T15:18:49.771 INFO:tasks.workunit.client.0.smithi096.stdout:5/558: write d4/d11/f22 [1582600,76977] 0 2024-04-22T15:18:49.774 INFO:tasks.workunit.client.0.smithi096.stdout:8/445: dwrite d4/d10/d25/f26 [0,4194304] 0 2024-04-22T15:18:49.781 INFO:tasks.workunit.client.0.smithi096.stdout:8/446: mknod d4/d1c/d40/d46/c9e 0 2024-04-22T15:18:49.782 INFO:tasks.workunit.client.0.smithi096.stdout:9/461: creat d1/d4/d9/d43/f97 x:0 0 0 2024-04-22T15:18:49.783 INFO:tasks.workunit.client.0.smithi096.stdout:9/462: stat d1/df 0 2024-04-22T15:18:49.783 INFO:tasks.workunit.client.0.smithi096.stdout:5/559: getdents d4/d11/d3d/d4d/d59/d8a 0 2024-04-22T15:18:49.783 INFO:tasks.workunit.client.0.smithi096.stdout:5/560: readlink d4/d11/l6d 0 2024-04-22T15:18:49.784 INFO:tasks.workunit.client.0.smithi096.stdout:5/561: stat d4/d11/c40 0 2024-04-22T15:18:49.785 INFO:tasks.workunit.client.0.smithi096.stdout:8/447: write d4/d10/f56 [1261803,13987] 0 2024-04-22T15:18:49.785 INFO:tasks.workunit.client.0.smithi096.stdout:9/463: truncate d1/d4/f4e 822739 0 2024-04-22T15:18:49.786 INFO:tasks.workunit.client.0.smithi096.stdout:8/448: chown d4/d10/d25/d4c/c6d 406439 1 2024-04-22T15:18:49.787 INFO:tasks.workunit.client.0.smithi096.stdout:5/562: creat d4/d11/d3d/fb6 x:0 0 0 2024-04-22T15:18:49.788 INFO:tasks.workunit.client.0.smithi096.stdout:9/464: mkdir d1/d6f/d98 0 2024-04-22T15:18:49.791 INFO:tasks.workunit.client.0.smithi096.stdout:9/465: rename d1/l27 to d1/d85/l99 0 2024-04-22T15:18:49.793 INFO:tasks.workunit.client.0.smithi096.stdout:5/563: dread d4/d1f/f60 [0,4194304] 0 2024-04-22T15:18:49.798 INFO:tasks.workunit.client.0.smithi096.stdout:8/449: write d4/d1c/f3a [950899,81523] 0 2024-04-22T15:18:49.800 INFO:tasks.workunit.client.0.smithi096.stdout:5/564: rename d4/d11/d3d/d4d/d59/d8a/cb5 to d4/d11/d3d/d4d/d59/d98/cb7 0 2024-04-22T15:18:49.801 INFO:tasks.workunit.client.0.smithi096.stdout:8/450: fdatasync d4/d2a/f3c 0 2024-04-22T15:18:49.802 INFO:tasks.workunit.client.0.smithi096.stdout:5/565: fsync d4/d11/d3d/d4d/f70 0 2024-04-22T15:18:49.803 INFO:tasks.workunit.client.0.smithi096.stdout:5/566: chown d4/d11/d3d/d4f 1 1 2024-04-22T15:18:49.803 INFO:tasks.workunit.client.1.smithi196.stdout:8/154: mknod d2/d5/d1a/c2b 0 2024-04-22T15:18:49.804 INFO:tasks.workunit.client.0.smithi096.stdout:9/466: write d1/d4/d9/dd/f2b [424834,78561] 0 2024-04-22T15:18:49.805 INFO:tasks.workunit.client.1.smithi196.stdout:2/153: truncate d1/d13/f1e 1227122 0 2024-04-22T15:18:49.808 INFO:tasks.workunit.client.0.smithi096.stdout:9/467: dread d1/d4/f11 [0,4194304] 0 2024-04-22T15:18:49.808 INFO:tasks.workunit.client.1.smithi196.stdout:1/114: dwrite d1/db/dc/de/f14 [0,4194304] 0 2024-04-22T15:18:49.808 INFO:tasks.workunit.client.1.smithi196.stdout:0/130: dread d3/de/d1c/f31 [0,4194304] 0 2024-04-22T15:18:49.808 INFO:tasks.workunit.client.1.smithi196.stdout:7/127: mkdir d14/d26/d28 0 2024-04-22T15:18:49.809 INFO:tasks.workunit.client.1.smithi196.stdout:1/115: chown d1/db/dc/de 7059875 1 2024-04-22T15:18:49.809 INFO:tasks.workunit.client.1.smithi196.stdout:7/128: read f4 [3135255,67995] 0 2024-04-22T15:18:49.811 INFO:tasks.workunit.client.1.smithi196.stdout:1/116: write d1/db/dc/de/f14 [364208,96356] 0 2024-04-22T15:18:49.814 INFO:tasks.workunit.client.1.smithi196.stdout:6/147: mknod dd/c2a 0 2024-04-22T15:18:49.816 INFO:tasks.workunit.client.1.smithi196.stdout:2/154: mknod d1/d13/d25/c3b 0 2024-04-22T15:18:49.818 INFO:tasks.workunit.client.1.smithi196.stdout:1/117: rename d1/db/dc/f18 to d1/db/dc/d26/f27 0 2024-04-22T15:18:49.819 INFO:tasks.workunit.client.1.smithi196.stdout:7/129: dread fd [0,4194304] 0 2024-04-22T15:18:49.820 INFO:tasks.workunit.client.1.smithi196.stdout:6/148: mkdir dd/d2b 0 2024-04-22T15:18:49.820 INFO:tasks.workunit.client.1.smithi196.stdout:2/155: read - d1/d23/f2f zero size 2024-04-22T15:18:49.820 INFO:tasks.workunit.client.1.smithi196.stdout:7/130: read - d14/f1e zero size 2024-04-22T15:18:49.821 INFO:tasks.workunit.client.1.smithi196.stdout:0/131: dwrite d3/de/f24 [0,4194304] 0 2024-04-22T15:18:49.821 INFO:tasks.workunit.client.1.smithi196.stdout:1/118: creat d1/db/dc/d26/f28 x:0 0 0 2024-04-22T15:18:49.822 INFO:tasks.workunit.client.1.smithi196.stdout:2/156: rmdir d1/d13 39 2024-04-22T15:18:49.824 INFO:tasks.workunit.client.1.smithi196.stdout:0/132: mkdir d3/d9/d36 0 2024-04-22T15:18:49.825 INFO:tasks.workunit.client.1.smithi196.stdout:6/149: rename dd/l19 to dd/d2b/l2c 0 2024-04-22T15:18:49.826 INFO:tasks.workunit.client.1.smithi196.stdout:8/155: dread d2/dc/f10 [0,4194304] 0 2024-04-22T15:18:49.826 INFO:tasks.workunit.client.1.smithi196.stdout:7/131: mkdir d14/d26/d28/d29 0 2024-04-22T15:18:49.826 INFO:tasks.workunit.client.0.smithi096.stdout:8/451: dread d4/fd [0,4194304] 0 2024-04-22T15:18:49.827 INFO:tasks.workunit.client.1.smithi196.stdout:1/119: write d1/db/dc/d26/f28 [873500,85059] 0 2024-04-22T15:18:49.828 INFO:tasks.workunit.client.1.smithi196.stdout:0/133: creat d3/de/f37 x:0 0 0 2024-04-22T15:18:49.828 INFO:tasks.workunit.client.1.smithi196.stdout:6/150: stat dd/f15 0 2024-04-22T15:18:49.830 INFO:tasks.workunit.client.1.smithi196.stdout:2/157: creat d1/d1d/f3c x:0 0 0 2024-04-22T15:18:49.832 INFO:tasks.workunit.client.1.smithi196.stdout:6/151: read f4 [4040223,11819] 0 2024-04-22T15:18:49.833 INFO:tasks.workunit.client.0.smithi096.stdout:7/742: sync 2024-04-22T15:18:49.834 INFO:tasks.workunit.client.0.smithi096.stdout:8/452: stat d4/d10/d25/f81 0 2024-04-22T15:18:49.837 INFO:tasks.workunit.client.1.smithi196.stdout:7/132: dwrite fe [0,4194304] 0 2024-04-22T15:18:49.837 INFO:tasks.workunit.client.1.smithi196.stdout:5/106: sync 2024-04-22T15:18:49.838 INFO:tasks.workunit.client.1.smithi196.stdout:0/134: dwrite d3/d6/f33 [0,4194304] 0 2024-04-22T15:18:49.840 INFO:tasks.workunit.client.0.smithi096.stdout:7/743: symlink d31/d32/d33/d6a/d83/dba/lee 0 2024-04-22T15:18:49.841 INFO:tasks.workunit.client.1.smithi196.stdout:6/152: chown dd/f12 248750328 1 2024-04-22T15:18:49.841 INFO:tasks.workunit.client.1.smithi196.stdout:6/153: fdatasync dd/f15 0 2024-04-22T15:18:49.841 INFO:tasks.workunit.client.0.smithi096.stdout:7/744: fsync d31/d32/d33/f5c 0 2024-04-22T15:18:49.843 INFO:tasks.workunit.client.0.smithi096.stdout:7/745: stat d31/d32/d33/d6a/d83/lb7 0 2024-04-22T15:18:49.845 INFO:tasks.workunit.client.1.smithi196.stdout:2/158: creat d1/d13/d25/f3d x:0 0 0 2024-04-22T15:18:49.847 INFO:tasks.workunit.client.1.smithi196.stdout:7/133: mkdir d14/d26/d2a 0 2024-04-22T15:18:49.848 INFO:tasks.workunit.client.0.smithi096.stdout:5/567: dread d4/f26 [0,4194304] 0 2024-04-22T15:18:49.849 INFO:tasks.workunit.client.0.smithi096.stdout:8/453: rename d4/d10/d25/d4c/d76/c91 to d4/d1c/c9f 0 2024-04-22T15:18:49.850 INFO:tasks.workunit.client.1.smithi196.stdout:9/130: write db/ff [271968,98313] 0 2024-04-22T15:18:49.850 INFO:tasks.workunit.client.0.smithi096.stdout:8/454: dread - d4/d1c/d40/f98 zero size 2024-04-22T15:18:49.851 INFO:tasks.workunit.client.1.smithi196.stdout:9/131: write db/d11/f25 [31775,45497] 0 2024-04-22T15:18:49.852 INFO:tasks.workunit.client.1.smithi196.stdout:9/132: readlink db/d18/l20 0 2024-04-22T15:18:49.854 INFO:tasks.workunit.client.0.smithi096.stdout:7/746: mkdir d31/d32/d33/d54/def 0 2024-04-22T15:18:49.854 INFO:tasks.workunit.client.1.smithi196.stdout:5/107: creat d1/d17/f24 x:0 0 0 2024-04-22T15:18:49.855 INFO:tasks.workunit.client.1.smithi196.stdout:5/108: write d1/d9/f11 [972319,41677] 0 2024-04-22T15:18:49.856 INFO:tasks.workunit.client.1.smithi196.stdout:1/120: creat d1/db/f29 x:0 0 0 2024-04-22T15:18:49.857 INFO:tasks.workunit.client.0.smithi096.stdout:8/455: creat d4/d1c/d40/d48/fa0 x:0 0 0 2024-04-22T15:18:49.858 INFO:tasks.workunit.client.0.smithi096.stdout:8/456: chown d4/d1c/d34/f4f 10 1 2024-04-22T15:18:49.859 INFO:tasks.workunit.client.1.smithi196.stdout:5/109: dread d1/f19 [0,4194304] 0 2024-04-22T15:18:49.859 INFO:tasks.workunit.client.0.smithi096.stdout:7/747: mkdir d31/d32/d33/d93/df0 0 2024-04-22T15:18:49.862 INFO:tasks.workunit.client.0.smithi096.stdout:9/468: dread d1/f38 [0,4194304] 0 2024-04-22T15:18:49.864 INFO:tasks.workunit.client.1.smithi196.stdout:9/133: dread fa [0,4194304] 0 2024-04-22T15:18:49.865 INFO:tasks.workunit.client.1.smithi196.stdout:9/134: truncate db/fc 4503203 0 2024-04-22T15:18:49.865 INFO:tasks.workunit.client.1.smithi196.stdout:9/135: dread - db/f28 zero size 2024-04-22T15:18:49.866 INFO:tasks.workunit.client.0.smithi096.stdout:9/469: write d1/f4f [1779619,50848] 0 2024-04-22T15:18:49.867 INFO:tasks.workunit.client.0.smithi096.stdout:9/470: write d1/f32 [3297762,102109] 0 2024-04-22T15:18:49.867 INFO:tasks.workunit.client.1.smithi196.stdout:8/156: write d2/f4 [4210752,88546] 0 2024-04-22T15:18:49.868 INFO:tasks.workunit.client.0.smithi096.stdout:9/471: stat d1/d85 0 2024-04-22T15:18:49.868 INFO:tasks.workunit.client.1.smithi196.stdout:8/157: chown d2/d5/c6 10 1 2024-04-22T15:18:49.873 INFO:tasks.workunit.client.0.smithi096.stdout:5/568: dread d4/f3c [0,4194304] 0 2024-04-22T15:18:49.875 INFO:tasks.workunit.client.1.smithi196.stdout:0/135: write d3/f5 [110106,114870] 0 2024-04-22T15:18:49.876 INFO:tasks.workunit.client.0.smithi096.stdout:9/472: unlink d1/df/c62 0 2024-04-22T15:18:49.878 INFO:tasks.workunit.client.1.smithi196.stdout:7/134: mkdir d14/d26/d2a/d2b 0 2024-04-22T15:18:49.879 INFO:tasks.workunit.client.1.smithi196.stdout:0/136: dread d3/de/f24 [0,4194304] 0 2024-04-22T15:18:49.879 INFO:tasks.workunit.client.1.smithi196.stdout:3/104: sync 2024-04-22T15:18:49.879 INFO:tasks.workunit.client.1.smithi196.stdout:4/146: sync 2024-04-22T15:18:49.880 INFO:tasks.workunit.client.1.smithi196.stdout:9/136: creat db/d11/d29/f2b x:0 0 0 2024-04-22T15:18:49.880 INFO:tasks.workunit.client.1.smithi196.stdout:0/137: stat d3/d9/d36 0 2024-04-22T15:18:49.883 INFO:tasks.workunit.client.0.smithi096.stdout:8/457: dread d4/f12 [0,4194304] 0 2024-04-22T15:18:49.885 INFO:tasks.workunit.client.1.smithi196.stdout:7/135: dwrite fa [0,4194304] 0 2024-04-22T15:18:49.887 INFO:tasks.workunit.client.1.smithi196.stdout:4/147: dwrite d6/d14/d18/f28 [0,4194304] 0 2024-04-22T15:18:49.887 INFO:tasks.workunit.client.1.smithi196.stdout:7/136: write d14/d15/f18 [507987,116730] 0 2024-04-22T15:18:49.889 INFO:tasks.workunit.client.1.smithi196.stdout:6/154: rename dd/c1d to dd/d20/c2d 0 2024-04-22T15:18:49.892 INFO:tasks.workunit.client.1.smithi196.stdout:2/159: creat d1/f3e x:0 0 0 2024-04-22T15:18:49.894 INFO:tasks.workunit.client.0.smithi096.stdout:7/748: rename d31/d32/d33/d6a/d83/f96 to d31/d32/d33/d54/ff1 0 2024-04-22T15:18:49.896 INFO:tasks.workunit.client.1.smithi196.stdout:7/137: dwrite d14/f1e [0,4194304] 0 2024-04-22T15:18:49.897 INFO:tasks.workunit.client.0.smithi096.stdout:9/473: creat d1/d4/d9/d43/d57/d5b/d73/f9a x:0 0 0 2024-04-22T15:18:49.900 INFO:tasks.workunit.client.0.smithi096.stdout:5/569: getdents d4/d2c 0 2024-04-22T15:18:49.902 INFO:tasks.workunit.client.1.smithi196.stdout:5/110: dwrite d1/fe [0,4194304] 0 2024-04-22T15:18:49.903 INFO:tasks.workunit.client.1.smithi196.stdout:0/138: creat d3/de/d1c/d27/f38 x:0 0 0 2024-04-22T15:18:49.905 INFO:tasks.workunit.client.0.smithi096.stdout:9/474: chown d1/d25/c5a 2 1 2024-04-22T15:18:49.907 INFO:tasks.workunit.client.0.smithi096.stdout:9/475: write d1/d4/d9/dd/f46 [113126,61925] 0 2024-04-22T15:18:49.911 INFO:tasks.workunit.client.1.smithi196.stdout:0/139: dwrite d3/de/d1c/d27/f38 [0,4194304] 0 2024-04-22T15:18:49.911 INFO:tasks.workunit.client.1.smithi196.stdout:4/148: stat d6/f19 0 2024-04-22T15:18:49.914 INFO:tasks.workunit.client.1.smithi196.stdout:6/155: creat dd/d2b/f2e x:0 0 0 2024-04-22T15:18:49.916 INFO:tasks.workunit.client.0.smithi096.stdout:9/476: truncate d1/d25/f7d 1458013 0 2024-04-22T15:18:49.920 INFO:tasks.workunit.client.0.smithi096.stdout:8/458: rename d4/d10/d4d/l8d to d4/d1c/d40/la1 0 2024-04-22T15:18:49.922 INFO:tasks.workunit.client.0.smithi096.stdout:8/459: dread - d4/d10/d25/d4c/d76/d7f/f92 zero size 2024-04-22T15:18:49.928 INFO:tasks.workunit.client.0.smithi096.stdout:8/460: dwrite d4/d10/d25/d38/f4b [0,4194304] 0 2024-04-22T15:18:49.929 INFO:tasks.workunit.client.0.smithi096.stdout:8/461: chown d4/d1c/d40/d46/d58/f7a 2390 1 2024-04-22T15:18:49.948 INFO:tasks.workunit.client.1.smithi196.stdout:2/160: creat d1/d13/d25/d2a/f3f x:0 0 0 2024-04-22T15:18:49.980 INFO:tasks.workunit.client.0.smithi096.stdout:7/749: write d31/d32/d33/d6a/f72 [2193962,88662] 0 2024-04-22T15:18:49.982 INFO:tasks.workunit.client.0.smithi096.stdout:8/462: rename d4/d1c/d40/d46/d58/f8c to d4/d1c/d62/fa2 0 2024-04-22T15:18:49.983 INFO:tasks.workunit.client.1.smithi196.stdout:7/138: mknod d14/d15/c2c 0 2024-04-22T15:18:49.986 INFO:tasks.workunit.client.0.smithi096.stdout:9/477: mkdir d1/d6f/d98/d9b 0 2024-04-22T15:18:49.988 INFO:tasks.workunit.client.0.smithi096.stdout:5/570: truncate d4/f3c 2493206 0 2024-04-22T15:18:49.988 INFO:tasks.workunit.client.0.smithi096.stdout:5/571: read - d4/d2c/f9f zero size 2024-04-22T15:18:49.990 INFO:tasks.workunit.client.0.smithi096.stdout:7/750: creat d31/d32/d33/d93/dd5/ff2 x:0 0 0 2024-04-22T15:18:49.991 INFO:tasks.workunit.client.0.smithi096.stdout:7/751: truncate d31/d32/fec 334139 0 2024-04-22T15:18:49.992 INFO:tasks.workunit.client.1.smithi196.stdout:9/137: mkdir db/dd/d1b/d2c 0 2024-04-22T15:18:49.993 INFO:tasks.workunit.client.0.smithi096.stdout:7/752: write d31/d68/f9f [4591435,126784] 0 2024-04-22T15:18:49.993 INFO:tasks.workunit.client.1.smithi196.stdout:9/138: chown db/d11/l12 1277 1 2024-04-22T15:18:49.999 INFO:tasks.workunit.client.1.smithi196.stdout:5/111: rename d1/d2/ff to d1/d2/f25 0 2024-04-22T15:18:50.003 INFO:tasks.workunit.client.0.smithi096.stdout:2/507: sync 2024-04-22T15:18:50.006 INFO:tasks.workunit.client.1.smithi196.stdout:1/121: link d1/la d1/db/l2a 0 2024-04-22T15:18:50.006 INFO:tasks.workunit.client.1.smithi196.stdout:1/122: readlink d1/db/l2a 0 2024-04-22T15:18:50.006 INFO:tasks.workunit.client.1.smithi196.stdout:1/123: fsync d1/db/d16/f1c 0 2024-04-22T15:18:50.007 INFO:tasks.workunit.client.1.smithi196.stdout:1/124: stat d1/db/dc/de/c1b 0 2024-04-22T15:18:50.009 INFO:tasks.workunit.client.1.smithi196.stdout:4/149: creat d6/d13/f2e x:0 0 0 2024-04-22T15:18:50.013 INFO:tasks.workunit.client.0.smithi096.stdout:7/753: rename d31/d32/d7c/c9e to d31/d32/d33/d5e/dc9/cf3 0 2024-04-22T15:18:50.014 INFO:tasks.workunit.client.1.smithi196.stdout:0/140: creat d3/de/d14/f39 x:0 0 0 2024-04-22T15:18:50.014 INFO:tasks.workunit.client.1.smithi196.stdout:0/141: dread - d3/de/d1c/d27/f32 zero size 2024-04-22T15:18:50.015 INFO:tasks.workunit.client.1.smithi196.stdout:6/156: dread - dd/f1b zero size 2024-04-22T15:18:50.018 INFO:tasks.workunit.client.0.smithi096.stdout:2/508: creat d3/d1f/f9f x:0 0 0 2024-04-22T15:18:50.018 INFO:tasks.workunit.client.0.smithi096.stdout:2/509: chown d3/db/d30/c88 16723 1 2024-04-22T15:18:50.019 INFO:tasks.workunit.client.1.smithi196.stdout:2/161: truncate d1/d23/f2f 835988 0 2024-04-22T15:18:50.020 INFO:tasks.workunit.client.1.smithi196.stdout:2/162: fsync d1/d13/d1f/f24 0 2024-04-22T15:18:50.023 INFO:tasks.workunit.client.0.smithi096.stdout:7/754: unlink f10 0 2024-04-22T15:18:50.025 INFO:tasks.workunit.client.1.smithi196.stdout:3/105: creat d0/f1e x:0 0 0 2024-04-22T15:18:50.025 INFO:tasks.workunit.client.0.smithi096.stdout:2/510: dwrite d3/d1f/f9f [0,4194304] 0 2024-04-22T15:18:50.027 INFO:tasks.workunit.client.1.smithi196.stdout:9/139: mknod db/d18/c2d 0 2024-04-22T15:18:50.031 INFO:tasks.workunit.client.1.smithi196.stdout:1/125: unlink d1/db/l23 0 2024-04-22T15:18:50.032 INFO:tasks.workunit.client.0.smithi096.stdout:2/511: symlink d3/db/df/d10/la0 0 2024-04-22T15:18:50.035 INFO:tasks.workunit.client.0.smithi096.stdout:7/755: mknod d31/d32/cf4 0 2024-04-22T15:18:50.037 INFO:tasks.workunit.client.1.smithi196.stdout:8/158: rename d2/d5/d1a/l1f to d2/l2c 0 2024-04-22T15:18:50.040 INFO:tasks.workunit.client.0.smithi096.stdout:7/756: creat d31/d32/d36/d75/ff5 x:0 0 0 2024-04-22T15:18:50.041 INFO:tasks.workunit.client.0.smithi096.stdout:7/757: chown d31/d68/f69 8294622 1 2024-04-22T15:18:50.041 INFO:tasks.workunit.client.1.smithi196.stdout:6/157: dread dd/fe [0,4194304] 0 2024-04-22T15:18:50.042 INFO:tasks.workunit.client.0.smithi096.stdout:2/512: creat d3/db/d2d/d46/d69/fa1 x:0 0 0 2024-04-22T15:18:50.043 INFO:tasks.workunit.client.0.smithi096.stdout:2/513: dread - d3/db/d2d/d46/d7e/d8d/f92 zero size 2024-04-22T15:18:50.044 INFO:tasks.workunit.client.1.smithi196.stdout:7/139: getdents d14/d25 0 2024-04-22T15:18:50.045 INFO:tasks.workunit.client.1.smithi196.stdout:7/140: dread fd [0,4194304] 0 2024-04-22T15:18:50.049 INFO:tasks.workunit.client.1.smithi196.stdout:1/126: creat d1/db/dc/d26/f2b x:0 0 0 2024-04-22T15:18:50.049 INFO:tasks.workunit.client.1.smithi196.stdout:7/141: dread fe [0,4194304] 0 2024-04-22T15:18:50.052 INFO:tasks.workunit.client.1.smithi196.stdout:7/142: dread f13 [0,4194304] 0 2024-04-22T15:18:50.053 INFO:tasks.workunit.client.1.smithi196.stdout:5/112: rename d1/f19 to d1/d4/f26 0 2024-04-22T15:18:50.053 INFO:tasks.workunit.client.1.smithi196.stdout:5/113: fdatasync d1/f1f 0 2024-04-22T15:18:50.055 INFO:tasks.workunit.client.1.smithi196.stdout:8/159: creat d2/d5/f2d x:0 0 0 2024-04-22T15:18:50.058 INFO:tasks.workunit.client.1.smithi196.stdout:6/158: chown dd/f11 451897 1 2024-04-22T15:18:50.061 INFO:tasks.workunit.client.0.smithi096.stdout:7/758: dread fe [0,4194304] 0 2024-04-22T15:18:50.061 INFO:tasks.workunit.client.0.smithi096.stdout:2/514: dread d3/db/f83 [0,4194304] 0 2024-04-22T15:18:50.063 INFO:tasks.workunit.client.0.smithi096.stdout:7/759: write d31/d32/d33/d44/da8/fe2 [1012534,3304] 0 2024-04-22T15:18:50.069 INFO:tasks.workunit.client.1.smithi196.stdout:9/140: rename db/dd/de/l21 to db/d11/d29/l2e 0 2024-04-22T15:18:50.070 INFO:tasks.workunit.client.1.smithi196.stdout:9/141: read - db/d11/d29/f2a zero size 2024-04-22T15:18:50.071 INFO:tasks.workunit.client.0.smithi096.stdout:7/760: creat d31/d32/d33/d44/da8/dbf/ddf/ff6 x:0 0 0 2024-04-22T15:18:50.073 INFO:tasks.workunit.client.0.smithi096.stdout:5/572: creat d4/d11/d3d/d35/fb8 x:0 0 0 2024-04-22T15:18:50.074 INFO:tasks.workunit.client.1.smithi196.stdout:8/160: mkdir d2/d5/d2e 0 2024-04-22T15:18:50.075 INFO:tasks.workunit.client.1.smithi196.stdout:0/142: link d3/de/d14/c20 d3/de/c3a 0 2024-04-22T15:18:50.078 INFO:tasks.workunit.client.0.smithi096.stdout:7/761: rename f11 to d31/d32/d33/d54/def/ff7 0 2024-04-22T15:18:50.079 INFO:tasks.workunit.client.1.smithi196.stdout:6/159: dread - dd/f1c zero size 2024-04-22T15:18:50.081 INFO:tasks.workunit.client.0.smithi096.stdout:5/573: mkdir d4/d11/d3d/d4d/d59/d98/db9 0 2024-04-22T15:18:50.086 INFO:tasks.workunit.client.0.smithi096.stdout:7/762: symlink d31/d32/d33/d5e/dc9/lf8 0 2024-04-22T15:18:50.087 INFO:tasks.workunit.client.1.smithi196.stdout:1/127: symlink d1/l2c 0 2024-04-22T15:18:50.088 INFO:tasks.workunit.client.0.smithi096.stdout:5/574: creat d4/d11/d3d/d35/fba x:0 0 0 2024-04-22T15:18:50.095 INFO:tasks.workunit.client.1.smithi196.stdout:9/142: rename db/d11/l12 to db/dd/d1b/l2f 0 2024-04-22T15:18:50.096 INFO:tasks.workunit.client.1.smithi196.stdout:9/143: chown db 1 1 2024-04-22T15:18:50.096 INFO:tasks.workunit.client.1.smithi196.stdout:9/144: stat db/c17 0 2024-04-22T15:18:50.097 INFO:tasks.workunit.client.1.smithi196.stdout:8/161: stat d2/d8/f27 0 2024-04-22T15:18:50.097 INFO:tasks.workunit.client.1.smithi196.stdout:8/162: write d2/dc/fd [3294118,37052] 0 2024-04-22T15:18:50.098 INFO:tasks.workunit.client.1.smithi196.stdout:8/163: read - d2/d5/f2d zero size 2024-04-22T15:18:50.098 INFO:tasks.workunit.client.1.smithi196.stdout:0/143: symlink d3/d9/l3b 0 2024-04-22T15:18:50.099 INFO:tasks.workunit.client.1.smithi196.stdout:0/144: chown d3/de/d1c/d27/d2c/d30 3334434 1 2024-04-22T15:18:50.100 INFO:tasks.workunit.client.0.smithi096.stdout:7/763: creat d31/d32/d33/d44/da8/ff9 x:0 0 0 2024-04-22T15:18:50.103 INFO:tasks.workunit.client.0.smithi096.stdout:3/407: sync 2024-04-22T15:18:50.104 INFO:tasks.workunit.client.0.smithi096.stdout:3/408: chown da/db/d12/d22 1280952 1 2024-04-22T15:18:50.105 INFO:tasks.workunit.client.1.smithi196.stdout:6/160: creat dd/d2b/f2f x:0 0 0 2024-04-22T15:18:50.106 INFO:tasks.workunit.client.1.smithi196.stdout:6/161: chown fc 103 1 2024-04-22T15:18:50.106 INFO:tasks.workunit.client.0.smithi096.stdout:3/409: read da/db/dc/d59/f79 [647664,89048] 0 2024-04-22T15:18:50.107 INFO:tasks.workunit.client.0.smithi096.stdout:7/764: dwrite d31/d32/dad/fe3 [0,4194304] 0 2024-04-22T15:18:50.107 INFO:tasks.workunit.client.1.smithi196.stdout:6/162: dread - dd/f1b zero size 2024-04-22T15:18:50.109 INFO:tasks.workunit.client.1.smithi196.stdout:0/145: dread d3/d6/dc/f15 [0,4194304] 0 2024-04-22T15:18:50.112 INFO:tasks.workunit.client.1.smithi196.stdout:2/163: getdents d1/d13/d1f 0 2024-04-22T15:18:50.117 INFO:tasks.workunit.client.1.smithi196.stdout:9/145: creat db/f30 x:0 0 0 2024-04-22T15:18:50.118 INFO:tasks.workunit.client.1.smithi196.stdout:9/146: read db/fc [4261327,56465] 0 2024-04-22T15:18:50.119 INFO:tasks.workunit.client.1.smithi196.stdout:8/164: creat d2/dc/f2f x:0 0 0 2024-04-22T15:18:50.120 INFO:tasks.workunit.client.0.smithi096.stdout:5/575: link d4/d2c/d2f/d31/d82/c85 d4/d2c/d2f/cbb 0 2024-04-22T15:18:50.124 INFO:tasks.workunit.client.0.smithi096.stdout:7/765: mkdir d31/d32/d33/dfa 0 2024-04-22T15:18:50.125 INFO:tasks.workunit.client.0.smithi096.stdout:8/463: dwrite d4/d10/d25/f37 [0,4194304] 0 2024-04-22T15:18:50.129 INFO:tasks.workunit.client.0.smithi096.stdout:5/576: dread d4/d11/d3d/d4d/d59/d8a/f97 [0,4194304] 0 2024-04-22T15:18:50.132 INFO:tasks.workunit.client.0.smithi096.stdout:3/410: symlink da/db/de/d19/d2d/d37/l83 0 2024-04-22T15:18:50.135 INFO:tasks.workunit.client.0.smithi096.stdout:9/478: dwrite d1/d4/d9/dd/f1a [0,4194304] 0 2024-04-22T15:18:50.139 INFO:tasks.workunit.client.0.smithi096.stdout:5/577: dwrite d4/fd [4194304,4194304] 0 2024-04-22T15:18:50.139 INFO:tasks.workunit.client.0.smithi096.stdout:8/464: rmdir d4/d10 39 2024-04-22T15:18:50.140 INFO:tasks.workunit.client.0.smithi096.stdout:8/465: write d4/d1c/f3a [293626,50054] 0 2024-04-22T15:18:50.148 INFO:tasks.workunit.client.0.smithi096.stdout:3/411: symlink da/db/d12/d63/l84 0 2024-04-22T15:18:50.149 INFO:tasks.workunit.client.0.smithi096.stdout:3/412: chown da/db/l36 5189952 1 2024-04-22T15:18:50.150 INFO:tasks.workunit.client.0.smithi096.stdout:3/413: write da/db/d12/d22/f60 [770741,9378] 0 2024-04-22T15:18:50.157 INFO:tasks.workunit.client.0.smithi096.stdout:8/466: chown d4/d10/c24 58270 1 2024-04-22T15:18:50.159 INFO:tasks.workunit.client.0.smithi096.stdout:8/467: write d4/f32 [2415307,102249] 0 2024-04-22T15:18:50.160 INFO:tasks.workunit.client.0.smithi096.stdout:9/479: dread d1/d4/d9/f24 [0,4194304] 0 2024-04-22T15:18:50.161 INFO:tasks.workunit.client.0.smithi096.stdout:9/480: readlink d1/d4/d79/l8b 0 2024-04-22T15:18:50.166 INFO:tasks.workunit.client.1.smithi196.stdout:4/150: write d6/f19 [1090170,121955] 0 2024-04-22T15:18:50.169 INFO:tasks.workunit.client.0.smithi096.stdout:4/548: sync 2024-04-22T15:18:50.169 INFO:tasks.workunit.client.0.smithi096.stdout:0/698: sync 2024-04-22T15:18:50.169 INFO:tasks.workunit.client.0.smithi096.stdout:1/667: sync 2024-04-22T15:18:50.169 INFO:tasks.workunit.client.0.smithi096.stdout:6/454: sync 2024-04-22T15:18:50.170 INFO:tasks.workunit.client.0.smithi096.stdout:4/549: write d5/f39 [8989844,95544] 0 2024-04-22T15:18:50.172 INFO:tasks.workunit.client.0.smithi096.stdout:4/550: fdatasync d5/d7/d2c/f9d 0 2024-04-22T15:18:50.172 INFO:tasks.workunit.client.0.smithi096.stdout:2/515: symlink d3/db/df/d24/la2 0 2024-04-22T15:18:50.173 INFO:tasks.workunit.client.0.smithi096.stdout:2/516: dread - d3/d1f/d2f/f9e zero size 2024-04-22T15:18:50.176 INFO:tasks.workunit.client.1.smithi196.stdout:3/106: write d0/d6/db/dc/f1c [795688,34255] 0 2024-04-22T15:18:50.182 INFO:tasks.workunit.client.0.smithi096.stdout:2/517: dread d3/db/f42 [0,4194304] 0 2024-04-22T15:18:50.186 INFO:tasks.workunit.client.0.smithi096.stdout:2/518: dwrite d3/db/f42 [0,4194304] 0 2024-04-22T15:18:50.190 INFO:tasks.workunit.client.1.smithi196.stdout:7/143: write f4 [5009604,75479] 0 2024-04-22T15:18:50.195 INFO:tasks.workunit.client.0.smithi096.stdout:5/578: fsync d4/d11/d3d/d35/fb8 0 2024-04-22T15:18:50.197 INFO:tasks.workunit.client.1.smithi196.stdout:7/144: dwrite ff [0,4194304] 0 2024-04-22T15:18:50.197 INFO:tasks.workunit.client.1.smithi196.stdout:7/145: chown d14/l1b 179070247 1 2024-04-22T15:18:50.219 INFO:tasks.workunit.client.0.smithi096.stdout:8/468: symlink d4/d1c/d62/la3 0 2024-04-22T15:18:50.225 INFO:tasks.workunit.client.0.smithi096.stdout:0/699: rmdir d7/d42/d4c/d55 39 2024-04-22T15:18:50.236 INFO:tasks.workunit.client.0.smithi096.stdout:2/519: mknod d3/db/d2d/d46/d7e/ca3 0 2024-04-22T15:18:50.237 INFO:tasks.workunit.client.0.smithi096.stdout:2/520: write d3/db/d30/f3e [2099905,123320] 0 2024-04-22T15:18:50.238 INFO:tasks.workunit.client.0.smithi096.stdout:2/521: rename d3 to d3/d1f/d5a/da4 22 2024-04-22T15:18:50.238 INFO:tasks.workunit.client.0.smithi096.stdout:5/579: mknod d4/d2c/cbc 0 2024-04-22T15:18:50.240 INFO:tasks.workunit.client.0.smithi096.stdout:3/414: link da/db/d12/d22/d68/f72 da/db/dc/d28/f85 0 2024-04-22T15:18:50.241 INFO:tasks.workunit.client.0.smithi096.stdout:3/415: write da/db/de/f40 [284185,40039] 0 2024-04-22T15:18:50.243 INFO:tasks.workunit.client.0.smithi096.stdout:9/481: unlink d1/df/l2a 0 2024-04-22T15:18:50.245 INFO:tasks.workunit.client.0.smithi096.stdout:6/455: chown d3/d28/dd/d11 6683159 1 2024-04-22T15:18:50.246 INFO:tasks.workunit.client.0.smithi096.stdout:6/456: stat d3/d28/dd/d11/l85 0 2024-04-22T15:18:50.253 INFO:tasks.workunit.client.0.smithi096.stdout:7/766: dwrite d31/d32/d33/f58 [0,4194304] 0 2024-04-22T15:18:50.266 INFO:tasks.workunit.client.0.smithi096.stdout:3/416: creat da/db/d12/d22/d26/d52/f86 x:0 0 0 2024-04-22T15:18:50.290 INFO:tasks.workunit.client.0.smithi096.stdout:2/522: creat d3/db/df/fa5 x:0 0 0 2024-04-22T15:18:50.294 INFO:tasks.workunit.client.0.smithi096.stdout:3/417: mkdir da/db/de/d87 0 2024-04-22T15:18:50.295 INFO:tasks.workunit.client.0.smithi096.stdout:0/700: getdents d7/de/d5b/d95/dc/d16 0 2024-04-22T15:18:50.300 INFO:tasks.workunit.client.0.smithi096.stdout:5/580: link d4/d11/d3d/d4d/f5d d4/d11/fbd 0 2024-04-22T15:18:50.302 INFO:tasks.workunit.client.0.smithi096.stdout:3/418: write da/f55 [286537,17087] 0 2024-04-22T15:18:50.303 INFO:tasks.workunit.client.0.smithi096.stdout:3/419: dread - da/db/de/d19/d2d/d37/f5f zero size 2024-04-22T15:18:50.306 INFO:tasks.workunit.client.0.smithi096.stdout:6/457: getdents d3/d28/d54 0 2024-04-22T15:18:50.310 INFO:tasks.workunit.client.0.smithi096.stdout:3/420: dwrite da/db/dc/f49 [0,4194304] 0 2024-04-22T15:18:50.319 INFO:tasks.workunit.client.0.smithi096.stdout:6/458: rename d3/d28/dd/d11/d2c/d44/d51/d78 to d3/d28/d4d/d82/d9e 0 2024-04-22T15:18:50.323 INFO:tasks.workunit.client.0.smithi096.stdout:0/701: creat d7/de/d5b/d95/dc/fed x:0 0 0 2024-04-22T15:18:50.325 INFO:tasks.workunit.client.0.smithi096.stdout:2/523: getdents d3 0 2024-04-22T15:18:50.329 INFO:tasks.workunit.client.0.smithi096.stdout:3/421: rename da/db/de/d19/d3e/f29 to da/db/d12/d22/f88 0 2024-04-22T15:18:50.331 INFO:tasks.workunit.client.0.smithi096.stdout:1/668: write d6/dc/d8a/d3f/d54/d32/d3b/d56/f6e [958077,82035] 0 2024-04-22T15:18:50.340 INFO:tasks.workunit.client.0.smithi096.stdout:4/551: dwrite d5/d7/d2c/d3d/d4f/f60 [0,4194304] 0 2024-04-22T15:18:50.340 INFO:tasks.workunit.client.0.smithi096.stdout:0/702: dwrite d7/de/d5b/d95/dc/d16/d65/d7c/f9b [4194304,4194304] 0 2024-04-22T15:18:50.342 INFO:tasks.workunit.client.0.smithi096.stdout:0/703: chown d7/de/d5b/d95/dc/l52 68 1 2024-04-22T15:18:50.344 INFO:tasks.workunit.client.0.smithi096.stdout:0/704: write d7/de/d5b/d95/dc/d16/d18/f69 [2789878,10224] 0 2024-04-22T15:18:50.345 INFO:tasks.workunit.client.0.smithi096.stdout:1/669: creat d6/fcf x:0 0 0 2024-04-22T15:18:50.347 INFO:tasks.workunit.client.0.smithi096.stdout:0/705: write d7/de/d5b/d95/dc/d16/d18/fd8 [868913,38107] 0 2024-04-22T15:18:50.355 INFO:tasks.workunit.client.0.smithi096.stdout:0/706: dwrite d7/de/d5b/d95/f45 [0,4194304] 0 2024-04-22T15:18:50.357 INFO:tasks.workunit.client.0.smithi096.stdout:0/707: read - d7/de/d5b/d95/dc/d26/fbc zero size 2024-04-22T15:18:50.358 INFO:tasks.workunit.client.0.smithi096.stdout:0/708: chown d7/de/d5b/d95/dc/d16/d65/d7c 11 1 2024-04-22T15:18:50.364 INFO:tasks.workunit.client.0.smithi096.stdout:4/552: symlink d5/d7/d1f/d84/lbc 0 2024-04-22T15:18:50.369 INFO:tasks.workunit.client.1.smithi196.stdout:6/163: mkdir dd/d20/d30 0 2024-04-22T15:18:50.372 INFO:tasks.workunit.client.0.smithi096.stdout:9/482: dwrite d1/d4/d7f/f83 [0,4194304] 0 2024-04-22T15:18:50.374 INFO:tasks.workunit.client.0.smithi096.stdout:7/767: write f18 [3262741,44659] 0 2024-04-22T15:18:50.377 INFO:tasks.workunit.client.1.smithi196.stdout:1/128: mkdir d1/db/d2d 0 2024-04-22T15:18:50.382 INFO:tasks.workunit.client.1.smithi196.stdout:9/147: mkdir db/dd/de/d15/d31 0 2024-04-22T15:18:50.382 INFO:tasks.workunit.client.1.smithi196.stdout:9/148: dread - db/d11/d29/f2a zero size 2024-04-22T15:18:50.384 INFO:tasks.workunit.client.0.smithi096.stdout:9/483: dwrite d1/d4/d9/dd/f46 [0,4194304] 0 2024-04-22T15:18:50.385 INFO:tasks.workunit.client.1.smithi196.stdout:9/149: dread db/dd/de/f10 [0,4194304] 0 2024-04-22T15:18:50.386 INFO:tasks.workunit.client.1.smithi196.stdout:9/150: fdatasync db/dd/f1a 0 2024-04-22T15:18:50.388 INFO:tasks.workunit.client.0.smithi096.stdout:4/553: rename d5/d7/d2c/d3d/d4f/d88/fa3 to d5/d7/fbd 0 2024-04-22T15:18:50.388 INFO:tasks.workunit.client.0.smithi096.stdout:7/768: dwrite d31/d32/d33/d44/da8/ff9 [0,4194304] 0 2024-04-22T15:18:50.391 INFO:tasks.workunit.client.0.smithi096.stdout:0/709: dread d7/de/d3a/f44 [4194304,4194304] 0 2024-04-22T15:18:50.393 INFO:tasks.workunit.client.1.smithi196.stdout:3/107: mknod d0/d6/c1f 0 2024-04-22T15:18:50.393 INFO:tasks.workunit.client.0.smithi096.stdout:7/769: dread d31/d32/d33/d6a/d83/fa6 [0,4194304] 0 2024-04-22T15:18:50.393 INFO:tasks.workunit.client.1.smithi196.stdout:3/108: readlink d0/d6/db/l1d 0 2024-04-22T15:18:50.394 INFO:tasks.workunit.client.1.smithi196.stdout:3/109: dread - d0/f1e zero size 2024-04-22T15:18:50.397 INFO:tasks.workunit.client.0.smithi096.stdout:4/554: dwrite d5/d7/d2c/f86 [0,4194304] 0 2024-04-22T15:18:50.399 INFO:tasks.workunit.client.1.smithi196.stdout:5/114: dread d1/d4/f26 [0,4194304] 0 2024-04-22T15:18:50.403 INFO:tasks.workunit.client.1.smithi196.stdout:5/115: dread d1/d2/f5 [0,4194304] 0 2024-04-22T15:18:50.406 INFO:tasks.workunit.client.0.smithi096.stdout:9/484: fdatasync d1/d4/d9/d43/f44 0 2024-04-22T15:18:50.408 INFO:tasks.workunit.client.0.smithi096.stdout:0/710: mknod d7/de/d5b/d95/d59/cee 0 2024-04-22T15:18:50.410 INFO:tasks.workunit.client.1.smithi196.stdout:7/146: dwrite f2 [0,4194304] 0 2024-04-22T15:18:50.410 INFO:tasks.workunit.client.0.smithi096.stdout:0/711: write d7/de/d3a/d61/d79/fa4 [898071,95087] 0 2024-04-22T15:18:50.411 INFO:tasks.workunit.client.1.smithi196.stdout:6/164: write dd/f18 [3861258,38737] 0 2024-04-22T15:18:50.412 INFO:tasks.workunit.client.1.smithi196.stdout:6/165: chown dd/d20 1276897855 1 2024-04-22T15:18:50.412 INFO:tasks.workunit.client.1.smithi196.stdout:6/166: dread - dd/d20/f24 zero size 2024-04-22T15:18:50.413 INFO:tasks.workunit.client.1.smithi196.stdout:6/167: chown dd/f15 1592 1 2024-04-22T15:18:50.415 INFO:tasks.workunit.client.0.smithi096.stdout:4/555: symlink d5/d7/d1f/d84/lbe 0 2024-04-22T15:18:50.416 INFO:tasks.workunit.client.0.smithi096.stdout:4/556: rename d5/d7 to d5/d7/d2c/d43/d56/dbf 22 2024-04-22T15:18:50.422 INFO:tasks.workunit.client.1.smithi196.stdout:1/129: write d1/db/dc/d26/f27 [1000653,11219] 0 2024-04-22T15:18:50.424 INFO:tasks.workunit.client.0.smithi096.stdout:5/581: dwrite d4/f6 [4194304,4194304] 0 2024-04-22T15:18:50.426 INFO:tasks.workunit.client.0.smithi096.stdout:9/485: fdatasync d1/f2f 0 2024-04-22T15:18:50.428 INFO:tasks.workunit.client.0.smithi096.stdout:9/486: chown d1/c3e 662 1 2024-04-22T15:18:50.429 INFO:tasks.workunit.client.1.smithi196.stdout:8/165: symlink d2/l30 0 2024-04-22T15:18:50.432 INFO:tasks.workunit.client.0.smithi096.stdout:0/712: dread - d7/d42/f7a zero size 2024-04-22T15:18:50.434 INFO:tasks.workunit.client.1.smithi196.stdout:4/151: link d6/d13/f1d d6/d14/d18/d26/f2f 0 2024-04-22T15:18:50.434 INFO:tasks.workunit.client.1.smithi196.stdout:4/152: read d6/d13/f16 [723234,10862] 0 2024-04-22T15:18:50.435 INFO:tasks.workunit.client.1.smithi196.stdout:4/153: chown d6/c9 2786 1 2024-04-22T15:18:50.435 INFO:tasks.workunit.client.1.smithi196.stdout:4/154: dread - d6/d13/f1b zero size 2024-04-22T15:18:50.436 INFO:tasks.workunit.client.0.smithi096.stdout:7/770: mknod d31/d32/cfb 0 2024-04-22T15:18:50.439 INFO:tasks.workunit.client.0.smithi096.stdout:4/557: creat d5/d7/d2c/d95/fc0 x:0 0 0 2024-04-22T15:18:50.439 INFO:tasks.workunit.client.0.smithi096.stdout:4/558: chown d5/d7/d2c/d43/l45 33174560 1 2024-04-22T15:18:50.440 INFO:tasks.workunit.client.0.smithi096.stdout:4/559: truncate d5/d7/f8e 633271 0 2024-04-22T15:18:50.441 INFO:tasks.workunit.client.0.smithi096.stdout:5/582: mkdir d4/d11/d3d/d35/d57/dbe 0 2024-04-22T15:18:50.442 INFO:tasks.workunit.client.0.smithi096.stdout:4/560: truncate d5/d7/f7b 4727546 0 2024-04-22T15:18:50.442 INFO:tasks.workunit.client.0.smithi096.stdout:3/422: write da/db/de/d19/f2a [3760533,19634] 0 2024-04-22T15:18:50.444 INFO:tasks.workunit.client.0.smithi096.stdout:6/459: truncate d3/d28/dd/d11/d2c/d44/d51/f60 2586545 0 2024-04-22T15:18:50.445 INFO:tasks.workunit.client.0.smithi096.stdout:6/460: stat d3/l74 0 2024-04-22T15:18:50.446 INFO:tasks.workunit.client.0.smithi096.stdout:6/461: fdatasync d3/d28/dd/d11/d2c/f95 0 2024-04-22T15:18:50.452 INFO:tasks.workunit.client.0.smithi096.stdout:2/524: truncate d3/f87 525319 0 2024-04-22T15:18:50.452 INFO:tasks.workunit.client.0.smithi096.stdout:9/487: dread d1/d4/d9/dd/f21 [0,4194304] 0 2024-04-22T15:18:50.454 INFO:tasks.workunit.client.0.smithi096.stdout:5/583: dread d4/d11/f22 [0,4194304] 0 2024-04-22T15:18:50.454 INFO:tasks.workunit.client.1.smithi196.stdout:0/146: link d3/de/c2d d3/de/d14/d29/c3c 0 2024-04-22T15:18:50.454 INFO:tasks.workunit.client.1.smithi196.stdout:0/147: dread - d3/de/f37 zero size 2024-04-22T15:18:50.458 INFO:tasks.workunit.client.0.smithi096.stdout:7/771: dread d31/d32/d33/d54/d43/f48 [0,4194304] 0 2024-04-22T15:18:50.459 INFO:tasks.workunit.client.0.smithi096.stdout:3/423: rename da/db/d12/d63 to da/db/dc/d28/d69/d89 0 2024-04-22T15:18:50.460 INFO:tasks.workunit.client.1.smithi196.stdout:0/148: dwrite d3/de/d1c/d27/f32 [0,4194304] 0 2024-04-22T15:18:50.461 INFO:tasks.workunit.client.1.smithi196.stdout:1/130: mkdir d1/db/dc/d26/d2e 0 2024-04-22T15:18:50.462 INFO:tasks.workunit.client.1.smithi196.stdout:8/166: readlink d2/l2c 0 2024-04-22T15:18:50.467 INFO:tasks.workunit.client.0.smithi096.stdout:0/713: creat d7/de/d5b/d95/d66/dce/fef x:0 0 0 2024-04-22T15:18:50.467 INFO:tasks.workunit.client.0.smithi096.stdout:0/714: dread - d7/de/d5b/fe2 zero size 2024-04-22T15:18:50.476 INFO:tasks.workunit.client.1.smithi196.stdout:5/116: symlink d1/d17/d1c/l27 0 2024-04-22T15:18:50.477 INFO:tasks.workunit.client.1.smithi196.stdout:5/117: chown d1/d17/d1a 484365961 1 2024-04-22T15:18:50.478 INFO:tasks.workunit.client.1.smithi196.stdout:7/147: creat d14/d26/d2a/d2b/f2d x:0 0 0 2024-04-22T15:18:50.480 INFO:tasks.workunit.client.1.smithi196.stdout:6/168: getdents dd/d20/d30 0 2024-04-22T15:18:50.482 INFO:tasks.workunit.client.1.smithi196.stdout:5/118: dwrite d1/d4/f6 [0,4194304] 0 2024-04-22T15:18:50.483 INFO:tasks.workunit.client.1.smithi196.stdout:0/149: rmdir d3/d9 39 2024-04-22T15:18:50.484 INFO:tasks.workunit.client.1.smithi196.stdout:5/119: readlink d1/d17/d1c/l27 0 2024-04-22T15:18:50.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:50 smithi096 ceph-mon[26688]: pgmap v31: 65 pgs: 65 active+clean; 1.5 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 44 MiB/s rd, 202 MiB/s wr, 446 op/s 2024-04-22T15:18:50.486 INFO:tasks.workunit.client.1.smithi196.stdout:5/120: readlink d1/d4/l18 0 2024-04-22T15:18:50.486 INFO:tasks.workunit.client.1.smithi196.stdout:7/148: dwrite f4 [4194304,4194304] 0 2024-04-22T15:18:50.486 INFO:tasks.workunit.client.1.smithi196.stdout:8/167: symlink d2/d8/l31 0 2024-04-22T15:18:50.488 INFO:tasks.workunit.client.1.smithi196.stdout:8/168: dread - d2/d8/f25 zero size 2024-04-22T15:18:50.490 INFO:tasks.workunit.client.1.smithi196.stdout:2/164: dwrite d1/d13/f1e [0,4194304] 0 2024-04-22T15:18:50.490 INFO:tasks.workunit.client.1.smithi196.stdout:8/169: dread - d2/dc/f2f zero size 2024-04-22T15:18:50.491 INFO:tasks.workunit.client.0.smithi096.stdout:7/772: mkdir d31/d32/d33/d93/dfc 0 2024-04-22T15:18:50.492 INFO:tasks.workunit.client.0.smithi096.stdout:3/424: write da/db/dc/d59/f79 [3608386,95945] 0 2024-04-22T15:18:50.492 INFO:tasks.workunit.client.0.smithi096.stdout:6/462: symlink d3/d28/l9f 0 2024-04-22T15:18:50.493 INFO:tasks.workunit.client.0.smithi096.stdout:0/715: creat d7/de/d5b/d95/d66/dce/ff0 x:0 0 0 2024-04-22T15:18:50.494 INFO:tasks.workunit.client.1.smithi196.stdout:5/121: dwrite d1/f1f [0,4194304] 0 2024-04-22T15:18:50.494 INFO:tasks.workunit.client.1.smithi196.stdout:4/155: rmdir d6/d13 39 2024-04-22T15:18:50.505 INFO:tasks.workunit.client.1.smithi196.stdout:3/110: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:18:50.508 INFO:tasks.workunit.client.0.smithi096.stdout:4/561: link d5/l17 d5/d7/d1a/d8b/lc1 0 2024-04-22T15:18:50.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:50 smithi196 ceph-mon[34717]: pgmap v31: 65 pgs: 65 active+clean; 1.5 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 44 MiB/s rd, 202 MiB/s wr, 446 op/s 2024-04-22T15:18:50.517 INFO:tasks.workunit.client.0.smithi096.stdout:0/716: mknod d7/d8f/cf1 0 2024-04-22T15:18:50.519 INFO:tasks.workunit.client.1.smithi196.stdout:9/151: write f5 [2539111,49917] 0 2024-04-22T15:18:50.520 INFO:tasks.workunit.client.1.smithi196.stdout:9/152: read db/dd/de/d15/f1e [7584363,107324] 0 2024-04-22T15:18:50.522 INFO:tasks.workunit.client.1.smithi196.stdout:9/153: dread db/dd/de/f10 [0,4194304] 0 2024-04-22T15:18:50.523 INFO:tasks.workunit.client.1.smithi196.stdout:5/122: rename d1/fe to d1/d2/f28 0 2024-04-22T15:18:50.524 INFO:tasks.workunit.client.1.smithi196.stdout:9/154: read f4 [5971795,21109] 0 2024-04-22T15:18:50.525 INFO:tasks.workunit.client.0.smithi096.stdout:4/562: creat d5/fc2 x:0 0 0 2024-04-22T15:18:50.526 INFO:tasks.workunit.client.1.smithi196.stdout:6/169: fdatasync dd/f14 0 2024-04-22T15:18:50.527 INFO:tasks.workunit.client.1.smithi196.stdout:6/170: chown c6 816891 1 2024-04-22T15:18:50.527 INFO:tasks.workunit.client.0.smithi096.stdout:7/773: symlink d31/d32/d33/dfa/lfd 0 2024-04-22T15:18:50.528 INFO:tasks.workunit.client.1.smithi196.stdout:6/171: write f5 [8516209,14380] 0 2024-04-22T15:18:50.528 INFO:tasks.workunit.client.1.smithi196.stdout:3/111: creat d0/d6/db/d13/f20 x:0 0 0 2024-04-22T15:18:50.531 INFO:tasks.workunit.client.0.smithi096.stdout:2/525: dwrite d3/d1f/f7a [0,4194304] 0 2024-04-22T15:18:50.531 INFO:tasks.workunit.client.0.smithi096.stdout:3/425: dread da/db/de/d19/f2a [0,4194304] 0 2024-04-22T15:18:50.534 INFO:tasks.workunit.client.0.smithi096.stdout:0/717: chown d7/de/d3a/d61/dbb/ce1 24758 1 2024-04-22T15:18:50.538 INFO:tasks.workunit.client.0.smithi096.stdout:0/718: stat d7/de/d5b/d95/dc/d16/d65/d7c/ce7 0 2024-04-22T15:18:50.539 INFO:tasks.workunit.client.0.smithi096.stdout:7/774: dwrite d31/d32/d33/d93/dd5/ff2 [0,4194304] 0 2024-04-22T15:18:50.547 INFO:tasks.workunit.client.0.smithi096.stdout:9/488: truncate d1/d25/f4c 2904314 0 2024-04-22T15:18:50.547 INFO:tasks.workunit.client.0.smithi096.stdout:6/463: link d3/d3a/l3d d3/d28/d54/d68/d6e/la0 0 2024-04-22T15:18:50.548 INFO:tasks.workunit.client.1.smithi196.stdout:3/112: fdatasync d0/d6/db/d13/f16 0 2024-04-22T15:18:50.548 INFO:tasks.workunit.client.1.smithi196.stdout:9/155: dwrite db/f28 [0,4194304] 0 2024-04-22T15:18:50.550 INFO:tasks.workunit.client.0.smithi096.stdout:9/489: truncate d1/f75 895922 0 2024-04-22T15:18:50.550 INFO:tasks.workunit.client.0.smithi096.stdout:1/670: dwrite d6/dc/d8a/d29/f59 [0,4194304] 0 2024-04-22T15:18:50.553 INFO:tasks.workunit.client.0.smithi096.stdout:9/490: write d1/d85/f94 [292644,107538] 0 2024-04-22T15:18:50.555 INFO:tasks.workunit.client.0.smithi096.stdout:4/563: mknod d5/d7/d2c/cc3 0 2024-04-22T15:18:50.556 INFO:tasks.workunit.client.0.smithi096.stdout:4/564: chown d5/d7/d1f/fa6 1 1 2024-04-22T15:18:50.559 INFO:tasks.workunit.client.0.smithi096.stdout:4/565: chown d5/d7/d2c/d43/d56/l8c 107 1 2024-04-22T15:18:50.560 INFO:tasks.workunit.client.0.smithi096.stdout:2/526: dread d3/d1f/d2f/f4e [0,4194304] 0 2024-04-22T15:18:50.562 INFO:tasks.workunit.client.1.smithi196.stdout:2/165: link d1/fd d1/d23/f40 0 2024-04-22T15:18:50.564 INFO:tasks.workunit.client.0.smithi096.stdout:9/491: dwrite d1/d4/d9/dd/f2b [0,4194304] 0 2024-04-22T15:18:50.567 INFO:tasks.workunit.client.0.smithi096.stdout:0/719: fsync d7/de/d5b/d95/dc/f96 0 2024-04-22T15:18:50.568 INFO:tasks.workunit.client.0.smithi096.stdout:7/775: rename d31/d32/d7c to d31/d32/d36/d75/dfe 0 2024-04-22T15:18:50.569 INFO:tasks.workunit.client.1.smithi196.stdout:8/170: creat d2/f32 x:0 0 0 2024-04-22T15:18:50.570 INFO:tasks.workunit.client.0.smithi096.stdout:7/776: write d31/d32/d36/d75/fc3 [5099404,3863] 0 2024-04-22T15:18:50.572 INFO:tasks.workunit.client.0.smithi096.stdout:3/426: fdatasync da/db/d12/d22/f88 0 2024-04-22T15:18:50.572 INFO:tasks.workunit.client.1.smithi196.stdout:7/149: truncate fc 3959316 0 2024-04-22T15:18:50.574 INFO:tasks.workunit.client.1.smithi196.stdout:4/156: link d6/d13/f1b d6/d13/d15/f30 0 2024-04-22T15:18:50.580 INFO:tasks.workunit.client.1.smithi196.stdout:5/123: link d1/d14/f16 d1/d2/f29 0 2024-04-22T15:18:50.582 INFO:tasks.workunit.client.1.smithi196.stdout:3/113: symlink d0/d15/l21 0 2024-04-22T15:18:50.585 INFO:tasks.workunit.client.0.smithi096.stdout:1/671: rmdir d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b 39 2024-04-22T15:18:50.586 INFO:tasks.workunit.client.0.smithi096.stdout:1/672: chown d6/dc/d8a/d3f/d54/d32/d82 122 1 2024-04-22T15:18:50.589 INFO:tasks.workunit.client.0.smithi096.stdout:4/566: dread f1 [4194304,4194304] 0 2024-04-22T15:18:50.589 INFO:tasks.workunit.client.0.smithi096.stdout:9/492: readlink d1/la 0 2024-04-22T15:18:50.591 INFO:tasks.workunit.client.1.smithi196.stdout:7/150: rename d14/d15/l21 to d14/l2e 0 2024-04-22T15:18:50.591 INFO:tasks.workunit.client.1.smithi196.stdout:7/151: stat d14/d15/c2c 0 2024-04-22T15:18:50.594 INFO:tasks.workunit.client.1.smithi196.stdout:5/124: creat d1/d9/f2a x:0 0 0 2024-04-22T15:18:50.595 INFO:tasks.workunit.client.1.smithi196.stdout:5/125: truncate d1/d17/f24 13601 0 2024-04-22T15:18:50.597 INFO:tasks.workunit.client.0.smithi096.stdout:0/720: write d7/de/d5b/d95/d15/d35/fb4 [4234426,96094] 0 2024-04-22T15:18:50.598 INFO:tasks.workunit.client.1.smithi196.stdout:2/166: dread d1/ff [0,4194304] 0 2024-04-22T15:18:50.598 INFO:tasks.workunit.client.1.smithi196.stdout:2/167: dread - d1/f3e zero size 2024-04-22T15:18:50.599 INFO:tasks.workunit.client.1.smithi196.stdout:9/156: unlink db/dd/d1b/l2f 0 2024-04-22T15:18:50.599 INFO:tasks.workunit.client.1.smithi196.stdout:9/157: readlink db/dd/de/l16 0 2024-04-22T15:18:50.602 INFO:tasks.workunit.client.0.smithi096.stdout:7/777: mkdir d31/d32/d36/d75/dff 0 2024-04-22T15:18:50.605 INFO:tasks.workunit.client.1.smithi196.stdout:4/157: truncate d6/d13/f1d 3120326 0 2024-04-22T15:18:50.606 INFO:tasks.workunit.client.0.smithi096.stdout:6/464: mknod d3/d28/dd/ca1 0 2024-04-22T15:18:50.607 INFO:tasks.workunit.client.1.smithi196.stdout:7/152: mknod d14/d26/d28/c2f 0 2024-04-22T15:18:50.608 INFO:tasks.workunit.client.0.smithi096.stdout:1/673: mkdir d6/dc/d8a/d3f/d62/dc2/dd0 0 2024-04-22T15:18:50.611 INFO:tasks.workunit.client.1.smithi196.stdout:4/158: dwrite d6/d14/f27 [0,4194304] 0 2024-04-22T15:18:50.613 INFO:tasks.workunit.client.0.smithi096.stdout:4/567: rename d5/c1e to d5/d7/d2c/d43/d56/cc4 0 2024-04-22T15:18:50.615 INFO:tasks.workunit.client.1.smithi196.stdout:5/126: dwrite d1/d2/f25 [0,4194304] 0 2024-04-22T15:18:50.616 INFO:tasks.workunit.client.1.smithi196.stdout:5/127: fsync d1/d17/f21 0 2024-04-22T15:18:50.619 INFO:tasks.workunit.client.1.smithi196.stdout:6/172: getdents dd/d2b 0 2024-04-22T15:18:50.622 INFO:tasks.workunit.client.0.smithi096.stdout:0/721: mknod d7/de/d5b/d95/dc/d16/d65/cf2 0 2024-04-22T15:18:50.629 INFO:tasks.workunit.client.1.smithi196.stdout:7/153: rename ff to d14/d26/f30 0 2024-04-22T15:18:50.632 INFO:tasks.workunit.client.1.smithi196.stdout:7/154: dread fa [0,4194304] 0 2024-04-22T15:18:50.633 INFO:tasks.workunit.client.1.smithi196.stdout:3/114: link d0/d15/l14 d0/d6/db/dc/l22 0 2024-04-22T15:18:50.634 INFO:tasks.workunit.client.1.smithi196.stdout:3/115: chown d0/d6/db/dc/d18/f1b 2219 1 2024-04-22T15:18:50.634 INFO:tasks.workunit.client.0.smithi096.stdout:6/465: symlink d3/d28/dd/d11/d2c/d44/d51/la2 0 2024-04-22T15:18:50.635 INFO:tasks.workunit.client.1.smithi196.stdout:7/155: dread f4 [0,4194304] 0 2024-04-22T15:18:50.636 INFO:tasks.workunit.client.1.smithi196.stdout:4/159: write d6/d14/fb [7399127,49500] 0 2024-04-22T15:18:50.637 INFO:tasks.workunit.client.0.smithi096.stdout:2/527: truncate d3/db/f42 4003998 0 2024-04-22T15:18:50.640 INFO:tasks.workunit.client.0.smithi096.stdout:1/674: chown d6/f1d 85 1 2024-04-22T15:18:50.641 INFO:tasks.workunit.client.1.smithi196.stdout:9/158: creat db/dd/de/d15/d31/f32 x:0 0 0 2024-04-22T15:18:50.641 INFO:tasks.workunit.client.0.smithi096.stdout:2/528: dread d3/db/d30/f82 [0,4194304] 0 2024-04-22T15:18:50.642 INFO:tasks.workunit.client.1.smithi196.stdout:4/160: dwrite d6/d13/f25 [0,4194304] 0 2024-04-22T15:18:50.643 INFO:tasks.workunit.client.1.smithi196.stdout:5/128: mkdir d1/d17/d2b 0 2024-04-22T15:18:50.644 INFO:tasks.workunit.client.1.smithi196.stdout:5/129: stat d1/d9/l23 0 2024-04-22T15:18:50.646 INFO:tasks.workunit.client.0.smithi096.stdout:0/722: creat d7/de/d5b/d95/dc/d16/d65/ff3 x:0 0 0 2024-04-22T15:18:50.646 INFO:tasks.workunit.client.0.smithi096.stdout:0/723: fdatasync d7/de/d5b/d64/fb6 0 2024-04-22T15:18:50.648 INFO:tasks.workunit.client.1.smithi196.stdout:8/171: getdents d2/d5 0 2024-04-22T15:18:50.648 INFO:tasks.workunit.client.0.smithi096.stdout:7/778: creat d31/d32/dc0/ddc/de9/f100 x:0 0 0 2024-04-22T15:18:50.648 INFO:tasks.workunit.client.1.smithi196.stdout:8/172: stat d2/dc/fe 0 2024-04-22T15:18:50.649 INFO:tasks.workunit.client.1.smithi196.stdout:8/173: fdatasync d2/f4 0 2024-04-22T15:18:50.649 INFO:tasks.workunit.client.1.smithi196.stdout:5/130: dread d1/d4/f26 [0,4194304] 0 2024-04-22T15:18:50.649 INFO:tasks.workunit.client.0.smithi096.stdout:7/779: read f1c [945007,40264] 0 2024-04-22T15:18:50.649 INFO:tasks.workunit.client.0.smithi096.stdout:3/427: creat da/f8a x:0 0 0 2024-04-22T15:18:50.650 INFO:tasks.workunit.client.1.smithi196.stdout:5/131: chown d1/d17/d1a/f1d 0 1 2024-04-22T15:18:50.650 INFO:tasks.workunit.client.1.smithi196.stdout:5/132: read d1/d2/f28 [1693699,92640] 0 2024-04-22T15:18:50.650 INFO:tasks.workunit.client.0.smithi096.stdout:3/428: readlink da/db/de/l25 0 2024-04-22T15:18:50.651 INFO:tasks.workunit.client.1.smithi196.stdout:5/133: chown d1/d9/f2a 29386496 1 2024-04-22T15:18:50.651 INFO:tasks.workunit.client.1.smithi196.stdout:5/134: stat d1/d17/d1c/l27 0 2024-04-22T15:18:50.652 INFO:tasks.workunit.client.0.smithi096.stdout:6/466: mkdir d3/d28/d4d/d82/d9e/da3 0 2024-04-22T15:18:50.655 INFO:tasks.workunit.client.1.smithi196.stdout:0/150: sync 2024-04-22T15:18:50.655 INFO:tasks.workunit.client.1.smithi196.stdout:1/131: sync 2024-04-22T15:18:50.656 INFO:tasks.workunit.client.1.smithi196.stdout:3/116: mknod d0/d6/c23 0 2024-04-22T15:18:50.656 INFO:tasks.workunit.client.1.smithi196.stdout:1/132: write d1/db/d16/f1c [635482,20597] 0 2024-04-22T15:18:50.657 INFO:tasks.workunit.client.1.smithi196.stdout:3/117: read d0/d6/db/dc/f1c [614834,28191] 0 2024-04-22T15:18:50.660 INFO:tasks.workunit.client.0.smithi096.stdout:9/493: creat d1/d4/d9/d43/f9c x:0 0 0 2024-04-22T15:18:50.660 INFO:tasks.workunit.client.1.smithi196.stdout:3/118: fdatasync d0/d6/db/d13/f20 0 2024-04-22T15:18:50.661 INFO:tasks.workunit.client.0.smithi096.stdout:9/494: read d1/d4/d7f/f83 [225676,78966] 0 2024-04-22T15:18:50.663 INFO:tasks.workunit.client.0.smithi096.stdout:4/568: dread d5/fc [0,4194304] 0 2024-04-22T15:18:50.664 INFO:tasks.workunit.client.1.smithi196.stdout:7/156: dwrite d14/d26/f30 [0,4194304] 0 2024-04-22T15:18:50.664 INFO:tasks.workunit.client.1.smithi196.stdout:1/133: dwrite d1/db/dc/d26/f2b [0,4194304] 0 2024-04-22T15:18:50.664 INFO:tasks.workunit.client.0.smithi096.stdout:3/429: fsync da/db/d12/d22/f88 0 2024-04-22T15:18:50.665 INFO:tasks.workunit.client.1.smithi196.stdout:7/157: truncate d14/d26/d2a/d2b/f2d 609638 0 2024-04-22T15:18:50.666 INFO:tasks.workunit.client.1.smithi196.stdout:7/158: fdatasync d14/f22 0 2024-04-22T15:18:50.667 INFO:tasks.workunit.client.0.smithi096.stdout:6/467: read d3/d3a/d75/d83/f86 [799478,23432] 0 2024-04-22T15:18:50.673 INFO:tasks.workunit.client.1.smithi196.stdout:9/159: rename db/f30 to db/d18/f33 0 2024-04-22T15:18:50.676 INFO:tasks.workunit.client.1.smithi196.stdout:6/173: creat dd/d20/d30/f31 x:0 0 0 2024-04-22T15:18:50.682 INFO:tasks.workunit.client.0.smithi096.stdout:7/780: mknod d31/d32/d36/d75/dff/c101 0 2024-04-22T15:18:50.683 INFO:tasks.workunit.client.1.smithi196.stdout:2/168: truncate d1/f7 293126 0 2024-04-22T15:18:50.686 INFO:tasks.workunit.client.1.smithi196.stdout:5/135: symlink d1/l2c 0 2024-04-22T15:18:50.687 INFO:tasks.workunit.client.1.smithi196.stdout:2/169: dread d1/d1d/f29 [0,4194304] 0 2024-04-22T15:18:50.689 INFO:tasks.workunit.client.1.smithi196.stdout:0/151: rename d3 to d3/d9/d36/d3d 22 2024-04-22T15:18:50.692 INFO:tasks.workunit.client.0.smithi096.stdout:3/430: dwrite da/db/d10/f33 [0,4194304] 0 2024-04-22T15:18:50.694 INFO:tasks.workunit.client.1.smithi196.stdout:3/119: creat d0/d6/db/dc/d18/f24 x:0 0 0 2024-04-22T15:18:50.695 INFO:tasks.workunit.client.0.smithi096.stdout:1/675: write d6/f14 [444315,36249] 0 2024-04-22T15:18:50.698 INFO:tasks.workunit.client.0.smithi096.stdout:9/495: mkdir d1/df/d95/d9d 0 2024-04-22T15:18:50.699 INFO:tasks.workunit.client.0.smithi096.stdout:9/496: stat d1/d4/d9/d43/d57/d5e/c93 0 2024-04-22T15:18:50.701 INFO:tasks.workunit.client.1.smithi196.stdout:1/134: symlink d1/db/d16/d1e/l2f 0 2024-04-22T15:18:50.701 INFO:tasks.workunit.client.1.smithi196.stdout:3/120: dwrite d0/f1 [4194304,4194304] 0 2024-04-22T15:18:50.703 INFO:tasks.workunit.client.1.smithi196.stdout:7/159: rmdir d14 39 2024-04-22T15:18:50.708 INFO:tasks.workunit.client.1.smithi196.stdout:9/160: mknod db/dd/d1b/c34 0 2024-04-22T15:18:50.711 INFO:tasks.workunit.client.0.smithi096.stdout:6/468: write d3/d3a/d75/d83/f86 [268765,26612] 0 2024-04-22T15:18:50.712 INFO:tasks.workunit.client.1.smithi196.stdout:6/174: dread dd/f16 [0,4194304] 0 2024-04-22T15:18:50.713 INFO:tasks.workunit.client.1.smithi196.stdout:8/174: symlink d2/d5/d2e/l33 0 2024-04-22T15:18:50.713 INFO:tasks.workunit.client.0.smithi096.stdout:4/569: dwrite d5/f13 [4194304,4194304] 0 2024-04-22T15:18:50.714 INFO:tasks.workunit.client.0.smithi096.stdout:6/469: dread - d3/d28/dd/d11/d2c/d3b/f79 zero size 2024-04-22T15:18:50.716 INFO:tasks.workunit.client.0.smithi096.stdout:2/529: getdents d3/db/df 0 2024-04-22T15:18:50.718 INFO:tasks.workunit.client.0.smithi096.stdout:4/570: chown d5/d7/d1f/d84/lbc 45576438 1 2024-04-22T15:18:50.719 INFO:tasks.workunit.client.0.smithi096.stdout:3/431: unlink da/db/de/c81 0 2024-04-22T15:18:50.722 INFO:tasks.workunit.client.0.smithi096.stdout:9/497: creat d1/d4/d9/d43/d57/d5b/d73/f9e x:0 0 0 2024-04-22T15:18:50.727 INFO:tasks.workunit.client.0.smithi096.stdout:7/781: dwrite d31/d32/d36/f70 [0,4194304] 0 2024-04-22T15:18:50.728 INFO:tasks.workunit.client.0.smithi096.stdout:6/470: mknod d3/d28/dd/d11/d2c/d6f/ca4 0 2024-04-22T15:18:50.729 INFO:tasks.workunit.client.0.smithi096.stdout:6/471: chown d3/d3a/d75/d83 2 1 2024-04-22T15:18:50.733 INFO:tasks.workunit.client.1.smithi196.stdout:3/121: unlink d0/d15/ff 0 2024-04-22T15:18:50.734 INFO:tasks.workunit.client.0.smithi096.stdout:4/571: symlink d5/d7/d2c/d67/lc5 0 2024-04-22T15:18:50.737 INFO:tasks.workunit.client.1.smithi196.stdout:1/135: rename d1/db/d16/f1c to d1/db/dc/d26/f30 0 2024-04-22T15:18:50.738 INFO:tasks.workunit.client.1.smithi196.stdout:2/170: write d1/ff [1345216,38653] 0 2024-04-22T15:18:50.739 INFO:tasks.workunit.client.0.smithi096.stdout:9/498: mkdir d1/d25/d9f 0 2024-04-22T15:18:50.745 INFO:tasks.workunit.client.0.smithi096.stdout:3/432: mkdir da/db/de/d8b 0 2024-04-22T15:18:50.747 INFO:tasks.workunit.client.0.smithi096.stdout:7/782: mknod d31/d32/c102 0 2024-04-22T15:18:50.751 INFO:tasks.workunit.client.1.smithi196.stdout:0/152: symlink d3/de/d14/d29/l3e 0 2024-04-22T15:18:50.752 INFO:tasks.workunit.client.1.smithi196.stdout:0/153: dread d3/de/f18 [0,4194304] 0 2024-04-22T15:18:50.754 INFO:tasks.workunit.client.1.smithi196.stdout:3/122: creat d0/d6/f25 x:0 0 0 2024-04-22T15:18:50.755 INFO:tasks.workunit.client.0.smithi096.stdout:2/530: getdents d3/db/d2d 0 2024-04-22T15:18:50.755 INFO:tasks.workunit.client.0.smithi096.stdout:2/531: chown d3/db/d2d/d46/d7e/d8d 12608941 1 2024-04-22T15:18:50.756 INFO:tasks.workunit.client.1.smithi196.stdout:1/136: mkdir d1/db/dc/d26/d31 0 2024-04-22T15:18:50.758 INFO:tasks.workunit.client.1.smithi196.stdout:9/161: mknod db/dd/d1b/d2c/c35 0 2024-04-22T15:18:50.759 INFO:tasks.workunit.client.0.smithi096.stdout:6/472: rename d3/d28/dd/f4a to d3/d28/dd/fa5 0 2024-04-22T15:18:50.760 INFO:tasks.workunit.client.1.smithi196.stdout:2/171: write d1/f3 [2573464,32521] 0 2024-04-22T15:18:50.763 INFO:tasks.workunit.client.1.smithi196.stdout:6/175: mknod dd/d1f/c32 0 2024-04-22T15:18:50.765 INFO:tasks.workunit.client.1.smithi196.stdout:5/136: link d1/l2c d1/d17/l2d 0 2024-04-22T15:18:50.768 INFO:tasks.workunit.client.1.smithi196.stdout:3/123: creat d0/d6/db/d13/f26 x:0 0 0 2024-04-22T15:18:50.769 INFO:tasks.workunit.client.1.smithi196.stdout:3/124: truncate d0/d7/f1a 834809 0 2024-04-22T15:18:50.769 INFO:tasks.workunit.client.0.smithi096.stdout:2/532: truncate d3/d1f/d2f/f47 2897159 0 2024-04-22T15:18:50.771 INFO:tasks.workunit.client.0.smithi096.stdout:7/783: truncate d31/d32/d33/d93/dd5/ff2 1422974 0 2024-04-22T15:18:50.772 INFO:tasks.workunit.client.0.smithi096.stdout:3/433: dwrite da/db/dc/d28/f85 [0,4194304] 0 2024-04-22T15:18:50.773 INFO:tasks.workunit.client.1.smithi196.stdout:3/125: dwrite d0/d15/f12 [0,4194304] 0 2024-04-22T15:18:50.777 INFO:tasks.workunit.client.1.smithi196.stdout:2/172: mkdir d1/d13/d41 0 2024-04-22T15:18:50.788 INFO:tasks.workunit.client.0.smithi096.stdout:6/473: truncate d3/d28/dd/d11/d2c/d6f/f89 2320324 0 2024-04-22T15:18:50.789 INFO:tasks.workunit.client.0.smithi096.stdout:3/434: rmdir da/db/dc 39 2024-04-22T15:18:50.790 INFO:tasks.workunit.client.0.smithi096.stdout:3/435: fdatasync da/db/d12/d22/f3a 0 2024-04-22T15:18:50.791 INFO:tasks.workunit.client.1.smithi196.stdout:5/137: unlink d1/d14/f16 0 2024-04-22T15:18:50.792 INFO:tasks.workunit.client.1.smithi196.stdout:9/162: write db/d18/f1f [1354415,52447] 0 2024-04-22T15:18:50.795 INFO:tasks.workunit.client.1.smithi196.stdout:0/154: truncate d3/de/d1c/f31 2410284 0 2024-04-22T15:18:50.795 INFO:tasks.workunit.client.1.smithi196.stdout:0/155: write d3/de/f37 [346985,69113] 0 2024-04-22T15:18:50.796 INFO:tasks.workunit.client.0.smithi096.stdout:2/533: write d3/db/d2d/d46/d69/f6c [838574,114078] 0 2024-04-22T15:18:50.796 INFO:tasks.workunit.client.1.smithi196.stdout:6/176: write f4 [5206626,65032] 0 2024-04-22T15:18:50.797 INFO:tasks.workunit.client.0.smithi096.stdout:2/534: fsync d3/d1f/f9f 0 2024-04-22T15:18:50.799 INFO:tasks.workunit.client.1.smithi196.stdout:4/161: sync 2024-04-22T15:18:50.800 INFO:tasks.workunit.client.1.smithi196.stdout:1/137: mkdir d1/db/dc/d26/d2e/d32 0 2024-04-22T15:18:50.800 INFO:tasks.workunit.client.1.smithi196.stdout:0/156: dread d3/de/d1c/d27/f38 [0,4194304] 0 2024-04-22T15:18:50.800 INFO:tasks.workunit.client.1.smithi196.stdout:4/162: readlink d6/d10/ld 0 2024-04-22T15:18:50.800 INFO:tasks.workunit.client.0.smithi096.stdout:3/436: write da/db/dc/d59/f79 [3811733,34646] 0 2024-04-22T15:18:50.800 INFO:tasks.workunit.client.1.smithi196.stdout:4/163: chown d6/d14/c1f 7894379 1 2024-04-22T15:18:50.801 INFO:tasks.workunit.client.1.smithi196.stdout:4/164: fsync d6/d14/f12 0 2024-04-22T15:18:50.805 INFO:tasks.workunit.client.1.smithi196.stdout:3/126: rename d0/d7/fa to d0/d7/f27 0 2024-04-22T15:18:50.810 INFO:tasks.workunit.client.0.smithi096.stdout:2/535: creat d3/db/df/d24/fa6 x:0 0 0 2024-04-22T15:18:50.810 INFO:tasks.workunit.client.1.smithi196.stdout:8/175: getdents d2/d5/d2e 0 2024-04-22T15:18:50.812 INFO:tasks.workunit.client.0.smithi096.stdout:7/784: dwrite d31/d32/f3f [0,4194304] 0 2024-04-22T15:18:50.818 INFO:tasks.workunit.client.1.smithi196.stdout:9/163: creat db/dd/de/d15/f36 x:0 0 0 2024-04-22T15:18:50.821 INFO:tasks.workunit.client.0.smithi096.stdout:3/437: mknod da/db/dc/d28/d69/c8c 0 2024-04-22T15:18:50.823 INFO:tasks.workunit.client.1.smithi196.stdout:6/177: stat dd/c13 0 2024-04-22T15:18:50.830 INFO:tasks.workunit.client.0.smithi096.stdout:7/785: chown d31/d32/d33/d6a/d83/dba/ccc 161175 1 2024-04-22T15:18:50.834 INFO:tasks.workunit.client.0.smithi096.stdout:3/438: unlink da/db/d12/d22/f60 0 2024-04-22T15:18:50.835 INFO:tasks.workunit.client.1.smithi196.stdout:3/127: mknod d0/d6/db/c28 0 2024-04-22T15:18:50.838 INFO:tasks.workunit.client.0.smithi096.stdout:3/439: dread da/db/d10/f33 [4194304,4194304] 0 2024-04-22T15:18:50.839 INFO:tasks.workunit.client.0.smithi096.stdout:3/440: readlink da/db/l48 0 2024-04-22T15:18:50.840 INFO:tasks.workunit.client.0.smithi096.stdout:7/786: unlink d31/d32/f47 0 2024-04-22T15:18:50.840 INFO:tasks.workunit.client.1.smithi196.stdout:3/128: dwrite d0/d6/db/d13/f20 [0,4194304] 0 2024-04-22T15:18:50.841 INFO:tasks.workunit.client.1.smithi196.stdout:8/176: creat d2/d5/f34 x:0 0 0 2024-04-22T15:18:50.843 INFO:tasks.workunit.client.1.smithi196.stdout:9/164: creat db/dd/d1b/d2c/f37 x:0 0 0 2024-04-22T15:18:50.845 INFO:tasks.workunit.client.0.smithi096.stdout:6/474: dread d3/d28/dd/d11/d2c/d6f/f89 [0,4194304] 0 2024-04-22T15:18:50.846 INFO:tasks.workunit.client.1.smithi196.stdout:5/138: rmdir d1 39 2024-04-22T15:18:50.846 INFO:tasks.workunit.client.0.smithi096.stdout:6/475: write d3/d28/dd/d11/d2c/f56 [4177565,105088] 0 2024-04-22T15:18:50.847 INFO:tasks.workunit.client.1.smithi196.stdout:7/160: sync 2024-04-22T15:18:50.847 INFO:tasks.workunit.client.0.smithi096.stdout:6/476: readlink d3/l6c 0 2024-04-22T15:18:50.849 INFO:tasks.workunit.client.1.smithi196.stdout:1/138: mknod d1/db/d2d/c33 0 2024-04-22T15:18:50.851 INFO:tasks.workunit.client.1.smithi196.stdout:9/165: dwrite db/dd/de/d15/f36 [0,4194304] 0 2024-04-22T15:18:50.852 INFO:tasks.workunit.client.0.smithi096.stdout:3/441: rmdir da/db/de 39 2024-04-22T15:18:50.854 INFO:tasks.workunit.client.0.smithi096.stdout:7/787: rmdir d31/d32/dad 39 2024-04-22T15:18:50.856 INFO:tasks.workunit.client.1.smithi196.stdout:6/178: dwrite dd/f1a [0,4194304] 0 2024-04-22T15:18:50.857 INFO:tasks.workunit.client.0.smithi096.stdout:2/536: link d3/d1f/d5a/l9a d3/db/df/d24/la7 0 2024-04-22T15:18:50.860 INFO:tasks.workunit.client.1.smithi196.stdout:3/129: fsync d0/d6/db/dc/f1c 0 2024-04-22T15:18:50.860 INFO:tasks.workunit.client.0.smithi096.stdout:6/477: fdatasync d3/d28/dd/d11/f41 0 2024-04-22T15:18:50.861 INFO:tasks.workunit.client.1.smithi196.stdout:3/130: dread d0/d7/f1a [0,4194304] 0 2024-04-22T15:18:50.861 INFO:tasks.workunit.client.1.smithi196.stdout:3/131: chown d0/d6/db/dc/d18 2544615 1 2024-04-22T15:18:50.862 INFO:tasks.workunit.client.0.smithi096.stdout:3/442: write da/db/de/d19/d3e/f4f [4409825,38682] 0 2024-04-22T15:18:50.863 INFO:tasks.workunit.client.0.smithi096.stdout:6/478: read d3/d28/dd/d11/d2c/f56 [3070648,109696] 0 2024-04-22T15:18:50.869 INFO:tasks.workunit.client.1.smithi196.stdout:9/166: creat db/d11/d29/f38 x:0 0 0 2024-04-22T15:18:50.874 INFO:tasks.workunit.client.0.smithi096.stdout:3/443: mkdir da/db/dc/d8d 0 2024-04-22T15:18:50.875 INFO:tasks.workunit.client.0.smithi096.stdout:3/444: fdatasync da/db/d12/d22/f61 0 2024-04-22T15:18:50.876 INFO:tasks.workunit.client.0.smithi096.stdout:6/479: chown d3/d28/dd/f19 0 1 2024-04-22T15:18:50.877 INFO:tasks.workunit.client.0.smithi096.stdout:3/445: fsync da/db/d12/d22/d26/d52/f86 0 2024-04-22T15:18:50.878 INFO:tasks.workunit.client.0.smithi096.stdout:6/480: dread - d3/d28/dd/d11/d2c/d44/d51/f88 zero size 2024-04-22T15:18:50.880 INFO:tasks.workunit.client.0.smithi096.stdout:2/537: fsync d3/f87 0 2024-04-22T15:18:50.883 INFO:tasks.workunit.client.0.smithi096.stdout:3/446: creat da/db/de/d19/d3e/f8e x:0 0 0 2024-04-22T15:18:50.884 INFO:tasks.workunit.client.1.smithi196.stdout:8/177: mkdir d2/d35 0 2024-04-22T15:18:50.886 INFO:tasks.workunit.client.0.smithi096.stdout:6/481: write d3/d28/d54/d68/d6e/f80 [2853651,80381] 0 2024-04-22T15:18:50.886 INFO:tasks.workunit.client.1.smithi196.stdout:3/132: symlink d0/d6/l29 0 2024-04-22T15:18:50.889 INFO:tasks.workunit.client.1.smithi196.stdout:2/173: sync 2024-04-22T15:18:50.889 INFO:tasks.workunit.client.1.smithi196.stdout:5/139: unlink d1/d4/l18 0 2024-04-22T15:18:50.890 INFO:tasks.workunit.client.0.smithi096.stdout:7/788: dwrite d31/d32/d33/d54/d43/f48 [0,4194304] 0 2024-04-22T15:18:50.892 INFO:tasks.workunit.client.0.smithi096.stdout:7/789: write d31/d32/d33/fb5 [1214443,57365] 0 2024-04-22T15:18:50.893 INFO:tasks.workunit.client.1.smithi196.stdout:9/167: mknod db/d18/c39 0 2024-04-22T15:18:50.894 INFO:tasks.workunit.client.0.smithi096.stdout:7/790: truncate d31/d32/d36/f87 1779076 0 2024-04-22T15:18:50.894 INFO:tasks.workunit.client.0.smithi096.stdout:6/482: symlink d3/d3a/d75/d83/la6 0 2024-04-22T15:18:50.896 INFO:tasks.workunit.client.1.smithi196.stdout:0/157: getdents d3/d6 0 2024-04-22T15:18:50.899 INFO:tasks.workunit.client.1.smithi196.stdout:1/139: link d1/db/d16/f17 d1/db/dc/d26/d2e/f34 0 2024-04-22T15:18:50.899 INFO:tasks.workunit.client.0.smithi096.stdout:2/538: creat d3/d1f/d5a/fa8 x:0 0 0 2024-04-22T15:18:50.901 INFO:tasks.workunit.client.1.smithi196.stdout:4/165: getdents d6/d14 0 2024-04-22T15:18:50.902 INFO:tasks.workunit.client.1.smithi196.stdout:6/179: symlink dd/l33 0 2024-04-22T15:18:50.903 INFO:tasks.workunit.client.1.smithi196.stdout:1/140: chown d1/db/d2d/c33 15173220 1 2024-04-22T15:18:50.903 INFO:tasks.workunit.client.1.smithi196.stdout:9/168: dwrite db/d18/f24 [0,4194304] 0 2024-04-22T15:18:50.904 INFO:tasks.workunit.client.1.smithi196.stdout:4/166: dread - d6/d13/d15/f30 zero size 2024-04-22T15:18:50.904 INFO:tasks.workunit.client.1.smithi196.stdout:6/180: fsync dd/d20/f24 0 2024-04-22T15:18:50.905 INFO:tasks.workunit.client.1.smithi196.stdout:8/178: rename d2/dc/c16 to d2/d5/d1a/c36 0 2024-04-22T15:18:50.906 INFO:tasks.workunit.client.0.smithi096.stdout:2/539: dwrite d3/db/d2d/d46/f7c [0,4194304] 0 2024-04-22T15:18:50.907 INFO:tasks.workunit.client.1.smithi196.stdout:3/133: mknod d0/d6/db/c2a 0 2024-04-22T15:18:50.908 INFO:tasks.workunit.client.0.smithi096.stdout:3/447: creat da/f8f x:0 0 0 2024-04-22T15:18:50.908 INFO:tasks.workunit.client.1.smithi196.stdout:0/158: dwrite d3/d9/f21 [0,4194304] 0 2024-04-22T15:18:50.910 INFO:tasks.workunit.client.0.smithi096.stdout:3/448: write da/db/dc/f7b [927262,111989] 0 2024-04-22T15:18:50.910 INFO:tasks.workunit.client.1.smithi196.stdout:3/134: write d0/d6/db/d13/f20 [3268279,46833] 0 2024-04-22T15:18:50.912 INFO:tasks.workunit.client.0.smithi096.stdout:3/449: write da/db/de/d19/d3e/f8e [761590,1628] 0 2024-04-22T15:18:50.913 INFO:tasks.workunit.client.1.smithi196.stdout:2/174: rmdir d1/d13/d25 39 2024-04-22T15:18:50.914 INFO:tasks.workunit.client.1.smithi196.stdout:5/140: rmdir d1/d14 39 2024-04-22T15:18:50.914 INFO:tasks.workunit.client.0.smithi096.stdout:3/450: write da/db/d12/d22/f61 [74860,127558] 0 2024-04-22T15:18:50.915 INFO:tasks.workunit.client.1.smithi196.stdout:7/161: mkdir d14/d25/d31 0 2024-04-22T15:18:50.915 INFO:tasks.workunit.client.1.smithi196.stdout:6/181: dwrite f4 [0,4194304] 0 2024-04-22T15:18:50.918 INFO:tasks.workunit.client.1.smithi196.stdout:2/175: write d1/d1d/f3c [352102,8344] 0 2024-04-22T15:18:50.918 INFO:tasks.workunit.client.1.smithi196.stdout:6/182: stat dd/c2a 0 2024-04-22T15:18:50.918 INFO:tasks.workunit.client.0.smithi096.stdout:7/791: creat d31/d32/f103 x:0 0 0 2024-04-22T15:18:50.922 INFO:tasks.workunit.client.1.smithi196.stdout:0/159: dwrite d3/d6/f33 [0,4194304] 0 2024-04-22T15:18:50.925 INFO:tasks.workunit.client.1.smithi196.stdout:1/141: write d1/db/d16/f17 [8840168,43255] 0 2024-04-22T15:18:50.928 INFO:tasks.workunit.client.0.smithi096.stdout:7/792: symlink d31/d32/dad/l104 0 2024-04-22T15:18:50.929 INFO:tasks.workunit.client.1.smithi196.stdout:9/169: symlink db/dd/de/d15/l3a 0 2024-04-22T15:18:50.931 INFO:tasks.workunit.client.1.smithi196.stdout:9/170: write db/dd/de/d15/d31/f32 [474076,30985] 0 2024-04-22T15:18:50.933 INFO:tasks.workunit.client.1.smithi196.stdout:3/135: mkdir d0/d7/d2b 0 2024-04-22T15:18:50.934 INFO:tasks.workunit.client.1.smithi196.stdout:3/136: chown d0/d15/l21 105458 1 2024-04-22T15:18:50.936 INFO:tasks.workunit.client.1.smithi196.stdout:8/179: dwrite d2/dc/fe [4194304,4194304] 0 2024-04-22T15:18:50.936 INFO:tasks.workunit.client.1.smithi196.stdout:5/141: creat d1/d17/f2e x:0 0 0 2024-04-22T15:18:50.937 INFO:tasks.workunit.client.1.smithi196.stdout:8/180: readlink d2/l30 0 2024-04-22T15:18:50.937 INFO:tasks.workunit.client.1.smithi196.stdout:6/183: fsync dd/fe 0 2024-04-22T15:18:50.938 INFO:tasks.workunit.client.0.smithi096.stdout:7/793: link d31/d32/d36/d5d/fbc d31/d32/d36/d5d/f105 0 2024-04-22T15:18:50.938 INFO:tasks.workunit.client.1.smithi196.stdout:5/142: chown d1/d9/c15 0 1 2024-04-22T15:18:50.939 INFO:tasks.workunit.client.1.smithi196.stdout:7/162: dread f10 [0,4194304] 0 2024-04-22T15:18:50.940 INFO:tasks.workunit.client.1.smithi196.stdout:0/160: write d3/de/d1c/d27/f38 [4432948,49642] 0 2024-04-22T15:18:50.940 INFO:tasks.workunit.client.1.smithi196.stdout:9/171: mknod db/dd/d1b/c3b 0 2024-04-22T15:18:50.941 INFO:tasks.workunit.client.1.smithi196.stdout:3/137: symlink d0/d6/db/dc/l2c 0 2024-04-22T15:18:50.941 INFO:tasks.workunit.client.1.smithi196.stdout:2/176: mknod d1/d23/d27/d2e/c42 0 2024-04-22T15:18:50.942 INFO:tasks.workunit.client.1.smithi196.stdout:0/161: chown d3/de/d1c/d27/d2c/d30/f35 58231 1 2024-04-22T15:18:50.968 INFO:tasks.workunit.client.0.smithi096.stdout:8/469: sync 2024-04-22T15:18:50.968 INFO:tasks.workunit.client.0.smithi096.stdout:5/584: sync 2024-04-22T15:18:50.968 INFO:tasks.workunit.client.1.smithi196.stdout:5/143: mkdir d1/d17/d1a/d2f 0 2024-04-22T15:18:50.968 INFO:tasks.workunit.client.1.smithi196.stdout:5/144: chown d1/d4/f26 600948 1 2024-04-22T15:18:50.972 INFO:tasks.workunit.client.1.smithi196.stdout:5/145: dread d1/d9/f11 [0,4194304] 0 2024-04-22T15:18:50.972 INFO:tasks.workunit.client.1.smithi196.stdout:7/163: rename d14/l24 to d14/d26/l32 0 2024-04-22T15:18:50.973 INFO:tasks.workunit.client.1.smithi196.stdout:4/167: sync 2024-04-22T15:18:50.974 INFO:tasks.workunit.client.1.smithi196.stdout:9/172: creat db/dd/de/d15/f3c x:0 0 0 2024-04-22T15:18:50.975 INFO:tasks.workunit.client.1.smithi196.stdout:7/164: dread f4 [4194304,4194304] 0 2024-04-22T15:18:50.975 INFO:tasks.workunit.client.1.smithi196.stdout:7/165: chown d14/d25 147595 1 2024-04-22T15:18:50.977 INFO:tasks.workunit.client.1.smithi196.stdout:2/177: write d1/f5 [395894,15921] 0 2024-04-22T15:18:50.978 INFO:tasks.workunit.client.1.smithi196.stdout:2/178: write d1/f5 [312033,18377] 0 2024-04-22T15:18:50.979 INFO:tasks.workunit.client.0.smithi096.stdout:5/585: rename d4/d2c/d2f/d6b/f77 to d4/d11/d3d/d35/fbf 0 2024-04-22T15:18:50.979 INFO:tasks.workunit.client.0.smithi096.stdout:2/540: truncate d3/d1f/d2f/f34 240672 0 2024-04-22T15:18:50.981 INFO:tasks.workunit.client.1.smithi196.stdout:8/181: link d2/f32 d2/d5/d1a/f37 0 2024-04-22T15:18:50.982 INFO:tasks.workunit.client.1.smithi196.stdout:0/162: mknod d3/de/d1c/d27/d2c/d30/c3f 0 2024-04-22T15:18:50.983 INFO:tasks.workunit.client.1.smithi196.stdout:0/163: readlink d3/d6/l8 0 2024-04-22T15:18:50.984 INFO:tasks.workunit.client.0.smithi096.stdout:5/586: fdatasync d4/d11/d3d/d4d/d59/d5a/f79 0 2024-04-22T15:18:50.984 INFO:tasks.workunit.client.1.smithi196.stdout:1/142: link d1/c6 d1/db/dc/d26/d2e/d32/c35 0 2024-04-22T15:18:50.985 INFO:tasks.workunit.client.0.smithi096.stdout:5/587: chown d4/d11/d3d/d35/c45 1104 1 2024-04-22T15:18:50.985 INFO:tasks.workunit.client.1.smithi196.stdout:1/143: read - d1/db/f29 zero size 2024-04-22T15:18:50.986 INFO:tasks.workunit.client.0.smithi096.stdout:8/470: getdents d4/d10 0 2024-04-22T15:18:50.987 INFO:tasks.workunit.client.1.smithi196.stdout:6/184: link dd/d2b/f2e dd/d20/d30/f34 0 2024-04-22T15:18:50.987 INFO:tasks.workunit.client.1.smithi196.stdout:6/185: stat dd/d20 0 2024-04-22T15:18:50.988 INFO:tasks.workunit.client.1.smithi196.stdout:6/186: chown dd/d20/f24 582 1 2024-04-22T15:18:50.989 INFO:tasks.workunit.client.1.smithi196.stdout:5/146: rename d1/d17/d2b to d1/d17/d1a/d2f/d30 0 2024-04-22T15:18:50.989 INFO:tasks.workunit.client.1.smithi196.stdout:4/168: rename d6/d13 to d6/d13/d15/d31 22 2024-04-22T15:18:50.990 INFO:tasks.workunit.client.1.smithi196.stdout:4/169: read f5 [1244131,6163] 0 2024-04-22T15:18:50.991 INFO:tasks.workunit.client.0.smithi096.stdout:8/471: mknod d4/d10/d4d/ca4 0 2024-04-22T15:18:50.992 INFO:tasks.workunit.client.0.smithi096.stdout:8/472: chown d4/d10/d4d/ca4 381 1 2024-04-22T15:18:50.992 INFO:tasks.workunit.client.1.smithi196.stdout:9/173: creat db/d18/f3d x:0 0 0 2024-04-22T15:18:50.992 INFO:tasks.workunit.client.0.smithi096.stdout:8/473: chown d4/d10/c9d 468558215 1 2024-04-22T15:18:50.994 INFO:tasks.workunit.client.0.smithi096.stdout:8/474: read d4/d10/d25/d38/f4b [5232490,8400] 0 2024-04-22T15:18:50.995 INFO:tasks.workunit.client.1.smithi196.stdout:3/138: truncate d0/f1 1649672 0 2024-04-22T15:18:50.997 INFO:tasks.workunit.client.0.smithi096.stdout:5/588: creat d4/d11/d3d/d4d/d59/fc0 x:0 0 0 2024-04-22T15:18:50.998 INFO:tasks.workunit.client.1.smithi196.stdout:7/166: dwrite f4 [4194304,4194304] 0 2024-04-22T15:18:50.999 INFO:tasks.workunit.client.0.smithi096.stdout:7/794: truncate d31/d32/d33/f3e 649717 0 2024-04-22T15:18:51.000 INFO:tasks.workunit.client.0.smithi096.stdout:8/475: creat d4/d1c/d62/fa5 x:0 0 0 2024-04-22T15:18:51.001 INFO:tasks.workunit.client.1.smithi196.stdout:2/179: readlink d1/l1a 0 2024-04-22T15:18:51.004 INFO:tasks.workunit.client.1.smithi196.stdout:0/164: rmdir d3/de/d1c 39 2024-04-22T15:18:51.005 INFO:tasks.workunit.client.0.smithi096.stdout:7/795: creat d31/d32/d33/d44/da8/dbf/f106 x:0 0 0 2024-04-22T15:18:51.008 INFO:tasks.workunit.client.0.smithi096.stdout:8/476: creat d4/d2a/d7e/fa6 x:0 0 0 2024-04-22T15:18:51.008 INFO:tasks.workunit.client.0.smithi096.stdout:8/477: chown d4/d10/d25 5355501 1 2024-04-22T15:18:51.010 INFO:tasks.workunit.client.0.smithi096.stdout:8/478: write d4/d1c/f43 [1783612,35134] 0 2024-04-22T15:18:51.017 INFO:tasks.workunit.client.1.smithi196.stdout:6/187: symlink dd/d1f/l35 0 2024-04-22T15:18:51.017 INFO:tasks.workunit.client.1.smithi196.stdout:6/188: truncate fc 4296700 0 2024-04-22T15:18:51.021 INFO:tasks.workunit.client.1.smithi196.stdout:5/147: creat d1/d17/f31 x:0 0 0 2024-04-22T15:18:51.022 INFO:tasks.workunit.client.0.smithi096.stdout:8/479: mkdir d4/d1c/d40/da7 0 2024-04-22T15:18:51.022 INFO:tasks.workunit.client.1.smithi196.stdout:6/189: dwrite dd/d20/f22 [0,4194304] 0 2024-04-22T15:18:51.028 INFO:tasks.workunit.client.1.smithi196.stdout:9/174: symlink db/dd/de/l3e 0 2024-04-22T15:18:51.031 INFO:tasks.workunit.client.1.smithi196.stdout:3/139: creat d0/d6/f2d x:0 0 0 2024-04-22T15:18:51.032 INFO:tasks.workunit.client.1.smithi196.stdout:7/167: creat d14/f33 x:0 0 0 2024-04-22T15:18:51.033 INFO:tasks.workunit.client.1.smithi196.stdout:7/168: chown d14/d15/c2c 3 1 2024-04-22T15:18:51.034 INFO:tasks.workunit.client.1.smithi196.stdout:7/169: chown d14/f1e 500 1 2024-04-22T15:18:51.035 INFO:tasks.workunit.client.0.smithi096.stdout:5/589: write d4/d11/d3d/d4d/fb1 [275212,6827] 0 2024-04-22T15:18:51.037 INFO:tasks.workunit.client.1.smithi196.stdout:0/165: rename d3/de/c1a to d3/d9/d25/c40 0 2024-04-22T15:18:51.038 INFO:tasks.workunit.client.1.smithi196.stdout:1/144: mkdir d1/db/dc/d26/d2e/d32/d36 0 2024-04-22T15:18:51.040 INFO:tasks.workunit.client.1.smithi196.stdout:2/180: dwrite d1/f15 [0,4194304] 0 2024-04-22T15:18:51.042 INFO:tasks.workunit.client.0.smithi096.stdout:5/590: mknod d4/d11/d3d/d4d/d59/d5a/cc1 0 2024-04-22T15:18:51.043 INFO:tasks.workunit.client.1.smithi196.stdout:0/166: dwrite d3/fd [4194304,4194304] 0 2024-04-22T15:18:51.044 INFO:tasks.workunit.client.1.smithi196.stdout:5/148: write d1/d14/f1e [449693,5251] 0 2024-04-22T15:18:51.044 INFO:tasks.workunit.client.0.smithi096.stdout:7/796: write f7 [643771,54017] 0 2024-04-22T15:18:51.045 INFO:tasks.workunit.client.1.smithi196.stdout:6/190: creat dd/d1f/f36 x:0 0 0 2024-04-22T15:18:51.045 INFO:tasks.workunit.client.0.smithi096.stdout:5/591: chown d4/d1f/f21 6917 1 2024-04-22T15:18:51.047 INFO:tasks.workunit.client.1.smithi196.stdout:4/170: truncate d6/d13/f1d 3006751 0 2024-04-22T15:18:51.047 INFO:tasks.workunit.client.1.smithi196.stdout:4/171: readlink d6/l17 0 2024-04-22T15:18:51.052 INFO:tasks.workunit.client.1.smithi196.stdout:3/140: creat d0/d6/db/f2e x:0 0 0 2024-04-22T15:18:51.053 INFO:tasks.workunit.client.0.smithi096.stdout:7/797: chown d31/l61 14 1 2024-04-22T15:18:51.053 INFO:tasks.workunit.client.1.smithi196.stdout:3/141: write d0/d15/f12 [3437939,40929] 0 2024-04-22T15:18:51.054 INFO:tasks.workunit.client.0.smithi096.stdout:7/798: dread - d31/d32/d33/d44/da8/dbf/ddf/ff6 zero size 2024-04-22T15:18:51.055 INFO:tasks.workunit.client.1.smithi196.stdout:7/170: symlink d14/d15/l34 0 2024-04-22T15:18:51.055 INFO:tasks.workunit.client.1.smithi196.stdout:4/172: dwrite d6/d13/f25 [0,4194304] 0 2024-04-22T15:18:51.057 INFO:tasks.workunit.client.1.smithi196.stdout:8/182: rename d2/f3 to d2/f38 0 2024-04-22T15:18:51.059 INFO:tasks.workunit.client.0.smithi096.stdout:7/799: mkdir d31/d32/d33/d6a/d83/d107 0 2024-04-22T15:18:51.060 INFO:tasks.workunit.client.0.smithi096.stdout:8/480: write d4/d10/d25/f6c [454413,60550] 0 2024-04-22T15:18:51.062 INFO:tasks.workunit.client.1.smithi196.stdout:1/145: rmdir d1/db/dc/de 39 2024-04-22T15:18:51.063 INFO:tasks.workunit.client.1.smithi196.stdout:2/181: fsync d1/f2b 0 2024-04-22T15:18:51.063 INFO:tasks.workunit.client.1.smithi196.stdout:2/182: chown d1/fd 2032609 1 2024-04-22T15:18:51.064 INFO:tasks.workunit.client.0.smithi096.stdout:7/800: creat d31/f108 x:0 0 0 2024-04-22T15:18:51.066 INFO:tasks.workunit.client.1.smithi196.stdout:0/167: fsync d3/fd 0 2024-04-22T15:18:51.069 INFO:tasks.workunit.client.1.smithi196.stdout:6/191: rename dd/d20/d30 to dd/d1f/d37 0 2024-04-22T15:18:51.071 INFO:tasks.workunit.client.0.smithi096.stdout:5/592: dread d4/d2c/d2f/d6b/f50 [0,4194304] 0 2024-04-22T15:18:51.077 INFO:tasks.workunit.client.0.smithi096.stdout:7/801: mknod d31/d32/d33/d93/df0/c109 0 2024-04-22T15:18:51.078 INFO:tasks.workunit.client.0.smithi096.stdout:5/593: rmdir d4/d11/d3d/d4d/d59/db0 39 2024-04-22T15:18:51.085 INFO:tasks.workunit.client.0.smithi096.stdout:8/481: link d4/d10/d4d/ca4 d4/d1c/ca8 0 2024-04-22T15:18:51.093 INFO:tasks.workunit.client.0.smithi096.stdout:8/482: creat d4/d10/d4d/fa9 x:0 0 0 2024-04-22T15:18:51.094 INFO:tasks.workunit.client.1.smithi196.stdout:0/168: symlink d3/d6/dc/l41 0 2024-04-22T15:18:51.095 INFO:tasks.workunit.client.1.smithi196.stdout:7/171: write f13 [894079,79780] 0 2024-04-22T15:18:51.095 INFO:tasks.workunit.client.1.smithi196.stdout:7/172: chown f7 270474 1 2024-04-22T15:18:51.099 INFO:tasks.workunit.client.0.smithi096.stdout:7/802: write d31/d32/d33/d93/dd5/ff2 [2312218,60203] 0 2024-04-22T15:18:51.101 INFO:tasks.workunit.client.1.smithi196.stdout:4/173: mkdir d6/d14/d18/d26/d32 0 2024-04-22T15:18:51.101 INFO:tasks.workunit.client.0.smithi096.stdout:7/803: dread - d31/d32/d33/d44/da8/dbf/f106 zero size 2024-04-22T15:18:51.102 INFO:tasks.workunit.client.1.smithi196.stdout:8/183: creat d2/d35/f39 x:0 0 0 2024-04-22T15:18:51.103 INFO:tasks.workunit.client.1.smithi196.stdout:8/184: read d2/f38 [6265352,110989] 0 2024-04-22T15:18:51.103 INFO:tasks.workunit.client.1.smithi196.stdout:1/146: getdents d1/db/dc/d26/d2e/d32/d36 0 2024-04-22T15:18:51.105 INFO:tasks.workunit.client.0.smithi096.stdout:7/804: mkdir d31/d32/d33/d6a/d10a 0 2024-04-22T15:18:51.106 INFO:tasks.workunit.client.0.smithi096.stdout:5/594: dwrite d4/d11/fbd [0,4194304] 0 2024-04-22T15:18:51.107 INFO:tasks.workunit.client.1.smithi196.stdout:3/142: truncate d0/d7/f11 185511 0 2024-04-22T15:18:51.108 INFO:tasks.workunit.client.1.smithi196.stdout:2/183: rmdir d1/d13/d25 39 2024-04-22T15:18:51.108 INFO:tasks.workunit.client.1.smithi196.stdout:3/143: write d0/d15/f12 [2187623,15710] 0 2024-04-22T15:18:51.109 INFO:tasks.workunit.client.1.smithi196.stdout:2/184: write d1/f5 [875291,112465] 0 2024-04-22T15:18:51.111 INFO:tasks.workunit.client.0.smithi096.stdout:7/805: rmdir d31/d32/d33/d44/da8/dbf 39 2024-04-22T15:18:51.113 INFO:tasks.workunit.client.0.smithi096.stdout:7/806: chown d31/d32/d33/d44/fc6 36137 1 2024-04-22T15:18:51.113 INFO:tasks.workunit.client.1.smithi196.stdout:7/173: creat d14/d26/f35 x:0 0 0 2024-04-22T15:18:51.114 INFO:tasks.workunit.client.0.smithi096.stdout:8/483: dwrite d4/d2a/f36 [0,4194304] 0 2024-04-22T15:18:51.114 INFO:tasks.workunit.client.0.smithi096.stdout:7/807: stat d31/d32/d36/l66 0 2024-04-22T15:18:51.114 INFO:tasks.workunit.client.0.smithi096.stdout:5/595: creat d4/d11/d3d/d4d/d99/fc2 x:0 0 0 2024-04-22T15:18:51.119 INFO:tasks.workunit.client.1.smithi196.stdout:4/174: dread d6/d14/f12 [0,4194304] 0 2024-04-22T15:18:51.121 INFO:tasks.workunit.client.0.smithi096.stdout:5/596: dwrite f2 [0,4194304] 0 2024-04-22T15:18:51.123 INFO:tasks.workunit.client.0.smithi096.stdout:8/484: read d4/d10/d25/f2e [3570604,55941] 0 2024-04-22T15:18:51.124 INFO:tasks.workunit.client.0.smithi096.stdout:8/485: stat d4/d1c/d62/fa2 0 2024-04-22T15:18:51.124 INFO:tasks.workunit.client.1.smithi196.stdout:2/185: readlink d1/le 0 2024-04-22T15:18:51.125 INFO:tasks.workunit.client.1.smithi196.stdout:2/186: write d1/d13/f1e [2532900,79506] 0 2024-04-22T15:18:51.127 INFO:tasks.workunit.client.0.smithi096.stdout:7/808: symlink d31/d32/d33/d93/dfc/l10b 0 2024-04-22T15:18:51.128 INFO:tasks.workunit.client.1.smithi196.stdout:3/144: rename d0/d6/l29 to d0/d6/db/l2f 0 2024-04-22T15:18:51.130 INFO:tasks.workunit.client.1.smithi196.stdout:7/174: truncate fd 249 0 2024-04-22T15:18:51.131 INFO:tasks.workunit.client.0.smithi096.stdout:8/486: creat d4/d1c/d40/d46/d58/faa x:0 0 0 2024-04-22T15:18:51.133 INFO:tasks.workunit.client.1.smithi196.stdout:0/169: getdents d3/d6 0 2024-04-22T15:18:51.134 INFO:tasks.workunit.client.0.smithi096.stdout:5/597: symlink d4/d11/d3d/d35/d57/dbe/lc3 0 2024-04-22T15:18:51.135 INFO:tasks.workunit.client.1.smithi196.stdout:2/187: dwrite d1/d13/d1f/f3a [0,4194304] 0 2024-04-22T15:18:51.137 INFO:tasks.workunit.client.0.smithi096.stdout:8/487: creat d4/d1c/d40/d48/fab x:0 0 0 2024-04-22T15:18:51.139 INFO:tasks.workunit.client.0.smithi096.stdout:7/809: rename d31/d32/d33/d54/f8b to d31/d32/d33/d6a/f10c 0 2024-04-22T15:18:51.141 INFO:tasks.workunit.client.0.smithi096.stdout:5/598: read d4/d2c/f7b [6279,35915] 0 2024-04-22T15:18:51.142 INFO:tasks.workunit.client.0.smithi096.stdout:5/599: fdatasync d4/d11/d3d/fb6 0 2024-04-22T15:18:51.143 INFO:tasks.workunit.client.0.smithi096.stdout:8/488: symlink d4/d10/d25/d4c/lac 0 2024-04-22T15:18:51.145 INFO:tasks.workunit.client.0.smithi096.stdout:5/600: read d4/d11/d3d/d4d/d59/f7f [2176966,26172] 0 2024-04-22T15:18:51.147 INFO:tasks.workunit.client.1.smithi196.stdout:0/170: chown d3/d6/c10 28 1 2024-04-22T15:18:51.147 INFO:tasks.workunit.client.1.smithi196.stdout:4/175: write d6/d13/d15/f30 [926265,61718] 0 2024-04-22T15:18:51.148 INFO:tasks.workunit.client.1.smithi196.stdout:4/176: chown c2 33 1 2024-04-22T15:18:51.148 INFO:tasks.workunit.client.1.smithi196.stdout:4/177: chown d6/d14/f2a 9 1 2024-04-22T15:18:51.162 INFO:tasks.workunit.client.1.smithi196.stdout:9/175: sync 2024-04-22T15:18:51.200 INFO:tasks.workunit.client.0.smithi096.stdout:0/724: sync 2024-04-22T15:18:51.200 INFO:tasks.workunit.client.0.smithi096.stdout:9/499: sync 2024-04-22T15:18:51.200 INFO:tasks.workunit.client.0.smithi096.stdout:4/572: sync 2024-04-22T15:18:51.201 INFO:tasks.workunit.client.0.smithi096.stdout:1/676: sync 2024-04-22T15:18:51.203 INFO:tasks.workunit.client.0.smithi096.stdout:1/677: readlink d6/dc/d8a/d3f/d54/d32/d3b/d94/lb0 0 2024-04-22T15:18:51.204 INFO:tasks.workunit.client.1.smithi196.stdout:2/188: write d1/f2c [689458,12245] 0 2024-04-22T15:18:51.208 INFO:tasks.workunit.client.1.smithi196.stdout:5/149: sync 2024-04-22T15:18:51.208 INFO:tasks.workunit.client.0.smithi096.stdout:7/810: unlink d31/d32/d33/d54/f51 0 2024-04-22T15:18:51.208 INFO:tasks.workunit.client.1.smithi196.stdout:3/145: getdents d0/d7/d2b 0 2024-04-22T15:18:51.210 INFO:tasks.workunit.client.1.smithi196.stdout:8/185: getdents d2/d5/d1a 0 2024-04-22T15:18:51.212 INFO:tasks.workunit.client.1.smithi196.stdout:3/146: dread d0/d6/db/dc/f1c [0,4194304] 0 2024-04-22T15:18:51.216 INFO:tasks.workunit.client.0.smithi096.stdout:8/489: dwrite d4/d10/d4d/f74 [0,4194304] 0 2024-04-22T15:18:51.218 INFO:tasks.workunit.client.1.smithi196.stdout:5/150: symlink d1/d9/l32 0 2024-04-22T15:18:51.228 INFO:tasks.workunit.client.1.smithi196.stdout:3/147: rename d0/d6/db/l1d to d0/d7/d8/l30 0 2024-04-22T15:18:51.230 INFO:tasks.workunit.client.1.smithi196.stdout:5/151: symlink d1/d17/d1a/l33 0 2024-04-22T15:18:51.231 INFO:tasks.workunit.client.1.smithi196.stdout:9/176: dread f4 [4194304,4194304] 0 2024-04-22T15:18:51.231 INFO:tasks.workunit.client.1.smithi196.stdout:9/177: rename db/dd to db/dd/de/d15/d3f 22 2024-04-22T15:18:51.232 INFO:tasks.workunit.client.1.smithi196.stdout:4/178: getdents d6/d14 0 2024-04-22T15:18:51.233 INFO:tasks.workunit.client.1.smithi196.stdout:3/148: mkdir d0/d15/d31 0 2024-04-22T15:18:51.235 INFO:tasks.workunit.client.1.smithi196.stdout:5/152: unlink d1/d4/f6 0 2024-04-22T15:18:51.235 INFO:tasks.workunit.client.1.smithi196.stdout:5/153: stat d1/d17/d1c/l27 0 2024-04-22T15:18:51.235 INFO:tasks.workunit.client.1.smithi196.stdout:5/154: fdatasync d1/d17/f24 0 2024-04-22T15:18:51.236 INFO:tasks.workunit.client.1.smithi196.stdout:9/178: rename f2 to db/dd/de/d15/f40 0 2024-04-22T15:18:51.243 INFO:tasks.workunit.client.1.smithi196.stdout:2/189: getdents d1/d13/d25/d2a 0 2024-04-22T15:18:51.243 INFO:tasks.workunit.client.1.smithi196.stdout:2/190: readlink d1/le 0 2024-04-22T15:18:51.243 INFO:tasks.workunit.client.1.smithi196.stdout:2/191: fsync d1/f3e 0 2024-04-22T15:18:51.245 INFO:tasks.workunit.client.1.smithi196.stdout:3/149: rename d0/c10 to d0/c32 0 2024-04-22T15:18:51.245 INFO:tasks.workunit.client.1.smithi196.stdout:3/150: read - d0/d6/f25 zero size 2024-04-22T15:18:51.246 INFO:tasks.workunit.client.0.smithi096.stdout:9/500: creat d1/d25/fa0 x:0 0 0 2024-04-22T15:18:51.247 INFO:tasks.workunit.client.1.smithi196.stdout:2/192: mknod d1/d13/c43 0 2024-04-22T15:18:51.248 INFO:tasks.workunit.client.1.smithi196.stdout:2/193: write d1/f15 [1352106,121938] 0 2024-04-22T15:18:51.248 INFO:tasks.workunit.client.1.smithi196.stdout:3/151: symlink d0/d6/db/dc/d18/l33 0 2024-04-22T15:18:51.251 INFO:tasks.workunit.client.1.smithi196.stdout:2/194: creat d1/d23/d27/f44 x:0 0 0 2024-04-22T15:18:51.253 INFO:tasks.workunit.client.1.smithi196.stdout:3/152: symlink d0/d7/d8/l34 0 2024-04-22T15:18:51.254 INFO:tasks.workunit.client.0.smithi096.stdout:4/573: dwrite d5/d7/d1a/d25/fb7 [0,4194304] 0 2024-04-22T15:18:51.254 INFO:tasks.workunit.client.1.smithi196.stdout:3/153: fsync d0/d7/d8/f19 0 2024-04-22T15:18:51.255 INFO:tasks.workunit.client.1.smithi196.stdout:2/195: mkdir d1/d23/d27/d2e/d45 0 2024-04-22T15:18:51.255 INFO:tasks.workunit.client.1.smithi196.stdout:2/196: readlink d1/l1a 0 2024-04-22T15:18:51.256 INFO:tasks.workunit.client.1.smithi196.stdout:3/154: unlink d0/d15/f17 0 2024-04-22T15:18:51.258 INFO:tasks.workunit.client.0.smithi096.stdout:8/490: unlink d4/d10/d25/c90 0 2024-04-22T15:18:51.259 INFO:tasks.workunit.client.0.smithi096.stdout:0/725: mknod d7/d42/dd4/cf4 0 2024-04-22T15:18:51.261 INFO:tasks.workunit.client.1.smithi196.stdout:3/155: rename d0/d6/f25 to d0/d7/d2b/f35 0 2024-04-22T15:18:51.262 INFO:tasks.workunit.client.0.smithi096.stdout:9/501: mkdir d1/d4/d9/d43/d57/d5b/da1 0 2024-04-22T15:18:51.266 INFO:tasks.workunit.client.1.smithi196.stdout:3/156: dwrite d0/d6/db/d13/f20 [0,4194304] 0 2024-04-22T15:18:51.269 INFO:tasks.workunit.client.1.smithi196.stdout:2/197: rmdir d1/d13/d25/d2a/d38 0 2024-04-22T15:18:51.270 INFO:tasks.workunit.client.0.smithi096.stdout:9/502: creat d1/d85/fa2 x:0 0 0 2024-04-22T15:18:51.271 INFO:tasks.workunit.client.0.smithi096.stdout:8/491: symlink d4/d1c/d40/lad 0 2024-04-22T15:18:51.272 INFO:tasks.workunit.client.0.smithi096.stdout:4/574: dread d5/d7/d1a/f38 [0,4194304] 0 2024-04-22T15:18:51.273 INFO:tasks.workunit.client.0.smithi096.stdout:4/575: chown d5/d7/d2c/f9d 24922903 1 2024-04-22T15:18:51.274 INFO:tasks.workunit.client.1.smithi196.stdout:2/198: symlink d1/d23/d27/d2e/l46 0 2024-04-22T15:18:51.274 INFO:tasks.workunit.client.1.smithi196.stdout:2/199: readlink d1/le 0 2024-04-22T15:18:51.274 INFO:tasks.workunit.client.0.smithi096.stdout:8/492: read d4/d10/f21 [791048,108833] 0 2024-04-22T15:18:51.274 INFO:tasks.workunit.client.0.smithi096.stdout:4/576: readlink d5/d7/d2c/l92 0 2024-04-22T15:18:51.275 INFO:tasks.workunit.client.1.smithi196.stdout:3/157: truncate d0/f1 39436 0 2024-04-22T15:18:51.275 INFO:tasks.workunit.client.1.smithi196.stdout:2/200: write d1/d1d/f22 [714883,109741] 0 2024-04-22T15:18:51.275 INFO:tasks.workunit.client.1.smithi196.stdout:3/158: dread - d0/d6/db/dc/d18/f24 zero size 2024-04-22T15:18:51.279 INFO:tasks.workunit.client.0.smithi096.stdout:1/678: getdents d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b 0 2024-04-22T15:18:51.280 INFO:tasks.workunit.client.1.smithi196.stdout:3/159: symlink d0/d6/l36 0 2024-04-22T15:18:51.281 INFO:tasks.workunit.client.1.smithi196.stdout:3/160: chown d0/d6/db/d13 9 1 2024-04-22T15:18:51.284 INFO:tasks.workunit.client.1.smithi196.stdout:3/161: dread d0/d15/f12 [0,4194304] 0 2024-04-22T15:18:51.287 INFO:tasks.workunit.client.0.smithi096.stdout:8/493: mknod d4/d10/d25/cae 0 2024-04-22T15:18:51.289 INFO:tasks.workunit.client.1.smithi196.stdout:3/162: symlink d0/d6/db/l37 0 2024-04-22T15:18:51.289 INFO:tasks.workunit.client.1.smithi196.stdout:3/163: readlink d0/d15/l21 0 2024-04-22T15:18:51.290 INFO:tasks.workunit.client.1.smithi196.stdout:0/171: truncate d3/d9/f21 3719339 0 2024-04-22T15:18:51.290 INFO:tasks.workunit.client.1.smithi196.stdout:2/201: link d1/f7 d1/d23/d27/d2e/f47 0 2024-04-22T15:18:51.293 INFO:tasks.workunit.client.1.smithi196.stdout:3/164: creat d0/d7/f38 x:0 0 0 2024-04-22T15:18:51.295 INFO:tasks.workunit.client.0.smithi096.stdout:1/679: creat d6/dc/d8a/d3f/d62/fd1 x:0 0 0 2024-04-22T15:18:51.296 INFO:tasks.workunit.client.1.smithi196.stdout:2/202: dwrite d1/d1d/f3c [0,4194304] 0 2024-04-22T15:18:51.297 INFO:tasks.workunit.client.0.smithi096.stdout:4/577: getdents d5/d7/d2c/d67 0 2024-04-22T15:18:51.298 INFO:tasks.workunit.client.1.smithi196.stdout:5/155: rmdir d1/d17 39 2024-04-22T15:18:51.299 INFO:tasks.workunit.client.1.smithi196.stdout:1/147: sync 2024-04-22T15:18:51.299 INFO:tasks.workunit.client.1.smithi196.stdout:6/192: sync 2024-04-22T15:18:51.299 INFO:tasks.workunit.client.1.smithi196.stdout:7/175: sync 2024-04-22T15:18:51.300 INFO:tasks.workunit.client.1.smithi196.stdout:6/193: read dd/d20/f28 [309947,12324] 0 2024-04-22T15:18:51.300 INFO:tasks.workunit.client.1.smithi196.stdout:8/186: truncate d2/dc/fd 1404604 0 2024-04-22T15:18:51.301 INFO:tasks.workunit.client.1.smithi196.stdout:7/176: truncate d14/d26/f30 4827578 0 2024-04-22T15:18:51.301 INFO:tasks.workunit.client.1.smithi196.stdout:0/172: mkdir d3/d42 0 2024-04-22T15:18:51.303 INFO:tasks.workunit.client.0.smithi096.stdout:1/680: symlink d6/dc/d8a/d3f/d62/d7d/ld2 0 2024-04-22T15:18:51.306 INFO:tasks.workunit.client.1.smithi196.stdout:5/156: chown d1/d17/l2d 340 1 2024-04-22T15:18:51.309 INFO:tasks.workunit.client.0.smithi096.stdout:8/494: getdents d4/d10/d25/d38/d8f 0 2024-04-22T15:18:51.313 INFO:tasks.workunit.client.1.smithi196.stdout:1/148: mknod d1/db/dc/d26/d2e/c37 0 2024-04-22T15:18:51.313 INFO:tasks.workunit.client.1.smithi196.stdout:4/179: write d6/d10/fe [259316,70388] 0 2024-04-22T15:18:51.314 INFO:tasks.workunit.client.1.smithi196.stdout:6/194: mkdir dd/d1f/d38 0 2024-04-22T15:18:51.316 INFO:tasks.workunit.client.1.smithi196.stdout:9/179: sync 2024-04-22T15:18:51.316 INFO:tasks.workunit.client.1.smithi196.stdout:8/187: unlink d2/d5/f34 0 2024-04-22T15:18:51.316 INFO:tasks.workunit.client.1.smithi196.stdout:9/180: fsync db/d18/f23 0 2024-04-22T15:18:51.317 INFO:tasks.workunit.client.0.smithi096.stdout:8/495: dwrite d4/d1c/d40/d48/fa0 [0,4194304] 0 2024-04-22T15:18:51.317 INFO:tasks.workunit.client.1.smithi196.stdout:8/188: chown d2/dc/f20 67732465 1 2024-04-22T15:18:51.317 INFO:tasks.workunit.client.1.smithi196.stdout:9/181: fsync db/dd/de/d15/d31/f32 0 2024-04-22T15:18:51.318 INFO:tasks.workunit.client.1.smithi196.stdout:0/173: fsync d3/de/d1c/d27/f32 0 2024-04-22T15:18:51.322 INFO:tasks.workunit.client.0.smithi096.stdout:8/496: mkdir d4/d10/d25/d38/d8f/d9c/daf 0 2024-04-22T15:18:51.322 INFO:tasks.workunit.client.0.smithi096.stdout:8/497: dread - d4/d1c/d40/d48/fab zero size 2024-04-22T15:18:51.323 INFO:tasks.workunit.client.1.smithi196.stdout:5/157: symlink d1/d14/l34 0 2024-04-22T15:18:51.325 INFO:tasks.workunit.client.1.smithi196.stdout:4/180: creat d6/d14/f33 x:0 0 0 2024-04-22T15:18:51.328 INFO:tasks.workunit.client.0.smithi096.stdout:8/498: dwrite d4/d10/f4e [0,4194304] 0 2024-04-22T15:18:51.330 INFO:tasks.workunit.client.1.smithi196.stdout:4/181: dread d6/d13/d15/f30 [0,4194304] 0 2024-04-22T15:18:51.331 INFO:tasks.workunit.client.1.smithi196.stdout:4/182: chown d6/d14/d18/f28 1379234 1 2024-04-22T15:18:51.332 INFO:tasks.workunit.client.1.smithi196.stdout:1/149: dwrite d1/db/d16/d1e/f22 [0,4194304] 0 2024-04-22T15:18:51.332 INFO:tasks.workunit.client.1.smithi196.stdout:9/182: mknod db/dd/d1b/d2c/c41 0 2024-04-22T15:18:51.335 INFO:tasks.workunit.client.1.smithi196.stdout:0/174: write d3/de/f24 [2522861,39576] 0 2024-04-22T15:18:51.340 INFO:tasks.workunit.client.1.smithi196.stdout:9/183: dwrite db/d18/f3d [0,4194304] 0 2024-04-22T15:18:51.356 INFO:tasks.workunit.client.0.smithi096.stdout:8/499: rmdir d4/d10/d25/d38/d8f/d9c/daf 0 2024-04-22T15:18:51.360 INFO:tasks.workunit.client.0.smithi096.stdout:8/500: rename d4/d10/d25/d4c/c5a to d4/d10/d4d/cb0 0 2024-04-22T15:18:51.360 INFO:tasks.workunit.client.0.smithi096.stdout:9/503: rmdir d1/d4/d9/d43 39 2024-04-22T15:18:51.362 INFO:tasks.workunit.client.1.smithi196.stdout:1/150: mknod d1/db/d2d/c38 0 2024-04-22T15:18:51.363 INFO:tasks.workunit.client.0.smithi096.stdout:8/501: creat d4/d10/d25/d4c/d76/fb1 x:0 0 0 2024-04-22T15:18:51.364 INFO:tasks.workunit.client.1.smithi196.stdout:0/175: fdatasync d3/d6/dc/f15 0 2024-04-22T15:18:51.365 INFO:tasks.workunit.client.0.smithi096.stdout:9/504: dread - d1/d4/d9/dd/f92 zero size 2024-04-22T15:18:51.366 INFO:tasks.workunit.client.1.smithi196.stdout:1/151: dread d1/db/d16/d1e/f22 [0,4194304] 0 2024-04-22T15:18:51.366 INFO:tasks.workunit.client.0.smithi096.stdout:9/505: chown d1/d85/fa2 0 1 2024-04-22T15:18:51.367 INFO:tasks.workunit.client.0.smithi096.stdout:8/502: chown d4/l29 1091562655 1 2024-04-22T15:18:51.370 INFO:tasks.workunit.client.1.smithi196.stdout:9/184: rename db/d11/f25 to db/dd/de/f42 0 2024-04-22T15:18:51.370 INFO:tasks.workunit.client.1.smithi196.stdout:1/152: dwrite d1/db/dc/d26/f2b [0,4194304] 0 2024-04-22T15:18:51.372 INFO:tasks.workunit.client.1.smithi196.stdout:1/153: write d1/f4 [4674709,115787] 0 2024-04-22T15:18:51.372 INFO:tasks.workunit.client.0.smithi096.stdout:8/503: symlink d4/d1c/d40/d48/lb2 0 2024-04-22T15:18:51.373 INFO:tasks.workunit.client.0.smithi096.stdout:8/504: fdatasync d4/d2a/d7e/f9a 0 2024-04-22T15:18:51.381 INFO:tasks.workunit.client.0.smithi096.stdout:8/505: mknod d4/d1c/d40/cb3 0 2024-04-22T15:18:51.383 INFO:tasks.workunit.client.1.smithi196.stdout:2/203: write d1/d1d/f29 [928148,37279] 0 2024-04-22T15:18:51.385 INFO:tasks.workunit.client.1.smithi196.stdout:0/176: mknod d3/de/d1c/d27/c43 0 2024-04-22T15:18:51.388 INFO:tasks.workunit.client.0.smithi096.stdout:1/681: dwrite d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fa1 [0,4194304] 0 2024-04-22T15:18:51.393 INFO:tasks.workunit.client.1.smithi196.stdout:5/158: link d1/d17/d1a/f1d d1/f35 0 2024-04-22T15:18:51.395 INFO:tasks.workunit.client.1.smithi196.stdout:9/185: rename db/f28 to db/dd/d1b/d2c/f43 0 2024-04-22T15:18:51.396 INFO:tasks.workunit.client.1.smithi196.stdout:8/189: getdents d2/d8 0 2024-04-22T15:18:51.396 INFO:tasks.workunit.client.1.smithi196.stdout:4/183: creat d6/f34 x:0 0 0 2024-04-22T15:18:51.398 INFO:tasks.workunit.client.1.smithi196.stdout:5/159: creat d1/d17/f36 x:0 0 0 2024-04-22T15:18:51.398 INFO:tasks.workunit.client.1.smithi196.stdout:5/160: chown d1/d9/la 86022 1 2024-04-22T15:18:51.403 INFO:tasks.workunit.client.1.smithi196.stdout:2/204: link d1/d13/d25/c3b d1/d23/d27/d2e/d45/c48 0 2024-04-22T15:18:51.404 INFO:tasks.workunit.client.1.smithi196.stdout:4/184: mkdir d6/d13/d15/d2c/d35 0 2024-04-22T15:18:51.405 INFO:tasks.workunit.client.1.smithi196.stdout:1/154: getdents d1 0 2024-04-22T15:18:51.406 INFO:tasks.workunit.client.1.smithi196.stdout:0/177: rename d3/de/d1c/d27/c43 to d3/de/c44 0 2024-04-22T15:18:51.407 INFO:tasks.workunit.client.1.smithi196.stdout:8/190: dwrite d2/f1d [0,4194304] 0 2024-04-22T15:18:51.409 INFO:tasks.workunit.client.1.smithi196.stdout:4/185: creat d6/d14/d18/f36 x:0 0 0 2024-04-22T15:18:51.411 INFO:tasks.workunit.client.1.smithi196.stdout:0/178: creat d3/d9/d25/f45 x:0 0 0 2024-04-22T15:18:51.411 INFO:tasks.workunit.client.1.smithi196.stdout:2/205: dwrite d1/d1d/f22 [0,4194304] 0 2024-04-22T15:18:51.413 INFO:tasks.workunit.client.1.smithi196.stdout:8/191: dread d2/f1d [0,4194304] 0 2024-04-22T15:18:51.414 INFO:tasks.workunit.client.1.smithi196.stdout:2/206: symlink d1/d13/l49 0 2024-04-22T15:18:51.415 INFO:tasks.workunit.client.1.smithi196.stdout:8/192: write d2/d8/f25 [688262,48136] 0 2024-04-22T15:18:51.417 INFO:tasks.workunit.client.1.smithi196.stdout:0/179: rename d3/d9/d25/f45 to d3/de/d1c/d27/d2c/d30/f46 0 2024-04-22T15:18:51.422 INFO:tasks.workunit.client.1.smithi196.stdout:0/180: creat d3/f47 x:0 0 0 2024-04-22T15:18:51.423 INFO:tasks.workunit.client.1.smithi196.stdout:0/181: stat d3/de/d14/d29 0 2024-04-22T15:18:51.423 INFO:tasks.workunit.client.1.smithi196.stdout:2/207: dwrite d1/d1d/f22 [0,4194304] 0 2024-04-22T15:18:51.423 INFO:tasks.workunit.client.1.smithi196.stdout:0/182: chown d3/de 520 1 2024-04-22T15:18:51.427 INFO:tasks.workunit.client.1.smithi196.stdout:0/183: creat d3/d6/f48 x:0 0 0 2024-04-22T15:18:51.427 INFO:tasks.workunit.client.1.smithi196.stdout:2/208: symlink d1/d13/d25/d2a/d35/l4a 0 2024-04-22T15:18:51.428 INFO:tasks.workunit.client.1.smithi196.stdout:0/184: chown d3/d6/l34 660 1 2024-04-22T15:18:51.455 INFO:tasks.workunit.client.1.smithi196.stdout:7/177: truncate d14/d15/f18 3373805 0 2024-04-22T15:18:51.460 INFO:tasks.workunit.client.1.smithi196.stdout:3/165: write d0/f3 [315024,51500] 0 2024-04-22T15:18:51.464 INFO:tasks.workunit.client.0.smithi096.stdout:9/506: truncate d1/d4/d9/d43/d57/d5b/f66 2350202 0 2024-04-22T15:18:51.464 INFO:tasks.workunit.client.0.smithi096.stdout:9/507: stat d1/d4/c1c 0 2024-04-22T15:18:51.470 INFO:tasks.workunit.client.0.smithi096.stdout:9/508: getdents d1/d4/d9 0 2024-04-22T15:18:51.471 INFO:tasks.workunit.client.0.smithi096.stdout:8/506: truncate d4/d10/d4d/f74 3501992 0 2024-04-22T15:18:51.473 INFO:tasks.workunit.client.0.smithi096.stdout:8/507: mkdir d4/d10/d25/d4c/db4 0 2024-04-22T15:18:51.475 INFO:tasks.workunit.client.0.smithi096.stdout:8/508: creat d4/d1c/d40/d46/d58/fb5 x:0 0 0 2024-04-22T15:18:51.476 INFO:tasks.workunit.client.1.smithi196.stdout:9/186: rmdir db/dd 39 2024-04-22T15:18:51.477 INFO:tasks.workunit.client.0.smithi096.stdout:8/509: write d4/d1c/d40/d48/fab [629908,88086] 0 2024-04-22T15:18:51.478 INFO:tasks.workunit.client.0.smithi096.stdout:8/510: readlink d4/d10/d25/d4c/l6a 0 2024-04-22T15:18:51.481 INFO:tasks.workunit.client.0.smithi096.stdout:1/682: dwrite d6/dc/d8a/d3f/f57 [0,4194304] 0 2024-04-22T15:18:51.484 INFO:tasks.workunit.client.0.smithi096.stdout:8/511: rename d4/d1c/d34 to d4/d1c/d40/da7/db6 0 2024-04-22T15:18:51.486 INFO:tasks.workunit.client.0.smithi096.stdout:1/683: mkdir d6/dc/d8a/d3f/d62/d7d/dd3 0 2024-04-22T15:18:51.486 INFO:tasks.workunit.client.1.smithi196.stdout:5/161: write d1/d2/f29 [1208478,32195] 0 2024-04-22T15:18:51.487 INFO:tasks.workunit.client.0.smithi096.stdout:8/512: mkdir d4/d2a/db7 0 2024-04-22T15:18:51.487 INFO:tasks.workunit.client.1.smithi196.stdout:4/186: rmdir d6 39 2024-04-22T15:18:51.489 INFO:tasks.workunit.client.1.smithi196.stdout:5/162: stat d1/d2/f28 0 2024-04-22T15:18:51.490 INFO:tasks.workunit.client.0.smithi096.stdout:6/483: sync 2024-04-22T15:18:51.490 INFO:tasks.workunit.client.0.smithi096.stdout:1/684: read d6/dc/d8a/d3f/d54/f18 [1953825,90958] 0 2024-04-22T15:18:51.499 INFO:tasks.workunit.client.0.smithi096.stdout:1/685: stat d6/dc/d8a/d3f/d54/d32/d8c/c33 0 2024-04-22T15:18:51.500 INFO:tasks.workunit.client.1.smithi196.stdout:0/185: fsync d3/de/d1c/d27/d2c/d30/f46 0 2024-04-22T15:18:51.501 INFO:tasks.workunit.client.1.smithi196.stdout:4/187: readlink d6/d10/ld 0 2024-04-22T15:18:51.502 INFO:tasks.workunit.client.1.smithi196.stdout:6/195: sync 2024-04-22T15:18:51.503 INFO:tasks.workunit.client.0.smithi096.stdout:6/484: mknod d3/d3a/d75/d83/ca7 0 2024-04-22T15:18:51.504 INFO:tasks.workunit.client.1.smithi196.stdout:4/188: dread d6/d14/f12 [0,4194304] 0 2024-04-22T15:18:51.504 INFO:tasks.workunit.client.1.smithi196.stdout:5/163: rename d1/l2c to d1/d14/l37 0 2024-04-22T15:18:51.506 INFO:tasks.workunit.client.1.smithi196.stdout:0/186: creat d3/d9/f49 x:0 0 0 2024-04-22T15:18:51.507 INFO:tasks.workunit.client.0.smithi096.stdout:1/686: dwrite d6/dc/dbd/fc9 [0,4194304] 0 2024-04-22T15:18:51.507 INFO:tasks.workunit.client.1.smithi196.stdout:2/209: write d1/f7 [468745,102377] 0 2024-04-22T15:18:51.510 INFO:tasks.workunit.client.1.smithi196.stdout:4/189: creat d6/d10/f37 x:0 0 0 2024-04-22T15:18:51.513 INFO:tasks.workunit.client.1.smithi196.stdout:7/178: write fc [139584,22736] 0 2024-04-22T15:18:51.514 INFO:tasks.workunit.client.1.smithi196.stdout:7/179: chown d14/d15/f1a 16232 1 2024-04-22T15:18:51.515 INFO:tasks.workunit.client.0.smithi096.stdout:6/485: mknod d3/d28/d54/d68/ca8 0 2024-04-22T15:18:51.515 INFO:tasks.workunit.client.1.smithi196.stdout:7/180: fsync d14/d15/f1a 0 2024-04-22T15:18:51.516 INFO:tasks.workunit.client.1.smithi196.stdout:7/181: chown d14/d25 1673435030 1 2024-04-22T15:18:51.516 INFO:tasks.workunit.client.1.smithi196.stdout:5/164: creat d1/d9/f38 x:0 0 0 2024-04-22T15:18:51.517 INFO:tasks.workunit.client.0.smithi096.stdout:1/687: dwrite d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b/dba/fc8 [0,4194304] 0 2024-04-22T15:18:51.518 INFO:tasks.workunit.client.0.smithi096.stdout:6/486: read - d3/d28/d4d/d82/d9e/f92 zero size 2024-04-22T15:18:51.519 INFO:tasks.workunit.client.1.smithi196.stdout:3/166: dwrite d0/f4 [0,4194304] 0 2024-04-22T15:18:51.523 INFO:tasks.workunit.client.1.smithi196.stdout:2/210: symlink d1/d23/d27/l4b 0 2024-04-22T15:18:51.523 INFO:tasks.workunit.client.1.smithi196.stdout:4/190: symlink d6/d13/d15/l38 0 2024-04-22T15:18:51.524 INFO:tasks.workunit.client.1.smithi196.stdout:2/211: read d1/f3 [1025521,6853] 0 2024-04-22T15:18:51.524 INFO:tasks.workunit.client.0.smithi096.stdout:1/688: rmdir d6/dc/d8a/d3f/d54/d32 39 2024-04-22T15:18:51.528 INFO:tasks.workunit.client.0.smithi096.stdout:6/487: dwrite d3/d28/dd/d11/d2c/d3b/f79 [0,4194304] 0 2024-04-22T15:18:51.528 INFO:tasks.workunit.client.1.smithi196.stdout:6/196: rename dd/d20/c2d to dd/d1f/d37/c39 0 2024-04-22T15:18:51.531 INFO:tasks.workunit.client.1.smithi196.stdout:5/165: symlink d1/d4/l39 0 2024-04-22T15:18:51.539 INFO:tasks.workunit.client.1.smithi196.stdout:4/191: mknod d6/d14/d18/c39 0 2024-04-22T15:18:51.543 INFO:tasks.workunit.client.1.smithi196.stdout:7/182: rename d14/d15/l27 to d14/d26/d28/l36 0 2024-04-22T15:18:51.545 INFO:tasks.workunit.client.0.smithi096.stdout:8/513: dwrite d4/d1c/d40/d48/f68 [8388608,4194304] 0 2024-04-22T15:18:51.549 INFO:tasks.workunit.client.1.smithi196.stdout:7/183: dwrite d14/d26/f35 [0,4194304] 0 2024-04-22T15:18:51.551 INFO:tasks.workunit.client.1.smithi196.stdout:5/166: symlink d1/d9/l3a 0 2024-04-22T15:18:51.551 INFO:tasks.workunit.client.0.smithi096.stdout:1/689: getdents d6/dc/d8a 0 2024-04-22T15:18:51.552 INFO:tasks.workunit.client.1.smithi196.stdout:5/167: dread - d1/d17/d1a/f1d zero size 2024-04-22T15:18:51.553 INFO:tasks.workunit.client.0.smithi096.stdout:8/514: dwrite d4/d10/d25/f81 [4194304,4194304] 0 2024-04-22T15:18:51.554 INFO:tasks.workunit.client.1.smithi196.stdout:3/167: chown d0/d6/db/dc/d18/f1b 246 1 2024-04-22T15:18:51.557 INFO:tasks.workunit.client.1.smithi196.stdout:1/155: sync 2024-04-22T15:18:51.560 INFO:tasks.workunit.client.1.smithi196.stdout:2/212: mknod d1/d13/d41/c4c 0 2024-04-22T15:18:51.562 INFO:tasks.workunit.client.0.smithi096.stdout:8/515: dwrite d4/d1c/f72 [0,4194304] 0 2024-04-22T15:18:51.563 INFO:tasks.workunit.client.1.smithi196.stdout:1/156: dwrite d1/db/f29 [0,4194304] 0 2024-04-22T15:18:51.565 INFO:tasks.workunit.client.1.smithi196.stdout:4/192: rename d6/d14/d18/f28 to d6/d14/d18/d26/f3a 0 2024-04-22T15:18:51.566 INFO:tasks.workunit.client.1.smithi196.stdout:4/193: dread - d6/d14/d18/f36 zero size 2024-04-22T15:18:51.570 INFO:tasks.workunit.client.1.smithi196.stdout:6/197: getdents dd/d1f/d38 0 2024-04-22T15:18:51.573 INFO:tasks.workunit.client.1.smithi196.stdout:7/184: mknod d14/d15/c37 0 2024-04-22T15:18:51.575 INFO:tasks.workunit.client.0.smithi096.stdout:1/690: dwrite d6/dc/d8a/d3f/d54/d32/d82/fc0 [0,4194304] 0 2024-04-22T15:18:51.578 INFO:tasks.workunit.client.1.smithi196.stdout:3/168: symlink d0/d6/l39 0 2024-04-22T15:18:51.579 INFO:tasks.workunit.client.1.smithi196.stdout:6/198: dwrite dd/d2b/f2f [0,4194304] 0 2024-04-22T15:18:51.579 INFO:tasks.workunit.client.1.smithi196.stdout:2/213: mknod d1/d1d/c4d 0 2024-04-22T15:18:51.581 INFO:tasks.workunit.client.1.smithi196.stdout:6/199: write dd/f12 [1139520,64308] 0 2024-04-22T15:18:51.581 INFO:tasks.workunit.client.1.smithi196.stdout:7/185: dwrite f4 [4194304,4194304] 0 2024-04-22T15:18:51.582 INFO:tasks.workunit.client.1.smithi196.stdout:6/200: dread - dd/f1b zero size 2024-04-22T15:18:51.582 INFO:tasks.workunit.client.1.smithi196.stdout:7/186: stat c9 0 2024-04-22T15:18:51.586 INFO:tasks.workunit.client.1.smithi196.stdout:5/168: rename d1/d9/f2a to d1/d17/d1c/f3b 0 2024-04-22T15:18:51.588 INFO:tasks.workunit.client.1.smithi196.stdout:5/169: write d1/d9/f38 [378288,79088] 0 2024-04-22T15:18:51.588 INFO:tasks.workunit.client.1.smithi196.stdout:5/170: readlink d1/d17/d1a/l33 0 2024-04-22T15:18:51.590 INFO:tasks.workunit.client.1.smithi196.stdout:3/169: unlink d0/d6/db/c2a 0 2024-04-22T15:18:51.593 INFO:tasks.workunit.client.1.smithi196.stdout:7/187: rmdir d14/d26 39 2024-04-22T15:18:51.596 INFO:tasks.workunit.client.1.smithi196.stdout:3/170: dwrite d0/d6/db/d13/f16 [0,4194304] 0 2024-04-22T15:18:51.596 INFO:tasks.workunit.client.1.smithi196.stdout:3/171: chown d0 2 1 2024-04-22T15:18:51.599 INFO:tasks.workunit.client.1.smithi196.stdout:2/214: rename d1/le to d1/d23/d27/d2e/l4e 0 2024-04-22T15:18:51.600 INFO:tasks.workunit.client.1.smithi196.stdout:5/171: unlink d1/d4/l39 0 2024-04-22T15:18:51.603 INFO:tasks.workunit.client.0.smithi096.stdout:8/516: dwrite d4/d10/f19 [0,4194304] 0 2024-04-22T15:18:51.606 INFO:tasks.workunit.client.1.smithi196.stdout:3/172: unlink d0/f4 0 2024-04-22T15:18:51.607 INFO:tasks.workunit.client.1.smithi196.stdout:3/173: truncate d0/f1e 988587 0 2024-04-22T15:18:51.610 INFO:tasks.workunit.client.0.smithi096.stdout:8/517: readlink d4/d1c/l1e 0 2024-04-22T15:18:51.611 INFO:tasks.workunit.client.1.smithi196.stdout:2/215: creat d1/d13/d25/f4f x:0 0 0 2024-04-22T15:18:51.611 INFO:tasks.workunit.client.1.smithi196.stdout:5/172: mknod d1/d4/c3c 0 2024-04-22T15:18:51.612 INFO:tasks.workunit.client.1.smithi196.stdout:5/173: dread - d1/d17/f36 zero size 2024-04-22T15:18:51.613 INFO:tasks.workunit.client.1.smithi196.stdout:6/201: creat dd/f3a x:0 0 0 2024-04-22T15:18:51.614 INFO:tasks.workunit.client.1.smithi196.stdout:3/174: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:18:51.615 INFO:tasks.workunit.client.1.smithi196.stdout:3/175: write d0/d6/f2d [139362,2021] 0 2024-04-22T15:18:51.617 INFO:tasks.workunit.client.1.smithi196.stdout:4/194: truncate d6/d14/fb 7882250 0 2024-04-22T15:18:51.617 INFO:tasks.workunit.client.1.smithi196.stdout:3/176: dread d0/f1e [0,4194304] 0 2024-04-22T15:18:51.618 INFO:tasks.workunit.client.0.smithi096.stdout:8/518: dwrite d4/d2a/d7e/f9a [0,4194304] 0 2024-04-22T15:18:51.622 INFO:tasks.workunit.client.1.smithi196.stdout:4/195: dwrite d6/d13/f25 [4194304,4194304] 0 2024-04-22T15:18:51.623 INFO:tasks.workunit.client.1.smithi196.stdout:5/174: mknod d1/d17/d1c/c3d 0 2024-04-22T15:18:51.625 INFO:tasks.workunit.client.1.smithi196.stdout:5/175: write d1/d2/f29 [1164161,87352] 0 2024-04-22T15:18:51.625 INFO:tasks.workunit.client.1.smithi196.stdout:6/202: symlink dd/d1f/d37/l3b 0 2024-04-22T15:18:51.627 INFO:tasks.workunit.client.1.smithi196.stdout:6/203: chown fa 2958951 1 2024-04-22T15:18:51.629 INFO:tasks.workunit.client.1.smithi196.stdout:5/176: dread - d1/d17/f2e zero size 2024-04-22T15:18:51.629 INFO:tasks.workunit.client.1.smithi196.stdout:3/177: rmdir d0/d15 39 2024-04-22T15:18:51.630 INFO:tasks.workunit.client.1.smithi196.stdout:3/178: stat d0/d7/f38 0 2024-04-22T15:18:51.631 INFO:tasks.workunit.client.0.smithi096.stdout:8/519: getdents d4/d1c 0 2024-04-22T15:18:51.634 INFO:tasks.workunit.client.1.smithi196.stdout:2/216: creat d1/d13/f50 x:0 0 0 2024-04-22T15:18:51.635 INFO:tasks.workunit.client.1.smithi196.stdout:5/177: mkdir d1/d14/d3e 0 2024-04-22T15:18:51.636 INFO:tasks.workunit.client.1.smithi196.stdout:5/178: stat d1/d9/f20 0 2024-04-22T15:18:51.637 INFO:tasks.workunit.client.0.smithi096.stdout:8/520: mkdir d4/d1c/d40/da7/db6/db8 0 2024-04-22T15:18:51.637 INFO:tasks.workunit.client.1.smithi196.stdout:5/179: truncate d1/d17/f24 778945 0 2024-04-22T15:18:51.638 INFO:tasks.workunit.client.1.smithi196.stdout:5/180: stat d1/d9/f20 0 2024-04-22T15:18:51.638 INFO:tasks.workunit.client.1.smithi196.stdout:2/217: dread d1/f5 [0,4194304] 0 2024-04-22T15:18:51.639 INFO:tasks.workunit.client.1.smithi196.stdout:4/196: creat d6/d13/d15/d2c/d35/f3b x:0 0 0 2024-04-22T15:18:51.640 INFO:tasks.workunit.client.0.smithi096.stdout:8/521: truncate d4/d2a/f31 896982 0 2024-04-22T15:18:51.640 INFO:tasks.workunit.client.1.smithi196.stdout:4/197: write d6/d14/f27 [4990206,65488] 0 2024-04-22T15:18:51.641 INFO:tasks.workunit.client.1.smithi196.stdout:3/179: symlink d0/l3a 0 2024-04-22T15:18:51.644 INFO:tasks.workunit.client.1.smithi196.stdout:8/193: sync 2024-04-22T15:18:51.644 INFO:tasks.workunit.client.1.smithi196.stdout:9/187: sync 2024-04-22T15:18:51.644 INFO:tasks.workunit.client.1.smithi196.stdout:7/188: truncate d14/d26/f30 3452533 0 2024-04-22T15:18:51.648 INFO:tasks.workunit.client.1.smithi196.stdout:3/180: dwrite d0/d6/db/d13/f26 [0,4194304] 0 2024-04-22T15:18:51.648 INFO:tasks.workunit.client.1.smithi196.stdout:5/181: symlink d1/d14/d3e/l3f 0 2024-04-22T15:18:51.648 INFO:tasks.workunit.client.0.smithi096.stdout:8/522: link d4/d1c/l3d d4/d2a/lb9 0 2024-04-22T15:18:51.651 INFO:tasks.workunit.client.1.smithi196.stdout:9/188: write db/dd/de/d15/f36 [408863,84759] 0 2024-04-22T15:18:51.651 INFO:tasks.workunit.client.1.smithi196.stdout:3/181: dread d0/f1 [0,4194304] 0 2024-04-22T15:18:51.655 INFO:tasks.workunit.client.0.smithi096.stdout:8/523: dwrite d4/d10/d25/f37 [0,4194304] 0 2024-04-22T15:18:51.657 INFO:tasks.workunit.client.1.smithi196.stdout:3/182: dwrite d0/d6/db/dc/d18/f1b [0,4194304] 0 2024-04-22T15:18:51.660 INFO:tasks.workunit.client.1.smithi196.stdout:5/182: truncate d1/d9/f11 552981 0 2024-04-22T15:18:51.662 INFO:tasks.workunit.client.0.smithi096.stdout:8/524: creat d4/d1c/d62/fba x:0 0 0 2024-04-22T15:18:51.663 INFO:tasks.workunit.client.1.smithi196.stdout:8/194: fdatasync d2/fa 0 2024-04-22T15:18:51.665 INFO:tasks.workunit.client.1.smithi196.stdout:2/218: rename d1/d23/d27/d2e/l46 to d1/d13/l51 0 2024-04-22T15:18:51.696 INFO:tasks.workunit.client.0.smithi096.stdout:3/451: sync 2024-04-22T15:18:51.697 INFO:tasks.workunit.client.0.smithi096.stdout:8/525: symlink d4/d2a/lbb 0 2024-04-22T15:18:51.702 INFO:tasks.workunit.client.1.smithi196.stdout:5/183: mknod d1/d17/c40 0 2024-04-22T15:18:51.705 INFO:tasks.workunit.client.0.smithi096.stdout:3/452: rename da/db/de/d19/d3e/f8e to da/db/d10/f90 0 2024-04-22T15:18:51.708 INFO:tasks.workunit.client.1.smithi196.stdout:8/195: creat d2/dc/f3a x:0 0 0 2024-04-22T15:18:51.710 INFO:tasks.workunit.client.1.smithi196.stdout:3/183: rename d0/d6/l39 to d0/d7/l3b 0 2024-04-22T15:18:51.711 INFO:tasks.workunit.client.1.smithi196.stdout:8/196: dread d2/d5/f19 [0,4194304] 0 2024-04-22T15:18:51.712 INFO:tasks.workunit.client.1.smithi196.stdout:2/219: mknod d1/d23/d27/d2e/c52 0 2024-04-22T15:18:51.716 INFO:tasks.workunit.client.1.smithi196.stdout:8/197: dwrite d2/d5/f2d [0,4194304] 0 2024-04-22T15:18:51.717 INFO:tasks.workunit.client.1.smithi196.stdout:8/198: dread - d2/d35/f39 zero size 2024-04-22T15:18:51.718 INFO:tasks.workunit.client.0.smithi096.stdout:8/526: getdents d4 0 2024-04-22T15:18:51.719 INFO:tasks.workunit.client.1.smithi196.stdout:5/184: mknod d1/d17/c41 0 2024-04-22T15:18:51.723 INFO:tasks.workunit.client.1.smithi196.stdout:3/184: creat d0/d6/db/d13/f3c x:0 0 0 2024-04-22T15:18:51.725 INFO:tasks.workunit.client.0.smithi096.stdout:3/453: link da/db/d10/f6e da/db/dc/f91 0 2024-04-22T15:18:51.726 INFO:tasks.workunit.client.0.smithi096.stdout:3/454: truncate da/f55 581712 0 2024-04-22T15:18:51.726 INFO:tasks.workunit.client.1.smithi196.stdout:2/220: unlink d1/d13/d41/c4c 0 2024-04-22T15:18:51.727 INFO:tasks.workunit.client.0.smithi096.stdout:8/527: dwrite d4/d1c/d40/d46/d58/faa [0,4194304] 0 2024-04-22T15:18:51.731 INFO:tasks.workunit.client.1.smithi196.stdout:4/198: unlink d6/d14/fb 0 2024-04-22T15:18:51.734 INFO:tasks.workunit.client.0.smithi096.stdout:8/528: creat d4/d1c/fbc x:0 0 0 2024-04-22T15:18:51.735 INFO:tasks.workunit.client.1.smithi196.stdout:8/199: mknod d2/d5/d2e/c3b 0 2024-04-22T15:18:51.736 INFO:tasks.workunit.client.1.smithi196.stdout:2/221: dread d1/d13/d1f/f24 [0,4194304] 0 2024-04-22T15:18:51.738 INFO:tasks.workunit.client.0.smithi096.stdout:8/529: rmdir d4/d1c/d40/d46 39 2024-04-22T15:18:51.742 INFO:tasks.workunit.client.0.smithi096.stdout:8/530: symlink d4/d1c/lbd 0 2024-04-22T15:18:51.742 INFO:tasks.workunit.client.0.smithi096.stdout:8/531: readlink d4/d10/d4d/l70 0 2024-04-22T15:18:51.743 INFO:tasks.workunit.client.1.smithi196.stdout:5/185: dread d1/d4/f22 [0,4194304] 0 2024-04-22T15:18:51.744 INFO:tasks.workunit.client.1.smithi196.stdout:4/199: rmdir d6/d13/d15/d2c/d35 39 2024-04-22T15:18:51.747 INFO:tasks.workunit.client.1.smithi196.stdout:8/200: rename d2/d5/c6 to d2/d8/c3c 0 2024-04-22T15:18:51.751 INFO:tasks.workunit.client.1.smithi196.stdout:2/222: mknod d1/d1d/c53 0 2024-04-22T15:18:51.752 INFO:tasks.workunit.client.1.smithi196.stdout:0/187: sync 2024-04-22T15:18:51.753 INFO:tasks.workunit.client.1.smithi196.stdout:1/157: sync 2024-04-22T15:18:51.754 INFO:tasks.workunit.client.1.smithi196.stdout:5/186: rmdir d1/d17/d1a/d2f 39 2024-04-22T15:18:51.754 INFO:tasks.workunit.client.0.smithi096.stdout:3/455: dread da/db/de/d19/d3e/f4f [0,4194304] 0 2024-04-22T15:18:51.755 INFO:tasks.workunit.client.0.smithi096.stdout:3/456: dread - da/db/d12/f77 zero size 2024-04-22T15:18:51.758 INFO:tasks.workunit.client.1.smithi196.stdout:2/223: rmdir d1/d13/d25 39 2024-04-22T15:18:51.761 INFO:tasks.workunit.client.1.smithi196.stdout:0/188: rename d3/d6/dc/f15 to d3/de/d14/f4a 0 2024-04-22T15:18:51.762 INFO:tasks.workunit.client.1.smithi196.stdout:5/187: creat d1/d14/f42 x:0 0 0 2024-04-22T15:18:51.763 INFO:tasks.workunit.client.1.smithi196.stdout:3/185: getdents d0/d6/db/dc 0 2024-04-22T15:18:51.764 INFO:tasks.workunit.client.0.smithi096.stdout:8/532: truncate d4/d2a/d7e/f9a 2937271 0 2024-04-22T15:18:51.765 INFO:tasks.workunit.client.1.smithi196.stdout:1/158: dread d1/db/d16/f24 [4194304,4194304] 0 2024-04-22T15:18:51.765 INFO:tasks.workunit.client.0.smithi096.stdout:8/533: stat d4/d10/d25/d38/f79 0 2024-04-22T15:18:51.766 INFO:tasks.workunit.client.1.smithi196.stdout:1/159: write d1/db/dc/d26/f27 [56146,52788] 0 2024-04-22T15:18:51.767 INFO:tasks.workunit.client.1.smithi196.stdout:4/200: write f5 [8087361,118220] 0 2024-04-22T15:18:51.767 INFO:tasks.workunit.client.1.smithi196.stdout:5/188: read d1/d9/f11 [133339,86259] 0 2024-04-22T15:18:51.767 INFO:tasks.workunit.client.1.smithi196.stdout:2/224: getdents d1/d13/d1f 0 2024-04-22T15:18:51.769 INFO:tasks.workunit.client.1.smithi196.stdout:5/189: truncate d1/d17/f31 553464 0 2024-04-22T15:18:51.772 INFO:tasks.workunit.client.0.smithi096.stdout:8/534: dread d4/d1c/d40/d46/d58/faa [0,4194304] 0 2024-04-22T15:18:51.773 INFO:tasks.workunit.client.1.smithi196.stdout:4/201: unlink d6/d10/f1c 0 2024-04-22T15:18:51.773 INFO:tasks.workunit.client.1.smithi196.stdout:4/202: stat d6/c9 0 2024-04-22T15:18:51.778 INFO:tasks.workunit.client.0.smithi096.stdout:8/535: dwrite d4/d1c/d40/da7/db6/f3e [0,4194304] 0 2024-04-22T15:18:51.778 INFO:tasks.workunit.client.1.smithi196.stdout:5/190: fdatasync d1/d2/f25 0 2024-04-22T15:18:51.779 INFO:tasks.workunit.client.0.smithi096.stdout:8/536: read - d4/d1c/d62/fa5 zero size 2024-04-22T15:18:51.780 INFO:tasks.workunit.client.1.smithi196.stdout:3/186: truncate d0/d6/db/dc/f1c 829001 0 2024-04-22T15:18:51.781 INFO:tasks.workunit.client.1.smithi196.stdout:0/189: dwrite d3/de/d1c/f31 [0,4194304] 0 2024-04-22T15:18:51.782 INFO:tasks.workunit.client.1.smithi196.stdout:0/190: write d3/d6/f33 [3535883,19617] 0 2024-04-22T15:18:51.786 INFO:tasks.workunit.client.0.smithi096.stdout:8/537: creat d4/d2a/fbe x:0 0 0 2024-04-22T15:18:51.787 INFO:tasks.workunit.client.1.smithi196.stdout:2/225: mknod d1/d13/d25/d2a/c54 0 2024-04-22T15:18:51.788 INFO:tasks.workunit.client.1.smithi196.stdout:2/226: chown d1/d1d 3 1 2024-04-22T15:18:51.794 INFO:tasks.workunit.client.0.smithi096.stdout:8/538: dwrite d4/fd [8388608,4194304] 0 2024-04-22T15:18:51.796 INFO:tasks.workunit.client.1.smithi196.stdout:3/187: mknod d0/d7/c3d 0 2024-04-22T15:18:51.796 INFO:tasks.workunit.client.1.smithi196.stdout:3/188: write d0/d6/f2d [948517,94714] 0 2024-04-22T15:18:51.797 INFO:tasks.workunit.client.0.smithi096.stdout:8/539: write d4/d1c/d62/fa2 [447908,119179] 0 2024-04-22T15:18:51.799 INFO:tasks.workunit.client.1.smithi196.stdout:4/203: rename f5 to d6/d13/d15/d2c/d35/f3c 0 2024-04-22T15:18:51.800 INFO:tasks.workunit.client.1.smithi196.stdout:4/204: truncate d6/f19 1746193 0 2024-04-22T15:18:51.801 INFO:tasks.workunit.client.1.smithi196.stdout:0/191: symlink d3/l4b 0 2024-04-22T15:18:51.805 INFO:tasks.workunit.client.0.smithi096.stdout:8/540: dwrite d4/d10/d25/d38/f54 [0,4194304] 0 2024-04-22T15:18:51.820 INFO:tasks.workunit.client.1.smithi196.stdout:6/204: sync 2024-04-22T15:18:51.825 INFO:tasks.workunit.client.1.smithi196.stdout:6/205: dwrite dd/d2b/f2f [4194304,4194304] 0 2024-04-22T15:18:51.863 INFO:tasks.workunit.client.1.smithi196.stdout:9/189: sync 2024-04-22T15:18:51.863 INFO:tasks.workunit.client.1.smithi196.stdout:7/189: sync 2024-04-22T15:18:51.863 INFO:tasks.workunit.client.1.smithi196.stdout:8/201: sync 2024-04-22T15:18:51.863 INFO:tasks.workunit.client.1.smithi196.stdout:7/190: readlink l12 0 2024-04-22T15:18:51.867 INFO:tasks.workunit.client.1.smithi196.stdout:7/191: dread f13 [0,4194304] 0 2024-04-22T15:18:51.894 INFO:tasks.workunit.client.1.smithi196.stdout:1/160: truncate d1/db/dc/d26/f27 409821 0 2024-04-22T15:18:51.895 INFO:tasks.workunit.client.1.smithi196.stdout:3/189: dread d0/f1 [0,4194304] 0 2024-04-22T15:18:51.897 INFO:tasks.workunit.client.1.smithi196.stdout:2/227: rename d1/d13/d25/d2a/d35 to d1/d23/d27/d2e/d55 0 2024-04-22T15:18:51.898 INFO:tasks.workunit.client.0.smithi096.stdout:2/541: sync 2024-04-22T15:18:51.899 INFO:tasks.workunit.client.1.smithi196.stdout:4/205: dread d6/d13/f1b [0,4194304] 0 2024-04-22T15:18:51.900 INFO:tasks.workunit.client.1.smithi196.stdout:4/206: dread d6/d13/d15/f30 [0,4194304] 0 2024-04-22T15:18:51.906 INFO:tasks.workunit.client.0.smithi096.stdout:8/541: write d4/d1c/d40/d46/f7d [257953,62376] 0 2024-04-22T15:18:51.907 INFO:tasks.workunit.client.1.smithi196.stdout:8/202: dwrite d2/f38 [4194304,4194304] 0 2024-04-22T15:18:51.908 INFO:tasks.workunit.client.1.smithi196.stdout:8/203: write d2/dc/f20 [835272,15223] 0 2024-04-22T15:18:51.909 INFO:tasks.workunit.client.1.smithi196.stdout:5/191: getdents d1/d17/d1a 0 2024-04-22T15:18:51.910 INFO:tasks.workunit.client.1.smithi196.stdout:5/192: truncate d1/d14/f42 804214 0 2024-04-22T15:18:51.911 INFO:tasks.workunit.client.1.smithi196.stdout:5/193: stat d1/d9/l32 0 2024-04-22T15:18:51.914 INFO:tasks.workunit.client.1.smithi196.stdout:1/161: unlink d1/db/d16/d1e/f22 0 2024-04-22T15:18:51.915 INFO:tasks.workunit.client.1.smithi196.stdout:6/206: dwrite dd/f11 [4194304,4194304] 0 2024-04-22T15:18:51.916 INFO:tasks.workunit.client.0.smithi096.stdout:8/542: dwrite d4/d10/f97 [0,4194304] 0 2024-04-22T15:18:51.919 INFO:tasks.workunit.client.0.smithi096.stdout:8/543: read d4/d10/d4d/f84 [91645,50033] 0 2024-04-22T15:18:51.921 INFO:tasks.workunit.client.1.smithi196.stdout:3/190: creat d0/d6/f3e x:0 0 0 2024-04-22T15:18:51.922 INFO:tasks.workunit.client.0.smithi096.stdout:2/542: dwrite d3/f2e [0,4194304] 0 2024-04-22T15:18:51.925 INFO:tasks.workunit.client.1.smithi196.stdout:7/192: write f7 [679098,36895] 0 2024-04-22T15:18:51.927 INFO:tasks.workunit.client.1.smithi196.stdout:0/192: creat d3/d9/d36/f4c x:0 0 0 2024-04-22T15:18:51.928 INFO:tasks.workunit.client.1.smithi196.stdout:0/193: read - d3/de/d1c/d27/d2c/d30/f35 zero size 2024-04-22T15:18:51.928 INFO:tasks.workunit.client.0.smithi096.stdout:8/544: creat d4/d2a/fbf x:0 0 0 2024-04-22T15:18:51.929 INFO:tasks.workunit.client.1.smithi196.stdout:0/194: write d3/de/d1c/d27/f32 [458465,50945] 0 2024-04-22T15:18:51.930 INFO:tasks.workunit.client.0.smithi096.stdout:2/543: mknod d3/db/df/d24/d58/ca9 0 2024-04-22T15:18:51.930 INFO:tasks.workunit.client.0.smithi096.stdout:8/545: chown d4/d1c/d40/d46/d58/f60 16315870 1 2024-04-22T15:18:51.931 INFO:tasks.workunit.client.1.smithi196.stdout:4/207: write d6/d13/d15/d2c/d35/f3c [10675682,61185] 0 2024-04-22T15:18:51.933 INFO:tasks.workunit.client.0.smithi096.stdout:2/544: mkdir d3/db/d2d/daa 0 2024-04-22T15:18:51.935 INFO:tasks.workunit.client.1.smithi196.stdout:8/204: symlink d2/d5/d1a/l3d 0 2024-04-22T15:18:51.936 INFO:tasks.workunit.client.1.smithi196.stdout:5/194: truncate d1/d4/f26 802607 0 2024-04-22T15:18:51.937 INFO:tasks.workunit.client.1.smithi196.stdout:8/205: write d2/dc/f2f [378119,88653] 0 2024-04-22T15:18:51.939 INFO:tasks.workunit.client.1.smithi196.stdout:4/208: dwrite d6/f2b [0,4194304] 0 2024-04-22T15:18:51.941 INFO:tasks.workunit.client.1.smithi196.stdout:6/207: mkdir dd/d1f/d37/d3c 0 2024-04-22T15:18:51.943 INFO:tasks.workunit.client.1.smithi196.stdout:3/191: fdatasync d0/d7/f1a 0 2024-04-22T15:18:51.945 INFO:tasks.workunit.client.0.smithi096.stdout:8/546: getdents d4/d1c/d40/da7/db6 0 2024-04-22T15:18:51.945 INFO:tasks.workunit.client.1.smithi196.stdout:2/228: mknod d1/d23/d27/d2e/d45/c56 0 2024-04-22T15:18:51.951 INFO:tasks.workunit.client.1.smithi196.stdout:2/229: dwrite d1/f2b [0,4194304] 0 2024-04-22T15:18:51.954 INFO:tasks.workunit.client.1.smithi196.stdout:5/195: unlink d1/d2/f29 0 2024-04-22T15:18:51.954 INFO:tasks.workunit.client.1.smithi196.stdout:5/196: chown d1/d14 468257497 1 2024-04-22T15:18:51.955 INFO:tasks.workunit.client.1.smithi196.stdout:5/197: write d1/d14/f1e [1338671,96994] 0 2024-04-22T15:18:51.956 INFO:tasks.workunit.client.1.smithi196.stdout:5/198: readlink d1/d17/d1c/l27 0 2024-04-22T15:18:51.956 INFO:tasks.workunit.client.1.smithi196.stdout:4/209: creat d6/d13/d15/f3d x:0 0 0 2024-04-22T15:18:51.957 INFO:tasks.workunit.client.1.smithi196.stdout:4/210: write d6/d10/f37 [197640,58029] 0 2024-04-22T15:18:51.958 INFO:tasks.workunit.client.1.smithi196.stdout:6/208: mknod dd/d20/c3d 0 2024-04-22T15:18:51.959 INFO:tasks.workunit.client.1.smithi196.stdout:6/209: write fc [5018836,32441] 0 2024-04-22T15:18:51.961 INFO:tasks.workunit.client.1.smithi196.stdout:7/193: getdents d14/d26/d28/d29 0 2024-04-22T15:18:51.962 INFO:tasks.workunit.client.1.smithi196.stdout:7/194: chown d14/f33 490226345 1 2024-04-22T15:18:51.963 INFO:tasks.workunit.client.1.smithi196.stdout:9/190: getdents db/d11/d29 0 2024-04-22T15:18:51.964 INFO:tasks.workunit.client.1.smithi196.stdout:2/230: creat d1/d1d/f57 x:0 0 0 2024-04-22T15:18:51.966 INFO:tasks.workunit.client.1.smithi196.stdout:3/192: dread d0/d7/f11 [0,4194304] 0 2024-04-22T15:18:51.967 INFO:tasks.workunit.client.1.smithi196.stdout:5/199: rmdir d1/d17/d1a 39 2024-04-22T15:18:51.970 INFO:tasks.workunit.client.1.smithi196.stdout:2/231: dwrite d1/f7 [0,4194304] 0 2024-04-22T15:18:51.970 INFO:tasks.workunit.client.1.smithi196.stdout:6/210: creat dd/d2b/f3e x:0 0 0 2024-04-22T15:18:51.971 INFO:tasks.workunit.client.1.smithi196.stdout:6/211: readlink dd/d1f/l35 0 2024-04-22T15:18:51.972 INFO:tasks.workunit.client.1.smithi196.stdout:6/212: chown dd/d1f/d37/f34 527997 1 2024-04-22T15:18:51.972 INFO:tasks.workunit.client.0.smithi096.stdout:2/545: dwrite d3/d1f/d2f/f34 [0,4194304] 0 2024-04-22T15:18:51.973 INFO:tasks.workunit.client.1.smithi196.stdout:0/195: write d3/de/f18 [87434,72610] 0 2024-04-22T15:18:51.973 INFO:tasks.workunit.client.1.smithi196.stdout:0/196: chown d3/c4 65 1 2024-04-22T15:18:51.974 INFO:tasks.workunit.client.1.smithi196.stdout:0/197: write d3/f5 [1157917,129886] 0 2024-04-22T15:18:51.975 INFO:tasks.workunit.client.1.smithi196.stdout:0/198: fdatasync d3/de/f37 0 2024-04-22T15:18:51.977 INFO:tasks.workunit.client.1.smithi196.stdout:8/206: rename d2/d5/f2d to d2/f3e 0 2024-04-22T15:18:51.980 INFO:tasks.workunit.client.0.smithi096.stdout:8/547: dwrite d4/d1c/d62/f6b [0,4194304] 0 2024-04-22T15:18:51.981 INFO:tasks.workunit.client.1.smithi196.stdout:0/199: dread d3/d6/f33 [0,4194304] 0 2024-04-22T15:18:51.982 INFO:tasks.workunit.client.1.smithi196.stdout:0/200: truncate d3/d6/f48 730272 0 2024-04-22T15:18:51.983 INFO:tasks.workunit.client.0.smithi096.stdout:2/546: rename d3/db/d2d/d46/l94 to d3/db/d2d/d46/d7e/lab 0 2024-04-22T15:18:51.983 INFO:tasks.workunit.client.0.smithi096.stdout:2/547: fsync d3/db/df/d24/fa6 0 2024-04-22T15:18:51.984 INFO:tasks.workunit.client.1.smithi196.stdout:7/195: creat d14/d25/d31/f38 x:0 0 0 2024-04-22T15:18:51.989 INFO:tasks.workunit.client.1.smithi196.stdout:2/232: creat d1/d23/d27/d2e/f58 x:0 0 0 2024-04-22T15:18:51.992 INFO:tasks.workunit.client.0.smithi096.stdout:2/548: creat d3/db/df/d24/d58/d5f/d6e/fac x:0 0 0 2024-04-22T15:18:51.995 INFO:tasks.workunit.client.1.smithi196.stdout:2/233: dwrite d1/f2c [0,4194304] 0 2024-04-22T15:18:51.997 INFO:tasks.workunit.client.1.smithi196.stdout:2/234: truncate d1/f3e 523037 0 2024-04-22T15:18:51.999 INFO:tasks.workunit.client.0.smithi096.stdout:2/549: dwrite d3/db/d30/f3a [4194304,4194304] 0 2024-04-22T15:18:52.001 INFO:tasks.workunit.client.1.smithi196.stdout:3/193: dwrite d0/f1 [0,4194304] 0 2024-04-22T15:18:52.002 INFO:tasks.workunit.client.1.smithi196.stdout:9/191: rename db/d11/c19 to db/dd/de/c44 0 2024-04-22T15:18:52.006 INFO:tasks.workunit.client.0.smithi096.stdout:2/550: creat d3/db/d2d/fad x:0 0 0 2024-04-22T15:18:52.007 INFO:tasks.workunit.client.0.smithi096.stdout:0/726: sync 2024-04-22T15:18:52.007 INFO:tasks.workunit.client.1.smithi196.stdout:0/201: mknod d3/d9/d36/c4d 0 2024-04-22T15:18:52.008 INFO:tasks.workunit.client.0.smithi096.stdout:2/551: read - d3/db/d2d/d46/d69/fa1 zero size 2024-04-22T15:18:52.009 INFO:tasks.workunit.client.0.smithi096.stdout:8/548: dread d4/d1c/d40/da7/db6/f4f [0,4194304] 0 2024-04-22T15:18:52.011 INFO:tasks.workunit.client.1.smithi196.stdout:6/213: link f2 dd/d1f/d37/f3f 0 2024-04-22T15:18:52.015 INFO:tasks.workunit.client.0.smithi096.stdout:2/552: mknod d3/db/df/d24/d58/d5f/d6e/cae 0 2024-04-22T15:18:52.017 INFO:tasks.workunit.client.0.smithi096.stdout:8/549: dwrite d4/d1c/f43 [0,4194304] 0 2024-04-22T15:18:52.019 INFO:tasks.workunit.client.1.smithi196.stdout:8/207: mkdir d2/d3f 0 2024-04-22T15:18:52.019 INFO:tasks.workunit.client.1.smithi196.stdout:5/200: write d1/d17/d1a/f1d [697983,106773] 0 2024-04-22T15:18:52.020 INFO:tasks.workunit.client.0.smithi096.stdout:0/727: truncate d7/de/d5b/d95/d15/f1a 4191446 0 2024-04-22T15:18:52.022 INFO:tasks.workunit.client.0.smithi096.stdout:2/553: rmdir d3/d13 39 2024-04-22T15:18:52.023 INFO:tasks.workunit.client.1.smithi196.stdout:3/194: symlink d0/d6/l3f 0 2024-04-22T15:18:52.024 INFO:tasks.workunit.client.0.smithi096.stdout:8/550: mkdir d4/dc0 0 2024-04-22T15:18:52.025 INFO:tasks.workunit.client.1.smithi196.stdout:9/192: creat db/dd/d1b/f45 x:0 0 0 2024-04-22T15:18:52.027 INFO:tasks.workunit.client.0.smithi096.stdout:2/554: readlink d3/db/d30/l97 0 2024-04-22T15:18:52.030 INFO:tasks.workunit.client.0.smithi096.stdout:0/728: dwrite d7/de/d5b/d95/dc/d16/d65/ff3 [0,4194304] 0 2024-04-22T15:18:52.031 INFO:tasks.workunit.client.0.smithi096.stdout:2/555: creat d3/d1f/d5a/faf x:0 0 0 2024-04-22T15:18:52.031 INFO:tasks.workunit.client.0.smithi096.stdout:0/729: chown d7/de/d3a/lc3 840277415 1 2024-04-22T15:18:52.032 INFO:tasks.workunit.client.1.smithi196.stdout:5/201: rmdir d1/d14 39 2024-04-22T15:18:52.032 INFO:tasks.workunit.client.0.smithi096.stdout:2/556: write d3/db/df/fa5 [271904,22491] 0 2024-04-22T15:18:52.036 INFO:tasks.workunit.client.1.smithi196.stdout:7/196: rename d14/d26/l32 to d14/d26/d2a/d2b/l39 0 2024-04-22T15:18:52.036 INFO:tasks.workunit.client.1.smithi196.stdout:7/197: chown d14/d26/f35 57955 1 2024-04-22T15:18:52.037 INFO:tasks.workunit.client.0.smithi096.stdout:0/730: readlink d7/d42/d4c/d55/l89 0 2024-04-22T15:18:52.038 INFO:tasks.workunit.client.1.smithi196.stdout:3/195: symlink d0/d6/l40 0 2024-04-22T15:18:52.039 INFO:tasks.workunit.client.0.smithi096.stdout:0/731: mknod d7/de/d3a/d61/dbb/cf5 0 2024-04-22T15:18:52.040 INFO:tasks.workunit.client.1.smithi196.stdout:0/202: symlink d3/d42/l4e 0 2024-04-22T15:18:52.040 INFO:tasks.workunit.client.0.smithi096.stdout:0/732: chown d7/de/ld 441906 1 2024-04-22T15:18:52.042 INFO:tasks.workunit.client.1.smithi196.stdout:6/214: mkdir dd/d1f/d38/d40 0 2024-04-22T15:18:52.043 INFO:tasks.workunit.client.1.smithi196.stdout:5/202: creat d1/d17/d1a/f43 x:0 0 0 2024-04-22T15:18:52.043 INFO:tasks.workunit.client.0.smithi096.stdout:2/557: link d3/l5 d3/db/d2d/d46/d7e/d84/lb0 0 2024-04-22T15:18:52.044 INFO:tasks.workunit.client.1.smithi196.stdout:6/215: write dd/d20/f22 [4173074,67775] 0 2024-04-22T15:18:52.045 INFO:tasks.workunit.client.0.smithi096.stdout:0/733: write d7/de/d5b/d95/dc/f2f [4587028,99541] 0 2024-04-22T15:18:52.046 INFO:tasks.workunit.client.1.smithi196.stdout:7/198: dwrite d14/d26/d2a/d2b/f2d [0,4194304] 0 2024-04-22T15:18:52.046 INFO:tasks.workunit.client.1.smithi196.stdout:6/216: stat dd/f1c 0 2024-04-22T15:18:52.046 INFO:tasks.workunit.client.1.smithi196.stdout:7/199: stat d14/d25 0 2024-04-22T15:18:52.046 INFO:tasks.workunit.client.0.smithi096.stdout:0/734: stat d7/de/d5b/d95/dc/f2f 0 2024-04-22T15:18:52.049 INFO:tasks.workunit.client.1.smithi196.stdout:9/193: dread fa [0,4194304] 0 2024-04-22T15:18:52.050 INFO:tasks.workunit.client.1.smithi196.stdout:3/196: symlink d0/d6/db/dc/l41 0 2024-04-22T15:18:52.050 INFO:tasks.workunit.client.1.smithi196.stdout:9/194: readlink db/dd/de/l16 0 2024-04-22T15:18:52.050 INFO:tasks.workunit.client.1.smithi196.stdout:1/162: sync 2024-04-22T15:18:52.054 INFO:tasks.workunit.client.1.smithi196.stdout:3/197: dread - d0/d7/d2b/f35 zero size 2024-04-22T15:18:52.054 INFO:tasks.workunit.client.1.smithi196.stdout:1/163: chown d1/db/d16/d1e 163781 1 2024-04-22T15:18:52.054 INFO:tasks.workunit.client.1.smithi196.stdout:1/164: truncate d1/db/d16/f17 9332940 0 2024-04-22T15:18:52.054 INFO:tasks.workunit.client.1.smithi196.stdout:0/203: rename d3/d9/f49 to d3/d42/f4f 0 2024-04-22T15:18:52.058 INFO:tasks.workunit.client.1.smithi196.stdout:2/235: dwrite d1/f5 [0,4194304] 0 2024-04-22T15:18:52.058 INFO:tasks.workunit.client.0.smithi096.stdout:0/735: rename d7/de/d5b/d95/f17 to d7/de/d5b/d62/ff6 0 2024-04-22T15:18:52.059 INFO:tasks.workunit.client.1.smithi196.stdout:2/236: stat d1/d23 0 2024-04-22T15:18:52.065 INFO:tasks.workunit.client.0.smithi096.stdout:0/736: mkdir d7/d42/d4c/d55/d9c/df7 0 2024-04-22T15:18:52.066 INFO:tasks.workunit.client.1.smithi196.stdout:8/208: link d2/d5/d1a/l3d d2/dc/l40 0 2024-04-22T15:18:52.070 INFO:tasks.workunit.client.0.smithi096.stdout:0/737: creat d7/d42/ff8 x:0 0 0 2024-04-22T15:18:52.071 INFO:tasks.workunit.client.1.smithi196.stdout:4/211: sync 2024-04-22T15:18:52.073 INFO:tasks.workunit.client.1.smithi196.stdout:9/195: mknod db/dd/d1b/c46 0 2024-04-22T15:18:52.073 INFO:tasks.workunit.client.0.smithi096.stdout:7/811: sync 2024-04-22T15:18:52.074 INFO:tasks.workunit.client.0.smithi096.stdout:5/601: sync 2024-04-22T15:18:52.077 INFO:tasks.workunit.client.1.smithi196.stdout:3/198: dread d0/d6/db/dc/f1c [0,4194304] 0 2024-04-22T15:18:52.077 INFO:tasks.workunit.client.0.smithi096.stdout:0/738: dwrite d7/de/d5b/d95/d15/f36 [0,4194304] 0 2024-04-22T15:18:52.077 INFO:tasks.workunit.client.0.smithi096.stdout:8/551: dread d4/d1c/f3a [0,4194304] 0 2024-04-22T15:18:52.078 INFO:tasks.workunit.client.0.smithi096.stdout:0/739: write d7/de/d5b/d95/dc/d16/f3b [3030556,92165] 0 2024-04-22T15:18:52.079 INFO:tasks.workunit.client.0.smithi096.stdout:7/812: creat d31/d68/f10d x:0 0 0 2024-04-22T15:18:52.079 INFO:tasks.workunit.client.0.smithi096.stdout:2/558: write d3/db/df/d10/f3b [355840,19259] 0 2024-04-22T15:18:52.081 INFO:tasks.workunit.client.0.smithi096.stdout:0/740: write d7/de/d5b/fe2 [563408,80012] 0 2024-04-22T15:18:52.084 INFO:tasks.workunit.client.0.smithi096.stdout:5/602: rename d4/d2c/d2f/d6b/f50 to d4/d11/d3d/d4d/d59/d98/db9/fc4 0 2024-04-22T15:18:52.089 INFO:tasks.workunit.client.0.smithi096.stdout:2/559: readlink d3/d1f/d5a/l9a 0 2024-04-22T15:18:52.091 INFO:tasks.workunit.client.1.smithi196.stdout:2/237: creat d1/d13/d25/f59 x:0 0 0 2024-04-22T15:18:52.091 INFO:tasks.workunit.client.1.smithi196.stdout:2/238: fsync d1/d1d/f29 0 2024-04-22T15:18:52.093 INFO:tasks.workunit.client.1.smithi196.stdout:5/203: creat d1/d14/d3e/f44 x:0 0 0 2024-04-22T15:18:52.094 INFO:tasks.workunit.client.0.smithi096.stdout:8/552: dwrite d4/d10/d25/d4c/d76/fb1 [0,4194304] 0 2024-04-22T15:18:52.097 INFO:tasks.workunit.client.1.smithi196.stdout:6/217: dwrite dd/f1c [0,4194304] 0 2024-04-22T15:18:52.098 INFO:tasks.workunit.client.0.smithi096.stdout:0/741: fdatasync d7/de/d5b/d62/d67/f70 0 2024-04-22T15:18:52.100 INFO:tasks.workunit.client.0.smithi096.stdout:0/742: chown d7/de/d5b/d95/dc/d16/d65/cf2 1 1 2024-04-22T15:18:52.102 INFO:tasks.workunit.client.1.smithi196.stdout:1/165: write d1/db/ff [1895840,2180] 0 2024-04-22T15:18:52.103 INFO:tasks.workunit.client.1.smithi196.stdout:4/212: symlink d6/d13/d15/d2c/d35/l3e 0 2024-04-22T15:18:52.108 INFO:tasks.workunit.client.0.smithi096.stdout:8/553: mknod d4/d1c/d40/da7/cc1 0 2024-04-22T15:18:52.110 INFO:tasks.workunit.client.1.smithi196.stdout:9/196: creat db/dd/de/d15/d31/f47 x:0 0 0 2024-04-22T15:18:52.111 INFO:tasks.workunit.client.1.smithi196.stdout:3/199: fsync d0/f1e 0 2024-04-22T15:18:52.113 INFO:tasks.workunit.client.0.smithi096.stdout:0/743: symlink d7/de/d5b/d95/d66/lf9 0 2024-04-22T15:18:52.113 INFO:tasks.workunit.client.1.smithi196.stdout:3/200: truncate d0/d6/db/dc/d18/f24 766044 0 2024-04-22T15:18:52.113 INFO:tasks.workunit.client.0.smithi096.stdout:0/744: fsync d7/de/d5b/d95/dc/f56 0 2024-04-22T15:18:52.123 INFO:tasks.workunit.client.0.smithi096.stdout:7/813: getdents d31/d32/d33/d6a/d83 0 2024-04-22T15:18:52.124 INFO:tasks.workunit.client.1.smithi196.stdout:8/209: creat d2/d3f/f41 x:0 0 0 2024-04-22T15:18:52.126 INFO:tasks.workunit.client.0.smithi096.stdout:0/745: mkdir d7/de/d3a/d61/d79/dfa 0 2024-04-22T15:18:52.127 INFO:tasks.workunit.client.0.smithi096.stdout:5/603: dread d4/d11/d25/f38 [0,4194304] 0 2024-04-22T15:18:52.127 INFO:tasks.workunit.client.1.smithi196.stdout:5/204: unlink d1/d9/l32 0 2024-04-22T15:18:52.128 INFO:tasks.workunit.client.0.smithi096.stdout:5/604: chown d4/d2c/d2f/d31/f8f 44703595 1 2024-04-22T15:18:52.130 INFO:tasks.workunit.client.0.smithi096.stdout:7/814: dwrite d31/d32/d33/d44/f80 [4194304,4194304] 0 2024-04-22T15:18:52.132 INFO:tasks.workunit.client.1.smithi196.stdout:6/218: fsync f1 0 2024-04-22T15:18:52.134 INFO:tasks.workunit.client.1.smithi196.stdout:7/200: creat d14/f3a x:0 0 0 2024-04-22T15:18:52.135 INFO:tasks.workunit.client.1.smithi196.stdout:1/166: mknod d1/db/d2d/c39 0 2024-04-22T15:18:52.136 INFO:tasks.workunit.client.1.smithi196.stdout:7/201: read f2 [3085579,56380] 0 2024-04-22T15:18:52.137 INFO:tasks.workunit.client.0.smithi096.stdout:5/605: mkdir d4/d11/dc5 0 2024-04-22T15:18:52.138 INFO:tasks.workunit.client.0.smithi096.stdout:7/815: mknod d31/d32/d33/c10e 0 2024-04-22T15:18:52.139 INFO:tasks.workunit.client.1.smithi196.stdout:7/202: truncate d14/d15/f1a 1109787 0 2024-04-22T15:18:52.139 INFO:tasks.workunit.client.0.smithi096.stdout:8/554: write d4/f12 [3676326,96855] 0 2024-04-22T15:18:52.140 INFO:tasks.workunit.client.1.smithi196.stdout:1/167: dread d1/db/dc/d26/f30 [0,4194304] 0 2024-04-22T15:18:52.142 INFO:tasks.workunit.client.1.smithi196.stdout:3/201: creat d0/d6/db/dc/d18/f42 x:0 0 0 2024-04-22T15:18:52.142 INFO:tasks.workunit.client.1.smithi196.stdout:2/239: dwrite d1/d23/f40 [0,4194304] 0 2024-04-22T15:18:52.143 INFO:tasks.workunit.client.1.smithi196.stdout:1/168: rename d1/db to d1/db/dc/d26/d2e/d32/d36/d3a 22 2024-04-22T15:18:52.143 INFO:tasks.workunit.client.0.smithi096.stdout:7/816: stat d31/d32/d33/f3e 0 2024-04-22T15:18:52.146 INFO:tasks.workunit.client.0.smithi096.stdout:8/555: mknod d4/d1c/d62/cc2 0 2024-04-22T15:18:52.147 INFO:tasks.workunit.client.0.smithi096.stdout:8/556: stat d4/d10/d4d/l55 0 2024-04-22T15:18:52.151 INFO:tasks.workunit.client.1.smithi196.stdout:8/210: dread d2/d5/f19 [0,4194304] 0 2024-04-22T15:18:52.151 INFO:tasks.workunit.client.1.smithi196.stdout:8/211: stat d2/dc/f3a 0 2024-04-22T15:18:52.152 INFO:tasks.workunit.client.1.smithi196.stdout:8/212: write d2/f4 [5127601,49798] 0 2024-04-22T15:18:52.153 INFO:tasks.workunit.client.1.smithi196.stdout:5/205: unlink d1/d17/f36 0 2024-04-22T15:18:52.154 INFO:tasks.workunit.client.0.smithi096.stdout:5/606: dwrite d4/d1f/f60 [0,4194304] 0 2024-04-22T15:18:52.156 INFO:tasks.workunit.client.0.smithi096.stdout:8/557: mkdir d4/d1c/d62/dc3 0 2024-04-22T15:18:52.156 INFO:tasks.workunit.client.0.smithi096.stdout:7/817: dread d31/d32/d33/fb5 [0,4194304] 0 2024-04-22T15:18:52.156 INFO:tasks.workunit.client.0.smithi096.stdout:7/818: chown d31/d68/f10d 88 1 2024-04-22T15:18:52.158 INFO:tasks.workunit.client.1.smithi196.stdout:8/213: dwrite d2/d5/f23 [0,4194304] 0 2024-04-22T15:18:52.161 INFO:tasks.workunit.client.1.smithi196.stdout:3/202: mknod d0/d7/c43 0 2024-04-22T15:18:52.164 INFO:tasks.workunit.client.1.smithi196.stdout:0/204: getdents d3/de 0 2024-04-22T15:18:52.166 INFO:tasks.workunit.client.1.smithi196.stdout:0/205: write d3/de/d1c/d27/f32 [1473256,31198] 0 2024-04-22T15:18:52.169 INFO:tasks.workunit.client.1.smithi196.stdout:6/219: mknod dd/c41 0 2024-04-22T15:18:52.169 INFO:tasks.workunit.client.1.smithi196.stdout:3/203: dwrite d0/d6/db/dc/d18/f42 [0,4194304] 0 2024-04-22T15:18:52.170 INFO:tasks.workunit.client.1.smithi196.stdout:6/220: write dd/f3a [627663,37945] 0 2024-04-22T15:18:52.174 INFO:tasks.workunit.client.1.smithi196.stdout:2/240: symlink d1/d13/l5a 0 2024-04-22T15:18:52.179 INFO:tasks.workunit.client.1.smithi196.stdout:0/206: chown d3/de/d14/d29/c3c 79156 1 2024-04-22T15:18:52.181 INFO:tasks.workunit.client.1.smithi196.stdout:3/204: rename d0/d6/db/d13 to d0/d7/d2b/d44 0 2024-04-22T15:18:52.182 INFO:tasks.workunit.client.1.smithi196.stdout:6/221: rmdir dd/d1f 39 2024-04-22T15:18:52.184 INFO:tasks.workunit.client.1.smithi196.stdout:8/214: unlink f0 0 2024-04-22T15:18:52.185 INFO:tasks.workunit.client.1.smithi196.stdout:8/215: readlink d2/d5/lb 0 2024-04-22T15:18:52.186 INFO:tasks.workunit.client.1.smithi196.stdout:1/169: dread d1/db/dc/de/f14 [0,4194304] 0 2024-04-22T15:18:52.186 INFO:tasks.workunit.client.1.smithi196.stdout:7/203: link f13 d14/d25/f3b 0 2024-04-22T15:18:52.187 INFO:tasks.workunit.client.1.smithi196.stdout:2/241: dread - d1/f10 zero size 2024-04-22T15:18:52.187 INFO:tasks.workunit.client.1.smithi196.stdout:2/242: chown d1/d23/d27/d2e/d55 657447006 1 2024-04-22T15:18:52.188 INFO:tasks.workunit.client.1.smithi196.stdout:5/206: symlink d1/d17/d1a/d2f/d30/l45 0 2024-04-22T15:18:52.190 INFO:tasks.workunit.client.1.smithi196.stdout:0/207: unlink d3/de/d1c/d27/d2c/d30/c3f 0 2024-04-22T15:18:52.191 INFO:tasks.workunit.client.1.smithi196.stdout:0/208: write d3/de/d14/f39 [202926,57945] 0 2024-04-22T15:18:52.191 INFO:tasks.workunit.client.1.smithi196.stdout:3/205: fsync d0/d7/f27 0 2024-04-22T15:18:52.192 INFO:tasks.workunit.client.1.smithi196.stdout:8/216: mknod d2/d8/c42 0 2024-04-22T15:18:52.193 INFO:tasks.workunit.client.1.smithi196.stdout:1/170: write d1/db/d16/f24 [5109690,48839] 0 2024-04-22T15:18:52.196 INFO:tasks.workunit.client.1.smithi196.stdout:0/209: symlink d3/d6/dc/l50 0 2024-04-22T15:18:52.197 INFO:tasks.workunit.client.1.smithi196.stdout:6/222: dread f1 [4194304,4194304] 0 2024-04-22T15:18:52.198 INFO:tasks.workunit.client.1.smithi196.stdout:8/217: creat d2/d5/d2e/f43 x:0 0 0 2024-04-22T15:18:52.200 INFO:tasks.workunit.client.1.smithi196.stdout:8/218: write d2/d3f/f41 [274036,67061] 0 2024-04-22T15:18:52.200 INFO:tasks.workunit.client.1.smithi196.stdout:1/171: symlink d1/db/dc/d26/l3b 0 2024-04-22T15:18:52.201 INFO:tasks.workunit.client.1.smithi196.stdout:8/219: dread - d2/dc/f28 zero size 2024-04-22T15:18:52.201 INFO:tasks.workunit.client.1.smithi196.stdout:2/243: getdents d1/d13/d1f 0 2024-04-22T15:18:52.202 INFO:tasks.workunit.client.1.smithi196.stdout:1/172: chown d1/db/dc/d26/d2e/d32 0 1 2024-04-22T15:18:52.202 INFO:tasks.workunit.client.0.smithi096.stdout:7/819: creat d31/d32/f10f x:0 0 0 2024-04-22T15:18:52.203 INFO:tasks.workunit.client.1.smithi196.stdout:3/206: dwrite d0/d7/d2b/f35 [0,4194304] 0 2024-04-22T15:18:52.203 INFO:tasks.workunit.client.1.smithi196.stdout:0/210: rmdir d3/d42 39 2024-04-22T15:18:52.203 INFO:tasks.workunit.client.1.smithi196.stdout:1/173: chown d1 25307493 1 2024-04-22T15:18:52.203 INFO:tasks.workunit.client.0.smithi096.stdout:8/558: link d4/d10/d25/d4c/f5e d4/d1c/fc4 0 2024-04-22T15:18:52.204 INFO:tasks.workunit.client.0.smithi096.stdout:5/607: getdents d4/d11/d3d/d4d/d59/d98 0 2024-04-22T15:18:52.205 INFO:tasks.workunit.client.1.smithi196.stdout:3/207: write d0/d7/d2b/d44/f16 [4381489,55386] 0 2024-04-22T15:18:52.206 INFO:tasks.workunit.client.0.smithi096.stdout:7/820: creat d31/d32/d33/d6a/d83/dba/dc7/f110 x:0 0 0 2024-04-22T15:18:52.207 INFO:tasks.workunit.client.1.smithi196.stdout:7/204: rename d14/d15/l23 to d14/l3c 0 2024-04-22T15:18:52.207 INFO:tasks.workunit.client.1.smithi196.stdout:6/223: chown dd/d1f/d37/l3b 396403 1 2024-04-22T15:18:52.207 INFO:tasks.workunit.client.0.smithi096.stdout:7/821: chown d31/d32/c102 1 1 2024-04-22T15:18:52.208 INFO:tasks.workunit.client.1.smithi196.stdout:6/224: chown dd/d2b/f2f 451527 1 2024-04-22T15:18:52.209 INFO:tasks.workunit.client.0.smithi096.stdout:8/559: symlink d4/d1c/d62/lc5 0 2024-04-22T15:18:52.210 INFO:tasks.workunit.client.1.smithi196.stdout:1/174: dread d1/db/dc/de/f14 [0,4194304] 0 2024-04-22T15:18:52.211 INFO:tasks.workunit.client.1.smithi196.stdout:9/197: sync 2024-04-22T15:18:52.211 INFO:tasks.workunit.client.1.smithi196.stdout:4/213: sync 2024-04-22T15:18:52.211 INFO:tasks.workunit.client.1.smithi196.stdout:8/220: symlink d2/d5/d1a/l44 0 2024-04-22T15:18:52.212 INFO:tasks.workunit.client.0.smithi096.stdout:7/822: dread - d31/d32/d33/d6a/d83/fc8 zero size 2024-04-22T15:18:52.212 INFO:tasks.workunit.client.1.smithi196.stdout:1/175: readlink d1/db/dc/d26/l3b 0 2024-04-22T15:18:52.212 INFO:tasks.workunit.client.1.smithi196.stdout:3/208: dread d0/d7/d2b/d44/f26 [0,4194304] 0 2024-04-22T15:18:52.212 INFO:tasks.workunit.client.1.smithi196.stdout:2/244: mkdir d1/d23/d27/d2e/d55/d5b 0 2024-04-22T15:18:52.213 INFO:tasks.workunit.client.0.smithi096.stdout:8/560: dread - d4/d1c/d40/f75 zero size 2024-04-22T15:18:52.214 INFO:tasks.workunit.client.1.smithi196.stdout:8/221: read d2/dc/f10 [106935,7745] 0 2024-04-22T15:18:52.218 INFO:tasks.workunit.client.1.smithi196.stdout:6/225: dwrite dd/f3a [0,4194304] 0 2024-04-22T15:18:52.218 INFO:tasks.workunit.client.0.smithi096.stdout:5/608: getdents d4/d11/d3d/d4d/d59/d98 0 2024-04-22T15:18:52.219 INFO:tasks.workunit.client.1.smithi196.stdout:5/207: link d1/d17/d1a/l33 d1/d9/l46 0 2024-04-22T15:18:52.220 INFO:tasks.workunit.client.1.smithi196.stdout:6/226: read f1 [6749294,23587] 0 2024-04-22T15:18:52.220 INFO:tasks.workunit.client.0.smithi096.stdout:5/609: write d4/d11/d3d/d4d/f56 [2070791,116388] 0 2024-04-22T15:18:52.221 INFO:tasks.workunit.client.1.smithi196.stdout:2/245: stat d1/cb 0 2024-04-22T15:18:52.224 INFO:tasks.workunit.client.0.smithi096.stdout:8/561: dwrite d4/d1c/f3a [0,4194304] 0 2024-04-22T15:18:52.225 INFO:tasks.workunit.client.1.smithi196.stdout:1/176: dwrite d1/f4 [4194304,4194304] 0 2024-04-22T15:18:52.225 INFO:tasks.workunit.client.1.smithi196.stdout:9/198: rename db/d11/d29/f2a to db/d11/d29/f48 0 2024-04-22T15:18:52.225 INFO:tasks.workunit.client.1.smithi196.stdout:4/214: dwrite d6/d13/d15/d2c/d35/f3b [0,4194304] 0 2024-04-22T15:18:52.226 INFO:tasks.workunit.client.1.smithi196.stdout:5/208: dread d1/d9/f38 [0,4194304] 0 2024-04-22T15:18:52.227 INFO:tasks.workunit.client.1.smithi196.stdout:2/246: dread d1/d23/f2f [0,4194304] 0 2024-04-22T15:18:52.229 INFO:tasks.workunit.client.1.smithi196.stdout:8/222: write d2/f3e [3694965,27422] 0 2024-04-22T15:18:52.229 INFO:tasks.workunit.client.0.smithi096.stdout:8/562: readlink d4/d2a/l2b 0 2024-04-22T15:18:52.230 INFO:tasks.workunit.client.0.smithi096.stdout:5/610: dwrite d4/f86 [0,4194304] 0 2024-04-22T15:18:52.230 INFO:tasks.workunit.client.1.smithi196.stdout:8/223: stat d2/f3e 0 2024-04-22T15:18:52.232 INFO:tasks.workunit.client.1.smithi196.stdout:0/211: dread d3/de/d1c/d27/f38 [0,4194304] 0 2024-04-22T15:18:52.232 INFO:tasks.workunit.client.1.smithi196.stdout:5/209: write d1/d17/f21 [603355,29261] 0 2024-04-22T15:18:52.232 INFO:tasks.workunit.client.1.smithi196.stdout:4/215: rmdir d6/d13/d15/d2c 39 2024-04-22T15:18:52.233 INFO:tasks.workunit.client.1.smithi196.stdout:6/227: creat dd/d1f/d38/d40/f42 x:0 0 0 2024-04-22T15:18:52.234 INFO:tasks.workunit.client.0.smithi096.stdout:5/611: creat d4/d11/d3d/fc6 x:0 0 0 2024-04-22T15:18:52.234 INFO:tasks.workunit.client.1.smithi196.stdout:2/247: symlink d1/d23/d27/d2e/l5c 0 2024-04-22T15:18:52.235 INFO:tasks.workunit.client.1.smithi196.stdout:6/228: write dd/d20/f24 [287858,7152] 0 2024-04-22T15:18:52.235 INFO:tasks.workunit.client.1.smithi196.stdout:2/248: dread - d1/d23/d27/f44 zero size 2024-04-22T15:18:52.236 INFO:tasks.workunit.client.1.smithi196.stdout:8/224: mkdir d2/dc/d45 0 2024-04-22T15:18:52.237 INFO:tasks.workunit.client.0.smithi096.stdout:7/823: dread d31/d32/d33/d44/da8/ff9 [0,4194304] 0 2024-04-22T15:18:52.238 INFO:tasks.workunit.client.1.smithi196.stdout:1/177: mknod d1/db/dc/d26/d2e/d32/c3c 0 2024-04-22T15:18:52.239 INFO:tasks.workunit.client.1.smithi196.stdout:1/178: chown d1/l2c 27975 1 2024-04-22T15:18:52.240 INFO:tasks.workunit.client.0.smithi096.stdout:8/563: dwrite d4/d1c/d62/fa5 [0,4194304] 0 2024-04-22T15:18:52.241 INFO:tasks.workunit.client.0.smithi096.stdout:7/824: write d31/d32/d33/d54/d43/f48 [4121929,33367] 0 2024-04-22T15:18:52.242 INFO:tasks.workunit.client.1.smithi196.stdout:5/210: unlink d1/d14/d3e/l3f 0 2024-04-22T15:18:52.243 INFO:tasks.workunit.client.1.smithi196.stdout:1/179: chown d1/db/dc/de/f14 12 1 2024-04-22T15:18:52.245 INFO:tasks.workunit.client.0.smithi096.stdout:5/612: rename d4/d11/d3d/fb6 to d4/d11/d3d/d4f/fc7 0 2024-04-22T15:18:52.247 INFO:tasks.workunit.client.0.smithi096.stdout:5/613: stat d4/d11/d3d/d4d/d59/d8a/c94 0 2024-04-22T15:18:52.249 INFO:tasks.workunit.client.1.smithi196.stdout:4/216: readlink d6/d13/d15/d2c/d35/l3e 0 2024-04-22T15:18:52.249 INFO:tasks.workunit.client.1.smithi196.stdout:6/229: dwrite dd/d1f/d38/d40/f42 [0,4194304] 0 2024-04-22T15:18:52.251 INFO:tasks.workunit.client.1.smithi196.stdout:4/217: truncate d6/d10/f37 913929 0 2024-04-22T15:18:52.251 INFO:tasks.workunit.client.1.smithi196.stdout:6/230: write dd/f1c [3328413,65544] 0 2024-04-22T15:18:52.253 INFO:tasks.workunit.client.1.smithi196.stdout:5/211: symlink d1/d14/l47 0 2024-04-22T15:18:52.253 INFO:tasks.workunit.client.1.smithi196.stdout:1/180: mknod d1/db/d16/d1e/c3d 0 2024-04-22T15:18:52.253 INFO:tasks.workunit.client.0.smithi096.stdout:8/564: mkdir d4/dc0/dc6 0 2024-04-22T15:18:52.255 INFO:tasks.workunit.client.0.smithi096.stdout:8/565: write d4/d1c/f43 [4783824,62624] 0 2024-04-22T15:18:52.255 INFO:tasks.workunit.client.1.smithi196.stdout:9/199: getdents db 0 2024-04-22T15:18:52.256 INFO:tasks.workunit.client.1.smithi196.stdout:9/200: chown db/d18/c39 15457 1 2024-04-22T15:18:52.257 INFO:tasks.workunit.client.1.smithi196.stdout:9/201: fsync db/d18/f24 0 2024-04-22T15:18:52.257 INFO:tasks.workunit.client.0.smithi096.stdout:7/825: link d31/d32/fed d31/d32/d36/d75/dff/f111 0 2024-04-22T15:18:52.260 INFO:tasks.workunit.client.1.smithi196.stdout:2/249: link d1/d13/d25/d2a/f3f d1/d23/d27/d2e/f5d 0 2024-04-22T15:18:52.262 INFO:tasks.workunit.client.1.smithi196.stdout:4/218: fsync d6/d14/d18/d26/f3a 0 2024-04-22T15:18:52.264 INFO:tasks.workunit.client.1.smithi196.stdout:9/202: dwrite db/dd/f1a [0,4194304] 0 2024-04-22T15:18:52.264 INFO:tasks.workunit.client.1.smithi196.stdout:4/219: write d6/d14/f12 [5048158,11276] 0 2024-04-22T15:18:52.266 INFO:tasks.workunit.client.0.smithi096.stdout:4/578: sync 2024-04-22T15:18:52.266 INFO:tasks.workunit.client.0.smithi096.stdout:3/457: sync 2024-04-22T15:18:52.266 INFO:tasks.workunit.client.0.smithi096.stdout:6/488: sync 2024-04-22T15:18:52.266 INFO:tasks.workunit.client.0.smithi096.stdout:9/509: sync 2024-04-22T15:18:52.266 INFO:tasks.workunit.client.0.smithi096.stdout:1/691: sync 2024-04-22T15:18:52.271 INFO:tasks.workunit.client.1.smithi196.stdout:6/231: creat dd/d1f/d38/d40/f43 x:0 0 0 2024-04-22T15:18:52.275 INFO:tasks.workunit.client.0.smithi096.stdout:7/826: fsync d31/d32/f6c 0 2024-04-22T15:18:52.279 INFO:tasks.workunit.client.1.smithi196.stdout:1/181: creat d1/db/dc/d26/d2e/f3e x:0 0 0 2024-04-22T15:18:52.281 INFO:tasks.workunit.client.0.smithi096.stdout:6/489: mknod d3/d28/d54/ca9 0 2024-04-22T15:18:52.284 INFO:tasks.workunit.client.1.smithi196.stdout:1/182: dwrite d1/db/d16/f24 [0,4194304] 0 2024-04-22T15:18:52.288 INFO:tasks.workunit.client.0.smithi096.stdout:5/614: getdents d4/d11/d3d/d4d/d59 0 2024-04-22T15:18:52.288 INFO:tasks.workunit.client.1.smithi196.stdout:1/183: dread d1/db/f29 [0,4194304] 0 2024-04-22T15:18:52.292 INFO:tasks.workunit.client.0.smithi096.stdout:3/458: dwrite da/db/de/d38/f78 [0,4194304] 0 2024-04-22T15:18:52.294 INFO:tasks.workunit.client.0.smithi096.stdout:7/827: creat d31/d32/dc0/ddc/f112 x:0 0 0 2024-04-22T15:18:52.297 INFO:tasks.workunit.client.0.smithi096.stdout:6/490: dread - d3/d28/dd/d11/d2c/d6f/f99 zero size 2024-04-22T15:18:52.307 INFO:tasks.workunit.client.0.smithi096.stdout:8/566: dread d4/d10/f19 [4194304,4194304] 0 2024-04-22T15:18:52.314 INFO:tasks.workunit.client.0.smithi096.stdout:4/579: dread d5/d7/d1a/f40 [4194304,4194304] 0 2024-04-22T15:18:52.314 INFO:tasks.workunit.client.1.smithi196.stdout:4/220: mkdir d6/d13/d15/d3f 0 2024-04-22T15:18:52.315 INFO:tasks.workunit.client.1.smithi196.stdout:4/221: write d6/d13/d15/d2c/d35/f3c [6767854,64367] 0 2024-04-22T15:18:52.318 INFO:tasks.workunit.client.0.smithi096.stdout:7/828: creat d31/d32/d33/d6a/f113 x:0 0 0 2024-04-22T15:18:52.318 INFO:tasks.workunit.client.1.smithi196.stdout:3/209: sync 2024-04-22T15:18:52.320 INFO:tasks.workunit.client.1.smithi196.stdout:3/210: dread d0/d6/db/dc/d18/f24 [0,4194304] 0 2024-04-22T15:18:52.320 INFO:tasks.workunit.client.0.smithi096.stdout:1/692: dread d6/dc/f41 [0,4194304] 0 2024-04-22T15:18:52.321 INFO:tasks.workunit.client.0.smithi096.stdout:6/491: mknod d3/d28/dd/d11/d2c/d44/d51/caa 0 2024-04-22T15:18:52.323 INFO:tasks.workunit.client.1.smithi196.stdout:5/212: getdents d1/d17/d1a 0 2024-04-22T15:18:52.323 INFO:tasks.workunit.client.1.smithi196.stdout:5/213: dread - d1/d17/d1c/f3b zero size 2024-04-22T15:18:52.325 INFO:tasks.workunit.client.0.smithi096.stdout:9/510: creat d1/d4/d9/d43/fa3 x:0 0 0 2024-04-22T15:18:52.328 INFO:tasks.workunit.client.0.smithi096.stdout:8/567: mknod d4/d1c/d40/da7/db6/cc7 0 2024-04-22T15:18:52.329 INFO:tasks.workunit.client.0.smithi096.stdout:8/568: dread - d4/d10/d25/d4c/d76/d7f/f92 zero size 2024-04-22T15:18:52.330 INFO:tasks.workunit.client.0.smithi096.stdout:8/569: write d4/d10/d25/d4c/d76/f89 [433778,75283] 0 2024-04-22T15:18:52.341 INFO:tasks.workunit.client.0.smithi096.stdout:8/570: creat d4/d1c/d40/d46/d58/fc8 x:0 0 0 2024-04-22T15:18:52.344 INFO:tasks.workunit.client.0.smithi096.stdout:9/511: truncate d1/d25/f4c 2644933 0 2024-04-22T15:18:52.344 INFO:tasks.workunit.client.1.smithi196.stdout:0/212: rmdir d3/d6/dc 39 2024-04-22T15:18:52.345 INFO:tasks.workunit.client.1.smithi196.stdout:4/222: rmdir d6/d13/d15/d3f 0 2024-04-22T15:18:52.346 INFO:tasks.workunit.client.1.smithi196.stdout:4/223: chown d6/l17 159735 1 2024-04-22T15:18:52.346 INFO:tasks.workunit.client.0.smithi096.stdout:4/580: link d5/d7/d1f/f50 d5/d7/d2c/d3d/d4f/d9c/fc6 0 2024-04-22T15:18:52.346 INFO:tasks.workunit.client.1.smithi196.stdout:4/224: dread - d6/d14/f33 zero size 2024-04-22T15:18:52.347 INFO:tasks.workunit.client.0.smithi096.stdout:4/581: chown d5/d7/f7b 8 1 2024-04-22T15:18:52.347 INFO:tasks.workunit.client.0.smithi096.stdout:4/582: chown d5/d7/d1a/f6c 2 1 2024-04-22T15:18:52.347 INFO:tasks.workunit.client.1.smithi196.stdout:4/225: chown d6/d10/fe 5494887 1 2024-04-22T15:18:52.347 INFO:tasks.workunit.client.1.smithi196.stdout:4/226: chown d6/d13/d15/d2c 5317450 1 2024-04-22T15:18:52.348 INFO:tasks.workunit.client.1.smithi196.stdout:0/213: mkdir d3/de/d1c/d27/d2c/d51 0 2024-04-22T15:18:52.348 INFO:tasks.workunit.client.1.smithi196.stdout:4/227: chown d6/d14/f2a 1063 1 2024-04-22T15:18:52.349 INFO:tasks.workunit.client.0.smithi096.stdout:8/571: mknod d4/d10/d25/d38/d8f/d9c/cc9 0 2024-04-22T15:18:52.349 INFO:tasks.workunit.client.1.smithi196.stdout:4/228: chown d6/d14/f33 1 1 2024-04-22T15:18:52.349 INFO:tasks.workunit.client.1.smithi196.stdout:5/214: dread d1/d2/f25 [0,4194304] 0 2024-04-22T15:18:52.350 INFO:tasks.workunit.client.1.smithi196.stdout:8/225: getdents d2/d5/d2e 0 2024-04-22T15:18:52.351 INFO:tasks.workunit.client.1.smithi196.stdout:0/214: fdatasync d3/de/d1c/d27/d2c/d30/f35 0 2024-04-22T15:18:52.352 INFO:tasks.workunit.client.0.smithi096.stdout:1/693: dread d6/dc/d8a/d29/f79 [0,4194304] 0 2024-04-22T15:18:52.352 INFO:tasks.workunit.client.1.smithi196.stdout:0/215: chown d3/d6/l8 11 1 2024-04-22T15:18:52.353 INFO:tasks.workunit.client.0.smithi096.stdout:9/512: unlink d1/d4/c1c 0 2024-04-22T15:18:52.354 INFO:tasks.workunit.client.0.smithi096.stdout:4/583: dwrite d5/d7/d2c/d3d/f75 [0,4194304] 0 2024-04-22T15:18:52.355 INFO:tasks.workunit.client.1.smithi196.stdout:5/215: dwrite d1/d17/d1a/f1d [0,4194304] 0 2024-04-22T15:18:52.357 INFO:tasks.workunit.client.0.smithi096.stdout:7/829: getdents d31/d32/d33/d93/dfc 0 2024-04-22T15:18:52.359 INFO:tasks.workunit.client.1.smithi196.stdout:7/205: chown d14/l3c 285 1 2024-04-22T15:18:52.360 INFO:tasks.workunit.client.0.smithi096.stdout:8/572: dread - d4/d1c/d40/d46/f7b zero size 2024-04-22T15:18:52.362 INFO:tasks.workunit.client.0.smithi096.stdout:8/573: write d4/d1c/d40/d46/d58/fb5 [216410,111990] 0 2024-04-22T15:18:52.367 INFO:tasks.workunit.client.0.smithi096.stdout:1/694: rename d6/dc/d8a/d3f/d62/fbb to d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b/fd4 0 2024-04-22T15:18:52.369 INFO:tasks.workunit.client.1.smithi196.stdout:5/216: creat d1/d14/d3e/f48 x:0 0 0 2024-04-22T15:18:52.371 INFO:tasks.workunit.client.0.smithi096.stdout:9/513: creat d1/d4/d9/d43/d57/d5b/fa4 x:0 0 0 2024-04-22T15:18:52.372 INFO:tasks.workunit.client.0.smithi096.stdout:4/584: truncate d5/d7/d1a/f38 23306 0 2024-04-22T15:18:52.373 INFO:tasks.workunit.client.1.smithi196.stdout:7/206: mkdir d14/d26/d2a/d3d 0 2024-04-22T15:18:52.374 INFO:tasks.workunit.client.0.smithi096.stdout:1/695: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d56/f44 [0,4194304] 0 2024-04-22T15:18:52.374 INFO:tasks.workunit.client.0.smithi096.stdout:7/830: creat d31/d32/dc0/f114 x:0 0 0 2024-04-22T15:18:52.382 INFO:tasks.workunit.client.0.smithi096.stdout:9/514: mkdir d1/d4/d79/da5 0 2024-04-22T15:18:52.382 INFO:tasks.workunit.client.1.smithi196.stdout:5/217: symlink d1/d17/d1a/d2f/d30/l49 0 2024-04-22T15:18:52.383 INFO:tasks.workunit.client.0.smithi096.stdout:8/574: dwrite d4/fd [8388608,4194304] 0 2024-04-22T15:18:52.383 INFO:tasks.workunit.client.1.smithi196.stdout:7/207: rmdir d14/d26/d28 39 2024-04-22T15:18:52.383 INFO:tasks.workunit.client.1.smithi196.stdout:0/216: dread d3/de/d1c/d27/f32 [0,4194304] 0 2024-04-22T15:18:52.384 INFO:tasks.workunit.client.0.smithi096.stdout:9/515: write d1/d4/d9/dd/f1a [2406053,98656] 0 2024-04-22T15:18:52.386 INFO:tasks.workunit.client.1.smithi196.stdout:5/218: dread d1/d17/d1a/f1d [0,4194304] 0 2024-04-22T15:18:52.387 INFO:tasks.workunit.client.1.smithi196.stdout:5/219: read d1/d9/f11 [383168,116684] 0 2024-04-22T15:18:52.388 INFO:tasks.workunit.client.0.smithi096.stdout:8/575: dread d4/d1c/d40/d46/f7d [0,4194304] 0 2024-04-22T15:18:52.390 INFO:tasks.workunit.client.0.smithi096.stdout:8/576: read d4/d1c/f43 [1741067,127186] 0 2024-04-22T15:18:52.391 INFO:tasks.workunit.client.0.smithi096.stdout:7/831: symlink d31/d32/d33/d5e/l115 0 2024-04-22T15:18:52.394 INFO:tasks.workunit.client.1.smithi196.stdout:8/226: getdents d2/dc 0 2024-04-22T15:18:52.397 INFO:tasks.workunit.client.0.smithi096.stdout:4/585: unlink d5/d7/d2c/d43/d56/cc4 0 2024-04-22T15:18:52.401 INFO:tasks.workunit.client.0.smithi096.stdout:9/516: creat d1/d4/d7b/fa6 x:0 0 0 2024-04-22T15:18:52.403 INFO:tasks.workunit.client.0.smithi096.stdout:1/696: mknod d6/dc/d8a/d3f/d54/d32/d8c/d4a/cd5 0 2024-04-22T15:18:52.407 INFO:tasks.workunit.client.1.smithi196.stdout:8/227: creat d2/d5/f46 x:0 0 0 2024-04-22T15:18:52.407 INFO:tasks.workunit.client.1.smithi196.stdout:2/250: rmdir d1/d13/d25 39 2024-04-22T15:18:52.408 INFO:tasks.workunit.client.0.smithi096.stdout:9/517: dwrite d1/d4/d9/d43/d57/d5b/f77 [0,4194304] 0 2024-04-22T15:18:52.409 INFO:tasks.workunit.client.0.smithi096.stdout:9/518: dread - d1/df/f65 zero size 2024-04-22T15:18:52.410 INFO:tasks.workunit.client.1.smithi196.stdout:5/220: mkdir d1/d17/d4a 0 2024-04-22T15:18:52.411 INFO:tasks.workunit.client.0.smithi096.stdout:1/697: dread d6/dc/d8a/d3f/d54/d32/d8c/d4a/d9b/dba/fc8 [0,4194304] 0 2024-04-22T15:18:52.411 INFO:tasks.workunit.client.1.smithi196.stdout:5/221: truncate d1/d14/d3e/f44 1023806 0 2024-04-22T15:18:52.415 INFO:tasks.workunit.client.1.smithi196.stdout:0/217: rename d3/d9/d25/c40 to d3/d6/dc/c52 0 2024-04-22T15:18:52.418 INFO:tasks.workunit.client.1.smithi196.stdout:8/228: truncate d2/fa 3795037 0 2024-04-22T15:18:52.419 INFO:tasks.workunit.client.1.smithi196.stdout:0/218: dread d3/de/d1c/d27/f32 [0,4194304] 0 2024-04-22T15:18:52.420 INFO:tasks.workunit.client.1.smithi196.stdout:9/203: write db/dd/f1a [4764660,104038] 0 2024-04-22T15:18:52.422 INFO:tasks.workunit.client.1.smithi196.stdout:0/219: truncate d3/de/d1c/d27/d2c/d30/f46 959298 0 2024-04-22T15:18:52.422 INFO:tasks.workunit.client.1.smithi196.stdout:8/229: mkdir d2/d8/d47 0 2024-04-22T15:18:52.424 INFO:tasks.workunit.client.1.smithi196.stdout:3/211: sync 2024-04-22T15:18:52.425 INFO:tasks.workunit.client.1.smithi196.stdout:6/232: write dd/f16 [1391381,75753] 0 2024-04-22T15:18:52.426 INFO:tasks.workunit.client.1.smithi196.stdout:0/220: fsync d3/d6/f33 0 2024-04-22T15:18:52.427 INFO:tasks.workunit.client.0.smithi096.stdout:3/459: write da/db/dc/d28/f45 [103760,99759] 0 2024-04-22T15:18:52.428 INFO:tasks.workunit.client.1.smithi196.stdout:3/212: fsync d0/d6/db/dc/d18/f42 0 2024-04-22T15:18:52.428 INFO:tasks.workunit.client.1.smithi196.stdout:0/221: chown d3/d9/d36/c4d 373094159 1 2024-04-22T15:18:52.430 INFO:tasks.workunit.client.1.smithi196.stdout:8/230: dwrite d2/d8/f25 [0,4194304] 0 2024-04-22T15:18:52.430 INFO:tasks.workunit.client.1.smithi196.stdout:3/213: write d0/d7/d2b/d44/f3c [583750,101538] 0 2024-04-22T15:18:52.431 INFO:tasks.workunit.client.0.smithi096.stdout:9/519: dread d1/f32 [0,4194304] 0 2024-04-22T15:18:52.433 INFO:tasks.workunit.client.1.smithi196.stdout:3/214: dread d0/d6/db/dc/f1c [0,4194304] 0 2024-04-22T15:18:52.436 INFO:tasks.workunit.client.1.smithi196.stdout:3/215: dread d0/d6/db/dc/d18/f1b [0,4194304] 0 2024-04-22T15:18:52.440 INFO:tasks.workunit.client.1.smithi196.stdout:1/184: truncate d1/db/d16/f24 3878348 0 2024-04-22T15:18:52.444 INFO:tasks.workunit.client.0.smithi096.stdout:8/577: rmdir d4/d1c/d40/d46/d9b 0 2024-04-22T15:18:52.448 INFO:tasks.workunit.client.0.smithi096.stdout:3/460: dread da/db/dc/d59/f79 [0,4194304] 0 2024-04-22T15:18:52.450 INFO:tasks.workunit.client.0.smithi096.stdout:2/560: sync 2024-04-22T15:18:52.450 INFO:tasks.workunit.client.1.smithi196.stdout:2/251: link d1/d23/d27/d2e/d55/l4a d1/l5e 0 2024-04-22T15:18:52.451 INFO:tasks.workunit.client.0.smithi096.stdout:3/461: fsync da/db/dc/d28/d69/d89/f65 0 2024-04-22T15:18:52.451 INFO:tasks.workunit.client.0.smithi096.stdout:8/578: dwrite d4/d10/d25/f6c [0,4194304] 0 2024-04-22T15:18:52.456 INFO:tasks.workunit.client.1.smithi196.stdout:4/229: dwrite d6/d13/f1b [0,4194304] 0 2024-04-22T15:18:52.458 INFO:tasks.workunit.client.1.smithi196.stdout:3/216: creat d0/d7/d2b/f45 x:0 0 0 2024-04-22T15:18:52.458 INFO:tasks.workunit.client.1.smithi196.stdout:3/217: fsync d0/d6/f3e 0 2024-04-22T15:18:52.460 INFO:tasks.workunit.client.1.smithi196.stdout:3/218: dread d0/d7/f11 [0,4194304] 0 2024-04-22T15:18:52.460 INFO:tasks.workunit.client.1.smithi196.stdout:3/219: chown d0/d6/db/l37 1 1 2024-04-22T15:18:52.465 INFO:tasks.workunit.client.1.smithi196.stdout:6/233: mknod dd/d1f/d37/d3c/c44 0 2024-04-22T15:18:52.469 INFO:tasks.workunit.client.0.smithi096.stdout:8/579: symlink d4/d10/d25/d38/d8f/lca 0 2024-04-22T15:18:52.470 INFO:tasks.workunit.client.1.smithi196.stdout:2/252: creat d1/d1d/f5f x:0 0 0 2024-04-22T15:18:52.471 INFO:tasks.workunit.client.1.smithi196.stdout:2/253: write d1/d23/f40 [3337601,88127] 0 2024-04-22T15:18:52.471 INFO:tasks.workunit.client.0.smithi096.stdout:3/462: rename da/db/dc/f7b to da/db/de/d8b/f92 0 2024-04-22T15:18:52.477 INFO:tasks.workunit.client.0.smithi096.stdout:1/698: link d6/f60 d6/dc/d8a/d3f/d62/d7d/dd3/fd6 0 2024-04-22T15:18:52.478 INFO:tasks.workunit.client.1.smithi196.stdout:4/230: mkdir d6/d14/d18/d40 0 2024-04-22T15:18:52.481 INFO:tasks.workunit.client.0.smithi096.stdout:2/561: dread d3/db/df/d24/d58/d5e/f6b [0,4194304] 0 2024-04-22T15:18:52.482 INFO:tasks.workunit.client.0.smithi096.stdout:8/580: fsync d4/d10/f19 0 2024-04-22T15:18:52.482 INFO:tasks.workunit.client.0.smithi096.stdout:8/581: chown d4/d2a/f3c 454495 1 2024-04-22T15:18:52.482 INFO:tasks.workunit.client.0.smithi096.stdout:2/562: truncate d3/db/d2d/d46/d69/f6c 1916492 0 2024-04-22T15:18:52.483 INFO:tasks.workunit.client.1.smithi196.stdout:1/185: creat d1/db/dc/d26/d2e/d32/d36/f3f x:0 0 0 2024-04-22T15:18:52.484 INFO:tasks.workunit.client.0.smithi096.stdout:4/586: write d5/d7/d1a/f97 [1730239,54563] 0 2024-04-22T15:18:52.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:52 smithi096 ceph-mon[26688]: pgmap v32: 65 pgs: 65 active+clean; 1.5 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 41 MiB/s rd, 146 MiB/s wr, 304 op/s 2024-04-22T15:18:52.486 INFO:tasks.workunit.client.1.smithi196.stdout:7/208: sync 2024-04-22T15:18:52.489 INFO:tasks.workunit.client.1.smithi196.stdout:6/234: dwrite dd/f26 [0,4194304] 0 2024-04-22T15:18:52.491 INFO:tasks.workunit.client.1.smithi196.stdout:2/254: creat d1/d13/d1f/f60 x:0 0 0 2024-04-22T15:18:52.492 INFO:tasks.workunit.client.1.smithi196.stdout:2/255: chown d1/d13/c37 13 1 2024-04-22T15:18:52.493 INFO:tasks.workunit.client.1.smithi196.stdout:4/231: symlink d6/d13/d15/d2c/d35/l41 0 2024-04-22T15:18:52.494 INFO:tasks.workunit.client.0.smithi096.stdout:8/582: chown d4/d10/d25/d4c/f5e 217 1 2024-04-22T15:18:52.495 INFO:tasks.workunit.client.0.smithi096.stdout:8/583: dread - d4/d1c/d40/d46/f7b zero size 2024-04-22T15:18:52.496 INFO:tasks.workunit.client.0.smithi096.stdout:8/584: readlink d4/d1c/d40/da7/db6/l51 0 2024-04-22T15:18:52.499 INFO:tasks.workunit.client.0.smithi096.stdout:7/832: truncate d31/d32/d33/d93/fae 1599181 0 2024-04-22T15:18:52.502 INFO:tasks.workunit.client.0.smithi096.stdout:3/463: link da/db/dc/f49 da/db/dc/d28/f93 0 2024-04-22T15:18:52.504 INFO:tasks.workunit.client.1.smithi196.stdout:8/231: getdents d2/d35 0 2024-04-22T15:18:52.504 INFO:tasks.workunit.client.0.smithi096.stdout:9/520: link d1/d4/d9/dd/f23 d1/df/fa7 0 2024-04-22T15:18:52.504 INFO:tasks.workunit.client.1.smithi196.stdout:9/204: dwrite db/dd/de/d15/f1e [4194304,4194304] 0 2024-04-22T15:18:52.506 INFO:tasks.workunit.client.1.smithi196.stdout:9/205: read - db/d11/d29/f48 zero size 2024-04-22T15:18:52.507 INFO:tasks.workunit.client.0.smithi096.stdout:1/699: fsync d6/dc/d8a/d3f/d54/d32/d3b/f9e 0 2024-04-22T15:18:52.507 INFO:tasks.workunit.client.1.smithi196.stdout:9/206: dread - db/dd/d1b/d2c/f37 zero size 2024-04-22T15:18:52.508 INFO:tasks.workunit.client.1.smithi196.stdout:9/207: write db/dd/d1b/f45 [22425,74216] 0 2024-04-22T15:18:52.508 INFO:tasks.workunit.client.1.smithi196.stdout:8/232: dread d2/f38 [4194304,4194304] 0 2024-04-22T15:18:52.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:52 smithi196 ceph-mon[34717]: pgmap v32: 65 pgs: 65 active+clean; 1.5 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 41 MiB/s rd, 146 MiB/s wr, 304 op/s 2024-04-22T15:18:52.509 INFO:tasks.workunit.client.1.smithi196.stdout:8/233: dread - d2/f32 zero size 2024-04-22T15:18:52.510 INFO:tasks.workunit.client.1.smithi196.stdout:8/234: write d2/d5/d2e/f43 [102942,4870] 0 2024-04-22T15:18:52.510 INFO:tasks.workunit.client.1.smithi196.stdout:8/235: fsync d2/f4 0 2024-04-22T15:18:52.511 INFO:tasks.workunit.client.0.smithi096.stdout:8/585: mkdir d4/d1c/d40/d46/d58/dcb 0 2024-04-22T15:18:52.517 INFO:tasks.workunit.client.0.smithi096.stdout:7/833: rename d31/d32/d36/d75/l91 to d31/d32/d33/d5e/dc9/l116 0 2024-04-22T15:18:52.518 INFO:tasks.workunit.client.0.smithi096.stdout:8/586: dwrite d4/d10/f97 [0,4194304] 0 2024-04-22T15:18:52.519 INFO:tasks.workunit.client.0.smithi096.stdout:8/587: stat d4/d1c/d40/c65 0 2024-04-22T15:18:52.519 INFO:tasks.workunit.client.0.smithi096.stdout:3/464: symlink da/db/d10/l94 0 2024-04-22T15:18:52.527 INFO:tasks.workunit.client.1.smithi196.stdout:9/208: creat db/dd/de/d15/d31/f49 x:0 0 0 2024-04-22T15:18:52.528 INFO:tasks.workunit.client.1.smithi196.stdout:9/209: read - db/d11/d29/f48 zero size 2024-04-22T15:18:52.528 INFO:tasks.workunit.client.0.smithi096.stdout:3/465: dwrite da/f8f [0,4194304] 0 2024-04-22T15:18:52.530 INFO:tasks.workunit.client.0.smithi096.stdout:3/466: write da/db/d12/d22/f3a [1329848,45754] 0 2024-04-22T15:18:52.530 INFO:tasks.workunit.client.1.smithi196.stdout:2/256: mknod d1/d13/c61 0 2024-04-22T15:18:52.535 INFO:tasks.workunit.client.0.smithi096.stdout:4/587: creat d5/d7/fc7 x:0 0 0 2024-04-22T15:18:52.536 INFO:tasks.workunit.client.1.smithi196.stdout:8/236: creat d2/d5/f48 x:0 0 0 2024-04-22T15:18:52.536 INFO:tasks.workunit.client.1.smithi196.stdout:1/186: rmdir d1/db/dc/d26/d2e/d32/d36 39 2024-04-22T15:18:52.537 INFO:tasks.workunit.client.1.smithi196.stdout:1/187: chown d1/db/dc/de/c1a 28 1 2024-04-22T15:18:52.539 INFO:tasks.workunit.client.1.smithi196.stdout:1/188: read d1/db/dc/d26/f30 [227518,24598] 0 2024-04-22T15:18:52.540 INFO:tasks.workunit.client.1.smithi196.stdout:1/189: chown d1/db/dc/d26/d2e/f34 9 1 2024-04-22T15:18:52.540 INFO:tasks.workunit.client.0.smithi096.stdout:8/588: mkdir d4/d10/dcc 0 2024-04-22T15:18:52.542 INFO:tasks.workunit.client.0.smithi096.stdout:7/834: dread d31/d32/d36/d75/fc3 [0,4194304] 0 2024-04-22T15:18:52.542 INFO:tasks.workunit.client.0.smithi096.stdout:7/835: fdatasync f18 0 2024-04-22T15:18:52.543 INFO:tasks.workunit.client.0.smithi096.stdout:2/563: write d3/f49 [3415734,37142] 0 2024-04-22T15:18:52.546 INFO:tasks.workunit.client.0.smithi096.stdout:8/589: dread d4/d10/d25/d38/f4b [4194304,4194304] 0 2024-04-22T15:18:52.548 INFO:tasks.workunit.client.1.smithi196.stdout:2/257: creat d1/d13/d41/f62 x:0 0 0 2024-04-22T15:18:52.548 INFO:tasks.workunit.client.1.smithi196.stdout:2/258: chown d1/d13/c39 15 1 2024-04-22T15:18:52.549 INFO:tasks.workunit.client.0.smithi096.stdout:3/467: symlink da/d71/l95 0 2024-04-22T15:18:52.549 INFO:tasks.workunit.client.1.smithi196.stdout:2/259: write d1/d1d/f3c [1236118,72439] 0 2024-04-22T15:18:52.550 INFO:tasks.workunit.client.1.smithi196.stdout:2/260: chown d1/d13/d41/f62 1505 1 2024-04-22T15:18:52.552 INFO:tasks.workunit.client.1.smithi196.stdout:5/222: sync 2024-04-22T15:18:52.552 INFO:tasks.workunit.client.0.smithi096.stdout:4/588: symlink d5/d7/d1f/d84/d6d/d98/lc8 0 2024-04-22T15:18:52.554 INFO:tasks.workunit.client.1.smithi196.stdout:4/232: creat d6/f42 x:0 0 0 2024-04-22T15:18:52.556 INFO:tasks.workunit.client.1.smithi196.stdout:6/235: dwrite dd/f3a [4194304,4194304] 0 2024-04-22T15:18:52.556 INFO:tasks.workunit.client.1.smithi196.stdout:6/236: chown dd/f26 46277 1 2024-04-22T15:18:52.558 INFO:tasks.workunit.client.1.smithi196.stdout:3/220: getdents d0/d15 0 2024-04-22T15:18:52.562 INFO:tasks.workunit.client.0.smithi096.stdout:8/590: rmdir d4/d1c/d40 39 2024-04-22T15:18:52.567 INFO:tasks.workunit.client.0.smithi096.stdout:3/468: mkdir da/db/dc/d28/d96 0 2024-04-22T15:18:52.568 INFO:tasks.workunit.client.1.smithi196.stdout:2/261: fsync d1/d13/d25/f4f 0 2024-04-22T15:18:52.569 INFO:tasks.workunit.client.0.smithi096.stdout:4/589: symlink d5/d7/d2c/d43/d56/lc9 0 2024-04-22T15:18:52.570 INFO:tasks.workunit.client.0.smithi096.stdout:4/590: chown d5/d7/d1f/f42 55441557 1 2024-04-22T15:18:52.571 INFO:tasks.workunit.client.1.smithi196.stdout:5/223: symlink d1/d17/d1c/l4b 0 2024-04-22T15:18:52.571 INFO:tasks.workunit.client.1.smithi196.stdout:5/224: chown d1/d17/f24 949260 1 2024-04-22T15:18:52.572 INFO:tasks.workunit.client.1.smithi196.stdout:4/233: creat d6/d14/d18/d26/f43 x:0 0 0 2024-04-22T15:18:52.572 INFO:tasks.workunit.client.1.smithi196.stdout:0/222: sync 2024-04-22T15:18:52.572 INFO:tasks.workunit.client.1.smithi196.stdout:7/209: sync 2024-04-22T15:18:52.576 INFO:tasks.workunit.client.1.smithi196.stdout:5/225: dwrite d1/d14/d3e/f44 [0,4194304] 0 2024-04-22T15:18:52.577 INFO:tasks.workunit.client.0.smithi096.stdout:4/591: dwrite d5/f21 [0,4194304] 0 2024-04-22T15:18:52.578 INFO:tasks.workunit.client.0.smithi096.stdout:1/700: getdents d6/dc/d8a/d3f/d62 0 2024-04-22T15:18:52.579 INFO:tasks.workunit.client.1.smithi196.stdout:3/221: creat d0/d7/d2b/f46 x:0 0 0 2024-04-22T15:18:52.583 INFO:tasks.workunit.client.1.smithi196.stdout:1/190: rmdir d1/db/dc/d26/d2e/d32/d36 39 2024-04-22T15:18:52.583 INFO:tasks.workunit.client.1.smithi196.stdout:8/237: write d2/d8/f25 [5132052,83524] 0 2024-04-22T15:18:52.585 INFO:tasks.workunit.client.1.smithi196.stdout:9/210: link db/dd/de/d15/l3a db/d11/d29/l4a 0 2024-04-22T15:18:52.589 INFO:tasks.workunit.client.0.smithi096.stdout:7/836: write d31/d68/f69 [589851,19146] 0 2024-04-22T15:18:52.593 INFO:tasks.workunit.client.1.smithi196.stdout:4/234: unlink d6/f34 0 2024-04-22T15:18:52.595 INFO:tasks.workunit.client.0.smithi096.stdout:4/592: creat d5/d7/d1f/fca x:0 0 0 2024-04-22T15:18:52.599 INFO:tasks.workunit.client.0.smithi096.stdout:2/564: write d3/db/df/d10/f21 [2351857,75688] 0 2024-04-22T15:18:52.599 INFO:tasks.workunit.client.0.smithi096.stdout:3/469: getdents da/db/de/d87 0 2024-04-22T15:18:52.602 INFO:tasks.workunit.client.0.smithi096.stdout:3/470: dread da/db/dc/d28/d69/d89/f65 [0,4194304] 0 2024-04-22T15:18:52.606 INFO:tasks.workunit.client.1.smithi196.stdout:3/222: creat d0/d6/db/dc/f47 x:0 0 0 2024-04-22T15:18:52.609 INFO:tasks.workunit.client.1.smithi196.stdout:2/262: dwrite d1/f3 [4194304,4194304] 0 2024-04-22T15:18:52.609 INFO:tasks.workunit.client.1.smithi196.stdout:1/191: creat d1/db/d16/f40 x:0 0 0 2024-04-22T15:18:52.612 INFO:tasks.workunit.client.1.smithi196.stdout:8/238: truncate d2/dc/fd 2324455 0 2024-04-22T15:18:52.613 INFO:tasks.workunit.client.1.smithi196.stdout:6/237: truncate f5 1279569 0 2024-04-22T15:18:52.615 INFO:tasks.workunit.client.1.smithi196.stdout:9/211: mkdir db/dd/d1b/d4b 0 2024-04-22T15:18:52.616 INFO:tasks.workunit.client.0.smithi096.stdout:1/701: write d6/dc/d8a/d3f/d54/d32/d8c/d45/f66 [281938,90240] 0 2024-04-22T15:18:52.617 INFO:tasks.workunit.client.0.smithi096.stdout:1/702: fdatasync d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fa1 0 2024-04-22T15:18:52.618 INFO:tasks.workunit.client.0.smithi096.stdout:2/565: write d3/d1f/d2f/f67 [4856883,110556] 0 2024-04-22T15:18:52.619 INFO:tasks.workunit.client.0.smithi096.stdout:4/593: mkdir d5/d7/dcb 0 2024-04-22T15:18:52.619 INFO:tasks.workunit.client.1.smithi196.stdout:8/239: dwrite d2/dc/f2f [0,4194304] 0 2024-04-22T15:18:52.620 INFO:tasks.workunit.client.1.smithi196.stdout:4/235: rmdir d6/d13 39 2024-04-22T15:18:52.622 INFO:tasks.workunit.client.0.smithi096.stdout:8/591: link d4/d10/d4d/cb0 d4/d1c/d40/d46/d58/ccd 0 2024-04-22T15:18:52.623 INFO:tasks.workunit.client.1.smithi196.stdout:0/223: creat d3/de/d1c/d27/d2c/d51/f53 x:0 0 0 2024-04-22T15:18:52.623 INFO:tasks.workunit.client.1.smithi196.stdout:3/223: mkdir d0/d15/d48 0 2024-04-22T15:18:52.624 INFO:tasks.workunit.client.0.smithi096.stdout:3/471: mkdir da/db/dc/d28/d96/d97 0 2024-04-22T15:18:52.624 INFO:tasks.workunit.client.1.smithi196.stdout:9/212: dwrite f5 [0,4194304] 0 2024-04-22T15:18:52.627 INFO:tasks.workunit.client.1.smithi196.stdout:2/263: symlink d1/d23/l63 0 2024-04-22T15:18:52.629 INFO:tasks.workunit.client.1.smithi196.stdout:8/240: dread d2/f3e [0,4194304] 0 2024-04-22T15:18:52.632 INFO:tasks.workunit.client.1.smithi196.stdout:8/241: read d2/f4 [2656976,114718] 0 2024-04-22T15:18:52.632 INFO:tasks.workunit.client.1.smithi196.stdout:6/238: symlink dd/d1f/l45 0 2024-04-22T15:18:52.634 INFO:tasks.workunit.client.1.smithi196.stdout:1/192: dread d1/db/dc/d26/f2b [0,4194304] 0 2024-04-22T15:18:52.641 INFO:tasks.workunit.client.0.smithi096.stdout:7/837: getdents d31/d32/d33/d54/d43 0 2024-04-22T15:18:52.643 INFO:tasks.workunit.client.1.smithi196.stdout:4/236: dwrite d6/d13/d15/f30 [0,4194304] 0 2024-04-22T15:18:52.643 INFO:tasks.workunit.client.1.smithi196.stdout:5/226: creat d1/f4c x:0 0 0 2024-04-22T15:18:52.643 INFO:tasks.workunit.client.0.smithi096.stdout:0/746: sync 2024-04-22T15:18:52.645 INFO:tasks.workunit.client.1.smithi196.stdout:5/227: dread d1/d9/f11 [0,4194304] 0 2024-04-22T15:18:52.646 INFO:tasks.workunit.client.1.smithi196.stdout:0/224: creat d3/d9/f54 x:0 0 0 2024-04-22T15:18:52.648 INFO:tasks.workunit.client.0.smithi096.stdout:2/566: symlink d3/db/d2d/d46/d59/lb1 0 2024-04-22T15:18:52.650 INFO:tasks.workunit.client.1.smithi196.stdout:9/213: creat db/d11/f4c x:0 0 0 2024-04-22T15:18:52.654 INFO:tasks.workunit.client.0.smithi096.stdout:8/592: link d4/d1c/f72 d4/d10/d25/d38/fce 0 2024-04-22T15:18:52.657 INFO:tasks.workunit.client.0.smithi096.stdout:2/567: dwrite d3/db/d2d/d46/d69/f6c [0,4194304] 0 2024-04-22T15:18:52.658 INFO:tasks.workunit.client.0.smithi096.stdout:8/593: write d4/d10/d25/d4c/d76/f89 [978115,20041] 0 2024-04-22T15:18:52.661 INFO:tasks.workunit.client.1.smithi196.stdout:2/264: symlink d1/d13/d25/l64 0 2024-04-22T15:18:52.662 INFO:tasks.workunit.client.1.smithi196.stdout:2/265: write d1/d1d/f3c [3927703,106668] 0 2024-04-22T15:18:52.662 INFO:tasks.workunit.client.0.smithi096.stdout:4/594: dread d5/d7/d2c/d3d/d4f/f5c [0,4194304] 0 2024-04-22T15:18:52.666 INFO:tasks.workunit.client.1.smithi196.stdout:6/239: dread dd/d20/f28 [0,4194304] 0 2024-04-22T15:18:52.666 INFO:tasks.workunit.client.1.smithi196.stdout:8/242: truncate d2/f1d 2668171 0 2024-04-22T15:18:52.668 INFO:tasks.workunit.client.0.smithi096.stdout:7/838: dread d31/d32/d36/d75/f86 [0,4194304] 0 2024-04-22T15:18:52.670 INFO:tasks.workunit.client.1.smithi196.stdout:1/193: creat d1/db/dc/d26/d2e/f41 x:0 0 0 2024-04-22T15:18:52.670 INFO:tasks.workunit.client.0.smithi096.stdout:3/472: dwrite da/db/dc/f5d [0,4194304] 0 2024-04-22T15:18:52.673 INFO:tasks.workunit.client.0.smithi096.stdout:3/473: write da/db/de/d19/f30 [8682144,81771] 0 2024-04-22T15:18:52.674 INFO:tasks.workunit.client.0.smithi096.stdout:3/474: fdatasync da/db/d12/d22/f53 0 2024-04-22T15:18:52.675 INFO:tasks.workunit.client.0.smithi096.stdout:1/703: getdents d6/dc 0 2024-04-22T15:18:52.675 INFO:tasks.workunit.client.0.smithi096.stdout:7/839: fsync fa 0 2024-04-22T15:18:52.677 INFO:tasks.workunit.client.1.smithi196.stdout:4/237: mkdir d6/d13/d15/d2c/d35/d44 0 2024-04-22T15:18:52.677 INFO:tasks.workunit.client.0.smithi096.stdout:3/475: truncate da/db/de/d38/f78 5225629 0 2024-04-22T15:18:52.678 INFO:tasks.workunit.client.1.smithi196.stdout:4/238: dread - d6/d14/f33 zero size 2024-04-22T15:18:52.679 INFO:tasks.workunit.client.0.smithi096.stdout:3/476: truncate da/f8a 71757 0 2024-04-22T15:18:52.679 INFO:tasks.workunit.client.0.smithi096.stdout:8/594: symlink d4/d10/d25/d4c/db4/lcf 0 2024-04-22T15:18:52.679 INFO:tasks.workunit.client.1.smithi196.stdout:3/224: dwrite d0/d6/db/dc/d18/f1b [0,4194304] 0 2024-04-22T15:18:52.684 INFO:tasks.workunit.client.1.smithi196.stdout:9/214: creat db/d11/f4d x:0 0 0 2024-04-22T15:18:52.688 INFO:tasks.workunit.client.1.smithi196.stdout:2/266: creat d1/d23/d27/f65 x:0 0 0 2024-04-22T15:18:52.690 INFO:tasks.workunit.client.0.smithi096.stdout:1/704: stat d6/c83 0 2024-04-22T15:18:52.691 INFO:tasks.workunit.client.0.smithi096.stdout:1/705: chown d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e/fa6 12107 1 2024-04-22T15:18:52.693 INFO:tasks.workunit.client.0.smithi096.stdout:7/840: rename d31/d32/d36/d75/ff5 to d31/d32/d36/d75/dfe/f117 0 2024-04-22T15:18:52.696 INFO:tasks.workunit.client.1.smithi196.stdout:8/243: mkdir d2/d8/d49 0 2024-04-22T15:18:52.696 INFO:tasks.workunit.client.0.smithi096.stdout:8/595: mkdir d4/d10/d25/d4c/db4/dd0 0 2024-04-22T15:18:52.698 INFO:tasks.workunit.client.1.smithi196.stdout:1/194: creat d1/db/d2d/f42 x:0 0 0 2024-04-22T15:18:52.698 INFO:tasks.workunit.client.0.smithi096.stdout:2/568: link d3/db/d2d/d46/l4c d3/db/d30/lb2 0 2024-04-22T15:18:52.698 INFO:tasks.workunit.client.1.smithi196.stdout:6/240: write dd/fe [183849,43818] 0 2024-04-22T15:18:52.699 INFO:tasks.workunit.client.0.smithi096.stdout:2/569: write d3/f49 [1608778,122165] 0 2024-04-22T15:18:52.699 INFO:tasks.workunit.client.1.smithi196.stdout:3/225: fsync d0/d7/f11 0 2024-04-22T15:18:52.700 INFO:tasks.workunit.client.0.smithi096.stdout:4/595: creat d5/d7/d1a/fcc x:0 0 0 2024-04-22T15:18:52.702 INFO:tasks.workunit.client.0.smithi096.stdout:4/596: chown d5/d7/d2c/d43/d56/f59 925 1 2024-04-22T15:18:52.703 INFO:tasks.workunit.client.1.smithi196.stdout:9/215: creat db/dd/de/f4e x:0 0 0 2024-04-22T15:18:52.703 INFO:tasks.workunit.client.0.smithi096.stdout:7/841: mkdir d31/d118 0 2024-04-22T15:18:52.704 INFO:tasks.workunit.client.1.smithi196.stdout:9/216: dread - db/dd/de/d15/f3c zero size 2024-04-22T15:18:52.704 INFO:tasks.workunit.client.0.smithi096.stdout:7/842: dread - d31/d32/d36/d5d/fd9 zero size 2024-04-22T15:18:52.705 INFO:tasks.workunit.client.1.smithi196.stdout:9/217: write db/d11/f4c [419224,91002] 0 2024-04-22T15:18:52.705 INFO:tasks.workunit.client.1.smithi196.stdout:2/267: mknod d1/d23/d27/d2e/d55/c66 0 2024-04-22T15:18:52.705 INFO:tasks.workunit.client.0.smithi096.stdout:7/843: stat d31/d32/d33/f58 0 2024-04-22T15:18:52.706 INFO:tasks.workunit.client.1.smithi196.stdout:9/218: chown db/dd/de/d15/f36 6929171 1 2024-04-22T15:18:52.706 INFO:tasks.workunit.client.1.smithi196.stdout:2/268: chown d1/d1d/f29 6 1 2024-04-22T15:18:52.706 INFO:tasks.workunit.client.1.smithi196.stdout:2/269: dread - d1/d23/d27/d2e/f58 zero size 2024-04-22T15:18:52.710 INFO:tasks.workunit.client.0.smithi096.stdout:2/570: mkdir d3/db/df/d98/db3 0 2024-04-22T15:18:52.713 INFO:tasks.workunit.client.1.smithi196.stdout:0/225: dwrite d3/de/d14/f4a [0,4194304] 0 2024-04-22T15:18:52.713 INFO:tasks.workunit.client.1.smithi196.stdout:3/226: unlink d0/d6/db/dc/d18/f24 0 2024-04-22T15:18:52.714 INFO:tasks.workunit.client.1.smithi196.stdout:7/210: sync 2024-04-22T15:18:52.716 INFO:tasks.workunit.client.1.smithi196.stdout:3/227: write d0/d6/f2d [945078,29890] 0 2024-04-22T15:18:52.717 INFO:tasks.workunit.client.1.smithi196.stdout:1/195: dwrite d1/db/d16/f17 [4194304,4194304] 0 2024-04-22T15:18:52.718 INFO:tasks.workunit.client.1.smithi196.stdout:9/219: symlink db/dd/de/d15/d31/l4f 0 2024-04-22T15:18:52.720 INFO:tasks.workunit.client.0.smithi096.stdout:3/477: getdents da/db/de/d19/d2d 0 2024-04-22T15:18:52.720 INFO:tasks.workunit.client.1.smithi196.stdout:1/196: read d1/db/d16/f1d [439070,106382] 0 2024-04-22T15:18:52.721 INFO:tasks.workunit.client.1.smithi196.stdout:6/241: dread dd/f14 [0,4194304] 0 2024-04-22T15:18:52.722 INFO:tasks.workunit.client.1.smithi196.stdout:1/197: readlink d1/db/dc/d26/l3b 0 2024-04-22T15:18:52.724 INFO:tasks.workunit.client.0.smithi096.stdout:8/596: truncate d4/d1c/d40/d46/d58/f60 1206487 0 2024-04-22T15:18:52.726 INFO:tasks.workunit.client.0.smithi096.stdout:2/571: mkdir d3/d13/db4 0 2024-04-22T15:18:52.727 INFO:tasks.workunit.client.0.smithi096.stdout:4/597: dwrite d5/d5b/f80 [0,4194304] 0 2024-04-22T15:18:52.728 INFO:tasks.workunit.client.1.smithi196.stdout:0/226: creat d3/de/d14/f55 x:0 0 0 2024-04-22T15:18:52.729 INFO:tasks.workunit.client.0.smithi096.stdout:2/572: stat d3/db/df/d10/f11 0 2024-04-22T15:18:52.730 INFO:tasks.workunit.client.0.smithi096.stdout:2/573: write d3/db/df/d10/f3b [1047158,118834] 0 2024-04-22T15:18:52.731 INFO:tasks.workunit.client.1.smithi196.stdout:7/211: read d14/d26/f30 [837752,89785] 0 2024-04-22T15:18:52.733 INFO:tasks.workunit.client.1.smithi196.stdout:3/228: rename d0/d6/l36 to d0/d6/l49 0 2024-04-22T15:18:52.736 INFO:tasks.workunit.client.0.smithi096.stdout:2/574: mkdir d3/db/df/d10/db5 0 2024-04-22T15:18:52.737 INFO:tasks.workunit.client.1.smithi196.stdout:9/220: symlink db/dd/d1b/d2c/l50 0 2024-04-22T15:18:52.738 INFO:tasks.workunit.client.1.smithi196.stdout:6/242: chown dd/f21 477 1 2024-04-22T15:18:52.738 INFO:tasks.workunit.client.1.smithi196.stdout:9/221: dread - db/d18/f33 zero size 2024-04-22T15:18:52.738 INFO:tasks.workunit.client.1.smithi196.stdout:8/244: creat d2/f4a x:0 0 0 2024-04-22T15:18:52.740 INFO:tasks.workunit.client.1.smithi196.stdout:1/198: mkdir d1/db/dc/de/d43 0 2024-04-22T15:18:52.740 INFO:tasks.workunit.client.1.smithi196.stdout:0/227: rmdir d3/de/d1c 39 2024-04-22T15:18:52.741 INFO:tasks.workunit.client.0.smithi096.stdout:4/598: dread d5/d7/d1a/f23 [0,4194304] 0 2024-04-22T15:18:52.741 INFO:tasks.workunit.client.1.smithi196.stdout:1/199: fdatasync d1/db/dc/d26/d2e/f41 0 2024-04-22T15:18:52.741 INFO:tasks.workunit.client.1.smithi196.stdout:0/228: write d3/f47 [160062,50824] 0 2024-04-22T15:18:52.745 INFO:tasks.workunit.client.0.smithi096.stdout:4/599: unlink d5/d7/d2c/d95/laf 0 2024-04-22T15:18:52.746 INFO:tasks.workunit.client.1.smithi196.stdout:8/245: mkdir d2/d3f/d4b 0 2024-04-22T15:18:52.749 INFO:tasks.workunit.client.1.smithi196.stdout:3/229: truncate d0/d7/f11 451769 0 2024-04-22T15:18:52.750 INFO:tasks.workunit.client.0.smithi096.stdout:4/600: creat d5/d7/d1f/d84/d6d/fcd x:0 0 0 2024-04-22T15:18:52.751 INFO:tasks.workunit.client.0.smithi096.stdout:4/601: write d5/d7/d2c/d67/fab [669355,72844] 0 2024-04-22T15:18:52.752 INFO:tasks.workunit.client.1.smithi196.stdout:7/212: mknod d14/c3e 0 2024-04-22T15:18:52.752 INFO:tasks.workunit.client.0.smithi096.stdout:4/602: write d5/d7/d1a/fcc [692946,114195] 0 2024-04-22T15:18:52.753 INFO:tasks.workunit.client.1.smithi196.stdout:7/213: truncate d14/d15/f1a 2099244 0 2024-04-22T15:18:52.754 INFO:tasks.workunit.client.0.smithi096.stdout:3/478: dread da/db/de/d19/f2a [0,4194304] 0 2024-04-22T15:18:52.754 INFO:tasks.workunit.client.0.smithi096.stdout:4/603: chown d5/d7/d1f/fca 141 1 2024-04-22T15:18:52.755 INFO:tasks.workunit.client.0.smithi096.stdout:8/597: write d4/d1c/fc4 [975150,22281] 0 2024-04-22T15:18:52.756 INFO:tasks.workunit.client.0.smithi096.stdout:8/598: stat d4/d1c/d40/d48/lb2 0 2024-04-22T15:18:52.756 INFO:tasks.workunit.client.1.smithi196.stdout:2/270: getdents d1/d23/d27/d2e 0 2024-04-22T15:18:52.757 INFO:tasks.workunit.client.1.smithi196.stdout:8/246: unlink d2/d8/l31 0 2024-04-22T15:18:52.758 INFO:tasks.workunit.client.1.smithi196.stdout:2/271: write d1/d13/d1f/f20 [1243156,32785] 0 2024-04-22T15:18:52.760 INFO:tasks.workunit.client.1.smithi196.stdout:2/272: write d1/d13/d1f/f20 [305830,36285] 0 2024-04-22T15:18:52.760 INFO:tasks.workunit.client.1.smithi196.stdout:6/243: link dd/fe dd/d1f/d37/d3c/f46 0 2024-04-22T15:18:52.761 INFO:tasks.workunit.client.0.smithi096.stdout:4/604: chown d5/d7/d2c/c2f 0 1 2024-04-22T15:18:52.761 INFO:tasks.workunit.client.1.smithi196.stdout:1/200: dwrite d1/db/dc/d26/d2e/d32/d36/f3f [0,4194304] 0 2024-04-22T15:18:52.762 INFO:tasks.workunit.client.1.smithi196.stdout:2/273: stat d1/d13/d25/d2a/f3f 0 2024-04-22T15:18:52.765 INFO:tasks.workunit.client.1.smithi196.stdout:2/274: write d1/d1d/f57 [869674,112581] 0 2024-04-22T15:18:52.766 INFO:tasks.workunit.client.1.smithi196.stdout:2/275: readlink d1/d13/l49 0 2024-04-22T15:18:52.767 INFO:tasks.workunit.client.1.smithi196.stdout:8/247: dwrite d2/dc/f3a [0,4194304] 0 2024-04-22T15:18:52.768 INFO:tasks.workunit.client.0.smithi096.stdout:4/605: creat d5/fce x:0 0 0 2024-04-22T15:18:52.771 INFO:tasks.workunit.client.1.smithi196.stdout:5/228: sync 2024-04-22T15:18:52.771 INFO:tasks.workunit.client.1.smithi196.stdout:0/229: rename d3/d42 to d3/d9/d36/d56 0 2024-04-22T15:18:52.771 INFO:tasks.workunit.client.0.smithi096.stdout:2/575: dwrite d3/f53 [0,4194304] 0 2024-04-22T15:18:52.771 INFO:tasks.workunit.client.0.smithi096.stdout:4/606: write d5/d5b/f80 [1189866,102921] 0 2024-04-22T15:18:52.772 INFO:tasks.workunit.client.1.smithi196.stdout:0/230: dread - d3/d9/d36/f4c zero size 2024-04-22T15:18:52.772 INFO:tasks.workunit.client.1.smithi196.stdout:5/229: chown d1/d9/c10 2 1 2024-04-22T15:18:52.772 INFO:tasks.workunit.client.1.smithi196.stdout:5/230: chown d1/d9/f38 0 1 2024-04-22T15:18:52.775 INFO:tasks.workunit.client.0.smithi096.stdout:8/599: rename d4/d1c/ca8 to d4/d1c/d40/d48/cd1 0 2024-04-22T15:18:52.776 INFO:tasks.workunit.client.1.smithi196.stdout:7/214: creat d14/d25/d31/f3f x:0 0 0 2024-04-22T15:18:52.785 INFO:tasks.workunit.client.1.smithi196.stdout:9/222: truncate db/dd/de/d15/f40 7685563 0 2024-04-22T15:18:52.786 INFO:tasks.workunit.client.1.smithi196.stdout:6/244: creat dd/d1f/f47 x:0 0 0 2024-04-22T15:18:52.788 INFO:tasks.workunit.client.1.smithi196.stdout:1/201: mkdir d1/db/d16/d44 0 2024-04-22T15:18:52.789 INFO:tasks.workunit.client.0.smithi096.stdout:8/600: creat d4/d2a/d7e/fd2 x:0 0 0 2024-04-22T15:18:52.791 INFO:tasks.workunit.client.1.smithi196.stdout:5/231: dread d1/f1f [0,4194304] 0 2024-04-22T15:18:52.791 INFO:tasks.workunit.client.0.smithi096.stdout:8/601: write d4/d10/f56 [1668285,125449] 0 2024-04-22T15:18:52.792 INFO:tasks.workunit.client.0.smithi096.stdout:3/479: getdents da/db/d10 0 2024-04-22T15:18:52.793 INFO:tasks.workunit.client.0.smithi096.stdout:3/480: chown da/db/d10/f33 8 1 2024-04-22T15:18:52.795 INFO:tasks.workunit.client.1.smithi196.stdout:6/245: dwrite dd/d1f/d37/f31 [0,4194304] 0 2024-04-22T15:18:52.796 INFO:tasks.workunit.client.1.smithi196.stdout:8/248: creat d2/d8/f4c x:0 0 0 2024-04-22T15:18:52.799 INFO:tasks.workunit.client.1.smithi196.stdout:5/232: dwrite d1/d17/f31 [0,4194304] 0 2024-04-22T15:18:52.801 INFO:tasks.workunit.client.0.smithi096.stdout:4/607: dread d5/f39 [4194304,4194304] 0 2024-04-22T15:18:52.802 INFO:tasks.workunit.client.0.smithi096.stdout:8/602: write d4/d1c/d40/da7/db6/f4f [2546418,108288] 0 2024-04-22T15:18:52.806 INFO:tasks.workunit.client.0.smithi096.stdout:3/481: mknod da/db/dc/d28/d69/d89/c98 0 2024-04-22T15:18:52.808 INFO:tasks.workunit.client.1.smithi196.stdout:9/223: rename db/d11/f4c to db/dd/d1b/f51 0 2024-04-22T15:18:52.816 INFO:tasks.workunit.client.0.smithi096.stdout:3/482: mkdir da/db/dc/d28/d96/d99 0 2024-04-22T15:18:52.817 INFO:tasks.workunit.client.0.smithi096.stdout:2/576: dwrite d3/db/f42 [0,4194304] 0 2024-04-22T15:18:52.819 INFO:tasks.workunit.client.0.smithi096.stdout:3/483: write da/db/de/d38/f78 [1466963,40069] 0 2024-04-22T15:18:52.819 INFO:tasks.workunit.client.0.smithi096.stdout:2/577: fdatasync d3/f2e 0 2024-04-22T15:18:52.819 INFO:tasks.workunit.client.0.smithi096.stdout:4/608: symlink d5/d7/d1a/lcf 0 2024-04-22T15:18:52.821 INFO:tasks.workunit.client.0.smithi096.stdout:3/484: write da/db/d12/d22/f61 [1180748,10902] 0 2024-04-22T15:18:52.821 INFO:tasks.workunit.client.0.smithi096.stdout:8/603: link d4/d1c/d40/d48/f94 d4/dc0/fd3 0 2024-04-22T15:18:52.823 INFO:tasks.workunit.client.1.smithi196.stdout:6/246: creat dd/d1f/f48 x:0 0 0 2024-04-22T15:18:52.824 INFO:tasks.workunit.client.1.smithi196.stdout:6/247: read - f9 zero size 2024-04-22T15:18:52.825 INFO:tasks.workunit.client.1.smithi196.stdout:6/248: chown dd/f15 18507013 1 2024-04-22T15:18:52.826 INFO:tasks.workunit.client.1.smithi196.stdout:8/249: stat d2/d8/c3c 0 2024-04-22T15:18:52.829 INFO:tasks.workunit.client.0.smithi096.stdout:4/609: creat d5/d5b/fd0 x:0 0 0 2024-04-22T15:18:52.830 INFO:tasks.workunit.client.1.smithi196.stdout:5/233: symlink d1/d4/l4d 0 2024-04-22T15:18:52.830 INFO:tasks.workunit.client.1.smithi196.stdout:6/249: dread dd/f26 [0,4194304] 0 2024-04-22T15:18:52.831 INFO:tasks.workunit.client.1.smithi196.stdout:7/215: dwrite d14/d26/f30 [0,4194304] 0 2024-04-22T15:18:52.832 INFO:tasks.workunit.client.1.smithi196.stdout:7/216: chown d14/d15/c1c 57166234 1 2024-04-22T15:18:52.832 INFO:tasks.workunit.client.0.smithi096.stdout:8/604: write d4/d1c/d40/d46/d58/faa [3182645,18687] 0 2024-04-22T15:18:52.832 INFO:tasks.workunit.client.1.smithi196.stdout:7/217: write d14/d25/d31/f38 [150430,72979] 0 2024-04-22T15:18:52.834 INFO:tasks.workunit.client.1.smithi196.stdout:7/218: write d14/d25/d31/f3f [749553,4422] 0 2024-04-22T15:18:52.836 INFO:tasks.workunit.client.1.smithi196.stdout:8/250: dread d2/dc/f10 [0,4194304] 0 2024-04-22T15:18:52.837 INFO:tasks.workunit.client.1.smithi196.stdout:9/224: creat db/dd/de/f52 x:0 0 0 2024-04-22T15:18:52.839 INFO:tasks.workunit.client.1.smithi196.stdout:3/230: getdents d0/d7/d2b 0 2024-04-22T15:18:52.840 INFO:tasks.workunit.client.0.smithi096.stdout:8/605: truncate d4/d10/d25/d4c/f87 717882 0 2024-04-22T15:18:52.840 INFO:tasks.workunit.client.0.smithi096.stdout:8/606: dread - d4/dc0/fd3 zero size 2024-04-22T15:18:52.841 INFO:tasks.workunit.client.0.smithi096.stdout:6/492: sync 2024-04-22T15:18:52.841 INFO:tasks.workunit.client.0.smithi096.stdout:9/521: sync 2024-04-22T15:18:52.842 INFO:tasks.workunit.client.0.smithi096.stdout:5/615: sync 2024-04-22T15:18:52.842 INFO:tasks.workunit.client.1.smithi196.stdout:8/251: dwrite d2/d5/f46 [0,4194304] 0 2024-04-22T15:18:52.843 INFO:tasks.workunit.client.1.smithi196.stdout:1/202: mkdir d1/db/dc/de/d43/d45 0 2024-04-22T15:18:52.844 INFO:tasks.workunit.client.0.smithi096.stdout:2/578: link d3/db/l48 d3/db/d2d/d46/d69/d7f/lb6 0 2024-04-22T15:18:52.845 INFO:tasks.workunit.client.0.smithi096.stdout:2/579: chown d3/db/df/c62 72739 1 2024-04-22T15:18:52.846 INFO:tasks.workunit.client.0.smithi096.stdout:2/580: readlink d3/db/d30/l97 0 2024-04-22T15:18:52.848 INFO:tasks.workunit.client.1.smithi196.stdout:1/203: dread d1/db/dc/d26/d2e/d32/d36/f3f [0,4194304] 0 2024-04-22T15:18:52.851 INFO:tasks.workunit.client.0.smithi096.stdout:4/610: dwrite d5/d7/f9 [0,4194304] 0 2024-04-22T15:18:52.852 INFO:tasks.workunit.client.1.smithi196.stdout:4/239: sync 2024-04-22T15:18:52.865 INFO:tasks.workunit.client.0.smithi096.stdout:2/581: dread d3/d1f/d2f/f67 [0,4194304] 0 2024-04-22T15:18:52.917 INFO:tasks.workunit.client.1.smithi196.stdout:0/231: sync 2024-04-22T15:18:52.921 INFO:tasks.workunit.client.0.smithi096.stdout:8/607: truncate d4/d1c/f43 3246636 0 2024-04-22T15:18:52.924 INFO:tasks.workunit.client.1.smithi196.stdout:9/225: symlink db/dd/de/d15/l53 0 2024-04-22T15:18:52.924 INFO:tasks.workunit.client.1.smithi196.stdout:9/226: stat db/dd/de/d15/d31/f49 0 2024-04-22T15:18:52.928 INFO:tasks.workunit.client.0.smithi096.stdout:4/611: symlink d5/d7/d1f/d84/d6d/d98/ld1 0 2024-04-22T15:18:52.929 INFO:tasks.workunit.client.1.smithi196.stdout:0/232: dread d3/fd [4194304,4194304] 0 2024-04-22T15:18:52.931 INFO:tasks.workunit.client.0.smithi096.stdout:2/582: symlink d3/db/d2d/d46/d7e/lb7 0 2024-04-22T15:18:52.932 INFO:tasks.workunit.client.1.smithi196.stdout:2/276: link d1/d13/d25/l64 d1/d13/l67 0 2024-04-22T15:18:52.933 INFO:tasks.workunit.client.1.smithi196.stdout:2/277: fsync d1/f7 0 2024-04-22T15:18:52.934 INFO:tasks.workunit.client.0.smithi096.stdout:8/608: mknod d4/d1c/d40/da7/cd4 0 2024-04-22T15:18:52.937 INFO:tasks.workunit.client.1.smithi196.stdout:1/204: rename d1/db/dc/de/d43/d45 to d1/db/d16/d1e/d46 0 2024-04-22T15:18:52.938 INFO:tasks.workunit.client.0.smithi096.stdout:2/583: dwrite d3/db/d2d/d46/d69/f6c [0,4194304] 0 2024-04-22T15:18:52.938 INFO:tasks.workunit.client.0.smithi096.stdout:9/522: creat d1/df/d95/d9d/fa8 x:0 0 0 2024-04-22T15:18:52.939 INFO:tasks.workunit.client.1.smithi196.stdout:4/240: mknod d6/d14/d18/c45 0 2024-04-22T15:18:52.939 INFO:tasks.workunit.client.0.smithi096.stdout:2/584: dread - d3/db/d2d/d46/d69/fa1 zero size 2024-04-22T15:18:52.940 INFO:tasks.workunit.client.0.smithi096.stdout:5/616: symlink d4/d11/d3d/d4d/d59/db0/db2/lc8 0 2024-04-22T15:18:52.941 INFO:tasks.workunit.client.0.smithi096.stdout:2/585: chown d3/d1f/d2f/f47 64985 1 2024-04-22T15:18:52.945 INFO:tasks.workunit.client.1.smithi196.stdout:4/241: dwrite d6/d14/f33 [0,4194304] 0 2024-04-22T15:18:52.947 INFO:tasks.workunit.client.1.smithi196.stdout:9/227: creat db/dd/de/f54 x:0 0 0 2024-04-22T15:18:52.947 INFO:tasks.workunit.client.1.smithi196.stdout:4/242: read d6/d13/d15/d2c/d35/f3c [7398648,93892] 0 2024-04-22T15:18:52.948 INFO:tasks.workunit.client.1.smithi196.stdout:4/243: dread - d6/d13/f2e zero size 2024-04-22T15:18:52.949 INFO:tasks.workunit.client.1.smithi196.stdout:3/231: truncate d0/d6/db/dc/d18/f42 3916706 0 2024-04-22T15:18:52.949 INFO:tasks.workunit.client.1.smithi196.stdout:4/244: fdatasync d6/f2b 0 2024-04-22T15:18:52.950 INFO:tasks.workunit.client.1.smithi196.stdout:8/252: getdents d2/d8/d49 0 2024-04-22T15:18:52.956 INFO:tasks.workunit.client.1.smithi196.stdout:8/253: dwrite d2/dc/f2f [0,4194304] 0 2024-04-22T15:18:52.956 INFO:tasks.workunit.client.0.smithi096.stdout:9/523: unlink d1/d4/d9/dd/f92 0 2024-04-22T15:18:52.957 INFO:tasks.workunit.client.1.smithi196.stdout:8/254: stat d2/d5/d2e/l33 0 2024-04-22T15:18:52.957 INFO:tasks.workunit.client.0.smithi096.stdout:6/493: truncate d3/d28/dd/d11/d2c/f5e 20886 0 2024-04-22T15:18:52.958 INFO:tasks.workunit.client.1.smithi196.stdout:8/255: stat d2/d5/d2e/f43 0 2024-04-22T15:18:52.958 INFO:tasks.workunit.client.1.smithi196.stdout:8/256: chown d2 235018441 1 2024-04-22T15:18:52.960 INFO:tasks.workunit.client.1.smithi196.stdout:5/234: dwrite d1/d4/f26 [0,4194304] 0 2024-04-22T15:18:52.961 INFO:tasks.workunit.client.0.smithi096.stdout:5/617: unlink d4/d2c/d2f/f9b 0 2024-04-22T15:18:52.962 INFO:tasks.workunit.client.1.smithi196.stdout:2/278: creat d1/d1d/f68 x:0 0 0 2024-04-22T15:18:52.964 INFO:tasks.workunit.client.1.smithi196.stdout:6/250: sync 2024-04-22T15:18:52.970 INFO:tasks.workunit.client.0.smithi096.stdout:8/609: dread d4/d71/f77 [0,4194304] 0 2024-04-22T15:18:52.978 INFO:tasks.workunit.client.0.smithi096.stdout:5/618: rename d4/d2c/d2f/d31/f9a to d4/d11/d3d/d35/fc9 0 2024-04-22T15:18:52.980 INFO:tasks.workunit.client.1.smithi196.stdout:9/228: symlink db/l55 0 2024-04-22T15:18:52.981 INFO:tasks.workunit.client.1.smithi196.stdout:1/205: dread d1/db/dc/d26/f27 [0,4194304] 0 2024-04-22T15:18:52.983 INFO:tasks.workunit.client.0.smithi096.stdout:2/586: dwrite d3/db/df/f61 [0,4194304] 0 2024-04-22T15:18:52.986 INFO:tasks.workunit.client.1.smithi196.stdout:3/232: rename d0/d7/d2b to d0/d6/db/dc/d4a 0 2024-04-22T15:18:52.989 INFO:tasks.workunit.client.0.smithi096.stdout:6/494: creat d3/d28/d4d/d82/d9e/da3/fab x:0 0 0 2024-04-22T15:18:52.990 INFO:tasks.workunit.client.1.smithi196.stdout:0/233: fsync d3/de/d14/f4a 0 2024-04-22T15:18:52.994 INFO:tasks.workunit.client.1.smithi196.stdout:8/257: rmdir d2/d5/d2e 39 2024-04-22T15:18:52.995 INFO:tasks.workunit.client.1.smithi196.stdout:5/235: creat d1/d14/d3e/f4e x:0 0 0 2024-04-22T15:18:52.997 INFO:tasks.workunit.client.1.smithi196.stdout:6/251: dread - dd/f23 zero size 2024-04-22T15:18:52.997 INFO:tasks.workunit.client.1.smithi196.stdout:0/234: dwrite d3/de/d14/f39 [0,4194304] 0 2024-04-22T15:18:52.998 INFO:tasks.workunit.client.1.smithi196.stdout:0/235: readlink d3/de/d14/d29/l3e 0 2024-04-22T15:18:52.999 INFO:tasks.workunit.client.1.smithi196.stdout:0/236: chown d3/d9/d36 1130597025 1 2024-04-22T15:18:53.000 INFO:tasks.workunit.client.1.smithi196.stdout:4/245: creat d6/d14/d18/d26/d32/f46 x:0 0 0 2024-04-22T15:18:53.000 INFO:tasks.workunit.client.1.smithi196.stdout:3/233: mknod d0/d7/c4b 0 2024-04-22T15:18:53.001 INFO:tasks.workunit.client.1.smithi196.stdout:2/279: mknod d1/d13/c69 0 2024-04-22T15:18:53.003 INFO:tasks.workunit.client.1.smithi196.stdout:7/219: getdents d14/d26/d28 0 2024-04-22T15:18:53.003 INFO:tasks.workunit.client.1.smithi196.stdout:1/206: mknod d1/db/dc/d26/d31/c47 0 2024-04-22T15:18:53.004 INFO:tasks.workunit.client.1.smithi196.stdout:4/246: rmdir d6/d14/d18 39 2024-04-22T15:18:53.005 INFO:tasks.workunit.client.1.smithi196.stdout:8/258: creat d2/d3f/d4b/f4d x:0 0 0 2024-04-22T15:18:53.005 INFO:tasks.workunit.client.1.smithi196.stdout:3/234: creat d0/d6/db/dc/d18/f4c x:0 0 0 2024-04-22T15:18:53.006 INFO:tasks.workunit.client.1.smithi196.stdout:2/280: mkdir d1/d1d/d6a 0 2024-04-22T15:18:53.006 INFO:tasks.workunit.client.1.smithi196.stdout:5/236: dread d1/f35 [0,4194304] 0 2024-04-22T15:18:53.007 INFO:tasks.workunit.client.1.smithi196.stdout:7/220: creat d14/d25/d31/f40 x:0 0 0 2024-04-22T15:18:53.008 INFO:tasks.workunit.client.1.smithi196.stdout:1/207: creat d1/db/dc/de/d43/f48 x:0 0 0 2024-04-22T15:18:53.009 INFO:tasks.workunit.client.1.smithi196.stdout:4/247: mknod d6/d13/c47 0 2024-04-22T15:18:53.010 INFO:tasks.workunit.client.1.smithi196.stdout:7/221: truncate d14/d15/f1a 2231370 0 2024-04-22T15:18:53.013 INFO:tasks.workunit.client.1.smithi196.stdout:0/237: dwrite d3/de/d1c/f31 [0,4194304] 0 2024-04-22T15:18:53.013 INFO:tasks.workunit.client.1.smithi196.stdout:1/208: unlink d1/db/dc/d26/l3b 0 2024-04-22T15:18:53.013 INFO:tasks.workunit.client.1.smithi196.stdout:8/259: link d2/f3e d2/d3f/f4e 0 2024-04-22T15:18:53.015 INFO:tasks.workunit.client.1.smithi196.stdout:7/222: symlink d14/d25/d31/l41 0 2024-04-22T15:18:53.018 INFO:tasks.workunit.client.1.smithi196.stdout:2/281: dwrite d1/f10 [0,4194304] 0 2024-04-22T15:18:53.018 INFO:tasks.workunit.client.1.smithi196.stdout:4/248: symlink d6/d14/d18/d40/l48 0 2024-04-22T15:18:53.018 INFO:tasks.workunit.client.1.smithi196.stdout:7/223: write d14/f1e [1927815,24071] 0 2024-04-22T15:18:53.023 INFO:tasks.workunit.client.1.smithi196.stdout:0/238: dwrite d3/f47 [0,4194304] 0 2024-04-22T15:18:53.024 INFO:tasks.workunit.client.1.smithi196.stdout:6/252: dread fc [4194304,4194304] 0 2024-04-22T15:18:53.025 INFO:tasks.workunit.client.0.smithi096.stdout:4/612: getdents d5/d7/d1f/d84/d6d 0 2024-04-22T15:18:53.027 INFO:tasks.workunit.client.0.smithi096.stdout:2/587: rmdir d3/db/d2d/d46/d7e/d84 39 2024-04-22T15:18:53.027 INFO:tasks.workunit.client.0.smithi096.stdout:8/610: dread d4/fd [0,4194304] 0 2024-04-22T15:18:53.028 INFO:tasks.workunit.client.0.smithi096.stdout:4/613: chown d5/d7/d2c/d3d/d4f/d9c/ca7 2 1 2024-04-22T15:18:53.028 INFO:tasks.workunit.client.1.smithi196.stdout:7/224: creat d14/d25/f42 x:0 0 0 2024-04-22T15:18:53.029 INFO:tasks.workunit.client.1.smithi196.stdout:6/253: dread dd/fe [0,4194304] 0 2024-04-22T15:18:53.029 INFO:tasks.workunit.client.1.smithi196.stdout:7/225: chown d14/d26/d2a/d2b/f2d 154 1 2024-04-22T15:18:53.030 INFO:tasks.workunit.client.0.smithi096.stdout:4/614: chown d5/d7/d1a/f40 115 1 2024-04-22T15:18:53.030 INFO:tasks.workunit.client.1.smithi196.stdout:6/254: chown dd/d1f/d37/f31 2940638 1 2024-04-22T15:18:53.033 INFO:tasks.workunit.client.0.smithi096.stdout:6/495: mkdir d3/d28/dd/d11/d2c/d3b/dac 0 2024-04-22T15:18:53.037 INFO:tasks.workunit.client.0.smithi096.stdout:2/588: creat d3/db/df/d24/d58/d5f/d6e/fb8 x:0 0 0 2024-04-22T15:18:53.037 INFO:tasks.workunit.client.1.smithi196.stdout:5/237: dread d1/d17/f21 [0,4194304] 0 2024-04-22T15:18:53.037 INFO:tasks.workunit.client.0.smithi096.stdout:2/589: stat d3/f2e 0 2024-04-22T15:18:53.038 INFO:tasks.workunit.client.1.smithi196.stdout:5/238: chown d1/d14/f42 235 1 2024-04-22T15:18:53.040 INFO:tasks.workunit.client.1.smithi196.stdout:7/226: write fa [585736,22766] 0 2024-04-22T15:18:53.041 INFO:tasks.workunit.client.0.smithi096.stdout:5/619: creat d4/d2c/d2f/d31/fca x:0 0 0 2024-04-22T15:18:53.042 INFO:tasks.workunit.client.1.smithi196.stdout:6/255: creat dd/d1f/d38/d40/f49 x:0 0 0 2024-04-22T15:18:53.043 INFO:tasks.workunit.client.1.smithi196.stdout:5/239: rename d1/d17/d1c/l4b to d1/d17/d1c/l4f 0 2024-04-22T15:18:53.044 INFO:tasks.workunit.client.0.smithi096.stdout:6/496: fsync d3/d28/dd/d11/f41 0 2024-04-22T15:18:53.045 INFO:tasks.workunit.client.1.smithi196.stdout:5/240: truncate d1/d17/d1c/f3b 150202 0 2024-04-22T15:18:53.045 INFO:tasks.workunit.client.0.smithi096.stdout:2/590: creat d3/db/d2d/d46/d69/fb9 x:0 0 0 2024-04-22T15:18:53.046 INFO:tasks.workunit.client.1.smithi196.stdout:6/256: symlink dd/d1f/d37/d3c/l4a 0 2024-04-22T15:18:53.047 INFO:tasks.workunit.client.1.smithi196.stdout:7/227: dwrite d14/d25/f42 [0,4194304] 0 2024-04-22T15:18:53.047 INFO:tasks.workunit.client.0.smithi096.stdout:8/611: rename d4/d10/d4d/cb0 to d4/d10/cd5 0 2024-04-22T15:18:53.048 INFO:tasks.workunit.client.1.smithi196.stdout:6/257: read - dd/d1f/d38/d40/f43 zero size 2024-04-22T15:18:53.049 INFO:tasks.workunit.client.1.smithi196.stdout:6/258: dread - dd/d1f/d38/d40/f43 zero size 2024-04-22T15:18:53.050 INFO:tasks.workunit.client.1.smithi196.stdout:5/241: chown d1/d14/l37 1 1 2024-04-22T15:18:53.052 INFO:tasks.workunit.client.1.smithi196.stdout:6/259: creat dd/d1f/d38/f4b x:0 0 0 2024-04-22T15:18:53.053 INFO:tasks.workunit.client.1.smithi196.stdout:4/249: dread d6/d14/f27 [0,4194304] 0 2024-04-22T15:18:53.055 INFO:tasks.workunit.client.0.smithi096.stdout:2/591: mknod d3/d13/cba 0 2024-04-22T15:18:53.057 INFO:tasks.workunit.client.1.smithi196.stdout:9/229: sync 2024-04-22T15:18:53.057 INFO:tasks.workunit.client.1.smithi196.stdout:7/228: fdatasync fd 0 2024-04-22T15:18:53.058 INFO:tasks.workunit.client.0.smithi096.stdout:8/612: write d4/d10/f21 [5112364,100628] 0 2024-04-22T15:18:53.060 INFO:tasks.workunit.client.1.smithi196.stdout:0/239: dread d3/d6/f33 [0,4194304] 0 2024-04-22T15:18:53.061 INFO:tasks.workunit.client.1.smithi196.stdout:6/260: dwrite dd/f15 [0,4194304] 0 2024-04-22T15:18:53.062 INFO:tasks.workunit.client.0.smithi096.stdout:6/497: symlink d3/lad 0 2024-04-22T15:18:53.062 INFO:tasks.workunit.client.1.smithi196.stdout:4/250: dread d6/d14/f33 [0,4194304] 0 2024-04-22T15:18:53.063 INFO:tasks.workunit.client.1.smithi196.stdout:4/251: chown d6/d14/d18/d26/d32/f46 1995 1 2024-04-22T15:18:53.064 INFO:tasks.workunit.client.1.smithi196.stdout:4/252: stat d6/d13/f25 0 2024-04-22T15:18:53.066 INFO:tasks.workunit.client.0.smithi096.stdout:8/613: mknod d4/d10/d25/d38/d8f/d9c/cd6 0 2024-04-22T15:18:53.071 INFO:tasks.workunit.client.0.smithi096.stdout:6/498: rmdir d3/d28/d54 39 2024-04-22T15:18:53.072 INFO:tasks.workunit.client.1.smithi196.stdout:7/229: mkdir d14/d26/d28/d43 0 2024-04-22T15:18:53.073 INFO:tasks.workunit.client.0.smithi096.stdout:2/592: creat d3/db/df/d98/db3/fbb x:0 0 0 2024-04-22T15:18:53.073 INFO:tasks.workunit.client.1.smithi196.stdout:0/240: rmdir d3/de 39 2024-04-22T15:18:53.074 INFO:tasks.workunit.client.0.smithi096.stdout:2/593: read d3/db/df/f61 [1001091,96092] 0 2024-04-22T15:18:53.074 INFO:tasks.workunit.client.0.smithi096.stdout:2/594: stat d3/db/df/c62 0 2024-04-22T15:18:53.075 INFO:tasks.workunit.client.1.smithi196.stdout:3/235: rename d0/d6 to d0/d15/d4d 0 2024-04-22T15:18:53.075 INFO:tasks.workunit.client.0.smithi096.stdout:2/595: stat d3/db/df/d10/f3b 0 2024-04-22T15:18:53.078 INFO:tasks.workunit.client.1.smithi196.stdout:3/236: fdatasync d0/d7/f27 0 2024-04-22T15:18:53.078 INFO:tasks.workunit.client.1.smithi196.stdout:6/261: write fc [6097374,106608] 0 2024-04-22T15:18:53.078 INFO:tasks.workunit.client.1.smithi196.stdout:7/230: dwrite d14/d26/d2a/d2b/f2d [0,4194304] 0 2024-04-22T15:18:53.079 INFO:tasks.workunit.client.0.smithi096.stdout:8/614: creat d4/d10/d25/d38/d8f/fd7 x:0 0 0 2024-04-22T15:18:53.079 INFO:tasks.workunit.client.0.smithi096.stdout:5/620: getdents d4/d11/d3d/d35 0 2024-04-22T15:18:53.080 INFO:tasks.workunit.client.0.smithi096.stdout:5/621: stat d4/d11/d25 0 2024-04-22T15:18:53.082 INFO:tasks.workunit.client.0.smithi096.stdout:2/596: dwrite d3/f49 [0,4194304] 0 2024-04-22T15:18:53.083 INFO:tasks.workunit.client.1.smithi196.stdout:3/237: dwrite d0/d15/d4d/db/dc/d18/f4c [0,4194304] 0 2024-04-22T15:18:53.091 INFO:tasks.workunit.client.1.smithi196.stdout:8/260: write d2/d5/f19 [424142,67588] 0 2024-04-22T15:18:53.093 INFO:tasks.workunit.client.1.smithi196.stdout:6/262: creat dd/d20/f4c x:0 0 0 2024-04-22T15:18:53.094 INFO:tasks.workunit.client.1.smithi196.stdout:6/263: chown dd/c2a 2116783774 1 2024-04-22T15:18:53.095 INFO:tasks.workunit.client.1.smithi196.stdout:6/264: chown dd/d1f/d38/d40/f42 413 1 2024-04-22T15:18:53.096 INFO:tasks.workunit.client.1.smithi196.stdout:6/265: truncate f2 2678474 0 2024-04-22T15:18:53.096 INFO:tasks.workunit.client.1.smithi196.stdout:6/266: dread - dd/d2b/f3e zero size 2024-04-22T15:18:53.097 INFO:tasks.workunit.client.1.smithi196.stdout:7/231: unlink d14/l3c 0 2024-04-22T15:18:53.097 INFO:tasks.workunit.client.1.smithi196.stdout:6/267: chown dd/d20/f24 442185 1 2024-04-22T15:18:53.098 INFO:tasks.workunit.client.1.smithi196.stdout:2/282: write d1/f18 [263785,111208] 0 2024-04-22T15:18:53.100 INFO:tasks.workunit.client.0.smithi096.stdout:2/597: read - d3/db/df/d10/f55 zero size 2024-04-22T15:18:53.101 INFO:tasks.workunit.client.1.smithi196.stdout:1/209: truncate d1/db/dc/d26/f30 420419 0 2024-04-22T15:18:53.102 INFO:tasks.workunit.client.1.smithi196.stdout:9/230: unlink db/dd/de/d15/d31/l4f 0 2024-04-22T15:18:53.102 INFO:tasks.workunit.client.1.smithi196.stdout:1/210: truncate d1/db/d2d/f42 250467 0 2024-04-22T15:18:53.104 INFO:tasks.workunit.client.1.smithi196.stdout:4/253: unlink d6/d14/d18/d26/f2f 0 2024-04-22T15:18:53.104 INFO:tasks.workunit.client.1.smithi196.stdout:2/283: dwrite d1/d13/d1f/f20 [0,4194304] 0 2024-04-22T15:18:53.107 INFO:tasks.workunit.client.1.smithi196.stdout:3/238: symlink d0/d15/d4d/db/dc/d4a/l4e 0 2024-04-22T15:18:53.107 INFO:tasks.workunit.client.1.smithi196.stdout:2/284: write d1/d13/d1f/f20 [1311164,78708] 0 2024-04-22T15:18:53.113 INFO:tasks.workunit.client.1.smithi196.stdout:9/231: dwrite db/dd/de/f54 [0,4194304] 0 2024-04-22T15:18:53.113 INFO:tasks.workunit.client.1.smithi196.stdout:8/261: write d2/dc/f18 [1853006,11924] 0 2024-04-22T15:18:53.115 INFO:tasks.workunit.client.1.smithi196.stdout:0/241: dread d3/de/f37 [0,4194304] 0 2024-04-22T15:18:53.116 INFO:tasks.workunit.client.0.smithi096.stdout:8/615: link d4/d2a/d7e/l88 d4/d10/d25/d38/ld8 0 2024-04-22T15:18:53.118 INFO:tasks.workunit.client.0.smithi096.stdout:4/615: truncate d5/d7/d2c/d67/fb3 1141749 0 2024-04-22T15:18:53.122 INFO:tasks.workunit.client.1.smithi196.stdout:9/232: read f4 [5590951,15717] 0 2024-04-22T15:18:53.125 INFO:tasks.workunit.client.0.smithi096.stdout:8/616: dwrite d4/d10/d25/d4c/d76/fb1 [0,4194304] 0 2024-04-22T15:18:53.125 INFO:tasks.workunit.client.1.smithi196.stdout:5/242: write d1/d9/f11 [612689,90039] 0 2024-04-22T15:18:53.135 INFO:tasks.workunit.client.0.smithi096.stdout:2/598: dread d3/f70 [0,4194304] 0 2024-04-22T15:18:53.136 INFO:tasks.workunit.client.0.smithi096.stdout:6/499: getdents d3/d28/d54/d68/d6e 0 2024-04-22T15:18:53.139 INFO:tasks.workunit.client.0.smithi096.stdout:8/617: truncate d4/d1c/d40/d46/f7d 497943 0 2024-04-22T15:18:53.139 INFO:tasks.workunit.client.0.smithi096.stdout:8/618: chown d4/d2a/d7e 30 1 2024-04-22T15:18:53.142 INFO:tasks.workunit.client.1.smithi196.stdout:3/239: rmdir d0/d15/d4d/db/dc/d4a 39 2024-04-22T15:18:53.143 INFO:tasks.workunit.client.0.smithi096.stdout:2/599: mknod d3/db/d2d/d46/d7e/d8d/cbc 0 2024-04-22T15:18:53.144 INFO:tasks.workunit.client.1.smithi196.stdout:2/285: dwrite d1/f1b [0,4194304] 0 2024-04-22T15:18:53.145 INFO:tasks.workunit.client.0.smithi096.stdout:2/600: chown d3/db/df/d24/d58/d5f/d6e/fac 1894619 1 2024-04-22T15:18:53.147 INFO:tasks.workunit.client.0.smithi096.stdout:8/619: chown d4/c4a 50682 1 2024-04-22T15:18:53.148 INFO:tasks.workunit.client.0.smithi096.stdout:5/622: dread d4/d11/d3d/d4d/d59/d5a/d5c/f89 [0,4194304] 0 2024-04-22T15:18:53.150 INFO:tasks.workunit.client.0.smithi096.stdout:2/601: mknod d3/db/df/d24/cbd 0 2024-04-22T15:18:53.153 INFO:tasks.workunit.client.0.smithi096.stdout:6/500: truncate d3/d28/dd/d11/d2c/d44/d51/f60 1027263 0 2024-04-22T15:18:53.154 INFO:tasks.workunit.client.1.smithi196.stdout:0/242: write d3/de/d1c/d27/f38 [4273202,99664] 0 2024-04-22T15:18:53.155 INFO:tasks.workunit.client.0.smithi096.stdout:8/620: dwrite d4/d2a/d7e/fa6 [0,4194304] 0 2024-04-22T15:18:53.159 INFO:tasks.workunit.client.0.smithi096.stdout:2/602: unlink d3/db/d30/f82 0 2024-04-22T15:18:53.160 INFO:tasks.workunit.client.0.smithi096.stdout:2/603: readlink d3/db/df/l3c 0 2024-04-22T15:18:53.162 INFO:tasks.workunit.client.1.smithi196.stdout:9/233: symlink db/dd/de/l56 0 2024-04-22T15:18:53.163 INFO:tasks.workunit.client.1.smithi196.stdout:9/234: chown db/dd/de/f42 0 1 2024-04-22T15:18:53.166 INFO:tasks.workunit.client.1.smithi196.stdout:6/268: dwrite dd/fe [0,4194304] 0 2024-04-22T15:18:53.167 INFO:tasks.workunit.client.1.smithi196.stdout:6/269: dread - dd/d1f/f36 zero size 2024-04-22T15:18:53.168 INFO:tasks.workunit.client.1.smithi196.stdout:6/270: fsync dd/d20/f22 0 2024-04-22T15:18:53.169 INFO:tasks.workunit.client.0.smithi096.stdout:6/501: creat d3/d28/d4d/d82/d9e/fae x:0 0 0 2024-04-22T15:18:53.169 INFO:tasks.workunit.client.0.smithi096.stdout:2/604: rename d3/db/df/c62 to d3/d1f/d2f/cbe 0 2024-04-22T15:18:53.170 INFO:tasks.workunit.client.0.smithi096.stdout:2/605: fdatasync d3/db/d2d/fad 0 2024-04-22T15:18:53.172 INFO:tasks.workunit.client.0.smithi096.stdout:0/747: sync 2024-04-22T15:18:53.174 INFO:tasks.workunit.client.1.smithi196.stdout:2/286: rmdir d1/d23/d27/d2e 39 2024-04-22T15:18:53.175 INFO:tasks.workunit.client.1.smithi196.stdout:6/271: dwrite f4 [0,4194304] 0 2024-04-22T15:18:53.176 INFO:tasks.workunit.client.1.smithi196.stdout:8/262: mkdir d2/d5/d4f 0 2024-04-22T15:18:53.177 INFO:tasks.workunit.client.0.smithi096.stdout:6/502: dwrite d3/d3a/d75/d83/f98 [0,4194304] 0 2024-04-22T15:18:53.177 INFO:tasks.workunit.client.1.smithi196.stdout:1/211: dwrite d1/db/f29 [0,4194304] 0 2024-04-22T15:18:53.177 INFO:tasks.workunit.client.1.smithi196.stdout:6/272: chown dd/d1f/d37/d3c 225 1 2024-04-22T15:18:53.179 INFO:tasks.workunit.client.0.smithi096.stdout:4/616: dwrite d5/d7/f20 [0,4194304] 0 2024-04-22T15:18:53.181 INFO:tasks.workunit.client.1.smithi196.stdout:0/243: creat d3/de/d14/d29/f57 x:0 0 0 2024-04-22T15:18:53.186 INFO:tasks.workunit.client.1.smithi196.stdout:5/243: mkdir d1/d17/d50 0 2024-04-22T15:18:53.190 INFO:tasks.workunit.client.1.smithi196.stdout:7/232: truncate d14/d26/f35 197606 0 2024-04-22T15:18:53.198 INFO:tasks.workunit.client.1.smithi196.stdout:3/240: link d0/f3 d0/d15/d4d/db/dc/d18/f4f 0 2024-04-22T15:18:53.198 INFO:tasks.workunit.client.1.smithi196.stdout:4/254: truncate d6/d14/f27 4609906 0 2024-04-22T15:18:53.199 INFO:tasks.workunit.client.1.smithi196.stdout:4/255: dread - d6/d14/f2a zero size 2024-04-22T15:18:53.200 INFO:tasks.workunit.client.1.smithi196.stdout:2/287: write d1/d23/d27/d2e/f58 [1016448,84716] 0 2024-04-22T15:18:53.205 INFO:tasks.workunit.client.1.smithi196.stdout:1/212: rename d1/db/d2d to d1/db/d16/d1e/d49 0 2024-04-22T15:18:53.205 INFO:tasks.workunit.client.1.smithi196.stdout:5/244: fdatasync d1/d17/d1c/f3b 0 2024-04-22T15:18:53.205 INFO:tasks.workunit.client.1.smithi196.stdout:8/263: chown d2/dc/fd 12965 1 2024-04-22T15:18:53.206 INFO:tasks.workunit.client.1.smithi196.stdout:5/245: fdatasync d1/d9/f11 0 2024-04-22T15:18:53.206 INFO:tasks.workunit.client.1.smithi196.stdout:8/264: dread - d2/dc/f28 zero size 2024-04-22T15:18:53.206 INFO:tasks.workunit.client.1.smithi196.stdout:1/213: write d1/db/dc/d26/d2e/f3e [845036,102221] 0 2024-04-22T15:18:53.208 INFO:tasks.workunit.client.1.smithi196.stdout:5/246: readlink d1/d17/d1a/d2f/d30/l49 0 2024-04-22T15:18:53.209 INFO:tasks.workunit.client.1.smithi196.stdout:4/256: mknod d6/d14/d18/d40/c49 0 2024-04-22T15:18:53.210 INFO:tasks.workunit.client.1.smithi196.stdout:4/257: write d6/d14/d18/d26/f43 [265530,126329] 0 2024-04-22T15:18:53.211 INFO:tasks.workunit.client.1.smithi196.stdout:0/244: link d3/f47 d3/d6/dc/f58 0 2024-04-22T15:18:53.212 INFO:tasks.workunit.client.1.smithi196.stdout:5/247: dread d1/d17/f31 [0,4194304] 0 2024-04-22T15:18:53.212 INFO:tasks.workunit.client.1.smithi196.stdout:0/245: fdatasync d3/f5 0 2024-04-22T15:18:53.213 INFO:tasks.workunit.client.1.smithi196.stdout:7/233: rename d14/d15/c1c to d14/d26/d2a/c44 0 2024-04-22T15:18:53.214 INFO:tasks.workunit.client.1.smithi196.stdout:7/234: truncate d14/f33 138052 0 2024-04-22T15:18:53.214 INFO:tasks.workunit.client.1.smithi196.stdout:2/288: mknod d1/d13/d25/d2a/c6b 0 2024-04-22T15:18:53.229 INFO:tasks.workunit.client.0.smithi096.stdout:5/623: getdents d4/d11/d3d/d4d/d59/db0 0 2024-04-22T15:18:53.231 INFO:tasks.workunit.client.1.smithi196.stdout:8/265: symlink d2/d8/d49/l50 0 2024-04-22T15:18:53.234 INFO:tasks.workunit.client.1.smithi196.stdout:4/258: creat d6/d13/d15/d2c/d35/d44/f4a x:0 0 0 2024-04-22T15:18:53.234 INFO:tasks.workunit.client.1.smithi196.stdout:4/259: dread - d6/f42 zero size 2024-04-22T15:18:53.236 INFO:tasks.workunit.client.1.smithi196.stdout:9/235: rename db/d18 to db/d11/d29/d57 0 2024-04-22T15:18:53.242 INFO:tasks.workunit.client.0.smithi096.stdout:4/617: rename d5/d7/d1f/d84/lad to d5/d7/d2c/d3d/d4f/d9c/ld2 0 2024-04-22T15:18:53.243 INFO:tasks.workunit.client.0.smithi096.stdout:4/618: chown d5/d7/d2c/d43/c57 1186912 1 2024-04-22T15:18:53.248 INFO:tasks.workunit.client.0.smithi096.stdout:6/503: dread f2 [0,4194304] 0 2024-04-22T15:18:53.249 INFO:tasks.workunit.client.0.smithi096.stdout:6/504: chown d3/d28/dd/d11/d2c/l35 702 1 2024-04-22T15:18:53.250 INFO:tasks.workunit.client.0.smithi096.stdout:6/505: read f1 [2405846,33677] 0 2024-04-22T15:18:53.251 INFO:tasks.workunit.client.0.smithi096.stdout:6/506: write d3/d28/f21 [542213,20810] 0 2024-04-22T15:18:53.251 INFO:tasks.workunit.client.0.smithi096.stdout:2/606: mkdir d3/d13/db4/dbf 0 2024-04-22T15:18:53.252 INFO:tasks.workunit.client.0.smithi096.stdout:6/507: dread - d3/d3a/d75/f93 zero size 2024-04-22T15:18:53.253 INFO:tasks.workunit.client.0.smithi096.stdout:0/748: mkdir d7/d42/d4c/d55/d9c/df7/dfb 0 2024-04-22T15:18:53.255 INFO:tasks.workunit.client.0.smithi096.stdout:4/619: rename d5/c8 to d5/d7/d1a/d25/cd3 0 2024-04-22T15:18:53.255 INFO:tasks.workunit.client.0.smithi096.stdout:4/620: chown d5/d7/d1a/d25 1496910 1 2024-04-22T15:18:53.256 INFO:tasks.workunit.client.0.smithi096.stdout:5/624: fdatasync d4/f3c 0 2024-04-22T15:18:53.258 INFO:tasks.workunit.client.0.smithi096.stdout:6/508: truncate d3/d28/f38 5126004 0 2024-04-22T15:18:53.259 INFO:tasks.workunit.client.0.smithi096.stdout:6/509: chown d3/l72 1473 1 2024-04-22T15:18:53.259 INFO:tasks.workunit.client.0.smithi096.stdout:0/749: mkdir d7/d42/d4c/d55/d9c/dfc 0 2024-04-22T15:18:53.261 INFO:tasks.workunit.client.0.smithi096.stdout:4/621: read - d5/d7/d1f/d84/d6d/d98/f9b zero size 2024-04-22T15:18:53.264 INFO:tasks.workunit.client.0.smithi096.stdout:0/750: mkdir d7/d42/dfd 0 2024-04-22T15:18:53.266 INFO:tasks.workunit.client.0.smithi096.stdout:2/607: mknod d3/db/df/d24/d58/d5f/d6e/cc0 0 2024-04-22T15:18:53.269 INFO:tasks.workunit.client.0.smithi096.stdout:4/622: dwrite d5/d7/d2c/d3d/d4f/f91 [0,4194304] 0 2024-04-22T15:18:53.270 INFO:tasks.workunit.client.0.smithi096.stdout:0/751: mknod d7/de/d3a/cfe 0 2024-04-22T15:18:53.270 INFO:tasks.workunit.client.0.smithi096.stdout:4/623: chown d5/d7/d1f/d84/c6b 544459 1 2024-04-22T15:18:53.272 INFO:tasks.workunit.client.0.smithi096.stdout:4/624: stat d5/d7/d1a/d25/cb5 0 2024-04-22T15:18:53.273 INFO:tasks.workunit.client.0.smithi096.stdout:5/625: dread d4/f7 [0,4194304] 0 2024-04-22T15:18:53.274 INFO:tasks.workunit.client.0.smithi096.stdout:6/510: rename d3/d28/dd/d11/d2c/d6f/d7c to d3/d28/daf 0 2024-04-22T15:18:53.274 INFO:tasks.workunit.client.0.smithi096.stdout:0/752: dread d7/de/d5b/d95/dc/d26/fb5 [0,4194304] 0 2024-04-22T15:18:53.275 INFO:tasks.workunit.client.0.smithi096.stdout:6/511: fsync d3/d28/d4d/d82/d9e/f92 0 2024-04-22T15:18:53.278 INFO:tasks.workunit.client.0.smithi096.stdout:4/625: creat d5/d7/d1a/d8b/fd4 x:0 0 0 2024-04-22T15:18:53.278 INFO:tasks.workunit.client.1.smithi196.stdout:4/260: mknod d6/d10/c4b 0 2024-04-22T15:18:53.279 INFO:tasks.workunit.client.1.smithi196.stdout:3/241: rename d0/d7/f27 to d0/d15/d4d/db/dc/d4a/f50 0 2024-04-22T15:18:53.280 INFO:tasks.workunit.client.0.smithi096.stdout:5/626: mknod d4/d2c/d2f/d30/ccb 0 2024-04-22T15:18:53.281 INFO:tasks.workunit.client.0.smithi096.stdout:6/512: rmdir d3/d28/d4d/d82 39 2024-04-22T15:18:53.281 INFO:tasks.workunit.client.1.smithi196.stdout:7/235: dread f2 [0,4194304] 0 2024-04-22T15:18:53.282 INFO:tasks.workunit.client.1.smithi196.stdout:9/236: dwrite db/dd/de/f10 [0,4194304] 0 2024-04-22T15:18:53.282 INFO:tasks.workunit.client.1.smithi196.stdout:0/246: truncate d3/d6/f48 496308 0 2024-04-22T15:18:53.282 INFO:tasks.workunit.client.0.smithi096.stdout:0/753: dwrite d7/da7/fd1 [0,4194304] 0 2024-04-22T15:18:53.283 INFO:tasks.workunit.client.1.smithi196.stdout:7/236: chown c3 36 1 2024-04-22T15:18:53.284 INFO:tasks.workunit.client.1.smithi196.stdout:7/237: write d14/f22 [386294,87334] 0 2024-04-22T15:18:53.285 INFO:tasks.workunit.client.0.smithi096.stdout:0/754: chown d7/de/d5b/d95/dc/d26/l53 4829 1 2024-04-22T15:18:53.286 INFO:tasks.workunit.client.0.smithi096.stdout:4/626: dwrite d5/d7/f7b [0,4194304] 0 2024-04-22T15:18:53.288 INFO:tasks.workunit.client.1.smithi196.stdout:8/266: mkdir d2/d5/d2e/d51 0 2024-04-22T15:18:53.290 INFO:tasks.workunit.client.1.smithi196.stdout:0/247: dread d3/d6/f33 [0,4194304] 0 2024-04-22T15:18:53.291 INFO:tasks.workunit.client.1.smithi196.stdout:8/267: truncate d2/d5/f46 5164331 0 2024-04-22T15:18:53.292 INFO:tasks.workunit.client.1.smithi196.stdout:9/237: dwrite db/d11/d29/d57/f3d [0,4194304] 0 2024-04-22T15:18:53.293 INFO:tasks.workunit.client.1.smithi196.stdout:0/248: write d3/de/d1c/f31 [1975577,56314] 0 2024-04-22T15:18:53.297 INFO:tasks.workunit.client.0.smithi096.stdout:5/627: creat d4/fcc x:0 0 0 2024-04-22T15:18:53.301 INFO:tasks.workunit.client.1.smithi196.stdout:1/214: rename d1/db/d16/f24 to d1/db/dc/de/f4a 0 2024-04-22T15:18:53.304 INFO:tasks.workunit.client.1.smithi196.stdout:4/261: creat d6/d13/d15/d2c/d35/f4c x:0 0 0 2024-04-22T15:18:53.305 INFO:tasks.workunit.client.0.smithi096.stdout:0/755: symlink d7/de/d3a/d61/d79/lff 0 2024-04-22T15:18:53.306 INFO:tasks.workunit.client.0.smithi096.stdout:0/756: write d7/de/d5b/d64/fb6 [184651,87357] 0 2024-04-22T15:18:53.307 INFO:tasks.workunit.client.0.smithi096.stdout:4/627: creat d5/d7/d1f/fd5 x:0 0 0 2024-04-22T15:18:53.307 INFO:tasks.workunit.client.1.smithi196.stdout:1/215: dwrite d1/db/f29 [4194304,4194304] 0 2024-04-22T15:18:53.309 INFO:tasks.workunit.client.1.smithi196.stdout:1/216: truncate d1/db/dc/de/d43/f48 969232 0 2024-04-22T15:18:53.312 INFO:tasks.workunit.client.0.smithi096.stdout:5/628: mknod d4/d11/d3d/ccd 0 2024-04-22T15:18:53.314 INFO:tasks.workunit.client.1.smithi196.stdout:9/238: creat db/dd/d1b/d2c/f58 x:0 0 0 2024-04-22T15:18:53.315 INFO:tasks.workunit.client.0.smithi096.stdout:2/608: dwrite d3/d1f/f51 [0,4194304] 0 2024-04-22T15:18:53.315 INFO:tasks.workunit.client.0.smithi096.stdout:6/513: mkdir d3/d28/d4d/d82/d9e/db0 0 2024-04-22T15:18:53.318 INFO:tasks.workunit.client.1.smithi196.stdout:0/249: creat d3/de/d1c/d27/d2c/d51/f59 x:0 0 0 2024-04-22T15:18:53.319 INFO:tasks.workunit.client.0.smithi096.stdout:6/514: truncate d3/d28/d4d/d82/d9e/fae 986832 0 2024-04-22T15:18:53.319 INFO:tasks.workunit.client.0.smithi096.stdout:0/757: unlink d7/de/d5b/d95/d66/dce/ff0 0 2024-04-22T15:18:53.320 INFO:tasks.workunit.client.1.smithi196.stdout:9/239: dwrite db/dd/d1b/f45 [0,4194304] 0 2024-04-22T15:18:53.320 INFO:tasks.workunit.client.1.smithi196.stdout:9/240: chown db/dd/de/f54 136 1 2024-04-22T15:18:53.321 INFO:tasks.workunit.client.1.smithi196.stdout:9/241: dread - db/d11/d29/d57/f23 zero size 2024-04-22T15:18:53.322 INFO:tasks.workunit.client.1.smithi196.stdout:9/242: readlink db/l55 0 2024-04-22T15:18:53.322 INFO:tasks.workunit.client.0.smithi096.stdout:4/628: unlink d5/d7/d2c/d43/d56/l8c 0 2024-04-22T15:18:53.325 INFO:tasks.workunit.client.0.smithi096.stdout:5/629: dread d4/d11/d3d/d4d/d59/d5a/f79 [4194304,4194304] 0 2024-04-22T15:18:53.330 INFO:tasks.workunit.client.1.smithi196.stdout:3/242: mkdir d0/d15/d48/d51 0 2024-04-22T15:18:53.331 INFO:tasks.workunit.client.0.smithi096.stdout:5/630: write d4/d11/d3d/d4d/d59/d5a/d5c/f83 [446303,57200] 0 2024-04-22T15:18:53.331 INFO:tasks.workunit.client.1.smithi196.stdout:6/273: sync 2024-04-22T15:18:53.332 INFO:tasks.workunit.client.0.smithi096.stdout:4/629: dwrite d5/d7/d1a/fcc [0,4194304] 0 2024-04-22T15:18:53.332 INFO:tasks.workunit.client.1.smithi196.stdout:6/274: write f9 [914593,81402] 0 2024-04-22T15:18:53.334 INFO:tasks.workunit.client.0.smithi096.stdout:4/630: chown d5/d7/d2c/d43/d56/lc9 2835 1 2024-04-22T15:18:53.334 INFO:tasks.workunit.client.1.smithi196.stdout:1/217: unlink d1/db/dc/d26/d2e/d32/c3c 0 2024-04-22T15:18:53.336 INFO:tasks.workunit.client.1.smithi196.stdout:1/218: fdatasync d1/db/dc/d26/f28 0 2024-04-22T15:18:53.339 INFO:tasks.workunit.client.1.smithi196.stdout:0/250: chown d3/de/c3a 30372 1 2024-04-22T15:18:53.341 INFO:tasks.workunit.client.1.smithi196.stdout:6/275: dwrite dd/d1f/d38/d40/f43 [0,4194304] 0 2024-04-22T15:18:53.341 INFO:tasks.workunit.client.1.smithi196.stdout:0/251: fdatasync d3/de/d1c/d27/f38 0 2024-04-22T15:18:53.342 INFO:tasks.workunit.client.1.smithi196.stdout:6/276: chown dd/d1f/c32 1239729 1 2024-04-22T15:18:53.342 INFO:tasks.workunit.client.1.smithi196.stdout:6/277: stat dd/d1f/f48 0 2024-04-22T15:18:53.344 INFO:tasks.workunit.client.0.smithi096.stdout:5/631: creat d4/d11/d3d/d35/d57/dbe/fce x:0 0 0 2024-04-22T15:18:53.347 INFO:tasks.workunit.client.0.smithi096.stdout:4/631: symlink d5/d7/d2c/d67/ld6 0 2024-04-22T15:18:53.347 INFO:tasks.workunit.client.0.smithi096.stdout:1/706: sync 2024-04-22T15:18:53.348 INFO:tasks.workunit.client.1.smithi196.stdout:6/278: dwrite dd/d2b/f2f [0,4194304] 0 2024-04-22T15:18:53.353 INFO:tasks.workunit.client.1.smithi196.stdout:1/219: creat d1/db/d16/f4b x:0 0 0 2024-04-22T15:18:53.354 INFO:tasks.workunit.client.1.smithi196.stdout:9/243: mknod db/c59 0 2024-04-22T15:18:53.355 INFO:tasks.workunit.client.1.smithi196.stdout:9/244: truncate db/d11/d29/f38 715614 0 2024-04-22T15:18:53.357 INFO:tasks.workunit.client.0.smithi096.stdout:4/632: mknod d5/d7/d2c/d67/cd7 0 2024-04-22T15:18:53.360 INFO:tasks.workunit.client.1.smithi196.stdout:9/245: dwrite db/dd/f1a [0,4194304] 0 2024-04-22T15:18:53.362 INFO:tasks.workunit.client.0.smithi096.stdout:4/633: chown d5/d7/d2c/d3d/d4f/d63/l83 6 1 2024-04-22T15:18:53.362 INFO:tasks.workunit.client.1.smithi196.stdout:3/243: write d0/f1e [1730740,66531] 0 2024-04-22T15:18:53.363 INFO:tasks.workunit.client.0.smithi096.stdout:0/758: dwrite d7/de/d3a/d61/d79/fb9 [0,4194304] 0 2024-04-22T15:18:53.366 INFO:tasks.workunit.client.1.smithi196.stdout:0/252: dwrite d3/de/f18 [0,4194304] 0 2024-04-22T15:18:53.367 INFO:tasks.workunit.client.1.smithi196.stdout:5/248: rename d1/d17/d4a to d1/d51 0 2024-04-22T15:18:53.368 INFO:tasks.workunit.client.0.smithi096.stdout:2/609: dwrite d3/db/df/d10/f37 [4194304,4194304] 0 2024-04-22T15:18:53.374 INFO:tasks.workunit.client.0.smithi096.stdout:6/515: link d3/d28/dd/d11/d2c/f39 d3/d28/dd/d11/d2c/fb1 0 2024-04-22T15:18:53.375 INFO:tasks.workunit.client.1.smithi196.stdout:2/289: sync 2024-04-22T15:18:53.375 INFO:tasks.workunit.client.1.smithi196.stdout:6/279: mkdir dd/d20/d4d 0 2024-04-22T15:18:53.376 INFO:tasks.workunit.client.1.smithi196.stdout:1/220: mknod d1/db/d16/d1e/d46/c4c 0 2024-04-22T15:18:53.377 INFO:tasks.workunit.client.0.smithi096.stdout:5/632: dwrite d4/d11/d3d/d35/d57/f92 [0,4194304] 0 2024-04-22T15:18:53.378 INFO:tasks.workunit.client.0.smithi096.stdout:0/759: creat d7/de/d5b/d95/d66/f100 x:0 0 0 2024-04-22T15:18:53.382 INFO:tasks.workunit.client.0.smithi096.stdout:4/634: unlink d5/d7/d1a/l5a 0 2024-04-22T15:18:53.384 INFO:tasks.workunit.client.0.smithi096.stdout:1/707: dwrite d6/dc/d8a/d3f/d62/d7d/f53 [0,4194304] 0 2024-04-22T15:18:53.387 INFO:tasks.workunit.client.1.smithi196.stdout:9/246: creat db/d11/d29/d57/f5a x:0 0 0 2024-04-22T15:18:53.387 INFO:tasks.workunit.client.1.smithi196.stdout:9/247: dread - db/dd/de/d15/f3c zero size 2024-04-22T15:18:53.388 INFO:tasks.workunit.client.1.smithi196.stdout:9/248: write f5 [3331901,50957] 0 2024-04-22T15:18:53.389 INFO:tasks.workunit.client.0.smithi096.stdout:2/610: creat d3/d1f/d5a/fc1 x:0 0 0 2024-04-22T15:18:53.389 INFO:tasks.workunit.client.1.smithi196.stdout:9/249: readlink db/dd/de/l16 0 2024-04-22T15:18:53.389 INFO:tasks.workunit.client.1.smithi196.stdout:9/250: write db/d11/f4d [97397,114576] 0 2024-04-22T15:18:53.390 INFO:tasks.workunit.client.1.smithi196.stdout:3/244: creat d0/d15/d4d/db/dc/d4a/f52 x:0 0 0 2024-04-22T15:18:53.390 INFO:tasks.workunit.client.1.smithi196.stdout:3/245: dread - d0/d15/d4d/f3e zero size 2024-04-22T15:18:53.392 INFO:tasks.workunit.client.0.smithi096.stdout:6/516: fsync d3/d28/dd/d11/d2c/f4e 0 2024-04-22T15:18:53.393 INFO:tasks.workunit.client.1.smithi196.stdout:0/253: mkdir d3/de/d14/d5a 0 2024-04-22T15:18:53.396 INFO:tasks.workunit.client.0.smithi096.stdout:2/611: dwrite d3/d1f/f56 [0,4194304] 0 2024-04-22T15:18:53.399 INFO:tasks.workunit.client.0.smithi096.stdout:5/633: creat d4/d2c/d2f/d30/fcf x:0 0 0 2024-04-22T15:18:53.399 INFO:tasks.workunit.client.1.smithi196.stdout:7/238: sync 2024-04-22T15:18:53.400 INFO:tasks.workunit.client.1.smithi196.stdout:8/268: rename d2/d5/lf to d2/d3f/l52 0 2024-04-22T15:18:53.401 INFO:tasks.workunit.client.0.smithi096.stdout:0/760: mknod d7/de/d5b/d95/dc/d16/d65/d7c/c101 0 2024-04-22T15:18:53.402 INFO:tasks.workunit.client.1.smithi196.stdout:6/280: rmdir dd/d1f/d38/d40 39 2024-04-22T15:18:53.403 INFO:tasks.workunit.client.0.smithi096.stdout:0/761: write d7/de/d5b/d95/dc/fd0 [646422,4590] 0 2024-04-22T15:18:53.405 INFO:tasks.workunit.client.1.smithi196.stdout:1/221: mknod d1/db/d16/d1e/d49/c4d 0 2024-04-22T15:18:53.407 INFO:tasks.workunit.client.1.smithi196.stdout:9/251: chown db/dd/de/c26 198913 1 2024-04-22T15:18:53.408 INFO:tasks.workunit.client.1.smithi196.stdout:3/246: readlink d0/d15/d4d/l49 0 2024-04-22T15:18:53.409 INFO:tasks.workunit.client.0.smithi096.stdout:6/517: mknod d3/d28/dd/d11/d2c/d6f/cb2 0 2024-04-22T15:18:53.411 INFO:tasks.workunit.client.0.smithi096.stdout:1/708: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d3d/f4b [4194304,4194304] 0 2024-04-22T15:18:53.412 INFO:tasks.workunit.client.0.smithi096.stdout:2/612: rename d3/db/df/d10/f1b to d3/db/df/d10/db5/fc2 0 2024-04-22T15:18:53.414 INFO:tasks.workunit.client.1.smithi196.stdout:7/239: creat d14/d26/d2a/d2b/f45 x:0 0 0 2024-04-22T15:18:53.414 INFO:tasks.workunit.client.1.smithi196.stdout:7/240: write d14/f22 [44855,123428] 0 2024-04-22T15:18:53.415 INFO:tasks.workunit.client.0.smithi096.stdout:5/634: creat d4/d11/d3d/d4d/d59/d98/fd0 x:0 0 0 2024-04-22T15:18:53.419 INFO:tasks.workunit.client.0.smithi096.stdout:4/635: symlink d5/d7/dcb/ld8 0 2024-04-22T15:18:53.420 INFO:tasks.workunit.client.1.smithi196.stdout:8/269: creat d2/d3f/f53 x:0 0 0 2024-04-22T15:18:53.422 INFO:tasks.workunit.client.0.smithi096.stdout:7/844: sync 2024-04-22T15:18:53.422 INFO:tasks.workunit.client.0.smithi096.stdout:8/621: sync 2024-04-22T15:18:53.422 INFO:tasks.workunit.client.0.smithi096.stdout:3/485: sync 2024-04-22T15:18:53.422 INFO:tasks.workunit.client.0.smithi096.stdout:9/524: sync 2024-04-22T15:18:53.423 INFO:tasks.workunit.client.1.smithi196.stdout:6/281: mkdir dd/d1f/d4e 0 2024-04-22T15:18:53.426 INFO:tasks.workunit.client.1.smithi196.stdout:3/247: unlink d0/f1e 0 2024-04-22T15:18:53.432 INFO:tasks.workunit.client.1.smithi196.stdout:5/249: write d1/f35 [3879256,1466] 0 2024-04-22T15:18:53.434 INFO:tasks.workunit.client.0.smithi096.stdout:5/635: unlink d4/d11/d3d/d35/c45 0 2024-04-22T15:18:53.436 INFO:tasks.workunit.client.1.smithi196.stdout:8/270: truncate d2/dc/fe 9385682 0 2024-04-22T15:18:53.437 INFO:tasks.workunit.client.1.smithi196.stdout:5/250: dwrite d1/d2/f25 [0,4194304] 0 2024-04-22T15:18:53.438 INFO:tasks.workunit.client.1.smithi196.stdout:6/282: read dd/f1c [2176219,19706] 0 2024-04-22T15:18:53.438 INFO:tasks.workunit.client.1.smithi196.stdout:6/283: read dd/f1c [2227935,42911] 0 2024-04-22T15:18:53.439 INFO:tasks.workunit.client.0.smithi096.stdout:0/762: symlink d7/d42/d4c/d55/d9c/dfc/l102 0 2024-04-22T15:18:53.439 INFO:tasks.workunit.client.1.smithi196.stdout:6/284: dread - dd/d2b/f2e zero size 2024-04-22T15:18:53.441 INFO:tasks.workunit.client.0.smithi096.stdout:5/636: dwrite d4/d11/d3d/d35/d57/fb4 [0,4194304] 0 2024-04-22T15:18:53.449 INFO:tasks.workunit.client.0.smithi096.stdout:9/525: creat d1/d4/d7f/fa9 x:0 0 0 2024-04-22T15:18:53.451 INFO:tasks.workunit.client.0.smithi096.stdout:9/526: chown d1/df/d95/d9d 4 1 2024-04-22T15:18:53.451 INFO:tasks.workunit.client.0.smithi096.stdout:7/845: dread - d31/d32/d33/d54/d43/fe8 zero size 2024-04-22T15:18:53.453 INFO:tasks.workunit.client.0.smithi096.stdout:8/622: dwrite d4/fd [8388608,4194304] 0 2024-04-22T15:18:53.454 INFO:tasks.workunit.client.1.smithi196.stdout:4/262: rename d6/d14/d18/d26/d32/f46 to d6/f4d 0 2024-04-22T15:18:53.455 INFO:tasks.workunit.client.0.smithi096.stdout:6/518: symlink d3/d28/d54/d7e/lb3 0 2024-04-22T15:18:53.455 INFO:tasks.workunit.client.1.smithi196.stdout:4/263: write d6/f19 [1423427,93950] 0 2024-04-22T15:18:53.455 INFO:tasks.workunit.client.0.smithi096.stdout:8/623: dread d4/d71/f77 [0,4194304] 0 2024-04-22T15:18:53.462 INFO:tasks.workunit.client.0.smithi096.stdout:4/636: mknod d5/d7/d1a/cd9 0 2024-04-22T15:18:53.464 INFO:tasks.workunit.client.0.smithi096.stdout:2/613: dwrite d3/db/d2d/d46/d59/f76 [0,4194304] 0 2024-04-22T15:18:53.466 INFO:tasks.workunit.client.1.smithi196.stdout:1/222: link d1/db/dc/l1f d1/db/d16/d1e/d46/l4e 0 2024-04-22T15:18:53.467 INFO:tasks.workunit.client.0.smithi096.stdout:0/763: symlink d7/de/d5b/d62/l103 0 2024-04-22T15:18:53.467 INFO:tasks.workunit.client.0.smithi096.stdout:3/486: mkdir da/db/d9a 0 2024-04-22T15:18:53.468 INFO:tasks.workunit.client.1.smithi196.stdout:3/248: creat d0/d15/d31/f53 x:0 0 0 2024-04-22T15:18:53.468 INFO:tasks.workunit.client.0.smithi096.stdout:1/709: dwrite d6/dc/d8a/d3f/d54/d32/d8c/d4a/fa7 [0,4194304] 0 2024-04-22T15:18:53.469 INFO:tasks.workunit.client.1.smithi196.stdout:3/249: read - d0/d15/d4d/db/dc/d4a/f46 zero size 2024-04-22T15:18:53.470 INFO:tasks.workunit.client.0.smithi096.stdout:7/846: readlink d31/d32/d33/d54/ldb 0 2024-04-22T15:18:53.472 INFO:tasks.workunit.client.0.smithi096.stdout:5/637: dread d4/d11/d3d/d4d/d59/d5a/f15 [0,4194304] 0 2024-04-22T15:18:53.478 INFO:tasks.workunit.client.1.smithi196.stdout:2/290: rename d1/f2b to d1/d13/d41/f6c 0 2024-04-22T15:18:53.478 INFO:tasks.workunit.client.1.smithi196.stdout:2/291: write d1/d13/d1f/f20 [2627734,22914] 0 2024-04-22T15:18:53.480 INFO:tasks.workunit.client.1.smithi196.stdout:8/271: mknod d2/d5/d4f/c54 0 2024-04-22T15:18:53.484 INFO:tasks.workunit.client.1.smithi196.stdout:4/264: dread d6/d14/d18/d26/f3a [0,4194304] 0 2024-04-22T15:18:53.485 INFO:tasks.workunit.client.0.smithi096.stdout:8/624: creat d4/d1c/d40/da7/fd9 x:0 0 0 2024-04-22T15:18:53.485 INFO:tasks.workunit.client.1.smithi196.stdout:4/265: write d6/f2b [3544135,99594] 0 2024-04-22T15:18:53.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:53 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:18:53.486 INFO:tasks.workunit.client.0.smithi096.stdout:3/487: dread da/db/dc/f16 [4194304,4194304] 0 2024-04-22T15:18:53.486 INFO:tasks.workunit.client.1.smithi196.stdout:6/285: creat dd/d1f/d38/d40/f4f x:0 0 0 2024-04-22T15:18:53.489 INFO:tasks.workunit.client.1.smithi196.stdout:7/241: getdents d14/d26/d28 0 2024-04-22T15:18:53.489 INFO:tasks.workunit.client.1.smithi196.stdout:7/242: chown d14/d26/d28/d43 111 1 2024-04-22T15:18:53.491 INFO:tasks.workunit.client.0.smithi096.stdout:9/527: creat d1/d4/d9/d43/d57/d5b/d8e/faa x:0 0 0 2024-04-22T15:18:53.492 INFO:tasks.workunit.client.0.smithi096.stdout:9/528: read d1/d4/f42 [3391624,1476] 0 2024-04-22T15:18:53.493 INFO:tasks.workunit.client.0.smithi096.stdout:9/529: write d1/f75 [543442,95048] 0 2024-04-22T15:18:53.494 INFO:tasks.workunit.client.0.smithi096.stdout:9/530: write d1/d4/f42 [338204,91202] 0 2024-04-22T15:18:53.495 INFO:tasks.workunit.client.0.smithi096.stdout:2/614: rmdir d3/db/d2d/d46/d7e 39 2024-04-22T15:18:53.498 INFO:tasks.workunit.client.0.smithi096.stdout:7/847: mknod d31/d32/d36/d5d/c119 0 2024-04-22T15:18:53.499 INFO:tasks.workunit.client.1.smithi196.stdout:0/254: rename d3/d9/d36/d56/f4f to d3/d9/d25/f5b 0 2024-04-22T15:18:53.501 INFO:tasks.workunit.client.0.smithi096.stdout:1/710: mkdir d6/dc/d8a/d3f/d62/d7d/dd7 0 2024-04-22T15:18:53.501 INFO:tasks.workunit.client.0.smithi096.stdout:2/615: dwrite d3/d1f/f9f [0,4194304] 0 2024-04-22T15:18:53.502 INFO:tasks.workunit.client.1.smithi196.stdout:5/251: write d1/d2/f5 [3203183,104821] 0 2024-04-22T15:18:53.504 INFO:tasks.workunit.client.0.smithi096.stdout:2/616: chown d3/d1f/f7a 0 1 2024-04-22T15:18:53.504 INFO:tasks.workunit.client.1.smithi196.stdout:2/292: write d1/d23/d27/d2e/d55/f36 [4408782,89076] 0 2024-04-22T15:18:53.504 INFO:tasks.workunit.client.0.smithi096.stdout:4/637: write d5/d7/d2c/f2e [3943297,69059] 0 2024-04-22T15:18:53.506 INFO:tasks.workunit.client.1.smithi196.stdout:8/272: mknod d2/dc/c55 0 2024-04-22T15:18:53.506 INFO:tasks.workunit.client.0.smithi096.stdout:6/519: dwrite d3/d28/d54/d68/f76 [0,4194304] 0 2024-04-22T15:18:53.508 INFO:tasks.workunit.client.0.smithi096.stdout:3/488: mkdir da/db/d12/d9b 0 2024-04-22T15:18:53.508 INFO:tasks.workunit.client.0.smithi096.stdout:8/625: stat d4/c15 0 2024-04-22T15:18:53.509 INFO:tasks.workunit.client.1.smithi196.stdout:8/273: dread d2/d5/f19 [0,4194304] 0 2024-04-22T15:18:53.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:53 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:18:53.509 INFO:tasks.workunit.client.0.smithi096.stdout:8/626: fdatasync d4/d1c/f3a 0 2024-04-22T15:18:53.511 INFO:tasks.workunit.client.0.smithi096.stdout:6/520: write d3/d28/d54/d68/d6e/f80 [4491957,40246] 0 2024-04-22T15:18:53.512 INFO:tasks.workunit.client.1.smithi196.stdout:4/266: rmdir d6/d13/d15/d2c/d35/d44 39 2024-04-22T15:18:53.516 INFO:tasks.workunit.client.1.smithi196.stdout:3/250: link d0/d15/d4d/db/dc/d18/f4c d0/d15/d48/d51/f54 0 2024-04-22T15:18:53.516 INFO:tasks.workunit.client.0.smithi096.stdout:9/531: creat d1/d85/fab x:0 0 0 2024-04-22T15:18:53.518 INFO:tasks.workunit.client.1.smithi196.stdout:9/252: rename db/dd/d1b/d2c/l50 to db/dd/de/d15/d31/l5b 0 2024-04-22T15:18:53.519 INFO:tasks.workunit.client.0.smithi096.stdout:7/848: creat d31/d32/dc0/f11a x:0 0 0 2024-04-22T15:18:53.521 INFO:tasks.workunit.client.1.smithi196.stdout:0/255: symlink d3/de/d1c/l5c 0 2024-04-22T15:18:53.521 INFO:tasks.workunit.client.1.smithi196.stdout:4/267: dread d6/f19 [0,4194304] 0 2024-04-22T15:18:53.521 INFO:tasks.workunit.client.1.smithi196.stdout:0/256: write d3/de/d14/f4a [743081,21960] 0 2024-04-22T15:18:53.524 INFO:tasks.workunit.client.1.smithi196.stdout:5/252: creat d1/d17/d1c/f52 x:0 0 0 2024-04-22T15:18:53.526 INFO:tasks.workunit.client.0.smithi096.stdout:2/617: mknod d3/db/d2d/d46/d59/cc3 0 2024-04-22T15:18:53.529 INFO:tasks.workunit.client.1.smithi196.stdout:0/257: dread d3/de/d1c/f31 [0,4194304] 0 2024-04-22T15:18:53.532 INFO:tasks.workunit.client.1.smithi196.stdout:3/251: mkdir d0/d15/d31/d55 0 2024-04-22T15:18:53.533 INFO:tasks.workunit.client.1.smithi196.stdout:6/286: dwrite dd/f26 [0,4194304] 0 2024-04-22T15:18:53.536 INFO:tasks.workunit.client.1.smithi196.stdout:9/253: read db/fc [2322602,93992] 0 2024-04-22T15:18:53.537 INFO:tasks.workunit.client.1.smithi196.stdout:4/268: creat d6/d14/d18/d40/f4e x:0 0 0 2024-04-22T15:18:53.538 INFO:tasks.workunit.client.1.smithi196.stdout:4/269: dread - d6/d14/f2a zero size 2024-04-22T15:18:53.538 INFO:tasks.workunit.client.1.smithi196.stdout:6/287: read dd/d1f/d37/f3f [2332070,77869] 0 2024-04-22T15:18:53.539 INFO:tasks.workunit.client.0.smithi096.stdout:9/532: chown d1/df/fa7 15 1 2024-04-22T15:18:53.540 INFO:tasks.workunit.client.0.smithi096.stdout:9/533: chown d1/f20 172102226 1 2024-04-22T15:18:53.541 INFO:tasks.workunit.client.1.smithi196.stdout:4/270: dread d6/d14/f12 [4194304,4194304] 0 2024-04-22T15:18:53.542 INFO:tasks.workunit.client.0.smithi096.stdout:7/849: rename d31/d32/d33/d5e/lda to d31/d32/dc0/ddc/de9/l11b 0 2024-04-22T15:18:53.544 INFO:tasks.workunit.client.0.smithi096.stdout:4/638: symlink d5/d7/d2c/lda 0 2024-04-22T15:18:53.547 INFO:tasks.workunit.client.1.smithi196.stdout:1/223: link d1/db/dc/de/c10 d1/db/c4f 0 2024-04-22T15:18:53.547 INFO:tasks.workunit.client.0.smithi096.stdout:3/489: write da/db/d12/d22/f88 [696191,121420] 0 2024-04-22T15:18:53.548 INFO:tasks.workunit.client.0.smithi096.stdout:6/521: symlink d3/d28/d54/d68/lb4 0 2024-04-22T15:18:53.551 INFO:tasks.workunit.client.0.smithi096.stdout:8/627: link d4/d10/d4d/f84 d4/d1c/d62/dc3/fda 0 2024-04-22T15:18:53.554 INFO:tasks.workunit.client.0.smithi096.stdout:7/850: creat d31/d32/d33/d44/f11c x:0 0 0 2024-04-22T15:18:53.556 INFO:tasks.workunit.client.0.smithi096.stdout:4/639: rename f3 to d5/d7/d1f/d84/d6d/fdb 0 2024-04-22T15:18:53.557 INFO:tasks.workunit.client.0.smithi096.stdout:4/640: stat d5/f39 0 2024-04-22T15:18:53.562 INFO:tasks.workunit.client.0.smithi096.stdout:3/490: symlink da/db/dc/l9c 0 2024-04-22T15:18:53.563 INFO:tasks.workunit.client.1.smithi196.stdout:4/271: write d6/d14/f33 [4642364,72092] 0 2024-04-22T15:18:53.565 INFO:tasks.workunit.client.1.smithi196.stdout:3/252: dwrite d0/d15/d48/d51/f54 [0,4194304] 0 2024-04-22T15:18:53.566 INFO:tasks.workunit.client.0.smithi096.stdout:8/628: mknod d4/d2a/d7e/cdb 0 2024-04-22T15:18:53.567 INFO:tasks.workunit.client.1.smithi196.stdout:3/253: chown d0/d15/d4d/db/dc/d18/f4c 5 1 2024-04-22T15:18:53.569 INFO:tasks.workunit.client.0.smithi096.stdout:3/491: dwrite da/db/d12/f77 [0,4194304] 0 2024-04-22T15:18:53.570 INFO:tasks.workunit.client.0.smithi096.stdout:6/522: dread d3/d28/dd/d11/d2c/d44/d51/f60 [0,4194304] 0 2024-04-22T15:18:53.571 INFO:tasks.workunit.client.1.smithi196.stdout:3/254: dwrite d0/d15/d4d/f3e [0,4194304] 0 2024-04-22T15:18:53.572 INFO:tasks.workunit.client.0.smithi096.stdout:2/618: truncate d3/db/d30/f3a 5612863 0 2024-04-22T15:18:53.572 INFO:tasks.workunit.client.1.smithi196.stdout:3/255: write d0/d15/d48/d51/f54 [1476539,380] 0 2024-04-22T15:18:53.573 INFO:tasks.workunit.client.0.smithi096.stdout:3/492: dread - da/db/de/d19/d2d/d37/f5f zero size 2024-04-22T15:18:53.574 INFO:tasks.workunit.client.0.smithi096.stdout:7/851: symlink d31/d32/d33/d93/dd5/l11d 0 2024-04-22T15:18:53.577 INFO:tasks.workunit.client.1.smithi196.stdout:7/243: rename d14/d26/d28/d43 to d14/d46 0 2024-04-22T15:18:53.579 INFO:tasks.workunit.client.0.smithi096.stdout:4/641: mknod d5/d7/d2c/d43/d56/cdc 0 2024-04-22T15:18:53.583 INFO:tasks.workunit.client.0.smithi096.stdout:3/493: dwrite da/db/dc/f5d [0,4194304] 0 2024-04-22T15:18:53.583 INFO:tasks.workunit.client.1.smithi196.stdout:4/272: stat d6/d13/f1d 0 2024-04-22T15:18:53.584 INFO:tasks.workunit.client.1.smithi196.stdout:5/253: getdents d1/d17/d1a 0 2024-04-22T15:18:53.584 INFO:tasks.workunit.client.0.smithi096.stdout:9/534: link d1/d4/d9/d59/c72 d1/d6f/cac 0 2024-04-22T15:18:53.585 INFO:tasks.workunit.client.1.smithi196.stdout:5/254: write d1/d14/f42 [804079,107829] 0 2024-04-22T15:18:53.585 INFO:tasks.workunit.client.0.smithi096.stdout:2/619: symlink d3/db/d30/lc4 0 2024-04-22T15:18:53.586 INFO:tasks.workunit.client.1.smithi196.stdout:5/255: truncate d1/d9/f20 4373368 0 2024-04-22T15:18:53.589 INFO:tasks.workunit.client.0.smithi096.stdout:4/642: unlink d5/d7/d1a/d25/fb7 0 2024-04-22T15:18:53.593 INFO:tasks.workunit.client.1.smithi196.stdout:2/293: rename d1/d23/d27/f65 to d1/d23/f6d 0 2024-04-22T15:18:53.594 INFO:tasks.workunit.client.1.smithi196.stdout:7/244: fdatasync f10 0 2024-04-22T15:18:53.594 INFO:tasks.workunit.client.0.smithi096.stdout:2/620: fdatasync d3/db/f42 0 2024-04-22T15:18:53.595 INFO:tasks.workunit.client.0.smithi096.stdout:2/621: fsync d3/db/d2d/fad 0 2024-04-22T15:18:53.596 INFO:tasks.workunit.client.0.smithi096.stdout:2/622: fdatasync d3/d1f/f9f 0 2024-04-22T15:18:53.598 INFO:tasks.workunit.client.1.smithi196.stdout:3/256: read d0/d15/d4d/db/dc/d18/f42 [534048,80181] 0 2024-04-22T15:18:53.599 INFO:tasks.workunit.client.0.smithi096.stdout:8/629: dread d4/d1c/d40/da7/db6/f3e [0,4194304] 0 2024-04-22T15:18:53.608 INFO:tasks.workunit.client.1.smithi196.stdout:8/274: sync 2024-04-22T15:18:53.675 INFO:tasks.workunit.client.0.smithi096.stdout:3/494: mknod da/c9d 0 2024-04-22T15:18:53.677 INFO:tasks.workunit.client.0.smithi096.stdout:9/535: symlink d1/d4/d9/d43/lad 0 2024-04-22T15:18:53.679 INFO:tasks.workunit.client.1.smithi196.stdout:0/258: rename d3/d6/c10 to d3/de/d1c/d27/d2c/d30/c5d 0 2024-04-22T15:18:53.679 INFO:tasks.workunit.client.1.smithi196.stdout:0/259: chown d3 3163 1 2024-04-22T15:18:53.680 INFO:tasks.workunit.client.1.smithi196.stdout:0/260: write d3/de/d1c/d27/d2c/d30/f35 [302765,68071] 0 2024-04-22T15:18:53.681 INFO:tasks.workunit.client.1.smithi196.stdout:0/261: write d3/de/f18 [977792,29238] 0 2024-04-22T15:18:53.683 INFO:tasks.workunit.client.1.smithi196.stdout:2/294: rmdir d1/d13/d1f 39 2024-04-22T15:18:53.684 INFO:tasks.workunit.client.0.smithi096.stdout:9/536: dwrite d1/d4/d9/d43/d57/d5b/d73/f9a [0,4194304] 0 2024-04-22T15:18:53.685 INFO:tasks.workunit.client.1.smithi196.stdout:6/288: truncate dd/f1a 2954198 0 2024-04-22T15:18:53.686 INFO:tasks.workunit.client.1.smithi196.stdout:6/289: write dd/fe [5130929,58679] 0 2024-04-22T15:18:53.687 INFO:tasks.workunit.client.1.smithi196.stdout:2/295: dread d1/d13/f19 [0,4194304] 0 2024-04-22T15:18:53.687 INFO:tasks.workunit.client.1.smithi196.stdout:6/290: dread - dd/d1f/d38/d40/f49 zero size 2024-04-22T15:18:53.688 INFO:tasks.workunit.client.1.smithi196.stdout:2/296: chown d1/d23/d27/d2e/l4e 0 1 2024-04-22T15:18:53.688 INFO:tasks.workunit.client.1.smithi196.stdout:6/291: chown dd/d1f/d38/f4b 67 1 2024-04-22T15:18:53.696 INFO:tasks.workunit.client.0.smithi096.stdout:2/623: dwrite d3/db/df/fa5 [0,4194304] 0 2024-04-22T15:18:53.698 INFO:tasks.workunit.client.1.smithi196.stdout:1/224: getdents d1/db/dc/d26/d2e/d32 0 2024-04-22T15:18:53.699 INFO:tasks.workunit.client.1.smithi196.stdout:1/225: write d1/db/d16/f40 [567347,27353] 0 2024-04-22T15:18:53.700 INFO:tasks.workunit.client.0.smithi096.stdout:3/495: rename da/db/d12/c14 to da/db/dc/d8d/c9e 0 2024-04-22T15:18:53.702 INFO:tasks.workunit.client.1.smithi196.stdout:5/256: mkdir d1/d53 0 2024-04-22T15:18:53.703 INFO:tasks.workunit.client.1.smithi196.stdout:4/273: write d6/d13/d15/d2c/d35/f3c [2387589,118104] 0 2024-04-22T15:18:53.705 INFO:tasks.workunit.client.1.smithi196.stdout:9/254: rename db/d11/d29/d57/f24 to db/dd/f5c 0 2024-04-22T15:18:53.706 INFO:tasks.workunit.client.0.smithi096.stdout:7/852: link d31/d32/d33/d6a/d83/dba/ccc d31/d32/d33/d44/da8/dbf/ddf/c11e 0 2024-04-22T15:18:53.707 INFO:tasks.workunit.client.0.smithi096.stdout:7/853: chown d31/d32/f4c 41209 1 2024-04-22T15:18:53.711 INFO:tasks.workunit.client.1.smithi196.stdout:0/262: creat d3/de/d14/d29/f5e x:0 0 0 2024-04-22T15:18:53.718 INFO:tasks.workunit.client.0.smithi096.stdout:4/643: truncate d5/f21 2357618 0 2024-04-22T15:18:53.719 INFO:tasks.workunit.client.1.smithi196.stdout:5/257: dread d1/d14/d3e/f44 [0,4194304] 0 2024-04-22T15:18:53.720 INFO:tasks.workunit.client.1.smithi196.stdout:6/292: unlink dd/d1f/f36 0 2024-04-22T15:18:53.722 INFO:tasks.workunit.client.1.smithi196.stdout:7/245: symlink d14/d26/d28/d29/l47 0 2024-04-22T15:18:53.723 INFO:tasks.workunit.client.0.smithi096.stdout:9/537: rename d1/d4/d9/d43/d57/d5e to d1/d4/d9/d43/d57/d5b/d8e/dae 0 2024-04-22T15:18:53.724 INFO:tasks.workunit.client.1.smithi196.stdout:5/258: dwrite d1/d2/f25 [0,4194304] 0 2024-04-22T15:18:53.724 INFO:tasks.workunit.client.1.smithi196.stdout:8/275: mknod d2/dc/d45/c56 0 2024-04-22T15:18:53.725 INFO:tasks.workunit.client.1.smithi196.stdout:5/259: chown d1/d14/d3e/f48 277041516 1 2024-04-22T15:18:53.726 INFO:tasks.workunit.client.0.smithi096.stdout:3/496: creat da/d3f/f9f x:0 0 0 2024-04-22T15:18:53.728 INFO:tasks.workunit.client.1.smithi196.stdout:1/226: unlink d1/l2c 0 2024-04-22T15:18:53.731 INFO:tasks.workunit.client.0.smithi096.stdout:7/854: creat d31/d32/d33/d93/f11f x:0 0 0 2024-04-22T15:18:53.733 INFO:tasks.workunit.client.0.smithi096.stdout:7/855: stat d31/d32/d33/d54/d43/lb8 0 2024-04-22T15:18:53.734 INFO:tasks.workunit.client.0.smithi096.stdout:8/630: rmdir d4/d1c/d40/da7/db6/db8 0 2024-04-22T15:18:53.737 INFO:tasks.workunit.client.1.smithi196.stdout:3/257: rename d0/l3a to d0/d15/d31/l56 0 2024-04-22T15:18:53.739 INFO:tasks.workunit.client.1.smithi196.stdout:9/255: symlink db/d11/d29/d57/l5d 0 2024-04-22T15:18:53.740 INFO:tasks.workunit.client.0.smithi096.stdout:4/644: mknod d5/d7/d1a/d25/cdd 0 2024-04-22T15:18:53.743 INFO:tasks.workunit.client.0.smithi096.stdout:9/538: rename d1/d4/d9/d43/lad to d1/d6f/laf 0 2024-04-22T15:18:53.745 INFO:tasks.workunit.client.0.smithi096.stdout:3/497: unlink da/d3f/c4b 0 2024-04-22T15:18:53.746 INFO:tasks.workunit.client.0.smithi096.stdout:7/856: symlink d31/d32/d36/d5d/l120 0 2024-04-22T15:18:53.750 INFO:tasks.workunit.client.0.smithi096.stdout:2/624: symlink d3/db/d2d/d46/d7e/lc5 0 2024-04-22T15:18:53.751 INFO:tasks.workunit.client.1.smithi196.stdout:7/246: mknod d14/d26/d28/c48 0 2024-04-22T15:18:53.752 INFO:tasks.workunit.client.0.smithi096.stdout:4/645: mknod d5/d7/d1f/d84/cde 0 2024-04-22T15:18:53.752 INFO:tasks.workunit.client.1.smithi196.stdout:8/276: chown d2/d5/d1a/l3d 10434 1 2024-04-22T15:18:53.753 INFO:tasks.workunit.client.0.smithi096.stdout:9/539: unlink d1/d4/d9/c51 0 2024-04-22T15:18:53.754 INFO:tasks.workunit.client.1.smithi196.stdout:5/260: fsync d1/f1f 0 2024-04-22T15:18:53.757 INFO:tasks.workunit.client.1.smithi196.stdout:8/277: dwrite d2/d3f/f41 [0,4194304] 0 2024-04-22T15:18:53.758 INFO:tasks.workunit.client.1.smithi196.stdout:4/274: rename d6/d13 to d6/d13/d15/d2c/d35/d4f 22 2024-04-22T15:18:53.759 INFO:tasks.workunit.client.1.smithi196.stdout:2/297: sync 2024-04-22T15:18:53.761 INFO:tasks.workunit.client.0.smithi096.stdout:4/646: creat d5/d7/d2c/d43/fdf x:0 0 0 2024-04-22T15:18:53.764 INFO:tasks.workunit.client.1.smithi196.stdout:4/275: dwrite d6/d13/d15/d2c/d35/f4c [0,4194304] 0 2024-04-22T15:18:53.765 INFO:tasks.workunit.client.1.smithi196.stdout:4/276: fsync d6/d13/d15/d2c/d35/f3c 0 2024-04-22T15:18:53.766 INFO:tasks.workunit.client.1.smithi196.stdout:0/263: creat d3/de/d14/d5a/f5f x:0 0 0 2024-04-22T15:18:53.768 INFO:tasks.workunit.client.1.smithi196.stdout:0/264: truncate d3/de/d14/d29/f57 913337 0 2024-04-22T15:18:53.769 INFO:tasks.workunit.client.0.smithi096.stdout:3/498: unlink da/d71/l76 0 2024-04-22T15:18:53.772 INFO:tasks.workunit.client.0.smithi096.stdout:7/857: link d31/d32/d36/d5d/fd9 d31/d32/d36/f121 0 2024-04-22T15:18:53.774 INFO:tasks.workunit.client.1.smithi196.stdout:6/293: truncate f1 2072695 0 2024-04-22T15:18:53.774 INFO:tasks.workunit.client.1.smithi196.stdout:6/294: dread - dd/d1f/d38/d40/f49 zero size 2024-04-22T15:18:53.775 INFO:tasks.workunit.client.1.smithi196.stdout:6/295: stat dd/d1f/d38/d40 0 2024-04-22T15:18:53.775 INFO:tasks.workunit.client.0.smithi096.stdout:8/631: creat d4/d1c/d40/fdc x:0 0 0 2024-04-22T15:18:53.778 INFO:tasks.workunit.client.0.smithi096.stdout:4/647: symlink d5/d7/d1f/le0 0 2024-04-22T15:18:53.779 INFO:tasks.workunit.client.0.smithi096.stdout:2/625: dread d3/db/d2d/d46/d69/f66 [4194304,4194304] 0 2024-04-22T15:18:53.780 INFO:tasks.workunit.client.1.smithi196.stdout:5/261: creat d1/d17/d1a/d2f/d30/f54 x:0 0 0 2024-04-22T15:18:53.783 INFO:tasks.workunit.client.1.smithi196.stdout:8/278: mknod d2/d35/c57 0 2024-04-22T15:18:53.784 INFO:tasks.workunit.client.0.smithi096.stdout:7/858: truncate d31/d32/d36/d75/f86 1787423 0 2024-04-22T15:18:53.788 INFO:tasks.workunit.client.0.smithi096.stdout:4/648: creat d5/d7/d2c/d3d/d4f/d88/fe1 x:0 0 0 2024-04-22T15:18:53.794 INFO:tasks.workunit.client.0.smithi096.stdout:3/499: dread da/db/dc/d28/f93 [0,4194304] 0 2024-04-22T15:18:53.794 INFO:tasks.workunit.client.0.smithi096.stdout:9/540: rmdir d1/d4/d79/da5 0 2024-04-22T15:18:53.796 INFO:tasks.workunit.client.0.smithi096.stdout:8/632: dread d4/d10/d25/f2e [0,4194304] 0 2024-04-22T15:18:53.800 INFO:tasks.workunit.client.1.smithi196.stdout:4/277: creat d6/d10/f50 x:0 0 0 2024-04-22T15:18:53.801 INFO:tasks.workunit.client.0.smithi096.stdout:1/711: sync 2024-04-22T15:18:53.801 INFO:tasks.workunit.client.0.smithi096.stdout:5/638: sync 2024-04-22T15:18:53.801 INFO:tasks.workunit.client.0.smithi096.stdout:0/764: sync 2024-04-22T15:18:53.801 INFO:tasks.workunit.client.0.smithi096.stdout:6/523: sync 2024-04-22T15:18:53.802 INFO:tasks.workunit.client.0.smithi096.stdout:0/765: chown d7/de/d3a/d61/dbb/ce1 18645 1 2024-04-22T15:18:53.806 INFO:tasks.workunit.client.0.smithi096.stdout:4/649: dread d5/d7/d1a/f6c [0,4194304] 0 2024-04-22T15:18:53.807 INFO:tasks.workunit.client.1.smithi196.stdout:2/298: dwrite d1/d13/f19 [0,4194304] 0 2024-04-22T15:18:53.807 INFO:tasks.workunit.client.1.smithi196.stdout:6/296: write dd/f17 [326340,80599] 0 2024-04-22T15:18:53.807 INFO:tasks.workunit.client.0.smithi096.stdout:3/500: truncate da/f34 3512888 0 2024-04-22T15:18:53.808 INFO:tasks.workunit.client.1.smithi196.stdout:2/299: write d1/f2c [2637031,91589] 0 2024-04-22T15:18:53.814 INFO:tasks.workunit.client.0.smithi096.stdout:7/859: write d31/d32/d33/d44/da8/dbf/fe1 [587716,39371] 0 2024-04-22T15:18:53.814 INFO:tasks.workunit.client.0.smithi096.stdout:0/766: dwrite d7/de/d5b/d95/dc/d16/d18/f69 [0,4194304] 0 2024-04-22T15:18:53.815 INFO:tasks.workunit.client.0.smithi096.stdout:2/626: dwrite d3/db/df/d24/f8f [0,4194304] 0 2024-04-22T15:18:53.816 INFO:tasks.workunit.client.1.smithi196.stdout:1/227: creat d1/db/dc/d26/d31/f50 x:0 0 0 2024-04-22T15:18:53.816 INFO:tasks.workunit.client.1.smithi196.stdout:1/228: read d1/db/f29 [4213521,47871] 0 2024-04-22T15:18:53.823 INFO:tasks.workunit.client.0.smithi096.stdout:1/712: chown d6/dc/d8a/d3f/d54/d32/f6d 181875 1 2024-04-22T15:18:53.824 INFO:tasks.workunit.client.1.smithi196.stdout:9/256: link db/d11/d29/d57/l20 db/dd/l5e 0 2024-04-22T15:18:53.825 INFO:tasks.workunit.client.1.smithi196.stdout:4/278: symlink d6/d14/d18/d26/l51 0 2024-04-22T15:18:53.830 INFO:tasks.workunit.client.1.smithi196.stdout:0/265: rename d3/d6/dc/c52 to d3/de/d1c/d27/c60 0 2024-04-22T15:18:53.830 INFO:tasks.workunit.client.0.smithi096.stdout:8/633: write d4/d1c/d62/dc3/fda [1078057,45723] 0 2024-04-22T15:18:53.832 INFO:tasks.workunit.client.0.smithi096.stdout:3/501: read - da/db/de/d19/f6f zero size 2024-04-22T15:18:53.832 INFO:tasks.workunit.client.1.smithi196.stdout:0/266: dread - d3/d9/d25/f5b zero size 2024-04-22T15:18:53.833 INFO:tasks.workunit.client.1.smithi196.stdout:7/247: dwrite d14/d15/f18 [0,4194304] 0 2024-04-22T15:18:53.834 INFO:tasks.workunit.client.1.smithi196.stdout:2/300: unlink d1/d13/d25/l64 0 2024-04-22T15:18:53.839 INFO:tasks.workunit.client.1.smithi196.stdout:5/262: symlink d1/d17/d50/l55 0 2024-04-22T15:18:53.840 INFO:tasks.workunit.client.0.smithi096.stdout:9/541: mknod d1/d4/d9/dd/cb0 0 2024-04-22T15:18:53.843 INFO:tasks.workunit.client.0.smithi096.stdout:1/713: mknod d6/dc/d8a/d3f/d54/d32/d8c/d45/d7b/cd8 0 2024-04-22T15:18:53.844 INFO:tasks.workunit.client.0.smithi096.stdout:1/714: chown d6/dc/d8a/d3f/d62/d7d/lbe 226 1 2024-04-22T15:18:53.845 INFO:tasks.workunit.client.1.smithi196.stdout:1/229: rename d1/db/dc/d26/d2e/f41 to d1/db/dc/d26/d2e/f51 0 2024-04-22T15:18:53.849 INFO:tasks.workunit.client.0.smithi096.stdout:8/634: rename d4/c4a to d4/d1c/d40/d48/cdd 0 2024-04-22T15:18:53.850 INFO:tasks.workunit.client.1.smithi196.stdout:3/258: getdents d0/d15/d4d/db/dc 0 2024-04-22T15:18:53.851 INFO:tasks.workunit.client.0.smithi096.stdout:3/502: mknod da/db/dc/ca0 0 2024-04-22T15:18:53.852 INFO:tasks.workunit.client.1.smithi196.stdout:3/259: truncate d0/d15/d4d/f2d 1303901 0 2024-04-22T15:18:53.853 INFO:tasks.workunit.client.0.smithi096.stdout:3/503: write da/db/de/d19/f30 [2740153,19719] 0 2024-04-22T15:18:53.853 INFO:tasks.workunit.client.1.smithi196.stdout:4/279: dread d6/d13/d15/d2c/d35/f3c [8388608,4194304] 0 2024-04-22T15:18:53.853 INFO:tasks.workunit.client.1.smithi196.stdout:3/260: read - d0/d7/f38 zero size 2024-04-22T15:18:53.853 INFO:tasks.workunit.client.1.smithi196.stdout:4/280: stat c2 0 2024-04-22T15:18:53.854 INFO:tasks.workunit.client.0.smithi096.stdout:3/504: readlink da/db/de/d19/d2d/l66 0 2024-04-22T15:18:53.854 INFO:tasks.workunit.client.0.smithi096.stdout:6/524: write d3/f49 [2829690,39363] 0 2024-04-22T15:18:53.855 INFO:tasks.workunit.client.1.smithi196.stdout:4/281: truncate d6/d10/f50 391146 0 2024-04-22T15:18:53.856 INFO:tasks.workunit.client.0.smithi096.stdout:3/505: chown da/db/d10/c18 10144 1 2024-04-22T15:18:53.858 INFO:tasks.workunit.client.0.smithi096.stdout:2/627: dwrite d3/db/f31 [0,4194304] 0 2024-04-22T15:18:53.859 INFO:tasks.workunit.client.0.smithi096.stdout:2/628: readlink d3/db/d2d/d46/d7e/lc5 0 2024-04-22T15:18:53.861 INFO:tasks.workunit.client.0.smithi096.stdout:9/542: unlink d1/d4/d9/dd/f1b 0 2024-04-22T15:18:53.861 INFO:tasks.workunit.client.0.smithi096.stdout:4/650: truncate d5/f39 3820052 0 2024-04-22T15:18:53.866 INFO:tasks.workunit.client.0.smithi096.stdout:1/715: truncate d6/dc/d8a/d3f/d54/fa5 114426 0 2024-04-22T15:18:53.867 INFO:tasks.workunit.client.1.smithi196.stdout:8/279: dwrite d2/dc/fe [0,4194304] 0 2024-04-22T15:18:53.869 INFO:tasks.workunit.client.0.smithi096.stdout:9/543: dwrite d1/d4/d7b/f81 [0,4194304] 0 2024-04-22T15:18:53.871 INFO:tasks.workunit.client.1.smithi196.stdout:1/230: mkdir d1/db/dc/de/d52 0 2024-04-22T15:18:53.871 INFO:tasks.workunit.client.0.smithi096.stdout:0/767: link d7/f32 d7/de/d5b/d95/d59/f104 0 2024-04-22T15:18:53.872 INFO:tasks.workunit.client.1.smithi196.stdout:1/231: chown d1/db/d16/f1d 1132908 1 2024-04-22T15:18:53.872 INFO:tasks.workunit.client.1.smithi196.stdout:1/232: chown d1/db/d16/d1e/l2f 14 1 2024-04-22T15:18:53.875 INFO:tasks.workunit.client.0.smithi096.stdout:6/525: creat d3/d28/dd/d11/d2c/d44/fb5 x:0 0 0 2024-04-22T15:18:53.876 INFO:tasks.workunit.client.1.smithi196.stdout:3/261: unlink d0/d15/d4d/db/dc/d4a/l4e 0 2024-04-22T15:18:53.877 INFO:tasks.workunit.client.1.smithi196.stdout:3/262: fsync d0/d15/d4d/db/dc/d4a/d44/f16 0 2024-04-22T15:18:53.884 INFO:tasks.workunit.client.1.smithi196.stdout:4/282: dread d6/d14/f12 [4194304,4194304] 0 2024-04-22T15:18:53.886 INFO:tasks.workunit.client.1.smithi196.stdout:6/297: link dd/d1f/d38/d40/f4f dd/f50 0 2024-04-22T15:18:53.887 INFO:tasks.workunit.client.1.smithi196.stdout:6/298: write dd/d1f/d37/d3c/f46 [4311194,2466] 0 2024-04-22T15:18:53.891 INFO:tasks.workunit.client.1.smithi196.stdout:0/267: rename d3/de/d1c/d27/c2b to d3/de/d14/d29/c61 0 2024-04-22T15:18:53.892 INFO:tasks.workunit.client.1.smithi196.stdout:0/268: read d3/de/d14/d29/f57 [252666,121053] 0 2024-04-22T15:18:53.892 INFO:tasks.workunit.client.0.smithi096.stdout:9/544: creat d1/d6f/d98/fb1 x:0 0 0 2024-04-22T15:18:53.893 INFO:tasks.workunit.client.1.smithi196.stdout:0/269: dread d3/de/f37 [0,4194304] 0 2024-04-22T15:18:53.893 INFO:tasks.workunit.client.1.smithi196.stdout:0/270: chown d3/d9/d25/f5b 132814684 1 2024-04-22T15:18:53.894 INFO:tasks.workunit.client.1.smithi196.stdout:0/271: write d3/de/d14/d29/f5e [291458,7760] 0 2024-04-22T15:18:53.898 INFO:tasks.workunit.client.0.smithi096.stdout:6/526: write d3/d28/dd/f70 [417906,8542] 0 2024-04-22T15:18:53.898 INFO:tasks.workunit.client.0.smithi096.stdout:3/506: getdents da/db/d10 0 2024-04-22T15:18:53.898 INFO:tasks.workunit.client.0.smithi096.stdout:2/629: rmdir d3/db/df/d98/db3 39 2024-04-22T15:18:53.900 INFO:tasks.workunit.client.0.smithi096.stdout:2/630: fdatasync d3/db/d2d/d46/d59/f76 0 2024-04-22T15:18:53.901 INFO:tasks.workunit.client.1.smithi196.stdout:9/257: creat db/f5f x:0 0 0 2024-04-22T15:18:53.901 INFO:tasks.workunit.client.1.smithi196.stdout:7/248: write d14/d25/f3b [255009,10096] 0 2024-04-22T15:18:53.902 INFO:tasks.workunit.client.1.smithi196.stdout:7/249: chown d14/d15/l34 5395563 1 2024-04-22T15:18:53.902 INFO:tasks.workunit.client.1.smithi196.stdout:1/233: stat d1/db/dc/d26/f30 0 2024-04-22T15:18:53.903 INFO:tasks.workunit.client.1.smithi196.stdout:7/250: write d14/d25/f42 [860741,76672] 0 2024-04-22T15:18:53.903 INFO:tasks.workunit.client.1.smithi196.stdout:1/234: readlink d1/db/d16/l20 0 2024-04-22T15:18:53.903 INFO:tasks.workunit.client.1.smithi196.stdout:7/251: read fd [6,129758] 0 2024-04-22T15:18:53.905 INFO:tasks.workunit.client.1.smithi196.stdout:5/263: truncate d1/d17/f24 406383 0 2024-04-22T15:18:53.906 INFO:tasks.workunit.client.0.smithi096.stdout:8/635: creat d4/d1c/d40/d46/d58/fde x:0 0 0 2024-04-22T15:18:53.907 INFO:tasks.workunit.client.1.smithi196.stdout:3/263: mknod d0/d15/d31/c57 0 2024-04-22T15:18:53.910 INFO:tasks.workunit.client.0.smithi096.stdout:9/545: rename d1/d6f to d1/d25/db2 0 2024-04-22T15:18:53.910 INFO:tasks.workunit.client.1.smithi196.stdout:5/264: dwrite d1/d17/d1a/f1d [0,4194304] 0 2024-04-22T15:18:53.912 INFO:tasks.workunit.client.0.smithi096.stdout:1/716: dwrite d6/f9d [0,4194304] 0 2024-04-22T15:18:53.917 INFO:tasks.workunit.client.0.smithi096.stdout:3/507: mknod da/db/dc/d59/ca1 0 2024-04-22T15:18:53.919 INFO:tasks.workunit.client.1.smithi196.stdout:4/283: write d6/d13/f16 [4149653,123677] 0 2024-04-22T15:18:53.921 INFO:tasks.workunit.client.1.smithi196.stdout:0/272: symlink d3/de/d1c/l62 0 2024-04-22T15:18:53.921 INFO:tasks.workunit.client.1.smithi196.stdout:0/273: stat d3/de/d1c/f31 0 2024-04-22T15:18:53.926 INFO:tasks.workunit.client.1.smithi196.stdout:1/235: creat d1/db/dc/d26/d2e/d32/f53 x:0 0 0 2024-04-22T15:18:53.928 INFO:tasks.workunit.client.0.smithi096.stdout:9/546: creat d1/d4/d79/fb3 x:0 0 0 2024-04-22T15:18:53.929 INFO:tasks.workunit.client.1.smithi196.stdout:7/252: symlink d14/d25/d31/l49 0 2024-04-22T15:18:53.929 INFO:tasks.workunit.client.1.smithi196.stdout:2/301: sync 2024-04-22T15:18:53.929 INFO:tasks.workunit.client.1.smithi196.stdout:2/302: fdatasync d1/d1d/f29 0 2024-04-22T15:18:53.930 INFO:tasks.workunit.client.0.smithi096.stdout:6/527: mknod d3/d28/d4d/d82/d9e/db0/cb6 0 2024-04-22T15:18:53.930 INFO:tasks.workunit.client.0.smithi096.stdout:8/636: dwrite d4/d10/f4e [0,4194304] 0 2024-04-22T15:18:53.931 INFO:tasks.workunit.client.1.smithi196.stdout:3/264: creat d0/d7/f58 x:0 0 0 2024-04-22T15:18:53.932 INFO:tasks.workunit.client.0.smithi096.stdout:6/528: write d3/d3a/d75/d83/f86 [1963487,25418] 0 2024-04-22T15:18:53.933 INFO:tasks.workunit.client.0.smithi096.stdout:6/529: chown d3/d28/dd/d11/l85 1 1 2024-04-22T15:18:53.934 INFO:tasks.workunit.client.0.smithi096.stdout:6/530: chown d3/d28/d4d/d82/c9a 379 1 2024-04-22T15:18:53.938 INFO:tasks.workunit.client.1.smithi196.stdout:3/265: dwrite d0/d15/d4d/db/f2e [0,4194304] 0 2024-04-22T15:18:53.938 INFO:tasks.workunit.client.1.smithi196.stdout:0/274: dread d3/f5 [0,4194304] 0 2024-04-22T15:18:53.940 INFO:tasks.workunit.client.1.smithi196.stdout:5/265: dwrite d1/d17/f21 [0,4194304] 0 2024-04-22T15:18:53.941 INFO:tasks.workunit.client.1.smithi196.stdout:4/284: mknod d6/d14/d18/d26/c52 0 2024-04-22T15:18:53.942 INFO:tasks.workunit.client.0.smithi096.stdout:8/637: dread d4/d71/f77 [0,4194304] 0 2024-04-22T15:18:53.944 INFO:tasks.workunit.client.1.smithi196.stdout:8/280: link d2/d35/c57 d2/d3f/c58 0 2024-04-22T15:18:53.948 INFO:tasks.workunit.client.0.smithi096.stdout:2/631: creat d3/db/df/fc6 x:0 0 0 2024-04-22T15:18:53.950 INFO:tasks.workunit.client.0.smithi096.stdout:8/638: dwrite d4/d10/d25/d38/f4b [0,4194304] 0 2024-04-22T15:18:53.950 INFO:tasks.workunit.client.1.smithi196.stdout:2/303: unlink d1/d1d/c53 0 2024-04-22T15:18:53.955 INFO:tasks.workunit.client.0.smithi096.stdout:9/547: creat d1/d4/d9/d43/d57/d5b/fb4 x:0 0 0 2024-04-22T15:18:53.959 INFO:tasks.workunit.client.0.smithi096.stdout:3/508: rmdir da/db/dc/d59/d7e 0 2024-04-22T15:18:53.959 INFO:tasks.workunit.client.0.smithi096.stdout:8/639: dread d4/d1c/d40/d46/d58/fb5 [0,4194304] 0 2024-04-22T15:18:53.959 INFO:tasks.workunit.client.1.smithi196.stdout:4/285: mkdir d6/d14/d18/d26/d32/d53 0 2024-04-22T15:18:53.960 INFO:tasks.workunit.client.0.smithi096.stdout:3/509: chown da/c9d 1533897174 1 2024-04-22T15:18:53.962 INFO:tasks.workunit.client.1.smithi196.stdout:5/266: creat d1/d4/f56 x:0 0 0 2024-04-22T15:18:53.962 INFO:tasks.workunit.client.1.smithi196.stdout:6/299: sync 2024-04-22T15:18:53.963 INFO:tasks.workunit.client.0.smithi096.stdout:2/632: truncate d3/d1f/d2f/f67 3736355 0 2024-04-22T15:18:53.963 INFO:tasks.workunit.client.1.smithi196.stdout:1/236: mkdir d1/d54 0 2024-04-22T15:18:53.964 INFO:tasks.workunit.client.0.smithi096.stdout:1/717: getdents d6/dc/d8a/d3f/d62/d7d/dd3 0 2024-04-22T15:18:53.967 INFO:tasks.workunit.client.1.smithi196.stdout:7/253: mkdir d14/d4a 0 2024-04-22T15:18:53.968 INFO:tasks.workunit.client.1.smithi196.stdout:6/300: dwrite dd/f15 [0,4194304] 0 2024-04-22T15:18:53.973 INFO:tasks.workunit.client.0.smithi096.stdout:3/510: creat da/d71/fa2 x:0 0 0 2024-04-22T15:18:53.977 INFO:tasks.workunit.client.0.smithi096.stdout:9/548: rename d1/d4/d9 to d1/df/d95/d9d/db5 0 2024-04-22T15:18:53.978 INFO:tasks.workunit.client.0.smithi096.stdout:9/549: chown d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae 25389 1 2024-04-22T15:18:53.978 INFO:tasks.workunit.client.1.smithi196.stdout:0/275: write d3/d6/f33 [3879205,12236] 0 2024-04-22T15:18:53.980 INFO:tasks.workunit.client.1.smithi196.stdout:9/258: getdents db/dd/de/d15 0 2024-04-22T15:18:53.981 INFO:tasks.workunit.client.0.smithi096.stdout:9/550: truncate d1/df/d95/d9d/db5/d43/d57/d5b/fb4 731804 0 2024-04-22T15:18:53.981 INFO:tasks.workunit.client.0.smithi096.stdout:1/718: dread - d6/dc/d8a/d3f/d54/d32/d3b/d94/fad zero size 2024-04-22T15:18:53.983 INFO:tasks.workunit.client.0.smithi096.stdout:6/531: write d3/d28/dd/d11/d2c/f5e [102004,103664] 0 2024-04-22T15:18:53.984 INFO:tasks.workunit.client.1.smithi196.stdout:5/267: creat d1/d17/d1c/f57 x:0 0 0 2024-04-22T15:18:53.984 INFO:tasks.workunit.client.0.smithi096.stdout:6/532: stat d3/d3a/d75/d83/f86 0 2024-04-22T15:18:53.990 INFO:tasks.workunit.client.0.smithi096.stdout:2/633: creat d3/db/df/d98/db3/fc7 x:0 0 0 2024-04-22T15:18:53.992 INFO:tasks.workunit.client.0.smithi096.stdout:8/640: write d4/d10/d25/d38/fce [1212918,67313] 0 2024-04-22T15:18:53.992 INFO:tasks.workunit.client.1.smithi196.stdout:8/281: link d2/d3f/f53 d2/d8/d49/f59 0 2024-04-22T15:18:53.992 INFO:tasks.workunit.client.1.smithi196.stdout:8/282: fdatasync d2/d8/f25 0 2024-04-22T15:18:53.993 INFO:tasks.workunit.client.1.smithi196.stdout:1/237: dwrite d1/db/dc/d26/d31/f50 [0,4194304] 0 2024-04-22T15:18:53.993 INFO:tasks.workunit.client.0.smithi096.stdout:8/641: write d4/d1c/d40/d48/fab [1127539,60553] 0 2024-04-22T15:18:53.994 INFO:tasks.workunit.client.0.smithi096.stdout:9/551: rename d1/df/d95/d9d/db5/dd/l26 to d1/d4/d79/lb6 0 2024-04-22T15:18:53.994 INFO:tasks.workunit.client.1.smithi196.stdout:8/283: dread - d2/d5/f48 zero size 2024-04-22T15:18:53.995 INFO:tasks.workunit.client.1.smithi196.stdout:1/238: read - d1/db/dc/d26/d2e/f51 zero size 2024-04-22T15:18:53.995 INFO:tasks.workunit.client.0.smithi096.stdout:8/642: write d4/d10/d25/d38/fce [3956795,69549] 0 2024-04-22T15:18:53.996 INFO:tasks.workunit.client.0.smithi096.stdout:1/719: rmdir d6/dc/d8a/d3f/d54/d32/d8c/dca 39 2024-04-22T15:18:53.996 INFO:tasks.workunit.client.0.smithi096.stdout:9/552: read - d1/df/d95/d9d/db5/d43/d57/d5b/fa4 zero size 2024-04-22T15:18:53.997 INFO:tasks.workunit.client.0.smithi096.stdout:8/643: fsync d4/d1c/d40/da7/db6/f4f 0 2024-04-22T15:18:53.998 INFO:tasks.workunit.client.0.smithi096.stdout:8/644: readlink d4/d10/l7c 0 2024-04-22T15:18:53.998 INFO:tasks.workunit.client.0.smithi096.stdout:9/553: dread - d1/df/d95/d9d/db5/d43/f97 zero size 2024-04-22T15:18:53.999 INFO:tasks.workunit.client.1.smithi196.stdout:7/254: creat d14/d15/f4b x:0 0 0 2024-04-22T15:18:54.000 INFO:tasks.workunit.client.1.smithi196.stdout:8/284: dwrite d2/d8/f27 [0,4194304] 0 2024-04-22T15:18:54.004 INFO:tasks.workunit.client.0.smithi096.stdout:3/511: write da/db/de/d19/f6f [496376,10970] 0 2024-04-22T15:18:54.005 INFO:tasks.workunit.client.1.smithi196.stdout:2/304: truncate d1/d23/d27/d2e/f47 975285 0 2024-04-22T15:18:54.005 INFO:tasks.workunit.client.0.smithi096.stdout:3/512: write da/db/de/d38/f78 [949090,10684] 0 2024-04-22T15:18:54.008 INFO:tasks.workunit.client.1.smithi196.stdout:4/286: creat d6/d14/d18/d26/d32/d53/f54 x:0 0 0 2024-04-22T15:18:54.014 INFO:tasks.workunit.client.1.smithi196.stdout:9/259: unlink db/dd/de/f52 0 2024-04-22T15:18:54.014 INFO:tasks.workunit.client.0.smithi096.stdout:1/720: creat d6/dc/d8a/d3f/d62/fd9 x:0 0 0 2024-04-22T15:18:54.015 INFO:tasks.workunit.client.1.smithi196.stdout:5/268: mknod d1/d17/d1c/c58 0 2024-04-22T15:18:54.015 INFO:tasks.workunit.client.1.smithi196.stdout:0/276: dwrite d3/d9/d25/f5b [0,4194304] 0 2024-04-22T15:18:54.015 INFO:tasks.workunit.client.1.smithi196.stdout:9/260: chown db/dd/de/d15/l53 14198034 1 2024-04-22T15:18:54.016 INFO:tasks.workunit.client.0.smithi096.stdout:6/533: write d3/d3a/d75/f5b [4821741,4638] 0 2024-04-22T15:18:54.018 INFO:tasks.workunit.client.1.smithi196.stdout:5/269: truncate d1/d17/d1a/f43 1045961 0 2024-04-22T15:18:54.019 INFO:tasks.workunit.client.1.smithi196.stdout:5/270: stat d1/d17/d1c/f3b 0 2024-04-22T15:18:54.043 INFO:tasks.workunit.client.0.smithi096.stdout:8/645: creat d4/d10/d25/d4c/d76/fdf x:0 0 0 2024-04-22T15:18:54.044 INFO:tasks.workunit.client.0.smithi096.stdout:2/634: write d3/f8e [612960,122426] 0 2024-04-22T15:18:54.065 INFO:tasks.workunit.client.0.smithi096.stdout:3/513: dread - da/db/d12/d22/f4e zero size 2024-04-22T15:18:54.070 INFO:tasks.workunit.client.0.smithi096.stdout:9/554: dwrite d1/df/d95/d9d/db5/d43/f44 [0,4194304] 0 2024-04-22T15:18:54.072 INFO:tasks.workunit.client.0.smithi096.stdout:9/555: fdatasync d1/df/d95/d9d/db5/d43/d57/d5b/d73/f9e 0 2024-04-22T15:18:54.072 INFO:tasks.workunit.client.1.smithi196.stdout:3/266: link d0/d7/l3b d0/d15/d4d/db/dc/d4a/l59 0 2024-04-22T15:18:54.079 INFO:tasks.workunit.client.1.smithi196.stdout:9/261: mknod db/d11/d29/d57/c60 0 2024-04-22T15:18:54.080 INFO:tasks.workunit.client.1.smithi196.stdout:0/277: rename d3/de/f37 to d3/d9/d25/f63 0 2024-04-22T15:18:54.081 INFO:tasks.workunit.client.1.smithi196.stdout:0/278: readlink d3/d6/l13 0 2024-04-22T15:18:54.081 INFO:tasks.workunit.client.0.smithi096.stdout:1/721: fdatasync d6/dc/d8a/d3f/d62/d7d/f84 0 2024-04-22T15:18:54.082 INFO:tasks.workunit.client.1.smithi196.stdout:5/271: write d1/d4/f22 [1665837,13173] 0 2024-04-22T15:18:54.082 INFO:tasks.workunit.client.0.smithi096.stdout:6/534: symlink d3/d28/d54/d68/lb7 0 2024-04-22T15:18:54.082 INFO:tasks.workunit.client.1.smithi196.stdout:0/279: fdatasync d3/de/d1c/d27/d2c/d30/f46 0 2024-04-22T15:18:54.083 INFO:tasks.workunit.client.1.smithi196.stdout:0/280: write d3/d9/d25/f5b [2982975,81325] 0 2024-04-22T15:18:54.084 INFO:tasks.workunit.client.0.smithi096.stdout:9/556: symlink d1/df/d95/d9d/db5/dd/lb7 0 2024-04-22T15:18:54.084 INFO:tasks.workunit.client.1.smithi196.stdout:4/287: sync 2024-04-22T15:18:54.084 INFO:tasks.workunit.client.1.smithi196.stdout:0/281: chown d3/d9/c12 56394585 1 2024-04-22T15:18:54.085 INFO:tasks.workunit.client.1.smithi196.stdout:4/288: truncate d6/d14/f2a 844615 0 2024-04-22T15:18:54.087 INFO:tasks.workunit.client.1.smithi196.stdout:1/239: creat d1/d54/f55 x:0 0 0 2024-04-22T15:18:54.090 INFO:tasks.workunit.client.0.smithi096.stdout:2/635: dwrite d3/fd [4194304,4194304] 0 2024-04-22T15:18:54.090 INFO:tasks.workunit.client.1.smithi196.stdout:8/285: fdatasync d2/f1d 0 2024-04-22T15:18:54.093 INFO:tasks.workunit.client.1.smithi196.stdout:2/305: write d1/d13/d25/f59 [933784,92882] 0 2024-04-22T15:18:54.093 INFO:tasks.workunit.client.1.smithi196.stdout:2/306: chown d1/f15 0 1 2024-04-22T15:18:54.096 INFO:tasks.workunit.client.0.smithi096.stdout:9/557: fsync d1/f38 0 2024-04-22T15:18:54.098 INFO:tasks.workunit.client.0.smithi096.stdout:8/646: creat d4/d2a/db7/fe0 x:0 0 0 2024-04-22T15:18:54.100 INFO:tasks.workunit.client.0.smithi096.stdout:8/647: chown d4/d1c/f72 72694431 1 2024-04-22T15:18:54.100 INFO:tasks.workunit.client.0.smithi096.stdout:9/558: write d1/df/d95/d9d/db5/dd/f46 [3511010,44436] 0 2024-04-22T15:18:54.102 INFO:tasks.workunit.client.0.smithi096.stdout:2/636: mkdir d3/db/df/d24/dc8 0 2024-04-22T15:18:54.104 INFO:tasks.workunit.client.0.smithi096.stdout:6/535: dwrite d3/d28/dd/d11/d2c/d6f/f89 [0,4194304] 0 2024-04-22T15:18:54.105 INFO:tasks.workunit.client.0.smithi096.stdout:8/648: truncate d4/d1c/d62/fba 264284 0 2024-04-22T15:18:54.105 INFO:tasks.workunit.client.0.smithi096.stdout:3/514: rmdir da/db/dc/d28/d96/d99 0 2024-04-22T15:18:54.107 INFO:tasks.workunit.client.1.smithi196.stdout:0/282: dread d3/d6/f48 [0,4194304] 0 2024-04-22T15:18:54.112 INFO:tasks.workunit.client.1.smithi196.stdout:6/301: getdents dd 0 2024-04-22T15:18:54.114 INFO:tasks.workunit.client.1.smithi196.stdout:6/302: read f4 [3298386,105211] 0 2024-04-22T15:18:54.116 INFO:tasks.workunit.client.0.smithi096.stdout:6/536: dwrite d3/d28/dd/f70 [0,4194304] 0 2024-04-22T15:18:54.116 INFO:tasks.workunit.client.0.smithi096.stdout:2/637: dwrite d3/db/df/d10/db5/fc2 [0,4194304] 0 2024-04-22T15:18:54.119 INFO:tasks.workunit.client.0.smithi096.stdout:8/649: dwrite d4/d1c/d40/da7/db6/f4f [0,4194304] 0 2024-04-22T15:18:54.119 INFO:tasks.workunit.client.0.smithi096.stdout:2/638: dread - d3/d1f/d5a/fc1 zero size 2024-04-22T15:18:54.121 INFO:tasks.workunit.client.1.smithi196.stdout:8/286: mknod d2/d8/d49/c5a 0 2024-04-22T15:18:54.122 INFO:tasks.workunit.client.0.smithi096.stdout:2/639: truncate d3/d1f/f56 4756655 0 2024-04-22T15:18:54.125 INFO:tasks.workunit.client.0.smithi096.stdout:1/722: creat d6/dc/d8a/d3f/fda x:0 0 0 2024-04-22T15:18:54.128 INFO:tasks.workunit.client.0.smithi096.stdout:6/537: dwrite d3/d28/f17 [0,4194304] 0 2024-04-22T15:18:54.137 INFO:tasks.workunit.client.0.smithi096.stdout:3/515: symlink da/db/de/d87/la3 0 2024-04-22T15:18:54.140 INFO:tasks.workunit.client.0.smithi096.stdout:1/723: truncate d6/dc/d8a/d3f/d54/d32/f6d 519564 0 2024-04-22T15:18:54.142 INFO:tasks.workunit.client.1.smithi196.stdout:4/289: mkdir d6/d13/d15/d2c/d35/d55 0 2024-04-22T15:18:54.143 INFO:tasks.workunit.client.1.smithi196.stdout:4/290: fsync d6/d14/f33 0 2024-04-22T15:18:54.144 INFO:tasks.workunit.client.0.smithi096.stdout:3/516: symlink da/db/d10/la4 0 2024-04-22T15:18:54.146 INFO:tasks.workunit.client.1.smithi196.stdout:7/255: getdents d14/d25 0 2024-04-22T15:18:54.147 INFO:tasks.workunit.client.1.smithi196.stdout:7/256: truncate d14/d15/f4b 527149 0 2024-04-22T15:18:54.148 INFO:tasks.workunit.client.1.smithi196.stdout:8/287: truncate d2/d5/f19 927209 0 2024-04-22T15:18:54.148 INFO:tasks.workunit.client.1.smithi196.stdout:7/257: chown f13 9 1 2024-04-22T15:18:54.149 INFO:tasks.workunit.client.1.smithi196.stdout:7/258: write d14/d25/f3b [741108,114049] 0 2024-04-22T15:18:54.149 INFO:tasks.workunit.client.1.smithi196.stdout:3/267: link d0/d15/d4d/db/dc/d4a/d44/f26 d0/d15/d4d/f5a 0 2024-04-22T15:18:54.152 INFO:tasks.workunit.client.1.smithi196.stdout:9/262: rename db/dd/de/d15/f3c to db/f61 0 2024-04-22T15:18:54.153 INFO:tasks.workunit.client.1.smithi196.stdout:9/263: stat db/dd/d1b/d2c/f43 0 2024-04-22T15:18:54.153 INFO:tasks.workunit.client.0.smithi096.stdout:3/517: creat da/d3f/fa5 x:0 0 0 2024-04-22T15:18:54.156 INFO:tasks.workunit.client.1.smithi196.stdout:9/264: dread db/dd/d1b/f45 [0,4194304] 0 2024-04-22T15:18:54.156 INFO:tasks.workunit.client.1.smithi196.stdout:0/283: symlink d3/de/l64 0 2024-04-22T15:18:54.157 INFO:tasks.workunit.client.1.smithi196.stdout:9/265: dread - db/d11/d29/f48 zero size 2024-04-22T15:18:54.158 INFO:tasks.workunit.client.1.smithi196.stdout:9/266: write f5 [238072,120353] 0 2024-04-22T15:18:54.158 INFO:tasks.workunit.client.0.smithi096.stdout:2/640: creat d3/db/fc9 x:0 0 0 2024-04-22T15:18:54.159 INFO:tasks.workunit.client.1.smithi196.stdout:9/267: write db/d11/d29/f38 [1688545,14414] 0 2024-04-22T15:18:54.163 INFO:tasks.workunit.client.0.smithi096.stdout:8/650: getdents d4/d1c/d40/d46 0 2024-04-22T15:18:54.165 INFO:tasks.workunit.client.0.smithi096.stdout:9/559: dwrite d1/df/d95/d9d/db5/d43/f6a [0,4194304] 0 2024-04-22T15:18:54.165 INFO:tasks.workunit.client.0.smithi096.stdout:8/651: write d4/d1c/d40/f98 [634062,45979] 0 2024-04-22T15:18:54.167 INFO:tasks.workunit.client.1.smithi196.stdout:8/288: creat d2/dc/d45/f5b x:0 0 0 2024-04-22T15:18:54.168 INFO:tasks.workunit.client.0.smithi096.stdout:8/652: dread - d4/d1c/d40/fdc zero size 2024-04-22T15:18:54.169 INFO:tasks.workunit.client.0.smithi096.stdout:6/538: getdents d3/d28/d54 0 2024-04-22T15:18:54.173 INFO:tasks.workunit.client.1.smithi196.stdout:7/259: truncate fe 3814813 0 2024-04-22T15:18:54.174 INFO:tasks.workunit.client.1.smithi196.stdout:6/303: dwrite dd/d1f/d37/f3f [0,4194304] 0 2024-04-22T15:18:54.176 INFO:tasks.workunit.client.1.smithi196.stdout:3/268: rename d0/d7/f58 to d0/d7/f5b 0 2024-04-22T15:18:54.176 INFO:tasks.workunit.client.1.smithi196.stdout:3/269: chown d0 143 1 2024-04-22T15:18:54.177 INFO:tasks.workunit.client.1.smithi196.stdout:6/304: dread dd/d20/f28 [0,4194304] 0 2024-04-22T15:18:54.177 INFO:tasks.workunit.client.1.smithi196.stdout:6/305: chown dd/d20 3155975 1 2024-04-22T15:18:54.177 INFO:tasks.workunit.client.0.smithi096.stdout:9/560: chown d1/df/d95/d9d/db5/f52 852922 1 2024-04-22T15:18:54.182 INFO:tasks.workunit.client.0.smithi096.stdout:1/724: dread d6/dc/d8a/d3f/d62/f70 [0,4194304] 0 2024-04-22T15:18:54.184 INFO:tasks.workunit.client.0.smithi096.stdout:1/725: fdatasync d6/dc/d8a/d3f/d54/d32/d3b/d56/f44 0 2024-04-22T15:18:54.185 INFO:tasks.workunit.client.1.smithi196.stdout:2/307: rename d1/d1d/f29 to d1/d23/d27/d2e/d55/d5b/f6e 0 2024-04-22T15:18:54.186 INFO:tasks.workunit.client.1.smithi196.stdout:1/240: sync 2024-04-22T15:18:54.187 INFO:tasks.workunit.client.1.smithi196.stdout:1/241: chown d1/db/d16/d1e/c3d 345597164 1 2024-04-22T15:18:54.188 INFO:tasks.workunit.client.1.smithi196.stdout:1/242: read d1/db/dc/d26/d31/f50 [2206629,119707] 0 2024-04-22T15:18:54.188 INFO:tasks.workunit.client.1.smithi196.stdout:5/272: link d1/l7 d1/d2/l59 0 2024-04-22T15:18:54.190 INFO:tasks.workunit.client.0.smithi096.stdout:3/518: dread da/db/de/d19/f2a [0,4194304] 0 2024-04-22T15:18:54.194 INFO:tasks.workunit.client.1.smithi196.stdout:5/273: dwrite d1/d17/d1c/f52 [0,4194304] 0 2024-04-22T15:18:54.197 INFO:tasks.workunit.client.1.smithi196.stdout:3/270: creat d0/d7/d8/f5c x:0 0 0 2024-04-22T15:18:54.198 INFO:tasks.workunit.client.1.smithi196.stdout:3/271: fsync d0/d15/d4d/db/dc/d4a/d44/f20 0 2024-04-22T15:18:54.199 INFO:tasks.workunit.client.0.smithi096.stdout:5/639: sync 2024-04-22T15:18:54.200 INFO:tasks.workunit.client.0.smithi096.stdout:5/640: chown d4/d11/d3d/d4f/l74 105 1 2024-04-22T15:18:54.201 INFO:tasks.workunit.client.1.smithi196.stdout:7/260: rename d14/d26/d28/d29/l47 to d14/d4a/l4c 0 2024-04-22T15:18:54.203 INFO:tasks.workunit.client.0.smithi096.stdout:2/641: truncate d3/db/df/d10/db5/fc2 98300 0 2024-04-22T15:18:54.204 INFO:tasks.workunit.client.0.smithi096.stdout:2/642: read d3/db/d30/f3e [3032089,26250] 0 2024-04-22T15:18:54.210 INFO:tasks.workunit.client.0.smithi096.stdout:2/643: dwrite d3/d1f/d2f/f34 [4194304,4194304] 0 2024-04-22T15:18:54.216 INFO:tasks.workunit.client.1.smithi196.stdout:9/268: creat db/dd/d1b/d4b/f62 x:0 0 0 2024-04-22T15:18:54.218 INFO:tasks.workunit.client.1.smithi196.stdout:0/284: dwrite d3/f47 [0,4194304] 0 2024-04-22T15:18:54.219 INFO:tasks.workunit.client.1.smithi196.stdout:1/243: dwrite d1/db/dc/de/f14 [0,4194304] 0 2024-04-22T15:18:54.219 INFO:tasks.workunit.client.1.smithi196.stdout:0/285: fdatasync d3/d9/d36/f4c 0 2024-04-22T15:18:54.221 INFO:tasks.workunit.client.1.smithi196.stdout:0/286: stat d3/d6/dc 0 2024-04-22T15:18:54.229 INFO:tasks.workunit.client.1.smithi196.stdout:9/269: dread db/d11/d29/f38 [0,4194304] 0 2024-04-22T15:18:54.232 INFO:tasks.workunit.client.1.smithi196.stdout:8/289: rmdir d2/d5/d2e/d51 0 2024-04-22T15:18:54.234 INFO:tasks.workunit.client.0.smithi096.stdout:4/651: sync 2024-04-22T15:18:54.234 INFO:tasks.workunit.client.0.smithi096.stdout:0/768: sync 2024-04-22T15:18:54.234 INFO:tasks.workunit.client.0.smithi096.stdout:7/860: sync 2024-04-22T15:18:54.235 INFO:tasks.workunit.client.1.smithi196.stdout:5/274: symlink d1/d53/l5a 0 2024-04-22T15:18:54.237 INFO:tasks.workunit.client.1.smithi196.stdout:3/272: mkdir d0/d15/d31/d55/d5d 0 2024-04-22T15:18:54.239 INFO:tasks.workunit.client.1.smithi196.stdout:3/273: chown d0/d15/d4d/db/dc/d4a/f45 5292 1 2024-04-22T15:18:54.239 INFO:tasks.workunit.client.1.smithi196.stdout:0/287: mkdir d3/d9/d36/d56/d65 0 2024-04-22T15:18:54.242 INFO:tasks.workunit.client.1.smithi196.stdout:4/291: sync 2024-04-22T15:18:54.243 INFO:tasks.workunit.client.1.smithi196.stdout:3/274: dwrite d0/d7/d8/f19 [0,4194304] 0 2024-04-22T15:18:54.244 INFO:tasks.workunit.client.1.smithi196.stdout:9/270: chown db/dd/de/d15/f1e 1431 1 2024-04-22T15:18:54.245 INFO:tasks.workunit.client.1.smithi196.stdout:8/290: mknod d2/d8/c5c 0 2024-04-22T15:18:54.246 INFO:tasks.workunit.client.1.smithi196.stdout:8/291: fdatasync d2/f4 0 2024-04-22T15:18:54.247 INFO:tasks.workunit.client.1.smithi196.stdout:5/275: mknod d1/d53/c5b 0 2024-04-22T15:18:54.248 INFO:tasks.workunit.client.1.smithi196.stdout:8/292: truncate d2/d5/d2e/f43 777889 0 2024-04-22T15:18:54.249 INFO:tasks.workunit.client.1.smithi196.stdout:1/244: truncate d1/db/dc/d26/f28 352203 0 2024-04-22T15:18:54.250 INFO:tasks.workunit.client.1.smithi196.stdout:1/245: write d1/db/dc/d26/d2e/f3e [728252,92416] 0 2024-04-22T15:18:54.251 INFO:tasks.workunit.client.1.smithi196.stdout:2/308: link d1/d13/d25/c3b d1/d13/d25/d2a/c6f 0 2024-04-22T15:18:54.252 INFO:tasks.workunit.client.1.smithi196.stdout:2/309: write d1/f3 [2141408,54431] 0 2024-04-22T15:18:54.256 INFO:tasks.workunit.client.1.smithi196.stdout:9/271: creat db/dd/de/d15/d31/f63 x:0 0 0 2024-04-22T15:18:54.257 INFO:tasks.workunit.client.1.smithi196.stdout:5/276: mknod d1/d4/c5c 0 2024-04-22T15:18:54.260 INFO:tasks.workunit.client.1.smithi196.stdout:8/293: mknod d2/d35/c5d 0 2024-04-22T15:18:54.262 INFO:tasks.workunit.client.1.smithi196.stdout:7/261: getdents d14/d26/d28 0 2024-04-22T15:18:54.263 INFO:tasks.workunit.client.0.smithi096.stdout:9/561: fdatasync d1/f20 0 2024-04-22T15:18:54.268 INFO:tasks.workunit.client.0.smithi096.stdout:1/726: rename d6/dc/d8a/d3f/d54/d32/d8c/d4a to d6/dc/d8a/d3f/d62/d7d/ddb 0 2024-04-22T15:18:54.269 INFO:tasks.workunit.client.0.smithi096.stdout:1/727: chown d6/dc/d8a/d3f/d54/d32/l3a 31027666 1 2024-04-22T15:18:54.269 INFO:tasks.workunit.client.1.smithi196.stdout:2/310: rename d1/d1d/c33 to d1/d13/d25/c70 0 2024-04-22T15:18:54.270 INFO:tasks.workunit.client.1.smithi196.stdout:2/311: chown d1/d13/d25/f3d 2316307 1 2024-04-22T15:18:54.270 INFO:tasks.workunit.client.1.smithi196.stdout:2/312: dread - d1/d13/d25/f3d zero size 2024-04-22T15:18:54.271 INFO:tasks.workunit.client.1.smithi196.stdout:3/275: mkdir d0/d15/d48/d51/d5e 0 2024-04-22T15:18:54.273 INFO:tasks.workunit.client.0.smithi096.stdout:5/641: creat d4/d11/d3d/d4d/d59/d98/fd1 x:0 0 0 2024-04-22T15:18:54.276 INFO:tasks.workunit.client.1.smithi196.stdout:1/246: symlink d1/db/l56 0 2024-04-22T15:18:54.276 INFO:tasks.workunit.client.0.smithi096.stdout:3/519: dwrite da/db/de/d19/d3e/f4f [0,4194304] 0 2024-04-22T15:18:54.280 INFO:tasks.workunit.client.0.smithi096.stdout:7/861: unlink d31/d32/d33/d54/d43/c94 0 2024-04-22T15:18:54.281 INFO:tasks.workunit.client.0.smithi096.stdout:7/862: chown d31/d32/d33/d54/d43/c9d 326 1 2024-04-22T15:18:54.283 INFO:tasks.workunit.client.1.smithi196.stdout:9/272: rename db/dd/d1b/d4b to db/dd/de/d15/d31/d64 0 2024-04-22T15:18:54.284 INFO:tasks.workunit.client.0.smithi096.stdout:4/652: dread d5/d7/d2c/d3d/d4f/d9c/fc6 [0,4194304] 0 2024-04-22T15:18:54.285 INFO:tasks.workunit.client.1.smithi196.stdout:9/273: write db/d11/d29/d57/f1f [1078143,92370] 0 2024-04-22T15:18:54.285 INFO:tasks.workunit.client.0.smithi096.stdout:4/653: readlink d5/d5b/l78 0 2024-04-22T15:18:54.289 INFO:tasks.workunit.client.0.smithi096.stdout:6/539: fdatasync d3/d28/dd/fa5 0 2024-04-22T15:18:54.290 INFO:tasks.workunit.client.0.smithi096.stdout:6/540: fdatasync f1 0 2024-04-22T15:18:54.290 INFO:tasks.workunit.client.1.smithi196.stdout:8/294: fsync d2/f3e 0 2024-04-22T15:18:54.291 INFO:tasks.workunit.client.1.smithi196.stdout:8/295: fsync d2/dc/f2f 0 2024-04-22T15:18:54.292 INFO:tasks.workunit.client.1.smithi196.stdout:6/306: sync 2024-04-22T15:18:54.293 INFO:tasks.workunit.client.0.smithi096.stdout:9/562: truncate d1/df/f58 4025220 0 2024-04-22T15:18:54.294 INFO:tasks.workunit.client.1.smithi196.stdout:9/274: dread db/d11/d29/d57/f1f [0,4194304] 0 2024-04-22T15:18:54.295 INFO:tasks.workunit.client.0.smithi096.stdout:8/653: creat d4/d10/d25/fe1 x:0 0 0 2024-04-22T15:18:54.295 INFO:tasks.workunit.client.1.smithi196.stdout:4/292: getdents d6/d14/d18/d26/d32/d53 0 2024-04-22T15:18:54.297 INFO:tasks.workunit.client.1.smithi196.stdout:6/307: dwrite dd/f15 [4194304,4194304] 0 2024-04-22T15:18:54.298 INFO:tasks.workunit.client.1.smithi196.stdout:6/308: chown dd/f12 32970798 1 2024-04-22T15:18:54.298 INFO:tasks.workunit.client.0.smithi096.stdout:5/642: fdatasync d4/d11/d3d/d4d/d99/fc2 0 2024-04-22T15:18:54.306 INFO:tasks.workunit.client.0.smithi096.stdout:7/863: creat d31/d32/d36/d75/dff/f122 x:0 0 0 2024-04-22T15:18:54.307 INFO:tasks.workunit.client.1.smithi196.stdout:3/276: mknod d0/d15/d48/d51/d5e/c5f 0 2024-04-22T15:18:54.309 INFO:tasks.workunit.client.0.smithi096.stdout:0/769: link d7/de/d5b/d95/f45 d7/de/d5b/dda/f105 0 2024-04-22T15:18:54.310 INFO:tasks.workunit.client.1.smithi196.stdout:5/277: creat d1/d17/f5d x:0 0 0 2024-04-22T15:18:54.310 INFO:tasks.workunit.client.0.smithi096.stdout:0/770: truncate d7/de/d5b/dda/f105 4578112 0 2024-04-22T15:18:54.311 INFO:tasks.workunit.client.1.smithi196.stdout:5/278: dread - d1/d17/f5d zero size 2024-04-22T15:18:54.311 INFO:tasks.workunit.client.0.smithi096.stdout:4/654: rename d5/d7/d2c/d3d/d4f/f91 to d5/d7/d2c/d3d/d4f/d88/fe2 0 2024-04-22T15:18:54.315 INFO:tasks.workunit.client.1.smithi196.stdout:5/279: dwrite d1/d4/f22 [0,4194304] 0 2024-04-22T15:18:54.318 INFO:tasks.workunit.client.0.smithi096.stdout:1/728: mknod d6/dc/d8a/d3f/d54/d32/d8c/cdc 0 2024-04-22T15:18:54.318 INFO:tasks.workunit.client.1.smithi196.stdout:0/288: sync 2024-04-22T15:18:54.320 INFO:tasks.workunit.client.0.smithi096.stdout:5/643: creat d4/d11/d25/fd2 x:0 0 0 2024-04-22T15:18:54.320 INFO:tasks.workunit.client.1.smithi196.stdout:0/289: read - d3/d9/f54 zero size 2024-04-22T15:18:54.322 INFO:tasks.workunit.client.1.smithi196.stdout:2/313: dwrite d1/f7 [0,4194304] 0 2024-04-22T15:18:54.322 INFO:tasks.workunit.client.1.smithi196.stdout:1/247: mknod d1/db/dc/c57 0 2024-04-22T15:18:54.323 INFO:tasks.workunit.client.1.smithi196.stdout:9/275: dread db/dd/d1b/d2c/f43 [0,4194304] 0 2024-04-22T15:18:54.324 INFO:tasks.workunit.client.0.smithi096.stdout:3/520: getdents da/db/d9a 0 2024-04-22T15:18:54.327 INFO:tasks.workunit.client.1.smithi196.stdout:4/293: dread - d6/d13/d15/d2c/d35/d44/f4a zero size 2024-04-22T15:18:54.328 INFO:tasks.workunit.client.0.smithi096.stdout:7/864: unlink d31/d32/dad/cde 0 2024-04-22T15:18:54.332 INFO:tasks.workunit.client.1.smithi196.stdout:1/248: dwrite d1/db/dc/d26/d2e/f3e [0,4194304] 0 2024-04-22T15:18:54.333 INFO:tasks.workunit.client.0.smithi096.stdout:0/771: read d7/d42/faa [127432,79432] 0 2024-04-22T15:18:54.334 INFO:tasks.workunit.client.1.smithi196.stdout:9/276: dwrite db/dd/de/d15/d31/f49 [0,4194304] 0 2024-04-22T15:18:54.335 INFO:tasks.workunit.client.0.smithi096.stdout:9/563: rename d1/df/d95/d9d/db5/d43/d57/d5b/f77 to d1/d25/db2/fb8 0 2024-04-22T15:18:54.339 INFO:tasks.workunit.client.0.smithi096.stdout:6/541: dwrite d3/d28/f1d [0,4194304] 0 2024-04-22T15:18:54.339 INFO:tasks.workunit.client.0.smithi096.stdout:4/655: creat d5/d7/d2c/d67/fe3 x:0 0 0 2024-04-22T15:18:54.348 INFO:tasks.workunit.client.1.smithi196.stdout:5/280: fdatasync d1/d14/d3e/f44 0 2024-04-22T15:18:54.348 INFO:tasks.workunit.client.0.smithi096.stdout:2/644: getdents d3/db/d2d/d46/d69 0 2024-04-22T15:18:54.352 INFO:tasks.workunit.client.0.smithi096.stdout:7/865: creat d31/d32/d33/d44/da8/dbf/f123 x:0 0 0 2024-04-22T15:18:54.355 INFO:tasks.workunit.client.0.smithi096.stdout:0/772: fdatasync d7/de/d3a/f44 0 2024-04-22T15:18:54.357 INFO:tasks.workunit.client.1.smithi196.stdout:7/262: rename d14/d26/d2a/d3d to d14/d4d 0 2024-04-22T15:18:54.358 INFO:tasks.workunit.client.0.smithi096.stdout:4/656: dread - d5/fa1 zero size 2024-04-22T15:18:54.360 INFO:tasks.workunit.client.0.smithi096.stdout:6/542: creat d3/d28/dd/d11/d2c/d44/d51/fb8 x:0 0 0 2024-04-22T15:18:54.360 INFO:tasks.workunit.client.0.smithi096.stdout:6/543: readlink d3/l74 0 2024-04-22T15:18:54.361 INFO:tasks.workunit.client.1.smithi196.stdout:9/277: readlink db/d11/d29/l4a 0 2024-04-22T15:18:54.361 INFO:tasks.workunit.client.1.smithi196.stdout:0/290: dread d3/f11 [0,4194304] 0 2024-04-22T15:18:54.361 INFO:tasks.workunit.client.1.smithi196.stdout:0/291: write d3/de/d14/d29/f5e [342591,1420] 0 2024-04-22T15:18:54.362 INFO:tasks.workunit.client.0.smithi096.stdout:2/645: mkdir d3/d13/dca 0 2024-04-22T15:18:54.363 INFO:tasks.workunit.client.1.smithi196.stdout:5/281: unlink d1/d2/f5 0 2024-04-22T15:18:54.364 INFO:tasks.workunit.client.1.smithi196.stdout:5/282: read d1/d14/d3e/f44 [1531013,120750] 0 2024-04-22T15:18:54.365 INFO:tasks.workunit.client.0.smithi096.stdout:3/521: link da/db/dc/f49 da/db/dc/d28/d96/d97/fa6 0 2024-04-22T15:18:54.366 INFO:tasks.workunit.client.1.smithi196.stdout:9/278: dwrite db/dd/de/d15/d31/f63 [0,4194304] 0 2024-04-22T15:18:54.367 INFO:tasks.workunit.client.1.smithi196.stdout:9/279: truncate db/d11/d29/d57/f5a 318686 0 2024-04-22T15:18:54.369 INFO:tasks.workunit.client.1.smithi196.stdout:4/294: mknod d6/d13/d15/d2c/d35/c56 0 2024-04-22T15:18:54.370 INFO:tasks.workunit.client.0.smithi096.stdout:9/564: mknod d1/d25/d9f/cb9 0 2024-04-22T15:18:54.371 INFO:tasks.workunit.client.0.smithi096.stdout:9/565: chown d1/df/d95/d9d/db5/dd/lb7 125162945 1 2024-04-22T15:18:54.372 INFO:tasks.workunit.client.0.smithi096.stdout:4/657: symlink d5/d7/d1f/d84/d6d/d98/le4 0 2024-04-22T15:18:54.376 INFO:tasks.workunit.client.0.smithi096.stdout:6/544: dread - d3/d28/dd/f2b zero size 2024-04-22T15:18:54.378 INFO:tasks.workunit.client.0.smithi096.stdout:6/545: write d3/d28/dd/d11/d2c/f5e [696240,6144] 0 2024-04-22T15:18:54.378 INFO:tasks.workunit.client.0.smithi096.stdout:5/644: write f1 [1387004,104052] 0 2024-04-22T15:18:54.379 INFO:tasks.workunit.client.0.smithi096.stdout:6/546: dread - d3/d28/d4d/d82/d9e/f92 zero size 2024-04-22T15:18:54.382 INFO:tasks.workunit.client.1.smithi196.stdout:3/277: link d0/d15/d4d/db/dc/d4a/d44/f3c d0/f60 0 2024-04-22T15:18:54.383 INFO:tasks.workunit.client.0.smithi096.stdout:3/522: dread da/db/de/d38/f78 [0,4194304] 0 2024-04-22T15:18:54.384 INFO:tasks.workunit.client.0.smithi096.stdout:8/654: rename d4/dc0/dc6 to d4/d1c/de2 0 2024-04-22T15:18:54.387 INFO:tasks.workunit.client.0.smithi096.stdout:9/566: mkdir d1/df/d95/d9d/db5/dd/dba 0 2024-04-22T15:18:54.389 INFO:tasks.workunit.client.0.smithi096.stdout:9/567: dread d1/df/d95/d9d/db5/d43/d57/d5b/fb4 [0,4194304] 0 2024-04-22T15:18:54.392 INFO:tasks.workunit.client.0.smithi096.stdout:2/646: mknod d3/d13/db4/dbf/ccb 0 2024-04-22T15:18:54.393 INFO:tasks.workunit.client.0.smithi096.stdout:2/647: chown d3/db/d2d/d46/d69/f6c 4070 1 2024-04-22T15:18:54.394 INFO:tasks.workunit.client.1.smithi196.stdout:0/292: dwrite d3/de/d1c/d27/f32 [0,4194304] 0 2024-04-22T15:18:54.397 INFO:tasks.workunit.client.1.smithi196.stdout:5/283: read d1/d17/f31 [2931690,128124] 0 2024-04-22T15:18:54.398 INFO:tasks.workunit.client.1.smithi196.stdout:8/296: getdents d2/d5/d2e 0 2024-04-22T15:18:54.399 INFO:tasks.workunit.client.0.smithi096.stdout:5/645: creat d4/d11/d3d/d4d/d59/d98/fd3 x:0 0 0 2024-04-22T15:18:54.399 INFO:tasks.workunit.client.0.smithi096.stdout:7/866: dwrite d31/d32/f59 [0,4194304] 0 2024-04-22T15:18:54.400 INFO:tasks.workunit.client.1.smithi196.stdout:5/284: chown d1/d14/d3e/f4e 58 1 2024-04-22T15:18:54.401 INFO:tasks.workunit.client.0.smithi096.stdout:7/867: dread - d31/d32/d33/d44/da8/dbf/f106 zero size 2024-04-22T15:18:54.401 INFO:tasks.workunit.client.1.smithi196.stdout:4/295: creat d6/d13/d15/d2c/d35/d44/f57 x:0 0 0 2024-04-22T15:18:54.403 INFO:tasks.workunit.client.1.smithi196.stdout:0/293: dwrite d3/de/d14/f39 [0,4194304] 0 2024-04-22T15:18:54.403 INFO:tasks.workunit.client.1.smithi196.stdout:4/296: fdatasync d6/d14/f33 0 2024-04-22T15:18:54.404 INFO:tasks.workunit.client.0.smithi096.stdout:6/547: creat d3/d28/d4d/d82/fb9 x:0 0 0 2024-04-22T15:18:54.407 INFO:tasks.workunit.client.1.smithi196.stdout:2/314: rename d1/c4 to d1/d23/d27/d2e/c71 0 2024-04-22T15:18:54.409 INFO:tasks.workunit.client.1.smithi196.stdout:2/315: write d1/d1d/f57 [1989240,111352] 0 2024-04-22T15:18:54.410 INFO:tasks.workunit.client.1.smithi196.stdout:7/263: unlink d14/d26/d2a/d2b/l39 0 2024-04-22T15:18:54.412 INFO:tasks.workunit.client.0.smithi096.stdout:6/548: dwrite d3/d28/f21 [0,4194304] 0 2024-04-22T15:18:54.414 INFO:tasks.workunit.client.1.smithi196.stdout:1/249: creat d1/db/f58 x:0 0 0 2024-04-22T15:18:54.415 INFO:tasks.workunit.client.1.smithi196.stdout:2/316: dwrite d1/d1d/f57 [0,4194304] 0 2024-04-22T15:18:54.416 INFO:tasks.workunit.client.1.smithi196.stdout:2/317: stat d1/d1d/d6a 0 2024-04-22T15:18:54.419 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:54 smithi096 ceph-mon[26688]: pgmap v33: 65 pgs: 65 active+clean; 1.5 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 41 MiB/s rd, 146 MiB/s wr, 304 op/s 2024-04-22T15:18:54.425 INFO:tasks.workunit.client.0.smithi096.stdout:9/568: symlink d1/d4/d7f/lbb 0 2024-04-22T15:18:54.427 INFO:tasks.workunit.client.1.smithi196.stdout:8/297: creat d2/dc/f5e x:0 0 0 2024-04-22T15:18:54.428 INFO:tasks.workunit.client.1.smithi196.stdout:8/298: dread - d2/dc/f5e zero size 2024-04-22T15:18:54.428 INFO:tasks.workunit.client.1.smithi196.stdout:8/299: read - d2/f4a zero size 2024-04-22T15:18:54.431 INFO:tasks.workunit.client.1.smithi196.stdout:9/280: rename db/dd/de/d15/d31/f49 to db/dd/d1b/f65 0 2024-04-22T15:18:54.433 INFO:tasks.workunit.client.0.smithi096.stdout:1/729: rename d6/dc/d8a/d3f/d54/d32/d8c/d45/f58 to d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/fdd 0 2024-04-22T15:18:54.435 INFO:tasks.workunit.client.0.smithi096.stdout:6/549: readlink d3/lb 0 2024-04-22T15:18:54.436 INFO:tasks.workunit.client.1.smithi196.stdout:1/250: write d1/f4 [7623302,70285] 0 2024-04-22T15:18:54.436 INFO:tasks.workunit.client.1.smithi196.stdout:6/309: link f1 dd/d1f/d38/f51 0 2024-04-22T15:18:54.437 INFO:tasks.workunit.client.1.smithi196.stdout:3/278: truncate d0/f3 2402763 0 2024-04-22T15:18:54.438 INFO:tasks.workunit.client.0.smithi096.stdout:9/569: rename d1/d85/fa2 to d1/d85/fbc 0 2024-04-22T15:18:54.439 INFO:tasks.workunit.client.1.smithi196.stdout:3/279: dread d0/d7/f1a [0,4194304] 0 2024-04-22T15:18:54.441 INFO:tasks.workunit.client.1.smithi196.stdout:4/297: dread d6/d14/f12 [0,4194304] 0 2024-04-22T15:18:54.444 INFO:tasks.workunit.client.0.smithi096.stdout:0/773: rename d7/de/d5b/d95/dc/l52 to d7/de/d5b/d95/d15/l106 0 2024-04-22T15:18:54.444 INFO:tasks.workunit.client.0.smithi096.stdout:3/523: rename da/db/dc/d28/d69/d89 to da/db/dc/d28/d69/d89/da7 22 2024-04-22T15:18:54.445 INFO:tasks.workunit.client.1.smithi196.stdout:8/300: rename d2/d35/f39 to d2/d35/f5f 0 2024-04-22T15:18:54.445 INFO:tasks.workunit.client.1.smithi196.stdout:8/301: write d2/f4 [5806950,61883] 0 2024-04-22T15:18:54.447 INFO:tasks.workunit.client.1.smithi196.stdout:8/302: write d2/dc/d45/f5b [62418,15844] 0 2024-04-22T15:18:54.447 INFO:tasks.workunit.client.1.smithi196.stdout:9/281: symlink db/dd/de/d15/d31/l66 0 2024-04-22T15:18:54.448 INFO:tasks.workunit.client.1.smithi196.stdout:9/282: chown db/d11/d29/d57/c22 225040279 1 2024-04-22T15:18:54.448 INFO:tasks.workunit.client.0.smithi096.stdout:1/730: creat d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e/fde x:0 0 0 2024-04-22T15:18:54.449 INFO:tasks.workunit.client.1.smithi196.stdout:9/283: write db/dd/d1b/d2c/f58 [610641,20521] 0 2024-04-22T15:18:54.450 INFO:tasks.workunit.client.1.smithi196.stdout:7/264: symlink d14/l4e 0 2024-04-22T15:18:54.454 INFO:tasks.workunit.client.1.smithi196.stdout:1/251: chown d1/db/dc/d26/d2e/d32/f53 69866 1 2024-04-22T15:18:54.458 INFO:tasks.workunit.client.0.smithi096.stdout:9/570: mkdir d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dbd 0 2024-04-22T15:18:54.460 INFO:tasks.workunit.client.0.smithi096.stdout:7/868: link c15 d31/d32/d36/c124 0 2024-04-22T15:18:54.461 INFO:tasks.workunit.client.1.smithi196.stdout:3/280: symlink d0/d15/d31/d55/l61 0 2024-04-22T15:18:54.464 INFO:tasks.workunit.client.0.smithi096.stdout:0/774: rename d7/de/fd7 to d7/de/d5b/d64/f107 0 2024-04-22T15:18:54.465 INFO:tasks.workunit.client.1.smithi196.stdout:3/281: dread d0/d15/d4d/db/dc/d4a/d44/f3c [0,4194304] 0 2024-04-22T15:18:54.465 INFO:tasks.workunit.client.0.smithi096.stdout:4/658: dwrite d5/d7/d1a/f38 [0,4194304] 0 2024-04-22T15:18:54.465 INFO:tasks.workunit.client.1.smithi196.stdout:0/294: write d3/f11 [2681017,11027] 0 2024-04-22T15:18:54.466 INFO:tasks.workunit.client.1.smithi196.stdout:0/295: fdatasync d3/de/d1c/d27/d2c/d51/f53 0 2024-04-22T15:18:54.467 INFO:tasks.workunit.client.0.smithi096.stdout:4/659: stat d5/d7/d1a/d25/c31 0 2024-04-22T15:18:54.467 INFO:tasks.workunit.client.0.smithi096.stdout:8/655: getdents d4/d10/d25/d4c/d76/d7f 0 2024-04-22T15:18:54.469 INFO:tasks.workunit.client.0.smithi096.stdout:0/775: chown d7/de/d3a/d61/dbb/fea 1947153 1 2024-04-22T15:18:54.470 INFO:tasks.workunit.client.1.smithi196.stdout:3/282: dread d0/d15/d4d/db/dc/d4a/f50 [0,4194304] 0 2024-04-22T15:18:54.470 INFO:tasks.workunit.client.0.smithi096.stdout:2/648: dwrite d3/db/df/fa5 [4194304,4194304] 0 2024-04-22T15:18:54.471 INFO:tasks.workunit.client.1.smithi196.stdout:3/283: write d0/d15/d4d/f2d [1483591,61512] 0 2024-04-22T15:18:54.472 INFO:tasks.workunit.client.1.smithi196.stdout:3/284: stat d0/d15/l14 0 2024-04-22T15:18:54.473 INFO:tasks.workunit.client.0.smithi096.stdout:3/524: creat da/db/de/d19/d2d/fa8 x:0 0 0 2024-04-22T15:18:54.475 INFO:tasks.workunit.client.0.smithi096.stdout:1/731: fsync d6/dc/d8a/d3f/d54/d32/d3b/d94/fad 0 2024-04-22T15:18:54.476 INFO:tasks.workunit.client.1.smithi196.stdout:9/284: fsync db/d11/d29/f38 0 2024-04-22T15:18:54.480 INFO:tasks.workunit.client.1.smithi196.stdout:7/265: symlink d14/d26/d2a/l4f 0 2024-04-22T15:18:54.481 INFO:tasks.workunit.client.0.smithi096.stdout:5/646: getdents d4/d11/d3d/d4d/d59/db0/db2 0 2024-04-22T15:18:54.487 INFO:tasks.workunit.client.1.smithi196.stdout:6/310: mknod dd/d1f/d4e/c52 0 2024-04-22T15:18:54.487 INFO:tasks.workunit.client.0.smithi096.stdout:3/525: dread da/db/d12/d22/f61 [0,4194304] 0 2024-04-22T15:18:54.488 INFO:tasks.workunit.client.0.smithi096.stdout:9/571: symlink d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/lbe 0 2024-04-22T15:18:54.489 INFO:tasks.workunit.client.1.smithi196.stdout:5/285: getdents d1/d17/d50 0 2024-04-22T15:18:54.491 INFO:tasks.workunit.client.1.smithi196.stdout:5/286: truncate d1/d9/f38 1481236 0 2024-04-22T15:18:54.491 INFO:tasks.workunit.client.1.smithi196.stdout:5/287: chown d1/d17/d1c 0 1 2024-04-22T15:18:54.492 INFO:tasks.workunit.client.1.smithi196.stdout:6/311: dwrite dd/f16 [0,4194304] 0 2024-04-22T15:18:54.494 INFO:tasks.workunit.client.0.smithi096.stdout:8/656: mknod d4/d2a/d7e/ce3 0 2024-04-22T15:18:54.495 INFO:tasks.workunit.client.0.smithi096.stdout:4/660: dread d5/d7/d1f/f62 [0,4194304] 0 2024-04-22T15:18:54.495 INFO:tasks.workunit.client.1.smithi196.stdout:4/298: mknod d6/d13/d15/d2c/d35/c58 0 2024-04-22T15:18:54.498 INFO:tasks.workunit.client.1.smithi196.stdout:5/288: dread d1/d17/d1c/f52 [0,4194304] 0 2024-04-22T15:18:54.500 INFO:tasks.workunit.client.0.smithi096.stdout:2/649: symlink d3/db/df/d24/d58/lcc 0 2024-04-22T15:18:54.502 INFO:tasks.workunit.client.1.smithi196.stdout:8/303: creat d2/d8/d47/f60 x:0 0 0 2024-04-22T15:18:54.505 INFO:tasks.workunit.client.0.smithi096.stdout:2/650: dread d3/d1f/f56 [4194304,4194304] 0 2024-04-22T15:18:54.505 INFO:tasks.workunit.client.0.smithi096.stdout:2/651: readlink d3/d1f/d2f/l43 0 2024-04-22T15:18:54.506 INFO:tasks.workunit.client.0.smithi096.stdout:6/550: creat d3/d28/dd/fba x:0 0 0 2024-04-22T15:18:54.507 INFO:tasks.workunit.client.1.smithi196.stdout:9/285: symlink db/dd/de/d15/l67 0 2024-04-22T15:18:54.507 INFO:tasks.workunit.client.0.smithi096.stdout:6/551: truncate d3/d28/dd/f70 4418388 0 2024-04-22T15:18:54.508 INFO:tasks.workunit.client.0.smithi096.stdout:5/647: readlink d4/d1f/l20 0 2024-04-22T15:18:54.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:54 smithi196 ceph-mon[34717]: pgmap v33: 65 pgs: 65 active+clean; 1.5 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 41 MiB/s rd, 146 MiB/s wr, 304 op/s 2024-04-22T15:18:54.509 INFO:tasks.workunit.client.0.smithi096.stdout:5/648: chown d4/d11/d3d/d4d/d59/d5a/cc1 1 1 2024-04-22T15:18:54.511 INFO:tasks.workunit.client.1.smithi196.stdout:6/312: dread dd/f11 [4194304,4194304] 0 2024-04-22T15:18:54.515 INFO:tasks.workunit.client.1.smithi196.stdout:5/289: dread d1/d9/f20 [0,4194304] 0 2024-04-22T15:18:54.516 INFO:tasks.workunit.client.1.smithi196.stdout:5/290: chown d1/d17/d50 5484845 1 2024-04-22T15:18:54.518 INFO:tasks.workunit.client.1.smithi196.stdout:0/296: symlink d3/d9/d36/d56/d65/l66 0 2024-04-22T15:18:54.519 INFO:tasks.workunit.client.1.smithi196.stdout:0/297: dread d3/de/d1c/d27/d2c/d30/f46 [0,4194304] 0 2024-04-22T15:18:54.521 INFO:tasks.workunit.client.0.smithi096.stdout:0/776: mkdir d7/de/d5b/d108 0 2024-04-22T15:18:54.522 INFO:tasks.workunit.client.0.smithi096.stdout:0/777: readlink d7/de/d5b/d95/d66/lc8 0 2024-04-22T15:18:54.528 INFO:tasks.workunit.client.1.smithi196.stdout:3/285: truncate d0/d15/d4d/db/dc/d4a/d44/f20 3926564 0 2024-04-22T15:18:54.529 INFO:tasks.workunit.client.0.smithi096.stdout:9/572: dwrite d1/df/d95/d9d/db5/dd/f41 [4194304,4194304] 0 2024-04-22T15:18:54.530 INFO:tasks.workunit.client.0.smithi096.stdout:2/652: chown d3/l5 12 1 2024-04-22T15:18:54.532 INFO:tasks.workunit.client.1.smithi196.stdout:9/286: creat db/dd/d1b/f68 x:0 0 0 2024-04-22T15:18:54.534 INFO:tasks.workunit.client.0.smithi096.stdout:5/649: creat d4/d11/d3d/d35/d57/fd4 x:0 0 0 2024-04-22T15:18:54.534 INFO:tasks.workunit.client.1.smithi196.stdout:1/252: getdents d1/db/dc/d26/d2e/d32 0 2024-04-22T15:18:54.536 INFO:tasks.workunit.client.0.smithi096.stdout:3/526: link da/db/dc/d28/f85 da/db/d12/fa9 0 2024-04-22T15:18:54.537 INFO:tasks.workunit.client.1.smithi196.stdout:6/313: mkdir dd/d1f/d4e/d53 0 2024-04-22T15:18:54.537 INFO:tasks.workunit.client.0.smithi096.stdout:7/869: link d31/d32/d36/d5d/l97 d31/d32/dad/l125 0 2024-04-22T15:18:54.538 INFO:tasks.workunit.client.1.smithi196.stdout:2/318: sync 2024-04-22T15:18:54.538 INFO:tasks.workunit.client.0.smithi096.stdout:8/657: fsync d4/d10/d4d/f74 0 2024-04-22T15:18:54.539 INFO:tasks.workunit.client.1.smithi196.stdout:9/287: dread fa [0,4194304] 0 2024-04-22T15:18:54.540 INFO:tasks.workunit.client.1.smithi196.stdout:5/291: mkdir d1/d17/d1a/d2f/d30/d5e 0 2024-04-22T15:18:54.541 INFO:tasks.workunit.client.0.smithi096.stdout:4/661: readlink d5/d7/d2c/d43/l54 0 2024-04-22T15:18:54.542 INFO:tasks.workunit.client.1.smithi196.stdout:4/299: symlink d6/d13/d15/l59 0 2024-04-22T15:18:54.543 INFO:tasks.workunit.client.0.smithi096.stdout:4/662: write d5/d7/d1a/fcc [1978801,43116] 0 2024-04-22T15:18:54.543 INFO:tasks.workunit.client.0.smithi096.stdout:3/527: dwrite da/db/d12/d22/f53 [0,4194304] 0 2024-04-22T15:18:54.544 INFO:tasks.workunit.client.0.smithi096.stdout:0/778: mknod d7/d42/d4c/d55/d9c/c109 0 2024-04-22T15:18:54.545 INFO:tasks.workunit.client.1.smithi196.stdout:0/298: unlink d3/de/d1c/l62 0 2024-04-22T15:18:54.548 INFO:tasks.workunit.client.1.smithi196.stdout:8/304: link d2/dc/f5e d2/d35/f61 0 2024-04-22T15:18:54.548 INFO:tasks.workunit.client.1.smithi196.stdout:4/300: dwrite d6/d13/f1b [0,4194304] 0 2024-04-22T15:18:54.554 INFO:tasks.workunit.client.1.smithi196.stdout:3/286: rename d0/d15/d4d/db/l37 to d0/d15/d4d/db/dc/d4a/l62 0 2024-04-22T15:18:54.555 INFO:tasks.workunit.client.0.smithi096.stdout:9/573: dwrite d1/f32 [0,4194304] 0 2024-04-22T15:18:54.556 INFO:tasks.workunit.client.1.smithi196.stdout:0/299: dwrite d3/d9/d36/f4c [0,4194304] 0 2024-04-22T15:18:54.556 INFO:tasks.workunit.client.0.smithi096.stdout:5/650: fsync d4/d2c/d2f/d31/f8f 0 2024-04-22T15:18:54.560 INFO:tasks.workunit.client.1.smithi196.stdout:1/253: dread d1/db/dc/d26/f28 [0,4194304] 0 2024-04-22T15:18:54.564 INFO:tasks.workunit.client.0.smithi096.stdout:9/574: dwrite d1/df/d95/d9d/db5/d43/f97 [0,4194304] 0 2024-04-22T15:18:54.566 INFO:tasks.workunit.client.0.smithi096.stdout:9/575: chown d1/df/d95/d9d/db5/dd/f46 1699 1 2024-04-22T15:18:54.566 INFO:tasks.workunit.client.1.smithi196.stdout:6/314: creat dd/d1f/d4e/f54 x:0 0 0 2024-04-22T15:18:54.567 INFO:tasks.workunit.client.1.smithi196.stdout:2/319: symlink d1/d23/d27/d2e/d55/l72 0 2024-04-22T15:18:54.569 INFO:tasks.workunit.client.1.smithi196.stdout:7/266: sync 2024-04-22T15:18:54.571 INFO:tasks.workunit.client.1.smithi196.stdout:7/267: truncate d14/d15/f18 4417125 0 2024-04-22T15:18:54.571 INFO:tasks.workunit.client.0.smithi096.stdout:3/528: rmdir da/d71 39 2024-04-22T15:18:54.575 INFO:tasks.workunit.client.0.smithi096.stdout:0/779: rmdir d7/d42/d4c/d55 39 2024-04-22T15:18:54.576 INFO:tasks.workunit.client.1.smithi196.stdout:9/288: dread db/dd/f5c [0,4194304] 0 2024-04-22T15:18:54.582 INFO:tasks.workunit.client.1.smithi196.stdout:8/305: rename d2/f4 to d2/d8/d47/f62 0 2024-04-22T15:18:54.583 INFO:tasks.workunit.client.0.smithi096.stdout:5/651: write d4/d11/d3d/d35/fc9 [81210,89275] 0 2024-04-22T15:18:54.584 INFO:tasks.workunit.client.1.smithi196.stdout:0/300: stat d3/de/d1c/d27/c60 0 2024-04-22T15:18:54.584 INFO:tasks.workunit.client.0.smithi096.stdout:5/652: chown d4/d11/dc5 755648 1 2024-04-22T15:18:54.584 INFO:tasks.workunit.client.1.smithi196.stdout:0/301: fdatasync d3/de/d14/f4a 0 2024-04-22T15:18:54.586 INFO:tasks.workunit.client.0.smithi096.stdout:9/576: creat d1/d25/db2/d98/fbf x:0 0 0 2024-04-22T15:18:54.587 INFO:tasks.workunit.client.0.smithi096.stdout:9/577: stat d1/df/d95/d9d/db5/d43/f6a 0 2024-04-22T15:18:54.587 INFO:tasks.workunit.client.1.smithi196.stdout:1/254: symlink d1/db/dc/d26/d2e/d32/d36/l59 0 2024-04-22T15:18:54.588 INFO:tasks.workunit.client.0.smithi096.stdout:3/529: truncate da/db/d12/d22/f31 3756415 0 2024-04-22T15:18:54.590 INFO:tasks.workunit.client.0.smithi096.stdout:4/663: symlink d5/d7/d2c/d43/le5 0 2024-04-22T15:18:54.594 INFO:tasks.workunit.client.0.smithi096.stdout:0/780: unlink d7/de/d5b/d95/d15/c97 0 2024-04-22T15:18:54.594 INFO:tasks.workunit.client.0.smithi096.stdout:4/664: write d5/d7/d2c/d95/fc0 [777861,36954] 0 2024-04-22T15:18:54.594 INFO:tasks.workunit.client.0.smithi096.stdout:9/578: dread d1/df/d95/d9d/db5/d43/d57/d5b/d73/f9a [0,4194304] 0 2024-04-22T15:18:54.595 INFO:tasks.workunit.client.0.smithi096.stdout:2/653: creat d3/db/df/fcd x:0 0 0 2024-04-22T15:18:54.596 INFO:tasks.workunit.client.0.smithi096.stdout:0/781: readlink d7/de/d5b/d95/lcb 0 2024-04-22T15:18:54.597 INFO:tasks.workunit.client.0.smithi096.stdout:6/552: getdents d3/d3a/d75 0 2024-04-22T15:18:54.599 INFO:tasks.workunit.client.0.smithi096.stdout:7/870: dwrite d31/d32/d33/f3e [0,4194304] 0 2024-04-22T15:18:54.600 INFO:tasks.workunit.client.1.smithi196.stdout:4/301: creat d6/d13/d15/d2c/d35/d55/f5a x:0 0 0 2024-04-22T15:18:54.601 INFO:tasks.workunit.client.0.smithi096.stdout:9/579: read d1/df/d95/d9d/db5/dd/f1a [2870952,58552] 0 2024-04-22T15:18:54.606 INFO:tasks.workunit.client.1.smithi196.stdout:9/289: rename db/dd/d1b/f65 to db/dd/f69 0 2024-04-22T15:18:54.606 INFO:tasks.workunit.client.1.smithi196.stdout:3/287: dread d0/d15/d4d/db/dc/d18/f4f [0,4194304] 0 2024-04-22T15:18:54.607 INFO:tasks.workunit.client.0.smithi096.stdout:3/530: rmdir da/db/d12/d22/d26 39 2024-04-22T15:18:54.610 INFO:tasks.workunit.client.1.smithi196.stdout:8/306: creat d2/d35/f63 x:0 0 0 2024-04-22T15:18:54.610 INFO:tasks.workunit.client.1.smithi196.stdout:8/307: chown d2/d8 3189632 1 2024-04-22T15:18:54.612 INFO:tasks.workunit.client.1.smithi196.stdout:8/308: truncate d2/d5/f48 849384 0 2024-04-22T15:18:54.612 INFO:tasks.workunit.client.1.smithi196.stdout:1/255: mkdir d1/db/dc/de/d5a 0 2024-04-22T15:18:54.613 INFO:tasks.workunit.client.1.smithi196.stdout:1/256: chown d1/db/dc/de/c1a 4864 1 2024-04-22T15:18:54.614 INFO:tasks.workunit.client.1.smithi196.stdout:6/315: dwrite dd/f1a [0,4194304] 0 2024-04-22T15:18:54.616 INFO:tasks.workunit.client.0.smithi096.stdout:4/665: dwrite d5/d7/d1a/f6c [0,4194304] 0 2024-04-22T15:18:54.618 INFO:tasks.workunit.client.1.smithi196.stdout:5/292: unlink d1/d2/f25 0 2024-04-22T15:18:54.619 INFO:tasks.workunit.client.0.smithi096.stdout:3/531: dread da/db/de/d8b/f92 [0,4194304] 0 2024-04-22T15:18:54.620 INFO:tasks.workunit.client.1.smithi196.stdout:8/309: dwrite d2/d5/f48 [0,4194304] 0 2024-04-22T15:18:54.620 INFO:tasks.workunit.client.0.smithi096.stdout:6/553: write d3/d28/dd/f16 [3832383,52673] 0 2024-04-22T15:18:54.620 INFO:tasks.workunit.client.1.smithi196.stdout:8/310: write d2/dc/f3a [1090732,44920] 0 2024-04-22T15:18:54.621 INFO:tasks.workunit.client.1.smithi196.stdout:2/320: mkdir d1/d23/d27/d73 0 2024-04-22T15:18:54.622 INFO:tasks.workunit.client.0.smithi096.stdout:6/554: chown d3/d28/dd/d11/d2c/d6f/f99 33348 1 2024-04-22T15:18:54.627 INFO:tasks.workunit.client.1.smithi196.stdout:8/311: dread d2/d5/d2e/f43 [0,4194304] 0 2024-04-22T15:18:54.628 INFO:tasks.workunit.client.1.smithi196.stdout:8/312: read - d2/d8/f4c zero size 2024-04-22T15:18:54.632 INFO:tasks.workunit.client.1.smithi196.stdout:9/290: creat db/d11/f6a x:0 0 0 2024-04-22T15:18:54.633 INFO:tasks.workunit.client.1.smithi196.stdout:3/288: creat d0/d15/d48/d51/d5e/f63 x:0 0 0 2024-04-22T15:18:54.644 INFO:tasks.workunit.client.0.smithi096.stdout:6/555: mkdir d3/d28/d54/d68/d6e/dbb 0 2024-04-22T15:18:54.645 INFO:tasks.workunit.client.0.smithi096.stdout:5/653: link d4/ce d4/d11/d3d/d35/d57/cd5 0 2024-04-22T15:18:54.648 INFO:tasks.workunit.client.0.smithi096.stdout:9/580: mknod d1/df/d95/d9d/db5/d59/cc0 0 2024-04-22T15:18:54.648 INFO:tasks.workunit.client.1.smithi196.stdout:7/268: dwrite d14/d26/f35 [0,4194304] 0 2024-04-22T15:18:54.649 INFO:tasks.workunit.client.0.smithi096.stdout:0/782: mknod d7/d42/d4c/d55/d98/c10a 0 2024-04-22T15:18:54.649 INFO:tasks.workunit.client.1.smithi196.stdout:2/321: dread d1/d23/f2f [0,4194304] 0 2024-04-22T15:18:54.652 INFO:tasks.workunit.client.0.smithi096.stdout:2/654: rename d3/db/d2d to d3/db/df/dce 0 2024-04-22T15:18:54.653 INFO:tasks.workunit.client.1.smithi196.stdout:7/269: dread d14/d25/d31/f3f [0,4194304] 0 2024-04-22T15:18:54.654 INFO:tasks.workunit.client.1.smithi196.stdout:7/270: write d14/f22 [745088,9537] 0 2024-04-22T15:18:54.655 INFO:tasks.workunit.client.1.smithi196.stdout:8/313: mknod d2/d5/d4f/c64 0 2024-04-22T15:18:54.656 INFO:tasks.workunit.client.0.smithi096.stdout:3/532: rmdir da/db/de/d19/d2d 39 2024-04-22T15:18:54.656 INFO:tasks.workunit.client.1.smithi196.stdout:8/314: write d2/dc/f18 [754491,52283] 0 2024-04-22T15:18:54.657 INFO:tasks.workunit.client.1.smithi196.stdout:8/315: fsync d2/d5/f23 0 2024-04-22T15:18:54.657 INFO:tasks.workunit.client.0.smithi096.stdout:4/666: write d5/d7/d1f/f50 [1396810,35902] 0 2024-04-22T15:18:54.657 INFO:tasks.workunit.client.0.smithi096.stdout:9/581: dwrite d1/df/d95/d9d/db5/d43/d57/d5b/fa4 [0,4194304] 0 2024-04-22T15:18:54.658 INFO:tasks.workunit.client.1.smithi196.stdout:8/316: stat d2/d8/d49/l50 0 2024-04-22T15:18:54.659 INFO:tasks.workunit.client.0.smithi096.stdout:4/667: chown d5/d7/f20 3 1 2024-04-22T15:18:54.660 INFO:tasks.workunit.client.0.smithi096.stdout:5/654: creat d4/d2c/d2f/d30/fd6 x:0 0 0 2024-04-22T15:18:54.665 INFO:tasks.workunit.client.0.smithi096.stdout:5/655: dwrite d4/d11/d3d/d4d/d59/d5a/f13 [0,4194304] 0 2024-04-22T15:18:54.665 INFO:tasks.workunit.client.1.smithi196.stdout:9/291: dwrite db/dd/d1b/f51 [0,4194304] 0 2024-04-22T15:18:54.666 INFO:tasks.workunit.client.0.smithi096.stdout:5/656: dread - d4/d11/d3d/d4d/d99/fc2 zero size 2024-04-22T15:18:54.667 INFO:tasks.workunit.client.1.smithi196.stdout:9/292: write db/dd/de/f54 [2402966,10360] 0 2024-04-22T15:18:54.670 INFO:tasks.workunit.client.1.smithi196.stdout:9/293: dread db/dd/f5c [0,4194304] 0 2024-04-22T15:18:54.671 INFO:tasks.workunit.client.1.smithi196.stdout:9/294: readlink db/dd/de/l56 0 2024-04-22T15:18:54.682 INFO:tasks.workunit.client.1.smithi196.stdout:6/316: mkdir dd/d1f/d4e/d53/d55 0 2024-04-22T15:18:54.682 INFO:tasks.workunit.client.1.smithi196.stdout:6/317: write dd/f18 [2411774,48219] 0 2024-04-22T15:18:54.683 INFO:tasks.workunit.client.1.smithi196.stdout:6/318: dread - dd/d1f/d4e/f54 zero size 2024-04-22T15:18:54.684 INFO:tasks.workunit.client.0.smithi096.stdout:7/871: rename d31/d68/f9f to d31/d32/d33/d44/da8/dbf/f126 0 2024-04-22T15:18:54.688 INFO:tasks.workunit.client.0.smithi096.stdout:2/655: creat d3/db/df/dce/d46/d7e/d8d/fcf x:0 0 0 2024-04-22T15:18:54.689 INFO:tasks.workunit.client.0.smithi096.stdout:2/656: write d3/db/df/dce/d46/d7e/d8d/f92 [982087,91668] 0 2024-04-22T15:18:54.691 INFO:tasks.workunit.client.0.smithi096.stdout:6/556: link d3/d28/dd/f19 d3/d28/d54/d68/d6e/dbb/fbc 0 2024-04-22T15:18:54.693 INFO:tasks.workunit.client.0.smithi096.stdout:9/582: symlink d1/d25/db2/lc1 0 2024-04-22T15:18:54.693 INFO:tasks.workunit.client.0.smithi096.stdout:6/557: read d3/d28/dd/d11/d2c/f5e [671091,31399] 0 2024-04-22T15:18:54.694 INFO:tasks.workunit.client.0.smithi096.stdout:9/583: write d1/d25/f2e [243081,60708] 0 2024-04-22T15:18:54.694 INFO:tasks.workunit.client.0.smithi096.stdout:6/558: fdatasync d3/d28/d4d/d82/d9e/f92 0 2024-04-22T15:18:54.698 INFO:tasks.workunit.client.1.smithi196.stdout:4/302: creat d6/d14/f5b x:0 0 0 2024-04-22T15:18:54.699 INFO:tasks.workunit.client.1.smithi196.stdout:4/303: write d6/d13/d15/d2c/d35/f3c [13238797,95668] 0 2024-04-22T15:18:54.706 INFO:tasks.workunit.client.1.smithi196.stdout:0/302: rename d3/d9/d36/c4d to d3/de/c67 0 2024-04-22T15:18:54.711 INFO:tasks.workunit.client.1.smithi196.stdout:9/295: mknod db/d11/c6b 0 2024-04-22T15:18:54.713 INFO:tasks.workunit.client.1.smithi196.stdout:6/319: mknod dd/d1f/d37/c56 0 2024-04-22T15:18:54.715 INFO:tasks.workunit.client.1.smithi196.stdout:4/304: mknod d6/d13/d15/d2c/d35/d55/c5c 0 2024-04-22T15:18:54.717 INFO:tasks.workunit.client.1.smithi196.stdout:2/322: rename d1/d23/d27/d2e/c52 to d1/d13/d25/c74 0 2024-04-22T15:18:54.718 INFO:tasks.workunit.client.1.smithi196.stdout:0/303: creat d3/de/d14/d5a/f68 x:0 0 0 2024-04-22T15:18:54.719 INFO:tasks.workunit.client.1.smithi196.stdout:8/317: unlink d2/c24 0 2024-04-22T15:18:54.721 INFO:tasks.workunit.client.1.smithi196.stdout:4/305: dwrite d6/f42 [0,4194304] 0 2024-04-22T15:18:54.721 INFO:tasks.workunit.client.1.smithi196.stdout:9/296: fsync db/dd/de/d15/f1e 0 2024-04-22T15:18:54.721 INFO:tasks.workunit.client.1.smithi196.stdout:4/306: readlink d6/d14/l21 0 2024-04-22T15:18:54.723 INFO:tasks.workunit.client.1.smithi196.stdout:9/297: write f5 [1089223,43337] 0 2024-04-22T15:18:54.723 INFO:tasks.workunit.client.1.smithi196.stdout:4/307: chown d6/d13/d15/d2c/d35 463621020 1 2024-04-22T15:18:54.723 INFO:tasks.workunit.client.1.smithi196.stdout:6/320: chown dd/d1f/c25 8902589 1 2024-04-22T15:18:54.726 INFO:tasks.workunit.client.1.smithi196.stdout:6/321: read dd/f15 [4326914,67609] 0 2024-04-22T15:18:54.726 INFO:tasks.workunit.client.1.smithi196.stdout:3/289: getdents d0/d15/d4d/db/dc/d4a 0 2024-04-22T15:18:54.727 INFO:tasks.workunit.client.1.smithi196.stdout:0/304: rename d3/c2f to d3/de/d1c/d27/d2c/d51/c69 0 2024-04-22T15:18:54.730 INFO:tasks.workunit.client.1.smithi196.stdout:3/290: mknod d0/d15/d31/c64 0 2024-04-22T15:18:54.730 INFO:tasks.workunit.client.1.smithi196.stdout:8/318: dwrite d2/d5/f48 [0,4194304] 0 2024-04-22T15:18:54.732 INFO:tasks.workunit.client.0.smithi096.stdout:3/533: truncate da/db/dc/d59/f79 3118718 0 2024-04-22T15:18:54.733 INFO:tasks.workunit.client.1.smithi196.stdout:6/322: mkdir dd/d1f/d4e/d53/d55/d57 0 2024-04-22T15:18:54.735 INFO:tasks.workunit.client.0.smithi096.stdout:2/657: mknod d3/d1f/d5a/cd0 0 2024-04-22T15:18:54.736 INFO:tasks.workunit.client.1.smithi196.stdout:6/323: dread - dd/d1f/d38/d40/f4f zero size 2024-04-22T15:18:54.736 INFO:tasks.workunit.client.0.smithi096.stdout:2/658: stat d3/d1f/d2f 0 2024-04-22T15:18:54.737 INFO:tasks.workunit.client.1.smithi196.stdout:9/298: dwrite db/dd/d1b/d2c/f37 [0,4194304] 0 2024-04-22T15:18:54.737 INFO:tasks.workunit.client.0.smithi096.stdout:2/659: chown d3/db/df/dce/d46/d59/l64 23 1 2024-04-22T15:18:54.738 INFO:tasks.workunit.client.1.smithi196.stdout:6/324: dread - dd/d1f/f47 zero size 2024-04-22T15:18:54.740 INFO:tasks.workunit.client.1.smithi196.stdout:4/308: rename d6/d13/d15/d2c/d35/c56 to d6/c5d 0 2024-04-22T15:18:54.740 INFO:tasks.workunit.client.1.smithi196.stdout:0/305: creat d3/de/d1c/d27/d2c/d51/f6a x:0 0 0 2024-04-22T15:18:54.742 INFO:tasks.workunit.client.0.smithi096.stdout:3/534: dread - da/db/d12/d22/d26/d52/f86 zero size 2024-04-22T15:18:54.743 INFO:tasks.workunit.client.1.smithi196.stdout:2/323: dread d1/fd [4194304,4194304] 0 2024-04-22T15:18:54.744 INFO:tasks.workunit.client.1.smithi196.stdout:2/324: write d1/f18 [1214755,42548] 0 2024-04-22T15:18:54.750 INFO:tasks.workunit.client.1.smithi196.stdout:6/325: rmdir dd/d1f/d38 39 2024-04-22T15:18:54.753 INFO:tasks.workunit.client.0.smithi096.stdout:5/657: creat d4/d11/d3d/d4d/fd7 x:0 0 0 2024-04-22T15:18:54.754 INFO:tasks.workunit.client.1.smithi196.stdout:3/291: rename d0/c5 to d0/d15/d4d/db/dc/d18/c65 0 2024-04-22T15:18:54.754 INFO:tasks.workunit.client.0.smithi096.stdout:5/658: chown d4/d11/d3d/d4d/d59/d5a/cc1 12428 1 2024-04-22T15:18:54.756 INFO:tasks.workunit.client.1.smithi196.stdout:4/309: rmdir d6/d14/d18/d26/d32/d53 39 2024-04-22T15:18:54.757 INFO:tasks.workunit.client.0.smithi096.stdout:9/584: mkdir d1/df/d95/d9d/db5/dd/dba/dc2 0 2024-04-22T15:18:54.758 INFO:tasks.workunit.client.0.smithi096.stdout:2/660: symlink d3/db/ld1 0 2024-04-22T15:18:54.759 INFO:tasks.workunit.client.1.smithi196.stdout:4/310: dread d6/d10/f37 [0,4194304] 0 2024-04-22T15:18:54.762 INFO:tasks.workunit.client.0.smithi096.stdout:4/668: dread d5/d7/d2c/d3d/f49 [0,4194304] 0 2024-04-22T15:18:54.762 INFO:tasks.workunit.client.1.smithi196.stdout:2/325: mknod d1/d13/d25/c75 0 2024-04-22T15:18:54.763 INFO:tasks.workunit.client.0.smithi096.stdout:3/535: mkdir da/db/de/daa 0 2024-04-22T15:18:54.763 INFO:tasks.workunit.client.0.smithi096.stdout:4/669: write d5/d7/fc7 [557977,74707] 0 2024-04-22T15:18:54.763 INFO:tasks.workunit.client.1.smithi196.stdout:9/299: symlink db/l6c 0 2024-04-22T15:18:54.764 INFO:tasks.workunit.client.1.smithi196.stdout:9/300: dread - db/f61 zero size 2024-04-22T15:18:54.764 INFO:tasks.workunit.client.0.smithi096.stdout:4/670: chown d5/d7/d1a/f23 366127 1 2024-04-22T15:18:54.765 INFO:tasks.workunit.client.0.smithi096.stdout:3/536: truncate da/db/dc/d28/f45 475457 0 2024-04-22T15:18:54.765 INFO:tasks.workunit.client.0.smithi096.stdout:4/671: chown d5/l34 9315246 1 2024-04-22T15:18:54.765 INFO:tasks.workunit.client.1.smithi196.stdout:6/326: mkdir dd/d1f/d37/d58 0 2024-04-22T15:18:54.768 INFO:tasks.workunit.client.0.smithi096.stdout:2/661: readlink d3/db/df/d24/d58/d5f/d6e/l8c 0 2024-04-22T15:18:54.770 INFO:tasks.workunit.client.1.smithi196.stdout:0/306: rename d3/d6/f33 to d3/d9/d36/d56/d65/f6b 0 2024-04-22T15:18:54.772 INFO:tasks.workunit.client.1.smithi196.stdout:3/292: creat d0/d7/d8/f66 x:0 0 0 2024-04-22T15:18:54.773 INFO:tasks.workunit.client.0.smithi096.stdout:7/872: rmdir d31/d32/d33/d44 39 2024-04-22T15:18:54.774 INFO:tasks.workunit.client.1.smithi196.stdout:1/257: truncate d1/db/d16/f40 393877 0 2024-04-22T15:18:54.775 INFO:tasks.workunit.client.1.smithi196.stdout:0/307: dwrite d3/de/d1c/d27/f32 [0,4194304] 0 2024-04-22T15:18:54.776 INFO:tasks.workunit.client.1.smithi196.stdout:2/326: dread - d1/d13/f50 zero size 2024-04-22T15:18:54.777 INFO:tasks.workunit.client.1.smithi196.stdout:2/327: chown d1/d13/d25/f3d 364 1 2024-04-22T15:18:54.778 INFO:tasks.workunit.client.1.smithi196.stdout:5/293: write d1/d2/f28 [1454572,91848] 0 2024-04-22T15:18:54.780 INFO:tasks.workunit.client.1.smithi196.stdout:9/301: creat db/dd/d1b/d2c/f6d x:0 0 0 2024-04-22T15:18:54.780 INFO:tasks.workunit.client.0.smithi096.stdout:4/672: rename d5/d7/d2c/d3d/f85 to d5/d7/d2c/d67/fe6 0 2024-04-22T15:18:54.781 INFO:tasks.workunit.client.1.smithi196.stdout:1/258: dread d1/db/dc/d26/d2e/f3e [0,4194304] 0 2024-04-22T15:18:54.782 INFO:tasks.workunit.client.1.smithi196.stdout:9/302: write db/d11/d29/d57/f3d [4252029,36806] 0 2024-04-22T15:18:54.784 INFO:tasks.workunit.client.1.smithi196.stdout:6/327: symlink dd/d2b/l59 0 2024-04-22T15:18:54.784 INFO:tasks.workunit.client.1.smithi196.stdout:9/303: chown db/dd/de/d15/f1e 1 1 2024-04-22T15:18:54.784 INFO:tasks.workunit.client.1.smithi196.stdout:7/271: write d14/d26/f35 [5178350,40698] 0 2024-04-22T15:18:54.785 INFO:tasks.workunit.client.1.smithi196.stdout:7/272: truncate d14/f3a 809850 0 2024-04-22T15:18:54.786 INFO:tasks.workunit.client.1.smithi196.stdout:8/319: getdents d2/d5/d1a 0 2024-04-22T15:18:54.787 INFO:tasks.workunit.client.1.smithi196.stdout:8/320: chown d2/dc/fd 133171 1 2024-04-22T15:18:54.787 INFO:tasks.workunit.client.1.smithi196.stdout:7/273: read f2 [732007,23464] 0 2024-04-22T15:18:54.790 INFO:tasks.workunit.client.0.smithi096.stdout:7/873: unlink d31/d68/fcb 0 2024-04-22T15:18:54.793 INFO:tasks.workunit.client.0.smithi096.stdout:0/783: truncate d7/de/d5b/d95/f45 1359932 0 2024-04-22T15:18:54.794 INFO:tasks.workunit.client.1.smithi196.stdout:8/321: dread d2/dc/f10 [0,4194304] 0 2024-04-22T15:18:54.802 INFO:tasks.workunit.client.1.smithi196.stdout:0/308: mkdir d3/de/d14/d29/d6c 0 2024-04-22T15:18:54.803 INFO:tasks.workunit.client.1.smithi196.stdout:0/309: read d3/de/d1c/d27/f32 [180937,18944] 0 2024-04-22T15:18:54.806 INFO:tasks.workunit.client.1.smithi196.stdout:5/294: chown d1/d17/d1c/l4f 2001192 1 2024-04-22T15:18:54.808 INFO:tasks.workunit.client.0.smithi096.stdout:9/585: getdents d1/df/d95/d9d/db5/d43 0 2024-04-22T15:18:54.808 INFO:tasks.workunit.client.1.smithi196.stdout:0/310: dread d3/de/d1c/d27/d2c/d30/f35 [0,4194304] 0 2024-04-22T15:18:54.809 INFO:tasks.workunit.client.0.smithi096.stdout:9/586: chown d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/c88 0 1 2024-04-22T15:18:54.811 INFO:tasks.workunit.client.1.smithi196.stdout:0/311: dread d3/d9/d25/f5b [0,4194304] 0 2024-04-22T15:18:54.811 INFO:tasks.workunit.client.1.smithi196.stdout:0/312: fdatasync d3/fd 0 2024-04-22T15:18:54.811 INFO:tasks.workunit.client.1.smithi196.stdout:1/259: unlink d1/db/d16/f17 0 2024-04-22T15:18:54.812 INFO:tasks.workunit.client.0.smithi096.stdout:4/673: dread d5/d7/d2c/d3d/d4f/d63/f89 [0,4194304] 0 2024-04-22T15:18:54.814 INFO:tasks.workunit.client.0.smithi096.stdout:3/537: rename da/d71 to da/dab 0 2024-04-22T15:18:54.814 INFO:tasks.workunit.client.0.smithi096.stdout:4/674: dread - d5/d5b/fd0 zero size 2024-04-22T15:18:54.815 INFO:tasks.workunit.client.0.smithi096.stdout:3/538: dread da/f8a [0,4194304] 0 2024-04-22T15:18:54.815 INFO:tasks.workunit.client.0.smithi096.stdout:4/675: truncate d5/d7/fbd 874787 0 2024-04-22T15:18:54.815 INFO:tasks.workunit.client.1.smithi196.stdout:8/322: dread d2/dc/f2f [0,4194304] 0 2024-04-22T15:18:54.816 INFO:tasks.workunit.client.0.smithi096.stdout:3/539: stat da/db/dc/c1a 0 2024-04-22T15:18:54.816 INFO:tasks.workunit.client.1.smithi196.stdout:9/304: creat db/dd/de/d15/f6e x:0 0 0 2024-04-22T15:18:54.816 INFO:tasks.workunit.client.0.smithi096.stdout:4/676: write d5/d7/d1a/f6c [4245849,128385] 0 2024-04-22T15:18:54.817 INFO:tasks.workunit.client.1.smithi196.stdout:0/313: dwrite d3/d6/dc/f58 [0,4194304] 0 2024-04-22T15:18:54.819 INFO:tasks.workunit.client.1.smithi196.stdout:6/328: rename c6 to dd/d1f/d4e/d53/d55/d57/c5a 0 2024-04-22T15:18:54.820 INFO:tasks.workunit.client.0.smithi096.stdout:7/874: link c25 d31/d32/d33/d6a/de7/c127 0 2024-04-22T15:18:54.821 INFO:tasks.workunit.client.1.smithi196.stdout:6/329: chown dd/f23 224974595 1 2024-04-22T15:18:54.822 INFO:tasks.workunit.client.0.smithi096.stdout:9/587: symlink d1/d25/db2/d98/lc3 0 2024-04-22T15:18:54.823 INFO:tasks.workunit.client.1.smithi196.stdout:8/323: dwrite d2/f4a [0,4194304] 0 2024-04-22T15:18:54.827 INFO:tasks.workunit.client.1.smithi196.stdout:8/324: dread d2/dc/f2f [0,4194304] 0 2024-04-22T15:18:54.830 INFO:tasks.workunit.client.0.smithi096.stdout:9/588: dwrite d1/df/d95/d9d/db5/dd/f46 [0,4194304] 0 2024-04-22T15:18:54.831 INFO:tasks.workunit.client.1.smithi196.stdout:2/328: mkdir d1/d76 0 2024-04-22T15:18:54.833 INFO:tasks.workunit.client.0.smithi096.stdout:4/677: mknod d5/d7/d2c/d3d/d4f/d63/ce7 0 2024-04-22T15:18:54.834 INFO:tasks.workunit.client.1.smithi196.stdout:8/325: dwrite d2/dc/d45/f5b [0,4194304] 0 2024-04-22T15:18:54.837 INFO:tasks.workunit.client.0.smithi096.stdout:7/875: chown d31/le5 27281 1 2024-04-22T15:18:54.839 INFO:tasks.workunit.client.0.smithi096.stdout:7/876: truncate d31/d32/dc0/f114 16305 0 2024-04-22T15:18:54.841 INFO:tasks.workunit.client.1.smithi196.stdout:5/295: unlink d1/f35 0 2024-04-22T15:18:54.845 INFO:tasks.workunit.client.0.smithi096.stdout:3/540: symlink da/db/de/daa/lac 0 2024-04-22T15:18:54.847 INFO:tasks.workunit.client.1.smithi196.stdout:9/305: creat db/d11/d29/f6f x:0 0 0 2024-04-22T15:18:54.848 INFO:tasks.workunit.client.0.smithi096.stdout:6/559: truncate d3/d28/f61 2021601 0 2024-04-22T15:18:54.851 INFO:tasks.workunit.client.0.smithi096.stdout:4/678: rmdir d5/d5b 39 2024-04-22T15:18:54.852 INFO:tasks.workunit.client.0.smithi096.stdout:4/679: readlink d5/d7/d2c/d43/le5 0 2024-04-22T15:18:54.854 INFO:tasks.workunit.client.0.smithi096.stdout:7/877: mkdir d31/d32/dc0/ddc/de9/d128 0 2024-04-22T15:18:54.856 INFO:tasks.workunit.client.1.smithi196.stdout:0/314: dwrite d3/d9/d25/f63 [0,4194304] 0 2024-04-22T15:18:54.860 INFO:tasks.workunit.client.0.smithi096.stdout:9/589: mkdir d1/df/d95/d9d/db5/dd/dba/dc2/dc4 0 2024-04-22T15:18:54.861 INFO:tasks.workunit.client.0.smithi096.stdout:5/659: truncate d4/d11/d3d/d35/d57/fb4 77593 0 2024-04-22T15:18:54.862 INFO:tasks.workunit.client.1.smithi196.stdout:2/329: dread d1/f3 [4194304,4194304] 0 2024-04-22T15:18:54.862 INFO:tasks.workunit.client.0.smithi096.stdout:5/660: chown d4/d11/d3d/d35/d57/l64 64886023 1 2024-04-22T15:18:54.863 INFO:tasks.workunit.client.1.smithi196.stdout:6/330: mknod dd/d1f/d4e/c5b 0 2024-04-22T15:18:54.869 INFO:tasks.workunit.client.0.smithi096.stdout:2/662: write d3/db/df/dce/d46/d69/f66 [2195262,53508] 0 2024-04-22T15:18:54.870 INFO:tasks.workunit.client.0.smithi096.stdout:9/590: symlink d1/d4/d7b/lc5 0 2024-04-22T15:18:54.873 INFO:tasks.workunit.client.0.smithi096.stdout:2/663: read d3/db/df/d24/d58/d5e/f6b [126115,5054] 0 2024-04-22T15:18:54.876 INFO:tasks.workunit.client.0.smithi096.stdout:5/661: unlink d4/d11/d3d/d4d/d59/d8a/f97 0 2024-04-22T15:18:54.877 INFO:tasks.workunit.client.0.smithi096.stdout:5/662: dread - d4/d11/d3d/d4d/d59/d98/fd0 zero size 2024-04-22T15:18:54.878 INFO:tasks.workunit.client.0.smithi096.stdout:5/663: chown d4/d11/d3d/d4f/fc7 487045 1 2024-04-22T15:18:54.880 INFO:tasks.workunit.client.0.smithi096.stdout:0/784: write d7/f78 [51973,89783] 0 2024-04-22T15:18:54.881 INFO:tasks.workunit.client.0.smithi096.stdout:9/591: creat d1/df/d95/d9d/db5/dd/dba/dc2/fc6 x:0 0 0 2024-04-22T15:18:54.882 INFO:tasks.workunit.client.1.smithi196.stdout:3/293: dwrite d0/d7/f1a [0,4194304] 0 2024-04-22T15:18:54.882 INFO:tasks.workunit.client.0.smithi096.stdout:9/592: stat d1/df/d95/d9d/db5/d59/cc0 0 2024-04-22T15:18:54.885 INFO:tasks.workunit.client.0.smithi096.stdout:6/560: rename d3/d28/d4d/d82/d9e/da3/fab to d3/d28/fbd 0 2024-04-22T15:18:54.888 INFO:tasks.workunit.client.1.smithi196.stdout:9/306: dwrite db/dd/f69 [4194304,4194304] 0 2024-04-22T15:18:54.888 INFO:tasks.workunit.client.1.smithi196.stdout:0/315: creat d3/d9/d36/f6d x:0 0 0 2024-04-22T15:18:54.890 INFO:tasks.workunit.client.1.smithi196.stdout:7/274: link d14/d15/c2c d14/d26/d2a/d2b/c50 0 2024-04-22T15:18:54.892 INFO:tasks.workunit.client.1.smithi196.stdout:7/275: write d14/f22 [1507766,94805] 0 2024-04-22T15:18:54.894 INFO:tasks.workunit.client.1.smithi196.stdout:2/330: creat d1/d13/d25/d2a/f77 x:0 0 0 2024-04-22T15:18:54.897 INFO:tasks.workunit.client.1.smithi196.stdout:2/331: write d1/d13/f50 [403269,30192] 0 2024-04-22T15:18:54.898 INFO:tasks.workunit.client.1.smithi196.stdout:0/316: dwrite d3/de/d1c/d27/d2c/d51/f59 [0,4194304] 0 2024-04-22T15:18:54.899 INFO:tasks.workunit.client.1.smithi196.stdout:2/332: write d1/d13/f50 [400158,44879] 0 2024-04-22T15:18:54.900 INFO:tasks.workunit.client.1.smithi196.stdout:2/333: write d1/d13/f50 [847467,92601] 0 2024-04-22T15:18:54.905 INFO:tasks.workunit.client.0.smithi096.stdout:4/680: rename d5/fa1 to d5/d7/d1f/d84/d6d/d98/fe8 0 2024-04-22T15:18:54.907 INFO:tasks.workunit.client.1.smithi196.stdout:3/294: symlink d0/d15/d48/d51/l67 0 2024-04-22T15:18:54.910 INFO:tasks.workunit.client.0.smithi096.stdout:2/664: unlink d3/db/df/l3c 0 2024-04-22T15:18:54.912 INFO:tasks.workunit.client.0.smithi096.stdout:2/665: dread - d3/db/df/d24/d58/d5f/d6e/fb8 zero size 2024-04-22T15:18:54.913 INFO:tasks.workunit.client.1.smithi196.stdout:9/307: rename db/d11/d29/l2e to db/dd/d1b/d2c/l70 0 2024-04-22T15:18:54.914 INFO:tasks.workunit.client.0.smithi096.stdout:7/878: getdents d31/d32/d36/d75/dff 0 2024-04-22T15:18:54.917 INFO:tasks.workunit.client.0.smithi096.stdout:9/593: dread d1/df/d95/d9d/db5/dd/f41 [0,4194304] 0 2024-04-22T15:18:54.918 INFO:tasks.workunit.client.0.smithi096.stdout:5/664: mknod d4/d11/d3d/d4d/cd8 0 2024-04-22T15:18:54.919 INFO:tasks.workunit.client.0.smithi096.stdout:9/594: write d1/df/d95/d9d/db5/d43/fa3 [535012,22314] 0 2024-04-22T15:18:54.920 INFO:tasks.workunit.client.0.smithi096.stdout:3/541: write da/db/d12/d22/f4e [983645,55933] 0 2024-04-22T15:18:54.921 INFO:tasks.workunit.client.1.smithi196.stdout:7/276: mknod d14/d26/d28/c51 0 2024-04-22T15:18:54.922 INFO:tasks.workunit.client.1.smithi196.stdout:4/311: sync 2024-04-22T15:18:54.922 INFO:tasks.workunit.client.1.smithi196.stdout:7/277: truncate d14/f22 2615393 0 2024-04-22T15:18:54.930 INFO:tasks.workunit.client.0.smithi096.stdout:1/732: sync 2024-04-22T15:18:54.930 INFO:tasks.workunit.client.0.smithi096.stdout:8/658: sync 2024-04-22T15:18:54.931 INFO:tasks.workunit.client.0.smithi096.stdout:8/659: chown d4/d1c/d62/la3 12910 1 2024-04-22T15:18:54.944 INFO:tasks.workunit.client.1.smithi196.stdout:7/278: dread fc [0,4194304] 0 2024-04-22T15:18:54.977 INFO:tasks.workunit.client.1.smithi196.stdout:1/260: sync 2024-04-22T15:18:54.992 INFO:tasks.workunit.client.0.smithi096.stdout:4/681: truncate d5/d7/d1f/f7a 675300 0 2024-04-22T15:18:54.996 INFO:tasks.workunit.client.0.smithi096.stdout:2/666: creat d3/d1f/d2f/fd2 x:0 0 0 2024-04-22T15:18:54.997 INFO:tasks.workunit.client.1.smithi196.stdout:0/317: dread d3/d9/d25/f5b [0,4194304] 0 2024-04-22T15:18:54.998 INFO:tasks.workunit.client.1.smithi196.stdout:0/318: chown d3/de/d14 7019 1 2024-04-22T15:18:54.998 INFO:tasks.workunit.client.0.smithi096.stdout:2/667: truncate d3/db/df/fcd 496366 0 2024-04-22T15:18:54.999 INFO:tasks.workunit.client.0.smithi096.stdout:7/879: symlink d31/d32/d36/d75/dfe/l129 0 2024-04-22T15:18:55.000 INFO:tasks.workunit.client.1.smithi196.stdout:3/295: mknod d0/d15/d48/d51/d5e/c68 0 2024-04-22T15:18:55.001 INFO:tasks.workunit.client.1.smithi196.stdout:0/319: dread d3/d9/d25/f63 [0,4194304] 0 2024-04-22T15:18:55.002 INFO:tasks.workunit.client.1.smithi196.stdout:0/320: chown d3/d6/l7 2 1 2024-04-22T15:18:55.002 INFO:tasks.workunit.client.0.smithi096.stdout:5/665: creat d4/d11/d3d/d4d/d59/db0/fd9 x:0 0 0 2024-04-22T15:18:55.004 INFO:tasks.workunit.client.1.smithi196.stdout:2/334: rename d1/d1d/f5f to d1/d23/d27/d2e/d55/d5b/f78 0 2024-04-22T15:18:55.005 INFO:tasks.workunit.client.0.smithi096.stdout:7/880: dwrite d31/d32/f59 [0,4194304] 0 2024-04-22T15:18:55.005 INFO:tasks.workunit.client.1.smithi196.stdout:9/308: truncate db/dd/d1b/f45 4279649 0 2024-04-22T15:18:55.006 INFO:tasks.workunit.client.0.smithi096.stdout:7/881: stat d31/d32/d33/d6a/de7 0 2024-04-22T15:18:55.007 INFO:tasks.workunit.client.0.smithi096.stdout:3/542: creat da/db/dc/d28/fad x:0 0 0 2024-04-22T15:18:55.008 INFO:tasks.workunit.client.1.smithi196.stdout:3/296: dwrite d0/d15/d4d/db/dc/d4a/d44/f16 [0,4194304] 0 2024-04-22T15:18:55.009 INFO:tasks.workunit.client.1.smithi196.stdout:0/321: dwrite d3/d9/d36/f4c [0,4194304] 0 2024-04-22T15:18:55.013 INFO:tasks.workunit.client.1.smithi196.stdout:5/296: sync 2024-04-22T15:18:55.014 INFO:tasks.workunit.client.1.smithi196.stdout:5/297: chown d1/d4/c5c 289 1 2024-04-22T15:18:55.017 INFO:tasks.workunit.client.1.smithi196.stdout:5/298: read d1/d2/f28 [2174677,24168] 0 2024-04-22T15:18:55.019 INFO:tasks.workunit.client.1.smithi196.stdout:0/322: dwrite d3/de/d1c/d27/d2c/d51/f59 [0,4194304] 0 2024-04-22T15:18:55.029 INFO:tasks.workunit.client.0.smithi096.stdout:2/668: rmdir d3/db/df/dce/d46/d69/d7f 39 2024-04-22T15:18:55.029 INFO:tasks.workunit.client.1.smithi196.stdout:8/326: truncate d2/d8/d47/f62 655950 0 2024-04-22T15:18:55.031 INFO:tasks.workunit.client.0.smithi096.stdout:0/785: dwrite d7/de/d3a/d61/fdd [0,4194304] 0 2024-04-22T15:18:55.039 INFO:tasks.workunit.client.0.smithi096.stdout:2/669: dwrite d3/db/df/dce/d46/d7e/d8d/f92 [0,4194304] 0 2024-04-22T15:18:55.047 INFO:tasks.workunit.client.0.smithi096.stdout:9/595: creat d1/d25/db2/d98/d9b/fc7 x:0 0 0 2024-04-22T15:18:55.048 INFO:tasks.workunit.client.0.smithi096.stdout:9/596: fsync d1/f4f 0 2024-04-22T15:18:55.049 INFO:tasks.workunit.client.0.smithi096.stdout:9/597: fsync d1/df/d95/d9d/db5/dd/f46 0 2024-04-22T15:18:55.052 INFO:tasks.workunit.client.1.smithi196.stdout:2/335: stat d1/d13/d1f/f20 0 2024-04-22T15:18:55.055 INFO:tasks.workunit.client.0.smithi096.stdout:7/882: rename d31/d32/d36/d5d/l7d to d31/d32/dc0/ddc/de9/l12a 0 2024-04-22T15:18:55.056 INFO:tasks.workunit.client.0.smithi096.stdout:7/883: chown d31/d32/d33/d54/ldb 485549110 1 2024-04-22T15:18:55.060 INFO:tasks.workunit.client.0.smithi096.stdout:6/561: link d3/d3a/l65 d3/d28/dd/d11/d2c/lbe 0 2024-04-22T15:18:55.066 INFO:tasks.workunit.client.1.smithi196.stdout:3/297: dwrite d0/d15/d4d/db/dc/d4a/f50 [0,4194304] 0 2024-04-22T15:18:55.079 INFO:tasks.workunit.client.1.smithi196.stdout:1/261: mkdir d1/db/dc/de/d5a/d5b 0 2024-04-22T15:18:55.080 INFO:tasks.workunit.client.1.smithi196.stdout:1/262: readlink d1/db/dc/de/l12 0 2024-04-22T15:18:55.082 INFO:tasks.workunit.client.1.smithi196.stdout:5/299: fdatasync d1/d9/f20 0 2024-04-22T15:18:55.086 INFO:tasks.workunit.client.1.smithi196.stdout:1/263: dwrite d1/db/dc/d26/d2e/d32/f53 [0,4194304] 0 2024-04-22T15:18:55.087 INFO:tasks.workunit.client.1.smithi196.stdout:6/331: getdents dd/d1f 0 2024-04-22T15:18:55.088 INFO:tasks.workunit.client.1.smithi196.stdout:1/264: dread - d1/db/dc/d26/d2e/f51 zero size 2024-04-22T15:18:55.088 INFO:tasks.workunit.client.1.smithi196.stdout:6/332: dread - dd/d1f/f48 zero size 2024-04-22T15:18:55.092 INFO:tasks.workunit.client.1.smithi196.stdout:8/327: read d2/f3e [2336803,61872] 0 2024-04-22T15:18:55.097 INFO:tasks.workunit.client.0.smithi096.stdout:9/598: rename d1/df/d95/d9d/db5/d43/f6a to d1/d85/fc8 0 2024-04-22T15:18:55.099 INFO:tasks.workunit.client.1.smithi196.stdout:2/336: creat d1/d23/d27/d2e/d45/f79 x:0 0 0 2024-04-22T15:18:55.100 INFO:tasks.workunit.client.1.smithi196.stdout:2/337: stat d1/d13/d1f 0 2024-04-22T15:18:55.102 INFO:tasks.workunit.client.1.smithi196.stdout:4/312: creat d6/d13/d15/f5e x:0 0 0 2024-04-22T15:18:55.102 INFO:tasks.workunit.client.1.smithi196.stdout:4/313: fdatasync d6/d13/d15/d2c/d35/d44/f4a 0 2024-04-22T15:18:55.103 INFO:tasks.workunit.client.0.smithi096.stdout:0/786: mknod d7/d42/d4c/d55/d9c/df7/dfb/c10b 0 2024-04-22T15:18:55.103 INFO:tasks.workunit.client.1.smithi196.stdout:4/314: write d6/d14/d18/f36 [445896,35924] 0 2024-04-22T15:18:55.109 INFO:tasks.workunit.client.1.smithi196.stdout:4/315: dwrite d6/d13/d15/d2c/d35/d44/f57 [0,4194304] 0 2024-04-22T15:18:55.111 INFO:tasks.workunit.client.0.smithi096.stdout:1/733: rmdir d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc 39 2024-04-22T15:18:55.112 INFO:tasks.workunit.client.0.smithi096.stdout:1/734: chown d6/dc/d8a/d3f/d62/d7d/dab 3049 1 2024-04-22T15:18:55.114 INFO:tasks.workunit.client.0.smithi096.stdout:7/884: rename d31/d32/d36/d75/lb3 to d31/d32/d33/dfa/l12b 0 2024-04-22T15:18:55.123 INFO:tasks.workunit.client.1.smithi196.stdout:5/300: symlink d1/d17/d1a/l5f 0 2024-04-22T15:18:55.124 INFO:tasks.workunit.client.1.smithi196.stdout:5/301: write d1/d14/f1e [687729,30774] 0 2024-04-22T15:18:55.124 INFO:tasks.workunit.client.1.smithi196.stdout:5/302: chown d1/cb 43554 1 2024-04-22T15:18:55.125 INFO:tasks.workunit.client.1.smithi196.stdout:5/303: readlink d1/d17/d1c/l27 0 2024-04-22T15:18:55.127 INFO:tasks.workunit.client.0.smithi096.stdout:6/562: creat d3/d28/d54/d68/d6e/dbb/fbf x:0 0 0 2024-04-22T15:18:55.128 INFO:tasks.workunit.client.0.smithi096.stdout:9/599: dwrite d1/d85/fc8 [0,4194304] 0 2024-04-22T15:18:55.129 INFO:tasks.workunit.client.0.smithi096.stdout:9/600: chown d1/df/d95/d9d 9 1 2024-04-22T15:18:55.134 INFO:tasks.workunit.client.1.smithi196.stdout:6/333: mkdir dd/d20/d5c 0 2024-04-22T15:18:55.134 INFO:tasks.workunit.client.0.smithi096.stdout:1/735: dread d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fa1 [0,4194304] 0 2024-04-22T15:18:55.145 INFO:tasks.workunit.client.1.smithi196.stdout:2/338: chown d1/c17 5 1 2024-04-22T15:18:55.150 INFO:tasks.workunit.client.1.smithi196.stdout:2/339: dwrite d1/f1b [0,4194304] 0 2024-04-22T15:18:55.151 INFO:tasks.workunit.client.0.smithi096.stdout:2/670: getdents d3/db/df/dce/d46/d69 0 2024-04-22T15:18:55.154 INFO:tasks.workunit.client.1.smithi196.stdout:4/316: symlink d6/d13/d15/d2c/d35/d44/l5f 0 2024-04-22T15:18:55.159 INFO:tasks.workunit.client.1.smithi196.stdout:2/340: dwrite d1/d23/d27/d2e/f58 [0,4194304] 0 2024-04-22T15:18:55.160 INFO:tasks.workunit.client.1.smithi196.stdout:4/317: dread d6/d13/f1b [0,4194304] 0 2024-04-22T15:18:55.163 INFO:tasks.workunit.client.0.smithi096.stdout:1/736: rename d6/fd to d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fdf 0 2024-04-22T15:18:55.169 INFO:tasks.workunit.client.1.smithi196.stdout:5/304: write d1/d17/d1c/f52 [3522376,79356] 0 2024-04-22T15:18:55.179 INFO:tasks.workunit.client.1.smithi196.stdout:2/341: creat d1/d1d/f7a x:0 0 0 2024-04-22T15:18:55.192 INFO:tasks.workunit.client.1.smithi196.stdout:2/342: chown d1/d13/d25/c3b 168015 1 2024-04-22T15:18:55.192 INFO:tasks.workunit.client.1.smithi196.stdout:2/343: dread - d1/d13/d25/d2a/f3f zero size 2024-04-22T15:18:55.205 INFO:tasks.workunit.client.1.smithi196.stdout:2/344: symlink d1/d23/d27/d73/l7b 0 2024-04-22T15:18:55.208 INFO:tasks.workunit.client.1.smithi196.stdout:2/345: mkdir d1/d23/d27/d2e/d55/d5b/d7c 0 2024-04-22T15:18:55.209 INFO:tasks.workunit.client.1.smithi196.stdout:2/346: readlink d1/d23/d27/d2e/l4e 0 2024-04-22T15:18:55.209 INFO:tasks.workunit.client.1.smithi196.stdout:2/347: chown d1/d23/d27/d2e/d55 0 1 2024-04-22T15:18:55.209 INFO:tasks.workunit.client.1.smithi196.stdout:2/348: chown d1/d1d/f7a 159 1 2024-04-22T15:18:55.215 INFO:tasks.workunit.client.1.smithi196.stdout:2/349: creat d1/d76/f7d x:0 0 0 2024-04-22T15:18:55.218 INFO:tasks.workunit.client.0.smithi096.stdout:8/660: dwrite d4/d1c/d40/d46/f7d [0,4194304] 0 2024-04-22T15:18:55.259 INFO:tasks.workunit.client.1.smithi196.stdout:7/279: truncate d14/d26/d2a/d2b/f2d 4185775 0 2024-04-22T15:18:55.262 INFO:tasks.workunit.client.1.smithi196.stdout:0/323: write d3/de/d14/d29/f57 [1452150,94798] 0 2024-04-22T15:18:55.262 INFO:tasks.workunit.client.1.smithi196.stdout:0/324: fsync d3/de/f18 0 2024-04-22T15:18:55.263 INFO:tasks.workunit.client.0.smithi096.stdout:3/543: symlink da/db/de/d19/lae 0 2024-04-22T15:18:55.273 INFO:tasks.workunit.client.0.smithi096.stdout:9/601: creat d1/df/d95/d9d/db5/d43/d57/d5b/da1/fc9 x:0 0 0 2024-04-22T15:18:55.275 INFO:tasks.workunit.client.0.smithi096.stdout:9/602: mkdir d1/d25/db2/dca 0 2024-04-22T15:18:55.275 INFO:tasks.workunit.client.0.smithi096.stdout:2/671: dwrite d3/db/d30/f3e [0,4194304] 0 2024-04-22T15:18:55.279 INFO:tasks.workunit.client.1.smithi196.stdout:0/325: symlink d3/de/l6e 0 2024-04-22T15:18:55.282 INFO:tasks.workunit.client.0.smithi096.stdout:6/563: write d3/d28/dd/d11/d2c/f4e [244141,89619] 0 2024-04-22T15:18:55.282 INFO:tasks.workunit.client.1.smithi196.stdout:0/326: mknod d3/d9/d36/d56/c6f 0 2024-04-22T15:18:55.282 INFO:tasks.workunit.client.0.smithi096.stdout:0/787: truncate d7/de/d5b/d95/dc/d16/d65/f84 3825063 0 2024-04-22T15:18:55.284 INFO:tasks.workunit.client.1.smithi196.stdout:3/298: rename d0/d15/d4d/db/l2f to d0/d15/l69 0 2024-04-22T15:18:55.286 INFO:tasks.workunit.client.0.smithi096.stdout:7/885: dwrite d31/d32/d33/d6a/d83/dba/dc7/fcd [0,4194304] 0 2024-04-22T15:18:55.286 INFO:tasks.workunit.client.1.smithi196.stdout:0/327: read d3/de/d1c/d27/f32 [3835150,48449] 0 2024-04-22T15:18:55.290 INFO:tasks.workunit.client.1.smithi196.stdout:4/318: getdents d6/d13/d15/d2c/d35/d44 0 2024-04-22T15:18:55.291 INFO:tasks.workunit.client.1.smithi196.stdout:0/328: dread d3/d6/dc/f58 [0,4194304] 0 2024-04-22T15:18:55.295 INFO:tasks.workunit.client.1.smithi196.stdout:4/319: unlink d6/d14/d18/d40/f4e 0 2024-04-22T15:18:55.299 INFO:tasks.workunit.client.1.smithi196.stdout:4/320: creat d6/d13/d15/d2c/d35/d44/f60 x:0 0 0 2024-04-22T15:18:55.299 INFO:tasks.workunit.client.1.smithi196.stdout:0/329: dwrite d3/de/d14/d29/f57 [0,4194304] 0 2024-04-22T15:18:55.300 INFO:tasks.workunit.client.1.smithi196.stdout:4/321: write d6/f2b [2314359,24373] 0 2024-04-22T15:18:55.306 INFO:tasks.workunit.client.0.smithi096.stdout:5/666: symlink d4/d2c/d2f/lda 0 2024-04-22T15:18:55.306 INFO:tasks.workunit.client.1.smithi196.stdout:0/330: dwrite d3/de/f18 [0,4194304] 0 2024-04-22T15:18:55.306 INFO:tasks.workunit.client.1.smithi196.stdout:8/328: unlink d2/f38 0 2024-04-22T15:18:55.307 INFO:tasks.workunit.client.0.smithi096.stdout:3/544: rename da/db/l36 to da/db/d12/laf 0 2024-04-22T15:18:55.309 INFO:tasks.workunit.client.1.smithi196.stdout:6/334: dwrite dd/f15 [4194304,4194304] 0 2024-04-22T15:18:55.310 INFO:tasks.workunit.client.1.smithi196.stdout:6/335: read - dd/f50 zero size 2024-04-22T15:18:55.312 INFO:tasks.workunit.client.1.smithi196.stdout:6/336: chown dd/d1f/d38/d40/f42 3 1 2024-04-22T15:18:55.312 INFO:tasks.workunit.client.1.smithi196.stdout:8/329: symlink d2/d35/l65 0 2024-04-22T15:18:55.313 INFO:tasks.workunit.client.1.smithi196.stdout:8/330: stat d2/d8/l21 0 2024-04-22T15:18:55.315 INFO:tasks.workunit.client.1.smithi196.stdout:0/331: mknod d3/d6/dc/c70 0 2024-04-22T15:18:55.317 INFO:tasks.workunit.client.1.smithi196.stdout:4/322: link d6/d13/d15/d2c/d35/c58 d6/d13/d15/d2c/d35/c61 0 2024-04-22T15:18:55.319 INFO:tasks.workunit.client.1.smithi196.stdout:6/337: dwrite dd/d2b/f3e [0,4194304] 0 2024-04-22T15:18:55.325 INFO:tasks.workunit.client.0.smithi096.stdout:5/667: rename d4/d11 to d4/ddb 0 2024-04-22T15:18:55.326 INFO:tasks.workunit.client.0.smithi096.stdout:7/886: symlink d31/d32/d33/d44/l12c 0 2024-04-22T15:18:55.330 INFO:tasks.workunit.client.0.smithi096.stdout:5/668: rmdir d4/d2c/d2f/d30 39 2024-04-22T15:18:55.330 INFO:tasks.workunit.client.0.smithi096.stdout:5/669: chown f1 11684 1 2024-04-22T15:18:55.334 INFO:tasks.workunit.client.0.smithi096.stdout:1/737: mknod d6/dc/d8a/d3f/d62/dc2/dcb/ce0 0 2024-04-22T15:18:55.335 INFO:tasks.workunit.client.0.smithi096.stdout:2/672: rename d3/db/df/fc6 to d3/db/d30/fd3 0 2024-04-22T15:18:55.338 INFO:tasks.workunit.client.0.smithi096.stdout:7/887: dwrite d31/d32/d33/d44/da8/fe2 [0,4194304] 0 2024-04-22T15:18:55.339 INFO:tasks.workunit.client.0.smithi096.stdout:1/738: write d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e/fde [703296,129466] 0 2024-04-22T15:18:55.340 INFO:tasks.workunit.client.0.smithi096.stdout:7/888: chown d31/d32/f4c 2297 1 2024-04-22T15:18:55.342 INFO:tasks.workunit.client.0.smithi096.stdout:6/564: rename d3/d28/dd/d11/f32 to d3/d28/dd/d11/d2c/d44/d51/fc0 0 2024-04-22T15:18:55.343 INFO:tasks.workunit.client.0.smithi096.stdout:2/673: creat d3/db/df/d24/d58/fd4 x:0 0 0 2024-04-22T15:18:55.344 INFO:tasks.workunit.client.0.smithi096.stdout:2/674: stat d3/db/df/dce/fad 0 2024-04-22T15:18:55.345 INFO:tasks.workunit.client.0.smithi096.stdout:7/889: mknod d31/d68/d98/c12d 0 2024-04-22T15:18:55.346 INFO:tasks.workunit.client.0.smithi096.stdout:7/890: read - d31/d32/d33/d54/d43/fe8 zero size 2024-04-22T15:18:55.347 INFO:tasks.workunit.client.0.smithi096.stdout:7/891: dread - d31/f108 zero size 2024-04-22T15:18:55.348 INFO:tasks.workunit.client.0.smithi096.stdout:1/739: symlink d6/dc/d8a/d3f/d54/d32/dcd/le1 0 2024-04-22T15:18:55.349 INFO:tasks.workunit.client.0.smithi096.stdout:5/670: dread d4/d2c/d2f/f3a [0,4194304] 0 2024-04-22T15:18:55.351 INFO:tasks.workunit.client.0.smithi096.stdout:0/788: rename d7/de/d5b/d62/d67 to d7/d42/d4c/d55/d9c/df7/d10c 0 2024-04-22T15:18:55.353 INFO:tasks.workunit.client.0.smithi096.stdout:6/565: symlink d3/d3a/d75/d83/lc1 0 2024-04-22T15:18:55.354 INFO:tasks.workunit.client.0.smithi096.stdout:2/675: dread - d3/db/df/d24/d58/d5e/f8b zero size 2024-04-22T15:18:55.355 INFO:tasks.workunit.client.0.smithi096.stdout:7/892: fsync d31/d32/d36/d75/dff/f111 0 2024-04-22T15:18:55.359 INFO:tasks.workunit.client.0.smithi096.stdout:1/740: chown d6/dc/d8a/d3f/d54/d32/d3b/d94/f96 92657394 1 2024-04-22T15:18:55.360 INFO:tasks.workunit.client.0.smithi096.stdout:1/741: chown d6/dc/d8a/d3f/d62/dc2/dd0 236996460 1 2024-04-22T15:18:55.361 INFO:tasks.workunit.client.0.smithi096.stdout:0/789: creat d7/d42/d4c/d55/d9c/df7/f10d x:0 0 0 2024-04-22T15:18:55.362 INFO:tasks.workunit.client.0.smithi096.stdout:6/566: symlink d3/d3a/d75/lc2 0 2024-04-22T15:18:55.368 INFO:tasks.workunit.client.1.smithi196.stdout:5/305: creat d1/d17/f60 x:0 0 0 2024-04-22T15:18:55.368 INFO:tasks.workunit.client.1.smithi196.stdout:5/306: chown d1/d17/f5d 1 1 2024-04-22T15:18:55.368 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:55 smithi096 ceph-mon[26688]: pgmap v34: 65 pgs: 65 active+clean; 2.1 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 72 MiB/s rd, 247 MiB/s wr, 496 op/s 2024-04-22T15:18:55.369 INFO:tasks.workunit.client.0.smithi096.stdout:5/671: dread d4/ddb/fbd [0,4194304] 0 2024-04-22T15:18:55.369 INFO:tasks.workunit.client.0.smithi096.stdout:0/790: mkdir d7/d8f/d10e 0 2024-04-22T15:18:55.370 INFO:tasks.workunit.client.0.smithi096.stdout:5/672: chown d4/ddb/d3d/d35/d57/dbe/lc3 3527 1 2024-04-22T15:18:55.372 INFO:tasks.workunit.client.1.smithi196.stdout:5/307: creat d1/d17/d1a/d2f/d30/f61 x:0 0 0 2024-04-22T15:18:55.373 INFO:tasks.workunit.client.1.smithi196.stdout:5/308: dread - d1/d14/d3e/f4e zero size 2024-04-22T15:18:55.376 INFO:tasks.workunit.client.1.smithi196.stdout:1/265: creat d1/db/d16/f5c x:0 0 0 2024-04-22T15:18:55.377 INFO:tasks.workunit.client.0.smithi096.stdout:5/673: dwrite d4/ddb/f22 [0,4194304] 0 2024-04-22T15:18:55.377 INFO:tasks.workunit.client.0.smithi096.stdout:7/893: truncate fa 1223324 0 2024-04-22T15:18:55.379 INFO:tasks.workunit.client.1.smithi196.stdout:5/309: unlink d1/d14/d3e/f48 0 2024-04-22T15:18:55.379 INFO:tasks.workunit.client.1.smithi196.stdout:5/310: stat d1/d14/d3e 0 2024-04-22T15:18:55.385 INFO:tasks.workunit.client.0.smithi096.stdout:6/567: symlink d3/d28/d54/d68/lc3 0 2024-04-22T15:18:55.385 INFO:tasks.workunit.client.1.smithi196.stdout:5/311: read d1/d2/f28 [3029118,54940] 0 2024-04-22T15:18:55.389 INFO:tasks.workunit.client.1.smithi196.stdout:2/350: rename d1/d23/d27/d2e/d55/d5b/f6e to d1/f7e 0 2024-04-22T15:18:55.391 INFO:tasks.workunit.client.1.smithi196.stdout:5/312: dwrite d1/d17/d1c/f52 [0,4194304] 0 2024-04-22T15:18:55.393 INFO:tasks.workunit.client.1.smithi196.stdout:0/332: rename d3/d9/d36/d56/l4e to d3/de/d1c/d27/d2c/d51/l71 0 2024-04-22T15:18:55.394 INFO:tasks.workunit.client.0.smithi096.stdout:7/894: creat d31/d32/d33/d44/f12e x:0 0 0 2024-04-22T15:18:55.396 INFO:tasks.workunit.client.1.smithi196.stdout:5/313: symlink d1/d17/d1a/d2f/d30/l62 0 2024-04-22T15:18:55.398 INFO:tasks.workunit.client.0.smithi096.stdout:6/568: unlink d3/d28/dd/d11/d2c/d44/d51/la2 0 2024-04-22T15:18:55.403 INFO:tasks.workunit.client.0.smithi096.stdout:7/895: mknod d31/d32/d33/d6a/de7/c12f 0 2024-04-22T15:18:55.406 INFO:tasks.workunit.client.0.smithi096.stdout:7/896: creat d31/d32/dc0/f130 x:0 0 0 2024-04-22T15:18:55.409 INFO:tasks.workunit.client.1.smithi196.stdout:4/323: rename d6/d13/d15/d2c/d35/d55 to d6/d13/d15/d2c/d62 0 2024-04-22T15:18:55.411 INFO:tasks.workunit.client.0.smithi096.stdout:7/897: symlink d31/d32/d33/d44/da8/dbf/ddf/l131 0 2024-04-22T15:18:55.415 INFO:tasks.workunit.client.0.smithi096.stdout:8/661: truncate d4/d10/d25/d38/f54 307423 0 2024-04-22T15:18:55.419 INFO:tasks.workunit.client.0.smithi096.stdout:7/898: link d31/d32/dc0/fc4 d31/d32/d33/d44/da8/f132 0 2024-04-22T15:18:55.421 INFO:tasks.workunit.client.0.smithi096.stdout:7/899: creat d31/d32/d33/d5e/f133 x:0 0 0 2024-04-22T15:18:55.422 INFO:tasks.workunit.client.1.smithi196.stdout:5/314: creat d1/d17/d1a/d2f/d30/d5e/f63 x:0 0 0 2024-04-22T15:18:55.422 INFO:tasks.workunit.client.1.smithi196.stdout:6/338: rename f4 to dd/d1f/d37/f5d 0 2024-04-22T15:18:55.422 INFO:tasks.workunit.client.1.smithi196.stdout:9/309: unlink db/l55 0 2024-04-22T15:18:55.423 INFO:tasks.workunit.client.0.smithi096.stdout:7/900: symlink d31/d32/d33/d6a/d83/l134 0 2024-04-22T15:18:55.424 INFO:tasks.workunit.client.0.smithi096.stdout:4/682: rmdir d5/d7/d2c/d67 39 2024-04-22T15:18:55.424 INFO:tasks.workunit.client.1.smithi196.stdout:4/324: dread d6/d14/d18/d26/f43 [0,4194304] 0 2024-04-22T15:18:55.426 INFO:tasks.workunit.client.1.smithi196.stdout:0/333: getdents d3/d9/d36/d56/d65 0 2024-04-22T15:18:55.427 INFO:tasks.workunit.client.1.smithi196.stdout:7/280: write fc [2244390,122267] 0 2024-04-22T15:18:55.428 INFO:tasks.workunit.client.1.smithi196.stdout:6/339: dwrite dd/f1c [0,4194304] 0 2024-04-22T15:18:55.430 INFO:tasks.workunit.client.1.smithi196.stdout:0/334: write d3/de/d14/d5a/f5f [125284,128494] 0 2024-04-22T15:18:55.431 INFO:tasks.workunit.client.0.smithi096.stdout:4/683: dread d5/d7/d2c/d3d/d4f/d9c/fc6 [0,4194304] 0 2024-04-22T15:18:55.434 INFO:tasks.workunit.client.1.smithi196.stdout:5/315: fsync d1/d2/f28 0 2024-04-22T15:18:55.435 INFO:tasks.workunit.client.1.smithi196.stdout:6/340: dwrite dd/d1f/d38/d40/f42 [4194304,4194304] 0 2024-04-22T15:18:55.436 INFO:tasks.workunit.client.1.smithi196.stdout:6/341: chown dd/d1f/f47 1121909 1 2024-04-22T15:18:55.439 INFO:tasks.workunit.client.1.smithi196.stdout:4/325: creat d6/d10/f63 x:0 0 0 2024-04-22T15:18:55.440 INFO:tasks.workunit.client.0.smithi096.stdout:4/684: fsync d5/d7/d2c/d3d/d4f/f5c 0 2024-04-22T15:18:55.443 INFO:tasks.workunit.client.1.smithi196.stdout:3/299: write d0/d15/d4d/db/dc/d4a/d44/f20 [2640496,10159] 0 2024-04-22T15:18:55.445 INFO:tasks.workunit.client.1.smithi196.stdout:8/331: sync 2024-04-22T15:18:55.446 INFO:tasks.workunit.client.1.smithi196.stdout:8/332: fdatasync d2/dc/f2f 0 2024-04-22T15:18:55.448 INFO:tasks.workunit.client.1.smithi196.stdout:4/326: dwrite d6/d13/d15/d2c/d35/d44/f60 [0,4194304] 0 2024-04-22T15:18:55.449 INFO:tasks.workunit.client.1.smithi196.stdout:0/335: mkdir d3/de/d14/d72 0 2024-04-22T15:18:55.449 INFO:tasks.workunit.client.1.smithi196.stdout:4/327: chown d6/d14/l21 23564 1 2024-04-22T15:18:55.449 INFO:tasks.workunit.client.1.smithi196.stdout:0/336: chown d3/d6/dc/l41 50 1 2024-04-22T15:18:55.455 INFO:tasks.workunit.client.1.smithi196.stdout:0/337: dwrite d3/de/d14/f39 [0,4194304] 0 2024-04-22T15:18:55.459 INFO:tasks.workunit.client.1.smithi196.stdout:3/300: mknod d0/d15/d31/c6a 0 2024-04-22T15:18:55.460 INFO:tasks.workunit.client.1.smithi196.stdout:3/301: dread d0/f60 [0,4194304] 0 2024-04-22T15:18:55.461 INFO:tasks.workunit.client.1.smithi196.stdout:3/302: readlink d0/d15/d4d/l49 0 2024-04-22T15:18:55.464 INFO:tasks.workunit.client.0.smithi096.stdout:3/545: dwrite da/db/de/d19/f73 [0,4194304] 0 2024-04-22T15:18:55.467 INFO:tasks.workunit.client.1.smithi196.stdout:4/328: unlink d6/d13/d15/d2c/d35/f3b 0 2024-04-22T15:18:55.472 INFO:tasks.workunit.client.0.smithi096.stdout:4/685: dread d5/d7/d2c/d3d/d4f/f60 [0,4194304] 0 2024-04-22T15:18:55.473 INFO:tasks.workunit.client.1.smithi196.stdout:5/316: unlink d1/d2/l59 0 2024-04-22T15:18:55.473 INFO:tasks.workunit.client.1.smithi196.stdout:4/329: dwrite d6/d14/f33 [0,4194304] 0 2024-04-22T15:18:55.475 INFO:tasks.workunit.client.1.smithi196.stdout:6/342: mkdir dd/d20/d5c/d5e 0 2024-04-22T15:18:55.480 INFO:tasks.workunit.client.1.smithi196.stdout:0/338: unlink d3/d9/d25/f5b 0 2024-04-22T15:18:55.480 INFO:tasks.workunit.client.1.smithi196.stdout:3/303: dread d0/d15/d4d/db/f2e [0,4194304] 0 2024-04-22T15:18:55.481 INFO:tasks.workunit.client.0.smithi096.stdout:3/546: truncate da/db/d12/d22/f31 1396961 0 2024-04-22T15:18:55.482 INFO:tasks.workunit.client.1.smithi196.stdout:4/330: dwrite d6/d14/f5b [0,4194304] 0 2024-04-22T15:18:55.483 INFO:tasks.workunit.client.1.smithi196.stdout:3/304: write d0/d7/f1a [3650250,55726] 0 2024-04-22T15:18:55.483 INFO:tasks.workunit.client.1.smithi196.stdout:4/331: chown d6/d14 21332 1 2024-04-22T15:18:55.484 INFO:tasks.workunit.client.1.smithi196.stdout:3/305: chown d0/d15/d4d/db/dc/d18 744192 1 2024-04-22T15:18:55.487 INFO:tasks.workunit.client.0.smithi096.stdout:3/547: mkdir da/db/d12/d22/d68/db0 0 2024-04-22T15:18:55.489 INFO:tasks.workunit.client.0.smithi096.stdout:3/548: truncate da/db/d10/f90 1410154 0 2024-04-22T15:18:55.489 INFO:tasks.workunit.client.0.smithi096.stdout:4/686: creat d5/fe9 x:0 0 0 2024-04-22T15:18:55.490 INFO:tasks.workunit.client.0.smithi096.stdout:3/549: chown da/db/dc/d28/d69/c8c 0 1 2024-04-22T15:18:55.491 INFO:tasks.workunit.client.0.smithi096.stdout:2/676: write d3/db/df/d24/d58/d5e/f8b [244407,111101] 0 2024-04-22T15:18:55.492 INFO:tasks.workunit.client.1.smithi196.stdout:4/332: dwrite d6/d14/d18/f36 [0,4194304] 0 2024-04-22T15:18:55.496 INFO:tasks.workunit.client.1.smithi196.stdout:2/351: sync 2024-04-22T15:18:55.496 INFO:tasks.workunit.client.1.smithi196.stdout:5/317: dread d1/d17/f24 [0,4194304] 0 2024-04-22T15:18:55.497 INFO:tasks.workunit.client.0.smithi096.stdout:1/742: write d6/dc/d8a/d3f/d54/d32/d8c/d45/d7b/f7e [662949,99922] 0 2024-04-22T15:18:55.503 INFO:tasks.workunit.client.0.smithi096.stdout:2/677: dwrite d3/f8e [0,4194304] 0 2024-04-22T15:18:55.505 INFO:tasks.workunit.client.0.smithi096.stdout:2/678: chown d3/db/df/d24/d58/d5f/d6e/c9d 173 1 2024-04-22T15:18:55.509 INFO:tasks.workunit.client.1.smithi196.stdout:0/339: read d3/de/f24 [2521580,55930] 0 2024-04-22T15:18:55.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:55 smithi196 ceph-mon[34717]: pgmap v34: 65 pgs: 65 active+clean; 2.1 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 72 MiB/s rd, 247 MiB/s wr, 496 op/s 2024-04-22T15:18:55.509 INFO:tasks.workunit.client.0.smithi096.stdout:0/791: write d7/de/d5b/d95/d66/f81 [1566189,119732] 0 2024-04-22T15:18:55.510 INFO:tasks.workunit.client.1.smithi196.stdout:0/340: read d3/d6/f48 [220198,22427] 0 2024-04-22T15:18:55.510 INFO:tasks.workunit.client.0.smithi096.stdout:1/743: truncate d6/dc/d8a/d3f/d54/f18 1714355 0 2024-04-22T15:18:55.511 INFO:tasks.workunit.client.0.smithi096.stdout:1/744: readlink d6/dc/l2a 0 2024-04-22T15:18:55.514 INFO:tasks.workunit.client.1.smithi196.stdout:1/266: truncate d1/db/dc/de/d43/f48 292818 0 2024-04-22T15:18:55.515 INFO:tasks.workunit.client.1.smithi196.stdout:1/267: chown d1/db/dc/de/d43 4237 1 2024-04-22T15:18:55.516 INFO:tasks.workunit.client.0.smithi096.stdout:5/674: write f1 [4432874,15691] 0 2024-04-22T15:18:55.517 INFO:tasks.workunit.client.1.smithi196.stdout:6/343: dread dd/d20/f22 [0,4194304] 0 2024-04-22T15:18:55.518 INFO:tasks.workunit.client.1.smithi196.stdout:3/306: mkdir d0/d7/d8/d6b 0 2024-04-22T15:18:55.518 INFO:tasks.workunit.client.0.smithi096.stdout:2/679: readlink d3/db/d30/l97 0 2024-04-22T15:18:55.519 INFO:tasks.workunit.client.1.smithi196.stdout:0/341: dread d3/f11 [0,4194304] 0 2024-04-22T15:18:55.521 INFO:tasks.workunit.client.0.smithi096.stdout:3/550: creat da/db/de/d19/d2d/d37/fb1 x:0 0 0 2024-04-22T15:18:55.522 INFO:tasks.workunit.client.1.smithi196.stdout:4/333: truncate d6/d13/f1b 577738 0 2024-04-22T15:18:55.523 INFO:tasks.workunit.client.1.smithi196.stdout:4/334: write d6/d13/d15/f5e [101958,72518] 0 2024-04-22T15:18:55.525 INFO:tasks.workunit.client.1.smithi196.stdout:2/352: rmdir d1/d13 39 2024-04-22T15:18:55.526 INFO:tasks.workunit.client.1.smithi196.stdout:4/335: fdatasync d6/d13/d15/f5e 0 2024-04-22T15:18:55.527 INFO:tasks.workunit.client.1.smithi196.stdout:5/318: symlink d1/d4/l64 0 2024-04-22T15:18:55.529 INFO:tasks.workunit.client.0.smithi096.stdout:4/687: link d5/l17 d5/d7/d1f/d84/lea 0 2024-04-22T15:18:55.532 INFO:tasks.workunit.client.0.smithi096.stdout:6/569: dwrite d3/d28/dd/d11/d2c/d3b/f63 [8388608,4194304] 0 2024-04-22T15:18:55.532 INFO:tasks.workunit.client.1.smithi196.stdout:5/319: dwrite d1/d17/f2e [0,4194304] 0 2024-04-22T15:18:55.534 INFO:tasks.workunit.client.0.smithi096.stdout:5/675: mknod d4/ddb/d3d/d4f/cdc 0 2024-04-22T15:18:55.544 INFO:tasks.workunit.client.1.smithi196.stdout:1/268: stat d1/db/d16/d1e/d46/l4e 0 2024-04-22T15:18:55.546 INFO:tasks.workunit.client.0.smithi096.stdout:3/551: dread - da/db/d12/d22/f7d zero size 2024-04-22T15:18:55.546 INFO:tasks.workunit.client.0.smithi096.stdout:8/662: write d4/d10/d25/f44 [340706,3122] 0 2024-04-22T15:18:55.547 INFO:tasks.workunit.client.0.smithi096.stdout:3/552: readlink da/db/de/l25 0 2024-04-22T15:18:55.548 INFO:tasks.workunit.client.1.smithi196.stdout:6/344: unlink dd/f21 0 2024-04-22T15:18:55.549 INFO:tasks.workunit.client.1.smithi196.stdout:6/345: write dd/d1f/f48 [433514,2314] 0 2024-04-22T15:18:55.549 INFO:tasks.workunit.client.0.smithi096.stdout:8/663: truncate d4/d10/d25/d4c/d76/f89 1846967 0 2024-04-22T15:18:55.550 INFO:tasks.workunit.client.1.smithi196.stdout:3/307: truncate d0/d15/d4d/db/dc/d18/f4f 114465 0 2024-04-22T15:18:55.553 INFO:tasks.workunit.client.1.smithi196.stdout:0/342: rename d3/de/d14/d29/f5e to d3/d9/f73 0 2024-04-22T15:18:55.554 INFO:tasks.workunit.client.0.smithi096.stdout:4/688: creat d5/d7/d1a/d25/feb x:0 0 0 2024-04-22T15:18:55.555 INFO:tasks.workunit.client.0.smithi096.stdout:7/901: write d31/d32/d36/d75/f86 [2560739,6925] 0 2024-04-22T15:18:55.556 INFO:tasks.workunit.client.0.smithi096.stdout:6/570: creat d3/d28/dd/d11/d2c/d44/fc4 x:0 0 0 2024-04-22T15:18:55.559 INFO:tasks.workunit.client.0.smithi096.stdout:2/680: mkdir d3/dd5 0 2024-04-22T15:18:55.563 INFO:tasks.workunit.client.0.smithi096.stdout:8/664: fdatasync d4/d10/d4d/f74 0 2024-04-22T15:18:55.564 INFO:tasks.workunit.client.1.smithi196.stdout:9/310: dwrite db/ff [0,4194304] 0 2024-04-22T15:18:55.566 INFO:tasks.workunit.client.0.smithi096.stdout:4/689: creat d5/d7/d1f/fec x:0 0 0 2024-04-22T15:18:55.569 INFO:tasks.workunit.client.0.smithi096.stdout:7/902: creat d31/d32/dad/f135 x:0 0 0 2024-04-22T15:18:55.569 INFO:tasks.workunit.client.1.smithi196.stdout:7/281: write fe [4459122,110463] 0 2024-04-22T15:18:55.578 INFO:tasks.workunit.client.1.smithi196.stdout:6/346: creat dd/d1f/d38/d40/f5f x:0 0 0 2024-04-22T15:18:55.579 INFO:tasks.workunit.client.1.smithi196.stdout:3/308: creat d0/d15/d4d/db/dc/d4a/d44/f6c x:0 0 0 2024-04-22T15:18:55.587 INFO:tasks.workunit.client.0.smithi096.stdout:8/665: rmdir d4/d10/d25/d4c/d76 39 2024-04-22T15:18:55.588 INFO:tasks.workunit.client.0.smithi096.stdout:8/666: write d4/d2a/d7e/fa6 [1562869,40723] 0 2024-04-22T15:18:55.589 INFO:tasks.workunit.client.1.smithi196.stdout:0/343: creat d3/d9/d36/d56/d65/f74 x:0 0 0 2024-04-22T15:18:55.591 INFO:tasks.workunit.client.1.smithi196.stdout:2/353: creat d1/d23/d27/d2e/d55/d5b/d7c/f7f x:0 0 0 2024-04-22T15:18:55.594 INFO:tasks.workunit.client.0.smithi096.stdout:6/571: mknod d3/d28/d4d/cc5 0 2024-04-22T15:18:55.599 INFO:tasks.workunit.client.1.smithi196.stdout:9/311: mknod db/dd/de/d15/d31/c71 0 2024-04-22T15:18:55.600 INFO:tasks.workunit.client.1.smithi196.stdout:9/312: chown db/d11/d29/f48 913 1 2024-04-22T15:18:55.603 INFO:tasks.workunit.client.0.smithi096.stdout:6/572: dwrite d3/d28/dd/d11/d2c/d44/d51/f88 [0,4194304] 0 2024-04-22T15:18:55.603 INFO:tasks.workunit.client.1.smithi196.stdout:9/313: dread db/dd/d1b/d2c/f37 [0,4194304] 0 2024-04-22T15:18:55.606 INFO:tasks.workunit.client.0.smithi096.stdout:4/690: truncate d5/f21 728508 0 2024-04-22T15:18:55.607 INFO:tasks.workunit.client.0.smithi096.stdout:4/691: fdatasync d5/d7/d1a/d25/feb 0 2024-04-22T15:18:55.609 INFO:tasks.workunit.client.0.smithi096.stdout:5/676: getdents d4/d2c/d2f/d31/d82/da0 0 2024-04-22T15:18:55.611 INFO:tasks.workunit.client.1.smithi196.stdout:4/336: fsync d6/d13/d15/f30 0 2024-04-22T15:18:55.611 INFO:tasks.workunit.client.0.smithi096.stdout:7/903: dread d31/d32/d33/da2/fb9 [0,4194304] 0 2024-04-22T15:18:55.612 INFO:tasks.workunit.client.1.smithi196.stdout:4/337: write d6/d13/d15/d2c/d35/d44/f60 [5082062,85935] 0 2024-04-22T15:18:55.613 INFO:tasks.workunit.client.0.smithi096.stdout:0/792: write d7/d42/f7a [838150,29490] 0 2024-04-22T15:18:55.613 INFO:tasks.workunit.client.0.smithi096.stdout:7/904: chown d31/d32/dad/l104 112241001 1 2024-04-22T15:18:55.613 INFO:tasks.workunit.client.1.smithi196.stdout:4/338: write d6/d13/f2e [1006335,31727] 0 2024-04-22T15:18:55.615 INFO:tasks.workunit.client.1.smithi196.stdout:6/347: creat dd/d1f/f60 x:0 0 0 2024-04-22T15:18:55.619 INFO:tasks.workunit.client.1.smithi196.stdout:4/339: dwrite d6/d13/d15/d2c/d35/f3c [0,4194304] 0 2024-04-22T15:18:55.621 INFO:tasks.workunit.client.0.smithi096.stdout:1/745: dwrite d6/dc/d8a/d3f/d54/d32/d82/fa4 [0,4194304] 0 2024-04-22T15:18:55.623 INFO:tasks.workunit.client.0.smithi096.stdout:5/677: dread d4/ddb/d3d/d4d/f6e [0,4194304] 0 2024-04-22T15:18:55.631 INFO:tasks.workunit.client.1.smithi196.stdout:9/314: symlink db/dd/l72 0 2024-04-22T15:18:55.638 INFO:tasks.workunit.client.0.smithi096.stdout:1/746: chown d6/dc/d8a/d3f/d54/d32/d8c/dca/fb1 0 1 2024-04-22T15:18:55.640 INFO:tasks.workunit.client.0.smithi096.stdout:5/678: fsync d4/d2c/d2f/f81 0 2024-04-22T15:18:55.642 INFO:tasks.workunit.client.0.smithi096.stdout:5/679: write d4/ddb/d25/fd2 [811077,121666] 0 2024-04-22T15:18:55.645 INFO:tasks.workunit.client.0.smithi096.stdout:7/905: mknod d31/d32/d33/c136 0 2024-04-22T15:18:55.646 INFO:tasks.workunit.client.1.smithi196.stdout:1/269: write d1/db/d16/f1d [2140682,67366] 0 2024-04-22T15:18:55.648 INFO:tasks.workunit.client.1.smithi196.stdout:4/340: dread d6/d13/f25 [0,4194304] 0 2024-04-22T15:18:55.649 INFO:tasks.workunit.client.0.smithi096.stdout:6/573: rename d3/d28/d54/ca9 to d3/d28/d4d/cc6 0 2024-04-22T15:18:55.651 INFO:tasks.workunit.client.0.smithi096.stdout:3/553: truncate da/db/d10/f90 649199 0 2024-04-22T15:18:55.651 INFO:tasks.workunit.client.0.smithi096.stdout:6/574: chown d3/d28/dd/d11/d2c/d3b/dac 780720 1 2024-04-22T15:18:55.651 INFO:tasks.workunit.client.1.smithi196.stdout:1/270: truncate d1/db/dc/d26/d2e/d32/f53 5192176 0 2024-04-22T15:18:55.651 INFO:tasks.workunit.client.1.smithi196.stdout:8/333: dwrite d2/dc/fe [8388608,4194304] 0 2024-04-22T15:18:55.653 INFO:tasks.workunit.client.0.smithi096.stdout:4/692: creat d5/d7/fed x:0 0 0 2024-04-22T15:18:55.657 INFO:tasks.workunit.client.0.smithi096.stdout:1/747: creat d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e/fe2 x:0 0 0 2024-04-22T15:18:55.659 INFO:tasks.workunit.client.1.smithi196.stdout:1/271: dwrite d1/db/d16/f4b [0,4194304] 0 2024-04-22T15:18:55.662 INFO:tasks.workunit.client.0.smithi096.stdout:5/680: write d4/ddb/fbd [3849952,55322] 0 2024-04-22T15:18:55.664 INFO:tasks.workunit.client.1.smithi196.stdout:5/320: write d1/d17/d1a/f1d [5227547,9696] 0 2024-04-22T15:18:55.665 INFO:tasks.workunit.client.0.smithi096.stdout:7/906: write d31/d32/d33/d54/ff1 [3881289,19742] 0 2024-04-22T15:18:55.665 INFO:tasks.workunit.client.1.smithi196.stdout:9/315: write db/d11/d29/f48 [380491,68016] 0 2024-04-22T15:18:55.665 INFO:tasks.workunit.client.0.smithi096.stdout:7/907: chown d31/d32/d33/d44 103248192 1 2024-04-22T15:18:55.668 INFO:tasks.workunit.client.0.smithi096.stdout:6/575: unlink d3/d3a/d75/f5b 0 2024-04-22T15:18:55.670 INFO:tasks.workunit.client.1.smithi196.stdout:4/341: fdatasync d6/f19 0 2024-04-22T15:18:55.670 INFO:tasks.workunit.client.0.smithi096.stdout:6/576: chown d3/d28/dd/d11/d2c/d3b/f63 1497 1 2024-04-22T15:18:55.672 INFO:tasks.workunit.client.0.smithi096.stdout:3/554: mkdir da/db/d12/d22/d26/db2 0 2024-04-22T15:18:55.675 INFO:tasks.workunit.client.1.smithi196.stdout:4/342: dwrite d6/d10/f63 [0,4194304] 0 2024-04-22T15:18:55.677 INFO:tasks.workunit.client.0.smithi096.stdout:6/577: dwrite d3/d28/f38 [0,4194304] 0 2024-04-22T15:18:55.680 INFO:tasks.workunit.client.1.smithi196.stdout:2/354: rename d1/d23/f40 to d1/f80 0 2024-04-22T15:18:55.680 INFO:tasks.workunit.client.0.smithi096.stdout:0/793: write d7/de/d3a/f76 [1941052,39694] 0 2024-04-22T15:18:55.682 INFO:tasks.workunit.client.1.smithi196.stdout:8/334: dwrite d2/d35/f5f [0,4194304] 0 2024-04-22T15:18:55.682 INFO:tasks.workunit.client.1.smithi196.stdout:3/309: sync 2024-04-22T15:18:55.683 INFO:tasks.workunit.client.1.smithi196.stdout:3/310: write d0/d7/d8/f66 [1025039,72896] 0 2024-04-22T15:18:55.684 INFO:tasks.workunit.client.0.smithi096.stdout:1/748: creat d6/dc/d8a/d3f/d54/d32/d3b/fe3 x:0 0 0 2024-04-22T15:18:55.690 INFO:tasks.workunit.client.1.smithi196.stdout:1/272: fsync d1/db/dc/d26/d31/f50 0 2024-04-22T15:18:55.691 INFO:tasks.workunit.client.1.smithi196.stdout:7/282: truncate d14/d15/f1a 846467 0 2024-04-22T15:18:55.691 INFO:tasks.workunit.client.1.smithi196.stdout:1/273: write d1/f4 [686970,106733] 0 2024-04-22T15:18:55.692 INFO:tasks.workunit.client.0.smithi096.stdout:5/681: creat d4/ddb/d3d/d4d/d59/db0/fdd x:0 0 0 2024-04-22T15:18:55.694 INFO:tasks.workunit.client.1.smithi196.stdout:1/274: write d1/db/d16/f5c [698208,30070] 0 2024-04-22T15:18:55.694 INFO:tasks.workunit.client.0.smithi096.stdout:7/908: mkdir d31/d32/dc0/ddc/de9/d137 0 2024-04-22T15:18:55.694 INFO:tasks.workunit.client.1.smithi196.stdout:1/275: read d1/db/d16/d1e/d49/f42 [23428,30238] 0 2024-04-22T15:18:55.695 INFO:tasks.workunit.client.1.smithi196.stdout:1/276: chown d1/db/l56 0 1 2024-04-22T15:18:55.697 INFO:tasks.workunit.client.1.smithi196.stdout:9/316: creat db/dd/de/d15/f73 x:0 0 0 2024-04-22T15:18:55.698 INFO:tasks.workunit.client.1.smithi196.stdout:1/277: dread d1/db/d16/f4b [0,4194304] 0 2024-04-22T15:18:55.699 INFO:tasks.workunit.client.1.smithi196.stdout:0/344: getdents d3 0 2024-04-22T15:18:55.700 INFO:tasks.workunit.client.1.smithi196.stdout:0/345: fsync d3/de/d1c/d27/d2c/d51/f59 0 2024-04-22T15:18:55.701 INFO:tasks.workunit.client.0.smithi096.stdout:3/555: mknod da/db/dc/d28/d69/cb3 0 2024-04-22T15:18:55.705 INFO:tasks.workunit.client.0.smithi096.stdout:4/693: rename d5/d7/d2c/d43/d56 to d5/d7/d2c/d95/dee 0 2024-04-22T15:18:55.709 INFO:tasks.workunit.client.0.smithi096.stdout:0/794: truncate d7/de/d5b/d95/dc/d16/d65/d7c/fbd 1064119 0 2024-04-22T15:18:55.712 INFO:tasks.workunit.client.0.smithi096.stdout:0/795: truncate d7/de/d5b/d95/dc/fed 854789 0 2024-04-22T15:18:55.713 INFO:tasks.workunit.client.0.smithi096.stdout:0/796: read d7/de/d5b/d64/fd9 [727506,14298] 0 2024-04-22T15:18:55.714 INFO:tasks.workunit.client.1.smithi196.stdout:2/355: unlink d1/f2c 0 2024-04-22T15:18:55.718 INFO:tasks.workunit.client.0.smithi096.stdout:3/556: dread da/f8a [0,4194304] 0 2024-04-22T15:18:55.719 INFO:tasks.workunit.client.1.smithi196.stdout:3/311: creat d0/d15/d4d/db/dc/d4a/d44/f6d x:0 0 0 2024-04-22T15:18:55.723 INFO:tasks.workunit.client.0.smithi096.stdout:4/694: dread - d5/d7/d2c/d67/fe3 zero size 2024-04-22T15:18:55.724 INFO:tasks.workunit.client.0.smithi096.stdout:4/695: dread d5/d7/f8e [0,4194304] 0 2024-04-22T15:18:55.729 INFO:tasks.workunit.client.0.smithi096.stdout:5/682: link d4/ddb/d25/fd2 d4/ddb/d3d/d35/d57/fde 0 2024-04-22T15:18:55.731 INFO:tasks.workunit.client.0.smithi096.stdout:0/797: unlink d7/de/d5b/d64/dc9/fcc 0 2024-04-22T15:18:55.734 INFO:tasks.workunit.client.1.smithi196.stdout:9/317: rmdir db/dd 39 2024-04-22T15:18:55.736 INFO:tasks.workunit.client.1.smithi196.stdout:9/318: dread db/d11/d29/d57/f5a [0,4194304] 0 2024-04-22T15:18:55.736 INFO:tasks.workunit.client.1.smithi196.stdout:1/278: symlink d1/db/dc/de/d5a/l5d 0 2024-04-22T15:18:55.736 INFO:tasks.workunit.client.1.smithi196.stdout:1/279: fsync d1/db/dc/d26/d31/f50 0 2024-04-22T15:18:55.741 INFO:tasks.workunit.client.0.smithi096.stdout:1/749: rename d6/dc/d8a/d3f/d62/l7c to d6/dc/d8a/d3f/d62/dc2/dd0/le4 0 2024-04-22T15:18:55.743 INFO:tasks.workunit.client.1.smithi196.stdout:0/346: rename d3/de/f18 to d3/de/d1c/d27/d2c/d51/f75 0 2024-04-22T15:18:55.744 INFO:tasks.workunit.client.1.smithi196.stdout:0/347: chown d3/de/d1c/d27/d2c/d30/f35 50 1 2024-04-22T15:18:55.744 INFO:tasks.workunit.client.1.smithi196.stdout:0/348: rename d3/de/d14/d72 to d3/de/d14/d72/d76 22 2024-04-22T15:18:55.744 INFO:tasks.workunit.client.1.smithi196.stdout:0/349: stat d3/de/d1c/f31 0 2024-04-22T15:18:55.745 INFO:tasks.workunit.client.1.smithi196.stdout:6/348: unlink dd/d1f/d37/f3f 0 2024-04-22T15:18:55.745 INFO:tasks.workunit.client.1.smithi196.stdout:8/335: write d2/f32 [814992,56165] 0 2024-04-22T15:18:55.746 INFO:tasks.workunit.client.1.smithi196.stdout:6/349: chown dd/d2b/f3e 1128101 1 2024-04-22T15:18:55.746 INFO:tasks.workunit.client.0.smithi096.stdout:6/578: creat d3/d28/dd/d11/d2c/fc7 x:0 0 0 2024-04-22T15:18:55.750 INFO:tasks.workunit.client.1.smithi196.stdout:0/350: dwrite d3/fd [4194304,4194304] 0 2024-04-22T15:18:55.751 INFO:tasks.workunit.client.0.smithi096.stdout:5/683: creat d4/ddb/d3d/d4d/d59/db0/db2/fdf x:0 0 0 2024-04-22T15:18:55.755 INFO:tasks.workunit.client.1.smithi196.stdout:4/343: sync 2024-04-22T15:18:55.755 INFO:tasks.workunit.client.1.smithi196.stdout:3/312: fsync d0/d15/d4d/db/dc/d4a/d44/f3c 0 2024-04-22T15:18:55.756 INFO:tasks.workunit.client.1.smithi196.stdout:3/313: chown d0/d15/d48/d51 29313 1 2024-04-22T15:18:55.758 INFO:tasks.workunit.client.1.smithi196.stdout:7/283: mknod d14/c52 0 2024-04-22T15:18:55.760 INFO:tasks.workunit.client.0.smithi096.stdout:3/557: mknod da/db/cb4 0 2024-04-22T15:18:55.760 INFO:tasks.workunit.client.1.smithi196.stdout:3/314: dread d0/d15/d4d/db/dc/d4a/d44/f16 [0,4194304] 0 2024-04-22T15:18:55.761 INFO:tasks.workunit.client.1.smithi196.stdout:3/315: chown d0/d15/d4d/db/dc/d18/f4c 14812 1 2024-04-22T15:18:55.761 INFO:tasks.workunit.client.0.smithi096.stdout:7/909: truncate d31/d32/d33/d6a/d83/dba/dc7/fcd 3645914 0 2024-04-22T15:18:55.763 INFO:tasks.workunit.client.0.smithi096.stdout:1/750: fsync d6/dc/d8a/f7a 0 2024-04-22T15:18:55.766 INFO:tasks.workunit.client.1.smithi196.stdout:6/350: dread dd/d2b/f2f [4194304,4194304] 0 2024-04-22T15:18:55.767 INFO:tasks.workunit.client.1.smithi196.stdout:6/351: dread - dd/d2b/f2e zero size 2024-04-22T15:18:55.767 INFO:tasks.workunit.client.1.smithi196.stdout:6/352: chown lb 241 1 2024-04-22T15:18:55.768 INFO:tasks.workunit.client.0.smithi096.stdout:6/579: unlink d3/d28/d4d/d82/d9e/db0/cb6 0 2024-04-22T15:18:55.768 INFO:tasks.workunit.client.1.smithi196.stdout:6/353: write dd/f14 [627151,101985] 0 2024-04-22T15:18:55.771 INFO:tasks.workunit.client.0.smithi096.stdout:4/696: mkdir d5/d7/d2c/d3d/def 0 2024-04-22T15:18:55.772 INFO:tasks.workunit.client.1.smithi196.stdout:1/280: mkdir d1/db/dc/de/d43/d5e 0 2024-04-22T15:18:55.773 INFO:tasks.workunit.client.1.smithi196.stdout:6/354: dwrite dd/f1a [0,4194304] 0 2024-04-22T15:18:55.773 INFO:tasks.workunit.client.1.smithi196.stdout:1/281: dread - d1/db/dc/d26/d2e/f51 zero size 2024-04-22T15:18:55.776 INFO:tasks.workunit.client.0.smithi096.stdout:5/684: creat d4/ddb/d3d/d4d/d59/db0/db2/fe0 x:0 0 0 2024-04-22T15:18:55.779 INFO:tasks.workunit.client.1.smithi196.stdout:2/356: dwrite d1/d23/f2f [0,4194304] 0 2024-04-22T15:18:55.781 INFO:tasks.workunit.client.0.smithi096.stdout:7/910: mkdir d31/d32/d33/d93/d138 0 2024-04-22T15:18:55.782 INFO:tasks.workunit.client.0.smithi096.stdout:7/911: chown d31/d32/dc0/f11a 949657319 1 2024-04-22T15:18:55.787 INFO:tasks.workunit.client.1.smithi196.stdout:0/351: mknod d3/de/d1c/d27/d2c/d30/c77 0 2024-04-22T15:18:55.788 INFO:tasks.workunit.client.0.smithi096.stdout:1/751: creat d6/dc/d8a/d3f/d54/d32/d82/fe5 x:0 0 0 2024-04-22T15:18:55.794 INFO:tasks.workunit.client.0.smithi096.stdout:5/685: symlink d4/d2c/d2f/d31/d82/le1 0 2024-04-22T15:18:55.796 INFO:tasks.workunit.client.1.smithi196.stdout:4/344: fdatasync d6/d13/d15/f30 0 2024-04-22T15:18:55.797 INFO:tasks.workunit.client.0.smithi096.stdout:5/686: dread - d4/ddb/d3d/d4d/d59/d98/fd1 zero size 2024-04-22T15:18:55.798 INFO:tasks.workunit.client.0.smithi096.stdout:0/798: dwrite d7/de/d5b/d95/dc/d16/d65/d7c/f85 [0,4194304] 0 2024-04-22T15:18:55.802 INFO:tasks.workunit.client.1.smithi196.stdout:7/284: truncate d14/d25/d31/f3f 1539141 0 2024-04-22T15:18:55.804 INFO:tasks.workunit.client.1.smithi196.stdout:3/316: fsync d0/d15/f12 0 2024-04-22T15:18:55.806 INFO:tasks.workunit.client.1.smithi196.stdout:5/321: getdents d1/d14 0 2024-04-22T15:18:55.809 INFO:tasks.workunit.client.0.smithi096.stdout:5/687: symlink d4/d2c/d2f/d31/d82/le2 0 2024-04-22T15:18:55.811 INFO:tasks.workunit.client.0.smithi096.stdout:5/688: truncate d4/ddb/d3d/d4d/d59/db0/db2/fdf 551991 0 2024-04-22T15:18:55.811 INFO:tasks.workunit.client.0.smithi096.stdout:3/558: write da/db/d12/d22/f31 [859289,62246] 0 2024-04-22T15:18:55.812 INFO:tasks.workunit.client.1.smithi196.stdout:9/319: symlink db/dd/d1b/d2c/l74 0 2024-04-22T15:18:55.814 INFO:tasks.workunit.client.0.smithi096.stdout:0/799: truncate d7/de/d3a/d61/dbb/fea 720076 0 2024-04-22T15:18:55.815 INFO:tasks.workunit.client.1.smithi196.stdout:1/282: read d1/db/d16/f40 [307703,85950] 0 2024-04-22T15:18:55.816 INFO:tasks.workunit.client.0.smithi096.stdout:7/912: truncate d31/f8f 291157 0 2024-04-22T15:18:55.816 INFO:tasks.workunit.client.1.smithi196.stdout:1/283: dread - d1/d54/f55 zero size 2024-04-22T15:18:55.819 INFO:tasks.workunit.client.1.smithi196.stdout:8/336: stat d2/dc/f2f 0 2024-04-22T15:18:55.820 INFO:tasks.workunit.client.0.smithi096.stdout:4/697: creat d5/d7/d2c/ff0 x:0 0 0 2024-04-22T15:18:55.822 INFO:tasks.workunit.client.1.smithi196.stdout:2/357: rename d1/d23/d27/d2e/f47 to d1/d1d/d6a/f81 0 2024-04-22T15:18:55.822 INFO:tasks.workunit.client.0.smithi096.stdout:5/689: chown d4/d2c/c44 0 1 2024-04-22T15:18:55.823 INFO:tasks.workunit.client.1.smithi196.stdout:2/358: fdatasync d1/ff 0 2024-04-22T15:18:55.824 INFO:tasks.workunit.client.1.smithi196.stdout:2/359: read d1/d23/d27/d2e/d55/f36 [2360813,40889] 0 2024-04-22T15:18:55.825 INFO:tasks.workunit.client.0.smithi096.stdout:3/559: readlink da/db/d12/d22/l3c 0 2024-04-22T15:18:55.826 INFO:tasks.workunit.client.0.smithi096.stdout:3/560: write da/db/de/d19/f73 [970325,24618] 0 2024-04-22T15:18:55.828 INFO:tasks.workunit.client.0.smithi096.stdout:3/561: write da/dab/fa2 [983746,81737] 0 2024-04-22T15:18:55.828 INFO:tasks.workunit.client.0.smithi096.stdout:0/800: rmdir d7/de/d3a/d61 39 2024-04-22T15:18:55.831 INFO:tasks.workunit.client.0.smithi096.stdout:3/562: truncate da/db/d12/d22/d26/d52/f86 483087 0 2024-04-22T15:18:55.833 INFO:tasks.workunit.client.0.smithi096.stdout:1/752: creat d6/fe6 x:0 0 0 2024-04-22T15:18:55.834 INFO:tasks.workunit.client.1.smithi196.stdout:4/345: unlink d6/d13/d15/d2c/d35/l3e 0 2024-04-22T15:18:55.834 INFO:tasks.workunit.client.1.smithi196.stdout:4/346: write d6/f42 [4601734,123558] 0 2024-04-22T15:18:55.835 INFO:tasks.workunit.client.0.smithi096.stdout:4/698: creat d5/d7/d2c/d67/ff1 x:0 0 0 2024-04-22T15:18:55.836 INFO:tasks.workunit.client.1.smithi196.stdout:7/285: symlink d14/d25/d31/l53 0 2024-04-22T15:18:55.837 INFO:tasks.workunit.client.1.smithi196.stdout:5/322: mkdir d1/d17/d1a/d2f/d65 0 2024-04-22T15:18:55.838 INFO:tasks.workunit.client.0.smithi096.stdout:3/563: dwrite da/db/d12/d22/f4e [0,4194304] 0 2024-04-22T15:18:55.838 INFO:tasks.workunit.client.1.smithi196.stdout:5/323: stat d1/d17/d1a/d2f/d30/f61 0 2024-04-22T15:18:55.839 INFO:tasks.workunit.client.0.smithi096.stdout:3/564: fsync da/db/d12/d22/f31 0 2024-04-22T15:18:55.841 INFO:tasks.workunit.client.0.smithi096.stdout:0/801: dread - d7/de/fe8 zero size 2024-04-22T15:18:55.841 INFO:tasks.workunit.client.1.smithi196.stdout:5/324: dread d1/d17/d1c/f52 [0,4194304] 0 2024-04-22T15:18:55.842 INFO:tasks.workunit.client.1.smithi196.stdout:6/355: rename dd/d1f/d4e/c52 to dd/d1f/d4e/c61 0 2024-04-22T15:18:55.843 INFO:tasks.workunit.client.0.smithi096.stdout:0/802: write d7/de/d5b/d95/dc/fed [110779,62322] 0 2024-04-22T15:18:55.844 INFO:tasks.workunit.client.0.smithi096.stdout:0/803: readlink d7/de/d5b/d95/dc/d16/l60 0 2024-04-22T15:18:55.849 INFO:tasks.workunit.client.1.smithi196.stdout:8/337: rmdir d2/d5/d2e 39 2024-04-22T15:18:55.853 INFO:tasks.workunit.client.0.smithi096.stdout:4/699: truncate d5/d7/d2c/d3d/d4f/fb0 605015 0 2024-04-22T15:18:55.856 INFO:tasks.workunit.client.1.smithi196.stdout:9/320: dwrite fa [0,4194304] 0 2024-04-22T15:18:55.856 INFO:tasks.workunit.client.1.smithi196.stdout:0/352: rename d3/de/d1c/d27/d2c/d51/f59 to d3/de/d1c/d27/d2c/d30/f78 0 2024-04-22T15:18:55.857 INFO:tasks.workunit.client.1.smithi196.stdout:9/321: truncate db/dd/de/f4e 156279 0 2024-04-22T15:18:55.861 INFO:tasks.workunit.client.1.smithi196.stdout:2/360: mknod d1/d23/d27/d2e/d55/d5b/c82 0 2024-04-22T15:18:55.863 INFO:tasks.workunit.client.0.smithi096.stdout:7/913: dwrite d31/d32/d33/d54/f64 [0,4194304] 0 2024-04-22T15:18:55.868 INFO:tasks.workunit.client.0.smithi096.stdout:5/690: rmdir d4/ddb/dc5 0 2024-04-22T15:18:55.869 INFO:tasks.workunit.client.0.smithi096.stdout:1/753: dread d6/dc/d8a/d3f/d54/d32/d3b/d3d/f8b [0,4194304] 0 2024-04-22T15:18:55.870 INFO:tasks.workunit.client.0.smithi096.stdout:4/700: fsync d5/d7/d1f/f50 0 2024-04-22T15:18:55.875 INFO:tasks.workunit.client.0.smithi096.stdout:7/914: mkdir d31/d32/d33/d93/dd5/d139 0 2024-04-22T15:18:55.879 INFO:tasks.workunit.client.1.smithi196.stdout:6/356: truncate dd/f11 5267035 0 2024-04-22T15:18:55.883 INFO:tasks.workunit.client.0.smithi096.stdout:4/701: chown d5/d7/d2c/d43/c4b 3530205 1 2024-04-22T15:18:55.883 INFO:tasks.workunit.client.1.smithi196.stdout:6/357: dwrite fc [4194304,4194304] 0 2024-04-22T15:18:55.888 INFO:tasks.workunit.client.0.smithi096.stdout:0/804: fsync d7/de/d3a/d61/dbb/fea 0 2024-04-22T15:18:55.890 INFO:tasks.workunit.client.0.smithi096.stdout:1/754: truncate d6/dc/d8a/d3f/d54/d32/d3b/d56/fb9 458385 0 2024-04-22T15:18:55.891 INFO:tasks.workunit.client.1.smithi196.stdout:0/353: dwrite d3/d6/dc/f58 [4194304,4194304] 0 2024-04-22T15:18:55.895 INFO:tasks.workunit.client.1.smithi196.stdout:9/322: dwrite db/fc [0,4194304] 0 2024-04-22T15:18:55.895 INFO:tasks.workunit.client.1.smithi196.stdout:3/317: sync 2024-04-22T15:18:55.897 INFO:tasks.workunit.client.1.smithi196.stdout:2/361: symlink d1/d23/d27/d2e/d45/l83 0 2024-04-22T15:18:55.898 INFO:tasks.workunit.client.1.smithi196.stdout:0/354: dread d3/de/d1c/d27/d2c/d51/f75 [0,4194304] 0 2024-04-22T15:18:55.899 INFO:tasks.workunit.client.1.smithi196.stdout:4/347: symlink d6/d13/d15/l64 0 2024-04-22T15:18:55.899 INFO:tasks.workunit.client.0.smithi096.stdout:5/691: write d4/d2c/d2f/d31/f8f [761766,84569] 0 2024-04-22T15:18:55.900 INFO:tasks.workunit.client.0.smithi096.stdout:3/565: dwrite da/db/de/d19/d2d/f2e [0,4194304] 0 2024-04-22T15:18:55.904 INFO:tasks.workunit.client.0.smithi096.stdout:5/692: chown d4/ddb/d3d/d4d/d59/d5a/d5c/c73 5826920 1 2024-04-22T15:18:55.908 INFO:tasks.workunit.client.1.smithi196.stdout:5/325: symlink d1/d2/l66 0 2024-04-22T15:18:55.910 INFO:tasks.workunit.client.0.smithi096.stdout:5/693: dwrite d4/ddb/d3d/d35/fb8 [0,4194304] 0 2024-04-22T15:18:55.917 INFO:tasks.workunit.client.1.smithi196.stdout:1/284: rename d1/c6 to d1/db/dc/de/d5a/c5f 0 2024-04-22T15:18:55.917 INFO:tasks.workunit.client.1.smithi196.stdout:1/285: stat d1/db/d16/d1e 0 2024-04-22T15:18:55.918 INFO:tasks.workunit.client.0.smithi096.stdout:7/915: dwrite f21 [0,4194304] 0 2024-04-22T15:18:55.919 INFO:tasks.workunit.client.1.smithi196.stdout:1/286: chown d1/db/dc/d26/d2e/d32/d36/l59 1 1 2024-04-22T15:18:55.919 INFO:tasks.workunit.client.1.smithi196.stdout:9/323: mkdir db/dd/de/d75 0 2024-04-22T15:18:55.921 INFO:tasks.workunit.client.0.smithi096.stdout:3/566: mkdir da/db/de/d8b/db5 0 2024-04-22T15:18:55.921 INFO:tasks.workunit.client.0.smithi096.stdout:3/567: stat da/db/d10/c18 0 2024-04-22T15:18:55.921 INFO:tasks.workunit.client.1.smithi196.stdout:1/287: chown d1/db/dc/de/c15 120082504 1 2024-04-22T15:18:55.921 INFO:tasks.workunit.client.1.smithi196.stdout:3/318: mkdir d0/d15/d31/d55/d6e 0 2024-04-22T15:18:55.922 INFO:tasks.workunit.client.1.smithi196.stdout:8/338: sync 2024-04-22T15:18:55.923 INFO:tasks.workunit.client.1.smithi196.stdout:3/319: chown d0/d15/d48/d51/d5e/f63 11 1 2024-04-22T15:18:55.927 INFO:tasks.workunit.client.0.smithi096.stdout:0/805: symlink d7/d8f/d10e/l10f 0 2024-04-22T15:18:55.927 INFO:tasks.workunit.client.1.smithi196.stdout:4/348: creat d6/d14/d18/d40/f65 x:0 0 0 2024-04-22T15:18:55.929 INFO:tasks.workunit.client.1.smithi196.stdout:4/349: write d6/d13/d15/d2c/d62/f5a [330919,58594] 0 2024-04-22T15:18:55.929 INFO:tasks.workunit.client.1.smithi196.stdout:7/286: link f13 d14/d26/f54 0 2024-04-22T15:18:55.930 INFO:tasks.workunit.client.1.smithi196.stdout:7/287: chown d14/f33 54907 1 2024-04-22T15:18:55.930 INFO:tasks.workunit.client.1.smithi196.stdout:2/362: dwrite d1/f7e [0,4194304] 0 2024-04-22T15:18:55.930 INFO:tasks.workunit.client.1.smithi196.stdout:5/326: mkdir d1/d17/d1c/d67 0 2024-04-22T15:18:55.932 INFO:tasks.workunit.client.0.smithi096.stdout:4/702: rename d5/d7/d2c/d95/dee/c71 to d5/d7/d2c/cf2 0 2024-04-22T15:18:55.934 INFO:tasks.workunit.client.1.smithi196.stdout:6/358: unlink dd/d1f/c32 0 2024-04-22T15:18:55.937 INFO:tasks.workunit.client.1.smithi196.stdout:9/324: symlink db/dd/de/d15/d31/l76 0 2024-04-22T15:18:55.937 INFO:tasks.workunit.client.0.smithi096.stdout:3/568: read da/db/dc/f91 [3888118,36742] 0 2024-04-22T15:18:55.938 INFO:tasks.workunit.client.0.smithi096.stdout:5/694: symlink d4/d2c/d2f/d30/le3 0 2024-04-22T15:18:55.938 INFO:tasks.workunit.client.0.smithi096.stdout:3/569: stat da/db/d12/d67/c74 0 2024-04-22T15:18:55.940 INFO:tasks.workunit.client.0.smithi096.stdout:0/806: creat d7/da7/f110 x:0 0 0 2024-04-22T15:18:55.941 INFO:tasks.workunit.client.1.smithi196.stdout:3/320: mkdir d0/d7/d6f 0 2024-04-22T15:18:55.943 INFO:tasks.workunit.client.0.smithi096.stdout:1/755: creat d6/dc/d8a/d3f/d54/d32/d8c/fe7 x:0 0 0 2024-04-22T15:18:55.944 INFO:tasks.workunit.client.0.smithi096.stdout:4/703: symlink d5/d7/dcb/lf3 0 2024-04-22T15:18:55.945 INFO:tasks.workunit.client.1.smithi196.stdout:0/355: mknod d3/de/c79 0 2024-04-22T15:18:55.945 INFO:tasks.workunit.client.0.smithi096.stdout:4/704: fsync d5/d7/d1f/f50 0 2024-04-22T15:18:55.946 INFO:tasks.workunit.client.0.smithi096.stdout:4/705: fdatasync d5/d7/d1f/fca 0 2024-04-22T15:18:55.948 INFO:tasks.workunit.client.0.smithi096.stdout:5/695: creat d4/ddb/d3d/d4f/fe4 x:0 0 0 2024-04-22T15:18:55.948 INFO:tasks.workunit.client.1.smithi196.stdout:4/350: mknod d6/d13/d15/d2c/d62/c66 0 2024-04-22T15:18:55.948 INFO:tasks.workunit.client.0.smithi096.stdout:5/696: read d4/ddb/fbd [3304168,77595] 0 2024-04-22T15:18:55.949 INFO:tasks.workunit.client.0.smithi096.stdout:3/570: fsync da/db/dc/d28/f93 0 2024-04-22T15:18:55.950 INFO:tasks.workunit.client.1.smithi196.stdout:5/327: mkdir d1/d9/d68 0 2024-04-22T15:18:55.952 INFO:tasks.workunit.client.1.smithi196.stdout:2/363: mkdir d1/d23/d84 0 2024-04-22T15:18:55.953 INFO:tasks.workunit.client.0.smithi096.stdout:1/756: chown d6/dc/d8a/d3f/d54/d32/d8c/d45/f50 1419725848 1 2024-04-22T15:18:55.954 INFO:tasks.workunit.client.1.smithi196.stdout:2/364: write d1/d23/d27/d2e/d45/f79 [207236,116384] 0 2024-04-22T15:18:55.954 INFO:tasks.workunit.client.1.smithi196.stdout:7/288: creat d14/d25/d31/f55 x:0 0 0 2024-04-22T15:18:55.957 INFO:tasks.workunit.client.1.smithi196.stdout:6/359: readlink dd/d1f/d37/d3c/l4a 0 2024-04-22T15:18:55.957 INFO:tasks.workunit.client.1.smithi196.stdout:5/328: dwrite d1/d17/d1a/d2f/d30/d5e/f63 [0,4194304] 0 2024-04-22T15:18:55.961 INFO:tasks.workunit.client.1.smithi196.stdout:9/325: rename db/l6c to db/dd/d1b/d2c/l77 0 2024-04-22T15:18:55.963 INFO:tasks.workunit.client.1.smithi196.stdout:7/289: dwrite d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:18:55.963 INFO:tasks.workunit.client.1.smithi196.stdout:7/290: stat fe 0 2024-04-22T15:18:55.965 INFO:tasks.workunit.client.1.smithi196.stdout:9/326: dread db/dd/de/d15/d31/f32 [0,4194304] 0 2024-04-22T15:18:55.966 INFO:tasks.workunit.client.1.smithi196.stdout:3/321: mkdir d0/d7/d8/d70 0 2024-04-22T15:18:55.970 INFO:tasks.workunit.client.0.smithi096.stdout:3/571: creat da/db/de/d87/fb6 x:0 0 0 2024-04-22T15:18:55.970 INFO:tasks.workunit.client.1.smithi196.stdout:9/327: dwrite db/d11/d29/f6f [0,4194304] 0 2024-04-22T15:18:55.978 INFO:tasks.workunit.client.1.smithi196.stdout:2/365: creat d1/d23/d27/d2e/f85 x:0 0 0 2024-04-22T15:18:55.979 INFO:tasks.workunit.client.1.smithi196.stdout:2/366: readlink d1/d23/d27/d2e/d45/l83 0 2024-04-22T15:18:55.979 INFO:tasks.workunit.client.0.smithi096.stdout:7/916: link c25 d31/d32/d36/d75/dff/c13a 0 2024-04-22T15:18:55.982 INFO:tasks.workunit.client.1.smithi196.stdout:6/360: unlink dd/c2a 0 2024-04-22T15:18:55.984 INFO:tasks.workunit.client.1.smithi196.stdout:2/367: dwrite d1/ff [4194304,4194304] 0 2024-04-22T15:18:55.984 INFO:tasks.workunit.client.1.smithi196.stdout:5/329: unlink d1/d9/c15 0 2024-04-22T15:18:55.991 INFO:tasks.workunit.client.0.smithi096.stdout:5/697: mkdir d4/ddb/d3d/d4d/d88/de5 0 2024-04-22T15:18:55.992 INFO:tasks.workunit.client.0.smithi096.stdout:5/698: stat d4/ddb/d3d/d4d/d59/d98/cb7 0 2024-04-22T15:18:55.994 INFO:tasks.workunit.client.1.smithi196.stdout:0/356: rename d3/de/l64 to d3/d9/d25/l7a 0 2024-04-22T15:18:55.995 INFO:tasks.workunit.client.0.smithi096.stdout:3/572: fsync da/db/de/d8b/f92 0 2024-04-22T15:18:55.998 INFO:tasks.workunit.client.0.smithi096.stdout:0/807: rmdir d7/de/d5b/d108 0 2024-04-22T15:18:55.999 INFO:tasks.workunit.client.0.smithi096.stdout:0/808: fdatasync d7/de/d5b/d95/d15/f36 0 2024-04-22T15:18:56.000 INFO:tasks.workunit.client.1.smithi196.stdout:3/322: mkdir d0/d15/d31/d55/d71 0 2024-04-22T15:18:56.001 INFO:tasks.workunit.client.0.smithi096.stdout:7/917: chown d31/d32/d36/cce 371213 1 2024-04-22T15:18:56.004 INFO:tasks.workunit.client.1.smithi196.stdout:6/361: dread dd/d1f/d37/f31 [0,4194304] 0 2024-04-22T15:18:56.005 INFO:tasks.workunit.client.0.smithi096.stdout:4/706: dwrite d5/fc [4194304,4194304] 0 2024-04-22T15:18:56.006 INFO:tasks.workunit.client.0.smithi096.stdout:3/573: mknod da/db/dc/d28/d96/cb7 0 2024-04-22T15:18:56.007 INFO:tasks.workunit.client.1.smithi196.stdout:4/351: mkdir d6/d14/d18/d26/d32/d67 0 2024-04-22T15:18:56.012 INFO:tasks.workunit.client.0.smithi096.stdout:0/809: symlink d7/de/d3a/d61/d79/l111 0 2024-04-22T15:18:56.013 INFO:tasks.workunit.client.1.smithi196.stdout:5/330: creat d1/d51/f69 x:0 0 0 2024-04-22T15:18:56.019 INFO:tasks.workunit.client.0.smithi096.stdout:3/574: fsync da/db/de/f57 0 2024-04-22T15:18:56.020 INFO:tasks.workunit.client.1.smithi196.stdout:1/288: getdents d1/db/dc/d26/d2e/d32/d36 0 2024-04-22T15:18:56.023 INFO:tasks.workunit.client.1.smithi196.stdout:1/289: dread d1/db/d16/f5c [0,4194304] 0 2024-04-22T15:18:56.024 INFO:tasks.workunit.client.1.smithi196.stdout:1/290: dread - d1/db/f58 zero size 2024-04-22T15:18:56.024 INFO:tasks.workunit.client.1.smithi196.stdout:6/362: rmdir dd/d1f/d37/d3c 39 2024-04-22T15:18:56.025 INFO:tasks.workunit.client.0.smithi096.stdout:4/707: chown d5/d5b/l77 473497 1 2024-04-22T15:18:56.026 INFO:tasks.workunit.client.1.smithi196.stdout:8/339: getdents d2/d5/d4f 0 2024-04-22T15:18:56.027 INFO:tasks.workunit.client.0.smithi096.stdout:0/810: write d7/de/d5b/d95/dc/d16/f5d [591194,88056] 0 2024-04-22T15:18:56.029 INFO:tasks.workunit.client.1.smithi196.stdout:4/352: unlink d6/d14/f2a 0 2024-04-22T15:18:56.029 INFO:tasks.workunit.client.0.smithi096.stdout:3/575: creat da/db/de/d38/fb8 x:0 0 0 2024-04-22T15:18:56.031 INFO:tasks.workunit.client.0.smithi096.stdout:7/918: rename d31/d32/d33/d54/def to d31/d32/d13b 0 2024-04-22T15:18:56.033 INFO:tasks.workunit.client.1.smithi196.stdout:7/291: rename d14/l1b to d14/d26/d28/l56 0 2024-04-22T15:18:56.035 INFO:tasks.workunit.client.1.smithi196.stdout:5/331: dwrite d1/d9/f38 [0,4194304] 0 2024-04-22T15:18:56.040 INFO:tasks.workunit.client.0.smithi096.stdout:5/699: rename d4/ddb/d3d/d4d/d99/fc2 to d4/d1f/d8b/fe6 0 2024-04-22T15:18:56.041 INFO:tasks.workunit.client.1.smithi196.stdout:7/292: dread f13 [0,4194304] 0 2024-04-22T15:18:56.042 INFO:tasks.workunit.client.1.smithi196.stdout:1/291: dread d1/db/dc/d26/f2b [0,4194304] 0 2024-04-22T15:18:56.043 INFO:tasks.workunit.client.0.smithi096.stdout:7/919: chown d31/d32/d36/f4d 1323 1 2024-04-22T15:18:56.044 INFO:tasks.workunit.client.1.smithi196.stdout:5/332: dwrite d1/d9/f38 [0,4194304] 0 2024-04-22T15:18:56.045 INFO:tasks.workunit.client.0.smithi096.stdout:4/708: fdatasync d5/d7/d2c/d95/dee/f94 0 2024-04-22T15:18:56.047 INFO:tasks.workunit.client.0.smithi096.stdout:4/709: write d5/d7/d2c/f86 [1563357,112804] 0 2024-04-22T15:18:56.047 INFO:tasks.workunit.client.1.smithi196.stdout:6/363: creat dd/d2b/f62 x:0 0 0 2024-04-22T15:18:56.048 INFO:tasks.workunit.client.0.smithi096.stdout:4/710: fdatasync d5/d7/d1a/f38 0 2024-04-22T15:18:56.048 INFO:tasks.workunit.client.1.smithi196.stdout:0/357: truncate d3/de/d1c/d27/d2c/d30/f78 1539732 0 2024-04-22T15:18:56.052 INFO:tasks.workunit.client.1.smithi196.stdout:8/340: symlink d2/d3f/d4b/l66 0 2024-04-22T15:18:56.053 INFO:tasks.workunit.client.1.smithi196.stdout:3/323: truncate d0/d15/d48/d51/f54 2796188 0 2024-04-22T15:18:56.060 INFO:tasks.workunit.client.0.smithi096.stdout:3/576: rename l3 to da/db/dc/d59/lb9 0 2024-04-22T15:18:56.062 INFO:tasks.workunit.client.0.smithi096.stdout:5/700: readlink d4/ddb/d3d/d4d/l78 0 2024-04-22T15:18:56.062 INFO:tasks.workunit.client.0.smithi096.stdout:0/811: dwrite d7/de/d5b/d64/f107 [0,4194304] 0 2024-04-22T15:18:56.063 INFO:tasks.workunit.client.1.smithi196.stdout:2/368: sync 2024-04-22T15:18:56.063 INFO:tasks.workunit.client.1.smithi196.stdout:9/328: sync 2024-04-22T15:18:56.063 INFO:tasks.workunit.client.0.smithi096.stdout:5/701: chown d4/ddb/d25/f90 216 1 2024-04-22T15:18:56.071 INFO:tasks.workunit.client.1.smithi196.stdout:3/324: dread d0/d15/d4d/db/dc/d4a/d44/f20 [0,4194304] 0 2024-04-22T15:18:56.072 INFO:tasks.workunit.client.1.smithi196.stdout:7/293: symlink d14/d25/l57 0 2024-04-22T15:18:56.073 INFO:tasks.workunit.client.1.smithi196.stdout:6/364: dread dd/f14 [0,4194304] 0 2024-04-22T15:18:56.076 INFO:tasks.workunit.client.1.smithi196.stdout:5/333: mkdir d1/d14/d6a 0 2024-04-22T15:18:56.078 INFO:tasks.workunit.client.1.smithi196.stdout:0/358: rmdir d3/de/d14 39 2024-04-22T15:18:56.079 INFO:tasks.workunit.client.0.smithi096.stdout:7/920: write d31/d32/d36/f121 [712954,51880] 0 2024-04-22T15:18:56.079 INFO:tasks.workunit.client.1.smithi196.stdout:1/292: dwrite d1/db/d16/f5c [0,4194304] 0 2024-04-22T15:18:56.080 INFO:tasks.workunit.client.1.smithi196.stdout:1/293: write d1/db/ff [3039478,119150] 0 2024-04-22T15:18:56.084 INFO:tasks.workunit.client.0.smithi096.stdout:5/702: symlink d4/ddb/d3d/d4d/d99/le7 0 2024-04-22T15:18:56.088 INFO:tasks.workunit.client.0.smithi096.stdout:3/577: write da/db/dc/f16 [4305501,98471] 0 2024-04-22T15:18:56.089 INFO:tasks.workunit.client.1.smithi196.stdout:4/353: sync 2024-04-22T15:18:56.090 INFO:tasks.workunit.client.1.smithi196.stdout:4/354: write d6/d13/f16 [3126289,21491] 0 2024-04-22T15:18:56.091 INFO:tasks.workunit.client.1.smithi196.stdout:4/355: stat d6/d13/f1d 0 2024-04-22T15:18:56.092 INFO:tasks.workunit.client.1.smithi196.stdout:4/356: read d6/d14/f33 [912289,22842] 0 2024-04-22T15:18:56.092 INFO:tasks.workunit.client.0.smithi096.stdout:7/921: rename d31/d32/d36/d75/dfe/ld0 to d31/d32/d33/d44/da8/dbf/l13c 0 2024-04-22T15:18:56.092 INFO:tasks.workunit.client.1.smithi196.stdout:4/357: chown d6/d13/d15/l64 3007 1 2024-04-22T15:18:56.093 INFO:tasks.workunit.client.1.smithi196.stdout:7/294: creat d14/d26/d28/f58 x:0 0 0 2024-04-22T15:18:56.094 INFO:tasks.workunit.client.1.smithi196.stdout:4/358: read d6/d13/d15/f5e [61117,57096] 0 2024-04-22T15:18:56.095 INFO:tasks.workunit.client.1.smithi196.stdout:6/365: truncate dd/d1f/d4e/f54 196172 0 2024-04-22T15:18:56.096 INFO:tasks.workunit.client.1.smithi196.stdout:3/325: write d0/d15/d4d/db/dc/d18/f42 [318532,26438] 0 2024-04-22T15:18:56.096 INFO:tasks.workunit.client.1.smithi196.stdout:6/366: stat dd/d1f/d4e/d53 0 2024-04-22T15:18:56.097 INFO:tasks.workunit.client.1.smithi196.stdout:6/367: chown dd/d2b 13391278 1 2024-04-22T15:18:56.098 INFO:tasks.workunit.client.1.smithi196.stdout:3/326: dread - d0/d15/d31/f53 zero size 2024-04-22T15:18:56.099 INFO:tasks.workunit.client.0.smithi096.stdout:3/578: rename da/db/d12/d22/d68/c6d to da/db/de/daa/cba 0 2024-04-22T15:18:56.099 INFO:tasks.workunit.client.0.smithi096.stdout:3/579: dread - da/d3f/f9f zero size 2024-04-22T15:18:56.100 INFO:tasks.workunit.client.1.smithi196.stdout:4/359: dwrite d6/d14/d18/d40/f65 [0,4194304] 0 2024-04-22T15:18:56.103 INFO:tasks.workunit.client.1.smithi196.stdout:0/359: read d3/de/d14/f4a [3039270,18891] 0 2024-04-22T15:18:56.103 INFO:tasks.workunit.client.0.smithi096.stdout:0/812: creat d7/f112 x:0 0 0 2024-04-22T15:18:56.104 INFO:tasks.workunit.client.1.smithi196.stdout:0/360: dread - d3/de/d1c/d27/d2c/d51/f6a zero size 2024-04-22T15:18:56.107 INFO:tasks.workunit.client.1.smithi196.stdout:1/294: fdatasync d1/db/dc/d26/f30 0 2024-04-22T15:18:56.109 INFO:tasks.workunit.client.0.smithi096.stdout:4/711: link d5/d7/f36 d5/d7/d2c/d43/ff4 0 2024-04-22T15:18:56.111 INFO:tasks.workunit.client.1.smithi196.stdout:8/341: creat d2/d5/d2e/f67 x:0 0 0 2024-04-22T15:18:56.114 INFO:tasks.workunit.client.1.smithi196.stdout:9/329: mknod db/c78 0 2024-04-22T15:18:56.116 INFO:tasks.workunit.client.0.smithi096.stdout:0/813: creat d7/d42/f113 x:0 0 0 2024-04-22T15:18:56.116 INFO:tasks.workunit.client.1.smithi196.stdout:2/369: stat d1/d13/d25/c70 0 2024-04-22T15:18:56.118 INFO:tasks.workunit.client.0.smithi096.stdout:5/703: creat d4/ddb/d3d/d4d/d59/d5a/fe8 x:0 0 0 2024-04-22T15:18:56.119 INFO:tasks.workunit.client.0.smithi096.stdout:4/712: creat d5/d7/d1a/d8b/ff5 x:0 0 0 2024-04-22T15:18:56.121 INFO:tasks.workunit.client.0.smithi096.stdout:3/580: mknod da/db/d12/d22/d26/db2/cbb 0 2024-04-22T15:18:56.122 INFO:tasks.workunit.client.1.smithi196.stdout:7/295: creat d14/d15/f59 x:0 0 0 2024-04-22T15:18:56.123 INFO:tasks.workunit.client.0.smithi096.stdout:0/814: mkdir d7/d8f/d10e/d114 0 2024-04-22T15:18:56.124 INFO:tasks.workunit.client.1.smithi196.stdout:5/334: chown d1/d17/c41 0 1 2024-04-22T15:18:56.126 INFO:tasks.workunit.client.0.smithi096.stdout:7/922: getdents d31/d32/d33/d93/dd5 0 2024-04-22T15:18:56.128 INFO:tasks.workunit.client.0.smithi096.stdout:4/713: fdatasync d5/d7/d2c/d95/dee/f96 0 2024-04-22T15:18:56.129 INFO:tasks.workunit.client.1.smithi196.stdout:8/342: dread d2/d5/f48 [0,4194304] 0 2024-04-22T15:18:56.132 INFO:tasks.workunit.client.0.smithi096.stdout:3/581: mkdir da/db/d12/d22/d68/dbc 0 2024-04-22T15:18:56.134 INFO:tasks.workunit.client.1.smithi196.stdout:4/360: creat d6/d13/f68 x:0 0 0 2024-04-22T15:18:56.135 INFO:tasks.workunit.client.1.smithi196.stdout:3/327: read d0/d15/d4d/db/dc/d18/f4c [318988,101524] 0 2024-04-22T15:18:56.135 INFO:tasks.workunit.client.1.smithi196.stdout:4/361: chown d6/d14/d18/d40 8 1 2024-04-22T15:18:56.136 INFO:tasks.workunit.client.1.smithi196.stdout:4/362: fdatasync d6/d10/f50 0 2024-04-22T15:18:56.139 INFO:tasks.workunit.client.0.smithi096.stdout:5/704: mknod d4/ddb/d3d/d4d/d59/d5a/ce9 0 2024-04-22T15:18:56.140 INFO:tasks.workunit.client.0.smithi096.stdout:9/603: sync 2024-04-22T15:18:56.140 INFO:tasks.workunit.client.0.smithi096.stdout:7/923: mknod d31/d32/d36/d75/dff/c13d 0 2024-04-22T15:18:56.144 INFO:tasks.workunit.client.0.smithi096.stdout:4/714: unlink d5/d7/d2c/d3d/d4f/d9c/fc6 0 2024-04-22T15:18:56.144 INFO:tasks.workunit.client.0.smithi096.stdout:4/715: chown d5/d7/d1f/d84/d6d/fdb 24 1 2024-04-22T15:18:56.146 INFO:tasks.workunit.client.0.smithi096.stdout:4/716: write d5/d7/d2c/d67/fe3 [957048,46180] 0 2024-04-22T15:18:56.149 INFO:tasks.workunit.client.0.smithi096.stdout:9/604: rmdir d1/df/d95/d9d/db5/d43 39 2024-04-22T15:18:56.150 INFO:tasks.workunit.client.0.smithi096.stdout:9/605: stat d1/df 0 2024-04-22T15:18:56.151 INFO:tasks.workunit.client.0.smithi096.stdout:7/924: rmdir d31/d32/dc0/ddc 39 2024-04-22T15:18:56.153 INFO:tasks.workunit.client.1.smithi196.stdout:5/335: fdatasync d1/d17/d1a/f43 0 2024-04-22T15:18:56.154 INFO:tasks.workunit.client.1.smithi196.stdout:6/368: symlink dd/d20/d5c/d5e/l63 0 2024-04-22T15:18:56.154 INFO:tasks.workunit.client.1.smithi196.stdout:6/369: fdatasync dd/d1f/f60 0 2024-04-22T15:18:56.162 INFO:tasks.workunit.client.0.smithi096.stdout:7/925: mkdir d31/d32/d33/d93/dd5/d13e 0 2024-04-22T15:18:56.163 INFO:tasks.workunit.client.0.smithi096.stdout:0/815: dwrite d7/de/d5b/d64/fd3 [0,4194304] 0 2024-04-22T15:18:56.164 INFO:tasks.workunit.client.1.smithi196.stdout:4/363: mkdir d6/d14/d18/d40/d69 0 2024-04-22T15:18:56.167 INFO:tasks.workunit.client.1.smithi196.stdout:8/343: dread d2/dc/d45/f5b [0,4194304] 0 2024-04-22T15:18:56.174 INFO:tasks.workunit.client.0.smithi096.stdout:9/606: write d1/df/d95/d9d/db5/d43/f44 [953093,115857] 0 2024-04-22T15:18:56.174 INFO:tasks.workunit.client.1.smithi196.stdout:0/361: write d3/d9/f21 [2990813,55498] 0 2024-04-22T15:18:56.176 INFO:tasks.workunit.client.1.smithi196.stdout:0/362: chown d3/d9/d36/d56/d65/l66 18 1 2024-04-22T15:18:56.177 INFO:tasks.workunit.client.1.smithi196.stdout:3/328: dwrite d0/d15/d4d/db/dc/d18/f4c [0,4194304] 0 2024-04-22T15:18:56.178 INFO:tasks.workunit.client.1.smithi196.stdout:3/329: stat d0/d15/d31/d55 0 2024-04-22T15:18:56.178 INFO:tasks.workunit.client.0.smithi096.stdout:3/582: getdents da/db/de/d87 0 2024-04-22T15:18:56.179 INFO:tasks.workunit.client.1.smithi196.stdout:3/330: chown d0/d15/d4d/db/dc/d4a/d44/f6d 22959 1 2024-04-22T15:18:56.181 INFO:tasks.workunit.client.1.smithi196.stdout:0/363: dread d3/de/d1c/d27/d2c/d51/f75 [0,4194304] 0 2024-04-22T15:18:56.185 INFO:tasks.workunit.client.1.smithi196.stdout:1/295: truncate d1/f4 3974966 0 2024-04-22T15:18:56.186 INFO:tasks.workunit.client.0.smithi096.stdout:4/717: truncate d5/d7/d2c/d3d/d4f/d88/fe2 3563728 0 2024-04-22T15:18:56.187 INFO:tasks.workunit.client.1.smithi196.stdout:5/336: creat d1/d17/d1a/d2f/d30/f6b x:0 0 0 2024-04-22T15:18:56.187 INFO:tasks.workunit.client.0.smithi096.stdout:5/705: link d4/ddb/d3d/d35/caf d4/ddb/d3d/d4d/d59/db0/cea 0 2024-04-22T15:18:56.190 INFO:tasks.workunit.client.0.smithi096.stdout:7/926: write d31/d32/f6c [859024,2759] 0 2024-04-22T15:18:56.193 INFO:tasks.workunit.client.1.smithi196.stdout:9/330: dwrite db/dd/d1b/f45 [0,4194304] 0 2024-04-22T15:18:56.194 INFO:tasks.workunit.client.1.smithi196.stdout:4/364: unlink d6/d13/d15/d2c/d62/c66 0 2024-04-22T15:18:56.199 INFO:tasks.workunit.client.1.smithi196.stdout:9/331: dwrite db/dd/f1a [0,4194304] 0 2024-04-22T15:18:56.210 INFO:tasks.workunit.client.0.smithi096.stdout:3/583: dread da/db/d10/f6e [0,4194304] 0 2024-04-22T15:18:56.211 INFO:tasks.workunit.client.0.smithi096.stdout:3/584: write da/f8f [4766897,95729] 0 2024-04-22T15:18:56.214 INFO:tasks.workunit.client.0.smithi096.stdout:9/607: dwrite d1/d25/f33 [0,4194304] 0 2024-04-22T15:18:56.214 INFO:tasks.workunit.client.1.smithi196.stdout:0/364: creat d3/d9/d36/d56/f7b x:0 0 0 2024-04-22T15:18:56.215 INFO:tasks.workunit.client.1.smithi196.stdout:0/365: chown d3/de/d1c/d27/d2c/d51 47 1 2024-04-22T15:18:56.216 INFO:tasks.workunit.client.1.smithi196.stdout:0/366: fdatasync d3/d9/d36/f4c 0 2024-04-22T15:18:56.218 INFO:tasks.workunit.client.1.smithi196.stdout:5/337: truncate d1/d2/f28 4555910 0 2024-04-22T15:18:56.221 INFO:tasks.workunit.client.1.smithi196.stdout:8/344: dwrite d2/d8/d49/f59 [0,4194304] 0 2024-04-22T15:18:56.225 INFO:tasks.workunit.client.1.smithi196.stdout:2/370: link d1/l5e d1/d23/d27/d73/l86 0 2024-04-22T15:18:56.226 INFO:tasks.workunit.client.1.smithi196.stdout:7/296: getdents d14/d25 0 2024-04-22T15:18:56.228 INFO:tasks.workunit.client.1.smithi196.stdout:3/331: fdatasync d0/d15/d4d/db/dc/d4a/d44/f26 0 2024-04-22T15:18:56.229 INFO:tasks.workunit.client.0.smithi096.stdout:2/681: sync 2024-04-22T15:18:56.230 INFO:tasks.workunit.client.1.smithi196.stdout:3/332: dread - d0/d15/d4d/db/dc/d4a/f45 zero size 2024-04-22T15:18:56.232 INFO:tasks.workunit.client.1.smithi196.stdout:3/333: write d0/d15/d4d/f3e [3394889,65885] 0 2024-04-22T15:18:56.233 INFO:tasks.workunit.client.1.smithi196.stdout:7/297: dwrite d14/d26/f30 [0,4194304] 0 2024-04-22T15:18:56.257 INFO:tasks.workunit.client.1.smithi196.stdout:8/345: dread d2/d8/f25 [0,4194304] 0 2024-04-22T15:18:56.257 INFO:tasks.workunit.client.1.smithi196.stdout:8/346: readlink d2/l30 0 2024-04-22T15:18:56.260 INFO:tasks.workunit.client.1.smithi196.stdout:9/332: getdents db/dd/de/d75 0 2024-04-22T15:18:56.261 INFO:tasks.workunit.client.1.smithi196.stdout:9/333: chown db/dd/de/d15/f40 25025550 1 2024-04-22T15:18:56.263 INFO:tasks.workunit.client.1.smithi196.stdout:4/365: dwrite d6/f19 [0,4194304] 0 2024-04-22T15:18:56.267 INFO:tasks.workunit.client.1.smithi196.stdout:1/296: truncate d1/db/dc/de/f14 2808543 0 2024-04-22T15:18:56.267 INFO:tasks.workunit.client.1.smithi196.stdout:1/297: readlink d1/db/d16/l20 0 2024-04-22T15:18:56.271 INFO:tasks.workunit.client.1.smithi196.stdout:3/334: mknod d0/d15/d31/d55/c72 0 2024-04-22T15:18:56.275 INFO:tasks.workunit.client.1.smithi196.stdout:0/367: creat d3/de/d14/d72/f7c x:0 0 0 2024-04-22T15:18:56.277 INFO:tasks.workunit.client.1.smithi196.stdout:3/335: dwrite d0/d15/d4d/f2d [0,4194304] 0 2024-04-22T15:18:56.279 INFO:tasks.workunit.client.1.smithi196.stdout:6/370: getdents dd/d1f/d4e/d53/d55 0 2024-04-22T15:18:56.279 INFO:tasks.workunit.client.0.smithi096.stdout:4/718: creat d5/d5b/ff6 x:0 0 0 2024-04-22T15:18:56.280 INFO:tasks.workunit.client.1.smithi196.stdout:2/371: write d1/d23/d27/d2e/d55/f36 [1785305,129702] 0 2024-04-22T15:18:56.281 INFO:tasks.workunit.client.1.smithi196.stdout:8/347: mkdir d2/d8/d47/d68 0 2024-04-22T15:18:56.281 INFO:tasks.workunit.client.1.smithi196.stdout:2/372: chown d1/d1d/f7a 580698 1 2024-04-22T15:18:56.286 INFO:tasks.workunit.client.1.smithi196.stdout:4/366: dread d6/d13/f16 [0,4194304] 0 2024-04-22T15:18:56.289 INFO:tasks.workunit.client.0.smithi096.stdout:3/585: mkdir da/dab/dbd 0 2024-04-22T15:18:56.290 INFO:tasks.workunit.client.1.smithi196.stdout:5/338: sync 2024-04-22T15:18:56.291 INFO:tasks.workunit.client.0.smithi096.stdout:9/608: truncate d1/df/d95/d9d/db5/dd/f2b 2144728 0 2024-04-22T15:18:56.292 INFO:tasks.workunit.client.1.smithi196.stdout:1/298: creat d1/db/d16/d1e/f60 x:0 0 0 2024-04-22T15:18:56.293 INFO:tasks.workunit.client.1.smithi196.stdout:1/299: readlink d1/db/l2a 0 2024-04-22T15:18:56.293 INFO:tasks.workunit.client.1.smithi196.stdout:1/300: dread - d1/d54/f55 zero size 2024-04-22T15:18:56.295 INFO:tasks.workunit.client.0.smithi096.stdout:0/816: write d7/de/d5b/d95/dc/d16/f3b [3764492,106405] 0 2024-04-22T15:18:56.295 INFO:tasks.workunit.client.0.smithi096.stdout:2/682: symlink d3/db/df/dce/ld6 0 2024-04-22T15:18:56.296 INFO:tasks.workunit.client.1.smithi196.stdout:7/298: mkdir d14/d26/d5a 0 2024-04-22T15:18:56.299 INFO:tasks.workunit.client.0.smithi096.stdout:4/719: rmdir d5/d7/d2c/d95/dee 39 2024-04-22T15:18:56.299 INFO:tasks.workunit.client.1.smithi196.stdout:9/334: write db/dd/de/d15/f1e [1794551,52553] 0 2024-04-22T15:18:56.300 INFO:tasks.workunit.client.0.smithi096.stdout:4/720: write d5/d7/f9 [1681403,55364] 0 2024-04-22T15:18:56.300 INFO:tasks.workunit.client.1.smithi196.stdout:9/335: chown db/dd/de/d15/f73 201660 1 2024-04-22T15:18:56.303 INFO:tasks.workunit.client.0.smithi096.stdout:7/927: creat d31/d32/d33/d6a/d10a/f13f x:0 0 0 2024-04-22T15:18:56.305 INFO:tasks.workunit.client.1.smithi196.stdout:8/348: write d2/dc/f10 [997010,7983] 0 2024-04-22T15:18:56.305 INFO:tasks.workunit.client.0.smithi096.stdout:3/586: mknod da/dab/cbe 0 2024-04-22T15:18:56.308 INFO:tasks.workunit.client.0.smithi096.stdout:9/609: rmdir d1/d4/d79 39 2024-04-22T15:18:56.310 INFO:tasks.workunit.client.0.smithi096.stdout:0/817: symlink d7/de/d3a/d61/d79/l115 0 2024-04-22T15:18:56.312 INFO:tasks.workunit.client.0.smithi096.stdout:2/683: truncate d3/db/df/d24/d58/d5f/d6e/f9b 740738 0 2024-04-22T15:18:56.313 INFO:tasks.workunit.client.0.smithi096.stdout:2/684: dread - d3/db/df/d98/db3/fc7 zero size 2024-04-22T15:18:56.313 INFO:tasks.workunit.client.1.smithi196.stdout:0/368: write d3/d6/f48 [201382,52955] 0 2024-04-22T15:18:56.314 INFO:tasks.workunit.client.1.smithi196.stdout:3/336: write d0/d15/d4d/db/dc/d4a/d44/f20 [1408451,53723] 0 2024-04-22T15:18:56.316 INFO:tasks.workunit.client.1.smithi196.stdout:5/339: symlink d1/d9/l6c 0 2024-04-22T15:18:56.319 INFO:tasks.workunit.client.1.smithi196.stdout:0/369: dread d3/de/d1c/d27/d2c/d51/f75 [0,4194304] 0 2024-04-22T15:18:56.320 INFO:tasks.workunit.client.0.smithi096.stdout:5/706: rename d4/d1f/f21 to d4/d2c/feb 0 2024-04-22T15:18:56.321 INFO:tasks.workunit.client.1.smithi196.stdout:0/370: dread - d3/de/d1c/d27/d2c/d51/f6a zero size 2024-04-22T15:18:56.321 INFO:tasks.workunit.client.1.smithi196.stdout:6/371: dwrite fa [0,4194304] 0 2024-04-22T15:18:56.322 INFO:tasks.workunit.client.1.smithi196.stdout:0/371: dread - d3/d9/d36/d56/f7b zero size 2024-04-22T15:18:56.323 INFO:tasks.workunit.client.0.smithi096.stdout:7/928: creat d31/d32/d33/d93/dd5/f140 x:0 0 0 2024-04-22T15:18:56.327 INFO:tasks.workunit.client.0.smithi096.stdout:3/587: rmdir da/db/de/d19/d2d 39 2024-04-22T15:18:56.328 INFO:tasks.workunit.client.0.smithi096.stdout:2/685: dread d3/f53 [0,4194304] 0 2024-04-22T15:18:56.329 INFO:tasks.workunit.client.0.smithi096.stdout:5/707: dread d4/ddb/d3d/d4d/d59/d5a/f79 [0,4194304] 0 2024-04-22T15:18:56.335 INFO:tasks.workunit.client.0.smithi096.stdout:7/929: dwrite d31/d32/d36/d75/dff/f122 [0,4194304] 0 2024-04-22T15:18:56.336 INFO:tasks.workunit.client.1.smithi196.stdout:1/301: dwrite d1/db/dc/d26/d2e/f3e [0,4194304] 0 2024-04-22T15:18:56.337 INFO:tasks.workunit.client.1.smithi196.stdout:1/302: chown d1/db/d16/f40 198473 1 2024-04-22T15:18:56.337 INFO:tasks.workunit.client.0.smithi096.stdout:0/818: rename d7/f9e to d7/de/d5b/d95/d59/f116 0 2024-04-22T15:18:56.338 INFO:tasks.workunit.client.1.smithi196.stdout:1/303: chown d1/db/dc/d26/f2b 6732950 1 2024-04-22T15:18:56.339 INFO:tasks.workunit.client.0.smithi096.stdout:0/819: stat d7/de/d5b/d95/dc/fd0 0 2024-04-22T15:18:56.340 INFO:tasks.workunit.client.1.smithi196.stdout:8/349: mknod d2/d35/c69 0 2024-04-22T15:18:56.343 INFO:tasks.workunit.client.0.smithi096.stdout:0/820: read d7/de/d5b/d62/ff6 [4586529,42747] 0 2024-04-22T15:18:56.345 INFO:tasks.workunit.client.1.smithi196.stdout:2/373: link d1/d13/d25/f4f d1/d23/d27/d2e/d55/f87 0 2024-04-22T15:18:56.347 INFO:tasks.workunit.client.1.smithi196.stdout:4/367: link d6/d14/d18/d40/f65 d6/d14/d18/d26/f6a 0 2024-04-22T15:18:56.348 INFO:tasks.workunit.client.0.smithi096.stdout:9/610: dwrite d1/df/d95/d9d/db5/d43/d57/f63 [0,4194304] 0 2024-04-22T15:18:56.349 INFO:tasks.workunit.client.1.smithi196.stdout:3/337: symlink d0/d15/d4d/db/dc/l73 0 2024-04-22T15:18:56.352 INFO:tasks.workunit.client.1.smithi196.stdout:5/340: readlink d1/l7 0 2024-04-22T15:18:56.353 INFO:tasks.workunit.client.0.smithi096.stdout:5/708: fdatasync d4/ddb/d3d/d4f/fc7 0 2024-04-22T15:18:56.354 INFO:tasks.workunit.client.0.smithi096.stdout:5/709: stat d4/ddb/d3d/d4d/d59/d98/c6f 0 2024-04-22T15:18:56.355 INFO:tasks.workunit.client.1.smithi196.stdout:7/299: rename d14/d25/d31/f3f to d14/d4a/f5b 0 2024-04-22T15:18:56.356 INFO:tasks.workunit.client.1.smithi196.stdout:6/372: creat dd/d20/d5c/d5e/f64 x:0 0 0 2024-04-22T15:18:56.357 INFO:tasks.workunit.client.0.smithi096.stdout:4/721: creat d5/d7/d2c/d3d/ff7 x:0 0 0 2024-04-22T15:18:56.358 INFO:tasks.workunit.client.1.smithi196.stdout:1/304: mkdir d1/db/dc/de/d43/d61 0 2024-04-22T15:18:56.361 INFO:tasks.workunit.client.1.smithi196.stdout:3/338: creat d0/d15/d31/d55/f74 x:0 0 0 2024-04-22T15:18:56.363 INFO:tasks.workunit.client.1.smithi196.stdout:7/300: stat d14/d15/f1a 0 2024-04-22T15:18:56.365 INFO:tasks.workunit.client.1.smithi196.stdout:2/374: rename d1/d13/d1f/f24 to d1/d23/d27/d2e/f88 0 2024-04-22T15:18:56.367 INFO:tasks.workunit.client.1.smithi196.stdout:3/339: dwrite d0/d15/d4d/db/dc/d18/f42 [0,4194304] 0 2024-04-22T15:18:56.368 INFO:tasks.workunit.client.1.smithi196.stdout:3/340: write d0/d7/f1a [443742,35980] 0 2024-04-22T15:18:56.369 INFO:tasks.workunit.client.0.smithi096.stdout:5/710: creat d4/ddb/d3d/d35/fec x:0 0 0 2024-04-22T15:18:56.376 INFO:tasks.workunit.client.1.smithi196.stdout:1/305: write d1/db/dc/d26/d2e/f51 [1039419,117686] 0 2024-04-22T15:18:56.378 INFO:tasks.workunit.client.1.smithi196.stdout:8/350: mkdir d2/d6a 0 2024-04-22T15:18:56.378 INFO:tasks.workunit.client.1.smithi196.stdout:8/351: dread - d2/d8/f4c zero size 2024-04-22T15:18:56.379 INFO:tasks.workunit.client.1.smithi196.stdout:8/352: chown d2/d35/c69 377199 1 2024-04-22T15:18:56.382 INFO:tasks.workunit.client.0.smithi096.stdout:7/930: write f23 [1707619,114842] 0 2024-04-22T15:18:56.384 INFO:tasks.workunit.client.1.smithi196.stdout:7/301: chown d14/d26/d2a/d2b/c50 154276 1 2024-04-22T15:18:56.386 INFO:tasks.workunit.client.0.smithi096.stdout:5/711: fdatasync d4/ddb/d3d/d35/fbf 0 2024-04-22T15:18:56.388 INFO:tasks.workunit.client.1.smithi196.stdout:4/368: rename d6/d14/f5b to d6/d13/f6b 0 2024-04-22T15:18:56.389 INFO:tasks.workunit.client.0.smithi096.stdout:4/722: creat d5/d7/d2c/d95/dee/ff8 x:0 0 0 2024-04-22T15:18:56.389 INFO:tasks.workunit.client.0.smithi096.stdout:0/821: dwrite d7/de/d5b/d95/dc/d16/f1f [0,4194304] 0 2024-04-22T15:18:56.391 INFO:tasks.workunit.client.0.smithi096.stdout:3/588: link da/l75 da/db/d12/d22/d68/lbf 0 2024-04-22T15:18:56.393 INFO:tasks.workunit.client.1.smithi196.stdout:0/372: creat d3/de/f7d x:0 0 0 2024-04-22T15:18:56.394 INFO:tasks.workunit.client.0.smithi096.stdout:2/686: write d3/db/df/f60 [542218,86555] 0 2024-04-22T15:18:56.396 INFO:tasks.workunit.client.0.smithi096.stdout:3/589: readlink da/l75 0 2024-04-22T15:18:56.397 INFO:tasks.workunit.client.1.smithi196.stdout:3/341: mknod d0/d15/d4d/c75 0 2024-04-22T15:18:56.397 INFO:tasks.workunit.client.0.smithi096.stdout:9/611: dwrite d1/df/d95/d9d/db5/dd/f21 [0,4194304] 0 2024-04-22T15:18:56.397 INFO:tasks.workunit.client.0.smithi096.stdout:3/590: stat da/db/dc/d59 0 2024-04-22T15:18:56.399 INFO:tasks.workunit.client.0.smithi096.stdout:9/612: write d1/d85/fab [257527,86452] 0 2024-04-22T15:18:56.403 INFO:tasks.workunit.client.1.smithi196.stdout:6/373: creat dd/d20/d4d/f65 x:0 0 0 2024-04-22T15:18:56.405 INFO:tasks.workunit.client.1.smithi196.stdout:9/336: getdents db 0 2024-04-22T15:18:56.406 INFO:tasks.workunit.client.1.smithi196.stdout:1/306: mknod d1/db/dc/d26/d2e/d32/d36/c62 0 2024-04-22T15:18:56.408 INFO:tasks.workunit.client.1.smithi196.stdout:8/353: creat d2/d5/d4f/f6b x:0 0 0 2024-04-22T15:18:56.411 INFO:tasks.workunit.client.0.smithi096.stdout:2/687: mkdir d3/db/df/dce/d46/d69/dd7 0 2024-04-22T15:18:56.411 INFO:tasks.workunit.client.1.smithi196.stdout:1/307: dread d1/db/dc/d26/d2e/f51 [0,4194304] 0 2024-04-22T15:18:56.412 INFO:tasks.workunit.client.1.smithi196.stdout:1/308: read d1/db/d16/f4b [2717422,70930] 0 2024-04-22T15:18:56.412 INFO:tasks.workunit.client.1.smithi196.stdout:7/302: symlink d14/d4d/l5c 0 2024-04-22T15:18:56.413 INFO:tasks.workunit.client.1.smithi196.stdout:7/303: write fc [153116,46067] 0 2024-04-22T15:18:56.413 INFO:tasks.workunit.client.1.smithi196.stdout:1/309: fdatasync d1/db/dc/d26/d2e/f51 0 2024-04-22T15:18:56.415 INFO:tasks.workunit.client.0.smithi096.stdout:3/591: mkdir da/db/dc/d28/d69/d89/dc0 0 2024-04-22T15:18:56.418 INFO:tasks.workunit.client.1.smithi196.stdout:5/341: rename d1/f1f to d1/d17/d50/f6d 0 2024-04-22T15:18:56.419 INFO:tasks.workunit.client.1.smithi196.stdout:4/369: creat d6/d14/d18/d40/f6c x:0 0 0 2024-04-22T15:18:56.420 INFO:tasks.workunit.client.1.smithi196.stdout:4/370: fsync d6/f42 0 2024-04-22T15:18:56.420 INFO:tasks.workunit.client.0.smithi096.stdout:8/667: sync 2024-04-22T15:18:56.421 INFO:tasks.workunit.client.0.smithi096.stdout:4/723: write d5/d7/d1f/d84/d6d/d98/f9b [215540,83272] 0 2024-04-22T15:18:56.421 INFO:tasks.workunit.client.1.smithi196.stdout:2/375: mkdir d1/d23/d27/d89 0 2024-04-22T15:18:56.424 INFO:tasks.workunit.client.0.smithi096.stdout:0/822: dwrite d7/de/d5b/d95/dc/f29 [0,4194304] 0 2024-04-22T15:18:56.426 INFO:tasks.workunit.client.1.smithi196.stdout:0/373: creat d3/d9/d36/f7e x:0 0 0 2024-04-22T15:18:56.426 INFO:tasks.workunit.client.0.smithi096.stdout:7/931: creat d31/d32/d33/d44/da8/f141 x:0 0 0 2024-04-22T15:18:56.431 INFO:tasks.workunit.client.0.smithi096.stdout:4/724: dwrite d5/d7/d2c/f86 [0,4194304] 0 2024-04-22T15:18:56.435 INFO:tasks.workunit.client.1.smithi196.stdout:9/337: creat db/dd/d1b/f79 x:0 0 0 2024-04-22T15:18:56.436 INFO:tasks.workunit.client.1.smithi196.stdout:9/338: fsync db/dd/de/f10 0 2024-04-22T15:18:56.438 INFO:tasks.workunit.client.1.smithi196.stdout:9/339: write db/dd/de/d15/f73 [267382,17510] 0 2024-04-22T15:18:56.438 INFO:tasks.workunit.client.1.smithi196.stdout:8/354: mkdir d2/d8/d49/d6c 0 2024-04-22T15:18:56.440 INFO:tasks.workunit.client.0.smithi096.stdout:2/688: symlink d3/db/df/dce/d46/ld8 0 2024-04-22T15:18:56.440 INFO:tasks.workunit.client.1.smithi196.stdout:8/355: dread - d2/d5/d2e/f67 zero size 2024-04-22T15:18:56.440 INFO:tasks.workunit.client.1.smithi196.stdout:7/304: rmdir d14/d4a 39 2024-04-22T15:18:56.441 INFO:tasks.workunit.client.1.smithi196.stdout:9/340: write db/dd/de/d15/f1e [6564734,35372] 0 2024-04-22T15:18:56.441 INFO:tasks.workunit.client.1.smithi196.stdout:8/356: readlink d2/l30 0 2024-04-22T15:18:56.442 INFO:tasks.workunit.client.1.smithi196.stdout:9/341: read - db/f61 zero size 2024-04-22T15:18:56.442 INFO:tasks.workunit.client.1.smithi196.stdout:8/357: dread - d2/d8/f4c zero size 2024-04-22T15:18:56.443 INFO:tasks.workunit.client.1.smithi196.stdout:1/310: mknod d1/db/d16/d1e/d49/c63 0 2024-04-22T15:18:56.444 INFO:tasks.workunit.client.0.smithi096.stdout:3/592: creat da/db/d9a/fc1 x:0 0 0 2024-04-22T15:18:56.444 INFO:tasks.workunit.client.0.smithi096.stdout:2/689: dread d3/f8e [0,4194304] 0 2024-04-22T15:18:56.447 INFO:tasks.workunit.client.0.smithi096.stdout:7/932: dread d31/d32/d33/d54/d43/f48 [0,4194304] 0 2024-04-22T15:18:56.447 INFO:tasks.workunit.client.1.smithi196.stdout:5/342: mkdir d1/d17/d1a/d2f/d30/d6e 0 2024-04-22T15:18:56.447 INFO:tasks.workunit.client.0.smithi096.stdout:7/933: stat d31/d32/d33/d44/da8/dbf/fe1 0 2024-04-22T15:18:56.450 INFO:tasks.workunit.client.1.smithi196.stdout:4/371: creat d6/d13/d15/d2c/d62/f6d x:0 0 0 2024-04-22T15:18:56.454 INFO:tasks.workunit.client.1.smithi196.stdout:0/374: unlink d3/de/d14/d29/f57 0 2024-04-22T15:18:56.455 INFO:tasks.workunit.client.0.smithi096.stdout:3/593: creat da/db/d12/d22/d26/db2/fc2 x:0 0 0 2024-04-22T15:18:56.457 INFO:tasks.workunit.client.0.smithi096.stdout:2/690: creat d3/db/df/d10/db5/fd9 x:0 0 0 2024-04-22T15:18:56.484 INFO:tasks.workunit.client.0.smithi096.stdout:0/823: creat d7/d42/d4c/d55/f117 x:0 0 0 2024-04-22T15:18:56.485 INFO:tasks.workunit.client.0.smithi096.stdout:7/934: creat d31/d32/d33/dfa/f142 x:0 0 0 2024-04-22T15:18:56.486 INFO:tasks.workunit.client.1.smithi196.stdout:7/305: mknod d14/d26/d28/d29/c5d 0 2024-04-22T15:18:56.487 INFO:tasks.workunit.client.1.smithi196.stdout:7/306: fdatasync d14/d15/f18 0 2024-04-22T15:18:56.489 INFO:tasks.workunit.client.0.smithi096.stdout:3/594: fdatasync da/db/de/d19/d2d/fa8 0 2024-04-22T15:18:56.489 INFO:tasks.workunit.client.1.smithi196.stdout:1/311: fdatasync d1/db/dc/d26/d2e/d32/d36/f3f 0 2024-04-22T15:18:56.490 INFO:tasks.workunit.client.0.smithi096.stdout:8/668: getdents d4/d2a/db7 0 2024-04-22T15:18:56.491 INFO:tasks.workunit.client.1.smithi196.stdout:5/343: write d1/d17/d1c/f3b [53928,110836] 0 2024-04-22T15:18:56.492 INFO:tasks.workunit.client.0.smithi096.stdout:8/669: write d4/d10/d25/f81 [3724139,122133] 0 2024-04-22T15:18:56.493 INFO:tasks.workunit.client.0.smithi096.stdout:2/691: mkdir d3/d13/db4/dbf/dda 0 2024-04-22T15:18:56.493 INFO:tasks.workunit.client.0.smithi096.stdout:9/613: dread d1/df/d95/d9d/db5/d43/d57/f8d [0,4194304] 0 2024-04-22T15:18:56.493 INFO:tasks.workunit.client.1.smithi196.stdout:4/372: truncate d6/d13/f16 2901265 0 2024-04-22T15:18:56.494 INFO:tasks.workunit.client.1.smithi196.stdout:4/373: write d6/d13/d15/d2c/d35/d44/f4a [521045,117424] 0 2024-04-22T15:18:56.496 INFO:tasks.workunit.client.0.smithi096.stdout:0/824: read d7/d8f/fca [588276,124613] 0 2024-04-22T15:18:56.496 INFO:tasks.workunit.client.0.smithi096.stdout:8/670: write d4/d1c/d40/fdc [387781,46279] 0 2024-04-22T15:18:56.497 INFO:tasks.workunit.client.1.smithi196.stdout:2/376: rename d1/l5e to d1/d1d/l8a 0 2024-04-22T15:18:56.498 INFO:tasks.workunit.client.1.smithi196.stdout:2/377: read d1/f80 [8256388,112876] 0 2024-04-22T15:18:56.499 INFO:tasks.workunit.client.0.smithi096.stdout:3/595: dwrite da/db/de/d19/d3e/f4f [0,4194304] 0 2024-04-22T15:18:56.500 INFO:tasks.workunit.client.1.smithi196.stdout:6/374: creat dd/d1f/d37/f66 x:0 0 0 2024-04-22T15:18:56.505 INFO:tasks.workunit.client.0.smithi096.stdout:7/935: link d31/d32/d33/fca d31/d32/d33/d93/dd5/f143 0 2024-04-22T15:18:56.506 INFO:tasks.workunit.client.0.smithi096.stdout:7/936: readlink d31/d32/d36/l7b 0 2024-04-22T15:18:56.513 INFO:tasks.workunit.client.0.smithi096.stdout:0/825: read d7/de/d5b/d95/dc/d16/d18/f30 [2825272,19723] 0 2024-04-22T15:18:56.518 INFO:tasks.workunit.client.1.smithi196.stdout:5/344: write d1/d17/f60 [358737,71812] 0 2024-04-22T15:18:56.518 INFO:tasks.workunit.client.1.smithi196.stdout:5/345: chown d1/d17/d50/l55 19007118 1 2024-04-22T15:18:56.518 INFO:tasks.workunit.client.0.smithi096.stdout:8/671: dread d4/d1c/d40/da7/db6/f49 [0,4194304] 0 2024-04-22T15:18:56.521 INFO:tasks.workunit.client.1.smithi196.stdout:0/375: rename d3/de/d1c/d27/d2c to d3/d9/d36/d56/d65/d7f 0 2024-04-22T15:18:56.523 INFO:tasks.workunit.client.0.smithi096.stdout:2/692: rename d3/db/df/dce/d46/l91 to d3/db/df/dce/d46/d7e/d84/ldb 0 2024-04-22T15:18:56.525 INFO:tasks.workunit.client.0.smithi096.stdout:9/614: mknod d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dbd/ccb 0 2024-04-22T15:18:56.527 INFO:tasks.workunit.client.0.smithi096.stdout:4/725: write d5/d7/d1f/d84/d6d/fbb [405756,25546] 0 2024-04-22T15:18:56.527 INFO:tasks.workunit.client.1.smithi196.stdout:7/307: mkdir d14/d26/d5e 0 2024-04-22T15:18:56.528 INFO:tasks.workunit.client.1.smithi196.stdout:7/308: dread d14/d25/f3b [0,4194304] 0 2024-04-22T15:18:56.529 INFO:tasks.workunit.client.1.smithi196.stdout:8/358: link d2/dc/f2f d2/f6d 0 2024-04-22T15:18:56.530 INFO:tasks.workunit.client.0.smithi096.stdout:3/596: mknod da/db/dc/d28/d69/d89/dc0/cc3 0 2024-04-22T15:18:56.530 INFO:tasks.workunit.client.1.smithi196.stdout:8/359: write d2/d5/d2e/f67 [494823,51729] 0 2024-04-22T15:18:56.530 INFO:tasks.workunit.client.0.smithi096.stdout:3/597: chown da/db/dc/l4a 3 1 2024-04-22T15:18:56.532 INFO:tasks.workunit.client.0.smithi096.stdout:8/672: truncate d4/d2a/f3c 106103 0 2024-04-22T15:18:56.535 INFO:tasks.workunit.client.0.smithi096.stdout:8/673: truncate d4/d10/d25/d38/d8f/fd7 664938 0 2024-04-22T15:18:56.537 INFO:tasks.workunit.client.0.smithi096.stdout:3/598: dwrite da/dab/fa2 [0,4194304] 0 2024-04-22T15:18:56.538 INFO:tasks.workunit.client.1.smithi196.stdout:3/342: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:18:56.539 INFO:tasks.workunit.client.1.smithi196.stdout:6/375: creat dd/d1f/d37/d3c/f67 x:0 0 0 2024-04-22T15:18:56.542 INFO:tasks.workunit.client.0.smithi096.stdout:0/826: creat d7/d42/d4c/dd2/f118 x:0 0 0 2024-04-22T15:18:56.543 INFO:tasks.workunit.client.0.smithi096.stdout:3/599: dread da/f55 [0,4194304] 0 2024-04-22T15:18:56.544 INFO:tasks.workunit.client.0.smithi096.stdout:0/827: dread - d7/de/d5b/d95/dc/f96 zero size 2024-04-22T15:18:56.546 INFO:tasks.workunit.client.1.smithi196.stdout:8/360: write d2/dc/f2f [572422,56615] 0 2024-04-22T15:18:56.547 INFO:tasks.workunit.client.1.smithi196.stdout:8/361: chown d2/d3f/d4b/f4d 9 1 2024-04-22T15:18:56.551 INFO:tasks.workunit.client.1.smithi196.stdout:5/346: mknod d1/d17/d1a/d2f/d30/d6e/c6f 0 2024-04-22T15:18:56.551 INFO:tasks.workunit.client.1.smithi196.stdout:5/347: chown d1/d17/d50/f6d 393 1 2024-04-22T15:18:56.555 INFO:tasks.workunit.client.0.smithi096.stdout:8/674: read d4/f12 [1460032,54031] 0 2024-04-22T15:18:56.556 INFO:tasks.workunit.client.0.smithi096.stdout:9/615: creat d1/d4/d79/fcc x:0 0 0 2024-04-22T15:18:56.557 INFO:tasks.workunit.client.0.smithi096.stdout:9/616: chown d1/d85/fc8 257 1 2024-04-22T15:18:56.559 INFO:tasks.workunit.client.1.smithi196.stdout:6/376: truncate dd/d1f/d38/d40/f49 980516 0 2024-04-22T15:18:56.560 INFO:tasks.workunit.client.0.smithi096.stdout:4/726: mkdir d5/d7/d2c/d95/df9 0 2024-04-22T15:18:56.560 INFO:tasks.workunit.client.1.smithi196.stdout:6/377: fdatasync f9 0 2024-04-22T15:18:56.560 INFO:tasks.workunit.client.1.smithi196.stdout:4/374: write d6/d13/f1b [1489794,52282] 0 2024-04-22T15:18:56.561 INFO:tasks.workunit.client.1.smithi196.stdout:7/309: mkdir d14/d26/d5a/d5f 0 2024-04-22T15:18:56.562 INFO:tasks.workunit.client.0.smithi096.stdout:8/675: dwrite d4/d10/d25/d38/f4b [4194304,4194304] 0 2024-04-22T15:18:56.562 INFO:tasks.workunit.client.1.smithi196.stdout:7/310: write f7 [287926,60142] 0 2024-04-22T15:18:56.563 INFO:tasks.workunit.client.1.smithi196.stdout:7/311: fsync d14/f3a 0 2024-04-22T15:18:56.566 INFO:tasks.workunit.client.0.smithi096.stdout:0/828: creat d7/de/d5b/d95/d66/f119 x:0 0 0 2024-04-22T15:18:56.566 INFO:tasks.workunit.client.1.smithi196.stdout:7/312: read d14/d15/f4b [485837,113067] 0 2024-04-22T15:18:56.568 INFO:tasks.workunit.client.1.smithi196.stdout:6/378: dwrite dd/d1f/d38/f4b [0,4194304] 0 2024-04-22T15:18:56.568 INFO:tasks.workunit.client.0.smithi096.stdout:7/937: dwrite fe [4194304,4194304] 0 2024-04-22T15:18:56.568 INFO:tasks.workunit.client.0.smithi096.stdout:2/693: creat d3/db/df/dce/d46/d7e/fdc x:0 0 0 2024-04-22T15:18:56.571 INFO:tasks.workunit.client.1.smithi196.stdout:1/312: rename d1/db/d16/d1e/c3d to d1/db/c64 0 2024-04-22T15:18:56.578 INFO:tasks.workunit.client.1.smithi196.stdout:4/375: dread d6/d14/f12 [0,4194304] 0 2024-04-22T15:18:56.580 INFO:tasks.workunit.client.1.smithi196.stdout:9/342: sync 2024-04-22T15:18:56.580 INFO:tasks.workunit.client.0.smithi096.stdout:9/617: rename d1/f75 to d1/d25/d9f/fcd 0 2024-04-22T15:18:56.580 INFO:tasks.workunit.client.1.smithi196.stdout:4/376: write d6/d13/d15/d2c/d35/d44/f4a [51326,68989] 0 2024-04-22T15:18:56.581 INFO:tasks.workunit.client.0.smithi096.stdout:9/618: chown d1/df/d95/d9d/db5/dd/f1a 2 1 2024-04-22T15:18:56.584 INFO:tasks.workunit.client.1.smithi196.stdout:0/376: dwrite d3/de/d1c/f31 [0,4194304] 0 2024-04-22T15:18:56.586 INFO:tasks.workunit.client.1.smithi196.stdout:9/343: read db/dd/de/d15/f40 [7137162,8494] 0 2024-04-22T15:18:56.586 INFO:tasks.workunit.client.1.smithi196.stdout:5/348: creat d1/d9/d68/f70 x:0 0 0 2024-04-22T15:18:56.587 INFO:tasks.workunit.client.1.smithi196.stdout:7/313: mkdir d14/d26/d5a/d60 0 2024-04-22T15:18:56.588 INFO:tasks.workunit.client.1.smithi196.stdout:5/349: dread - d1/d17/d1a/d2f/d30/f54 zero size 2024-04-22T15:18:56.589 INFO:tasks.workunit.client.1.smithi196.stdout:7/314: dread d14/d25/f3b [0,4194304] 0 2024-04-22T15:18:56.589 INFO:tasks.workunit.client.1.smithi196.stdout:7/315: truncate f4 9329751 0 2024-04-22T15:18:56.590 INFO:tasks.workunit.client.1.smithi196.stdout:2/378: rename d1/d13/d25/c31 to d1/d23/c8b 0 2024-04-22T15:18:56.593 INFO:tasks.workunit.client.1.smithi196.stdout:1/313: mkdir d1/db/d16/d1e/d65 0 2024-04-22T15:18:56.599 INFO:tasks.workunit.client.0.smithi096.stdout:3/600: write da/db/d12/d22/f7d [242016,45429] 0 2024-04-22T15:18:56.600 INFO:tasks.workunit.client.0.smithi096.stdout:3/601: stat da/d3f/f5b 0 2024-04-22T15:18:56.600 INFO:tasks.workunit.client.1.smithi196.stdout:4/377: creat d6/d14/d18/d26/f6e x:0 0 0 2024-04-22T15:18:56.601 INFO:tasks.workunit.client.1.smithi196.stdout:4/378: readlink d6/d13/d15/d2c/d35/d44/l5f 0 2024-04-22T15:18:56.602 INFO:tasks.workunit.client.0.smithi096.stdout:9/619: unlink d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dbd/ccb 0 2024-04-22T15:18:56.606 INFO:tasks.workunit.client.0.smithi096.stdout:3/602: dwrite da/db/dc/f5d [0,4194304] 0 2024-04-22T15:18:56.608 INFO:tasks.workunit.client.1.smithi196.stdout:0/377: symlink d3/de/d14/d5a/l80 0 2024-04-22T15:18:56.609 INFO:tasks.workunit.client.0.smithi096.stdout:2/694: mknod d3/d13/dca/cdd 0 2024-04-22T15:18:56.611 INFO:tasks.workunit.client.1.smithi196.stdout:9/344: dread db/dd/d1b/f51 [0,4194304] 0 2024-04-22T15:18:56.612 INFO:tasks.workunit.client.1.smithi196.stdout:5/350: unlink d1/ld 0 2024-04-22T15:18:56.613 INFO:tasks.workunit.client.0.smithi096.stdout:3/603: dwrite da/d3f/fa5 [0,4194304] 0 2024-04-22T15:18:56.616 INFO:tasks.workunit.client.0.smithi096.stdout:0/829: link d7/de/fbf d7/de/d5b/d95/d15/d35/f11a 0 2024-04-22T15:18:56.619 INFO:tasks.workunit.client.1.smithi196.stdout:7/316: rename d14/d15/f18 to d14/d26/d28/d29/f61 0 2024-04-22T15:18:56.620 INFO:tasks.workunit.client.1.smithi196.stdout:7/317: write d14/f22 [2612055,120883] 0 2024-04-22T15:18:56.622 INFO:tasks.workunit.client.0.smithi096.stdout:7/938: creat d31/d32/f144 x:0 0 0 2024-04-22T15:18:56.623 INFO:tasks.workunit.client.1.smithi196.stdout:3/343: sync 2024-04-22T15:18:56.625 INFO:tasks.workunit.client.1.smithi196.stdout:8/362: getdents d2/d8/d47 0 2024-04-22T15:18:56.626 INFO:tasks.workunit.client.0.smithi096.stdout:7/939: dread d31/d32/d33/d44/da8/fe2 [0,4194304] 0 2024-04-22T15:18:56.627 INFO:tasks.workunit.client.0.smithi096.stdout:2/695: dread d3/db/f12 [0,4194304] 0 2024-04-22T15:18:56.628 INFO:tasks.workunit.client.1.smithi196.stdout:3/344: dread d0/d15/d48/d51/f54 [0,4194304] 0 2024-04-22T15:18:56.628 INFO:tasks.workunit.client.0.smithi096.stdout:8/676: truncate d4/d1c/d40/d48/f68 10938665 0 2024-04-22T15:18:56.631 INFO:tasks.workunit.client.0.smithi096.stdout:3/604: readlink da/db/de/d19/d2d/l66 0 2024-04-22T15:18:56.632 INFO:tasks.workunit.client.1.smithi196.stdout:9/345: mkdir db/d11/d29/d7a 0 2024-04-22T15:18:56.633 INFO:tasks.workunit.client.1.smithi196.stdout:9/346: write db/dd/f1a [5393571,44203] 0 2024-04-22T15:18:56.634 INFO:tasks.workunit.client.1.smithi196.stdout:5/351: symlink d1/d17/d1a/l71 0 2024-04-22T15:18:56.635 INFO:tasks.workunit.client.1.smithi196.stdout:5/352: write d1/d17/d1c/f3b [494956,92738] 0 2024-04-22T15:18:56.636 INFO:tasks.workunit.client.1.smithi196.stdout:6/379: link dd/d20/c3d dd/d20/c68 0 2024-04-22T15:18:56.638 INFO:tasks.workunit.client.0.smithi096.stdout:3/605: dwrite da/db/de/d19/d2d/d37/f5f [0,4194304] 0 2024-04-22T15:18:56.641 INFO:tasks.workunit.client.0.smithi096.stdout:7/940: unlink d31/d32/d36/f99 0 2024-04-22T15:18:56.645 INFO:tasks.workunit.client.1.smithi196.stdout:0/378: rename d3/d9/d36/d56/d65/d7f/d30/f35 to d3/de/d14/f81 0 2024-04-22T15:18:56.645 INFO:tasks.workunit.client.1.smithi196.stdout:3/345: dread d0/d7/f1a [0,4194304] 0 2024-04-22T15:18:56.647 INFO:tasks.workunit.client.1.smithi196.stdout:2/379: creat d1/d23/d27/d89/f8c x:0 0 0 2024-04-22T15:18:56.648 INFO:tasks.workunit.client.1.smithi196.stdout:2/380: write d1/d23/d27/d2e/f58 [1453357,6434] 0 2024-04-22T15:18:56.649 INFO:tasks.workunit.client.0.smithi096.stdout:4/727: rename d5/d7/d1a/d25/cd3 to d5/d7/d2c/d95/cfa 0 2024-04-22T15:18:56.649 INFO:tasks.workunit.client.1.smithi196.stdout:3/346: dread d0/d7/f1a [0,4194304] 0 2024-04-22T15:18:56.650 INFO:tasks.workunit.client.1.smithi196.stdout:7/318: symlink d14/d46/l62 0 2024-04-22T15:18:56.653 INFO:tasks.workunit.client.1.smithi196.stdout:1/314: creat d1/db/d16/d1e/d65/f66 x:0 0 0 2024-04-22T15:18:56.654 INFO:tasks.workunit.client.0.smithi096.stdout:0/830: symlink d7/d8f/d10e/d114/l11b 0 2024-04-22T15:18:56.654 INFO:tasks.workunit.client.0.smithi096.stdout:0/831: stat d7/de/d3a/lc3 0 2024-04-22T15:18:56.656 INFO:tasks.workunit.client.0.smithi096.stdout:9/620: truncate d1/df/d95/d9d/db5/dd/f21 1063126 0 2024-04-22T15:18:56.658 INFO:tasks.workunit.client.0.smithi096.stdout:9/621: read d1/df/d95/d9d/db5/d43/d57/d5b/d73/f9a [2955995,40847] 0 2024-04-22T15:18:56.658 INFO:tasks.workunit.client.1.smithi196.stdout:8/363: creat d2/d5/d4f/f6e x:0 0 0 2024-04-22T15:18:56.661 INFO:tasks.workunit.client.0.smithi096.stdout:3/606: write da/db/de/d19/f2a [4247457,67166] 0 2024-04-22T15:18:56.661 INFO:tasks.workunit.client.0.smithi096.stdout:3/607: chown da/dab 784762453 1 2024-04-22T15:18:56.665 INFO:tasks.workunit.client.1.smithi196.stdout:0/379: dread d3/de/d14/f4a [0,4194304] 0 2024-04-22T15:18:56.668 INFO:tasks.workunit.client.0.smithi096.stdout:4/728: creat d5/d7/d2c/d3d/d4f/d9c/ffb x:0 0 0 2024-04-22T15:18:56.673 INFO:tasks.workunit.client.0.smithi096.stdout:0/832: dread - d7/de/d5b/d95/dc/d16/feb zero size 2024-04-22T15:18:56.675 INFO:tasks.workunit.client.0.smithi096.stdout:9/622: mknod d1/df/d95/d9d/db5/d43/d57/d5b/da1/cce 0 2024-04-22T15:18:56.676 INFO:tasks.workunit.client.0.smithi096.stdout:0/833: read d7/d8f/fca [919285,61758] 0 2024-04-22T15:18:56.677 INFO:tasks.workunit.client.1.smithi196.stdout:2/381: symlink d1/d13/d25/l8d 0 2024-04-22T15:18:56.678 INFO:tasks.workunit.client.0.smithi096.stdout:3/608: chown da/f34 211 1 2024-04-22T15:18:56.679 INFO:tasks.workunit.client.0.smithi096.stdout:8/677: truncate d4/d10/d25/d38/fce 2951194 0 2024-04-22T15:18:56.680 INFO:tasks.workunit.client.0.smithi096.stdout:7/941: dwrite d31/d32/d36/d75/f9a [0,4194304] 0 2024-04-22T15:18:56.680 INFO:tasks.workunit.client.1.smithi196.stdout:7/319: creat d14/d25/d31/f63 x:0 0 0 2024-04-22T15:18:56.680 INFO:tasks.workunit.client.1.smithi196.stdout:7/320: readlink d14/d25/l57 0 2024-04-22T15:18:56.683 INFO:tasks.workunit.client.0.smithi096.stdout:2/696: link d3/db/d30/fd3 d3/db/df/fde 0 2024-04-22T15:18:56.685 INFO:tasks.workunit.client.1.smithi196.stdout:1/315: truncate d1/db/d16/f40 589120 0 2024-04-22T15:18:56.689 INFO:tasks.workunit.client.1.smithi196.stdout:4/379: creat d6/d13/d15/f6f x:0 0 0 2024-04-22T15:18:56.690 INFO:tasks.workunit.client.0.smithi096.stdout:8/678: dwrite d4/d10/f21 [0,4194304] 0 2024-04-22T15:18:56.691 INFO:tasks.workunit.client.1.smithi196.stdout:0/380: symlink d3/d9/l82 0 2024-04-22T15:18:56.692 INFO:tasks.workunit.client.1.smithi196.stdout:0/381: write d3/f47 [1896953,10937] 0 2024-04-22T15:18:56.692 INFO:tasks.workunit.client.1.smithi196.stdout:9/347: mkdir db/d11/d29/d7a/d7b 0 2024-04-22T15:18:56.693 INFO:tasks.workunit.client.1.smithi196.stdout:0/382: dread - d3/d9/d36/f6d zero size 2024-04-22T15:18:56.695 INFO:tasks.workunit.client.1.smithi196.stdout:6/380: mknod dd/d1f/d37/c69 0 2024-04-22T15:18:56.701 INFO:tasks.workunit.client.0.smithi096.stdout:2/697: mkdir d3/db/df/d24/d58/d5f/ddf 0 2024-04-22T15:18:56.702 INFO:tasks.workunit.client.0.smithi096.stdout:2/698: stat d3/db/df/d10/f37 0 2024-04-22T15:18:56.702 INFO:tasks.workunit.client.0.smithi096.stdout:0/834: dread d7/f20 [0,4194304] 0 2024-04-22T15:18:56.706 INFO:tasks.workunit.client.1.smithi196.stdout:0/383: dread d3/f5 [0,4194304] 0 2024-04-22T15:18:56.706 INFO:tasks.workunit.client.0.smithi096.stdout:9/623: creat d1/d25/db2/dca/fcf x:0 0 0 2024-04-22T15:18:56.707 INFO:tasks.workunit.client.0.smithi096.stdout:3/609: dread da/f8f [0,4194304] 0 2024-04-22T15:18:56.711 INFO:tasks.workunit.client.0.smithi096.stdout:3/610: dread da/db/d12/d22/f4e [0,4194304] 0 2024-04-22T15:18:56.713 INFO:tasks.workunit.client.0.smithi096.stdout:2/699: unlink d3/db/df/dce/d46/d7e/l90 0 2024-04-22T15:18:56.714 INFO:tasks.workunit.client.1.smithi196.stdout:9/348: creat db/dd/d1b/d2c/f7c x:0 0 0 2024-04-22T15:18:56.715 INFO:tasks.workunit.client.1.smithi196.stdout:6/381: mkdir dd/d20/d5c/d6a 0 2024-04-22T15:18:56.716 INFO:tasks.workunit.client.0.smithi096.stdout:0/835: symlink d7/d42/d4c/d55/d9c/l11c 0 2024-04-22T15:18:56.717 INFO:tasks.workunit.client.0.smithi096.stdout:4/729: write d5/d7/d1f/d84/d6d/d98/fe8 [721800,96360] 0 2024-04-22T15:18:56.718 INFO:tasks.workunit.client.0.smithi096.stdout:9/624: read d1/f6 [2086634,15888] 0 2024-04-22T15:18:56.719 INFO:tasks.workunit.client.1.smithi196.stdout:3/347: rmdir d0/d15/d31/d55/d71 0 2024-04-22T15:18:56.721 INFO:tasks.workunit.client.1.smithi196.stdout:0/384: symlink d3/d9/d25/l83 0 2024-04-22T15:18:56.722 INFO:tasks.workunit.client.0.smithi096.stdout:3/611: creat da/db/d12/d22/d26/db2/fc4 x:0 0 0 2024-04-22T15:18:56.723 INFO:tasks.workunit.client.0.smithi096.stdout:2/700: creat d3/db/df/d24/fe0 x:0 0 0 2024-04-22T15:18:56.724 INFO:tasks.workunit.client.1.smithi196.stdout:8/364: getdents d2/d8/d47 0 2024-04-22T15:18:56.725 INFO:tasks.workunit.client.1.smithi196.stdout:5/353: link d1/d9/l46 d1/d9/l72 0 2024-04-22T15:18:56.729 INFO:tasks.workunit.client.0.smithi096.stdout:9/625: creat d1/d25/db2/d98/fd0 x:0 0 0 2024-04-22T15:18:56.733 INFO:tasks.workunit.client.1.smithi196.stdout:7/321: rename d14/d26/d28/c2f to d14/c64 0 2024-04-22T15:18:56.733 INFO:tasks.workunit.client.0.smithi096.stdout:7/942: getdents d31/d32/d33/d6a/de7 0 2024-04-22T15:18:56.734 INFO:tasks.workunit.client.1.smithi196.stdout:0/385: mknod d3/de/d14/d5a/c84 0 2024-04-22T15:18:56.737 INFO:tasks.workunit.client.1.smithi196.stdout:8/365: creat d2/d3f/d4b/f6f x:0 0 0 2024-04-22T15:18:56.737 INFO:tasks.workunit.client.1.smithi196.stdout:8/366: fsync d2/f4a 0 2024-04-22T15:18:56.738 INFO:tasks.workunit.client.0.smithi096.stdout:0/836: mkdir d7/de/d5b/d95/d11d 0 2024-04-22T15:18:56.740 INFO:tasks.workunit.client.1.smithi196.stdout:5/354: creat d1/d17/d50/f73 x:0 0 0 2024-04-22T15:18:56.740 INFO:tasks.workunit.client.1.smithi196.stdout:5/355: fsync d1/d4/f22 0 2024-04-22T15:18:56.741 INFO:tasks.workunit.client.0.smithi096.stdout:8/679: getdents d4/d10/d25 0 2024-04-22T15:18:56.741 INFO:tasks.workunit.client.1.smithi196.stdout:5/356: chown d1/d17/d1a/d2f/d65 3 1 2024-04-22T15:18:56.741 INFO:tasks.workunit.client.1.smithi196.stdout:9/349: link db/dd/d1b/d2c/f6d db/d11/d29/d7a/f7d 0 2024-04-22T15:18:56.742 INFO:tasks.workunit.client.0.smithi096.stdout:9/626: dread - d1/f50 zero size 2024-04-22T15:18:56.742 INFO:tasks.workunit.client.1.smithi196.stdout:6/382: getdents dd/d1f/d37/d58 0 2024-04-22T15:18:56.743 INFO:tasks.workunit.client.1.smithi196.stdout:2/382: getdents d1/d1d 0 2024-04-22T15:18:56.744 INFO:tasks.workunit.client.0.smithi096.stdout:8/680: dread d4/d1c/d62/fa2 [0,4194304] 0 2024-04-22T15:18:56.745 INFO:tasks.workunit.client.0.smithi096.stdout:6/580: sync 2024-04-22T15:18:56.745 INFO:tasks.workunit.client.1.smithi196.stdout:3/348: creat d0/d7/d6f/f76 x:0 0 0 2024-04-22T15:18:56.745 INFO:tasks.workunit.client.1.smithi196.stdout:3/349: stat d0/d7/l3b 0 2024-04-22T15:18:56.746 INFO:tasks.workunit.client.1.smithi196.stdout:7/322: rmdir d14/d46 39 2024-04-22T15:18:56.746 INFO:tasks.workunit.client.0.smithi096.stdout:0/837: read - d7/de/d5b/d95/d66/dce/fef zero size 2024-04-22T15:18:56.747 INFO:tasks.workunit.client.1.smithi196.stdout:0/386: mknod d3/d9/d36/d56/d65/c85 0 2024-04-22T15:18:56.748 INFO:tasks.workunit.client.1.smithi196.stdout:0/387: write d3/d9/d36/d56/f7b [644972,41331] 0 2024-04-22T15:18:56.748 INFO:tasks.workunit.client.1.smithi196.stdout:1/316: getdents d1/db/dc/d26/d31 0 2024-04-22T15:18:56.749 INFO:tasks.workunit.client.0.smithi096.stdout:4/730: link d5/d7/d1f/fa6 d5/d7/d2c/d3d/d4f/d88/ffc 0 2024-04-22T15:18:56.749 INFO:tasks.workunit.client.1.smithi196.stdout:0/388: read d3/f5 [753982,67702] 0 2024-04-22T15:18:56.750 INFO:tasks.workunit.client.0.smithi096.stdout:9/627: symlink d1/df/d95/d9d/db5/d43/d57/d5b/da1/ld1 0 2024-04-22T15:18:56.751 INFO:tasks.workunit.client.1.smithi196.stdout:8/367: unlink d2/d5/l17 0 2024-04-22T15:18:56.752 INFO:tasks.workunit.client.1.smithi196.stdout:8/368: truncate d2/dc/f10 1578584 0 2024-04-22T15:18:56.755 INFO:tasks.workunit.client.1.smithi196.stdout:9/350: unlink db/dd/f69 0 2024-04-22T15:18:56.756 INFO:tasks.workunit.client.0.smithi096.stdout:8/681: symlink d4/d10/d4d/le4 0 2024-04-22T15:18:56.758 INFO:tasks.workunit.client.1.smithi196.stdout:6/383: symlink dd/d1f/d4e/d53/d55/d57/l6b 0 2024-04-22T15:18:56.758 INFO:tasks.workunit.client.1.smithi196.stdout:6/384: write dd/d2b/f62 [650769,71991] 0 2024-04-22T15:18:56.761 INFO:tasks.workunit.client.0.smithi096.stdout:7/943: write d31/d32/d33/d44/da8/f132 [1242255,123055] 0 2024-04-22T15:18:56.762 INFO:tasks.workunit.client.1.smithi196.stdout:1/317: symlink d1/db/dc/d26/d2e/d32/l67 0 2024-04-22T15:18:56.763 INFO:tasks.workunit.client.1.smithi196.stdout:1/318: readlink d1/db/d16/d1e/l2f 0 2024-04-22T15:18:56.764 INFO:tasks.workunit.client.0.smithi096.stdout:9/628: symlink d1/d25/db2/d98/d9b/ld2 0 2024-04-22T15:18:56.764 INFO:tasks.workunit.client.1.smithi196.stdout:1/319: write d1/db/f58 [263390,85559] 0 2024-04-22T15:18:56.764 INFO:tasks.workunit.client.1.smithi196.stdout:0/389: mkdir d3/de/d14/d86 0 2024-04-22T15:18:56.765 INFO:tasks.workunit.client.1.smithi196.stdout:8/369: chown d2/c9 14816296 1 2024-04-22T15:18:56.767 INFO:tasks.workunit.client.1.smithi196.stdout:9/351: creat db/d11/d29/d7a/f7e x:0 0 0 2024-04-22T15:18:56.768 INFO:tasks.workunit.client.1.smithi196.stdout:9/352: write db/d11/d29/d7a/f7e [171813,26464] 0 2024-04-22T15:18:56.770 INFO:tasks.workunit.client.1.smithi196.stdout:6/385: symlink dd/d2b/l6c 0 2024-04-22T15:18:56.772 INFO:tasks.workunit.client.1.smithi196.stdout:4/380: rename d6/d13/c47 to d6/d13/d15/d2c/d35/c70 0 2024-04-22T15:18:56.772 INFO:tasks.workunit.client.1.smithi196.stdout:5/357: rename d1/d17/d1a to d1/d17/d1a/d2f/d74 22 2024-04-22T15:18:56.774 INFO:tasks.workunit.client.1.smithi196.stdout:3/350: link d0/d7/f5b d0/d7/d8/f77 0 2024-04-22T15:18:56.774 INFO:tasks.workunit.client.1.smithi196.stdout:4/381: dread d6/d10/f50 [0,4194304] 0 2024-04-22T15:18:56.775 INFO:tasks.workunit.client.1.smithi196.stdout:1/320: unlink d1/db/d16/d1e/d49/c4d 0 2024-04-22T15:18:56.777 INFO:tasks.workunit.client.1.smithi196.stdout:6/386: dwrite dd/d20/d4d/f65 [0,4194304] 0 2024-04-22T15:18:56.777 INFO:tasks.workunit.client.1.smithi196.stdout:2/383: getdents d1/d1d 0 2024-04-22T15:18:56.781 INFO:tasks.workunit.client.1.smithi196.stdout:7/323: rename d14/d15/f4b to d14/d25/d31/f65 0 2024-04-22T15:18:56.781 INFO:tasks.workunit.client.1.smithi196.stdout:5/358: creat d1/d9/f75 x:0 0 0 2024-04-22T15:18:56.782 INFO:tasks.workunit.client.1.smithi196.stdout:4/382: dwrite d6/d14/d18/d26/f3a [0,4194304] 0 2024-04-22T15:18:56.786 INFO:tasks.workunit.client.1.smithi196.stdout:1/321: creat d1/db/dc/de/d43/f68 x:0 0 0 2024-04-22T15:18:56.789 INFO:tasks.workunit.client.1.smithi196.stdout:6/387: unlink dd/f1b 0 2024-04-22T15:18:56.790 INFO:tasks.workunit.client.1.smithi196.stdout:9/353: symlink db/d11/d29/d7a/d7b/l7f 0 2024-04-22T15:18:56.793 INFO:tasks.workunit.client.1.smithi196.stdout:1/322: dwrite d1/db/d16/f1d [0,4194304] 0 2024-04-22T15:18:56.793 INFO:tasks.workunit.client.1.smithi196.stdout:8/370: link d2/d3f/c58 d2/d5/d1a/c70 0 2024-04-22T15:18:56.797 INFO:tasks.workunit.client.1.smithi196.stdout:2/384: symlink d1/d13/l8e 0 2024-04-22T15:18:56.798 INFO:tasks.workunit.client.1.smithi196.stdout:9/354: symlink db/d11/d29/d7a/l80 0 2024-04-22T15:18:56.798 INFO:tasks.workunit.client.1.smithi196.stdout:7/324: getdents d14/d26/d5e 0 2024-04-22T15:18:56.799 INFO:tasks.workunit.client.1.smithi196.stdout:0/390: getdents d3/de/d14/d29 0 2024-04-22T15:18:56.799 INFO:tasks.workunit.client.1.smithi196.stdout:5/359: fdatasync d1/d2/f28 0 2024-04-22T15:18:56.800 INFO:tasks.workunit.client.1.smithi196.stdout:8/371: symlink d2/d3f/l71 0 2024-04-22T15:18:56.802 INFO:tasks.workunit.client.1.smithi196.stdout:6/388: unlink dd/d20/c68 0 2024-04-22T15:18:56.804 INFO:tasks.workunit.client.1.smithi196.stdout:6/389: truncate dd/d20/f24 909351 0 2024-04-22T15:18:56.805 INFO:tasks.workunit.client.1.smithi196.stdout:9/355: mkdir db/d11/d29/d7a/d7b/d81 0 2024-04-22T15:18:56.805 INFO:tasks.workunit.client.1.smithi196.stdout:1/323: mkdir d1/db/dc/de/d5a/d5b/d69 0 2024-04-22T15:18:56.806 INFO:tasks.workunit.client.1.smithi196.stdout:1/324: fsync d1/db/ff 0 2024-04-22T15:18:56.807 INFO:tasks.workunit.client.1.smithi196.stdout:8/372: creat d2/d3f/d4b/f72 x:0 0 0 2024-04-22T15:18:56.809 INFO:tasks.workunit.client.1.smithi196.stdout:9/356: symlink db/d11/l82 0 2024-04-22T15:18:56.812 INFO:tasks.workunit.client.1.smithi196.stdout:8/373: creat d2/d8/f73 x:0 0 0 2024-04-22T15:18:56.815 INFO:tasks.workunit.client.1.smithi196.stdout:1/325: rename d1/db/dc/d26/d31 to d1/db/d16/d44/d6a 0 2024-04-22T15:18:56.815 INFO:tasks.workunit.client.1.smithi196.stdout:9/357: dwrite db/dd/de/f10 [0,4194304] 0 2024-04-22T15:18:56.818 INFO:tasks.workunit.client.1.smithi196.stdout:8/374: unlink d2/d5/d2e/c3b 0 2024-04-22T15:18:56.820 INFO:tasks.workunit.client.1.smithi196.stdout:1/326: fsync d1/db/dc/d26/f28 0 2024-04-22T15:18:56.821 INFO:tasks.workunit.client.1.smithi196.stdout:0/391: dread d3/d9/f21 [0,4194304] 0 2024-04-22T15:18:56.822 INFO:tasks.workunit.client.1.smithi196.stdout:9/358: rename db/dd/de/d15/l3a to db/d11/d29/l83 0 2024-04-22T15:18:56.823 INFO:tasks.workunit.client.1.smithi196.stdout:7/325: truncate fe 941974 0 2024-04-22T15:18:56.825 INFO:tasks.workunit.client.1.smithi196.stdout:9/359: creat db/d11/d29/f84 x:0 0 0 2024-04-22T15:18:56.825 INFO:tasks.workunit.client.1.smithi196.stdout:6/390: dread dd/f3a [0,4194304] 0 2024-04-22T15:18:56.827 INFO:tasks.workunit.client.1.smithi196.stdout:1/327: link d1/db/d16/d44/d6a/f50 d1/db/dc/de/d43/f6b 0 2024-04-22T15:18:56.827 INFO:tasks.workunit.client.1.smithi196.stdout:2/385: truncate d1/d23/d27/d2e/d55/f36 3196165 0 2024-04-22T15:18:56.829 INFO:tasks.workunit.client.1.smithi196.stdout:2/386: write d1/d23/d27/d2e/d45/f79 [1345594,82284] 0 2024-04-22T15:18:56.830 INFO:tasks.workunit.client.1.smithi196.stdout:7/326: creat d14/d26/d2a/f66 x:0 0 0 2024-04-22T15:18:56.831 INFO:tasks.workunit.client.1.smithi196.stdout:9/360: fsync db/d11/d29/d7a/f7d 0 2024-04-22T15:18:56.832 INFO:tasks.workunit.client.1.smithi196.stdout:5/360: truncate d1/d17/d1a/f1d 2635096 0 2024-04-22T15:18:56.833 INFO:tasks.workunit.client.1.smithi196.stdout:6/391: mkdir dd/d1f/d4e/d53/d55/d6d 0 2024-04-22T15:18:56.833 INFO:tasks.workunit.client.1.smithi196.stdout:5/361: chown d1/d51/f69 268217 1 2024-04-22T15:18:56.835 INFO:tasks.workunit.client.1.smithi196.stdout:0/392: fsync d3/de/d14/f81 0 2024-04-22T15:18:56.836 INFO:tasks.workunit.client.0.smithi096.stdout:3/612: getdents da/db/d12 0 2024-04-22T15:18:56.836 INFO:tasks.workunit.client.0.smithi096.stdout:8/682: rmdir d4/d10/d25/d4c 39 2024-04-22T15:18:56.837 INFO:tasks.workunit.client.0.smithi096.stdout:2/701: rename d3/db/df/dce/d46/d59/f76 to d3/db/df/dce/d46/d69/fe1 0 2024-04-22T15:18:56.838 INFO:tasks.workunit.client.1.smithi196.stdout:2/387: fsync d1/d13/d41/f62 0 2024-04-22T15:18:56.839 INFO:tasks.workunit.client.0.smithi096.stdout:8/683: write d4/d1c/d62/f6b [3578732,24106] 0 2024-04-22T15:18:56.843 INFO:tasks.workunit.client.0.smithi096.stdout:0/838: link d7/f20 d7/de/d3a/d61/d79/f11e 0 2024-04-22T15:18:56.845 INFO:tasks.workunit.client.1.smithi196.stdout:9/361: symlink db/d11/d29/d7a/l85 0 2024-04-22T15:18:56.846 INFO:tasks.workunit.client.0.smithi096.stdout:9/629: truncate d1/df/d95/d9d/db5/d43/d57/d5b/d73/f9e 679066 0 2024-04-22T15:18:56.846 INFO:tasks.workunit.client.1.smithi196.stdout:2/388: dwrite d1/f3e [0,4194304] 0 2024-04-22T15:18:56.846 INFO:tasks.workunit.client.0.smithi096.stdout:0/839: dread - d7/de/fe8 zero size 2024-04-22T15:18:56.848 INFO:tasks.workunit.client.0.smithi096.stdout:9/630: chown d1/d85/fab 369 1 2024-04-22T15:18:56.849 INFO:tasks.workunit.client.0.smithi096.stdout:7/944: dwrite d31/d32/fe6 [0,4194304] 0 2024-04-22T15:18:56.850 INFO:tasks.workunit.client.0.smithi096.stdout:3/613: dwrite da/db/dc/d28/d69/d89/f65 [0,4194304] 0 2024-04-22T15:18:56.851 INFO:tasks.workunit.client.1.smithi196.stdout:1/328: dread d1/db/dc/d26/d2e/d32/f53 [0,4194304] 0 2024-04-22T15:18:56.862 INFO:tasks.workunit.client.0.smithi096.stdout:4/731: getdents d5/d7/d1f 0 2024-04-22T15:18:56.864 INFO:tasks.workunit.client.0.smithi096.stdout:0/840: creat d7/de/d3a/d61/f11f x:0 0 0 2024-04-22T15:18:56.865 INFO:tasks.workunit.client.0.smithi096.stdout:0/841: chown d7/de/d3a/d61/d79/fbe 81 1 2024-04-22T15:18:56.866 INFO:tasks.workunit.client.0.smithi096.stdout:0/842: chown d7/de/d5b/d64/f107 97 1 2024-04-22T15:18:56.866 INFO:tasks.workunit.client.0.smithi096.stdout:3/614: creat da/db/dc/d8d/fc5 x:0 0 0 2024-04-22T15:18:56.868 INFO:tasks.workunit.client.0.smithi096.stdout:3/615: write da/db/dc/d28/f45 [1108826,62214] 0 2024-04-22T15:18:56.868 INFO:tasks.workunit.client.1.smithi196.stdout:9/362: rename db/dd/de/d15/f36 to db/dd/de/d75/f86 0 2024-04-22T15:18:56.869 INFO:tasks.workunit.client.0.smithi096.stdout:3/616: chown da/db/dc/d28/fad 2173626 1 2024-04-22T15:18:56.870 INFO:tasks.workunit.client.0.smithi096.stdout:9/631: rename d1/d4/d79/fb3 to d1/df/d95/d9d/db5/d43/d57/d5b/da1/fd3 0 2024-04-22T15:18:56.870 INFO:tasks.workunit.client.0.smithi096.stdout:3/617: fdatasync da/db/d12/d22/f88 0 2024-04-22T15:18:56.870 INFO:tasks.workunit.client.1.smithi196.stdout:9/363: write db/dd/d1b/d2c/f58 [1597932,21330] 0 2024-04-22T15:18:56.872 INFO:tasks.workunit.client.1.smithi196.stdout:6/392: unlink dd/d20/c3d 0 2024-04-22T15:18:56.874 INFO:tasks.workunit.client.1.smithi196.stdout:5/362: mknod d1/d17/c76 0 2024-04-22T15:18:56.877 INFO:tasks.workunit.client.0.smithi096.stdout:4/732: symlink d5/d7/dcb/lfd 0 2024-04-22T15:18:56.877 INFO:tasks.workunit.client.0.smithi096.stdout:4/733: dread - d5/d7/d2c/d67/fe6 zero size 2024-04-22T15:18:56.878 INFO:tasks.workunit.client.0.smithi096.stdout:0/843: mkdir d7/d8f/d10e/d114/d120 0 2024-04-22T15:18:56.879 INFO:tasks.workunit.client.1.smithi196.stdout:1/329: dread d1/db/dc/d26/d2e/d32/f53 [0,4194304] 0 2024-04-22T15:18:56.879 INFO:tasks.workunit.client.0.smithi096.stdout:0/844: chown d7/de/d5b/d95/dc/d16/d65/d7c/c101 274 1 2024-04-22T15:18:56.880 INFO:tasks.workunit.client.1.smithi196.stdout:1/330: stat d1/db/dc/d26/d2e/d32/d36/c62 0 2024-04-22T15:18:56.882 INFO:tasks.workunit.client.0.smithi096.stdout:9/632: truncate d1/d4/f7a 4311675 0 2024-04-22T15:18:56.883 INFO:tasks.workunit.client.1.smithi196.stdout:7/327: creat d14/f67 x:0 0 0 2024-04-22T15:18:56.884 INFO:tasks.workunit.client.0.smithi096.stdout:3/618: mkdir da/db/d12/d22/d26/dc6 0 2024-04-22T15:18:56.888 INFO:tasks.workunit.client.0.smithi096.stdout:2/702: creat d3/db/d30/fe2 x:0 0 0 2024-04-22T15:18:56.890 INFO:tasks.workunit.client.1.smithi196.stdout:0/393: truncate d3/d6/dc/f58 5091138 0 2024-04-22T15:18:56.891 INFO:tasks.workunit.client.0.smithi096.stdout:3/619: dwrite da/db/d12/d22/f53 [0,4194304] 0 2024-04-22T15:18:56.893 INFO:tasks.workunit.client.1.smithi196.stdout:2/389: rename d1/d23/d27/d2e/l5c to d1/d23/d84/l8f 0 2024-04-22T15:18:56.895 INFO:tasks.workunit.client.0.smithi096.stdout:0/845: readlink d7/de/d5b/d95/dc/d16/l50 0 2024-04-22T15:18:56.895 INFO:tasks.workunit.client.1.smithi196.stdout:2/390: dread - d1/d23/d27/d2e/d55/d5b/d7c/f7f zero size 2024-04-22T15:18:56.896 INFO:tasks.workunit.client.1.smithi196.stdout:9/364: symlink db/d11/d29/d57/l87 0 2024-04-22T15:18:56.896 INFO:tasks.workunit.client.1.smithi196.stdout:0/394: dwrite d3/de/d14/f55 [0,4194304] 0 2024-04-22T15:18:56.897 INFO:tasks.workunit.client.0.smithi096.stdout:9/633: creat d1/df/d95/fd4 x:0 0 0 2024-04-22T15:18:56.898 INFO:tasks.workunit.client.1.smithi196.stdout:3/351: sync 2024-04-22T15:18:56.898 INFO:tasks.workunit.client.1.smithi196.stdout:9/365: write db/dd/de/f54 [3976590,2224] 0 2024-04-22T15:18:56.900 INFO:tasks.workunit.client.1.smithi196.stdout:6/393: rmdir dd/d1f/d37 39 2024-04-22T15:18:56.901 INFO:tasks.workunit.client.0.smithi096.stdout:0/846: dwrite d7/da7/f110 [0,4194304] 0 2024-04-22T15:18:56.903 INFO:tasks.workunit.client.1.smithi196.stdout:6/394: write dd/d1f/d38/d40/f43 [2231409,50415] 0 2024-04-22T15:18:56.929 INFO:tasks.workunit.client.1.smithi196.stdout:5/363: dread d1/d17/f24 [0,4194304] 0 2024-04-22T15:18:56.930 INFO:tasks.workunit.client.1.smithi196.stdout:5/364: fsync d1/d14/f1e 0 2024-04-22T15:18:56.930 INFO:tasks.workunit.client.1.smithi196.stdout:5/365: chown d1/d17/d1a/d2f/d30/d5e/f63 3614 1 2024-04-22T15:18:56.931 INFO:tasks.workunit.client.0.smithi096.stdout:7/945: truncate d31/d32/d33/d44/da8/fe2 3995493 0 2024-04-22T15:18:56.932 INFO:tasks.workunit.client.0.smithi096.stdout:7/946: readlink d31/d32/d33/d6a/l81 0 2024-04-22T15:18:56.935 INFO:tasks.workunit.client.0.smithi096.stdout:8/684: creat d4/d10/d25/d4c/db4/dd0/fe5 x:0 0 0 2024-04-22T15:18:56.937 INFO:tasks.workunit.client.1.smithi196.stdout:1/331: read d1/f4 [2811089,1584] 0 2024-04-22T15:18:56.938 INFO:tasks.workunit.client.1.smithi196.stdout:7/328: creat d14/d26/d28/d29/f68 x:0 0 0 2024-04-22T15:18:56.939 INFO:tasks.workunit.client.0.smithi096.stdout:7/947: dwrite d31/d68/f10d [0,4194304] 0 2024-04-22T15:18:56.939 INFO:tasks.workunit.client.0.smithi096.stdout:7/948: readlink d31/l61 0 2024-04-22T15:18:56.940 INFO:tasks.workunit.client.1.smithi196.stdout:7/329: dread d14/d25/f3b [0,4194304] 0 2024-04-22T15:18:56.940 INFO:tasks.workunit.client.1.smithi196.stdout:4/383: sync 2024-04-22T15:18:56.940 INFO:tasks.workunit.client.1.smithi196.stdout:8/375: sync 2024-04-22T15:18:56.941 INFO:tasks.workunit.client.0.smithi096.stdout:9/634: creat d1/d25/fd5 x:0 0 0 2024-04-22T15:18:56.942 INFO:tasks.workunit.client.0.smithi096.stdout:1/757: sync 2024-04-22T15:18:56.942 INFO:tasks.workunit.client.0.smithi096.stdout:5/712: sync 2024-04-22T15:18:56.943 INFO:tasks.workunit.client.1.smithi196.stdout:0/395: fsync d3/d9/d25/f63 0 2024-04-22T15:18:56.943 INFO:tasks.workunit.client.0.smithi096.stdout:5/713: chown d4/ddb/f96 0 1 2024-04-22T15:18:56.944 INFO:tasks.workunit.client.1.smithi196.stdout:0/396: dread - d3/de/f7d zero size 2024-04-22T15:18:56.945 INFO:tasks.workunit.client.1.smithi196.stdout:4/384: dread d6/d13/d15/d2c/d62/f5a [0,4194304] 0 2024-04-22T15:18:56.951 INFO:tasks.workunit.client.0.smithi096.stdout:8/685: creat d4/d2a/fe6 x:0 0 0 2024-04-22T15:18:56.952 INFO:tasks.workunit.client.1.smithi196.stdout:9/366: dwrite db/dd/de/d75/f86 [0,4194304] 0 2024-04-22T15:18:56.953 INFO:tasks.workunit.client.0.smithi096.stdout:3/620: mknod da/db/dc/d28/d96/d97/cc7 0 2024-04-22T15:18:56.954 INFO:tasks.workunit.client.1.smithi196.stdout:6/395: fdatasync dd/d1f/d38/d40/f4f 0 2024-04-22T15:18:56.956 INFO:tasks.workunit.client.0.smithi096.stdout:4/734: creat d5/d7/d2c/d43/ffe x:0 0 0 2024-04-22T15:18:56.956 INFO:tasks.workunit.client.0.smithi096.stdout:4/735: read - d5/d7/d2c/ff0 zero size 2024-04-22T15:18:56.958 INFO:tasks.workunit.client.0.smithi096.stdout:7/949: rmdir d31/d32/d33/d44/da8/dbf/ddf 39 2024-04-22T15:18:56.960 INFO:tasks.workunit.client.1.smithi196.stdout:1/332: chown d1/db/dc/d26/d2e/d32/c35 38 1 2024-04-22T15:18:56.960 INFO:tasks.workunit.client.0.smithi096.stdout:0/847: write d7/d42/d4c/d55/d9c/fb7 [145728,2753] 0 2024-04-22T15:18:56.963 INFO:tasks.workunit.client.0.smithi096.stdout:9/635: read d1/df/d95/d9d/db5/f12 [4157149,13713] 0 2024-04-22T15:18:56.968 INFO:tasks.workunit.client.1.smithi196.stdout:8/376: dread d2/f6d [0,4194304] 0 2024-04-22T15:18:56.969 INFO:tasks.workunit.client.1.smithi196.stdout:2/391: rename d1/d13/d25/c74 to d1/d13/d1f/c90 0 2024-04-22T15:18:56.971 INFO:tasks.workunit.client.0.smithi096.stdout:8/686: dread - d4/d10/d25/d4c/d76/fdf zero size 2024-04-22T15:18:56.974 INFO:tasks.workunit.client.0.smithi096.stdout:7/950: symlink d31/d32/d33/d6a/de7/l145 0 2024-04-22T15:18:56.975 INFO:tasks.workunit.client.1.smithi196.stdout:3/352: dread d0/d15/d4d/db/dc/d4a/d44/f26 [0,4194304] 0 2024-04-22T15:18:56.975 INFO:tasks.workunit.client.1.smithi196.stdout:9/367: stat db/dd/d1b/d2c/l70 0 2024-04-22T15:18:56.975 INFO:tasks.workunit.client.1.smithi196.stdout:9/368: chown db/dd/de/d15/d31/l66 461143 1 2024-04-22T15:18:56.976 INFO:tasks.workunit.client.0.smithi096.stdout:0/848: readlink d7/l2c 0 2024-04-22T15:18:56.979 INFO:tasks.workunit.client.1.smithi196.stdout:6/396: chown dd/d1f/l35 30456 1 2024-04-22T15:18:56.981 INFO:tasks.workunit.client.1.smithi196.stdout:9/369: dwrite db/ff [0,4194304] 0 2024-04-22T15:18:56.982 INFO:tasks.workunit.client.1.smithi196.stdout:1/333: mknod d1/db/d16/d1e/d49/c6c 0 2024-04-22T15:18:56.984 INFO:tasks.workunit.client.0.smithi096.stdout:9/636: rename d1/df/d95/d9d/db5/d43/d57/d5b/d73/f9a to d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dbd/fd6 0 2024-04-22T15:18:56.986 INFO:tasks.workunit.client.0.smithi096.stdout:2/703: creat d3/db/df/dce/d46/d69/fe3 x:0 0 0 2024-04-22T15:18:56.988 INFO:tasks.workunit.client.0.smithi096.stdout:2/704: readlink d3/db/df/dce/d46/ld8 0 2024-04-22T15:18:56.988 INFO:tasks.workunit.client.1.smithi196.stdout:4/385: dwrite d6/d10/f37 [0,4194304] 0 2024-04-22T15:18:56.989 INFO:tasks.workunit.client.0.smithi096.stdout:3/621: dread da/db/dc/d59/f79 [0,4194304] 0 2024-04-22T15:18:56.991 INFO:tasks.workunit.client.0.smithi096.stdout:0/849: rmdir d7/de/d5b/d95/dc/d16/d65 39 2024-04-22T15:18:56.992 INFO:tasks.workunit.client.1.smithi196.stdout:9/370: dwrite db/dd/de/f54 [0,4194304] 0 2024-04-22T15:18:56.992 INFO:tasks.workunit.client.0.smithi096.stdout:5/714: dread d4/fd [4194304,4194304] 0 2024-04-22T15:18:56.997 INFO:tasks.workunit.client.0.smithi096.stdout:1/758: creat d6/dc/d8a/d3f/d62/dc2/dd0/fe8 x:0 0 0 2024-04-22T15:18:57.002 INFO:tasks.workunit.client.1.smithi196.stdout:2/392: creat d1/d23/d27/d2e/d55/d5b/d7c/f91 x:0 0 0 2024-04-22T15:18:57.002 INFO:tasks.workunit.client.0.smithi096.stdout:8/687: mknod d4/d10/d25/d38/d8f/d9c/ce7 0 2024-04-22T15:18:57.003 INFO:tasks.workunit.client.1.smithi196.stdout:2/393: write d1/f5 [1090983,130600] 0 2024-04-22T15:18:57.005 INFO:tasks.workunit.client.0.smithi096.stdout:8/688: write d4/d1c/d62/fa5 [4772052,108932] 0 2024-04-22T15:18:57.007 INFO:tasks.workunit.client.0.smithi096.stdout:8/689: write d4/d2a/db7/fe0 [548030,24354] 0 2024-04-22T15:18:57.007 INFO:tasks.workunit.client.0.smithi096.stdout:4/736: dread d5/d7/d2c/d95/dee/f59 [0,4194304] 0 2024-04-22T15:18:57.010 INFO:tasks.workunit.client.0.smithi096.stdout:3/622: mknod da/db/dc/d28/d69/cc8 0 2024-04-22T15:18:57.015 INFO:tasks.workunit.client.0.smithi096.stdout:0/850: rename d7/d42/d4c/d55/d9c/df7/d10c/fe5 to d7/de/d5b/d95/dc/d16/d18/f121 0 2024-04-22T15:18:57.015 INFO:tasks.workunit.client.0.smithi096.stdout:5/715: mkdir d4/ddb/d3d/ded 0 2024-04-22T15:18:57.018 INFO:tasks.workunit.client.0.smithi096.stdout:9/637: creat d1/df/d95/d9d/db5/dd/dba/dc2/dc4/fd7 x:0 0 0 2024-04-22T15:18:57.020 INFO:tasks.workunit.client.1.smithi196.stdout:1/334: mkdir d1/db/d16/d6d 0 2024-04-22T15:18:57.021 INFO:tasks.workunit.client.0.smithi096.stdout:7/951: link d31/d32/d33/d44/cd8 d31/d32/d36/d75/dff/c146 0 2024-04-22T15:18:57.023 INFO:tasks.workunit.client.1.smithi196.stdout:7/330: unlink d14/d4a/l4c 0 2024-04-22T15:18:57.024 INFO:tasks.workunit.client.1.smithi196.stdout:7/331: read - d14/d26/d28/f58 zero size 2024-04-22T15:18:57.024 INFO:tasks.workunit.client.0.smithi096.stdout:0/851: dwrite d7/d42/d4c/d55/d9c/df7/f10d [0,4194304] 0 2024-04-22T15:18:57.024 INFO:tasks.workunit.client.0.smithi096.stdout:2/705: fsync d3/db/df/f61 0 2024-04-22T15:18:57.030 INFO:tasks.workunit.client.1.smithi196.stdout:4/386: mknod d6/d13/d15/d2c/d35/d44/c71 0 2024-04-22T15:18:57.035 INFO:tasks.workunit.client.1.smithi196.stdout:9/371: rename db/dd/f1a to db/dd/f88 0 2024-04-22T15:18:57.036 INFO:tasks.workunit.client.0.smithi096.stdout:3/623: fdatasync da/db/d12/d22/f4e 0 2024-04-22T15:18:57.037 INFO:tasks.workunit.client.0.smithi096.stdout:3/624: stat da/db/d10/c18 0 2024-04-22T15:18:57.038 INFO:tasks.workunit.client.0.smithi096.stdout:3/625: fdatasync da/db/de/d19/d3e/f4f 0 2024-04-22T15:18:57.039 INFO:tasks.workunit.client.1.smithi196.stdout:5/366: rename d1/d17/d1a/f1d to d1/d17/d1a/d2f/d30/d5e/f77 0 2024-04-22T15:18:57.040 INFO:tasks.workunit.client.1.smithi196.stdout:5/367: chown d1/d17/d1a/d2f/d30/d5e/f63 10 1 2024-04-22T15:18:57.043 INFO:tasks.workunit.client.0.smithi096.stdout:5/716: mkdir d4/ddb/d3d/d4d/d59/d98/dee 0 2024-04-22T15:18:57.045 INFO:tasks.workunit.client.1.smithi196.stdout:2/394: mknod d1/d23/d27/d2e/d55/d5b/d7c/c92 0 2024-04-22T15:18:57.048 INFO:tasks.workunit.client.0.smithi096.stdout:7/952: dread d31/d32/dc0/fc4 [0,4194304] 0 2024-04-22T15:18:57.051 INFO:tasks.workunit.client.1.smithi196.stdout:1/335: truncate d1/db/dc/d26/d2e/f51 848930 0 2024-04-22T15:18:57.052 INFO:tasks.workunit.client.0.smithi096.stdout:2/706: readlink d3/db/df/dce/d46/d7e/d84/ldb 0 2024-04-22T15:18:57.053 INFO:tasks.workunit.client.1.smithi196.stdout:8/377: dwrite d2/dc/fd [0,4194304] 0 2024-04-22T15:18:57.055 INFO:tasks.workunit.client.1.smithi196.stdout:8/378: write d2/d3f/d4b/f72 [820472,31647] 0 2024-04-22T15:18:57.057 INFO:tasks.workunit.client.1.smithi196.stdout:4/387: write d6/d14/d18/d26/d32/d53/f54 [530323,12510] 0 2024-04-22T15:18:57.057 INFO:tasks.workunit.client.0.smithi096.stdout:4/737: dwrite d5/d7/d2c/d95/dee/f94 [0,4194304] 0 2024-04-22T15:18:57.062 INFO:tasks.workunit.client.0.smithi096.stdout:8/690: dwrite d4/d1c/d40/d46/d58/f7a [0,4194304] 0 2024-04-22T15:18:57.062 INFO:tasks.workunit.client.1.smithi196.stdout:4/388: dwrite d6/d13/f1b [0,4194304] 0 2024-04-22T15:18:57.062 INFO:tasks.workunit.client.1.smithi196.stdout:9/372: mknod db/dd/d1b/c89 0 2024-04-22T15:18:57.064 INFO:tasks.workunit.client.1.smithi196.stdout:9/373: write db/f5f [98096,109566] 0 2024-04-22T15:18:57.065 INFO:tasks.workunit.client.0.smithi096.stdout:8/691: chown d4/d1c/c9f 192 1 2024-04-22T15:18:57.067 INFO:tasks.workunit.client.1.smithi196.stdout:7/332: rename d14/d26/f54 to d14/d25/d31/f69 0 2024-04-22T15:18:57.070 INFO:tasks.workunit.client.1.smithi196.stdout:2/395: mkdir d1/d23/d27/d2e/d55/d5b/d93 0 2024-04-22T15:18:57.072 INFO:tasks.workunit.client.1.smithi196.stdout:0/397: getdents d3/d9/d36/d56/d65/d7f/d30 0 2024-04-22T15:18:57.072 INFO:tasks.workunit.client.1.smithi196.stdout:4/389: dwrite d6/d13/d15/d2c/d35/f3c [0,4194304] 0 2024-04-22T15:18:57.072 INFO:tasks.workunit.client.0.smithi096.stdout:5/717: read - d4/ddb/d3d/d35/fba zero size 2024-04-22T15:18:57.075 INFO:tasks.workunit.client.1.smithi196.stdout:6/397: creat dd/d1f/d37/d58/f6e x:0 0 0 2024-04-22T15:18:57.077 INFO:tasks.workunit.client.1.smithi196.stdout:1/336: mkdir d1/db/dc/de/d5a/d5b/d6e 0 2024-04-22T15:18:57.080 INFO:tasks.workunit.client.1.smithi196.stdout:6/398: write dd/d1f/d38/d40/f5f [905059,113548] 0 2024-04-22T15:18:57.084 INFO:tasks.workunit.client.1.smithi196.stdout:7/333: creat d14/d4a/f6a x:0 0 0 2024-04-22T15:18:57.086 INFO:tasks.workunit.client.1.smithi196.stdout:0/398: dwrite d3/fd [0,4194304] 0 2024-04-22T15:18:57.086 INFO:tasks.workunit.client.1.smithi196.stdout:2/396: fsync d1/d23/d27/d2e/f5d 0 2024-04-22T15:18:57.086 INFO:tasks.workunit.client.1.smithi196.stdout:7/334: chown d14/d25/d31 4 1 2024-04-22T15:18:57.086 INFO:tasks.workunit.client.1.smithi196.stdout:4/390: dwrite d6/d13/f68 [0,4194304] 0 2024-04-22T15:18:57.087 INFO:tasks.workunit.client.0.smithi096.stdout:1/759: rename d6/dc/d8a/d3f/d54/d32/d3b/d3d/d4e to d6/dc/d8a/d3f/de9 0 2024-04-22T15:18:57.089 INFO:tasks.workunit.client.1.smithi196.stdout:7/335: truncate d14/d15/f59 906327 0 2024-04-22T15:18:57.090 INFO:tasks.workunit.client.1.smithi196.stdout:3/353: getdents d0/d7/d8 0 2024-04-22T15:18:57.091 INFO:tasks.workunit.client.1.smithi196.stdout:3/354: chown d0/d7/c3d 15555 1 2024-04-22T15:18:57.091 INFO:tasks.workunit.client.0.smithi096.stdout:5/718: chown d4/ddb/d3d/d4d/f5d 1376 1 2024-04-22T15:18:57.092 INFO:tasks.workunit.client.0.smithi096.stdout:9/638: link d1/df/d95/d9d/db5/d43/d57/d5b/d8e/faa d1/df/d95/d9d/db5/d43/fd8 0 2024-04-22T15:18:57.094 INFO:tasks.workunit.client.0.smithi096.stdout:7/953: mknod d31/d32/d33/c147 0 2024-04-22T15:18:57.095 INFO:tasks.workunit.client.1.smithi196.stdout:1/337: write d1/db/f29 [3554845,98241] 0 2024-04-22T15:18:57.096 INFO:tasks.workunit.client.0.smithi096.stdout:7/954: fdatasync d31/d32/d36/f121 0 2024-04-22T15:18:57.099 INFO:tasks.workunit.client.1.smithi196.stdout:9/374: mknod db/d11/d29/d7a/d7b/d81/c8a 0 2024-04-22T15:18:57.101 INFO:tasks.workunit.client.1.smithi196.stdout:9/375: read db/dd/de/d75/f86 [2990821,114027] 0 2024-04-22T15:18:57.102 INFO:tasks.workunit.client.1.smithi196.stdout:9/376: fdatasync db/d11/d29/d7a/f7e 0 2024-04-22T15:18:57.102 INFO:tasks.workunit.client.1.smithi196.stdout:6/399: mknod dd/d1f/d4e/c6f 0 2024-04-22T15:18:57.103 INFO:tasks.workunit.client.0.smithi096.stdout:1/760: dwrite d6/dc/f81 [4194304,4194304] 0 2024-04-22T15:18:57.103 INFO:tasks.workunit.client.0.smithi096.stdout:7/955: dread - d31/d32/d33/d93/dd5/f140 zero size 2024-04-22T15:18:57.104 INFO:tasks.workunit.client.1.smithi196.stdout:6/400: stat f9 0 2024-04-22T15:18:57.104 INFO:tasks.workunit.client.1.smithi196.stdout:9/377: stat db/d11/d29/d7a/d7b/d81 0 2024-04-22T15:18:57.104 INFO:tasks.workunit.client.1.smithi196.stdout:6/401: fsync fc 0 2024-04-22T15:18:57.105 INFO:tasks.workunit.client.0.smithi096.stdout:2/707: dread d3/db/f31 [0,4194304] 0 2024-04-22T15:18:57.105 INFO:tasks.workunit.client.0.smithi096.stdout:9/639: truncate d1/df/d95/d9d/db5/dd/dba/dc2/fc6 778053 0 2024-04-22T15:18:57.105 INFO:tasks.workunit.client.0.smithi096.stdout:4/738: unlink d5/d7/d1f/f50 0 2024-04-22T15:18:57.106 INFO:tasks.workunit.client.0.smithi096.stdout:3/626: creat da/db/d12/d22/fc9 x:0 0 0 2024-04-22T15:18:57.108 INFO:tasks.workunit.client.0.smithi096.stdout:0/852: rename d7/d42/d4c/d55/d98 to d7/d42/d4c/d55/d9c/df7/dfb/d122 0 2024-04-22T15:18:57.110 INFO:tasks.workunit.client.0.smithi096.stdout:0/853: fdatasync d7/de/d5b/fe2 0 2024-04-22T15:18:57.114 INFO:tasks.workunit.client.0.smithi096.stdout:3/627: dread da/d3f/fa5 [0,4194304] 0 2024-04-22T15:18:57.114 INFO:tasks.workunit.client.1.smithi196.stdout:2/397: read d1/f16 [1202708,13601] 0 2024-04-22T15:18:57.115 INFO:tasks.workunit.client.1.smithi196.stdout:0/399: rename d3/de/d14/d29 to d3/d9/d36/d56/d65/d7f/d87 0 2024-04-22T15:18:57.116 INFO:tasks.workunit.client.1.smithi196.stdout:2/398: write d1/d13/d1f/f60 [40910,77810] 0 2024-04-22T15:18:57.116 INFO:tasks.workunit.client.1.smithi196.stdout:5/368: sync 2024-04-22T15:18:57.117 INFO:tasks.workunit.client.1.smithi196.stdout:2/399: chown d1/d23/d27/d2e 69149 1 2024-04-22T15:18:57.123 INFO:tasks.workunit.client.0.smithi096.stdout:7/956: unlink d31/d32/d33/d44/fdd 0 2024-04-22T15:18:57.124 INFO:tasks.workunit.client.1.smithi196.stdout:0/400: dwrite d3/de/d1c/d27/f38 [0,4194304] 0 2024-04-22T15:18:57.125 INFO:tasks.workunit.client.1.smithi196.stdout:3/355: mknod d0/d15/d4d/db/dc/d18/c78 0 2024-04-22T15:18:57.126 INFO:tasks.workunit.client.1.smithi196.stdout:3/356: chown d0/d15/l14 0 1 2024-04-22T15:18:57.127 INFO:tasks.workunit.client.1.smithi196.stdout:3/357: read - d0/d15/d31/d55/f74 zero size 2024-04-22T15:18:57.129 INFO:tasks.workunit.client.1.smithi196.stdout:8/379: creat d2/d5/f74 x:0 0 0 2024-04-22T15:18:57.129 INFO:tasks.workunit.client.1.smithi196.stdout:3/358: dread - d0/d15/d4d/db/dc/d4a/f46 zero size 2024-04-22T15:18:57.129 INFO:tasks.workunit.client.0.smithi096.stdout:8/692: write d4/d1c/d40/d46/d58/fb5 [1184723,63569] 0 2024-04-22T15:18:57.130 INFO:tasks.workunit.client.1.smithi196.stdout:8/380: chown d2/dc/fd 20941429 1 2024-04-22T15:18:57.135 INFO:tasks.workunit.client.1.smithi196.stdout:8/381: dwrite d2/dc/f10 [0,4194304] 0 2024-04-22T15:18:57.137 INFO:tasks.workunit.client.1.smithi196.stdout:8/382: write d2/d8/f73 [976193,127837] 0 2024-04-22T15:18:57.140 INFO:tasks.workunit.client.0.smithi096.stdout:2/708: dwrite d3/d1f/f56 [4194304,4194304] 0 2024-04-22T15:18:57.142 INFO:tasks.workunit.client.0.smithi096.stdout:9/640: dread d1/df/d95/d9d/db5/d43/f44 [0,4194304] 0 2024-04-22T15:18:57.145 INFO:tasks.workunit.client.1.smithi196.stdout:6/402: write dd/f3a [1533108,127869] 0 2024-04-22T15:18:57.145 INFO:tasks.workunit.client.0.smithi096.stdout:1/761: dread d6/dc/d8a/d3f/d54/d32/d3b/f6f [4194304,4194304] 0 2024-04-22T15:18:57.146 INFO:tasks.workunit.client.1.smithi196.stdout:6/403: fsync dd/d1f/d38/d40/f43 0 2024-04-22T15:18:57.146 INFO:tasks.workunit.client.0.smithi096.stdout:1/762: stat d6/fcf 0 2024-04-22T15:18:57.149 INFO:tasks.workunit.client.0.smithi096.stdout:3/628: dread da/db/d12/fa9 [0,4194304] 0 2024-04-22T15:18:57.150 INFO:tasks.workunit.client.1.smithi196.stdout:9/378: rename db/dd/de/l56 to db/d11/d29/l8b 0 2024-04-22T15:18:57.151 INFO:tasks.workunit.client.1.smithi196.stdout:9/379: readlink db/l27 0 2024-04-22T15:18:57.151 INFO:tasks.workunit.client.1.smithi196.stdout:9/380: stat db/dd/de/c26 0 2024-04-22T15:18:57.153 INFO:tasks.workunit.client.1.smithi196.stdout:5/369: unlink d1/d9/l3a 0 2024-04-22T15:18:57.156 INFO:tasks.workunit.client.0.smithi096.stdout:4/739: symlink d5/d7/d2c/d3d/lff 0 2024-04-22T15:18:57.157 INFO:tasks.workunit.client.1.smithi196.stdout:2/400: symlink d1/d23/d27/d2e/d45/l94 0 2024-04-22T15:18:57.158 INFO:tasks.workunit.client.0.smithi096.stdout:8/693: symlink d4/d10/d25/d4c/db4/le8 0 2024-04-22T15:18:57.158 INFO:tasks.workunit.client.0.smithi096.stdout:4/740: fdatasync d5/d7/d2c/ff0 0 2024-04-22T15:18:57.161 INFO:tasks.workunit.client.0.smithi096.stdout:5/719: dwrite d4/ddb/f7a [4194304,4194304] 0 2024-04-22T15:18:57.165 INFO:tasks.workunit.client.1.smithi196.stdout:2/401: dread d1/f18 [0,4194304] 0 2024-04-22T15:18:57.167 INFO:tasks.workunit.client.0.smithi096.stdout:9/641: mkdir d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/dd9 0 2024-04-22T15:18:57.170 INFO:tasks.workunit.client.0.smithi096.stdout:0/854: creat d7/de/d5b/d95/dc/d16/d65/f123 x:0 0 0 2024-04-22T15:18:57.171 INFO:tasks.workunit.client.0.smithi096.stdout:0/855: write d7/de/d5b/d95/dc/d16/f1f [3463165,43465] 0 2024-04-22T15:18:57.175 INFO:tasks.workunit.client.0.smithi096.stdout:3/629: symlink da/db/de/d87/lca 0 2024-04-22T15:18:57.178 INFO:tasks.workunit.client.0.smithi096.stdout:7/957: write d31/d32/f3f [2868477,22185] 0 2024-04-22T15:18:57.178 INFO:tasks.workunit.client.0.smithi096.stdout:8/694: symlink d4/d2a/d7e/le9 0 2024-04-22T15:18:57.183 INFO:tasks.workunit.client.0.smithi096.stdout:5/720: mkdir d4/d2c/d2f/d31/d82/def 0 2024-04-22T15:18:57.189 INFO:tasks.workunit.client.0.smithi096.stdout:9/642: mknod d1/df/d95/d9d/db5/d43/d57/d5b/da1/cda 0 2024-04-22T15:18:57.191 INFO:tasks.workunit.client.0.smithi096.stdout:0/856: symlink d7/de/d5b/d95/d15/l124 0 2024-04-22T15:18:57.193 INFO:tasks.workunit.client.0.smithi096.stdout:3/630: write da/f8f [637498,61319] 0 2024-04-22T15:18:57.193 INFO:tasks.workunit.client.1.smithi196.stdout:4/391: truncate d6/d13/d15/d2c/d35/d44/f57 2496112 0 2024-04-22T15:18:57.196 INFO:tasks.workunit.client.1.smithi196.stdout:9/381: unlink db/d11/l13 0 2024-04-22T15:18:57.199 INFO:tasks.workunit.client.1.smithi196.stdout:7/336: creat d14/f6b x:0 0 0 2024-04-22T15:18:57.200 INFO:tasks.workunit.client.0.smithi096.stdout:4/741: dwrite d5/d7/d1a/f90 [0,4194304] 0 2024-04-22T15:18:57.204 INFO:tasks.workunit.client.1.smithi196.stdout:3/359: dwrite d0/d15/d4d/f5a [4194304,4194304] 0 2024-04-22T15:18:57.204 INFO:tasks.workunit.client.1.smithi196.stdout:7/337: dread d14/d25/f3b [0,4194304] 0 2024-04-22T15:18:57.205 INFO:tasks.workunit.client.0.smithi096.stdout:5/721: dread d4/ddb/d3d/d4d/d59/f7f [0,4194304] 0 2024-04-22T15:18:57.206 INFO:tasks.workunit.client.0.smithi096.stdout:5/722: write d4/d2c/d2f/d30/fd6 [1038030,88320] 0 2024-04-22T15:18:57.206 INFO:tasks.workunit.client.1.smithi196.stdout:9/382: dwrite db/ff [0,4194304] 0 2024-04-22T15:18:57.207 INFO:tasks.workunit.client.0.smithi096.stdout:2/709: dread d3/db/df/dce/d46/d69/fe1 [0,4194304] 0 2024-04-22T15:18:57.293 INFO:tasks.workunit.client.1.smithi196.stdout:0/401: mkdir d3/d9/d36/d56/d65/d7f/d87/d6c/d88 0 2024-04-22T15:18:57.300 INFO:tasks.workunit.client.0.smithi096.stdout:7/958: write d31/d32/d33/da2/fb9 [272843,56820] 0 2024-04-22T15:18:57.300 INFO:tasks.workunit.client.1.smithi196.stdout:8/383: creat d2/d8/d49/d6c/f75 x:0 0 0 2024-04-22T15:18:57.301 INFO:tasks.workunit.client.0.smithi096.stdout:7/959: chown d31/d32/d36/d75/dfe/f117 1359 1 2024-04-22T15:18:57.301 INFO:tasks.workunit.client.1.smithi196.stdout:8/384: read d2/d8/d49/f59 [862810,30500] 0 2024-04-22T15:18:57.302 INFO:tasks.workunit.client.1.smithi196.stdout:6/404: fsync f5 0 2024-04-22T15:18:57.305 INFO:tasks.workunit.client.1.smithi196.stdout:5/370: mkdir d1/d78 0 2024-04-22T15:18:57.306 INFO:tasks.workunit.client.0.smithi096.stdout:7/960: dwrite d31/d32/f3f [0,4194304] 0 2024-04-22T15:18:57.307 INFO:tasks.workunit.client.1.smithi196.stdout:8/385: dwrite d2/d8/d47/f60 [0,4194304] 0 2024-04-22T15:18:57.308 INFO:tasks.workunit.client.1.smithi196.stdout:7/338: creat d14/d26/d28/d29/f6c x:0 0 0 2024-04-22T15:18:57.311 INFO:tasks.workunit.client.1.smithi196.stdout:9/383: dread db/d11/d29/d57/f5a [0,4194304] 0 2024-04-22T15:18:57.311 INFO:tasks.workunit.client.1.smithi196.stdout:9/384: stat db/dd/de/c44 0 2024-04-22T15:18:57.313 INFO:tasks.workunit.client.1.smithi196.stdout:3/360: rename d0/d15/d4d/db/dc/l41 to d0/d15/d4d/db/dc/l79 0 2024-04-22T15:18:57.320 INFO:tasks.workunit.client.1.smithi196.stdout:1/338: link d1/db/d16/d44/d6a/f50 d1/db/dc/f6f 0 2024-04-22T15:18:57.326 INFO:tasks.workunit.client.0.smithi096.stdout:1/763: rename d6/dc/d8a/d3f/d54/d32/d82/fc0 to d6/fea 0 2024-04-22T15:18:57.326 INFO:tasks.workunit.client.1.smithi196.stdout:1/339: dwrite d1/db/d16/f5c [4194304,4194304] 0 2024-04-22T15:18:57.327 INFO:tasks.workunit.client.1.smithi196.stdout:6/405: mkdir dd/d70 0 2024-04-22T15:18:57.328 INFO:tasks.workunit.client.0.smithi096.stdout:9/643: creat d1/df/d95/d9d/db5/d43/d57/d5b/fdb x:0 0 0 2024-04-22T15:18:57.329 INFO:tasks.workunit.client.0.smithi096.stdout:0/857: dread - d7/de/d5b/d95/dc/d16/d18/f121 zero size 2024-04-22T15:18:57.331 INFO:tasks.workunit.client.1.smithi196.stdout:4/392: dwrite d6/d14/f33 [0,4194304] 0 2024-04-22T15:18:57.333 INFO:tasks.workunit.client.1.smithi196.stdout:4/393: chown d6/d13/d15/f5e 369183161 1 2024-04-22T15:18:57.335 INFO:tasks.workunit.client.1.smithi196.stdout:4/394: write d6/d14/d18/d40/f6c [311203,92794] 0 2024-04-22T15:18:57.335 INFO:tasks.workunit.client.1.smithi196.stdout:8/386: truncate d2/dc/f5e 9505 0 2024-04-22T15:18:57.337 INFO:tasks.workunit.client.1.smithi196.stdout:4/395: chown d6/d13/d15/f30 0 1 2024-04-22T15:18:57.340 INFO:tasks.workunit.client.1.smithi196.stdout:9/385: write db/dd/d1b/d2c/f43 [4427231,94438] 0 2024-04-22T15:18:57.340 INFO:tasks.workunit.client.0.smithi096.stdout:4/742: mknod d5/d7/d1a/d8b/c100 0 2024-04-22T15:18:57.341 INFO:tasks.workunit.client.1.smithi196.stdout:3/361: creat d0/d15/f7a x:0 0 0 2024-04-22T15:18:57.342 INFO:tasks.workunit.client.1.smithi196.stdout:3/362: chown d0/d15/d4d/db/dc/d4a/f45 172 1 2024-04-22T15:18:57.343 INFO:tasks.workunit.client.1.smithi196.stdout:2/402: truncate d1/f18 914132 0 2024-04-22T15:18:57.344 INFO:tasks.workunit.client.1.smithi196.stdout:2/403: chown d1/d23/d27 137 1 2024-04-22T15:18:57.344 INFO:tasks.workunit.client.0.smithi096.stdout:9/644: dread d1/d25/db2/fb8 [0,4194304] 0 2024-04-22T15:18:57.345 INFO:tasks.workunit.client.0.smithi096.stdout:2/710: mknod d3/db/df/d24/d58/d5f/ce4 0 2024-04-22T15:18:57.345 INFO:tasks.workunit.client.1.smithi196.stdout:0/402: write d3/f11 [3488876,58373] 0 2024-04-22T15:18:57.348 INFO:tasks.workunit.client.0.smithi096.stdout:7/961: mknod d31/d32/d33/d54/d43/c148 0 2024-04-22T15:18:57.352 INFO:tasks.workunit.client.0.smithi096.stdout:0/858: truncate d7/de/fe8 693497 0 2024-04-22T15:18:57.354 INFO:tasks.workunit.client.1.smithi196.stdout:1/340: dread d1/db/d16/f5c [0,4194304] 0 2024-04-22T15:18:57.355 INFO:tasks.workunit.client.1.smithi196.stdout:7/339: write d14/d15/f1a [1261166,65326] 0 2024-04-22T15:18:57.356 INFO:tasks.workunit.client.1.smithi196.stdout:7/340: chown d14/f22 4546328 1 2024-04-22T15:18:57.356 INFO:tasks.workunit.client.1.smithi196.stdout:1/341: write d1/db/dc/d26/d2e/d32/d36/f3f [1630582,39610] 0 2024-04-22T15:18:57.357 INFO:tasks.workunit.client.0.smithi096.stdout:5/723: write d4/f6 [4933498,89987] 0 2024-04-22T15:18:57.360 INFO:tasks.workunit.client.1.smithi196.stdout:8/387: creat d2/d8/f76 x:0 0 0 2024-04-22T15:18:57.360 INFO:tasks.workunit.client.0.smithi096.stdout:2/711: dread d3/f53 [0,4194304] 0 2024-04-22T15:18:57.361 INFO:tasks.workunit.client.1.smithi196.stdout:8/388: read - d2/d5/f74 zero size 2024-04-22T15:18:57.363 INFO:tasks.workunit.client.1.smithi196.stdout:9/386: creat db/d11/d29/d7a/f8c x:0 0 0 2024-04-22T15:18:57.364 INFO:tasks.workunit.client.0.smithi096.stdout:1/764: symlink d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/leb 0 2024-04-22T15:18:57.366 INFO:tasks.workunit.client.1.smithi196.stdout:8/389: dwrite d2/d8/f27 [0,4194304] 0 2024-04-22T15:18:57.367 INFO:tasks.workunit.client.1.smithi196.stdout:8/390: dread - d2/d5/d4f/f6b zero size 2024-04-22T15:18:57.368 INFO:tasks.workunit.client.1.smithi196.stdout:3/363: dread d0/d15/d4d/db/dc/d4a/d44/f20 [0,4194304] 0 2024-04-22T15:18:57.383 INFO:tasks.workunit.client.0.smithi096.stdout:0/859: creat d7/de/d5b/d64/f125 x:0 0 0 2024-04-22T15:18:57.385 INFO:tasks.workunit.client.1.smithi196.stdout:2/404: creat d1/d13/d25/d2a/f95 x:0 0 0 2024-04-22T15:18:57.385 INFO:tasks.workunit.client.0.smithi096.stdout:3/631: link da/db/d12/d22/d26/db2/cbb da/db/de/d87/ccb 0 2024-04-22T15:18:57.386 INFO:tasks.workunit.client.0.smithi096.stdout:3/632: chown da/db/d12/d22/d26/d52/f6c 61474715 1 2024-04-22T15:18:57.386 INFO:tasks.workunit.client.1.smithi196.stdout:0/403: symlink d3/d9/d36/l89 0 2024-04-22T15:18:57.387 INFO:tasks.workunit.client.0.smithi096.stdout:8/695: link d4/d10/d25/f81 d4/d10/d25/fea 0 2024-04-22T15:18:57.388 INFO:tasks.workunit.client.0.smithi096.stdout:4/743: symlink d5/d7/d2c/d3d/def/l101 0 2024-04-22T15:18:57.389 INFO:tasks.workunit.client.1.smithi196.stdout:5/371: creat d1/d17/f79 x:0 0 0 2024-04-22T15:18:57.390 INFO:tasks.workunit.client.0.smithi096.stdout:5/724: creat d4/ddb/d3d/d4d/d59/d98/db9/ff0 x:0 0 0 2024-04-22T15:18:57.390 INFO:tasks.workunit.client.1.smithi196.stdout:4/396: dwrite d6/d13/f1b [4194304,4194304] 0 2024-04-22T15:18:57.393 INFO:tasks.workunit.client.1.smithi196.stdout:1/342: mkdir d1/d54/d70 0 2024-04-22T15:18:57.393 INFO:tasks.workunit.client.0.smithi096.stdout:7/962: mkdir d31/d32/d33/d44/da8/dbf/d149 0 2024-04-22T15:18:57.395 INFO:tasks.workunit.client.1.smithi196.stdout:9/387: creat db/dd/f8d x:0 0 0 2024-04-22T15:18:57.398 INFO:tasks.workunit.client.1.smithi196.stdout:8/391: truncate d2/dc/f28 923843 0 2024-04-22T15:18:57.399 INFO:tasks.workunit.client.0.smithi096.stdout:8/696: unlink d4/d10/c67 0 2024-04-22T15:18:57.404 INFO:tasks.workunit.client.0.smithi096.stdout:5/725: creat d4/ddb/d3d/d4d/d59/d98/ff1 x:0 0 0 2024-04-22T15:18:57.405 INFO:tasks.workunit.client.0.smithi096.stdout:9/645: link d1/df/fa7 d1/df/d95/fdc 0 2024-04-22T15:18:57.406 INFO:tasks.workunit.client.1.smithi196.stdout:0/404: rmdir d3 39 2024-04-22T15:18:57.407 INFO:tasks.workunit.client.0.smithi096.stdout:1/765: mknod d6/dc/d8a/d3f/d54/cec 0 2024-04-22T15:18:57.410 INFO:tasks.workunit.client.0.smithi096.stdout:8/697: mkdir d4/d10/d4d/deb 0 2024-04-22T15:18:57.412 INFO:tasks.workunit.client.1.smithi196.stdout:5/372: mkdir d1/d17/d1a/d2f/d30/d6e/d7a 0 2024-04-22T15:18:57.414 INFO:tasks.workunit.client.0.smithi096.stdout:4/744: unlink d5/d7/d1a/l48 0 2024-04-22T15:18:57.414 INFO:tasks.workunit.client.0.smithi096.stdout:8/698: dread d4/d1c/d40/d46/d58/f7a [0,4194304] 0 2024-04-22T15:18:57.414 INFO:tasks.workunit.client.0.smithi096.stdout:4/745: readlink d5/d7/d2c/l92 0 2024-04-22T15:18:57.415 INFO:tasks.workunit.client.1.smithi196.stdout:4/397: mknod d6/d13/d15/d2c/d35/d44/c72 0 2024-04-22T15:18:57.415 INFO:tasks.workunit.client.0.smithi096.stdout:4/746: chown d5/d5b/fd0 80272 1 2024-04-22T15:18:57.416 INFO:tasks.workunit.client.0.smithi096.stdout:4/747: stat d5/d7/d2c/d3d/d4f/fb6 0 2024-04-22T15:18:57.417 INFO:tasks.workunit.client.0.smithi096.stdout:4/748: truncate d5/d7/d1f/fec 157227 0 2024-04-22T15:18:57.417 INFO:tasks.workunit.client.1.smithi196.stdout:1/343: creat d1/db/dc/d26/f71 x:0 0 0 2024-04-22T15:18:57.418 INFO:tasks.workunit.client.0.smithi096.stdout:4/749: fdatasync d5/d7/d1a/f90 0 2024-04-22T15:18:57.419 INFO:tasks.workunit.client.0.smithi096.stdout:9/646: creat d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/fdd x:0 0 0 2024-04-22T15:18:57.420 INFO:tasks.workunit.client.1.smithi196.stdout:9/388: rmdir db/dd/d1b 39 2024-04-22T15:18:57.422 INFO:tasks.workunit.client.0.smithi096.stdout:0/860: dwrite d7/de/d5b/d95/dc/d26/fb5 [0,4194304] 0 2024-04-22T15:18:57.423 INFO:tasks.workunit.client.0.smithi096.stdout:0/861: dread - d7/de/d5b/d95/dc/f96 zero size 2024-04-22T15:18:57.423 INFO:tasks.workunit.client.0.smithi096.stdout:1/766: unlink d6/dc/d8a/d3f/d54/f98 0 2024-04-22T15:18:57.425 INFO:tasks.workunit.client.1.smithi196.stdout:8/392: mknod d2/d3f/c77 0 2024-04-22T15:18:57.427 INFO:tasks.workunit.client.1.smithi196.stdout:8/393: truncate d2/d5/d4f/f6e 778054 0 2024-04-22T15:18:57.427 INFO:tasks.workunit.client.1.smithi196.stdout:8/394: truncate d2/d8/f73 1833401 0 2024-04-22T15:18:57.429 INFO:tasks.workunit.client.1.smithi196.stdout:3/364: mknod d0/d15/d31/d55/d5d/c7b 0 2024-04-22T15:18:57.434 INFO:tasks.workunit.client.1.smithi196.stdout:6/406: sync 2024-04-22T15:18:57.435 INFO:tasks.workunit.client.1.smithi196.stdout:8/395: dwrite d2/d5/d4f/f6e [0,4194304] 0 2024-04-22T15:18:57.438 INFO:tasks.workunit.client.1.smithi196.stdout:3/365: dwrite d0/d15/d4d/db/dc/f47 [0,4194304] 0 2024-04-22T15:18:57.441 INFO:tasks.workunit.client.1.smithi196.stdout:1/344: stat d1/db/dc/d26/d2e/f34 0 2024-04-22T15:18:57.443 INFO:tasks.workunit.client.1.smithi196.stdout:9/389: write db/d11/d29/d57/f33 [608731,8633] 0 2024-04-22T15:18:57.444 INFO:tasks.workunit.client.1.smithi196.stdout:9/390: dread - db/dd/de/d15/d31/d64/f62 zero size 2024-04-22T15:18:57.445 INFO:tasks.workunit.client.1.smithi196.stdout:9/391: stat db/dd/de/d15/d31/f63 0 2024-04-22T15:18:57.448 INFO:tasks.workunit.client.1.smithi196.stdout:0/405: symlink d3/d9/d36/d56/l8a 0 2024-04-22T15:18:57.450 INFO:tasks.workunit.client.0.smithi096.stdout:8/699: mkdir d4/d1c/d62/dc3/dec 0 2024-04-22T15:18:57.451 INFO:tasks.workunit.client.1.smithi196.stdout:7/341: getdents d14/d15 0 2024-04-22T15:18:57.452 INFO:tasks.workunit.client.1.smithi196.stdout:0/406: dread d3/d9/d36/d56/f7b [0,4194304] 0 2024-04-22T15:18:57.453 INFO:tasks.workunit.client.0.smithi096.stdout:5/726: creat d4/ddb/ff2 x:0 0 0 2024-04-22T15:18:57.455 INFO:tasks.workunit.client.1.smithi196.stdout:5/373: getdents d1/d14/d6a 0 2024-04-22T15:18:57.455 INFO:tasks.workunit.client.1.smithi196.stdout:7/342: dread d14/d15/f1a [0,4194304] 0 2024-04-22T15:18:57.457 INFO:tasks.workunit.client.1.smithi196.stdout:6/407: symlink dd/d1f/d4e/d53/d55/l71 0 2024-04-22T15:18:57.457 INFO:tasks.workunit.client.1.smithi196.stdout:5/374: stat d1/d17/d1a/d2f 0 2024-04-22T15:18:57.457 INFO:tasks.workunit.client.0.smithi096.stdout:0/862: dwrite d7/de/d5b/d95/dc/f96 [0,4194304] 0 2024-04-22T15:18:57.458 INFO:tasks.workunit.client.1.smithi196.stdout:2/405: dwrite d1/f18 [0,4194304] 0 2024-04-22T15:18:57.459 INFO:tasks.workunit.client.0.smithi096.stdout:8/700: write d4/d10/d25/d4c/f5d [4712085,113581] 0 2024-04-22T15:18:57.460 INFO:tasks.workunit.client.1.smithi196.stdout:8/396: rename d2/c9 to d2/d8/d47/d68/c78 0 2024-04-22T15:18:57.460 INFO:tasks.workunit.client.1.smithi196.stdout:8/397: chown d2/d8/l21 10 1 2024-04-22T15:18:57.461 INFO:tasks.workunit.client.1.smithi196.stdout:3/366: symlink d0/d15/d48/l7c 0 2024-04-22T15:18:57.463 INFO:tasks.workunit.client.1.smithi196.stdout:1/345: fdatasync d1/db/dc/de/f4a 0 2024-04-22T15:18:57.463 INFO:tasks.workunit.client.0.smithi096.stdout:5/727: dwrite d4/ddb/d3d/d4d/d59/d98/ff1 [0,4194304] 0 2024-04-22T15:18:57.463 INFO:tasks.workunit.client.0.smithi096.stdout:6/581: sync 2024-04-22T15:18:57.464 INFO:tasks.workunit.client.1.smithi196.stdout:9/392: fdatasync db/dd/de/f42 0 2024-04-22T15:18:57.465 INFO:tasks.workunit.client.0.smithi096.stdout:6/582: fdatasync d3/d28/dd/d11/d2c/f4e 0 2024-04-22T15:18:57.467 INFO:tasks.workunit.client.0.smithi096.stdout:1/767: creat d6/dc/d8a/d3f/d54/d32/fed x:0 0 0 2024-04-22T15:18:57.473 INFO:tasks.workunit.client.0.smithi096.stdout:8/701: fdatasync d4/d1c/d40/f75 0 2024-04-22T15:18:57.474 INFO:tasks.workunit.client.0.smithi096.stdout:9/647: link d1/df/f65 d1/d25/d9f/fde 0 2024-04-22T15:18:57.474 INFO:tasks.workunit.client.0.smithi096.stdout:8/702: readlink d4/d10/l7c 0 2024-04-22T15:18:57.476 INFO:tasks.workunit.client.0.smithi096.stdout:9/648: write d1/df/d95/d9d/db5/d43/d57/d5b/fdb [878706,106252] 0 2024-04-22T15:18:57.477 INFO:tasks.workunit.client.0.smithi096.stdout:4/750: getdents d5/d7/d2c/d3d 0 2024-04-22T15:18:57.479 INFO:tasks.workunit.client.0.smithi096.stdout:9/649: dread - d1/df/d95/d9d/db5/d43/d57/d5b/da1/fd3 zero size 2024-04-22T15:18:57.482 INFO:tasks.workunit.client.0.smithi096.stdout:8/703: dwrite d4/d1c/d40/d46/d58/fb5 [0,4194304] 0 2024-04-22T15:18:57.484 INFO:tasks.workunit.client.1.smithi196.stdout:6/408: symlink dd/d1f/d37/d58/l72 0 2024-04-22T15:18:57.485 INFO:tasks.workunit.client.1.smithi196.stdout:6/409: chown dd/d20 11776059 1 2024-04-22T15:18:57.486 INFO:tasks.workunit.client.0.smithi096.stdout:0/863: mknod d7/de/d5b/dda/c126 0 2024-04-22T15:18:57.486 INFO:tasks.workunit.client.1.smithi196.stdout:2/406: fsync d1/f16 0 2024-04-22T15:18:57.486 INFO:tasks.workunit.client.1.smithi196.stdout:6/410: write dd/d1f/d38/d40/f43 [4119440,96363] 0 2024-04-22T15:18:57.488 INFO:tasks.workunit.client.0.smithi096.stdout:1/768: mknod d6/dc/d8a/d3f/d54/d32/d3b/d94/cee 0 2024-04-22T15:18:57.489 INFO:tasks.workunit.client.1.smithi196.stdout:8/398: mkdir d2/d3f/d4b/d79 0 2024-04-22T15:18:57.492 INFO:tasks.workunit.client.1.smithi196.stdout:3/367: rmdir d0/d15/d4d/db/dc 39 2024-04-22T15:18:57.494 INFO:tasks.workunit.client.0.smithi096.stdout:0/864: dwrite d7/f78 [0,4194304] 0 2024-04-22T15:18:57.495 INFO:tasks.workunit.client.0.smithi096.stdout:0/865: write d7/d42/f7a [638370,66621] 0 2024-04-22T15:18:57.495 INFO:tasks.workunit.client.1.smithi196.stdout:1/346: creat d1/db/dc/de/f72 x:0 0 0 2024-04-22T15:18:57.496 INFO:tasks.workunit.client.1.smithi196.stdout:1/347: fsync d1/db/d16/d1e/f60 0 2024-04-22T15:18:57.496 INFO:tasks.workunit.client.1.smithi196.stdout:9/393: dread db/dd/de/d15/d31/f63 [0,4194304] 0 2024-04-22T15:18:57.498 INFO:tasks.workunit.client.1.smithi196.stdout:1/348: truncate d1/db/dc/d26/d2e/f34 8813785 0 2024-04-22T15:18:57.498 INFO:tasks.workunit.client.0.smithi096.stdout:4/751: unlink d5/d7/d2c/d67/l7c 0 2024-04-22T15:18:57.501 INFO:tasks.workunit.client.0.smithi096.stdout:9/650: creat d1/d4/d7b/fdf x:0 0 0 2024-04-22T15:18:57.503 INFO:tasks.workunit.client.0.smithi096.stdout:9/651: read d1/df/d95/d9d/db5/d43/f44 [3523929,88216] 0 2024-04-22T15:18:57.504 INFO:tasks.workunit.client.0.smithi096.stdout:8/704: rmdir d4/d10/d25/d38/d8f 39 2024-04-22T15:18:57.506 INFO:tasks.workunit.client.0.smithi096.stdout:6/583: dwrite f2 [0,4194304] 0 2024-04-22T15:18:57.507 INFO:tasks.workunit.client.1.smithi196.stdout:0/407: symlink d3/de/l8b 0 2024-04-22T15:18:57.509 INFO:tasks.workunit.client.1.smithi196.stdout:4/398: link d6/d13/d15/c1a d6/d13/d15/d2c/d35/c73 0 2024-04-22T15:18:57.511 INFO:tasks.workunit.client.0.smithi096.stdout:9/652: dwrite d1/df/d95/d9d/db5/dd/f46 [0,4194304] 0 2024-04-22T15:18:57.513 INFO:tasks.workunit.client.1.smithi196.stdout:2/407: symlink d1/d1d/d6a/l96 0 2024-04-22T15:18:57.513 INFO:tasks.workunit.client.0.smithi096.stdout:1/769: mkdir d6/dc/d8a/d3f/d54/d32/d3b/def 0 2024-04-22T15:18:57.514 INFO:tasks.workunit.client.0.smithi096.stdout:1/770: dread - d6/dc/d8a/d3f/d54/d32/d8c/fe7 zero size 2024-04-22T15:18:57.514 INFO:tasks.workunit.client.1.smithi196.stdout:6/411: chown dd/d1f/d4e/c61 39814511 1 2024-04-22T15:18:57.515 INFO:tasks.workunit.client.0.smithi096.stdout:0/866: creat d7/d42/f127 x:0 0 0 2024-04-22T15:18:57.515 INFO:tasks.workunit.client.1.smithi196.stdout:8/399: fsync d2/dc/f18 0 2024-04-22T15:18:57.516 INFO:tasks.workunit.client.0.smithi096.stdout:4/752: creat d5/d7/d1a/d8b/f102 x:0 0 0 2024-04-22T15:18:57.518 INFO:tasks.workunit.client.1.smithi196.stdout:3/368: fdatasync d0/d15/d4d/db/dc/d18/f42 0 2024-04-22T15:18:57.519 INFO:tasks.workunit.client.0.smithi096.stdout:8/705: chown d4/d1c/d40/c82 522815 1 2024-04-22T15:18:57.519 INFO:tasks.workunit.client.1.smithi196.stdout:3/369: chown d0/d15/d4d/c1f 13348737 1 2024-04-22T15:18:57.520 INFO:tasks.workunit.client.0.smithi096.stdout:5/728: link d4/d1f/d80/c95 d4/ddb/d3d/d4d/d99/cf3 0 2024-04-22T15:18:57.521 INFO:tasks.workunit.client.0.smithi096.stdout:5/729: chown d4/d2c/d2f/d31/fca 638802 1 2024-04-22T15:18:57.522 INFO:tasks.workunit.client.1.smithi196.stdout:1/349: rmdir d1/d54 39 2024-04-22T15:18:57.523 INFO:tasks.workunit.client.0.smithi096.stdout:5/730: fsync d4/ddb/d3d/d4d/d59/d98/fd1 0 2024-04-22T15:18:57.524 INFO:tasks.workunit.client.1.smithi196.stdout:3/370: dread d0/d15/d4d/db/dc/d18/f4c [0,4194304] 0 2024-04-22T15:18:57.526 INFO:tasks.workunit.client.1.smithi196.stdout:0/408: fsync d3/d9/d36/d56/d65/f6b 0 2024-04-22T15:18:57.527 INFO:tasks.workunit.client.1.smithi196.stdout:0/409: write d3/f5 [2235634,58135] 0 2024-04-22T15:18:57.529 INFO:tasks.workunit.client.0.smithi096.stdout:8/706: symlink d4/d2a/d7e/led 0 2024-04-22T15:18:57.529 INFO:tasks.workunit.client.1.smithi196.stdout:0/410: write d3/de/d14/d5a/f68 [37146,120726] 0 2024-04-22T15:18:57.529 INFO:tasks.workunit.client.1.smithi196.stdout:0/411: fdatasync d3/d6/f48 0 2024-04-22T15:18:57.531 INFO:tasks.workunit.client.1.smithi196.stdout:4/399: mkdir d6/d14/d18/d40/d74 0 2024-04-22T15:18:57.531 INFO:tasks.workunit.client.0.smithi096.stdout:6/584: symlink d3/d28/d4d/lc8 0 2024-04-22T15:18:57.532 INFO:tasks.workunit.client.1.smithi196.stdout:7/343: write f10 [682325,5942] 0 2024-04-22T15:18:57.532 INFO:tasks.workunit.client.1.smithi196.stdout:7/344: fdatasync d14/d25/f42 0 2024-04-22T15:18:57.533 INFO:tasks.workunit.client.1.smithi196.stdout:7/345: dread - d14/d25/d31/f63 zero size 2024-04-22T15:18:57.537 INFO:tasks.workunit.client.1.smithi196.stdout:6/412: creat dd/d1f/d4e/f73 x:0 0 0 2024-04-22T15:18:57.538 INFO:tasks.workunit.client.0.smithi096.stdout:6/585: dwrite f1 [0,4194304] 0 2024-04-22T15:18:57.538 INFO:tasks.workunit.client.0.smithi096.stdout:4/753: mkdir d5/d7/d1a/d103 0 2024-04-22T15:18:57.540 INFO:tasks.workunit.client.1.smithi196.stdout:9/394: write db/dd/de/d15/f40 [5907335,100439] 0 2024-04-22T15:18:57.541 INFO:tasks.workunit.client.1.smithi196.stdout:0/412: dread d3/d9/d36/d56/d65/d7f/d30/f78 [0,4194304] 0 2024-04-22T15:18:57.541 INFO:tasks.workunit.client.1.smithi196.stdout:9/395: dread db/d11/d29/d57/f5a [0,4194304] 0 2024-04-22T15:18:57.542 INFO:tasks.workunit.client.1.smithi196.stdout:1/350: creat d1/db/d16/f73 x:0 0 0 2024-04-22T15:18:57.543 INFO:tasks.workunit.client.1.smithi196.stdout:3/371: unlink d0/d15/d4d/db/dc/d4a/f35 0 2024-04-22T15:18:57.544 INFO:tasks.workunit.client.1.smithi196.stdout:0/413: write d3/d9/d36/d56/d65/d7f/d51/f53 [477520,119617] 0 2024-04-22T15:18:57.545 INFO:tasks.workunit.client.1.smithi196.stdout:0/414: chown d3/de/f24 70 1 2024-04-22T15:18:57.548 INFO:tasks.workunit.client.1.smithi196.stdout:2/408: mkdir d1/d23/d27/d2e/d55/d5b/d93/d97 0 2024-04-22T15:18:57.549 INFO:tasks.workunit.client.1.smithi196.stdout:2/409: chown d1/d23/d27/f44 3111865 1 2024-04-22T15:18:57.549 INFO:tasks.workunit.client.0.smithi096.stdout:6/586: mkdir d3/d28/dc9 0 2024-04-22T15:18:57.550 INFO:tasks.workunit.client.1.smithi196.stdout:2/410: dread - d1/d13/d25/f3d zero size 2024-04-22T15:18:57.552 INFO:tasks.workunit.client.0.smithi096.stdout:6/587: write d3/d3a/d75/d83/f98 [349347,22865] 0 2024-04-22T15:18:57.552 INFO:tasks.workunit.client.0.smithi096.stdout:4/754: symlink d5/d7/d2c/d43/l104 0 2024-04-22T15:18:57.553 INFO:tasks.workunit.client.0.smithi096.stdout:1/771: dwrite d6/dc/d8a/d3f/d54/fa5 [0,4194304] 0 2024-04-22T15:18:57.555 INFO:tasks.workunit.client.0.smithi096.stdout:1/772: dread - d6/dc/d8a/d3f/de9/fe2 zero size 2024-04-22T15:18:57.555 INFO:tasks.workunit.client.0.smithi096.stdout:5/731: getdents d4/ddb/d3d/d4d/d59/db0/db2 0 2024-04-22T15:18:57.556 INFO:tasks.workunit.client.1.smithi196.stdout:2/411: dwrite d1/d23/d27/d2e/d55/d5b/d7c/f91 [0,4194304] 0 2024-04-22T15:18:57.560 INFO:tasks.workunit.client.0.smithi096.stdout:6/588: symlink d3/d3a/lca 0 2024-04-22T15:18:57.562 INFO:tasks.workunit.client.0.smithi096.stdout:4/755: chown d5/d7/d2c/d43/l45 681753683 1 2024-04-22T15:18:57.568 INFO:tasks.workunit.client.0.smithi096.stdout:5/732: mknod d4/ddb/d3d/d4d/d88/cf4 0 2024-04-22T15:18:57.570 INFO:tasks.workunit.client.1.smithi196.stdout:0/415: mkdir d3/d6/dc/d8c 0 2024-04-22T15:18:57.572 INFO:tasks.workunit.client.1.smithi196.stdout:7/346: mknod d14/d26/d5e/c6d 0 2024-04-22T15:18:57.573 INFO:tasks.workunit.client.0.smithi096.stdout:1/773: write d6/dc/d8a/f67 [2292371,90462] 0 2024-04-22T15:18:57.573 INFO:tasks.workunit.client.1.smithi196.stdout:7/347: chown d14/d15/c37 16185568 1 2024-04-22T15:18:57.574 INFO:tasks.workunit.client.1.smithi196.stdout:7/348: chown d14/d4d 32154 1 2024-04-22T15:18:57.574 INFO:tasks.workunit.client.0.smithi096.stdout:1/774: stat d6/dc/d8a/d3f/f73 0 2024-04-22T15:18:57.574 INFO:tasks.workunit.client.1.smithi196.stdout:7/349: write f7 [218081,36075] 0 2024-04-22T15:18:57.576 INFO:tasks.workunit.client.0.smithi096.stdout:8/707: dread d4/d2a/f36 [0,4194304] 0 2024-04-22T15:18:57.577 INFO:tasks.workunit.client.1.smithi196.stdout:1/351: symlink d1/l74 0 2024-04-22T15:18:57.579 INFO:tasks.workunit.client.1.smithi196.stdout:9/396: symlink db/dd/l8e 0 2024-04-22T15:18:57.580 INFO:tasks.workunit.client.0.smithi096.stdout:6/589: rename d3/d3a/l3d to d3/d28/dd/d11/d2c/lcb 0 2024-04-22T15:18:57.582 INFO:tasks.workunit.client.1.smithi196.stdout:3/372: link d0/d7/d8/f19 d0/d7/f7d 0 2024-04-22T15:18:57.583 INFO:tasks.workunit.client.0.smithi096.stdout:8/708: dread - d4/d10/f99 zero size 2024-04-22T15:18:57.584 INFO:tasks.workunit.client.1.smithi196.stdout:4/400: link d6/d13/d15/d2c/d35/d44/f60 d6/d13/d15/d2c/f75 0 2024-04-22T15:18:57.586 INFO:tasks.workunit.client.0.smithi096.stdout:6/590: rmdir d3/d28/d54/d68/d6e 39 2024-04-22T15:18:57.587 INFO:tasks.workunit.client.1.smithi196.stdout:6/413: write dd/d1f/d38/f51 [905892,15004] 0 2024-04-22T15:18:57.588 INFO:tasks.workunit.client.0.smithi096.stdout:6/591: fdatasync d3/d28/dd/d11/d2c/d6f/f89 0 2024-04-22T15:18:57.588 INFO:tasks.workunit.client.1.smithi196.stdout:6/414: chown dd/d1f/d4e/f73 1387971 1 2024-04-22T15:18:57.588 INFO:tasks.workunit.client.0.smithi096.stdout:6/592: fdatasync d3/d28/f1d 0 2024-04-22T15:18:57.589 INFO:tasks.workunit.client.0.smithi096.stdout:6/593: dread - d3/d28/dd/d11/f41 zero size 2024-04-22T15:18:57.590 INFO:tasks.workunit.client.0.smithi096.stdout:8/709: mknod d4/cee 0 2024-04-22T15:18:57.591 INFO:tasks.workunit.client.0.smithi096.stdout:8/710: write d4/d10/d25/d4c/d76/f89 [510386,2497] 0 2024-04-22T15:18:57.595 INFO:tasks.workunit.client.1.smithi196.stdout:5/375: sync 2024-04-22T15:18:57.595 INFO:tasks.workunit.client.1.smithi196.stdout:5/376: stat d1/d17/d50 0 2024-04-22T15:18:57.596 INFO:tasks.workunit.client.1.smithi196.stdout:4/401: write d6/d14/d18/d26/f43 [1349277,37600] 0 2024-04-22T15:18:57.597 INFO:tasks.workunit.client.1.smithi196.stdout:0/416: mkdir d3/d6/dc/d8c/d8d 0 2024-04-22T15:18:57.599 INFO:tasks.workunit.client.0.smithi096.stdout:1/775: dread d6/dc/d8a/d3f/d62/d7d/ddb/d9b/dba/fc8 [0,4194304] 0 2024-04-22T15:18:57.599 INFO:tasks.workunit.client.1.smithi196.stdout:7/350: mkdir d14/d26/d5a/d5f/d6e 0 2024-04-22T15:18:57.605 INFO:tasks.workunit.client.1.smithi196.stdout:1/352: dread d1/db/dc/de/f4a [0,4194304] 0 2024-04-22T15:18:57.606 INFO:tasks.workunit.client.0.smithi096.stdout:1/776: read d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fdf [899082,129067] 0 2024-04-22T15:18:57.606 INFO:tasks.workunit.client.1.smithi196.stdout:7/351: dwrite d14/d26/d28/d29/f68 [0,4194304] 0 2024-04-22T15:18:57.607 INFO:tasks.workunit.client.1.smithi196.stdout:5/377: rmdir d1/d17/d50 39 2024-04-22T15:18:57.608 INFO:tasks.workunit.client.1.smithi196.stdout:5/378: dread - d1/d4/f56 zero size 2024-04-22T15:18:57.609 INFO:tasks.workunit.client.1.smithi196.stdout:0/417: write d3/d9/f73 [1105980,91960] 0 2024-04-22T15:18:57.609 INFO:tasks.workunit.client.1.smithi196.stdout:5/379: chown d1/d17/c41 509650393 1 2024-04-22T15:18:57.611 INFO:tasks.workunit.client.1.smithi196.stdout:2/412: rename d1/d13/l67 to d1/l98 0 2024-04-22T15:18:57.612 INFO:tasks.workunit.client.1.smithi196.stdout:0/418: chown d3/de/d1c/f31 0 1 2024-04-22T15:18:57.612 INFO:tasks.workunit.client.0.smithi096.stdout:6/594: dwrite d3/d28/dd/f19 [4194304,4194304] 0 2024-04-22T15:18:57.613 INFO:tasks.workunit.client.0.smithi096.stdout:8/711: truncate d4/d10/d25/d38/f4b 751017 0 2024-04-22T15:18:57.615 INFO:tasks.workunit.client.0.smithi096.stdout:1/777: fsync d6/dc/d8a/d3f/d62/d7d/ddb/d9b/dba/fc8 0 2024-04-22T15:18:57.620 INFO:tasks.workunit.client.0.smithi096.stdout:6/595: creat d3/d28/d4d/d82/d9e/fcc x:0 0 0 2024-04-22T15:18:57.620 INFO:tasks.workunit.client.1.smithi196.stdout:1/353: read d1/db/d16/f40 [21158,2489] 0 2024-04-22T15:18:57.622 INFO:tasks.workunit.client.1.smithi196.stdout:3/373: dwrite d0/d7/f11 [0,4194304] 0 2024-04-22T15:18:57.623 INFO:tasks.workunit.client.1.smithi196.stdout:4/402: symlink d6/d14/d18/d26/d32/l76 0 2024-04-22T15:18:57.627 INFO:tasks.workunit.client.1.smithi196.stdout:6/415: creat dd/d1f/d37/f74 x:0 0 0 2024-04-22T15:18:57.631 INFO:tasks.workunit.client.1.smithi196.stdout:1/354: dwrite d1/db/d16/d1e/f60 [0,4194304] 0 2024-04-22T15:18:57.632 INFO:tasks.workunit.client.1.smithi196.stdout:1/355: dread - d1/db/dc/de/f72 zero size 2024-04-22T15:18:57.633 INFO:tasks.workunit.client.1.smithi196.stdout:8/400: sync 2024-04-22T15:18:57.633 INFO:tasks.workunit.client.1.smithi196.stdout:2/413: symlink d1/d23/d27/d89/l99 0 2024-04-22T15:18:57.635 INFO:tasks.workunit.client.1.smithi196.stdout:3/374: dwrite d0/d15/d4d/db/dc/d4a/f50 [0,4194304] 0 2024-04-22T15:18:57.636 INFO:tasks.workunit.client.1.smithi196.stdout:0/419: truncate d3/d9/d25/f63 3887362 0 2024-04-22T15:18:57.639 INFO:tasks.workunit.client.0.smithi096.stdout:1/778: write d6/dc/d8a/d3f/d54/d32/d3b/d56/fb9 [578792,109881] 0 2024-04-22T15:18:57.642 INFO:tasks.workunit.client.0.smithi096.stdout:8/712: dwrite d4/fd [4194304,4194304] 0 2024-04-22T15:18:57.642 INFO:tasks.workunit.client.1.smithi196.stdout:3/375: dwrite d0/d7/f38 [0,4194304] 0 2024-04-22T15:18:57.644 INFO:tasks.workunit.client.1.smithi196.stdout:6/416: symlink dd/d1f/d4e/l75 0 2024-04-22T15:18:57.648 INFO:tasks.workunit.client.1.smithi196.stdout:5/380: unlink d1/d9/lc 0 2024-04-22T15:18:57.649 INFO:tasks.workunit.client.1.smithi196.stdout:8/401: creat d2/d8/d49/f7a x:0 0 0 2024-04-22T15:18:57.652 INFO:tasks.workunit.client.1.smithi196.stdout:8/402: write d2/d8/f4c [389483,19838] 0 2024-04-22T15:18:57.654 INFO:tasks.workunit.client.1.smithi196.stdout:6/417: dread f9 [0,4194304] 0 2024-04-22T15:18:57.654 INFO:tasks.workunit.client.1.smithi196.stdout:9/397: getdents db/dd/d1b/d2c 0 2024-04-22T15:18:57.656 INFO:tasks.workunit.client.1.smithi196.stdout:3/376: read d0/d7/f1a [2466960,48589] 0 2024-04-22T15:18:57.656 INFO:tasks.workunit.client.1.smithi196.stdout:8/403: chown d2/d35/c5d 199446 1 2024-04-22T15:18:57.657 INFO:tasks.workunit.client.1.smithi196.stdout:4/403: link d6/d13/f25 d6/d14/d18/d26/d32/d67/f77 0 2024-04-22T15:18:57.659 INFO:tasks.workunit.client.1.smithi196.stdout:1/356: mknod d1/db/dc/c75 0 2024-04-22T15:18:57.661 INFO:tasks.workunit.client.0.smithi096.stdout:6/596: dread d3/d28/dd/d11/d2c/d3b/f79 [0,4194304] 0 2024-04-22T15:18:57.662 INFO:tasks.workunit.client.1.smithi196.stdout:5/381: dwrite d1/d17/f60 [0,4194304] 0 2024-04-22T15:18:57.663 INFO:tasks.workunit.client.1.smithi196.stdout:7/352: getdents d14/d26/d5a/d5f 0 2024-04-22T15:18:57.664 INFO:tasks.workunit.client.0.smithi096.stdout:6/597: rmdir d3/d28/d4d 39 2024-04-22T15:18:57.666 INFO:tasks.workunit.client.0.smithi096.stdout:6/598: symlink d3/d3a/lcd 0 2024-04-22T15:18:57.669 INFO:tasks.workunit.client.0.smithi096.stdout:6/599: mknod d3/d28/dd/d11/d2c/d44/d51/cce 0 2024-04-22T15:18:57.669 INFO:tasks.workunit.client.1.smithi196.stdout:4/404: dread d6/d14/d18/d40/f6c [0,4194304] 0 2024-04-22T15:18:57.669 INFO:tasks.workunit.client.1.smithi196.stdout:6/418: dread dd/d20/f29 [0,4194304] 0 2024-04-22T15:18:57.671 INFO:tasks.workunit.client.1.smithi196.stdout:5/382: truncate d1/d17/d1c/f52 678636 0 2024-04-22T15:18:57.673 INFO:tasks.workunit.client.1.smithi196.stdout:3/377: link d0/d7/f7d d0/d7/d8/d70/f7e 0 2024-04-22T15:18:57.676 INFO:tasks.workunit.client.1.smithi196.stdout:9/398: dwrite db/dd/d1b/d2c/f43 [0,4194304] 0 2024-04-22T15:18:57.676 INFO:tasks.workunit.client.1.smithi196.stdout:2/414: write d1/d13/d25/d2a/f3f [508787,84800] 0 2024-04-22T15:18:57.677 INFO:tasks.workunit.client.1.smithi196.stdout:5/383: mkdir d1/d4/d7b 0 2024-04-22T15:18:57.677 INFO:tasks.workunit.client.1.smithi196.stdout:4/405: readlink d6/d14/d18/d26/d32/l76 0 2024-04-22T15:18:57.677 INFO:tasks.workunit.client.0.smithi096.stdout:8/713: dread d4/d1c/d40/da7/db6/f49 [0,4194304] 0 2024-04-22T15:18:57.679 INFO:tasks.workunit.client.1.smithi196.stdout:1/357: dwrite d1/db/d16/f73 [0,4194304] 0 2024-04-22T15:18:57.679 INFO:tasks.workunit.client.1.smithi196.stdout:6/419: mkdir dd/d20/d5c/d5e/d76 0 2024-04-22T15:18:57.681 INFO:tasks.workunit.client.1.smithi196.stdout:9/399: chown db/dd/c1d 949296 1 2024-04-22T15:18:57.682 INFO:tasks.workunit.client.1.smithi196.stdout:7/353: dwrite d14/d15/f59 [0,4194304] 0 2024-04-22T15:18:57.683 INFO:tasks.workunit.client.1.smithi196.stdout:9/400: truncate db/dd/de/d15/f1e 8235840 0 2024-04-22T15:18:57.688 INFO:tasks.workunit.client.1.smithi196.stdout:5/384: stat d1/d4/c3c 0 2024-04-22T15:18:57.689 INFO:tasks.workunit.client.1.smithi196.stdout:8/404: write d2/d35/f61 [498396,67477] 0 2024-04-22T15:18:57.690 INFO:tasks.workunit.client.1.smithi196.stdout:3/378: chown d0/d15/d4d/db/dc/l2c 351 1 2024-04-22T15:18:57.692 INFO:tasks.workunit.client.1.smithi196.stdout:2/415: mknod d1/d13/d1f/c9a 0 2024-04-22T15:18:57.694 INFO:tasks.workunit.client.1.smithi196.stdout:3/379: chown d0/d15/d48/d51/d5e 1483 1 2024-04-22T15:18:57.695 INFO:tasks.workunit.client.1.smithi196.stdout:7/354: symlink d14/d26/d5a/d5f/l6f 0 2024-04-22T15:18:57.698 INFO:tasks.workunit.client.1.smithi196.stdout:8/405: read d2/d3f/f4e [2413526,89667] 0 2024-04-22T15:18:57.701 INFO:tasks.workunit.client.1.smithi196.stdout:2/416: mknod d1/d76/c9b 0 2024-04-22T15:18:57.703 INFO:tasks.workunit.client.1.smithi196.stdout:0/420: dwrite d3/f47 [0,4194304] 0 2024-04-22T15:18:57.703 INFO:tasks.workunit.client.0.smithi096.stdout:6/600: dread d3/d28/f1a [0,4194304] 0 2024-04-22T15:18:57.703 INFO:tasks.workunit.client.1.smithi196.stdout:7/355: chown d14/d26/d5e/c6d 12791129 1 2024-04-22T15:18:57.704 INFO:tasks.workunit.client.1.smithi196.stdout:1/358: rename d1/db/dc/d26/d2e/f51 to d1/db/d16/d1e/d65/f76 0 2024-04-22T15:18:57.704 INFO:tasks.workunit.client.1.smithi196.stdout:9/401: dwrite db/f61 [0,4194304] 0 2024-04-22T15:18:57.705 INFO:tasks.workunit.client.0.smithi096.stdout:6/601: unlink d3/d28/d54/d68/c7b 0 2024-04-22T15:18:57.708 INFO:tasks.workunit.client.0.smithi096.stdout:8/714: dread d4/d1c/f72 [0,4194304] 0 2024-04-22T15:18:57.708 INFO:tasks.workunit.client.1.smithi196.stdout:3/380: mkdir d0/d15/d48/d51/d7f 0 2024-04-22T15:18:57.709 INFO:tasks.workunit.client.0.smithi096.stdout:8/715: stat d4/d10/d25/d38/l80 0 2024-04-22T15:18:57.710 INFO:tasks.workunit.client.1.smithi196.stdout:1/359: truncate d1/db/dc/de/d43/f68 187019 0 2024-04-22T15:18:57.711 INFO:tasks.workunit.client.1.smithi196.stdout:5/385: stat d1/d9/d68/f70 0 2024-04-22T15:18:57.714 INFO:tasks.workunit.client.1.smithi196.stdout:3/381: write d0/d15/d48/d51/d5e/f63 [887301,85775] 0 2024-04-22T15:18:57.714 INFO:tasks.workunit.client.1.smithi196.stdout:0/421: rmdir d3/d9/d36 39 2024-04-22T15:18:57.716 INFO:tasks.workunit.client.1.smithi196.stdout:2/417: unlink d1/d13/d41/f62 0 2024-04-22T15:18:57.716 INFO:tasks.workunit.client.1.smithi196.stdout:0/422: chown d3/de/d14/d5a 2354941 1 2024-04-22T15:18:57.717 INFO:tasks.workunit.client.1.smithi196.stdout:3/382: dread - d0/d15/d31/d55/f74 zero size 2024-04-22T15:18:57.717 INFO:tasks.workunit.client.1.smithi196.stdout:7/356: creat d14/d26/d2a/d2b/f70 x:0 0 0 2024-04-22T15:18:57.718 INFO:tasks.workunit.client.1.smithi196.stdout:9/402: symlink db/dd/d1b/d2c/l8f 0 2024-04-22T15:18:57.719 INFO:tasks.workunit.client.1.smithi196.stdout:0/423: readlink d3/d6/dc/l50 0 2024-04-22T15:18:57.720 INFO:tasks.workunit.client.1.smithi196.stdout:6/420: write dd/d1f/d37/f5d [871224,85248] 0 2024-04-22T15:18:57.723 INFO:tasks.workunit.client.1.smithi196.stdout:6/421: chown dd/d70 7338 1 2024-04-22T15:18:57.723 INFO:tasks.workunit.client.1.smithi196.stdout:7/357: creat d14/d4a/f71 x:0 0 0 2024-04-22T15:18:57.723 INFO:tasks.workunit.client.1.smithi196.stdout:3/383: mknod d0/d15/d4d/c80 0 2024-04-22T15:18:57.723 INFO:tasks.workunit.client.1.smithi196.stdout:2/418: write d1/d13/d41/f6c [5073935,78244] 0 2024-04-22T15:18:57.724 INFO:tasks.workunit.client.1.smithi196.stdout:9/403: chown db/dd/de/l3e 0 1 2024-04-22T15:18:57.725 INFO:tasks.workunit.client.1.smithi196.stdout:5/386: dwrite d1/d17/d1c/f57 [0,4194304] 0 2024-04-22T15:18:57.728 INFO:tasks.workunit.client.1.smithi196.stdout:4/406: dread d6/d13/f16 [0,4194304] 0 2024-04-22T15:18:57.730 INFO:tasks.workunit.client.1.smithi196.stdout:1/360: dwrite d1/db/dc/d26/d2e/f3e [0,4194304] 0 2024-04-22T15:18:57.730 INFO:tasks.workunit.client.1.smithi196.stdout:7/358: mkdir d14/d26/d2a/d72 0 2024-04-22T15:18:57.731 INFO:tasks.workunit.client.1.smithi196.stdout:2/419: write d1/d13/d1f/f60 [4194,91950] 0 2024-04-22T15:18:57.732 INFO:tasks.workunit.client.1.smithi196.stdout:7/359: write d14/d26/d28/d29/f6c [1016612,110597] 0 2024-04-22T15:18:57.733 INFO:tasks.workunit.client.1.smithi196.stdout:9/404: symlink db/d11/d29/d7a/d7b/l90 0 2024-04-22T15:18:57.734 INFO:tasks.workunit.client.1.smithi196.stdout:6/422: rmdir dd/d1f/d4e/d53/d55/d57 39 2024-04-22T15:18:57.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:57 smithi096 ceph-mon[26688]: pgmap v35: 65 pgs: 65 active+clean; 2.1 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 51 MiB/s rd, 180 MiB/s wr, 339 op/s 2024-04-22T15:18:57.736 INFO:tasks.workunit.client.1.smithi196.stdout:6/423: write dd/d20/d5c/d5e/f64 [376132,80713] 0 2024-04-22T15:18:57.740 INFO:tasks.workunit.client.1.smithi196.stdout:8/406: truncate d2/d8/f27 1074261 0 2024-04-22T15:18:57.741 INFO:tasks.workunit.client.1.smithi196.stdout:8/407: stat d2/dc/d45 0 2024-04-22T15:18:57.742 INFO:tasks.workunit.client.1.smithi196.stdout:6/424: dwrite dd/d1f/d38/d40/f42 [0,4194304] 0 2024-04-22T15:18:57.742 INFO:tasks.workunit.client.1.smithi196.stdout:8/408: readlink d2/d8/d49/l50 0 2024-04-22T15:18:57.742 INFO:tasks.workunit.client.1.smithi196.stdout:4/407: mkdir d6/d13/d15/d2c/d62/d78 0 2024-04-22T15:18:57.744 INFO:tasks.workunit.client.1.smithi196.stdout:6/425: write dd/d1f/d4e/f73 [948420,64819] 0 2024-04-22T15:18:57.746 INFO:tasks.workunit.client.1.smithi196.stdout:2/420: creat d1/d1d/d6a/f9c x:0 0 0 2024-04-22T15:18:57.746 INFO:tasks.workunit.client.1.smithi196.stdout:3/384: dread d0/d15/d4d/db/dc/d4a/d44/f3c [0,4194304] 0 2024-04-22T15:18:57.747 INFO:tasks.workunit.client.1.smithi196.stdout:9/405: creat db/dd/d1b/f91 x:0 0 0 2024-04-22T15:18:57.748 INFO:tasks.workunit.client.1.smithi196.stdout:9/406: chown db/d11/d29/d57/c60 3703 1 2024-04-22T15:18:57.748 INFO:tasks.workunit.client.1.smithi196.stdout:3/385: write d0/d15/d48/d51/d5e/f63 [1019495,10928] 0 2024-04-22T15:18:57.749 INFO:tasks.workunit.client.1.smithi196.stdout:0/424: link d3/d9/d25/l83 d3/d9/l8e 0 2024-04-22T15:18:57.751 INFO:tasks.workunit.client.1.smithi196.stdout:4/408: mknod d6/d14/d18/c79 0 2024-04-22T15:18:57.752 INFO:tasks.workunit.client.1.smithi196.stdout:6/426: stat dd/d1f/c25 0 2024-04-22T15:18:57.753 INFO:tasks.workunit.client.1.smithi196.stdout:2/421: creat d1/d13/d1f/f9d x:0 0 0 2024-04-22T15:18:57.755 INFO:tasks.workunit.client.1.smithi196.stdout:4/409: dread d6/d14/d18/d26/f43 [0,4194304] 0 2024-04-22T15:18:57.756 INFO:tasks.workunit.client.1.smithi196.stdout:3/386: unlink d0/d7/f11 0 2024-04-22T15:18:57.756 INFO:tasks.workunit.client.1.smithi196.stdout:4/410: stat d6/d13/f68 0 2024-04-22T15:18:57.756 INFO:tasks.workunit.client.1.smithi196.stdout:9/407: creat db/d11/d29/f92 x:0 0 0 2024-04-22T15:18:57.757 INFO:tasks.workunit.client.1.smithi196.stdout:3/387: readlink d0/d15/d48/d51/l67 0 2024-04-22T15:18:57.757 INFO:tasks.workunit.client.1.smithi196.stdout:8/409: mkdir d2/d6a/d7b 0 2024-04-22T15:18:57.758 INFO:tasks.workunit.client.1.smithi196.stdout:6/427: fdatasync dd/d20/f29 0 2024-04-22T15:18:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:57 smithi196 ceph-mon[34717]: pgmap v35: 65 pgs: 65 active+clean; 2.1 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 51 MiB/s rd, 180 MiB/s wr, 339 op/s 2024-04-22T15:18:57.759 INFO:tasks.workunit.client.1.smithi196.stdout:2/422: rmdir d1/d23/d27/d2e/d55 39 2024-04-22T15:18:57.759 INFO:tasks.workunit.client.1.smithi196.stdout:7/360: link d14/c52 d14/d26/d2a/d2b/c73 0 2024-04-22T15:18:57.762 INFO:tasks.workunit.client.1.smithi196.stdout:9/408: unlink db/dd/de/c44 0 2024-04-22T15:18:57.763 INFO:tasks.workunit.client.1.smithi196.stdout:7/361: write d14/d26/d28/f58 [588383,100969] 0 2024-04-22T15:18:57.763 INFO:tasks.workunit.client.1.smithi196.stdout:6/428: truncate dd/d20/f24 1560183 0 2024-04-22T15:18:57.764 INFO:tasks.workunit.client.1.smithi196.stdout:2/423: chown d1/d13/d25/d2a/f95 0 1 2024-04-22T15:18:57.765 INFO:tasks.workunit.client.1.smithi196.stdout:0/425: rename d3/d6/dc/l50 to d3/de/l8f 0 2024-04-22T15:18:57.766 INFO:tasks.workunit.client.1.smithi196.stdout:9/409: mknod db/d11/c93 0 2024-04-22T15:18:57.768 INFO:tasks.workunit.client.1.smithi196.stdout:2/424: chown d1/d23/d27/d2e/d55 118155 1 2024-04-22T15:18:57.769 INFO:tasks.workunit.client.1.smithi196.stdout:8/410: dwrite d2/d3f/f41 [0,4194304] 0 2024-04-22T15:18:57.769 INFO:tasks.workunit.client.1.smithi196.stdout:3/388: dwrite d0/d15/d4d/db/dc/d18/f1b [0,4194304] 0 2024-04-22T15:18:57.771 INFO:tasks.workunit.client.1.smithi196.stdout:7/362: truncate d14/d15/f1a 82655 0 2024-04-22T15:18:57.775 INFO:tasks.workunit.client.1.smithi196.stdout:0/426: symlink d3/de/d14/d86/l90 0 2024-04-22T15:18:57.775 INFO:tasks.workunit.client.1.smithi196.stdout:5/387: dread d1/d17/f2e [0,4194304] 0 2024-04-22T15:18:57.777 INFO:tasks.workunit.client.1.smithi196.stdout:7/363: fdatasync d14/f6b 0 2024-04-22T15:18:57.777 INFO:tasks.workunit.client.1.smithi196.stdout:8/411: fsync d2/d35/f5f 0 2024-04-22T15:18:57.779 INFO:tasks.workunit.client.1.smithi196.stdout:3/389: creat d0/d7/d6f/f81 x:0 0 0 2024-04-22T15:18:57.779 INFO:tasks.workunit.client.1.smithi196.stdout:7/364: fsync d14/f6b 0 2024-04-22T15:18:57.780 INFO:tasks.workunit.client.1.smithi196.stdout:5/388: dread d1/d17/d1a/f43 [0,4194304] 0 2024-04-22T15:18:57.781 INFO:tasks.workunit.client.1.smithi196.stdout:5/389: dread - d1/d51/f69 zero size 2024-04-22T15:18:57.783 INFO:tasks.workunit.client.1.smithi196.stdout:5/390: chown d1/d14/d3e/f44 3 1 2024-04-22T15:18:57.787 INFO:tasks.workunit.client.1.smithi196.stdout:8/412: creat d2/d8/d47/d68/f7c x:0 0 0 2024-04-22T15:18:57.788 INFO:tasks.workunit.client.1.smithi196.stdout:0/427: mknod d3/d6/dc/c91 0 2024-04-22T15:18:57.788 INFO:tasks.workunit.client.1.smithi196.stdout:8/413: dread - d2/d8/f76 zero size 2024-04-22T15:18:57.788 INFO:tasks.workunit.client.1.smithi196.stdout:1/361: dwrite d1/db/dc/f6f [0,4194304] 0 2024-04-22T15:18:57.788 INFO:tasks.workunit.client.1.smithi196.stdout:5/391: write d1/d9/f75 [333257,32510] 0 2024-04-22T15:18:57.789 INFO:tasks.workunit.client.1.smithi196.stdout:8/414: write d2/d5/f23 [4272418,45416] 0 2024-04-22T15:18:57.789 INFO:tasks.workunit.client.1.smithi196.stdout:2/425: rename d1/d23/d27/d2e/d55/d5b/d7c/c92 to d1/c9e 0 2024-04-22T15:18:57.790 INFO:tasks.workunit.client.1.smithi196.stdout:0/428: creat d3/d6/dc/f92 x:0 0 0 2024-04-22T15:18:57.791 INFO:tasks.workunit.client.1.smithi196.stdout:8/415: chown d2/d8/c2a 955732756 1 2024-04-22T15:18:57.792 INFO:tasks.workunit.client.1.smithi196.stdout:8/416: chown d2/f1d 1445409 1 2024-04-22T15:18:57.793 INFO:tasks.workunit.client.1.smithi196.stdout:1/362: symlink d1/db/d16/d44/d6a/l77 0 2024-04-22T15:18:57.795 INFO:tasks.workunit.client.1.smithi196.stdout:2/426: write d1/f80 [8947216,25413] 0 2024-04-22T15:18:57.798 INFO:tasks.workunit.client.1.smithi196.stdout:5/392: dwrite d1/d4/f22 [0,4194304] 0 2024-04-22T15:18:57.799 INFO:tasks.workunit.client.1.smithi196.stdout:7/365: rename d14/d26/f30 to d14/d15/f74 0 2024-04-22T15:18:57.799 INFO:tasks.workunit.client.0.smithi096.stdout:8/716: getdents d4/d10/d25/d4c/d76/d7f 0 2024-04-22T15:18:57.802 INFO:tasks.workunit.client.0.smithi096.stdout:6/602: getdents d3/d28/d4d/d82/d9e/db0 0 2024-04-22T15:18:57.802 INFO:tasks.workunit.client.1.smithi196.stdout:0/429: dwrite d3/de/d14/f39 [0,4194304] 0 2024-04-22T15:18:57.804 INFO:tasks.workunit.client.1.smithi196.stdout:0/430: stat d3/de/d14/f4a 0 2024-04-22T15:18:57.804 INFO:tasks.workunit.client.1.smithi196.stdout:4/411: dread d6/d13/f25 [4194304,4194304] 0 2024-04-22T15:18:57.804 INFO:tasks.workunit.client.1.smithi196.stdout:8/417: write d2/d3f/f53 [2855138,104315] 0 2024-04-22T15:18:57.808 INFO:tasks.workunit.client.1.smithi196.stdout:5/393: dread d1/d4/f22 [0,4194304] 0 2024-04-22T15:18:57.811 INFO:tasks.workunit.client.1.smithi196.stdout:6/429: truncate fa 1726457 0 2024-04-22T15:18:57.815 INFO:tasks.workunit.client.0.smithi096.stdout:8/717: link d4/d10/d25/fea d4/d10/d25/d4c/d76/fef 0 2024-04-22T15:18:57.815 INFO:tasks.workunit.client.1.smithi196.stdout:3/390: rename d0/d15/d48/d51/l67 to d0/d15/d31/d55/l82 0 2024-04-22T15:18:57.815 INFO:tasks.workunit.client.1.smithi196.stdout:0/431: dwrite d3/fd [0,4194304] 0 2024-04-22T15:18:57.825 INFO:tasks.workunit.client.0.smithi096.stdout:8/718: creat d4/d10/d4d/deb/ff0 x:0 0 0 2024-04-22T15:18:57.826 INFO:tasks.workunit.client.1.smithi196.stdout:6/430: dwrite dd/f3a [0,4194304] 0 2024-04-22T15:18:57.826 INFO:tasks.workunit.client.1.smithi196.stdout:9/410: dwrite db/dd/de/d15/d31/f47 [0,4194304] 0 2024-04-22T15:18:57.828 INFO:tasks.workunit.client.1.smithi196.stdout:2/427: truncate d1/d23/d27/d2e/d55/f36 1228440 0 2024-04-22T15:18:57.829 INFO:tasks.workunit.client.0.smithi096.stdout:8/719: dread - d4/d10/d4d/fa9 zero size 2024-04-22T15:18:57.832 INFO:tasks.workunit.client.1.smithi196.stdout:0/432: dread d3/f11 [0,4194304] 0 2024-04-22T15:18:57.832 INFO:tasks.workunit.client.1.smithi196.stdout:3/391: creat d0/d15/d31/d55/f83 x:0 0 0 2024-04-22T15:18:57.833 INFO:tasks.workunit.client.0.smithi096.stdout:8/720: fsync d4/d10/f99 0 2024-04-22T15:18:57.835 INFO:tasks.workunit.client.1.smithi196.stdout:2/428: dwrite d1/d13/f50 [0,4194304] 0 2024-04-22T15:18:57.844 INFO:tasks.workunit.client.1.smithi196.stdout:7/366: truncate d14/d25/d31/f65 1134698 0 2024-04-22T15:18:57.845 INFO:tasks.workunit.client.1.smithi196.stdout:4/412: creat d6/d14/d18/d40/d74/f7a x:0 0 0 2024-04-22T15:18:57.847 INFO:tasks.workunit.client.1.smithi196.stdout:6/431: creat dd/d2b/f77 x:0 0 0 2024-04-22T15:18:57.849 INFO:tasks.workunit.client.1.smithi196.stdout:6/432: chown dd/d1f/d38/d40 29 1 2024-04-22T15:18:57.850 INFO:tasks.workunit.client.1.smithi196.stdout:0/433: creat d3/de/d1c/d27/f93 x:0 0 0 2024-04-22T15:18:57.857 INFO:tasks.workunit.client.1.smithi196.stdout:1/363: rename d1/db/d16/f4b to d1/f78 0 2024-04-22T15:18:57.859 INFO:tasks.workunit.client.1.smithi196.stdout:7/367: mknod d14/c75 0 2024-04-22T15:18:57.860 INFO:tasks.workunit.client.1.smithi196.stdout:9/411: rmdir db/dd/de/d15/d31/d64 39 2024-04-22T15:18:57.862 INFO:tasks.workunit.client.1.smithi196.stdout:5/394: rename d1/d2/l66 to d1/d9/d68/l7c 0 2024-04-22T15:18:57.864 INFO:tasks.workunit.client.0.smithi096.stdout:8/721: dread d4/d1c/d40/d48/f68 [0,4194304] 0 2024-04-22T15:18:57.864 INFO:tasks.workunit.client.1.smithi196.stdout:8/418: truncate d2/dc/fd 2628956 0 2024-04-22T15:18:57.865 INFO:tasks.workunit.client.1.smithi196.stdout:6/433: dwrite dd/d1f/d37/f31 [4194304,4194304] 0 2024-04-22T15:18:57.865 INFO:tasks.workunit.client.1.smithi196.stdout:1/364: read d1/db/dc/de/f14 [472055,93552] 0 2024-04-22T15:18:57.866 INFO:tasks.workunit.client.1.smithi196.stdout:6/434: readlink dd/d2b/l6c 0 2024-04-22T15:18:57.867 INFO:tasks.workunit.client.1.smithi196.stdout:7/368: mknod d14/d26/c76 0 2024-04-22T15:18:57.867 INFO:tasks.workunit.client.1.smithi196.stdout:5/395: chown d1/d17/d1a/d2f/d30/f6b 255420 1 2024-04-22T15:18:57.868 INFO:tasks.workunit.client.1.smithi196.stdout:6/435: dread - dd/d1f/d37/f34 zero size 2024-04-22T15:18:57.869 INFO:tasks.workunit.client.1.smithi196.stdout:3/392: write d0/d15/d4d/f2d [4268606,60807] 0 2024-04-22T15:18:57.870 INFO:tasks.workunit.client.1.smithi196.stdout:2/429: write d1/f3 [1271943,124182] 0 2024-04-22T15:18:57.871 INFO:tasks.workunit.client.0.smithi096.stdout:8/722: creat d4/d10/d25/ff1 x:0 0 0 2024-04-22T15:18:57.871 INFO:tasks.workunit.client.0.smithi096.stdout:8/723: readlink d4/d10/l1b 0 2024-04-22T15:18:57.872 INFO:tasks.workunit.client.1.smithi196.stdout:3/393: stat d0/d7/d6f 0 2024-04-22T15:18:57.873 INFO:tasks.workunit.client.1.smithi196.stdout:1/365: creat d1/db/dc/de/d5a/d5b/f79 x:0 0 0 2024-04-22T15:18:57.875 INFO:tasks.workunit.client.1.smithi196.stdout:5/396: mkdir d1/d17/d1a/d2f/d30/d7d 0 2024-04-22T15:18:57.875 INFO:tasks.workunit.client.1.smithi196.stdout:7/369: dwrite fc [0,4194304] 0 2024-04-22T15:18:57.876 INFO:tasks.workunit.client.1.smithi196.stdout:4/413: dread d6/d13/f6b [0,4194304] 0 2024-04-22T15:18:57.876 INFO:tasks.workunit.client.1.smithi196.stdout:5/397: chown d1/d14/f1e 938015073 1 2024-04-22T15:18:57.877 INFO:tasks.workunit.client.0.smithi096.stdout:8/724: fsync d4/d10/d25/d38/f54 0 2024-04-22T15:18:57.877 INFO:tasks.workunit.client.1.smithi196.stdout:4/414: read d6/d13/f1b [4321904,56607] 0 2024-04-22T15:18:57.877 INFO:tasks.workunit.client.1.smithi196.stdout:5/398: dread - d1/d14/d3e/f4e zero size 2024-04-22T15:18:57.880 INFO:tasks.workunit.client.1.smithi196.stdout:9/412: getdents db/d11 0 2024-04-22T15:18:57.880 INFO:tasks.workunit.client.1.smithi196.stdout:2/430: mkdir d1/d13/d25/d9f 0 2024-04-22T15:18:57.883 INFO:tasks.workunit.client.1.smithi196.stdout:3/394: creat d0/d15/d4d/db/dc/d18/f84 x:0 0 0 2024-04-22T15:18:57.884 INFO:tasks.workunit.client.1.smithi196.stdout:7/370: creat d14/d26/d2a/d2b/f77 x:0 0 0 2024-04-22T15:18:57.885 INFO:tasks.workunit.client.1.smithi196.stdout:6/436: symlink dd/d20/d5c/d5e/d76/l78 0 2024-04-22T15:18:57.885 INFO:tasks.workunit.client.1.smithi196.stdout:8/419: getdents d2/d3f 0 2024-04-22T15:18:57.886 INFO:tasks.workunit.client.1.smithi196.stdout:6/437: readlink dd/d20/d5c/d5e/l63 0 2024-04-22T15:18:57.886 INFO:tasks.workunit.client.1.smithi196.stdout:9/413: unlink db/d11/d29/d7a/f8c 0 2024-04-22T15:18:57.887 INFO:tasks.workunit.client.1.smithi196.stdout:5/399: rename d1/d17/d1c/l4f to d1/d17/d1a/d2f/d30/d7d/l7e 0 2024-04-22T15:18:57.888 INFO:tasks.workunit.client.1.smithi196.stdout:1/366: mkdir d1/db/dc/de/d43/d5e/d7a 0 2024-04-22T15:18:57.888 INFO:tasks.workunit.client.1.smithi196.stdout:5/400: fsync d1/d17/d1a/d2f/d30/d5e/f63 0 2024-04-22T15:18:57.891 INFO:tasks.workunit.client.0.smithi096.stdout:2/712: sync 2024-04-22T15:18:57.894 INFO:tasks.workunit.client.1.smithi196.stdout:6/438: write dd/d1f/f48 [1361249,111648] 0 2024-04-22T15:18:57.894 INFO:tasks.workunit.client.1.smithi196.stdout:9/414: fdatasync db/dd/de/f42 0 2024-04-22T15:18:57.897 INFO:tasks.workunit.client.0.smithi096.stdout:2/713: creat d3/d13/fe5 x:0 0 0 2024-04-22T15:18:57.902 INFO:tasks.workunit.client.0.smithi096.stdout:2/714: creat d3/db/df/dce/d46/d7e/d8d/fe6 x:0 0 0 2024-04-22T15:18:57.902 INFO:tasks.workunit.client.0.smithi096.stdout:2/715: chown d3/d1f/c6d 10428 1 2024-04-22T15:18:57.903 INFO:tasks.workunit.client.1.smithi196.stdout:4/415: rmdir d6/d14/d18/d40/d69 0 2024-04-22T15:18:57.904 INFO:tasks.workunit.client.1.smithi196.stdout:7/371: symlink d14/d26/d5a/d5f/d6e/l78 0 2024-04-22T15:18:57.906 INFO:tasks.workunit.client.0.smithi096.stdout:2/716: creat d3/db/df/d98/db3/fe7 x:0 0 0 2024-04-22T15:18:57.907 INFO:tasks.workunit.client.0.smithi096.stdout:2/717: write d3/db/df/d24/d58/d5f/d6e/fb8 [784505,104512] 0 2024-04-22T15:18:57.910 INFO:tasks.workunit.client.1.smithi196.stdout:9/415: symlink db/d11/d29/d7a/d7b/l94 0 2024-04-22T15:18:57.911 INFO:tasks.workunit.client.0.smithi096.stdout:8/725: truncate d4/d10/d4d/f84 128725 0 2024-04-22T15:18:57.912 INFO:tasks.workunit.client.0.smithi096.stdout:2/718: symlink d3/db/df/dce/d46/d59/le8 0 2024-04-22T15:18:57.913 INFO:tasks.workunit.client.0.smithi096.stdout:2/719: read d3/db/df/fcd [352744,9767] 0 2024-04-22T15:18:57.914 INFO:tasks.workunit.client.1.smithi196.stdout:2/431: rename d1/d23/c8b to d1/d23/d27/d2e/d55/d5b/ca0 0 2024-04-22T15:18:57.915 INFO:tasks.workunit.client.1.smithi196.stdout:2/432: fsync d1/d1d/f7a 0 2024-04-22T15:18:57.915 INFO:tasks.workunit.client.1.smithi196.stdout:1/367: getdents d1/db/dc/de/d43/d5e/d7a 0 2024-04-22T15:18:57.917 INFO:tasks.workunit.client.0.smithi096.stdout:8/726: truncate d4/d10/d25/d38/f54 307281 0 2024-04-22T15:18:57.918 INFO:tasks.workunit.client.1.smithi196.stdout:5/401: dread d1/d17/d1a/d2f/d30/d5e/f63 [0,4194304] 0 2024-04-22T15:18:57.920 INFO:tasks.workunit.client.0.smithi096.stdout:2/720: dwrite d3/db/df/d98/db3/fc7 [0,4194304] 0 2024-04-22T15:18:57.920 INFO:tasks.workunit.client.1.smithi196.stdout:8/420: dread d2/dc/f3a [0,4194304] 0 2024-04-22T15:18:57.921 INFO:tasks.workunit.client.1.smithi196.stdout:7/372: creat d14/d25/f79 x:0 0 0 2024-04-22T15:18:57.922 INFO:tasks.workunit.client.1.smithi196.stdout:8/421: fdatasync d2/d5/f74 0 2024-04-22T15:18:57.922 INFO:tasks.workunit.client.0.smithi096.stdout:2/721: stat d3/db/df/f61 0 2024-04-22T15:18:57.922 INFO:tasks.workunit.client.1.smithi196.stdout:1/368: dwrite d1/db/f58 [0,4194304] 0 2024-04-22T15:18:57.925 INFO:tasks.workunit.client.0.smithi096.stdout:8/727: dwrite d4/d1c/f3a [4194304,4194304] 0 2024-04-22T15:18:57.927 INFO:tasks.workunit.client.0.smithi096.stdout:2/722: chown d3/db/df/dce/d46/d69/d7f 1120436 1 2024-04-22T15:18:57.932 INFO:tasks.workunit.client.1.smithi196.stdout:9/416: mknod db/d11/c95 0 2024-04-22T15:18:57.936 INFO:tasks.workunit.client.0.smithi096.stdout:8/728: rename d4/d1c/d40/l47 to d4/d10/d25/d4c/db4/lf2 0 2024-04-22T15:18:57.937 INFO:tasks.workunit.client.0.smithi096.stdout:8/729: write d4/d10/d25/d4c/f5e [1994702,66597] 0 2024-04-22T15:18:57.940 INFO:tasks.workunit.client.1.smithi196.stdout:2/433: rename d1/d23/d27/d2e/d45 to d1/d1d/d6a/da1 0 2024-04-22T15:18:57.944 INFO:tasks.workunit.client.0.smithi096.stdout:2/723: symlink d3/db/df/d24/dc8/le9 0 2024-04-22T15:18:57.949 INFO:tasks.workunit.client.0.smithi096.stdout:2/724: creat d3/d13/db4/dbf/fea x:0 0 0 2024-04-22T15:18:57.950 INFO:tasks.workunit.client.1.smithi196.stdout:8/422: read d2/d5/f48 [2364393,35917] 0 2024-04-22T15:18:57.951 INFO:tasks.workunit.client.1.smithi196.stdout:7/373: creat d14/d25/f7a x:0 0 0 2024-04-22T15:18:57.951 INFO:tasks.workunit.client.1.smithi196.stdout:7/374: read d14/d15/f59 [3590174,85977] 0 2024-04-22T15:18:57.952 INFO:tasks.workunit.client.1.smithi196.stdout:7/375: write d14/d25/f42 [3298186,9944] 0 2024-04-22T15:18:57.953 INFO:tasks.workunit.client.1.smithi196.stdout:6/439: creat dd/d1f/d38/f79 x:0 0 0 2024-04-22T15:18:57.954 INFO:tasks.workunit.client.0.smithi096.stdout:2/725: mknod d3/db/df/ceb 0 2024-04-22T15:18:57.957 INFO:tasks.workunit.client.1.smithi196.stdout:5/402: mkdir d1/d17/d7f 0 2024-04-22T15:18:57.960 INFO:tasks.workunit.client.1.smithi196.stdout:8/423: dread d2/d5/f19 [0,4194304] 0 2024-04-22T15:18:57.961 INFO:tasks.workunit.client.1.smithi196.stdout:4/416: truncate d6/d13/d15/d2c/d35/d44/f4a 34760 0 2024-04-22T15:18:57.962 INFO:tasks.workunit.client.1.smithi196.stdout:7/376: creat d14/d26/d5e/f7b x:0 0 0 2024-04-22T15:18:57.963 INFO:tasks.workunit.client.0.smithi096.stdout:8/730: write d4/f12 [4974720,85937] 0 2024-04-22T15:18:57.965 INFO:tasks.workunit.client.1.smithi196.stdout:7/377: dread d14/d26/d28/d29/f6c [0,4194304] 0 2024-04-22T15:18:57.967 INFO:tasks.workunit.client.0.smithi096.stdout:8/731: creat d4/d1c/de2/ff3 x:0 0 0 2024-04-22T15:18:57.971 INFO:tasks.workunit.client.0.smithi096.stdout:8/732: mkdir d4/d10/d25/d4c/db4/df4 0 2024-04-22T15:18:57.974 INFO:tasks.workunit.client.0.smithi096.stdout:2/726: dwrite d3/db/d30/fd3 [0,4194304] 0 2024-04-22T15:18:57.978 INFO:tasks.workunit.client.0.smithi096.stdout:2/727: dread - d3/db/df/dce/d46/d69/f6f zero size 2024-04-22T15:18:57.980 INFO:tasks.workunit.client.0.smithi096.stdout:2/728: chown d3/db/df/d24/f8f 5747790 1 2024-04-22T15:18:57.982 INFO:tasks.workunit.client.0.smithi096.stdout:2/729: mknod d3/db/cec 0 2024-04-22T15:18:57.986 INFO:tasks.workunit.client.0.smithi096.stdout:2/730: truncate d3/db/d30/f39 962184 0 2024-04-22T15:18:57.988 INFO:tasks.workunit.client.0.smithi096.stdout:8/733: dwrite d4/d10/f19 [0,4194304] 0 2024-04-22T15:18:57.988 INFO:tasks.workunit.client.0.smithi096.stdout:2/731: write d3/d1f/f56 [4749867,358] 0 2024-04-22T15:18:57.990 INFO:tasks.workunit.client.0.smithi096.stdout:8/734: truncate d4/d1c/d40/d46/d58/fc8 294090 0 2024-04-22T15:18:57.995 INFO:tasks.workunit.client.0.smithi096.stdout:8/735: symlink d4/d2a/db7/lf5 0 2024-04-22T15:18:57.995 INFO:tasks.workunit.client.0.smithi096.stdout:8/736: dread - d4/dc0/fd3 zero size 2024-04-22T15:18:57.996 INFO:tasks.workunit.client.0.smithi096.stdout:2/732: getdents d3/db/df/dce 0 2024-04-22T15:18:57.997 INFO:tasks.workunit.client.0.smithi096.stdout:2/733: dread - d3/db/df/d10/f55 zero size 2024-04-22T15:18:57.999 INFO:tasks.workunit.client.0.smithi096.stdout:8/737: fdatasync d4/d10/d25/fea 0 2024-04-22T15:18:58.000 INFO:tasks.workunit.client.0.smithi096.stdout:2/734: rmdir d3/d13/db4 39 2024-04-22T15:18:58.001 INFO:tasks.workunit.client.0.smithi096.stdout:8/738: chown d4/d1c/f43 42 1 2024-04-22T15:18:58.003 INFO:tasks.workunit.client.0.smithi096.stdout:8/739: symlink d4/d71/lf6 0 2024-04-22T15:18:58.004 INFO:tasks.workunit.client.0.smithi096.stdout:2/735: fsync d3/db/d30/f3a 0 2024-04-22T15:18:58.007 INFO:tasks.workunit.client.0.smithi096.stdout:2/736: chown d3/db/d30/l78 213208 1 2024-04-22T15:18:58.009 INFO:tasks.workunit.client.0.smithi096.stdout:2/737: symlink d3/d13/db4/led 0 2024-04-22T15:18:58.010 INFO:tasks.workunit.client.0.smithi096.stdout:8/740: write d4/d10/d4d/fa9 [827461,2048] 0 2024-04-22T15:18:58.010 INFO:tasks.workunit.client.0.smithi096.stdout:8/741: stat d4/d10/d4d/l5f 0 2024-04-22T15:18:58.011 INFO:tasks.workunit.client.0.smithi096.stdout:2/738: mknod d3/db/df/d24/d58/d5f/d6e/cee 0 2024-04-22T15:18:58.013 INFO:tasks.workunit.client.0.smithi096.stdout:2/739: dread - d3/db/df/dce/d46/d59/f79 zero size 2024-04-22T15:18:58.016 INFO:tasks.workunit.client.0.smithi096.stdout:8/742: dread d4/d1c/d40/d48/fab [0,4194304] 0 2024-04-22T15:18:58.018 INFO:tasks.workunit.client.0.smithi096.stdout:8/743: mkdir d4/d10/d25/df7 0 2024-04-22T15:18:58.019 INFO:tasks.workunit.client.0.smithi096.stdout:2/740: truncate d3/f53 3019867 0 2024-04-22T15:18:58.021 INFO:tasks.workunit.client.0.smithi096.stdout:2/741: truncate d3/f8e 3426941 0 2024-04-22T15:18:58.021 INFO:tasks.workunit.client.0.smithi096.stdout:2/742: truncate d3/fd 8971200 0 2024-04-22T15:18:58.024 INFO:tasks.workunit.client.0.smithi096.stdout:2/743: read d3/db/df/f35 [83552,24830] 0 2024-04-22T15:18:58.027 INFO:tasks.workunit.client.0.smithi096.stdout:8/744: truncate d4/d1c/f5c 5187825 0 2024-04-22T15:18:58.027 INFO:tasks.workunit.client.1.smithi196.stdout:2/434: creat d1/fa2 x:0 0 0 2024-04-22T15:18:58.029 INFO:tasks.workunit.client.1.smithi196.stdout:8/424: mknod d2/d5/c7d 0 2024-04-22T15:18:58.030 INFO:tasks.workunit.client.1.smithi196.stdout:4/417: unlink d6/d13/d15/d2c/d35/c58 0 2024-04-22T15:18:58.031 INFO:tasks.workunit.client.0.smithi096.stdout:8/745: mkdir d4/d1c/df8 0 2024-04-22T15:18:58.032 INFO:tasks.workunit.client.1.smithi196.stdout:8/425: chown d2/d8/l12 287 1 2024-04-22T15:18:58.033 INFO:tasks.workunit.client.0.smithi096.stdout:2/744: getdents d3/db/df/d24/d58/d5f 0 2024-04-22T15:18:58.035 INFO:tasks.workunit.client.1.smithi196.stdout:2/435: rename d1/d23/d27/d2e/d55/l72 to d1/d13/d1f/la3 0 2024-04-22T15:18:58.036 INFO:tasks.workunit.client.0.smithi096.stdout:2/745: symlink d3/db/df/dce/daa/lef 0 2024-04-22T15:18:58.037 INFO:tasks.workunit.client.0.smithi096.stdout:2/746: fdatasync d3/db/df/d24/f8f 0 2024-04-22T15:18:58.038 INFO:tasks.workunit.client.1.smithi196.stdout:1/369: dwrite d1/db/d16/d1e/d65/f76 [0,4194304] 0 2024-04-22T15:18:58.039 INFO:tasks.workunit.client.1.smithi196.stdout:5/403: dwrite d1/d17/f24 [0,4194304] 0 2024-04-22T15:18:58.040 INFO:tasks.workunit.client.1.smithi196.stdout:4/418: creat d6/d13/d15/d2c/d62/d78/f7b x:0 0 0 2024-04-22T15:18:58.041 INFO:tasks.workunit.client.1.smithi196.stdout:7/378: dwrite d14/d15/f74 [0,4194304] 0 2024-04-22T15:18:58.041 INFO:tasks.workunit.client.1.smithi196.stdout:2/436: chown d1/d23/d27/d2e/d55/d5b/f78 886 1 2024-04-22T15:18:58.042 INFO:tasks.workunit.client.1.smithi196.stdout:5/404: chown d1/d17/d1a/d2f 72992 1 2024-04-22T15:18:58.042 INFO:tasks.workunit.client.1.smithi196.stdout:5/405: readlink d1/d4/l4d 0 2024-04-22T15:18:58.044 INFO:tasks.workunit.client.1.smithi196.stdout:8/426: rename d2/dc/f5e to d2/d6a/f7e 0 2024-04-22T15:18:58.044 INFO:tasks.workunit.client.1.smithi196.stdout:2/437: truncate d1/d1d/d6a/f9c 45625 0 2024-04-22T15:18:58.045 INFO:tasks.workunit.client.0.smithi096.stdout:2/747: write d3/db/df/dce/d46/d69/fb9 [229055,40921] 0 2024-04-22T15:18:58.046 INFO:tasks.workunit.client.1.smithi196.stdout:8/427: write d2/dc/f18 [88531,64873] 0 2024-04-22T15:18:58.048 INFO:tasks.workunit.client.1.smithi196.stdout:8/428: chown d2/d5/d1a/c26 1065817923 1 2024-04-22T15:18:58.050 INFO:tasks.workunit.client.0.smithi096.stdout:2/748: creat d3/db/df/d10/db5/ff0 x:0 0 0 2024-04-22T15:18:58.052 INFO:tasks.workunit.client.0.smithi096.stdout:2/749: readlink d3/db/df/dce/d46/d59/le8 0 2024-04-22T15:18:58.056 INFO:tasks.workunit.client.1.smithi196.stdout:5/406: creat d1/d17/d1a/d2f/d30/f80 x:0 0 0 2024-04-22T15:18:58.057 INFO:tasks.workunit.client.1.smithi196.stdout:5/407: write d1/d51/f69 [546863,27482] 0 2024-04-22T15:18:58.057 INFO:tasks.workunit.client.1.smithi196.stdout:0/434: sync 2024-04-22T15:18:58.063 INFO:tasks.workunit.client.1.smithi196.stdout:2/438: dwrite d1/d13/f19 [0,4194304] 0 2024-04-22T15:18:58.066 INFO:tasks.workunit.client.1.smithi196.stdout:7/379: symlink d14/d26/d2a/d72/l7c 0 2024-04-22T15:18:58.068 INFO:tasks.workunit.client.1.smithi196.stdout:5/408: symlink d1/d17/d1a/d2f/d30/d6e/l81 0 2024-04-22T15:18:58.068 INFO:tasks.workunit.client.1.smithi196.stdout:7/380: chown c8 646 1 2024-04-22T15:18:58.070 INFO:tasks.workunit.client.1.smithi196.stdout:4/419: creat d6/d13/d15/f7c x:0 0 0 2024-04-22T15:18:58.071 INFO:tasks.workunit.client.1.smithi196.stdout:2/439: mkdir d1/da4 0 2024-04-22T15:18:58.072 INFO:tasks.workunit.client.1.smithi196.stdout:0/435: write d3/d9/d36/d56/d65/f74 [133367,106396] 0 2024-04-22T15:18:58.073 INFO:tasks.workunit.client.0.smithi096.stdout:2/750: write d3/d1f/d2f/f67 [4187315,83363] 0 2024-04-22T15:18:58.074 INFO:tasks.workunit.client.1.smithi196.stdout:5/409: mknod d1/c82 0 2024-04-22T15:18:58.074 INFO:tasks.workunit.client.1.smithi196.stdout:7/381: mknod d14/c7d 0 2024-04-22T15:18:58.075 INFO:tasks.workunit.client.1.smithi196.stdout:1/370: dwrite d1/f4 [0,4194304] 0 2024-04-22T15:18:58.075 INFO:tasks.workunit.client.1.smithi196.stdout:2/440: fdatasync d1/d13/d1f/f60 0 2024-04-22T15:18:58.079 INFO:tasks.workunit.client.1.smithi196.stdout:8/429: dread d2/dc/f10 [0,4194304] 0 2024-04-22T15:18:58.081 INFO:tasks.workunit.client.1.smithi196.stdout:0/436: creat d3/de/d14/d86/f94 x:0 0 0 2024-04-22T15:18:58.082 INFO:tasks.workunit.client.1.smithi196.stdout:1/371: creat d1/db/dc/de/d43/f7b x:0 0 0 2024-04-22T15:18:58.083 INFO:tasks.workunit.client.1.smithi196.stdout:4/420: creat d6/d14/d18/d26/d32/f7d x:0 0 0 2024-04-22T15:18:58.085 INFO:tasks.workunit.client.1.smithi196.stdout:0/437: chown d3/d6/l23 139 1 2024-04-22T15:18:58.086 INFO:tasks.workunit.client.1.smithi196.stdout:2/441: link d1/d23/f6d d1/d23/d27/d2e/d55/d5b/d7c/fa5 0 2024-04-22T15:18:58.087 INFO:tasks.workunit.client.0.smithi096.stdout:2/751: dwrite d3/d1f/f7a [0,4194304] 0 2024-04-22T15:18:58.088 INFO:tasks.workunit.client.1.smithi196.stdout:7/382: dwrite d14/d15/f74 [0,4194304] 0 2024-04-22T15:18:58.090 INFO:tasks.workunit.client.1.smithi196.stdout:5/410: dwrite d1/d17/d1a/d2f/d30/d5e/f63 [0,4194304] 0 2024-04-22T15:18:58.090 INFO:tasks.workunit.client.1.smithi196.stdout:1/372: creat d1/db/d16/d6d/f7c x:0 0 0 2024-04-22T15:18:58.093 INFO:tasks.workunit.client.1.smithi196.stdout:3/395: sync 2024-04-22T15:18:58.096 INFO:tasks.workunit.client.1.smithi196.stdout:3/396: stat d0/d15/d4d/db/c28 0 2024-04-22T15:18:58.100 INFO:tasks.workunit.client.1.smithi196.stdout:0/438: dwrite d3/d9/f21 [0,4194304] 0 2024-04-22T15:18:58.101 INFO:tasks.workunit.client.1.smithi196.stdout:2/442: dwrite d1/d13/f1e [4194304,4194304] 0 2024-04-22T15:18:58.102 INFO:tasks.workunit.client.1.smithi196.stdout:2/443: fdatasync d1/d23/d27/d2e/f85 0 2024-04-22T15:18:58.113 INFO:tasks.workunit.client.1.smithi196.stdout:0/439: dwrite d3/d6/dc/f92 [0,4194304] 0 2024-04-22T15:18:58.119 INFO:tasks.workunit.client.1.smithi196.stdout:1/373: dread d1/db/d16/f1d [0,4194304] 0 2024-04-22T15:18:58.119 INFO:tasks.workunit.client.1.smithi196.stdout:1/374: fsync d1/db/f58 0 2024-04-22T15:18:58.122 INFO:tasks.workunit.client.1.smithi196.stdout:1/375: dread d1/db/dc/d26/d2e/f3e [0,4194304] 0 2024-04-22T15:18:58.126 INFO:tasks.workunit.client.0.smithi096.stdout:2/752: fdatasync d3/f70 0 2024-04-22T15:18:58.127 INFO:tasks.workunit.client.1.smithi196.stdout:8/430: dread d2/d3f/d4b/f72 [0,4194304] 0 2024-04-22T15:18:58.128 INFO:tasks.workunit.client.1.smithi196.stdout:8/431: read d2/dc/f3a [2832429,50576] 0 2024-04-22T15:18:58.128 INFO:tasks.workunit.client.1.smithi196.stdout:1/376: stat d1/db/dc/d26/d2e/f3e 0 2024-04-22T15:18:58.128 INFO:tasks.workunit.client.0.smithi096.stdout:2/753: chown d3/db/df/dce/d46/d7e/d84/lb0 27 1 2024-04-22T15:18:58.128 INFO:tasks.workunit.client.1.smithi196.stdout:8/432: readlink d2/d3f/l52 0 2024-04-22T15:18:58.129 INFO:tasks.workunit.client.1.smithi196.stdout:8/433: read - d2/d8/d49/f7a zero size 2024-04-22T15:18:58.129 INFO:tasks.workunit.client.1.smithi196.stdout:8/434: fdatasync d2/d8/f25 0 2024-04-22T15:18:58.134 INFO:tasks.workunit.client.1.smithi196.stdout:6/440: sync 2024-04-22T15:18:58.134 INFO:tasks.workunit.client.1.smithi196.stdout:9/417: sync 2024-04-22T15:18:58.135 INFO:tasks.workunit.client.1.smithi196.stdout:2/444: dread d1/f3e [0,4194304] 0 2024-04-22T15:18:58.138 INFO:tasks.workunit.client.1.smithi196.stdout:4/421: rmdir d6/d14/d18/d26 39 2024-04-22T15:18:58.139 INFO:tasks.workunit.client.1.smithi196.stdout:4/422: write d6/f19 [3353614,10771] 0 2024-04-22T15:18:58.144 INFO:tasks.workunit.client.1.smithi196.stdout:3/397: mkdir d0/d15/d4d/db/dc/d85 0 2024-04-22T15:18:58.146 INFO:tasks.workunit.client.1.smithi196.stdout:0/440: chown d3/de/c67 45658 1 2024-04-22T15:18:58.148 INFO:tasks.workunit.client.1.smithi196.stdout:0/441: write d3/d9/f54 [379971,44465] 0 2024-04-22T15:18:58.149 INFO:tasks.workunit.client.1.smithi196.stdout:3/398: dread d0/d15/d4d/db/dc/d18/f1b [0,4194304] 0 2024-04-22T15:18:58.150 INFO:tasks.workunit.client.1.smithi196.stdout:3/399: readlink d0/d7/d8/l34 0 2024-04-22T15:18:58.151 INFO:tasks.workunit.client.1.smithi196.stdout:0/442: dread d3/d9/d36/d56/d65/f74 [0,4194304] 0 2024-04-22T15:18:58.156 INFO:tasks.workunit.client.1.smithi196.stdout:5/411: mknod d1/d17/d1c/d67/c83 0 2024-04-22T15:18:58.158 INFO:tasks.workunit.client.1.smithi196.stdout:6/441: mknod dd/d1f/d37/d3c/c7a 0 2024-04-22T15:18:58.159 INFO:tasks.workunit.client.1.smithi196.stdout:2/445: creat d1/d23/d27/d2e/d55/fa6 x:0 0 0 2024-04-22T15:18:58.160 INFO:tasks.workunit.client.1.smithi196.stdout:2/446: truncate d1/d13/d1f/f9d 1029848 0 2024-04-22T15:18:58.165 INFO:tasks.workunit.client.1.smithi196.stdout:3/400: read d0/d7/d8/f19 [1044877,74547] 0 2024-04-22T15:18:58.165 INFO:tasks.workunit.client.1.smithi196.stdout:1/377: symlink d1/db/dc/l7d 0 2024-04-22T15:18:58.167 INFO:tasks.workunit.client.1.smithi196.stdout:2/447: dwrite d1/d76/f7d [0,4194304] 0 2024-04-22T15:18:58.167 INFO:tasks.workunit.client.1.smithi196.stdout:5/412: fdatasync d1/d17/d50/f73 0 2024-04-22T15:18:58.168 INFO:tasks.workunit.client.1.smithi196.stdout:1/378: chown d1/db/d16/d1e/l2f 61 1 2024-04-22T15:18:58.169 INFO:tasks.workunit.client.1.smithi196.stdout:1/379: write d1/db/ff [1505857,65549] 0 2024-04-22T15:18:58.170 INFO:tasks.workunit.client.1.smithi196.stdout:5/413: dread d1/d17/d1a/f43 [0,4194304] 0 2024-04-22T15:18:58.175 INFO:tasks.workunit.client.1.smithi196.stdout:0/443: mkdir d3/d9/d36/d56/d65/d7f/d87/d6c/d88/d95 0 2024-04-22T15:18:58.177 INFO:tasks.workunit.client.1.smithi196.stdout:0/444: write d3/de/d1c/d27/f93 [70379,111929] 0 2024-04-22T15:18:58.178 INFO:tasks.workunit.client.1.smithi196.stdout:7/383: getdents d14/d26/d5a/d5f 0 2024-04-22T15:18:58.178 INFO:tasks.workunit.client.1.smithi196.stdout:0/445: chown d3/d6/dc/f58 514055737 1 2024-04-22T15:18:58.178 INFO:tasks.workunit.client.0.smithi096.stdout:2/754: dwrite d3/db/df/dce/d46/d59/f79 [0,4194304] 0 2024-04-22T15:18:58.179 INFO:tasks.workunit.client.1.smithi196.stdout:0/446: fsync d3/d9/f54 0 2024-04-22T15:18:58.181 INFO:tasks.workunit.client.0.smithi096.stdout:2/755: truncate d3/f2e 5209664 0 2024-04-22T15:18:58.186 INFO:tasks.workunit.client.1.smithi196.stdout:3/401: creat d0/d15/d4d/db/dc/d4a/f86 x:0 0 0 2024-04-22T15:18:58.186 INFO:tasks.workunit.client.1.smithi196.stdout:7/384: dwrite d14/d26/d2a/f66 [0,4194304] 0 2024-04-22T15:18:58.186 INFO:tasks.workunit.client.1.smithi196.stdout:6/442: mkdir dd/d1f/d37/d7b 0 2024-04-22T15:18:58.186 INFO:tasks.workunit.client.1.smithi196.stdout:3/402: readlink d0/d15/l21 0 2024-04-22T15:18:58.188 INFO:tasks.workunit.client.1.smithi196.stdout:6/443: chown dd/d20/f24 1240286145 1 2024-04-22T15:18:58.188 INFO:tasks.workunit.client.0.smithi096.stdout:2/756: mknod d3/db/df/d10/db5/cf1 0 2024-04-22T15:18:58.189 INFO:tasks.workunit.client.0.smithi096.stdout:2/757: chown d3/d13/db4 99002745 1 2024-04-22T15:18:58.194 INFO:tasks.workunit.client.1.smithi196.stdout:2/448: chown d1/d23/d27/d2e/d55/d5b/ca0 83 1 2024-04-22T15:18:58.196 INFO:tasks.workunit.client.1.smithi196.stdout:1/380: write d1/db/dc/d26/d2e/f34 [4750548,95620] 0 2024-04-22T15:18:58.197 INFO:tasks.workunit.client.1.smithi196.stdout:6/444: dwrite dd/d1f/d4e/f54 [0,4194304] 0 2024-04-22T15:18:58.197 INFO:tasks.workunit.client.1.smithi196.stdout:1/381: chown d1/db/d16/d6d 537547522 1 2024-04-22T15:18:58.199 INFO:tasks.workunit.client.1.smithi196.stdout:1/382: stat d1/db/dc/d26/d2e/c37 0 2024-04-22T15:18:58.200 INFO:tasks.workunit.client.1.smithi196.stdout:5/414: creat d1/d51/f84 x:0 0 0 2024-04-22T15:18:58.205 INFO:tasks.workunit.client.1.smithi196.stdout:8/435: getdents d2/d8/d49/d6c 0 2024-04-22T15:18:58.208 INFO:tasks.workunit.client.1.smithi196.stdout:1/383: dwrite d1/db/d16/f73 [0,4194304] 0 2024-04-22T15:18:58.209 INFO:tasks.workunit.client.1.smithi196.stdout:5/415: dwrite d1/d14/d3e/f44 [0,4194304] 0 2024-04-22T15:18:58.210 INFO:tasks.workunit.client.1.smithi196.stdout:5/416: dread - d1/d4/f56 zero size 2024-04-22T15:18:58.212 INFO:tasks.workunit.client.1.smithi196.stdout:5/417: dread - d1/d17/d1a/d2f/d30/f54 zero size 2024-04-22T15:18:58.215 INFO:tasks.workunit.client.1.smithi196.stdout:9/418: getdents db/dd 0 2024-04-22T15:18:58.218 INFO:tasks.workunit.client.1.smithi196.stdout:7/385: rename f13 to d14/d4d/f7e 0 2024-04-22T15:18:58.220 INFO:tasks.workunit.client.1.smithi196.stdout:3/403: mkdir d0/d15/d4d/d87 0 2024-04-22T15:18:58.223 INFO:tasks.workunit.client.1.smithi196.stdout:4/423: getdents d6/d14/d18/d40 0 2024-04-22T15:18:58.224 INFO:tasks.workunit.client.1.smithi196.stdout:4/424: fsync d6/d13/f2e 0 2024-04-22T15:18:58.228 INFO:tasks.workunit.client.1.smithi196.stdout:2/449: unlink d1/d23/l63 0 2024-04-22T15:18:58.229 INFO:tasks.workunit.client.1.smithi196.stdout:2/450: write d1/d23/d27/d2e/f5d [811240,90668] 0 2024-04-22T15:18:58.230 INFO:tasks.workunit.client.1.smithi196.stdout:6/445: creat dd/d20/d5c/d5e/f7c x:0 0 0 2024-04-22T15:18:58.230 INFO:tasks.workunit.client.1.smithi196.stdout:0/447: dread d3/de/d1c/f31 [0,4194304] 0 2024-04-22T15:18:58.231 INFO:tasks.workunit.client.1.smithi196.stdout:2/451: chown d1/f10 14787 1 2024-04-22T15:18:58.239 INFO:tasks.workunit.client.1.smithi196.stdout:7/386: creat d14/d4d/f7f x:0 0 0 2024-04-22T15:18:58.248 INFO:tasks.workunit.client.1.smithi196.stdout:6/446: creat dd/d20/f7d x:0 0 0 2024-04-22T15:18:58.255 INFO:tasks.workunit.client.1.smithi196.stdout:1/384: rename d1/d54/f55 to d1/db/d16/f7e 0 2024-04-22T15:18:58.256 INFO:tasks.workunit.client.1.smithi196.stdout:1/385: fdatasync d1/db/dc/de/d5a/d5b/f79 0 2024-04-22T15:18:58.258 INFO:tasks.workunit.client.1.smithi196.stdout:1/386: dread d1/db/dc/d26/f28 [0,4194304] 0 2024-04-22T15:18:58.265 INFO:tasks.workunit.client.1.smithi196.stdout:4/425: dread d6/d14/d18/d26/f43 [0,4194304] 0 2024-04-22T15:18:58.267 INFO:tasks.workunit.client.1.smithi196.stdout:6/447: rmdir dd/d1f 39 2024-04-22T15:18:58.268 INFO:tasks.workunit.client.1.smithi196.stdout:3/404: write d0/d7/f7d [2553560,74188] 0 2024-04-22T15:18:58.269 INFO:tasks.workunit.client.1.smithi196.stdout:3/405: truncate d0/d7/f5b 17202 0 2024-04-22T15:18:58.281 INFO:tasks.workunit.client.1.smithi196.stdout:3/406: dread d0/d7/d8/f19 [0,4194304] 0 2024-04-22T15:18:58.345 INFO:tasks.workunit.client.0.smithi096.stdout:2/758: symlink d3/db/df/d24/d58/d5f/lf2 0 2024-04-22T15:18:58.349 INFO:tasks.workunit.client.1.smithi196.stdout:5/418: link d1/d17/d1a/d2f/d30/l49 d1/d53/l85 0 2024-04-22T15:18:58.351 INFO:tasks.workunit.client.1.smithi196.stdout:7/387: unlink d14/d26/d2a/c44 0 2024-04-22T15:18:58.351 INFO:tasks.workunit.client.0.smithi096.stdout:2/759: dwrite d3/db/df/dce/d46/d7e/d8d/fcf [0,4194304] 0 2024-04-22T15:18:58.352 INFO:tasks.workunit.client.1.smithi196.stdout:8/436: write d2/dc/f20 [1713080,43731] 0 2024-04-22T15:18:58.352 INFO:tasks.workunit.client.0.smithi096.stdout:2/760: readlink d3/db/df/dce/d46/d7e/lc5 0 2024-04-22T15:18:58.355 INFO:tasks.workunit.client.1.smithi196.stdout:6/448: mknod dd/d20/c7e 0 2024-04-22T15:18:58.355 INFO:tasks.workunit.client.1.smithi196.stdout:0/448: dwrite d3/d9/d36/d56/d65/f74 [0,4194304] 0 2024-04-22T15:18:58.356 INFO:tasks.workunit.client.1.smithi196.stdout:0/449: truncate d3/d9/f54 1312066 0 2024-04-22T15:18:58.360 INFO:tasks.workunit.client.1.smithi196.stdout:9/419: truncate fa 164600 0 2024-04-22T15:18:58.361 INFO:tasks.workunit.client.0.smithi096.stdout:2/761: mknod d3/db/cf3 0 2024-04-22T15:18:58.363 INFO:tasks.workunit.client.1.smithi196.stdout:7/388: rmdir d14/d26/d5e 39 2024-04-22T15:18:58.367 INFO:tasks.workunit.client.1.smithi196.stdout:6/449: unlink dd/f14 0 2024-04-22T15:18:58.369 INFO:tasks.workunit.client.0.smithi096.stdout:2/762: dread d3/db/df/f60 [0,4194304] 0 2024-04-22T15:18:58.369 INFO:tasks.workunit.client.1.smithi196.stdout:1/387: sync 2024-04-22T15:18:58.370 INFO:tasks.workunit.client.1.smithi196.stdout:2/452: truncate d1/d13/d41/f6c 3443855 0 2024-04-22T15:18:58.370 INFO:tasks.workunit.client.1.smithi196.stdout:1/388: write d1/db/dc/de/d43/f7b [721477,77115] 0 2024-04-22T15:18:58.372 INFO:tasks.workunit.client.1.smithi196.stdout:1/389: read - d1/db/d16/f7e zero size 2024-04-22T15:18:58.372 INFO:tasks.workunit.client.1.smithi196.stdout:9/420: write db/dd/de/d15/d31/d64/f62 [509277,41964] 0 2024-04-22T15:18:58.373 INFO:tasks.workunit.client.1.smithi196.stdout:7/389: readlink d14/l2e 0 2024-04-22T15:18:58.373 INFO:tasks.workunit.client.1.smithi196.stdout:7/390: readlink d14/d26/d28/l36 0 2024-04-22T15:18:58.375 INFO:tasks.workunit.client.1.smithi196.stdout:4/426: rename d6/d13/d15/d2c/d62/c5c to d6/d13/d15/d2c/c7e 0 2024-04-22T15:18:58.377 INFO:tasks.workunit.client.1.smithi196.stdout:6/450: stat dd/d1f/d4e/d53 0 2024-04-22T15:18:58.378 INFO:tasks.workunit.client.1.smithi196.stdout:0/450: symlink d3/d9/d36/d56/d65/d7f/d87/d6c/d88/d95/l96 0 2024-04-22T15:18:58.380 INFO:tasks.workunit.client.1.smithi196.stdout:1/390: mkdir d1/db/d16/d1e/d49/d7f 0 2024-04-22T15:18:58.381 INFO:tasks.workunit.client.1.smithi196.stdout:9/421: mknod db/d11/d29/d7a/c96 0 2024-04-22T15:18:58.383 INFO:tasks.workunit.client.1.smithi196.stdout:7/391: mkdir d14/d25/d80 0 2024-04-22T15:18:58.385 INFO:tasks.workunit.client.1.smithi196.stdout:9/422: dread db/dd/de/d15/d31/d64/f62 [0,4194304] 0 2024-04-22T15:18:58.385 INFO:tasks.workunit.client.1.smithi196.stdout:5/419: rename d1/d14/f42 to d1/d53/f86 0 2024-04-22T15:18:58.386 INFO:tasks.workunit.client.1.smithi196.stdout:9/423: chown db/dd/d1b/d2c/l77 1034544 1 2024-04-22T15:18:58.386 INFO:tasks.workunit.client.1.smithi196.stdout:5/420: chown d1/d17/d1a/d2f/d30/d6e 933699 1 2024-04-22T15:18:58.388 INFO:tasks.workunit.client.0.smithi096.stdout:2/763: truncate d3/db/d30/f3e 2701765 0 2024-04-22T15:18:58.389 INFO:tasks.workunit.client.0.smithi096.stdout:2/764: fsync d3/d1f/f56 0 2024-04-22T15:18:58.389 INFO:tasks.workunit.client.1.smithi196.stdout:0/451: rmdir d3/d6/dc 39 2024-04-22T15:18:58.391 INFO:tasks.workunit.client.1.smithi196.stdout:2/453: symlink d1/la7 0 2024-04-22T15:18:58.393 INFO:tasks.workunit.client.1.smithi196.stdout:3/407: getdents d0/d15/d4d/db/dc 0 2024-04-22T15:18:58.394 INFO:tasks.workunit.client.0.smithi096.stdout:2/765: mknod d3/db/df/d24/dc8/cf4 0 2024-04-22T15:18:58.398 INFO:tasks.workunit.client.0.smithi096.stdout:2/766: mknod d3/db/df/d98/db3/cf5 0 2024-04-22T15:18:58.398 INFO:tasks.workunit.client.1.smithi196.stdout:1/391: dread d1/db/d16/d1e/f60 [0,4194304] 0 2024-04-22T15:18:58.402 INFO:tasks.workunit.client.1.smithi196.stdout:8/437: rename d2/d3f/d4b/l66 to d2/dc/d45/l7f 0 2024-04-22T15:18:58.402 INFO:tasks.workunit.client.1.smithi196.stdout:8/438: readlink d2/d8/d49/l50 0 2024-04-22T15:18:58.405 INFO:tasks.workunit.client.1.smithi196.stdout:5/421: symlink d1/d53/l87 0 2024-04-22T15:18:58.407 INFO:tasks.workunit.client.1.smithi196.stdout:0/452: unlink d3/d9/d36/d56/d65/f74 0 2024-04-22T15:18:58.410 INFO:tasks.workunit.client.1.smithi196.stdout:3/408: dread d0/d7/d8/d70/f7e [0,4194304] 0 2024-04-22T15:18:58.411 INFO:tasks.workunit.client.1.smithi196.stdout:2/454: rename d1/d13/d25/f59 to d1/d13/d41/fa8 0 2024-04-22T15:18:58.411 INFO:tasks.workunit.client.1.smithi196.stdout:4/427: link d6/d10/c4b d6/d14/d18/d40/d74/c7f 0 2024-04-22T15:18:58.412 INFO:tasks.workunit.client.1.smithi196.stdout:9/424: mknod db/dd/de/d75/c97 0 2024-04-22T15:18:58.413 INFO:tasks.workunit.client.1.smithi196.stdout:5/422: creat d1/d17/d1c/f88 x:0 0 0 2024-04-22T15:18:58.416 INFO:tasks.workunit.client.1.smithi196.stdout:1/392: mknod d1/db/dc/c80 0 2024-04-22T15:18:58.418 INFO:tasks.workunit.client.1.smithi196.stdout:3/409: mknod d0/d15/d4d/db/dc/d4a/c88 0 2024-04-22T15:18:58.418 INFO:tasks.workunit.client.1.smithi196.stdout:8/439: getdents d2/d3f/d4b/d79 0 2024-04-22T15:18:58.418 INFO:tasks.workunit.client.1.smithi196.stdout:7/392: write d14/d26/d2a/d2b/f2d [1193009,121578] 0 2024-04-22T15:18:58.419 INFO:tasks.workunit.client.1.smithi196.stdout:4/428: mknod d6/d10/c80 0 2024-04-22T15:18:58.420 INFO:tasks.workunit.client.1.smithi196.stdout:6/451: getdents dd/d1f/d4e/d53/d55 0 2024-04-22T15:18:58.422 INFO:tasks.workunit.client.1.smithi196.stdout:0/453: symlink d3/de/l97 0 2024-04-22T15:18:58.422 INFO:tasks.workunit.client.1.smithi196.stdout:3/410: unlink d0/d15/d31/d55/f83 0 2024-04-22T15:18:58.429 INFO:tasks.workunit.client.1.smithi196.stdout:8/440: dwrite d2/d5/d2e/f67 [0,4194304] 0 2024-04-22T15:18:58.429 INFO:tasks.workunit.client.1.smithi196.stdout:5/423: link d1/d17/c41 d1/d14/c89 0 2024-04-22T15:18:58.429 INFO:tasks.workunit.client.1.smithi196.stdout:1/393: dread d1/db/dc/de/f14 [0,4194304] 0 2024-04-22T15:18:58.431 INFO:tasks.workunit.client.1.smithi196.stdout:4/429: stat d6/d13/d15/d2c/d35/f4c 0 2024-04-22T15:18:58.432 INFO:tasks.workunit.client.1.smithi196.stdout:9/425: getdents db/dd/d1b 0 2024-04-22T15:18:58.433 INFO:tasks.workunit.client.1.smithi196.stdout:9/426: write db/d11/d29/d57/f33 [1045637,4492] 0 2024-04-22T15:18:58.434 INFO:tasks.workunit.client.1.smithi196.stdout:3/411: rename d0/d15/d31/d55/d5d/c7b to d0/d15/d31/d55/d5d/c89 0 2024-04-22T15:18:58.436 INFO:tasks.workunit.client.1.smithi196.stdout:7/393: link d14/d15/c2c d14/d15/c81 0 2024-04-22T15:18:58.440 INFO:tasks.workunit.client.1.smithi196.stdout:6/452: write f5 [164379,91230] 0 2024-04-22T15:18:58.441 INFO:tasks.workunit.client.1.smithi196.stdout:9/427: dwrite db/d11/f6a [0,4194304] 0 2024-04-22T15:18:58.444 INFO:tasks.workunit.client.1.smithi196.stdout:3/412: unlink d0/d15/d4d/db/dc/d4a/f46 0 2024-04-22T15:18:58.445 INFO:tasks.workunit.client.1.smithi196.stdout:2/455: dwrite d1/d23/d27/d2e/d55/f36 [0,4194304] 0 2024-04-22T15:18:58.445 INFO:tasks.workunit.client.1.smithi196.stdout:0/454: truncate d3/f5 1762309 0 2024-04-22T15:18:58.449 INFO:tasks.workunit.client.1.smithi196.stdout:1/394: write d1/db/d16/f40 [1520705,54574] 0 2024-04-22T15:18:58.449 INFO:tasks.workunit.client.1.smithi196.stdout:7/394: rmdir d14/d25 39 2024-04-22T15:18:58.452 INFO:tasks.workunit.client.1.smithi196.stdout:8/441: link d2/f32 d2/d3f/d4b/d79/f80 0 2024-04-22T15:18:58.453 INFO:tasks.workunit.client.1.smithi196.stdout:1/395: chown d1/db/dc/d26 6 1 2024-04-22T15:18:58.453 INFO:tasks.workunit.client.1.smithi196.stdout:8/442: chown d2/d8/d49 1 1 2024-04-22T15:18:58.453 INFO:tasks.workunit.client.1.smithi196.stdout:5/424: symlink d1/d17/d1a/d2f/d30/d6e/d7a/l8a 0 2024-04-22T15:18:58.454 INFO:tasks.workunit.client.1.smithi196.stdout:1/396: chown d1/db/d16/d1e 2609195 1 2024-04-22T15:18:58.454 INFO:tasks.workunit.client.1.smithi196.stdout:4/430: creat d6/d13/d15/d2c/d35/d44/f81 x:0 0 0 2024-04-22T15:18:58.455 INFO:tasks.workunit.client.1.smithi196.stdout:6/453: mkdir dd/d1f/d4e/d53/d7f 0 2024-04-22T15:18:58.455 INFO:tasks.workunit.client.1.smithi196.stdout:9/428: unlink db/d11/d29/d57/l87 0 2024-04-22T15:18:58.456 INFO:tasks.workunit.client.1.smithi196.stdout:3/413: rename d0/d7/f1a to d0/d15/d48/d51/f8a 0 2024-04-22T15:18:58.458 INFO:tasks.workunit.client.1.smithi196.stdout:1/397: write d1/db/dc/d26/d2e/d32/d36/f3f [155711,106532] 0 2024-04-22T15:18:58.458 INFO:tasks.workunit.client.1.smithi196.stdout:7/395: symlink d14/d26/d5a/l82 0 2024-04-22T15:18:58.460 INFO:tasks.workunit.client.1.smithi196.stdout:8/443: write d2/d3f/d4b/f72 [958147,84029] 0 2024-04-22T15:18:58.460 INFO:tasks.workunit.client.1.smithi196.stdout:2/456: symlink d1/d23/d27/d2e/d55/d5b/d93/d97/la9 0 2024-04-22T15:18:58.461 INFO:tasks.workunit.client.1.smithi196.stdout:4/431: truncate d6/d13/d15/d2c/d35/f4c 4641827 0 2024-04-22T15:18:58.461 INFO:tasks.workunit.client.1.smithi196.stdout:9/429: mkdir db/dd/de/d98 0 2024-04-22T15:18:58.464 INFO:tasks.workunit.client.1.smithi196.stdout:7/396: dread - d14/d25/f79 zero size 2024-04-22T15:18:58.464 INFO:tasks.workunit.client.1.smithi196.stdout:2/457: chown d1/f3e 3 1 2024-04-22T15:18:58.464 INFO:tasks.workunit.client.1.smithi196.stdout:5/425: truncate d1/d9/f11 264196 0 2024-04-22T15:18:58.465 INFO:tasks.workunit.client.1.smithi196.stdout:9/430: write db/d11/d29/f92 [721068,21707] 0 2024-04-22T15:18:58.466 INFO:tasks.workunit.client.1.smithi196.stdout:3/414: getdents d0/d15/d4d/db/dc/d18 0 2024-04-22T15:18:58.470 INFO:tasks.workunit.client.1.smithi196.stdout:6/454: dwrite dd/f18 [4194304,4194304] 0 2024-04-22T15:18:58.470 INFO:tasks.workunit.client.1.smithi196.stdout:5/426: fsync d1/d4/f22 0 2024-04-22T15:18:58.470 INFO:tasks.workunit.client.1.smithi196.stdout:9/431: fdatasync db/d11/d29/f2b 0 2024-04-22T15:18:58.473 INFO:tasks.workunit.client.1.smithi196.stdout:3/415: mkdir d0/d15/d31/d55/d8b 0 2024-04-22T15:18:58.476 INFO:tasks.workunit.client.1.smithi196.stdout:7/397: rename d14/d26/d28/d29/f6c to d14/d26/f83 0 2024-04-22T15:18:58.478 INFO:tasks.workunit.client.1.smithi196.stdout:8/444: dwrite d2/d35/f5f [0,4194304] 0 2024-04-22T15:18:58.479 INFO:tasks.workunit.client.1.smithi196.stdout:2/458: dread d1/d13/d25/d2a/f3f [0,4194304] 0 2024-04-22T15:18:58.480 INFO:tasks.workunit.client.1.smithi196.stdout:9/432: truncate db/dd/d1b/f45 4707942 0 2024-04-22T15:18:58.481 INFO:tasks.workunit.client.1.smithi196.stdout:6/455: truncate dd/d1f/d38/d40/f43 5020138 0 2024-04-22T15:18:58.481 INFO:tasks.workunit.client.1.smithi196.stdout:3/416: mkdir d0/d15/d4d/db/dc/d4a/d44/d8c 0 2024-04-22T15:18:58.484 INFO:tasks.workunit.client.1.smithi196.stdout:5/427: dread d1/d17/f24 [0,4194304] 0 2024-04-22T15:18:58.485 INFO:tasks.workunit.client.1.smithi196.stdout:8/445: rmdir d2/dc/d45 39 2024-04-22T15:18:58.485 INFO:tasks.workunit.client.1.smithi196.stdout:2/459: mknod d1/d13/d41/caa 0 2024-04-22T15:18:58.487 INFO:tasks.workunit.client.1.smithi196.stdout:9/433: fdatasync db/dd/de/f4e 0 2024-04-22T15:18:58.489 INFO:tasks.workunit.client.1.smithi196.stdout:7/398: dread d14/d15/f59 [0,4194304] 0 2024-04-22T15:18:58.490 INFO:tasks.workunit.client.1.smithi196.stdout:6/456: getdents dd/d1f/d37/d7b 0 2024-04-22T15:18:58.490 INFO:tasks.workunit.client.1.smithi196.stdout:3/417: creat d0/d15/d31/d55/d6e/f8d x:0 0 0 2024-04-22T15:18:58.492 INFO:tasks.workunit.client.1.smithi196.stdout:8/446: mkdir d2/d8/d47/d68/d81 0 2024-04-22T15:18:58.493 INFO:tasks.workunit.client.1.smithi196.stdout:3/418: chown d0/d15/d48/d51/d7f 13 1 2024-04-22T15:18:58.493 INFO:tasks.workunit.client.1.smithi196.stdout:4/432: write d6/d13/d15/f3d [496165,126548] 0 2024-04-22T15:18:58.493 INFO:tasks.workunit.client.1.smithi196.stdout:5/428: fdatasync d1/d17/d1c/f52 0 2024-04-22T15:18:58.495 INFO:tasks.workunit.client.1.smithi196.stdout:0/455: dwrite d3/de/f24 [4194304,4194304] 0 2024-04-22T15:18:58.498 INFO:tasks.workunit.client.1.smithi196.stdout:1/398: truncate d1/db/dc/de/d43/f6b 706243 0 2024-04-22T15:18:58.500 INFO:tasks.workunit.client.1.smithi196.stdout:6/457: dread dd/d1f/d37/f31 [4194304,4194304] 0 2024-04-22T15:18:58.500 INFO:tasks.workunit.client.1.smithi196.stdout:3/419: creat d0/d15/f8e x:0 0 0 2024-04-22T15:18:58.507 INFO:tasks.workunit.client.1.smithi196.stdout:7/399: fdatasync d14/d26/f83 0 2024-04-22T15:18:58.509 INFO:tasks.workunit.client.0.smithi096.stdout:3/633: sync 2024-04-22T15:18:58.510 INFO:tasks.workunit.client.1.smithi196.stdout:1/399: dwrite d1/db/d16/d1e/d65/f66 [0,4194304] 0 2024-04-22T15:18:58.510 INFO:tasks.workunit.client.0.smithi096.stdout:3/634: readlink da/db/de/daa/lac 0 2024-04-22T15:18:58.512 INFO:tasks.workunit.client.1.smithi196.stdout:0/456: getdents d3/d9/d36/d56/d65/d7f/d30 0 2024-04-22T15:18:58.517 INFO:tasks.workunit.client.1.smithi196.stdout:8/447: dwrite d2/dc/d45/f5b [4194304,4194304] 0 2024-04-22T15:18:58.518 INFO:tasks.workunit.client.0.smithi096.stdout:3/635: dread da/db/dc/d28/f45 [0,4194304] 0 2024-04-22T15:18:58.519 INFO:tasks.workunit.client.1.smithi196.stdout:6/458: symlink dd/d1f/d37/l80 0 2024-04-22T15:18:58.519 INFO:tasks.workunit.client.1.smithi196.stdout:2/460: dwrite d1/d13/d1f/f3a [4194304,4194304] 0 2024-04-22T15:18:58.520 INFO:tasks.workunit.client.1.smithi196.stdout:9/434: truncate db/d11/f6a 1728156 0 2024-04-22T15:18:58.520 INFO:tasks.workunit.client.1.smithi196.stdout:6/459: write dd/d1f/d38/f51 [1053617,121547] 0 2024-04-22T15:18:58.522 INFO:tasks.workunit.client.1.smithi196.stdout:3/420: write d0/d7/d8/f19 [3151002,13473] 0 2024-04-22T15:18:58.523 INFO:tasks.workunit.client.1.smithi196.stdout:4/433: write d6/d14/f27 [5576583,43664] 0 2024-04-22T15:18:58.524 INFO:tasks.workunit.client.1.smithi196.stdout:1/400: mknod d1/db/d16/d44/c81 0 2024-04-22T15:18:58.526 INFO:tasks.workunit.client.0.smithi096.stdout:3/636: rename da/db/d12/d22/d68/lbf to da/db/d12/d22/d26/lcc 0 2024-04-22T15:18:58.528 INFO:tasks.workunit.client.1.smithi196.stdout:8/448: rmdir d2/d3f 39 2024-04-22T15:18:58.530 INFO:tasks.workunit.client.0.smithi096.stdout:3/637: truncate da/d3f/f5b 1012370 0 2024-04-22T15:18:58.531 INFO:tasks.workunit.client.1.smithi196.stdout:4/434: dwrite d6/d13/d15/d2c/d62/f6d [0,4194304] 0 2024-04-22T15:18:58.532 INFO:tasks.workunit.client.1.smithi196.stdout:2/461: mknod d1/d13/d41/cab 0 2024-04-22T15:18:58.539 INFO:tasks.workunit.client.1.smithi196.stdout:0/457: unlink d3/d6/dc/c2a 0 2024-04-22T15:18:58.540 INFO:tasks.workunit.client.1.smithi196.stdout:3/421: rmdir d0/d15/d4d/db/dc 39 2024-04-22T15:18:58.540 INFO:tasks.workunit.client.1.smithi196.stdout:0/458: fsync d3/d9/d36/d56/d65/d7f/d51/f6a 0 2024-04-22T15:18:58.541 INFO:tasks.workunit.client.1.smithi196.stdout:3/422: readlink d0/d7/d8/l30 0 2024-04-22T15:18:58.542 INFO:tasks.workunit.client.1.smithi196.stdout:1/401: mknod d1/db/d16/d1e/c82 0 2024-04-22T15:18:58.542 INFO:tasks.workunit.client.1.smithi196.stdout:3/423: chown d0/d15/d48/d51/f54 218966099 1 2024-04-22T15:18:58.543 INFO:tasks.workunit.client.1.smithi196.stdout:1/402: chown d1/db/dc/de/d43/f7b 29385369 1 2024-04-22T15:18:58.544 INFO:tasks.workunit.client.1.smithi196.stdout:1/403: chown d1/f4 162213 1 2024-04-22T15:18:58.544 INFO:tasks.workunit.client.1.smithi196.stdout:2/462: mknod d1/d1d/d6a/da1/cac 0 2024-04-22T15:18:58.545 INFO:tasks.workunit.client.1.smithi196.stdout:5/429: link d1/d9/f20 d1/d17/f8b 0 2024-04-22T15:18:58.545 INFO:tasks.workunit.client.1.smithi196.stdout:5/430: dread - d1/d4/f56 zero size 2024-04-22T15:18:58.546 INFO:tasks.workunit.client.1.smithi196.stdout:6/460: creat dd/d1f/d4e/d53/d7f/f81 x:0 0 0 2024-04-22T15:18:58.547 INFO:tasks.workunit.client.1.smithi196.stdout:5/431: read d1/d17/d1a/f43 [484337,98895] 0 2024-04-22T15:18:58.547 INFO:tasks.workunit.client.1.smithi196.stdout:9/435: mkdir db/dd/de/d15/d31/d99 0 2024-04-22T15:18:58.547 INFO:tasks.workunit.client.1.smithi196.stdout:6/461: fdatasync dd/d20/d4d/f65 0 2024-04-22T15:18:58.549 INFO:tasks.workunit.client.1.smithi196.stdout:3/424: rename d0/d7/d6f/f76 to d0/d15/f8f 0 2024-04-22T15:18:58.550 INFO:tasks.workunit.client.1.smithi196.stdout:8/449: rmdir d2/d3f/d4b 39 2024-04-22T15:18:58.552 INFO:tasks.workunit.client.1.smithi196.stdout:2/463: fsync d1/d23/d27/d2e/d55/f87 0 2024-04-22T15:18:58.552 INFO:tasks.workunit.client.1.smithi196.stdout:0/459: mkdir d3/d6/dc/d8c/d8d/d98 0 2024-04-22T15:18:58.555 INFO:tasks.workunit.client.1.smithi196.stdout:8/450: fdatasync d2/d8/d47/f62 0 2024-04-22T15:18:58.556 INFO:tasks.workunit.client.1.smithi196.stdout:2/464: mknod d1/d23/cad 0 2024-04-22T15:18:58.557 INFO:tasks.workunit.client.1.smithi196.stdout:5/432: link d1/d17/d1a/d2f/d30/d6e/c6f d1/d17/d1a/d2f/d30/d5e/c8c 0 2024-04-22T15:18:58.560 INFO:tasks.workunit.client.1.smithi196.stdout:8/451: dread - d2/d5/f74 zero size 2024-04-22T15:18:58.564 INFO:tasks.workunit.client.1.smithi196.stdout:5/433: write d1/d17/f5d [501640,126264] 0 2024-04-22T15:18:58.566 INFO:tasks.workunit.client.0.smithi096.stdout:3/638: dread da/db/dc/d28/d69/d89/f65 [0,4194304] 0 2024-04-22T15:18:58.566 INFO:tasks.workunit.client.1.smithi196.stdout:2/465: rename d1/d23/d27/l4b to d1/d23/lae 0 2024-04-22T15:18:58.569 INFO:tasks.workunit.client.1.smithi196.stdout:3/425: dwrite d0/d7/d8/f5c [0,4194304] 0 2024-04-22T15:18:58.570 INFO:tasks.workunit.client.0.smithi096.stdout:3/639: unlink da/db/d12/d22/f3a 0 2024-04-22T15:18:58.573 INFO:tasks.workunit.client.0.smithi096.stdout:3/640: creat da/db/de/d8b/db5/fcd x:0 0 0 2024-04-22T15:18:58.574 INFO:tasks.workunit.client.1.smithi196.stdout:1/404: dwrite d1/db/d16/f7e [0,4194304] 0 2024-04-22T15:18:58.575 INFO:tasks.workunit.client.1.smithi196.stdout:2/466: stat d1/f7e 0 2024-04-22T15:18:58.575 INFO:tasks.workunit.client.1.smithi196.stdout:9/436: dwrite db/dd/f5c [4194304,4194304] 0 2024-04-22T15:18:58.577 INFO:tasks.workunit.client.1.smithi196.stdout:5/434: symlink d1/d17/d1a/d2f/d30/d5e/l8d 0 2024-04-22T15:18:58.581 INFO:tasks.workunit.client.1.smithi196.stdout:1/405: chown d1/db/dc/de/d5a/c5f 0 1 2024-04-22T15:18:58.581 INFO:tasks.workunit.client.1.smithi196.stdout:0/460: dwrite d3/de/f7d [0,4194304] 0 2024-04-22T15:18:58.582 INFO:tasks.workunit.client.1.smithi196.stdout:9/437: fsync db/dd/de/d15/f40 0 2024-04-22T15:18:58.586 INFO:tasks.workunit.client.1.smithi196.stdout:3/426: rename d0/d15/d4d/c23 to d0/d15/d31/d55/c90 0 2024-04-22T15:18:58.589 INFO:tasks.workunit.client.1.smithi196.stdout:0/461: rmdir d3 39 2024-04-22T15:18:58.590 INFO:tasks.workunit.client.1.smithi196.stdout:3/427: truncate d0/d15/d31/f53 425879 0 2024-04-22T15:18:58.590 INFO:tasks.workunit.client.1.smithi196.stdout:5/435: truncate d1/d17/d1a/d2f/d30/f54 464743 0 2024-04-22T15:18:58.591 INFO:tasks.workunit.client.1.smithi196.stdout:9/438: symlink db/d11/d29/d7a/d7b/l9a 0 2024-04-22T15:18:58.592 INFO:tasks.workunit.client.1.smithi196.stdout:2/467: creat d1/d13/d25/d9f/faf x:0 0 0 2024-04-22T15:18:58.593 INFO:tasks.workunit.client.1.smithi196.stdout:7/400: dwrite d14/d4a/f5b [0,4194304] 0 2024-04-22T15:18:58.596 INFO:tasks.workunit.client.0.smithi096.stdout:3/641: dread da/db/d12/d22/f88 [0,4194304] 0 2024-04-22T15:18:58.599 INFO:tasks.workunit.client.1.smithi196.stdout:5/436: read d1/d17/f8b [3195600,113779] 0 2024-04-22T15:18:58.599 INFO:tasks.workunit.client.1.smithi196.stdout:0/462: unlink d3/f11 0 2024-04-22T15:18:58.600 INFO:tasks.workunit.client.1.smithi196.stdout:2/468: creat d1/d23/d27/d89/fb0 x:0 0 0 2024-04-22T15:18:58.601 INFO:tasks.workunit.client.1.smithi196.stdout:2/469: fdatasync d1/d1d/d6a/f9c 0 2024-04-22T15:18:58.602 INFO:tasks.workunit.client.1.smithi196.stdout:7/401: creat d14/d26/d2a/f84 x:0 0 0 2024-04-22T15:18:58.603 INFO:tasks.workunit.client.0.smithi096.stdout:3/642: mknod da/db/d12/d9b/cce 0 2024-04-22T15:18:58.606 INFO:tasks.workunit.client.1.smithi196.stdout:5/437: unlink d1/d53/c5b 0 2024-04-22T15:18:58.607 INFO:tasks.workunit.client.1.smithi196.stdout:8/452: dread d2/dc/f2f [0,4194304] 0 2024-04-22T15:18:58.607 INFO:tasks.workunit.client.0.smithi096.stdout:3/643: creat da/db/de/d87/fcf x:0 0 0 2024-04-22T15:18:58.608 INFO:tasks.workunit.client.0.smithi096.stdout:3/644: write da/db/de/d8b/db5/fcd [60916,51247] 0 2024-04-22T15:18:58.608 INFO:tasks.workunit.client.1.smithi196.stdout:0/463: unlink d3/d6/dc/l19 0 2024-04-22T15:18:58.609 INFO:tasks.workunit.client.1.smithi196.stdout:8/453: dread d2/d5/f19 [0,4194304] 0 2024-04-22T15:18:58.610 INFO:tasks.workunit.client.1.smithi196.stdout:2/470: symlink d1/d23/d27/d2e/d55/d5b/d93/lb1 0 2024-04-22T15:18:58.612 INFO:tasks.workunit.client.1.smithi196.stdout:7/402: symlink d14/d26/d2a/l85 0 2024-04-22T15:18:58.613 INFO:tasks.workunit.client.1.smithi196.stdout:3/428: creat d0/d15/d4d/db/dc/d85/f91 x:0 0 0 2024-04-22T15:18:58.614 INFO:tasks.workunit.client.1.smithi196.stdout:5/438: truncate d1/d9/f20 3505127 0 2024-04-22T15:18:58.616 INFO:tasks.workunit.client.1.smithi196.stdout:9/439: write f4 [5063305,76420] 0 2024-04-22T15:18:58.616 INFO:tasks.workunit.client.1.smithi196.stdout:2/471: mknod d1/d13/d41/cb2 0 2024-04-22T15:18:58.618 INFO:tasks.workunit.client.1.smithi196.stdout:3/429: creat d0/d15/f92 x:0 0 0 2024-04-22T15:18:58.618 INFO:tasks.workunit.client.1.smithi196.stdout:9/440: dread - db/d11/d29/d7a/f7d zero size 2024-04-22T15:18:58.619 INFO:tasks.workunit.client.1.smithi196.stdout:9/441: chown db/d11/d29/d57/l20 170825741 1 2024-04-22T15:18:58.620 INFO:tasks.workunit.client.1.smithi196.stdout:7/403: symlink d14/l86 0 2024-04-22T15:18:58.620 INFO:tasks.workunit.client.1.smithi196.stdout:7/404: chown d14/f3a 5784917 1 2024-04-22T15:18:58.623 INFO:tasks.workunit.client.1.smithi196.stdout:8/454: rename d2/d5/f48 to d2/dc/f82 0 2024-04-22T15:18:58.624 INFO:tasks.workunit.client.1.smithi196.stdout:9/442: creat db/d11/d29/f9b x:0 0 0 2024-04-22T15:18:58.625 INFO:tasks.workunit.client.1.smithi196.stdout:3/430: dread d0/d15/d4d/db/dc/d4a/d44/f20 [0,4194304] 0 2024-04-22T15:18:58.632 INFO:tasks.workunit.client.1.smithi196.stdout:5/439: rename d1/d17/l2d to d1/d4/l8e 0 2024-04-22T15:18:58.637 INFO:tasks.workunit.client.0.smithi096.stdout:3/645: dread da/db/d12/d22/f4e [0,4194304] 0 2024-04-22T15:18:58.637 INFO:tasks.workunit.client.1.smithi196.stdout:9/443: getdents db/dd/de 0 2024-04-22T15:18:58.638 INFO:tasks.workunit.client.1.smithi196.stdout:2/472: write d1/d1d/f68 [562044,70219] 0 2024-04-22T15:18:58.640 INFO:tasks.workunit.client.1.smithi196.stdout:0/464: dwrite d3/de/d14/f81 [0,4194304] 0 2024-04-22T15:18:58.641 INFO:tasks.workunit.client.1.smithi196.stdout:0/465: dread - d3/d9/d36/f7e zero size 2024-04-22T15:18:58.642 INFO:tasks.workunit.client.1.smithi196.stdout:7/405: truncate f10 674208 0 2024-04-22T15:18:58.645 INFO:tasks.workunit.client.1.smithi196.stdout:2/473: creat d1/d23/d27/d2e/d55/d5b/fb3 x:0 0 0 2024-04-22T15:18:58.646 INFO:tasks.workunit.client.1.smithi196.stdout:3/431: dread d0/f3 [0,4194304] 0 2024-04-22T15:18:58.647 INFO:tasks.workunit.client.1.smithi196.stdout:9/444: unlink db/d11/d29/l83 0 2024-04-22T15:18:58.647 INFO:tasks.workunit.client.0.smithi096.stdout:3/646: link da/db/de/d87/lca da/db/d6a/ld0 0 2024-04-22T15:18:58.648 INFO:tasks.workunit.client.1.smithi196.stdout:9/445: write db/dd/de/f4e [236231,8218] 0 2024-04-22T15:18:58.648 INFO:tasks.workunit.client.1.smithi196.stdout:5/440: write d1/d17/d1a/d2f/d30/d5e/f77 [60514,82423] 0 2024-04-22T15:18:58.649 INFO:tasks.workunit.client.1.smithi196.stdout:0/466: unlink d3/d9/f21 0 2024-04-22T15:18:58.651 INFO:tasks.workunit.client.1.smithi196.stdout:2/474: creat d1/d13/d25/fb4 x:0 0 0 2024-04-22T15:18:58.655 INFO:tasks.workunit.client.1.smithi196.stdout:9/446: creat db/d11/d29/d7a/f9c x:0 0 0 2024-04-22T15:18:58.656 INFO:tasks.workunit.client.0.smithi096.stdout:3/647: getdents da 0 2024-04-22T15:18:58.657 INFO:tasks.workunit.client.1.smithi196.stdout:5/441: rmdir d1/d17/d1a/d2f 39 2024-04-22T15:18:58.657 INFO:tasks.workunit.client.1.smithi196.stdout:0/467: dwrite d3/d9/d36/d56/d65/d7f/d30/f46 [0,4194304] 0 2024-04-22T15:18:58.657 INFO:tasks.workunit.client.1.smithi196.stdout:7/406: mknod d14/c87 0 2024-04-22T15:18:58.658 INFO:tasks.workunit.client.1.smithi196.stdout:3/432: mknod d0/d15/d31/d55/d8b/c93 0 2024-04-22T15:18:58.659 INFO:tasks.workunit.client.1.smithi196.stdout:5/442: readlink d1/d4/l4d 0 2024-04-22T15:18:58.660 INFO:tasks.workunit.client.1.smithi196.stdout:9/447: write db/dd/f5c [2712314,106787] 0 2024-04-22T15:18:58.663 INFO:tasks.workunit.client.1.smithi196.stdout:2/475: dwrite d1/d23/d27/d2e/d55/f36 [4194304,4194304] 0 2024-04-22T15:18:58.665 INFO:tasks.workunit.client.1.smithi196.stdout:0/468: creat d3/f99 x:0 0 0 2024-04-22T15:18:58.672 INFO:tasks.workunit.client.1.smithi196.stdout:0/469: dread d3/d6/dc/f92 [0,4194304] 0 2024-04-22T15:18:58.672 INFO:tasks.workunit.client.1.smithi196.stdout:9/448: rename db/d11/d29/d57/l20 to db/d11/d29/d7a/d7b/d81/l9d 0 2024-04-22T15:18:58.675 INFO:tasks.workunit.client.1.smithi196.stdout:5/443: dwrite d1/d17/d1a/d2f/d30/f54 [0,4194304] 0 2024-04-22T15:18:58.676 INFO:tasks.workunit.client.1.smithi196.stdout:2/476: mkdir d1/d23/d27/d2e/db5 0 2024-04-22T15:18:58.676 INFO:tasks.workunit.client.1.smithi196.stdout:0/470: write d3/d9/d36/d56/d65/d7f/d51/f53 [1441851,44199] 0 2024-04-22T15:18:58.679 INFO:tasks.workunit.client.1.smithi196.stdout:2/477: read - d1/d23/d27/d2e/d55/d5b/fb3 zero size 2024-04-22T15:18:58.681 INFO:tasks.workunit.client.1.smithi196.stdout:2/478: write d1/f16 [3515363,15379] 0 2024-04-22T15:18:58.683 INFO:tasks.workunit.client.0.smithi096.stdout:3/648: dwrite da/db/dc/d28/f93 [0,4194304] 0 2024-04-22T15:18:58.685 INFO:tasks.workunit.client.1.smithi196.stdout:9/449: rmdir db/dd/de/d15/d31/d64 39 2024-04-22T15:18:58.689 INFO:tasks.workunit.client.1.smithi196.stdout:9/450: dread db/dd/de/d15/d31/f32 [0,4194304] 0 2024-04-22T15:18:58.690 INFO:tasks.workunit.client.0.smithi096.stdout:3/649: mkdir da/db/d9a/dd1 0 2024-04-22T15:18:58.693 INFO:tasks.workunit.client.1.smithi196.stdout:5/444: rename d1/d17/d1c/c3d to d1/d17/d1a/d2f/d30/d6e/c8f 0 2024-04-22T15:18:58.693 INFO:tasks.workunit.client.1.smithi196.stdout:3/433: write d0/d15/d4d/db/f2e [4860852,88727] 0 2024-04-22T15:18:58.695 INFO:tasks.workunit.client.1.smithi196.stdout:0/471: creat d3/d9/d36/d56/d65/d7f/f9a x:0 0 0 2024-04-22T15:18:58.696 INFO:tasks.workunit.client.0.smithi096.stdout:3/650: rmdir da/db/de/d87 39 2024-04-22T15:18:58.696 INFO:tasks.workunit.client.0.smithi096.stdout:3/651: chown da/db/de/d38 631666 1 2024-04-22T15:18:58.697 INFO:tasks.workunit.client.0.smithi096.stdout:3/652: chown da/db/d12/d9b/cce 254709889 1 2024-04-22T15:18:58.697 INFO:tasks.workunit.client.1.smithi196.stdout:5/445: write d1/d17/f5d [62245,30400] 0 2024-04-22T15:18:58.698 INFO:tasks.workunit.client.1.smithi196.stdout:2/479: symlink d1/d13/d1f/lb6 0 2024-04-22T15:18:58.698 INFO:tasks.workunit.client.1.smithi196.stdout:3/434: chown d0/d15/d4d/db/dc/l73 822 1 2024-04-22T15:18:58.700 INFO:tasks.workunit.client.1.smithi196.stdout:9/451: dwrite db/dd/d1b/f68 [0,4194304] 0 2024-04-22T15:18:58.701 INFO:tasks.workunit.client.1.smithi196.stdout:7/407: dwrite f10 [0,4194304] 0 2024-04-22T15:18:58.703 INFO:tasks.workunit.client.1.smithi196.stdout:2/480: dread - d1/d23/d27/d2e/f85 zero size 2024-04-22T15:18:58.704 INFO:tasks.workunit.client.0.smithi096.stdout:3/653: mkdir da/db/d12/d22/d26/dc6/dd2 0 2024-04-22T15:18:58.705 INFO:tasks.workunit.client.1.smithi196.stdout:0/472: unlink d3/de/c79 0 2024-04-22T15:18:58.706 INFO:tasks.workunit.client.1.smithi196.stdout:3/435: write d0/d15/d4d/db/dc/d4a/d44/f26 [7840764,56792] 0 2024-04-22T15:18:58.707 INFO:tasks.workunit.client.0.smithi096.stdout:3/654: readlink da/db/d12/laf 0 2024-04-22T15:18:58.708 INFO:tasks.workunit.client.1.smithi196.stdout:0/473: stat d3/d9/d36/d56/d65/l66 0 2024-04-22T15:18:58.711 INFO:tasks.workunit.client.1.smithi196.stdout:0/474: read d3/d9/d36/d56/d65/d7f/d30/f46 [3352593,27545] 0 2024-04-22T15:18:58.711 INFO:tasks.workunit.client.1.smithi196.stdout:9/452: dwrite db/dd/de/f54 [0,4194304] 0 2024-04-22T15:18:58.713 INFO:tasks.workunit.client.1.smithi196.stdout:2/481: rename d1/d13/d25/c32 to d1/d13/d25/d2a/cb7 0 2024-04-22T15:18:58.713 INFO:tasks.workunit.client.1.smithi196.stdout:0/475: write d3/de/d14/f55 [41536,96494] 0 2024-04-22T15:18:58.714 INFO:tasks.workunit.client.1.smithi196.stdout:3/436: creat d0/d15/d4d/db/dc/d85/f94 x:0 0 0 2024-04-22T15:18:58.716 INFO:tasks.workunit.client.1.smithi196.stdout:7/408: rmdir d14/d26/d5a/d60 0 2024-04-22T15:18:58.719 INFO:tasks.workunit.client.1.smithi196.stdout:9/453: creat db/dd/f9e x:0 0 0 2024-04-22T15:18:58.719 INFO:tasks.workunit.client.1.smithi196.stdout:7/409: write d14/d26/f35 [3709235,71663] 0 2024-04-22T15:18:58.719 INFO:tasks.workunit.client.1.smithi196.stdout:5/446: getdents d1/d17/d1c/d67 0 2024-04-22T15:18:58.720 INFO:tasks.workunit.client.0.smithi096.stdout:7/963: sync 2024-04-22T15:18:58.722 INFO:tasks.workunit.client.0.smithi096.stdout:7/964: truncate d31/d32/d36/d75/dfe/f117 709170 0 2024-04-22T15:18:58.730 INFO:tasks.workunit.client.0.smithi096.stdout:3/655: write da/db/de/d19/d3e/f42 [1834950,25753] 0 2024-04-22T15:18:58.731 INFO:tasks.workunit.client.0.smithi096.stdout:3/656: mknod da/db/dc/d8d/cd3 0 2024-04-22T15:18:58.734 INFO:tasks.workunit.client.0.smithi096.stdout:3/657: unlink da/db/d12/f77 0 2024-04-22T15:18:58.736 INFO:tasks.workunit.client.0.smithi096.stdout:3/658: creat da/db/d12/d22/d26/d52/fd4 x:0 0 0 2024-04-22T15:18:58.737 INFO:tasks.workunit.client.0.smithi096.stdout:3/659: chown da/db/d12/f56 510355829 1 2024-04-22T15:18:58.739 INFO:tasks.workunit.client.0.smithi096.stdout:3/660: write da/db/d12/d22/f88 [5090503,104213] 0 2024-04-22T15:18:58.739 INFO:tasks.workunit.client.1.smithi196.stdout:4/435: sync 2024-04-22T15:18:58.743 INFO:tasks.workunit.client.0.smithi096.stdout:7/965: dwrite fa [0,4194304] 0 2024-04-22T15:18:58.746 INFO:tasks.workunit.client.0.smithi096.stdout:1/779: sync 2024-04-22T15:18:58.746 INFO:tasks.workunit.client.0.smithi096.stdout:0/867: sync 2024-04-22T15:18:58.746 INFO:tasks.workunit.client.0.smithi096.stdout:4/756: sync 2024-04-22T15:18:58.746 INFO:tasks.workunit.client.0.smithi096.stdout:6/603: sync 2024-04-22T15:18:58.746 INFO:tasks.workunit.client.0.smithi096.stdout:5/733: sync 2024-04-22T15:18:58.746 INFO:tasks.workunit.client.0.smithi096.stdout:9/653: sync 2024-04-22T15:18:58.747 INFO:tasks.workunit.client.0.smithi096.stdout:2/767: sync 2024-04-22T15:18:58.747 INFO:tasks.workunit.client.0.smithi096.stdout:8/746: sync 2024-04-22T15:18:58.748 INFO:tasks.workunit.client.0.smithi096.stdout:8/747: dread - d4/d2a/fe6 zero size 2024-04-22T15:18:58.749 INFO:tasks.workunit.client.0.smithi096.stdout:2/768: dread - d3/db/d30/fe2 zero size 2024-04-22T15:18:58.750 INFO:tasks.workunit.client.0.smithi096.stdout:3/661: write da/db/d12/f56 [1000644,33902] 0 2024-04-22T15:18:58.751 INFO:tasks.workunit.client.0.smithi096.stdout:6/604: mknod d3/d3a/d75/d83/ccf 0 2024-04-22T15:18:58.751 INFO:tasks.workunit.client.0.smithi096.stdout:1/780: fsync f2 0 2024-04-22T15:18:58.753 INFO:tasks.workunit.client.0.smithi096.stdout:7/966: getdents d31/d32/d13b 0 2024-04-22T15:18:58.754 INFO:tasks.workunit.client.1.smithi196.stdout:7/410: stat d14/d26/d2a/d2b/f2d 0 2024-04-22T15:18:58.755 INFO:tasks.workunit.client.1.smithi196.stdout:3/437: fdatasync d0/d15/d4d/db/dc/d4a/d44/f3c 0 2024-04-22T15:18:58.755 INFO:tasks.workunit.client.0.smithi096.stdout:5/734: rename d4/ddb/d3d/d4d/d59/f7f to d4/ff5 0 2024-04-22T15:18:58.755 INFO:tasks.workunit.client.1.smithi196.stdout:9/454: readlink db/dd/de/d15/l67 0 2024-04-22T15:18:58.756 INFO:tasks.workunit.client.0.smithi096.stdout:4/757: write d5/d7/d2c/d3d/f49 [5084016,1021] 0 2024-04-22T15:18:58.757 INFO:tasks.workunit.client.1.smithi196.stdout:2/482: dwrite d1/d23/d27/d89/f8c [0,4194304] 0 2024-04-22T15:18:58.758 INFO:tasks.workunit.client.0.smithi096.stdout:8/748: fsync d4/d10/d25/fe1 0 2024-04-22T15:18:58.758 INFO:tasks.workunit.client.0.smithi096.stdout:9/654: dwrite d1/f38 [4194304,4194304] 0 2024-04-22T15:18:58.759 INFO:tasks.workunit.client.0.smithi096.stdout:0/868: creat d7/d42/dfd/f128 x:0 0 0 2024-04-22T15:18:58.767 INFO:tasks.workunit.client.1.smithi196.stdout:7/411: dwrite f10 [4194304,4194304] 0 2024-04-22T15:18:58.768 INFO:tasks.workunit.client.0.smithi096.stdout:0/869: dwrite d7/de/d5b/d95/dc/fed [0,4194304] 0 2024-04-22T15:18:58.772 INFO:tasks.workunit.client.1.smithi196.stdout:9/455: creat db/dd/d1b/d2c/f9f x:0 0 0 2024-04-22T15:18:58.776 INFO:tasks.workunit.client.1.smithi196.stdout:2/483: symlink d1/d1d/d6a/lb8 0 2024-04-22T15:18:58.779 INFO:tasks.workunit.client.1.smithi196.stdout:7/412: unlink d14/d4a/f5b 0 2024-04-22T15:18:58.781 INFO:tasks.workunit.client.1.smithi196.stdout:7/413: chown d14/d25/d31/f40 27873482 1 2024-04-22T15:18:58.784 INFO:tasks.workunit.client.1.smithi196.stdout:3/438: dread d0/d15/d4d/f5a [0,4194304] 0 2024-04-22T15:18:58.785 INFO:tasks.workunit.client.1.smithi196.stdout:2/484: rename d1/d13/f50 to d1/d76/fb9 0 2024-04-22T15:18:58.785 INFO:tasks.workunit.client.1.smithi196.stdout:2/485: fdatasync d1/f16 0 2024-04-22T15:18:58.787 INFO:tasks.workunit.client.1.smithi196.stdout:9/456: chown db/d11/d29/f6f 968745 1 2024-04-22T15:18:58.788 INFO:tasks.workunit.client.1.smithi196.stdout:7/414: mknod d14/d26/d5a/d5f/c88 0 2024-04-22T15:18:58.790 INFO:tasks.workunit.client.1.smithi196.stdout:7/415: write d14/d26/d2a/d2b/f77 [611088,128560] 0 2024-04-22T15:18:58.790 INFO:tasks.workunit.client.1.smithi196.stdout:3/439: dwrite d0/d15/f12 [0,4194304] 0 2024-04-22T15:18:58.791 INFO:tasks.workunit.client.1.smithi196.stdout:7/416: readlink d14/d26/d2a/l4f 0 2024-04-22T15:18:58.792 INFO:tasks.workunit.client.1.smithi196.stdout:2/486: link d1/d23/d27/d2e/d55/d5b/d7c/f91 d1/d23/fba 0 2024-04-22T15:18:58.793 INFO:tasks.workunit.client.1.smithi196.stdout:7/417: dread d14/f33 [0,4194304] 0 2024-04-22T15:18:58.793 INFO:tasks.workunit.client.1.smithi196.stdout:2/487: chown d1/d23/d27/d2e/f85 30 1 2024-04-22T15:18:58.794 INFO:tasks.workunit.client.1.smithi196.stdout:9/457: rename db/c78 to db/d11/d29/d7a/d7b/ca0 0 2024-04-22T15:18:58.794 INFO:tasks.workunit.client.1.smithi196.stdout:2/488: chown d1/d13/d1f/c26 76438039 1 2024-04-22T15:18:58.796 INFO:tasks.workunit.client.1.smithi196.stdout:7/418: rmdir d14/d26/d5a/d5f 39 2024-04-22T15:18:58.797 INFO:tasks.workunit.client.1.smithi196.stdout:9/458: truncate db/d11/d29/d7a/f9c 296655 0 2024-04-22T15:18:58.798 INFO:tasks.workunit.client.1.smithi196.stdout:2/489: creat d1/d1d/fbb x:0 0 0 2024-04-22T15:18:58.806 INFO:tasks.workunit.client.1.smithi196.stdout:4/436: write d6/d13/d15/d2c/d35/d44/f4a [359502,60127] 0 2024-04-22T15:18:58.806 INFO:tasks.workunit.client.1.smithi196.stdout:7/419: dwrite d14/d25/f7a [0,4194304] 0 2024-04-22T15:18:58.807 INFO:tasks.workunit.client.1.smithi196.stdout:5/447: truncate d1/d9/f38 1655865 0 2024-04-22T15:18:58.807 INFO:tasks.workunit.client.0.smithi096.stdout:5/735: creat d4/ddb/d25/ff6 x:0 0 0 2024-04-22T15:18:58.807 INFO:tasks.workunit.client.0.smithi096.stdout:2/769: truncate d3/f53 3602399 0 2024-04-22T15:18:58.807 INFO:tasks.workunit.client.0.smithi096.stdout:3/662: write da/f55 [698976,65659] 0 2024-04-22T15:18:58.808 INFO:tasks.workunit.client.0.smithi096.stdout:1/781: write d6/fcf [987173,22443] 0 2024-04-22T15:18:58.808 INFO:tasks.workunit.client.0.smithi096.stdout:8/749: truncate d4/d71/f77 578936 0 2024-04-22T15:18:58.811 INFO:tasks.workunit.client.1.smithi196.stdout:7/420: write d14/d25/d31/f69 [1514946,10977] 0 2024-04-22T15:18:58.811 INFO:tasks.workunit.client.0.smithi096.stdout:4/758: mkdir d5/d7/d1a/d105 0 2024-04-22T15:18:58.811 INFO:tasks.workunit.client.0.smithi096.stdout:6/605: link d3/d3a/d75/l66 d3/d28/d54/d7e/ld0 0 2024-04-22T15:18:58.811 INFO:tasks.workunit.client.0.smithi096.stdout:0/870: mknod d7/de/d5b/d95/dc/c129 0 2024-04-22T15:18:58.812 INFO:tasks.workunit.client.1.smithi196.stdout:4/437: mknod d6/d14/d18/d26/d32/d67/c82 0 2024-04-22T15:18:58.813 INFO:tasks.workunit.client.1.smithi196.stdout:5/448: dread d1/d17/f24 [0,4194304] 0 2024-04-22T15:18:58.813 INFO:tasks.workunit.client.1.smithi196.stdout:7/421: fsync d14/d4a/f71 0 2024-04-22T15:18:58.814 INFO:tasks.workunit.client.1.smithi196.stdout:4/438: rename d6/d14/d18/d40 to d6/d14/d18/d40/d74/d83 22 2024-04-22T15:18:58.814 INFO:tasks.workunit.client.1.smithi196.stdout:5/449: dread - d1/d17/d1a/d2f/d30/f80 zero size 2024-04-22T15:18:58.815 INFO:tasks.workunit.client.1.smithi196.stdout:7/422: truncate d14/d4d/f7f 763572 0 2024-04-22T15:18:58.818 INFO:tasks.workunit.client.1.smithi196.stdout:3/440: dread d0/d15/d4d/db/dc/d4a/f50 [0,4194304] 0 2024-04-22T15:18:58.819 INFO:tasks.workunit.client.0.smithi096.stdout:3/663: dread da/db/d10/f90 [0,4194304] 0 2024-04-22T15:18:58.819 INFO:tasks.workunit.client.1.smithi196.stdout:4/439: mkdir d6/d14/d18/d40/d74/d84 0 2024-04-22T15:18:58.820 INFO:tasks.workunit.client.0.smithi096.stdout:0/871: dwrite d7/d42/f113 [0,4194304] 0 2024-04-22T15:18:58.822 INFO:tasks.workunit.client.1.smithi196.stdout:5/450: dwrite d1/d17/d1a/d2f/d30/f6b [0,4194304] 0 2024-04-22T15:18:58.823 INFO:tasks.workunit.client.0.smithi096.stdout:8/750: mkdir d4/d1c/d62/dc3/df9 0 2024-04-22T15:18:58.825 INFO:tasks.workunit.client.0.smithi096.stdout:1/782: mknod d6/dc/d8a/d3f/d54/d32/d3b/cf0 0 2024-04-22T15:18:58.829 INFO:tasks.workunit.client.1.smithi196.stdout:4/440: mkdir d6/d14/d18/d40/d85 0 2024-04-22T15:18:58.832 INFO:tasks.workunit.client.1.smithi196.stdout:3/441: symlink d0/l95 0 2024-04-22T15:18:58.832 INFO:tasks.workunit.client.0.smithi096.stdout:7/967: write d31/d32/d33/d6a/d83/dba/dc7/fcd [2800077,49875] 0 2024-04-22T15:18:58.833 INFO:tasks.workunit.client.0.smithi096.stdout:7/968: stat d31/d32/l78 0 2024-04-22T15:18:58.834 INFO:tasks.workunit.client.1.smithi196.stdout:9/459: dwrite db/d11/f6a [0,4194304] 0 2024-04-22T15:18:58.835 INFO:tasks.workunit.client.1.smithi196.stdout:5/451: write d1/d17/d50/f6d [4649352,47219] 0 2024-04-22T15:18:58.837 INFO:tasks.workunit.client.1.smithi196.stdout:4/441: stat d6/c5d 0 2024-04-22T15:18:58.838 INFO:tasks.workunit.client.1.smithi196.stdout:5/452: write d1/d51/f69 [552920,89914] 0 2024-04-22T15:18:58.840 INFO:tasks.workunit.client.1.smithi196.stdout:7/423: write d14/f1e [2916684,5033] 0 2024-04-22T15:18:58.840 INFO:tasks.workunit.client.1.smithi196.stdout:4/442: chown d6/d14/d18/d26/d32/l76 7985 1 2024-04-22T15:18:58.842 INFO:tasks.workunit.client.0.smithi096.stdout:5/736: fsync d4/ddb/d3d/d4d/f70 0 2024-04-22T15:18:58.844 INFO:tasks.workunit.client.1.smithi196.stdout:3/442: mkdir d0/d15/d4d/db/dc/d4a/d44/d96 0 2024-04-22T15:18:58.846 INFO:tasks.workunit.client.0.smithi096.stdout:3/664: mkdir da/db/d12/dd5 0 2024-04-22T15:18:58.847 INFO:tasks.workunit.client.0.smithi096.stdout:3/665: write da/db/d12/d22/f53 [3778029,46141] 0 2024-04-22T15:18:58.847 INFO:tasks.workunit.client.1.smithi196.stdout:5/453: dwrite d1/d51/f69 [0,4194304] 0 2024-04-22T15:18:58.848 INFO:tasks.workunit.client.1.smithi196.stdout:9/460: rename db/dd/d1b/d2c/f6d to db/dd/de/d15/d31/d99/fa1 0 2024-04-22T15:18:58.850 INFO:tasks.workunit.client.1.smithi196.stdout:5/454: read d1/d17/f2e [2589261,33135] 0 2024-04-22T15:18:58.850 INFO:tasks.workunit.client.0.smithi096.stdout:8/751: symlink d4/d1c/d40/d48/lfa 0 2024-04-22T15:18:58.852 INFO:tasks.workunit.client.1.smithi196.stdout:4/443: mknod d6/d13/d15/d2c/d62/d78/c86 0 2024-04-22T15:18:58.854 INFO:tasks.workunit.client.0.smithi096.stdout:3/666: dwrite da/f55 [0,4194304] 0 2024-04-22T15:18:58.854 INFO:tasks.workunit.client.0.smithi096.stdout:1/783: creat d6/dc/d8a/d3f/d54/d32/d3b/d56/ff1 x:0 0 0 2024-04-22T15:18:58.857 INFO:tasks.workunit.client.0.smithi096.stdout:6/606: fdatasync d3/d28/dd/d11/d2c/f39 0 2024-04-22T15:18:58.861 INFO:tasks.workunit.client.1.smithi196.stdout:9/461: unlink db/dd/de/d15/d31/f47 0 2024-04-22T15:18:58.862 INFO:tasks.workunit.client.0.smithi096.stdout:4/759: write d5/d7/d1f/d84/d6d/fae [269537,3035] 0 2024-04-22T15:18:58.863 INFO:tasks.workunit.client.0.smithi096.stdout:7/969: dread d31/d32/d33/d5e/f6d [0,4194304] 0 2024-04-22T15:18:58.864 INFO:tasks.workunit.client.0.smithi096.stdout:5/737: rmdir d4/ddb/d3d/d4d/d59/db0/db2 39 2024-04-22T15:18:58.865 INFO:tasks.workunit.client.1.smithi196.stdout:3/443: dread d0/d7/f38 [0,4194304] 0 2024-04-22T15:18:58.866 INFO:tasks.workunit.client.1.smithi196.stdout:6/462: sync 2024-04-22T15:18:58.866 INFO:tasks.workunit.client.1.smithi196.stdout:1/406: sync 2024-04-22T15:18:58.867 INFO:tasks.workunit.client.1.smithi196.stdout:3/444: dread - d0/d15/d4d/db/dc/d4a/d44/f6c zero size 2024-04-22T15:18:58.867 INFO:tasks.workunit.client.1.smithi196.stdout:1/407: write d1/db/dc/de/d5a/d5b/f79 [359325,98490] 0 2024-04-22T15:18:58.868 INFO:tasks.workunit.client.1.smithi196.stdout:3/445: stat d0/d7/d8/f5c 0 2024-04-22T15:18:58.872 INFO:tasks.workunit.client.0.smithi096.stdout:3/667: creat da/db/d12/d9b/fd6 x:0 0 0 2024-04-22T15:18:58.875 INFO:tasks.workunit.client.0.smithi096.stdout:1/784: symlink d6/dc/d8a/d3f/d54/d32/dcd/lf2 0 2024-04-22T15:18:58.875 INFO:tasks.workunit.client.0.smithi096.stdout:2/770: truncate d3/db/f31 3468282 0 2024-04-22T15:18:58.879 INFO:tasks.workunit.client.1.smithi196.stdout:5/455: rename d1/d17/d1a/d2f/d30/d6e/c6f to d1/d9/d68/c90 0 2024-04-22T15:18:58.882 INFO:tasks.workunit.client.0.smithi096.stdout:7/970: creat d31/d32/d33/d5e/f14a x:0 0 0 2024-04-22T15:18:58.886 INFO:tasks.workunit.client.1.smithi196.stdout:7/424: truncate d14/d25/f42 4138983 0 2024-04-22T15:18:58.887 INFO:tasks.workunit.client.1.smithi196.stdout:7/425: write d14/d25/f3b [1986285,53883] 0 2024-04-22T15:18:58.888 INFO:tasks.workunit.client.1.smithi196.stdout:9/462: creat db/d11/d29/d57/fa2 x:0 0 0 2024-04-22T15:18:58.890 INFO:tasks.workunit.client.1.smithi196.stdout:9/463: chown db/d11/l82 28002 1 2024-04-22T15:18:58.891 INFO:tasks.workunit.client.0.smithi096.stdout:2/771: unlink d3/f49 0 2024-04-22T15:18:58.892 INFO:tasks.workunit.client.1.smithi196.stdout:3/446: dwrite d0/f1 [0,4194304] 0 2024-04-22T15:18:58.892 INFO:tasks.workunit.client.0.smithi096.stdout:2/772: dread - d3/db/df/dce/d46/d69/f6f zero size 2024-04-22T15:18:58.895 INFO:tasks.workunit.client.1.smithi196.stdout:4/444: creat d6/d14/f87 x:0 0 0 2024-04-22T15:18:58.895 INFO:tasks.workunit.client.0.smithi096.stdout:1/785: symlink d6/dc/d8a/d3f/d62/d7d/ddb/d9b/lf3 0 2024-04-22T15:18:58.896 INFO:tasks.workunit.client.1.smithi196.stdout:1/408: dread d1/db/dc/de/f4a [0,4194304] 0 2024-04-22T15:18:58.897 INFO:tasks.workunit.client.0.smithi096.stdout:6/607: write f0 [554615,51276] 0 2024-04-22T15:18:58.897 INFO:tasks.workunit.client.1.smithi196.stdout:6/463: creat dd/d20/d5c/d6a/f82 x:0 0 0 2024-04-22T15:18:58.899 INFO:tasks.workunit.client.0.smithi096.stdout:6/608: truncate f0 1470527 0 2024-04-22T15:18:58.902 INFO:tasks.workunit.client.1.smithi196.stdout:7/426: dwrite d14/d15/f74 [0,4194304] 0 2024-04-22T15:18:58.902 INFO:tasks.workunit.client.1.smithi196.stdout:8/455: sync 2024-04-22T15:18:58.902 INFO:tasks.workunit.client.1.smithi196.stdout:0/476: sync 2024-04-22T15:18:58.903 INFO:tasks.workunit.client.1.smithi196.stdout:1/409: read d1/db/dc/d26/f2b [283014,113967] 0 2024-04-22T15:18:58.904 INFO:tasks.workunit.client.0.smithi096.stdout:4/760: dwrite d5/d7/d2c/d3d/d4f/fb6 [0,4194304] 0 2024-04-22T15:18:58.905 INFO:tasks.workunit.client.1.smithi196.stdout:4/445: dread d6/d13/d15/f3d [0,4194304] 0 2024-04-22T15:18:58.906 INFO:tasks.workunit.client.1.smithi196.stdout:5/456: dread d1/d9/f38 [0,4194304] 0 2024-04-22T15:18:58.906 INFO:tasks.workunit.client.0.smithi096.stdout:5/738: unlink d4/ddb/d3d/d4d/fae 0 2024-04-22T15:18:58.908 INFO:tasks.workunit.client.1.smithi196.stdout:9/464: dwrite db/dd/de/f10 [0,4194304] 0 2024-04-22T15:18:58.909 INFO:tasks.workunit.client.1.smithi196.stdout:3/447: dread d0/d7/d8/f5c [0,4194304] 0 2024-04-22T15:18:58.909 INFO:tasks.workunit.client.0.smithi096.stdout:5/739: write d4/d2c/d2f/d30/fcf [438090,27172] 0 2024-04-22T15:18:58.909 INFO:tasks.workunit.client.0.smithi096.stdout:3/668: write da/db/de/f40 [1140554,126002] 0 2024-04-22T15:18:58.910 INFO:tasks.workunit.client.0.smithi096.stdout:0/872: link d7/de/d5b/d95/f24 d7/d42/d4c/d55/f12a 0 2024-04-22T15:18:58.911 INFO:tasks.workunit.client.0.smithi096.stdout:0/873: truncate d7/de/d5b/d64/fb6 454663 0 2024-04-22T15:18:58.911 INFO:tasks.workunit.client.0.smithi096.stdout:3/669: dread da/db/d10/f90 [0,4194304] 0 2024-04-22T15:18:58.915 INFO:tasks.workunit.client.0.smithi096.stdout:7/971: dread d31/d68/f69 [0,4194304] 0 2024-04-22T15:18:58.916 INFO:tasks.workunit.client.0.smithi096.stdout:2/773: creat d3/db/df/dce/d46/d7e/d84/ff6 x:0 0 0 2024-04-22T15:18:58.918 INFO:tasks.workunit.client.0.smithi096.stdout:1/786: creat d6/dc/d8a/d3f/d62/dc2/ff4 x:0 0 0 2024-04-22T15:18:58.918 INFO:tasks.workunit.client.0.smithi096.stdout:7/972: chown d31/d32/d36/d75/fc3 58 1 2024-04-22T15:18:58.918 INFO:tasks.workunit.client.1.smithi196.stdout:3/448: chown d0/d15/d31/c64 281574 1 2024-04-22T15:18:58.920 INFO:tasks.workunit.client.0.smithi096.stdout:6/609: unlink d3/d28/f1a 0 2024-04-22T15:18:58.920 INFO:tasks.workunit.client.1.smithi196.stdout:3/449: stat d0/d15/d4d/db/dc/d4a/d44/f16 0 2024-04-22T15:18:58.921 INFO:tasks.workunit.client.1.smithi196.stdout:8/456: mkdir d2/dc/d45/d83 0 2024-04-22T15:18:58.923 INFO:tasks.workunit.client.1.smithi196.stdout:1/410: dwrite d1/db/d16/f40 [0,4194304] 0 2024-04-22T15:18:58.923 INFO:tasks.workunit.client.0.smithi096.stdout:4/761: symlink d5/d7/d1a/d25/l106 0 2024-04-22T15:18:58.924 INFO:tasks.workunit.client.1.smithi196.stdout:5/457: read d1/d9/f11 [52004,53420] 0 2024-04-22T15:18:58.926 INFO:tasks.workunit.client.1.smithi196.stdout:9/465: rmdir db/d11/d29/d57 39 2024-04-22T15:18:58.930 INFO:tasks.workunit.client.0.smithi096.stdout:2/774: symlink d3/db/df/dce/d46/d7e/d8d/lf7 0 2024-04-22T15:18:58.934 INFO:tasks.workunit.client.1.smithi196.stdout:6/464: dread f1 [0,4194304] 0 2024-04-22T15:18:58.936 INFO:tasks.workunit.client.1.smithi196.stdout:6/465: write dd/f1c [2508801,100648] 0 2024-04-22T15:18:58.936 INFO:tasks.workunit.client.1.smithi196.stdout:1/411: read d1/db/dc/de/d43/f48 [127267,122183] 0 2024-04-22T15:18:58.937 INFO:tasks.workunit.client.0.smithi096.stdout:4/762: dread - d5/d7/d2c/f9d zero size 2024-04-22T15:18:58.937 INFO:tasks.workunit.client.0.smithi096.stdout:0/874: dwrite d7/de/d5b/d62/ff6 [4194304,4194304] 0 2024-04-22T15:18:58.937 INFO:tasks.workunit.client.0.smithi096.stdout:6/610: write d3/d28/d54/d68/d6e/dbb/fbc [6457131,20705] 0 2024-04-22T15:18:58.939 INFO:tasks.workunit.client.0.smithi096.stdout:5/740: unlink d4/d2c/feb 0 2024-04-22T15:18:58.939 INFO:tasks.workunit.client.1.smithi196.stdout:9/466: creat db/dd/d1b/d2c/fa3 x:0 0 0 2024-04-22T15:18:58.940 INFO:tasks.workunit.client.1.smithi196.stdout:9/467: readlink db/d11/l82 0 2024-04-22T15:18:58.941 INFO:tasks.workunit.client.1.smithi196.stdout:9/468: write db/dd/d1b/f79 [573693,122732] 0 2024-04-22T15:18:58.941 INFO:tasks.workunit.client.0.smithi096.stdout:2/775: dwrite d3/d1f/f51 [0,4194304] 0 2024-04-22T15:18:58.944 INFO:tasks.workunit.client.1.smithi196.stdout:7/427: fsync d14/d15/f74 0 2024-04-22T15:18:58.945 INFO:tasks.workunit.client.1.smithi196.stdout:3/450: creat d0/d7/d8/d70/f97 x:0 0 0 2024-04-22T15:18:58.946 INFO:tasks.workunit.client.0.smithi096.stdout:5/741: truncate d4/ddb/d3d/d4d/d59/d98/db9/ff0 732679 0 2024-04-22T15:18:58.946 INFO:tasks.workunit.client.1.smithi196.stdout:3/451: dread - d0/d15/f8f zero size 2024-04-22T15:18:58.947 INFO:tasks.workunit.client.1.smithi196.stdout:3/452: stat d0/d15/d4d/l40 0 2024-04-22T15:18:58.948 INFO:tasks.workunit.client.0.smithi096.stdout:4/763: write d5/d7/d1f/fac [850545,82696] 0 2024-04-22T15:18:58.949 INFO:tasks.workunit.client.1.smithi196.stdout:3/453: chown d0/d15/d4d/db/dc/d18/f4f 8981599 1 2024-04-22T15:18:58.950 INFO:tasks.workunit.client.1.smithi196.stdout:3/454: dread - d0/d15/d31/d55/f74 zero size 2024-04-22T15:18:58.952 INFO:tasks.workunit.client.1.smithi196.stdout:2/490: sync 2024-04-22T15:18:58.955 INFO:tasks.workunit.client.0.smithi096.stdout:1/787: link d6/dc/d8a/d3f/d54/c5a d6/dc/dbd/cf5 0 2024-04-22T15:18:58.955 INFO:tasks.workunit.client.1.smithi196.stdout:4/446: dread d6/d13/d15/d2c/d35/f4c [0,4194304] 0 2024-04-22T15:18:58.955 INFO:tasks.workunit.client.1.smithi196.stdout:2/491: chown d1/d1d/d6a/da1/c48 1496 1 2024-04-22T15:18:58.958 INFO:tasks.workunit.client.0.smithi096.stdout:1/788: write d6/dc/d8a/d3f/d54/d32/d3b/d56/fb9 [962564,27838] 0 2024-04-22T15:18:58.958 INFO:tasks.workunit.client.1.smithi196.stdout:9/469: stat db/d11/d29/l4a 0 2024-04-22T15:18:58.959 INFO:tasks.workunit.client.0.smithi096.stdout:7/973: dread d31/d32/dc0/fc4 [0,4194304] 0 2024-04-22T15:18:58.961 INFO:tasks.workunit.client.0.smithi096.stdout:1/789: readlink d6/dc/d8a/d3f/d54/d32/d3b/d94/lb0 0 2024-04-22T15:18:58.961 INFO:tasks.workunit.client.0.smithi096.stdout:7/974: write d31/d32/d36/d75/f86 [1943403,94636] 0 2024-04-22T15:18:58.963 INFO:tasks.workunit.client.1.smithi196.stdout:0/477: link d3/d9/d36/d56/d65/d7f/d87/d6c/d88/d95/l96 d3/d9/d36/d56/d65/d7f/d87/d6c/d88/l9b 0 2024-04-22T15:18:58.965 INFO:tasks.workunit.client.0.smithi096.stdout:6/611: link f1 d3/d28/d4d/d82/fd1 0 2024-04-22T15:18:58.966 INFO:tasks.workunit.client.1.smithi196.stdout:7/428: dread d14/f22 [0,4194304] 0 2024-04-22T15:18:58.970 INFO:tasks.workunit.client.0.smithi096.stdout:2/776: dread d3/db/df/d24/d58/d5e/f6b [0,4194304] 0 2024-04-22T15:18:58.972 INFO:tasks.workunit.client.0.smithi096.stdout:2/777: truncate d3/db/df/fa5 8611590 0 2024-04-22T15:18:58.972 INFO:tasks.workunit.client.0.smithi096.stdout:2/778: fsync d3/db/fc9 0 2024-04-22T15:18:58.976 INFO:tasks.workunit.client.1.smithi196.stdout:5/458: rename d1/d17/d1a/d2f/d30/d6e/d7a/l8a to d1/d2/l91 0 2024-04-22T15:18:58.976 INFO:tasks.workunit.client.1.smithi196.stdout:5/459: readlink d1/d53/l87 0 2024-04-22T15:18:58.977 INFO:tasks.workunit.client.1.smithi196.stdout:3/455: mknod d0/d15/d48/c98 0 2024-04-22T15:18:58.978 INFO:tasks.workunit.client.0.smithi096.stdout:1/790: mknod d6/dc/d8a/d3f/d62/dc2/cf6 0 2024-04-22T15:18:58.980 INFO:tasks.workunit.client.1.smithi196.stdout:8/457: write d2/d8/d47/f62 [139829,12177] 0 2024-04-22T15:18:58.981 INFO:tasks.workunit.client.1.smithi196.stdout:8/458: dread - d2/d5/f74 zero size 2024-04-22T15:18:58.982 INFO:tasks.workunit.client.0.smithi096.stdout:7/975: creat d31/d32/d33/d5e/dc9/f14b x:0 0 0 2024-04-22T15:18:58.984 INFO:tasks.workunit.client.1.smithi196.stdout:1/412: creat d1/db/dc/de/d43/d5e/d7a/f83 x:0 0 0 2024-04-22T15:18:58.987 INFO:tasks.workunit.client.1.smithi196.stdout:1/413: dread d1/db/ff [4194304,4194304] 0 2024-04-22T15:18:58.987 INFO:tasks.workunit.client.1.smithi196.stdout:9/470: symlink db/d11/d29/d7a/d7b/d81/la4 0 2024-04-22T15:18:58.989 INFO:tasks.workunit.client.0.smithi096.stdout:2/779: symlink d3/db/df/d10/lf8 0 2024-04-22T15:18:58.989 INFO:tasks.workunit.client.0.smithi096.stdout:2/780: chown d3/d1f/f56 410 1 2024-04-22T15:18:58.991 INFO:tasks.workunit.client.1.smithi196.stdout:7/429: unlink d14/d25/d31/f40 0 2024-04-22T15:18:58.999 INFO:tasks.workunit.client.0.smithi096.stdout:5/742: dread d4/f26 [0,4194304] 0 2024-04-22T15:18:59.000 INFO:tasks.workunit.client.1.smithi196.stdout:4/447: rename d6/d13/f2e to d6/d13/d15/d2c/d62/f88 0 2024-04-22T15:18:59.000 INFO:tasks.workunit.client.1.smithi196.stdout:6/466: write dd/f11 [221926,74662] 0 2024-04-22T15:18:59.001 INFO:tasks.workunit.client.0.smithi096.stdout:6/612: mknod d3/d28/daf/d94/cd2 0 2024-04-22T15:18:59.005 INFO:tasks.workunit.client.0.smithi096.stdout:4/764: getdents d5/d7 0 2024-04-22T15:18:59.008 INFO:tasks.workunit.client.0.smithi096.stdout:7/976: dread d31/d32/d33/f39 [0,4194304] 0 2024-04-22T15:18:59.009 INFO:tasks.workunit.client.1.smithi196.stdout:5/460: mkdir d1/d4/d92 0 2024-04-22T15:18:59.010 INFO:tasks.workunit.client.1.smithi196.stdout:2/492: write d1/d23/f6d [198577,100550] 0 2024-04-22T15:18:59.010 INFO:tasks.workunit.client.1.smithi196.stdout:5/461: write d1/d14/f1e [667251,125933] 0 2024-04-22T15:18:59.012 INFO:tasks.workunit.client.0.smithi096.stdout:6/613: creat d3/d3a/d75/fd3 x:0 0 0 2024-04-22T15:18:59.013 INFO:tasks.workunit.client.1.smithi196.stdout:1/414: symlink d1/db/dc/d26/l84 0 2024-04-22T15:18:59.019 INFO:tasks.workunit.client.0.smithi096.stdout:7/977: dwrite d31/d32/d36/d75/fc3 [4194304,4194304] 0 2024-04-22T15:18:59.031 INFO:tasks.workunit.client.1.smithi196.stdout:4/448: mkdir d6/d13/d15/d2c/d62/d78/d89 0 2024-04-22T15:18:59.032 INFO:tasks.workunit.client.1.smithi196.stdout:6/467: creat dd/d1f/d4e/d53/d7f/f83 x:0 0 0 2024-04-22T15:18:59.032 INFO:tasks.workunit.client.1.smithi196.stdout:4/449: write d6/d14/f33 [5053166,43220] 0 2024-04-22T15:18:59.035 INFO:tasks.workunit.client.1.smithi196.stdout:2/493: rmdir d1/d23 39 2024-04-22T15:18:59.037 INFO:tasks.workunit.client.0.smithi096.stdout:6/614: getdents d3/d3a/d75 0 2024-04-22T15:18:59.039 INFO:tasks.workunit.client.1.smithi196.stdout:1/415: chown d1/db/dc/de/d43/f6b 1036920 1 2024-04-22T15:18:59.039 INFO:tasks.workunit.client.0.smithi096.stdout:5/743: mknod d4/ddb/d25/cf7 0 2024-04-22T15:18:59.040 INFO:tasks.workunit.client.0.smithi096.stdout:7/978: write d31/d32/d33/d6a/f72 [700236,122665] 0 2024-04-22T15:18:59.041 INFO:tasks.workunit.client.0.smithi096.stdout:5/744: write d4/ddb/d3d/d35/fec [976660,15964] 0 2024-04-22T15:18:59.041 INFO:tasks.workunit.client.0.smithi096.stdout:6/615: mkdir d3/d28/dd/d11/d2c/d6f/dd4 0 2024-04-22T15:18:59.044 INFO:tasks.workunit.client.0.smithi096.stdout:6/616: write d3/d28/dd/d11/d2c/f91 [2126755,125857] 0 2024-04-22T15:18:59.046 INFO:tasks.workunit.client.0.smithi096.stdout:5/745: unlink d4/ddb/d3d/d4d/d59/d5a/ce9 0 2024-04-22T15:18:59.047 INFO:tasks.workunit.client.0.smithi096.stdout:7/979: write d31/d32/d36/d5d/f84 [1222225,38254] 0 2024-04-22T15:18:59.048 INFO:tasks.workunit.client.1.smithi196.stdout:6/468: mknod dd/d20/c84 0 2024-04-22T15:18:59.048 INFO:tasks.workunit.client.1.smithi196.stdout:6/469: chown dd/d1f/d37/l80 259690057 1 2024-04-22T15:18:59.049 INFO:tasks.workunit.client.0.smithi096.stdout:6/617: creat d3/d3a/d75/fd5 x:0 0 0 2024-04-22T15:18:59.050 INFO:tasks.workunit.client.1.smithi196.stdout:5/462: getdents d1/d4/d92 0 2024-04-22T15:18:59.053 INFO:tasks.workunit.client.1.smithi196.stdout:0/478: getdents d3/d6/dc 0 2024-04-22T15:18:59.054 INFO:tasks.workunit.client.0.smithi096.stdout:5/746: dwrite d4/ddb/d25/ff6 [0,4194304] 0 2024-04-22T15:18:59.060 INFO:tasks.workunit.client.1.smithi196.stdout:2/494: mkdir d1/d13/dbc 0 2024-04-22T15:18:59.061 INFO:tasks.workunit.client.0.smithi096.stdout:5/747: dwrite d4/ddb/f7a [4194304,4194304] 0 2024-04-22T15:18:59.062 INFO:tasks.workunit.client.1.smithi196.stdout:1/416: dwrite d1/db/dc/de/d43/f6b [0,4194304] 0 2024-04-22T15:18:59.063 INFO:tasks.workunit.client.1.smithi196.stdout:6/470: rename dd/d1f/d37/d3c/f67 to dd/d20/d5c/d5e/d76/f85 0 2024-04-22T15:18:59.064 INFO:tasks.workunit.client.1.smithi196.stdout:1/417: write d1/db/dc/de/d43/f68 [618693,37969] 0 2024-04-22T15:18:59.064 INFO:tasks.workunit.client.0.smithi096.stdout:7/980: getdents d31/d32/d33/d93/dd5 0 2024-04-22T15:18:59.065 INFO:tasks.workunit.client.1.smithi196.stdout:5/463: chown d1/d17/d1a/d2f/d30/l49 100 1 2024-04-22T15:18:59.065 INFO:tasks.workunit.client.0.smithi096.stdout:7/981: chown d31/d68/fea 1534782 1 2024-04-22T15:18:59.065 INFO:tasks.workunit.client.1.smithi196.stdout:9/471: getdents db/dd/de/d15 0 2024-04-22T15:18:59.068 INFO:tasks.workunit.client.0.smithi096.stdout:7/982: unlink d31/d32/d33/d6a/d83/lb7 0 2024-04-22T15:18:59.069 INFO:tasks.workunit.client.0.smithi096.stdout:7/983: readlink d31/ld4 0 2024-04-22T15:18:59.069 INFO:tasks.workunit.client.0.smithi096.stdout:6/618: write d3/d28/dd/d11/f41 [49464,102070] 0 2024-04-22T15:18:59.070 INFO:tasks.workunit.client.1.smithi196.stdout:3/456: sync 2024-04-22T15:18:59.070 INFO:tasks.workunit.client.1.smithi196.stdout:8/459: sync 2024-04-22T15:18:59.073 INFO:tasks.workunit.client.0.smithi096.stdout:7/984: creat d31/d32/d36/d75/dfe/f14c x:0 0 0 2024-04-22T15:18:59.078 INFO:tasks.workunit.client.0.smithi096.stdout:6/619: rename d3/d28/dd/d11/d2c/d44 to d3/d28/dd/d11/d2c/d6f/dd4/dd6 0 2024-04-22T15:18:59.079 INFO:tasks.workunit.client.0.smithi096.stdout:5/748: write d4/ddb/d3d/d4d/d59/d5a/d5c/f71 [1833876,20958] 0 2024-04-22T15:18:59.082 INFO:tasks.workunit.client.0.smithi096.stdout:7/985: rename d31/d32/d33/d5e/dc9/f14b to d31/d118/f14d 0 2024-04-22T15:18:59.083 INFO:tasks.workunit.client.1.smithi196.stdout:6/471: write dd/d20/d5c/d5e/d76/f85 [322863,61823] 0 2024-04-22T15:18:59.084 INFO:tasks.workunit.client.0.smithi096.stdout:6/620: mknod d3/d3a/d75/d83/cd7 0 2024-04-22T15:18:59.085 INFO:tasks.workunit.client.0.smithi096.stdout:6/621: dread - d3/d28/dd/fba zero size 2024-04-22T15:18:59.085 INFO:tasks.workunit.client.1.smithi196.stdout:1/418: fdatasync d1/db/dc/d26/f30 0 2024-04-22T15:18:59.086 INFO:tasks.workunit.client.1.smithi196.stdout:1/419: write d1/f4 [3984678,18665] 0 2024-04-22T15:18:59.087 INFO:tasks.workunit.client.1.smithi196.stdout:9/472: dread - db/d11/d29/d57/fa2 zero size 2024-04-22T15:18:59.088 INFO:tasks.workunit.client.1.smithi196.stdout:9/473: stat db/dd/d1b/f91 0 2024-04-22T15:18:59.091 INFO:tasks.workunit.client.1.smithi196.stdout:0/479: symlink d3/de/l9c 0 2024-04-22T15:18:59.092 INFO:tasks.workunit.client.0.smithi096.stdout:7/986: symlink d31/d32/d33/da2/l14e 0 2024-04-22T15:18:59.093 INFO:tasks.workunit.client.1.smithi196.stdout:7/430: getdents d14/d26/d5a/d5f/d6e 0 2024-04-22T15:18:59.093 INFO:tasks.workunit.client.1.smithi196.stdout:7/431: stat d14/d26/d5a/l82 0 2024-04-22T15:18:59.094 INFO:tasks.workunit.client.0.smithi096.stdout:6/622: truncate d3/d28/dd/d11/d2c/fb1 1242569 0 2024-04-22T15:18:59.095 INFO:tasks.workunit.client.0.smithi096.stdout:5/749: write d4/ddb/d25/f90 [268941,75387] 0 2024-04-22T15:18:59.095 INFO:tasks.workunit.client.1.smithi196.stdout:4/450: link c2 d6/d13/d15/d2c/d35/c8a 0 2024-04-22T15:18:59.098 INFO:tasks.workunit.client.1.smithi196.stdout:3/457: rmdir d0/d15/d4d/db/dc/d4a/d44 39 2024-04-22T15:18:59.099 INFO:tasks.workunit.client.1.smithi196.stdout:9/474: dread f5 [0,4194304] 0 2024-04-22T15:18:59.100 INFO:tasks.workunit.client.1.smithi196.stdout:2/495: mkdir d1/d23/d27/d2e/d55/d5b/d7c/dbd 0 2024-04-22T15:18:59.101 INFO:tasks.workunit.client.0.smithi096.stdout:5/750: mknod d4/ddb/d3d/cf8 0 2024-04-22T15:18:59.101 INFO:tasks.workunit.client.1.smithi196.stdout:9/475: write db/d11/d29/d57/f33 [215254,30358] 0 2024-04-22T15:18:59.102 INFO:tasks.workunit.client.1.smithi196.stdout:6/472: unlink f1 0 2024-04-22T15:18:59.106 INFO:tasks.workunit.client.1.smithi196.stdout:5/464: mknod d1/d4/d92/c93 0 2024-04-22T15:18:59.108 INFO:tasks.workunit.client.1.smithi196.stdout:5/465: write d1/d17/d1c/f3b [7833,52253] 0 2024-04-22T15:18:59.108 INFO:tasks.workunit.client.1.smithi196.stdout:8/460: dread d2/f1d [0,4194304] 0 2024-04-22T15:18:59.110 INFO:tasks.workunit.client.0.smithi096.stdout:6/623: dwrite d3/d28/f4f [0,4194304] 0 2024-04-22T15:18:59.110 INFO:tasks.workunit.client.1.smithi196.stdout:7/432: dread d14/d26/d2a/d2b/f2d [0,4194304] 0 2024-04-22T15:18:59.111 INFO:tasks.workunit.client.1.smithi196.stdout:7/433: stat d14/d25/d31 0 2024-04-22T15:18:59.112 INFO:tasks.workunit.client.1.smithi196.stdout:7/434: write d14/d26/d28/f58 [110308,90309] 0 2024-04-22T15:18:59.113 INFO:tasks.workunit.client.1.smithi196.stdout:7/435: stat f4 0 2024-04-22T15:18:59.113 INFO:tasks.workunit.client.0.smithi096.stdout:5/751: stat d4/d2c/d2f/cbb 0 2024-04-22T15:18:59.113 INFO:tasks.workunit.client.1.smithi196.stdout:7/436: write d14/d25/d31/f69 [2050549,59453] 0 2024-04-22T15:18:59.116 INFO:tasks.workunit.client.0.smithi096.stdout:7/987: link d31/d32/d33/d54/f62 d31/d32/d33/d93/dfc/f14f 0 2024-04-22T15:18:59.118 INFO:tasks.workunit.client.0.smithi096.stdout:6/624: mkdir d3/d28/d4d/d82/dd8 0 2024-04-22T15:18:59.120 INFO:tasks.workunit.client.1.smithi196.stdout:4/451: mkdir d6/d13/d15/d2c/d62/d78/d8b 0 2024-04-22T15:18:59.120 INFO:tasks.workunit.client.1.smithi196.stdout:4/452: stat d6/c9 0 2024-04-22T15:18:59.120 INFO:tasks.workunit.client.1.smithi196.stdout:4/453: readlink d6/d10/ld 0 2024-04-22T15:18:59.121 INFO:tasks.workunit.client.0.smithi096.stdout:5/752: write d4/fcc [587253,1913] 0 2024-04-22T15:18:59.121 INFO:tasks.workunit.client.1.smithi196.stdout:3/458: mknod d0/d15/d31/c99 0 2024-04-22T15:18:59.122 INFO:tasks.workunit.client.0.smithi096.stdout:7/988: fdatasync d31/d32/d33/d93/dd5/f143 0 2024-04-22T15:18:59.122 INFO:tasks.workunit.client.1.smithi196.stdout:3/459: dread - d0/d15/d4d/db/dc/d18/f84 zero size 2024-04-22T15:18:59.123 INFO:tasks.workunit.client.0.smithi096.stdout:7/989: write d31/d32/f59 [511755,1046] 0 2024-04-22T15:18:59.123 INFO:tasks.workunit.client.1.smithi196.stdout:2/496: symlink d1/d1d/d6a/lbe 0 2024-04-22T15:18:59.123 INFO:tasks.workunit.client.1.smithi196.stdout:3/460: stat d0/d7/d8/ld 0 2024-04-22T15:18:59.124 INFO:tasks.workunit.client.1.smithi196.stdout:3/461: chown d0/d15/d4d/c1f 1903 1 2024-04-22T15:18:59.124 INFO:tasks.workunit.client.1.smithi196.stdout:2/497: fsync d1/d1d/d6a/f9c 0 2024-04-22T15:18:59.124 INFO:tasks.workunit.client.0.smithi096.stdout:6/625: mknod d3/d28/dd/d11/d2c/d6f/dd4/dd6/d51/cd9 0 2024-04-22T15:18:59.125 INFO:tasks.workunit.client.1.smithi196.stdout:2/498: read d1/d1d/f68 [578641,4129] 0 2024-04-22T15:18:59.129 INFO:tasks.workunit.client.1.smithi196.stdout:9/476: chown db/d11/d29/d7a/d7b/ca0 47930 1 2024-04-22T15:18:59.130 INFO:tasks.workunit.client.1.smithi196.stdout:9/477: dread - db/dd/f8d zero size 2024-04-22T15:18:59.136 INFO:tasks.workunit.client.0.smithi096.stdout:7/990: rename d31/d32/d33/d5e to d31/d32/d33/d6a/d83/d107/d150 0 2024-04-22T15:18:59.137 INFO:tasks.workunit.client.0.smithi096.stdout:7/991: truncate d31/d32/d33/d93/dd5/f140 56289 0 2024-04-22T15:18:59.141 INFO:tasks.workunit.client.1.smithi196.stdout:7/437: fdatasync f2 0 2024-04-22T15:18:59.144 INFO:tasks.workunit.client.0.smithi096.stdout:5/753: dwrite d4/d2c/d2f/d31/f3b [0,4194304] 0 2024-04-22T15:18:59.147 INFO:tasks.workunit.client.1.smithi196.stdout:7/438: dwrite d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:18:59.186 INFO:tasks.workunit.client.1.smithi196.stdout:8/461: sync 2024-04-22T15:18:59.197 INFO:tasks.workunit.client.1.smithi196.stdout:8/462: dread d2/dc/f10 [0,4194304] 0 2024-04-22T15:18:59.198 INFO:tasks.workunit.client.1.smithi196.stdout:8/463: truncate d2/d5/d4f/f6b 16099 0 2024-04-22T15:18:59.235 INFO:tasks.workunit.client.1.smithi196.stdout:4/454: truncate d6/d13/f16 126557 0 2024-04-22T15:18:59.237 INFO:tasks.workunit.client.1.smithi196.stdout:3/462: rmdir d0 39 2024-04-22T15:18:59.240 INFO:tasks.workunit.client.1.smithi196.stdout:2/499: rmdir d1/d1d/d6a/da1 39 2024-04-22T15:18:59.241 INFO:tasks.workunit.client.1.smithi196.stdout:2/500: chown d1/d76/f7d 24 1 2024-04-22T15:18:59.245 INFO:tasks.workunit.client.1.smithi196.stdout:4/455: dread d6/d14/f12 [0,4194304] 0 2024-04-22T15:18:59.262 INFO:tasks.workunit.client.1.smithi196.stdout:9/478: chown db/d11/d29/f92 1637 1 2024-04-22T15:18:59.270 INFO:tasks.workunit.client.1.smithi196.stdout:7/439: rename d14/d26/d28/f58 to d14/d26/d28/f89 0 2024-04-22T15:18:59.271 INFO:tasks.workunit.client.1.smithi196.stdout:7/440: fdatasync d14/d26/d2a/f84 0 2024-04-22T15:18:59.272 INFO:tasks.workunit.client.1.smithi196.stdout:8/464: rmdir d2/d8/d47/d68 39 2024-04-22T15:18:59.284 INFO:tasks.workunit.client.1.smithi196.stdout:5/466: truncate d1/d17/d1a/d2f/d30/d5e/f63 789185 0 2024-04-22T15:18:59.285 INFO:tasks.workunit.client.1.smithi196.stdout:9/479: creat db/d11/d29/d7a/fa5 x:0 0 0 2024-04-22T15:18:59.290 INFO:tasks.workunit.client.1.smithi196.stdout:6/473: creat dd/f86 x:0 0 0 2024-04-22T15:18:59.291 INFO:tasks.workunit.client.1.smithi196.stdout:6/474: truncate dd/d1f/d4e/d53/d7f/f81 61080 0 2024-04-22T15:18:59.291 INFO:tasks.workunit.client.1.smithi196.stdout:6/475: chown f9 0 1 2024-04-22T15:18:59.292 INFO:tasks.workunit.client.1.smithi196.stdout:1/420: getdents d1/db/d16/d1e/d65 0 2024-04-22T15:18:59.299 INFO:tasks.workunit.client.1.smithi196.stdout:8/465: chown d2/dc/c55 1317 1 2024-04-22T15:18:59.300 INFO:tasks.workunit.client.1.smithi196.stdout:0/480: link d3/d9/d25/l7a d3/d9/d36/d56/d65/d7f/l9d 0 2024-04-22T15:18:59.301 INFO:tasks.workunit.client.1.smithi196.stdout:3/463: symlink d0/d15/d31/l9a 0 2024-04-22T15:18:59.304 INFO:tasks.workunit.client.1.smithi196.stdout:9/480: fdatasync db/d11/d29/d57/f1f 0 2024-04-22T15:18:59.305 INFO:tasks.workunit.client.1.smithi196.stdout:6/476: creat dd/d1f/d4e/d53/d55/f87 x:0 0 0 2024-04-22T15:18:59.306 INFO:tasks.workunit.client.1.smithi196.stdout:6/477: write dd/d1f/d37/f66 [455997,38835] 0 2024-04-22T15:18:59.306 INFO:tasks.workunit.client.1.smithi196.stdout:1/421: write d1/db/dc/d26/d2e/d32/f53 [2811172,19323] 0 2024-04-22T15:18:59.308 INFO:tasks.workunit.client.1.smithi196.stdout:9/481: dread db/d11/f6a [0,4194304] 0 2024-04-22T15:18:59.310 INFO:tasks.workunit.client.1.smithi196.stdout:7/441: rename d14/c52 to d14/d26/d2a/c8a 0 2024-04-22T15:18:59.310 INFO:tasks.workunit.client.1.smithi196.stdout:7/442: write fe [301951,55954] 0 2024-04-22T15:18:59.311 INFO:tasks.workunit.client.0.smithi096.stdout:5/754: creat d4/ddb/d3d/d4d/d59/d98/dee/ff9 x:0 0 0 2024-04-22T15:18:59.312 INFO:tasks.workunit.client.1.smithi196.stdout:0/481: creat d3/d9/d36/d56/d65/d7f/d87/d6c/d88/d95/f9e x:0 0 0 2024-04-22T15:18:59.313 INFO:tasks.workunit.client.1.smithi196.stdout:3/464: rmdir d0 39 2024-04-22T15:18:59.314 INFO:tasks.workunit.client.0.smithi096.stdout:5/755: readlink d4/ddb/l1b 0 2024-04-22T15:18:59.318 INFO:tasks.workunit.client.0.smithi096.stdout:5/756: truncate d4/f86 1306199 0 2024-04-22T15:18:59.319 INFO:tasks.workunit.client.1.smithi196.stdout:4/456: rmdir d6/d13/d15/d2c/d62/d78/d89 0 2024-04-22T15:18:59.322 INFO:tasks.workunit.client.1.smithi196.stdout:1/422: fsync d1/db/ff 0 2024-04-22T15:18:59.323 INFO:tasks.workunit.client.1.smithi196.stdout:6/478: mknod dd/d1f/d4e/d53/d7f/c88 0 2024-04-22T15:18:59.324 INFO:tasks.workunit.client.1.smithi196.stdout:7/443: fsync d14/d26/d28/f89 0 2024-04-22T15:18:59.329 INFO:tasks.workunit.client.1.smithi196.stdout:4/457: unlink d6/d13/d15/f30 0 2024-04-22T15:18:59.329 INFO:tasks.workunit.client.1.smithi196.stdout:4/458: write d6/d14/f33 [2970220,27458] 0 2024-04-22T15:18:59.330 INFO:tasks.workunit.client.1.smithi196.stdout:1/423: write d1/db/d16/d1e/f60 [3850520,116219] 0 2024-04-22T15:18:59.332 INFO:tasks.workunit.client.1.smithi196.stdout:3/465: dwrite d0/d7/d8/f19 [0,4194304] 0 2024-04-22T15:18:59.333 INFO:tasks.workunit.client.1.smithi196.stdout:1/424: write d1/db/dc/de/d43/f7b [640877,58252] 0 2024-04-22T15:18:59.335 INFO:tasks.workunit.client.1.smithi196.stdout:2/501: truncate d1/d1d/d6a/da1/f79 575415 0 2024-04-22T15:18:59.336 INFO:tasks.workunit.client.1.smithi196.stdout:1/425: write d1/db/dc/de/d43/f7b [1460944,40467] 0 2024-04-22T15:18:59.339 INFO:tasks.workunit.client.1.smithi196.stdout:4/459: getdents d6/d10 0 2024-04-22T15:18:59.339 INFO:tasks.workunit.client.1.smithi196.stdout:1/426: chown d1/db/dc/de/d5a/d5b/d69 625687792 1 2024-04-22T15:18:59.343 INFO:tasks.workunit.client.1.smithi196.stdout:4/460: chown d6/d13/d15/d2c/d35/d44/f4a 23754035 1 2024-04-22T15:18:59.344 INFO:tasks.workunit.client.1.smithi196.stdout:3/466: dwrite d0/d15/d4d/f5a [0,4194304] 0 2024-04-22T15:18:59.345 INFO:tasks.workunit.client.1.smithi196.stdout:4/461: stat d6/d10/c4b 0 2024-04-22T15:18:59.346 INFO:tasks.workunit.client.1.smithi196.stdout:2/502: dwrite d1/d13/d25/d2a/f77 [0,4194304] 0 2024-04-22T15:18:59.348 INFO:tasks.workunit.client.1.smithi196.stdout:4/462: mkdir d6/d13/d15/d2c/d8c 0 2024-04-22T15:18:59.349 INFO:tasks.workunit.client.1.smithi196.stdout:3/467: symlink d0/d15/d4d/db/dc/d4a/d44/d8c/l9b 0 2024-04-22T15:18:59.349 INFO:tasks.workunit.client.1.smithi196.stdout:3/468: stat d0/d15/d4d/db/dc/d85 0 2024-04-22T15:18:59.350 INFO:tasks.workunit.client.1.smithi196.stdout:4/463: mkdir d6/d13/d15/d2c/d62/d78/d8d 0 2024-04-22T15:18:59.351 INFO:tasks.workunit.client.1.smithi196.stdout:2/503: getdents d1/d76 0 2024-04-22T15:18:59.351 INFO:tasks.workunit.client.1.smithi196.stdout:3/469: write d0/d15/d4d/db/dc/f47 [3130810,115664] 0 2024-04-22T15:18:59.353 INFO:tasks.workunit.client.1.smithi196.stdout:3/470: fsync d0/d15/d31/d55/f74 0 2024-04-22T15:18:59.354 INFO:tasks.workunit.client.0.smithi096.stdout:9/655: sync 2024-04-22T15:18:59.364 INFO:tasks.workunit.client.0.smithi096.stdout:9/656: chown d1/df/d95/d9d/db5/dd/f21 14 1 2024-04-22T15:18:59.366 INFO:tasks.workunit.client.1.smithi196.stdout:4/464: truncate d6/d13/d15/d2c/d35/d44/f60 4977673 0 2024-04-22T15:18:59.367 INFO:tasks.workunit.client.0.smithi096.stdout:9/657: write d1/df/d95/d9d/db5/dd/dba/dc2/fc6 [121099,55535] 0 2024-04-22T15:18:59.367 INFO:tasks.workunit.client.0.smithi096.stdout:5/757: write d4/ddb/d3d/d35/fba [112603,121077] 0 2024-04-22T15:18:59.368 INFO:tasks.workunit.client.1.smithi196.stdout:5/467: sync 2024-04-22T15:18:59.375 INFO:tasks.workunit.client.1.smithi196.stdout:5/468: mkdir d1/d14/d94 0 2024-04-22T15:18:59.375 INFO:tasks.workunit.client.0.smithi096.stdout:5/758: rmdir d4/d2c/d2f/d31 39 2024-04-22T15:18:59.378 INFO:tasks.workunit.client.1.smithi196.stdout:4/465: creat d6/d13/d15/d2c/d62/d78/d8b/f8e x:0 0 0 2024-04-22T15:18:59.378 INFO:tasks.workunit.client.1.smithi196.stdout:0/482: write d3/d9/d36/d56/d65/d7f/d30/f78 [1612023,25133] 0 2024-04-22T15:18:59.379 INFO:tasks.workunit.client.0.smithi096.stdout:9/658: unlink d1/df/l53 0 2024-04-22T15:18:59.380 INFO:tasks.workunit.client.0.smithi096.stdout:5/759: symlink d4/ddb/d3d/d4d/d59/d98/db9/lfa 0 2024-04-22T15:18:59.382 INFO:tasks.workunit.client.0.smithi096.stdout:9/659: dread - d1/d4/d7f/fa9 zero size 2024-04-22T15:18:59.383 INFO:tasks.workunit.client.1.smithi196.stdout:8/466: dwrite d2/d3f/d4b/f4d [0,4194304] 0 2024-04-22T15:18:59.385 INFO:tasks.workunit.client.0.smithi096.stdout:5/760: symlink d4/ddb/d3d/d4d/d99/lfb 0 2024-04-22T15:18:59.385 INFO:tasks.workunit.client.1.smithi196.stdout:0/483: mkdir d3/d9/d36/d9f 0 2024-04-22T15:18:59.388 INFO:tasks.workunit.client.1.smithi196.stdout:2/504: fsync d1/d1d/d6a/da1/f79 0 2024-04-22T15:18:59.389 INFO:tasks.workunit.client.1.smithi196.stdout:7/444: sync 2024-04-22T15:18:59.391 INFO:tasks.workunit.client.1.smithi196.stdout:9/482: dread db/dd/de/d15/f1e [4194304,4194304] 0 2024-04-22T15:18:59.395 INFO:tasks.workunit.client.1.smithi196.stdout:6/479: dwrite dd/d20/f29 [0,4194304] 0 2024-04-22T15:18:59.395 INFO:tasks.workunit.client.0.smithi096.stdout:9/660: creat d1/df/d95/d9d/db5/d43/fe0 x:0 0 0 2024-04-22T15:18:59.397 INFO:tasks.workunit.client.1.smithi196.stdout:3/471: write d0/d7/f5b [897557,1288] 0 2024-04-22T15:18:59.398 INFO:tasks.workunit.client.1.smithi196.stdout:6/480: dread - dd/d20/f7d zero size 2024-04-22T15:18:59.398 INFO:tasks.workunit.client.1.smithi196.stdout:1/427: dread d1/db/ff [0,4194304] 0 2024-04-22T15:18:59.400 INFO:tasks.workunit.client.1.smithi196.stdout:8/467: dwrite d2/d5/d2e/f67 [0,4194304] 0 2024-04-22T15:18:59.401 INFO:tasks.workunit.client.1.smithi196.stdout:0/484: creat d3/d9/d36/fa0 x:0 0 0 2024-04-22T15:18:59.402 INFO:tasks.workunit.client.1.smithi196.stdout:0/485: chown d3/de/d1c 24325 1 2024-04-22T15:18:59.402 INFO:tasks.workunit.client.1.smithi196.stdout:8/468: truncate d2/d35/f61 1524363 0 2024-04-22T15:18:59.403 INFO:tasks.workunit.client.1.smithi196.stdout:0/486: chown d3/de/c3a 62 1 2024-04-22T15:18:59.403 INFO:tasks.workunit.client.1.smithi196.stdout:8/469: fdatasync d2/d5/f23 0 2024-04-22T15:18:59.404 INFO:tasks.workunit.client.1.smithi196.stdout:8/470: dread d2/d5/d4f/f6b [0,4194304] 0 2024-04-22T15:18:59.408 INFO:tasks.workunit.client.1.smithi196.stdout:5/469: dread d1/d4/f22 [0,4194304] 0 2024-04-22T15:18:59.410 INFO:tasks.workunit.client.1.smithi196.stdout:5/470: write d1/d17/d50/f6d [2270345,120818] 0 2024-04-22T15:18:59.414 INFO:tasks.workunit.client.1.smithi196.stdout:4/466: creat d6/d13/d15/d2c/d35/f8f x:0 0 0 2024-04-22T15:18:59.415 INFO:tasks.workunit.client.0.smithi096.stdout:5/761: link d4/d2c/c36 d4/d2c/d2f/d30/cfc 0 2024-04-22T15:18:59.418 INFO:tasks.workunit.client.1.smithi196.stdout:7/445: rmdir d14/d46 39 2024-04-22T15:18:59.418 INFO:tasks.workunit.client.1.smithi196.stdout:5/471: dwrite d1/d17/d1a/d2f/d30/f54 [0,4194304] 0 2024-04-22T15:18:59.418 INFO:tasks.workunit.client.1.smithi196.stdout:2/505: dread d1/fd [4194304,4194304] 0 2024-04-22T15:18:59.418 INFO:tasks.workunit.client.1.smithi196.stdout:1/428: rmdir d1/db/d16/d1e/d49 39 2024-04-22T15:18:59.418 INFO:tasks.workunit.client.1.smithi196.stdout:2/506: fsync d1/f3 0 2024-04-22T15:18:59.420 INFO:tasks.workunit.client.1.smithi196.stdout:4/467: truncate d6/d13/d15/d2c/d62/d78/d8b/f8e 158707 0 2024-04-22T15:18:59.420 INFO:tasks.workunit.client.1.smithi196.stdout:6/481: mkdir dd/d1f/d37/d3c/d89 0 2024-04-22T15:18:59.420 INFO:tasks.workunit.client.1.smithi196.stdout:4/468: fdatasync d6/f2b 0 2024-04-22T15:18:59.422 INFO:tasks.workunit.client.1.smithi196.stdout:3/472: creat d0/d15/d4d/db/dc/d85/f9c x:0 0 0 2024-04-22T15:18:59.423 INFO:tasks.workunit.client.1.smithi196.stdout:3/473: chown d0/d15/d4d/db/f2e 0 1 2024-04-22T15:18:59.425 INFO:tasks.workunit.client.0.smithi096.stdout:5/762: creat d4/d2c/d2f/d31/d82/def/ffd x:0 0 0 2024-04-22T15:18:59.428 INFO:tasks.workunit.client.0.smithi096.stdout:5/763: creat d4/d2c/d2f/d6b/ffe x:0 0 0 2024-04-22T15:18:59.432 INFO:tasks.workunit.client.1.smithi196.stdout:7/446: rmdir d14/d25/d31 39 2024-04-22T15:18:59.435 INFO:tasks.workunit.client.1.smithi196.stdout:5/472: symlink d1/d17/d1a/d2f/d30/l95 0 2024-04-22T15:18:59.436 INFO:tasks.workunit.client.0.smithi096.stdout:9/661: dwrite d1/df/d95/d9d/db5/f24 [0,4194304] 0 2024-04-22T15:18:59.436 INFO:tasks.workunit.client.1.smithi196.stdout:1/429: mkdir d1/db/dc/de/d5a/d5b/d85 0 2024-04-22T15:18:59.440 INFO:tasks.workunit.client.1.smithi196.stdout:1/430: dread d1/db/d16/d44/d6a/f50 [0,4194304] 0 2024-04-22T15:18:59.441 INFO:tasks.workunit.client.1.smithi196.stdout:2/507: creat d1/d23/d27/d2e/d55/d5b/d93/fbf x:0 0 0 2024-04-22T15:18:59.442 INFO:tasks.workunit.client.1.smithi196.stdout:1/431: write d1/db/dc/de/d43/f6b [358808,43997] 0 2024-04-22T15:18:59.442 INFO:tasks.workunit.client.1.smithi196.stdout:2/508: chown d1/d23/d27/d2e 142659 1 2024-04-22T15:18:59.443 INFO:tasks.workunit.client.1.smithi196.stdout:2/509: stat d1/d76/fb9 0 2024-04-22T15:18:59.443 INFO:tasks.workunit.client.1.smithi196.stdout:1/432: fdatasync d1/db/dc/de/d5a/d5b/f79 0 2024-04-22T15:18:59.446 INFO:tasks.workunit.client.1.smithi196.stdout:3/474: mkdir d0/d15/d31/d9d 0 2024-04-22T15:18:59.447 INFO:tasks.workunit.client.1.smithi196.stdout:0/487: mknod d3/d6/ca1 0 2024-04-22T15:18:59.450 INFO:tasks.workunit.client.0.smithi096.stdout:5/764: dread f2 [0,4194304] 0 2024-04-22T15:18:59.452 INFO:tasks.workunit.client.1.smithi196.stdout:5/473: symlink d1/d17/d1a/d2f/d30/l96 0 2024-04-22T15:18:59.455 INFO:tasks.workunit.client.0.smithi096.stdout:5/765: creat d4/ddb/d25/fff x:0 0 0 2024-04-22T15:18:59.455 INFO:tasks.workunit.client.0.smithi096.stdout:5/766: rename d4/d2c to d4/d2c/d2f/d30/d100 22 2024-04-22T15:18:59.457 INFO:tasks.workunit.client.1.smithi196.stdout:6/482: creat dd/d1f/d37/d7b/f8a x:0 0 0 2024-04-22T15:18:59.458 INFO:tasks.workunit.client.0.smithi096.stdout:9/662: write d1/f8a [1318771,36098] 0 2024-04-22T15:18:59.462 INFO:tasks.workunit.client.1.smithi196.stdout:7/447: write d14/d26/f83 [815067,29694] 0 2024-04-22T15:18:59.463 INFO:tasks.workunit.client.0.smithi096.stdout:9/663: rmdir d1/d25/db2/d98/d9b 39 2024-04-22T15:18:59.464 INFO:tasks.workunit.client.1.smithi196.stdout:7/448: write fe [3168666,112097] 0 2024-04-22T15:18:59.464 INFO:tasks.workunit.client.1.smithi196.stdout:6/483: dwrite dd/f11 [0,4194304] 0 2024-04-22T15:18:59.464 INFO:tasks.workunit.client.0.smithi096.stdout:9/664: chown d1/df/d95/d9d/db5/f24 3500783 1 2024-04-22T15:18:59.465 INFO:tasks.workunit.client.1.smithi196.stdout:2/510: dread d1/d23/d27/d2e/f58 [0,4194304] 0 2024-04-22T15:18:59.465 INFO:tasks.workunit.client.0.smithi096.stdout:5/767: creat d4/ddb/d3d/d4d/d59/d5a/f101 x:0 0 0 2024-04-22T15:18:59.468 INFO:tasks.workunit.client.1.smithi196.stdout:9/483: getdents db/d11/d29/d7a 0 2024-04-22T15:18:59.469 INFO:tasks.workunit.client.0.smithi096.stdout:5/768: rmdir d4/ddb/d3d/d35/d57 39 2024-04-22T15:18:59.470 INFO:tasks.workunit.client.1.smithi196.stdout:1/433: write d1/db/dc/d26/d2e/f3e [501602,37439] 0 2024-04-22T15:18:59.476 INFO:tasks.workunit.client.0.smithi096.stdout:5/769: dread d4/d2c/d2f/d30/fcf [0,4194304] 0 2024-04-22T15:18:59.476 INFO:tasks.workunit.client.0.smithi096.stdout:9/665: dwrite d1/d25/d9f/fcd [0,4194304] 0 2024-04-22T15:18:59.476 INFO:tasks.workunit.client.1.smithi196.stdout:3/475: mkdir d0/d15/d31/d9d/d9e 0 2024-04-22T15:18:59.479 INFO:tasks.workunit.client.1.smithi196.stdout:7/449: creat d14/d26/d28/d29/f8b x:0 0 0 2024-04-22T15:18:59.480 INFO:tasks.workunit.client.1.smithi196.stdout:9/484: rename c6 to db/d11/d29/d7a/ca6 0 2024-04-22T15:18:59.484 INFO:tasks.workunit.client.0.smithi096.stdout:9/666: creat d1/df/d95/d9d/db5/dd/fe1 x:0 0 0 2024-04-22T15:18:59.485 INFO:tasks.workunit.client.1.smithi196.stdout:1/434: fsync d1/db/dc/d26/f27 0 2024-04-22T15:18:59.486 INFO:tasks.workunit.client.1.smithi196.stdout:2/511: dread d1/d1d/d6a/f81 [0,4194304] 0 2024-04-22T15:18:59.486 INFO:tasks.workunit.client.1.smithi196.stdout:1/435: write d1/db/f58 [3403174,38114] 0 2024-04-22T15:18:59.486 INFO:tasks.workunit.client.1.smithi196.stdout:2/512: stat d1/d13/d25/d2a/c6b 0 2024-04-22T15:18:59.487 INFO:tasks.workunit.client.0.smithi096.stdout:5/770: rename d4/ddb/d3d/d4d/d59/d5a/d5c/f83 to d4/ddb/d3d/d4d/d59/d5a/d5c/f102 0 2024-04-22T15:18:59.489 INFO:tasks.workunit.client.1.smithi196.stdout:2/513: read d1/d23/d27/d2e/d55/d5b/d7c/f91 [2410996,34901] 0 2024-04-22T15:18:59.489 INFO:tasks.workunit.client.1.smithi196.stdout:3/476: rename d0/d15/d4d/c80 to d0/d15/d4d/c9f 0 2024-04-22T15:18:59.490 INFO:tasks.workunit.client.1.smithi196.stdout:2/514: chown d1/d23/cad 58 1 2024-04-22T15:18:59.506 INFO:tasks.workunit.client.0.smithi096.stdout:5/771: write d4/ddb/d3d/d4d/d59/d5a/d5c/f102 [191094,45823] 0 2024-04-22T15:18:59.507 INFO:tasks.workunit.client.0.smithi096.stdout:5/772: stat d4/ddb/d3d/d4d/d59/db0/fd9 0 2024-04-22T15:18:59.508 INFO:tasks.workunit.client.0.smithi096.stdout:5/773: chown d4/d2c/d2f/d31/d82/def 127958 1 2024-04-22T15:18:59.511 INFO:tasks.workunit.client.0.smithi096.stdout:5/774: link d4/d2c/d2f/f81 d4/d2c/d2f/d6b/f103 0 2024-04-22T15:18:59.512 INFO:tasks.workunit.client.0.smithi096.stdout:5/775: readlink d4/d2c/d2f/l52 0 2024-04-22T15:18:59.513 INFO:tasks.workunit.client.0.smithi096.stdout:5/776: symlink d4/d2c/d2f/d6b/l104 0 2024-04-22T15:18:59.515 INFO:tasks.workunit.client.0.smithi096.stdout:5/777: read d4/d1f/f60 [1762217,19795] 0 2024-04-22T15:18:59.518 INFO:tasks.workunit.client.0.smithi096.stdout:5/778: fsync d4/ddb/d3d/d35/d57/fb4 0 2024-04-22T15:18:59.520 INFO:tasks.workunit.client.0.smithi096.stdout:5/779: mkdir d4/ddb/d3d/d4d/d59/d98/dee/d105 0 2024-04-22T15:18:59.521 INFO:tasks.workunit.client.0.smithi096.stdout:5/780: dread - d4/ddb/d25/fff zero size 2024-04-22T15:18:59.524 INFO:tasks.workunit.client.0.smithi096.stdout:5/781: rename d4/d2c/d2f/d31/d82/da0 to d4/d1f/d8b/d106 0 2024-04-22T15:18:59.529 INFO:tasks.workunit.client.0.smithi096.stdout:5/782: write d4/ddb/d3d/fc6 [323467,88336] 0 2024-04-22T15:18:59.529 INFO:tasks.workunit.client.1.smithi196.stdout:8/471: sync 2024-04-22T15:18:59.530 INFO:tasks.workunit.client.0.smithi096.stdout:5/783: chown d4/ddb/d3d/d35/d57/c91 3362021 1 2024-04-22T15:18:59.532 INFO:tasks.workunit.client.0.smithi096.stdout:5/784: symlink d4/d1f/d8b/d106/l107 0 2024-04-22T15:18:59.532 INFO:tasks.workunit.client.1.smithi196.stdout:7/450: creat d14/d4a/f8c x:0 0 0 2024-04-22T15:18:59.532 INFO:tasks.workunit.client.1.smithi196.stdout:5/474: link d1/d17/d1a/l33 d1/d51/l97 0 2024-04-22T15:18:59.534 INFO:tasks.workunit.client.0.smithi096.stdout:5/785: creat d4/d2c/d2f/d30/f108 x:0 0 0 2024-04-22T15:18:59.534 INFO:tasks.workunit.client.1.smithi196.stdout:5/475: chown d1/d14 1 1 2024-04-22T15:18:59.534 INFO:tasks.workunit.client.1.smithi196.stdout:8/472: dread - d2/d8/d49/f7a zero size 2024-04-22T15:18:59.534 INFO:tasks.workunit.client.1.smithi196.stdout:0/488: getdents d3/de 0 2024-04-22T15:18:59.535 INFO:tasks.workunit.client.1.smithi196.stdout:1/436: write d1/db/dc/d26/f27 [1440959,20603] 0 2024-04-22T15:18:59.535 INFO:tasks.workunit.client.1.smithi196.stdout:2/515: dread - d1/d23/d27/d89/fb0 zero size 2024-04-22T15:18:59.536 INFO:tasks.workunit.client.1.smithi196.stdout:3/477: creat d0/d15/d31/d55/d5d/fa0 x:0 0 0 2024-04-22T15:18:59.539 INFO:tasks.workunit.client.1.smithi196.stdout:0/489: mkdir d3/d6/da2 0 2024-04-22T15:18:59.540 INFO:tasks.workunit.client.1.smithi196.stdout:3/478: dread - d0/d15/f8f zero size 2024-04-22T15:18:59.540 INFO:tasks.workunit.client.1.smithi196.stdout:2/516: symlink d1/d13/d25/lc0 0 2024-04-22T15:18:59.544 INFO:tasks.workunit.client.1.smithi196.stdout:1/437: getdents d1/db/dc/de/d5a/d5b/d85 0 2024-04-22T15:18:59.546 INFO:tasks.workunit.client.1.smithi196.stdout:3/479: readlink d0/d15/d4d/l3f 0 2024-04-22T15:18:59.547 INFO:tasks.workunit.client.1.smithi196.stdout:7/451: dwrite f7 [0,4194304] 0 2024-04-22T15:18:59.551 INFO:tasks.workunit.client.1.smithi196.stdout:5/476: dwrite d1/d51/f84 [0,4194304] 0 2024-04-22T15:18:59.551 INFO:tasks.workunit.client.1.smithi196.stdout:8/473: dwrite d2/d35/f63 [0,4194304] 0 2024-04-22T15:18:59.553 INFO:tasks.workunit.client.1.smithi196.stdout:7/452: write d14/d15/f74 [434976,114364] 0 2024-04-22T15:18:59.553 INFO:tasks.workunit.client.1.smithi196.stdout:9/485: dread db/dd/d1b/d2c/f43 [0,4194304] 0 2024-04-22T15:18:59.556 INFO:tasks.workunit.client.1.smithi196.stdout:3/480: creat d0/d15/d31/d55/d6e/fa1 x:0 0 0 2024-04-22T15:18:59.559 INFO:tasks.workunit.client.1.smithi196.stdout:9/486: stat db/dd/d1b/d2c/c41 0 2024-04-22T15:18:59.561 INFO:tasks.workunit.client.1.smithi196.stdout:1/438: getdents d1/db/dc/de/d5a/d5b/d69 0 2024-04-22T15:18:59.561 INFO:tasks.workunit.client.1.smithi196.stdout:9/487: write db/f5f [132048,87130] 0 2024-04-22T15:18:59.565 INFO:tasks.workunit.client.1.smithi196.stdout:9/488: dread - db/dd/f9e zero size 2024-04-22T15:18:59.565 INFO:tasks.workunit.client.1.smithi196.stdout:8/474: dread d2/f1d [0,4194304] 0 2024-04-22T15:18:59.565 INFO:tasks.workunit.client.1.smithi196.stdout:3/481: mkdir d0/d15/d31/d9d/da2 0 2024-04-22T15:18:59.567 INFO:tasks.workunit.client.1.smithi196.stdout:0/490: dread d3/fd [4194304,4194304] 0 2024-04-22T15:18:59.567 INFO:tasks.workunit.client.1.smithi196.stdout:5/477: rename d1/d14/d94 to d1/d2/d98 0 2024-04-22T15:18:59.568 INFO:tasks.workunit.client.1.smithi196.stdout:1/439: creat d1/f86 x:0 0 0 2024-04-22T15:18:59.570 INFO:tasks.workunit.client.1.smithi196.stdout:1/440: write d1/db/d16/f7e [2591661,68339] 0 2024-04-22T15:18:59.570 INFO:tasks.workunit.client.1.smithi196.stdout:9/489: mkdir db/d11/d29/d57/da7 0 2024-04-22T15:18:59.572 INFO:tasks.workunit.client.1.smithi196.stdout:3/482: unlink d0/d15/d4d/db/dc/l22 0 2024-04-22T15:18:59.572 INFO:tasks.workunit.client.1.smithi196.stdout:3/483: fsync d0/d15/f92 0 2024-04-22T15:18:59.574 INFO:tasks.workunit.client.1.smithi196.stdout:7/453: rename d14/d4a/f6a to d14/d26/d5a/d5f/d6e/f8d 0 2024-04-22T15:18:59.575 INFO:tasks.workunit.client.1.smithi196.stdout:5/478: dwrite d1/d14/d3e/f44 [0,4194304] 0 2024-04-22T15:18:59.577 INFO:tasks.workunit.client.1.smithi196.stdout:1/441: mknod d1/db/d16/c87 0 2024-04-22T15:18:59.577 INFO:tasks.workunit.client.1.smithi196.stdout:9/490: symlink db/d11/d29/d57/la8 0 2024-04-22T15:18:59.578 INFO:tasks.workunit.client.1.smithi196.stdout:1/442: readlink d1/db/dc/de/d5a/l5d 0 2024-04-22T15:18:59.580 INFO:tasks.workunit.client.1.smithi196.stdout:6/484: sync 2024-04-22T15:18:59.580 INFO:tasks.workunit.client.1.smithi196.stdout:4/469: sync 2024-04-22T15:18:59.581 INFO:tasks.workunit.client.1.smithi196.stdout:9/491: write db/dd/de/f4e [369037,103640] 0 2024-04-22T15:18:59.581 INFO:tasks.workunit.client.1.smithi196.stdout:8/475: rmdir d2/dc/d45/d83 0 2024-04-22T15:18:59.583 INFO:tasks.workunit.client.1.smithi196.stdout:6/485: chown dd/d2b/f3e 1714 1 2024-04-22T15:18:59.583 INFO:tasks.workunit.client.1.smithi196.stdout:5/479: stat d1/d17/d1a/d2f/d30/d5e/f63 0 2024-04-22T15:18:59.586 INFO:tasks.workunit.client.1.smithi196.stdout:6/486: fsync dd/d20/f29 0 2024-04-22T15:18:59.586 INFO:tasks.workunit.client.1.smithi196.stdout:2/517: dread d1/d1d/f22 [0,4194304] 0 2024-04-22T15:18:59.586 INFO:tasks.workunit.client.1.smithi196.stdout:3/484: link d0/d15/d4d/db/f2e d0/d15/d4d/fa3 0 2024-04-22T15:18:59.587 INFO:tasks.workunit.client.1.smithi196.stdout:4/470: fdatasync d6/d14/d18/d26/f6a 0 2024-04-22T15:18:59.593 INFO:tasks.workunit.client.1.smithi196.stdout:6/487: dwrite dd/d1f/d4e/d53/d7f/f81 [0,4194304] 0 2024-04-22T15:18:59.596 INFO:tasks.workunit.client.1.smithi196.stdout:3/485: symlink d0/d7/d6f/la4 0 2024-04-22T15:18:59.597 INFO:tasks.workunit.client.1.smithi196.stdout:2/518: fdatasync d1/d13/d41/fa8 0 2024-04-22T15:18:59.600 INFO:tasks.workunit.client.1.smithi196.stdout:9/492: mkdir db/dd/de/d98/da9 0 2024-04-22T15:18:59.600 INFO:tasks.workunit.client.1.smithi196.stdout:0/491: dread d3/de/d1c/d27/f32 [0,4194304] 0 2024-04-22T15:18:59.601 INFO:tasks.workunit.client.1.smithi196.stdout:1/443: creat d1/db/d16/d1e/f88 x:0 0 0 2024-04-22T15:18:59.603 INFO:tasks.workunit.client.1.smithi196.stdout:8/476: symlink d2/d6a/d7b/l84 0 2024-04-22T15:18:59.603 INFO:tasks.workunit.client.1.smithi196.stdout:1/444: read d1/db/d16/d1e/d65/f66 [895402,120951] 0 2024-04-22T15:18:59.605 INFO:tasks.workunit.client.1.smithi196.stdout:7/454: dwrite d14/d25/d31/f38 [0,4194304] 0 2024-04-22T15:18:59.607 INFO:tasks.workunit.client.1.smithi196.stdout:5/480: write d1/d17/d1c/f52 [1103355,123787] 0 2024-04-22T15:18:59.608 INFO:tasks.workunit.client.1.smithi196.stdout:3/486: symlink d0/d15/d4d/db/dc/d85/la5 0 2024-04-22T15:18:59.609 INFO:tasks.workunit.client.1.smithi196.stdout:1/445: write d1/db/dc/d26/d2e/d32/d36/f3f [2149958,104811] 0 2024-04-22T15:18:59.609 INFO:tasks.workunit.client.1.smithi196.stdout:2/519: rename d1/d13/d25/d2a/f77 to d1/d23/d27/d89/fc1 0 2024-04-22T15:18:59.614 INFO:tasks.workunit.client.1.smithi196.stdout:8/477: dwrite d2/d3f/d4b/f72 [0,4194304] 0 2024-04-22T15:18:59.616 INFO:tasks.workunit.client.1.smithi196.stdout:8/478: dread - d2/d8/f76 zero size 2024-04-22T15:18:59.618 INFO:tasks.workunit.client.1.smithi196.stdout:5/481: dwrite d1/d9/f75 [0,4194304] 0 2024-04-22T15:18:59.624 INFO:tasks.workunit.client.1.smithi196.stdout:8/479: dread d2/dc/f10 [0,4194304] 0 2024-04-22T15:18:59.625 INFO:tasks.workunit.client.1.smithi196.stdout:2/520: readlink d1/d13/d1f/la3 0 2024-04-22T15:18:59.628 INFO:tasks.workunit.client.1.smithi196.stdout:5/482: rename d1/d51/f69 to d1/d17/d1a/d2f/f99 0 2024-04-22T15:18:59.630 INFO:tasks.workunit.client.1.smithi196.stdout:0/492: rmdir d3/d6/da2 0 2024-04-22T15:18:59.631 INFO:tasks.workunit.client.1.smithi196.stdout:8/480: mknod d2/d3f/d4b/c85 0 2024-04-22T15:18:59.631 INFO:tasks.workunit.client.1.smithi196.stdout:8/481: dread - d2/d3f/d4b/f6f zero size 2024-04-22T15:18:59.632 INFO:tasks.workunit.client.1.smithi196.stdout:1/446: creat d1/d54/d70/f89 x:0 0 0 2024-04-22T15:18:59.633 INFO:tasks.workunit.client.1.smithi196.stdout:6/488: link dd/f18 dd/d1f/d4e/d53/d55/f8b 0 2024-04-22T15:18:59.633 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:18:59 smithi096 ceph-mon[26688]: pgmap v36: 65 pgs: 65 active+clean; 2.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 82 MiB/s rd, 266 MiB/s wr, 501 op/s 2024-04-22T15:18:59.637 INFO:tasks.workunit.client.1.smithi196.stdout:0/493: creat d3/d9/d36/d56/d65/d7f/d87/d6c/d88/fa3 x:0 0 0 2024-04-22T15:18:59.638 INFO:tasks.workunit.client.1.smithi196.stdout:8/482: creat d2/d8/d49/d6c/f86 x:0 0 0 2024-04-22T15:18:59.642 INFO:tasks.workunit.client.1.smithi196.stdout:8/483: rmdir d2/dc/d45 39 2024-04-22T15:18:59.644 INFO:tasks.workunit.client.1.smithi196.stdout:6/489: rmdir dd/d1f/d4e/d53/d55/d57 39 2024-04-22T15:18:59.645 INFO:tasks.workunit.client.1.smithi196.stdout:3/487: dwrite d0/d15/d4d/db/dc/d18/f1b [4194304,4194304] 0 2024-04-22T15:18:59.648 INFO:tasks.workunit.client.1.smithi196.stdout:7/455: dwrite d14/d15/f1a [0,4194304] 0 2024-04-22T15:18:59.649 INFO:tasks.workunit.client.1.smithi196.stdout:0/494: symlink d3/d6/dc/d8c/d8d/d98/la4 0 2024-04-22T15:18:59.651 INFO:tasks.workunit.client.1.smithi196.stdout:8/484: dwrite d2/d3f/f41 [0,4194304] 0 2024-04-22T15:18:59.697 INFO:tasks.workunit.client.1.smithi196.stdout:6/490: creat dd/d20/d4d/f8c x:0 0 0 2024-04-22T15:18:59.697 INFO:tasks.workunit.client.1.smithi196.stdout:2/521: link d1/d13/d25/c3b d1/d1d/d6a/cc2 0 2024-04-22T15:18:59.697 INFO:tasks.workunit.client.1.smithi196.stdout:5/483: getdents d1/d4/d92 0 2024-04-22T15:18:59.697 INFO:tasks.workunit.client.1.smithi196.stdout:0/495: mknod d3/d9/d36/d56/d65/d7f/ca5 0 2024-04-22T15:18:59.697 INFO:tasks.workunit.client.1.smithi196.stdout:1/447: truncate d1/db/d16/f5c 733363 0 2024-04-22T15:18:59.698 INFO:tasks.workunit.client.1.smithi196.stdout:2/522: write d1/d23/f2f [4535133,98937] 0 2024-04-22T15:18:59.701 INFO:tasks.workunit.client.1.smithi196.stdout:0/496: unlink d3/d9/d36/l89 0 2024-04-22T15:18:59.737 INFO:tasks.workunit.client.0.smithi096.stdout:7/992: sync 2024-04-22T15:18:59.737 INFO:tasks.workunit.client.0.smithi096.stdout:3/670: sync 2024-04-22T15:18:59.737 INFO:tasks.workunit.client.0.smithi096.stdout:8/752: sync 2024-04-22T15:18:59.737 INFO:tasks.workunit.client.0.smithi096.stdout:0/875: sync 2024-04-22T15:18:59.739 INFO:tasks.workunit.client.0.smithi096.stdout:0/876: read - d7/d42/ff8 zero size 2024-04-22T15:18:59.741 INFO:tasks.workunit.client.0.smithi096.stdout:8/753: creat d4/d1c/d40/d46/d58/ffb x:0 0 0 2024-04-22T15:18:59.742 INFO:tasks.workunit.client.1.smithi196.stdout:4/471: sync 2024-04-22T15:18:59.742 INFO:tasks.workunit.client.0.smithi096.stdout:7/993: unlink d31/d32/d33/d93/dfc/f14f 0 2024-04-22T15:18:59.743 INFO:tasks.workunit.client.1.smithi196.stdout:8/485: creat d2/d5/f87 x:0 0 0 2024-04-22T15:18:59.743 INFO:tasks.workunit.client.1.smithi196.stdout:1/448: symlink d1/db/d16/l8a 0 2024-04-22T15:18:59.744 INFO:tasks.workunit.client.0.smithi096.stdout:0/877: rename d7/de/d5b/d95/d66/f119 to d7/d42/f12b 0 2024-04-22T15:18:59.744 INFO:tasks.workunit.client.1.smithi196.stdout:5/484: write d1/d17/d1a/d2f/f99 [3701744,127215] 0 2024-04-22T15:18:59.744 INFO:tasks.workunit.client.1.smithi196.stdout:2/523: symlink d1/d13/d25/d2a/lc3 0 2024-04-22T15:18:59.745 INFO:tasks.workunit.client.0.smithi096.stdout:3/671: link da/db/d12/d9b/fd6 da/db/dc/d59/fd7 0 2024-04-22T15:18:59.746 INFO:tasks.workunit.client.0.smithi096.stdout:8/754: fsync d4/d10/f56 0 2024-04-22T15:18:59.747 INFO:tasks.workunit.client.0.smithi096.stdout:8/755: dread - d4/d1c/d40/f75 zero size 2024-04-22T15:18:59.749 INFO:tasks.workunit.client.0.smithi096.stdout:7/994: creat d31/d32/d33/dfa/f151 x:0 0 0 2024-04-22T15:18:59.750 INFO:tasks.workunit.client.1.smithi196.stdout:6/491: dwrite dd/d20/f4c [0,4194304] 0 2024-04-22T15:18:59.751 INFO:tasks.workunit.client.1.smithi196.stdout:6/492: dread - dd/d20/d5c/d5e/f7c zero size 2024-04-22T15:18:59.752 INFO:tasks.workunit.client.1.smithi196.stdout:5/485: creat d1/d17/d50/f9a x:0 0 0 2024-04-22T15:18:59.756 INFO:tasks.workunit.client.0.smithi096.stdout:8/756: truncate d4/d10/f97 5047977 0 2024-04-22T15:18:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:18:59 smithi196 ceph-mon[34717]: pgmap v36: 65 pgs: 65 active+clean; 2.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 82 MiB/s rd, 266 MiB/s wr, 501 op/s 2024-04-22T15:18:59.764 INFO:tasks.workunit.client.0.smithi096.stdout:8/757: truncate d4/d1c/f72 1862404 0 2024-04-22T15:18:59.766 INFO:tasks.workunit.client.1.smithi196.stdout:4/472: truncate d6/f42 1535565 0 2024-04-22T15:18:59.766 INFO:tasks.workunit.client.0.smithi096.stdout:7/995: mkdir d31/d32/d33/d54/d152 0 2024-04-22T15:18:59.767 INFO:tasks.workunit.client.0.smithi096.stdout:0/878: write d7/de/fe8 [996841,119028] 0 2024-04-22T15:18:59.767 INFO:tasks.workunit.client.0.smithi096.stdout:7/996: chown d31/d32/d33/d6a/de7 1686907 1 2024-04-22T15:18:59.768 INFO:tasks.workunit.client.0.smithi096.stdout:0/879: readlink d7/de/d5b/d95/dc/d16/l50 0 2024-04-22T15:18:59.770 INFO:tasks.workunit.client.0.smithi096.stdout:0/880: write d7/de/d5b/d64/f107 [1639970,126114] 0 2024-04-22T15:18:59.770 INFO:tasks.workunit.client.1.smithi196.stdout:0/497: link d3/d9/d36/d56/d65/d7f/d87/d6c/d88/d95/f9e d3/d6/dc/d8c/d8d/d98/fa6 0 2024-04-22T15:18:59.771 INFO:tasks.workunit.client.0.smithi096.stdout:3/672: link da/db/dc/d28/d69/d89/l84 da/db/dc/d8d/ld8 0 2024-04-22T15:18:59.772 INFO:tasks.workunit.client.1.smithi196.stdout:2/524: mknod d1/d23/d27/d2e/d55/d5b/d7c/dbd/cc4 0 2024-04-22T15:18:59.774 INFO:tasks.workunit.client.1.smithi196.stdout:8/486: creat d2/d8/d47/d68/d81/f88 x:0 0 0 2024-04-22T15:18:59.777 INFO:tasks.workunit.client.0.smithi096.stdout:8/758: dread d4/d1c/d62/dc3/fda [0,4194304] 0 2024-04-22T15:18:59.779 INFO:tasks.workunit.client.0.smithi096.stdout:3/673: dwrite da/db/de/d19/d2d/fa8 [0,4194304] 0 2024-04-22T15:18:59.781 INFO:tasks.workunit.client.0.smithi096.stdout:7/997: rename d31/d32/dad/fd3 to d31/d32/d33/d6a/d83/dba/dc7/f153 0 2024-04-22T15:18:59.783 INFO:tasks.workunit.client.0.smithi096.stdout:0/881: chown d7/de/d5b/dda/f105 45870 1 2024-04-22T15:18:59.787 INFO:tasks.workunit.client.0.smithi096.stdout:0/882: read d7/de/d5b/d95/dc/d16/d65/d7c/f85 [1569381,107463] 0 2024-04-22T15:18:59.787 INFO:tasks.workunit.client.1.smithi196.stdout:5/486: mknod d1/d2/c9b 0 2024-04-22T15:18:59.788 INFO:tasks.workunit.client.1.smithi196.stdout:5/487: fdatasync d1/d17/f60 0 2024-04-22T15:18:59.789 INFO:tasks.workunit.client.1.smithi196.stdout:5/488: write d1/d17/d1a/d2f/d30/f54 [2091336,91752] 0 2024-04-22T15:18:59.790 INFO:tasks.workunit.client.1.smithi196.stdout:2/525: truncate d1/d23/d27/d2e/d55/f87 966593 0 2024-04-22T15:18:59.790 INFO:tasks.workunit.client.1.smithi196.stdout:1/449: truncate d1/db/d16/d1e/d65/f76 1014590 0 2024-04-22T15:18:59.791 INFO:tasks.workunit.client.1.smithi196.stdout:2/526: readlink d1/d13/d25/d2a/lc3 0 2024-04-22T15:18:59.792 INFO:tasks.workunit.client.1.smithi196.stdout:1/450: write d1/db/dc/de/d5a/d5b/f79 [879657,107008] 0 2024-04-22T15:18:59.792 INFO:tasks.workunit.client.1.smithi196.stdout:8/487: unlink d2/d8/l12 0 2024-04-22T15:18:59.794 INFO:tasks.workunit.client.1.smithi196.stdout:8/488: write d2/d8/d49/f59 [3160545,104707] 0 2024-04-22T15:18:59.794 INFO:tasks.workunit.client.0.smithi096.stdout:8/759: mknod d4/d10/d25/d4c/d76/cfc 0 2024-04-22T15:18:59.795 INFO:tasks.workunit.client.1.smithi196.stdout:8/489: chown d2/d5/d1a/l44 0 1 2024-04-22T15:18:59.795 INFO:tasks.workunit.client.1.smithi196.stdout:8/490: truncate d2/d5/f87 346594 0 2024-04-22T15:18:59.798 INFO:tasks.workunit.client.1.smithi196.stdout:0/498: dread d3/de/d1c/f31 [0,4194304] 0 2024-04-22T15:18:59.799 INFO:tasks.workunit.client.0.smithi096.stdout:1/791: sync 2024-04-22T15:18:59.800 INFO:tasks.workunit.client.0.smithi096.stdout:2/781: sync 2024-04-22T15:18:59.801 INFO:tasks.workunit.client.0.smithi096.stdout:7/998: mkdir d31/d32/d33/d6a/d10a/d154 0 2024-04-22T15:18:59.806 INFO:tasks.workunit.client.0.smithi096.stdout:0/883: mknod d7/d8f/d10e/d114/c12c 0 2024-04-22T15:18:59.807 INFO:tasks.workunit.client.1.smithi196.stdout:5/489: creat d1/d4/f9c x:0 0 0 2024-04-22T15:18:59.808 INFO:tasks.workunit.client.0.smithi096.stdout:2/782: dwrite d3/db/df/dce/d46/d69/fb9 [0,4194304] 0 2024-04-22T15:18:59.808 INFO:tasks.workunit.client.1.smithi196.stdout:5/490: dread d1/d17/d1a/f43 [0,4194304] 0 2024-04-22T15:18:59.808 INFO:tasks.workunit.client.0.smithi096.stdout:0/884: write d7/de/d5b/d64/fb6 [77429,88400] 0 2024-04-22T15:18:59.810 INFO:tasks.workunit.client.0.smithi096.stdout:8/760: rename d4/d1c/d40/d48/c6e to d4/d1c/d40/d48/cfd 0 2024-04-22T15:18:59.813 INFO:tasks.workunit.client.1.smithi196.stdout:2/527: mknod d1/d1d/d6a/da1/cc5 0 2024-04-22T15:18:59.814 INFO:tasks.workunit.client.0.smithi096.stdout:3/674: mknod da/db/de/d19/d2d/cd9 0 2024-04-22T15:18:59.814 INFO:tasks.workunit.client.1.smithi196.stdout:2/528: chown d1/d1d/f68 3 1 2024-04-22T15:18:59.815 INFO:tasks.workunit.client.1.smithi196.stdout:5/491: dwrite d1/d17/d1a/f43 [0,4194304] 0 2024-04-22T15:18:59.815 INFO:tasks.workunit.client.0.smithi096.stdout:3/675: read - da/db/d9a/fc1 zero size 2024-04-22T15:18:59.816 INFO:tasks.workunit.client.1.smithi196.stdout:5/492: write d1/d4/f56 [371487,60530] 0 2024-04-22T15:18:59.817 INFO:tasks.workunit.client.1.smithi196.stdout:1/451: truncate d1/db/dc/de/f14 3146206 0 2024-04-22T15:18:59.818 INFO:tasks.workunit.client.1.smithi196.stdout:9/493: sync 2024-04-22T15:18:59.819 INFO:tasks.workunit.client.1.smithi196.stdout:1/452: chown d1/db/dc/de/d43/d5e/d7a/f83 0 1 2024-04-22T15:18:59.819 INFO:tasks.workunit.client.1.smithi196.stdout:9/494: chown db/d11/d29/d57/c39 2 1 2024-04-22T15:18:59.821 INFO:tasks.workunit.client.0.smithi096.stdout:1/792: mknod d6/dc/cf7 0 2024-04-22T15:18:59.823 INFO:tasks.workunit.client.0.smithi096.stdout:3/676: dwrite da/db/d12/d22/f53 [0,4194304] 0 2024-04-22T15:18:59.827 INFO:tasks.workunit.client.0.smithi096.stdout:3/677: dread da/db/dc/d28/f45 [0,4194304] 0 2024-04-22T15:18:59.828 INFO:tasks.workunit.client.1.smithi196.stdout:0/499: mknod d3/de/ca7 0 2024-04-22T15:18:59.833 INFO:tasks.workunit.client.1.smithi196.stdout:4/473: rename d6/d13/d15/d2c/d35/d44 to d6/d90 0 2024-04-22T15:18:59.836 INFO:tasks.workunit.client.0.smithi096.stdout:2/783: creat d3/d1f/d2f/ff9 x:0 0 0 2024-04-22T15:18:59.838 INFO:tasks.workunit.client.0.smithi096.stdout:0/885: creat d7/de/d3a/d61/f12d x:0 0 0 2024-04-22T15:18:59.840 INFO:tasks.workunit.client.1.smithi196.stdout:2/529: creat d1/d23/d27/d73/fc6 x:0 0 0 2024-04-22T15:18:59.841 INFO:tasks.workunit.client.0.smithi096.stdout:8/761: creat d4/d1c/d40/da7/ffe x:0 0 0 2024-04-22T15:18:59.842 INFO:tasks.workunit.client.0.smithi096.stdout:0/886: read d7/d42/d4c/d55/d9c/df7/f10d [947805,27077] 0 2024-04-22T15:18:59.848 INFO:tasks.workunit.client.0.smithi096.stdout:1/793: creat d6/dc/d8a/d29/ff8 x:0 0 0 2024-04-22T15:18:59.849 INFO:tasks.workunit.client.1.smithi196.stdout:9/495: creat db/dd/de/d75/faa x:0 0 0 2024-04-22T15:18:59.850 INFO:tasks.workunit.client.0.smithi096.stdout:7/999: dwrite d31/d32/d33/d44/fc6 [0,4194304] 0 2024-04-22T15:18:59.855 INFO:tasks.workunit.client.0.smithi096.stdout:2/784: mkdir d3/db/df/dce/d46/d7e/d8d/dfa 0 2024-04-22T15:18:59.858 INFO:tasks.workunit.client.0.smithi096.stdout:3/678: dwrite da/db/de/d87/fb6 [0,4194304] 0 2024-04-22T15:18:59.858 INFO:tasks.workunit.client.1.smithi196.stdout:3/488: sync 2024-04-22T15:18:59.858 INFO:tasks.workunit.client.1.smithi196.stdout:7/456: sync 2024-04-22T15:18:59.858 INFO:tasks.workunit.client.1.smithi196.stdout:6/493: sync 2024-04-22T15:18:59.859 INFO:tasks.workunit.client.0.smithi096.stdout:3/679: chown da/db/d12/d22/d26/db2/fc2 14311 1 2024-04-22T15:18:59.862 INFO:tasks.workunit.client.1.smithi196.stdout:4/474: write d6/d14/d18/d40/f6c [697860,48120] 0 2024-04-22T15:18:59.862 INFO:tasks.workunit.client.0.smithi096.stdout:8/762: creat d4/d71/fff x:0 0 0 2024-04-22T15:18:59.863 INFO:tasks.workunit.client.0.smithi096.stdout:8/763: dread - d4/d10/d25/fe1 zero size 2024-04-22T15:18:59.863 INFO:tasks.workunit.client.1.smithi196.stdout:4/475: chown d6/d13/d15/d2c/d62/d78 7129703 1 2024-04-22T15:18:59.865 INFO:tasks.workunit.client.0.smithi096.stdout:0/887: mknod d7/d8f/d10e/d114/c12e 0 2024-04-22T15:18:59.867 INFO:tasks.workunit.client.1.smithi196.stdout:4/476: readlink d6/d13/d15/l64 0 2024-04-22T15:18:59.867 INFO:tasks.workunit.client.0.smithi096.stdout:3/680: dread da/db/d12/d22/f7d [0,4194304] 0 2024-04-22T15:18:59.868 INFO:tasks.workunit.client.1.smithi196.stdout:6/494: dwrite dd/d20/d4d/f8c [0,4194304] 0 2024-04-22T15:18:59.868 INFO:tasks.workunit.client.0.smithi096.stdout:1/794: mknod d6/dc/d8a/d3f/d54/d32/d8c/dca/cf9 0 2024-04-22T15:18:59.871 INFO:tasks.workunit.client.1.smithi196.stdout:9/496: chown db/d11/d29/d7a/f7d 123 1 2024-04-22T15:18:59.872 INFO:tasks.workunit.client.0.smithi096.stdout:2/785: fsync d3/db/df/d10/f3b 0 2024-04-22T15:18:59.874 INFO:tasks.workunit.client.1.smithi196.stdout:1/453: dwrite d1/db/dc/de/d43/f48 [0,4194304] 0 2024-04-22T15:18:59.875 INFO:tasks.workunit.client.0.smithi096.stdout:1/795: read d6/dc/d8a/d3f/d54/d32/d3b/d3d/f4b [2923067,91356] 0 2024-04-22T15:18:59.875 INFO:tasks.workunit.client.1.smithi196.stdout:8/491: link d2/dc/fe d2/d8/d47/d68/d81/f89 0 2024-04-22T15:18:59.876 INFO:tasks.workunit.client.0.smithi096.stdout:1/796: write d6/fe6 [568638,25499] 0 2024-04-22T15:18:59.879 INFO:tasks.workunit.client.1.smithi196.stdout:4/477: dread d6/d14/f27 [4194304,4194304] 0 2024-04-22T15:18:59.880 INFO:tasks.workunit.client.1.smithi196.stdout:7/457: dread - d14/d26/d5e/f7b zero size 2024-04-22T15:18:59.880 INFO:tasks.workunit.client.1.smithi196.stdout:0/500: write d3/de/d14/d72/f7c [30279,116718] 0 2024-04-22T15:18:59.882 INFO:tasks.workunit.client.1.smithi196.stdout:8/492: dread d2/f1d [0,4194304] 0 2024-04-22T15:18:59.882 INFO:tasks.workunit.client.1.smithi196.stdout:4/478: chown d6/d14/c1e 58472 1 2024-04-22T15:18:59.884 INFO:tasks.workunit.client.1.smithi196.stdout:8/493: chown d2/dc/f2f 553696914 1 2024-04-22T15:18:59.886 INFO:tasks.workunit.client.1.smithi196.stdout:8/494: truncate d2/d8/d49/f7a 812218 0 2024-04-22T15:18:59.887 INFO:tasks.workunit.client.1.smithi196.stdout:5/493: dwrite d1/d17/f21 [4194304,4194304] 0 2024-04-22T15:18:59.888 INFO:tasks.workunit.client.1.smithi196.stdout:5/494: dread - d1/d9/d68/f70 zero size 2024-04-22T15:18:59.890 INFO:tasks.workunit.client.0.smithi096.stdout:8/764: fsync d4/d10/d25/f37 0 2024-04-22T15:18:59.892 INFO:tasks.workunit.client.1.smithi196.stdout:6/495: mkdir dd/d2b/d8d 0 2024-04-22T15:18:59.893 INFO:tasks.workunit.client.1.smithi196.stdout:6/496: write dd/d1f/d37/f74 [731312,102794] 0 2024-04-22T15:18:59.897 INFO:tasks.workunit.client.1.smithi196.stdout:7/458: creat d14/d25/d31/f8e x:0 0 0 2024-04-22T15:18:59.898 INFO:tasks.workunit.client.1.smithi196.stdout:0/501: fsync d3/d6/dc/f92 0 2024-04-22T15:18:59.900 INFO:tasks.workunit.client.1.smithi196.stdout:6/497: dwrite dd/f15 [4194304,4194304] 0 2024-04-22T15:18:59.900 INFO:tasks.workunit.client.0.smithi096.stdout:1/797: dread d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fdf [0,4194304] 0 2024-04-22T15:18:59.901 INFO:tasks.workunit.client.1.smithi196.stdout:3/489: write d0/d15/d4d/db/f2e [2845419,59401] 0 2024-04-22T15:18:59.901 INFO:tasks.workunit.client.1.smithi196.stdout:6/498: chown dd/d2b/f62 14083 1 2024-04-22T15:18:59.902 INFO:tasks.workunit.client.1.smithi196.stdout:9/497: read fa [50333,119138] 0 2024-04-22T15:18:59.906 INFO:tasks.workunit.client.1.smithi196.stdout:3/490: dread d0/d15/d4d/db/dc/f1c [0,4194304] 0 2024-04-22T15:18:59.909 INFO:tasks.workunit.client.1.smithi196.stdout:2/530: rename d1/d1d/fbb to d1/d13/fc7 0 2024-04-22T15:18:59.910 INFO:tasks.workunit.client.0.smithi096.stdout:2/786: dwrite d3/db/df/d98/db3/fbb [0,4194304] 0 2024-04-22T15:18:59.910 INFO:tasks.workunit.client.0.smithi096.stdout:0/888: link d7/de/d5b/d95/dc/f56 d7/d8f/f12f 0 2024-04-22T15:18:59.915 INFO:tasks.workunit.client.0.smithi096.stdout:1/798: truncate d6/dc/d8a/d3f/d54/d32/d3b/d94/fad 381178 0 2024-04-22T15:18:59.919 INFO:tasks.workunit.client.1.smithi196.stdout:4/479: write d6/d13/d15/d2c/d35/f4c [2099663,115903] 0 2024-04-22T15:18:59.920 INFO:tasks.workunit.client.1.smithi196.stdout:8/495: dread d2/dc/f20 [0,4194304] 0 2024-04-22T15:18:59.920 INFO:tasks.workunit.client.0.smithi096.stdout:3/681: truncate da/db/dc/f5d 939376 0 2024-04-22T15:18:59.920 INFO:tasks.workunit.client.0.smithi096.stdout:0/889: dwrite d7/d42/f127 [0,4194304] 0 2024-04-22T15:18:59.922 INFO:tasks.workunit.client.0.smithi096.stdout:0/890: chown d7/d42/d4c/d55/d9c/df7/dfb 3 1 2024-04-22T15:18:59.923 INFO:tasks.workunit.client.0.smithi096.stdout:2/787: creat d3/db/df/d24/dc8/ffb x:0 0 0 2024-04-22T15:18:59.923 INFO:tasks.workunit.client.0.smithi096.stdout:0/891: fsync d7/da7/f110 0 2024-04-22T15:18:59.931 INFO:tasks.workunit.client.1.smithi196.stdout:2/531: dread d1/d23/f2f [0,4194304] 0 2024-04-22T15:18:59.942 INFO:tasks.workunit.client.0.smithi096.stdout:4/765: sync 2024-04-22T15:18:59.943 INFO:tasks.workunit.client.0.smithi096.stdout:9/667: sync 2024-04-22T15:18:59.943 INFO:tasks.workunit.client.0.smithi096.stdout:5/786: sync 2024-04-22T15:18:59.943 INFO:tasks.workunit.client.0.smithi096.stdout:6/626: sync 2024-04-22T15:18:59.948 INFO:tasks.workunit.client.1.smithi196.stdout:6/499: dread fc [4194304,4194304] 0 2024-04-22T15:18:59.949 INFO:tasks.workunit.client.0.smithi096.stdout:9/668: fsync d1/df/d95/d9d/db5/d43/d57/d5b/fa4 0 2024-04-22T15:18:59.954 INFO:tasks.workunit.client.0.smithi096.stdout:3/682: dread da/db/dc/d28/f85 [0,4194304] 0 2024-04-22T15:18:59.956 INFO:tasks.workunit.client.0.smithi096.stdout:5/787: dread d4/fcc [0,4194304] 0 2024-04-22T15:18:59.957 INFO:tasks.workunit.client.0.smithi096.stdout:6/627: dwrite d3/d28/f21 [0,4194304] 0 2024-04-22T15:18:59.959 INFO:tasks.workunit.client.0.smithi096.stdout:5/788: stat d4/d2c/d2f/d30/le3 0 2024-04-22T15:18:59.959 INFO:tasks.workunit.client.0.smithi096.stdout:6/628: dread - d3/d28/dd/f2b zero size 2024-04-22T15:18:59.960 INFO:tasks.workunit.client.0.smithi096.stdout:5/789: write d4/ddb/d3d/d4d/fd7 [827641,69516] 0 2024-04-22T15:19:00.017 INFO:tasks.workunit.client.0.smithi096.stdout:8/765: write d4/d10/d25/d4c/d76/fef [1127840,79373] 0 2024-04-22T15:19:00.019 INFO:tasks.workunit.client.0.smithi096.stdout:8/766: write d4/d10/f19 [321248,118382] 0 2024-04-22T15:19:00.032 INFO:tasks.workunit.client.1.smithi196.stdout:9/498: unlink db/dd/l72 0 2024-04-22T15:19:00.032 INFO:tasks.workunit.client.1.smithi196.stdout:9/499: fdatasync db/dd/f5c 0 2024-04-22T15:19:00.052 INFO:tasks.workunit.client.0.smithi096.stdout:5/790: dwrite d4/f1c [0,4194304] 0 2024-04-22T15:19:00.065 INFO:tasks.workunit.client.0.smithi096.stdout:8/767: dread - d4/d1c/d40/da7/fd9 zero size 2024-04-22T15:19:00.066 INFO:tasks.workunit.client.1.smithi196.stdout:2/532: rmdir d1/d13/d41 39 2024-04-22T15:19:00.082 INFO:tasks.workunit.client.0.smithi096.stdout:4/766: symlink d5/d7/d2c/d95/df9/l107 0 2024-04-22T15:19:00.085 INFO:tasks.workunit.client.0.smithi096.stdout:5/791: dread - d4/d2c/d2f/d6b/f103 zero size 2024-04-22T15:19:00.085 INFO:tasks.workunit.client.0.smithi096.stdout:5/792: chown d4/ddb/f22 113536 1 2024-04-22T15:19:00.086 INFO:tasks.workunit.client.1.smithi196.stdout:3/491: mkdir d0/d15/d4d/d87/da6 0 2024-04-22T15:19:00.091 INFO:tasks.workunit.client.1.smithi196.stdout:6/500: rmdir dd/d1f/d37/d3c 39 2024-04-22T15:19:00.093 INFO:tasks.workunit.client.0.smithi096.stdout:8/768: fdatasync d4/d10/f99 0 2024-04-22T15:19:00.094 INFO:tasks.workunit.client.0.smithi096.stdout:8/769: write d4/d1c/fc4 [3530123,118783] 0 2024-04-22T15:19:00.095 INFO:tasks.workunit.client.1.smithi196.stdout:5/495: getdents d1/d9 0 2024-04-22T15:19:00.095 INFO:tasks.workunit.client.1.smithi196.stdout:5/496: fdatasync d1/d9/f75 0 2024-04-22T15:19:00.095 INFO:tasks.workunit.client.0.smithi096.stdout:2/788: link d3/db/df/dce/d46/l57 d3/db/df/dce/d46/d69/dd7/lfc 0 2024-04-22T15:19:00.096 INFO:tasks.workunit.client.1.smithi196.stdout:5/497: chown d1/cb 2154991 1 2024-04-22T15:19:00.096 INFO:tasks.workunit.client.0.smithi096.stdout:8/770: dread - d4/d10/d25/fe1 zero size 2024-04-22T15:19:00.098 INFO:tasks.workunit.client.1.smithi196.stdout:8/496: creat d2/d3f/d4b/d79/f8a x:0 0 0 2024-04-22T15:19:00.098 INFO:tasks.workunit.client.1.smithi196.stdout:8/497: chown d2/d8/l21 0 1 2024-04-22T15:19:00.100 INFO:tasks.workunit.client.0.smithi096.stdout:4/767: symlink d5/d7/l108 0 2024-04-22T15:19:00.100 INFO:tasks.workunit.client.0.smithi096.stdout:4/768: dread - d5/d5b/fd0 zero size 2024-04-22T15:19:00.103 INFO:tasks.workunit.client.0.smithi096.stdout:5/793: mkdir d4/ddb/d3d/d4d/d59/d98/dee/d109 0 2024-04-22T15:19:00.105 INFO:tasks.workunit.client.0.smithi096.stdout:1/799: getdents d6/dc/d8a/d3f/d62/d7d/d8e 0 2024-04-22T15:19:00.107 INFO:tasks.workunit.client.1.smithi196.stdout:7/459: creat d14/d26/d5a/f8f x:0 0 0 2024-04-22T15:19:00.108 INFO:tasks.workunit.client.1.smithi196.stdout:4/480: mknod d6/d14/d18/d26/d32/c91 0 2024-04-22T15:19:00.108 INFO:tasks.workunit.client.1.smithi196.stdout:4/481: chown d6/d13/f24 943425869 1 2024-04-22T15:19:00.110 INFO:tasks.workunit.client.0.smithi096.stdout:5/794: dwrite d4/ddb/f22 [0,4194304] 0 2024-04-22T15:19:00.112 INFO:tasks.workunit.client.0.smithi096.stdout:9/669: mknod d1/df/d95/d9d/db5/d43/d57/d5b/d8e/ce2 0 2024-04-22T15:19:00.112 INFO:tasks.workunit.client.0.smithi096.stdout:5/795: chown d4/ddb/d3d/d35/d57/fb4 122 1 2024-04-22T15:19:00.113 INFO:tasks.workunit.client.1.smithi196.stdout:9/500: mkdir db/dd/de/d15/d31/d64/dab 0 2024-04-22T15:19:00.125 INFO:tasks.workunit.client.1.smithi196.stdout:5/498: creat d1/d4/f9d x:0 0 0 2024-04-22T15:19:00.127 INFO:tasks.workunit.client.1.smithi196.stdout:8/498: mkdir d2/d8/d49/d8b 0 2024-04-22T15:19:00.132 INFO:tasks.workunit.client.1.smithi196.stdout:1/454: link d1/l11 d1/db/d16/d1e/d49/d7f/l8b 0 2024-04-22T15:19:00.133 INFO:tasks.workunit.client.0.smithi096.stdout:5/796: mknod d4/ddb/d3d/d4f/c10a 0 2024-04-22T15:19:00.136 INFO:tasks.workunit.client.1.smithi196.stdout:4/482: mknod d6/d10/c92 0 2024-04-22T15:19:00.138 INFO:tasks.workunit.client.0.smithi096.stdout:1/800: mknod d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/cfa 0 2024-04-22T15:19:00.140 INFO:tasks.workunit.client.1.smithi196.stdout:4/483: dread d6/d13/d15/d2c/d62/f88 [0,4194304] 0 2024-04-22T15:19:00.140 INFO:tasks.workunit.client.1.smithi196.stdout:0/502: getdents d3/de/d14/d86 0 2024-04-22T15:19:00.144 INFO:tasks.workunit.client.1.smithi196.stdout:9/501: mknod db/dd/cac 0 2024-04-22T15:19:00.144 INFO:tasks.workunit.client.1.smithi196.stdout:4/484: dread d6/d14/d18/d40/f6c [0,4194304] 0 2024-04-22T15:19:00.148 INFO:tasks.workunit.client.0.smithi096.stdout:1/801: creat d6/dc/dbd/ffb x:0 0 0 2024-04-22T15:19:00.152 INFO:tasks.workunit.client.0.smithi096.stdout:5/797: fsync d4/ff5 0 2024-04-22T15:19:00.154 INFO:tasks.workunit.client.0.smithi096.stdout:5/798: read d4/ddb/d3d/d4d/d59/d5a/f13 [2469873,19013] 0 2024-04-22T15:19:00.155 INFO:tasks.workunit.client.0.smithi096.stdout:5/799: chown d4/ddb/d3d/d4f/c10a 4 1 2024-04-22T15:19:00.159 INFO:tasks.workunit.client.1.smithi196.stdout:8/499: creat d2/d6a/f8c x:0 0 0 2024-04-22T15:19:00.160 INFO:tasks.workunit.client.0.smithi096.stdout:1/802: readlink d6/l64 0 2024-04-22T15:19:00.179 INFO:tasks.workunit.client.0.smithi096.stdout:5/800: creat d4/d2c/d2f/d31/f10b x:0 0 0 2024-04-22T15:19:00.180 INFO:tasks.workunit.client.1.smithi196.stdout:0/503: chown d3/de/d1c/f31 4 1 2024-04-22T15:19:00.181 INFO:tasks.workunit.client.1.smithi196.stdout:0/504: chown d3/d9/d36/d56/d65/d7f/d30 466844 1 2024-04-22T15:19:00.182 INFO:tasks.workunit.client.1.smithi196.stdout:0/505: fdatasync d3/d9/d36/d56/d65/f6b 0 2024-04-22T15:19:00.183 INFO:tasks.workunit.client.1.smithi196.stdout:6/501: creat dd/d1f/d37/f8e x:0 0 0 2024-04-22T15:19:00.183 INFO:tasks.workunit.client.1.smithi196.stdout:6/502: chown dd/d2b/f77 65 1 2024-04-22T15:19:00.184 INFO:tasks.workunit.client.1.smithi196.stdout:3/492: unlink d0/d15/d31/l9a 0 2024-04-22T15:19:00.186 INFO:tasks.workunit.client.1.smithi196.stdout:1/455: symlink d1/l8c 0 2024-04-22T15:19:00.187 INFO:tasks.workunit.client.1.smithi196.stdout:7/460: creat d14/d25/f90 x:0 0 0 2024-04-22T15:19:00.191 INFO:tasks.workunit.client.1.smithi196.stdout:7/461: dread d14/d15/f1a [0,4194304] 0 2024-04-22T15:19:00.191 INFO:tasks.workunit.client.1.smithi196.stdout:7/462: stat d14/d15 0 2024-04-22T15:19:00.192 INFO:tasks.workunit.client.1.smithi196.stdout:6/503: rmdir dd/d20/d5c 39 2024-04-22T15:19:00.193 INFO:tasks.workunit.client.1.smithi196.stdout:8/500: link d2/d5/d4f/f6e d2/d8/d47/d68/d81/f8d 0 2024-04-22T15:19:00.196 INFO:tasks.workunit.client.1.smithi196.stdout:1/456: mknod d1/d54/d70/c8d 0 2024-04-22T15:19:00.199 INFO:tasks.workunit.client.1.smithi196.stdout:4/485: creat d6/f93 x:0 0 0 2024-04-22T15:19:00.204 INFO:tasks.workunit.client.1.smithi196.stdout:7/463: mkdir d14/d15/d91 0 2024-04-22T15:19:00.204 INFO:tasks.workunit.client.1.smithi196.stdout:7/464: chown d14/d25/f3b 8722565 1 2024-04-22T15:19:00.211 INFO:tasks.workunit.client.1.smithi196.stdout:1/457: mkdir d1/db/d16/d8e 0 2024-04-22T15:19:00.222 INFO:tasks.workunit.client.0.smithi096.stdout:2/789: creat d3/db/df/dce/ffd x:0 0 0 2024-04-22T15:19:00.223 INFO:tasks.workunit.client.1.smithi196.stdout:6/504: rmdir dd/d1f/d4e/d53/d55/d57 39 2024-04-22T15:19:00.225 INFO:tasks.workunit.client.1.smithi196.stdout:2/533: sync 2024-04-22T15:19:00.226 INFO:tasks.workunit.client.1.smithi196.stdout:8/501: symlink d2/dc/l8e 0 2024-04-22T15:19:00.226 INFO:tasks.workunit.client.1.smithi196.stdout:2/534: stat d1/d23/d27/d2e/d55 0 2024-04-22T15:19:00.227 INFO:tasks.workunit.client.1.smithi196.stdout:1/458: readlink d1/la 0 2024-04-22T15:19:00.228 INFO:tasks.workunit.client.0.smithi096.stdout:6/629: write d3/d28/d54/d68/f76 [2429678,114328] 0 2024-04-22T15:19:00.229 INFO:tasks.workunit.client.1.smithi196.stdout:6/505: dwrite dd/d1f/d4e/d53/d55/f87 [0,4194304] 0 2024-04-22T15:19:00.230 INFO:tasks.workunit.client.1.smithi196.stdout:1/459: fdatasync d1/db/dc/d26/d2e/d32/f53 0 2024-04-22T15:19:00.232 INFO:tasks.workunit.client.1.smithi196.stdout:0/506: getdents d3/d9/d36/d56/d65/d7f/d87/d6c/d88 0 2024-04-22T15:19:00.238 INFO:tasks.workunit.client.1.smithi196.stdout:2/535: dwrite d1/d23/d27/d2e/d55/f36 [8388608,4194304] 0 2024-04-22T15:19:00.239 INFO:tasks.workunit.client.1.smithi196.stdout:0/507: truncate d3/d9/d36/d56/d65/d7f/d51/f6a 81978 0 2024-04-22T15:19:00.241 INFO:tasks.workunit.client.1.smithi196.stdout:0/508: dread - d3/d9/d36/d56/d65/d7f/d87/d6c/d88/d95/f9e zero size 2024-04-22T15:19:00.243 INFO:tasks.workunit.client.1.smithi196.stdout:6/506: chown dd/d20/d5c 9636968 1 2024-04-22T15:19:00.246 INFO:tasks.workunit.client.1.smithi196.stdout:8/502: dread d2/d8/d47/f60 [0,4194304] 0 2024-04-22T15:19:00.254 INFO:tasks.workunit.client.1.smithi196.stdout:0/509: mkdir d3/d6/dc/d8c/d8d/da8 0 2024-04-22T15:19:00.258 INFO:tasks.workunit.client.1.smithi196.stdout:1/460: symlink d1/db/dc/de/d5a/d5b/d85/l8f 0 2024-04-22T15:19:00.259 INFO:tasks.workunit.client.1.smithi196.stdout:9/502: sync 2024-04-22T15:19:00.260 INFO:tasks.workunit.client.1.smithi196.stdout:8/503: rmdir d2/dc 39 2024-04-22T15:19:00.265 INFO:tasks.workunit.client.1.smithi196.stdout:1/461: dwrite d1/db/dc/d26/d2e/d32/d36/f3f [0,4194304] 0 2024-04-22T15:19:00.267 INFO:tasks.workunit.client.1.smithi196.stdout:0/510: dread d3/d6/dc/f58 [0,4194304] 0 2024-04-22T15:19:00.270 INFO:tasks.workunit.client.1.smithi196.stdout:8/504: fsync d2/d5/d2e/f43 0 2024-04-22T15:19:00.273 INFO:tasks.workunit.client.1.smithi196.stdout:6/507: dread f2 [0,4194304] 0 2024-04-22T15:19:00.275 INFO:tasks.workunit.client.1.smithi196.stdout:1/462: mkdir d1/db/d16/d1e/d49/d90 0 2024-04-22T15:19:00.280 INFO:tasks.workunit.client.1.smithi196.stdout:0/511: creat d3/d9/d36/d56/d65/d7f/d87/d6c/d88/fa9 x:0 0 0 2024-04-22T15:19:00.282 INFO:tasks.workunit.client.0.smithi096.stdout:8/771: dwrite d4/d1c/f43 [0,4194304] 0 2024-04-22T15:19:00.284 INFO:tasks.workunit.client.1.smithi196.stdout:8/505: symlink d2/d6a/l8f 0 2024-04-22T15:19:00.284 INFO:tasks.workunit.client.1.smithi196.stdout:0/512: dread d3/de/d14/d5a/f68 [0,4194304] 0 2024-04-22T15:19:00.286 INFO:tasks.workunit.client.1.smithi196.stdout:6/508: mkdir dd/d2b/d8f 0 2024-04-22T15:19:00.290 INFO:tasks.workunit.client.0.smithi096.stdout:0/892: rename d7/de/d5b/d95/dc/f96 to d7/de/d5b/d95/d15/d35/f130 0 2024-04-22T15:19:00.296 INFO:tasks.workunit.client.0.smithi096.stdout:2/790: mkdir d3/db/df/d24/d58/d5f/dfe 0 2024-04-22T15:19:00.298 INFO:tasks.workunit.client.0.smithi096.stdout:6/630: rmdir d3/d28/dd/d11 39 2024-04-22T15:19:00.299 INFO:tasks.workunit.client.1.smithi196.stdout:0/513: write d3/de/f7d [2901620,61551] 0 2024-04-22T15:19:00.300 INFO:tasks.workunit.client.0.smithi096.stdout:0/893: dwrite d7/d42/f127 [0,4194304] 0 2024-04-22T15:19:00.305 INFO:tasks.workunit.client.1.smithi196.stdout:1/463: symlink d1/db/dc/de/d52/l91 0 2024-04-22T15:19:00.307 INFO:tasks.workunit.client.0.smithi096.stdout:9/670: dwrite d1/f2f [0,4194304] 0 2024-04-22T15:19:00.309 INFO:tasks.workunit.client.0.smithi096.stdout:0/894: fsync d7/de/d5b/d95/dc/d16/d65/f84 0 2024-04-22T15:19:00.312 INFO:tasks.workunit.client.0.smithi096.stdout:0/895: fdatasync d7/de/d3a/d61/f12d 0 2024-04-22T15:19:00.316 INFO:tasks.workunit.client.0.smithi096.stdout:2/791: dwrite d3/db/df/dce/ffd [0,4194304] 0 2024-04-22T15:19:00.320 INFO:tasks.workunit.client.1.smithi196.stdout:8/506: mknod d2/d5/c90 0 2024-04-22T15:19:00.323 INFO:tasks.workunit.client.1.smithi196.stdout:8/507: dread d2/d5/d2e/f43 [0,4194304] 0 2024-04-22T15:19:00.328 INFO:tasks.workunit.client.0.smithi096.stdout:3/683: mkdir da/db/dc/d28/dda 0 2024-04-22T15:19:00.329 INFO:tasks.workunit.client.1.smithi196.stdout:5/499: dwrite d1/d17/d1a/d2f/d30/d5e/f63 [0,4194304] 0 2024-04-22T15:19:00.329 INFO:tasks.workunit.client.0.smithi096.stdout:2/792: dwrite d3/db/df/d24/f8f [0,4194304] 0 2024-04-22T15:19:00.330 INFO:tasks.workunit.client.1.smithi196.stdout:5/500: write d1/d17/d50/f6d [1929015,78966] 0 2024-04-22T15:19:00.331 INFO:tasks.workunit.client.1.smithi196.stdout:5/501: write d1/d17/d50/f9a [932746,39023] 0 2024-04-22T15:19:00.333 INFO:tasks.workunit.client.1.smithi196.stdout:0/514: creat d3/d9/d36/d56/d65/d7f/d30/faa x:0 0 0 2024-04-22T15:19:00.336 INFO:tasks.workunit.client.1.smithi196.stdout:6/509: read dd/d1f/d38/d40/f42 [2219350,8176] 0 2024-04-22T15:19:00.337 INFO:tasks.workunit.client.1.smithi196.stdout:5/502: dwrite d1/d14/f1e [0,4194304] 0 2024-04-22T15:19:00.339 INFO:tasks.workunit.client.1.smithi196.stdout:6/510: truncate dd/d1f/d4e/d53/d55/f8b 9087443 0 2024-04-22T15:19:00.344 INFO:tasks.workunit.client.0.smithi096.stdout:9/671: mkdir d1/df/d95/d9d/db5/d43/d57/d5b/d73/de3 0 2024-04-22T15:19:00.346 INFO:tasks.workunit.client.1.smithi196.stdout:3/493: rename d0/d15/d4d/db/dc to d0/d15/da7 0 2024-04-22T15:19:00.347 INFO:tasks.workunit.client.0.smithi096.stdout:0/896: dread d7/f20 [0,4194304] 0 2024-04-22T15:19:00.347 INFO:tasks.workunit.client.0.smithi096.stdout:1/803: write d6/dc/d8a/d3f/d54/d32/d3b/d3d/f63 [1462359,9799] 0 2024-04-22T15:19:00.348 INFO:tasks.workunit.client.0.smithi096.stdout:9/672: chown d1/df/d95/d9d/db5/d43/d57/d5b/d73/f9e 321380677 1 2024-04-22T15:19:00.360 INFO:tasks.workunit.client.1.smithi196.stdout:9/503: sync 2024-04-22T15:19:00.369 INFO:tasks.workunit.client.0.smithi096.stdout:3/684: rename da/db/dc/d28/d69/c8c to da/db/d12/d22/d26/db2/cdb 0 2024-04-22T15:19:00.374 INFO:tasks.workunit.client.0.smithi096.stdout:3/685: dread da/db/d12/fa9 [0,4194304] 0 2024-04-22T15:19:00.374 INFO:tasks.workunit.client.0.smithi096.stdout:3/686: readlink da/l1c 0 2024-04-22T15:19:00.375 INFO:tasks.workunit.client.0.smithi096.stdout:9/673: creat d1/df/d95/fe4 x:0 0 0 2024-04-22T15:19:00.379 INFO:tasks.workunit.client.1.smithi196.stdout:0/515: mkdir d3/d6/dc/d8c/d8d/dab 0 2024-04-22T15:19:00.386 INFO:tasks.workunit.client.1.smithi196.stdout:7/465: rename d14/d26/d28/d29/c5d to d14/d26/d2a/c92 0 2024-04-22T15:19:00.387 INFO:tasks.workunit.client.1.smithi196.stdout:7/466: write d14/d25/d31/f69 [843743,26724] 0 2024-04-22T15:19:00.388 INFO:tasks.workunit.client.1.smithi196.stdout:7/467: truncate f7 4961759 0 2024-04-22T15:19:00.391 INFO:tasks.workunit.client.1.smithi196.stdout:3/494: creat d0/d15/da7/fa8 x:0 0 0 2024-04-22T15:19:00.392 INFO:tasks.workunit.client.1.smithi196.stdout:3/495: write d0/d15/d48/d51/d5e/f63 [103767,90402] 0 2024-04-22T15:19:00.398 INFO:tasks.workunit.client.1.smithi196.stdout:9/504: truncate db/d11/d29/d7a/f7d 1012923 0 2024-04-22T15:19:00.400 INFO:tasks.workunit.client.1.smithi196.stdout:1/464: link d1/db/ff d1/db/d16/d44/f92 0 2024-04-22T15:19:00.400 INFO:tasks.workunit.client.1.smithi196.stdout:4/486: write d6/d14/d18/d40/f65 [2316721,4995] 0 2024-04-22T15:19:00.402 INFO:tasks.workunit.client.1.smithi196.stdout:5/503: dread d1/d17/f8b [0,4194304] 0 2024-04-22T15:19:00.402 INFO:tasks.workunit.client.1.smithi196.stdout:1/465: read - d1/db/d16/d1e/f88 zero size 2024-04-22T15:19:00.403 INFO:tasks.workunit.client.0.smithi096.stdout:6/631: rmdir d3/d28/dd/d11/d2c/d3b/dac 0 2024-04-22T15:19:00.404 INFO:tasks.workunit.client.1.smithi196.stdout:5/504: write d1/d17/f79 [289593,69855] 0 2024-04-22T15:19:00.404 INFO:tasks.workunit.client.1.smithi196.stdout:9/505: dwrite db/d11/d29/f92 [0,4194304] 0 2024-04-22T15:19:00.413 INFO:tasks.workunit.client.1.smithi196.stdout:0/516: symlink d3/d9/d36/d56/d65/d7f/d30/lac 0 2024-04-22T15:19:00.414 INFO:tasks.workunit.client.1.smithi196.stdout:0/517: truncate d3/d9/d36/d56/d65/d7f/d51/f6a 110593 0 2024-04-22T15:19:00.415 INFO:tasks.workunit.client.1.smithi196.stdout:0/518: write d3/d6/f48 [938466,97685] 0 2024-04-22T15:19:00.416 INFO:tasks.workunit.client.0.smithi096.stdout:4/769: link d5/d7/d2c/d3d/d4f/d63/l74 d5/l109 0 2024-04-22T15:19:00.417 INFO:tasks.workunit.client.1.smithi196.stdout:6/511: symlink dd/l90 0 2024-04-22T15:19:00.422 INFO:tasks.workunit.client.1.smithi196.stdout:2/536: rename d1/d1d/d6a to d1/d23/d84/dc8 0 2024-04-22T15:19:00.426 INFO:tasks.workunit.client.1.smithi196.stdout:5/505: dread d1/d17/f21 [0,4194304] 0 2024-04-22T15:19:00.426 INFO:tasks.workunit.client.1.smithi196.stdout:5/506: stat d1/f4c 0 2024-04-22T15:19:00.427 INFO:tasks.workunit.client.1.smithi196.stdout:5/507: read - d1/d17/d1a/d2f/d30/f61 zero size 2024-04-22T15:19:00.428 INFO:tasks.workunit.client.0.smithi096.stdout:9/674: fdatasync d1/df/f58 0 2024-04-22T15:19:00.428 INFO:tasks.workunit.client.1.smithi196.stdout:5/508: write d1/d17/d1c/f88 [983715,98264] 0 2024-04-22T15:19:00.433 INFO:tasks.workunit.client.1.smithi196.stdout:7/468: creat d14/d26/d28/f93 x:0 0 0 2024-04-22T15:19:00.433 INFO:tasks.workunit.client.1.smithi196.stdout:5/509: dwrite d1/d17/d1c/f3b [0,4194304] 0 2024-04-22T15:19:00.434 INFO:tasks.workunit.client.1.smithi196.stdout:7/469: truncate d14/d25/f90 1021378 0 2024-04-22T15:19:00.443 INFO:tasks.workunit.client.0.smithi096.stdout:0/897: link d7/de/d5b/d95/dc/f29 d7/f131 0 2024-04-22T15:19:00.447 INFO:tasks.workunit.client.1.smithi196.stdout:4/487: read d6/d13/d15/d2c/d62/f5a [13147,104530] 0 2024-04-22T15:19:00.452 INFO:tasks.workunit.client.0.smithi096.stdout:9/675: unlink d1/d25/c5a 0 2024-04-22T15:19:00.460 INFO:tasks.workunit.client.1.smithi196.stdout:0/519: mkdir d3/d6/dc/d8c/d8d/dad 0 2024-04-22T15:19:00.460 INFO:tasks.workunit.client.1.smithi196.stdout:0/520: chown d3/de 48256 1 2024-04-22T15:19:00.461 INFO:tasks.workunit.client.1.smithi196.stdout:0/521: stat d3/d9/l3b 0 2024-04-22T15:19:00.467 INFO:tasks.workunit.client.1.smithi196.stdout:3/496: rename d0/d15/d4d/fa3 to d0/d15/d48/d51/fa9 0 2024-04-22T15:19:00.469 INFO:tasks.workunit.client.1.smithi196.stdout:2/537: mknod d1/d23/d27/d89/cc9 0 2024-04-22T15:19:00.470 INFO:tasks.workunit.client.0.smithi096.stdout:8/772: dwrite d4/d10/f97 [4194304,4194304] 0 2024-04-22T15:19:00.475 INFO:tasks.workunit.client.0.smithi096.stdout:9/676: truncate d1/df/d95/d9d/db5/d43/d57/d5b/fa4 1722318 0 2024-04-22T15:19:00.475 INFO:tasks.workunit.client.1.smithi196.stdout:2/538: dwrite d1/d13/d1f/f9d [0,4194304] 0 2024-04-22T15:19:00.477 INFO:tasks.workunit.client.0.smithi096.stdout:2/793: write d3/d1f/d2f/f9e [229725,43360] 0 2024-04-22T15:19:00.479 INFO:tasks.workunit.client.1.smithi196.stdout:5/510: chown d1/d9/l23 131128064 1 2024-04-22T15:19:00.480 INFO:tasks.workunit.client.0.smithi096.stdout:6/632: rename d3/d28/dd/c46 to d3/d28/dd/d11/cda 0 2024-04-22T15:19:00.482 INFO:tasks.workunit.client.1.smithi196.stdout:7/470: creat d14/d15/f94 x:0 0 0 2024-04-22T15:19:00.487 INFO:tasks.workunit.client.1.smithi196.stdout:8/508: getdents d2/d3f 0 2024-04-22T15:19:00.489 INFO:tasks.workunit.client.1.smithi196.stdout:3/497: dread d0/d15/da7/d4a/d44/f26 [4194304,4194304] 0 2024-04-22T15:19:00.493 INFO:tasks.workunit.client.0.smithi096.stdout:9/677: rmdir d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae 39 2024-04-22T15:19:00.494 INFO:tasks.workunit.client.1.smithi196.stdout:4/488: dread - d6/d13/f24 zero size 2024-04-22T15:19:00.494 INFO:tasks.workunit.client.0.smithi096.stdout:3/687: write da/db/de/d19/f30 [2607840,39634] 0 2024-04-22T15:19:00.501 INFO:tasks.workunit.client.0.smithi096.stdout:1/804: write d6/f60 [670836,92484] 0 2024-04-22T15:19:00.503 INFO:tasks.workunit.client.0.smithi096.stdout:8/773: fdatasync d4/f32 0 2024-04-22T15:19:00.503 INFO:tasks.workunit.client.1.smithi196.stdout:7/471: dread fe [0,4194304] 0 2024-04-22T15:19:00.504 INFO:tasks.workunit.client.0.smithi096.stdout:1/805: chown d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/leb 5762 1 2024-04-22T15:19:00.506 INFO:tasks.workunit.client.1.smithi196.stdout:9/506: link db/fc db/dd/de/d15/d31/d64/fad 0 2024-04-22T15:19:00.508 INFO:tasks.workunit.client.1.smithi196.stdout:7/472: dwrite d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:19:00.509 INFO:tasks.workunit.client.0.smithi096.stdout:9/678: mkdir d1/df/d95/d9d/db5/dd/dba/dc2/de5 0 2024-04-22T15:19:00.520 INFO:tasks.workunit.client.0.smithi096.stdout:4/770: write d5/d7/d2c/d3d/d4f/d88/ffc [1053408,75930] 0 2024-04-22T15:19:00.522 INFO:tasks.workunit.client.1.smithi196.stdout:6/512: write dd/f18 [1832006,94429] 0 2024-04-22T15:19:00.528 INFO:tasks.workunit.client.0.smithi096.stdout:2/794: creat d3/db/df/dce/d46/fff x:0 0 0 2024-04-22T15:19:00.529 INFO:tasks.workunit.client.0.smithi096.stdout:2/795: stat d3/db/df/dce/d46/d7e/d84 0 2024-04-22T15:19:00.530 INFO:tasks.workunit.client.1.smithi196.stdout:5/511: mknod d1/d9/c9e 0 2024-04-22T15:19:00.531 INFO:tasks.workunit.client.0.smithi096.stdout:9/679: symlink d1/df/d95/d9d/db5/d43/d57/d5b/d73/le6 0 2024-04-22T15:19:00.535 INFO:tasks.workunit.client.1.smithi196.stdout:5/512: dwrite d1/d17/f5d [0,4194304] 0 2024-04-22T15:19:00.538 INFO:tasks.workunit.client.1.smithi196.stdout:3/498: write d0/d15/da7/d4a/f50 [130565,3387] 0 2024-04-22T15:19:00.539 INFO:tasks.workunit.client.0.smithi096.stdout:8/774: truncate d4/d1c/d40/da7/db6/f3e 5364313 0 2024-04-22T15:19:00.540 INFO:tasks.workunit.client.1.smithi196.stdout:3/499: write d0/d15/d31/f53 [111668,9973] 0 2024-04-22T15:19:00.541 INFO:tasks.workunit.client.1.smithi196.stdout:3/500: chown d0/d15/da7/d4a 411 1 2024-04-22T15:19:00.541 INFO:tasks.workunit.client.1.smithi196.stdout:1/466: dwrite d1/db/d16/d44/f92 [4194304,4194304] 0 2024-04-22T15:19:00.546 INFO:tasks.workunit.client.0.smithi096.stdout:6/633: dwrite d3/d28/d4d/d82/fd1 [4194304,4194304] 0 2024-04-22T15:19:00.546 INFO:tasks.workunit.client.1.smithi196.stdout:1/467: write d1/db/dc/de/d43/f7b [1165015,70718] 0 2024-04-22T15:19:00.548 INFO:tasks.workunit.client.1.smithi196.stdout:4/489: rename d6/d13/d15/f7c to d6/d14/d18/d26/d32/d53/f94 0 2024-04-22T15:19:00.553 INFO:tasks.workunit.client.0.smithi096.stdout:2/796: creat d3/db/df/d24/d58/d5f/d6e/f100 x:0 0 0 2024-04-22T15:19:00.554 INFO:tasks.workunit.client.0.smithi096.stdout:2/797: chown d3/d13/c3d 695041 1 2024-04-22T15:19:00.557 INFO:tasks.workunit.client.1.smithi196.stdout:0/522: mkdir d3/d6/dae 0 2024-04-22T15:19:00.560 INFO:tasks.workunit.client.1.smithi196.stdout:2/539: write d1/d23/d84/dc8/da1/f79 [667213,75328] 0 2024-04-22T15:19:00.561 INFO:tasks.workunit.client.1.smithi196.stdout:2/540: fdatasync d1/d23/d27/d2e/d55/d5b/d93/fbf 0 2024-04-22T15:19:00.563 INFO:tasks.workunit.client.0.smithi096.stdout:3/688: rename da/db/d12/d22 to da/db/de/ddc 0 2024-04-22T15:19:00.568 INFO:tasks.workunit.client.0.smithi096.stdout:4/771: symlink d5/d7/dcb/l10a 0 2024-04-22T15:19:00.570 INFO:tasks.workunit.client.1.smithi196.stdout:6/513: creat dd/d20/d4d/f91 x:0 0 0 2024-04-22T15:19:00.572 INFO:tasks.workunit.client.1.smithi196.stdout:8/509: dwrite d2/d5/d4f/f6b [0,4194304] 0 2024-04-22T15:19:00.576 INFO:tasks.workunit.client.0.smithi096.stdout:1/806: write d6/f4f [86914,118349] 0 2024-04-22T15:19:00.577 INFO:tasks.workunit.client.1.smithi196.stdout:4/490: dread d6/d14/d18/d26/f6a [0,4194304] 0 2024-04-22T15:19:00.578 INFO:tasks.workunit.client.0.smithi096.stdout:1/807: chown d6/dc/d8a/d3f/d54/f21 6582 1 2024-04-22T15:19:00.579 INFO:tasks.workunit.client.0.smithi096.stdout:1/808: chown d6/dc/l5d 33278 1 2024-04-22T15:19:00.582 INFO:tasks.workunit.client.0.smithi096.stdout:8/775: dread d4/d1c/f72 [0,4194304] 0 2024-04-22T15:19:00.582 INFO:tasks.workunit.client.0.smithi096.stdout:2/798: creat d3/db/df/d24/f101 x:0 0 0 2024-04-22T15:19:00.587 INFO:tasks.workunit.client.1.smithi196.stdout:3/501: creat d0/d15/da7/d4a/d44/d8c/faa x:0 0 0 2024-04-22T15:19:00.590 INFO:tasks.workunit.client.0.smithi096.stdout:2/799: dwrite d3/db/df/d98/db3/fbb [0,4194304] 0 2024-04-22T15:19:00.597 INFO:tasks.workunit.client.0.smithi096.stdout:3/689: write da/f55 [5048716,8313] 0 2024-04-22T15:19:00.598 INFO:tasks.workunit.client.0.smithi096.stdout:9/680: write d1/d25/f7d [979679,128315] 0 2024-04-22T15:19:00.600 INFO:tasks.workunit.client.1.smithi196.stdout:5/513: rename d1/d17/d1a/d2f/d30/d6e/l81 to d1/d17/d50/l9f 0 2024-04-22T15:19:00.601 INFO:tasks.workunit.client.0.smithi096.stdout:1/809: creat d6/dc/d8a/d3f/d54/d32/d3b/d56/ffc x:0 0 0 2024-04-22T15:19:00.603 INFO:tasks.workunit.client.1.smithi196.stdout:4/491: dread d6/d13/f25 [4194304,4194304] 0 2024-04-22T15:19:00.604 INFO:tasks.workunit.client.1.smithi196.stdout:2/541: mknod d1/d23/d27/d73/cca 0 2024-04-22T15:19:00.605 INFO:tasks.workunit.client.0.smithi096.stdout:2/800: creat d3/d13/dca/f102 x:0 0 0 2024-04-22T15:19:00.607 INFO:tasks.workunit.client.0.smithi096.stdout:2/801: chown d3/d13/db4/dbf/fea 92 1 2024-04-22T15:19:00.610 INFO:tasks.workunit.client.0.smithi096.stdout:1/810: symlink d6/dc/d8a/d3f/d62/d7d/lfd 0 2024-04-22T15:19:00.613 INFO:tasks.workunit.client.1.smithi196.stdout:8/510: dwrite d2/d35/f61 [0,4194304] 0 2024-04-22T15:19:00.613 INFO:tasks.workunit.client.1.smithi196.stdout:8/511: readlink d2/l30 0 2024-04-22T15:19:00.614 INFO:tasks.workunit.client.0.smithi096.stdout:4/772: dread d5/d7/d2c/f3a [0,4194304] 0 2024-04-22T15:19:00.619 INFO:tasks.workunit.client.1.smithi196.stdout:0/523: rename d3/de/d14/d72 to d3/d6/dc/d8c/daf 0 2024-04-22T15:19:00.619 INFO:tasks.workunit.client.1.smithi196.stdout:0/524: rename d3/d9/d36 to d3/d9/d36/d56/d65/d7f/d51/db0 22 2024-04-22T15:19:00.620 INFO:tasks.workunit.client.0.smithi096.stdout:3/690: write da/db/dc/d28/fad [986465,3444] 0 2024-04-22T15:19:00.622 INFO:tasks.workunit.client.0.smithi096.stdout:2/802: truncate d3/db/df/d24/d58/d5e/f6b 1359136 0 2024-04-22T15:19:00.627 INFO:tasks.workunit.client.0.smithi096.stdout:9/681: read d1/df/d95/d9d/db5/d43/fa3 [322357,93403] 0 2024-04-22T15:19:00.629 INFO:tasks.workunit.client.0.smithi096.stdout:2/803: mkdir d3/db/df/d24/dc8/d103 0 2024-04-22T15:19:00.630 INFO:tasks.workunit.client.1.smithi196.stdout:2/542: symlink d1/d23/d27/d2e/d55/lcb 0 2024-04-22T15:19:00.632 INFO:tasks.workunit.client.0.smithi096.stdout:1/811: mkdir d6/dc/d8a/d3f/dfe 0 2024-04-22T15:19:00.638 INFO:tasks.workunit.client.1.smithi196.stdout:1/468: rmdir d1/db/dc/de/d43/d61 0 2024-04-22T15:19:00.638 INFO:tasks.workunit.client.1.smithi196.stdout:5/514: dwrite d1/d9/f38 [0,4194304] 0 2024-04-22T15:19:00.639 INFO:tasks.workunit.client.1.smithi196.stdout:5/515: dread - d1/d4/f9d zero size 2024-04-22T15:19:00.640 INFO:tasks.workunit.client.1.smithi196.stdout:8/512: mknod d2/d8/d47/d68/c91 0 2024-04-22T15:19:00.641 INFO:tasks.workunit.client.1.smithi196.stdout:5/516: chown d1/d17/d1c/l27 17702108 1 2024-04-22T15:19:00.642 INFO:tasks.workunit.client.0.smithi096.stdout:9/682: chown d1/df/d95/d9d/db5/d43/d57/d5b/l68 1 1 2024-04-22T15:19:00.644 INFO:tasks.workunit.client.1.smithi196.stdout:0/525: write d3/f47 [5088182,112259] 0 2024-04-22T15:19:00.645 INFO:tasks.workunit.client.1.smithi196.stdout:4/492: dwrite d6/d13/d15/f3d [0,4194304] 0 2024-04-22T15:19:00.649 INFO:tasks.workunit.client.0.smithi096.stdout:2/804: creat d3/d13/db4/f104 x:0 0 0 2024-04-22T15:19:00.649 INFO:tasks.workunit.client.0.smithi096.stdout:2/805: chown d3/db/d30/l97 6 1 2024-04-22T15:19:00.650 INFO:tasks.workunit.client.1.smithi196.stdout:7/473: getdents d14/d25/d31 0 2024-04-22T15:19:00.652 INFO:tasks.workunit.client.0.smithi096.stdout:9/683: mknod d1/df/d95/d9d/db5/dd/dba/dc2/ce7 0 2024-04-22T15:19:00.653 INFO:tasks.workunit.client.0.smithi096.stdout:9/684: chown d1/d25/f7d 856536832 1 2024-04-22T15:19:00.654 INFO:tasks.workunit.client.1.smithi196.stdout:6/514: creat dd/d1f/d38/f92 x:0 0 0 2024-04-22T15:19:00.655 INFO:tasks.workunit.client.0.smithi096.stdout:9/685: write d1/df/d95/d9d/db5/dd/fe1 [275360,103585] 0 2024-04-22T15:19:00.657 INFO:tasks.workunit.client.1.smithi196.stdout:1/469: fdatasync d1/db/dc/de/f4a 0 2024-04-22T15:19:00.659 INFO:tasks.workunit.client.1.smithi196.stdout:6/515: dwrite dd/d1f/f48 [0,4194304] 0 2024-04-22T15:19:00.661 INFO:tasks.workunit.client.1.smithi196.stdout:8/513: creat d2/d35/f92 x:0 0 0 2024-04-22T15:19:00.661 INFO:tasks.workunit.client.0.smithi096.stdout:1/812: link d6/dc/d8a/d3f/d54/d32/d3b/f9e d6/dc/d8a/d3f/d62/dc2/dd0/fff 0 2024-04-22T15:19:00.661 INFO:tasks.workunit.client.0.smithi096.stdout:4/773: dwrite d5/d7/f8e [0,4194304] 0 2024-04-22T15:19:00.665 INFO:tasks.workunit.client.1.smithi196.stdout:9/507: sync 2024-04-22T15:19:00.666 INFO:tasks.workunit.client.0.smithi096.stdout:9/686: truncate d1/d4/d7b/f81 1895040 0 2024-04-22T15:19:00.668 INFO:tasks.workunit.client.0.smithi096.stdout:1/813: truncate d6/dc/d8a/d3f/d54/d32/d82/fe5 252046 0 2024-04-22T15:19:00.670 INFO:tasks.workunit.client.1.smithi196.stdout:2/543: mknod d1/d23/d27/ccc 0 2024-04-22T15:19:00.670 INFO:tasks.workunit.client.0.smithi096.stdout:9/687: fdatasync d1/df/d95/d9d/db5/dd/dba/dc2/dc4/fd7 0 2024-04-22T15:19:00.671 INFO:tasks.workunit.client.1.smithi196.stdout:8/514: write d2/d8/d47/d68/f7c [313130,125478] 0 2024-04-22T15:19:00.671 INFO:tasks.workunit.client.1.smithi196.stdout:9/508: truncate db/dd/f8d 983122 0 2024-04-22T15:19:00.673 INFO:tasks.workunit.client.1.smithi196.stdout:6/516: truncate dd/f23 290657 0 2024-04-22T15:19:00.678 INFO:tasks.workunit.client.1.smithi196.stdout:2/544: dwrite d1/f5 [0,4194304] 0 2024-04-22T15:19:00.678 INFO:tasks.workunit.client.1.smithi196.stdout:8/515: dread d2/d5/d4f/f6b [0,4194304] 0 2024-04-22T15:19:00.678 INFO:tasks.workunit.client.0.smithi096.stdout:1/814: mkdir d6/dc/d8a/d29/d100 0 2024-04-22T15:19:00.679 INFO:tasks.workunit.client.0.smithi096.stdout:1/815: chown d6/dc/d8a/l99 58381303 1 2024-04-22T15:19:00.681 INFO:tasks.workunit.client.0.smithi096.stdout:2/806: dread d3/db/df/dce/d46/d69/fe1 [0,4194304] 0 2024-04-22T15:19:00.684 INFO:tasks.workunit.client.1.smithi196.stdout:4/493: mknod d6/d13/d15/d2c/d8c/c95 0 2024-04-22T15:19:00.686 INFO:tasks.workunit.client.1.smithi196.stdout:1/470: fsync d1/db/dc/de/f14 0 2024-04-22T15:19:00.686 INFO:tasks.workunit.client.1.smithi196.stdout:5/517: dwrite d1/d17/f21 [4194304,4194304] 0 2024-04-22T15:19:00.718 INFO:tasks.workunit.client.0.smithi096.stdout:2/807: dwrite d3/d13/db4/dbf/fea [0,4194304] 0 2024-04-22T15:19:00.719 INFO:tasks.workunit.client.0.smithi096.stdout:2/808: chown d3/d1f/f7a 385261036 1 2024-04-22T15:19:00.721 INFO:tasks.workunit.client.1.smithi196.stdout:3/502: sync 2024-04-22T15:19:00.724 INFO:tasks.workunit.client.1.smithi196.stdout:9/509: symlink db/dd/de/d15/d31/d64/lae 0 2024-04-22T15:19:00.730 INFO:tasks.workunit.client.1.smithi196.stdout:0/526: creat d3/d9/fb1 x:0 0 0 2024-04-22T15:19:00.731 INFO:tasks.workunit.client.0.smithi096.stdout:4/774: link d5/ld d5/d7/d1a/d25/l10b 0 2024-04-22T15:19:00.732 INFO:tasks.workunit.client.0.smithi096.stdout:4/775: write d5/d7/d2c/d3d/d4f/d88/ffc [1265032,5365] 0 2024-04-22T15:19:00.734 INFO:tasks.workunit.client.0.smithi096.stdout:1/816: creat d6/dc/d8a/d3f/f101 x:0 0 0 2024-04-22T15:19:00.734 INFO:tasks.workunit.client.1.smithi196.stdout:4/494: creat d6/d13/d15/d2c/d62/f96 x:0 0 0 2024-04-22T15:19:00.736 INFO:tasks.workunit.client.1.smithi196.stdout:5/518: mkdir d1/d51/da0 0 2024-04-22T15:19:00.737 INFO:tasks.workunit.client.0.smithi096.stdout:4/776: fdatasync d5/d7/d2c/d95/dee/f59 0 2024-04-22T15:19:00.739 INFO:tasks.workunit.client.0.smithi096.stdout:1/817: creat d6/dc/d8a/d3f/d54/d32/dcd/f102 x:0 0 0 2024-04-22T15:19:00.741 INFO:tasks.workunit.client.1.smithi196.stdout:5/519: dwrite d1/d17/d1a/f43 [0,4194304] 0 2024-04-22T15:19:00.744 INFO:tasks.workunit.client.1.smithi196.stdout:9/510: unlink db/dd/d1b/d2c/fa3 0 2024-04-22T15:19:00.746 INFO:tasks.workunit.client.1.smithi196.stdout:9/511: chown db/d11/d29/d7a/d7b/d81/c8a 25 1 2024-04-22T15:19:00.746 INFO:tasks.workunit.client.1.smithi196.stdout:6/517: creat dd/d2b/d8d/f93 x:0 0 0 2024-04-22T15:19:00.754 INFO:tasks.workunit.client.0.smithi096.stdout:1/818: symlink d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/l103 0 2024-04-22T15:19:00.756 INFO:tasks.workunit.client.1.smithi196.stdout:0/527: rename d3/l4b to d3/de/d14/d86/lb2 0 2024-04-22T15:19:00.757 INFO:tasks.workunit.client.1.smithi196.stdout:8/516: dread d2/d3f/f4e [0,4194304] 0 2024-04-22T15:19:00.758 INFO:tasks.workunit.client.0.smithi096.stdout:1/819: truncate d6/dc/d8a/d3f/d54/d32/d82/fe5 489507 0 2024-04-22T15:19:00.760 INFO:tasks.workunit.client.1.smithi196.stdout:7/474: sync 2024-04-22T15:19:00.760 INFO:tasks.workunit.client.1.smithi196.stdout:4/495: creat d6/d13/d15/d2c/d62/f97 x:0 0 0 2024-04-22T15:19:00.761 INFO:tasks.workunit.client.0.smithi096.stdout:9/688: dwrite d1/df/d95/d9d/db5/dd/f1a [4194304,4194304] 0 2024-04-22T15:19:00.765 INFO:tasks.workunit.client.1.smithi196.stdout:7/475: dwrite d14/d26/d2a/f66 [4194304,4194304] 0 2024-04-22T15:19:00.766 INFO:tasks.workunit.client.1.smithi196.stdout:1/471: mkdir d1/db/dc/d93 0 2024-04-22T15:19:00.775 INFO:tasks.workunit.client.1.smithi196.stdout:5/520: dwrite d1/d53/f86 [0,4194304] 0 2024-04-22T15:19:00.776 INFO:tasks.workunit.client.1.smithi196.stdout:2/545: dwrite d1/d13/d1f/f20 [4194304,4194304] 0 2024-04-22T15:19:00.777 INFO:tasks.workunit.client.1.smithi196.stdout:9/512: mknod db/d11/d29/d7a/d7b/caf 0 2024-04-22T15:19:00.778 INFO:tasks.workunit.client.0.smithi096.stdout:9/689: dwrite d1/d25/db2/fb8 [0,4194304] 0 2024-04-22T15:19:00.785 INFO:tasks.workunit.client.0.smithi096.stdout:1/820: mkdir d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/d104 0 2024-04-22T15:19:00.787 INFO:tasks.workunit.client.1.smithi196.stdout:6/518: stat dd/d1f/d38/f4b 0 2024-04-22T15:19:00.789 INFO:tasks.workunit.client.0.smithi096.stdout:9/690: unlink d1/d4/d7b/c8f 0 2024-04-22T15:19:00.795 INFO:tasks.workunit.client.1.smithi196.stdout:3/503: dwrite d0/d15/da7/d4a/d44/f26 [4194304,4194304] 0 2024-04-22T15:19:00.802 INFO:tasks.workunit.client.1.smithi196.stdout:4/496: dread d6/d13/f1d [0,4194304] 0 2024-04-22T15:19:00.803 INFO:tasks.workunit.client.0.smithi096.stdout:4/777: truncate d5/d7/f7f 5597695 0 2024-04-22T15:19:00.813 INFO:tasks.workunit.client.1.smithi196.stdout:1/472: mkdir d1/db/dc/de/d5a/d94 0 2024-04-22T15:19:00.818 INFO:tasks.workunit.client.1.smithi196.stdout:5/521: mknod d1/d17/d1a/d2f/d30/d5e/ca1 0 2024-04-22T15:19:00.819 INFO:tasks.workunit.client.0.smithi096.stdout:1/821: link d6/f8f d6/dc/d8a/d29/f105 0 2024-04-22T15:19:00.820 INFO:tasks.workunit.client.0.smithi096.stdout:1/822: stat d6/dc/d8a/d3f/d54/c27 0 2024-04-22T15:19:00.821 INFO:tasks.workunit.client.1.smithi196.stdout:2/546: rmdir d1/d76 39 2024-04-22T15:19:00.821 INFO:tasks.workunit.client.1.smithi196.stdout:2/547: stat d1/d13/d1f/f9d 0 2024-04-22T15:19:00.823 INFO:tasks.workunit.client.0.smithi096.stdout:4/778: mknod d5/d7/c10c 0 2024-04-22T15:19:00.826 INFO:tasks.workunit.client.1.smithi196.stdout:9/513: rmdir db/dd/d1b/d2c 39 2024-04-22T15:19:00.828 INFO:tasks.workunit.client.0.smithi096.stdout:9/691: link d1/l80 d1/df/d95/d9d/db5/d43/le8 0 2024-04-22T15:19:00.831 INFO:tasks.workunit.client.0.smithi096.stdout:1/823: dread - d6/dc/d8a/d3f/d54/d32/fa3 zero size 2024-04-22T15:19:00.832 INFO:tasks.workunit.client.1.smithi196.stdout:0/528: mknod d3/d6/dae/cb3 0 2024-04-22T15:19:00.838 INFO:tasks.workunit.client.0.smithi096.stdout:9/692: creat d1/df/d95/d9d/db5/dd/dba/dc2/dc4/fe9 x:0 0 0 2024-04-22T15:19:00.840 INFO:tasks.workunit.client.1.smithi196.stdout:8/517: unlink d2/dc/f2f 0 2024-04-22T15:19:00.842 INFO:tasks.workunit.client.1.smithi196.stdout:8/518: truncate d2/d5/d1a/f37 1448497 0 2024-04-22T15:19:00.845 INFO:tasks.workunit.client.1.smithi196.stdout:7/476: mkdir d14/d26/d95 0 2024-04-22T15:19:00.847 INFO:tasks.workunit.client.1.smithi196.stdout:1/473: mknod d1/db/dc/de/d5a/c95 0 2024-04-22T15:19:00.847 INFO:tasks.workunit.client.0.smithi096.stdout:1/824: dread d6/dc/d8a/d3f/d54/d32/d3b/d3d/f63 [4194304,4194304] 0 2024-04-22T15:19:00.848 INFO:tasks.workunit.client.1.smithi196.stdout:1/474: fdatasync d1/db/dc/d26/d2e/f34 0 2024-04-22T15:19:00.849 INFO:tasks.workunit.client.1.smithi196.stdout:5/522: creat d1/d17/d1a/d2f/d30/d6e/fa2 x:0 0 0 2024-04-22T15:19:00.850 INFO:tasks.workunit.client.0.smithi096.stdout:1/825: chown d6/dc/d8a/d3f/d54/d32/dcd/f102 2480 1 2024-04-22T15:19:00.852 INFO:tasks.workunit.client.0.smithi096.stdout:4/779: rename d5/d7/d2c/d3d/ff7 to d5/d7/d1a/d103/f10d 0 2024-04-22T15:19:00.853 INFO:tasks.workunit.client.0.smithi096.stdout:4/780: dread - d5/fe9 zero size 2024-04-22T15:19:00.854 INFO:tasks.workunit.client.0.smithi096.stdout:4/781: readlink d5/d7/d2c/lda 0 2024-04-22T15:19:00.854 INFO:tasks.workunit.client.1.smithi196.stdout:9/514: creat db/d11/d29/d57/fb0 x:0 0 0 2024-04-22T15:19:00.858 INFO:tasks.workunit.client.1.smithi196.stdout:0/529: mkdir d3/db4 0 2024-04-22T15:19:00.861 INFO:tasks.workunit.client.1.smithi196.stdout:3/504: fsync d0/d15/d48/d51/f8a 0 2024-04-22T15:19:00.862 INFO:tasks.workunit.client.1.smithi196.stdout:3/505: readlink d0/d15/d48/l7c 0 2024-04-22T15:19:00.862 INFO:tasks.workunit.client.0.smithi096.stdout:9/693: write d1/d4/d79/f86 [839775,35454] 0 2024-04-22T15:19:00.864 INFO:tasks.workunit.client.1.smithi196.stdout:7/477: creat d14/d15/f96 x:0 0 0 2024-04-22T15:19:00.865 INFO:tasks.workunit.client.1.smithi196.stdout:7/478: read d14/d25/d31/f38 [3255320,65575] 0 2024-04-22T15:19:00.866 INFO:tasks.workunit.client.1.smithi196.stdout:7/479: write d14/d4d/f7e [11972,64712] 0 2024-04-22T15:19:00.866 INFO:tasks.workunit.client.0.smithi096.stdout:4/782: unlink d5/d7/d1f/d84/lbc 0 2024-04-22T15:19:00.867 INFO:tasks.workunit.client.1.smithi196.stdout:7/480: truncate d14/d26/d5a/d5f/d6e/f8d 652198 0 2024-04-22T15:19:00.868 INFO:tasks.workunit.client.1.smithi196.stdout:1/475: mkdir d1/db/dc/de/d96 0 2024-04-22T15:19:00.871 INFO:tasks.workunit.client.1.smithi196.stdout:5/523: creat d1/d17/d1a/d2f/d30/d6e/d7a/fa3 x:0 0 0 2024-04-22T15:19:00.872 INFO:tasks.workunit.client.0.smithi096.stdout:4/783: symlink d5/d7/d1f/d84/d6d/l10e 0 2024-04-22T15:19:00.873 INFO:tasks.workunit.client.0.smithi096.stdout:9/694: mknod d1/df/d95/d9d/db5/dd/dba/dc2/de5/cea 0 2024-04-22T15:19:00.875 INFO:tasks.workunit.client.1.smithi196.stdout:6/519: creat dd/d1f/d38/f94 x:0 0 0 2024-04-22T15:19:00.875 INFO:tasks.workunit.client.1.smithi196.stdout:6/520: fdatasync dd/d1f/d38/f94 0 2024-04-22T15:19:00.876 INFO:tasks.workunit.client.1.smithi196.stdout:6/521: write dd/d1f/d37/f74 [938453,18567] 0 2024-04-22T15:19:00.877 INFO:tasks.workunit.client.1.smithi196.stdout:6/522: write dd/d1f/d38/d40/f49 [614372,120972] 0 2024-04-22T15:19:00.877 INFO:tasks.workunit.client.1.smithi196.stdout:9/515: rename db/dd/de/l16 to db/d11/d29/d7a/d7b/d81/lb1 0 2024-04-22T15:19:00.878 INFO:tasks.workunit.client.1.smithi196.stdout:6/523: stat dd/d1f/l35 0 2024-04-22T15:19:00.879 INFO:tasks.workunit.client.1.smithi196.stdout:0/530: fsync d3/d9/d36/d56/d65/d7f/d51/f75 0 2024-04-22T15:19:00.879 INFO:tasks.workunit.client.1.smithi196.stdout:0/531: dread - d3/f99 zero size 2024-04-22T15:19:00.880 INFO:tasks.workunit.client.1.smithi196.stdout:0/532: fdatasync d3/d9/d36/d56/d65/d7f/d87/d6c/d88/fa3 0 2024-04-22T15:19:00.885 INFO:tasks.workunit.client.0.smithi096.stdout:9/695: creat d1/d25/db2/d98/d9b/feb x:0 0 0 2024-04-22T15:19:00.886 INFO:tasks.workunit.client.0.smithi096.stdout:9/696: truncate d1/d4/d7b/fdf 443641 0 2024-04-22T15:19:00.888 INFO:tasks.workunit.client.0.smithi096.stdout:4/784: fsync d5/d7/d2c/d3d/f75 0 2024-04-22T15:19:00.891 INFO:tasks.workunit.client.1.smithi196.stdout:8/519: mkdir d2/d5/d93 0 2024-04-22T15:19:00.894 INFO:tasks.workunit.client.1.smithi196.stdout:4/497: creat d6/d14/f98 x:0 0 0 2024-04-22T15:19:00.896 INFO:tasks.workunit.client.0.smithi096.stdout:9/697: rename d1/df/d95/d9d/db5/dd/f46 to d1/d4/d7b/fec 0 2024-04-22T15:19:00.897 INFO:tasks.workunit.client.1.smithi196.stdout:6/524: dread dd/f1c [0,4194304] 0 2024-04-22T15:19:00.904 INFO:tasks.workunit.client.0.smithi096.stdout:9/698: dwrite d1/df/d95/d9d/db5/f12 [0,4194304] 0 2024-04-22T15:19:00.905 INFO:tasks.workunit.client.1.smithi196.stdout:7/481: readlink d14/d26/d28/l56 0 2024-04-22T15:19:00.905 INFO:tasks.workunit.client.1.smithi196.stdout:7/482: write d14/d26/f35 [4689409,68020] 0 2024-04-22T15:19:00.910 INFO:tasks.workunit.client.0.smithi096.stdout:4/785: symlink d5/d7/d2c/d3d/def/l10f 0 2024-04-22T15:19:00.911 INFO:tasks.workunit.client.0.smithi096.stdout:9/699: rmdir d1/df 39 2024-04-22T15:19:00.914 INFO:tasks.workunit.client.0.smithi096.stdout:4/786: creat d5/d7/d1f/d84/d6d/d98/f110 x:0 0 0 2024-04-22T15:19:00.915 INFO:tasks.workunit.client.1.smithi196.stdout:5/524: rename d1/d17/d50/f9a to d1/d17/d1c/fa4 0 2024-04-22T15:19:00.918 INFO:tasks.workunit.client.0.smithi096.stdout:9/700: write d1/df/d95/d9d/db5/f24 [1823931,15282] 0 2024-04-22T15:19:00.920 INFO:tasks.workunit.client.1.smithi196.stdout:0/533: mkdir d3/d6/dc/d8c/db5 0 2024-04-22T15:19:00.922 INFO:tasks.workunit.client.1.smithi196.stdout:3/506: creat d0/d15/d31/d9d/d9e/fab x:0 0 0 2024-04-22T15:19:00.923 INFO:tasks.workunit.client.0.smithi096.stdout:9/701: mknod d1/d4/d79/ced 0 2024-04-22T15:19:00.924 INFO:tasks.workunit.client.0.smithi096.stdout:9/702: readlink d1/df/d95/d9d/db5/dd/l6d 0 2024-04-22T15:19:00.925 INFO:tasks.workunit.client.0.smithi096.stdout:9/703: write d1/d25/fd5 [655775,21837] 0 2024-04-22T15:19:00.927 INFO:tasks.workunit.client.1.smithi196.stdout:1/476: mkdir d1/db/d97 0 2024-04-22T15:19:00.928 INFO:tasks.workunit.client.1.smithi196.stdout:1/477: write d1/db/d16/f7e [3124537,123599] 0 2024-04-22T15:19:00.929 INFO:tasks.workunit.client.1.smithi196.stdout:2/548: link d1/d13/d25/d2a/c6f d1/d23/d27/d89/ccd 0 2024-04-22T15:19:00.930 INFO:tasks.workunit.client.0.smithi096.stdout:0/898: sync 2024-04-22T15:19:00.930 INFO:tasks.workunit.client.0.smithi096.stdout:5/801: sync 2024-04-22T15:19:00.931 INFO:tasks.workunit.client.0.smithi096.stdout:4/787: link d5/d7/ca d5/d7/d2c/d3d/d4f/d9c/c111 0 2024-04-22T15:19:00.931 INFO:tasks.workunit.client.1.smithi196.stdout:0/534: dread d3/de/f7d [0,4194304] 0 2024-04-22T15:19:00.933 INFO:tasks.workunit.client.1.smithi196.stdout:4/498: rename d6/f4d to d6/d13/d15/d2c/d62/f99 0 2024-04-22T15:19:00.933 INFO:tasks.workunit.client.1.smithi196.stdout:4/499: write d6/f93 [258378,85555] 0 2024-04-22T15:19:00.934 INFO:tasks.workunit.client.0.smithi096.stdout:5/802: read - d4/ddb/d3d/d35/d57/fd4 zero size 2024-04-22T15:19:00.934 INFO:tasks.workunit.client.1.smithi196.stdout:5/525: write d1/d17/f2e [4457942,97095] 0 2024-04-22T15:19:00.938 INFO:tasks.workunit.client.1.smithi196.stdout:9/516: link db/dd/f5c db/d11/fb2 0 2024-04-22T15:19:00.940 INFO:tasks.workunit.client.0.smithi096.stdout:9/704: creat d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/dd9/fee x:0 0 0 2024-04-22T15:19:00.944 INFO:tasks.workunit.client.0.smithi096.stdout:4/788: truncate d5/d7/d2c/d67/fab 1457230 0 2024-04-22T15:19:00.946 INFO:tasks.workunit.client.1.smithi196.stdout:8/520: read d2/d8/d47/d68/d81/f8d [611004,6670] 0 2024-04-22T15:19:00.948 INFO:tasks.workunit.client.1.smithi196.stdout:7/483: symlink d14/d26/d95/l97 0 2024-04-22T15:19:00.951 INFO:tasks.workunit.client.1.smithi196.stdout:7/484: dread d14/d26/d2a/f66 [4194304,4194304] 0 2024-04-22T15:19:00.953 INFO:tasks.workunit.client.0.smithi096.stdout:4/789: fdatasync d5/d7/d2c/f2e 0 2024-04-22T15:19:00.955 INFO:tasks.workunit.client.0.smithi096.stdout:0/899: write d7/de/d5b/d95/dc/f8d [959636,86948] 0 2024-04-22T15:19:00.958 INFO:tasks.workunit.client.0.smithi096.stdout:5/803: creat d4/ddb/d3d/d4d/d59/d98/dee/d109/f10c x:0 0 0 2024-04-22T15:19:00.958 INFO:tasks.workunit.client.1.smithi196.stdout:2/549: creat d1/d23/d27/d2e/d55/fce x:0 0 0 2024-04-22T15:19:00.960 INFO:tasks.workunit.client.1.smithi196.stdout:0/535: symlink d3/d6/dc/d8c/d8d/lb6 0 2024-04-22T15:19:00.961 INFO:tasks.workunit.client.1.smithi196.stdout:0/536: chown d3/de/d1c/d27/f38 232 1 2024-04-22T15:19:00.962 INFO:tasks.workunit.client.0.smithi096.stdout:4/790: creat d5/d7/d2c/d3d/d4f/d63/f112 x:0 0 0 2024-04-22T15:19:00.965 INFO:tasks.workunit.client.1.smithi196.stdout:3/507: rename d0/d15/f8f to d0/d15/da7/d18/fac 0 2024-04-22T15:19:00.967 INFO:tasks.workunit.client.0.smithi096.stdout:9/705: dread d1/df/d95/d9d/db5/d43/f5f [0,4194304] 0 2024-04-22T15:19:00.968 INFO:tasks.workunit.client.0.smithi096.stdout:0/900: truncate d7/de/d3a/d61/dbb/fea 18345 0 2024-04-22T15:19:00.972 INFO:tasks.workunit.client.0.smithi096.stdout:5/804: truncate d4/ddb/d3d/d4d/f5d 3244656 0 2024-04-22T15:19:00.973 INFO:tasks.workunit.client.1.smithi196.stdout:9/517: chown db/dd/d1b/d2c/l74 2425 1 2024-04-22T15:19:00.975 INFO:tasks.workunit.client.0.smithi096.stdout:4/791: symlink d5/d7/d1a/d103/l113 0 2024-04-22T15:19:00.978 INFO:tasks.workunit.client.0.smithi096.stdout:9/706: creat d1/df/d95/d9d/db5/dd/dba/dc2/dc4/fef x:0 0 0 2024-04-22T15:19:00.979 INFO:tasks.workunit.client.0.smithi096.stdout:9/707: chown d1/df/d95/d9d/db5/d43/d57/d5b/f84 283877 1 2024-04-22T15:19:00.984 INFO:tasks.workunit.client.1.smithi196.stdout:4/500: write d6/d13/f1d [2833989,107744] 0 2024-04-22T15:19:00.989 INFO:tasks.workunit.client.1.smithi196.stdout:8/521: dread d2/d5/f87 [0,4194304] 0 2024-04-22T15:19:00.992 INFO:tasks.workunit.client.0.smithi096.stdout:5/805: mknod d4/ddb/d3d/c10d 0 2024-04-22T15:19:00.992 INFO:tasks.workunit.client.0.smithi096.stdout:5/806: write d4/d2c/d2f/d6b/ffe [949751,61809] 0 2024-04-22T15:19:00.992 INFO:tasks.workunit.client.0.smithi096.stdout:4/792: creat d5/d7/d1a/d103/f114 x:0 0 0 2024-04-22T15:19:00.992 INFO:tasks.workunit.client.0.smithi096.stdout:5/807: chown d4/ddb/d3d/d4d/cd8 1475878 1 2024-04-22T15:19:00.993 INFO:tasks.workunit.client.1.smithi196.stdout:6/525: creat dd/d1f/d4e/d53/d55/f95 x:0 0 0 2024-04-22T15:19:00.993 INFO:tasks.workunit.client.1.smithi196.stdout:6/526: chown dd/d1f/d37/f34 1 1 2024-04-22T15:19:00.994 INFO:tasks.workunit.client.1.smithi196.stdout:6/527: write dd/d2b/f77 [354072,52767] 0 2024-04-22T15:19:00.995 INFO:tasks.workunit.client.0.smithi096.stdout:9/708: mknod d1/d4/d79/cf0 0 2024-04-22T15:19:00.995 INFO:tasks.workunit.client.1.smithi196.stdout:6/528: dread - dd/d1f/d38/f92 zero size 2024-04-22T15:19:00.995 INFO:tasks.workunit.client.0.smithi096.stdout:0/901: write d7/de/d5b/d95/d66/f100 [758944,118850] 0 2024-04-22T15:19:01.003 INFO:tasks.workunit.client.0.smithi096.stdout:5/808: unlink d4/ddb/d3d/d4d/d59/d5a/f101 0 2024-04-22T15:19:01.004 INFO:tasks.workunit.client.1.smithi196.stdout:0/537: fsync d3/fd 0 2024-04-22T15:19:01.005 INFO:tasks.workunit.client.1.smithi196.stdout:0/538: readlink d3/d6/l7 0 2024-04-22T15:19:01.005 INFO:tasks.workunit.client.1.smithi196.stdout:0/539: write d3/d9/f54 [730575,40333] 0 2024-04-22T15:19:01.006 INFO:tasks.workunit.client.1.smithi196.stdout:0/540: chown d3/d9/fb1 2085692 1 2024-04-22T15:19:01.007 INFO:tasks.workunit.client.1.smithi196.stdout:0/541: write d3/de/d14/d5a/f5f [379859,41116] 0 2024-04-22T15:19:01.009 INFO:tasks.workunit.client.1.smithi196.stdout:7/485: rename d14/d25/l57 to d14/d26/d5a/d5f/d6e/l98 0 2024-04-22T15:19:01.010 INFO:tasks.workunit.client.1.smithi196.stdout:7/486: dread - d14/d26/d2a/f84 zero size 2024-04-22T15:19:01.011 INFO:tasks.workunit.client.0.smithi096.stdout:9/709: readlink d1/d85/l99 0 2024-04-22T15:19:01.013 INFO:tasks.workunit.client.1.smithi196.stdout:3/508: write d0/d15/d48/d51/fa9 [3651756,47241] 0 2024-04-22T15:19:01.014 INFO:tasks.workunit.client.0.smithi096.stdout:0/902: creat d7/d42/d4c/d55/d9c/df7/d10c/f132 x:0 0 0 2024-04-22T15:19:01.016 INFO:tasks.workunit.client.1.smithi196.stdout:1/478: dwrite d1/db/d16/d1e/d65/f76 [0,4194304] 0 2024-04-22T15:19:01.016 INFO:tasks.workunit.client.1.smithi196.stdout:5/526: mknod d1/d17/d1a/ca5 0 2024-04-22T15:19:01.018 INFO:tasks.workunit.client.1.smithi196.stdout:5/527: readlink d1/d9/l6c 0 2024-04-22T15:19:01.018 INFO:tasks.workunit.client.1.smithi196.stdout:9/518: chown db/d11/d29/l8b 9 1 2024-04-22T15:19:01.019 INFO:tasks.workunit.client.1.smithi196.stdout:7/487: dwrite d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:19:01.020 INFO:tasks.workunit.client.1.smithi196.stdout:4/501: mkdir d6/d14/d18/d26/d32/d67/d9a 0 2024-04-22T15:19:01.022 INFO:tasks.workunit.client.1.smithi196.stdout:0/542: dread d3/de/d1c/d27/f32 [0,4194304] 0 2024-04-22T15:19:01.029 INFO:tasks.workunit.client.0.smithi096.stdout:4/793: link d5/d7/d2c/d3d/def/l101 d5/d7/d1a/d8b/l115 0 2024-04-22T15:19:01.031 INFO:tasks.workunit.client.1.smithi196.stdout:1/479: dread d1/db/d16/f40 [0,4194304] 0 2024-04-22T15:19:01.032 INFO:tasks.workunit.client.0.smithi096.stdout:9/710: mknod d1/df/d95/d9d/db5/d43/d57/d5b/d73/de3/cf1 0 2024-04-22T15:19:01.035 INFO:tasks.workunit.client.0.smithi096.stdout:5/809: creat d4/ddb/d3d/d35/f10e x:0 0 0 2024-04-22T15:19:01.037 INFO:tasks.workunit.client.1.smithi196.stdout:9/519: symlink db/dd/d1b/d2c/lb3 0 2024-04-22T15:19:01.038 INFO:tasks.workunit.client.1.smithi196.stdout:9/520: write db/dd/f9e [706968,82558] 0 2024-04-22T15:19:01.039 INFO:tasks.workunit.client.0.smithi096.stdout:9/711: creat d1/df/d95/d9d/db5/d43/d57/d5b/d73/ff2 x:0 0 0 2024-04-22T15:19:01.049 INFO:tasks.workunit.client.0.smithi096.stdout:5/810: creat d4/d1f/d8b/d106/f10f x:0 0 0 2024-04-22T15:19:01.049 INFO:tasks.workunit.client.0.smithi096.stdout:9/712: dwrite d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/fdd [0,4194304] 0 2024-04-22T15:19:01.050 INFO:tasks.workunit.client.1.smithi196.stdout:7/488: creat d14/d26/d2a/d72/f99 x:0 0 0 2024-04-22T15:19:01.053 INFO:tasks.workunit.client.0.smithi096.stdout:0/903: write d7/d42/d4c/d55/d9c/df7/d10c/f70 [4047026,95377] 0 2024-04-22T15:19:01.055 INFO:tasks.workunit.client.0.smithi096.stdout:9/713: unlink d1/f2f 0 2024-04-22T15:19:01.058 INFO:tasks.workunit.client.1.smithi196.stdout:0/543: mkdir d3/d9/d36/d56/d65/d7f/d87/d6c/d88/db7 0 2024-04-22T15:19:01.059 INFO:tasks.workunit.client.0.smithi096.stdout:5/811: unlink d4/ddb/d3d/d4d/d59/d5a/f79 0 2024-04-22T15:19:01.061 INFO:tasks.workunit.client.1.smithi196.stdout:6/529: mkdir dd/d1f/d37/d3c/d96 0 2024-04-22T15:19:01.066 INFO:tasks.workunit.client.1.smithi196.stdout:6/530: dwrite dd/d2b/f77 [0,4194304] 0 2024-04-22T15:19:01.067 INFO:tasks.workunit.client.0.smithi096.stdout:9/714: rmdir d1/d25/db2/dca 39 2024-04-22T15:19:01.072 INFO:tasks.workunit.client.0.smithi096.stdout:4/794: dwrite d5/d7/d2c/d95/dee/f59 [0,4194304] 0 2024-04-22T15:19:01.075 INFO:tasks.workunit.client.0.smithi096.stdout:5/812: mknod d4/ddb/d3d/d35/d57/dbe/c110 0 2024-04-22T15:19:01.077 INFO:tasks.workunit.client.1.smithi196.stdout:5/528: creat d1/d17/d7f/fa6 x:0 0 0 2024-04-22T15:19:01.078 INFO:tasks.workunit.client.1.smithi196.stdout:5/529: chown d1/d14/d3e/f4e 391381671 1 2024-04-22T15:19:01.079 INFO:tasks.workunit.client.1.smithi196.stdout:9/521: symlink db/dd/de/lb4 0 2024-04-22T15:19:01.080 INFO:tasks.workunit.client.1.smithi196.stdout:9/522: chown db/dd/de/d15/d31/d64/dab 0 1 2024-04-22T15:19:01.080 INFO:tasks.workunit.client.1.smithi196.stdout:9/523: write db/d11/d29/d7a/f9c [191449,73414] 0 2024-04-22T15:19:01.082 INFO:tasks.workunit.client.1.smithi196.stdout:4/502: mknod d6/d14/d18/d26/d32/c9b 0 2024-04-22T15:19:01.083 INFO:tasks.workunit.client.1.smithi196.stdout:2/550: write d1/d13/d41/f6c [82123,35393] 0 2024-04-22T15:19:01.084 INFO:tasks.workunit.client.1.smithi196.stdout:2/551: chown d1/d1d 3179 1 2024-04-22T15:19:01.086 INFO:tasks.workunit.client.0.smithi096.stdout:4/795: symlink d5/d7/d2c/d67/l116 0 2024-04-22T15:19:01.086 INFO:tasks.workunit.client.0.smithi096.stdout:6/634: sync 2024-04-22T15:19:01.088 INFO:tasks.workunit.client.0.smithi096.stdout:5/813: symlink d4/ddb/d3d/d4f/l111 0 2024-04-22T15:19:01.090 INFO:tasks.workunit.client.0.smithi096.stdout:0/904: dwrite d7/de/d5b/dda/f105 [0,4194304] 0 2024-04-22T15:19:01.091 INFO:tasks.workunit.client.1.smithi196.stdout:9/524: dwrite db/dd/de/f10 [4194304,4194304] 0 2024-04-22T15:19:01.092 INFO:tasks.workunit.client.0.smithi096.stdout:9/715: truncate d1/d4/f11 499058 0 2024-04-22T15:19:01.093 INFO:tasks.workunit.client.1.smithi196.stdout:1/480: dwrite d1/db/dc/de/f14 [0,4194304] 0 2024-04-22T15:19:01.094 INFO:tasks.workunit.client.0.smithi096.stdout:5/814: chown d4/ddb/d3d/d4f/c10a 458927833 1 2024-04-22T15:19:01.094 INFO:tasks.workunit.client.1.smithi196.stdout:4/503: dwrite d6/d13/f1d [0,4194304] 0 2024-04-22T15:19:01.101 INFO:tasks.workunit.client.1.smithi196.stdout:2/552: dwrite d1/d23/d27/d2e/d55/d5b/d7c/fa5 [0,4194304] 0 2024-04-22T15:19:01.102 INFO:tasks.workunit.client.0.smithi096.stdout:9/716: mkdir d1/d25/d9f/df3 0 2024-04-22T15:19:01.103 INFO:tasks.workunit.client.1.smithi196.stdout:4/504: dread d6/d90/f4a [0,4194304] 0 2024-04-22T15:19:01.103 INFO:tasks.workunit.client.1.smithi196.stdout:5/530: write d1/d17/f24 [2040693,114282] 0 2024-04-22T15:19:01.104 INFO:tasks.workunit.client.1.smithi196.stdout:2/553: stat d1/d23/d27/d2e/d55/d5b/d93 0 2024-04-22T15:19:01.105 INFO:tasks.workunit.client.0.smithi096.stdout:4/796: rename d5/d7/ca to d5/d7/d2c/d3d/d4f/d9c/c117 0 2024-04-22T15:19:01.109 INFO:tasks.workunit.client.0.smithi096.stdout:5/815: mknod d4/ddb/d3d/d4d/d88/de5/c112 0 2024-04-22T15:19:01.110 INFO:tasks.workunit.client.1.smithi196.stdout:5/531: dwrite d1/d17/d50/f73 [0,4194304] 0 2024-04-22T15:19:01.111 INFO:tasks.workunit.client.0.smithi096.stdout:9/717: mkdir d1/d85/df4 0 2024-04-22T15:19:01.113 INFO:tasks.workunit.client.1.smithi196.stdout:8/522: getdents d2/d5/d4f 0 2024-04-22T15:19:01.120 INFO:tasks.workunit.client.1.smithi196.stdout:9/525: creat db/dd/d1b/d2c/fb5 x:0 0 0 2024-04-22T15:19:01.121 INFO:tasks.workunit.client.0.smithi096.stdout:5/816: mkdir d4/ddb/d3d/d35/d57/dbe/d113 0 2024-04-22T15:19:01.124 INFO:tasks.workunit.client.1.smithi196.stdout:2/554: mknod d1/d23/d27/d2e/d55/d5b/d93/ccf 0 2024-04-22T15:19:01.125 INFO:tasks.workunit.client.1.smithi196.stdout:2/555: chown d1/d23/f2f 20826 1 2024-04-22T15:19:01.129 INFO:tasks.workunit.client.0.smithi096.stdout:0/905: getdents d7/d42/d4c/d55/d9c/df7/dfb 0 2024-04-22T15:19:01.130 INFO:tasks.workunit.client.0.smithi096.stdout:8/776: sync 2024-04-22T15:19:01.130 INFO:tasks.workunit.client.0.smithi096.stdout:3/691: sync 2024-04-22T15:19:01.130 INFO:tasks.workunit.client.0.smithi096.stdout:2/809: sync 2024-04-22T15:19:01.130 INFO:tasks.workunit.client.0.smithi096.stdout:1/826: sync 2024-04-22T15:19:01.131 INFO:tasks.workunit.client.1.smithi196.stdout:0/544: creat d3/d9/fb8 x:0 0 0 2024-04-22T15:19:01.131 INFO:tasks.workunit.client.0.smithi096.stdout:2/810: stat c2 0 2024-04-22T15:19:01.132 INFO:tasks.workunit.client.1.smithi196.stdout:3/509: sync 2024-04-22T15:19:01.134 INFO:tasks.workunit.client.0.smithi096.stdout:3/692: chown da/db/de/daa 0 1 2024-04-22T15:19:01.137 INFO:tasks.workunit.client.1.smithi196.stdout:1/481: mknod d1/db/d16/d1e/c98 0 2024-04-22T15:19:01.138 INFO:tasks.workunit.client.1.smithi196.stdout:1/482: readlink d1/db/d16/d1e/d46/l4e 0 2024-04-22T15:19:01.140 INFO:tasks.workunit.client.1.smithi196.stdout:1/483: write d1/db/dc/de/f14 [717958,42808] 0 2024-04-22T15:19:01.142 INFO:tasks.workunit.client.1.smithi196.stdout:0/545: dwrite d3/d9/f54 [0,4194304] 0 2024-04-22T15:19:01.143 INFO:tasks.workunit.client.1.smithi196.stdout:9/526: dread - db/d11/d29/f2b zero size 2024-04-22T15:19:01.144 INFO:tasks.workunit.client.1.smithi196.stdout:9/527: write db/ff [4714570,125087] 0 2024-04-22T15:19:01.145 INFO:tasks.workunit.client.0.smithi096.stdout:1/827: rmdir d6/dc/d8a/d3f/d54/d32/d82 39 2024-04-22T15:19:01.147 INFO:tasks.workunit.client.1.smithi196.stdout:5/532: dread d1/d9/f75 [0,4194304] 0 2024-04-22T15:19:01.150 INFO:tasks.workunit.client.1.smithi196.stdout:9/528: dwrite db/d11/d29/f92 [0,4194304] 0 2024-04-22T15:19:01.152 INFO:tasks.workunit.client.1.smithi196.stdout:2/556: creat d1/d13/d25/fd0 x:0 0 0 2024-04-22T15:19:01.154 INFO:tasks.workunit.client.0.smithi096.stdout:6/635: dwrite d3/d28/dd/d11/d2c/d6f/dd4/dd6/fb5 [0,4194304] 0 2024-04-22T15:19:01.155 INFO:tasks.workunit.client.0.smithi096.stdout:9/718: dread d1/d4/d7b/fec [0,4194304] 0 2024-04-22T15:19:01.155 INFO:tasks.workunit.client.0.smithi096.stdout:6/636: chown f1 24148 1 2024-04-22T15:19:01.157 INFO:tasks.workunit.client.0.smithi096.stdout:9/719: chown d1/df/d95/d9d/db5/d59 177 1 2024-04-22T15:19:01.158 INFO:tasks.workunit.client.0.smithi096.stdout:0/906: symlink d7/de/d5b/d95/l133 0 2024-04-22T15:19:01.158 INFO:tasks.workunit.client.1.smithi196.stdout:7/489: getdents d14/d4a 0 2024-04-22T15:19:01.159 INFO:tasks.workunit.client.0.smithi096.stdout:9/720: chown d1/la 71494 1 2024-04-22T15:19:01.163 INFO:tasks.workunit.client.1.smithi196.stdout:8/523: creat d2/d6a/d7b/f94 x:0 0 0 2024-04-22T15:19:01.165 INFO:tasks.workunit.client.0.smithi096.stdout:4/797: write d5/d7/d1f/d84/fa0 [335703,66009] 0 2024-04-22T15:19:01.168 INFO:tasks.workunit.client.1.smithi196.stdout:6/531: truncate dd/f15 2818180 0 2024-04-22T15:19:01.174 INFO:tasks.workunit.client.1.smithi196.stdout:4/505: dwrite d6/d14/f27 [4194304,4194304] 0 2024-04-22T15:19:01.175 INFO:tasks.workunit.client.1.smithi196.stdout:6/532: dwrite dd/d1f/d4e/f54 [0,4194304] 0 2024-04-22T15:19:01.177 INFO:tasks.workunit.client.0.smithi096.stdout:1/828: fsync d6/dc/d8a/d3f/d62/d7d/d8e/fa0 0 2024-04-22T15:19:01.185 INFO:tasks.workunit.client.0.smithi096.stdout:5/817: dwrite d4/ddb/d3d/d35/d57/fd4 [0,4194304] 0 2024-04-22T15:19:01.185 INFO:tasks.workunit.client.0.smithi096.stdout:9/721: fsync d1/df/d95/d9d/db5/d43/d57/f8d 0 2024-04-22T15:19:01.208 INFO:tasks.workunit.client.1.smithi196.stdout:3/510: chown d0/d15/d4d/c9f 6436 1 2024-04-22T15:19:01.209 INFO:tasks.workunit.client.0.smithi096.stdout:3/693: dwrite da/db/de/d19/f6f [0,4194304] 0 2024-04-22T15:19:01.212 INFO:tasks.workunit.client.0.smithi096.stdout:8/777: dwrite d4/d10/d25/d4c/d76/d7f/f92 [0,4194304] 0 2024-04-22T15:19:01.217 INFO:tasks.workunit.client.0.smithi096.stdout:2/811: dwrite d3/db/df/f61 [0,4194304] 0 2024-04-22T15:19:01.218 INFO:tasks.workunit.client.0.smithi096.stdout:6/637: truncate d3/d28/dd/f16 2545389 0 2024-04-22T15:19:01.218 INFO:tasks.workunit.client.0.smithi096.stdout:0/907: link d7/de/d5b/d95/d66/f81 d7/d42/f134 0 2024-04-22T15:19:01.220 INFO:tasks.workunit.client.0.smithi096.stdout:5/818: unlink d4/ddb/d3d/d4f/cdc 0 2024-04-22T15:19:01.226 INFO:tasks.workunit.client.1.smithi196.stdout:9/529: symlink db/d11/d29/d7a/lb6 0 2024-04-22T15:19:01.228 INFO:tasks.workunit.client.1.smithi196.stdout:2/557: creat d1/d23/fd1 x:0 0 0 2024-04-22T15:19:01.229 INFO:tasks.workunit.client.1.smithi196.stdout:2/558: write d1/d13/f1e [9132,104971] 0 2024-04-22T15:19:01.238 INFO:tasks.workunit.client.0.smithi096.stdout:0/908: dread d7/de/d5b/d95/dc/d16/d18/f92 [0,4194304] 0 2024-04-22T15:19:01.238 INFO:tasks.workunit.client.0.smithi096.stdout:0/909: stat d7/de/d5b/d62 0 2024-04-22T15:19:01.240 INFO:tasks.workunit.client.1.smithi196.stdout:6/533: symlink dd/d20/d5c/d5e/d76/l97 0 2024-04-22T15:19:01.243 INFO:tasks.workunit.client.1.smithi196.stdout:1/484: creat d1/db/d16/d1e/d49/d90/f99 x:0 0 0 2024-04-22T15:19:01.245 INFO:tasks.workunit.client.1.smithi196.stdout:5/533: mkdir d1/d14/d6a/da7 0 2024-04-22T15:19:01.245 INFO:tasks.workunit.client.1.smithi196.stdout:7/490: sync 2024-04-22T15:19:01.246 INFO:tasks.workunit.client.1.smithi196.stdout:7/491: read d14/d25/f90 [156199,73628] 0 2024-04-22T15:19:01.253 INFO:tasks.workunit.client.1.smithi196.stdout:2/559: mknod d1/d23/d27/d89/cd2 0 2024-04-22T15:19:01.253 INFO:tasks.workunit.client.1.smithi196.stdout:0/546: write d3/de/f7d [632034,5788] 0 2024-04-22T15:19:01.254 INFO:tasks.workunit.client.1.smithi196.stdout:4/506: creat d6/d13/d15/d2c/d62/d78/d8d/f9c x:0 0 0 2024-04-22T15:19:01.262 INFO:tasks.workunit.client.1.smithi196.stdout:1/485: mkdir d1/db/dc/de/d5a/d5b/d9a 0 2024-04-22T15:19:01.262 INFO:tasks.workunit.client.1.smithi196.stdout:1/486: stat d1/db/d16/d44/d6a 0 2024-04-22T15:19:01.264 INFO:tasks.workunit.client.1.smithi196.stdout:5/534: creat d1/d14/d6a/fa8 x:0 0 0 2024-04-22T15:19:01.271 INFO:tasks.workunit.client.1.smithi196.stdout:8/524: symlink d2/dc/l95 0 2024-04-22T15:19:01.272 INFO:tasks.workunit.client.1.smithi196.stdout:7/492: dread f10 [4194304,4194304] 0 2024-04-22T15:19:01.274 INFO:tasks.workunit.client.1.smithi196.stdout:7/493: truncate d14/d4a/f71 293795 0 2024-04-22T15:19:01.274 INFO:tasks.workunit.client.1.smithi196.stdout:2/560: creat d1/d13/d41/fd3 x:0 0 0 2024-04-22T15:19:01.275 INFO:tasks.workunit.client.1.smithi196.stdout:8/525: dread d2/d5/f19 [0,4194304] 0 2024-04-22T15:19:01.276 INFO:tasks.workunit.client.1.smithi196.stdout:6/534: write dd/f12 [1869301,43888] 0 2024-04-22T15:19:01.277 INFO:tasks.workunit.client.1.smithi196.stdout:4/507: fsync d6/d10/f63 0 2024-04-22T15:19:01.277 INFO:tasks.workunit.client.1.smithi196.stdout:1/487: dread d1/db/dc/de/d43/f6b [0,4194304] 0 2024-04-22T15:19:01.278 INFO:tasks.workunit.client.1.smithi196.stdout:3/511: link d0/d7/l3b d0/d15/da7/d4a/d44/lad 0 2024-04-22T15:19:01.281 INFO:tasks.workunit.client.1.smithi196.stdout:5/535: rename d1/d53 to d1/d9/da9 0 2024-04-22T15:19:01.285 INFO:tasks.workunit.client.1.smithi196.stdout:9/530: link db/dd/de/d15/d31/f32 db/dd/de/d15/d31/fb7 0 2024-04-22T15:19:01.289 INFO:tasks.workunit.client.1.smithi196.stdout:0/547: mkdir d3/d9/d36/d9f/db9 0 2024-04-22T15:19:01.289 INFO:tasks.workunit.client.1.smithi196.stdout:0/548: dread - d3/d6/dc/d8c/d8d/d98/fa6 zero size 2024-04-22T15:19:01.290 INFO:tasks.workunit.client.1.smithi196.stdout:8/526: stat d2/d3f/d4b/f72 0 2024-04-22T15:19:01.290 INFO:tasks.workunit.client.1.smithi196.stdout:8/527: stat d2/d5/f19 0 2024-04-22T15:19:01.296 INFO:tasks.workunit.client.1.smithi196.stdout:3/512: write d0/d15/d48/d51/f54 [104013,96535] 0 2024-04-22T15:19:01.296 INFO:tasks.workunit.client.1.smithi196.stdout:2/561: dread d1/f10 [0,4194304] 0 2024-04-22T15:19:01.300 INFO:tasks.workunit.client.1.smithi196.stdout:5/536: mknod d1/d14/caa 0 2024-04-22T15:19:01.302 INFO:tasks.workunit.client.1.smithi196.stdout:2/562: dwrite d1/d13/d41/fa8 [0,4194304] 0 2024-04-22T15:19:01.302 INFO:tasks.workunit.client.1.smithi196.stdout:9/531: read f5 [1859141,49450] 0 2024-04-22T15:19:01.303 INFO:tasks.workunit.client.1.smithi196.stdout:9/532: chown db/dd/de/d15/l67 44735 1 2024-04-22T15:19:01.304 INFO:tasks.workunit.client.1.smithi196.stdout:7/494: mkdir d14/d15/d91/d9a 0 2024-04-22T15:19:01.304 INFO:tasks.workunit.client.1.smithi196.stdout:2/563: dread - d1/d23/d27/d2e/d55/d5b/d93/fbf zero size 2024-04-22T15:19:01.305 INFO:tasks.workunit.client.1.smithi196.stdout:2/564: chown d1/d23/d27/d89/fb0 2616 1 2024-04-22T15:19:01.306 INFO:tasks.workunit.client.1.smithi196.stdout:0/549: mkdir d3/d9/d36/d56/d65/d7f/d87/d6c/dba 0 2024-04-22T15:19:01.307 INFO:tasks.workunit.client.1.smithi196.stdout:0/550: chown d3/d9/d36/d56/d65/d7f/d51/f75 166 1 2024-04-22T15:19:01.309 INFO:tasks.workunit.client.1.smithi196.stdout:0/551: truncate d3/d9/fb8 591859 0 2024-04-22T15:19:01.312 INFO:tasks.workunit.client.1.smithi196.stdout:3/513: creat d0/d15/d48/fae x:0 0 0 2024-04-22T15:19:01.313 INFO:tasks.workunit.client.1.smithi196.stdout:6/535: dwrite dd/d1f/f48 [4194304,4194304] 0 2024-04-22T15:19:01.318 INFO:tasks.workunit.client.1.smithi196.stdout:0/552: dwrite d3/de/f7d [0,4194304] 0 2024-04-22T15:19:01.318 INFO:tasks.workunit.client.1.smithi196.stdout:5/537: mknod d1/d17/d1c/d67/cab 0 2024-04-22T15:19:01.319 INFO:tasks.workunit.client.1.smithi196.stdout:5/538: read - d1/d4/f9c zero size 2024-04-22T15:19:01.323 INFO:tasks.workunit.client.1.smithi196.stdout:6/536: dwrite dd/f86 [0,4194304] 0 2024-04-22T15:19:01.324 INFO:tasks.workunit.client.1.smithi196.stdout:6/537: chown dd/d1f/d37/f5d 60096 1 2024-04-22T15:19:01.325 INFO:tasks.workunit.client.1.smithi196.stdout:2/565: creat d1/d13/d41/fd4 x:0 0 0 2024-04-22T15:19:01.326 INFO:tasks.workunit.client.1.smithi196.stdout:6/538: readlink dd/d1f/l35 0 2024-04-22T15:19:01.328 INFO:tasks.workunit.client.1.smithi196.stdout:8/528: mkdir d2/d35/d96 0 2024-04-22T15:19:01.330 INFO:tasks.workunit.client.1.smithi196.stdout:4/508: creat d6/d13/d15/d2c/d35/f9d x:0 0 0 2024-04-22T15:19:01.335 INFO:tasks.workunit.client.1.smithi196.stdout:3/514: creat d0/d15/d31/faf x:0 0 0 2024-04-22T15:19:01.336 INFO:tasks.workunit.client.1.smithi196.stdout:3/515: stat d0/d15/da7/d4a/d44/d8c 0 2024-04-22T15:19:01.337 INFO:tasks.workunit.client.0.smithi096.stdout:8/778: chown d4/d10/d25/c52 72 1 2024-04-22T15:19:01.337 INFO:tasks.workunit.client.1.smithi196.stdout:3/516: write d0/d15/da7/fa8 [437736,24821] 0 2024-04-22T15:19:01.338 INFO:tasks.workunit.client.1.smithi196.stdout:1/488: link d1/db/dc/d26/f30 d1/db/dc/d26/d2e/d32/f9b 0 2024-04-22T15:19:01.339 INFO:tasks.workunit.client.1.smithi196.stdout:1/489: fdatasync d1/db/dc/d26/d2e/f34 0 2024-04-22T15:19:01.341 INFO:tasks.workunit.client.1.smithi196.stdout:9/533: symlink db/d11/d29/lb8 0 2024-04-22T15:19:01.343 INFO:tasks.workunit.client.1.smithi196.stdout:5/539: mkdir d1/d2/d98/dac 0 2024-04-22T15:19:01.345 INFO:tasks.workunit.client.1.smithi196.stdout:7/495: rename d14/d26/d2a/d2b/c50 to d14/d4a/c9b 0 2024-04-22T15:19:01.345 INFO:tasks.workunit.client.1.smithi196.stdout:7/496: readlink d14/d4d/l5c 0 2024-04-22T15:19:01.351 INFO:tasks.workunit.client.1.smithi196.stdout:8/529: creat d2/d3f/f97 x:0 0 0 2024-04-22T15:19:01.354 INFO:tasks.workunit.client.0.smithi096.stdout:0/910: creat d7/de/d3a/d61/dbb/f135 x:0 0 0 2024-04-22T15:19:01.356 INFO:tasks.workunit.client.1.smithi196.stdout:3/517: unlink d0/d15/da7/d18/f4f 0 2024-04-22T15:19:01.357 INFO:tasks.workunit.client.1.smithi196.stdout:3/518: write d0/d15/da7/d4a/d44/f26 [3368424,1166] 0 2024-04-22T15:19:01.360 INFO:tasks.workunit.client.1.smithi196.stdout:6/539: dread dd/f15 [0,4194304] 0 2024-04-22T15:19:01.361 INFO:tasks.workunit.client.1.smithi196.stdout:1/490: rename d1/d54/d70/c8d to d1/db/dc/de/d5a/d94/c9c 0 2024-04-22T15:19:01.363 INFO:tasks.workunit.client.0.smithi096.stdout:6/638: dwrite d3/d28/dd/d11/d2c/f5e [0,4194304] 0 2024-04-22T15:19:01.363 INFO:tasks.workunit.client.1.smithi196.stdout:3/519: dwrite d0/d15/da7/f47 [0,4194304] 0 2024-04-22T15:19:01.363 INFO:tasks.workunit.client.0.smithi096.stdout:2/812: mknod d3/db/c105 0 2024-04-22T15:19:01.365 INFO:tasks.workunit.client.0.smithi096.stdout:5/819: write d4/ddb/d25/f7e [1772224,45574] 0 2024-04-22T15:19:01.366 INFO:tasks.workunit.client.1.smithi196.stdout:0/553: mknod d3/d9/d36/d56/d65/d7f/d87/d6c/d88/db7/cbb 0 2024-04-22T15:19:01.367 INFO:tasks.workunit.client.1.smithi196.stdout:7/497: dread d14/f33 [0,4194304] 0 2024-04-22T15:19:01.368 INFO:tasks.workunit.client.1.smithi196.stdout:1/491: dwrite d1/db/f29 [0,4194304] 0 2024-04-22T15:19:01.368 INFO:tasks.workunit.client.0.smithi096.stdout:9/722: creat d1/df/ff5 x:0 0 0 2024-04-22T15:19:01.372 INFO:tasks.workunit.client.0.smithi096.stdout:0/911: symlink d7/d42/d4c/d55/d9c/df7/dfb/d122/l136 0 2024-04-22T15:19:01.374 INFO:tasks.workunit.client.0.smithi096.stdout:3/694: rename da/db/cd to da/db/de/d19/d2d/cdd 0 2024-04-22T15:19:01.378 INFO:tasks.workunit.client.0.smithi096.stdout:8/779: mknod d4/d1c/d62/dc3/dec/c100 0 2024-04-22T15:19:01.382 INFO:tasks.workunit.client.0.smithi096.stdout:8/780: truncate d4/d10/d25/d4c/db4/dd0/fe5 609402 0 2024-04-22T15:19:01.382 INFO:tasks.workunit.client.0.smithi096.stdout:1/829: getdents d6/dc/d8a/d3f/d62/d7d/ddb 0 2024-04-22T15:19:01.384 INFO:tasks.workunit.client.1.smithi196.stdout:2/566: link d1/d1d/c21 d1/d23/d27/d89/cd5 0 2024-04-22T15:19:01.386 INFO:tasks.workunit.client.1.smithi196.stdout:0/554: rmdir d3/d9/d36/d56/d65/d7f/d87/d6c/d88/d95 39 2024-04-22T15:19:01.388 INFO:tasks.workunit.client.0.smithi096.stdout:3/695: rmdir da/db/de/d19 39 2024-04-22T15:19:01.389 INFO:tasks.workunit.client.1.smithi196.stdout:4/509: link d6/d13/f24 d6/d14/d18/d26/d32/d67/d9a/f9e 0 2024-04-22T15:19:01.390 INFO:tasks.workunit.client.1.smithi196.stdout:8/530: mkdir d2/dc/d45/d98 0 2024-04-22T15:19:01.390 INFO:tasks.workunit.client.1.smithi196.stdout:5/540: link d1/d4/l64 d1/d2/d98/dac/lad 0 2024-04-22T15:19:01.391 INFO:tasks.workunit.client.0.smithi096.stdout:4/798: sync 2024-04-22T15:19:01.391 INFO:tasks.workunit.client.0.smithi096.stdout:8/781: mkdir d4/d1c/d40/d48/d101 0 2024-04-22T15:19:01.392 INFO:tasks.workunit.client.0.smithi096.stdout:8/782: dread - d4/d1c/d40/f75 zero size 2024-04-22T15:19:01.393 INFO:tasks.workunit.client.0.smithi096.stdout:1/830: creat d6/dc/d8a/d3f/d62/d7d/dd3/f106 x:0 0 0 2024-04-22T15:19:01.396 INFO:tasks.workunit.client.1.smithi196.stdout:7/498: mknod d14/d26/c9c 0 2024-04-22T15:19:01.399 INFO:tasks.workunit.client.0.smithi096.stdout:2/813: creat d3/f106 x:0 0 0 2024-04-22T15:19:01.401 INFO:tasks.workunit.client.0.smithi096.stdout:4/799: write d5/d7/d2c/d67/fe6 [235528,40252] 0 2024-04-22T15:19:01.402 INFO:tasks.workunit.client.1.smithi196.stdout:3/520: rename d0/d15/da7/d18/l33 to d0/lb0 0 2024-04-22T15:19:01.403 INFO:tasks.workunit.client.0.smithi096.stdout:8/783: dwrite d4/d10/f97 [0,4194304] 0 2024-04-22T15:19:01.405 INFO:tasks.workunit.client.0.smithi096.stdout:9/723: dread d1/df/d95/d9d/db5/dd/f1a [0,4194304] 0 2024-04-22T15:19:01.406 INFO:tasks.workunit.client.1.smithi196.stdout:2/567: mknod d1/d23/d27/d2e/db5/cd6 0 2024-04-22T15:19:01.411 INFO:tasks.workunit.client.1.smithi196.stdout:0/555: mknod d3/de/cbc 0 2024-04-22T15:19:01.413 INFO:tasks.workunit.client.0.smithi096.stdout:3/696: creat da/db/de/d19/d3e/fde x:0 0 0 2024-04-22T15:19:01.414 INFO:tasks.workunit.client.0.smithi096.stdout:3/697: chown da/db/de/d19/l39 4477 1 2024-04-22T15:19:01.418 INFO:tasks.workunit.client.1.smithi196.stdout:6/540: getdents dd/d20/d5c/d6a 0 2024-04-22T15:19:01.422 INFO:tasks.workunit.client.1.smithi196.stdout:1/492: rename d1/db/d16/d44/d6a to d1/db/dc/de/d96/d9d 0 2024-04-22T15:19:01.423 INFO:tasks.workunit.client.1.smithi196.stdout:1/493: fsync d1/db/dc/d26/f27 0 2024-04-22T15:19:01.428 INFO:tasks.workunit.client.0.smithi096.stdout:2/814: dwrite d3/db/df/f60 [0,4194304] 0 2024-04-22T15:19:01.430 INFO:tasks.workunit.client.1.smithi196.stdout:9/534: sync 2024-04-22T15:19:01.430 INFO:tasks.workunit.client.1.smithi196.stdout:8/531: readlink d2/d3f/l52 0 2024-04-22T15:19:01.431 INFO:tasks.workunit.client.0.smithi096.stdout:1/831: dread d6/dc/d8a/d3f/d62/d7d/ddb/fa7 [0,4194304] 0 2024-04-22T15:19:01.432 INFO:tasks.workunit.client.0.smithi096.stdout:9/724: dread - d1/df/f65 zero size 2024-04-22T15:19:01.434 INFO:tasks.workunit.client.0.smithi096.stdout:3/698: dread da/f8f [0,4194304] 0 2024-04-22T15:19:01.434 INFO:tasks.workunit.client.1.smithi196.stdout:2/568: creat d1/d23/d27/d2e/d55/d5b/d93/d97/fd7 x:0 0 0 2024-04-22T15:19:01.434 INFO:tasks.workunit.client.1.smithi196.stdout:9/535: dread db/dd/d1b/f79 [0,4194304] 0 2024-04-22T15:19:01.435 INFO:tasks.workunit.client.1.smithi196.stdout:2/569: stat d1/ff 0 2024-04-22T15:19:01.436 INFO:tasks.workunit.client.0.smithi096.stdout:3/699: write da/db/dc/f49 [2900498,75838] 0 2024-04-22T15:19:01.437 INFO:tasks.workunit.client.0.smithi096.stdout:0/912: link d7/de/d5b/d95/dc/d16/d65/d7c/fbd d7/de/d5b/d95/dc/d16/d18/f137 0 2024-04-22T15:19:01.440 INFO:tasks.workunit.client.1.smithi196.stdout:0/556: dread d3/de/d14/f81 [0,4194304] 0 2024-04-22T15:19:01.442 INFO:tasks.workunit.client.0.smithi096.stdout:4/800: fsync d5/fc 0 2024-04-22T15:19:01.444 INFO:tasks.workunit.client.0.smithi096.stdout:6/639: dwrite d3/d28/dd/d11/d2c/d3b/f63 [0,4194304] 0 2024-04-22T15:19:01.446 INFO:tasks.workunit.client.0.smithi096.stdout:5/820: dwrite d4/ddb/f96 [0,4194304] 0 2024-04-22T15:19:01.446 INFO:tasks.workunit.client.0.smithi096.stdout:8/784: mknod d4/d10/dcc/c102 0 2024-04-22T15:19:01.450 INFO:tasks.workunit.client.0.smithi096.stdout:2/815: dread - d3/db/df/d24/d58/d5f/d6e/fac zero size 2024-04-22T15:19:01.452 INFO:tasks.workunit.client.0.smithi096.stdout:2/816: dread - d3/db/df/d10/db5/ff0 zero size 2024-04-22T15:19:01.454 INFO:tasks.workunit.client.1.smithi196.stdout:1/494: truncate d1/db/d16/d1e/d49/f42 78847 0 2024-04-22T15:19:01.457 INFO:tasks.workunit.client.0.smithi096.stdout:9/725: write d1/d4/f42 [2575540,53108] 0 2024-04-22T15:19:01.461 INFO:tasks.workunit.client.1.smithi196.stdout:7/499: dwrite d14/d25/d31/f65 [0,4194304] 0 2024-04-22T15:19:01.462 INFO:tasks.workunit.client.0.smithi096.stdout:8/785: dread d4/d10/d4d/fa9 [0,4194304] 0 2024-04-22T15:19:01.463 INFO:tasks.workunit.client.1.smithi196.stdout:7/500: truncate d14/d15/f94 788281 0 2024-04-22T15:19:01.463 INFO:tasks.workunit.client.0.smithi096.stdout:3/700: symlink da/dab/ldf 0 2024-04-22T15:19:01.465 INFO:tasks.workunit.client.1.smithi196.stdout:7/501: truncate d14/d4a/f8c 274092 0 2024-04-22T15:19:01.466 INFO:tasks.workunit.client.0.smithi096.stdout:0/913: mkdir d7/de/d5b/d64/d138 0 2024-04-22T15:19:01.467 INFO:tasks.workunit.client.0.smithi096.stdout:0/914: chown d7/de/d5b/d95/dc/f13 1888072703 1 2024-04-22T15:19:01.469 INFO:tasks.workunit.client.1.smithi196.stdout:3/521: dwrite d0/d15/da7/d4a/d44/f16 [0,4194304] 0 2024-04-22T15:19:01.469 INFO:tasks.workunit.client.1.smithi196.stdout:0/557: unlink d3/d6/dc/c70 0 2024-04-22T15:19:01.470 INFO:tasks.workunit.client.0.smithi096.stdout:1/832: dread d6/f14 [0,4194304] 0 2024-04-22T15:19:01.471 INFO:tasks.workunit.client.1.smithi196.stdout:6/541: symlink dd/d2b/d8f/l98 0 2024-04-22T15:19:01.471 INFO:tasks.workunit.client.1.smithi196.stdout:3/522: truncate d0/d7/d8/d70/f7e 4945126 0 2024-04-22T15:19:01.480 INFO:tasks.workunit.client.1.smithi196.stdout:8/532: creat d2/d8/d49/d8b/f99 x:0 0 0 2024-04-22T15:19:01.484 INFO:tasks.workunit.client.0.smithi096.stdout:9/726: rmdir d1/df/d95 39 2024-04-22T15:19:01.485 INFO:tasks.workunit.client.0.smithi096.stdout:9/727: readlink d1/d4/d7b/lc5 0 2024-04-22T15:19:01.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:01 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:01.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:01 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:01.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:01 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:19:01.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:01 smithi096 ceph-mon[26688]: pgmap v37: 65 pgs: 65 active+clean; 2.7 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 68 MiB/s rd, 202 MiB/s wr, 400 op/s 2024-04-22T15:19:01.489 INFO:tasks.workunit.client.0.smithi096.stdout:3/701: dread da/db/de/f57 [0,4194304] 0 2024-04-22T15:19:01.490 INFO:tasks.workunit.client.1.smithi196.stdout:0/558: mkdir d3/d6/dc/d8c/d8d/d98/dbd 0 2024-04-22T15:19:01.493 INFO:tasks.workunit.client.0.smithi096.stdout:0/915: creat d7/de/d5b/d64/dc9/f139 x:0 0 0 2024-04-22T15:19:01.496 INFO:tasks.workunit.client.0.smithi096.stdout:1/833: stat d6/dc/d8a/d29/f105 0 2024-04-22T15:19:01.498 INFO:tasks.workunit.client.0.smithi096.stdout:3/702: dread da/db/d12/f56 [0,4194304] 0 2024-04-22T15:19:01.502 INFO:tasks.workunit.client.1.smithi196.stdout:7/502: symlink d14/d15/d91/d9a/l9d 0 2024-04-22T15:19:01.503 INFO:tasks.workunit.client.1.smithi196.stdout:7/503: chown d14/d26/d2a/d72/l7c 102 1 2024-04-22T15:19:01.504 INFO:tasks.workunit.client.0.smithi096.stdout:6/640: write d3/f49 [704644,32213] 0 2024-04-22T15:19:01.505 INFO:tasks.workunit.client.1.smithi196.stdout:0/559: rmdir d3/d9/d36/d56/d65/d7f/d30 39 2024-04-22T15:19:01.508 INFO:tasks.workunit.client.0.smithi096.stdout:2/817: mkdir d3/d107 0 2024-04-22T15:19:01.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:01 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:01.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:01 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:01.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:01 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:19:01.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:01 smithi196 ceph-mon[34717]: pgmap v37: 65 pgs: 65 active+clean; 2.7 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 68 MiB/s rd, 202 MiB/s wr, 400 op/s 2024-04-22T15:19:01.509 INFO:tasks.workunit.client.1.smithi196.stdout:7/504: dwrite d14/d25/d31/f8e [0,4194304] 0 2024-04-22T15:19:01.511 INFO:tasks.workunit.client.1.smithi196.stdout:4/510: rename d6/d90/f57 to d6/f9f 0 2024-04-22T15:19:01.511 INFO:tasks.workunit.client.0.smithi096.stdout:6/641: dwrite d3/d28/d54/d68/d6e/dbb/fbc [4194304,4194304] 0 2024-04-22T15:19:01.514 INFO:tasks.workunit.client.1.smithi196.stdout:3/523: mknod d0/d15/da7/d4a/d44/d96/cb1 0 2024-04-22T15:19:01.515 INFO:tasks.workunit.client.1.smithi196.stdout:3/524: chown d0/d15/f92 0 1 2024-04-22T15:19:01.516 INFO:tasks.workunit.client.1.smithi196.stdout:7/505: dwrite d14/d26/f35 [0,4194304] 0 2024-04-22T15:19:01.517 INFO:tasks.workunit.client.1.smithi196.stdout:7/506: chown d14/d26/d95/l97 1755 1 2024-04-22T15:19:01.519 INFO:tasks.workunit.client.0.smithi096.stdout:4/801: write d5/d7/f36 [813663,64728] 0 2024-04-22T15:19:01.520 INFO:tasks.workunit.client.0.smithi096.stdout:8/786: dwrite d4/f32 [0,4194304] 0 2024-04-22T15:19:01.523 INFO:tasks.workunit.client.1.smithi196.stdout:2/570: getdents d1/d23/d27/d2e 0 2024-04-22T15:19:01.523 INFO:tasks.workunit.client.0.smithi096.stdout:1/834: readlink d6/dc/d8a/d3f/d54/d32/l51 0 2024-04-22T15:19:01.524 INFO:tasks.workunit.client.1.smithi196.stdout:2/571: readlink d1/d23/d27/d2e/l4e 0 2024-04-22T15:19:01.525 INFO:tasks.workunit.client.1.smithi196.stdout:9/536: sync 2024-04-22T15:19:01.527 INFO:tasks.workunit.client.1.smithi196.stdout:8/533: dwrite d2/d5/d4f/f6e [4194304,4194304] 0 2024-04-22T15:19:01.529 INFO:tasks.workunit.client.0.smithi096.stdout:5/821: link d4/ddb/d3d/d35/f63 d4/ddb/d3d/d4d/d59/d98/f114 0 2024-04-22T15:19:01.529 INFO:tasks.workunit.client.1.smithi196.stdout:0/560: unlink d3/d9/d36/f7e 0 2024-04-22T15:19:01.530 INFO:tasks.workunit.client.1.smithi196.stdout:5/541: rename d1/d17/d1a/ca5 to d1/d78/cae 0 2024-04-22T15:19:01.530 INFO:tasks.workunit.client.1.smithi196.stdout:0/561: write d3/d9/f54 [2303139,75611] 0 2024-04-22T15:19:01.531 INFO:tasks.workunit.client.1.smithi196.stdout:5/542: chown d1/d78 62281 1 2024-04-22T15:19:01.533 INFO:tasks.workunit.client.1.smithi196.stdout:5/543: stat d1/d17/d1a/d2f/f99 0 2024-04-22T15:19:01.536 INFO:tasks.workunit.client.0.smithi096.stdout:9/728: creat d1/df/d95/d9d/db5/dd/dba/dc2/ff6 x:0 0 0 2024-04-22T15:19:01.538 INFO:tasks.workunit.client.1.smithi196.stdout:1/495: getdents d1/db/dc/de/d43/d5e 0 2024-04-22T15:19:01.539 INFO:tasks.workunit.client.0.smithi096.stdout:4/802: rmdir d5/d5b 39 2024-04-22T15:19:01.542 INFO:tasks.workunit.client.1.smithi196.stdout:2/572: creat d1/d23/d27/d2e/fd8 x:0 0 0 2024-04-22T15:19:01.543 INFO:tasks.workunit.client.0.smithi096.stdout:8/787: rename d4/d1c/d40/da7/db6/l6f to d4/d2a/d7e/l103 0 2024-04-22T15:19:01.544 INFO:tasks.workunit.client.1.smithi196.stdout:9/537: symlink db/dd/lb9 0 2024-04-22T15:19:01.545 INFO:tasks.workunit.client.1.smithi196.stdout:9/538: fdatasync db/dd/d1b/d2c/f9f 0 2024-04-22T15:19:01.546 INFO:tasks.workunit.client.0.smithi096.stdout:3/703: mknod da/db/de/ddc/d68/db0/ce0 0 2024-04-22T15:19:01.547 INFO:tasks.workunit.client.1.smithi196.stdout:8/534: read d2/f32 [495601,86685] 0 2024-04-22T15:19:01.552 INFO:tasks.workunit.client.0.smithi096.stdout:8/788: symlink d4/d10/d25/d4c/db4/dd0/l104 0 2024-04-22T15:19:01.556 INFO:tasks.workunit.client.0.smithi096.stdout:3/704: mknod da/db/de/d19/d3e/ce1 0 2024-04-22T15:19:01.558 INFO:tasks.workunit.client.0.smithi096.stdout:6/642: link d3/d3a/d75/d83/ccf d3/d28/d54/d68/d6e/cdb 0 2024-04-22T15:19:01.558 INFO:tasks.workunit.client.0.smithi096.stdout:3/705: chown da/dab/l95 321856 1 2024-04-22T15:19:01.559 INFO:tasks.workunit.client.1.smithi196.stdout:1/496: fsync d1/db/dc/d26/f28 0 2024-04-22T15:19:01.560 INFO:tasks.workunit.client.0.smithi096.stdout:3/706: dread - da/db/dc/d59/fd7 zero size 2024-04-22T15:19:01.561 INFO:tasks.workunit.client.1.smithi196.stdout:1/497: fdatasync d1/db/dc/d26/d2e/d32/d36/f3f 0 2024-04-22T15:19:01.562 INFO:tasks.workunit.client.0.smithi096.stdout:3/707: chown da/db/dc/d28/d69/c7c 8300 1 2024-04-22T15:19:01.562 INFO:tasks.workunit.client.0.smithi096.stdout:9/729: symlink d1/df/d95/d9d/db5/d43/lf7 0 2024-04-22T15:19:01.564 INFO:tasks.workunit.client.0.smithi096.stdout:2/818: dread d3/db/df/d10/f37 [0,4194304] 0 2024-04-22T15:19:01.564 INFO:tasks.workunit.client.0.smithi096.stdout:9/730: dread - d1/df/d95/fd4 zero size 2024-04-22T15:19:01.565 INFO:tasks.workunit.client.0.smithi096.stdout:0/916: getdents d7/d42/d4c/d55 0 2024-04-22T15:19:01.566 INFO:tasks.workunit.client.0.smithi096.stdout:0/917: fdatasync d7/d42/dfd/f128 0 2024-04-22T15:19:01.568 INFO:tasks.workunit.client.1.smithi196.stdout:9/539: dread db/dd/f5c [0,4194304] 0 2024-04-22T15:19:01.569 INFO:tasks.workunit.client.1.smithi196.stdout:9/540: chown db/dd/de/d15/d31/f63 76345 1 2024-04-22T15:19:01.570 INFO:tasks.workunit.client.0.smithi096.stdout:8/789: rename d4/d10/d25/d4c/db4/dd0/fe5 to d4/d1c/d40/d48/f105 0 2024-04-22T15:19:01.575 INFO:tasks.workunit.client.1.smithi196.stdout:8/535: readlink d2/l30 0 2024-04-22T15:19:01.578 INFO:tasks.workunit.client.0.smithi096.stdout:6/643: dread d3/d28/f38 [0,4194304] 0 2024-04-22T15:19:01.579 INFO:tasks.workunit.client.1.smithi196.stdout:4/511: creat d6/d13/d15/d2c/fa0 x:0 0 0 2024-04-22T15:19:01.581 INFO:tasks.workunit.client.0.smithi096.stdout:3/708: dread - da/d3f/f9f zero size 2024-04-22T15:19:01.582 INFO:tasks.workunit.client.0.smithi096.stdout:3/709: write da/db/de/ddc/d26/db2/fc2 [380880,108792] 0 2024-04-22T15:19:01.583 INFO:tasks.workunit.client.0.smithi096.stdout:2/819: rmdir d3/db/df/d24/dc8 39 2024-04-22T15:19:01.585 INFO:tasks.workunit.client.0.smithi096.stdout:5/822: write d4/ddb/f62 [888719,46672] 0 2024-04-22T15:19:01.587 INFO:tasks.workunit.client.0.smithi096.stdout:9/731: creat d1/df/d95/d9d/db5/d43/d57/d5b/da1/ff8 x:0 0 0 2024-04-22T15:19:01.591 INFO:tasks.workunit.client.1.smithi196.stdout:3/525: truncate d0/d15/d48/d51/fa9 707999 0 2024-04-22T15:19:01.593 INFO:tasks.workunit.client.0.smithi096.stdout:8/790: creat d4/d71/f106 x:0 0 0 2024-04-22T15:19:01.593 INFO:tasks.workunit.client.0.smithi096.stdout:1/835: write d6/f1d [240367,35976] 0 2024-04-22T15:19:01.594 INFO:tasks.workunit.client.1.smithi196.stdout:1/498: creat d1/db/d16/d44/f9e x:0 0 0 2024-04-22T15:19:01.600 INFO:tasks.workunit.client.0.smithi096.stdout:2/820: read d3/d1f/d2f/f4e [374923,22627] 0 2024-04-22T15:19:01.601 INFO:tasks.workunit.client.1.smithi196.stdout:6/542: rename lb to dd/d1f/d37/l99 0 2024-04-22T15:19:01.602 INFO:tasks.workunit.client.1.smithi196.stdout:6/543: readlink dd/d1f/d37/l3b 0 2024-04-22T15:19:01.603 INFO:tasks.workunit.client.1.smithi196.stdout:2/573: stat d1/d23/d27/d2e/d55/d5b/ca0 0 2024-04-22T15:19:01.606 INFO:tasks.workunit.client.0.smithi096.stdout:5/823: dread d4/ddb/d3d/d4d/d59/d5a/f13 [0,4194304] 0 2024-04-22T15:19:01.606 INFO:tasks.workunit.client.1.smithi196.stdout:4/512: creat d6/d14/d18/d40/fa1 x:0 0 0 2024-04-22T15:19:01.608 INFO:tasks.workunit.client.0.smithi096.stdout:8/791: creat d4/d10/dcc/f107 x:0 0 0 2024-04-22T15:19:01.609 INFO:tasks.workunit.client.0.smithi096.stdout:8/792: readlink d4/d10/l59 0 2024-04-22T15:19:01.609 INFO:tasks.workunit.client.1.smithi196.stdout:5/544: creat d1/d2/faf x:0 0 0 2024-04-22T15:19:01.610 INFO:tasks.workunit.client.0.smithi096.stdout:1/836: truncate d6/dc/f23 1133171 0 2024-04-22T15:19:01.610 INFO:tasks.workunit.client.1.smithi196.stdout:7/507: link d14/d26/d5a/d5f/d6e/l98 d14/d15/d91/d9a/l9e 0 2024-04-22T15:19:01.612 INFO:tasks.workunit.client.1.smithi196.stdout:7/508: chown d14/d26/d2a/d2b/f70 30493653 1 2024-04-22T15:19:01.612 INFO:tasks.workunit.client.0.smithi096.stdout:1/837: readlink d6/dc/d8a/d3f/d54/d32/dcd/lf2 0 2024-04-22T15:19:01.613 INFO:tasks.workunit.client.1.smithi196.stdout:3/526: creat d0/d15/d31/d55/d6e/fb2 x:0 0 0 2024-04-22T15:19:01.614 INFO:tasks.workunit.client.0.smithi096.stdout:3/710: getdents da/db/dc/d28/dda 0 2024-04-22T15:19:01.614 INFO:tasks.workunit.client.1.smithi196.stdout:3/527: dread - d0/d15/da7/d85/f9c zero size 2024-04-22T15:19:01.615 INFO:tasks.workunit.client.1.smithi196.stdout:1/499: mkdir d1/db/dc/de/d43/d5e/d7a/d9f 0 2024-04-22T15:19:01.617 INFO:tasks.workunit.client.0.smithi096.stdout:2/821: unlink d3/d1f/d2f/l5b 0 2024-04-22T15:19:01.617 INFO:tasks.workunit.client.0.smithi096.stdout:1/838: dread d6/dc/d8a/d3f/d62/d7d/dd3/fd6 [0,4194304] 0 2024-04-22T15:19:01.618 INFO:tasks.workunit.client.1.smithi196.stdout:4/513: dread d6/d14/d18/d26/f3a [0,4194304] 0 2024-04-22T15:19:01.619 INFO:tasks.workunit.client.1.smithi196.stdout:9/541: creat db/dd/de/d15/d31/d64/dab/fba x:0 0 0 2024-04-22T15:19:01.621 INFO:tasks.workunit.client.0.smithi096.stdout:9/732: truncate d1/df/d95/d9d/db5/d43/f9c 807699 0 2024-04-22T15:19:01.622 INFO:tasks.workunit.client.1.smithi196.stdout:8/536: truncate d2/d3f/f4e 2355843 0 2024-04-22T15:19:01.623 INFO:tasks.workunit.client.1.smithi196.stdout:9/542: dread db/dd/de/f42 [0,4194304] 0 2024-04-22T15:19:01.624 INFO:tasks.workunit.client.1.smithi196.stdout:9/543: stat db/d11/d29/d7a/f7e 0 2024-04-22T15:19:01.626 INFO:tasks.workunit.client.0.smithi096.stdout:2/822: dwrite d3/db/df/d24/d58/d5f/d6e/fb8 [0,4194304] 0 2024-04-22T15:19:01.626 INFO:tasks.workunit.client.1.smithi196.stdout:0/562: rename d3/d9/d36/d56/d65/d7f/d51/f75 to d3/d6/dc/d8c/d8d/da8/fbe 0 2024-04-22T15:19:01.627 INFO:tasks.workunit.client.1.smithi196.stdout:0/563: chown d3/d6/dc/d8c/d8d/da8 235 1 2024-04-22T15:19:01.627 INFO:tasks.workunit.client.0.smithi096.stdout:8/793: dread d4/d10/f19 [0,4194304] 0 2024-04-22T15:19:01.629 INFO:tasks.workunit.client.0.smithi096.stdout:2/823: write d3/d13/fe5 [485763,110056] 0 2024-04-22T15:19:01.631 INFO:tasks.workunit.client.0.smithi096.stdout:6/644: truncate d3/d28/dd/d11/d2c/f56 2099952 0 2024-04-22T15:19:01.634 INFO:tasks.workunit.client.1.smithi196.stdout:0/564: dwrite d3/f99 [0,4194304] 0 2024-04-22T15:19:01.646 INFO:tasks.workunit.client.0.smithi096.stdout:5/824: dread d4/d2c/f4e [0,4194304] 0 2024-04-22T15:19:01.760 INFO:tasks.workunit.client.1.smithi196.stdout:7/509: rmdir d14/d25 39 2024-04-22T15:19:01.760 INFO:tasks.workunit.client.1.smithi196.stdout:7/510: stat d14/c87 0 2024-04-22T15:19:01.761 INFO:tasks.workunit.client.1.smithi196.stdout:7/511: write d14/d26/d2a/d2b/f77 [340812,102157] 0 2024-04-22T15:19:01.762 INFO:tasks.workunit.client.1.smithi196.stdout:3/528: mknod d0/d7/d6f/cb3 0 2024-04-22T15:19:01.764 INFO:tasks.workunit.client.0.smithi096.stdout:9/733: mknod d1/df/d95/cf9 0 2024-04-22T15:19:01.765 INFO:tasks.workunit.client.1.smithi196.stdout:1/500: creat d1/db/dc/de/d43/d5e/d7a/fa0 x:0 0 0 2024-04-22T15:19:01.765 INFO:tasks.workunit.client.1.smithi196.stdout:1/501: chown d1/db/dc/de/d43 1 1 2024-04-22T15:19:01.766 INFO:tasks.workunit.client.1.smithi196.stdout:1/502: write d1/d54/d70/f89 [244874,38936] 0 2024-04-22T15:19:01.768 INFO:tasks.workunit.client.1.smithi196.stdout:1/503: read d1/db/d16/d1e/d65/f76 [3675886,29740] 0 2024-04-22T15:19:01.768 INFO:tasks.workunit.client.1.smithi196.stdout:4/514: unlink d6/d13/d15/l64 0 2024-04-22T15:19:01.773 INFO:tasks.workunit.client.0.smithi096.stdout:9/734: dwrite d1/df/d95/d9d/db5/d43/d57/d5b/da1/ff8 [0,4194304] 0 2024-04-22T15:19:01.781 INFO:tasks.workunit.client.0.smithi096.stdout:2/824: creat d3/db/df/d24/d58/d5e/f108 x:0 0 0 2024-04-22T15:19:01.781 INFO:tasks.workunit.client.0.smithi096.stdout:1/839: dread d6/dc/d8a/d3f/f57 [0,4194304] 0 2024-04-22T15:19:01.781 INFO:tasks.workunit.client.0.smithi096.stdout:8/794: dwrite d4/d10/d25/d38/d8f/fd7 [0,4194304] 0 2024-04-22T15:19:01.782 INFO:tasks.workunit.client.1.smithi196.stdout:3/529: dread d0/d15/d4d/f5a [0,4194304] 0 2024-04-22T15:19:01.783 INFO:tasks.workunit.client.1.smithi196.stdout:3/530: truncate d0/d7/d8/f77 1488270 0 2024-04-22T15:19:01.787 INFO:tasks.workunit.client.0.smithi096.stdout:1/840: dread - d6/dc/d8a/d3f/d54/d32/fed zero size 2024-04-22T15:19:01.788 INFO:tasks.workunit.client.1.smithi196.stdout:5/545: rename d1/d51 to d1/d78/db0 0 2024-04-22T15:19:01.788 INFO:tasks.workunit.client.1.smithi196.stdout:7/512: rename d14 to d14/d26/d2a/d72/d9f 22 2024-04-22T15:19:01.789 INFO:tasks.workunit.client.1.smithi196.stdout:7/513: stat d14 0 2024-04-22T15:19:01.789 INFO:tasks.workunit.client.1.smithi196.stdout:5/546: dread - d1/d14/d3e/f4e zero size 2024-04-22T15:19:01.791 INFO:tasks.workunit.client.0.smithi096.stdout:5/825: creat d4/ddb/d3d/d4f/f115 x:0 0 0 2024-04-22T15:19:01.792 INFO:tasks.workunit.client.0.smithi096.stdout:1/841: stat d6/dc/d8a/d3f/d62/d7d/ddb/fa7 0 2024-04-22T15:19:01.792 INFO:tasks.workunit.client.0.smithi096.stdout:3/711: dwrite da/db/d12/f56 [0,4194304] 0 2024-04-22T15:19:01.792 INFO:tasks.workunit.client.0.smithi096.stdout:8/795: read d4/d1c/d40/da7/db6/f49 [1749690,56229] 0 2024-04-22T15:19:01.797 INFO:tasks.workunit.client.1.smithi196.stdout:7/514: dwrite d14/d26/d5a/f8f [0,4194304] 0 2024-04-22T15:19:01.802 INFO:tasks.workunit.client.1.smithi196.stdout:0/565: creat d3/d9/d36/d56/d65/d7f/d87/d6c/fbf x:0 0 0 2024-04-22T15:19:01.802 INFO:tasks.workunit.client.0.smithi096.stdout:3/712: dwrite da/db/de/ddc/d26/d52/fd4 [0,4194304] 0 2024-04-22T15:19:01.812 INFO:tasks.workunit.client.0.smithi096.stdout:6/645: rename d3/d28/dd/d11/l5c to d3/d28/d4d/d82/d9e/db0/ldc 0 2024-04-22T15:19:01.813 INFO:tasks.workunit.client.0.smithi096.stdout:6/646: dread - d3/d3a/d75/fd3 zero size 2024-04-22T15:19:01.817 INFO:tasks.workunit.client.1.smithi196.stdout:1/504: creat d1/db/dc/de/d43/d5e/d7a/fa1 x:0 0 0 2024-04-22T15:19:01.818 INFO:tasks.workunit.client.1.smithi196.stdout:1/505: chown d1/db/d16/f7e 757 1 2024-04-22T15:19:01.819 INFO:tasks.workunit.client.1.smithi196.stdout:1/506: write d1/db/d16/d1e/f60 [4960477,115421] 0 2024-04-22T15:19:01.820 INFO:tasks.workunit.client.1.smithi196.stdout:4/515: symlink d6/d14/d18/d26/d32/d53/la2 0 2024-04-22T15:19:01.823 INFO:tasks.workunit.client.1.smithi196.stdout:8/537: mknod d2/d3f/d4b/d79/c9a 0 2024-04-22T15:19:01.823 INFO:tasks.workunit.client.0.smithi096.stdout:8/796: rename d4/d1c/d40/da7/db6/l51 to d4/d1c/d40/da7/db6/l108 0 2024-04-22T15:19:01.824 INFO:tasks.workunit.client.1.smithi196.stdout:3/531: readlink d0/lb0 0 2024-04-22T15:19:01.824 INFO:tasks.workunit.client.1.smithi196.stdout:3/532: chown d0/d15/d31/d9d/d9e/fab 11851 1 2024-04-22T15:19:01.826 INFO:tasks.workunit.client.0.smithi096.stdout:9/735: dread d1/df/d95/d9d/db5/d43/d57/d5b/f66 [0,4194304] 0 2024-04-22T15:19:01.827 INFO:tasks.workunit.client.1.smithi196.stdout:5/547: symlink d1/d17/d1a/d2f/d30/d6e/lb1 0 2024-04-22T15:19:01.828 INFO:tasks.workunit.client.0.smithi096.stdout:3/713: dread da/f34 [0,4194304] 0 2024-04-22T15:19:01.829 INFO:tasks.workunit.client.0.smithi096.stdout:6/647: mkdir d3/d28/dd/d11/d2c/d6f/dd4/ddd 0 2024-04-22T15:19:01.829 INFO:tasks.workunit.client.0.smithi096.stdout:3/714: fdatasync da/f55 0 2024-04-22T15:19:01.832 INFO:tasks.workunit.client.1.smithi196.stdout:6/544: creat dd/d1f/d4e/d53/d55/f9a x:0 0 0 2024-04-22T15:19:01.832 INFO:tasks.workunit.client.1.smithi196.stdout:6/545: fdatasync dd/d1f/d4e/d53/d55/f8b 0 2024-04-22T15:19:01.833 INFO:tasks.workunit.client.1.smithi196.stdout:2/574: link d1/d13/d1f/la3 d1/da4/ld9 0 2024-04-22T15:19:01.833 INFO:tasks.workunit.client.0.smithi096.stdout:9/736: dread d1/df/d95/d9d/db5/d43/d57/d5b/da1/ff8 [0,4194304] 0 2024-04-22T15:19:01.834 INFO:tasks.workunit.client.1.smithi196.stdout:6/546: write dd/d2b/f62 [1511895,20827] 0 2024-04-22T15:19:01.835 INFO:tasks.workunit.client.1.smithi196.stdout:6/547: dread - dd/d1f/d4e/d53/d55/f9a zero size 2024-04-22T15:19:01.835 INFO:tasks.workunit.client.0.smithi096.stdout:8/797: truncate d4/dc0/fd3 227577 0 2024-04-22T15:19:01.836 INFO:tasks.workunit.client.0.smithi096.stdout:1/842: dread d6/dc/d8a/d3f/d62/dc2/dd0/fff [0,4194304] 0 2024-04-22T15:19:01.836 INFO:tasks.workunit.client.0.smithi096.stdout:8/798: stat d4/d1c/d40/c65 0 2024-04-22T15:19:01.836 INFO:tasks.workunit.client.1.smithi196.stdout:7/515: symlink d14/d26/d2a/d2b/la0 0 2024-04-22T15:19:01.837 INFO:tasks.workunit.client.0.smithi096.stdout:8/799: chown d4/d2a/d7e/fa6 6 1 2024-04-22T15:19:01.838 INFO:tasks.workunit.client.1.smithi196.stdout:6/548: dread f9 [0,4194304] 0 2024-04-22T15:19:01.841 INFO:tasks.workunit.client.0.smithi096.stdout:3/715: symlink da/db/dc/d28/le2 0 2024-04-22T15:19:01.844 INFO:tasks.workunit.client.0.smithi096.stdout:2/825: link d3/db/df/d10/lf8 d3/d13/db4/l109 0 2024-04-22T15:19:01.845 INFO:tasks.workunit.client.0.smithi096.stdout:6/648: mknod d3/d28/d4d/d82/cde 0 2024-04-22T15:19:01.847 INFO:tasks.workunit.client.0.smithi096.stdout:9/737: rmdir d1/df/d95/d9d/db5/d43/d57/d5b 39 2024-04-22T15:19:01.848 INFO:tasks.workunit.client.0.smithi096.stdout:5/826: write d4/ddb/d3d/d4d/f70 [59091,29470] 0 2024-04-22T15:19:01.853 INFO:tasks.workunit.client.0.smithi096.stdout:8/800: rmdir d4/d1c/d62 39 2024-04-22T15:19:01.854 INFO:tasks.workunit.client.0.smithi096.stdout:8/801: chown d4/d1c/d40/d48 30745355 1 2024-04-22T15:19:01.855 INFO:tasks.workunit.client.1.smithi196.stdout:3/533: symlink d0/d15/da7/d4a/d44/d8c/lb4 0 2024-04-22T15:19:01.855 INFO:tasks.workunit.client.1.smithi196.stdout:5/548: mkdir d1/d17/d1a/d2f/d30/d6e/db2 0 2024-04-22T15:19:01.856 INFO:tasks.workunit.client.1.smithi196.stdout:6/549: write dd/d20/d5c/d5e/d76/f85 [941784,130449] 0 2024-04-22T15:19:01.856 INFO:tasks.workunit.client.1.smithi196.stdout:9/544: truncate db/dd/f5c 1865440 0 2024-04-22T15:19:01.857 INFO:tasks.workunit.client.1.smithi196.stdout:6/550: stat f2 0 2024-04-22T15:19:01.857 INFO:tasks.workunit.client.1.smithi196.stdout:3/534: truncate d0/d15/f92 1003444 0 2024-04-22T15:19:01.857 INFO:tasks.workunit.client.1.smithi196.stdout:5/549: write d1/d17/d1c/f88 [471650,75082] 0 2024-04-22T15:19:01.858 INFO:tasks.workunit.client.1.smithi196.stdout:9/545: write db/dd/de/d15/f40 [7422813,110006] 0 2024-04-22T15:19:01.859 INFO:tasks.workunit.client.1.smithi196.stdout:9/546: stat db/d11/d29 0 2024-04-22T15:19:01.860 INFO:tasks.workunit.client.0.smithi096.stdout:3/716: symlink da/db/dc/d8d/le3 0 2024-04-22T15:19:01.860 INFO:tasks.workunit.client.0.smithi096.stdout:8/802: dwrite d4/d71/f106 [0,4194304] 0 2024-04-22T15:19:01.865 INFO:tasks.workunit.client.0.smithi096.stdout:2/826: write d3/db/f83 [3561295,44691] 0 2024-04-22T15:19:01.869 INFO:tasks.workunit.client.1.smithi196.stdout:2/575: rename d1/d23/d27/d2e/d55/d5b/d93/d97/fd7 to d1/d23/d27/d73/fda 0 2024-04-22T15:19:01.870 INFO:tasks.workunit.client.0.smithi096.stdout:6/649: rmdir d3/d28/d4d/d82/d9e 39 2024-04-22T15:19:01.870 INFO:tasks.workunit.client.1.smithi196.stdout:2/576: truncate d1/d23/d27/d2e/d55/d5b/d7c/f7f 776317 0 2024-04-22T15:19:01.871 INFO:tasks.workunit.client.1.smithi196.stdout:7/516: creat d14/d26/d28/fa1 x:0 0 0 2024-04-22T15:19:01.874 INFO:tasks.workunit.client.1.smithi196.stdout:0/566: link d3/d9/d36/d56/d65/d7f/d87/d6c/d88/fa3 d3/d9/d36/d56/d65/d7f/d87/d6c/fc0 0 2024-04-22T15:19:01.881 INFO:tasks.workunit.client.0.smithi096.stdout:3/717: dread da/dab/fa2 [0,4194304] 0 2024-04-22T15:19:01.882 INFO:tasks.workunit.client.0.smithi096.stdout:3/718: stat da/db/d9a 0 2024-04-22T15:19:01.883 INFO:tasks.workunit.client.0.smithi096.stdout:3/719: chown da/d3f 2 1 2024-04-22T15:19:01.884 INFO:tasks.workunit.client.0.smithi096.stdout:3/720: chown da/db/de/ddc/d68/dbc 1722432175 1 2024-04-22T15:19:01.885 INFO:tasks.workunit.client.1.smithi196.stdout:6/551: creat dd/d20/d5c/d5e/f9b x:0 0 0 2024-04-22T15:19:01.890 INFO:tasks.workunit.client.0.smithi096.stdout:1/843: dwrite d6/dc/d8a/f7a [4194304,4194304] 0 2024-04-22T15:19:01.901 INFO:tasks.workunit.client.1.smithi196.stdout:8/538: dwrite d2/f1d [0,4194304] 0 2024-04-22T15:19:01.902 INFO:tasks.workunit.client.0.smithi096.stdout:2/827: symlink d3/d1f/l10a 0 2024-04-22T15:19:01.903 INFO:tasks.workunit.client.1.smithi196.stdout:5/550: creat d1/d17/d1a/d2f/d30/fb3 x:0 0 0 2024-04-22T15:19:01.905 INFO:tasks.workunit.client.0.smithi096.stdout:6/650: creat d3/d28/d54/d68/d6e/fdf x:0 0 0 2024-04-22T15:19:01.906 INFO:tasks.workunit.client.0.smithi096.stdout:5/827: truncate d4/f26 2091833 0 2024-04-22T15:19:01.907 INFO:tasks.workunit.client.0.smithi096.stdout:8/803: dwrite d4/d1c/d40/d46/f7d [0,4194304] 0 2024-04-22T15:19:01.907 INFO:tasks.workunit.client.0.smithi096.stdout:6/651: chown d3/d28/fbd 2210823 1 2024-04-22T15:19:01.913 INFO:tasks.workunit.client.0.smithi096.stdout:9/738: fsync d1/df/d95/d9d/db5/d43/d57/d5b/f84 0 2024-04-22T15:19:01.919 INFO:tasks.workunit.client.1.smithi196.stdout:2/577: dread d1/f1b [0,4194304] 0 2024-04-22T15:19:01.920 INFO:tasks.workunit.client.1.smithi196.stdout:2/578: fsync d1/d13/d25/f3d 0 2024-04-22T15:19:01.925 INFO:tasks.workunit.client.1.smithi196.stdout:0/567: creat d3/d6/dc/d8c/d8d/fc1 x:0 0 0 2024-04-22T15:19:01.931 INFO:tasks.workunit.client.0.smithi096.stdout:3/721: rename da/db/de/d19/d3e/c5a to da/db/d12/dd5/ce4 0 2024-04-22T15:19:01.935 INFO:tasks.workunit.client.1.smithi196.stdout:9/547: dwrite db/d11/f4d [0,4194304] 0 2024-04-22T15:19:01.942 INFO:tasks.workunit.client.1.smithi196.stdout:9/548: dwrite db/dd/de/d15/f73 [0,4194304] 0 2024-04-22T15:19:01.952 INFO:tasks.workunit.client.0.smithi096.stdout:1/844: mkdir d6/dc/d8a/d3f/d54/d32/d3b/d56/d107 0 2024-04-22T15:19:01.953 INFO:tasks.workunit.client.0.smithi096.stdout:1/845: chown d6/dc/d8a/d3f/d54/d32/d8c/d45 480 1 2024-04-22T15:19:01.955 INFO:tasks.workunit.client.0.smithi096.stdout:6/652: write d3/f49 [2224103,53209] 0 2024-04-22T15:19:01.955 INFO:tasks.workunit.client.1.smithi196.stdout:8/539: creat d2/d3f/d4b/f9b x:0 0 0 2024-04-22T15:19:01.956 INFO:tasks.workunit.client.1.smithi196.stdout:8/540: fsync d2/d6a/d7b/f94 0 2024-04-22T15:19:01.956 INFO:tasks.workunit.client.1.smithi196.stdout:8/541: read - d2/d35/f92 zero size 2024-04-22T15:19:01.957 INFO:tasks.workunit.client.1.smithi196.stdout:5/551: unlink d1/d17/d7f/fa6 0 2024-04-22T15:19:01.957 INFO:tasks.workunit.client.1.smithi196.stdout:5/552: chown d1/d17/d50 36885832 1 2024-04-22T15:19:01.958 INFO:tasks.workunit.client.1.smithi196.stdout:2/579: dread - d1/d23/d27/f44 zero size 2024-04-22T15:19:01.959 INFO:tasks.workunit.client.1.smithi196.stdout:0/568: chown d3/de/l8b 885 1 2024-04-22T15:19:01.960 INFO:tasks.workunit.client.0.smithi096.stdout:3/722: readlink da/db/de/d87/lca 0 2024-04-22T15:19:01.961 INFO:tasks.workunit.client.0.smithi096.stdout:3/723: chown da/db/de/ddc/d26/db2/cbb 1347659 1 2024-04-22T15:19:01.963 INFO:tasks.workunit.client.1.smithi196.stdout:5/553: dwrite d1/d17/d1a/d2f/d30/f6b [0,4194304] 0 2024-04-22T15:19:01.963 INFO:tasks.workunit.client.1.smithi196.stdout:3/535: rmdir d0/d15/d31/d9d/da2 0 2024-04-22T15:19:01.964 INFO:tasks.workunit.client.0.smithi096.stdout:1/846: creat d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/f108 x:0 0 0 2024-04-22T15:19:01.965 INFO:tasks.workunit.client.1.smithi196.stdout:3/536: truncate d0/d15/da7/d85/f9c 243670 0 2024-04-22T15:19:01.967 INFO:tasks.workunit.client.1.smithi196.stdout:9/549: creat db/d11/d29/d7a/fbb x:0 0 0 2024-04-22T15:19:01.967 INFO:tasks.workunit.client.1.smithi196.stdout:1/507: sync 2024-04-22T15:19:01.971 INFO:tasks.workunit.client.0.smithi096.stdout:5/828: symlink d4/l116 0 2024-04-22T15:19:01.971 INFO:tasks.workunit.client.0.smithi096.stdout:1/847: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d56/fb9 [0,4194304] 0 2024-04-22T15:19:01.976 INFO:tasks.workunit.client.1.smithi196.stdout:3/537: dwrite d0/d15/d31/d55/d6e/fb2 [0,4194304] 0 2024-04-22T15:19:01.978 INFO:tasks.workunit.client.1.smithi196.stdout:9/550: dwrite db/f5f [0,4194304] 0 2024-04-22T15:19:01.978 INFO:tasks.workunit.client.1.smithi196.stdout:1/508: dwrite d1/d54/d70/f89 [0,4194304] 0 2024-04-22T15:19:01.980 INFO:tasks.workunit.client.0.smithi096.stdout:9/739: creat d1/d85/df4/ffa x:0 0 0 2024-04-22T15:19:01.983 INFO:tasks.workunit.client.1.smithi196.stdout:2/580: mkdir d1/d13/d25/d2a/ddb 0 2024-04-22T15:19:01.984 INFO:tasks.workunit.client.1.smithi196.stdout:1/509: write d1/db/d16/d44/f9e [512731,25167] 0 2024-04-22T15:19:01.984 INFO:tasks.workunit.client.1.smithi196.stdout:0/569: creat d3/d6/dc/fc2 x:0 0 0 2024-04-22T15:19:01.986 INFO:tasks.workunit.client.0.smithi096.stdout:3/724: unlink da/db/de/d19/d3e/f4f 0 2024-04-22T15:19:01.986 INFO:tasks.workunit.client.0.smithi096.stdout:3/725: dread da/f8a [0,4194304] 0 2024-04-22T15:19:01.988 INFO:tasks.workunit.client.0.smithi096.stdout:3/726: chown da/db/de/d19/d3e/ce1 7 1 2024-04-22T15:19:01.989 INFO:tasks.workunit.client.0.smithi096.stdout:3/727: stat da/db/dc/d28/d69/d89 0 2024-04-22T15:19:01.990 INFO:tasks.workunit.client.0.smithi096.stdout:6/653: mkdir d3/d28/dd/de0 0 2024-04-22T15:19:01.991 INFO:tasks.workunit.client.1.smithi196.stdout:1/510: dread d1/db/dc/de/d43/f6b [0,4194304] 0 2024-04-22T15:19:01.991 INFO:tasks.workunit.client.0.smithi096.stdout:6/654: dread - d3/d28/d54/d68/d6e/fdf zero size 2024-04-22T15:19:01.995 INFO:tasks.workunit.client.1.smithi196.stdout:8/542: fsync d2/f3e 0 2024-04-22T15:19:01.998 INFO:tasks.workunit.client.0.smithi096.stdout:9/740: fsync d1/df/d95/d9d/db5/dd/f40 0 2024-04-22T15:19:02.001 INFO:tasks.workunit.client.1.smithi196.stdout:7/517: link d14/d25/d31/f69 d14/d26/fa2 0 2024-04-22T15:19:02.002 INFO:tasks.workunit.client.0.smithi096.stdout:2/828: rename d3/d1f/d5a/cd0 to d3/db/d30/c10b 0 2024-04-22T15:19:02.003 INFO:tasks.workunit.client.1.smithi196.stdout:3/538: mkdir d0/d7/d6f/db5 0 2024-04-22T15:19:02.003 INFO:tasks.workunit.client.1.smithi196.stdout:5/554: dread d1/d17/d1a/d2f/f99 [0,4194304] 0 2024-04-22T15:19:02.004 INFO:tasks.workunit.client.0.smithi096.stdout:2/829: write d3/db/df/dce/d46/d7e/d8d/fe6 [966562,33487] 0 2024-04-22T15:19:02.004 INFO:tasks.workunit.client.1.smithi196.stdout:3/539: dread - d0/d15/da7/d85/f91 zero size 2024-04-22T15:19:02.005 INFO:tasks.workunit.client.0.smithi096.stdout:8/804: getdents d4/d10/d4d 0 2024-04-22T15:19:02.005 INFO:tasks.workunit.client.1.smithi196.stdout:3/540: fdatasync d0/d15/da7/d4a/f86 0 2024-04-22T15:19:02.006 INFO:tasks.workunit.client.1.smithi196.stdout:3/541: readlink d0/d15/d4d/l3f 0 2024-04-22T15:19:02.007 INFO:tasks.workunit.client.1.smithi196.stdout:3/542: fdatasync d0/d15/f92 0 2024-04-22T15:19:02.008 INFO:tasks.workunit.client.1.smithi196.stdout:3/543: dread - d0/d15/d31/d55/d5d/fa0 zero size 2024-04-22T15:19:02.010 INFO:tasks.workunit.client.0.smithi096.stdout:5/829: symlink d4/ddb/d3d/d4d/d59/d98/dee/l117 0 2024-04-22T15:19:02.013 INFO:tasks.workunit.client.1.smithi196.stdout:3/544: dread d0/d7/f38 [0,4194304] 0 2024-04-22T15:19:02.015 INFO:tasks.workunit.client.1.smithi196.stdout:8/543: unlink d2/d6a/d7b/l84 0 2024-04-22T15:19:02.017 INFO:tasks.workunit.client.0.smithi096.stdout:1/848: rename d6/dc/d8a/d3f/d54/c42 to d6/dc/d8a/d3f/d62/d7d/dd3/c109 0 2024-04-22T15:19:02.017 INFO:tasks.workunit.client.0.smithi096.stdout:1/849: readlink d6/dc/d8a/l2d 0 2024-04-22T15:19:02.018 INFO:tasks.workunit.client.1.smithi196.stdout:7/518: mkdir d14/d4d/da3 0 2024-04-22T15:19:02.020 INFO:tasks.workunit.client.1.smithi196.stdout:9/551: dread db/dd/de/d15/f40 [0,4194304] 0 2024-04-22T15:19:02.021 INFO:tasks.workunit.client.0.smithi096.stdout:8/805: creat d4/d10/d4d/f109 x:0 0 0 2024-04-22T15:19:02.030 INFO:tasks.workunit.client.0.smithi096.stdout:9/741: fsync d1/df/d95/d9d/db5/dd/f2b 0 2024-04-22T15:19:02.032 INFO:tasks.workunit.client.1.smithi196.stdout:3/545: creat d0/d15/d31/d55/fb6 x:0 0 0 2024-04-22T15:19:02.032 INFO:tasks.workunit.client.1.smithi196.stdout:3/546: dread - d0/d15/da7/d18/f84 zero size 2024-04-22T15:19:02.034 INFO:tasks.workunit.client.1.smithi196.stdout:3/547: fsync d0/d15/d48/d51/d5e/f63 0 2024-04-22T15:19:02.035 INFO:tasks.workunit.client.0.smithi096.stdout:2/830: mkdir d3/db/df/d24/d58/d5f/ddf/d10c 0 2024-04-22T15:19:02.039 INFO:tasks.workunit.client.1.smithi196.stdout:5/555: rename d1/d14/l37 to d1/d9/d68/lb4 0 2024-04-22T15:19:02.043 INFO:tasks.workunit.client.1.smithi196.stdout:0/570: link d3/c4 d3/d9/d36/cc3 0 2024-04-22T15:19:02.043 INFO:tasks.workunit.client.1.smithi196.stdout:0/571: chown d3/d6/dc/c91 1626 1 2024-04-22T15:19:02.044 INFO:tasks.workunit.client.1.smithi196.stdout:1/511: link d1/db/d16/d1e/c82 d1/db/dc/de/d5a/d5b/d9a/ca2 0 2024-04-22T15:19:02.045 INFO:tasks.workunit.client.0.smithi096.stdout:3/728: truncate da/db/de/ddc/d26/d52/fd4 465172 0 2024-04-22T15:19:02.047 INFO:tasks.workunit.client.0.smithi096.stdout:9/742: dread - d1/d4/d7b/fa6 zero size 2024-04-22T15:19:02.050 INFO:tasks.workunit.client.1.smithi196.stdout:8/544: dread d2/d3f/f53 [0,4194304] 0 2024-04-22T15:19:02.051 INFO:tasks.workunit.client.1.smithi196.stdout:3/548: creat d0/d15/d48/d51/fb7 x:0 0 0 2024-04-22T15:19:02.052 INFO:tasks.workunit.client.0.smithi096.stdout:1/850: write d6/dc/d8a/d3f/d62/d7d/dab/fc1 [783543,111885] 0 2024-04-22T15:19:02.053 INFO:tasks.workunit.client.0.smithi096.stdout:8/806: creat d4/d1c/d62/f10a x:0 0 0 2024-04-22T15:19:02.053 INFO:tasks.workunit.client.1.smithi196.stdout:5/556: dread d1/d17/f8b [0,4194304] 0 2024-04-22T15:19:02.056 INFO:tasks.workunit.client.1.smithi196.stdout:4/516: sync 2024-04-22T15:19:02.056 INFO:tasks.workunit.client.1.smithi196.stdout:6/552: sync 2024-04-22T15:19:02.057 INFO:tasks.workunit.client.0.smithi096.stdout:5/830: write d4/ddb/fbd [4031110,58952] 0 2024-04-22T15:19:02.057 INFO:tasks.workunit.client.0.smithi096.stdout:3/729: symlink da/db/dc/d8d/le5 0 2024-04-22T15:19:02.059 INFO:tasks.workunit.client.0.smithi096.stdout:3/730: chown da/db/de/ddc/c5e 5641717 1 2024-04-22T15:19:02.059 INFO:tasks.workunit.client.1.smithi196.stdout:5/557: dwrite d1/d17/d1a/d2f/d30/f54 [0,4194304] 0 2024-04-22T15:19:02.060 INFO:tasks.workunit.client.0.smithi096.stdout:2/831: fsync d3/db/df/dce/d46/d69/f6c 0 2024-04-22T15:19:02.063 INFO:tasks.workunit.client.1.smithi196.stdout:9/552: link db/dd/de/d15/d31/f63 db/d11/d29/d57/fbc 0 2024-04-22T15:19:02.066 INFO:tasks.workunit.client.0.smithi096.stdout:5/831: creat d4/ddb/d3d/d4d/d88/de5/f118 x:0 0 0 2024-04-22T15:19:02.070 INFO:tasks.workunit.client.1.smithi196.stdout:0/572: readlink d3/d9/d36/d56/d65/d7f/d87/d6c/d88/d95/l96 0 2024-04-22T15:19:02.073 INFO:tasks.workunit.client.0.smithi096.stdout:3/731: mknod da/db/de/d87/ce6 0 2024-04-22T15:19:02.075 INFO:tasks.workunit.client.1.smithi196.stdout:4/517: unlink d6/d13/d15/d2c/d62/f6d 0 2024-04-22T15:19:02.076 INFO:tasks.workunit.client.1.smithi196.stdout:1/512: mknod d1/db/ca3 0 2024-04-22T15:19:02.079 INFO:tasks.workunit.client.1.smithi196.stdout:8/545: mkdir d2/d5/d9c 0 2024-04-22T15:19:02.081 INFO:tasks.workunit.client.0.smithi096.stdout:9/743: truncate d1/d25/d9f/fcd 2969610 0 2024-04-22T15:19:02.082 INFO:tasks.workunit.client.1.smithi196.stdout:5/558: mkdir d1/d4/d92/db5 0 2024-04-22T15:19:02.084 INFO:tasks.workunit.client.1.smithi196.stdout:3/549: mkdir d0/d7/d6f/db5/db8 0 2024-04-22T15:19:02.086 INFO:tasks.workunit.client.1.smithi196.stdout:5/559: dread d1/d17/f79 [0,4194304] 0 2024-04-22T15:19:02.087 INFO:tasks.workunit.client.1.smithi196.stdout:6/553: dwrite dd/f50 [0,4194304] 0 2024-04-22T15:19:02.088 INFO:tasks.workunit.client.0.smithi096.stdout:5/832: creat d4/ddb/d3d/d4d/d59/d8a/f119 x:0 0 0 2024-04-22T15:19:02.091 INFO:tasks.workunit.client.0.smithi096.stdout:8/807: truncate d4/f32 2774305 0 2024-04-22T15:19:02.092 INFO:tasks.workunit.client.1.smithi196.stdout:5/560: dread d1/d17/d50/f73 [0,4194304] 0 2024-04-22T15:19:02.095 INFO:tasks.workunit.client.0.smithi096.stdout:1/851: dwrite d6/dc/d8a/d3f/d62/f85 [4194304,4194304] 0 2024-04-22T15:19:02.103 INFO:tasks.workunit.client.1.smithi196.stdout:9/553: dwrite db/dd/de/d15/f6e [0,4194304] 0 2024-04-22T15:19:02.103 INFO:tasks.workunit.client.1.smithi196.stdout:1/513: dread d1/db/d16/d1e/d49/f42 [0,4194304] 0 2024-04-22T15:19:02.113 INFO:tasks.workunit.client.1.smithi196.stdout:3/550: rmdir d0/d15/d31/d9d/d9e 39 2024-04-22T15:19:02.117 INFO:tasks.workunit.client.0.smithi096.stdout:8/808: creat d4/d1c/d40/da7/f10b x:0 0 0 2024-04-22T15:19:02.118 INFO:tasks.workunit.client.0.smithi096.stdout:1/852: creat d6/dc/d8a/d3f/d54/d32/d8c/dca/f10a x:0 0 0 2024-04-22T15:19:02.119 INFO:tasks.workunit.client.0.smithi096.stdout:9/744: symlink d1/d25/db2/lfb 0 2024-04-22T15:19:02.120 INFO:tasks.workunit.client.1.smithi196.stdout:5/561: truncate d1/d4/f26 5170389 0 2024-04-22T15:19:02.121 INFO:tasks.workunit.client.1.smithi196.stdout:5/562: write d1/d17/d1a/d2f/d30/d5e/f63 [4254418,22826] 0 2024-04-22T15:19:02.122 INFO:tasks.workunit.client.0.smithi096.stdout:2/832: truncate d3/db/df/f61 180792 0 2024-04-22T15:19:02.124 INFO:tasks.workunit.client.0.smithi096.stdout:5/833: rename d4/ddb/d3d/d4d/d59/d98/f8d to d4/d1f/d8b/d106/f11a 0 2024-04-22T15:19:02.125 INFO:tasks.workunit.client.0.smithi096.stdout:8/809: truncate d4/d10/d25/d38/f79 885686 0 2024-04-22T15:19:02.126 INFO:tasks.workunit.client.0.smithi096.stdout:3/732: write da/d3f/f5b [73843,31025] 0 2024-04-22T15:19:02.129 INFO:tasks.workunit.client.0.smithi096.stdout:1/853: unlink d6/dc/d8a/d3f/d54/d32/dcd/f102 0 2024-04-22T15:19:02.131 INFO:tasks.workunit.client.1.smithi196.stdout:1/514: unlink d1/db/dc/d26/f71 0 2024-04-22T15:19:02.134 INFO:tasks.workunit.client.1.smithi196.stdout:8/546: mknod d2/d5/c9d 0 2024-04-22T15:19:02.134 INFO:tasks.workunit.client.1.smithi196.stdout:8/547: fdatasync d2/d8/d47/d68/d81/f88 0 2024-04-22T15:19:02.137 INFO:tasks.workunit.client.1.smithi196.stdout:7/519: sync 2024-04-22T15:19:02.137 INFO:tasks.workunit.client.1.smithi196.stdout:2/581: sync 2024-04-22T15:19:02.139 INFO:tasks.workunit.client.0.smithi096.stdout:2/833: readlink d3/db/d30/lb2 0 2024-04-22T15:19:02.140 INFO:tasks.workunit.client.0.smithi096.stdout:2/834: readlink d3/db/df/dce/d46/d59/le8 0 2024-04-22T15:19:02.141 INFO:tasks.workunit.client.1.smithi196.stdout:2/582: readlink d1/d23/d84/dc8/da1/l94 0 2024-04-22T15:19:02.146 INFO:tasks.workunit.client.1.smithi196.stdout:9/554: dwrite db/d11/d29/f6f [0,4194304] 0 2024-04-22T15:19:02.146 INFO:tasks.workunit.client.0.smithi096.stdout:3/733: symlink da/db/d9a/le7 0 2024-04-22T15:19:02.148 INFO:tasks.workunit.client.0.smithi096.stdout:9/745: write d1/df/d95/d9d/db5/d43/d57/d5b/f76 [741173,20861] 0 2024-04-22T15:19:02.150 INFO:tasks.workunit.client.1.smithi196.stdout:5/563: chown d1/d17/c41 48 1 2024-04-22T15:19:02.151 INFO:tasks.workunit.client.1.smithi196.stdout:5/564: fdatasync d1/d17/d1a/d2f/d30/fb3 0 2024-04-22T15:19:02.151 INFO:tasks.workunit.client.0.smithi096.stdout:1/854: symlink d6/dc/d8a/d3f/d54/d32/d3b/d94/l10b 0 2024-04-22T15:19:02.153 INFO:tasks.workunit.client.1.smithi196.stdout:4/518: rmdir d6/d14/d18/d40/d74/d84 0 2024-04-22T15:19:02.154 INFO:tasks.workunit.client.0.smithi096.stdout:2/835: fdatasync d3/db/df/d10/db5/fc2 0 2024-04-22T15:19:02.156 INFO:tasks.workunit.client.0.smithi096.stdout:2/836: chown d3/db/f12 41447 1 2024-04-22T15:19:02.157 INFO:tasks.workunit.client.0.smithi096.stdout:5/834: mknod d4/d1f/c11b 0 2024-04-22T15:19:02.157 INFO:tasks.workunit.client.0.smithi096.stdout:9/746: dread d1/f8a [0,4194304] 0 2024-04-22T15:19:02.158 INFO:tasks.workunit.client.0.smithi096.stdout:5/835: chown d4/ddb/d3d/d4f/ca3 8343776 1 2024-04-22T15:19:02.159 INFO:tasks.workunit.client.1.smithi196.stdout:1/515: rename d1/f4 to d1/db/dc/de/d5a/d94/fa4 0 2024-04-22T15:19:02.161 INFO:tasks.workunit.client.0.smithi096.stdout:8/810: creat d4/d10/d25/d4c/db4/df4/f10c x:0 0 0 2024-04-22T15:19:02.162 INFO:tasks.workunit.client.0.smithi096.stdout:3/734: symlink da/db/dc/d28/d96/le8 0 2024-04-22T15:19:02.164 INFO:tasks.workunit.client.1.smithi196.stdout:6/554: sync 2024-04-22T15:19:02.164 INFO:tasks.workunit.client.1.smithi196.stdout:1/516: dwrite d1/db/f58 [0,4194304] 0 2024-04-22T15:19:02.173 INFO:tasks.workunit.client.1.smithi196.stdout:8/548: dread d2/f4a [0,4194304] 0 2024-04-22T15:19:02.173 INFO:tasks.workunit.client.1.smithi196.stdout:8/549: chown d2/d35/c5d 20 1 2024-04-22T15:19:02.175 INFO:tasks.workunit.client.0.smithi096.stdout:1/855: write d6/dc/d8a/d3f/d54/d32/d3b/d56/f44 [3878492,112516] 0 2024-04-22T15:19:02.176 INFO:tasks.workunit.client.0.smithi096.stdout:8/811: creat d4/d10/d25/d38/d8f/d9c/f10d x:0 0 0 2024-04-22T15:19:02.177 INFO:tasks.workunit.client.0.smithi096.stdout:8/812: chown d4/d10/d4d 192706819 1 2024-04-22T15:19:02.178 INFO:tasks.workunit.client.0.smithi096.stdout:4/803: sync 2024-04-22T15:19:02.183 INFO:tasks.workunit.client.0.smithi096.stdout:9/747: write d1/df/f65 [342524,23862] 0 2024-04-22T15:19:02.185 INFO:tasks.workunit.client.0.smithi096.stdout:5/836: mkdir d4/ddb/d3d/d35/d57/dbe/d113/d11c 0 2024-04-22T15:19:02.187 INFO:tasks.workunit.client.0.smithi096.stdout:8/813: chown d4/c27 23844399 1 2024-04-22T15:19:02.189 INFO:tasks.workunit.client.0.smithi096.stdout:5/837: dread d4/ddb/f62 [0,4194304] 0 2024-04-22T15:19:02.190 INFO:tasks.workunit.client.0.smithi096.stdout:4/804: mkdir d5/d7/d1f/d84/d6d/d118 0 2024-04-22T15:19:02.192 INFO:tasks.workunit.client.0.smithi096.stdout:2/837: rename d3/db/df/dce/d46/d7e/d8d to d3/db/d10d 0 2024-04-22T15:19:02.199 INFO:tasks.workunit.client.0.smithi096.stdout:3/735: link da/db/de/ddc/d26/l47 da/db/dc/d28/dda/le9 0 2024-04-22T15:19:02.199 INFO:tasks.workunit.client.0.smithi096.stdout:3/736: stat da/db/dc/f91 0 2024-04-22T15:19:02.203 INFO:tasks.workunit.client.0.smithi096.stdout:4/805: symlink d5/d7/d2c/d95/df9/l119 0 2024-04-22T15:19:02.208 INFO:tasks.workunit.client.1.smithi196.stdout:7/520: mknod d14/d4a/ca4 0 2024-04-22T15:19:02.212 INFO:tasks.workunit.client.1.smithi196.stdout:0/573: getdents d3 0 2024-04-22T15:19:02.213 INFO:tasks.workunit.client.0.smithi096.stdout:3/737: symlink da/db/d12/d67/lea 0 2024-04-22T15:19:02.214 INFO:tasks.workunit.client.1.smithi196.stdout:9/555: creat db/d11/d29/d57/fbd x:0 0 0 2024-04-22T15:19:02.217 INFO:tasks.workunit.client.1.smithi196.stdout:5/565: creat d1/d2/d98/fb6 x:0 0 0 2024-04-22T15:19:02.218 INFO:tasks.workunit.client.0.smithi096.stdout:4/806: creat d5/d7/d1f/d84/d6d/d98/f11a x:0 0 0 2024-04-22T15:19:02.218 INFO:tasks.workunit.client.1.smithi196.stdout:5/566: truncate d1/d17/d1a/d2f/d30/d6e/d7a/fa3 985120 0 2024-04-22T15:19:02.221 INFO:tasks.workunit.client.0.smithi096.stdout:9/748: rmdir d1/d25/d9f/df3 0 2024-04-22T15:19:02.223 INFO:tasks.workunit.client.0.smithi096.stdout:9/749: dread - d1/d85/fbc zero size 2024-04-22T15:19:02.224 INFO:tasks.workunit.client.0.smithi096.stdout:1/856: dwrite d6/f9d [0,4194304] 0 2024-04-22T15:19:02.225 INFO:tasks.workunit.client.0.smithi096.stdout:1/857: readlink d6/dc/d8a/l28 0 2024-04-22T15:19:02.226 INFO:tasks.workunit.client.1.smithi196.stdout:9/556: dread db/dd/de/f10 [4194304,4194304] 0 2024-04-22T15:19:02.227 INFO:tasks.workunit.client.1.smithi196.stdout:9/557: dread - db/d11/d29/f2b zero size 2024-04-22T15:19:02.229 INFO:tasks.workunit.client.0.smithi096.stdout:3/738: dread da/db/de/d19/d2d/fa8 [0,4194304] 0 2024-04-22T15:19:02.229 INFO:tasks.workunit.client.1.smithi196.stdout:6/555: symlink dd/d20/d5c/d5e/l9c 0 2024-04-22T15:19:02.233 INFO:tasks.workunit.client.1.smithi196.stdout:3/551: rmdir d0/d7/d6f/db5/db8 0 2024-04-22T15:19:02.235 INFO:tasks.workunit.client.0.smithi096.stdout:5/838: dwrite d4/d2c/f4e [0,4194304] 0 2024-04-22T15:19:02.240 INFO:tasks.workunit.client.1.smithi196.stdout:7/521: dwrite d14/d26/d2a/f66 [0,4194304] 0 2024-04-22T15:19:02.240 INFO:tasks.workunit.client.1.smithi196.stdout:5/567: rename d1/d4/l4d to d1/d17/d1c/d67/lb7 0 2024-04-22T15:19:02.244 INFO:tasks.workunit.client.1.smithi196.stdout:7/522: fdatasync d14/d4a/f8c 0 2024-04-22T15:19:02.244 INFO:tasks.workunit.client.1.smithi196.stdout:2/583: dwrite d1/d23/d27/d2e/f5d [0,4194304] 0 2024-04-22T15:19:02.245 INFO:tasks.workunit.client.1.smithi196.stdout:6/556: unlink dd/d1f/d4e/l75 0 2024-04-22T15:19:02.245 INFO:tasks.workunit.client.1.smithi196.stdout:4/519: dread d6/d90/f60 [0,4194304] 0 2024-04-22T15:19:02.247 INFO:tasks.workunit.client.1.smithi196.stdout:3/552: rmdir d0/d15/d4d 39 2024-04-22T15:19:02.249 INFO:tasks.workunit.client.1.smithi196.stdout:7/523: read d14/d26/fa2 [1060209,119661] 0 2024-04-22T15:19:02.253 INFO:tasks.workunit.client.1.smithi196.stdout:6/557: dwrite dd/f11 [0,4194304] 0 2024-04-22T15:19:02.255 INFO:tasks.workunit.client.1.smithi196.stdout:3/553: dread d0/d15/d31/f53 [0,4194304] 0 2024-04-22T15:19:02.258 INFO:tasks.workunit.client.1.smithi196.stdout:9/558: unlink db/d11/d29/d7a/d7b/d81/l9d 0 2024-04-22T15:19:02.260 INFO:tasks.workunit.client.1.smithi196.stdout:9/559: readlink db/d11/d29/d7a/d7b/l7f 0 2024-04-22T15:19:02.260 INFO:tasks.workunit.client.1.smithi196.stdout:5/568: creat d1/d14/d6a/fb8 x:0 0 0 2024-04-22T15:19:02.261 INFO:tasks.workunit.client.1.smithi196.stdout:6/558: dwrite dd/d2b/f62 [0,4194304] 0 2024-04-22T15:19:02.277 INFO:tasks.workunit.client.1.smithi196.stdout:4/520: write d6/d14/d18/d26/d32/d53/f94 [535607,55656] 0 2024-04-22T15:19:02.284 INFO:tasks.workunit.client.1.smithi196.stdout:1/517: dwrite d1/db/d16/f5c [0,4194304] 0 2024-04-22T15:19:02.284 INFO:tasks.workunit.client.1.smithi196.stdout:1/518: stat d1/db/dc/d26/d2e/d32/d36 0 2024-04-22T15:19:02.287 INFO:tasks.workunit.client.1.smithi196.stdout:0/574: truncate d3/d6/dc/d8c/d8d/da8/fbe 4088227 0 2024-04-22T15:19:02.290 INFO:tasks.workunit.client.1.smithi196.stdout:9/560: mkdir db/dd/de/d15/d31/d64/dbe 0 2024-04-22T15:19:02.291 INFO:tasks.workunit.client.1.smithi196.stdout:8/550: getdents d2/d8/d49 0 2024-04-22T15:19:02.296 INFO:tasks.workunit.client.1.smithi196.stdout:8/551: dwrite d2/d8/f73 [0,4194304] 0 2024-04-22T15:19:02.299 INFO:tasks.workunit.client.1.smithi196.stdout:3/554: rename d0/d15/da7/d4a/d44/lad to d0/d15/d48/d51/d7f/lb9 0 2024-04-22T15:19:02.301 INFO:tasks.workunit.client.1.smithi196.stdout:3/555: truncate d0/d15/d31/d55/f74 585005 0 2024-04-22T15:19:02.304 INFO:tasks.workunit.client.1.smithi196.stdout:5/569: dread d1/d17/f21 [0,4194304] 0 2024-04-22T15:19:02.307 INFO:tasks.workunit.client.1.smithi196.stdout:9/561: creat db/dd/de/d75/fbf x:0 0 0 2024-04-22T15:19:02.307 INFO:tasks.workunit.client.1.smithi196.stdout:9/562: stat db/dd/de/d15/l67 0 2024-04-22T15:19:02.308 INFO:tasks.workunit.client.1.smithi196.stdout:5/570: read d1/d2/f28 [4149280,25796] 0 2024-04-22T15:19:02.312 INFO:tasks.workunit.client.1.smithi196.stdout:7/524: creat d14/fa5 x:0 0 0 2024-04-22T15:19:02.312 INFO:tasks.workunit.client.1.smithi196.stdout:7/525: readlink d14/d25/d31/l53 0 2024-04-22T15:19:02.313 INFO:tasks.workunit.client.1.smithi196.stdout:7/526: chown fe 44423 1 2024-04-22T15:19:02.317 INFO:tasks.workunit.client.1.smithi196.stdout:3/556: mknod d0/d15/da7/d4a/d44/cba 0 2024-04-22T15:19:02.321 INFO:tasks.workunit.client.1.smithi196.stdout:0/575: fsync d3/d9/d36/d56/d65/d7f/d30/f46 0 2024-04-22T15:19:02.323 INFO:tasks.workunit.client.1.smithi196.stdout:9/563: symlink db/dd/de/d15/d31/d64/dab/lc0 0 2024-04-22T15:19:02.328 INFO:tasks.workunit.client.1.smithi196.stdout:5/571: dread d1/d17/d1c/f3b [0,4194304] 0 2024-04-22T15:19:02.329 INFO:tasks.workunit.client.0.smithi096.stdout:6/655: sync 2024-04-22T15:19:02.329 INFO:tasks.workunit.client.0.smithi096.stdout:0/918: sync 2024-04-22T15:19:02.330 INFO:tasks.workunit.client.0.smithi096.stdout:0/919: chown d7/de/d5b/d95/dc/f2f 1150794 1 2024-04-22T15:19:02.332 INFO:tasks.workunit.client.0.smithi096.stdout:8/814: creat d4/d1c/d40/f10e x:0 0 0 2024-04-22T15:19:02.333 INFO:tasks.workunit.client.1.smithi196.stdout:3/557: mknod d0/d15/d31/d55/d8b/cbb 0 2024-04-22T15:19:02.333 INFO:tasks.workunit.client.1.smithi196.stdout:5/572: dwrite d1/d17/d1a/d2f/d30/f6b [0,4194304] 0 2024-04-22T15:19:02.338 INFO:tasks.workunit.client.1.smithi196.stdout:0/576: fdatasync d3/d9/d36/d56/d65/d7f/d30/f46 0 2024-04-22T15:19:02.339 INFO:tasks.workunit.client.1.smithi196.stdout:6/559: dwrite dd/d1f/d38/f4b [0,4194304] 0 2024-04-22T15:19:02.341 INFO:tasks.workunit.client.1.smithi196.stdout:0/577: truncate d3/d6/dc/d8c/d8d/fc1 276028 0 2024-04-22T15:19:02.341 INFO:tasks.workunit.client.1.smithi196.stdout:6/560: chown dd/d1f/d4e/d53/d7f/f83 1162189451 1 2024-04-22T15:19:02.342 INFO:tasks.workunit.client.0.smithi096.stdout:4/807: rename d5/d7/d1a/d25/c70 to d5/d7/d2c/d95/df9/c11b 0 2024-04-22T15:19:02.344 INFO:tasks.workunit.client.0.smithi096.stdout:9/750: mkdir d1/df/d95/d9d/db5/d43/d57/d5b/da1/dfc 0 2024-04-22T15:19:02.346 INFO:tasks.workunit.client.1.smithi196.stdout:5/573: dread d1/d17/d1c/f52 [0,4194304] 0 2024-04-22T15:19:02.347 INFO:tasks.workunit.client.1.smithi196.stdout:2/584: truncate d1/d23/d27/d2e/d55/d5b/d7c/fa5 3264142 0 2024-04-22T15:19:02.347 INFO:tasks.workunit.client.0.smithi096.stdout:9/751: readlink d1/df/d95/d9d/db5/d43/d57/d5b/da1/ld1 0 2024-04-22T15:19:02.348 INFO:tasks.workunit.client.1.smithi196.stdout:2/585: stat d1/d1d/f3c 0 2024-04-22T15:19:02.348 INFO:tasks.workunit.client.0.smithi096.stdout:3/739: fsync da/db/de/d19/f2a 0 2024-04-22T15:19:02.350 INFO:tasks.workunit.client.0.smithi096.stdout:4/808: dwrite d5/d7/d1f/d84/d6d/d98/f110 [0,4194304] 0 2024-04-22T15:19:02.360 INFO:tasks.workunit.client.1.smithi196.stdout:1/519: write d1/db/d16/f1d [5130363,81476] 0 2024-04-22T15:19:02.360 INFO:tasks.workunit.client.1.smithi196.stdout:0/578: rmdir d3/d9/d36/d9f 39 2024-04-22T15:19:02.360 INFO:tasks.workunit.client.1.smithi196.stdout:1/520: read d1/db/f58 [2630733,77823] 0 2024-04-22T15:19:02.361 INFO:tasks.workunit.client.1.smithi196.stdout:1/521: stat d1/db/d16/d1e/d46 0 2024-04-22T15:19:02.364 INFO:tasks.workunit.client.1.smithi196.stdout:5/574: creat d1/d17/d1a/d2f/d30/d6e/fb9 x:0 0 0 2024-04-22T15:19:02.366 INFO:tasks.workunit.client.1.smithi196.stdout:8/552: link d2/d8/f27 d2/d35/f9e 0 2024-04-22T15:19:02.367 INFO:tasks.workunit.client.1.smithi196.stdout:8/553: fsync d2/dc/d45/f5b 0 2024-04-22T15:19:02.367 INFO:tasks.workunit.client.0.smithi096.stdout:6/656: creat d3/d28/d4d/d82/dd8/fe1 x:0 0 0 2024-04-22T15:19:02.371 INFO:tasks.workunit.client.0.smithi096.stdout:3/740: mknod da/db/de/ddc/d26/ceb 0 2024-04-22T15:19:02.372 INFO:tasks.workunit.client.1.smithi196.stdout:8/554: dwrite d2/d8/f76 [0,4194304] 0 2024-04-22T15:19:02.373 INFO:tasks.workunit.client.0.smithi096.stdout:1/858: truncate d6/dc/d8a/d3f/d62/d7d/f53 1853499 0 2024-04-22T15:19:02.373 INFO:tasks.workunit.client.0.smithi096.stdout:3/741: readlink da/db/dc/d8d/le5 0 2024-04-22T15:19:02.373 INFO:tasks.workunit.client.0.smithi096.stdout:2/838: sync 2024-04-22T15:19:02.373 INFO:tasks.workunit.client.1.smithi196.stdout:8/555: truncate d2/d3f/d4b/d79/f8a 835330 0 2024-04-22T15:19:02.374 INFO:tasks.workunit.client.1.smithi196.stdout:8/556: read d2/d8/d49/f59 [2884736,56122] 0 2024-04-22T15:19:02.376 INFO:tasks.workunit.client.0.smithi096.stdout:8/815: dread d4/d1c/d62/f6b [0,4194304] 0 2024-04-22T15:19:02.377 INFO:tasks.workunit.client.1.smithi196.stdout:1/522: symlink d1/db/d16/la5 0 2024-04-22T15:19:02.378 INFO:tasks.workunit.client.0.smithi096.stdout:5/839: creat d4/d1f/f11d x:0 0 0 2024-04-22T15:19:02.380 INFO:tasks.workunit.client.1.smithi196.stdout:5/575: unlink d1/d17/d1c/fa4 0 2024-04-22T15:19:02.382 INFO:tasks.workunit.client.1.smithi196.stdout:7/527: write d14/f67 [624023,106204] 0 2024-04-22T15:19:02.383 INFO:tasks.workunit.client.1.smithi196.stdout:5/576: dread d1/d9/f11 [0,4194304] 0 2024-04-22T15:19:02.384 INFO:tasks.workunit.client.1.smithi196.stdout:5/577: write d1/d4/f22 [3029103,60408] 0 2024-04-22T15:19:02.395 INFO:tasks.workunit.client.1.smithi196.stdout:4/521: sync 2024-04-22T15:19:02.396 INFO:tasks.workunit.client.1.smithi196.stdout:0/579: mknod d3/d9/d36/d56/d65/d7f/d87/d6c/d88/cc4 0 2024-04-22T15:19:02.396 INFO:tasks.workunit.client.1.smithi196.stdout:0/580: stat d3/de/d1c/d27/f38 0 2024-04-22T15:19:02.399 INFO:tasks.workunit.client.0.smithi096.stdout:3/742: dread da/db/de/ddc/f53 [0,4194304] 0 2024-04-22T15:19:02.400 INFO:tasks.workunit.client.0.smithi096.stdout:5/840: creat d4/ddb/d3d/d4d/d59/d98/dee/d109/f11e x:0 0 0 2024-04-22T15:19:02.400 INFO:tasks.workunit.client.0.smithi096.stdout:8/816: dread d4/d1c/d40/fdc [0,4194304] 0 2024-04-22T15:19:02.401 INFO:tasks.workunit.client.1.smithi196.stdout:6/561: creat dd/d1f/d38/f9d x:0 0 0 2024-04-22T15:19:02.402 INFO:tasks.workunit.client.1.smithi196.stdout:6/562: chown dd/f18 246428 1 2024-04-22T15:19:02.402 INFO:tasks.workunit.client.1.smithi196.stdout:6/563: read - dd/d20/d4d/f91 zero size 2024-04-22T15:19:02.403 INFO:tasks.workunit.client.0.smithi096.stdout:6/657: unlink d3/d28/dd/l62 0 2024-04-22T15:19:02.404 INFO:tasks.workunit.client.1.smithi196.stdout:1/523: mknod d1/db/dc/de/d5a/d5b/ca6 0 2024-04-22T15:19:02.405 INFO:tasks.workunit.client.0.smithi096.stdout:9/752: link d1/d25/db2/lfb d1/df/d95/d9d/db5/dd/dba/dc2/dc4/lfd 0 2024-04-22T15:19:02.408 INFO:tasks.workunit.client.1.smithi196.stdout:5/578: mkdir d1/d14/d6a/dba 0 2024-04-22T15:19:02.408 INFO:tasks.workunit.client.0.smithi096.stdout:0/920: dwrite d7/da7/fa8 [4194304,4194304] 0 2024-04-22T15:19:02.408 INFO:tasks.workunit.client.0.smithi096.stdout:4/809: link d5/d7/d2c/d67/ld6 d5/d7/d1a/d103/l11c 0 2024-04-22T15:19:02.412 INFO:tasks.workunit.client.0.smithi096.stdout:2/839: fdatasync d3/db/df/dce/d46/d69/f66 0 2024-04-22T15:19:02.414 INFO:tasks.workunit.client.1.smithi196.stdout:3/558: sync 2024-04-22T15:19:02.414 INFO:tasks.workunit.client.1.smithi196.stdout:3/559: dread - d0/d7/d8/d70/f97 zero size 2024-04-22T15:19:02.415 INFO:tasks.workunit.client.0.smithi096.stdout:1/859: getdents d6/dc/d8a/d3f/d54/d32/d3b/d56/d107 0 2024-04-22T15:19:02.419 INFO:tasks.workunit.client.1.smithi196.stdout:0/581: mknod d3/d6/dc/d8c/d8d/cc5 0 2024-04-22T15:19:02.420 INFO:tasks.workunit.client.1.smithi196.stdout:8/557: mkdir d2/d35/d96/d9f 0 2024-04-22T15:19:02.421 INFO:tasks.workunit.client.1.smithi196.stdout:2/586: truncate d1/d23/d27/d2e/d55/f36 3815491 0 2024-04-22T15:19:02.422 INFO:tasks.workunit.client.0.smithi096.stdout:5/841: creat d4/ddb/d3d/d4d/d59/d98/dee/d109/f11f x:0 0 0 2024-04-22T15:19:02.426 INFO:tasks.workunit.client.0.smithi096.stdout:8/817: creat d4/d10/d25/d4c/d76/f10f x:0 0 0 2024-04-22T15:19:02.432 INFO:tasks.workunit.client.1.smithi196.stdout:5/579: unlink d1/d17/d50/f73 0 2024-04-22T15:19:02.433 INFO:tasks.workunit.client.0.smithi096.stdout:4/810: mkdir d5/d7/d2c/d67/d11d 0 2024-04-22T15:19:02.434 INFO:tasks.workunit.client.0.smithi096.stdout:4/811: fdatasync d5/d7/d1f/d84/d6d/d98/f110 0 2024-04-22T15:19:02.437 INFO:tasks.workunit.client.1.smithi196.stdout:8/558: dread d2/d5/f23 [0,4194304] 0 2024-04-22T15:19:02.438 INFO:tasks.workunit.client.1.smithi196.stdout:4/522: mkdir d6/d14/d18/d26/d32/da3 0 2024-04-22T15:19:02.439 INFO:tasks.workunit.client.1.smithi196.stdout:0/582: symlink d3/de/d14/d5a/lc6 0 2024-04-22T15:19:02.441 INFO:tasks.workunit.client.1.smithi196.stdout:2/587: readlink d1/d23/d27/d73/l86 0 2024-04-22T15:19:02.443 INFO:tasks.workunit.client.0.smithi096.stdout:5/842: unlink d4/ddb/f96 0 2024-04-22T15:19:02.445 INFO:tasks.workunit.client.0.smithi096.stdout:8/818: creat d4/d10/d25/d4c/d76/d7f/f110 x:0 0 0 2024-04-22T15:19:02.448 INFO:tasks.workunit.client.1.smithi196.stdout:1/524: link d1/db/dc/de/d96/d9d/f50 d1/db/dc/de/d43/d5e/fa7 0 2024-04-22T15:19:02.451 INFO:tasks.workunit.client.0.smithi096.stdout:0/921: write d7/de/d5b/d95/d66/f81 [3345413,65884] 0 2024-04-22T15:19:02.453 INFO:tasks.workunit.client.1.smithi196.stdout:6/564: dwrite dd/d1f/d37/f34 [0,4194304] 0 2024-04-22T15:19:02.454 INFO:tasks.workunit.client.1.smithi196.stdout:6/565: chown dd/d1f/d38/d40/f4f 0 1 2024-04-22T15:19:02.457 INFO:tasks.workunit.client.1.smithi196.stdout:8/559: fsync d2/dc/f3a 0 2024-04-22T15:19:02.458 INFO:tasks.workunit.client.1.smithi196.stdout:3/560: dread d0/d15/d4d/f5a [0,4194304] 0 2024-04-22T15:19:02.464 INFO:tasks.workunit.client.1.smithi196.stdout:2/588: unlink d1/d13/d25/d2a/lc3 0 2024-04-22T15:19:02.466 INFO:tasks.workunit.client.1.smithi196.stdout:3/561: dread d0/d15/da7/f1c [0,4194304] 0 2024-04-22T15:19:02.467 INFO:tasks.workunit.client.1.smithi196.stdout:3/562: fsync d0/d15/d31/faf 0 2024-04-22T15:19:02.469 INFO:tasks.workunit.client.1.smithi196.stdout:7/528: getdents d14/d26 0 2024-04-22T15:19:02.469 INFO:tasks.workunit.client.1.smithi196.stdout:7/529: stat d14/d4a/f71 0 2024-04-22T15:19:02.471 INFO:tasks.workunit.client.1.smithi196.stdout:5/580: mkdir d1/d2/dbb 0 2024-04-22T15:19:02.473 INFO:tasks.workunit.client.0.smithi096.stdout:2/840: creat d3/db/df/f10e x:0 0 0 2024-04-22T15:19:02.474 INFO:tasks.workunit.client.0.smithi096.stdout:2/841: stat d3/db/df/d24/d58 0 2024-04-22T15:19:02.478 INFO:tasks.workunit.client.0.smithi096.stdout:2/842: readlink d3/db/df/dce/d46/d59/lb1 0 2024-04-22T15:19:02.478 INFO:tasks.workunit.client.0.smithi096.stdout:1/860: creat d6/f10c x:0 0 0 2024-04-22T15:19:02.478 INFO:tasks.workunit.client.0.smithi096.stdout:3/743: getdents da/db/d12/d9b 0 2024-04-22T15:19:02.479 INFO:tasks.workunit.client.1.smithi196.stdout:9/564: creat db/dd/d1b/fc1 x:0 0 0 2024-04-22T15:19:02.481 INFO:tasks.workunit.client.0.smithi096.stdout:0/922: dwrite d7/de/d5b/d95/d15/d35/fb4 [4194304,4194304] 0 2024-04-22T15:19:02.482 INFO:tasks.workunit.client.0.smithi096.stdout:0/923: chown d7/de/d5b/d95/f45 1733495509 1 2024-04-22T15:19:02.484 INFO:tasks.workunit.client.1.smithi196.stdout:4/523: creat d6/d14/d18/d26/d32/da3/fa4 x:0 0 0 2024-04-22T15:19:02.486 INFO:tasks.workunit.client.0.smithi096.stdout:8/819: truncate d4/d1c/fc4 435342 0 2024-04-22T15:19:02.488 INFO:tasks.workunit.client.1.smithi196.stdout:0/583: mkdir d3/db4/dc7 0 2024-04-22T15:19:02.490 INFO:tasks.workunit.client.1.smithi196.stdout:0/584: readlink d3/d9/l82 0 2024-04-22T15:19:02.490 INFO:tasks.workunit.client.1.smithi196.stdout:2/589: mknod d1/d23/d27/d2e/d55/d5b/cdc 0 2024-04-22T15:19:02.491 INFO:tasks.workunit.client.0.smithi096.stdout:3/744: stat da/db/dc/f5d 0 2024-04-22T15:19:02.491 INFO:tasks.workunit.client.1.smithi196.stdout:8/560: dwrite d2/d5/f46 [0,4194304] 0 2024-04-22T15:19:02.492 INFO:tasks.workunit.client.0.smithi096.stdout:3/745: chown da/db/de/d19/d2d/fa8 1475 1 2024-04-22T15:19:02.492 INFO:tasks.workunit.client.1.smithi196.stdout:6/566: dwrite dd/d1f/d38/d40/f42 [4194304,4194304] 0 2024-04-22T15:19:02.493 INFO:tasks.workunit.client.0.smithi096.stdout:1/861: mkdir d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/d10d 0 2024-04-22T15:19:02.497 INFO:tasks.workunit.client.1.smithi196.stdout:3/563: rename d0/d15/d31/faf to d0/d15/da7/d4a/d44/d8c/fbc 0 2024-04-22T15:19:02.500 INFO:tasks.workunit.client.0.smithi096.stdout:4/812: rename d5/ld to d5/d7/d2c/d43/l11e 0 2024-04-22T15:19:02.501 INFO:tasks.workunit.client.1.smithi196.stdout:5/581: symlink d1/d17/d1c/d67/lbc 0 2024-04-22T15:19:02.501 INFO:tasks.workunit.client.0.smithi096.stdout:8/820: symlink d4/d10/dcc/l111 0 2024-04-22T15:19:02.503 INFO:tasks.workunit.client.1.smithi196.stdout:9/565: read db/dd/de/d15/f73 [2681143,14088] 0 2024-04-22T15:19:02.504 INFO:tasks.workunit.client.0.smithi096.stdout:3/746: symlink da/db/dc/d59/lec 0 2024-04-22T15:19:02.507 INFO:tasks.workunit.client.0.smithi096.stdout:5/843: getdents d4/ddb/d3d/d4d/d59/d98 0 2024-04-22T15:19:02.510 INFO:tasks.workunit.client.1.smithi196.stdout:0/585: truncate d3/f5 1364678 0 2024-04-22T15:19:02.513 INFO:tasks.workunit.client.0.smithi096.stdout:4/813: truncate d5/d7/d2c/f86 340369 0 2024-04-22T15:19:02.516 INFO:tasks.workunit.client.1.smithi196.stdout:1/525: creat d1/db/fa8 x:0 0 0 2024-04-22T15:19:02.518 INFO:tasks.workunit.client.1.smithi196.stdout:7/530: symlink d14/d25/la6 0 2024-04-22T15:19:02.518 INFO:tasks.workunit.client.0.smithi096.stdout:5/844: dread - d4/d2c/d2f/d31/fca zero size 2024-04-22T15:19:02.519 INFO:tasks.workunit.client.1.smithi196.stdout:2/590: dread d1/d23/d84/dc8/f81 [0,4194304] 0 2024-04-22T15:19:02.519 INFO:tasks.workunit.client.1.smithi196.stdout:2/591: chown d1/d13/d25/l8d 266704187 1 2024-04-22T15:19:02.520 INFO:tasks.workunit.client.1.smithi196.stdout:5/582: creat d1/d9/d68/fbd x:0 0 0 2024-04-22T15:19:02.521 INFO:tasks.workunit.client.0.smithi096.stdout:1/862: symlink d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/d10d/l10e 0 2024-04-22T15:19:02.527 INFO:tasks.workunit.client.1.smithi196.stdout:4/524: mkdir d6/d14/d18/d26/d32/d67/d9a/da5 0 2024-04-22T15:19:02.528 INFO:tasks.workunit.client.1.smithi196.stdout:4/525: write d6/d13/d15/d2c/fa0 [469248,107690] 0 2024-04-22T15:19:02.530 INFO:tasks.workunit.client.1.smithi196.stdout:8/561: write d2/f3e [1528260,122832] 0 2024-04-22T15:19:02.530 INFO:tasks.workunit.client.1.smithi196.stdout:0/586: chown d3/d9/d25/l7a 3519070 1 2024-04-22T15:19:02.531 INFO:tasks.workunit.client.1.smithi196.stdout:8/562: stat d2/d3f/d4b/c85 0 2024-04-22T15:19:02.531 INFO:tasks.workunit.client.1.smithi196.stdout:0/587: chown d3/d6/dc/l41 1739 1 2024-04-22T15:19:02.535 INFO:tasks.workunit.client.0.smithi096.stdout:0/924: dwrite d7/de/d5b/d95/dc/d16/d65/d7c/fbd [0,4194304] 0 2024-04-22T15:19:02.537 INFO:tasks.workunit.client.0.smithi096.stdout:3/747: dread da/db/de/d19/d2d/d37/f5f [0,4194304] 0 2024-04-22T15:19:02.537 INFO:tasks.workunit.client.1.smithi196.stdout:6/567: mknod dd/d20/c9e 0 2024-04-22T15:19:02.538 INFO:tasks.workunit.client.0.smithi096.stdout:1/863: unlink d6/dc/d8a/d3f/d54/cec 0 2024-04-22T15:19:02.542 INFO:tasks.workunit.client.1.smithi196.stdout:6/568: dwrite fc [4194304,4194304] 0 2024-04-22T15:19:02.544 INFO:tasks.workunit.client.1.smithi196.stdout:7/531: mkdir d14/d26/d2a/d2b/da7 0 2024-04-22T15:19:02.546 INFO:tasks.workunit.client.0.smithi096.stdout:2/843: link d3/db/df/dce/d46/d69/dd7/lfc d3/db/df/d24/l10f 0 2024-04-22T15:19:02.549 INFO:tasks.workunit.client.1.smithi196.stdout:2/592: mkdir d1/d13/d25/d9f/ddd 0 2024-04-22T15:19:02.554 INFO:tasks.workunit.client.0.smithi096.stdout:8/821: truncate d4/d1c/d40/d48/fab 857070 0 2024-04-22T15:19:02.557 INFO:tasks.workunit.client.0.smithi096.stdout:8/822: readlink d4/d1c/d62/la3 0 2024-04-22T15:19:02.558 INFO:tasks.workunit.client.0.smithi096.stdout:4/814: dwrite d5/d7/d2c/ff0 [0,4194304] 0 2024-04-22T15:19:02.563 INFO:tasks.workunit.client.1.smithi196.stdout:8/563: symlink d2/d5/d1a/la0 0 2024-04-22T15:19:02.564 INFO:tasks.workunit.client.1.smithi196.stdout:8/564: write d2/d35/f92 [1025431,14752] 0 2024-04-22T15:19:02.564 INFO:tasks.workunit.client.1.smithi196.stdout:8/565: stat d2/d35/c69 0 2024-04-22T15:19:02.569 INFO:tasks.workunit.client.1.smithi196.stdout:1/526: write d1/db/dc/de/d96/d9d/f50 [2515431,9918] 0 2024-04-22T15:19:02.570 INFO:tasks.workunit.client.1.smithi196.stdout:6/569: rmdir dd/d2b/d8f 39 2024-04-22T15:19:02.574 INFO:tasks.workunit.client.0.smithi096.stdout:4/815: dread - d5/d7/d1f/d84/d6d/fcd zero size 2024-04-22T15:19:02.574 INFO:tasks.workunit.client.1.smithi196.stdout:3/564: sync 2024-04-22T15:19:02.580 INFO:tasks.workunit.client.0.smithi096.stdout:8/823: mknod d4/d10/d25/d4c/d76/d7f/c112 0 2024-04-22T15:19:02.581 INFO:tasks.workunit.client.0.smithi096.stdout:3/748: write da/d3f/f9f [34748,111849] 0 2024-04-22T15:19:02.581 INFO:tasks.workunit.client.0.smithi096.stdout:2/844: dread d3/db/d30/f3a [0,4194304] 0 2024-04-22T15:19:02.582 INFO:tasks.workunit.client.1.smithi196.stdout:5/583: mknod d1/d2/cbe 0 2024-04-22T15:19:02.585 INFO:tasks.workunit.client.0.smithi096.stdout:0/925: dwrite d7/d42/fa6 [0,4194304] 0 2024-04-22T15:19:02.586 INFO:tasks.workunit.client.0.smithi096.stdout:1/864: rename d6/dc/d8a/d3f/de9/c75 to d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/d10d/c10f 0 2024-04-22T15:19:02.589 INFO:tasks.workunit.client.0.smithi096.stdout:5/845: getdents d4/ddb/d3d/d4d/d59/d98/db9 0 2024-04-22T15:19:02.594 INFO:tasks.workunit.client.1.smithi196.stdout:8/566: rename d2/d8/d49/l50 to d2/d3f/d4b/la1 0 2024-04-22T15:19:02.598 INFO:tasks.workunit.client.1.smithi196.stdout:7/532: dwrite f2 [0,4194304] 0 2024-04-22T15:19:02.598 INFO:tasks.workunit.client.1.smithi196.stdout:1/527: unlink d1/db/d16/d1e/d49/d90/f99 0 2024-04-22T15:19:02.605 INFO:tasks.workunit.client.1.smithi196.stdout:6/570: mknod dd/d1f/d38/d40/c9f 0 2024-04-22T15:19:02.608 INFO:tasks.workunit.client.1.smithi196.stdout:8/567: dread d2/d35/f5f [0,4194304] 0 2024-04-22T15:19:02.611 INFO:tasks.workunit.client.0.smithi096.stdout:1/865: fsync d6/dc/d8a/d3f/d54/d32/d3b/f89 0 2024-04-22T15:19:02.612 INFO:tasks.workunit.client.0.smithi096.stdout:0/926: creat d7/d42/d4c/d55/da3/f13a x:0 0 0 2024-04-22T15:19:02.613 INFO:tasks.workunit.client.0.smithi096.stdout:5/846: creat d4/ddb/d3d/d4d/d59/d98/db9/f120 x:0 0 0 2024-04-22T15:19:02.614 INFO:tasks.workunit.client.1.smithi196.stdout:8/568: dwrite d2/dc/f3a [0,4194304] 0 2024-04-22T15:19:02.614 INFO:tasks.workunit.client.1.smithi196.stdout:9/566: getdents db/dd/de/d15/d31 0 2024-04-22T15:19:02.617 INFO:tasks.workunit.client.1.smithi196.stdout:4/526: creat d6/d13/d15/fa6 x:0 0 0 2024-04-22T15:19:02.620 INFO:tasks.workunit.client.1.smithi196.stdout:8/569: dwrite d2/d5/f23 [0,4194304] 0 2024-04-22T15:19:02.622 INFO:tasks.workunit.client.0.smithi096.stdout:6/658: sync 2024-04-22T15:19:02.622 INFO:tasks.workunit.client.0.smithi096.stdout:9/753: sync 2024-04-22T15:19:02.625 INFO:tasks.workunit.client.0.smithi096.stdout:3/749: creat da/db/de/ddc/d26/dc6/dd2/fed x:0 0 0 2024-04-22T15:19:02.625 INFO:tasks.workunit.client.1.smithi196.stdout:7/533: mknod d14/d4d/ca8 0 2024-04-22T15:19:02.627 INFO:tasks.workunit.client.1.smithi196.stdout:7/534: dread d14/d4a/f8c [0,4194304] 0 2024-04-22T15:19:02.627 INFO:tasks.workunit.client.1.smithi196.stdout:7/535: chown d14/d15 2852 1 2024-04-22T15:19:02.628 INFO:tasks.workunit.client.0.smithi096.stdout:3/750: readlink da/db/de/d87/lca 0 2024-04-22T15:19:02.628 INFO:tasks.workunit.client.0.smithi096.stdout:3/751: dread - da/db/dc/d59/fd7 zero size 2024-04-22T15:19:02.629 INFO:tasks.workunit.client.0.smithi096.stdout:3/752: chown da/db/de/d87/la3 211338 1 2024-04-22T15:19:02.631 INFO:tasks.workunit.client.0.smithi096.stdout:8/824: truncate d4/f12 1664813 0 2024-04-22T15:19:02.632 INFO:tasks.workunit.client.1.smithi196.stdout:1/528: mknod d1/db/dc/de/d43/ca9 0 2024-04-22T15:19:02.635 INFO:tasks.workunit.client.1.smithi196.stdout:3/565: creat d0/d15/d4d/d87/da6/fbd x:0 0 0 2024-04-22T15:19:02.636 INFO:tasks.workunit.client.1.smithi196.stdout:2/593: getdents d1/d23/d27/d2e 0 2024-04-22T15:19:02.637 INFO:tasks.workunit.client.0.smithi096.stdout:5/847: mkdir d4/ddb/d3d/d4d/d59/d8a/d121 0 2024-04-22T15:19:02.640 INFO:tasks.workunit.client.1.smithi196.stdout:9/567: symlink db/dd/de/d15/d31/lc2 0 2024-04-22T15:19:02.642 INFO:tasks.workunit.client.1.smithi196.stdout:3/566: dwrite d0/d15/d48/d51/f54 [0,4194304] 0 2024-04-22T15:19:02.643 INFO:tasks.workunit.client.1.smithi196.stdout:0/588: link d3/d9/d36/d56/d65/d7f/d30/c77 d3/d9/d36/d56/d65/d7f/d87/d6c/d88/cc8 0 2024-04-22T15:19:02.644 INFO:tasks.workunit.client.1.smithi196.stdout:4/527: read d6/d13/f1b [3522987,51163] 0 2024-04-22T15:19:02.646 INFO:tasks.workunit.client.1.smithi196.stdout:7/536: dread fc [0,4194304] 0 2024-04-22T15:19:02.649 INFO:tasks.workunit.client.0.smithi096.stdout:8/825: fsync d4/d10/f19 0 2024-04-22T15:19:02.654 INFO:tasks.workunit.client.1.smithi196.stdout:1/529: rmdir d1/db/dc/de 39 2024-04-22T15:19:02.660 INFO:tasks.workunit.client.0.smithi096.stdout:2/845: truncate d3/db/df/dce/d46/d69/fb9 3466325 0 2024-04-22T15:19:02.661 INFO:tasks.workunit.client.0.smithi096.stdout:0/927: dread d7/de/d5b/d95/dc/d16/d65/d7c/f9b [0,4194304] 0 2024-04-22T15:19:02.662 INFO:tasks.workunit.client.1.smithi196.stdout:6/571: dwrite dd/d2b/f3e [0,4194304] 0 2024-04-22T15:19:02.663 INFO:tasks.workunit.client.0.smithi096.stdout:6/659: fsync d3/d28/dd/f25 0 2024-04-22T15:19:02.666 INFO:tasks.workunit.client.1.smithi196.stdout:5/584: creat d1/fbf x:0 0 0 2024-04-22T15:19:02.671 INFO:tasks.workunit.client.1.smithi196.stdout:8/570: dwrite d2/d3f/d4b/d79/f80 [0,4194304] 0 2024-04-22T15:19:02.672 INFO:tasks.workunit.client.1.smithi196.stdout:3/567: mkdir d0/d7/d8/d70/dbe 0 2024-04-22T15:19:02.672 INFO:tasks.workunit.client.0.smithi096.stdout:9/754: dwrite d1/df/d95/d9d/db5/f52 [0,4194304] 0 2024-04-22T15:19:02.672 INFO:tasks.workunit.client.1.smithi196.stdout:3/568: stat d0/d15/da7/d4a/d44/f16 0 2024-04-22T15:19:02.678 INFO:tasks.workunit.client.0.smithi096.stdout:3/753: dwrite da/db/d12/d9b/fd6 [0,4194304] 0 2024-04-22T15:19:02.680 INFO:tasks.workunit.client.0.smithi096.stdout:3/754: chown da/db/dc/l4a 247536110 1 2024-04-22T15:19:02.684 INFO:tasks.workunit.client.1.smithi196.stdout:4/528: rename d6/d14/l21 to d6/d14/d18/d40/d85/la7 0 2024-04-22T15:19:02.684 INFO:tasks.workunit.client.0.smithi096.stdout:5/848: dwrite d4/ddb/d3d/d4d/d59/d5a/d5c/f5f [0,4194304] 0 2024-04-22T15:19:02.687 INFO:tasks.workunit.client.1.smithi196.stdout:7/537: mknod d14/d26/d5e/ca9 0 2024-04-22T15:19:02.689 INFO:tasks.workunit.client.0.smithi096.stdout:4/816: getdents d5/d7/dcb 0 2024-04-22T15:19:02.694 INFO:tasks.workunit.client.1.smithi196.stdout:2/594: mknod d1/d13/d25/d2a/ddb/cde 0 2024-04-22T15:19:02.695 INFO:tasks.workunit.client.1.smithi196.stdout:5/585: symlink d1/d78/lc0 0 2024-04-22T15:19:02.697 INFO:tasks.workunit.client.1.smithi196.stdout:5/586: truncate d1/d17/d1a/d2f/d30/f54 4224993 0 2024-04-22T15:19:02.698 INFO:tasks.workunit.client.1.smithi196.stdout:5/587: chown d1/d14/d6a/da7 2094410 1 2024-04-22T15:19:02.700 INFO:tasks.workunit.client.1.smithi196.stdout:5/588: write d1/d4/f9d [815561,17151] 0 2024-04-22T15:19:02.701 INFO:tasks.workunit.client.1.smithi196.stdout:3/569: mkdir d0/d15/d31/d55/d8b/dbf 0 2024-04-22T15:19:02.702 INFO:tasks.workunit.client.0.smithi096.stdout:1/866: getdents d6/dc/d8a/d29 0 2024-04-22T15:19:02.703 INFO:tasks.workunit.client.0.smithi096.stdout:9/755: creat d1/df/d95/d9d/db5/d43/d57/d5b/ffe x:0 0 0 2024-04-22T15:19:02.703 INFO:tasks.workunit.client.1.smithi196.stdout:3/570: write d0/d15/f12 [2334282,95210] 0 2024-04-22T15:19:02.703 INFO:tasks.workunit.client.1.smithi196.stdout:3/571: fsync d0/d15/f8e 0 2024-04-22T15:19:02.704 INFO:tasks.workunit.client.1.smithi196.stdout:3/572: chown d0/d15/d48/d51 962855 1 2024-04-22T15:19:02.706 INFO:tasks.workunit.client.1.smithi196.stdout:0/589: rename d3/d9/d36/d9f/db9 to d3/d9/d36/d56/d65/d7f/d87/d6c/dc9 0 2024-04-22T15:19:02.707 INFO:tasks.workunit.client.1.smithi196.stdout:6/572: dread dd/d20/f4c [0,4194304] 0 2024-04-22T15:19:02.708 INFO:tasks.workunit.client.1.smithi196.stdout:0/590: dread - d3/d9/d36/d56/d65/d7f/d87/d6c/fbf zero size 2024-04-22T15:19:02.708 INFO:tasks.workunit.client.1.smithi196.stdout:7/538: creat d14/d26/d5a/d5f/faa x:0 0 0 2024-04-22T15:19:02.710 INFO:tasks.workunit.client.1.smithi196.stdout:9/568: mknod db/dd/de/d15/d31/cc3 0 2024-04-22T15:19:02.712 INFO:tasks.workunit.client.0.smithi096.stdout:6/660: rename d3/d28/dd/d11/d2c/d6f/f89 to d3/d28/d4d/d82/dd8/fe2 0 2024-04-22T15:19:02.713 INFO:tasks.workunit.client.1.smithi196.stdout:8/571: symlink d2/dc/la2 0 2024-04-22T15:19:02.715 INFO:tasks.workunit.client.1.smithi196.stdout:5/589: creat d1/d17/d1a/d2f/d30/d6e/fc1 x:0 0 0 2024-04-22T15:19:02.717 INFO:tasks.workunit.client.0.smithi096.stdout:9/756: mknod d1/d4/d79/cff 0 2024-04-22T15:19:02.717 INFO:tasks.workunit.client.1.smithi196.stdout:9/569: dwrite db/dd/d1b/d2c/fb5 [0,4194304] 0 2024-04-22T15:19:02.719 INFO:tasks.workunit.client.0.smithi096.stdout:8/826: write d4/d2a/db7/fe0 [1546062,67401] 0 2024-04-22T15:19:02.720 INFO:tasks.workunit.client.0.smithi096.stdout:5/849: symlink d4/ddb/d3d/d4d/l122 0 2024-04-22T15:19:02.722 INFO:tasks.workunit.client.0.smithi096.stdout:8/827: truncate d4/d10/d25/ff1 436792 0 2024-04-22T15:19:02.723 INFO:tasks.workunit.client.0.smithi096.stdout:2/846: mkdir d3/db/df/d24/dc8/d103/d110 0 2024-04-22T15:19:02.724 INFO:tasks.workunit.client.0.smithi096.stdout:8/828: fsync d4/d10/d25/d4c/d76/d7f/f110 0 2024-04-22T15:19:02.725 INFO:tasks.workunit.client.1.smithi196.stdout:2/595: rename d1/d23/d84/dc8 to d1/d13/d1f/ddf 0 2024-04-22T15:19:02.726 INFO:tasks.workunit.client.1.smithi196.stdout:2/596: truncate d1/d13/fc7 658282 0 2024-04-22T15:19:02.729 INFO:tasks.workunit.client.0.smithi096.stdout:9/757: rename d1/df/d95/d9d/db5/d43/d57/d5b/d73/f9e to d1/df/d95/d9d/db5/d43/d57/d5b/da1/f100 0 2024-04-22T15:19:02.735 INFO:tasks.workunit.client.1.smithi196.stdout:8/572: creat d2/d8/d47/d68/fa3 x:0 0 0 2024-04-22T15:19:02.736 INFO:tasks.workunit.client.0.smithi096.stdout:3/755: dread da/d3f/fa5 [0,4194304] 0 2024-04-22T15:19:02.736 INFO:tasks.workunit.client.1.smithi196.stdout:5/590: creat d1/d4/fc2 x:0 0 0 2024-04-22T15:19:02.737 INFO:tasks.workunit.client.1.smithi196.stdout:5/591: stat d1/d4/c3c 0 2024-04-22T15:19:02.737 INFO:tasks.workunit.client.1.smithi196.stdout:5/592: stat d1/d78/db0/f84 0 2024-04-22T15:19:02.739 INFO:tasks.workunit.client.0.smithi096.stdout:3/756: chown da/db/dc/d59/f79 6 1 2024-04-22T15:19:02.740 INFO:tasks.workunit.client.0.smithi096.stdout:8/829: mkdir d4/d113 0 2024-04-22T15:19:02.741 INFO:tasks.workunit.client.0.smithi096.stdout:4/817: dread d5/d7/f9 [0,4194304] 0 2024-04-22T15:19:02.741 INFO:tasks.workunit.client.1.smithi196.stdout:7/539: rename d14/d25/d31/f69 to d14/d26/d28/fab 0 2024-04-22T15:19:02.742 INFO:tasks.workunit.client.1.smithi196.stdout:7/540: chown d14/d26/d2a/d72/f99 1025041 1 2024-04-22T15:19:02.742 INFO:tasks.workunit.client.1.smithi196.stdout:7/541: chown d14/f67 5513323 1 2024-04-22T15:19:02.744 INFO:tasks.workunit.client.1.smithi196.stdout:3/573: link d0/d7/d8/d70/f97 d0/d15/d31/d55/d8b/fc0 0 2024-04-22T15:19:02.745 INFO:tasks.workunit.client.0.smithi096.stdout:3/757: symlink da/db/de/d38/lee 0 2024-04-22T15:19:02.746 INFO:tasks.workunit.client.0.smithi096.stdout:3/758: readlink da/dab/ldf 0 2024-04-22T15:19:02.749 INFO:tasks.workunit.client.0.smithi096.stdout:0/928: dwrite d7/de/d5b/d64/fd3 [0,4194304] 0 2024-04-22T15:19:02.750 INFO:tasks.workunit.client.0.smithi096.stdout:9/758: dread d1/d4/f47 [0,4194304] 0 2024-04-22T15:19:02.751 INFO:tasks.workunit.client.0.smithi096.stdout:0/929: chown d7/d42/d4c/d55/d9c/df7/d10c/ca2 64792166 1 2024-04-22T15:19:02.807 INFO:tasks.workunit.client.1.smithi196.stdout:6/573: sync 2024-04-22T15:19:02.834 INFO:tasks.workunit.client.1.smithi196.stdout:2/597: creat d1/d13/d41/fe0 x:0 0 0 2024-04-22T15:19:02.835 INFO:tasks.workunit.client.1.smithi196.stdout:2/598: chown d1/f15 190209229 1 2024-04-22T15:19:02.835 INFO:tasks.workunit.client.1.smithi196.stdout:2/599: dread - d1/d23/fd1 zero size 2024-04-22T15:19:02.838 INFO:tasks.workunit.client.0.smithi096.stdout:8/830: unlink d4/d1c/d62/f10a 0 2024-04-22T15:19:02.839 INFO:tasks.workunit.client.1.smithi196.stdout:1/530: dwrite d1/db/dc/d26/f28 [0,4194304] 0 2024-04-22T15:19:02.844 INFO:tasks.workunit.client.1.smithi196.stdout:8/573: mkdir d2/d3f/d4b/d79/da4 0 2024-04-22T15:19:02.844 INFO:tasks.workunit.client.1.smithi196.stdout:8/574: readlink l1 0 2024-04-22T15:19:02.846 INFO:tasks.workunit.client.0.smithi096.stdout:2/847: rmdir d3/db/df/d24/d58/d5f/ddf/d10c 0 2024-04-22T15:19:02.848 INFO:tasks.workunit.client.1.smithi196.stdout:5/593: chown d1/d17/c1b 0 1 2024-04-22T15:19:02.853 INFO:tasks.workunit.client.0.smithi096.stdout:9/759: unlink d1/f32 0 2024-04-22T15:19:02.854 INFO:tasks.workunit.client.1.smithi196.stdout:3/574: mkdir d0/d15/d31/d55/d5d/dc1 0 2024-04-22T15:19:02.856 INFO:tasks.workunit.client.0.smithi096.stdout:0/930: symlink d7/de/d5b/dda/l13b 0 2024-04-22T15:19:02.859 INFO:tasks.workunit.client.0.smithi096.stdout:1/867: write d6/dc/d8a/d3f/d62/d7d/f84 [1623995,1289] 0 2024-04-22T15:19:02.865 INFO:tasks.workunit.client.1.smithi196.stdout:4/529: link d6/d14/d18/d26/f6a d6/d13/d15/d2c/fa8 0 2024-04-22T15:19:02.867 INFO:tasks.workunit.client.0.smithi096.stdout:4/818: dread d5/d7/f41 [0,4194304] 0 2024-04-22T15:19:02.867 INFO:tasks.workunit.client.1.smithi196.stdout:1/531: creat d1/db/d16/d44/faa x:0 0 0 2024-04-22T15:19:02.872 INFO:tasks.workunit.client.1.smithi196.stdout:5/594: unlink d1/d17/d1a/d2f/d30/f61 0 2024-04-22T15:19:02.874 INFO:tasks.workunit.client.1.smithi196.stdout:0/591: dwrite d3/d9/d36/d56/f7b [0,4194304] 0 2024-04-22T15:19:02.875 INFO:tasks.workunit.client.1.smithi196.stdout:0/592: write d3/de/d14/f39 [3834200,22625] 0 2024-04-22T15:19:02.879 INFO:tasks.workunit.client.1.smithi196.stdout:3/575: unlink d0/d15/d4d/c75 0 2024-04-22T15:19:02.886 INFO:tasks.workunit.client.1.smithi196.stdout:6/574: getdents dd/d1f/d37/d3c/d96 0 2024-04-22T15:19:02.886 INFO:tasks.workunit.client.0.smithi096.stdout:2/848: creat d3/d107/f111 x:0 0 0 2024-04-22T15:19:02.886 INFO:tasks.workunit.client.0.smithi096.stdout:2/849: chown d3/db/df/d24/d58/lcc 108781 1 2024-04-22T15:19:02.888 INFO:tasks.workunit.client.1.smithi196.stdout:1/532: fsync d1/db/dc/d26/f30 0 2024-04-22T15:19:02.889 INFO:tasks.workunit.client.0.smithi096.stdout:3/759: write da/db/d10/f6e [2213093,78595] 0 2024-04-22T15:19:02.891 INFO:tasks.workunit.client.0.smithi096.stdout:1/868: truncate d6/dc/d8a/d3f/d62/f70 1077671 0 2024-04-22T15:19:02.893 INFO:tasks.workunit.client.1.smithi196.stdout:7/542: dwrite f10 [0,4194304] 0 2024-04-22T15:19:02.893 INFO:tasks.workunit.client.0.smithi096.stdout:1/869: readlink d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/d10d/l10e 0 2024-04-22T15:19:02.894 INFO:tasks.workunit.client.0.smithi096.stdout:4/819: mkdir d5/d5b/d11f 0 2024-04-22T15:19:02.895 INFO:tasks.workunit.client.0.smithi096.stdout:4/820: readlink d5/d7/d2c/d3d/d4f/l73 0 2024-04-22T15:19:02.896 INFO:tasks.workunit.client.0.smithi096.stdout:2/850: dread - d3/db/df/dce/d46/d69/fe3 zero size 2024-04-22T15:19:02.898 INFO:tasks.workunit.client.1.smithi196.stdout:5/595: symlink d1/d17/d1c/lc3 0 2024-04-22T15:19:02.899 INFO:tasks.workunit.client.0.smithi096.stdout:3/760: mkdir da/db/d9a/def 0 2024-04-22T15:19:02.899 INFO:tasks.workunit.client.0.smithi096.stdout:9/760: write d1/d85/f94 [196688,103933] 0 2024-04-22T15:19:02.900 INFO:tasks.workunit.client.0.smithi096.stdout:9/761: chown d1/d85/df4/ffa 106 1 2024-04-22T15:19:02.901 INFO:tasks.workunit.client.0.smithi096.stdout:6/661: write d3/d28/d4d/d82/dd8/fe2 [1578817,3018] 0 2024-04-22T15:19:02.903 INFO:tasks.workunit.client.0.smithi096.stdout:1/870: creat d6/dc/d8a/d3f/d54/d32/d3b/d3d/f110 x:0 0 0 2024-04-22T15:19:02.903 INFO:tasks.workunit.client.0.smithi096.stdout:8/831: dwrite d4/d2a/fbe [0,4194304] 0 2024-04-22T15:19:02.908 INFO:tasks.workunit.client.1.smithi196.stdout:0/593: creat d3/d6/dc/d8c/d8d/d98/fca x:0 0 0 2024-04-22T15:19:02.908 INFO:tasks.workunit.client.1.smithi196.stdout:0/594: chown d3/de/d1c/d27 461 1 2024-04-22T15:19:02.912 INFO:tasks.workunit.client.0.smithi096.stdout:0/931: dwrite d7/de/d5b/d95/d15/f36 [0,4194304] 0 2024-04-22T15:19:02.914 INFO:tasks.workunit.client.0.smithi096.stdout:9/762: read d1/df/d95/d9d/db5/dd/f2b [1242833,50981] 0 2024-04-22T15:19:02.915 INFO:tasks.workunit.client.0.smithi096.stdout:2/851: chown d3/db/df/l25 2958 1 2024-04-22T15:19:02.915 INFO:tasks.workunit.client.1.smithi196.stdout:6/575: creat dd/d1f/d38/d40/fa0 x:0 0 0 2024-04-22T15:19:02.917 INFO:tasks.workunit.client.1.smithi196.stdout:5/596: dread d1/d17/d1a/d2f/d30/d5e/f77 [0,4194304] 0 2024-04-22T15:19:02.917 INFO:tasks.workunit.client.1.smithi196.stdout:6/576: readlink dd/d1f/d37/d58/l72 0 2024-04-22T15:19:02.917 INFO:tasks.workunit.client.0.smithi096.stdout:3/761: dread da/f8f [4194304,4194304] 0 2024-04-22T15:19:02.923 INFO:tasks.workunit.client.1.smithi196.stdout:5/597: dwrite d1/d9/f75 [0,4194304] 0 2024-04-22T15:19:02.923 INFO:tasks.workunit.client.0.smithi096.stdout:2/852: dwrite d3/db/f83 [0,4194304] 0 2024-04-22T15:19:02.924 INFO:tasks.workunit.client.1.smithi196.stdout:8/575: creat d2/d3f/fa5 x:0 0 0 2024-04-22T15:19:02.927 INFO:tasks.workunit.client.0.smithi096.stdout:4/821: write d5/d7/d1f/d84/d6d/d98/f9b [1287374,118465] 0 2024-04-22T15:19:02.929 INFO:tasks.workunit.client.1.smithi196.stdout:7/543: mknod d14/d15/cac 0 2024-04-22T15:19:02.931 INFO:tasks.workunit.client.0.smithi096.stdout:0/932: unlink d7/de/d5b/d95/dc/d16/d65/d7c/ce7 0 2024-04-22T15:19:02.932 INFO:tasks.workunit.client.1.smithi196.stdout:9/570: dwrite db/dd/de/d15/d31/fb7 [0,4194304] 0 2024-04-22T15:19:02.935 INFO:tasks.workunit.client.1.smithi196.stdout:0/595: creat d3/d9/d36/d9f/fcb x:0 0 0 2024-04-22T15:19:02.937 INFO:tasks.workunit.client.1.smithi196.stdout:3/576: creat d0/d15/d31/d9d/d9e/fc2 x:0 0 0 2024-04-22T15:19:02.939 INFO:tasks.workunit.client.1.smithi196.stdout:3/577: dread d0/d15/f92 [0,4194304] 0 2024-04-22T15:19:02.940 INFO:tasks.workunit.client.1.smithi196.stdout:2/600: link d1/d13/d25/c30 d1/d13/d1f/ddf/da1/ce1 0 2024-04-22T15:19:02.940 INFO:tasks.workunit.client.1.smithi196.stdout:3/578: chown d0/d15/da7 9 1 2024-04-22T15:19:02.941 INFO:tasks.workunit.client.1.smithi196.stdout:2/601: chown d1/d23/d27/d2e/d55/d5b/fb3 58628361 1 2024-04-22T15:19:02.941 INFO:tasks.workunit.client.0.smithi096.stdout:0/933: dwrite d7/d42/dfd/f128 [0,4194304] 0 2024-04-22T15:19:02.941 INFO:tasks.workunit.client.0.smithi096.stdout:3/762: creat da/db/d9a/dd1/ff0 x:0 0 0 2024-04-22T15:19:02.942 INFO:tasks.workunit.client.1.smithi196.stdout:4/530: link d6/d90/c71 d6/d90/ca9 0 2024-04-22T15:19:02.944 INFO:tasks.workunit.client.0.smithi096.stdout:0/934: write d7/d42/d4c/d55/da3/f13a [315092,37102] 0 2024-04-22T15:19:02.945 INFO:tasks.workunit.client.0.smithi096.stdout:0/935: chown d7/de/d5b/d95/dc/d16/c7f 24 1 2024-04-22T15:19:02.946 INFO:tasks.workunit.client.1.smithi196.stdout:5/598: dread d1/d17/d1c/f52 [0,4194304] 0 2024-04-22T15:19:02.947 INFO:tasks.workunit.client.0.smithi096.stdout:3/763: dread - da/db/de/d19/d2d/d37/fb1 zero size 2024-04-22T15:19:02.949 INFO:tasks.workunit.client.1.smithi196.stdout:1/533: unlink d1/db/dc/de/d96/d9d/l77 0 2024-04-22T15:19:02.950 INFO:tasks.workunit.client.0.smithi096.stdout:2/853: creat d3/db/f112 x:0 0 0 2024-04-22T15:19:02.950 INFO:tasks.workunit.client.1.smithi196.stdout:1/534: readlink d1/db/dc/l7d 0 2024-04-22T15:19:02.951 INFO:tasks.workunit.client.0.smithi096.stdout:8/832: link d4/d10/d25/d38/f79 d4/d1c/de2/f114 0 2024-04-22T15:19:02.952 INFO:tasks.workunit.client.0.smithi096.stdout:8/833: chown d4/d2a/d7e 6810258 1 2024-04-22T15:19:02.954 INFO:tasks.workunit.client.0.smithi096.stdout:6/662: dread d3/d28/dd/d11/d2c/fb1 [0,4194304] 0 2024-04-22T15:19:02.955 INFO:tasks.workunit.client.1.smithi196.stdout:9/571: symlink db/dd/d1b/d2c/lc4 0 2024-04-22T15:19:02.959 INFO:tasks.workunit.client.0.smithi096.stdout:3/764: creat da/db/de/ddc/d68/dbc/ff1 x:0 0 0 2024-04-22T15:19:02.959 INFO:tasks.workunit.client.0.smithi096.stdout:3/765: stat da/db/d12 0 2024-04-22T15:19:02.960 INFO:tasks.workunit.client.1.smithi196.stdout:3/579: write d0/d7/f38 [4790860,101915] 0 2024-04-22T15:19:02.961 INFO:tasks.workunit.client.1.smithi196.stdout:2/602: dread d1/f1b [0,4194304] 0 2024-04-22T15:19:02.961 INFO:tasks.workunit.client.1.smithi196.stdout:3/580: dread d0/d15/da7/d85/f9c [0,4194304] 0 2024-04-22T15:19:02.963 INFO:tasks.workunit.client.0.smithi096.stdout:8/834: fsync d4/d10/d25/d38/f54 0 2024-04-22T15:19:02.964 INFO:tasks.workunit.client.1.smithi196.stdout:5/599: symlink d1/d17/d1a/d2f/d30/d6e/d7a/lc4 0 2024-04-22T15:19:02.965 INFO:tasks.workunit.client.0.smithi096.stdout:3/766: creat da/db/d12/d9b/ff2 x:0 0 0 2024-04-22T15:19:02.966 INFO:tasks.workunit.client.0.smithi096.stdout:3/767: readlink da/db/l48 0 2024-04-22T15:19:02.966 INFO:tasks.workunit.client.1.smithi196.stdout:1/535: mkdir d1/db/dc/de/d5a/d94/dab 0 2024-04-22T15:19:02.969 INFO:tasks.workunit.client.1.smithi196.stdout:2/603: dread d1/d23/d27/d2e/f88 [0,4194304] 0 2024-04-22T15:19:02.969 INFO:tasks.workunit.client.0.smithi096.stdout:3/768: write da/db/d10/f6e [1223198,108695] 0 2024-04-22T15:19:02.970 INFO:tasks.workunit.client.0.smithi096.stdout:6/663: write d3/d28/d4d/d82/d9e/fcc [409,75058] 0 2024-04-22T15:19:02.972 INFO:tasks.workunit.client.1.smithi196.stdout:6/577: rename dd/d20/d5c/d5e/d76/l97 to dd/la1 0 2024-04-22T15:19:02.973 INFO:tasks.workunit.client.0.smithi096.stdout:8/835: truncate d4/d10/d25/d4c/f5d 2269740 0 2024-04-22T15:19:02.976 INFO:tasks.workunit.client.1.smithi196.stdout:4/531: truncate d6/d13/f16 318144 0 2024-04-22T15:19:02.976 INFO:tasks.workunit.client.1.smithi196.stdout:4/532: chown d6/d14/d18/f36 1085 1 2024-04-22T15:19:02.977 INFO:tasks.workunit.client.0.smithi096.stdout:6/664: rmdir d3/d28/d4d/d82/d9e/db0 39 2024-04-22T15:19:02.977 INFO:tasks.workunit.client.1.smithi196.stdout:4/533: readlink d6/d14/d18/d40/l48 0 2024-04-22T15:19:02.978 INFO:tasks.workunit.client.0.smithi096.stdout:6/665: write d3/d28/d4d/d82/dd8/fe2 [1332601,19341] 0 2024-04-22T15:19:02.978 INFO:tasks.workunit.client.1.smithi196.stdout:3/581: creat d0/d15/d48/d51/fc3 x:0 0 0 2024-04-22T15:19:02.978 INFO:tasks.workunit.client.1.smithi196.stdout:3/582: dread - d0/d15/da7/d4a/d44/d8c/faa zero size 2024-04-22T15:19:02.981 INFO:tasks.workunit.client.1.smithi196.stdout:1/536: mknod d1/db/dc/de/d43/d5e/cac 0 2024-04-22T15:19:02.982 INFO:tasks.workunit.client.1.smithi196.stdout:9/572: symlink db/dd/d1b/lc5 0 2024-04-22T15:19:02.983 INFO:tasks.workunit.client.1.smithi196.stdout:1/537: write d1/db/d16/d1e/f88 [996522,52977] 0 2024-04-22T15:19:02.983 INFO:tasks.workunit.client.0.smithi096.stdout:6/666: creat d3/d3a/fe3 x:0 0 0 2024-04-22T15:19:02.984 INFO:tasks.workunit.client.1.smithi196.stdout:3/583: dwrite d0/d15/da7/d18/f42 [0,4194304] 0 2024-04-22T15:19:02.984 INFO:tasks.workunit.client.0.smithi096.stdout:6/667: fsync f0 0 2024-04-22T15:19:02.985 INFO:tasks.workunit.client.1.smithi196.stdout:2/604: mknod d1/d13/d25/d9f/ce2 0 2024-04-22T15:19:02.986 INFO:tasks.workunit.client.0.smithi096.stdout:8/836: getdents d4/d1c/d62/dc3/df9 0 2024-04-22T15:19:02.988 INFO:tasks.workunit.client.1.smithi196.stdout:6/578: creat dd/d20/d5c/d6a/fa2 x:0 0 0 2024-04-22T15:19:02.991 INFO:tasks.workunit.client.0.smithi096.stdout:3/769: rename da/db/d12/fa9 to da/db/de/ddc/ff3 0 2024-04-22T15:19:02.992 INFO:tasks.workunit.client.0.smithi096.stdout:3/770: stat da/db/d6a/ld0 0 2024-04-22T15:19:02.993 INFO:tasks.workunit.client.1.smithi196.stdout:7/544: dwrite fd [0,4194304] 0 2024-04-22T15:19:02.997 INFO:tasks.workunit.client.1.smithi196.stdout:4/534: fdatasync d6/d10/f63 0 2024-04-22T15:19:02.998 INFO:tasks.workunit.client.0.smithi096.stdout:6/668: fdatasync d3/d28/dd/d11/d2c/d6f/dd4/dd6/d51/f88 0 2024-04-22T15:19:02.998 INFO:tasks.workunit.client.0.smithi096.stdout:2/854: write d3/d1f/d2f/f47 [770114,106219] 0 2024-04-22T15:19:02.999 INFO:tasks.workunit.client.1.smithi196.stdout:4/535: truncate d6/d13/d15/d2c/d62/f96 760702 0 2024-04-22T15:19:03.000 INFO:tasks.workunit.client.0.smithi096.stdout:0/936: dwrite d7/f112 [0,4194304] 0 2024-04-22T15:19:03.001 INFO:tasks.workunit.client.0.smithi096.stdout:6/669: stat d3/d28/daf 0 2024-04-22T15:19:03.003 INFO:tasks.workunit.client.0.smithi096.stdout:3/771: creat da/db/d6a/ff4 x:0 0 0 2024-04-22T15:19:03.003 INFO:tasks.workunit.client.1.smithi196.stdout:8/576: truncate d2/d8/f76 3700201 0 2024-04-22T15:19:03.004 INFO:tasks.workunit.client.1.smithi196.stdout:0/596: dwrite d3/de/d1c/d27/f38 [0,4194304] 0 2024-04-22T15:19:03.004 INFO:tasks.workunit.client.1.smithi196.stdout:0/597: write d3/f47 [5488028,125084] 0 2024-04-22T15:19:03.007 INFO:tasks.workunit.client.1.smithi196.stdout:2/605: symlink d1/d23/d27/d89/le3 0 2024-04-22T15:19:03.011 INFO:tasks.workunit.client.1.smithi196.stdout:6/579: rename dd/d1f/c25 to dd/d1f/d38/d40/ca3 0 2024-04-22T15:19:03.011 INFO:tasks.workunit.client.1.smithi196.stdout:0/598: dwrite d3/de/d14/d5a/f5f [0,4194304] 0 2024-04-22T15:19:03.011 INFO:tasks.workunit.client.1.smithi196.stdout:4/536: rename d6/d13/d15/d2c/d62/d78/d8d to d6/d13/d15/d2c/d62/d78/d8d/daa 22 2024-04-22T15:19:03.012 INFO:tasks.workunit.client.1.smithi196.stdout:6/580: fdatasync dd/d1f/f48 0 2024-04-22T15:19:03.013 INFO:tasks.workunit.client.0.smithi096.stdout:2/855: creat d3/db/df/d10/db5/f113 x:0 0 0 2024-04-22T15:19:03.014 INFO:tasks.workunit.client.1.smithi196.stdout:7/545: mknod d14/d15/cad 0 2024-04-22T15:19:03.015 INFO:tasks.workunit.client.0.smithi096.stdout:0/937: symlink d7/da7/l13c 0 2024-04-22T15:19:03.018 INFO:tasks.workunit.client.1.smithi196.stdout:5/600: link d1/d9/l23 d1/d4/d92/db5/lc5 0 2024-04-22T15:19:03.019 INFO:tasks.workunit.client.0.smithi096.stdout:8/837: creat d4/f115 x:0 0 0 2024-04-22T15:19:03.023 INFO:tasks.workunit.client.0.smithi096.stdout:0/938: creat d7/de/d3a/d61/d79/f13d x:0 0 0 2024-04-22T15:19:03.027 INFO:tasks.workunit.client.1.smithi196.stdout:2/606: creat d1/d23/d27/d2e/d55/d5b/d7c/fe4 x:0 0 0 2024-04-22T15:19:03.028 INFO:tasks.workunit.client.0.smithi096.stdout:0/939: creat d7/d42/d4c/d55/da3/f13e x:0 0 0 2024-04-22T15:19:03.032 INFO:tasks.workunit.client.1.smithi196.stdout:4/537: mknod d6/d13/d15/d2c/d62/d78/d8b/cab 0 2024-04-22T15:19:03.035 INFO:tasks.workunit.client.0.smithi096.stdout:0/940: rename d7/f32 to d7/de/d5b/d95/d15/f13f 0 2024-04-22T15:19:03.037 INFO:tasks.workunit.client.1.smithi196.stdout:6/581: creat dd/d1f/d4e/d53/fa4 x:0 0 0 2024-04-22T15:19:03.038 INFO:tasks.workunit.client.1.smithi196.stdout:7/546: creat d14/d25/d31/fae x:0 0 0 2024-04-22T15:19:03.040 INFO:tasks.workunit.client.0.smithi096.stdout:0/941: mknod d7/d8f/c140 0 2024-04-22T15:19:03.040 INFO:tasks.workunit.client.1.smithi196.stdout:5/601: fsync d1/d17/f21 0 2024-04-22T15:19:03.042 INFO:tasks.workunit.client.1.smithi196.stdout:9/573: creat db/dd/fc6 x:0 0 0 2024-04-22T15:19:03.043 INFO:tasks.workunit.client.0.smithi096.stdout:0/942: mkdir d7/d42/dd4/d141 0 2024-04-22T15:19:03.046 INFO:tasks.workunit.client.1.smithi196.stdout:4/538: dread d6/d13/f6b [0,4194304] 0 2024-04-22T15:19:03.046 INFO:tasks.workunit.client.0.smithi096.stdout:0/943: chown d7/de/fbf 3753966 1 2024-04-22T15:19:03.048 INFO:tasks.workunit.client.1.smithi196.stdout:0/599: unlink d3/de/l8f 0 2024-04-22T15:19:03.050 INFO:tasks.workunit.client.1.smithi196.stdout:6/582: mkdir dd/d1f/d4e/d53/d7f/da5 0 2024-04-22T15:19:03.050 INFO:tasks.workunit.client.1.smithi196.stdout:6/583: readlink dd/d1f/d37/d3c/l4a 0 2024-04-22T15:19:03.053 INFO:tasks.workunit.client.1.smithi196.stdout:8/577: dread d2/dc/fe [4194304,4194304] 0 2024-04-22T15:19:03.056 INFO:tasks.workunit.client.1.smithi196.stdout:1/538: getdents d1/db/dc/d26 0 2024-04-22T15:19:03.058 INFO:tasks.workunit.client.1.smithi196.stdout:1/539: write d1/db/d16/d6d/f7c [911947,38082] 0 2024-04-22T15:19:03.063 INFO:tasks.workunit.client.1.smithi196.stdout:8/578: read - d2/d3f/d4b/f6f zero size 2024-04-22T15:19:03.065 INFO:tasks.workunit.client.1.smithi196.stdout:9/574: symlink db/d11/d29/d57/da7/lc7 0 2024-04-22T15:19:03.065 INFO:tasks.workunit.client.1.smithi196.stdout:2/607: link d1/d23/d27/d2e/d55/d5b/d93/d97/la9 d1/d1d/le5 0 2024-04-22T15:19:03.066 INFO:tasks.workunit.client.0.smithi096.stdout:0/944: dread d7/de/d5b/d95/dc/d26/fb5 [0,4194304] 0 2024-04-22T15:19:03.068 INFO:tasks.workunit.client.1.smithi196.stdout:4/539: creat d6/d14/d18/d26/d32/d67/d9a/da5/fac x:0 0 0 2024-04-22T15:19:03.068 INFO:tasks.workunit.client.1.smithi196.stdout:7/547: dread d14/d15/f1a [0,4194304] 0 2024-04-22T15:19:03.069 INFO:tasks.workunit.client.1.smithi196.stdout:0/600: symlink d3/de/lcc 0 2024-04-22T15:19:03.070 INFO:tasks.workunit.client.1.smithi196.stdout:0/601: write d3/f47 [6654299,91040] 0 2024-04-22T15:19:03.071 INFO:tasks.workunit.client.0.smithi096.stdout:3/772: dwrite da/db/de/ddc/f61 [0,4194304] 0 2024-04-22T15:19:03.073 INFO:tasks.workunit.client.1.smithi196.stdout:3/584: sync 2024-04-22T15:19:03.073 INFO:tasks.workunit.client.1.smithi196.stdout:7/548: dwrite f10 [4194304,4194304] 0 2024-04-22T15:19:03.074 INFO:tasks.workunit.client.1.smithi196.stdout:7/549: fdatasync d14/d26/d5e/f7b 0 2024-04-22T15:19:03.075 INFO:tasks.workunit.client.0.smithi096.stdout:2/856: truncate d3/db/d10d/fcf 415406 0 2024-04-22T15:19:03.076 INFO:tasks.workunit.client.0.smithi096.stdout:3/773: mknod da/db/de/ddc/d68/dbc/cf5 0 2024-04-22T15:19:03.078 INFO:tasks.workunit.client.1.smithi196.stdout:5/602: creat d1/d2/fc6 x:0 0 0 2024-04-22T15:19:03.086 INFO:tasks.workunit.client.1.smithi196.stdout:5/603: dwrite d1/d4/fc2 [0,4194304] 0 2024-04-22T15:19:03.128 INFO:tasks.workunit.client.0.smithi096.stdout:0/945: rename d7/de/d5b/d95/d66/lf9 to d7/de/d5b/d95/dc/d16/l142 0 2024-04-22T15:19:03.132 INFO:tasks.workunit.client.1.smithi196.stdout:2/608: truncate d1/d23/d27/d2e/d55/d5b/d7c/f91 1680710 0 2024-04-22T15:19:03.135 INFO:tasks.workunit.client.0.smithi096.stdout:2/857: rename d3/db/df/dce/ffd to d3/d1f/d2f/f114 0 2024-04-22T15:19:03.148 INFO:tasks.workunit.client.1.smithi196.stdout:3/585: mknod d0/d7/d6f/cc4 0 2024-04-22T15:19:03.149 INFO:tasks.workunit.client.1.smithi196.stdout:6/584: truncate dd/d2b/f3e 2163820 0 2024-04-22T15:19:03.150 INFO:tasks.workunit.client.1.smithi196.stdout:3/586: read d0/d7/d8/d70/f7e [890042,12724] 0 2024-04-22T15:19:03.151 INFO:tasks.workunit.client.1.smithi196.stdout:7/550: creat d14/d15/d91/d9a/faf x:0 0 0 2024-04-22T15:19:03.158 INFO:tasks.workunit.client.1.smithi196.stdout:9/575: mknod db/dd/d1b/d2c/cc8 0 2024-04-22T15:19:03.160 INFO:tasks.workunit.client.0.smithi096.stdout:3/774: write da/d3f/fa5 [3194949,49156] 0 2024-04-22T15:19:03.164 INFO:tasks.workunit.client.0.smithi096.stdout:2/858: rename d3/d1f to d3/db/df/d24/d58/d5f/dfe/d115 0 2024-04-22T15:19:03.165 INFO:tasks.workunit.client.0.smithi096.stdout:0/946: write d7/d42/d4c/d55/f12a [3608569,79366] 0 2024-04-22T15:19:03.167 INFO:tasks.workunit.client.1.smithi196.stdout:5/604: sync 2024-04-22T15:19:03.167 INFO:tasks.workunit.client.0.smithi096.stdout:2/859: dread d3/d13/fe5 [0,4194304] 0 2024-04-22T15:19:03.170 INFO:tasks.workunit.client.0.smithi096.stdout:0/947: creat d7/de/d3a/d61/dbb/f143 x:0 0 0 2024-04-22T15:19:03.176 INFO:tasks.workunit.client.0.smithi096.stdout:2/860: mknod d3/d13/c116 0 2024-04-22T15:19:03.180 INFO:tasks.workunit.client.0.smithi096.stdout:2/861: symlink d3/db/df/dce/l117 0 2024-04-22T15:19:03.181 INFO:tasks.workunit.client.0.smithi096.stdout:2/862: dread - d3/db/df/f10e zero size 2024-04-22T15:19:03.183 INFO:tasks.workunit.client.1.smithi196.stdout:4/540: write d6/d14/d18/d26/f6a [4966628,104725] 0 2024-04-22T15:19:03.185 INFO:tasks.workunit.client.0.smithi096.stdout:2/863: creat d3/db/df/d24/d58/d5f/dfe/d115/f118 x:0 0 0 2024-04-22T15:19:03.189 INFO:tasks.workunit.client.0.smithi096.stdout:2/864: chown d3/db/d30/l4f 2747 1 2024-04-22T15:19:03.189 INFO:tasks.workunit.client.0.smithi096.stdout:2/865: chown d3/db/d30/f3a 0 1 2024-04-22T15:19:03.190 INFO:tasks.workunit.client.0.smithi096.stdout:3/775: write da/db/de/ddc/d26/d52/f86 [414603,15637] 0 2024-04-22T15:19:03.192 INFO:tasks.workunit.client.0.smithi096.stdout:3/776: write da/db/dc/f91 [1643132,67] 0 2024-04-22T15:19:03.193 INFO:tasks.workunit.client.0.smithi096.stdout:2/866: creat d3/db/df/d24/d58/d5f/dfe/f119 x:0 0 0 2024-04-22T15:19:03.196 INFO:tasks.workunit.client.0.smithi096.stdout:3/777: dread da/db/d10/f90 [0,4194304] 0 2024-04-22T15:19:03.200 INFO:tasks.workunit.client.0.smithi096.stdout:0/948: write d7/de/d5b/d95/dc/d16/d65/d7c/f9b [3707939,63050] 0 2024-04-22T15:19:03.202 INFO:tasks.workunit.client.0.smithi096.stdout:3/778: creat da/db/de/d8b/ff6 x:0 0 0 2024-04-22T15:19:03.204 INFO:tasks.workunit.client.0.smithi096.stdout:2/867: creat d3/db/df/d24/dc8/d103/d110/f11a x:0 0 0 2024-04-22T15:19:03.206 INFO:tasks.workunit.client.0.smithi096.stdout:3/779: mknod da/d3f/cf7 0 2024-04-22T15:19:03.208 INFO:tasks.workunit.client.0.smithi096.stdout:0/949: symlink d7/de/d5b/d95/d11d/l144 0 2024-04-22T15:19:03.210 INFO:tasks.workunit.client.0.smithi096.stdout:0/950: chown d7/d42/d4c/d55/d9c/df7/d10c/lb8 182177 1 2024-04-22T15:19:03.211 INFO:tasks.workunit.client.0.smithi096.stdout:2/868: dwrite d3/db/df/d24/d58/d5f/dfe/d115/f51 [0,4194304] 0 2024-04-22T15:19:03.213 INFO:tasks.workunit.client.1.smithi196.stdout:1/540: rename d1/db/dc/d26/d2e/d32 to d1/db/dc/dad 0 2024-04-22T15:19:03.213 INFO:tasks.workunit.client.1.smithi196.stdout:0/602: mknod d3/db4/dc7/ccd 0 2024-04-22T15:19:03.217 INFO:tasks.workunit.client.1.smithi196.stdout:3/587: unlink d0/d15/d48/d51/d5e/c5f 0 2024-04-22T15:19:03.218 INFO:tasks.workunit.client.1.smithi196.stdout:7/551: symlink d14/d26/d2a/d2b/lb0 0 2024-04-22T15:19:03.222 INFO:tasks.workunit.client.0.smithi096.stdout:0/951: dread d7/de/d5b/d95/dc/d26/fb5 [0,4194304] 0 2024-04-22T15:19:03.222 INFO:tasks.workunit.client.0.smithi096.stdout:3/780: dread da/db/de/d8b/f92 [0,4194304] 0 2024-04-22T15:19:03.223 INFO:tasks.workunit.client.1.smithi196.stdout:1/541: dwrite d1/db/dc/d26/d2e/f3e [0,4194304] 0 2024-04-22T15:19:03.223 INFO:tasks.workunit.client.1.smithi196.stdout:0/603: dwrite d3/d6/dc/f58 [0,4194304] 0 2024-04-22T15:19:03.225 INFO:tasks.workunit.client.1.smithi196.stdout:1/542: fdatasync d1/db/fa8 0 2024-04-22T15:19:03.230 INFO:tasks.workunit.client.1.smithi196.stdout:4/541: symlink d6/d13/d15/d2c/d62/lad 0 2024-04-22T15:19:03.231 INFO:tasks.workunit.client.1.smithi196.stdout:4/542: truncate d6/d13/d15/d2c/d62/f96 1622232 0 2024-04-22T15:19:03.237 INFO:tasks.workunit.client.1.smithi196.stdout:7/552: mkdir d14/d26/d2a/d72/db1 0 2024-04-22T15:19:03.243 INFO:tasks.workunit.client.1.smithi196.stdout:6/585: truncate dd/d1f/d37/f5d 16694 0 2024-04-22T15:19:03.246 INFO:tasks.workunit.client.1.smithi196.stdout:0/604: symlink d3/d9/d25/lce 0 2024-04-22T15:19:03.250 INFO:tasks.workunit.client.1.smithi196.stdout:1/543: dread d1/db/dc/d26/f30 [0,4194304] 0 2024-04-22T15:19:03.252 INFO:tasks.workunit.client.1.smithi196.stdout:0/605: dwrite d3/d6/dc/d8c/d8d/d98/fca [0,4194304] 0 2024-04-22T15:19:03.252 INFO:tasks.workunit.client.1.smithi196.stdout:1/544: write d1/db/d16/d44/f92 [922504,65265] 0 2024-04-22T15:19:03.258 INFO:tasks.workunit.client.1.smithi196.stdout:0/606: dread d3/de/d14/d5a/f5f [0,4194304] 0 2024-04-22T15:19:03.259 INFO:tasks.workunit.client.1.smithi196.stdout:3/588: mkdir d0/d15/da7/d4a/dc5 0 2024-04-22T15:19:03.260 INFO:tasks.workunit.client.1.smithi196.stdout:7/553: fdatasync d14/d26/d2a/d2b/f70 0 2024-04-22T15:19:03.262 INFO:tasks.workunit.client.1.smithi196.stdout:7/554: write d14/d26/d28/d29/f8b [251920,96293] 0 2024-04-22T15:19:03.262 INFO:tasks.workunit.client.1.smithi196.stdout:7/555: dread - d14/d25/f79 zero size 2024-04-22T15:19:03.267 INFO:tasks.workunit.client.1.smithi196.stdout:9/576: link db/dd/d1b/c46 db/d11/d29/d57/da7/cc9 0 2024-04-22T15:19:03.268 INFO:tasks.workunit.client.1.smithi196.stdout:9/577: write db/d11/f4d [4843360,27627] 0 2024-04-22T15:19:03.275 INFO:tasks.workunit.client.1.smithi196.stdout:4/543: symlink d6/d14/d18/lae 0 2024-04-22T15:19:03.278 INFO:tasks.workunit.client.1.smithi196.stdout:0/607: mknod d3/ccf 0 2024-04-22T15:19:03.280 INFO:tasks.workunit.client.1.smithi196.stdout:4/544: dwrite d6/d14/d18/d26/d32/d53/f94 [0,4194304] 0 2024-04-22T15:19:03.283 INFO:tasks.workunit.client.1.smithi196.stdout:7/556: symlink d14/d15/d91/d9a/lb2 0 2024-04-22T15:19:03.284 INFO:tasks.workunit.client.1.smithi196.stdout:7/557: write d14/d26/d28/f93 [754543,6307] 0 2024-04-22T15:19:03.287 INFO:tasks.workunit.client.0.smithi096.stdout:2/869: creat d3/db/df/d10/f11b x:0 0 0 2024-04-22T15:19:03.287 INFO:tasks.workunit.client.1.smithi196.stdout:2/609: write d1/d23/d27/d2e/d55/d5b/d7c/fa5 [430821,104125] 0 2024-04-22T15:19:03.289 INFO:tasks.workunit.client.1.smithi196.stdout:9/578: read - db/d11/d29/f9b zero size 2024-04-22T15:19:03.290 INFO:tasks.workunit.client.1.smithi196.stdout:7/558: dwrite f10 [4194304,4194304] 0 2024-04-22T15:19:03.290 INFO:tasks.workunit.client.0.smithi096.stdout:3/781: symlink da/db/dc/lf8 0 2024-04-22T15:19:03.292 INFO:tasks.workunit.client.0.smithi096.stdout:3/782: write da/db/de/d8b/ff6 [341170,128674] 0 2024-04-22T15:19:03.294 INFO:tasks.workunit.client.1.smithi196.stdout:1/545: symlink d1/db/dc/d93/lae 0 2024-04-22T15:19:03.298 INFO:tasks.workunit.client.1.smithi196.stdout:3/589: symlink d0/d15/d31/d55/d5d/dc1/lc6 0 2024-04-22T15:19:03.302 INFO:tasks.workunit.client.0.smithi096.stdout:0/952: link d7/de/d5b/d95/dc/d16/d65/d7c/f9b d7/de/f145 0 2024-04-22T15:19:03.302 INFO:tasks.workunit.client.1.smithi196.stdout:0/608: dwrite d3/de/d14/f81 [0,4194304] 0 2024-04-22T15:19:03.304 INFO:tasks.workunit.client.0.smithi096.stdout:2/870: rename d3/db/df/dce/d46/d59/f79 to d3/db/d30/f11c 0 2024-04-22T15:19:03.309 INFO:tasks.workunit.client.1.smithi196.stdout:9/579: fsync db/dd/f8d 0 2024-04-22T15:19:03.309 INFO:tasks.workunit.client.1.smithi196.stdout:7/559: fsync d14/f3a 0 2024-04-22T15:19:03.310 INFO:tasks.workunit.client.1.smithi196.stdout:0/609: dwrite d3/d9/fb1 [0,4194304] 0 2024-04-22T15:19:03.313 INFO:tasks.workunit.client.1.smithi196.stdout:1/546: unlink d1/db/d16/d1e/d46/c4c 0 2024-04-22T15:19:03.320 INFO:tasks.workunit.client.1.smithi196.stdout:5/605: truncate d1/d17/d1a/d2f/d30/f54 1267351 0 2024-04-22T15:19:03.321 INFO:tasks.workunit.client.1.smithi196.stdout:9/580: creat db/d11/d29/d7a/fca x:0 0 0 2024-04-22T15:19:03.323 INFO:tasks.workunit.client.0.smithi096.stdout:3/783: dread da/db/de/d19/f73 [0,4194304] 0 2024-04-22T15:19:03.323 INFO:tasks.workunit.client.1.smithi196.stdout:7/560: rename f4 to d14/d26/d28/d29/fb3 0 2024-04-22T15:19:03.324 INFO:tasks.workunit.client.1.smithi196.stdout:0/610: mkdir d3/d9/d36/d56/d65/d7f/d87/d6c/d88/d95/dd0 0 2024-04-22T15:19:03.325 INFO:tasks.workunit.client.1.smithi196.stdout:3/590: getdents d0/d15/d31/d55/d8b/dbf 0 2024-04-22T15:19:03.325 INFO:tasks.workunit.client.1.smithi196.stdout:0/611: write d3/fd [2370606,102494] 0 2024-04-22T15:19:03.327 INFO:tasks.workunit.client.1.smithi196.stdout:0/612: fdatasync d3/d9/d36/d9f/fcb 0 2024-04-22T15:19:03.329 INFO:tasks.workunit.client.1.smithi196.stdout:1/547: rmdir d1/db/dc/de/d5a/d5b 39 2024-04-22T15:19:03.330 INFO:tasks.workunit.client.0.smithi096.stdout:3/784: dread da/db/de/d19/d2d/fa8 [0,4194304] 0 2024-04-22T15:19:03.331 INFO:tasks.workunit.client.1.smithi196.stdout:2/610: creat d1/d13/fe6 x:0 0 0 2024-04-22T15:19:03.332 INFO:tasks.workunit.client.1.smithi196.stdout:2/611: write d1/d13/d1f/f3a [4721839,101075] 0 2024-04-22T15:19:03.333 INFO:tasks.workunit.client.1.smithi196.stdout:9/581: write db/dd/d1b/d2c/f43 [1804523,22432] 0 2024-04-22T15:19:03.335 INFO:tasks.workunit.client.1.smithi196.stdout:7/561: symlink d14/d26/d28/d29/lb4 0 2024-04-22T15:19:03.336 INFO:tasks.workunit.client.0.smithi096.stdout:3/785: symlink da/db/d9a/def/lf9 0 2024-04-22T15:19:03.338 INFO:tasks.workunit.client.0.smithi096.stdout:3/786: write da/db/de/d87/fb6 [3307365,80036] 0 2024-04-22T15:19:03.338 INFO:tasks.workunit.client.1.smithi196.stdout:6/586: dwrite dd/d1f/d37/d3c/f46 [0,4194304] 0 2024-04-22T15:19:03.338 INFO:tasks.workunit.client.1.smithi196.stdout:3/591: creat d0/d15/da7/d85/fc7 x:0 0 0 2024-04-22T15:19:03.339 INFO:tasks.workunit.client.0.smithi096.stdout:0/953: dread d7/f20 [0,4194304] 0 2024-04-22T15:19:03.341 INFO:tasks.workunit.client.1.smithi196.stdout:0/613: truncate d3/d9/d36/d56/d65/d7f/d30/f46 1157930 0 2024-04-22T15:19:03.344 INFO:tasks.workunit.client.0.smithi096.stdout:2/871: dread d3/db/df/d24/d58/d5f/dfe/d115/d2f/f34 [0,4194304] 0 2024-04-22T15:19:03.346 INFO:tasks.workunit.client.0.smithi096.stdout:2/872: write d3/f106 [911704,81215] 0 2024-04-22T15:19:03.346 INFO:tasks.workunit.client.0.smithi096.stdout:3/787: read da/db/de/ddc/d26/d52/f6c [7308073,84592] 0 2024-04-22T15:19:03.348 INFO:tasks.workunit.client.1.smithi196.stdout:6/587: dwrite dd/d20/d5c/d5e/d76/f85 [0,4194304] 0 2024-04-22T15:19:03.352 INFO:tasks.workunit.client.1.smithi196.stdout:9/582: unlink db/d11/d29/d57/fb0 0 2024-04-22T15:19:03.352 INFO:tasks.workunit.client.1.smithi196.stdout:0/614: dwrite d3/d9/d36/d56/d65/d7f/d87/d6c/fbf [0,4194304] 0 2024-04-22T15:19:03.357 INFO:tasks.workunit.client.1.smithi196.stdout:7/562: rename fa to d14/d26/d28/d29/fb5 0 2024-04-22T15:19:03.357 INFO:tasks.workunit.client.1.smithi196.stdout:7/563: read - d14/d26/d5e/f7b zero size 2024-04-22T15:19:03.357 INFO:tasks.workunit.client.0.smithi096.stdout:3/788: mknod da/db/dc/d28/cfa 0 2024-04-22T15:19:03.359 INFO:tasks.workunit.client.1.smithi196.stdout:8/579: write d2/d8/f76 [2759181,87696] 0 2024-04-22T15:19:03.361 INFO:tasks.workunit.client.1.smithi196.stdout:8/580: chown d2/d6a 61552 1 2024-04-22T15:19:03.361 INFO:tasks.workunit.client.0.smithi096.stdout:0/954: link d7/d42/d4c/d55/l89 d7/de/d5b/d95/d66/l146 0 2024-04-22T15:19:03.364 INFO:tasks.workunit.client.0.smithi096.stdout:3/789: creat da/db/dc/ffb x:0 0 0 2024-04-22T15:19:03.365 INFO:tasks.workunit.client.1.smithi196.stdout:1/548: dread d1/db/dc/de/d43/f48 [0,4194304] 0 2024-04-22T15:19:03.367 INFO:tasks.workunit.client.0.smithi096.stdout:0/955: mknod d7/d42/d4c/d55/d9c/c147 0 2024-04-22T15:19:03.368 INFO:tasks.workunit.client.0.smithi096.stdout:3/790: stat da/db/de/ddc/d26/d52/fd4 0 2024-04-22T15:19:03.369 INFO:tasks.workunit.client.1.smithi196.stdout:4/545: write d6/d14/d18/d26/d32/d53/f94 [4176651,107374] 0 2024-04-22T15:19:03.370 INFO:tasks.workunit.client.1.smithi196.stdout:6/588: unlink dd/d1f/d4e/c5b 0 2024-04-22T15:19:03.371 INFO:tasks.workunit.client.1.smithi196.stdout:6/589: fsync dd/f12 0 2024-04-22T15:19:03.376 INFO:tasks.workunit.client.0.smithi096.stdout:3/791: unlink da/db/de/d38/fb8 0 2024-04-22T15:19:03.378 INFO:tasks.workunit.client.1.smithi196.stdout:0/615: readlink d3/d9/d25/l83 0 2024-04-22T15:19:03.379 INFO:tasks.workunit.client.1.smithi196.stdout:7/564: rmdir d14/d26/d5a/d5f 39 2024-04-22T15:19:03.381 INFO:tasks.workunit.client.0.smithi096.stdout:3/792: mkdir da/db/de/d38/dfc 0 2024-04-22T15:19:03.383 INFO:tasks.workunit.client.1.smithi196.stdout:8/581: rename d2/d8/d47/f60 to d2/dc/d45/d98/fa6 0 2024-04-22T15:19:03.383 INFO:tasks.workunit.client.1.smithi196.stdout:3/592: dread d0/d7/d8/f5c [0,4194304] 0 2024-04-22T15:19:03.384 INFO:tasks.workunit.client.0.smithi096.stdout:2/873: dread d3/f8e [0,4194304] 0 2024-04-22T15:19:03.385 INFO:tasks.workunit.client.0.smithi096.stdout:2/874: fsync d3/db/df/d24/d58/d5f/dfe/d115/f118 0 2024-04-22T15:19:03.391 INFO:tasks.workunit.client.1.smithi196.stdout:6/590: symlink dd/d1f/d4e/la6 0 2024-04-22T15:19:03.392 INFO:tasks.workunit.client.1.smithi196.stdout:6/591: read dd/d20/f29 [1825386,128129] 0 2024-04-22T15:19:03.406 INFO:tasks.workunit.client.0.smithi096.stdout:2/875: dread d3/f2e [0,4194304] 0 2024-04-22T15:19:03.408 INFO:tasks.workunit.client.0.smithi096.stdout:2/876: truncate d3/db/df/d10/f21 1179595 0 2024-04-22T15:19:03.409 INFO:tasks.workunit.client.0.smithi096.stdout:2/877: readlink d3/db/df/d10/la0 0 2024-04-22T15:19:03.410 INFO:tasks.workunit.client.0.smithi096.stdout:0/956: write d7/de/d5b/d95/dc/d16/f5d [564677,28653] 0 2024-04-22T15:19:03.411 INFO:tasks.workunit.client.1.smithi196.stdout:0/616: rename d3/d9/d36/d56 to d3/d9/d36/d9f/dd1 0 2024-04-22T15:19:03.412 INFO:tasks.workunit.client.0.smithi096.stdout:2/878: mkdir d3/db/df/d24/d58/d5f/d11d 0 2024-04-22T15:19:03.412 INFO:tasks.workunit.client.1.smithi196.stdout:1/549: unlink d1/db/dc/de/d5a/d94/c9c 0 2024-04-22T15:19:03.413 INFO:tasks.workunit.client.1.smithi196.stdout:5/606: dwrite d1/d17/f79 [0,4194304] 0 2024-04-22T15:19:03.413 INFO:tasks.workunit.client.1.smithi196.stdout:7/565: mkdir d14/d26/d5a/db6 0 2024-04-22T15:19:03.415 INFO:tasks.workunit.client.1.smithi196.stdout:1/550: fsync d1/db/dc/d26/d2e/f34 0 2024-04-22T15:19:03.416 INFO:tasks.workunit.client.1.smithi196.stdout:5/607: write d1/d2/d98/fb6 [445444,62707] 0 2024-04-22T15:19:03.416 INFO:tasks.workunit.client.1.smithi196.stdout:0/617: dread - d3/d9/d36/d9f/dd1/d65/d7f/d30/faa zero size 2024-04-22T15:19:03.418 INFO:tasks.workunit.client.0.smithi096.stdout:3/793: write da/db/dc/d8d/fc5 [888009,125863] 0 2024-04-22T15:19:03.418 INFO:tasks.workunit.client.0.smithi096.stdout:5/850: sync 2024-04-22T15:19:03.419 INFO:tasks.workunit.client.1.smithi196.stdout:5/608: write d1/d14/d3e/f44 [3445863,26838] 0 2024-04-22T15:19:03.420 INFO:tasks.workunit.client.0.smithi096.stdout:2/879: dwrite d3/db/fc9 [0,4194304] 0 2024-04-22T15:19:03.421 INFO:tasks.workunit.client.0.smithi096.stdout:3/794: write da/db/dc/d8d/fc5 [1465747,36646] 0 2024-04-22T15:19:03.421 INFO:tasks.workunit.client.1.smithi196.stdout:9/583: dwrite db/f61 [0,4194304] 0 2024-04-22T15:19:03.424 INFO:tasks.workunit.client.1.smithi196.stdout:7/566: mkdir d14/d26/db7 0 2024-04-22T15:19:03.435 INFO:tasks.workunit.client.0.smithi096.stdout:5/851: dread d4/fcc [0,4194304] 0 2024-04-22T15:19:03.435 INFO:tasks.workunit.client.1.smithi196.stdout:4/546: write d6/d13/f24 [1019511,78673] 0 2024-04-22T15:19:03.436 INFO:tasks.workunit.client.0.smithi096.stdout:3/795: symlink da/db/dc/d28/lfd 0 2024-04-22T15:19:03.441 INFO:tasks.workunit.client.0.smithi096.stdout:0/957: write d7/de/d5b/d95/dc/d16/d18/fd8 [1367600,93084] 0 2024-04-22T15:19:03.444 INFO:tasks.workunit.client.0.smithi096.stdout:5/852: truncate d4/ddb/ff2 112183 0 2024-04-22T15:19:03.446 INFO:tasks.workunit.client.0.smithi096.stdout:3/796: creat da/db/de/ddc/d26/ffe x:0 0 0 2024-04-22T15:19:03.447 INFO:tasks.workunit.client.1.smithi196.stdout:3/593: getdents d0/d15/d31/d9d/d9e 0 2024-04-22T15:19:03.451 INFO:tasks.workunit.client.1.smithi196.stdout:6/592: rename dd/d1f/d37/l3b to dd/d1f/la7 0 2024-04-22T15:19:03.456 INFO:tasks.workunit.client.0.smithi096.stdout:5/853: mknod d4/d2c/d2f/c123 0 2024-04-22T15:19:03.457 INFO:tasks.workunit.client.0.smithi096.stdout:2/880: dwrite d3/db/df/d24/d58/fd4 [0,4194304] 0 2024-04-22T15:19:03.460 INFO:tasks.workunit.client.1.smithi196.stdout:7/567: dwrite d14/d15/f59 [0,4194304] 0 2024-04-22T15:19:03.460 INFO:tasks.workunit.client.0.smithi096.stdout:3/797: link da/db/de/d8b/db5/fcd da/db/d9a/dd1/fff 0 2024-04-22T15:19:03.464 INFO:tasks.workunit.client.1.smithi196.stdout:5/609: symlink d1/d14/d6a/da7/lc7 0 2024-04-22T15:19:03.465 INFO:tasks.workunit.client.1.smithi196.stdout:5/610: write d1/d14/d6a/fa8 [995454,44533] 0 2024-04-22T15:19:03.466 INFO:tasks.workunit.client.0.smithi096.stdout:5/854: mknod d4/ddb/d25/c124 0 2024-04-22T15:19:03.468 INFO:tasks.workunit.client.0.smithi096.stdout:0/958: dwrite d7/d42/d4c/dd2/f118 [0,4194304] 0 2024-04-22T15:19:03.470 INFO:tasks.workunit.client.1.smithi196.stdout:3/594: rename d0/d7/d8/f66 to d0/d15/da7/fc8 0 2024-04-22T15:19:03.470 INFO:tasks.workunit.client.1.smithi196.stdout:2/612: sync 2024-04-22T15:19:03.470 INFO:tasks.workunit.client.1.smithi196.stdout:6/593: rename dd to dd/d1f/d4e/d53/da8 22 2024-04-22T15:19:03.470 INFO:tasks.workunit.client.1.smithi196.stdout:6/594: chown fc 31022 1 2024-04-22T15:19:03.472 INFO:tasks.workunit.client.1.smithi196.stdout:1/551: rmdir d1/db/dc/de/d43/d5e/d7a/d9f 0 2024-04-22T15:19:03.472 INFO:tasks.workunit.client.1.smithi196.stdout:6/595: dread - dd/d1f/d4e/d53/d7f/f83 zero size 2024-04-22T15:19:03.472 INFO:tasks.workunit.client.1.smithi196.stdout:1/552: stat d1/db/dc/d26/f28 0 2024-04-22T15:19:03.474 INFO:tasks.workunit.client.1.smithi196.stdout:1/553: fsync d1/db/dc/dad/d36/f3f 0 2024-04-22T15:19:03.480 INFO:tasks.workunit.client.0.smithi096.stdout:9/763: sync 2024-04-22T15:19:03.480 INFO:tasks.workunit.client.0.smithi096.stdout:6/670: sync 2024-04-22T15:19:03.480 INFO:tasks.workunit.client.0.smithi096.stdout:8/838: sync 2024-04-22T15:19:03.480 INFO:tasks.workunit.client.0.smithi096.stdout:1/871: sync 2024-04-22T15:19:03.480 INFO:tasks.workunit.client.0.smithi096.stdout:4/822: sync 2024-04-22T15:19:03.482 INFO:tasks.workunit.client.0.smithi096.stdout:1/872: fdatasync d6/f9d 0 2024-04-22T15:19:03.485 INFO:tasks.workunit.client.0.smithi096.stdout:5/855: creat d4/ddb/d3d/d35/d57/dbe/d113/f125 x:0 0 0 2024-04-22T15:19:03.487 INFO:tasks.workunit.client.1.smithi196.stdout:9/584: creat db/d11/d29/fcb x:0 0 0 2024-04-22T15:19:03.488 INFO:tasks.workunit.client.0.smithi096.stdout:3/798: dread da/db/dc/d28/f93 [0,4194304] 0 2024-04-22T15:19:03.488 INFO:tasks.workunit.client.0.smithi096.stdout:9/764: fsync d1/df/d95/d9d/db5/dd/dba/dc2/fc6 0 2024-04-22T15:19:03.490 INFO:tasks.workunit.client.1.smithi196.stdout:5/611: mknod d1/d14/d6a/cc8 0 2024-04-22T15:19:03.492 INFO:tasks.workunit.client.0.smithi096.stdout:2/881: truncate d3/db/df/d24/d58/d5f/dfe/d115/d2f/f34 89619 0 2024-04-22T15:19:03.493 INFO:tasks.workunit.client.0.smithi096.stdout:1/873: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d3d/f110 [0,4194304] 0 2024-04-22T15:19:03.494 INFO:tasks.workunit.client.0.smithi096.stdout:2/882: readlink d3/db/df/d10/l16 0 2024-04-22T15:19:03.494 INFO:tasks.workunit.client.0.smithi096.stdout:2/883: chown d3/db/df/dce/d46/d59/l64 161243 1 2024-04-22T15:19:03.495 INFO:tasks.workunit.client.1.smithi196.stdout:3/595: mknod d0/d15/da7/d18/cc9 0 2024-04-22T15:19:03.495 INFO:tasks.workunit.client.1.smithi196.stdout:5/612: read d1/d17/f8b [61499,13156] 0 2024-04-22T15:19:03.496 INFO:tasks.workunit.client.0.smithi096.stdout:2/884: chown d3/d13/db4/dbf/fea 477927 1 2024-04-22T15:19:03.497 INFO:tasks.workunit.client.1.smithi196.stdout:5/613: write d1/d4/f22 [83295,43849] 0 2024-04-22T15:19:03.501 INFO:tasks.workunit.client.1.smithi196.stdout:5/614: chown d1/d17/f79 1633 1 2024-04-22T15:19:03.502 INFO:tasks.workunit.client.0.smithi096.stdout:0/959: dwrite d7/f20 [4194304,4194304] 0 2024-04-22T15:19:03.502 INFO:tasks.workunit.client.1.smithi196.stdout:4/547: dwrite d6/d13/d15/d2c/d35/f3c [8388608,4194304] 0 2024-04-22T15:19:03.504 INFO:tasks.workunit.client.0.smithi096.stdout:8/839: unlink d4/d1c/fbc 0 2024-04-22T15:19:03.506 INFO:tasks.workunit.client.1.smithi196.stdout:0/618: getdents d3/d6/dc/d8c/daf 0 2024-04-22T15:19:03.508 INFO:tasks.workunit.client.1.smithi196.stdout:0/619: truncate d3/d9/d36/d9f/dd1/d65/d7f/d30/faa 139963 0 2024-04-22T15:19:03.509 INFO:tasks.workunit.client.0.smithi096.stdout:4/823: rename d5/d7/d2c/d95/df9/l107 to d5/d7/d1f/d84/d6d/l120 0 2024-04-22T15:19:03.510 INFO:tasks.workunit.client.1.smithi196.stdout:9/585: truncate db/dd/d1b/d2c/f37 2640552 0 2024-04-22T15:19:03.514 INFO:tasks.workunit.client.1.smithi196.stdout:2/613: rename d1/d13/l49 to d1/d23/d27/d2e/d55/d5b/le7 0 2024-04-22T15:19:03.515 INFO:tasks.workunit.client.1.smithi196.stdout:2/614: dread - d1/d13/d25/fb4 zero size 2024-04-22T15:19:03.516 INFO:tasks.workunit.client.1.smithi196.stdout:6/596: creat dd/d1f/d37/d3c/d89/fa9 x:0 0 0 2024-04-22T15:19:03.517 INFO:tasks.workunit.client.1.smithi196.stdout:6/597: chown dd/d1f/d38/d40/f42 3 1 2024-04-22T15:19:03.518 INFO:tasks.workunit.client.0.smithi096.stdout:1/874: read d6/f60 [321692,9680] 0 2024-04-22T15:19:03.519 INFO:tasks.workunit.client.1.smithi196.stdout:7/568: creat d14/d25/fb8 x:0 0 0 2024-04-22T15:19:03.520 INFO:tasks.workunit.client.1.smithi196.stdout:4/548: symlink d6/d13/d15/d2c/d62/d78/d8b/laf 0 2024-04-22T15:19:03.523 INFO:tasks.workunit.client.0.smithi096.stdout:0/960: creat d7/de/d3a/d61/dbb/f148 x:0 0 0 2024-04-22T15:19:03.527 INFO:tasks.workunit.client.0.smithi096.stdout:8/840: creat d4/d1c/d62/dc3/dec/f116 x:0 0 0 2024-04-22T15:19:03.528 INFO:tasks.workunit.client.0.smithi096.stdout:8/841: truncate d4/d2a/fe6 355014 0 2024-04-22T15:19:03.530 INFO:tasks.workunit.client.0.smithi096.stdout:8/842: write d4/d1c/d40/d46/d58/ffb [90995,127736] 0 2024-04-22T15:19:03.531 INFO:tasks.workunit.client.1.smithi196.stdout:5/615: mknod d1/d4/d92/db5/cc9 0 2024-04-22T15:19:03.532 INFO:tasks.workunit.client.1.smithi196.stdout:5/616: chown d1/d17/f79 1716768 1 2024-04-22T15:19:03.533 INFO:tasks.workunit.client.0.smithi096.stdout:5/856: mkdir d4/ddb/d3d/d4d/d59/db0/d126 0 2024-04-22T15:19:03.536 INFO:tasks.workunit.client.1.smithi196.stdout:8/582: sync 2024-04-22T15:19:03.536 INFO:tasks.workunit.client.1.smithi196.stdout:6/598: rename dd/d20/f24 to dd/d20/d5c/d6a/faa 0 2024-04-22T15:19:03.538 INFO:tasks.workunit.client.0.smithi096.stdout:3/799: creat da/db/dc/d28/dda/f100 x:0 0 0 2024-04-22T15:19:03.541 INFO:tasks.workunit.client.0.smithi096.stdout:9/765: fsync d1/df/fa7 0 2024-04-22T15:19:03.543 INFO:tasks.workunit.client.1.smithi196.stdout:4/549: read d6/d14/d18/d26/f3a [1245732,86732] 0 2024-04-22T15:19:03.544 INFO:tasks.workunit.client.0.smithi096.stdout:1/875: symlink d6/dc/d8a/d3f/d62/d7d/ddb/d9b/l111 0 2024-04-22T15:19:03.546 INFO:tasks.workunit.client.0.smithi096.stdout:4/824: dread d5/d7/d2c/d3d/f49 [0,4194304] 0 2024-04-22T15:19:03.547 INFO:tasks.workunit.client.1.smithi196.stdout:9/586: creat db/dd/de/d75/fcc x:0 0 0 2024-04-22T15:19:03.548 INFO:tasks.workunit.client.0.smithi096.stdout:6/671: creat d3/d28/d54/d68/fe4 x:0 0 0 2024-04-22T15:19:03.551 INFO:tasks.workunit.client.0.smithi096.stdout:2/885: dwrite d3/fd [0,4194304] 0 2024-04-22T15:19:03.552 INFO:tasks.workunit.client.1.smithi196.stdout:3/596: write d0/d15/d48/d51/fa9 [415030,59214] 0 2024-04-22T15:19:03.553 INFO:tasks.workunit.client.0.smithi096.stdout:2/886: write d3/db/f12 [1411682,46039] 0 2024-04-22T15:19:03.554 INFO:tasks.workunit.client.0.smithi096.stdout:0/961: truncate d7/de/d5b/d95/dc/d16/f1f 1411246 0 2024-04-22T15:19:03.554 INFO:tasks.workunit.client.1.smithi196.stdout:3/597: truncate d0/d15/d31/d55/d8b/fc0 243529 0 2024-04-22T15:19:03.555 INFO:tasks.workunit.client.0.smithi096.stdout:2/887: chown d3/db/df/dce/d46/d59/l64 1038206300 1 2024-04-22T15:19:03.555 INFO:tasks.workunit.client.1.smithi196.stdout:5/617: chown d1/d9/d68/c90 1 1 2024-04-22T15:19:03.556 INFO:tasks.workunit.client.1.smithi196.stdout:5/618: dread - d1/d4/f9c zero size 2024-04-22T15:19:03.558 INFO:tasks.workunit.client.1.smithi196.stdout:8/583: creat d2/d5/d1a/fa7 x:0 0 0 2024-04-22T15:19:03.559 INFO:tasks.workunit.client.1.smithi196.stdout:5/619: truncate d1/d17/d1c/f3b 5054525 0 2024-04-22T15:19:03.560 INFO:tasks.workunit.client.1.smithi196.stdout:6/599: creat dd/d1f/d4e/fab x:0 0 0 2024-04-22T15:19:03.561 INFO:tasks.workunit.client.1.smithi196.stdout:6/600: dread - dd/d2b/d8d/f93 zero size 2024-04-22T15:19:03.563 INFO:tasks.workunit.client.1.smithi196.stdout:7/569: truncate d14/d26/d2a/f84 225192 0 2024-04-22T15:19:03.565 INFO:tasks.workunit.client.0.smithi096.stdout:9/766: readlink d1/d25/db2/lfb 0 2024-04-22T15:19:03.566 INFO:tasks.workunit.client.1.smithi196.stdout:0/620: creat d3/d9/d36/fd2 x:0 0 0 2024-04-22T15:19:03.568 INFO:tasks.workunit.client.1.smithi196.stdout:2/615: write d1/d23/d27/d2e/d55/f87 [1934645,1093] 0 2024-04-22T15:19:03.568 INFO:tasks.workunit.client.1.smithi196.stdout:1/554: sync 2024-04-22T15:19:03.568 INFO:tasks.workunit.client.1.smithi196.stdout:9/587: rmdir db/dd/de 39 2024-04-22T15:19:03.571 INFO:tasks.workunit.client.0.smithi096.stdout:4/825: dread d5/d7/d2c/d3d/d4f/fb0 [0,4194304] 0 2024-04-22T15:19:03.573 INFO:tasks.workunit.client.1.smithi196.stdout:3/598: mknod d0/d15/d31/cca 0 2024-04-22T15:19:03.574 INFO:tasks.workunit.client.1.smithi196.stdout:3/599: write d0/d15/d48/d51/fa9 [268816,7778] 0 2024-04-22T15:19:03.574 INFO:tasks.workunit.client.0.smithi096.stdout:6/672: creat d3/d28/dd/d11/d2c/d6f/dd4/dd6/d51/fe5 x:0 0 0 2024-04-22T15:19:03.577 INFO:tasks.workunit.client.0.smithi096.stdout:6/673: write d3/d28/dd/d11/f41 [849310,90764] 0 2024-04-22T15:19:03.580 INFO:tasks.workunit.client.1.smithi196.stdout:5/620: rename d1/d17/d1a/d2f/d30/fb3 to d1/d17/d1a/d2f/d30/d5e/fca 0 2024-04-22T15:19:03.580 INFO:tasks.workunit.client.1.smithi196.stdout:8/584: dread d2/d8/d49/f59 [0,4194304] 0 2024-04-22T15:19:03.580 INFO:tasks.workunit.client.0.smithi096.stdout:3/800: dwrite da/db/de/d19/f73 [0,4194304] 0 2024-04-22T15:19:03.581 INFO:tasks.workunit.client.1.smithi196.stdout:6/601: truncate f2 1755754 0 2024-04-22T15:19:03.582 INFO:tasks.workunit.client.0.smithi096.stdout:3/801: write da/db/d12/f56 [3644976,99330] 0 2024-04-22T15:19:03.582 INFO:tasks.workunit.client.1.smithi196.stdout:3/600: dwrite d0/d15/da7/d85/fc7 [0,4194304] 0 2024-04-22T15:19:03.583 INFO:tasks.workunit.client.1.smithi196.stdout:8/585: fdatasync d2/d8/d49/d6c/f86 0 2024-04-22T15:19:03.584 INFO:tasks.workunit.client.0.smithi096.stdout:8/843: link d4/d1c/d40/d46/d58/ffb d4/d10/d25/d4c/db4/f117 0 2024-04-22T15:19:03.586 INFO:tasks.workunit.client.1.smithi196.stdout:7/570: creat d14/d26/d2a/d2b/fb9 x:0 0 0 2024-04-22T15:19:03.586 INFO:tasks.workunit.client.1.smithi196.stdout:3/601: read - d0/d15/d31/d55/d6e/fa1 zero size 2024-04-22T15:19:03.588 INFO:tasks.workunit.client.0.smithi096.stdout:1/876: creat d6/dc/d8a/d3f/d54/d32/d3b/d56/d107/f112 x:0 0 0 2024-04-22T15:19:03.589 INFO:tasks.workunit.client.1.smithi196.stdout:3/602: write d0/d15/da7/d18/f4c [712600,83275] 0 2024-04-22T15:19:03.590 INFO:tasks.workunit.client.0.smithi096.stdout:1/877: chown d6/dc/c92 13 1 2024-04-22T15:19:03.590 INFO:tasks.workunit.client.0.smithi096.stdout:2/888: dread d3/db/df/dce/d46/d69/fe1 [0,4194304] 0 2024-04-22T15:19:03.592 INFO:tasks.workunit.client.1.smithi196.stdout:1/555: dread d1/db/d16/f5c [0,4194304] 0 2024-04-22T15:19:03.592 INFO:tasks.workunit.client.0.smithi096.stdout:9/767: symlink d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/dd9/l101 0 2024-04-22T15:19:03.592 INFO:tasks.workunit.client.1.smithi196.stdout:9/588: symlink db/d11/d29/d57/da7/lcd 0 2024-04-22T15:19:03.599 INFO:tasks.workunit.client.1.smithi196.stdout:5/621: mkdir d1/d17/d1a/d2f/d30/d5e/dcb 0 2024-04-22T15:19:03.600 INFO:tasks.workunit.client.0.smithi096.stdout:3/802: creat da/db/dc/d28/dda/f101 x:0 0 0 2024-04-22T15:19:03.601 INFO:tasks.workunit.client.0.smithi096.stdout:8/844: mknod d4/d2a/db7/c118 0 2024-04-22T15:19:03.603 INFO:tasks.workunit.client.1.smithi196.stdout:8/586: unlink d2/d3f/f97 0 2024-04-22T15:19:03.604 INFO:tasks.workunit.client.1.smithi196.stdout:8/587: chown d2/d5/f87 5880 1 2024-04-22T15:19:03.604 INFO:tasks.workunit.client.1.smithi196.stdout:0/621: dread d3/fd [0,4194304] 0 2024-04-22T15:19:03.604 INFO:tasks.workunit.client.0.smithi096.stdout:1/878: creat d6/dc/d8a/d3f/d62/dc2/f113 x:0 0 0 2024-04-22T15:19:03.605 INFO:tasks.workunit.client.1.smithi196.stdout:4/550: creat d6/d13/d15/d2c/d35/fb0 x:0 0 0 2024-04-22T15:19:03.606 INFO:tasks.workunit.client.0.smithi096.stdout:4/826: mknod d5/c121 0 2024-04-22T15:19:03.608 INFO:tasks.workunit.client.0.smithi096.stdout:9/768: stat d1/df/d95/d9d/db5/d43/l4b 0 2024-04-22T15:19:03.609 INFO:tasks.workunit.client.0.smithi096.stdout:2/889: symlink d3/db/df/d24/d58/l11e 0 2024-04-22T15:19:03.618 INFO:tasks.workunit.client.1.smithi196.stdout:1/556: rmdir d1/d54 39 2024-04-22T15:19:03.619 INFO:tasks.workunit.client.1.smithi196.stdout:1/557: write d1/db/d16/d1e/f60 [2711927,80140] 0 2024-04-22T15:19:03.622 INFO:tasks.workunit.client.0.smithi096.stdout:4/827: symlink d5/d7/d1f/d84/d6d/l122 0 2024-04-22T15:19:03.623 INFO:tasks.workunit.client.1.smithi196.stdout:9/589: creat db/d11/d29/d7a/fce x:0 0 0 2024-04-22T15:19:03.624 INFO:tasks.workunit.client.0.smithi096.stdout:9/769: creat d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dbd/f102 x:0 0 0 2024-04-22T15:19:03.625 INFO:tasks.workunit.client.0.smithi096.stdout:6/674: write d3/d28/dd/d11/d2c/d3b/f79 [1077423,57162] 0 2024-04-22T15:19:03.625 INFO:tasks.workunit.client.1.smithi196.stdout:6/602: symlink dd/d1f/d37/d3c/d96/lac 0 2024-04-22T15:19:03.627 INFO:tasks.workunit.client.0.smithi096.stdout:8/845: rename d4/d1c/d40/d48/cdd to d4/d10/d25/d4c/db4/df4/c119 0 2024-04-22T15:19:03.631 INFO:tasks.workunit.client.1.smithi196.stdout:8/588: symlink d2/d8/d47/d68/d81/la8 0 2024-04-22T15:19:03.632 INFO:tasks.workunit.client.0.smithi096.stdout:4/828: creat d5/d5b/f123 x:0 0 0 2024-04-22T15:19:03.634 INFO:tasks.workunit.client.1.smithi196.stdout:0/622: rename d3/d9/c12 to d3/d9/d36/d9f/cd3 0 2024-04-22T15:19:03.638 INFO:tasks.workunit.client.1.smithi196.stdout:7/571: symlink d14/d26/d28/d29/lba 0 2024-04-22T15:19:03.640 INFO:tasks.workunit.client.0.smithi096.stdout:8/846: mkdir d4/d1c/d40/d46/d58/d11a 0 2024-04-22T15:19:03.647 INFO:tasks.workunit.client.1.smithi196.stdout:9/590: unlink db/dd/d1b/f79 0 2024-04-22T15:19:03.647 INFO:tasks.workunit.client.1.smithi196.stdout:9/591: readlink db/dd/d1b/d2c/l77 0 2024-04-22T15:19:03.649 INFO:tasks.workunit.client.1.smithi196.stdout:8/589: rmdir d2/d5/d4f 39 2024-04-22T15:19:03.652 INFO:tasks.workunit.client.0.smithi096.stdout:6/675: creat d3/d28/d4d/d82/fe6 x:0 0 0 2024-04-22T15:19:03.653 INFO:tasks.workunit.client.1.smithi196.stdout:7/572: dread - d14/d26/d2a/d2b/f45 zero size 2024-04-22T15:19:03.658 INFO:tasks.workunit.client.0.smithi096.stdout:8/847: link d4/d1c/f72 d4/d1c/d40/f11b 0 2024-04-22T15:19:03.659 INFO:tasks.workunit.client.1.smithi196.stdout:6/603: truncate fa 1050371 0 2024-04-22T15:19:03.660 INFO:tasks.workunit.client.1.smithi196.stdout:8/590: rmdir d2/d3f/d4b 39 2024-04-22T15:19:03.661 INFO:tasks.workunit.client.1.smithi196.stdout:7/573: stat d14/d26/d5e/c6d 0 2024-04-22T15:19:03.662 INFO:tasks.workunit.client.0.smithi096.stdout:8/848: mknod d4/d10/d25/df7/c11c 0 2024-04-22T15:19:03.663 INFO:tasks.workunit.client.1.smithi196.stdout:3/603: getdents d0/d15/d4d/d87 0 2024-04-22T15:19:03.664 INFO:tasks.workunit.client.1.smithi196.stdout:1/558: symlink d1/db/dc/de/d5a/d5b/d6e/laf 0 2024-04-22T15:19:03.665 INFO:tasks.workunit.client.1.smithi196.stdout:3/604: read d0/d15/da7/d4a/d44/f26 [3763776,29353] 0 2024-04-22T15:19:03.666 INFO:tasks.workunit.client.0.smithi096.stdout:8/849: creat d4/d10/d25/d4c/d76/d7f/f11d x:0 0 0 2024-04-22T15:19:03.666 INFO:tasks.workunit.client.1.smithi196.stdout:6/604: symlink dd/d1f/d37/d3c/d96/lad 0 2024-04-22T15:19:03.667 INFO:tasks.workunit.client.0.smithi096.stdout:6/676: dread d3/d28/d54/d68/d6e/dbb/fbc [0,4194304] 0 2024-04-22T15:19:03.668 INFO:tasks.workunit.client.1.smithi196.stdout:8/591: symlink d2/d8/d47/d68/la9 0 2024-04-22T15:19:03.669 INFO:tasks.workunit.client.1.smithi196.stdout:8/592: read d2/d8/d47/d68/d81/f89 [12434472,92793] 0 2024-04-22T15:19:03.670 INFO:tasks.workunit.client.0.smithi096.stdout:6/677: symlink d3/d28/d54/d68/d6e/le7 0 2024-04-22T15:19:03.671 INFO:tasks.workunit.client.1.smithi196.stdout:3/605: mkdir d0/d15/d31/dcb 0 2024-04-22T15:19:03.672 INFO:tasks.workunit.client.1.smithi196.stdout:0/623: getdents d3/d6/dc/d8c 0 2024-04-22T15:19:03.672 INFO:tasks.workunit.client.1.smithi196.stdout:3/606: write d0/d15/d31/d55/d6e/fa1 [23716,68604] 0 2024-04-22T15:19:03.675 INFO:tasks.workunit.client.1.smithi196.stdout:0/624: mkdir d3/de/d1c/dd4 0 2024-04-22T15:19:03.675 INFO:tasks.workunit.client.1.smithi196.stdout:8/593: fdatasync d2/d3f/f53 0 2024-04-22T15:19:03.675 INFO:tasks.workunit.client.1.smithi196.stdout:8/594: readlink d2/l30 0 2024-04-22T15:19:03.676 INFO:tasks.workunit.client.1.smithi196.stdout:0/625: write d3/d9/d36/fd2 [669165,72330] 0 2024-04-22T15:19:03.676 INFO:tasks.workunit.client.1.smithi196.stdout:6/605: creat dd/d1f/d37/fae x:0 0 0 2024-04-22T15:19:03.676 INFO:tasks.workunit.client.1.smithi196.stdout:8/595: chown d2/d8/d49/d8b 0 1 2024-04-22T15:19:03.678 INFO:tasks.workunit.client.1.smithi196.stdout:8/596: write d2/d5/d4f/f6e [8859941,35284] 0 2024-04-22T15:19:03.680 INFO:tasks.workunit.client.1.smithi196.stdout:8/597: read d2/d35/f9e [932663,46025] 0 2024-04-22T15:19:03.681 INFO:tasks.workunit.client.1.smithi196.stdout:6/606: rename dd/d1f/l45 to dd/d1f/d4e/d53/d55/laf 0 2024-04-22T15:19:03.683 INFO:tasks.workunit.client.1.smithi196.stdout:6/607: symlink dd/lb0 0 2024-04-22T15:19:03.683 INFO:tasks.workunit.client.1.smithi196.stdout:8/598: creat d2/d5/faa x:0 0 0 2024-04-22T15:19:03.683 INFO:tasks.workunit.client.1.smithi196.stdout:8/599: dread - d2/d6a/d7b/f94 zero size 2024-04-22T15:19:03.685 INFO:tasks.workunit.client.1.smithi196.stdout:8/600: stat d2/dc/f82 0 2024-04-22T15:19:03.685 INFO:tasks.workunit.client.1.smithi196.stdout:6/608: getdents dd/d1f/d37/d3c/d89 0 2024-04-22T15:19:03.686 INFO:tasks.workunit.client.1.smithi196.stdout:6/609: fsync dd/f86 0 2024-04-22T15:19:03.688 INFO:tasks.workunit.client.1.smithi196.stdout:8/601: dread d2/d35/f92 [0,4194304] 0 2024-04-22T15:19:03.688 INFO:tasks.workunit.client.1.smithi196.stdout:8/602: stat d2/d6a/l8f 0 2024-04-22T15:19:03.693 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:03 smithi096 ceph-mon[26688]: pgmap v38: 65 pgs: 65 active+clean; 2.7 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 68 MiB/s rd, 202 MiB/s wr, 400 op/s 2024-04-22T15:19:03.696 INFO:tasks.workunit.client.1.smithi196.stdout:6/610: link dd/d1f/d37/f66 dd/d1f/d4e/d53/d7f/da5/fb1 0 2024-04-22T15:19:03.699 INFO:tasks.workunit.client.0.smithi096.stdout:4/829: dread d5/d7/d2c/d3d/d4f/d88/ffc [0,4194304] 0 2024-04-22T15:19:03.701 INFO:tasks.workunit.client.1.smithi196.stdout:5/622: dread d1/d17/d1a/d2f/f99 [0,4194304] 0 2024-04-22T15:19:03.701 INFO:tasks.workunit.client.0.smithi096.stdout:4/830: chown d5/d5b/fd0 24361389 1 2024-04-22T15:19:03.701 INFO:tasks.workunit.client.1.smithi196.stdout:2/616: truncate d1/d13/d1f/f3a 5523667 0 2024-04-22T15:19:03.702 INFO:tasks.workunit.client.0.smithi096.stdout:4/831: dread - d5/d7/d2c/f9d zero size 2024-04-22T15:19:03.706 INFO:tasks.workunit.client.0.smithi096.stdout:3/803: dwrite da/db/de/d19/d2d/d37/f5f [0,4194304] 0 2024-04-22T15:19:03.706 INFO:tasks.workunit.client.0.smithi096.stdout:3/804: dread - da/db/dc/ffb zero size 2024-04-22T15:19:03.706 INFO:tasks.workunit.client.0.smithi096.stdout:3/805: readlink da/db/d12/laf 0 2024-04-22T15:19:03.707 INFO:tasks.workunit.client.1.smithi196.stdout:6/611: dwrite dd/d1f/d38/f94 [0,4194304] 0 2024-04-22T15:19:03.707 INFO:tasks.workunit.client.1.smithi196.stdout:2/617: dread - d1/d23/d27/d2e/d55/d5b/d7c/fe4 zero size 2024-04-22T15:19:03.708 INFO:tasks.workunit.client.1.smithi196.stdout:8/603: dwrite d2/d3f/d4b/f72 [0,4194304] 0 2024-04-22T15:19:03.711 INFO:tasks.workunit.client.1.smithi196.stdout:1/559: dread d1/db/f58 [0,4194304] 0 2024-04-22T15:19:03.713 INFO:tasks.workunit.client.1.smithi196.stdout:6/612: creat dd/d2b/d8d/fb2 x:0 0 0 2024-04-22T15:19:03.713 INFO:tasks.workunit.client.0.smithi096.stdout:1/879: dwrite d6/dc/d8a/d3f/d54/d32/f6d [0,4194304] 0 2024-04-22T15:19:03.714 INFO:tasks.workunit.client.1.smithi196.stdout:4/551: write d6/d14/d18/d26/d32/d53/f54 [1151725,60781] 0 2024-04-22T15:19:03.715 INFO:tasks.workunit.client.1.smithi196.stdout:4/552: chown d6/d14/f87 708 1 2024-04-22T15:19:03.718 INFO:tasks.workunit.client.1.smithi196.stdout:5/623: link d1/d17/d1a/d2f/d30/d6e/fa2 d1/d2/d98/fcc 0 2024-04-22T15:19:03.797 INFO:tasks.workunit.client.1.smithi196.stdout:1/560: dwrite d1/db/d16/f73 [0,4194304] 0 2024-04-22T15:19:03.797 INFO:tasks.workunit.client.1.smithi196.stdout:1/561: dread - d1/db/fa8 zero size 2024-04-22T15:19:03.797 INFO:tasks.workunit.client.1.smithi196.stdout:8/604: symlink d2/d35/d96/lab 0 2024-04-22T15:19:03.797 INFO:tasks.workunit.client.1.smithi196.stdout:8/605: stat d2/dc/fe 0 2024-04-22T15:19:03.797 INFO:tasks.workunit.client.1.smithi196.stdout:8/606: stat d2/d5/d1a 0 2024-04-22T15:19:03.797 INFO:tasks.workunit.client.1.smithi196.stdout:6/613: fsync dd/f16 0 2024-04-22T15:19:03.797 INFO:tasks.workunit.client.1.smithi196.stdout:9/592: write db/d11/d29/d57/f23 [945299,20838] 0 2024-04-22T15:19:03.798 INFO:tasks.workunit.client.1.smithi196.stdout:4/553: creat d6/d10/fb1 x:0 0 0 2024-04-22T15:19:03.798 INFO:tasks.workunit.client.1.smithi196.stdout:7/574: write d14/d4d/f7e [2913811,79287] 0 2024-04-22T15:19:03.798 INFO:tasks.workunit.client.1.smithi196.stdout:7/575: chown d14/d25/f3b 6922 1 2024-04-22T15:19:03.798 INFO:tasks.workunit.client.1.smithi196.stdout:8/607: mkdir d2/d6a/dac 0 2024-04-22T15:19:03.798 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:03 smithi196 ceph-mon[34717]: pgmap v38: 65 pgs: 65 active+clean; 2.7 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 68 MiB/s rd, 202 MiB/s wr, 400 op/s 2024-04-22T15:19:03.799 INFO:tasks.workunit.client.0.smithi096.stdout:1/880: creat d6/dc/d8a/d3f/d62/d7d/ddb/d9b/dba/f114 x:0 0 0 2024-04-22T15:19:03.799 INFO:tasks.workunit.client.0.smithi096.stdout:9/770: dwrite d1/df/f58 [0,4194304] 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:9/771: mknod d1/d25/db2/dca/c103 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:8/850: dwrite d4/d10/d25/d38/f54 [0,4194304] 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:4/832: dread d5/d7/d2c/d3d/d4f/d88/fe2 [0,4194304] 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:9/772: dwrite d1/d25/fd5 [0,4194304] 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:9/773: dread - d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/dd9/fee zero size 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:9/774: dread - d1/df/d95/d9d/db5/d43/fe0 zero size 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:3/806: write da/db/de/d19/d2d/fa8 [4854402,125190] 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:3/807: write da/db/de/d19/d2d/fa8 [831688,129538] 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:3/808: chown da/db/de/ddc/d68/db0/ce0 3 1 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:8/851: truncate d4/d1c/d40/f11b 1474120 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:4/833: link d5/d7/d1f/d84/d6d/fae d5/d7/d1f/d84/d6d/d118/f124 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:4/834: stat d5/d7/d1f/d84/d6d/fbb 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:1/881: write d6/dc/d8a/d3f/d54/f18 [263465,36565] 0 2024-04-22T15:19:03.800 INFO:tasks.workunit.client.0.smithi096.stdout:3/809: rmdir da/db/de 39 2024-04-22T15:19:03.801 INFO:tasks.workunit.client.0.smithi096.stdout:8/852: chown d4/l29 14 1 2024-04-22T15:19:03.801 INFO:tasks.workunit.client.0.smithi096.stdout:3/810: truncate da/db/de/ddc/d26/dc6/dd2/fed 346817 0 2024-04-22T15:19:03.801 INFO:tasks.workunit.client.1.smithi196.stdout:6/614: dread dd/d1f/d38/d40/f43 [0,4194304] 0 2024-04-22T15:19:03.801 INFO:tasks.workunit.client.1.smithi196.stdout:2/618: write d1/fa2 [45384,13827] 0 2024-04-22T15:19:03.801 INFO:tasks.workunit.client.1.smithi196.stdout:5/624: rename d1/d17/f31 to d1/d17/d50/fcd 0 2024-04-22T15:19:03.802 INFO:tasks.workunit.client.1.smithi196.stdout:8/608: chown d2/dc/fd 0 1 2024-04-22T15:19:03.802 INFO:tasks.workunit.client.1.smithi196.stdout:1/562: dwrite d1/db/dc/d26/f30 [0,4194304] 0 2024-04-22T15:19:03.802 INFO:tasks.workunit.client.1.smithi196.stdout:3/607: sync 2024-04-22T15:19:03.802 INFO:tasks.workunit.client.1.smithi196.stdout:0/626: sync 2024-04-22T15:19:03.802 INFO:tasks.workunit.client.1.smithi196.stdout:3/608: fsync d0/d15/da7/d18/f4c 0 2024-04-22T15:19:03.802 INFO:tasks.workunit.client.1.smithi196.stdout:3/609: stat d0/d15/d48/d51/d7f 0 2024-04-22T15:19:03.802 INFO:tasks.workunit.client.1.smithi196.stdout:2/619: mknod d1/d13/d25/ce8 0 2024-04-22T15:19:03.802 INFO:tasks.workunit.client.0.smithi096.stdout:8/853: creat d4/d1c/de2/f11e x:0 0 0 2024-04-22T15:19:03.803 INFO:tasks.workunit.client.1.smithi196.stdout:5/625: fsync d1/d17/d1a/d2f/d30/d5e/f77 0 2024-04-22T15:19:03.803 INFO:tasks.workunit.client.1.smithi196.stdout:4/554: dwrite d6/d14/d18/d40/f6c [0,4194304] 0 2024-04-22T15:19:03.810 INFO:tasks.workunit.client.0.smithi096.stdout:9/775: dwrite d1/d25/db2/d98/fd0 [0,4194304] 0 2024-04-22T15:19:03.811 INFO:tasks.workunit.client.0.smithi096.stdout:3/811: mkdir da/db/de/ddc/d102 0 2024-04-22T15:19:03.815 INFO:tasks.workunit.client.0.smithi096.stdout:1/882: write d6/dc/d8a/d29/f2f [828669,76497] 0 2024-04-22T15:19:03.815 INFO:tasks.workunit.client.0.smithi096.stdout:9/776: chown d1/df/d95/d9d/db5/d43/f44 80916 1 2024-04-22T15:19:03.815 INFO:tasks.workunit.client.1.smithi196.stdout:6/615: write dd/d20/d5c/d5e/d76/f85 [4787951,109278] 0 2024-04-22T15:19:03.818 INFO:tasks.workunit.client.0.smithi096.stdout:3/812: rmdir da/db/dc/d8d 39 2024-04-22T15:19:03.818 INFO:tasks.workunit.client.1.smithi196.stdout:1/563: write d1/db/dc/de/d5a/d94/fa4 [363416,51630] 0 2024-04-22T15:19:03.822 INFO:tasks.workunit.client.0.smithi096.stdout:1/883: unlink d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/cfa 0 2024-04-22T15:19:03.829 INFO:tasks.workunit.client.1.smithi196.stdout:0/627: mkdir d3/de/d14/d86/dd5 0 2024-04-22T15:19:03.830 INFO:tasks.workunit.client.0.smithi096.stdout:1/884: mknod d6/dc/d8a/d3f/d54/d32/d3b/d3d/c115 0 2024-04-22T15:19:03.834 INFO:tasks.workunit.client.1.smithi196.stdout:9/593: write db/dd/de/d15/d31/d64/f62 [900392,72718] 0 2024-04-22T15:19:03.834 INFO:tasks.workunit.client.1.smithi196.stdout:9/594: fsync db/ff 0 2024-04-22T15:19:03.835 INFO:tasks.workunit.client.0.smithi096.stdout:1/885: mkdir d6/dc/d8a/d3f/d54/d32/d8c/d45/d7b/d116 0 2024-04-22T15:19:03.838 INFO:tasks.workunit.client.0.smithi096.stdout:3/813: link da/db/de/d8b/db5/fcd da/db/de/ddc/d26/dc6/f103 0 2024-04-22T15:19:03.839 INFO:tasks.workunit.client.0.smithi096.stdout:3/814: stat da/db/dc/d28/le2 0 2024-04-22T15:19:03.840 INFO:tasks.workunit.client.0.smithi096.stdout:3/815: write da/db/de/ddc/d26/ffe [1032547,118425] 0 2024-04-22T15:19:03.841 INFO:tasks.workunit.client.0.smithi096.stdout:3/816: chown da/db/de/d19/d2d 1 1 2024-04-22T15:19:03.843 INFO:tasks.workunit.client.0.smithi096.stdout:1/886: symlink d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/dc3/l117 0 2024-04-22T15:19:03.845 INFO:tasks.workunit.client.1.smithi196.stdout:8/609: creat d2/d5/d93/fad x:0 0 0 2024-04-22T15:19:03.845 INFO:tasks.workunit.client.1.smithi196.stdout:3/610: dwrite d0/d7/d8/f5c [0,4194304] 0 2024-04-22T15:19:03.846 INFO:tasks.workunit.client.1.smithi196.stdout:8/610: stat d2/d6a/d7b/f94 0 2024-04-22T15:19:03.846 INFO:tasks.workunit.client.1.smithi196.stdout:2/620: write d1/f18 [2107516,54248] 0 2024-04-22T15:19:03.848 INFO:tasks.workunit.client.0.smithi096.stdout:1/887: symlink d6/dc/d8a/d3f/d54/d32/d3b/l118 0 2024-04-22T15:19:03.848 INFO:tasks.workunit.client.1.smithi196.stdout:8/611: truncate d2/d5/faa 148617 0 2024-04-22T15:19:03.850 INFO:tasks.workunit.client.1.smithi196.stdout:1/564: mknod d1/db/d16/d6d/cb0 0 2024-04-22T15:19:03.850 INFO:tasks.workunit.client.0.smithi096.stdout:3/817: link da/db/de/ddc/f53 da/db/d12/d9b/f104 0 2024-04-22T15:19:03.853 INFO:tasks.workunit.client.1.smithi196.stdout:6/616: write dd/f15 [2969656,8454] 0 2024-04-22T15:19:03.854 INFO:tasks.workunit.client.1.smithi196.stdout:0/628: mknod d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/db7/cd6 0 2024-04-22T15:19:03.856 INFO:tasks.workunit.client.1.smithi196.stdout:1/565: dwrite d1/db/dc/dad/f9b [0,4194304] 0 2024-04-22T15:19:03.857 INFO:tasks.workunit.client.0.smithi096.stdout:3/818: link da/db/dc/d28/d96/d97/fa6 da/db/de/ddc/d26/dc6/f105 0 2024-04-22T15:19:03.857 INFO:tasks.workunit.client.0.smithi096.stdout:1/888: read d6/dc/d8a/d3f/d54/fa5 [255086,105081] 0 2024-04-22T15:19:03.859 INFO:tasks.workunit.client.1.smithi196.stdout:3/611: symlink d0/d7/d8/lcc 0 2024-04-22T15:19:03.860 INFO:tasks.workunit.client.1.smithi196.stdout:3/612: dread - d0/d15/d31/d9d/d9e/fc2 zero size 2024-04-22T15:19:03.863 INFO:tasks.workunit.client.1.smithi196.stdout:8/612: unlink d2/d8/d49/d6c/f75 0 2024-04-22T15:19:03.864 INFO:tasks.workunit.client.1.smithi196.stdout:8/613: stat d2/d5/d1a 0 2024-04-22T15:19:03.865 INFO:tasks.workunit.client.0.smithi096.stdout:3/819: dwrite da/db/de/ddc/f53 [0,4194304] 0 2024-04-22T15:19:03.867 INFO:tasks.workunit.client.1.smithi196.stdout:3/613: dwrite d0/d15/da7/d4a/d44/d8c/faa [0,4194304] 0 2024-04-22T15:19:03.867 INFO:tasks.workunit.client.1.smithi196.stdout:6/617: read dd/f26 [3679453,20268] 0 2024-04-22T15:19:03.869 INFO:tasks.workunit.client.1.smithi196.stdout:1/566: mkdir d1/db/dc/de/d96/d9d/db1 0 2024-04-22T15:19:03.870 INFO:tasks.workunit.client.1.smithi196.stdout:7/576: rename d14/d26/d5a/d5f/d6e to d14/d26/d5a/dbb 0 2024-04-22T15:19:03.871 INFO:tasks.workunit.client.0.smithi096.stdout:3/820: mkdir da/db/de/d8b/d106 0 2024-04-22T15:19:03.871 INFO:tasks.workunit.client.1.smithi196.stdout:3/614: write d0/d7/f38 [179217,118904] 0 2024-04-22T15:19:03.873 INFO:tasks.workunit.client.1.smithi196.stdout:2/621: creat d1/da4/fe9 x:0 0 0 2024-04-22T15:19:03.875 INFO:tasks.workunit.client.1.smithi196.stdout:8/614: fdatasync d2/dc/fe 0 2024-04-22T15:19:03.875 INFO:tasks.workunit.client.0.smithi096.stdout:3/821: creat da/db/de/d8b/f107 x:0 0 0 2024-04-22T15:19:03.875 INFO:tasks.workunit.client.1.smithi196.stdout:8/615: chown d2/dc/f20 3 1 2024-04-22T15:19:03.876 INFO:tasks.workunit.client.0.smithi096.stdout:3/822: write da/db/d12/f56 [1439081,52771] 0 2024-04-22T15:19:03.881 INFO:tasks.workunit.client.1.smithi196.stdout:6/618: creat dd/d1f/d37/d3c/d96/fb3 x:0 0 0 2024-04-22T15:19:03.883 INFO:tasks.workunit.client.1.smithi196.stdout:1/567: dwrite d1/d54/d70/f89 [0,4194304] 0 2024-04-22T15:19:03.884 INFO:tasks.workunit.client.1.smithi196.stdout:1/568: chown d1/db/l2a 15522 1 2024-04-22T15:19:03.886 INFO:tasks.workunit.client.1.smithi196.stdout:0/629: rename d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/cc4 to d3/d6/dc/d8c/d8d/da8/cd7 0 2024-04-22T15:19:03.891 INFO:tasks.workunit.client.1.smithi196.stdout:0/630: dread d3/d6/dc/d8c/d8d/d98/fca [0,4194304] 0 2024-04-22T15:19:03.891 INFO:tasks.workunit.client.0.smithi096.stdout:1/889: truncate d6/dc/d8a/d3f/d62/d7d/f84 347744 0 2024-04-22T15:19:03.893 INFO:tasks.workunit.client.1.smithi196.stdout:2/622: creat d1/d23/d27/d2e/d55/d5b/d7c/dbd/fea x:0 0 0 2024-04-22T15:19:03.895 INFO:tasks.workunit.client.1.smithi196.stdout:2/623: chown d1/f18 722403 1 2024-04-22T15:19:03.896 INFO:tasks.workunit.client.1.smithi196.stdout:6/619: creat dd/d1f/d37/d7b/fb4 x:0 0 0 2024-04-22T15:19:03.898 INFO:tasks.workunit.client.0.smithi096.stdout:1/890: creat d6/dc/d8a/d3f/d62/dc2/dcb/f119 x:0 0 0 2024-04-22T15:19:03.899 INFO:tasks.workunit.client.0.smithi096.stdout:1/891: chown d6/dc/d8a/d3f/f73 5244611 1 2024-04-22T15:19:03.900 INFO:tasks.workunit.client.1.smithi196.stdout:9/595: dwrite db/d11/d29/d57/fbc [0,4194304] 0 2024-04-22T15:19:03.900 INFO:tasks.workunit.client.1.smithi196.stdout:0/631: dwrite d3/de/d1c/d27/f93 [0,4194304] 0 2024-04-22T15:19:03.900 INFO:tasks.workunit.client.1.smithi196.stdout:8/616: rename d2/d6a/dac to d2/d3f/d4b/d79/dae 0 2024-04-22T15:19:03.904 INFO:tasks.workunit.client.0.smithi096.stdout:1/892: read d6/dc/d8a/d3f/d54/d32/d3b/d56/f7f [257609,120637] 0 2024-04-22T15:19:03.905 INFO:tasks.workunit.client.0.smithi096.stdout:3/823: write da/f8f [483052,78733] 0 2024-04-22T15:19:03.907 INFO:tasks.workunit.client.0.smithi096.stdout:3/824: write da/db/de/d19/f30 [4211567,90913] 0 2024-04-22T15:19:03.910 INFO:tasks.workunit.client.1.smithi196.stdout:6/620: unlink dd/d20/d4d/f91 0 2024-04-22T15:19:03.914 INFO:tasks.workunit.client.1.smithi196.stdout:3/615: write d0/d15/da7/d4a/d44/f3c [1163748,25642] 0 2024-04-22T15:19:03.914 INFO:tasks.workunit.client.1.smithi196.stdout:3/616: fsync d0/d7/f38 0 2024-04-22T15:19:03.914 INFO:tasks.workunit.client.1.smithi196.stdout:2/624: dwrite d1/d13/fc7 [0,4194304] 0 2024-04-22T15:19:03.914 INFO:tasks.workunit.client.1.smithi196.stdout:2/625: dread - d1/d23/d27/d73/fda zero size 2024-04-22T15:19:03.914 INFO:tasks.workunit.client.1.smithi196.stdout:2/626: stat d1/da4/fe9 0 2024-04-22T15:19:03.915 INFO:tasks.workunit.client.0.smithi096.stdout:1/893: creat d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/f11a x:0 0 0 2024-04-22T15:19:03.922 INFO:tasks.workunit.client.1.smithi196.stdout:7/577: dread d14/d25/f3b [0,4194304] 0 2024-04-22T15:19:03.924 INFO:tasks.workunit.client.1.smithi196.stdout:4/555: sync 2024-04-22T15:19:03.924 INFO:tasks.workunit.client.1.smithi196.stdout:5/626: sync 2024-04-22T15:19:03.925 INFO:tasks.workunit.client.1.smithi196.stdout:4/556: chown d6/d14/d18/d26/d32/d53/f54 86 1 2024-04-22T15:19:03.927 INFO:tasks.workunit.client.0.smithi096.stdout:1/894: creat d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/d104/f11b x:0 0 0 2024-04-22T15:19:03.928 INFO:tasks.workunit.client.0.smithi096.stdout:1/895: fdatasync d6/f1d 0 2024-04-22T15:19:03.930 INFO:tasks.workunit.client.0.smithi096.stdout:1/896: write d6/dc/dbd/ffb [436676,1633] 0 2024-04-22T15:19:03.930 INFO:tasks.workunit.client.0.smithi096.stdout:3/825: link da/db/dc/d59/lec da/db/d12/l108 0 2024-04-22T15:19:03.931 INFO:tasks.workunit.client.1.smithi196.stdout:1/569: creat d1/fb2 x:0 0 0 2024-04-22T15:19:03.932 INFO:tasks.workunit.client.0.smithi096.stdout:1/897: write d6/dc/d8a/d3f/d54/d32/d3b/d56/ffc [470086,34298] 0 2024-04-22T15:19:03.934 INFO:tasks.workunit.client.0.smithi096.stdout:3/826: creat da/db/d9a/f109 x:0 0 0 2024-04-22T15:19:03.936 INFO:tasks.workunit.client.1.smithi196.stdout:3/617: symlink d0/d15/d4d/d87/lcd 0 2024-04-22T15:19:03.936 INFO:tasks.workunit.client.1.smithi196.stdout:6/621: dread dd/d20/d5c/d6a/faa [0,4194304] 0 2024-04-22T15:19:03.937 INFO:tasks.workunit.client.0.smithi096.stdout:3/827: creat da/db/de/ddc/d26/db2/f10a x:0 0 0 2024-04-22T15:19:03.938 INFO:tasks.workunit.client.1.smithi196.stdout:9/596: unlink db/dd/lb9 0 2024-04-22T15:19:03.942 INFO:tasks.workunit.client.1.smithi196.stdout:7/578: unlink l12 0 2024-04-22T15:19:03.944 INFO:tasks.workunit.client.0.smithi096.stdout:1/898: getdents d6/dc/d8a/d3f/d62 0 2024-04-22T15:19:03.947 INFO:tasks.workunit.client.0.smithi096.stdout:3/828: getdents da/db/de/ddc/d26 0 2024-04-22T15:19:03.947 INFO:tasks.workunit.client.1.smithi196.stdout:1/570: symlink d1/db/dc/de/d96/lb3 0 2024-04-22T15:19:03.949 INFO:tasks.workunit.client.1.smithi196.stdout:2/627: symlink d1/d13/d1f/ddf/leb 0 2024-04-22T15:19:03.950 INFO:tasks.workunit.client.0.smithi096.stdout:3/829: fsync da/db/de/ddc/f31 0 2024-04-22T15:19:03.955 INFO:tasks.workunit.client.0.smithi096.stdout:3/830: symlink da/db/de/d8b/l10b 0 2024-04-22T15:19:03.956 INFO:tasks.workunit.client.1.smithi196.stdout:8/617: creat d2/d5/faf x:0 0 0 2024-04-22T15:19:03.958 INFO:tasks.workunit.client.1.smithi196.stdout:7/579: dread d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:19:03.960 INFO:tasks.workunit.client.1.smithi196.stdout:0/632: creat d3/d9/d36/d9f/dd1/d65/d7f/fd8 x:0 0 0 2024-04-22T15:19:03.962 INFO:tasks.workunit.client.1.smithi196.stdout:1/571: creat d1/db/d16/d1e/d49/d90/fb4 x:0 0 0 2024-04-22T15:19:03.963 INFO:tasks.workunit.client.1.smithi196.stdout:1/572: chown d1/db/d16/f7e 2559 1 2024-04-22T15:19:03.963 INFO:tasks.workunit.client.1.smithi196.stdout:2/628: write d1/d23/d27/d2e/f88 [540759,98489] 0 2024-04-22T15:19:03.963 INFO:tasks.workunit.client.1.smithi196.stdout:6/622: symlink dd/d70/lb5 0 2024-04-22T15:19:03.963 INFO:tasks.workunit.client.0.smithi096.stdout:3/831: write da/db/de/d19/f2a [3535793,111197] 0 2024-04-22T15:19:03.964 INFO:tasks.workunit.client.1.smithi196.stdout:6/623: fsync dd/d20/d4d/f65 0 2024-04-22T15:19:03.968 INFO:tasks.workunit.client.1.smithi196.stdout:8/618: mknod d2/d5/d1a/cb0 0 2024-04-22T15:19:03.968 INFO:tasks.workunit.client.1.smithi196.stdout:8/619: fdatasync d2/d5/f23 0 2024-04-22T15:19:03.969 INFO:tasks.workunit.client.0.smithi096.stdout:3/832: symlink da/db/de/d38/dfc/l10c 0 2024-04-22T15:19:03.972 INFO:tasks.workunit.client.1.smithi196.stdout:5/627: rmdir d1/d14/d6a/dba 0 2024-04-22T15:19:03.973 INFO:tasks.workunit.client.1.smithi196.stdout:4/557: creat d6/d13/d15/fb2 x:0 0 0 2024-04-22T15:19:03.974 INFO:tasks.workunit.client.1.smithi196.stdout:4/558: chown d6/d14/f87 11 1 2024-04-22T15:19:03.975 INFO:tasks.workunit.client.0.smithi096.stdout:3/833: unlink da/db/de/d19/d2d/d37/fb1 0 2024-04-22T15:19:03.977 INFO:tasks.workunit.client.1.smithi196.stdout:9/597: dwrite db/dd/d1b/d2c/f37 [0,4194304] 0 2024-04-22T15:19:03.977 INFO:tasks.workunit.client.1.smithi196.stdout:0/633: mknod d3/db4/dc7/cd9 0 2024-04-22T15:19:03.978 INFO:tasks.workunit.client.1.smithi196.stdout:9/598: write db/dd/d1b/d2c/f37 [677378,29869] 0 2024-04-22T15:19:03.980 INFO:tasks.workunit.client.0.smithi096.stdout:3/834: rmdir da/db/dc/d59 39 2024-04-22T15:19:03.980 INFO:tasks.workunit.client.1.smithi196.stdout:0/634: chown d3/d9/d36/d9f/dd1/d65/d7f/d51/f6a 7 1 2024-04-22T15:19:03.981 INFO:tasks.workunit.client.1.smithi196.stdout:1/573: unlink d1/db/dc/dad/f53 0 2024-04-22T15:19:03.982 INFO:tasks.workunit.client.0.smithi096.stdout:3/835: dread da/db/de/ddc/d26/dc6/dd2/fed [0,4194304] 0 2024-04-22T15:19:03.984 INFO:tasks.workunit.client.1.smithi196.stdout:6/624: unlink dd/l90 0 2024-04-22T15:19:03.985 INFO:tasks.workunit.client.1.smithi196.stdout:6/625: dread - dd/d20/d5c/d5e/f7c zero size 2024-04-22T15:19:03.985 INFO:tasks.workunit.client.1.smithi196.stdout:3/618: link d0/d15/d48/d51/d5e/c68 d0/d7/d8/cce 0 2024-04-22T15:19:03.986 INFO:tasks.workunit.client.1.smithi196.stdout:3/619: dread - d0/d15/d31/d55/d6e/f8d zero size 2024-04-22T15:19:03.986 INFO:tasks.workunit.client.0.smithi096.stdout:3/836: mkdir da/db/d12/dd5/d10d 0 2024-04-22T15:19:03.987 INFO:tasks.workunit.client.1.smithi196.stdout:6/626: truncate dd/d1f/d4e/d53/d55/f9a 601702 0 2024-04-22T15:19:03.987 INFO:tasks.workunit.client.1.smithi196.stdout:3/620: dread - d0/d15/da7/d4a/f86 zero size 2024-04-22T15:19:03.987 INFO:tasks.workunit.client.1.smithi196.stdout:3/621: stat d0/d15/da7/d4a/f45 0 2024-04-22T15:19:03.988 INFO:tasks.workunit.client.0.smithi096.stdout:3/837: write da/db/d12/d9b/f104 [3035752,45115] 0 2024-04-22T15:19:03.988 INFO:tasks.workunit.client.0.smithi096.stdout:3/838: chown da/db/de/d8b/ff6 45245 1 2024-04-22T15:19:03.990 INFO:tasks.workunit.client.1.smithi196.stdout:7/580: creat d14/d4d/da3/fbc x:0 0 0 2024-04-22T15:19:03.992 INFO:tasks.workunit.client.0.smithi096.stdout:3/839: rmdir da/db/dc/d28/d96 39 2024-04-22T15:19:03.994 INFO:tasks.workunit.client.1.smithi196.stdout:4/559: mkdir d6/d13/d15/d2c/d8c/db3 0 2024-04-22T15:19:03.999 INFO:tasks.workunit.client.0.smithi096.stdout:3/840: mknod da/db/de/ddc/c10e 0 2024-04-22T15:19:04.001 INFO:tasks.workunit.client.1.smithi196.stdout:1/574: symlink d1/db/dc/dad/lb5 0 2024-04-22T15:19:04.001 INFO:tasks.workunit.client.0.smithi096.stdout:3/841: write da/db/de/ddc/d68/dbc/ff1 [217914,105400] 0 2024-04-22T15:19:04.002 INFO:tasks.workunit.client.1.smithi196.stdout:2/629: truncate d1/d23/d27/d2e/d55/d5b/d7c/f91 390710 0 2024-04-22T15:19:04.003 INFO:tasks.workunit.client.1.smithi196.stdout:6/627: chown dd/d1f/d37/l99 5172 1 2024-04-22T15:19:04.004 INFO:tasks.workunit.client.0.smithi096.stdout:3/842: rmdir da/db/d9a 39 2024-04-22T15:19:04.008 INFO:tasks.workunit.client.1.smithi196.stdout:2/630: dwrite d1/d13/d41/fd4 [0,4194304] 0 2024-04-22T15:19:04.010 INFO:tasks.workunit.client.1.smithi196.stdout:5/628: write d1/d17/d1a/d2f/d30/d6e/fa2 [445818,75820] 0 2024-04-22T15:19:04.011 INFO:tasks.workunit.client.0.smithi096.stdout:3/843: dwrite da/db/de/d19/d2d/fa8 [4194304,4194304] 0 2024-04-22T15:19:04.014 INFO:tasks.workunit.client.1.smithi196.stdout:1/575: creat d1/db/dc/d93/fb6 x:0 0 0 2024-04-22T15:19:04.015 INFO:tasks.workunit.client.0.smithi096.stdout:3/844: creat da/db/dc/d28/d69/f10f x:0 0 0 2024-04-22T15:19:04.016 INFO:tasks.workunit.client.1.smithi196.stdout:1/576: write d1/db/d16/d1e/d49/d90/fb4 [782988,79026] 0 2024-04-22T15:19:04.019 INFO:tasks.workunit.client.1.smithi196.stdout:3/622: write d0/d15/da7/d4a/f52 [910033,60179] 0 2024-04-22T15:19:04.020 INFO:tasks.workunit.client.1.smithi196.stdout:2/631: rmdir d1/d23/d27/d2e/d55/d5b/d93/d97 39 2024-04-22T15:19:04.021 INFO:tasks.workunit.client.1.smithi196.stdout:8/620: truncate d2/d35/f92 556768 0 2024-04-22T15:19:04.023 INFO:tasks.workunit.client.1.smithi196.stdout:7/581: creat d14/d25/d80/fbd x:0 0 0 2024-04-22T15:19:04.024 INFO:tasks.workunit.client.1.smithi196.stdout:4/560: mknod d6/d14/d18/d26/d32/cb4 0 2024-04-22T15:19:04.025 INFO:tasks.workunit.client.1.smithi196.stdout:9/599: fdatasync db/dd/de/d15/d31/d99/fa1 0 2024-04-22T15:19:04.026 INFO:tasks.workunit.client.1.smithi196.stdout:3/623: dwrite d0/d15/da7/d18/f84 [0,4194304] 0 2024-04-22T15:19:04.029 INFO:tasks.workunit.client.1.smithi196.stdout:9/600: stat db/d11/d29/d57/f23 0 2024-04-22T15:19:04.030 INFO:tasks.workunit.client.1.smithi196.stdout:9/601: chown db/d11/d29/d57/f33 0 1 2024-04-22T15:19:04.031 INFO:tasks.workunit.client.1.smithi196.stdout:7/582: dread d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:19:04.033 INFO:tasks.workunit.client.0.smithi096.stdout:3/845: getdents da/db/d9a/def 0 2024-04-22T15:19:04.038 INFO:tasks.workunit.client.1.smithi196.stdout:7/583: dwrite d14/d26/d2a/d72/f99 [0,4194304] 0 2024-04-22T15:19:04.039 INFO:tasks.workunit.client.1.smithi196.stdout:7/584: fsync d14/fa5 0 2024-04-22T15:19:04.041 INFO:tasks.workunit.client.1.smithi196.stdout:6/628: truncate dd/f11 4344231 0 2024-04-22T15:19:04.042 INFO:tasks.workunit.client.1.smithi196.stdout:6/629: chown dd/d1f/d37/c56 1883 1 2024-04-22T15:19:04.043 INFO:tasks.workunit.client.1.smithi196.stdout:1/577: dread d1/db/dc/de/d5a/d94/fa4 [0,4194304] 0 2024-04-22T15:19:04.045 INFO:tasks.workunit.client.1.smithi196.stdout:3/624: rename d0/d7 to d0/d15/d31/d55/dcf 0 2024-04-22T15:19:04.048 INFO:tasks.workunit.client.1.smithi196.stdout:4/561: dwrite d6/d13/d15/d2c/d62/f99 [0,4194304] 0 2024-04-22T15:19:04.049 INFO:tasks.workunit.client.1.smithi196.stdout:4/562: fsync d6/d13/d15/f3d 0 2024-04-22T15:19:04.050 INFO:tasks.workunit.client.1.smithi196.stdout:4/563: readlink d6/d14/d18/d26/d32/l76 0 2024-04-22T15:19:04.050 INFO:tasks.workunit.client.1.smithi196.stdout:9/602: rmdir db/dd/de/d15/d31/d99 39 2024-04-22T15:19:04.051 INFO:tasks.workunit.client.1.smithi196.stdout:8/621: link d2/d8/d47/d68/fa3 d2/d6a/d7b/fb1 0 2024-04-22T15:19:04.051 INFO:tasks.workunit.client.0.smithi096.stdout:3/846: getdents da/db/de/ddc/d68/dbc 0 2024-04-22T15:19:04.052 INFO:tasks.workunit.client.0.smithi096.stdout:3/847: dread - da/db/dc/d28/dda/f100 zero size 2024-04-22T15:19:04.057 INFO:tasks.workunit.client.1.smithi196.stdout:5/629: link d1/d17/d1a/l5f d1/d4/lce 0 2024-04-22T15:19:04.057 INFO:tasks.workunit.client.1.smithi196.stdout:0/635: sync 2024-04-22T15:19:04.058 INFO:tasks.workunit.client.0.smithi096.stdout:3/848: fsync da/db/dc/d8d/fc5 0 2024-04-22T15:19:04.060 INFO:tasks.workunit.client.1.smithi196.stdout:2/632: link d1/f3 d1/d23/d27/fec 0 2024-04-22T15:19:04.062 INFO:tasks.workunit.client.1.smithi196.stdout:6/630: mkdir dd/d1f/d37/d7b/db6 0 2024-04-22T15:19:04.066 INFO:tasks.workunit.client.1.smithi196.stdout:3/625: rename d0/d15/d48 to d0/d15/d31/dd0 0 2024-04-22T15:19:04.066 INFO:tasks.workunit.client.0.smithi096.stdout:3/849: dwrite da/db/dc/d28/d69/d89/f65 [0,4194304] 0 2024-04-22T15:19:04.068 INFO:tasks.workunit.client.1.smithi196.stdout:6/631: dwrite dd/d1f/d37/d7b/fb4 [0,4194304] 0 2024-04-22T15:19:04.068 INFO:tasks.workunit.client.0.smithi096.stdout:3/850: fdatasync da/db/d9a/f109 0 2024-04-22T15:19:04.069 INFO:tasks.workunit.client.1.smithi196.stdout:4/564: fdatasync d6/d90/f60 0 2024-04-22T15:19:04.072 INFO:tasks.workunit.client.1.smithi196.stdout:8/622: unlink d2/d5/d1a/la0 0 2024-04-22T15:19:04.073 INFO:tasks.workunit.client.0.smithi096.stdout:3/851: symlink da/db/dc/d28/d69/l110 0 2024-04-22T15:19:04.075 INFO:tasks.workunit.client.1.smithi196.stdout:0/636: mknod d3/d6/dc/d8c/cda 0 2024-04-22T15:19:04.078 INFO:tasks.workunit.client.1.smithi196.stdout:2/633: symlink d1/d23/led 0 2024-04-22T15:19:04.083 INFO:tasks.workunit.client.1.smithi196.stdout:0/637: dwrite d3/d9/d36/d9f/dd1/d65/d7f/f9a [0,4194304] 0 2024-04-22T15:19:04.087 INFO:tasks.workunit.client.0.smithi096.stdout:3/852: link da/db/d12/laf da/db/de/d19/d3e/l111 0 2024-04-22T15:19:04.091 INFO:tasks.workunit.client.1.smithi196.stdout:6/632: rmdir dd/d2b/d8d 39 2024-04-22T15:19:04.092 INFO:tasks.workunit.client.1.smithi196.stdout:6/633: write dd/d1f/d4e/d53/d55/f87 [4938153,30206] 0 2024-04-22T15:19:04.099 INFO:tasks.workunit.client.1.smithi196.stdout:7/585: write d14/d26/d28/d29/fb3 [1113829,46305] 0 2024-04-22T15:19:04.101 INFO:tasks.workunit.client.1.smithi196.stdout:1/578: truncate d1/d54/d70/f89 2992518 0 2024-04-22T15:19:04.103 INFO:tasks.workunit.client.1.smithi196.stdout:2/634: truncate d1/d13/d1f/f60 545303 0 2024-04-22T15:19:04.104 INFO:tasks.workunit.client.1.smithi196.stdout:3/626: write d0/d15/d31/d55/dcf/d8/d70/f7e [5044763,47738] 0 2024-04-22T15:19:04.106 INFO:tasks.workunit.client.0.smithi096.stdout:3/853: write da/db/d10/f33 [2319253,45262] 0 2024-04-22T15:19:04.108 INFO:tasks.workunit.client.1.smithi196.stdout:6/634: creat dd/d20/d5c/fb7 x:0 0 0 2024-04-22T15:19:04.108 INFO:tasks.workunit.client.1.smithi196.stdout:5/630: truncate d1/d4/fc2 2605057 0 2024-04-22T15:19:04.111 INFO:tasks.workunit.client.0.smithi096.stdout:3/854: chown l1 1089689 1 2024-04-22T15:19:04.116 INFO:tasks.workunit.client.0.smithi096.stdout:3/855: creat da/db/de/d87/f112 x:0 0 0 2024-04-22T15:19:04.122 INFO:tasks.workunit.client.1.smithi196.stdout:0/638: mkdir d3/d9/d36/d9f/dd1/d65/d7f/ddb 0 2024-04-22T15:19:04.122 INFO:tasks.workunit.client.1.smithi196.stdout:9/603: write db/dd/de/d15/d31/d64/fad [4020618,15645] 0 2024-04-22T15:19:04.122 INFO:tasks.workunit.client.0.smithi096.stdout:3/856: dwrite da/db/de/d19/f6f [0,4194304] 0 2024-04-22T15:19:04.123 INFO:tasks.workunit.client.0.smithi096.stdout:3/857: write da/db/de/d19/d3e/f42 [4176772,101610] 0 2024-04-22T15:19:04.124 INFO:tasks.workunit.client.1.smithi196.stdout:2/635: symlink d1/d23/d27/d2e/d55/lee 0 2024-04-22T15:19:04.125 INFO:tasks.workunit.client.1.smithi196.stdout:2/636: write d1/d23/d27/d2e/fd8 [40912,110510] 0 2024-04-22T15:19:04.127 INFO:tasks.workunit.client.1.smithi196.stdout:6/635: chown dd/d1f/d4e/d53/d55/laf 24072 1 2024-04-22T15:19:04.129 INFO:tasks.workunit.client.0.smithi096.stdout:3/858: creat da/db/dc/d8d/f113 x:0 0 0 2024-04-22T15:19:04.130 INFO:tasks.workunit.client.0.smithi096.stdout:3/859: fsync da/d3f/f5b 0 2024-04-22T15:19:04.130 INFO:tasks.workunit.client.1.smithi196.stdout:5/631: rmdir d1/d78/db0 39 2024-04-22T15:19:04.131 INFO:tasks.workunit.client.1.smithi196.stdout:4/565: link d6/d14/d18/d26/d32/d67/c82 d6/d14/d18/d26/d32/d67/d9a/da5/cb5 0 2024-04-22T15:19:04.133 INFO:tasks.workunit.client.1.smithi196.stdout:8/623: link d2/d3f/d4b/d79/c9a d2/d8/d49/d8b/cb2 0 2024-04-22T15:19:04.135 INFO:tasks.workunit.client.1.smithi196.stdout:8/624: chown d2/d8/d49/d6c 41328785 1 2024-04-22T15:19:04.135 INFO:tasks.workunit.client.0.smithi096.stdout:3/860: mkdir da/db/de/d38/d114 0 2024-04-22T15:19:04.136 INFO:tasks.workunit.client.1.smithi196.stdout:7/586: mknod d14/d26/d5a/cbe 0 2024-04-22T15:19:04.136 INFO:tasks.workunit.client.0.smithi096.stdout:3/861: chown da/db/d10/f33 0 1 2024-04-22T15:19:04.136 INFO:tasks.workunit.client.1.smithi196.stdout:5/632: dwrite d1/d4/f9c [0,4194304] 0 2024-04-22T15:19:04.137 INFO:tasks.workunit.client.1.smithi196.stdout:5/633: readlink d1/d17/d1a/d2f/d30/d6e/d7a/lc4 0 2024-04-22T15:19:04.138 INFO:tasks.workunit.client.1.smithi196.stdout:1/579: creat d1/db/d16/d1e/d49/d7f/fb7 x:0 0 0 2024-04-22T15:19:04.139 INFO:tasks.workunit.client.1.smithi196.stdout:0/639: rename d3/d6/dc/d8c/db5 to d3/de/d1c/d27/ddc 0 2024-04-22T15:19:04.139 INFO:tasks.workunit.client.1.smithi196.stdout:5/634: chown d1/d14/d6a/fb8 4012 1 2024-04-22T15:19:04.143 INFO:tasks.workunit.client.0.smithi096.stdout:3/862: rename da/db/de/ddc/f7d to da/db/dc/d28/dda/f115 0 2024-04-22T15:19:04.143 INFO:tasks.workunit.client.0.smithi096.stdout:3/863: stat da/dab/fa2 0 2024-04-22T15:19:04.144 INFO:tasks.workunit.client.0.smithi096.stdout:3/864: chown da/db/dc/d28/d69/cc8 101 1 2024-04-22T15:19:04.149 INFO:tasks.workunit.client.1.smithi196.stdout:2/637: symlink d1/d23/lef 0 2024-04-22T15:19:04.151 INFO:tasks.workunit.client.1.smithi196.stdout:2/638: dread - d1/d13/d41/fd3 zero size 2024-04-22T15:19:04.151 INFO:tasks.workunit.client.1.smithi196.stdout:1/580: dread d1/db/dc/de/d43/f7b [0,4194304] 0 2024-04-22T15:19:04.151 INFO:tasks.workunit.client.0.smithi096.stdout:3/865: truncate da/db/dc/d8d/fc5 675508 0 2024-04-22T15:19:04.152 INFO:tasks.workunit.client.1.smithi196.stdout:8/625: creat d2/d8/d47/d68/fb3 x:0 0 0 2024-04-22T15:19:04.155 INFO:tasks.workunit.client.1.smithi196.stdout:2/639: read d1/d23/d27/d2e/fd8 [19908,120174] 0 2024-04-22T15:19:04.156 INFO:tasks.workunit.client.0.smithi096.stdout:3/866: symlink da/db/de/ddc/d68/l116 0 2024-04-22T15:19:04.161 INFO:tasks.workunit.client.1.smithi196.stdout:7/587: dwrite d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:19:04.161 INFO:tasks.workunit.client.0.smithi096.stdout:3/867: rmdir da/db/de/ddc/d26/dc6 39 2024-04-22T15:19:04.162 INFO:tasks.workunit.client.0.smithi096.stdout:3/868: chown da/db/de/daa/lac 664702 1 2024-04-22T15:19:04.164 INFO:tasks.workunit.client.1.smithi196.stdout:4/566: dread d6/d14/f27 [4194304,4194304] 0 2024-04-22T15:19:04.164 INFO:tasks.workunit.client.1.smithi196.stdout:5/635: rmdir d1/d17/d1a/d2f/d30/d5e 39 2024-04-22T15:19:04.166 INFO:tasks.workunit.client.0.smithi096.stdout:3/869: dread da/db/dc/f49 [0,4194304] 0 2024-04-22T15:19:04.167 INFO:tasks.workunit.client.0.smithi096.stdout:3/870: readlink da/db/dc/d8d/le5 0 2024-04-22T15:19:04.168 INFO:tasks.workunit.client.1.smithi196.stdout:3/627: write d0/d15/da7/fc8 [90510,8044] 0 2024-04-22T15:19:04.173 INFO:tasks.workunit.client.0.smithi096.stdout:3/871: mknod da/db/de/ddc/d26/d52/c117 0 2024-04-22T15:19:04.177 INFO:tasks.workunit.client.0.smithi096.stdout:3/872: symlink da/db/de/d19/d2d/d37/l118 0 2024-04-22T15:19:04.180 INFO:tasks.workunit.client.0.smithi096.stdout:3/873: mkdir da/db/de/d38/d119 0 2024-04-22T15:19:04.181 INFO:tasks.workunit.client.0.smithi096.stdout:3/874: read da/db/de/d19/f6f [3261249,19650] 0 2024-04-22T15:19:04.182 INFO:tasks.workunit.client.0.smithi096.stdout:3/875: chown da/db/de/d38/d119 3496815 1 2024-04-22T15:19:04.187 INFO:tasks.workunit.client.0.smithi096.stdout:3/876: mknod da/dab/c11a 0 2024-04-22T15:19:04.190 INFO:tasks.workunit.client.0.smithi096.stdout:3/877: creat da/db/d10/f11b x:0 0 0 2024-04-22T15:19:04.191 INFO:tasks.workunit.client.0.smithi096.stdout:3/878: write da/d3f/f5b [2023324,78355] 0 2024-04-22T15:19:04.193 INFO:tasks.workunit.client.0.smithi096.stdout:3/879: symlink da/db/d12/d9b/l11c 0 2024-04-22T15:19:04.193 INFO:tasks.workunit.client.0.smithi096.stdout:3/880: chown da/db/de/c23 110 1 2024-04-22T15:19:04.197 INFO:tasks.workunit.client.0.smithi096.stdout:3/881: mknod da/dab/c11d 0 2024-04-22T15:19:04.197 INFO:tasks.workunit.client.0.smithi096.stdout:3/882: chown da/db/de/ddc/c5e 1182 1 2024-04-22T15:19:04.274 INFO:tasks.workunit.client.1.smithi196.stdout:1/581: creat d1/db/d16/d44/fb8 x:0 0 0 2024-04-22T15:19:04.275 INFO:tasks.workunit.client.1.smithi196.stdout:8/626: fsync d2/d5/f74 0 2024-04-22T15:19:04.275 INFO:tasks.workunit.client.1.smithi196.stdout:1/582: read d1/db/dc/dad/f9b [3342800,100214] 0 2024-04-22T15:19:04.276 INFO:tasks.workunit.client.1.smithi196.stdout:2/640: chown d1/c9e 5 1 2024-04-22T15:19:04.278 INFO:tasks.workunit.client.1.smithi196.stdout:9/604: creat db/dd/de/d15/d31/fcf x:0 0 0 2024-04-22T15:19:04.285 INFO:tasks.workunit.client.1.smithi196.stdout:4/567: truncate d6/d14/d18/d26/f6e 242699 0 2024-04-22T15:19:04.287 INFO:tasks.workunit.client.1.smithi196.stdout:3/628: readlink d0/d15/d31/d55/l82 0 2024-04-22T15:19:04.289 INFO:tasks.workunit.client.1.smithi196.stdout:6/636: creat dd/d1f/d37/fb8 x:0 0 0 2024-04-22T15:19:04.294 INFO:tasks.workunit.client.1.smithi196.stdout:8/627: fsync d2/dc/d45/d98/fa6 0 2024-04-22T15:19:04.299 INFO:tasks.workunit.client.1.smithi196.stdout:6/637: dread dd/d2b/f62 [0,4194304] 0 2024-04-22T15:19:04.300 INFO:tasks.workunit.client.1.smithi196.stdout:7/588: fdatasync d14/d25/f42 0 2024-04-22T15:19:04.301 INFO:tasks.workunit.client.1.smithi196.stdout:1/583: dread d1/db/dc/de/d43/f68 [0,4194304] 0 2024-04-22T15:19:04.302 INFO:tasks.workunit.client.1.smithi196.stdout:1/584: readlink d1/l74 0 2024-04-22T15:19:04.311 INFO:tasks.workunit.client.1.smithi196.stdout:8/628: fsync d2/dc/fd 0 2024-04-22T15:19:04.312 INFO:tasks.workunit.client.1.smithi196.stdout:0/640: getdents d3 0 2024-04-22T15:19:04.313 INFO:tasks.workunit.client.1.smithi196.stdout:0/641: write d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/fbf [1896555,12580] 0 2024-04-22T15:19:04.316 INFO:tasks.workunit.client.1.smithi196.stdout:2/641: creat d1/d23/d27/d2e/d55/d5b/d93/d97/ff0 x:0 0 0 2024-04-22T15:19:04.316 INFO:tasks.workunit.client.1.smithi196.stdout:2/642: write d1/d23/d27/d2e/f88 [2341203,81277] 0 2024-04-22T15:19:04.326 INFO:tasks.workunit.client.1.smithi196.stdout:1/585: creat d1/db/dc/dad/fb9 x:0 0 0 2024-04-22T15:19:04.328 INFO:tasks.workunit.client.1.smithi196.stdout:3/629: symlink d0/d15/da7/d4a/ld1 0 2024-04-22T15:19:04.328 INFO:tasks.workunit.client.1.smithi196.stdout:3/630: fdatasync d0/d15/da7/fc8 0 2024-04-22T15:19:04.332 INFO:tasks.workunit.client.1.smithi196.stdout:8/629: chown d2/d3f/f4e 3213 1 2024-04-22T15:19:04.333 INFO:tasks.workunit.client.1.smithi196.stdout:8/630: chown d2/d8/d49/d8b/cb2 444 1 2024-04-22T15:19:04.333 INFO:tasks.workunit.client.0.smithi096.stdout:3/883: symlink da/db/d12/dd5/d10d/l11e 0 2024-04-22T15:19:04.336 INFO:tasks.workunit.client.0.smithi096.stdout:3/884: mkdir da/db/dc/d28/d69/d11f 0 2024-04-22T15:19:04.341 INFO:tasks.workunit.client.0.smithi096.stdout:3/885: creat da/db/de/ddc/d68/db0/f120 x:0 0 0 2024-04-22T15:19:04.377 INFO:tasks.workunit.client.1.smithi196.stdout:5/636: write d1/d17/d1c/f52 [544234,43737] 0 2024-04-22T15:19:04.377 INFO:tasks.workunit.client.1.smithi196.stdout:8/631: fsync d2/d8/f4c 0 2024-04-22T15:19:04.379 INFO:tasks.workunit.client.1.smithi196.stdout:2/643: link d1/d13/d41/fe0 d1/d13/dbc/ff1 0 2024-04-22T15:19:04.379 INFO:tasks.workunit.client.1.smithi196.stdout:1/586: dread d1/db/dc/de/d43/f7b [0,4194304] 0 2024-04-22T15:19:04.380 INFO:tasks.workunit.client.1.smithi196.stdout:7/589: rename d14/d4a/ca4 to d14/cbf 0 2024-04-22T15:19:04.407 INFO:tasks.workunit.client.1.smithi196.stdout:8/632: rmdir d2/d3f/d4b 39 2024-04-22T15:19:04.408 INFO:tasks.workunit.client.1.smithi196.stdout:1/587: mkdir d1/db/d16/d1e/d46/dba 0 2024-04-22T15:19:04.408 INFO:tasks.workunit.client.1.smithi196.stdout:2/644: dread - d1/d23/d27/d2e/d55/d5b/fb3 zero size 2024-04-22T15:19:04.409 INFO:tasks.workunit.client.1.smithi196.stdout:6/638: rename dd/d1f/d38/d40/fa0 to dd/d2b/d8d/fb9 0 2024-04-22T15:19:04.410 INFO:tasks.workunit.client.1.smithi196.stdout:7/590: mkdir d14/d15/dc0 0 2024-04-22T15:19:04.411 INFO:tasks.workunit.client.1.smithi196.stdout:5/637: dwrite d1/d17/d1a/d2f/d30/d6e/d7a/fa3 [0,4194304] 0 2024-04-22T15:19:04.416 INFO:tasks.workunit.client.1.smithi196.stdout:2/645: unlink d1/d13/c61 0 2024-04-22T15:19:04.418 INFO:tasks.workunit.client.1.smithi196.stdout:5/638: chown d1/d17/d1a/d2f/d30/d5e/f77 24 1 2024-04-22T15:19:04.420 INFO:tasks.workunit.client.1.smithi196.stdout:2/646: creat d1/d23/d27/d2e/d55/d5b/d93/d97/ff2 x:0 0 0 2024-04-22T15:19:04.423 INFO:tasks.workunit.client.1.smithi196.stdout:9/605: sync 2024-04-22T15:19:04.423 INFO:tasks.workunit.client.1.smithi196.stdout:4/568: sync 2024-04-22T15:19:04.423 INFO:tasks.workunit.client.1.smithi196.stdout:8/633: dread d2/d5/f46 [0,4194304] 0 2024-04-22T15:19:04.423 INFO:tasks.workunit.client.1.smithi196.stdout:2/647: truncate d1/d23/d27/d2e/f85 726572 0 2024-04-22T15:19:04.425 INFO:tasks.workunit.client.1.smithi196.stdout:7/591: write d14/d25/f90 [1828577,73849] 0 2024-04-22T15:19:04.426 INFO:tasks.workunit.client.1.smithi196.stdout:5/639: getdents d1/d4/d92 0 2024-04-22T15:19:04.428 INFO:tasks.workunit.client.1.smithi196.stdout:9/606: mkdir db/dd/dd0 0 2024-04-22T15:19:04.430 INFO:tasks.workunit.client.1.smithi196.stdout:2/648: truncate d1/d23/d27/f44 268805 0 2024-04-22T15:19:04.430 INFO:tasks.workunit.client.1.smithi196.stdout:7/592: stat d14/d26/d95/l97 0 2024-04-22T15:19:04.431 INFO:tasks.workunit.client.1.smithi196.stdout:8/634: rmdir d2/d35/d96/d9f 0 2024-04-22T15:19:04.432 INFO:tasks.workunit.client.1.smithi196.stdout:1/588: dwrite d1/db/f58 [4194304,4194304] 0 2024-04-22T15:19:04.433 INFO:tasks.workunit.client.1.smithi196.stdout:5/640: unlink d1/d17/d1a/d2f/d30/d6e/lb1 0 2024-04-22T15:19:04.434 INFO:tasks.workunit.client.1.smithi196.stdout:4/569: getdents d6/d13/d15/d2c/d62 0 2024-04-22T15:19:04.437 INFO:tasks.workunit.client.1.smithi196.stdout:5/641: truncate d1/d17/d1a/d2f/d30/d6e/fa2 1363725 0 2024-04-22T15:19:04.437 INFO:tasks.workunit.client.1.smithi196.stdout:7/593: creat d14/d26/d5a/d5f/fc1 x:0 0 0 2024-04-22T15:19:04.438 INFO:tasks.workunit.client.1.smithi196.stdout:7/594: truncate d14/d4a/f71 1030040 0 2024-04-22T15:19:04.439 INFO:tasks.workunit.client.1.smithi196.stdout:6/639: dread dd/f3a [4194304,4194304] 0 2024-04-22T15:19:04.440 INFO:tasks.workunit.client.1.smithi196.stdout:2/649: rename d1/d23/d27/d89/l99 to d1/d23/d27/d2e/d55/lf3 0 2024-04-22T15:19:04.443 INFO:tasks.workunit.client.1.smithi196.stdout:5/642: rmdir d1/d14/d6a/da7 39 2024-04-22T15:19:04.445 INFO:tasks.workunit.client.1.smithi196.stdout:6/640: mknod dd/d2b/d8d/cba 0 2024-04-22T15:19:04.447 INFO:tasks.workunit.client.1.smithi196.stdout:1/589: rename d1/db/dc/l7d to d1/db/dc/de/d96/lbb 0 2024-04-22T15:19:04.448 INFO:tasks.workunit.client.1.smithi196.stdout:8/635: link d2/d8/d47/d68/f7c d2/d35/fb4 0 2024-04-22T15:19:04.451 INFO:tasks.workunit.client.1.smithi196.stdout:2/650: rename d1/d13/d1f/ddf/da1/l94 to d1/d13/dbc/lf4 0 2024-04-22T15:19:04.452 INFO:tasks.workunit.client.1.smithi196.stdout:1/590: mknod d1/db/d16/d1e/d49/cbc 0 2024-04-22T15:19:04.452 INFO:tasks.workunit.client.1.smithi196.stdout:5/643: dread d1/d17/d1c/f52 [0,4194304] 0 2024-04-22T15:19:04.454 INFO:tasks.workunit.client.1.smithi196.stdout:5/644: chown d1/d17/c76 76253456 1 2024-04-22T15:19:04.454 INFO:tasks.workunit.client.1.smithi196.stdout:6/641: creat dd/d2b/d8f/fbb x:0 0 0 2024-04-22T15:19:04.455 INFO:tasks.workunit.client.1.smithi196.stdout:2/651: unlink d1/d13/fc7 0 2024-04-22T15:19:04.456 INFO:tasks.workunit.client.1.smithi196.stdout:4/570: getdents d6/d14/d18/d26/d32/d67/d9a 0 2024-04-22T15:19:04.458 INFO:tasks.workunit.client.1.smithi196.stdout:7/595: getdents d14/d26/d5a/d5f 0 2024-04-22T15:19:04.458 INFO:tasks.workunit.client.1.smithi196.stdout:0/642: sync 2024-04-22T15:19:04.459 INFO:tasks.workunit.client.1.smithi196.stdout:0/643: chown d3/d9 30 1 2024-04-22T15:19:04.461 INFO:tasks.workunit.client.1.smithi196.stdout:9/607: dread db/dd/d1b/d2c/f43 [0,4194304] 0 2024-04-22T15:19:04.463 INFO:tasks.workunit.client.1.smithi196.stdout:2/652: dwrite d1/d13/fe6 [0,4194304] 0 2024-04-22T15:19:04.463 INFO:tasks.workunit.client.1.smithi196.stdout:4/571: mkdir d6/d14/d18/d26/d32/d53/db6 0 2024-04-22T15:19:04.464 INFO:tasks.workunit.client.1.smithi196.stdout:1/591: creat d1/db/d16/d8e/fbd x:0 0 0 2024-04-22T15:19:04.465 INFO:tasks.workunit.client.1.smithi196.stdout:1/592: write d1/db/d16/d8e/fbd [861595,115652] 0 2024-04-22T15:19:04.466 INFO:tasks.workunit.client.1.smithi196.stdout:8/636: truncate d2/d5/f46 4097466 0 2024-04-22T15:19:04.467 INFO:tasks.workunit.client.1.smithi196.stdout:8/637: fsync d2/d8/f73 0 2024-04-22T15:19:04.467 INFO:tasks.workunit.client.1.smithi196.stdout:5/645: mkdir d1/d78/dcf 0 2024-04-22T15:19:04.468 INFO:tasks.workunit.client.1.smithi196.stdout:6/642: mknod dd/d20/cbc 0 2024-04-22T15:19:04.471 INFO:tasks.workunit.client.0.smithi096.stdout:5/857: sync 2024-04-22T15:19:04.471 INFO:tasks.workunit.client.0.smithi096.stdout:0/962: sync 2024-04-22T15:19:04.478 INFO:tasks.workunit.client.0.smithi096.stdout:5/858: unlink d4/ddb/d3d/d35/caf 0 2024-04-22T15:19:04.479 INFO:tasks.workunit.client.0.smithi096.stdout:5/859: chown d4/ddb/d3d/d4d/d59/d98/db9/f120 192764124 1 2024-04-22T15:19:04.486 INFO:tasks.workunit.client.0.smithi096.stdout:5/860: getdents d4/d1f 0 2024-04-22T15:19:04.491 INFO:tasks.workunit.client.0.smithi096.stdout:0/963: dwrite d7/de/d3a/d61/d79/fb9 [0,4194304] 0 2024-04-22T15:19:04.494 INFO:tasks.workunit.client.0.smithi096.stdout:5/861: dwrite d4/ddb/d3d/d4d/d99/fa1 [0,4194304] 0 2024-04-22T15:19:04.500 INFO:tasks.workunit.client.1.smithi196.stdout:4/572: fdatasync d6/f19 0 2024-04-22T15:19:04.501 INFO:tasks.workunit.client.1.smithi196.stdout:0/644: creat d3/d9/d36/d9f/dd1/d65/d7f/d30/fdd x:0 0 0 2024-04-22T15:19:04.504 INFO:tasks.workunit.client.1.smithi196.stdout:8/638: creat d2/d35/fb5 x:0 0 0 2024-04-22T15:19:04.504 INFO:tasks.workunit.client.1.smithi196.stdout:4/573: readlink d6/d13/d15/l59 0 2024-04-22T15:19:04.504 INFO:tasks.workunit.client.0.smithi096.stdout:5/862: dwrite d4/ddb/d3d/d4d/d59/d98/fd1 [0,4194304] 0 2024-04-22T15:19:04.505 INFO:tasks.workunit.client.1.smithi196.stdout:6/643: rmdir dd/d1f/d37 39 2024-04-22T15:19:04.507 INFO:tasks.workunit.client.0.smithi096.stdout:5/863: symlink d4/d1f/l127 0 2024-04-22T15:19:04.508 INFO:tasks.workunit.client.1.smithi196.stdout:5/646: dwrite d1/d4/f22 [4194304,4194304] 0 2024-04-22T15:19:04.509 INFO:tasks.workunit.client.1.smithi196.stdout:1/593: symlink d1/db/dc/de/d5a/d94/dab/lbe 0 2024-04-22T15:19:04.512 INFO:tasks.workunit.client.1.smithi196.stdout:8/639: symlink d2/dc/d45/lb6 0 2024-04-22T15:19:04.514 INFO:tasks.workunit.client.1.smithi196.stdout:9/608: link db/d11/d29/d57/c39 db/dd/de/d15/d31/d64/cd1 0 2024-04-22T15:19:04.517 INFO:tasks.workunit.client.1.smithi196.stdout:7/596: getdents d14/d4a 0 2024-04-22T15:19:04.520 INFO:tasks.workunit.client.1.smithi196.stdout:8/640: fsync d2/d3f/d4b/d79/f8a 0 2024-04-22T15:19:04.522 INFO:tasks.workunit.client.1.smithi196.stdout:4/574: dwrite d6/d13/d15/fb2 [0,4194304] 0 2024-04-22T15:19:04.522 INFO:tasks.workunit.client.1.smithi196.stdout:6/644: dwrite dd/d1f/d4e/d53/d55/f87 [4194304,4194304] 0 2024-04-22T15:19:04.522 INFO:tasks.workunit.client.1.smithi196.stdout:4/575: stat d6/d14 0 2024-04-22T15:19:04.523 INFO:tasks.workunit.client.1.smithi196.stdout:1/594: creat d1/db/dc/de/d43/d5e/fbf x:0 0 0 2024-04-22T15:19:04.526 INFO:tasks.workunit.client.1.smithi196.stdout:0/645: link d3/d9/d36/d9f/dd1/d65/d7f/d30/lac d3/de/d14/lde 0 2024-04-22T15:19:04.530 INFO:tasks.workunit.client.1.smithi196.stdout:4/576: mkdir d6/d14/d18/d26/d32/da3/db7 0 2024-04-22T15:19:04.534 INFO:tasks.workunit.client.1.smithi196.stdout:4/577: truncate d6/d14/d18/d40/fa1 370769 0 2024-04-22T15:19:04.534 INFO:tasks.workunit.client.1.smithi196.stdout:9/609: dwrite db/dd/f88 [0,4194304] 0 2024-04-22T15:19:04.535 INFO:tasks.workunit.client.0.smithi096.stdout:5/864: dwrite d4/f3c [0,4194304] 0 2024-04-22T15:19:04.539 INFO:tasks.workunit.client.1.smithi196.stdout:4/578: symlink d6/d14/d18/d26/lb8 0 2024-04-22T15:19:04.540 INFO:tasks.workunit.client.1.smithi196.stdout:9/610: mknod db/d11/d29/d7a/d7b/d81/cd2 0 2024-04-22T15:19:04.542 INFO:tasks.workunit.client.1.smithi196.stdout:0/646: rename d3/de/l97 to d3/d9/d36/ldf 0 2024-04-22T15:19:04.542 INFO:tasks.workunit.client.1.smithi196.stdout:1/595: getdents d1/db/dc/de/d5a/d5b/d6e 0 2024-04-22T15:19:04.545 INFO:tasks.workunit.client.1.smithi196.stdout:9/611: creat db/d11/d29/d7a/d7b/d81/fd3 x:0 0 0 2024-04-22T15:19:04.546 INFO:tasks.workunit.client.1.smithi196.stdout:0/647: rmdir d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95 39 2024-04-22T15:19:04.549 INFO:tasks.workunit.client.1.smithi196.stdout:3/631: sync 2024-04-22T15:19:04.550 INFO:tasks.workunit.client.1.smithi196.stdout:1/596: rename d1/db/dc/de/f72 to d1/db/fc0 0 2024-04-22T15:19:04.551 INFO:tasks.workunit.client.1.smithi196.stdout:6/645: dread dd/d20/d4d/f65 [0,4194304] 0 2024-04-22T15:19:04.551 INFO:tasks.workunit.client.1.smithi196.stdout:7/597: write d14/f6b [359730,106455] 0 2024-04-22T15:19:04.551 INFO:tasks.workunit.client.1.smithi196.stdout:4/579: dread d6/d13/d15/f5e [0,4194304] 0 2024-04-22T15:19:04.553 INFO:tasks.workunit.client.1.smithi196.stdout:0/648: creat d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dba/fe0 x:0 0 0 2024-04-22T15:19:04.554 INFO:tasks.workunit.client.1.smithi196.stdout:5/647: truncate d1/d17/d1a/d2f/d30/d6e/d7a/fa3 4068340 0 2024-04-22T15:19:04.555 INFO:tasks.workunit.client.1.smithi196.stdout:7/598: mkdir d14/d26/d5a/d5f/dc2 0 2024-04-22T15:19:04.555 INFO:tasks.workunit.client.1.smithi196.stdout:5/648: chown d1/d4/d7b 63916941 1 2024-04-22T15:19:04.556 INFO:tasks.workunit.client.1.smithi196.stdout:4/580: symlink d6/d14/d18/d26/d32/d53/lb9 0 2024-04-22T15:19:04.557 INFO:tasks.workunit.client.1.smithi196.stdout:5/649: write d1/d14/d6a/fa8 [1733276,123015] 0 2024-04-22T15:19:04.559 INFO:tasks.workunit.client.1.smithi196.stdout:3/632: rename d0/d15/d31/d55/dcf/d6f/la4 to d0/d15/d31/d55/dcf/d6f/ld2 0 2024-04-22T15:19:04.560 INFO:tasks.workunit.client.1.smithi196.stdout:6/646: creat dd/d1f/fbd x:0 0 0 2024-04-22T15:19:04.562 INFO:tasks.workunit.client.1.smithi196.stdout:4/581: creat d6/d14/d18/d26/d32/d53/fba x:0 0 0 2024-04-22T15:19:04.562 INFO:tasks.workunit.client.0.smithi096.stdout:5/865: write d4/ddb/d3d/d4d/d59/d98/f114 [1087239,37260] 0 2024-04-22T15:19:04.563 INFO:tasks.workunit.client.1.smithi196.stdout:5/650: creat d1/d17/d1a/d2f/d30/d5e/fd0 x:0 0 0 2024-04-22T15:19:04.565 INFO:tasks.workunit.client.1.smithi196.stdout:0/649: dwrite d3/d9/d36/d9f/dd1/d65/d7f/fd8 [0,4194304] 0 2024-04-22T15:19:04.570 INFO:tasks.workunit.client.0.smithi096.stdout:5/866: chown d4/ddb/d3d/d4d/d59/db0/db2/lc8 0 1 2024-04-22T15:19:04.572 INFO:tasks.workunit.client.1.smithi196.stdout:6/647: unlink dd/d20/c7e 0 2024-04-22T15:19:04.572 INFO:tasks.workunit.client.0.smithi096.stdout:5/867: write d4/ddb/d3d/d4d/d59/d98/dee/d109/f10c [170916,72139] 0 2024-04-22T15:19:04.575 INFO:tasks.workunit.client.1.smithi196.stdout:3/633: creat d0/d15/d31/d55/dcf/d6f/db5/fd3 x:0 0 0 2024-04-22T15:19:04.577 INFO:tasks.workunit.client.1.smithi196.stdout:9/612: dwrite db/d11/f6a [4194304,4194304] 0 2024-04-22T15:19:04.578 INFO:tasks.workunit.client.1.smithi196.stdout:3/634: truncate d0/d15/da7/d4a/d44/d8c/fbc 153791 0 2024-04-22T15:19:04.583 INFO:tasks.workunit.client.1.smithi196.stdout:0/650: mknod d3/db4/dc7/ce1 0 2024-04-22T15:19:04.583 INFO:tasks.workunit.client.0.smithi096.stdout:5/868: dwrite d4/ddb/d3d/d4d/d59/d8a/f119 [0,4194304] 0 2024-04-22T15:19:04.584 INFO:tasks.workunit.client.1.smithi196.stdout:7/599: dread f2 [0,4194304] 0 2024-04-22T15:19:04.586 INFO:tasks.workunit.client.1.smithi196.stdout:4/582: truncate d6/d13/d15/d2c/d62/d78/f7b 983662 0 2024-04-22T15:19:04.588 INFO:tasks.workunit.client.1.smithi196.stdout:1/597: write d1/f78 [3057218,67918] 0 2024-04-22T15:19:04.590 INFO:tasks.workunit.client.1.smithi196.stdout:9/613: symlink db/d11/d29/d57/da7/ld4 0 2024-04-22T15:19:04.594 INFO:tasks.workunit.client.0.smithi096.stdout:5/869: truncate d4/d1f/d8b/fe6 42588 0 2024-04-22T15:19:04.595 INFO:tasks.workunit.client.1.smithi196.stdout:4/583: symlink d6/d14/d18/d26/d32/d53/lbb 0 2024-04-22T15:19:04.596 INFO:tasks.workunit.client.1.smithi196.stdout:1/598: creat d1/db/d16/d1e/d46/fc1 x:0 0 0 2024-04-22T15:19:04.597 INFO:tasks.workunit.client.1.smithi196.stdout:5/651: dread d1/d17/d1a/d2f/d30/f54 [0,4194304] 0 2024-04-22T15:19:04.597 INFO:tasks.workunit.client.1.smithi196.stdout:1/599: fsync d1/db/dc/de/d5a/d5b/f79 0 2024-04-22T15:19:04.600 INFO:tasks.workunit.client.1.smithi196.stdout:4/584: creat d6/d14/d18/d26/d32/da3/fbc x:0 0 0 2024-04-22T15:19:04.602 INFO:tasks.workunit.client.0.smithi096.stdout:5/870: link f1 d4/ddb/d3d/f128 0 2024-04-22T15:19:04.603 INFO:tasks.workunit.client.1.smithi196.stdout:5/652: creat d1/d14/d3e/fd1 x:0 0 0 2024-04-22T15:19:04.603 INFO:tasks.workunit.client.1.smithi196.stdout:5/653: chown d1/d9/la 2629523 1 2024-04-22T15:19:04.604 INFO:tasks.workunit.client.1.smithi196.stdout:1/600: creat d1/db/dc/de/d5a/d94/dab/fc2 x:0 0 0 2024-04-22T15:19:04.604 INFO:tasks.workunit.client.0.smithi096.stdout:5/871: mkdir d4/d2c/d2f/d31/d82/d129 0 2024-04-22T15:19:04.605 INFO:tasks.workunit.client.0.smithi096.stdout:5/872: stat d4/ddb/f7a 0 2024-04-22T15:19:04.607 INFO:tasks.workunit.client.1.smithi196.stdout:1/601: rename d1/db/d16/d6d/cb0 to d1/db/dc/de/cc3 0 2024-04-22T15:19:04.608 INFO:tasks.workunit.client.1.smithi196.stdout:4/585: dread d6/d14/d18/d26/f43 [0,4194304] 0 2024-04-22T15:19:04.609 INFO:tasks.workunit.client.1.smithi196.stdout:1/602: mkdir d1/db/dc/d26/dc4 0 2024-04-22T15:19:04.611 INFO:tasks.workunit.client.1.smithi196.stdout:4/586: truncate d6/d13/d15/d2c/d62/f5a 1034797 0 2024-04-22T15:19:04.613 INFO:tasks.workunit.client.1.smithi196.stdout:1/603: symlink d1/db/dc/de/d5a/d5b/lc5 0 2024-04-22T15:19:04.614 INFO:tasks.workunit.client.1.smithi196.stdout:1/604: readlink d1/db/l56 0 2024-04-22T15:19:04.615 INFO:tasks.workunit.client.1.smithi196.stdout:1/605: chown d1/f86 4 1 2024-04-22T15:19:04.616 INFO:tasks.workunit.client.1.smithi196.stdout:4/587: dwrite d6/f2b [0,4194304] 0 2024-04-22T15:19:04.618 INFO:tasks.workunit.client.1.smithi196.stdout:4/588: dread - d6/d14/f98 zero size 2024-04-22T15:19:04.620 INFO:tasks.workunit.client.1.smithi196.stdout:1/606: rename d1/db/d16/d1e/d49/c63 to d1/db/d16/d1e/d65/cc6 0 2024-04-22T15:19:04.621 INFO:tasks.workunit.client.1.smithi196.stdout:1/607: chown d1/db/dc/de/d96/lb3 24 1 2024-04-22T15:19:04.622 INFO:tasks.workunit.client.1.smithi196.stdout:1/608: chown d1/db/dc/d26/f30 3 1 2024-04-22T15:19:04.622 INFO:tasks.workunit.client.1.smithi196.stdout:1/609: readlink d1/db/dc/de/d52/l91 0 2024-04-22T15:19:04.625 INFO:tasks.workunit.client.1.smithi196.stdout:7/600: write d14/f33 [234731,76167] 0 2024-04-22T15:19:04.626 INFO:tasks.workunit.client.1.smithi196.stdout:4/589: read d6/d14/d18/d40/f65 [969604,124848] 0 2024-04-22T15:19:04.627 INFO:tasks.workunit.client.0.smithi096.stdout:5/873: dread d4/ddb/d3d/d4d/d59/d5a/d5c/f71 [0,4194304] 0 2024-04-22T15:19:04.628 INFO:tasks.workunit.client.1.smithi196.stdout:9/614: write db/dd/d1b/f91 [437743,90600] 0 2024-04-22T15:19:04.629 INFO:tasks.workunit.client.1.smithi196.stdout:5/654: dread d1/d17/d1a/d2f/d30/f6b [0,4194304] 0 2024-04-22T15:19:04.633 INFO:tasks.workunit.client.1.smithi196.stdout:7/601: rename d14/d26/d28/d29/f8b to d14/d26/d2a/d2b/da7/fc3 0 2024-04-22T15:19:04.634 INFO:tasks.workunit.client.1.smithi196.stdout:1/610: dwrite d1/db/dc/de/d96/d9d/f50 [0,4194304] 0 2024-04-22T15:19:04.634 INFO:tasks.workunit.client.1.smithi196.stdout:2/653: sync 2024-04-22T15:19:04.634 INFO:tasks.workunit.client.1.smithi196.stdout:9/615: chown db/dd/de/d15/d31/lc2 41882601 1 2024-04-22T15:19:04.636 INFO:tasks.workunit.client.1.smithi196.stdout:4/590: creat d6/d13/d15/d2c/d62/d78/d8d/fbd x:0 0 0 2024-04-22T15:19:04.636 INFO:tasks.workunit.client.1.smithi196.stdout:2/654: chown d1/f5 395 1 2024-04-22T15:19:04.637 INFO:tasks.workunit.client.1.smithi196.stdout:0/651: dwrite d3/de/f24 [0,4194304] 0 2024-04-22T15:19:04.638 INFO:tasks.workunit.client.1.smithi196.stdout:4/591: chown d6/d14/d18/d26/d32/l76 10847 1 2024-04-22T15:19:04.640 INFO:tasks.workunit.client.1.smithi196.stdout:7/602: creat d14/d15/d91/d9a/fc4 x:0 0 0 2024-04-22T15:19:04.641 INFO:tasks.workunit.client.1.smithi196.stdout:4/592: read d6/f93 [18374,14716] 0 2024-04-22T15:19:04.641 INFO:tasks.workunit.client.1.smithi196.stdout:5/655: creat d1/d78/fd2 x:0 0 0 2024-04-22T15:19:04.648 INFO:tasks.workunit.client.1.smithi196.stdout:7/603: creat d14/d26/d2a/d72/db1/fc5 x:0 0 0 2024-04-22T15:19:04.648 INFO:tasks.workunit.client.1.smithi196.stdout:5/656: creat d1/d2/d98/dac/fd3 x:0 0 0 2024-04-22T15:19:04.667 INFO:tasks.workunit.client.1.smithi196.stdout:6/648: sync 2024-04-22T15:19:04.667 INFO:tasks.workunit.client.1.smithi196.stdout:8/641: sync 2024-04-22T15:19:04.669 INFO:tasks.workunit.client.1.smithi196.stdout:6/649: write dd/d2b/f2e [1146713,72581] 0 2024-04-22T15:19:04.674 INFO:tasks.workunit.client.0.smithi096.stdout:2/890: sync 2024-04-22T15:19:04.674 INFO:tasks.workunit.client.0.smithi096.stdout:6/678: sync 2024-04-22T15:19:04.676 INFO:tasks.workunit.client.1.smithi196.stdout:9/616: rmdir db/dd/de/d15/d31/d64/dbe 0 2024-04-22T15:19:04.676 INFO:tasks.workunit.client.1.smithi196.stdout:9/617: chown f4 0 1 2024-04-22T15:19:04.678 INFO:tasks.workunit.client.0.smithi096.stdout:6/679: creat d3/d28/d4d/d82/dd8/fe8 x:0 0 0 2024-04-22T15:19:04.679 INFO:tasks.workunit.client.1.smithi196.stdout:4/593: creat d6/d13/d15/d2c/d35/fbe x:0 0 0 2024-04-22T15:19:04.682 INFO:tasks.workunit.client.1.smithi196.stdout:7/604: mkdir d14/d26/d2a/d2b/da7/dc6 0 2024-04-22T15:19:04.683 INFO:tasks.workunit.client.1.smithi196.stdout:5/657: creat d1/d14/d3e/fd4 x:0 0 0 2024-04-22T15:19:04.686 INFO:tasks.workunit.client.0.smithi096.stdout:6/680: rename d3/d28/dd/d11/d2c/d6f/dd4/dd6/d51 to d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9 0 2024-04-22T15:19:04.689 INFO:tasks.workunit.client.0.smithi096.stdout:6/681: truncate d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/f60 1886103 0 2024-04-22T15:19:04.690 INFO:tasks.workunit.client.0.smithi096.stdout:2/891: dread d3/db/df/d24/d58/d5f/d6e/fb8 [0,4194304] 0 2024-04-22T15:19:04.691 INFO:tasks.workunit.client.1.smithi196.stdout:8/642: mkdir d2/d5/d1a/db7 0 2024-04-22T15:19:04.691 INFO:tasks.workunit.client.1.smithi196.stdout:8/643: readlink d2/d6a/l8f 0 2024-04-22T15:19:04.692 INFO:tasks.workunit.client.1.smithi196.stdout:0/652: getdents d3/d9/d36/d9f/dd1 0 2024-04-22T15:19:04.693 INFO:tasks.workunit.client.1.smithi196.stdout:9/618: dread db/dd/de/d15/f6e [0,4194304] 0 2024-04-22T15:19:04.696 INFO:tasks.workunit.client.0.smithi096.stdout:2/892: symlink d3/db/df/d24/dc8/d103/d110/l11f 0 2024-04-22T15:19:04.698 INFO:tasks.workunit.client.1.smithi196.stdout:5/658: dread - d1/d17/d1a/d2f/d30/f80 zero size 2024-04-22T15:19:04.700 INFO:tasks.workunit.client.1.smithi196.stdout:8/644: mkdir d2/d8/d47/d68/d81/db8 0 2024-04-22T15:19:04.702 INFO:tasks.workunit.client.0.smithi096.stdout:2/893: rename d3/d13/db4/dbf/fea to d3/db/df/d24/d58/d5f/dfe/d115/d5a/f120 0 2024-04-22T15:19:04.704 INFO:tasks.workunit.client.1.smithi196.stdout:9/619: mknod db/d11/cd5 0 2024-04-22T15:19:04.704 INFO:tasks.workunit.client.0.smithi096.stdout:2/894: write d3/db/df/d10/db5/f113 [57801,10370] 0 2024-04-22T15:19:04.705 INFO:tasks.workunit.client.1.smithi196.stdout:9/620: truncate db/dd/de/d75/faa 898773 0 2024-04-22T15:19:04.707 INFO:tasks.workunit.client.1.smithi196.stdout:0/653: mknod d3/d9/d36/d9f/dd1/d65/d7f/d51/ce2 0 2024-04-22T15:19:04.708 INFO:tasks.workunit.client.0.smithi096.stdout:2/895: mknod d3/d13/db4/c121 0 2024-04-22T15:19:04.710 INFO:tasks.workunit.client.1.smithi196.stdout:7/605: dread d14/d26/f35 [0,4194304] 0 2024-04-22T15:19:04.711 INFO:tasks.workunit.client.1.smithi196.stdout:8/645: symlink d2/dc/d45/lb9 0 2024-04-22T15:19:04.712 INFO:tasks.workunit.client.1.smithi196.stdout:2/655: dwrite d1/f3 [4194304,4194304] 0 2024-04-22T15:19:04.714 INFO:tasks.workunit.client.1.smithi196.stdout:1/611: dwrite d1/d54/d70/f89 [0,4194304] 0 2024-04-22T15:19:04.717 INFO:tasks.workunit.client.0.smithi096.stdout:2/896: dwrite d3/db/df/d24/d58/d5f/dfe/d115/d5a/f120 [0,4194304] 0 2024-04-22T15:19:04.720 INFO:tasks.workunit.client.1.smithi196.stdout:8/646: creat d2/dc/d45/d98/fba x:0 0 0 2024-04-22T15:19:04.724 INFO:tasks.workunit.client.1.smithi196.stdout:2/656: rmdir d1/d23/d27/d2e/d55/d5b/d7c 39 2024-04-22T15:19:04.726 INFO:tasks.workunit.client.0.smithi096.stdout:2/897: mknod d3/db/d10d/dfa/c122 0 2024-04-22T15:19:04.726 INFO:tasks.workunit.client.0.smithi096.stdout:6/682: dwrite d3/d28/f4f [0,4194304] 0 2024-04-22T15:19:04.735 INFO:tasks.workunit.client.1.smithi196.stdout:1/612: mknod d1/db/dc/cc7 0 2024-04-22T15:19:04.735 INFO:tasks.workunit.client.1.smithi196.stdout:1/613: fsync d1/db/d16/d1e/f88 0 2024-04-22T15:19:04.735 INFO:tasks.workunit.client.0.smithi096.stdout:2/898: mkdir d3/db/df/d123 0 2024-04-22T15:19:04.740 INFO:tasks.workunit.client.1.smithi196.stdout:0/654: mkdir d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/dd0/de3 0 2024-04-22T15:19:04.744 INFO:tasks.workunit.client.1.smithi196.stdout:1/614: creat d1/db/dc/de/d5a/d5b/d85/fc8 x:0 0 0 2024-04-22T15:19:04.746 INFO:tasks.workunit.client.1.smithi196.stdout:0/655: mknod d3/d9/d36/d9f/dd1/d65/d7f/d51/ce4 0 2024-04-22T15:19:04.747 INFO:tasks.workunit.client.1.smithi196.stdout:1/615: creat d1/db/dc/dad/d36/fc9 x:0 0 0 2024-04-22T15:19:04.750 INFO:tasks.workunit.client.1.smithi196.stdout:1/616: creat d1/db/dc/de/d5a/d5b/d69/fca x:0 0 0 2024-04-22T15:19:04.751 INFO:tasks.workunit.client.0.smithi096.stdout:6/683: write d3/d28/dd/d11/d2c/d6f/dd4/dd6/fc4 [784912,124510] 0 2024-04-22T15:19:04.752 INFO:tasks.workunit.client.1.smithi196.stdout:9/621: truncate db/dd/f9e 740088 0 2024-04-22T15:19:04.754 INFO:tasks.workunit.client.0.smithi096.stdout:6/684: mknod d3/d28/dd/d11/d2c/d3b/cea 0 2024-04-22T15:19:04.756 INFO:tasks.workunit.client.1.smithi196.stdout:2/657: dwrite d1/f16 [0,4194304] 0 2024-04-22T15:19:04.757 INFO:tasks.workunit.client.1.smithi196.stdout:8/647: truncate d2/f1d 3973757 0 2024-04-22T15:19:04.758 INFO:tasks.workunit.client.1.smithi196.stdout:0/656: dread d3/f47 [4194304,4194304] 0 2024-04-22T15:19:04.758 INFO:tasks.workunit.client.1.smithi196.stdout:2/658: read d1/f10 [1106292,31840] 0 2024-04-22T15:19:04.760 INFO:tasks.workunit.client.1.smithi196.stdout:5/659: dread d1/d2/d98/fb6 [0,4194304] 0 2024-04-22T15:19:04.760 INFO:tasks.workunit.client.0.smithi096.stdout:6/685: creat d3/d28/dc9/feb x:0 0 0 2024-04-22T15:19:04.761 INFO:tasks.workunit.client.1.smithi196.stdout:8/648: mknod d2/dc/d45/d98/cbb 0 2024-04-22T15:19:04.763 INFO:tasks.workunit.client.1.smithi196.stdout:0/657: creat d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/fe5 x:0 0 0 2024-04-22T15:19:04.764 INFO:tasks.workunit.client.1.smithi196.stdout:5/660: chown d1/d4/l8e 577413956 1 2024-04-22T15:19:04.764 INFO:tasks.workunit.client.1.smithi196.stdout:5/661: chown d1/d17/f2e 163878 1 2024-04-22T15:19:04.766 INFO:tasks.workunit.client.0.smithi096.stdout:6/686: creat d3/d28/dc9/fec x:0 0 0 2024-04-22T15:19:04.769 INFO:tasks.workunit.client.1.smithi196.stdout:9/622: dread f4 [0,4194304] 0 2024-04-22T15:19:04.770 INFO:tasks.workunit.client.1.smithi196.stdout:9/623: dread - db/d11/d29/fcb zero size 2024-04-22T15:19:04.770 INFO:tasks.workunit.client.1.smithi196.stdout:8/649: rename d2/d3f/d4b/d79 to d2/d8/d47/d68/dbc 0 2024-04-22T15:19:04.771 INFO:tasks.workunit.client.0.smithi096.stdout:6/687: readlink d3/l2a 0 2024-04-22T15:19:04.771 INFO:tasks.workunit.client.1.smithi196.stdout:5/662: dwrite d1/d14/d3e/f4e [0,4194304] 0 2024-04-22T15:19:04.772 INFO:tasks.workunit.client.1.smithi196.stdout:8/650: chown d2/d6a/f7e 32 1 2024-04-22T15:19:04.772 INFO:tasks.workunit.client.1.smithi196.stdout:7/606: dread d14/d26/d2a/d2b/f77 [0,4194304] 0 2024-04-22T15:19:04.773 INFO:tasks.workunit.client.1.smithi196.stdout:0/658: truncate d3/de/d14/d86/f94 62438 0 2024-04-22T15:19:04.774 INFO:tasks.workunit.client.1.smithi196.stdout:8/651: truncate d2/d3f/d4b/f6f 251337 0 2024-04-22T15:19:04.779 INFO:tasks.workunit.client.1.smithi196.stdout:3/635: sync 2024-04-22T15:19:04.821 INFO:tasks.workunit.client.0.smithi096.stdout:4/835: sync 2024-04-22T15:19:04.822 INFO:tasks.workunit.client.0.smithi096.stdout:4/836: readlink d5/d7/d1f/d84/lbe 0 2024-04-22T15:19:04.828 INFO:tasks.workunit.client.1.smithi196.stdout:6/650: sync 2024-04-22T15:19:04.828 INFO:tasks.workunit.client.1.smithi196.stdout:4/594: sync 2024-04-22T15:19:04.847 INFO:tasks.workunit.client.1.smithi196.stdout:1/617: truncate d1/db/dc/d26/d2e/f3e 3064690 0 2024-04-22T15:19:04.848 INFO:tasks.workunit.client.1.smithi196.stdout:1/618: dread - d1/db/dc/de/d5a/d94/dab/fc2 zero size 2024-04-22T15:19:04.848 INFO:tasks.workunit.client.1.smithi196.stdout:9/624: mknod db/d11/d29/d7a/d7b/d81/cd6 0 2024-04-22T15:19:04.848 INFO:tasks.workunit.client.1.smithi196.stdout:9/625: chown db/dd/dd0 3550736 1 2024-04-22T15:19:04.849 INFO:tasks.workunit.client.1.smithi196.stdout:1/619: write d1/db/dc/de/d43/d5e/d7a/fa0 [768489,36462] 0 2024-04-22T15:19:04.850 INFO:tasks.workunit.client.1.smithi196.stdout:5/663: readlink d1/d9/l72 0 2024-04-22T15:19:04.852 INFO:tasks.workunit.client.1.smithi196.stdout:7/607: unlink d14/d25/d31/fae 0 2024-04-22T15:19:04.854 INFO:tasks.workunit.client.0.smithi096.stdout:4/837: creat d5/d7/d1f/d84/d6d/f125 x:0 0 0 2024-04-22T15:19:04.854 INFO:tasks.workunit.client.1.smithi196.stdout:2/659: getdents d1/d13/dbc 0 2024-04-22T15:19:04.855 INFO:tasks.workunit.client.1.smithi196.stdout:2/660: fdatasync d1/d23/d27/fec 0 2024-04-22T15:19:04.857 INFO:tasks.workunit.client.1.smithi196.stdout:8/652: chown d2/d35/f92 1980448 1 2024-04-22T15:19:04.861 INFO:tasks.workunit.client.1.smithi196.stdout:3/636: creat d0/d15/d31/d55/dcf/d8/fd4 x:0 0 0 2024-04-22T15:19:04.862 INFO:tasks.workunit.client.0.smithi096.stdout:4/838: mknod d5/d7/d2c/d67/d11d/c126 0 2024-04-22T15:19:04.865 INFO:tasks.workunit.client.1.smithi196.stdout:6/651: fsync dd/d1f/d38/d40/f43 0 2024-04-22T15:19:04.866 INFO:tasks.workunit.client.1.smithi196.stdout:4/595: truncate d6/d10/f37 913874 0 2024-04-22T15:19:04.868 INFO:tasks.workunit.client.0.smithi096.stdout:6/688: truncate d3/d28/dd/f16 2146036 0 2024-04-22T15:19:04.869 INFO:tasks.workunit.client.1.smithi196.stdout:9/626: mkdir db/dd/de/dd7 0 2024-04-22T15:19:04.871 INFO:tasks.workunit.client.0.smithi096.stdout:6/689: mknod d3/d28/dd/d11/d2c/d3b/ced 0 2024-04-22T15:19:04.872 INFO:tasks.workunit.client.0.smithi096.stdout:6/690: readlink d3/d28/d54/d68/d6e/le7 0 2024-04-22T15:19:04.873 INFO:tasks.workunit.client.1.smithi196.stdout:7/608: mkdir d14/d26/d2a/d2b/dc7 0 2024-04-22T15:19:04.873 INFO:tasks.workunit.client.1.smithi196.stdout:7/609: stat d14/d26/d5e/ca9 0 2024-04-22T15:19:04.876 INFO:tasks.workunit.client.0.smithi096.stdout:6/691: fdatasync d3/d28/dd/d11/f18 0 2024-04-22T15:19:04.877 INFO:tasks.workunit.client.0.smithi096.stdout:6/692: dread - d3/d28/d4d/d82/dd8/fe8 zero size 2024-04-22T15:19:04.878 INFO:tasks.workunit.client.1.smithi196.stdout:0/659: creat d3/d6/dc/d8c/d8d/d98/dbd/fe6 x:0 0 0 2024-04-22T15:19:04.881 INFO:tasks.workunit.client.0.smithi096.stdout:4/839: dread d5/d7/d2c/d3d/d4f/fb6 [0,4194304] 0 2024-04-22T15:19:04.884 INFO:tasks.workunit.client.1.smithi196.stdout:3/637: fdatasync d0/d15/d31/d55/d8b/fc0 0 2024-04-22T15:19:04.885 INFO:tasks.workunit.client.1.smithi196.stdout:3/638: write d0/d15/da7/d4a/f50 [4998828,61089] 0 2024-04-22T15:19:04.886 INFO:tasks.workunit.client.0.smithi096.stdout:4/840: mkdir d5/d7/d1f/d127 0 2024-04-22T15:19:04.890 INFO:tasks.workunit.client.1.smithi196.stdout:9/627: truncate f5 2396562 0 2024-04-22T15:19:04.891 INFO:tasks.workunit.client.1.smithi196.stdout:1/620: mkdir d1/db/dc/de/d5a/dcb 0 2024-04-22T15:19:04.891 INFO:tasks.workunit.client.1.smithi196.stdout:1/621: readlink d1/db/l56 0 2024-04-22T15:19:04.891 INFO:tasks.workunit.client.0.smithi096.stdout:6/693: getdents d3/d28/dd/d11/d2c/d6f/dd4 0 2024-04-22T15:19:04.893 INFO:tasks.workunit.client.1.smithi196.stdout:5/664: rename d1/d17/d50/l9f to d1/d14/d3e/ld5 0 2024-04-22T15:19:04.894 INFO:tasks.workunit.client.1.smithi196.stdout:7/610: mkdir d14/d4d/da3/dc8 0 2024-04-22T15:19:04.895 INFO:tasks.workunit.client.0.smithi096.stdout:6/694: mknod d3/d3a/d75/cee 0 2024-04-22T15:19:04.897 INFO:tasks.workunit.client.1.smithi196.stdout:1/622: dwrite d1/db/d16/d1e/d49/d90/fb4 [0,4194304] 0 2024-04-22T15:19:04.897 INFO:tasks.workunit.client.1.smithi196.stdout:7/611: dread d14/d26/d5a/dbb/f8d [0,4194304] 0 2024-04-22T15:19:04.898 INFO:tasks.workunit.client.1.smithi196.stdout:8/653: symlink d2/lbd 0 2024-04-22T15:19:04.901 INFO:tasks.workunit.client.0.smithi096.stdout:6/695: dwrite d3/d28/d54/d68/d6e/fdf [0,4194304] 0 2024-04-22T15:19:04.903 INFO:tasks.workunit.client.0.smithi096.stdout:6/696: chown d3/d28/d54/d68/d6e/fdf 57 1 2024-04-22T15:19:04.904 INFO:tasks.workunit.client.1.smithi196.stdout:6/652: mknod dd/d1f/cbe 0 2024-04-22T15:19:04.904 INFO:tasks.workunit.client.1.smithi196.stdout:6/653: readlink dd/d2b/l59 0 2024-04-22T15:19:04.909 INFO:tasks.workunit.client.0.smithi096.stdout:6/697: creat d3/d28/dd/de0/fef x:0 0 0 2024-04-22T15:19:04.912 INFO:tasks.workunit.client.1.smithi196.stdout:5/665: mknod d1/d17/d50/cd6 0 2024-04-22T15:19:04.914 INFO:tasks.workunit.client.0.smithi096.stdout:6/698: rmdir d3/d28/dd/d11/d2c 39 2024-04-22T15:19:04.933 INFO:tasks.workunit.client.1.smithi196.stdout:5/666: dread d1/d17/d1a/d2f/d30/d5e/f63 [0,4194304] 0 2024-04-22T15:19:05.031 INFO:tasks.workunit.client.1.smithi196.stdout:6/654: creat dd/d2b/d8f/fbf x:0 0 0 2024-04-22T15:19:05.032 INFO:tasks.workunit.client.1.smithi196.stdout:4/596: link d6/d90/f60 d6/d14/d18/d40/d85/fbf 0 2024-04-22T15:19:05.034 INFO:tasks.workunit.client.0.smithi096.stdout:9/777: sync 2024-04-22T15:19:05.034 INFO:tasks.workunit.client.0.smithi096.stdout:8/854: sync 2024-04-22T15:19:05.035 INFO:tasks.workunit.client.0.smithi096.stdout:6/699: link d3/l45 d3/d28/dd/d11/lf0 0 2024-04-22T15:19:05.037 INFO:tasks.workunit.client.1.smithi196.stdout:6/655: dwrite dd/d1f/d4e/d53/d55/f9a [0,4194304] 0 2024-04-22T15:19:05.040 INFO:tasks.workunit.client.1.smithi196.stdout:6/656: write dd/d2b/d8d/f93 [624124,18175] 0 2024-04-22T15:19:05.041 INFO:tasks.workunit.client.1.smithi196.stdout:2/661: getdents d1/d13 0 2024-04-22T15:19:05.044 INFO:tasks.workunit.client.1.smithi196.stdout:7/612: symlink d14/d26/db7/lc9 0 2024-04-22T15:19:05.044 INFO:tasks.workunit.client.0.smithi096.stdout:9/778: dread d1/df/d95/d9d/db5/d43/d57/d5b/f66 [0,4194304] 0 2024-04-22T15:19:05.045 INFO:tasks.workunit.client.1.smithi196.stdout:7/613: stat d14/d26/f35 0 2024-04-22T15:19:05.045 INFO:tasks.workunit.client.1.smithi196.stdout:7/614: chown d14/d4a/f8c 20175 1 2024-04-22T15:19:05.046 INFO:tasks.workunit.client.1.smithi196.stdout:1/623: write d1/db/dc/d26/f2b [3372285,60588] 0 2024-04-22T15:19:05.052 INFO:tasks.workunit.client.0.smithi096.stdout:9/779: rename d1/d25/f48 to d1/d25/d9f/f104 0 2024-04-22T15:19:05.056 INFO:tasks.workunit.client.1.smithi196.stdout:5/667: creat d1/d17/d1a/d2f/d30/d7d/fd7 x:0 0 0 2024-04-22T15:19:05.056 INFO:tasks.workunit.client.1.smithi196.stdout:8/654: dwrite d2/d6a/d7b/fb1 [0,4194304] 0 2024-04-22T15:19:05.058 INFO:tasks.workunit.client.0.smithi096.stdout:6/700: dread d3/d28/d4d/d82/fd1 [0,4194304] 0 2024-04-22T15:19:05.066 INFO:tasks.workunit.client.0.smithi096.stdout:9/780: rename d1/df/d95/d9d/db5/dd/fe1 to d1/df/d95/d9d/db5/dd/dba/f105 0 2024-04-22T15:19:05.067 INFO:tasks.workunit.client.0.smithi096.stdout:9/781: write d1/d25/f7d [1880122,1159] 0 2024-04-22T15:19:05.068 INFO:tasks.workunit.client.1.smithi196.stdout:4/597: symlink d6/d14/d18/d26/lc0 0 2024-04-22T15:19:05.069 INFO:tasks.workunit.client.1.smithi196.stdout:4/598: chown d6/d13/d15/d2c/d62/d78/d8b/cab 139212385 1 2024-04-22T15:19:05.071 INFO:tasks.workunit.client.1.smithi196.stdout:9/628: creat db/d11/d29/fd8 x:0 0 0 2024-04-22T15:19:05.072 INFO:tasks.workunit.client.1.smithi196.stdout:4/599: read d6/d14/d18/d26/d32/d53/f54 [119900,115028] 0 2024-04-22T15:19:05.073 INFO:tasks.workunit.client.1.smithi196.stdout:4/600: stat d6/d14/d18/d26/d32/d53/la2 0 2024-04-22T15:19:05.075 INFO:tasks.workunit.client.1.smithi196.stdout:4/601: truncate d6/d14/d18/d26/d32/da3/fbc 1011424 0 2024-04-22T15:19:05.075 INFO:tasks.workunit.client.1.smithi196.stdout:4/602: stat d6/d14/d18/d26/d32/da3/fbc 0 2024-04-22T15:19:05.076 INFO:tasks.workunit.client.1.smithi196.stdout:4/603: chown d6/d13/d15/d2c/d62/d78/c86 1 1 2024-04-22T15:19:05.080 INFO:tasks.workunit.client.1.smithi196.stdout:6/657: truncate dd/d1f/d4e/f73 1578979 0 2024-04-22T15:19:05.086 INFO:tasks.workunit.client.1.smithi196.stdout:7/615: mknod d14/d26/d2a/d72/db1/cca 0 2024-04-22T15:19:05.088 INFO:tasks.workunit.client.1.smithi196.stdout:1/624: readlink d1/db/d16/d1e/d49/d7f/l8b 0 2024-04-22T15:19:05.092 INFO:tasks.workunit.client.1.smithi196.stdout:3/639: getdents d0/d15/d31/d55/d5d 0 2024-04-22T15:19:05.094 INFO:tasks.workunit.client.1.smithi196.stdout:8/655: rmdir d2/d3f/d4b 39 2024-04-22T15:19:05.096 INFO:tasks.workunit.client.1.smithi196.stdout:9/629: symlink db/dd/de/d15/ld9 0 2024-04-22T15:19:05.097 INFO:tasks.workunit.client.1.smithi196.stdout:4/604: mknod d6/d14/d18/d40/d85/cc1 0 2024-04-22T15:19:05.099 INFO:tasks.workunit.client.0.smithi096.stdout:1/899: sync 2024-04-22T15:19:05.099 INFO:tasks.workunit.client.0.smithi096.stdout:5/874: sync 2024-04-22T15:19:05.103 INFO:tasks.workunit.client.1.smithi196.stdout:6/658: dwrite dd/d1f/d37/d7b/fb4 [0,4194304] 0 2024-04-22T15:19:05.105 INFO:tasks.workunit.client.0.smithi096.stdout:1/900: symlink d6/dc/d8a/d3f/d62/dc2/dd0/l11c 0 2024-04-22T15:19:05.109 INFO:tasks.workunit.client.1.smithi196.stdout:8/656: creat d2/d5/d2e/fbe x:0 0 0 2024-04-22T15:19:05.111 INFO:tasks.workunit.client.0.smithi096.stdout:1/901: mknod d6/dc/d8a/d3f/d62/d7d/d8e/c11d 0 2024-04-22T15:19:05.118 INFO:tasks.workunit.client.1.smithi196.stdout:0/660: getdents d3 0 2024-04-22T15:19:05.118 INFO:tasks.workunit.client.1.smithi196.stdout:3/640: mknod d0/d15/da7/d4a/dc5/cd5 0 2024-04-22T15:19:05.122 INFO:tasks.workunit.client.1.smithi196.stdout:4/605: symlink d6/d13/d15/d2c/d8c/db3/lc2 0 2024-04-22T15:19:05.124 INFO:tasks.workunit.client.1.smithi196.stdout:6/659: mkdir dd/d1f/d4e/d53/d55/d57/dc0 0 2024-04-22T15:19:05.127 INFO:tasks.workunit.client.1.smithi196.stdout:0/661: link d3/d6/f48 d3/d9/d36/d9f/dd1/d65/d7f/d30/fe7 0 2024-04-22T15:19:05.128 INFO:tasks.workunit.client.1.smithi196.stdout:3/641: creat d0/d15/d31/d55/dcf/d8/d6b/fd6 x:0 0 0 2024-04-22T15:19:05.130 INFO:tasks.workunit.client.1.smithi196.stdout:4/606: fsync d6/d14/d18/d40/d85/fbf 0 2024-04-22T15:19:05.132 INFO:tasks.workunit.client.1.smithi196.stdout:6/660: mknod dd/d1f/d4e/d53/d55/cc1 0 2024-04-22T15:19:05.134 INFO:tasks.workunit.client.1.smithi196.stdout:3/642: dread d0/d15/d31/d55/dcf/d8/f77 [0,4194304] 0 2024-04-22T15:19:05.134 INFO:tasks.workunit.client.1.smithi196.stdout:4/607: truncate d6/d14/f12 4885850 0 2024-04-22T15:19:05.135 INFO:tasks.workunit.client.1.smithi196.stdout:0/662: mkdir d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8 0 2024-04-22T15:19:05.136 INFO:tasks.workunit.client.1.smithi196.stdout:0/663: chown d3/d6 543059 1 2024-04-22T15:19:05.137 INFO:tasks.workunit.client.1.smithi196.stdout:6/661: unlink dd/d1f/d37/d3c/d89/fa9 0 2024-04-22T15:19:05.142 INFO:tasks.workunit.client.1.smithi196.stdout:3/643: mkdir d0/d15/d31/d55/dcf/d6f/db5/dd7 0 2024-04-22T15:19:05.144 INFO:tasks.workunit.client.1.smithi196.stdout:0/664: creat d3/db4/fe9 x:0 0 0 2024-04-22T15:19:05.145 INFO:tasks.workunit.client.1.smithi196.stdout:0/665: fsync d3/d6/dc/d8c/daf/f7c 0 2024-04-22T15:19:05.146 INFO:tasks.workunit.client.1.smithi196.stdout:0/666: write d3/d9/f54 [2354819,125000] 0 2024-04-22T15:19:05.146 INFO:tasks.workunit.client.1.smithi196.stdout:4/608: symlink d6/d14/d18/d26/d32/d53/db6/lc3 0 2024-04-22T15:19:05.153 INFO:tasks.workunit.client.1.smithi196.stdout:0/667: mkdir d3/de/d14/d86/dea 0 2024-04-22T15:19:05.156 INFO:tasks.workunit.client.1.smithi196.stdout:6/662: dread dd/d1f/d4e/d53/d55/f87 [0,4194304] 0 2024-04-22T15:19:05.160 INFO:tasks.workunit.client.1.smithi196.stdout:4/609: dwrite d6/d13/d15/d2c/d62/d78/d8d/f9c [0,4194304] 0 2024-04-22T15:19:05.160 INFO:tasks.workunit.client.1.smithi196.stdout:3/644: dread d0/d15/d31/d55/dcf/f7d [0,4194304] 0 2024-04-22T15:19:05.161 INFO:tasks.workunit.client.1.smithi196.stdout:4/610: chown d6/d14/d18/d26/d32/d67/d9a/da5 3645 1 2024-04-22T15:19:05.164 INFO:tasks.workunit.client.1.smithi196.stdout:4/611: write d6/d14/d18/d26/d32/da3/fbc [1951731,92784] 0 2024-04-22T15:19:05.165 INFO:tasks.workunit.client.0.smithi096.stdout:0/964: sync 2024-04-22T15:19:05.165 INFO:tasks.workunit.client.0.smithi096.stdout:2/899: sync 2024-04-22T15:19:05.165 INFO:tasks.workunit.client.0.smithi096.stdout:3/886: sync 2024-04-22T15:19:05.165 INFO:tasks.workunit.client.0.smithi096.stdout:4/841: sync 2024-04-22T15:19:05.166 INFO:tasks.workunit.client.1.smithi196.stdout:4/612: dread d6/d13/d15/d2c/d62/f96 [0,4194304] 0 2024-04-22T15:19:05.179 INFO:tasks.workunit.client.1.smithi196.stdout:0/668: dread d3/fd [0,4194304] 0 2024-04-22T15:19:05.180 INFO:tasks.workunit.client.1.smithi196.stdout:0/669: write d3/d9/d36/d9f/dd1/f7b [4901826,33737] 0 2024-04-22T15:19:05.198 INFO:tasks.workunit.client.0.smithi096.stdout:3/887: link da/db/de/ddc/f88 da/db/dc/d28/f121 0 2024-04-22T15:19:05.200 INFO:tasks.workunit.client.0.smithi096.stdout:3/888: read da/db/d12/d9b/fd6 [2928272,15774] 0 2024-04-22T15:19:05.202 INFO:tasks.workunit.client.1.smithi196.stdout:2/662: mkdir d1/d23/df5 0 2024-04-22T15:19:05.203 INFO:tasks.workunit.client.0.smithi096.stdout:3/889: symlink da/l122 0 2024-04-22T15:19:05.219 INFO:tasks.workunit.client.1.smithi196.stdout:6/663: write dd/d20/f7d [336980,35836] 0 2024-04-22T15:19:05.221 INFO:tasks.workunit.client.1.smithi196.stdout:3/645: mkdir d0/d15/da7/d4a/d44/d96/dd8 0 2024-04-22T15:19:05.226 INFO:tasks.workunit.client.1.smithi196.stdout:4/613: dread d6/d90/f4a [0,4194304] 0 2024-04-22T15:19:05.228 INFO:tasks.workunit.client.0.smithi096.stdout:6/701: creat d3/d28/d54/d68/ff1 x:0 0 0 2024-04-22T15:19:05.229 INFO:tasks.workunit.client.0.smithi096.stdout:6/702: chown d3/d28/dd/f25 4234 1 2024-04-22T15:19:05.229 INFO:tasks.workunit.client.1.smithi196.stdout:0/670: symlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9/leb 0 2024-04-22T15:19:05.231 INFO:tasks.workunit.client.1.smithi196.stdout:2/663: mknod d1/d23/d27/d73/cf6 0 2024-04-22T15:19:05.237 INFO:tasks.workunit.client.1.smithi196.stdout:2/664: dwrite d1/d13/d25/f4f [0,4194304] 0 2024-04-22T15:19:05.240 INFO:tasks.workunit.client.0.smithi096.stdout:8/855: write d4/d10/d4d/f74 [2845298,77272] 0 2024-04-22T15:19:05.241 INFO:tasks.workunit.client.1.smithi196.stdout:2/665: dread d1/d23/d27/d2e/d55/f87 [0,4194304] 0 2024-04-22T15:19:05.248 INFO:tasks.workunit.client.1.smithi196.stdout:0/671: dread d3/de/d1c/d27/f93 [0,4194304] 0 2024-04-22T15:19:05.256 INFO:tasks.workunit.client.1.smithi196.stdout:3/646: mknod d0/d15/d4d/db/cd9 0 2024-04-22T15:19:05.259 INFO:tasks.workunit.client.1.smithi196.stdout:4/614: dread d6/d14/f33 [0,4194304] 0 2024-04-22T15:19:05.261 INFO:tasks.workunit.client.1.smithi196.stdout:2/666: write d1/d23/d27/d73/fc6 [300112,32681] 0 2024-04-22T15:19:05.265 INFO:tasks.workunit.client.1.smithi196.stdout:0/672: creat d3/de/d1c/d27/ddc/fec x:0 0 0 2024-04-22T15:19:05.266 INFO:tasks.workunit.client.1.smithi196.stdout:0/673: dread - d3/d6/dc/d8c/d8d/d98/dbd/fe6 zero size 2024-04-22T15:19:05.266 INFO:tasks.workunit.client.1.smithi196.stdout:0/674: write d3/db4/fe9 [695418,26807] 0 2024-04-22T15:19:05.269 INFO:tasks.workunit.client.1.smithi196.stdout:1/625: rmdir d1 39 2024-04-22T15:19:05.272 INFO:tasks.workunit.client.1.smithi196.stdout:0/675: mkdir d3/de/d14/ded 0 2024-04-22T15:19:05.274 INFO:tasks.workunit.client.1.smithi196.stdout:1/626: write d1/db/d16/f1d [2066596,9166] 0 2024-04-22T15:19:05.278 INFO:tasks.workunit.client.1.smithi196.stdout:4/615: link d6/d10/c4b d6/d14/d18/d26/d32/d67/cc4 0 2024-04-22T15:19:05.280 INFO:tasks.workunit.client.0.smithi096.stdout:4/842: mknod d5/d7/c128 0 2024-04-22T15:19:05.285 INFO:tasks.workunit.client.1.smithi196.stdout:4/616: dread d6/d13/d15/d2c/d62/f88 [0,4194304] 0 2024-04-22T15:19:05.287 INFO:tasks.workunit.client.0.smithi096.stdout:6/703: fdatasync d3/d28/dd/d11/d2c/f5e 0 2024-04-22T15:19:05.291 INFO:tasks.workunit.client.1.smithi196.stdout:4/617: dwrite d6/d14/d18/d26/d32/d53/fba [0,4194304] 0 2024-04-22T15:19:05.291 INFO:tasks.workunit.client.0.smithi096.stdout:4/843: truncate d5/d7/d1a/f38 4563312 0 2024-04-22T15:19:05.294 INFO:tasks.workunit.client.0.smithi096.stdout:6/704: chown d3/d28/ca 11 1 2024-04-22T15:19:05.297 INFO:tasks.workunit.client.1.smithi196.stdout:4/618: symlink d6/d14/d18/d26/d32/d67/d9a/da5/lc5 0 2024-04-22T15:19:05.298 INFO:tasks.workunit.client.0.smithi096.stdout:9/782: rename d1/d25/c60 to d1/df/d95/d9d/db5/c106 0 2024-04-22T15:19:05.299 INFO:tasks.workunit.client.0.smithi096.stdout:4/844: read - d5/d7/d2c/d3d/d4f/d9c/ffb zero size 2024-04-22T15:19:05.300 INFO:tasks.workunit.client.0.smithi096.stdout:4/845: stat d5/d7/d1a/d25/c3c 0 2024-04-22T15:19:05.302 INFO:tasks.workunit.client.0.smithi096.stdout:4/846: chown d5/d7/d1f/d84/d6d/fdb 0 1 2024-04-22T15:19:05.304 INFO:tasks.workunit.client.0.smithi096.stdout:9/783: readlink d1/d4/d79/lb6 0 2024-04-22T15:19:05.306 INFO:tasks.workunit.client.0.smithi096.stdout:6/705: fdatasync d3/d28/dd/f16 0 2024-04-22T15:19:05.309 INFO:tasks.workunit.client.0.smithi096.stdout:4/847: symlink d5/d7/d2c/d3d/d4f/d9c/l129 0 2024-04-22T15:19:05.311 INFO:tasks.workunit.client.0.smithi096.stdout:6/706: dread d3/d28/d54/d68/d6e/dbb/fbc [4194304,4194304] 0 2024-04-22T15:19:05.312 INFO:tasks.workunit.client.0.smithi096.stdout:5/875: rename d4/d2c/d2f/d30/fd6 to d4/f12a 0 2024-04-22T15:19:05.314 INFO:tasks.workunit.client.0.smithi096.stdout:6/707: dread d3/d28/d4d/d82/d9e/fcc [0,4194304] 0 2024-04-22T15:19:05.319 INFO:tasks.workunit.client.0.smithi096.stdout:5/876: dwrite d4/ddb/d3d/d4d/f70 [0,4194304] 0 2024-04-22T15:19:05.321 INFO:tasks.workunit.client.0.smithi096.stdout:6/708: creat d3/d3a/d75/d83/ff2 x:0 0 0 2024-04-22T15:19:05.322 INFO:tasks.workunit.client.1.smithi196.stdout:5/668: truncate d1/d17/d1c/f52 701288 0 2024-04-22T15:19:05.323 INFO:tasks.workunit.client.1.smithi196.stdout:5/669: chown d1/d17/d1a/d2f/d30/d5e/c8c 689900565 1 2024-04-22T15:19:05.324 INFO:tasks.workunit.client.1.smithi196.stdout:5/670: write d1/d4/f9d [1271467,24127] 0 2024-04-22T15:19:05.324 INFO:tasks.workunit.client.0.smithi096.stdout:6/709: write d3/d28/dd/d11/d2c/d6f/dd4/dd6/fc4 [869747,59127] 0 2024-04-22T15:19:05.325 INFO:tasks.workunit.client.1.smithi196.stdout:5/671: dread - d1/d17/d1a/d2f/d30/d5e/fd0 zero size 2024-04-22T15:19:05.325 INFO:tasks.workunit.client.0.smithi096.stdout:6/710: readlink d3/d28/d54/d68/d6e/le7 0 2024-04-22T15:19:05.329 INFO:tasks.workunit.client.0.smithi096.stdout:5/877: unlink d4/ddb/d3d/d4f/fe4 0 2024-04-22T15:19:05.330 INFO:tasks.workunit.client.0.smithi096.stdout:5/878: chown f1 165 1 2024-04-22T15:19:05.333 INFO:tasks.workunit.client.0.smithi096.stdout:6/711: dread - d3/d28/dd/fba zero size 2024-04-22T15:19:05.335 INFO:tasks.workunit.client.1.smithi196.stdout:9/630: dwrite db/d11/d29/d7a/f7d [0,4194304] 0 2024-04-22T15:19:05.337 INFO:tasks.workunit.client.0.smithi096.stdout:1/902: rename d6/dc/f23 to d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/f11e 0 2024-04-22T15:19:05.341 INFO:tasks.workunit.client.1.smithi196.stdout:9/631: write db/dd/de/d15/d31/fb7 [4628694,100714] 0 2024-04-22T15:19:05.341 INFO:tasks.workunit.client.0.smithi096.stdout:5/879: rename d4/d2c/d2f/c123 to d4/d2c/d2f/d30/c12b 0 2024-04-22T15:19:05.344 INFO:tasks.workunit.client.0.smithi096.stdout:6/712: mknod d3/d28/d4d/d82/d9e/cf3 0 2024-04-22T15:19:05.347 INFO:tasks.workunit.client.0.smithi096.stdout:1/903: unlink d6/dc/d8a/d3f/d54/d32/d3b/d94/f96 0 2024-04-22T15:19:05.350 INFO:tasks.workunit.client.0.smithi096.stdout:6/713: rename d3/d3a/lcd to d3/d28/d54/d68/d6e/lf4 0 2024-04-22T15:19:05.354 INFO:tasks.workunit.client.0.smithi096.stdout:1/904: read d6/dc/d8a/d3f/d54/d32/d3b/f6f [8071317,55652] 0 2024-04-22T15:19:05.355 INFO:tasks.workunit.client.0.smithi096.stdout:1/905: chown d6/dc/d8a/d3f/d54/d32/d3b/d94/fb3 21199 1 2024-04-22T15:19:05.356 INFO:tasks.workunit.client.0.smithi096.stdout:5/880: creat d4/d1f/f12c x:0 0 0 2024-04-22T15:19:05.357 INFO:tasks.workunit.client.0.smithi096.stdout:1/906: stat d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/dc3/l117 0 2024-04-22T15:19:05.361 INFO:tasks.workunit.client.0.smithi096.stdout:5/881: mkdir d4/ddb/d3d/d35/d57/dbe/d113/d12d 0 2024-04-22T15:19:05.363 INFO:tasks.workunit.client.0.smithi096.stdout:5/882: mkdir d4/ddb/d3d/d35/d57/d12e 0 2024-04-22T15:19:05.365 INFO:tasks.workunit.client.0.smithi096.stdout:5/883: rmdir d4/d1f/d80 39 2024-04-22T15:19:05.375 INFO:tasks.workunit.client.1.smithi196.stdout:4/619: symlink d6/d14/lc6 0 2024-04-22T15:19:05.376 INFO:tasks.workunit.client.0.smithi096.stdout:0/965: write d7/de/d5b/d95/d15/f13f [1282322,82207] 0 2024-04-22T15:19:05.380 INFO:tasks.workunit.client.0.smithi096.stdout:0/966: rename d7/de/d3a/d61/d79/dfa to d7/d42/d4c/d55/d149 0 2024-04-22T15:19:05.388 INFO:tasks.workunit.client.0.smithi096.stdout:3/890: unlink da/db/dc/d8d/ld8 0 2024-04-22T15:19:05.390 INFO:tasks.workunit.client.1.smithi196.stdout:6/664: rename dd/d1f/d37/f74 to dd/d20/d5c/fc2 0 2024-04-22T15:19:05.391 INFO:tasks.workunit.client.1.smithi196.stdout:6/665: fsync dd/d20/f7d 0 2024-04-22T15:19:05.393 INFO:tasks.workunit.client.0.smithi096.stdout:0/967: dread d7/de/d5b/d95/f24 [0,4194304] 0 2024-04-22T15:19:05.394 INFO:tasks.workunit.client.1.smithi196.stdout:3/647: rename d0/d15/d4d/f3e to d0/d15/d31/d55/dcf/d6f/db5/dd7/fda 0 2024-04-22T15:19:05.395 INFO:tasks.workunit.client.1.smithi196.stdout:6/666: dread dd/d1f/d4e/d53/d55/f9a [0,4194304] 0 2024-04-22T15:19:05.398 INFO:tasks.workunit.client.0.smithi096.stdout:3/891: getdents da/db/de/d19/d2d 0 2024-04-22T15:19:05.400 INFO:tasks.workunit.client.0.smithi096.stdout:3/892: truncate da/db/d9a/f109 496450 0 2024-04-22T15:19:05.401 INFO:tasks.workunit.client.0.smithi096.stdout:0/968: symlink d7/de/l14a 0 2024-04-22T15:19:05.401 INFO:tasks.workunit.client.0.smithi096.stdout:0/969: readlink d7/d42/d4c/lc5 0 2024-04-22T15:19:05.401 INFO:tasks.workunit.client.1.smithi196.stdout:3/648: mknod d0/d15/da7/d18/cdb 0 2024-04-22T15:19:05.405 INFO:tasks.workunit.client.1.smithi196.stdout:6/667: symlink dd/d20/lc3 0 2024-04-22T15:19:05.406 INFO:tasks.workunit.client.1.smithi196.stdout:6/668: fdatasync dd/d1f/d37/d3c/d96/fb3 0 2024-04-22T15:19:05.407 INFO:tasks.workunit.client.0.smithi096.stdout:0/970: fsync d7/de/d5b/d95/dc/f56 0 2024-04-22T15:19:05.408 INFO:tasks.workunit.client.1.smithi196.stdout:3/649: truncate d0/d15/d31/d55/dcf/d8/d70/f97 843461 0 2024-04-22T15:19:05.409 INFO:tasks.workunit.client.0.smithi096.stdout:3/893: creat da/db/de/d19/f123 x:0 0 0 2024-04-22T15:19:05.411 INFO:tasks.workunit.client.1.smithi196.stdout:0/676: rename d3/db4/dc7/cd9 to d3/d9/d36/d9f/dd1/d65/d7f/cee 0 2024-04-22T15:19:05.412 INFO:tasks.workunit.client.1.smithi196.stdout:0/677: chown d3/d6/dc/f58 2753558 1 2024-04-22T15:19:05.413 INFO:tasks.workunit.client.1.smithi196.stdout:0/678: read - d3/d6/dc/d8c/d8d/d98/fa6 zero size 2024-04-22T15:19:05.413 INFO:tasks.workunit.client.1.smithi196.stdout:0/679: chown d3/d9/d36/d9f/dd1/d65/d7f/d30 8747517 1 2024-04-22T15:19:05.417 INFO:tasks.workunit.client.1.smithi196.stdout:1/627: rename d1/db/dc/de/d5a/dcb to d1/db/d16/d44/dcc 0 2024-04-22T15:19:05.417 INFO:tasks.workunit.client.1.smithi196.stdout:5/672: mknod d1/d4/cd8 0 2024-04-22T15:19:05.421 INFO:tasks.workunit.client.1.smithi196.stdout:0/680: symlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9/lef 0 2024-04-22T15:19:05.424 INFO:tasks.workunit.client.1.smithi196.stdout:5/673: mknod d1/d78/db0/da0/cd9 0 2024-04-22T15:19:05.425 INFO:tasks.workunit.client.1.smithi196.stdout:5/674: chown d1/d9/da9/l5a 190095439 1 2024-04-22T15:19:05.427 INFO:tasks.workunit.client.0.smithi096.stdout:3/894: dread da/db/de/ddc/d26/d52/f6c [4194304,4194304] 0 2024-04-22T15:19:05.428 INFO:tasks.workunit.client.1.smithi196.stdout:6/669: mkdir dd/dc4 0 2024-04-22T15:19:05.428 INFO:tasks.workunit.client.1.smithi196.stdout:1/628: dread d1/db/dc/dad/d36/f3f [0,4194304] 0 2024-04-22T15:19:05.431 INFO:tasks.workunit.client.0.smithi096.stdout:3/895: mkdir da/db/d10/d124 0 2024-04-22T15:19:05.435 INFO:tasks.workunit.client.1.smithi196.stdout:7/616: rmdir d14/d26/d5a/d5f 39 2024-04-22T15:19:05.436 INFO:tasks.workunit.client.0.smithi096.stdout:3/896: dwrite da/db/de/d19/f30 [4194304,4194304] 0 2024-04-22T15:19:05.439 INFO:tasks.workunit.client.1.smithi196.stdout:7/617: read d14/d26/d28/d29/fb5 [365025,76805] 0 2024-04-22T15:19:05.440 INFO:tasks.workunit.client.1.smithi196.stdout:1/629: unlink d1/db/dc/f6f 0 2024-04-22T15:19:05.443 INFO:tasks.workunit.client.1.smithi196.stdout:7/618: symlink d14/d26/d5a/db6/lcb 0 2024-04-22T15:19:05.443 INFO:tasks.workunit.client.1.smithi196.stdout:7/619: readlink d14/d26/d5a/l82 0 2024-04-22T15:19:05.444 INFO:tasks.workunit.client.1.smithi196.stdout:1/630: mknod d1/db/dc/de/ccd 0 2024-04-22T15:19:05.446 INFO:tasks.workunit.client.1.smithi196.stdout:8/657: mknod d2/d8/d47/d68/cbf 0 2024-04-22T15:19:05.446 INFO:tasks.workunit.client.1.smithi196.stdout:1/631: chown d1/db/d16/d8e/fbd 36658426 1 2024-04-22T15:19:05.447 INFO:tasks.workunit.client.1.smithi196.stdout:7/620: rename d14/d15/d91/d9a/faf to d14/d15/fcc 0 2024-04-22T15:19:05.449 INFO:tasks.workunit.client.1.smithi196.stdout:7/621: truncate d14/d26/d2a/d72/db1/fc5 166179 0 2024-04-22T15:19:05.451 INFO:tasks.workunit.client.1.smithi196.stdout:1/632: rename d1/db/dc/dad/lb5 to d1/db/dc/de/d5a/d5b/d69/lce 0 2024-04-22T15:19:05.452 INFO:tasks.workunit.client.0.smithi096.stdout:3/897: fsync da/db/de/d19/f30 0 2024-04-22T15:19:05.452 INFO:tasks.workunit.client.1.smithi196.stdout:8/658: dwrite d2/d5/d1a/f37 [0,4194304] 0 2024-04-22T15:19:05.452 INFO:tasks.workunit.client.1.smithi196.stdout:7/622: rmdir d14/d15/d91 39 2024-04-22T15:19:05.453 INFO:tasks.workunit.client.1.smithi196.stdout:1/633: write d1/f78 [4293759,75337] 0 2024-04-22T15:19:05.455 INFO:tasks.workunit.client.1.smithi196.stdout:6/670: dread dd/f1a [0,4194304] 0 2024-04-22T15:19:05.455 INFO:tasks.workunit.client.0.smithi096.stdout:3/898: write da/f8f [4305878,9203] 0 2024-04-22T15:19:05.456 INFO:tasks.workunit.client.1.smithi196.stdout:8/659: mknod d2/d35/d96/cc0 0 2024-04-22T15:19:05.464 INFO:tasks.workunit.client.1.smithi196.stdout:8/660: rename d2/d35/f61 to d2/d8/d49/d6c/fc1 0 2024-04-22T15:19:05.466 INFO:tasks.workunit.client.1.smithi196.stdout:8/661: dread - d2/d5/d93/fad zero size 2024-04-22T15:19:05.466 INFO:tasks.workunit.client.1.smithi196.stdout:7/623: creat d14/d26/d5a/d5f/fcd x:0 0 0 2024-04-22T15:19:05.466 INFO:tasks.workunit.client.0.smithi096.stdout:3/899: chown da/db/dc/d59/lb9 109478251 1 2024-04-22T15:19:05.470 INFO:tasks.workunit.client.1.smithi196.stdout:7/624: truncate d14/d15/f74 2737139 0 2024-04-22T15:19:05.471 INFO:tasks.workunit.client.1.smithi196.stdout:6/671: symlink dd/d1f/d4e/d53/d55/d6d/lc5 0 2024-04-22T15:19:05.475 INFO:tasks.workunit.client.1.smithi196.stdout:6/672: unlink dd/d1f/d4e/d53/d55/cc1 0 2024-04-22T15:19:05.480 INFO:tasks.workunit.client.0.smithi096.stdout:8/856: write d4/d1c/d40/d48/fa0 [2089915,78407] 0 2024-04-22T15:19:05.480 INFO:tasks.workunit.client.0.smithi096.stdout:4/848: truncate d5/d7/d1a/f38 4538412 0 2024-04-22T15:19:05.481 INFO:tasks.workunit.client.0.smithi096.stdout:8/857: read - d4/d10/d25/d4c/d76/d7f/f11d zero size 2024-04-22T15:19:05.483 INFO:tasks.workunit.client.0.smithi096.stdout:4/849: mkdir d5/d7/dcb/d12a 0 2024-04-22T15:19:05.487 INFO:tasks.workunit.client.0.smithi096.stdout:4/850: creat d5/d7/d2c/d95/df9/f12b x:0 0 0 2024-04-22T15:19:05.494 INFO:tasks.workunit.client.1.smithi196.stdout:9/632: symlink db/dd/lda 0 2024-04-22T15:19:05.495 INFO:tasks.workunit.client.1.smithi196.stdout:9/633: dread - db/d11/d29/d7a/d7b/d81/fd3 zero size 2024-04-22T15:19:05.497 INFO:tasks.workunit.client.1.smithi196.stdout:9/634: mknod db/dd/de/d15/d31/d64/cdb 0 2024-04-22T15:19:05.501 INFO:tasks.workunit.client.0.smithi096.stdout:6/714: write d3/d28/dd/d11/d2c/f39 [205284,77018] 0 2024-04-22T15:19:05.502 INFO:tasks.workunit.client.0.smithi096.stdout:4/851: dread d5/d7/d2c/ff0 [0,4194304] 0 2024-04-22T15:19:05.505 INFO:tasks.workunit.client.0.smithi096.stdout:4/852: chown d5/d7/d2c/d43/l76 0 1 2024-04-22T15:19:05.506 INFO:tasks.workunit.client.1.smithi196.stdout:2/667: sync 2024-04-22T15:19:05.507 INFO:tasks.workunit.client.0.smithi096.stdout:6/715: symlink d3/d28/d4d/d82/lf5 0 2024-04-22T15:19:05.509 INFO:tasks.workunit.client.1.smithi196.stdout:2/668: unlink d1/d23/d27/c28 0 2024-04-22T15:19:05.509 INFO:tasks.workunit.client.0.smithi096.stdout:5/884: dwrite d4/d1f/d8b/d106/f11a [0,4194304] 0 2024-04-22T15:19:05.510 INFO:tasks.workunit.client.1.smithi196.stdout:4/620: dwrite d6/d14/f33 [0,4194304] 0 2024-04-22T15:19:05.512 INFO:tasks.workunit.client.0.smithi096.stdout:4/853: symlink d5/d7/d2c/d95/dee/l12c 0 2024-04-22T15:19:05.513 INFO:tasks.workunit.client.0.smithi096.stdout:4/854: chown d5/d7/d2c/d95/dee/lc9 1871513469 1 2024-04-22T15:19:05.514 INFO:tasks.workunit.client.1.smithi196.stdout:4/621: dread d6/d14/f33 [0,4194304] 0 2024-04-22T15:19:05.515 INFO:tasks.workunit.client.0.smithi096.stdout:6/716: dread d3/d28/dd/d11/d2c/d6f/dd4/dd6/fc4 [0,4194304] 0 2024-04-22T15:19:05.523 INFO:tasks.workunit.client.0.smithi096.stdout:4/855: rename d5/d7/d1a/d25/feb to d5/d7/dcb/f12d 0 2024-04-22T15:19:05.530 INFO:tasks.workunit.client.0.smithi096.stdout:4/856: stat d5/d7/d1a/d8b/l115 0 2024-04-22T15:19:05.531 INFO:tasks.workunit.client.0.smithi096.stdout:0/971: dwrite d7/d8f/f12f [0,4194304] 0 2024-04-22T15:19:05.532 INFO:tasks.workunit.client.1.smithi196.stdout:4/622: mknod d6/d14/cc7 0 2024-04-22T15:19:05.537 INFO:tasks.workunit.client.0.smithi096.stdout:0/972: creat d7/de/d3a/d61/d79/f14b x:0 0 0 2024-04-22T15:19:05.539 INFO:tasks.workunit.client.0.smithi096.stdout:4/857: symlink d5/d7/d1f/d127/l12e 0 2024-04-22T15:19:05.542 INFO:tasks.workunit.client.0.smithi096.stdout:2/900: sync 2024-04-22T15:19:05.543 INFO:tasks.workunit.client.0.smithi096.stdout:6/717: getdents d3/d28/dd/d11/d2c/d6f/dd4/ddd 0 2024-04-22T15:19:05.544 INFO:tasks.workunit.client.0.smithi096.stdout:6/718: chown d3/d28/dd/d11/f1e 7 1 2024-04-22T15:19:05.547 INFO:tasks.workunit.client.0.smithi096.stdout:2/901: mknod d3/db/df/d98/db3/c124 0 2024-04-22T15:19:05.548 INFO:tasks.workunit.client.0.smithi096.stdout:4/858: symlink d5/d7/d1f/d84/d6d/d118/l12f 0 2024-04-22T15:19:05.548 INFO:tasks.workunit.client.0.smithi096.stdout:2/902: write d3/db/df/d10/db5/f113 [916346,124200] 0 2024-04-22T15:19:05.549 INFO:tasks.workunit.client.0.smithi096.stdout:6/719: creat d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/ff6 x:0 0 0 2024-04-22T15:19:05.556 INFO:tasks.workunit.client.0.smithi096.stdout:6/720: dwrite d3/f49 [0,4194304] 0 2024-04-22T15:19:05.557 INFO:tasks.workunit.client.0.smithi096.stdout:6/721: dread - d3/d28/d54/d68/fe4 zero size 2024-04-22T15:19:05.560 INFO:tasks.workunit.client.0.smithi096.stdout:6/722: fsync d3/d3a/f3e 0 2024-04-22T15:19:05.562 INFO:tasks.workunit.client.0.smithi096.stdout:6/723: chown d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/cd9 87844 1 2024-04-22T15:19:05.563 INFO:tasks.workunit.client.0.smithi096.stdout:3/900: write da/db/dc/f5d [802070,32099] 0 2024-04-22T15:19:05.564 INFO:tasks.workunit.client.0.smithi096.stdout:3/901: chown da/db/de 144 1 2024-04-22T15:19:05.566 INFO:tasks.workunit.client.0.smithi096.stdout:6/724: fsync d3/d28/dd/f2b 0 2024-04-22T15:19:05.572 INFO:tasks.workunit.client.0.smithi096.stdout:3/902: rmdir da/db/d10/d124 0 2024-04-22T15:19:05.574 INFO:tasks.workunit.client.0.smithi096.stdout:8/858: write d4/d1c/d40/fdc [288677,6634] 0 2024-04-22T15:19:05.583 INFO:tasks.workunit.client.0.smithi096.stdout:5/885: write d4/d1f/d80/f87 [170059,126431] 0 2024-04-22T15:19:05.588 INFO:tasks.workunit.client.0.smithi096.stdout:5/886: creat d4/ddb/d3d/d4d/d59/d8a/f12f x:0 0 0 2024-04-22T15:19:05.588 INFO:tasks.workunit.client.0.smithi096.stdout:8/859: dread d4/d1c/d40/d46/f7d [0,4194304] 0 2024-04-22T15:19:05.590 INFO:tasks.workunit.client.0.smithi096.stdout:8/860: stat d4/d1c/f43 0 2024-04-22T15:19:05.590 INFO:tasks.workunit.client.0.smithi096.stdout:5/887: dread - d4/d1f/d8b/d106/f10f zero size 2024-04-22T15:19:05.591 INFO:tasks.workunit.client.0.smithi096.stdout:4/859: write d5/d7/d1f/d84/d6d/d118/f124 [109437,33816] 0 2024-04-22T15:19:05.591 INFO:tasks.workunit.client.0.smithi096.stdout:0/973: dwrite d7/de/d5b/fe2 [0,4194304] 0 2024-04-22T15:19:05.596 INFO:tasks.workunit.client.1.smithi196.stdout:5/675: truncate d1/d14/d6a/fa8 1081541 0 2024-04-22T15:19:05.597 INFO:tasks.workunit.client.1.smithi196.stdout:6/673: write dd/f11 [3530426,80148] 0 2024-04-22T15:19:05.598 INFO:tasks.workunit.client.1.smithi196.stdout:2/669: truncate d1/d23/d27/d2e/f88 234098 0 2024-04-22T15:19:05.600 INFO:tasks.workunit.client.0.smithi096.stdout:3/903: write da/db/d12/d9b/fd6 [4721835,15901] 0 2024-04-22T15:19:05.601 INFO:tasks.workunit.client.0.smithi096.stdout:6/725: dwrite d3/d3a/f3e [4194304,4194304] 0 2024-04-22T15:19:05.602 INFO:tasks.workunit.client.1.smithi196.stdout:7/625: dwrite d14/d26/d2a/d2b/f70 [0,4194304] 0 2024-04-22T15:19:05.602 INFO:tasks.workunit.client.1.smithi196.stdout:4/623: creat d6/d14/d18/d26/d32/d67/fc8 x:0 0 0 2024-04-22T15:19:05.604 INFO:tasks.workunit.client.0.smithi096.stdout:8/861: creat d4/d1c/d40/d48/d101/f11f x:0 0 0 2024-04-22T15:19:05.609 INFO:tasks.workunit.client.0.smithi096.stdout:4/860: creat d5/d7/f130 x:0 0 0 2024-04-22T15:19:05.616 INFO:tasks.workunit.client.0.smithi096.stdout:5/888: write d4/ddb/d3d/d35/d57/fb4 [934017,17028] 0 2024-04-22T15:19:05.618 INFO:tasks.workunit.client.1.smithi196.stdout:2/670: creat d1/d13/d1f/ddf/da1/ff7 x:0 0 0 2024-04-22T15:19:05.619 INFO:tasks.workunit.client.0.smithi096.stdout:6/726: dread - d3/d28/dd/d11/d2c/d6f/f99 zero size 2024-04-22T15:19:05.619 INFO:tasks.workunit.client.0.smithi096.stdout:6/727: chown d3/d28/d4d/d82/dd8/fe2 32974394 1 2024-04-22T15:19:05.621 INFO:tasks.workunit.client.1.smithi196.stdout:5/676: write d1/d17/f8b [4485663,104773] 0 2024-04-22T15:19:05.623 INFO:tasks.workunit.client.1.smithi196.stdout:7/626: creat d14/d26/d2a/d72/db1/fce x:0 0 0 2024-04-22T15:19:05.624 INFO:tasks.workunit.client.0.smithi096.stdout:4/861: creat d5/d7/d2c/d3d/d4f/d63/f131 x:0 0 0 2024-04-22T15:19:05.625 INFO:tasks.workunit.client.0.smithi096.stdout:6/728: dread d3/d28/d4d/d82/fd1 [0,4194304] 0 2024-04-22T15:19:05.625 INFO:tasks.workunit.client.0.smithi096.stdout:4/862: readlink d5/d7/d1a/d25/l37 0 2024-04-22T15:19:05.625 INFO:tasks.workunit.client.0.smithi096.stdout:0/974: creat d7/de/d5b/d95/dc/d16/f14c x:0 0 0 2024-04-22T15:19:05.627 INFO:tasks.workunit.client.0.smithi096.stdout:3/904: dread da/db/dc/d28/f45 [0,4194304] 0 2024-04-22T15:19:05.627 INFO:tasks.workunit.client.0.smithi096.stdout:6/729: readlink d3/d28/dd/d11/d2c/l34 0 2024-04-22T15:19:05.627 INFO:tasks.workunit.client.0.smithi096.stdout:5/889: fsync d4/ddb/d3d/d4f/fc7 0 2024-04-22T15:19:05.630 INFO:tasks.workunit.client.1.smithi196.stdout:3/650: sync 2024-04-22T15:19:05.631 INFO:tasks.workunit.client.1.smithi196.stdout:2/671: read d1/d13/d1f/f60 [57094,127367] 0 2024-04-22T15:19:05.633 INFO:tasks.workunit.client.1.smithi196.stdout:2/672: write d1/d23/d27/d73/fc6 [1026158,27185] 0 2024-04-22T15:19:05.634 INFO:tasks.workunit.client.1.smithi196.stdout:6/674: dread dd/f26 [0,4194304] 0 2024-04-22T15:19:05.635 INFO:tasks.workunit.client.0.smithi096.stdout:3/905: dread da/db/de/ddc/d26/ffe [0,4194304] 0 2024-04-22T15:19:05.637 INFO:tasks.workunit.client.1.smithi196.stdout:5/677: dread d1/d17/d1a/d2f/d30/d6e/d7a/fa3 [0,4194304] 0 2024-04-22T15:19:05.637 INFO:tasks.workunit.client.1.smithi196.stdout:5/678: read - d1/d17/d1a/d2f/d30/f80 zero size 2024-04-22T15:19:05.638 INFO:tasks.workunit.client.1.smithi196.stdout:5/679: dread - d1/d2/d98/dac/fd3 zero size 2024-04-22T15:19:05.639 INFO:tasks.workunit.client.0.smithi096.stdout:3/906: dread da/db/de/d8b/ff6 [0,4194304] 0 2024-04-22T15:19:05.639 INFO:tasks.workunit.client.1.smithi196.stdout:5/680: fdatasync d1/d14/f1e 0 2024-04-22T15:19:05.639 INFO:tasks.workunit.client.1.smithi196.stdout:3/651: read - d0/d15/da7/d4a/f45 zero size 2024-04-22T15:19:05.640 INFO:tasks.workunit.client.1.smithi196.stdout:3/652: stat d0/d15/d4d/d87 0 2024-04-22T15:19:05.640 INFO:tasks.workunit.client.0.smithi096.stdout:5/890: symlink d4/d2c/d2f/d31/d82/l130 0 2024-04-22T15:19:05.644 INFO:tasks.workunit.client.1.smithi196.stdout:4/624: dwrite d6/d14/f12 [0,4194304] 0 2024-04-22T15:19:05.644 INFO:tasks.workunit.client.0.smithi096.stdout:6/730: rename d3/d28/dd/de0 to d3/d3a/d75/d83/df7 0 2024-04-22T15:19:05.645 INFO:tasks.workunit.client.0.smithi096.stdout:8/862: dwrite d4/d10/d25/d4c/f87 [0,4194304] 0 2024-04-22T15:19:05.645 INFO:tasks.workunit.client.1.smithi196.stdout:4/625: stat d6/d14/c1e 0 2024-04-22T15:19:05.645 INFO:tasks.workunit.client.1.smithi196.stdout:5/681: dwrite d1/d9/d68/fbd [0,4194304] 0 2024-04-22T15:19:05.649 INFO:tasks.workunit.client.0.smithi096.stdout:3/907: dwrite da/db/dc/d59/fd7 [4194304,4194304] 0 2024-04-22T15:19:05.650 INFO:tasks.workunit.client.0.smithi096.stdout:3/908: readlink da/db/dc/d8d/le5 0 2024-04-22T15:19:05.653 INFO:tasks.workunit.client.0.smithi096.stdout:5/891: read - d4/ddb/d3d/d35/fbf zero size 2024-04-22T15:19:05.654 INFO:tasks.workunit.client.1.smithi196.stdout:2/673: dread d1/d13/d25/f4f [0,4194304] 0 2024-04-22T15:19:05.654 INFO:tasks.workunit.client.1.smithi196.stdout:5/682: readlink d1/d9/l23 0 2024-04-22T15:19:05.657 INFO:tasks.workunit.client.1.smithi196.stdout:6/675: mkdir dd/d1f/dc6 0 2024-04-22T15:19:05.658 INFO:tasks.workunit.client.0.smithi096.stdout:8/863: mkdir d4/d1c/d40/d46/d120 0 2024-04-22T15:19:05.661 INFO:tasks.workunit.client.0.smithi096.stdout:3/909: creat da/db/d12/d67/f125 x:0 0 0 2024-04-22T15:19:05.661 INFO:tasks.workunit.client.1.smithi196.stdout:4/626: mkdir d6/d14/d18/d26/d32/da3/db7/dc9 0 2024-04-22T15:19:05.662 INFO:tasks.workunit.client.0.smithi096.stdout:5/892: chown d4/f7 64159849 1 2024-04-22T15:19:05.663 INFO:tasks.workunit.client.1.smithi196.stdout:2/674: read d1/d13/d1f/ddf/f9c [11133,53101] 0 2024-04-22T15:19:05.663 INFO:tasks.workunit.client.1.smithi196.stdout:2/675: readlink d1/d23/d27/d2e/d55/lee 0 2024-04-22T15:19:05.664 INFO:tasks.workunit.client.0.smithi096.stdout:0/975: rename d7/d42/d4c/d55/f12a to d7/de/d5b/d95/d59/f14d 0 2024-04-22T15:19:05.667 INFO:tasks.workunit.client.1.smithi196.stdout:5/683: rename d1/d14/caa to d1/d2/dbb/cda 0 2024-04-22T15:19:05.668 INFO:tasks.workunit.client.1.smithi196.stdout:5/684: write d1/d17/f5d [2421988,126759] 0 2024-04-22T15:19:05.669 INFO:tasks.workunit.client.0.smithi096.stdout:4/863: getdents d5/d7/d1f/d84/d6d/d98 0 2024-04-22T15:19:05.671 INFO:tasks.workunit.client.1.smithi196.stdout:6/676: creat dd/d1f/d4e/d53/d55/d6d/fc7 x:0 0 0 2024-04-22T15:19:05.673 INFO:tasks.workunit.client.0.smithi096.stdout:3/910: mknod da/db/dc/d8d/c126 0 2024-04-22T15:19:05.674 INFO:tasks.workunit.client.1.smithi196.stdout:4/627: rename d6/d14/d18/d26/d32/cb4 to d6/d10/cca 0 2024-04-22T15:19:05.675 INFO:tasks.workunit.client.1.smithi196.stdout:4/628: stat d6/d14/d18/d26/d32/f7d 0 2024-04-22T15:19:05.677 INFO:tasks.workunit.client.1.smithi196.stdout:5/685: creat d1/d17/d1a/d2f/d30/d6e/d7a/fdb x:0 0 0 2024-04-22T15:19:05.677 INFO:tasks.workunit.client.0.smithi096.stdout:0/976: truncate d7/de/d5b/d95/dc/d16/d18/f92 1140780 0 2024-04-22T15:19:05.678 INFO:tasks.workunit.client.1.smithi196.stdout:5/686: chown d1/d17/d1c/lc3 0 1 2024-04-22T15:19:05.678 INFO:tasks.workunit.client.0.smithi096.stdout:8/864: mknod d4/d1c/d40/c121 0 2024-04-22T15:19:05.679 INFO:tasks.workunit.client.1.smithi196.stdout:4/629: dread d6/d13/d15/fb2 [0,4194304] 0 2024-04-22T15:19:05.679 INFO:tasks.workunit.client.0.smithi096.stdout:8/865: chown d4/d10/d25/d4c/l6a 0 1 2024-04-22T15:19:05.680 INFO:tasks.workunit.client.1.smithi196.stdout:2/676: rename d1/d13/c37 to d1/d23/d27/d2e/d55/cf8 0 2024-04-22T15:19:05.683 INFO:tasks.workunit.client.0.smithi096.stdout:5/893: mknod d4/ddb/d3d/ded/c131 0 2024-04-22T15:19:05.687 INFO:tasks.workunit.client.0.smithi096.stdout:6/731: rename d3/d3a/d75/l66 to d3/d28/daf/lf8 0 2024-04-22T15:19:05.688 INFO:tasks.workunit.client.1.smithi196.stdout:5/687: symlink d1/d17/d1a/d2f/d30/d6e/d7a/ldc 0 2024-04-22T15:19:05.691 INFO:tasks.workunit.client.1.smithi196.stdout:4/630: unlink d6/d90/f4a 0 2024-04-22T15:19:05.693 INFO:tasks.workunit.client.1.smithi196.stdout:4/631: stat d6/d13/d15/fb2 0 2024-04-22T15:19:05.693 INFO:tasks.workunit.client.1.smithi196.stdout:7/627: dwrite d14/d15/f74 [0,4194304] 0 2024-04-22T15:19:05.694 INFO:tasks.workunit.client.0.smithi096.stdout:8/866: unlink d4/d10/d25/d4c/db4/df4/f10c 0 2024-04-22T15:19:05.694 INFO:tasks.workunit.client.1.smithi196.stdout:7/628: readlink d14/d4d/l5c 0 2024-04-22T15:19:05.695 INFO:tasks.workunit.client.1.smithi196.stdout:7/629: fdatasync d14/d25/d31/f55 0 2024-04-22T15:19:05.697 INFO:tasks.workunit.client.0.smithi096.stdout:3/911: dread da/db/de/ddc/f88 [0,4194304] 0 2024-04-22T15:19:05.699 INFO:tasks.workunit.client.0.smithi096.stdout:4/864: rename d5/d7/d1a/lcf to d5/d7/d2c/d3d/d4f/l132 0 2024-04-22T15:19:05.700 INFO:tasks.workunit.client.1.smithi196.stdout:2/677: dread d1/f7 [0,4194304] 0 2024-04-22T15:19:05.702 INFO:tasks.workunit.client.0.smithi096.stdout:6/732: write d3/d28/f17 [3455178,38433] 0 2024-04-22T15:19:05.703 INFO:tasks.workunit.client.0.smithi096.stdout:6/733: chown d3/d28/dd/d11/l85 7087 1 2024-04-22T15:19:05.704 INFO:tasks.workunit.client.0.smithi096.stdout:6/734: dread - d3/d3a/d75/fd3 zero size 2024-04-22T15:19:05.705 INFO:tasks.workunit.client.1.smithi196.stdout:4/632: unlink d6/d14/d18/d26/f43 0 2024-04-22T15:19:05.711 INFO:tasks.workunit.client.0.smithi096.stdout:8/867: fdatasync d4/d1c/d40/da7/db6/f49 0 2024-04-22T15:19:05.714 INFO:tasks.workunit.client.0.smithi096.stdout:5/894: truncate d4/f1c 2803279 0 2024-04-22T15:19:05.717 INFO:tasks.workunit.client.0.smithi096.stdout:3/912: rename da/db/dc/d28/f45 to da/db/de/ddc/d68/db0/f127 0 2024-04-22T15:19:05.718 INFO:tasks.workunit.client.0.smithi096.stdout:4/865: rmdir d5/d7/dcb 39 2024-04-22T15:19:05.719 INFO:tasks.workunit.client.0.smithi096.stdout:6/735: truncate f2 3117723 0 2024-04-22T15:19:05.721 INFO:tasks.workunit.client.1.smithi196.stdout:5/688: creat d1/d17/d1a/d2f/d30/fdd x:0 0 0 2024-04-22T15:19:05.723 INFO:tasks.workunit.client.1.smithi196.stdout:2/678: unlink d1/d23/d27/d2e/d55/d5b/d7c/fa5 0 2024-04-22T15:19:05.724 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:05 smithi096 ceph-mon[26688]: pgmap v39: 65 pgs: 65 active+clean; 3.1 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 113 MiB/s rd, 285 MiB/s wr, 602 op/s 2024-04-22T15:19:05.727 INFO:tasks.workunit.client.1.smithi196.stdout:6/677: dwrite dd/f23 [0,4194304] 0 2024-04-22T15:19:05.728 INFO:tasks.workunit.client.0.smithi096.stdout:9/784: sync 2024-04-22T15:19:05.728 INFO:tasks.workunit.client.0.smithi096.stdout:1/907: sync 2024-04-22T15:19:05.729 INFO:tasks.workunit.client.1.smithi196.stdout:5/689: dwrite d1/d17/d1a/d2f/d30/fdd [0,4194304] 0 2024-04-22T15:19:05.733 INFO:tasks.workunit.client.0.smithi096.stdout:3/913: mknod da/db/de/d19/d2d/c128 0 2024-04-22T15:19:05.734 INFO:tasks.workunit.client.1.smithi196.stdout:7/630: write d14/d26/d2a/d2b/f45 [844601,88064] 0 2024-04-22T15:19:05.736 INFO:tasks.workunit.client.1.smithi196.stdout:4/633: rename d6/d90/f81 to d6/d14/d18/d26/d32/fcb 0 2024-04-22T15:19:05.742 INFO:tasks.workunit.client.1.smithi196.stdout:6/678: rmdir dd/d20/d5c/d6a 39 2024-04-22T15:19:05.744 INFO:tasks.workunit.client.1.smithi196.stdout:2/679: fsync d1/d76/fb9 0 2024-04-22T15:19:05.745 INFO:tasks.workunit.client.1.smithi196.stdout:5/690: fdatasync d1/d17/d1a/d2f/d30/f54 0 2024-04-22T15:19:05.747 INFO:tasks.workunit.client.1.smithi196.stdout:7/631: symlink d14/d25/d31/lcf 0 2024-04-22T15:19:05.748 INFO:tasks.workunit.client.1.smithi196.stdout:7/632: truncate d14/fa5 77619 0 2024-04-22T15:19:05.750 INFO:tasks.workunit.client.1.smithi196.stdout:6/679: truncate dd/d20/d5c/d6a/fa2 631160 0 2024-04-22T15:19:05.750 INFO:tasks.workunit.client.1.smithi196.stdout:5/691: getdents d1/d78/dcf 0 2024-04-22T15:19:05.750 INFO:tasks.workunit.client.1.smithi196.stdout:6/680: chown dd/f12 1 1 2024-04-22T15:19:05.751 INFO:tasks.workunit.client.1.smithi196.stdout:7/633: creat d14/d26/fd0 x:0 0 0 2024-04-22T15:19:05.752 INFO:tasks.workunit.client.1.smithi196.stdout:5/692: mkdir d1/d2/d98/dde 0 2024-04-22T15:19:05.753 INFO:tasks.workunit.client.1.smithi196.stdout:7/634: creat d14/d26/d2a/d2b/dc7/fd1 x:0 0 0 2024-04-22T15:19:05.754 INFO:tasks.workunit.client.1.smithi196.stdout:7/635: chown d14/d15/fcc 1 1 2024-04-22T15:19:05.754 INFO:tasks.workunit.client.1.smithi196.stdout:5/693: mknod d1/d2/d98/dac/cdf 0 2024-04-22T15:19:05.755 INFO:tasks.workunit.client.1.smithi196.stdout:1/634: sync 2024-04-22T15:19:05.756 INFO:tasks.workunit.client.1.smithi196.stdout:0/681: sync 2024-04-22T15:19:05.756 INFO:tasks.workunit.client.1.smithi196.stdout:6/681: getdents dd/d2b/d8f 0 2024-04-22T15:19:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:05 smithi196 ceph-mon[34717]: pgmap v39: 65 pgs: 65 active+clean; 3.1 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 113 MiB/s rd, 285 MiB/s wr, 602 op/s 2024-04-22T15:19:05.759 INFO:tasks.workunit.client.1.smithi196.stdout:6/682: creat dd/d1f/d4e/d53/d55/fc8 x:0 0 0 2024-04-22T15:19:05.759 INFO:tasks.workunit.client.1.smithi196.stdout:7/636: dwrite f7 [0,4194304] 0 2024-04-22T15:19:05.761 INFO:tasks.workunit.client.1.smithi196.stdout:6/683: write dd/d1f/d37/d3c/f46 [1223355,60942] 0 2024-04-22T15:19:05.761 INFO:tasks.workunit.client.1.smithi196.stdout:1/635: link d1/f78 d1/db/dc/de/d43/d5e/d7a/fcf 0 2024-04-22T15:19:05.763 INFO:tasks.workunit.client.1.smithi196.stdout:6/684: rename dd/d70/lb5 to dd/d70/lc9 0 2024-04-22T15:19:05.763 INFO:tasks.workunit.client.1.smithi196.stdout:6/685: dread - dd/d20/d5c/d5e/f9b zero size 2024-04-22T15:19:05.768 INFO:tasks.workunit.client.1.smithi196.stdout:1/636: dwrite d1/db/dc/de/d43/f68 [0,4194304] 0 2024-04-22T15:19:05.769 INFO:tasks.workunit.client.1.smithi196.stdout:1/637: write d1/db/d16/d1e/d46/fc1 [186355,34616] 0 2024-04-22T15:19:05.771 INFO:tasks.workunit.client.1.smithi196.stdout:1/638: creat d1/db/dc/de/d5a/d94/dab/fd0 x:0 0 0 2024-04-22T15:19:05.773 INFO:tasks.workunit.client.1.smithi196.stdout:1/639: creat d1/db/d16/fd1 x:0 0 0 2024-04-22T15:19:05.797 INFO:tasks.workunit.client.0.smithi096.stdout:0/977: dwrite d7/de/d5b/d95/dc/d16/f1f [0,4194304] 0 2024-04-22T15:19:05.798 INFO:tasks.workunit.client.0.smithi096.stdout:4/866: readlink d5/d7/d2c/d3d/lff 0 2024-04-22T15:19:05.801 INFO:tasks.workunit.client.0.smithi096.stdout:1/908: rename d6/dc/d8a/d3f/d54/d32/d3b/d56/l88 to d6/dc/d8a/d3f/d54/d32/d8c/dca/l11f 0 2024-04-22T15:19:05.809 INFO:tasks.workunit.client.0.smithi096.stdout:5/895: dwrite d4/ddb/d3d/d4d/d59/d98/fd3 [0,4194304] 0 2024-04-22T15:19:05.811 INFO:tasks.workunit.client.0.smithi096.stdout:1/909: truncate d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/f108 947917 0 2024-04-22T15:19:05.812 INFO:tasks.workunit.client.0.smithi096.stdout:8/868: rename d4/d1c/d40/d46/d58/d11a to d4/d1c/d40/d48/d101/d122 0 2024-04-22T15:19:05.814 INFO:tasks.workunit.client.0.smithi096.stdout:4/867: creat d5/d7/d1a/f133 x:0 0 0 2024-04-22T15:19:05.814 INFO:tasks.workunit.client.0.smithi096.stdout:4/868: dread - d5/d7/d1a/d103/f10d zero size 2024-04-22T15:19:05.815 INFO:tasks.workunit.client.0.smithi096.stdout:4/869: fdatasync d5/d7/d1a/d103/f114 0 2024-04-22T15:19:05.818 INFO:tasks.workunit.client.0.smithi096.stdout:0/978: mknod d7/d8f/d10e/d114/d120/c14e 0 2024-04-22T15:19:05.818 INFO:tasks.workunit.client.0.smithi096.stdout:9/785: write d1/df/fa7 [1061681,120314] 0 2024-04-22T15:19:05.819 INFO:tasks.workunit.client.0.smithi096.stdout:9/786: dread - d1/d4/d7b/fa6 zero size 2024-04-22T15:19:05.825 INFO:tasks.workunit.client.0.smithi096.stdout:3/914: link da/db/d6a/c82 da/d3f/c129 0 2024-04-22T15:19:05.827 INFO:tasks.workunit.client.0.smithi096.stdout:8/869: read d4/d1c/fc4 [15987,23199] 0 2024-04-22T15:19:05.827 INFO:tasks.workunit.client.0.smithi096.stdout:9/787: truncate d1/df/d95/d9d/db5/d43/d57/d5b/f84 826322 0 2024-04-22T15:19:05.828 INFO:tasks.workunit.client.0.smithi096.stdout:8/870: chown d4/d10/d25/d4c/c6d 17 1 2024-04-22T15:19:05.829 INFO:tasks.workunit.client.0.smithi096.stdout:1/910: dread d6/dc/d8a/f67 [0,4194304] 0 2024-04-22T15:19:05.829 INFO:tasks.workunit.client.0.smithi096.stdout:3/915: mknod da/db/d9a/c12a 0 2024-04-22T15:19:05.830 INFO:tasks.workunit.client.0.smithi096.stdout:8/871: chown d4/d10/d25/df7/c11c 0 1 2024-04-22T15:19:05.831 INFO:tasks.workunit.client.0.smithi096.stdout:1/911: stat d6/dc/d8a/d3f/d54/d32/d8c 0 2024-04-22T15:19:05.831 INFO:tasks.workunit.client.0.smithi096.stdout:5/896: rename d4/d1f/d80/c84 to d4/ddb/c132 0 2024-04-22T15:19:05.833 INFO:tasks.workunit.client.0.smithi096.stdout:9/788: mknod d1/d25/c107 0 2024-04-22T15:19:05.835 INFO:tasks.workunit.client.1.smithi196.stdout:0/682: dread d3/d9/f73 [0,4194304] 0 2024-04-22T15:19:05.835 INFO:tasks.workunit.client.1.smithi196.stdout:4/634: dwrite d6/d90/f60 [4194304,4194304] 0 2024-04-22T15:19:05.835 INFO:tasks.workunit.client.0.smithi096.stdout:3/916: dread da/f8a [0,4194304] 0 2024-04-22T15:19:05.836 INFO:tasks.workunit.client.0.smithi096.stdout:8/872: mkdir d4/dc0/d123 0 2024-04-22T15:19:05.836 INFO:tasks.workunit.client.1.smithi196.stdout:4/635: readlink d6/d14/d18/d26/d32/d53/la2 0 2024-04-22T15:19:05.838 INFO:tasks.workunit.client.0.smithi096.stdout:4/870: rename d5/d7/d1a/d103/f114 to d5/d7/d1f/f134 0 2024-04-22T15:19:05.852 INFO:tasks.workunit.client.0.smithi096.stdout:1/912: truncate d6/dc/d8a/d3f/d54/d32/d3b/d94/fad 1028422 0 2024-04-22T15:19:05.857 INFO:tasks.workunit.client.0.smithi096.stdout:8/873: symlink d4/d1c/d40/d46/l124 0 2024-04-22T15:19:05.859 INFO:tasks.workunit.client.1.smithi196.stdout:8/662: sync 2024-04-22T15:19:05.859 INFO:tasks.workunit.client.1.smithi196.stdout:9/635: sync 2024-04-22T15:19:05.859 INFO:tasks.workunit.client.0.smithi096.stdout:1/913: truncate d6/dc/d8a/d29/f79 3839910 0 2024-04-22T15:19:05.861 INFO:tasks.workunit.client.0.smithi096.stdout:5/897: link d4/ddb/d3d/d4d/d59/d5a/d5c/f89 d4/ddb/d3d/d4d/d59/d98/dee/d109/f133 0 2024-04-22T15:19:05.861 INFO:tasks.workunit.client.1.smithi196.stdout:0/683: symlink d3/d6/dc/d8c/d8d/d98/lf0 0 2024-04-22T15:19:05.861 INFO:tasks.workunit.client.1.smithi196.stdout:6/686: write dd/d1f/d4e/d53/d7f/f81 [2175174,116223] 0 2024-04-22T15:19:05.862 INFO:tasks.workunit.client.0.smithi096.stdout:1/914: stat d6/dc/d8a/d3f/d62/dc2/dd0/l11c 0 2024-04-22T15:19:05.862 INFO:tasks.workunit.client.1.smithi196.stdout:1/640: write d1/db/dc/de/d43/f6b [1872354,34833] 0 2024-04-22T15:19:05.863 INFO:tasks.workunit.client.0.smithi096.stdout:5/898: chown d4/ddb/d3d/c10d 490 1 2024-04-22T15:19:05.864 INFO:tasks.workunit.client.1.smithi196.stdout:1/641: write d1/db/d16/d1e/f88 [70169,38176] 0 2024-04-22T15:19:05.865 INFO:tasks.workunit.client.0.smithi096.stdout:8/874: dwrite d4/d10/f97 [0,4194304] 0 2024-04-22T15:19:05.867 INFO:tasks.workunit.client.0.smithi096.stdout:1/915: rename d6/dc/d8a/d3f/d54/d32/d8c to d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/d104/d120 22 2024-04-22T15:19:05.867 INFO:tasks.workunit.client.1.smithi196.stdout:4/636: dwrite d6/d14/d18/d26/d32/d53/fba [0,4194304] 0 2024-04-22T15:19:05.871 INFO:tasks.workunit.client.1.smithi196.stdout:9/636: chown db/d11/d29/d7a/ca6 0 1 2024-04-22T15:19:05.874 INFO:tasks.workunit.client.0.smithi096.stdout:9/789: write d1/d25/db2/d98/d9b/fc7 [893313,41256] 0 2024-04-22T15:19:05.875 INFO:tasks.workunit.client.0.smithi096.stdout:9/790: stat d1/d25/db2/dca 0 2024-04-22T15:19:05.877 INFO:tasks.workunit.client.0.smithi096.stdout:3/917: getdents da/db/de/ddc/d68/db0 0 2024-04-22T15:19:05.882 INFO:tasks.workunit.client.0.smithi096.stdout:4/871: write d5/d7/d1a/f38 [3977154,19091] 0 2024-04-22T15:19:05.886 INFO:tasks.workunit.client.0.smithi096.stdout:1/916: creat d6/dc/dbd/f121 x:0 0 0 2024-04-22T15:19:05.890 INFO:tasks.workunit.client.0.smithi096.stdout:8/875: rename d4/d1c/d62/dc3/dec to d4/d1c/de2/d125 0 2024-04-22T15:19:05.892 INFO:tasks.workunit.client.1.smithi196.stdout:4/637: unlink d6/d14/d18/d26/d32/da3/fbc 0 2024-04-22T15:19:05.893 INFO:tasks.workunit.client.1.smithi196.stdout:4/638: dread - d6/d13/d15/d2c/d62/d78/d8d/fbd zero size 2024-04-22T15:19:05.898 INFO:tasks.workunit.client.0.smithi096.stdout:5/899: mkdir d4/ddb/d3d/d4d/d59/d8a/d121/d134 0 2024-04-22T15:19:05.901 INFO:tasks.workunit.client.1.smithi196.stdout:0/684: dwrite d3/de/d14/f55 [0,4194304] 0 2024-04-22T15:19:05.903 INFO:tasks.workunit.client.0.smithi096.stdout:1/917: mkdir d6/dc/d8a/d122 0 2024-04-22T15:19:05.903 INFO:tasks.workunit.client.1.smithi196.stdout:0/685: chown d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/db7/cbb 15 1 2024-04-22T15:19:05.906 INFO:tasks.workunit.client.1.smithi196.stdout:0/686: read d3/f47 [1974600,93948] 0 2024-04-22T15:19:05.908 INFO:tasks.workunit.client.0.smithi096.stdout:8/876: symlink d4/d1c/d40/da7/l126 0 2024-04-22T15:19:05.910 INFO:tasks.workunit.client.1.smithi196.stdout:6/687: truncate dd/d1f/d37/d3c/f46 1178001 0 2024-04-22T15:19:05.916 INFO:tasks.workunit.client.0.smithi096.stdout:3/918: rename da/db/de/ddc/d26/dc6/f105 to da/db/d12/f12b 0 2024-04-22T15:19:05.916 INFO:tasks.workunit.client.0.smithi096.stdout:4/872: write d5/d7/d2c/d3d/d4f/f60 [4979308,37100] 0 2024-04-22T15:19:05.919 INFO:tasks.workunit.client.1.smithi196.stdout:1/642: truncate d1/db/f58 3882620 0 2024-04-22T15:19:05.920 INFO:tasks.workunit.client.0.smithi096.stdout:4/873: dread d5/d7/d2c/d3d/d4f/d88/ffc [0,4194304] 0 2024-04-22T15:19:05.920 INFO:tasks.workunit.client.1.smithi196.stdout:1/643: fsync d1/db/d16/d1e/f88 0 2024-04-22T15:19:05.921 INFO:tasks.workunit.client.1.smithi196.stdout:6/688: dread dd/f16 [0,4194304] 0 2024-04-22T15:19:05.922 INFO:tasks.workunit.client.1.smithi196.stdout:6/689: write dd/d2b/d8f/fbf [63415,120726] 0 2024-04-22T15:19:05.924 INFO:tasks.workunit.client.1.smithi196.stdout:8/663: rename d2/d35/f9e to d2/dc/fc2 0 2024-04-22T15:19:05.926 INFO:tasks.workunit.client.0.smithi096.stdout:8/877: dread d4/d10/d25/d38/f79 [0,4194304] 0 2024-04-22T15:19:05.927 INFO:tasks.workunit.client.1.smithi196.stdout:6/690: dwrite dd/d1f/d4e/d53/d55/f8b [0,4194304] 0 2024-04-22T15:19:05.928 INFO:tasks.workunit.client.1.smithi196.stdout:0/687: unlink d3/d6/l13 0 2024-04-22T15:19:05.929 INFO:tasks.workunit.client.1.smithi196.stdout:3/653: sync 2024-04-22T15:19:05.930 INFO:tasks.workunit.client.1.smithi196.stdout:2/680: sync 2024-04-22T15:19:05.930 INFO:tasks.workunit.client.1.smithi196.stdout:0/688: write d3/d9/d36/d9f/dd1/d65/d7f/d30/faa [1172782,11464] 0 2024-04-22T15:19:05.934 INFO:tasks.workunit.client.1.smithi196.stdout:0/689: dread d3/fd [0,4194304] 0 2024-04-22T15:19:05.936 INFO:tasks.workunit.client.1.smithi196.stdout:9/637: truncate db/fc 4325658 0 2024-04-22T15:19:05.942 INFO:tasks.workunit.client.1.smithi196.stdout:9/638: dwrite db/dd/de/d15/d31/fb7 [0,4194304] 0 2024-04-22T15:19:05.943 INFO:tasks.workunit.client.1.smithi196.stdout:4/639: write d6/d13/d15/d2c/d62/d78/d8b/f8e [790402,4055] 0 2024-04-22T15:19:05.943 INFO:tasks.workunit.client.1.smithi196.stdout:1/644: mknod d1/d54/d70/cd2 0 2024-04-22T15:19:05.944 INFO:tasks.workunit.client.0.smithi096.stdout:2/903: sync 2024-04-22T15:19:05.945 INFO:tasks.workunit.client.1.smithi196.stdout:4/640: write d6/d13/d15/d2c/d62/d78/d8b/f8e [1132712,4081] 0 2024-04-22T15:19:05.949 INFO:tasks.workunit.client.1.smithi196.stdout:8/664: fdatasync d2/d5/f87 0 2024-04-22T15:19:05.950 INFO:tasks.workunit.client.1.smithi196.stdout:8/665: chown d2/d8/d49/d6c 204689 1 2024-04-22T15:19:05.952 INFO:tasks.workunit.client.1.smithi196.stdout:6/691: mkdir dd/d1f/d37/d3c/d96/dca 0 2024-04-22T15:19:05.954 INFO:tasks.workunit.client.0.smithi096.stdout:4/874: truncate d5/d7/fc7 1231307 0 2024-04-22T15:19:05.954 INFO:tasks.workunit.client.0.smithi096.stdout:4/875: readlink d5/d5b/l69 0 2024-04-22T15:19:05.961 INFO:tasks.workunit.client.0.smithi096.stdout:3/919: dwrite da/db/dc/f16 [0,4194304] 0 2024-04-22T15:19:05.963 INFO:tasks.workunit.client.0.smithi096.stdout:2/904: symlink d3/db/df/d24/d58/d5f/dfe/d115/d2f/l125 0 2024-04-22T15:19:05.964 INFO:tasks.workunit.client.0.smithi096.stdout:2/905: chown d3/db/df/d24/d58/d5f/d6e/f100 4 1 2024-04-22T15:19:05.965 INFO:tasks.workunit.client.0.smithi096.stdout:5/900: write d4/ddb/d3d/d4d/d59/d5a/f13 [966856,80669] 0 2024-04-22T15:19:05.966 INFO:tasks.workunit.client.0.smithi096.stdout:2/906: write d3/d107/f111 [1020754,18438] 0 2024-04-22T15:19:05.966 INFO:tasks.workunit.client.0.smithi096.stdout:9/791: link d1/d25/db2/cac d1/df/d95/d9d/db5/d43/d57/d5b/d8e/c108 0 2024-04-22T15:19:05.969 INFO:tasks.workunit.client.1.smithi196.stdout:3/654: symlink d0/d15/d31/d55/dcf/d8/d6b/ldc 0 2024-04-22T15:19:05.969 INFO:tasks.workunit.client.1.smithi196.stdout:4/641: dread d6/d13/f6b [0,4194304] 0 2024-04-22T15:19:05.971 INFO:tasks.workunit.client.0.smithi096.stdout:4/876: truncate d5/d7/d2c/d3d/d4f/d9c/ffb 827795 0 2024-04-22T15:19:05.971 INFO:tasks.workunit.client.0.smithi096.stdout:4/877: chown d5/fe9 4 1 2024-04-22T15:19:05.972 INFO:tasks.workunit.client.0.smithi096.stdout:4/878: stat d5/d7/d1a/d103/f10d 0 2024-04-22T15:19:05.973 INFO:tasks.workunit.client.1.smithi196.stdout:0/690: mkdir d3/de/d1c/df1 0 2024-04-22T15:19:05.977 INFO:tasks.workunit.client.1.smithi196.stdout:7/637: sync 2024-04-22T15:19:05.977 INFO:tasks.workunit.client.1.smithi196.stdout:5/694: sync 2024-04-22T15:19:05.978 INFO:tasks.workunit.client.0.smithi096.stdout:2/907: rmdir d3/d13/db4 39 2024-04-22T15:19:05.978 INFO:tasks.workunit.client.1.smithi196.stdout:7/638: chown d14/d25/d31/f55 60909022 1 2024-04-22T15:19:05.980 INFO:tasks.workunit.client.1.smithi196.stdout:1/645: stat d1/c19 0 2024-04-22T15:19:05.980 INFO:tasks.workunit.client.0.smithi096.stdout:5/901: mknod d4/ddb/d3d/d4d/d88/c135 0 2024-04-22T15:19:05.982 INFO:tasks.workunit.client.1.smithi196.stdout:9/639: rmdir db/d11/d29/d57/da7 39 2024-04-22T15:19:05.986 INFO:tasks.workunit.client.0.smithi096.stdout:9/792: rename d1/d25/db2/dca/fcf to d1/d4/d79/f109 0 2024-04-22T15:19:05.990 INFO:tasks.workunit.client.0.smithi096.stdout:4/879: rmdir d5/d7/d2c/d67 39 2024-04-22T15:19:05.991 INFO:tasks.workunit.client.0.smithi096.stdout:4/880: fdatasync d5/d5b/f123 0 2024-04-22T15:19:05.993 INFO:tasks.workunit.client.0.smithi096.stdout:3/920: creat da/db/de/d38/d119/f12c x:0 0 0 2024-04-22T15:19:05.996 INFO:tasks.workunit.client.0.smithi096.stdout:5/902: creat d4/d1f/d8b/d106/f136 x:0 0 0 2024-04-22T15:19:06.000 INFO:tasks.workunit.client.0.smithi096.stdout:4/881: symlink d5/d7/d1f/d84/d6d/d118/l135 0 2024-04-22T15:19:06.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.000+0000 7fba7dd0f700 1 -- 172.21.15.96:0/1715898492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 msgr2=0x7fba78104310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:06.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.000+0000 7fba7dd0f700 1 --2- 172.21.15.96:0/1715898492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 0x7fba78104310 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fba68009910 tx=0x7fba68009c20 comp rx=0 tx=0).stop 2024-04-22T15:19:06.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.000+0000 7fba7dd0f700 1 -- 172.21.15.96:0/1715898492 shutdown_connections 2024-04-22T15:19:06.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.000+0000 7fba7dd0f700 1 --2- 172.21.15.96:0/1715898492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 0x7fba78104310 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.000+0000 7fba7dd0f700 1 --2- 172.21.15.96:0/1715898492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba780ff510 0x7fba78101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.000+0000 7fba7dd0f700 1 -- 172.21.15.96:0/1715898492 >> 172.21.15.96:0/1715898492 conn(0x7fba780fb110 msgr2=0x7fba780fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:06.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.000+0000 7fba7dd0f700 1 -- 172.21.15.96:0/1715898492 shutdown_connections 2024-04-22T15:19:06.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.000+0000 7fba7dd0f700 1 -- 172.21.15.96:0/1715898492 wait complete. 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.001+0000 7fba7dd0f700 1 Processor -- start 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.001+0000 7fba7dd0f700 1 -- start start 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.002+0000 7fba7dd0f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba780ff510 0x7fba7819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.002+0000 7fba7dd0f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 0x7fba7819d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.002+0000 7fba7dd0f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7819d7f0 con 0x7fba78101e80 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.002+0000 7fba7dd0f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba781a21b0 con 0x7fba780ff510 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.002+0000 7fba76ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 0x7fba7819d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.002+0000 7fba76ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 0x7fba7819d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34830/0 (socket says 172.21.15.96:34830) 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.002+0000 7fba76ffd700 1 -- 172.21.15.96:0/3510739948 learned_addr learned my addr 172.21.15.96:0/3510739948 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.002+0000 7fba777fe700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba780ff510 0x7fba7819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:06.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.003+0000 7fba76ffd700 1 -- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba780ff510 msgr2=0x7fba7819cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:06.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.003+0000 7fba76ffd700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba780ff510 0x7fba7819cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.003+0000 7fba76ffd700 1 -- 172.21.15.96:0/3510739948 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba680095f0 con 0x7fba78101e80 2024-04-22T15:19:06.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.003+0000 7fba777fe700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba780ff510 0x7fba7819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:19:06.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.003+0000 7fba76ffd700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 0x7fba7819d1d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fba6800bd70 tx=0x7fba6800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:06.007 INFO:tasks.workunit.client.1.smithi196.stdout:0/691: symlink d3/db4/lf2 0 2024-04-22T15:19:06.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.003+0000 7fba74ff9700 1 -- 172.21.15.96:0/3510739948 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba6801d070 con 0x7fba78101e80 2024-04-22T15:19:06.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.003+0000 7fba74ff9700 1 -- 172.21.15.96:0/3510739948 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba6800f460 con 0x7fba78101e80 2024-04-22T15:19:06.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.003+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba781a2350 con 0x7fba78101e80 2024-04-22T15:19:06.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.003+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba781a2810 con 0x7fba78101e80 2024-04-22T15:19:06.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.004+0000 7fba74ff9700 1 -- 172.21.15.96:0/3510739948 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba68021640 con 0x7fba78101e80 2024-04-22T15:19:06.009 INFO:tasks.workunit.client.0.smithi096.stdout:2/908: mknod d3/db/df/c126 0 2024-04-22T15:19:06.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.005+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba640052f0 con 0x7fba78101e80 2024-04-22T15:19:06.009 INFO:tasks.workunit.client.0.smithi096.stdout:2/909: readlink d3/db/df/dce/d46/l4c 0 2024-04-22T15:19:06.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.006+0000 7fba74ff9700 1 -- 172.21.15.96:0/3510739948 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fba6800f5d0 con 0x7fba78101e80 2024-04-22T15:19:06.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.007+0000 7fba74ff9700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fba6006da70 0x7fba6006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:06.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.007+0000 7fba74ff9700 1 -- 172.21.15.96:0/3510739948 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fba6808e880 con 0x7fba78101e80 2024-04-22T15:19:06.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.007+0000 7fba777fe700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fba6006da70 0x7fba6006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:06.011 INFO:tasks.workunit.client.1.smithi196.stdout:3/655: write d0/d15/d31/f53 [1255453,96447] 0 2024-04-22T15:19:06.011 INFO:tasks.workunit.client.1.smithi196.stdout:5/695: creat d1/d17/d7f/fe0 x:0 0 0 2024-04-22T15:19:06.011 INFO:tasks.workunit.client.1.smithi196.stdout:5/696: write d1/d4/f9c [2212248,99982] 0 2024-04-22T15:19:06.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.008+0000 7fba777fe700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fba6006da70 0x7fba6006ff30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fba6c005950 tx=0x7fba6c00a300 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:06.012 INFO:tasks.workunit.client.0.smithi096.stdout:3/921: write da/db/f3d [1482137,7717] 0 2024-04-22T15:19:06.014 INFO:tasks.workunit.client.0.smithi096.stdout:3/922: stat da/db/de/ddc/d26/db2/fc2 0 2024-04-22T15:19:06.014 INFO:tasks.workunit.client.0.smithi096.stdout:9/793: symlink d1/df/d95/d9d/db5/l10a 0 2024-04-22T15:19:06.015 INFO:tasks.workunit.client.1.smithi196.stdout:3/656: dwrite d0/d15/da7/d18/f42 [0,4194304] 0 2024-04-22T15:19:06.016 INFO:tasks.workunit.client.1.smithi196.stdout:1/646: rmdir d1/db/d16/d1e/d49/d7f 39 2024-04-22T15:19:06.018 INFO:tasks.workunit.client.0.smithi096.stdout:2/910: dread d3/db/f12 [0,4194304] 0 2024-04-22T15:19:06.018 INFO:tasks.workunit.client.0.smithi096.stdout:4/882: dread d5/d7/d1f/d84/d6d/fdb [0,4194304] 0 2024-04-22T15:19:06.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.016+0000 7fba74ff9700 1 -- 172.21.15.96:0/3510739948 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fba68057b60 con 0x7fba78101e80 2024-04-22T15:19:06.020 INFO:tasks.workunit.client.1.smithi196.stdout:4/642: dwrite d6/d13/f1b [4194304,4194304] 0 2024-04-22T15:19:06.020 INFO:tasks.workunit.client.1.smithi196.stdout:3/657: truncate d0/d15/d31/dd0/d51/fa9 1261563 0 2024-04-22T15:19:06.020 INFO:tasks.workunit.client.1.smithi196.stdout:6/692: truncate dd/d1f/d38/f79 738708 0 2024-04-22T15:19:06.022 INFO:tasks.workunit.client.0.smithi096.stdout:9/794: mknod d1/df/d95/d9d/db5/dd/dba/c10b 0 2024-04-22T15:19:06.024 INFO:tasks.workunit.client.1.smithi196.stdout:2/681: rename d1/d23/d27/d2e/f58 to d1/d13/d25/ff9 0 2024-04-22T15:19:06.026 INFO:tasks.workunit.client.0.smithi096.stdout:5/903: write d4/ddb/d25/fd2 [151969,25113] 0 2024-04-22T15:19:06.027 INFO:tasks.workunit.client.0.smithi096.stdout:3/923: mkdir da/db/de/d38/d114/d12d 0 2024-04-22T15:19:06.031 INFO:tasks.workunit.client.1.smithi196.stdout:0/692: symlink d3/d9/d36/d9f/dd1/d65/lf3 0 2024-04-22T15:19:06.033 INFO:tasks.workunit.client.0.smithi096.stdout:2/911: symlink d3/db/df/d24/d58/d5f/dfe/d115/d2f/l127 0 2024-04-22T15:19:06.034 INFO:tasks.workunit.client.0.smithi096.stdout:9/795: mknod d1/d25/c10c 0 2024-04-22T15:19:06.035 INFO:tasks.workunit.client.0.smithi096.stdout:9/796: fdatasync d1/d4/d7b/fdf 0 2024-04-22T15:19:06.039 INFO:tasks.workunit.client.0.smithi096.stdout:5/904: mknod d4/d2c/d2f/d6b/c137 0 2024-04-22T15:19:06.040 INFO:tasks.workunit.client.0.smithi096.stdout:5/905: chown d4/ddb/d3d/d35/d57/l64 3046195 1 2024-04-22T15:19:06.046 INFO:tasks.workunit.client.1.smithi196.stdout:9/640: sync 2024-04-22T15:19:06.048 INFO:tasks.workunit.client.0.smithi096.stdout:2/912: mkdir d3/db/d10d/d128 0 2024-04-22T15:19:06.051 INFO:tasks.workunit.client.0.smithi096.stdout:4/883: dwrite d5/d7/d2c/d95/dee/ff8 [0,4194304] 0 2024-04-22T15:19:06.054 INFO:tasks.workunit.client.1.smithi196.stdout:3/658: mkdir d0/d15/d31/dd0/d51/ddd 0 2024-04-22T15:19:06.059 INFO:tasks.workunit.client.1.smithi196.stdout:1/647: rename d1/f78 to d1/db/d97/fd3 0 2024-04-22T15:19:06.060 INFO:tasks.workunit.client.1.smithi196.stdout:4/643: write d6/d13/d15/f5e [413036,79296] 0 2024-04-22T15:19:06.060 INFO:tasks.workunit.client.1.smithi196.stdout:1/648: write d1/db/dc/dad/d36/f3f [3716702,68181] 0 2024-04-22T15:19:06.062 INFO:tasks.workunit.client.0.smithi096.stdout:3/924: dwrite da/db/de/ddc/d26/d52/f6c [8388608,4194304] 0 2024-04-22T15:19:06.066 INFO:tasks.workunit.client.0.smithi096.stdout:2/913: unlink d3/db/df/d10/f11 0 2024-04-22T15:19:06.069 INFO:tasks.workunit.client.0.smithi096.stdout:2/914: chown d3/db/d30/l97 40681690 1 2024-04-22T15:19:06.071 INFO:tasks.workunit.client.1.smithi196.stdout:5/697: creat d1/d4/d7b/fe1 x:0 0 0 2024-04-22T15:19:06.073 INFO:tasks.workunit.client.0.smithi096.stdout:3/925: dwrite da/db/d10/f11b [0,4194304] 0 2024-04-22T15:19:06.075 INFO:tasks.workunit.client.1.smithi196.stdout:7/639: creat d14/d26/fd2 x:0 0 0 2024-04-22T15:19:06.078 INFO:tasks.workunit.client.1.smithi196.stdout:8/666: getdents d2/d5/d2e 0 2024-04-22T15:19:06.080 INFO:tasks.workunit.client.1.smithi196.stdout:1/649: dread d1/db/d16/f40 [0,4194304] 0 2024-04-22T15:19:06.081 INFO:tasks.workunit.client.0.smithi096.stdout:2/915: fsync d3/db/df/d24/fe0 0 2024-04-22T15:19:06.083 INFO:tasks.workunit.client.1.smithi196.stdout:7/640: dread d14/d26/d2a/d2b/f70 [0,4194304] 0 2024-04-22T15:19:06.083 INFO:tasks.workunit.client.0.smithi096.stdout:2/916: write d3/db/df/d24/d58/d5f/dfe/d115/d5a/f120 [195731,46751] 0 2024-04-22T15:19:06.084 INFO:tasks.workunit.client.0.smithi096.stdout:3/926: mknod da/db/d12/c12e 0 2024-04-22T15:19:06.085 INFO:tasks.workunit.client.0.smithi096.stdout:5/906: dread d4/ddb/d3d/d4d/d59/d5a/f67 [0,4194304] 0 2024-04-22T15:19:06.085 INFO:tasks.workunit.client.1.smithi196.stdout:9/641: write db/dd/d1b/d2c/f43 [231333,90572] 0 2024-04-22T15:19:06.085 INFO:tasks.workunit.client.1.smithi196.stdout:6/693: mkdir dd/d1f/d37/dcb 0 2024-04-22T15:19:06.086 INFO:tasks.workunit.client.0.smithi096.stdout:2/917: chown d3/db/df/f60 18931 1 2024-04-22T15:19:06.086 INFO:tasks.workunit.client.0.smithi096.stdout:2/918: fdatasync d3/fd 0 2024-04-22T15:19:06.087 INFO:tasks.workunit.client.1.smithi196.stdout:2/682: dwrite d1/d1d/f7a [0,4194304] 0 2024-04-22T15:19:06.090 INFO:tasks.workunit.client.1.smithi196.stdout:3/659: mknod d0/d15/d31/d55/dcf/d6f/db5/cde 0 2024-04-22T15:19:06.091 INFO:tasks.workunit.client.0.smithi096.stdout:4/884: dread d5/d7/d1a/f90 [0,4194304] 0 2024-04-22T15:19:06.092 INFO:tasks.workunit.client.0.smithi096.stdout:9/797: getdents d1/df/d95/d9d/db5 0 2024-04-22T15:19:06.093 INFO:tasks.workunit.client.0.smithi096.stdout:4/885: stat d5/d7/d1a/d25/c31 0 2024-04-22T15:19:06.093 INFO:tasks.workunit.client.1.smithi196.stdout:1/650: dwrite d1/db/dc/de/d43/f6b [0,4194304] 0 2024-04-22T15:19:06.094 INFO:tasks.workunit.client.1.smithi196.stdout:4/644: creat d6/d13/d15/d2c/d62/d78/fcc x:0 0 0 2024-04-22T15:19:06.094 INFO:tasks.workunit.client.0.smithi096.stdout:5/907: readlink d4/ddb/d3d/d4d/d59/d5a/l6a 0 2024-04-22T15:19:06.097 INFO:tasks.workunit.client.1.smithi196.stdout:0/693: dwrite d3/de/d1c/f31 [0,4194304] 0 2024-04-22T15:19:06.097 INFO:tasks.workunit.client.0.smithi096.stdout:9/798: mknod d1/df/d95/d9d/db5/d43/d57/d5b/d73/c10d 0 2024-04-22T15:19:06.098 INFO:tasks.workunit.client.0.smithi096.stdout:4/886: dread d5/d7/d2c/d95/dee/ff8 [0,4194304] 0 2024-04-22T15:19:06.099 INFO:tasks.workunit.client.0.smithi096.stdout:4/887: dread - d5/d7/d1a/f133 zero size 2024-04-22T15:19:06.099 INFO:tasks.workunit.client.1.smithi196.stdout:7/641: dread d14/f33 [0,4194304] 0 2024-04-22T15:19:06.099 INFO:tasks.workunit.client.1.smithi196.stdout:5/698: unlink d1/d17/d1a/d2f/d30/d5e/fd0 0 2024-04-22T15:19:06.105 INFO:tasks.workunit.client.0.smithi096.stdout:5/908: rename d4/ddb/d3d/d4d/d59/d5a/d5c/f71 to d4/ddb/d3d/f138 0 2024-04-22T15:19:06.109 INFO:tasks.workunit.client.0.smithi096.stdout:5/909: truncate d4/ddb/d3d/d4d/d59/d98/db9/fc4 5184783 0 2024-04-22T15:19:06.109 INFO:tasks.workunit.client.0.smithi096.stdout:9/799: read d1/d4/f7a [3984326,86653] 0 2024-04-22T15:19:06.112 INFO:tasks.workunit.client.0.smithi096.stdout:9/800: dread - d1/df/d95/d9d/db5/d43/fe0 zero size 2024-04-22T15:19:06.112 INFO:tasks.workunit.client.1.smithi196.stdout:2/683: dread d1/f80 [4194304,4194304] 0 2024-04-22T15:19:06.114 INFO:tasks.workunit.client.1.smithi196.stdout:9/642: chown db/dd/l8e 1391895 1 2024-04-22T15:19:06.117 INFO:tasks.workunit.client.0.smithi096.stdout:5/910: dwrite d4/d2c/f4e [4194304,4194304] 0 2024-04-22T15:19:06.118 INFO:tasks.workunit.client.0.smithi096.stdout:5/911: stat d4/ddb/d3d/d4d/f6e 0 2024-04-22T15:19:06.124 INFO:tasks.workunit.client.1.smithi196.stdout:4/645: mkdir d6/d13/d15/d2c/d62/dcd 0 2024-04-22T15:19:06.125 INFO:tasks.workunit.client.1.smithi196.stdout:4/646: chown d6/d14/d18/d26/d32/d53/f54 111 1 2024-04-22T15:19:06.126 INFO:tasks.workunit.client.1.smithi196.stdout:4/647: dread - d6/d10/fb1 zero size 2024-04-22T15:19:06.127 INFO:tasks.workunit.client.1.smithi196.stdout:1/651: mknod d1/db/dc/de/d5a/d5b/d85/cd4 0 2024-04-22T15:19:06.128 INFO:tasks.workunit.client.1.smithi196.stdout:1/652: write d1/db/dc/de/d43/f6b [4581286,39063] 0 2024-04-22T15:19:06.129 INFO:tasks.workunit.client.0.smithi096.stdout:5/912: mknod d4/ddb/d3d/d4d/d88/de5/c139 0 2024-04-22T15:19:06.131 INFO:tasks.workunit.client.0.smithi096.stdout:9/801: truncate d1/d25/d9f/fcd 1323479 0 2024-04-22T15:19:06.131 INFO:tasks.workunit.client.0.smithi096.stdout:4/888: write d5/d7/d1f/d84/d6d/d98/fe8 [464950,98768] 0 2024-04-22T15:19:06.131 INFO:tasks.workunit.client.1.smithi196.stdout:1/653: dread d1/db/d16/d1e/d49/d90/fb4 [0,4194304] 0 2024-04-22T15:19:06.133 INFO:tasks.workunit.client.1.smithi196.stdout:7/642: creat d14/d26/d95/fd3 x:0 0 0 2024-04-22T15:19:06.135 INFO:tasks.workunit.client.0.smithi096.stdout:4/889: mkdir d5/d7/d1a/d8b/d136 0 2024-04-22T15:19:06.138 INFO:tasks.workunit.client.0.smithi096.stdout:9/802: read d1/df/d95/d9d/db5/d43/d57/d5b/fdb [447123,88563] 0 2024-04-22T15:19:06.138 INFO:tasks.workunit.client.1.smithi196.stdout:3/660: dwrite d0/d15/da7/d18/f1b [8388608,4194304] 0 2024-04-22T15:19:06.139 INFO:tasks.workunit.client.0.smithi096.stdout:5/913: mkdir d4/ddb/d3d/d35/db3/d13a 0 2024-04-22T15:19:06.142 INFO:tasks.workunit.client.1.smithi196.stdout:2/684: creat d1/d13/d25/ffa x:0 0 0 2024-04-22T15:19:06.145 INFO:tasks.workunit.client.1.smithi196.stdout:3/661: dread d0/d15/da7/d4a/f52 [0,4194304] 0 2024-04-22T15:19:06.145 INFO:tasks.workunit.client.1.smithi196.stdout:3/662: chown d0/d15/d4d/d87/da6 5 1 2024-04-22T15:19:06.148 INFO:tasks.workunit.client.1.smithi196.stdout:9/643: mknod db/cdc 0 2024-04-22T15:19:06.156 INFO:tasks.workunit.client.0.smithi096.stdout:9/803: creat d1/df/d95/d9d/db5/dd/dba/dc2/dc4/f10e x:0 0 0 2024-04-22T15:19:06.157 INFO:tasks.workunit.client.1.smithi196.stdout:8/667: dwrite d2/fa [0,4194304] 0 2024-04-22T15:19:06.163 INFO:tasks.workunit.client.1.smithi196.stdout:1/654: unlink d1/db/dc/d26/f27 0 2024-04-22T15:19:06.164 INFO:tasks.workunit.client.0.smithi096.stdout:5/914: dread d4/f1c [0,4194304] 0 2024-04-22T15:19:06.168 INFO:tasks.workunit.client.0.smithi096.stdout:4/890: creat d5/d7/d1a/f137 x:0 0 0 2024-04-22T15:19:06.175 INFO:tasks.workunit.client.1.smithi196.stdout:2/685: rename d1/d23/lef to d1/d13/d25/d2a/ddb/lfb 0 2024-04-22T15:19:06.177 INFO:tasks.workunit.client.1.smithi196.stdout:3/663: creat d0/d15/da7/fdf x:0 0 0 2024-04-22T15:19:06.179 INFO:tasks.workunit.client.1.smithi196.stdout:6/694: sync 2024-04-22T15:19:06.180 INFO:tasks.workunit.client.1.smithi196.stdout:6/695: readlink dd/d1f/l35 0 2024-04-22T15:19:06.182 INFO:tasks.workunit.client.1.smithi196.stdout:4/648: write d6/d13/d15/fb2 [1328073,104345] 0 2024-04-22T15:19:06.182 INFO:tasks.workunit.client.1.smithi196.stdout:6/696: fsync dd/d1f/d37/d3c/d96/fb3 0 2024-04-22T15:19:06.184 INFO:tasks.workunit.client.1.smithi196.stdout:9/644: dread db/d11/d29/f48 [0,4194304] 0 2024-04-22T15:19:06.185 INFO:tasks.workunit.client.1.smithi196.stdout:9/645: write db/f61 [1240444,8807] 0 2024-04-22T15:19:06.192 INFO:tasks.workunit.client.0.smithi096.stdout:5/915: write d4/fcc [1075321,29608] 0 2024-04-22T15:19:06.193 INFO:tasks.workunit.client.0.smithi096.stdout:9/804: dwrite d1/d4/f22 [0,4194304] 0 2024-04-22T15:19:06.198 INFO:tasks.workunit.client.0.smithi096.stdout:4/891: write d5/d7/d1f/d84/d6d/fcd [806100,7242] 0 2024-04-22T15:19:06.201 INFO:tasks.workunit.client.0.smithi096.stdout:9/805: mknod d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/c10f 0 2024-04-22T15:19:06.203 INFO:tasks.workunit.client.0.smithi096.stdout:5/916: mknod d4/ddb/d3d/d4d/d59/d8a/d121/d134/c13b 0 2024-04-22T15:19:06.205 INFO:tasks.workunit.client.0.smithi096.stdout:4/892: dread - d5/fce zero size 2024-04-22T15:19:06.207 INFO:tasks.workunit.client.0.smithi096.stdout:9/806: creat d1/d25/f110 x:0 0 0 2024-04-22T15:19:06.209 INFO:tasks.workunit.client.0.smithi096.stdout:9/807: chown d1/df/d95/d9d/db5/d43/d57/d5b/f76 32538563 1 2024-04-22T15:19:06.209 INFO:tasks.workunit.client.0.smithi096.stdout:5/917: mknod d4/ddb/d3d/d4d/d59/d98/dee/c13c 0 2024-04-22T15:19:06.212 INFO:tasks.workunit.client.0.smithi096.stdout:5/918: fsync d4/ddb/d3d/f128 0 2024-04-22T15:19:06.214 INFO:tasks.workunit.client.0.smithi096.stdout:9/808: truncate d1/d4/f47 2597844 0 2024-04-22T15:19:06.217 INFO:tasks.workunit.client.0.smithi096.stdout:4/893: rename d5/d5b/d11f to d5/d7/dcb/d138 0 2024-04-22T15:19:06.220 INFO:tasks.workunit.client.0.smithi096.stdout:5/919: symlink d4/d2c/d2f/d31/d82/def/l13d 0 2024-04-22T15:19:06.221 INFO:tasks.workunit.client.0.smithi096.stdout:5/920: chown d4/d2c/d2f/d30/fcf 577 1 2024-04-22T15:19:06.230 INFO:tasks.workunit.client.0.smithi096.stdout:9/809: unlink d1/df/l30 0 2024-04-22T15:19:06.232 INFO:tasks.workunit.client.0.smithi096.stdout:5/921: rename d4/ddb/d3d/d4d/d59/c8e to d4/ddb/d3d/d35/d57/c13e 0 2024-04-22T15:19:06.239 INFO:tasks.workunit.client.1.smithi196.stdout:8/668: symlink d2/d8/d49/d8b/lc3 0 2024-04-22T15:19:06.242 INFO:tasks.workunit.client.0.smithi096.stdout:4/894: write d5/d7/d2c/d3d/d4f/d63/f7e [1290940,120727] 0 2024-04-22T15:19:06.243 INFO:tasks.workunit.client.1.smithi196.stdout:1/655: symlink d1/db/dc/de/d52/ld5 0 2024-04-22T15:19:06.246 INFO:tasks.workunit.client.1.smithi196.stdout:0/694: sync 2024-04-22T15:19:06.247 INFO:tasks.workunit.client.1.smithi196.stdout:7/643: creat d14/d26/d2a/d2b/da7/dc6/fd4 x:0 0 0 2024-04-22T15:19:06.251 INFO:tasks.workunit.client.1.smithi196.stdout:2/686: read - d1/d13/d1f/ddf/da1/ff7 zero size 2024-04-22T15:19:06.253 INFO:tasks.workunit.client.1.smithi196.stdout:3/664: symlink d0/d15/da7/d85/le0 0 2024-04-22T15:19:06.254 INFO:tasks.workunit.client.1.smithi196.stdout:8/669: dread d2/d6a/f7e [0,4194304] 0 2024-04-22T15:19:06.255 INFO:tasks.workunit.client.1.smithi196.stdout:1/656: dread d1/db/d16/f5c [0,4194304] 0 2024-04-22T15:19:06.256 INFO:tasks.workunit.client.1.smithi196.stdout:4/649: rmdir d6/d13/d15 39 2024-04-22T15:19:06.261 INFO:tasks.workunit.client.1.smithi196.stdout:6/697: dread dd/d1f/d38/d40/f49 [0,4194304] 0 2024-04-22T15:19:06.264 INFO:tasks.workunit.client.1.smithi196.stdout:7/644: creat d14/d25/d31/fd5 x:0 0 0 2024-04-22T15:19:06.264 INFO:tasks.workunit.client.1.smithi196.stdout:7/645: chown d14/d26/d28/d29/f61 246155 1 2024-04-22T15:19:06.266 INFO:tasks.workunit.client.1.smithi196.stdout:2/687: dread - d1/d13/d25/fd0 zero size 2024-04-22T15:19:06.268 INFO:tasks.workunit.client.1.smithi196.stdout:3/665: mkdir d0/d15/d31/d55/dcf/d6f/db5/dd7/de1 0 2024-04-22T15:19:06.269 INFO:tasks.workunit.client.0.smithi096.stdout:6/736: sync 2024-04-22T15:19:06.269 INFO:tasks.workunit.client.0.smithi096.stdout:0/979: sync 2024-04-22T15:19:06.270 INFO:tasks.workunit.client.1.smithi196.stdout:0/695: dread d3/d9/d36/d9f/dd1/d65/d7f/d30/f46 [0,4194304] 0 2024-04-22T15:19:06.272 INFO:tasks.workunit.client.1.smithi196.stdout:6/698: rmdir dd/d2b/d8d 39 2024-04-22T15:19:06.273 INFO:tasks.workunit.client.1.smithi196.stdout:9/646: unlink db/dd/f9e 0 2024-04-22T15:19:06.275 INFO:tasks.workunit.client.1.smithi196.stdout:2/688: rmdir d1/d13 39 2024-04-22T15:19:06.276 INFO:tasks.workunit.client.1.smithi196.stdout:0/696: dwrite d3/d6/dc/d8c/d8d/fc1 [0,4194304] 0 2024-04-22T15:19:06.276 INFO:tasks.workunit.client.1.smithi196.stdout:8/670: mknod d2/d5/cc4 0 2024-04-22T15:19:06.277 INFO:tasks.workunit.client.1.smithi196.stdout:3/666: rename d0/d15/d31/d55/dcf/d8/d70/dbe to d0/d15/d31/dd0/de2 0 2024-04-22T15:19:06.279 INFO:tasks.workunit.client.1.smithi196.stdout:5/699: sync 2024-04-22T15:19:06.280 INFO:tasks.workunit.client.1.smithi196.stdout:1/657: truncate d1/db/d16/d1e/f88 564504 0 2024-04-22T15:19:06.282 INFO:tasks.workunit.client.1.smithi196.stdout:9/647: dwrite db/d11/d29/d57/f23 [0,4194304] 0 2024-04-22T15:19:06.282 INFO:tasks.workunit.client.1.smithi196.stdout:4/650: mkdir d6/d13/d15/d2c/d8c/db3/dce 0 2024-04-22T15:19:06.285 INFO:tasks.workunit.client.1.smithi196.stdout:7/646: mkdir d14/d25/dd6 0 2024-04-22T15:19:06.286 INFO:tasks.workunit.client.1.smithi196.stdout:7/647: write d14/d26/d2a/d72/db1/fc5 [63578,126705] 0 2024-04-22T15:19:06.287 INFO:tasks.workunit.client.1.smithi196.stdout:7/648: truncate d14/d26/d2a/d2b/fb9 142264 0 2024-04-22T15:19:06.292 INFO:tasks.workunit.client.1.smithi196.stdout:8/671: creat d2/d6a/d7b/fc5 x:0 0 0 2024-04-22T15:19:06.293 INFO:tasks.workunit.client.1.smithi196.stdout:0/697: dwrite d3/f47 [0,4194304] 0 2024-04-22T15:19:06.296 INFO:tasks.workunit.client.1.smithi196.stdout:6/699: rename dd/d20/f29 to dd/d1f/d37/d3c/fcc 0 2024-04-22T15:19:06.300 INFO:tasks.workunit.client.1.smithi196.stdout:3/667: read d0/d15/d4d/f2d [2318443,90544] 0 2024-04-22T15:19:06.301 INFO:tasks.workunit.client.1.smithi196.stdout:1/658: stat d1/db/dc/de/d96/lbb 0 2024-04-22T15:19:06.302 INFO:tasks.workunit.client.1.smithi196.stdout:4/651: creat d6/d13/d15/d2c/d8c/fcf x:0 0 0 2024-04-22T15:19:06.302 INFO:tasks.workunit.client.1.smithi196.stdout:3/668: truncate d0/d15/d31/f53 1740638 0 2024-04-22T15:19:06.303 INFO:tasks.workunit.client.0.smithi096.stdout:9/810: creat d1/d4/d7f/f111 x:0 0 0 2024-04-22T15:19:06.308 INFO:tasks.workunit.client.0.smithi096.stdout:4/895: creat d5/d7/dcb/f139 x:0 0 0 2024-04-22T15:19:06.311 INFO:tasks.workunit.client.1.smithi196.stdout:7/649: chown d14/d15/d91/d9a/l9d 76745 1 2024-04-22T15:19:06.312 INFO:tasks.workunit.client.0.smithi096.stdout:0/980: readlink d7/d42/d4c/d55/d9c/dfc/l102 0 2024-04-22T15:19:06.313 INFO:tasks.workunit.client.1.smithi196.stdout:8/672: creat d2/d8/d49/d6c/fc6 x:0 0 0 2024-04-22T15:19:06.317 INFO:tasks.workunit.client.0.smithi096.stdout:9/811: mkdir d1/d25/d9f/d112 0 2024-04-22T15:19:06.317 INFO:tasks.workunit.client.0.smithi096.stdout:9/812: stat d1/d25 0 2024-04-22T15:19:06.318 INFO:tasks.workunit.client.0.smithi096.stdout:0/981: dwrite d7/de/d5b/d64/fd3 [0,4194304] 0 2024-04-22T15:19:06.319 INFO:tasks.workunit.client.0.smithi096.stdout:0/982: chown d7/d42/ff8 2957 1 2024-04-22T15:19:06.323 INFO:tasks.workunit.client.0.smithi096.stdout:4/896: mknod d5/d7/d1f/d84/d6d/d118/c13a 0 2024-04-22T15:19:06.333 INFO:tasks.workunit.client.0.smithi096.stdout:0/983: creat d7/de/d5b/d95/dc/d16/d65/f14f x:0 0 0 2024-04-22T15:19:06.334 INFO:tasks.workunit.client.0.smithi096.stdout:0/984: read - d7/de/d3a/d61/dbb/f148 zero size 2024-04-22T15:19:06.335 INFO:tasks.workunit.client.0.smithi096.stdout:0/985: chown d7/d42/d4c/d55/d9c/c147 257606547 1 2024-04-22T15:19:06.338 INFO:tasks.workunit.client.1.smithi196.stdout:2/689: creat d1/ffc x:0 0 0 2024-04-22T15:19:06.340 INFO:tasks.workunit.client.1.smithi196.stdout:8/673: fsync d2/d5/d4f/f6b 0 2024-04-22T15:19:06.343 INFO:tasks.workunit.client.0.smithi096.stdout:9/813: dread d1/df/d95/d9d/db5/f52 [0,4194304] 0 2024-04-22T15:19:06.346 INFO:tasks.workunit.client.0.smithi096.stdout:6/737: getdents d3/d28/dd/d11/d2c/d3b 0 2024-04-22T15:19:06.347 INFO:tasks.workunit.client.0.smithi096.stdout:6/738: chown d3/d3a/d75/d83/df7 84898 1 2024-04-22T15:19:06.348 INFO:tasks.workunit.client.1.smithi196.stdout:1/659: link d1/db/d16/d44/faa d1/db/dc/de/d5a/d5b/d9a/fd6 0 2024-04-22T15:19:06.350 INFO:tasks.workunit.client.1.smithi196.stdout:1/660: chown d1/db/dc/de/d5a/d94/dab 0 1 2024-04-22T15:19:06.350 INFO:tasks.workunit.client.1.smithi196.stdout:5/700: dwrite d1/d9/da9/f86 [0,4194304] 0 2024-04-22T15:19:06.351 INFO:tasks.workunit.client.1.smithi196.stdout:1/661: chown d1/db/dc/de/d52 54 1 2024-04-22T15:19:06.352 INFO:tasks.workunit.client.1.smithi196.stdout:4/652: getdents d6/d13/d15/d2c/d62/dcd 0 2024-04-22T15:19:06.355 INFO:tasks.workunit.client.0.smithi096.stdout:9/814: unlink d1/df/d95/d9d/db5/d43/d57/d5b/l68 0 2024-04-22T15:19:06.356 INFO:tasks.workunit.client.1.smithi196.stdout:9/648: link db/dd/d1b/c34 db/dd/de/d15/d31/d64/dab/cdd 0 2024-04-22T15:19:06.356 INFO:tasks.workunit.client.0.smithi096.stdout:6/739: dwrite d3/d3a/d75/fd3 [0,4194304] 0 2024-04-22T15:19:06.359 INFO:tasks.workunit.client.0.smithi096.stdout:5/922: dwrite d4/d2c/d2f/d31/f10b [0,4194304] 0 2024-04-22T15:19:06.361 INFO:tasks.workunit.client.1.smithi196.stdout:2/690: mknod d1/d23/d27/d89/cfd 0 2024-04-22T15:19:06.363 INFO:tasks.workunit.client.0.smithi096.stdout:8/878: sync 2024-04-22T15:19:06.363 INFO:tasks.workunit.client.0.smithi096.stdout:1/918: sync 2024-04-22T15:19:06.366 INFO:tasks.workunit.client.0.smithi096.stdout:8/879: chown d4/d10/dcc/f107 2214 1 2024-04-22T15:19:06.366 INFO:tasks.workunit.client.0.smithi096.stdout:1/919: stat d6/dc/d8a/d3f/d62/dc2/cf6 0 2024-04-22T15:19:06.366 INFO:tasks.workunit.client.0.smithi096.stdout:1/920: read - d6/dc/d8a/d3f/d62/d7d/dd3/f106 zero size 2024-04-22T15:19:06.367 INFO:tasks.workunit.client.0.smithi096.stdout:1/921: stat d6/dc/d8a/d122 0 2024-04-22T15:19:06.372 INFO:tasks.workunit.client.0.smithi096.stdout:5/923: readlink d4/l16 0 2024-04-22T15:19:06.373 INFO:tasks.workunit.client.0.smithi096.stdout:5/924: chown d4/ddb/d3d/d4d/d88/c8c 139565 1 2024-04-22T15:19:06.374 INFO:tasks.workunit.client.1.smithi196.stdout:8/674: unlink d2/d8/d47/d68/d81/f8d 0 2024-04-22T15:19:06.374 INFO:tasks.workunit.client.1.smithi196.stdout:6/700: link dd/d2b/f77 dd/d1f/d38/d40/fcd 0 2024-04-22T15:19:06.376 INFO:tasks.workunit.client.0.smithi096.stdout:8/880: creat d4/d10/f127 x:0 0 0 2024-04-22T15:19:06.379 INFO:tasks.workunit.client.1.smithi196.stdout:0/698: dwrite d3/f5 [0,4194304] 0 2024-04-22T15:19:06.379 INFO:tasks.workunit.client.1.smithi196.stdout:3/669: dwrite d0/d15/f7a [0,4194304] 0 2024-04-22T15:19:06.381 INFO:tasks.workunit.client.0.smithi096.stdout:1/922: rmdir d6/dc/d8a/d3f/d62/d7d/ddb/d9b 39 2024-04-22T15:19:06.382 INFO:tasks.workunit.client.1.smithi196.stdout:1/662: mkdir d1/db/d97/dd7 0 2024-04-22T15:19:06.382 INFO:tasks.workunit.client.0.smithi096.stdout:4/897: dwrite d5/d7/d2c/d3d/d4f/fb6 [0,4194304] 0 2024-04-22T15:19:06.384 INFO:tasks.workunit.client.0.smithi096.stdout:1/923: write d6/dc/d8a/d3f/d54/f18 [2116605,13494] 0 2024-04-22T15:19:06.385 INFO:tasks.workunit.client.1.smithi196.stdout:4/653: fdatasync d6/d14/d18/d26/d32/d67/f77 0 2024-04-22T15:19:06.387 INFO:tasks.workunit.client.0.smithi096.stdout:0/986: truncate d7/de/d5b/d95/d15/f13f 3699513 0 2024-04-22T15:19:06.389 INFO:tasks.workunit.client.1.smithi196.stdout:9/649: fsync db/d11/d29/d57/f33 0 2024-04-22T15:19:06.389 INFO:tasks.workunit.client.1.smithi196.stdout:7/650: sync 2024-04-22T15:19:06.390 INFO:tasks.workunit.client.0.smithi096.stdout:8/881: readlink d4/d10/d25/d4c/db4/lf2 0 2024-04-22T15:19:06.393 INFO:tasks.workunit.client.0.smithi096.stdout:5/925: dread d4/d2c/d2f/f3a [0,4194304] 0 2024-04-22T15:19:06.394 INFO:tasks.workunit.client.1.smithi196.stdout:3/670: dwrite d0/d15/da7/d4a/f50 [4194304,4194304] 0 2024-04-22T15:19:06.394 INFO:tasks.workunit.client.1.smithi196.stdout:1/663: dread d1/db/dc/de/d43/d5e/fa7 [0,4194304] 0 2024-04-22T15:19:06.395 INFO:tasks.workunit.client.0.smithi096.stdout:5/926: readlink d4/ddb/d3d/d35/d57/l65 0 2024-04-22T15:19:06.396 INFO:tasks.workunit.client.1.smithi196.stdout:1/664: readlink d1/db/dc/de/d5a/d5b/d6e/laf 0 2024-04-22T15:19:06.400 INFO:tasks.workunit.client.1.smithi196.stdout:3/671: write d0/d15/d31/d55/dcf/d8/fd4 [436309,108532] 0 2024-04-22T15:19:06.400 INFO:tasks.workunit.client.1.smithi196.stdout:6/701: creat dd/d1f/d37/d58/fce x:0 0 0 2024-04-22T15:19:06.401 INFO:tasks.workunit.client.0.smithi096.stdout:6/740: creat d3/d28/d4d/ff9 x:0 0 0 2024-04-22T15:19:06.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.400+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba64000bc0 con 0x7fba6006da70 2024-04-22T15:19:06.404 INFO:tasks.workunit.client.0.smithi096.stdout:3/927: sync 2024-04-22T15:19:06.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.402+0000 7fba74ff9700 1 -- 172.21.15.96:0/3510739948 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fba64000bc0 con 0x7fba6006da70 2024-04-22T15:19:06.405 INFO:tasks.workunit.client.0.smithi096.stdout:3/928: stat da/db/d9a 0 2024-04-22T15:19:06.407 INFO:tasks.workunit.client.0.smithi096.stdout:0/987: rename d7/de/d5b/d62/ff6 to d7/d42/d4c/d55/d9c/df7/dfb/f150 0 2024-04-22T15:19:06.408 INFO:tasks.workunit.client.0.smithi096.stdout:0/988: fdatasync d7/f20 0 2024-04-22T15:19:06.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.405+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fba6006da70 msgr2=0x7fba6006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:06.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.405+0000 7fba7dd0f700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fba6006da70 0x7fba6006ff30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fba6c005950 tx=0x7fba6c00a300 comp rx=0 tx=0).stop 2024-04-22T15:19:06.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.405+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 msgr2=0x7fba7819d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:06.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.405+0000 7fba7dd0f700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 0x7fba7819d1d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fba6800bd70 tx=0x7fba6800bda0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.406+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 shutdown_connections 2024-04-22T15:19:06.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.406+0000 7fba7dd0f700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fba6006da70 0x7fba6006ff30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.406+0000 7fba7dd0f700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba780ff510 0x7fba7819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.406+0000 7fba7dd0f700 1 --2- 172.21.15.96:0/3510739948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101e80 0x7fba7819d1d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.406+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 >> 172.21.15.96:0/3510739948 conn(0x7fba780fb110 msgr2=0x7fba780fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:06.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.406+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 shutdown_connections 2024-04-22T15:19:06.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.406+0000 7fba7dd0f700 1 -- 172.21.15.96:0/3510739948 wait complete. 2024-04-22T15:19:06.412 INFO:tasks.workunit.client.1.smithi196.stdout:0/699: symlink d3/d6/dc/d8c/d8d/d98/lf4 0 2024-04-22T15:19:06.413 INFO:tasks.workunit.client.1.smithi196.stdout:0/700: write d3/de/f24 [1562184,12539] 0 2024-04-22T15:19:06.414 INFO:tasks.workunit.client.0.smithi096.stdout:6/741: creat d3/d28/daf/d94/ffa x:0 0 0 2024-04-22T15:19:06.416 INFO:tasks.workunit.client.0.smithi096.stdout:3/929: symlink da/db/d12/dd5/l12f 0 2024-04-22T15:19:06.417 INFO:tasks.workunit.client.1.smithi196.stdout:4/654: write d6/d14/d18/d26/f6a [5709015,58187] 0 2024-04-22T15:19:06.418 INFO:tasks.workunit.client.0.smithi096.stdout:8/882: creat d4/dc0/d123/f128 x:0 0 0 2024-04-22T15:19:06.421 INFO:tasks.workunit.client.0.smithi096.stdout:0/989: readlink d7/de/d5b/d95/d15/l106 0 2024-04-22T15:19:06.422 INFO:tasks.workunit.client.0.smithi096.stdout:9/815: link d1/d4/d7b/f81 d1/df/d95/d9d/db5/d43/d57/f113 0 2024-04-22T15:19:06.426 INFO:tasks.workunit.client.0.smithi096.stdout:1/924: dread d6/dc/d8a/d3f/f57 [0,4194304] 0 2024-04-22T15:19:06.427 INFO:tasks.workunit.client.0.smithi096.stdout:3/930: dwrite da/db/de/d87/fb6 [0,4194304] 0 2024-04-22T15:19:06.429 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:19:06.430 INFO:tasks.workunit.client.0.smithi096.stdout:5/927: creat d4/ddb/d3d/d4d/d59/db0/d126/f13f x:0 0 0 2024-04-22T15:19:06.437 INFO:tasks.workunit.client.0.smithi096.stdout:9/816: dread d1/df/d95/d9d/db5/dd/dba/f105 [0,4194304] 0 2024-04-22T15:19:06.440 INFO:tasks.workunit.client.0.smithi096.stdout:4/898: dwrite d5/d7/d1f/d84/f93 [0,4194304] 0 2024-04-22T15:19:06.444 INFO:tasks.workunit.client.0.smithi096.stdout:8/883: mknod d4/d10/dcc/c129 0 2024-04-22T15:19:06.445 INFO:tasks.workunit.client.1.smithi196.stdout:3/672: mkdir d0/d15/d31/dd0/d51/d5e/de3 0 2024-04-22T15:19:06.448 INFO:tasks.workunit.client.0.smithi096.stdout:0/990: fdatasync d7/de/d5b/d95/dc/fed 0 2024-04-22T15:19:06.449 INFO:tasks.workunit.client.1.smithi196.stdout:5/701: creat d1/d14/d6a/fe2 x:0 0 0 2024-04-22T15:19:06.450 INFO:tasks.workunit.client.0.smithi096.stdout:3/931: mknod da/db/de/ddc/d26/dc6/dd2/c130 0 2024-04-22T15:19:06.454 INFO:tasks.workunit.client.1.smithi196.stdout:5/702: dwrite d1/d17/d7f/fe0 [0,4194304] 0 2024-04-22T15:19:06.457 INFO:tasks.workunit.client.1.smithi196.stdout:0/701: symlink d3/d9/d36/d9f/dd1/d65/d7f/ddb/lf5 0 2024-04-22T15:19:06.458 INFO:tasks.workunit.client.1.smithi196.stdout:2/691: dwrite d1/d23/f6d [0,4194304] 0 2024-04-22T15:19:06.458 INFO:tasks.workunit.client.1.smithi196.stdout:6/702: write dd/d20/f28 [702051,96843] 0 2024-04-22T15:19:06.460 INFO:tasks.workunit.client.1.smithi196.stdout:4/655: symlink d6/d13/d15/d2c/d62/d78/ld0 0 2024-04-22T15:19:06.461 INFO:tasks.workunit.client.1.smithi196.stdout:4/656: chown d6/d13/d15/d2c/d35/f4c 387 1 2024-04-22T15:19:06.461 INFO:tasks.workunit.client.1.smithi196.stdout:4/657: readlink d6/d14/d18/d26/d32/d53/la2 0 2024-04-22T15:19:06.464 INFO:tasks.workunit.client.1.smithi196.stdout:7/651: mknod d14/cd7 0 2024-04-22T15:19:06.465 INFO:tasks.workunit.client.1.smithi196.stdout:7/652: chown d14/d25/d31/f63 27 1 2024-04-22T15:19:06.465 INFO:tasks.workunit.client.0.smithi096.stdout:9/817: fsync d1/d4/d7b/fa6 0 2024-04-22T15:19:06.475 INFO:tasks.workunit.client.0.smithi096.stdout:8/884: creat d4/d10/d4d/f12a x:0 0 0 2024-04-22T15:19:06.476 INFO:tasks.workunit.client.1.smithi196.stdout:8/675: creat d2/dc/fc7 x:0 0 0 2024-04-22T15:19:06.484 INFO:tasks.workunit.client.1.smithi196.stdout:5/703: write d1/d2/d98/fb6 [834221,8936] 0 2024-04-22T15:19:06.484 INFO:tasks.workunit.client.1.smithi196.stdout:9/650: dwrite db/dd/f5c [0,4194304] 0 2024-04-22T15:19:06.486 INFO:tasks.workunit.client.1.smithi196.stdout:9/651: chown db/d11/c95 37964515 1 2024-04-22T15:19:06.486 INFO:tasks.workunit.client.1.smithi196.stdout:5/704: dread - d1/d17/d1a/d2f/d30/d7d/fd7 zero size 2024-04-22T15:19:06.487 INFO:tasks.workunit.client.1.smithi196.stdout:5/705: write d1/d14/d6a/fb8 [896134,104943] 0 2024-04-22T15:19:06.488 INFO:tasks.workunit.client.0.smithi096.stdout:4/899: dwrite d5/d7/d2c/d95/dee/f94 [0,4194304] 0 2024-04-22T15:19:06.494 INFO:tasks.workunit.client.1.smithi196.stdout:0/702: rename d3/d6/dc/d8c/d8d/d98/la4 to d3/db4/lf6 0 2024-04-22T15:19:06.495 INFO:tasks.workunit.client.0.smithi096.stdout:6/742: creat d3/d28/d4d/d82/ffb x:0 0 0 2024-04-22T15:19:06.498 INFO:tasks.workunit.client.0.smithi096.stdout:9/818: fsync d1/d4/d7f/fa9 0 2024-04-22T15:19:06.498 INFO:tasks.workunit.client.1.smithi196.stdout:0/703: read d3/d9/d36/d9f/dd1/d65/d7f/d30/fe7 [642460,68982] 0 2024-04-22T15:19:06.503 INFO:tasks.workunit.client.0.smithi096.stdout:0/991: rmdir d7/d42/d4c/d55/da3 39 2024-04-22T15:19:06.504 INFO:tasks.workunit.client.1.smithi196.stdout:3/673: creat d0/d15/d31/dd0/d51/ddd/fe4 x:0 0 0 2024-04-22T15:19:06.504 INFO:tasks.workunit.client.1.smithi196.stdout:3/674: chown d0/d15/d31/dd0/d51/d5e/f63 7974 1 2024-04-22T15:19:06.505 INFO:tasks.workunit.client.1.smithi196.stdout:3/675: stat d0/d15/d31/d55/dcf 0 2024-04-22T15:19:06.505 INFO:tasks.workunit.client.1.smithi196.stdout:3/676: stat d0/d15/d31/cca 0 2024-04-22T15:19:06.506 INFO:tasks.workunit.client.1.smithi196.stdout:3/677: readlink d0/d15/d31/d55/dcf/d8/d6b/ldc 0 2024-04-22T15:19:06.507 INFO:tasks.workunit.client.1.smithi196.stdout:3/678: write d0/d15/d31/d55/d6e/fa1 [1034889,19369] 0 2024-04-22T15:19:06.514 INFO:tasks.workunit.client.1.smithi196.stdout:8/676: rename d2/d3f/fa5 to d2/d5/d93/fc8 0 2024-04-22T15:19:06.515 INFO:tasks.workunit.client.1.smithi196.stdout:6/703: dread dd/d1f/d37/f31 [0,4194304] 0 2024-04-22T15:19:06.522 INFO:tasks.workunit.client.0.smithi096.stdout:9/819: symlink d1/d25/db2/d98/d9b/l114 0 2024-04-22T15:19:06.523 INFO:tasks.workunit.client.1.smithi196.stdout:6/704: dwrite dd/d1f/d37/d7b/fb4 [0,4194304] 0 2024-04-22T15:19:06.524 INFO:tasks.workunit.client.1.smithi196.stdout:7/653: mknod d14/d25/dd6/cd8 0 2024-04-22T15:19:06.528 INFO:tasks.workunit.client.1.smithi196.stdout:5/706: dread d1/d9/f38 [0,4194304] 0 2024-04-22T15:19:06.529 INFO:tasks.workunit.client.1.smithi196.stdout:8/677: dread d2/d8/f73 [0,4194304] 0 2024-04-22T15:19:06.530 INFO:tasks.workunit.client.1.smithi196.stdout:9/652: dread db/dd/d1b/d2c/f43 [0,4194304] 0 2024-04-22T15:19:06.531 INFO:tasks.workunit.client.1.smithi196.stdout:9/653: chown db/d11/d29/d7a/d7b/d81/la4 735921 1 2024-04-22T15:19:06.534 INFO:tasks.workunit.client.1.smithi196.stdout:8/678: dwrite d2/dc/d45/d98/fa6 [0,4194304] 0 2024-04-22T15:19:06.534 INFO:tasks.workunit.client.0.smithi096.stdout:4/900: write d5/fc2 [553884,34170] 0 2024-04-22T15:19:06.535 INFO:tasks.workunit.client.0.smithi096.stdout:3/932: mknod da/db/de/d19/d2d/c131 0 2024-04-22T15:19:06.538 INFO:tasks.workunit.client.1.smithi196.stdout:1/665: getdents d1/db/dc/de 0 2024-04-22T15:19:06.538 INFO:tasks.workunit.client.0.smithi096.stdout:5/928: getdents d4/d2c/d2f/d31 0 2024-04-22T15:19:06.539 INFO:tasks.workunit.client.0.smithi096.stdout:5/929: readlink d4/l16 0 2024-04-22T15:19:06.543 INFO:tasks.workunit.client.0.smithi096.stdout:1/925: rename d6/dc/d8a/d3f/d54/d32/d8c/d45/f66 to d6/dc/d8a/d3f/d62/d7d/ddb/f123 0 2024-04-22T15:19:06.543 INFO:tasks.workunit.client.1.smithi196.stdout:1/666: dwrite d1/db/d16/d1e/d46/fc1 [0,4194304] 0 2024-04-22T15:19:06.547 INFO:tasks.workunit.client.0.smithi096.stdout:9/820: creat d1/df/d95/d9d/db5/dd/dba/dc2/dc4/f115 x:0 0 0 2024-04-22T15:19:06.550 INFO:tasks.workunit.client.1.smithi196.stdout:3/679: rename d0/d15/d31/dd0/d51/d5e/de3 to d0/d15/d31/d55/dcf/d6f/de5 0 2024-04-22T15:19:06.553 INFO:tasks.workunit.client.0.smithi096.stdout:0/992: dwrite d7/de/d5b/d64/f125 [0,4194304] 0 2024-04-22T15:19:06.555 INFO:tasks.workunit.client.0.smithi096.stdout:6/743: rename d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/f88 to d3/d28/d54/ffc 0 2024-04-22T15:19:06.556 INFO:tasks.workunit.client.1.smithi196.stdout:7/654: creat d14/d26/d2a/d72/db1/fd9 x:0 0 0 2024-04-22T15:19:06.557 INFO:tasks.workunit.client.1.smithi196.stdout:4/658: sync 2024-04-22T15:19:06.558 INFO:tasks.workunit.client.1.smithi196.stdout:6/705: unlink dd/d1f/d4e/d53/d7f/f81 0 2024-04-22T15:19:06.559 INFO:tasks.workunit.client.1.smithi196.stdout:6/706: fsync dd/f23 0 2024-04-22T15:19:06.560 INFO:tasks.workunit.client.1.smithi196.stdout:5/707: creat d1/d17/d1a/d2f/d30/d6e/d7a/fe3 x:0 0 0 2024-04-22T15:19:06.564 INFO:tasks.workunit.client.0.smithi096.stdout:6/744: dwrite d3/d28/d4d/d82/dd8/fe8 [0,4194304] 0 2024-04-22T15:19:06.564 INFO:tasks.workunit.client.1.smithi196.stdout:0/704: mkdir d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8/df7 0 2024-04-22T15:19:06.568 INFO:tasks.workunit.client.0.smithi096.stdout:5/930: rename d4/ddb/d3d/d4d/d59/d8a/f12f to d4/ddb/d3d/d4f/f140 0 2024-04-22T15:19:06.571 INFO:tasks.workunit.client.0.smithi096.stdout:5/931: read d4/ddb/d3d/d4d/d59/d98/db9/fc4 [3358233,22844] 0 2024-04-22T15:19:06.575 INFO:tasks.workunit.client.1.smithi196.stdout:1/667: creat d1/db/dc/dad/d36/fd8 x:0 0 0 2024-04-22T15:19:06.576 INFO:tasks.workunit.client.1.smithi196.stdout:1/668: fdatasync d1/db/d16/d44/fb8 0 2024-04-22T15:19:06.577 INFO:tasks.workunit.client.0.smithi096.stdout:3/933: write da/db/de/ddc/f88 [2127478,14495] 0 2024-04-22T15:19:06.578 INFO:tasks.workunit.client.0.smithi096.stdout:4/901: dread d5/d7/d2c/d3d/d4f/f5c [0,4194304] 0 2024-04-22T15:19:06.580 INFO:tasks.workunit.client.0.smithi096.stdout:4/902: fsync d5/d7/d1f/d84/d6d/d98/fe8 0 2024-04-22T15:19:06.582 INFO:tasks.workunit.client.0.smithi096.stdout:0/993: mknod d7/d42/d4c/d55/d149/c151 0 2024-04-22T15:19:06.591 INFO:tasks.workunit.client.0.smithi096.stdout:9/821: dwrite d1/df/d95/d9d/db5/dd/dba/dc2/dc4/fef [0,4194304] 0 2024-04-22T15:19:06.594 INFO:tasks.workunit.client.1.smithi196.stdout:4/659: creat d6/d13/d15/d2c/d62/d78/fd1 x:0 0 0 2024-04-22T15:19:06.600 INFO:tasks.workunit.client.1.smithi196.stdout:3/680: dwrite d0/d15/da7/d85/f94 [0,4194304] 0 2024-04-22T15:19:06.602 INFO:tasks.workunit.client.1.smithi196.stdout:3/681: read - d0/d15/da7/d85/f91 zero size 2024-04-22T15:19:06.602 INFO:tasks.workunit.client.1.smithi196.stdout:3/682: chown d0/d15/d31/d55/dcf/f5b 674561 1 2024-04-22T15:19:06.603 INFO:tasks.workunit.client.1.smithi196.stdout:3/683: fsync d0/d15/d31/dd0/fae 0 2024-04-22T15:19:06.604 INFO:tasks.workunit.client.1.smithi196.stdout:3/684: fsync d0/d15/da7/d4a/d44/f16 0 2024-04-22T15:19:06.604 INFO:tasks.workunit.client.0.smithi096.stdout:3/934: rename da/db/de/d8b to da/db/d9a/d132 0 2024-04-22T15:19:06.604 INFO:tasks.workunit.client.0.smithi096.stdout:5/932: dwrite d4/ddb/d3d/d4d/d59/d5a/f13 [0,4194304] 0 2024-04-22T15:19:06.605 INFO:tasks.workunit.client.1.smithi196.stdout:3/685: write d0/d15/d31/d55/fb6 [367661,27558] 0 2024-04-22T15:19:06.606 INFO:tasks.workunit.client.1.smithi196.stdout:3/686: chown d0/d15/d31/dd0/d51/ddd 118084 1 2024-04-22T15:19:06.615 INFO:tasks.workunit.client.0.smithi096.stdout:0/994: truncate d7/de/d5b/d64/fb6 974349 0 2024-04-22T15:19:06.617 INFO:tasks.workunit.client.1.smithi196.stdout:0/705: mknod d3/d6/dc/d8c/d8d/cf8 0 2024-04-22T15:19:06.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.614+0000 7fa50afc6700 1 -- 172.21.15.96:0/3214053914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 msgr2=0x7fa5040fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:06.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.614+0000 7fa50afc6700 1 --2- 172.21.15.96:0/3214053914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 0x7fa5040fffb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4009910 tx=0x7fa4f4009c20 comp rx=0 tx=0).stop 2024-04-22T15:19:06.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.614+0000 7fa50afc6700 1 -- 172.21.15.96:0/3214053914 shutdown_connections 2024-04-22T15:19:06.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.614+0000 7fa50afc6700 1 --2- 172.21.15.96:0/3214053914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa504100d90 0x7fa5041011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.614+0000 7fa50afc6700 1 --2- 172.21.15.96:0/3214053914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 0x7fa5040fffb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.614+0000 7fa50afc6700 1 -- 172.21.15.96:0/3214053914 >> 172.21.15.96:0/3214053914 conn(0x7fa5040fb110 msgr2=0x7fa5040fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:06.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.614+0000 7fa50afc6700 1 -- 172.21.15.96:0/3214053914 shutdown_connections 2024-04-22T15:19:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.614+0000 7fa50afc6700 1 -- 172.21.15.96:0/3214053914 wait complete. 2024-04-22T15:19:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.615+0000 7fa50afc6700 1 Processor -- start 2024-04-22T15:19:06.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.616+0000 7fa50afc6700 1 -- start start 2024-04-22T15:19:06.620 INFO:tasks.workunit.client.1.smithi196.stdout:1/669: rmdir d1/db/dc/de/d5a/d5b/d85 39 2024-04-22T15:19:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.616+0000 7fa50afc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 0x7fa504196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.617+0000 7fa50afc6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa504100d90 0x7fa504196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.617+0000 7fa508d62700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 0x7fa504196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.617+0000 7fa50afc6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5041972a0 con 0x7fa5040ffb90 2024-04-22T15:19:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.617+0000 7fa508d62700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 0x7fa504196740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34860/0 (socket says 172.21.15.96:34860) 2024-04-22T15:19:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.617+0000 7fa508d62700 1 -- 172.21.15.96:0/2570126455 learned_addr learned my addr 172.21.15.96:0/2570126455 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:06.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.617+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5041973e0 con 0x7fa504100d90 2024-04-22T15:19:06.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.618+0000 7fa508d62700 1 -- 172.21.15.96:0/2570126455 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa504100d90 msgr2=0x7fa504196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:06.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.618+0000 7fa508d62700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa504100d90 0x7fa504196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:06.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.618+0000 7fa508d62700 1 -- 172.21.15.96:0/2570126455 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f40095f0 con 0x7fa5040ffb90 2024-04-22T15:19:06.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.618+0000 7fa508d62700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 0x7fa504196740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400bd70 tx=0x7fa4f400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:06.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.619+0000 7fa501ffb700 1 -- 172.21.15.96:0/2570126455 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f401d070 con 0x7fa5040ffb90 2024-04-22T15:19:06.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.619+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50419bde0 con 0x7fa5040ffb90 2024-04-22T15:19:06.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.619+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50419c2f0 con 0x7fa5040ffb90 2024-04-22T15:19:06.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.619+0000 7fa501ffb700 1 -- 172.21.15.96:0/2570126455 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4f400f460 con 0x7fa5040ffb90 2024-04-22T15:19:06.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.619+0000 7fa501ffb700 1 -- 172.21.15.96:0/2570126455 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f4017610 con 0x7fa5040ffb90 2024-04-22T15:19:06.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.620+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f0005320 con 0x7fa5040ffb90 2024-04-22T15:19:06.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.621+0000 7fa501ffb700 1 -- 172.21.15.96:0/2570126455 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fa4f400f5d0 con 0x7fa5040ffb90 2024-04-22T15:19:06.625 INFO:tasks.workunit.client.0.smithi096.stdout:9/822: truncate d1/df/d95/d9d/db5/d43/d57/d5b/f66 1303121 0 2024-04-22T15:19:06.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.622+0000 7fa501ffb700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fa4ec06dc50 0x7fa4ec070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:06.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.622+0000 7fa501ffb700 1 -- 172.21.15.96:0/2570126455 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fa4f408eb00 con 0x7fa5040ffb90 2024-04-22T15:19:06.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.622+0000 7fa503fff700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fa4ec06dc50 0x7fa4ec070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:06.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.623+0000 7fa503fff700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fa4ec06dc50 0x7fa4ec070110 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009770 tx=0x7fa4f8006cd0 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:06.626 INFO:tasks.workunit.client.1.smithi196.stdout:1/670: dwrite d1/db/dc/de/d5a/d94/dab/fd0 [0,4194304] 0 2024-04-22T15:19:06.627 INFO:tasks.workunit.client.1.smithi196.stdout:7/655: creat d14/d26/d5a/d5f/dc2/fda x:0 0 0 2024-04-22T15:19:06.627 INFO:tasks.workunit.client.0.smithi096.stdout:5/933: fdatasync d4/ddb/d3d/d4d/d59/d98/fd0 0 2024-04-22T15:19:06.627 INFO:tasks.workunit.client.1.smithi196.stdout:7/656: truncate d14/d26/d2a/f84 613257 0 2024-04-22T15:19:06.629 INFO:tasks.workunit.client.1.smithi196.stdout:4/660: unlink d6/d13/d15/d2c/d8c/c95 0 2024-04-22T15:19:06.630 INFO:tasks.workunit.client.1.smithi196.stdout:4/661: chown d6/d14/d18/d26/d32/d67 876 1 2024-04-22T15:19:06.630 INFO:tasks.workunit.client.0.smithi096.stdout:3/935: symlink da/db/de/ddc/d68/dbc/l133 0 2024-04-22T15:19:06.631 INFO:tasks.workunit.client.1.smithi196.stdout:4/662: stat d6/d13/d15/d2c/d35/fbe 0 2024-04-22T15:19:06.631 INFO:tasks.workunit.client.1.smithi196.stdout:4/663: readlink d6/d14/d18/d26/lb8 0 2024-04-22T15:19:06.632 INFO:tasks.workunit.client.0.smithi096.stdout:4/903: mkdir d5/d7/d2c/d95/d13b 0 2024-04-22T15:19:06.633 INFO:tasks.workunit.client.0.smithi096.stdout:2/919: sync 2024-04-22T15:19:06.633 INFO:tasks.workunit.client.1.smithi196.stdout:4/664: write d6/d13/d15/d2c/d62/d78/fd1 [787753,66514] 0 2024-04-22T15:19:06.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:06.630+0000 7fa501ffb700 1 -- 172.21.15.96:0/2570126455 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fa4f405b870 con 0x7fa5040ffb90 2024-04-22T15:19:06.634 INFO:tasks.workunit.client.1.smithi196.stdout:4/665: read - d6/d13/d15/fa6 zero size 2024-04-22T15:19:06.636 INFO:tasks.workunit.client.0.smithi096.stdout:0/995: rename d7/de/d3a/d61/d79/lff to d7/d42/d4c/d55/d9c/l152 0 2024-04-22T15:19:06.637 INFO:tasks.workunit.client.1.smithi196.stdout:5/708: dwrite d1/d17/d1a/d2f/d30/d5e/f77 [0,4194304] 0 2024-04-22T15:19:06.645 INFO:tasks.workunit.client.0.smithi096.stdout:6/745: getdents d3/d28/d4d/d82 0 2024-04-22T15:19:06.647 INFO:tasks.workunit.client.1.smithi196.stdout:9/654: creat db/d11/d29/fde x:0 0 0 2024-04-22T15:19:06.649 INFO:tasks.workunit.client.0.smithi096.stdout:9/823: creat d1/df/d95/d9d/db5/f116 x:0 0 0 2024-04-22T15:19:06.650 INFO:tasks.workunit.client.1.smithi196.stdout:8/679: link d2/d8/f25 d2/d6a/d7b/fc9 0 2024-04-22T15:19:06.651 INFO:tasks.workunit.client.0.smithi096.stdout:9/824: write d1/d25/f110 [772743,77220] 0 2024-04-22T15:19:06.653 INFO:tasks.workunit.client.0.smithi096.stdout:4/904: unlink d5/d7/d2c/d95/dee/f96 0 2024-04-22T15:19:06.659 INFO:tasks.workunit.client.0.smithi096.stdout:5/934: write d4/ddb/d3d/d35/fc9 [1163238,107136] 0 2024-04-22T15:19:06.660 INFO:tasks.workunit.client.1.smithi196.stdout:1/671: mknod d1/db/dc/dad/cd9 0 2024-04-22T15:19:06.661 INFO:tasks.workunit.client.0.smithi096.stdout:0/996: mknod d7/de/d5b/d95/d15/c153 0 2024-04-22T15:19:06.663 INFO:tasks.workunit.client.0.smithi096.stdout:0/997: write d7/de/d5b/d95/dc/d16/f14c [753355,1175] 0 2024-04-22T15:19:06.664 INFO:tasks.workunit.client.0.smithi096.stdout:1/926: link d6/dc/d8a/d3f/d54/d32/d3b/d3d/c115 d6/dc/d8a/d3f/d62/d7d/ddb/d9b/c124 0 2024-04-22T15:19:06.667 INFO:tasks.workunit.client.1.smithi196.stdout:1/672: dwrite d1/db/dc/d26/f28 [4194304,4194304] 0 2024-04-22T15:19:06.668 INFO:tasks.workunit.client.1.smithi196.stdout:6/707: rmdir dd/dc4 0 2024-04-22T15:19:06.669 INFO:tasks.workunit.client.1.smithi196.stdout:0/706: dwrite d3/d9/f73 [0,4194304] 0 2024-04-22T15:19:06.672 INFO:tasks.workunit.client.0.smithi096.stdout:0/998: dwrite d7/de/d3a/d61/d79/fb9 [0,4194304] 0 2024-04-22T15:19:06.675 INFO:tasks.workunit.client.1.smithi196.stdout:9/655: dread db/d11/f4d [0,4194304] 0 2024-04-22T15:19:06.679 INFO:tasks.workunit.client.1.smithi196.stdout:3/687: mkdir d0/d15/d4d/db/de6 0 2024-04-22T15:19:06.680 INFO:tasks.workunit.client.1.smithi196.stdout:5/709: write d1/d9/f11 [668236,41042] 0 2024-04-22T15:19:06.683 INFO:tasks.workunit.client.0.smithi096.stdout:4/905: truncate d5/d7/d1a/f97 3955372 0 2024-04-22T15:19:06.687 INFO:tasks.workunit.client.1.smithi196.stdout:2/692: rename d1/d1d/le5 to d1/d23/d27/d2e/d55/d5b/d93/lfe 0 2024-04-22T15:19:06.688 INFO:tasks.workunit.client.1.smithi196.stdout:3/688: dwrite d0/d15/d31/d55/f74 [0,4194304] 0 2024-04-22T15:19:06.689 INFO:tasks.workunit.client.0.smithi096.stdout:4/906: dwrite d5/d7/d2c/d43/ff4 [0,4194304] 0 2024-04-22T15:19:06.693 INFO:tasks.workunit.client.1.smithi196.stdout:7/657: mknod d14/d4d/da3/dc8/cdb 0 2024-04-22T15:19:06.695 INFO:tasks.workunit.client.1.smithi196.stdout:7/658: truncate d14/d15/d91/d9a/fc4 809554 0 2024-04-22T15:19:06.695 INFO:tasks.workunit.client.0.smithi096.stdout:6/746: dread d3/d28/f21 [0,4194304] 0 2024-04-22T15:19:06.696 INFO:tasks.workunit.client.1.smithi196.stdout:7/659: stat d14/d26/d5a/f8f 0 2024-04-22T15:19:06.696 INFO:tasks.workunit.client.0.smithi096.stdout:6/747: dread - d3/d3a/fe3 zero size 2024-04-22T15:19:06.697 INFO:tasks.workunit.client.1.smithi196.stdout:4/666: link d6/d14/d18/d26/f6a d6/d13/d15/d2c/d62/d78/d8b/fd2 0 2024-04-22T15:19:06.700 INFO:tasks.workunit.client.0.smithi096.stdout:3/936: rename da/db/d12/f12b to da/db/de/ddc/d68/db0/f134 0 2024-04-22T15:19:06.703 INFO:tasks.workunit.client.1.smithi196.stdout:6/708: creat dd/d1f/d4e/d53/fcf x:0 0 0 2024-04-22T15:19:06.704 INFO:tasks.workunit.client.0.smithi096.stdout:0/999: rmdir d7/de/d5b/d95/d11d 39 2024-04-22T15:19:06.708 INFO:tasks.workunit.client.0.smithi096.stdout:2/920: truncate d3/db/df/d24/d58/fd4 3869887 0 2024-04-22T15:19:06.713 INFO:tasks.workunit.client.0.smithi096.stdout:6/748: creat d3/d28/daf/d94/ffd x:0 0 0 2024-04-22T15:19:06.714 INFO:tasks.workunit.client.0.smithi096.stdout:6/749: fsync d3/d28/dd/d11/d2c/d3b/f63 0 2024-04-22T15:19:06.715 INFO:tasks.workunit.client.0.smithi096.stdout:3/937: symlink da/db/d12/dd5/l135 0 2024-04-22T15:19:06.718 INFO:tasks.workunit.client.1.smithi196.stdout:8/680: rename d2/lbd to d2/d5/d4f/lca 0 2024-04-22T15:19:06.719 INFO:tasks.workunit.client.0.smithi096.stdout:9/825: chown d1/df/d95/d9d/db5/d43/f5c 11 1 2024-04-22T15:19:06.720 INFO:tasks.workunit.client.1.smithi196.stdout:3/689: mkdir d0/d15/d4d/de7 0 2024-04-22T15:19:06.720 INFO:tasks.workunit.client.0.smithi096.stdout:2/921: mknod d3/db/df/d24/d58/d5f/dfe/c129 0 2024-04-22T15:19:06.721 INFO:tasks.workunit.client.1.smithi196.stdout:1/673: mknod d1/db/dc/de/d5a/d5b/d9a/cda 0 2024-04-22T15:19:06.722 INFO:tasks.workunit.client.0.smithi096.stdout:2/922: chown d3/db/df/d24/d58/d5f/dfe/d115/f118 1 1 2024-04-22T15:19:06.728 INFO:tasks.workunit.client.0.smithi096.stdout:3/938: mkdir da/db/de/d38/d136 0 2024-04-22T15:19:06.730 INFO:tasks.workunit.client.1.smithi196.stdout:7/660: rename d14/d4d/da3/dc8/cdb to d14/d26/d28/cdc 0 2024-04-22T15:19:06.730 INFO:tasks.workunit.client.1.smithi196.stdout:5/710: dread d1/d17/d1c/f57 [0,4194304] 0 2024-04-22T15:19:06.733 INFO:tasks.workunit.client.1.smithi196.stdout:3/690: mknod d0/d15/d31/d9d/d9e/ce8 0 2024-04-22T15:19:06.735 INFO:tasks.workunit.client.0.smithi096.stdout:1/927: creat d6/dc/d8a/d3f/d54/d32/d8c/f125 x:0 0 0 2024-04-22T15:19:06.737 INFO:tasks.workunit.client.1.smithi196.stdout:1/674: mkdir d1/db/d97/ddb 0 2024-04-22T15:19:06.739 INFO:tasks.workunit.client.1.smithi196.stdout:8/681: dread d2/f4a [0,4194304] 0 2024-04-22T15:19:06.741 INFO:tasks.workunit.client.1.smithi196.stdout:6/709: link dd/d1f/fbd dd/d1f/d37/d7b/db6/fd0 0 2024-04-22T15:19:06.743 INFO:tasks.workunit.client.1.smithi196.stdout:0/707: creat d3/d6/ff9 x:0 0 0 2024-04-22T15:19:06.745 INFO:tasks.workunit.client.0.smithi096.stdout:2/923: mknod d3/db/df/d24/d58/d5e/c12a 0 2024-04-22T15:19:06.745 INFO:tasks.workunit.client.1.smithi196.stdout:0/708: dread - d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/fe5 zero size 2024-04-22T15:19:06.746 INFO:tasks.workunit.client.1.smithi196.stdout:0/709: chown d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/f9e 136206 1 2024-04-22T15:19:06.746 INFO:tasks.workunit.client.1.smithi196.stdout:9/656: link db/c59 db/dd/d1b/d2c/cdf 0 2024-04-22T15:19:06.747 INFO:tasks.workunit.client.1.smithi196.stdout:0/710: stat d3/de/d1c/d27/ddc 0 2024-04-22T15:19:06.747 INFO:tasks.workunit.client.0.smithi096.stdout:3/939: dread - da/db/d9a/fc1 zero size 2024-04-22T15:19:06.748 INFO:tasks.workunit.client.0.smithi096.stdout:5/935: write d4/ddb/d3d/d4d/fa4 [6736210,76589] 0 2024-04-22T15:19:06.748 INFO:tasks.workunit.client.1.smithi196.stdout:9/657: dread db/dd/de/f42 [0,4194304] 0 2024-04-22T15:19:06.749 INFO:tasks.workunit.client.1.smithi196.stdout:9/658: dread db/d11/d29/f48 [0,4194304] 0 2024-04-22T15:19:06.752 INFO:tasks.workunit.client.0.smithi096.stdout:1/928: unlink d6/dc/d8a/d3f/d54/d32/d3b/d56/f44 0 2024-04-22T15:19:06.753 INFO:tasks.workunit.client.1.smithi196.stdout:3/691: mkdir d0/d15/da7/d4a/d44/d8c/de9 0 2024-04-22T15:19:06.755 INFO:tasks.workunit.client.1.smithi196.stdout:1/675: write d1/db/d16/f40 [3217281,102581] 0 2024-04-22T15:19:06.764 INFO:tasks.workunit.client.0.smithi096.stdout:6/750: write d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/f8b [405020,118547] 0 2024-04-22T15:19:06.764 INFO:tasks.workunit.client.0.smithi096.stdout:8/885: sync 2024-04-22T15:19:06.765 INFO:tasks.workunit.client.0.smithi096.stdout:3/940: creat da/db/d9a/d132/f137 x:0 0 0 2024-04-22T15:19:06.768 INFO:tasks.workunit.client.1.smithi196.stdout:5/711: mknod d1/d2/d98/dde/ce4 0 2024-04-22T15:19:06.769 INFO:tasks.workunit.client.0.smithi096.stdout:9/826: write d1/d25/db2/d98/d9b/feb [782830,104222] 0 2024-04-22T15:19:06.770 INFO:tasks.workunit.client.0.smithi096.stdout:6/751: dread d3/d28/dd/d11/d2c/f95 [0,4194304] 0 2024-04-22T15:19:06.770 INFO:tasks.workunit.client.0.smithi096.stdout:1/929: read d6/dc/d8a/d3f/d62/d7d/ddb/d9b/fb5 [128784,65625] 0 2024-04-22T15:19:06.772 INFO:tasks.workunit.client.0.smithi096.stdout:6/752: truncate d3/d28/d4d/d82/ffb 48573 0 2024-04-22T15:19:06.773 INFO:tasks.workunit.client.1.smithi196.stdout:2/693: sync 2024-04-22T15:19:06.774 INFO:tasks.workunit.client.0.smithi096.stdout:4/907: getdents d5/d7/d2c/d67 0 2024-04-22T15:19:06.775 INFO:tasks.workunit.client.1.smithi196.stdout:9/659: dwrite db/dd/de/d15/d31/f32 [0,4194304] 0 2024-04-22T15:19:06.776 INFO:tasks.workunit.client.1.smithi196.stdout:3/692: rmdir d0/d15/d31/dd0/d51/ddd 39 2024-04-22T15:19:06.777 INFO:tasks.workunit.client.0.smithi096.stdout:6/753: dread d3/d28/dd/f19 [4194304,4194304] 0 2024-04-22T15:19:06.779 INFO:tasks.workunit.client.1.smithi196.stdout:1/676: mkdir d1/db/d16/d1e/d46/ddc 0 2024-04-22T15:19:06.780 INFO:tasks.workunit.client.1.smithi196.stdout:1/677: chown d1/db/d16/la5 3 1 2024-04-22T15:19:06.784 INFO:tasks.workunit.client.0.smithi096.stdout:9/827: rmdir d1/df/d95 39 2024-04-22T15:19:06.787 INFO:tasks.workunit.client.1.smithi196.stdout:8/682: dwrite d2/d5/d2e/f67 [4194304,4194304] 0 2024-04-22T15:19:06.787 INFO:tasks.workunit.client.1.smithi196.stdout:4/667: rename d6/d13/f1d to d6/d13/d15/d2c/fd3 0 2024-04-22T15:19:06.788 INFO:tasks.workunit.client.0.smithi096.stdout:1/930: dread d6/dc/d8a/d3f/d54/d32/d3b/d56/f7f [0,4194304] 0 2024-04-22T15:19:06.790 INFO:tasks.workunit.client.1.smithi196.stdout:0/711: dwrite d3/d9/d36/d9f/dd1/d65/d7f/d30/fe7 [0,4194304] 0 2024-04-22T15:19:06.794 INFO:tasks.workunit.client.0.smithi096.stdout:5/936: write d4/ddb/ff2 [369283,110393] 0 2024-04-22T15:19:06.795 INFO:tasks.workunit.client.0.smithi096.stdout:2/924: creat d3/db/df/d98/f12b x:0 0 0 2024-04-22T15:19:06.798 INFO:tasks.workunit.client.1.smithi196.stdout:9/660: fdatasync db/d11/d29/f48 0 2024-04-22T15:19:06.799 INFO:tasks.workunit.client.1.smithi196.stdout:9/661: chown db/dd/d1b/d2c/f9f 9 1 2024-04-22T15:19:06.800 INFO:tasks.workunit.client.0.smithi096.stdout:6/754: mkdir d3/d28/d4d/d82/dfe 0 2024-04-22T15:19:06.803 INFO:tasks.workunit.client.1.smithi196.stdout:3/693: creat d0/d15/d31/d55/dcf/d8/fea x:0 0 0 2024-04-22T15:19:06.803 INFO:tasks.workunit.client.1.smithi196.stdout:1/678: dread - d1/db/fc0 zero size 2024-04-22T15:19:06.804 INFO:tasks.workunit.client.0.smithi096.stdout:8/886: creat d4/d113/f12b x:0 0 0 2024-04-22T15:19:06.810 INFO:tasks.workunit.client.1.smithi196.stdout:1/679: dwrite d1/db/dc/d93/fb6 [0,4194304] 0 2024-04-22T15:19:06.812 INFO:tasks.workunit.client.1.smithi196.stdout:4/668: truncate d6/d10/f63 1421359 0 2024-04-22T15:19:06.814 INFO:tasks.workunit.client.0.smithi096.stdout:5/937: creat d4/ddb/d3d/d4d/d59/d8a/d121/f141 x:0 0 0 2024-04-22T15:19:06.817 INFO:tasks.workunit.client.1.smithi196.stdout:0/712: creat d3/d9/d36/d9f/dd1/d65/d7f/ddb/ffa x:0 0 0 2024-04-22T15:19:06.817 INFO:tasks.workunit.client.1.smithi196.stdout:0/713: stat d3/d9/d36/d9f/dd1/l8a 0 2024-04-22T15:19:06.819 INFO:tasks.workunit.client.0.smithi096.stdout:6/755: stat d3/d28/d54/ffc 0 2024-04-22T15:19:06.822 INFO:tasks.workunit.client.1.smithi196.stdout:9/662: truncate db/dd/de/f4e 551406 0 2024-04-22T15:19:06.822 INFO:tasks.workunit.client.1.smithi196.stdout:9/663: stat db/dd/de/d15/d31/d64/dab 0 2024-04-22T15:19:06.823 INFO:tasks.workunit.client.0.smithi096.stdout:9/828: symlink d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dbd/l117 0 2024-04-22T15:19:06.823 INFO:tasks.workunit.client.1.smithi196.stdout:3/694: readlink d0/d15/da7/d4a/l59 0 2024-04-22T15:19:06.824 INFO:tasks.workunit.client.0.smithi096.stdout:6/756: dread d3/d3a/d75/fd3 [0,4194304] 0 2024-04-22T15:19:06.826 INFO:tasks.workunit.client.0.smithi096.stdout:4/908: rmdir d5/d7/d2c/d95/d13b 0 2024-04-22T15:19:06.827 INFO:tasks.workunit.client.1.smithi196.stdout:9/664: dwrite db/d11/d29/f6f [0,4194304] 0 2024-04-22T15:19:06.830 INFO:tasks.workunit.client.0.smithi096.stdout:5/938: rename d4/ddb/d3d/d4d/d88/de5/c112 to d4/ddb/d3d/d4d/d59/d98/c142 0 2024-04-22T15:19:06.834 INFO:tasks.workunit.client.0.smithi096.stdout:2/925: creat d3/dd5/f12c x:0 0 0 2024-04-22T15:19:06.836 INFO:tasks.workunit.client.1.smithi196.stdout:5/712: creat d1/d17/d1a/fe5 x:0 0 0 2024-04-22T15:19:06.837 INFO:tasks.workunit.client.1.smithi196.stdout:2/694: dwrite d1/d23/d27/d2e/f88 [0,4194304] 0 2024-04-22T15:19:06.840 INFO:tasks.workunit.client.1.smithi196.stdout:1/680: mkdir d1/db/d16/d8e/ddd 0 2024-04-22T15:19:06.841 INFO:tasks.workunit.client.1.smithi196.stdout:1/681: readlink d1/db/dc/de/d5a/d94/dab/lbe 0 2024-04-22T15:19:06.842 INFO:tasks.workunit.client.1.smithi196.stdout:1/682: readlink d1/db/dc/de/d96/lb3 0 2024-04-22T15:19:06.843 INFO:tasks.workunit.client.1.smithi196.stdout:1/683: fdatasync d1/db/dc/de/d43/d5e/d7a/fa1 0 2024-04-22T15:19:06.843 INFO:tasks.workunit.client.1.smithi196.stdout:1/684: readlink d1/db/dc/d93/lae 0 2024-04-22T15:19:06.844 INFO:tasks.workunit.client.1.smithi196.stdout:1/685: readlink d1/db/dc/de/d96/lbb 0 2024-04-22T15:19:06.847 INFO:tasks.workunit.client.1.smithi196.stdout:4/669: mkdir d6/d13/d15/d2c/d62/d78/d8d/dd4 0 2024-04-22T15:19:06.853 INFO:tasks.workunit.client.0.smithi096.stdout:4/909: creat d5/d7/d1a/d8b/f13c x:0 0 0 2024-04-22T15:19:06.853 INFO:tasks.workunit.client.0.smithi096.stdout:4/910: fsync d5/d7/d1f/d84/d6d/f125 0 2024-04-22T15:19:06.855 INFO:tasks.workunit.client.0.smithi096.stdout:5/939: mknod d4/ddb/d3d/d4d/d59/db0/db2/c143 0 2024-04-22T15:19:06.857 INFO:tasks.workunit.client.0.smithi096.stdout:2/926: creat d3/db/df/dce/d46/d69/d7f/f12d x:0 0 0 2024-04-22T15:19:06.860 INFO:tasks.workunit.client.0.smithi096.stdout:4/911: dwrite d5/d7/d2c/d3d/d4f/f60 [0,4194304] 0 2024-04-22T15:19:06.860 INFO:tasks.workunit.client.1.smithi196.stdout:7/661: rename d14/d26/d5a/db6 to d14/ddd 0 2024-04-22T15:19:06.861 INFO:tasks.workunit.client.1.smithi196.stdout:8/683: write d2/d3f/d4b/f4d [2792098,59948] 0 2024-04-22T15:19:06.862 INFO:tasks.workunit.client.1.smithi196.stdout:0/714: dread d3/d9/d25/f63 [0,4194304] 0 2024-04-22T15:19:06.862 INFO:tasks.workunit.client.1.smithi196.stdout:8/684: truncate d2/d35/fb4 1059495 0 2024-04-22T15:19:06.864 INFO:tasks.workunit.client.1.smithi196.stdout:9/665: creat db/dd/de/d15/fe0 x:0 0 0 2024-04-22T15:19:06.864 INFO:tasks.workunit.client.1.smithi196.stdout:9/666: dread - db/dd/de/d15/d31/fcf zero size 2024-04-22T15:19:06.866 INFO:tasks.workunit.client.1.smithi196.stdout:9/667: read db/dd/de/d15/d31/fb7 [4560428,81407] 0 2024-04-22T15:19:06.867 INFO:tasks.workunit.client.0.smithi096.stdout:1/931: getdents d6/dc/d8a/d3f/d54/d32/d3b/d94 0 2024-04-22T15:19:06.871 INFO:tasks.workunit.client.0.smithi096.stdout:6/757: symlink d3/d28/dd/d11/d2c/lff 0 2024-04-22T15:19:06.872 INFO:tasks.workunit.client.0.smithi096.stdout:6/758: stat d3/d28/dc9/fec 0 2024-04-22T15:19:06.874 INFO:tasks.workunit.client.0.smithi096.stdout:5/940: symlink d4/d2c/d2f/d31/d82/l144 0 2024-04-22T15:19:06.875 INFO:tasks.workunit.client.0.smithi096.stdout:8/887: dwrite d4/d1c/d40/d48/f94 [0,4194304] 0 2024-04-22T15:19:06.875 INFO:tasks.workunit.client.1.smithi196.stdout:4/670: creat d6/d14/d18/d26/d32/d53/db6/fd5 x:0 0 0 2024-04-22T15:19:06.876 INFO:tasks.workunit.client.0.smithi096.stdout:6/759: truncate d3/d3a/d75/d83/df7/fef 428866 0 2024-04-22T15:19:06.878 INFO:tasks.workunit.client.0.smithi096.stdout:2/927: mkdir d3/db/df/dce/d46/d7e/d84/d12e 0 2024-04-22T15:19:06.878 INFO:tasks.workunit.client.1.smithi196.stdout:3/695: symlink d0/d15/leb 0 2024-04-22T15:19:06.882 INFO:tasks.workunit.client.1.smithi196.stdout:6/710: rename dd/d1f/d37/d3c/d96 to dd/d1f/d4e/d53/d55/d6d/dd1 0 2024-04-22T15:19:06.884 INFO:tasks.workunit.client.1.smithi196.stdout:7/662: read d14/f22 [628355,8995] 0 2024-04-22T15:19:06.884 INFO:tasks.workunit.client.1.smithi196.stdout:7/663: fdatasync d14/d25/d31/f8e 0 2024-04-22T15:19:06.890 INFO:tasks.workunit.client.1.smithi196.stdout:9/668: dread db/dd/de/d15/f40 [4194304,4194304] 0 2024-04-22T15:19:06.894 INFO:tasks.workunit.client.1.smithi196.stdout:4/671: unlink d6/d14/c1f 0 2024-04-22T15:19:06.895 INFO:tasks.workunit.client.1.smithi196.stdout:4/672: write d6/d13/d15/f3d [3641988,23925] 0 2024-04-22T15:19:06.895 INFO:tasks.workunit.client.0.smithi096.stdout:2/928: mknod d3/db/df/d24/d58/c12f 0 2024-04-22T15:19:06.896 INFO:tasks.workunit.client.1.smithi196.stdout:4/673: write d6/f2b [2059113,24928] 0 2024-04-22T15:19:06.898 INFO:tasks.workunit.client.1.smithi196.stdout:3/696: rename d0/d15/da7/d85/le0 to d0/d15/da7/d85/lec 0 2024-04-22T15:19:06.902 INFO:tasks.workunit.client.1.smithi196.stdout:5/713: rename d1/f4c to d1/d17/d7f/fe6 0 2024-04-22T15:19:06.905 INFO:tasks.workunit.client.0.smithi096.stdout:9/829: link d1/f6 d1/df/d95/d9d/db5/dd/dba/f118 0 2024-04-22T15:19:06.907 INFO:tasks.workunit.client.0.smithi096.stdout:6/760: dread d3/d28/dd/f70 [0,4194304] 0 2024-04-22T15:19:06.908 INFO:tasks.workunit.client.1.smithi196.stdout:7/664: unlink d14/d26/d5a/dbb/l98 0 2024-04-22T15:19:06.908 INFO:tasks.workunit.client.1.smithi196.stdout:7/665: fdatasync d14/d26/d5a/f8f 0 2024-04-22T15:19:06.909 INFO:tasks.workunit.client.1.smithi196.stdout:7/666: stat d14/d26/d2a/d72/db1/cca 0 2024-04-22T15:19:06.909 INFO:tasks.workunit.client.1.smithi196.stdout:0/715: getdents d3/d6/dc/d8c/daf 0 2024-04-22T15:19:06.910 INFO:tasks.workunit.client.1.smithi196.stdout:7/667: write d14/d26/d2a/d2b/fb9 [535174,24090] 0 2024-04-22T15:19:06.910 INFO:tasks.workunit.client.1.smithi196.stdout:8/685: write d2/d8/d49/f7a [1152762,118740] 0 2024-04-22T15:19:06.914 INFO:tasks.workunit.client.0.smithi096.stdout:2/929: symlink d3/db/df/d10/l130 0 2024-04-22T15:19:06.914 INFO:tasks.workunit.client.0.smithi096.stdout:6/761: dwrite d3/d28/f38 [0,4194304] 0 2024-04-22T15:19:06.920 INFO:tasks.workunit.client.1.smithi196.stdout:1/686: link d1/db/dc/c75 d1/db/d16/d8e/ddd/cde 0 2024-04-22T15:19:06.921 INFO:tasks.workunit.client.1.smithi196.stdout:1/687: fsync d1/db/d16/d44/f92 0 2024-04-22T15:19:06.923 INFO:tasks.workunit.client.0.smithi096.stdout:9/830: read d1/f38 [5610106,19899] 0 2024-04-22T15:19:06.925 INFO:tasks.workunit.client.1.smithi196.stdout:4/674: mkdir d6/d13/d15/d2c/d62/dd6 0 2024-04-22T15:19:06.926 INFO:tasks.workunit.client.0.smithi096.stdout:9/831: truncate d1/df/f65 473825 0 2024-04-22T15:19:06.926 INFO:tasks.workunit.client.0.smithi096.stdout:4/912: dwrite d5/d7/fed [0,4194304] 0 2024-04-22T15:19:06.927 INFO:tasks.workunit.client.1.smithi196.stdout:4/675: chown d6/d13/d15/d2c/d62/d78/d8d/f9c 326 1 2024-04-22T15:19:06.929 INFO:tasks.workunit.client.0.smithi096.stdout:5/941: creat d4/ddb/d3d/d4d/d59/db0/f145 x:0 0 0 2024-04-22T15:19:06.931 INFO:tasks.workunit.client.0.smithi096.stdout:5/942: write d4/ddb/d3d/d4d/d59/d98/fd3 [619748,63368] 0 2024-04-22T15:19:06.931 INFO:tasks.workunit.client.1.smithi196.stdout:5/714: rmdir d1/d14/d6a 39 2024-04-22T15:19:06.932 INFO:tasks.workunit.client.1.smithi196.stdout:9/669: dwrite db/d11/d29/f92 [0,4194304] 0 2024-04-22T15:19:06.934 INFO:tasks.workunit.client.0.smithi096.stdout:2/930: unlink d3/db/df/d24/d58/d5f/dfe/d115/f118 0 2024-04-22T15:19:06.935 INFO:tasks.workunit.client.0.smithi096.stdout:2/931: write d3/db/f83 [3406089,74020] 0 2024-04-22T15:19:06.935 INFO:tasks.workunit.client.0.smithi096.stdout:2/932: stat d3/l72 0 2024-04-22T15:19:06.937 INFO:tasks.workunit.client.1.smithi196.stdout:7/668: creat d14/d26/d2a/d72/fde x:0 0 0 2024-04-22T15:19:06.938 INFO:tasks.workunit.client.0.smithi096.stdout:1/932: getdents d6/dc/d8a/d3f/d54/d32/d8c/d45/da9 0 2024-04-22T15:19:06.939 INFO:tasks.workunit.client.0.smithi096.stdout:6/762: fsync d3/d28/f38 0 2024-04-22T15:19:06.940 INFO:tasks.workunit.client.1.smithi196.stdout:1/688: dread d1/db/d97/fd3 [0,4194304] 0 2024-04-22T15:19:06.940 INFO:tasks.workunit.client.1.smithi196.stdout:0/716: creat d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8/ffb x:0 0 0 2024-04-22T15:19:06.940 INFO:tasks.workunit.client.0.smithi096.stdout:6/763: write d3/d28/dd/d11/d2c/fb1 [436252,20776] 0 2024-04-22T15:19:06.941 INFO:tasks.workunit.client.1.smithi196.stdout:0/717: chown d3/d9/d36/f6d 1079749 1 2024-04-22T15:19:06.941 INFO:tasks.workunit.client.1.smithi196.stdout:1/689: dread - d1/db/fc0 zero size 2024-04-22T15:19:06.949 INFO:tasks.workunit.client.0.smithi096.stdout:8/888: rename d4/l2f to d4/d10/d25/d38/l12c 0 2024-04-22T15:19:06.949 INFO:tasks.workunit.client.1.smithi196.stdout:3/697: creat d0/d15/da7/d4a/d44/d8c/de9/fed x:0 0 0 2024-04-22T15:19:06.950 INFO:tasks.workunit.client.0.smithi096.stdout:8/889: chown d4/d1c/d40/da7/db6 7908879 1 2024-04-22T15:19:06.950 INFO:tasks.workunit.client.0.smithi096.stdout:5/943: symlink d4/ddb/d3d/d4d/d59/d8a/l146 0 2024-04-22T15:19:06.952 INFO:tasks.workunit.client.0.smithi096.stdout:5/944: chown d4/d2c/d2f/d30/le3 654 1 2024-04-22T15:19:06.952 INFO:tasks.workunit.client.1.smithi196.stdout:2/695: sync 2024-04-22T15:19:06.952 INFO:tasks.workunit.client.1.smithi196.stdout:6/711: sync 2024-04-22T15:19:06.956 INFO:tasks.workunit.client.1.smithi196.stdout:4/676: rename d6/d14/d18/d40/d85/cc1 to d6/d13/d15/d2c/d62/d78/cd7 0 2024-04-22T15:19:06.960 INFO:tasks.workunit.client.0.smithi096.stdout:1/933: chown d6/dc/d8a/d3f/d54/d32/d82/fa4 88 1 2024-04-22T15:19:06.963 INFO:tasks.workunit.client.1.smithi196.stdout:0/718: unlink d3/d9/d36/f6d 0 2024-04-22T15:19:06.965 INFO:tasks.workunit.client.0.smithi096.stdout:4/913: dwrite d5/d7/d2c/d67/fb3 [0,4194304] 0 2024-04-22T15:19:06.967 INFO:tasks.workunit.client.0.smithi096.stdout:9/832: unlink d1/d4/d7b/f81 0 2024-04-22T15:19:06.969 INFO:tasks.workunit.client.0.smithi096.stdout:9/833: chown d1/df/d95/d9d/db5/d43/d57/d5b/d8e 206174 1 2024-04-22T15:19:06.977 INFO:tasks.workunit.client.0.smithi096.stdout:5/945: symlink d4/ddb/d3d/d4d/d59/d5a/d5c/l147 0 2024-04-22T15:19:06.978 INFO:tasks.workunit.client.0.smithi096.stdout:2/933: unlink d3/d13/db4/led 0 2024-04-22T15:19:06.979 INFO:tasks.workunit.client.0.smithi096.stdout:2/934: write d3/db/df/d98/f12b [702491,5329] 0 2024-04-22T15:19:06.980 INFO:tasks.workunit.client.0.smithi096.stdout:1/934: read d6/fcf [605376,115232] 0 2024-04-22T15:19:06.980 INFO:tasks.workunit.client.1.smithi196.stdout:4/677: symlink d6/d14/d18/d26/d32/da3/ld8 0 2024-04-22T15:19:06.981 INFO:tasks.workunit.client.0.smithi096.stdout:3/941: sync 2024-04-22T15:19:06.989 INFO:tasks.workunit.client.1.smithi196.stdout:0/719: mknod d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/db7/cfc 0 2024-04-22T15:19:06.990 INFO:tasks.workunit.client.0.smithi096.stdout:3/942: dwrite da/db/de/ddc/d68/dbc/ff1 [0,4194304] 0 2024-04-22T15:19:06.994 INFO:tasks.workunit.client.0.smithi096.stdout:9/834: mknod d1/d85/c119 0 2024-04-22T15:19:06.995 INFO:tasks.workunit.client.0.smithi096.stdout:9/835: write d1/df/d95/d9d/db5/d43/d57/d5b/ffe [70736,24040] 0 2024-04-22T15:19:06.995 INFO:tasks.workunit.client.1.smithi196.stdout:1/690: symlink d1/db/d97/dd7/ldf 0 2024-04-22T15:19:07.001 INFO:tasks.workunit.client.0.smithi096.stdout:5/946: unlink d4/ddb/d3d/d4d/f6e 0 2024-04-22T15:19:07.002 INFO:tasks.workunit.client.1.smithi196.stdout:8/686: dwrite d2/d5/d4f/f6e [4194304,4194304] 0 2024-04-22T15:19:07.002 INFO:tasks.workunit.client.1.smithi196.stdout:3/698: getdents d0/d15/d4d/de7 0 2024-04-22T15:19:07.005 INFO:tasks.workunit.client.1.smithi196.stdout:5/715: truncate d1/d17/d1a/d2f/f99 815398 0 2024-04-22T15:19:07.005 INFO:tasks.workunit.client.1.smithi196.stdout:9/670: write db/dd/de/d15/d31/d64/fad [4037127,73590] 0 2024-04-22T15:19:07.006 INFO:tasks.workunit.client.1.smithi196.stdout:6/712: dwrite dd/d1f/d37/f8e [0,4194304] 0 2024-04-22T15:19:07.006 INFO:tasks.workunit.client.0.smithi096.stdout:2/935: mknod d3/db/df/d24/d58/d5e/c131 0 2024-04-22T15:19:07.009 INFO:tasks.workunit.client.1.smithi196.stdout:2/696: mkdir d1/d13/d25/d2a/ddb/dff 0 2024-04-22T15:19:07.009 INFO:tasks.workunit.client.1.smithi196.stdout:8/687: write d2/d5/d4f/f6e [3161809,46157] 0 2024-04-22T15:19:07.009 INFO:tasks.workunit.client.1.smithi196.stdout:1/691: dread d1/db/d16/f40 [0,4194304] 0 2024-04-22T15:19:07.011 INFO:tasks.workunit.client.1.smithi196.stdout:1/692: fsync d1/db/d16/fd1 0 2024-04-22T15:19:07.011 INFO:tasks.workunit.client.0.smithi096.stdout:6/764: mknod d3/d28/d4d/d82/d9e/da3/c100 0 2024-04-22T15:19:07.012 INFO:tasks.workunit.client.1.smithi196.stdout:1/693: chown d1/db/d16/d1e/d46/l4e 2985 1 2024-04-22T15:19:07.015 INFO:tasks.workunit.client.1.smithi196.stdout:0/720: symlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9/lfd 0 2024-04-22T15:19:07.017 INFO:tasks.workunit.client.1.smithi196.stdout:4/678: dread d6/f93 [0,4194304] 0 2024-04-22T15:19:07.017 INFO:tasks.workunit.client.0.smithi096.stdout:3/943: dread da/db/de/ddc/d26/dc6/dd2/fed [0,4194304] 0 2024-04-22T15:19:07.019 INFO:tasks.workunit.client.0.smithi096.stdout:3/944: write da/db/dc/f16 [4900017,130798] 0 2024-04-22T15:19:07.022 INFO:tasks.workunit.client.0.smithi096.stdout:8/890: link d4/d1c/d62/la3 d4/d1c/d62/dc3/df9/l12d 0 2024-04-22T15:19:07.023 INFO:tasks.workunit.client.0.smithi096.stdout:8/891: chown d4/d71/f106 6395 1 2024-04-22T15:19:07.023 INFO:tasks.workunit.client.1.smithi196.stdout:4/679: dwrite d6/d13/d15/d2c/d62/d78/d8d/f9c [0,4194304] 0 2024-04-22T15:19:07.025 INFO:tasks.workunit.client.0.smithi096.stdout:5/947: creat d4/ddb/d3d/d4d/d59/db0/db2/f148 x:0 0 0 2024-04-22T15:19:07.026 INFO:tasks.workunit.client.1.smithi196.stdout:2/697: rmdir d1/d76 39 2024-04-22T15:19:07.028 INFO:tasks.workunit.client.1.smithi196.stdout:7/669: rename d14/d26/d2a/d2b/fb9 to d14/d26/fdf 0 2024-04-22T15:19:07.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.028+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4f0000bf0 con 0x7fa4ec06dc50 2024-04-22T15:19:07.032 INFO:tasks.workunit.client.1.smithi196.stdout:6/713: creat dd/d70/fd2 x:0 0 0 2024-04-22T15:19:07.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.029+0000 7fa501ffb700 1 -- 172.21.15.96:0/2570126455 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fa4f0000bf0 con 0x7fa4ec06dc50 2024-04-22T15:19:07.034 INFO:tasks.workunit.client.0.smithi096.stdout:2/936: unlink d3/db/df/d24/d58/d5f/dfe/d115/d2f/f5d 0 2024-04-22T15:19:07.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fa4ec06dc50 msgr2=0x7fa4ec070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:07.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fa4ec06dc50 0x7fa4ec070110 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009770 tx=0x7fa4f8006cd0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 msgr2=0x7fa504196740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:07.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 0x7fa504196740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400bd70 tx=0x7fa4f400be50 comp rx=0 tx=0).stop 2024-04-22T15:19:07.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 shutdown_connections 2024-04-22T15:19:07.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fa4ec06dc50 0x7fa4ec070110 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5040ffb90 0x7fa504196740 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 --2- 172.21.15.96:0/2570126455 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa504100d90 0x7fa504196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 >> 172.21.15.96:0/2570126455 conn(0x7fa5040fb110 msgr2=0x7fa5040685c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:07.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 shutdown_connections 2024-04-22T15:19:07.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.033+0000 7fa50afc6700 1 -- 172.21.15.96:0/2570126455 wait complete. 2024-04-22T15:19:07.045 INFO:tasks.workunit.client.1.smithi196.stdout:9/671: symlink db/dd/de/le1 0 2024-04-22T15:19:07.052 INFO:tasks.workunit.client.1.smithi196.stdout:1/694: creat d1/db/dc/de/d96/d9d/fe0 x:0 0 0 2024-04-22T15:19:07.057 INFO:tasks.workunit.client.0.smithi096.stdout:4/914: dwrite d5/d7/d2c/f86 [0,4194304] 0 2024-04-22T15:19:07.060 INFO:tasks.workunit.client.0.smithi096.stdout:8/892: unlink d4/d2a/d7e/led 0 2024-04-22T15:19:07.065 INFO:tasks.workunit.client.0.smithi096.stdout:5/948: rename d4/d2c/d2f/l52 to d4/ddb/d3d/d4d/d59/d5a/d5c/l149 0 2024-04-22T15:19:07.068 INFO:tasks.workunit.client.0.smithi096.stdout:1/935: link d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/l103 d6/dc/d8a/d3f/d54/d32/d3b/d56/l126 0 2024-04-22T15:19:07.074 INFO:tasks.workunit.client.0.smithi096.stdout:2/937: unlink d3/db/df/d24/c4b 0 2024-04-22T15:19:07.077 INFO:tasks.workunit.client.0.smithi096.stdout:3/945: dread da/f34 [0,4194304] 0 2024-04-22T15:19:07.077 INFO:tasks.workunit.client.1.smithi196.stdout:5/716: write d1/d9/f38 [647934,97199] 0 2024-04-22T15:19:07.079 INFO:tasks.workunit.client.1.smithi196.stdout:3/699: rename d0/d15/da7/d4a/d44/f26 to d0/d15/d4d/db/de6/fee 0 2024-04-22T15:19:07.084 INFO:tasks.workunit.client.0.smithi096.stdout:6/765: dwrite d3/d28/dd/d11/d2c/d6f/f99 [0,4194304] 0 2024-04-22T15:19:07.085 INFO:tasks.workunit.client.1.smithi196.stdout:7/670: creat d14/d26/d2a/d2b/dc7/fe0 x:0 0 0 2024-04-22T15:19:07.086 INFO:tasks.workunit.client.1.smithi196.stdout:7/671: chown d14/d25/d31/f8e 1 1 2024-04-22T15:19:07.087 INFO:tasks.workunit.client.1.smithi196.stdout:7/672: fsync d14/d26/d5a/d5f/dc2/fda 0 2024-04-22T15:19:07.088 INFO:tasks.workunit.client.1.smithi196.stdout:6/714: mkdir dd/d1f/d38/d40/dd3 0 2024-04-22T15:19:07.090 INFO:tasks.workunit.client.0.smithi096.stdout:4/915: truncate f1 4251602 0 2024-04-22T15:19:07.096 INFO:tasks.workunit.client.1.smithi196.stdout:6/715: dwrite dd/d20/f7d [0,4194304] 0 2024-04-22T15:19:07.098 INFO:tasks.workunit.client.0.smithi096.stdout:5/949: rename d4/ddb/d3d/d4f to d4/ddb/d3d/d35/d57/dbe/d113/d14a 0 2024-04-22T15:19:07.102 INFO:tasks.workunit.client.1.smithi196.stdout:0/721: link d3/d6/dc/d8c/d8d/fc1 d3/de/d14/ded/ffe 0 2024-04-22T15:19:07.105 INFO:tasks.workunit.client.1.smithi196.stdout:4/680: creat d6/d14/d18/d26/d32/da3/db7/dc9/fd9 x:0 0 0 2024-04-22T15:19:07.106 INFO:tasks.workunit.client.1.smithi196.stdout:4/681: truncate d6/d13/f24 1697847 0 2024-04-22T15:19:07.106 INFO:tasks.workunit.client.1.smithi196.stdout:4/682: dread - d6/d14/d18/d26/d32/d67/fc8 zero size 2024-04-22T15:19:07.107 INFO:tasks.workunit.client.0.smithi096.stdout:3/946: creat da/db/de/d38/f138 x:0 0 0 2024-04-22T15:19:07.107 INFO:tasks.workunit.client.1.smithi196.stdout:4/683: write d6/d13/d15/d2c/d62/f99 [4808935,106862] 0 2024-04-22T15:19:07.108 INFO:tasks.workunit.client.0.smithi096.stdout:9/836: link d1/d25/db2/l74 d1/df/d95/d9d/db5/l11a 0 2024-04-22T15:19:07.110 INFO:tasks.workunit.client.1.smithi196.stdout:9/672: rename db/d11/d29/d57/l5d to db/d11/d29/d7a/d7b/d81/le2 0 2024-04-22T15:19:07.111 INFO:tasks.workunit.client.1.smithi196.stdout:3/700: fsync d0/d15/d31/dd0/d51/ddd/fe4 0 2024-04-22T15:19:07.111 INFO:tasks.workunit.client.1.smithi196.stdout:9/673: truncate db/d11/d29/fcb 973954 0 2024-04-22T15:19:07.112 INFO:tasks.workunit.client.1.smithi196.stdout:9/674: chown db/dd/de/d15/ld9 3404 1 2024-04-22T15:19:07.113 INFO:tasks.workunit.client.0.smithi096.stdout:6/766: creat d3/d28/d54/d7e/f101 x:0 0 0 2024-04-22T15:19:07.113 INFO:tasks.workunit.client.0.smithi096.stdout:6/767: write f0 [1100635,75372] 0 2024-04-22T15:19:07.114 INFO:tasks.workunit.client.0.smithi096.stdout:8/893: mkdir d4/d1c/df8/d12e 0 2024-04-22T15:19:07.115 INFO:tasks.workunit.client.1.smithi196.stdout:7/673: write d14/d26/d28/fab [907858,118509] 0 2024-04-22T15:19:07.119 INFO:tasks.workunit.client.1.smithi196.stdout:8/688: link d2/dc/f20 d2/d3f/d4b/fcb 0 2024-04-22T15:19:07.120 INFO:tasks.workunit.client.1.smithi196.stdout:7/674: dread d14/d26/d2a/d2b/f77 [0,4194304] 0 2024-04-22T15:19:07.121 INFO:tasks.workunit.client.1.smithi196.stdout:8/689: read d2/d8/d47/d68/dbc/f80 [924293,78783] 0 2024-04-22T15:19:07.122 INFO:tasks.workunit.client.0.smithi096.stdout:3/947: symlink da/db/d9a/d132/l139 0 2024-04-22T15:19:07.123 INFO:tasks.workunit.client.1.smithi196.stdout:6/716: write dd/d20/f4c [4803633,128329] 0 2024-04-22T15:19:07.125 INFO:tasks.workunit.client.1.smithi196.stdout:2/698: sync 2024-04-22T15:19:07.128 INFO:tasks.workunit.client.1.smithi196.stdout:6/717: dwrite dd/d1f/d38/d40/f4f [0,4194304] 0 2024-04-22T15:19:07.132 INFO:tasks.workunit.client.0.smithi096.stdout:2/938: dwrite d3/db/df/dce/d46/f80 [0,4194304] 0 2024-04-22T15:19:07.133 INFO:tasks.workunit.client.0.smithi096.stdout:1/936: dread d6/dc/d8a/d3f/d54/d32/d3b/d3d/f63 [4194304,4194304] 0 2024-04-22T15:19:07.134 INFO:tasks.workunit.client.1.smithi196.stdout:4/684: truncate d6/d13/d15/f6f 436500 0 2024-04-22T15:19:07.134 INFO:tasks.workunit.client.0.smithi096.stdout:2/939: stat d3/db/df/d24/d58/d5f/dfe/d115/d5a/fc1 0 2024-04-22T15:19:07.138 INFO:tasks.workunit.client.0.smithi096.stdout:3/948: rmdir da/db/dc/d59 39 2024-04-22T15:19:07.140 INFO:tasks.workunit.client.1.smithi196.stdout:9/675: fdatasync db/dd/de/d15/d31/fb7 0 2024-04-22T15:19:07.141 INFO:tasks.workunit.client.1.smithi196.stdout:9/676: write db/d11/d29/d57/f23 [389947,29310] 0 2024-04-22T15:19:07.142 INFO:tasks.workunit.client.1.smithi196.stdout:9/677: write db/dd/d1b/d2c/fb5 [2689943,54480] 0 2024-04-22T15:19:07.146 INFO:tasks.workunit.client.0.smithi096.stdout:3/949: dwrite da/db/d9a/d132/f137 [0,4194304] 0 2024-04-22T15:19:07.158 INFO:tasks.workunit.client.1.smithi196.stdout:1/695: sync 2024-04-22T15:19:07.159 INFO:tasks.workunit.client.0.smithi096.stdout:4/916: rename d5/d7/d2c/d67 to d5/d7/d2c/d3d/d13d 0 2024-04-22T15:19:07.162 INFO:tasks.workunit.client.1.smithi196.stdout:1/696: write d1/db/dc/de/d5a/d94/dab/fd0 [1667734,119379] 0 2024-04-22T15:19:07.169 INFO:tasks.workunit.client.1.smithi196.stdout:4/685: rmdir d6/d14 39 2024-04-22T15:19:07.173 INFO:tasks.workunit.client.1.smithi196.stdout:3/701: fsync d0/f1 0 2024-04-22T15:19:07.179 INFO:tasks.workunit.client.0.smithi096.stdout:3/950: fsync da/db/d9a/d132/f137 0 2024-04-22T15:19:07.181 INFO:tasks.workunit.client.0.smithi096.stdout:4/917: mkdir d5/d7/d1f/d84/d6d/d118/d13e 0 2024-04-22T15:19:07.185 INFO:tasks.workunit.client.1.smithi196.stdout:7/675: truncate d14/d25/d31/f55 1989441 0 2024-04-22T15:19:07.187 INFO:tasks.workunit.client.0.smithi096.stdout:8/894: creat d4/d1c/d40/f12f x:0 0 0 2024-04-22T15:19:07.187 INFO:tasks.workunit.client.0.smithi096.stdout:1/937: dwrite d6/dc/dbd/fc9 [0,4194304] 0 2024-04-22T15:19:07.188 INFO:tasks.workunit.client.1.smithi196.stdout:6/718: dwrite dd/d1f/d37/f31 [4194304,4194304] 0 2024-04-22T15:19:07.191 INFO:tasks.workunit.client.0.smithi096.stdout:6/768: link d3/d28/d54/d68/d6e/cdb d3/d3a/d75/c102 0 2024-04-22T15:19:07.192 INFO:tasks.workunit.client.1.smithi196.stdout:7/676: dwrite d14/d15/f74 [0,4194304] 0 2024-04-22T15:19:07.193 INFO:tasks.workunit.client.1.smithi196.stdout:6/719: dread dd/d1f/d38/d40/f4f [0,4194304] 0 2024-04-22T15:19:07.194 INFO:tasks.workunit.client.1.smithi196.stdout:7/677: read d14/f22 [1566133,26592] 0 2024-04-22T15:19:07.196 INFO:tasks.workunit.client.0.smithi096.stdout:5/950: rename d4/fd to d4/ddb/d3d/d4d/d99/f14b 0 2024-04-22T15:19:07.196 INFO:tasks.workunit.client.0.smithi096.stdout:2/940: dwrite d3/db/df/d24/d58/d5f/dfe/d115/f7a [0,4194304] 0 2024-04-22T15:19:07.198 INFO:tasks.workunit.client.0.smithi096.stdout:4/918: unlink d5/d7/d1a/d25/c33 0 2024-04-22T15:19:07.199 INFO:tasks.workunit.client.1.smithi196.stdout:2/699: mkdir d1/d23/d27/d100 0 2024-04-22T15:19:07.200 INFO:tasks.workunit.client.0.smithi096.stdout:9/837: dread d1/df/d95/d9d/db5/d43/d57/f113 [0,4194304] 0 2024-04-22T15:19:07.202 INFO:tasks.workunit.client.0.smithi096.stdout:1/938: read - d6/dc/d8a/d3f/fda zero size 2024-04-22T15:19:07.203 INFO:tasks.workunit.client.0.smithi096.stdout:1/939: dread - d6/f10c zero size 2024-04-22T15:19:07.205 INFO:tasks.workunit.client.0.smithi096.stdout:1/940: dread d6/dc/d8a/d3f/d54/d32/d3b/d3d/f8b [0,4194304] 0 2024-04-22T15:19:07.212 INFO:tasks.workunit.client.0.smithi096.stdout:6/769: dread - d3/d28/d4d/d82/dd8/fe1 zero size 2024-04-22T15:19:07.216 INFO:tasks.workunit.client.0.smithi096.stdout:3/951: mknod da/db/d9a/d132/d106/c13a 0 2024-04-22T15:19:07.217 INFO:tasks.workunit.client.0.smithi096.stdout:3/952: chown da/db/dc/ffb 739412 1 2024-04-22T15:19:07.218 INFO:tasks.workunit.client.0.smithi096.stdout:3/953: fsync da/db/de/d19/d3e/fde 0 2024-04-22T15:19:07.219 INFO:tasks.workunit.client.1.smithi196.stdout:3/702: dread d0/d15/f12 [0,4194304] 0 2024-04-22T15:19:07.219 INFO:tasks.workunit.client.1.smithi196.stdout:3/703: fdatasync d0/f3 0 2024-04-22T15:19:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.222+0000 7fb10b001700 1 -- 172.21.15.96:0/2311011466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc095c50 msgr2=0x7fb0fc0960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.222+0000 7fb10b001700 1 --2- 172.21.15.96:0/2311011466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc095c50 0x7fb0fc0960b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009910 tx=0x7fb0f4009c20 comp rx=0 tx=0).stop 2024-04-22T15:19:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.222+0000 7fb10b001700 1 -- 172.21.15.96:0/2311011466 shutdown_connections 2024-04-22T15:19:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.222+0000 7fb10b001700 1 --2- 172.21.15.96:0/2311011466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc095c50 0x7fb0fc0960b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.222+0000 7fb10b001700 1 --2- 172.21.15.96:0/2311011466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0fc094a50 0x7fb0fc094e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.222+0000 7fb10b001700 1 -- 172.21.15.96:0/2311011466 >> 172.21.15.96:0/2311011466 conn(0x7fb0fc08fff0 msgr2=0x7fb0fc092430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.223+0000 7fb10b001700 1 -- 172.21.15.96:0/2311011466 shutdown_connections 2024-04-22T15:19:07.227 INFO:tasks.workunit.client.0.smithi096.stdout:4/919: chown d5/d7/d2c/d3d/d13d/ld6 0 1 2024-04-22T15:19:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.223+0000 7fb10b001700 1 -- 172.21.15.96:0/2311011466 wait complete. 2024-04-22T15:19:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb10b001700 1 Processor -- start 2024-04-22T15:19:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb10b001700 1 -- start start 2024-04-22T15:19:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb10b001700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc094a50 0x7fb0fc12a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb10b001700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0fc095c50 0x7fb0fc12a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb10b001700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc12af90 con 0x7fb0fc094a50 2024-04-22T15:19:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb10b001700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc12b0d0 con 0x7fb0fc095c50 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb109fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc094a50 0x7fb0fc12a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb109fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc094a50 0x7fb0fc12a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34874/0 (socket says 172.21.15.96:34874) 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb109fff700 1 -- 172.21.15.96:0/2354780486 learned_addr learned my addr 172.21.15.96:0/2354780486 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb1097fe700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0fc095c50 0x7fb0fc12a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb1097fe700 1 -- 172.21.15.96:0/2354780486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc094a50 msgr2=0x7fb0fc12a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb1097fe700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc094a50 0x7fb0fc12a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.224+0000 7fb1097fe700 1 -- 172.21.15.96:0/2354780486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f40095f0 con 0x7fb0fc095c50 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.225+0000 7fb109fff700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc094a50 0x7fb0fc12a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.225+0000 7fb1097fe700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0fc095c50 0x7fb0fc12a970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40059d0 tx=0x7fb0f400bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.225+0000 7fb0faffd700 1 -- 172.21.15.96:0/2354780486 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0f401d070 con 0x7fb0fc095c50 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.225+0000 7fb0faffd700 1 -- 172.21.15.96:0/2354780486 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0f4003d50 con 0x7fb0fc095c50 2024-04-22T15:19:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.225+0000 7fb0faffd700 1 -- 172.21.15.96:0/2354780486 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0f4021410 con 0x7fb0fc095c50 2024-04-22T15:19:07.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.225+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0fc006160 con 0x7fb0fc095c50 2024-04-22T15:19:07.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.225+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0fc0066d0 con 0x7fb0fc095c50 2024-04-22T15:19:07.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.227+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc0055a0 con 0x7fb0fc095c50 2024-04-22T15:19:07.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.227+0000 7fb0faffd700 1 -- 172.21.15.96:0/2354780486 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7fb0f400f460 con 0x7fb0fc095c50 2024-04-22T15:19:07.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.228+0000 7fb0faffd700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fb0f006dc50 0x7fb0f0070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:07.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.228+0000 7fb0faffd700 1 -- 172.21.15.96:0/2354780486 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fb0f408e8a0 con 0x7fb0fc095c50 2024-04-22T15:19:07.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.229+0000 7fb109fff700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fb0f006dc50 0x7fb0f0070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:07.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.229+0000 7fb109fff700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fb0f006dc50 0x7fb0f0070110 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb100009490 tx=0x7fb10000d6b0 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:07.234 INFO:tasks.workunit.client.0.smithi096.stdout:5/951: chown d4/d2c/d2f/d30/c12b 9736 1 2024-04-22T15:19:07.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.233+0000 7fb0faffd700 1 -- 172.21.15.96:0/2354780486 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fb0f405b610 con 0x7fb0fc095c50 2024-04-22T15:19:07.244 INFO:tasks.workunit.client.0.smithi096.stdout:8/895: rename d4/d1c/d40/d46/c9e to d4/d10/d4d/deb/c130 0 2024-04-22T15:19:07.253 INFO:tasks.workunit.client.0.smithi096.stdout:2/941: write d3/db/d30/f11c [516234,4068] 0 2024-04-22T15:19:07.256 INFO:tasks.workunit.client.0.smithi096.stdout:3/954: write da/db/de/ddc/d68/f80 [724352,51119] 0 2024-04-22T15:19:07.259 INFO:tasks.workunit.client.0.smithi096.stdout:3/955: truncate da/db/de/d19/d3e/fde 879470 0 2024-04-22T15:19:07.276 INFO:tasks.workunit.client.1.smithi196.stdout:5/717: link d1/d2/dbb/cda d1/d78/db0/ce7 0 2024-04-22T15:19:07.282 INFO:tasks.workunit.client.1.smithi196.stdout:0/722: rename d3/d9/d36/d9f/dd1/d65/d7f/d51/ce4 to d3/de/cff 0 2024-04-22T15:19:07.289 INFO:tasks.workunit.client.1.smithi196.stdout:9/678: mknod db/dd/de/dd7/ce3 0 2024-04-22T15:19:07.300 INFO:tasks.workunit.client.1.smithi196.stdout:2/700: read d1/d1d/f3c [3627724,88317] 0 2024-04-22T15:19:07.304 INFO:tasks.workunit.client.1.smithi196.stdout:4/686: unlink d6/d14/d18/lae 0 2024-04-22T15:19:07.308 INFO:tasks.workunit.client.1.smithi196.stdout:8/690: rename d2/d5/d9c to d2/d5/d1a/dcc 0 2024-04-22T15:19:07.308 INFO:tasks.workunit.client.1.smithi196.stdout:3/704: rename d0/d15/d31 to d0/d15/d31/dd0/de2/def 22 2024-04-22T15:19:07.309 INFO:tasks.workunit.client.1.smithi196.stdout:8/691: read d2/d8/d47/d68/d81/f89 [11804817,105298] 0 2024-04-22T15:19:07.311 INFO:tasks.workunit.client.1.smithi196.stdout:8/692: write d2/d8/d47/d68/d81/f89 [5879013,10113] 0 2024-04-22T15:19:07.318 INFO:tasks.workunit.client.0.smithi096.stdout:1/941: rename d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/d10d/l10e to d6/dc/d8a/d3f/d62/dc2/dd0/l127 0 2024-04-22T15:19:07.319 INFO:tasks.workunit.client.1.smithi196.stdout:9/679: truncate db/dd/d1b/d2c/f7c 289642 0 2024-04-22T15:19:07.320 INFO:tasks.workunit.client.0.smithi096.stdout:1/942: chown d6/dc/d8a/d3f/d54/d32/d3b/d56/ff1 679518 1 2024-04-22T15:19:07.322 INFO:tasks.workunit.client.0.smithi096.stdout:8/896: creat d4/d10/d25/d4c/db4/dd0/f131 x:0 0 0 2024-04-22T15:19:07.323 INFO:tasks.workunit.client.0.smithi096.stdout:5/952: dread d4/ddb/d3d/d35/fb8 [0,4194304] 0 2024-04-22T15:19:07.327 INFO:tasks.workunit.client.1.smithi196.stdout:6/720: creat dd/d1f/d37/dcb/fd4 x:0 0 0 2024-04-22T15:19:07.327 INFO:tasks.workunit.client.1.smithi196.stdout:6/721: dread - dd/d1f/d37/fae zero size 2024-04-22T15:19:07.333 INFO:tasks.workunit.client.1.smithi196.stdout:8/693: dread d2/dc/d45/f5b [4194304,4194304] 0 2024-04-22T15:19:07.335 INFO:tasks.workunit.client.1.smithi196.stdout:1/697: creat d1/db/dc/de/d5a/d5b/fe1 x:0 0 0 2024-04-22T15:19:07.339 INFO:tasks.workunit.client.1.smithi196.stdout:4/687: rename d6/d14/f27 to d6/d14/d18/d26/d32/da3/db7/dc9/fda 0 2024-04-22T15:19:07.339 INFO:tasks.workunit.client.1.smithi196.stdout:6/722: dread dd/d1f/d38/d40/f42 [4194304,4194304] 0 2024-04-22T15:19:07.340 INFO:tasks.workunit.client.1.smithi196.stdout:0/723: symlink d3/de/d14/d86/dea/l100 0 2024-04-22T15:19:07.340 INFO:tasks.workunit.client.1.smithi196.stdout:7/678: dwrite d14/d25/d31/f65 [0,4194304] 0 2024-04-22T15:19:07.344 INFO:tasks.workunit.client.1.smithi196.stdout:2/701: mkdir d1/d13/d101 0 2024-04-22T15:19:07.346 INFO:tasks.workunit.client.1.smithi196.stdout:0/724: dread d3/de/d1c/d27/f38 [4194304,4194304] 0 2024-04-22T15:19:07.346 INFO:tasks.workunit.client.1.smithi196.stdout:8/694: symlink d2/d5/d1a/dcc/lcd 0 2024-04-22T15:19:07.352 INFO:tasks.workunit.client.1.smithi196.stdout:4/688: creat d6/d13/d15/d2c/d62/d78/d8d/fdb x:0 0 0 2024-04-22T15:19:07.353 INFO:tasks.workunit.client.1.smithi196.stdout:6/723: read dd/d1f/d37/f5d [12757,122821] 0 2024-04-22T15:19:07.354 INFO:tasks.workunit.client.1.smithi196.stdout:7/679: truncate d14/f33 1266463 0 2024-04-22T15:19:07.355 INFO:tasks.workunit.client.1.smithi196.stdout:7/680: chown d14/d26/d28/fab 1 1 2024-04-22T15:19:07.356 INFO:tasks.workunit.client.1.smithi196.stdout:4/689: dread d6/d13/d15/d2c/d62/d78/d8d/f9c [0,4194304] 0 2024-04-22T15:19:07.359 INFO:tasks.workunit.client.1.smithi196.stdout:5/718: creat d1/d14/d6a/fe8 x:0 0 0 2024-04-22T15:19:07.361 INFO:tasks.workunit.client.1.smithi196.stdout:4/690: dwrite d6/d14/d18/d26/d32/d53/f94 [0,4194304] 0 2024-04-22T15:19:07.407 INFO:tasks.workunit.client.1.smithi196.stdout:3/705: sync 2024-04-22T15:19:07.418 INFO:tasks.workunit.client.0.smithi096.stdout:2/942: creat d3/db/df/d24/d58/d5f/dfe/d115/f132 x:0 0 0 2024-04-22T15:19:07.419 INFO:tasks.workunit.client.0.smithi096.stdout:8/897: creat d4/d1c/d40/d46/d58/f132 x:0 0 0 2024-04-22T15:19:07.420 INFO:tasks.workunit.client.1.smithi196.stdout:8/695: rename d2/dc/la2 to d2/d8/d49/d6c/lce 0 2024-04-22T15:19:07.423 INFO:tasks.workunit.client.1.smithi196.stdout:8/696: dread d2/d35/fb4 [0,4194304] 0 2024-04-22T15:19:07.427 INFO:tasks.workunit.client.1.smithi196.stdout:6/724: mknod dd/d1f/d37/d58/cd5 0 2024-04-22T15:19:07.429 INFO:tasks.workunit.client.1.smithi196.stdout:9/680: link db/dd/de/d15/d31/d64/lae db/dd/de/d15/le4 0 2024-04-22T15:19:07.430 INFO:tasks.workunit.client.1.smithi196.stdout:7/681: truncate d14/d15/f1a 439052 0 2024-04-22T15:19:07.433 INFO:tasks.workunit.client.1.smithi196.stdout:2/702: mkdir d1/d23/d27/d2e/db5/d102 0 2024-04-22T15:19:07.435 INFO:tasks.workunit.client.1.smithi196.stdout:1/698: dwrite d1/db/dc/de/d43/d5e/d7a/f83 [0,4194304] 0 2024-04-22T15:19:07.436 INFO:tasks.workunit.client.1.smithi196.stdout:4/691: truncate d6/d13/d15/d2c/d62/f88 233392 0 2024-04-22T15:19:07.438 INFO:tasks.workunit.client.1.smithi196.stdout:3/706: creat d0/d15/da7/d4a/d44/d8c/ff0 x:0 0 0 2024-04-22T15:19:07.444 INFO:tasks.workunit.client.1.smithi196.stdout:0/725: dwrite d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/fc0 [0,4194304] 0 2024-04-22T15:19:07.445 INFO:tasks.workunit.client.1.smithi196.stdout:6/725: dread dd/d20/d4d/f65 [0,4194304] 0 2024-04-22T15:19:07.446 INFO:tasks.workunit.client.1.smithi196.stdout:7/682: mkdir d14/d26/d2a/d2b/dc7/de1 0 2024-04-22T15:19:07.447 INFO:tasks.workunit.client.1.smithi196.stdout:2/703: readlink d1/d13/l51 0 2024-04-22T15:19:07.449 INFO:tasks.workunit.client.1.smithi196.stdout:5/719: fsync d1/d14/d6a/fa8 0 2024-04-22T15:19:07.451 INFO:tasks.workunit.client.1.smithi196.stdout:4/692: mkdir d6/d13/d15/d2c/d62/d78/ddc 0 2024-04-22T15:19:07.453 INFO:tasks.workunit.client.1.smithi196.stdout:3/707: creat d0/d15/d31/d55/dcf/d6f/db5/dd7/ff1 x:0 0 0 2024-04-22T15:19:07.456 INFO:tasks.workunit.client.1.smithi196.stdout:9/681: creat db/dd/dd0/fe5 x:0 0 0 2024-04-22T15:19:07.460 INFO:tasks.workunit.client.1.smithi196.stdout:1/699: dread d1/db/dc/d26/d2e/f34 [0,4194304] 0 2024-04-22T15:19:07.461 INFO:tasks.workunit.client.1.smithi196.stdout:0/726: mknod d3/d9/d36/d9f/dd1/d65/c101 0 2024-04-22T15:19:07.462 INFO:tasks.workunit.client.1.smithi196.stdout:1/700: write d1/db/dc/de/d43/f68 [3043471,95137] 0 2024-04-22T15:19:07.462 INFO:tasks.workunit.client.1.smithi196.stdout:7/683: unlink d14/d26/d5a/d5f/l6f 0 2024-04-22T15:19:07.463 INFO:tasks.workunit.client.1.smithi196.stdout:1/701: write d1/db/dc/de/f4a [785252,105340] 0 2024-04-22T15:19:07.465 INFO:tasks.workunit.client.1.smithi196.stdout:7/684: write d14/d26/d5a/d5f/faa [871033,115331] 0 2024-04-22T15:19:07.467 INFO:tasks.workunit.client.1.smithi196.stdout:2/704: fsync d1/d13/d1f/f9d 0 2024-04-22T15:19:07.467 INFO:tasks.workunit.client.1.smithi196.stdout:7/685: read d14/d25/f42 [4072434,107505] 0 2024-04-22T15:19:07.474 INFO:tasks.workunit.client.1.smithi196.stdout:3/708: dread d0/d15/d31/d55/d8b/fc0 [0,4194304] 0 2024-04-22T15:19:07.474 INFO:tasks.workunit.client.1.smithi196.stdout:7/686: dread - d14/d26/d28/fa1 zero size 2024-04-22T15:19:07.476 INFO:tasks.workunit.client.1.smithi196.stdout:4/693: fdatasync d6/f42 0 2024-04-22T15:19:07.477 INFO:tasks.workunit.client.1.smithi196.stdout:9/682: mkdir db/dd/d1b/de6 0 2024-04-22T15:19:07.478 INFO:tasks.workunit.client.1.smithi196.stdout:2/705: dwrite d1/da4/fe9 [0,4194304] 0 2024-04-22T15:19:07.482 INFO:tasks.workunit.client.1.smithi196.stdout:8/697: dwrite d2/d8/f25 [0,4194304] 0 2024-04-22T15:19:07.482 INFO:tasks.workunit.client.1.smithi196.stdout:3/709: mkdir d0/d15/d31/d55/d8b/df2 0 2024-04-22T15:19:07.483 INFO:tasks.workunit.client.1.smithi196.stdout:3/710: stat d0/d15/d31/c6a 0 2024-04-22T15:19:07.485 INFO:tasks.workunit.client.1.smithi196.stdout:4/694: symlink d6/d14/d18/d40/ldd 0 2024-04-22T15:19:07.486 INFO:tasks.workunit.client.1.smithi196.stdout:5/720: write d1/d17/f60 [4890278,122093] 0 2024-04-22T15:19:07.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:07 smithi096 ceph-mon[26688]: pgmap v40: 65 pgs: 65 active+clean; 3.1 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 82 MiB/s rd, 183 MiB/s wr, 409 op/s 2024-04-22T15:19:07.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:07 smithi096 ceph-mon[26688]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:07.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:07 smithi096 ceph-mon[26688]: from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:07.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:07 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:19:07.487 INFO:tasks.workunit.client.1.smithi196.stdout:6/726: dwrite dd/d1f/d37/f66 [0,4194304] 0 2024-04-22T15:19:07.488 INFO:tasks.workunit.client.1.smithi196.stdout:2/706: stat d1/d13/d41/f6c 0 2024-04-22T15:19:07.488 INFO:tasks.workunit.client.1.smithi196.stdout:6/727: chown dd/d2b/f2f 24 1 2024-04-22T15:19:07.489 INFO:tasks.workunit.client.1.smithi196.stdout:7/687: unlink d14/d46/l62 0 2024-04-22T15:19:07.489 INFO:tasks.workunit.client.1.smithi196.stdout:6/728: read - dd/d1f/d37/fae zero size 2024-04-22T15:19:07.490 INFO:tasks.workunit.client.1.smithi196.stdout:7/688: readlink d14/d25/d31/l41 0 2024-04-22T15:19:07.493 INFO:tasks.workunit.client.1.smithi196.stdout:9/683: unlink db/d11/d29/d7a/d7b/ca0 0 2024-04-22T15:19:07.494 INFO:tasks.workunit.client.1.smithi196.stdout:1/702: getdents d1/db/d16/d6d 0 2024-04-22T15:19:07.498 INFO:tasks.workunit.client.1.smithi196.stdout:8/698: rmdir d2/d8/d49/d6c 39 2024-04-22T15:19:07.504 INFO:tasks.workunit.client.0.smithi096.stdout:5/953: dwrite d4/ddb/d3d/d4d/f70 [0,4194304] 0 2024-04-22T15:19:07.508 INFO:tasks.workunit.client.1.smithi196.stdout:3/711: creat d0/d15/da7/d4a/d44/ff3 x:0 0 0 2024-04-22T15:19:07.510 INFO:tasks.workunit.client.1.smithi196.stdout:4/695: dread - d6/d13/d15/d2c/d62/f97 zero size 2024-04-22T15:19:07.511 INFO:tasks.workunit.client.0.smithi096.stdout:6/770: dwrite d3/d28/f61 [0,4194304] 0 2024-04-22T15:19:07.512 INFO:tasks.workunit.client.1.smithi196.stdout:5/721: write d1/d17/d1a/d2f/d30/d5e/fca [604228,45968] 0 2024-04-22T15:19:07.514 INFO:tasks.workunit.client.1.smithi196.stdout:6/729: symlink dd/d20/d5c/d6a/ld6 0 2024-04-22T15:19:07.516 INFO:tasks.workunit.client.0.smithi096.stdout:4/920: dwrite d5/d7/d2c/d3d/fa5 [0,4194304] 0 2024-04-22T15:19:07.517 INFO:tasks.workunit.client.0.smithi096.stdout:9/838: dwrite d1/d25/d9f/f104 [0,4194304] 0 2024-04-22T15:19:07.521 INFO:tasks.workunit.client.0.smithi096.stdout:2/943: truncate d3/db/df/d10/f37 1850109 0 2024-04-22T15:19:07.522 INFO:tasks.workunit.client.0.smithi096.stdout:1/943: dread d6/dc/d8a/d3f/d62/d7d/ddb/f123 [0,4194304] 0 2024-04-22T15:19:07.522 INFO:tasks.workunit.client.0.smithi096.stdout:6/771: mknod d3/d28/daf/c103 0 2024-04-22T15:19:07.526 INFO:tasks.workunit.client.0.smithi096.stdout:2/944: chown d3/db/df/d24/d58/d5f/dfe/d115/c68 899315 1 2024-04-22T15:19:07.527 INFO:tasks.workunit.client.0.smithi096.stdout:9/839: chown d1/df/l91 1 1 2024-04-22T15:19:07.528 INFO:tasks.workunit.client.0.smithi096.stdout:6/772: mknod d3/d3a/c104 0 2024-04-22T15:19:07.528 INFO:tasks.workunit.client.0.smithi096.stdout:9/840: readlink d1/d4/d79/l8b 0 2024-04-22T15:19:07.529 INFO:tasks.workunit.client.0.smithi096.stdout:9/841: stat d1 0 2024-04-22T15:19:07.531 INFO:tasks.workunit.client.0.smithi096.stdout:4/921: fsync d5/d7/f7f 0 2024-04-22T15:19:07.538 INFO:tasks.workunit.client.0.smithi096.stdout:6/773: readlink d3/d28/dd/l9d 0 2024-04-22T15:19:07.539 INFO:tasks.workunit.client.1.smithi196.stdout:5/722: write d1/d17/d1a/d2f/d30/d6e/fa2 [1608424,110715] 0 2024-04-22T15:19:07.540 INFO:tasks.workunit.client.0.smithi096.stdout:6/774: readlink d3/d3a/d75/d83/l8f 0 2024-04-22T15:19:07.540 INFO:tasks.workunit.client.1.smithi196.stdout:1/703: mkdir d1/d54/de2 0 2024-04-22T15:19:07.541 INFO:tasks.workunit.client.1.smithi196.stdout:4/696: rename d6/d14/f33 to d6/d13/d15/d2c/d62/dd6/fde 0 2024-04-22T15:19:07.542 INFO:tasks.workunit.client.0.smithi096.stdout:4/922: dwrite d5/d7/fed [0,4194304] 0 2024-04-22T15:19:07.542 INFO:tasks.workunit.client.1.smithi196.stdout:2/707: symlink d1/d13/d25/d2a/ddb/dff/l103 0 2024-04-22T15:19:07.544 INFO:tasks.workunit.client.1.smithi196.stdout:4/697: chown d6/f2b 1698420 1 2024-04-22T15:19:07.545 INFO:tasks.workunit.client.0.smithi096.stdout:5/954: getdents d4/ddb/d3d/d4d/d59/db0/db2 0 2024-04-22T15:19:07.545 INFO:tasks.workunit.client.1.smithi196.stdout:2/708: chown d1/d23/d27/d2e/d55/d5b/d93 51927 1 2024-04-22T15:19:07.549 INFO:tasks.workunit.client.0.smithi096.stdout:2/945: dread d3/db/d30/f39 [0,4194304] 0 2024-04-22T15:19:07.550 INFO:tasks.workunit.client.1.smithi196.stdout:9/684: dwrite db/dd/de/d15/d31/f63 [0,4194304] 0 2024-04-22T15:19:07.553 INFO:tasks.workunit.client.0.smithi096.stdout:8/898: dread d4/d10/d25/f37 [0,4194304] 0 2024-04-22T15:19:07.554 INFO:tasks.workunit.client.0.smithi096.stdout:6/775: dwrite d3/d28/dc9/feb [0,4194304] 0 2024-04-22T15:19:07.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.553+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0fc124120 con 0x7fb0f006dc50 2024-04-22T15:19:07.562 INFO:tasks.workunit.client.0.smithi096.stdout:5/955: dwrite d4/ddb/d3d/d4d/d59/d98/dee/d109/f10c [0,4194304] 0 2024-04-22T15:19:07.564 INFO:tasks.workunit.client.0.smithi096.stdout:1/944: write d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/fc6 [4352521,49199] 0 2024-04-22T15:19:07.567 INFO:tasks.workunit.client.1.smithi196.stdout:1/704: mknod d1/db/d16/d1e/d46/ddc/ce3 0 2024-04-22T15:19:07.569 INFO:tasks.workunit.client.1.smithi196.stdout:1/705: write d1/db/dc/de/d5a/d94/dab/fc2 [431676,106916] 0 2024-04-22T15:19:07.573 INFO:tasks.workunit.client.1.smithi196.stdout:9/685: creat db/dd/de/dd7/fe7 x:0 0 0 2024-04-22T15:19:07.574 INFO:tasks.workunit.client.0.smithi096.stdout:4/923: creat d5/d7/dcb/d12a/f13f x:0 0 0 2024-04-22T15:19:07.574 INFO:tasks.workunit.client.0.smithi096.stdout:9/842: link d1/f4f d1/df/d95/d9d/db5/dd/dba/dc2/de5/f11b 0 2024-04-22T15:19:07.575 INFO:tasks.workunit.client.1.smithi196.stdout:9/686: chown db/dd/de/d75/fbf 17677357 1 2024-04-22T15:19:07.575 INFO:tasks.workunit.client.1.smithi196.stdout:6/730: rmdir dd/d1f/d4e/d53/d55/d57/dc0 0 2024-04-22T15:19:07.576 INFO:tasks.workunit.client.1.smithi196.stdout:3/712: dread d0/d15/d31/d55/dcf/d6f/db5/dd7/fda [0,4194304] 0 2024-04-22T15:19:07.576 INFO:tasks.workunit.client.1.smithi196.stdout:9/687: stat db/dd/de/f10 0 2024-04-22T15:19:07.578 INFO:tasks.workunit.client.1.smithi196.stdout:4/698: dread d6/d14/d18/d40/f6c [0,4194304] 0 2024-04-22T15:19:07.579 INFO:tasks.workunit.client.0.smithi096.stdout:6/776: creat d3/d3a/d75/f105 x:0 0 0 2024-04-22T15:19:07.579 INFO:tasks.workunit.client.1.smithi196.stdout:5/723: dread d1/d4/fc2 [0,4194304] 0 2024-04-22T15:19:07.580 INFO:tasks.workunit.client.1.smithi196.stdout:6/731: dread - dd/d1f/f47 zero size 2024-04-22T15:19:07.580 INFO:tasks.workunit.client.1.smithi196.stdout:5/724: chown d1/d17/d1a/d2f/d30/f54 431715 1 2024-04-22T15:19:07.581 INFO:tasks.workunit.client.0.smithi096.stdout:1/945: rename d6/dc/d8a/d3f/d62/l68 to d6/dc/d8a/d3f/d54/d32/dcd/l128 0 2024-04-22T15:19:07.581 INFO:tasks.workunit.client.1.smithi196.stdout:5/725: fsync d1/d14/d6a/fb8 0 2024-04-22T15:19:07.582 INFO:tasks.workunit.client.1.smithi196.stdout:9/688: read db/d11/d29/d7a/f7d [494222,127475] 0 2024-04-22T15:19:07.583 INFO:tasks.workunit.client.0.smithi096.stdout:1/946: write d6/dc/d8a/d3f/d54/d32/d3b/d56/ffc [922984,130868] 0 2024-04-22T15:19:07.584 INFO:tasks.workunit.client.0.smithi096.stdout:9/843: symlink d1/d25/db2/dca/l11c 0 2024-04-22T15:19:07.585 INFO:tasks.workunit.client.1.smithi196.stdout:5/726: dread d1/d17/d1c/f57 [0,4194304] 0 2024-04-22T15:19:07.586 INFO:tasks.workunit.client.1.smithi196.stdout:4/699: creat d6/d13/d15/d2c/d8c/db3/fdf x:0 0 0 2024-04-22T15:19:07.589 INFO:tasks.workunit.client.1.smithi196.stdout:4/700: read - d6/d14/d18/d26/d32/d53/db6/fd5 zero size 2024-04-22T15:19:07.591 INFO:tasks.workunit.client.0.smithi096.stdout:1/947: fdatasync d6/dc/d8a/d29/ff8 0 2024-04-22T15:19:07.592 INFO:tasks.workunit.client.1.smithi196.stdout:3/713: symlink d0/d15/d4d/de7/lf4 0 2024-04-22T15:19:07.593 INFO:tasks.workunit.client.1.smithi196.stdout:0/727: sync 2024-04-22T15:19:07.596 INFO:tasks.workunit.client.1.smithi196.stdout:5/727: mknod d1/d2/d98/dac/ce9 0 2024-04-22T15:19:07.598 INFO:tasks.workunit.client.1.smithi196.stdout:6/732: fsync dd/d1f/d38/f79 0 2024-04-22T15:19:07.599 INFO:tasks.workunit.client.1.smithi196.stdout:3/714: dwrite d0/d15/d31/d55/dcf/d8/d6b/fd6 [0,4194304] 0 2024-04-22T15:19:07.600 INFO:tasks.workunit.client.0.smithi096.stdout:8/899: dread d4/f32 [0,4194304] 0 2024-04-22T15:19:07.602 INFO:tasks.workunit.client.0.smithi096.stdout:2/946: dread d3/db/df/dce/d46/f7c [0,4194304] 0 2024-04-22T15:19:07.603 INFO:tasks.workunit.client.1.smithi196.stdout:2/709: write d1/d1d/f57 [3750446,26992] 0 2024-04-22T15:19:07.604 INFO:tasks.workunit.client.0.smithi096.stdout:9/844: symlink d1/df/d95/d9d/db5/d43/d57/d5b/d73/l11d 0 2024-04-22T15:19:07.605 INFO:tasks.workunit.client.0.smithi096.stdout:2/947: write d3/db/df/d24/d58/d5f/dfe/d115/d5a/f120 [3215626,3423] 0 2024-04-22T15:19:07.606 INFO:tasks.workunit.client.1.smithi196.stdout:5/728: dwrite d1/d2/d98/fcc [0,4194304] 0 2024-04-22T15:19:07.609 INFO:tasks.workunit.client.1.smithi196.stdout:0/728: mkdir d3/de/d14/d86/dea/d102 0 2024-04-22T15:19:07.612 INFO:tasks.workunit.client.1.smithi196.stdout:1/706: dread d1/db/dc/de/d43/f6b [0,4194304] 0 2024-04-22T15:19:07.614 INFO:tasks.workunit.client.1.smithi196.stdout:2/710: dread d1/d23/d27/d73/fc6 [0,4194304] 0 2024-04-22T15:19:07.616 INFO:tasks.workunit.client.0.smithi096.stdout:6/777: write d3/d28/f1c [3751704,130469] 0 2024-04-22T15:19:07.617 INFO:tasks.workunit.client.0.smithi096.stdout:6/778: chown d3/d3a/d75/d83 65783 1 2024-04-22T15:19:07.619 INFO:tasks.workunit.client.1.smithi196.stdout:6/733: rmdir dd/d1f/d4e/d53/d55/d6d/dd1 39 2024-04-22T15:19:07.620 INFO:tasks.workunit.client.1.smithi196.stdout:2/711: dwrite d1/d23/f6d [4194304,4194304] 0 2024-04-22T15:19:07.620 INFO:tasks.workunit.client.1.smithi196.stdout:6/734: write dd/f11 [2678689,82635] 0 2024-04-22T15:19:07.622 INFO:tasks.workunit.client.1.smithi196.stdout:3/715: mkdir d0/d15/d31/d55/dcf/df5 0 2024-04-22T15:19:07.622 INFO:tasks.workunit.client.1.smithi196.stdout:3/716: stat d0/l95 0 2024-04-22T15:19:07.624 INFO:tasks.workunit.client.1.smithi196.stdout:9/689: creat db/dd/de/d15/d31/fe8 x:0 0 0 2024-04-22T15:19:07.625 INFO:tasks.workunit.client.0.smithi096.stdout:6/779: dwrite d3/d28/dd/d11/d2c/f39 [0,4194304] 0 2024-04-22T15:19:07.625 INFO:tasks.workunit.client.0.smithi096.stdout:5/956: link d4/ddb/d3d/d4d/d59/d5a/d5c/f89 d4/d1f/f14c 0 2024-04-22T15:19:07.626 INFO:tasks.workunit.client.1.smithi196.stdout:9/690: fdatasync db/d11/d29/d57/f23 0 2024-04-22T15:19:07.628 INFO:tasks.workunit.client.1.smithi196.stdout:5/729: mknod d1/d2/d98/dac/cea 0 2024-04-22T15:19:07.630 INFO:tasks.workunit.client.1.smithi196.stdout:3/717: dwrite d0/d15/f7a [0,4194304] 0 2024-04-22T15:19:07.632 INFO:tasks.workunit.client.1.smithi196.stdout:3/718: chown d0/d15/da7/d4a/d44/f6d 2025 1 2024-04-22T15:19:07.636 INFO:tasks.workunit.client.0.smithi096.stdout:2/948: creat d3/db/df/d123/f133 x:0 0 0 2024-04-22T15:19:07.640 INFO:tasks.workunit.client.1.smithi196.stdout:4/701: creat d6/d13/d15/d2c/d35/fe0 x:0 0 0 2024-04-22T15:19:07.641 INFO:tasks.workunit.client.0.smithi096.stdout:8/900: dwrite d4/d2a/f36 [0,4194304] 0 2024-04-22T15:19:07.642 INFO:tasks.workunit.client.1.smithi196.stdout:3/719: dwrite d0/d15/d31/d55/dcf/d8/fd4 [0,4194304] 0 2024-04-22T15:19:07.645 INFO:tasks.workunit.client.1.smithi196.stdout:7/689: sync 2024-04-22T15:19:07.647 INFO:tasks.workunit.client.1.smithi196.stdout:2/712: readlink d1/d23/d84/l8f 0 2024-04-22T15:19:07.649 INFO:tasks.workunit.client.0.smithi096.stdout:9/845: dwrite d1/df/d95/d9d/fa8 [0,4194304] 0 2024-04-22T15:19:07.651 INFO:tasks.workunit.client.0.smithi096.stdout:1/948: getdents d6/d97 0 2024-04-22T15:19:07.652 INFO:tasks.workunit.client.1.smithi196.stdout:9/691: chown db/c59 68 1 2024-04-22T15:19:07.655 INFO:tasks.workunit.client.0.smithi096.stdout:8/901: mknod d4/d10/d4d/deb/c133 0 2024-04-22T15:19:07.656 INFO:tasks.workunit.client.1.smithi196.stdout:0/729: dwrite d3/f99 [0,4194304] 0 2024-04-22T15:19:07.656 INFO:tasks.workunit.client.0.smithi096.stdout:2/949: creat d3/db/df/dce/d46/d69/dd7/f134 x:0 0 0 2024-04-22T15:19:07.658 INFO:tasks.workunit.client.0.smithi096.stdout:2/950: write d3/db/df/d24/d58/d5f/dfe/d115/d5a/f120 [1294185,46101] 0 2024-04-22T15:19:07.660 INFO:tasks.workunit.client.1.smithi196.stdout:1/707: truncate d1/db/d16/d1e/f88 131226 0 2024-04-22T15:19:07.661 INFO:tasks.workunit.client.1.smithi196.stdout:1/708: write d1/db/dc/d26/f28 [2323717,42079] 0 2024-04-22T15:19:07.661 INFO:tasks.workunit.client.0.smithi096.stdout:6/780: rename d3/d28/d54/d7e to d3/d28/dd/d11/d2c/d106 0 2024-04-22T15:19:07.663 INFO:tasks.workunit.client.0.smithi096.stdout:6/781: write d3/d3a/f3e [8709063,98841] 0 2024-04-22T15:19:07.663 INFO:tasks.workunit.client.0.smithi096.stdout:9/846: creat d1/df/d95/d9d/db5/dd/dba/dc2/f11e x:0 0 0 2024-04-22T15:19:07.664 INFO:tasks.workunit.client.0.smithi096.stdout:8/902: rmdir d4/d10/d25/d38/d8f 39 2024-04-22T15:19:07.665 INFO:tasks.workunit.client.1.smithi196.stdout:3/720: readlink d0/d15/da7/d85/lec 0 2024-04-22T15:19:07.667 INFO:tasks.workunit.client.1.smithi196.stdout:7/690: symlink d14/d26/d95/le2 0 2024-04-22T15:19:07.668 INFO:tasks.workunit.client.0.smithi096.stdout:5/957: getdents d4/ddb/d3d/d4d/d88 0 2024-04-22T15:19:07.669 INFO:tasks.workunit.client.1.smithi196.stdout:7/691: stat d14/d25/d80/fbd 0 2024-04-22T15:19:07.672 INFO:tasks.workunit.client.1.smithi196.stdout:3/721: dwrite d0/d15/d31/d55/d6e/fa1 [0,4194304] 0 2024-04-22T15:19:07.672 INFO:tasks.workunit.client.0.smithi096.stdout:8/903: dwrite d4/d10/d25/d38/f54 [0,4194304] 0 2024-04-22T15:19:07.673 INFO:tasks.workunit.client.0.smithi096.stdout:8/904: fsync d4/d10/d4d/f12a 0 2024-04-22T15:19:07.678 INFO:tasks.workunit.client.0.smithi096.stdout:1/949: rename d6/dc/d8a/d3f/de9/fde to d6/dc/d8a/d3f/d54/d32/d3b/d56/f129 0 2024-04-22T15:19:07.680 INFO:tasks.workunit.client.0.smithi096.stdout:1/950: readlink d6/dc/d8a/d3f/d54/d32/d3b/l118 0 2024-04-22T15:19:07.694 INFO:tasks.workunit.client.0.smithi096.stdout:6/782: read d3/d28/dd/f19 [5072498,2802] 0 2024-04-22T15:19:07.696 INFO:tasks.workunit.client.0.smithi096.stdout:9/847: truncate d1/df/d95/d9d/db5/dd/f41 2512619 0 2024-04-22T15:19:07.696 INFO:tasks.workunit.client.1.smithi196.stdout:8/699: sync 2024-04-22T15:19:07.696 INFO:tasks.workunit.client.0.smithi096.stdout:6/783: truncate d3/d28/dd/d11/d2c/d6f/dd4/dd6/fc4 1332360 0 2024-04-22T15:19:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:07 smithi196 ceph-mon[34717]: pgmap v40: 65 pgs: 65 active+clean; 3.1 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 82 MiB/s rd, 183 MiB/s wr, 409 op/s 2024-04-22T15:19:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:07 smithi196 ceph-mon[34717]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:07 smithi196 ceph-mon[34717]: from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:07 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:19:07.809 INFO:tasks.workunit.client.1.smithi196.stdout:8/700: sync 2024-04-22T15:19:07.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.812+0000 7fb0faffd700 1 -- 172.21.15.96:0/2354780486 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb0fc124120 con 0x7fb0f006dc50 2024-04-22T15:19:07.815 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:19:07.815 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (7m) 62s ago 9m 24.7M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:19:07.815 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (9m) 62s ago 9m 7838k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:19:07.815 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (8m) 63s ago 8m 7625k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:19:07.815 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (9m) 62s ago 9m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (8m) 63s ago 8m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (7m) 62s ago 8m 84.4M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (3m) 62s ago 3m 123M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (3m) 62s ago 3m 84.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (3m) 63s ago 3m 21.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (3m) 63s ago 3m 21.4M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 starting - - - - 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (95s) 63s ago 8m 491M - 19.0.0-3223-g5883d494 3d983c26e278 8e82f21cd80c 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (10m) 62s ago 10m 47.7M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (7m) 63s ago 7m 41.1M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (9m) 62s ago 9m 14.4M - 1.5.0 0da6a335fe13 852e13a0c299 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (8m) 63s ago 8m 16.1M - 1.5.0 0da6a335fe13 4d02817c7cd0 2024-04-22T15:19:07.816 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (7m) 62s ago 7m 75.6M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:19:07.817 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (6m) 62s ago 6m 75.3M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:19:07.817 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (6m) 62s ago 6m 68.2M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:19:07.817 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (5m) 63s ago 5m 74.6M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:19:07.817 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (5m) 63s ago 5m 64.5M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:19:07.817 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (4m) 63s ago 4m 64.0M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:19:07.817 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 starting - - - - 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.814+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fb0f006dc50 msgr2=0x7fb0f0070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.814+0000 7fb10b001700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fb0f006dc50 0x7fb0f0070110 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb100009490 tx=0x7fb10000d6b0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.814+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0fc095c50 msgr2=0x7fb0fc12a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.814+0000 7fb10b001700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0fc095c50 0x7fb0fc12a970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40059d0 tx=0x7fb0f400bd70 comp rx=0 tx=0).stop 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.814+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 shutdown_connections 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.814+0000 7fb10b001700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7fb0f006dc50 0x7fb0f0070110 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.814+0000 7fb10b001700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0fc094a50 0x7fb0fc12a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.814+0000 7fb10b001700 1 --2- 172.21.15.96:0/2354780486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0fc095c50 0x7fb0fc12a970 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.815+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 >> 172.21.15.96:0/2354780486 conn(0x7fb0fc08fff0 msgr2=0x7fb0fc099200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.815+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 shutdown_connections 2024-04-22T15:19:07.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.815+0000 7fb10b001700 1 -- 172.21.15.96:0/2354780486 wait complete. 2024-04-22T15:19:07.853 INFO:tasks.workunit.client.0.smithi096.stdout:8/905: creat d4/d10/d25/df7/f134 x:0 0 0 2024-04-22T15:19:07.855 INFO:tasks.workunit.client.0.smithi096.stdout:5/958: rename d4/d1f/d80/f87 to d4/d2c/d2f/d6b/f14d 0 2024-04-22T15:19:07.856 INFO:tasks.workunit.client.0.smithi096.stdout:5/959: write d4/fcc [1075810,22335] 0 2024-04-22T15:19:07.857 INFO:tasks.workunit.client.0.smithi096.stdout:1/951: fsync d6/dc/d8a/d3f/d54/d32/d3b/d94/fad 0 2024-04-22T15:19:07.860 INFO:tasks.workunit.client.1.smithi196.stdout:2/713: getdents d1/d13/d25/d9f/ddd 0 2024-04-22T15:19:07.861 INFO:tasks.workunit.client.0.smithi096.stdout:2/951: rmdir d3/db 39 2024-04-22T15:19:07.864 INFO:tasks.workunit.client.1.smithi196.stdout:7/692: rename d14/d26/d28/fab to d14/d25/d31/fe3 0 2024-04-22T15:19:07.864 INFO:tasks.workunit.client.1.smithi196.stdout:7/693: chown d14/l86 1521161 1 2024-04-22T15:19:07.869 INFO:tasks.workunit.client.0.smithi096.stdout:8/906: read - d4/d71/fff zero size 2024-04-22T15:19:07.871 INFO:tasks.workunit.client.0.smithi096.stdout:8/907: read d4/d10/d25/d4c/f5e [366715,57654] 0 2024-04-22T15:19:07.872 INFO:tasks.workunit.client.1.smithi196.stdout:9/692: link db/fc db/dd/de/fe9 0 2024-04-22T15:19:07.872 INFO:tasks.workunit.client.1.smithi196.stdout:6/735: write dd/d1f/d38/f51 [1578287,27121] 0 2024-04-22T15:19:07.873 INFO:tasks.workunit.client.1.smithi196.stdout:9/693: chown db/dd/de/d15/d31/d64/dab 967363753 1 2024-04-22T15:19:07.873 INFO:tasks.workunit.client.1.smithi196.stdout:9/694: dread - db/d11/d29/d57/fbd zero size 2024-04-22T15:19:07.876 INFO:tasks.workunit.client.1.smithi196.stdout:0/730: symlink d3/de/d14/ded/l103 0 2024-04-22T15:19:07.877 INFO:tasks.workunit.client.1.smithi196.stdout:5/730: write d1/d17/d50/fcd [1635663,49757] 0 2024-04-22T15:19:07.877 INFO:tasks.workunit.client.1.smithi196.stdout:5/731: readlink d1/d4/l8e 0 2024-04-22T15:19:07.879 INFO:tasks.workunit.client.1.smithi196.stdout:1/709: symlink d1/db/d16/d1e/d49/le4 0 2024-04-22T15:19:07.880 INFO:tasks.workunit.client.1.smithi196.stdout:8/701: mknod d2/d6a/ccf 0 2024-04-22T15:19:07.883 INFO:tasks.workunit.client.1.smithi196.stdout:4/702: truncate d6/d14/d18/d26/d32/d53/fba 2404413 0 2024-04-22T15:19:07.884 INFO:tasks.workunit.client.1.smithi196.stdout:2/714: mknod d1/d23/d27/d73/c104 0 2024-04-22T15:19:07.887 INFO:tasks.workunit.client.0.smithi096.stdout:8/908: fdatasync d4/d10/d25/fe1 0 2024-04-22T15:19:07.893 INFO:tasks.workunit.client.0.smithi096.stdout:9/848: creat d1/f11f x:0 0 0 2024-04-22T15:19:07.894 INFO:tasks.workunit.client.1.smithi196.stdout:6/736: rmdir dd/d1f/d37/dcb 39 2024-04-22T15:19:07.895 INFO:tasks.workunit.client.1.smithi196.stdout:6/737: write dd/f86 [1052347,120499] 0 2024-04-22T15:19:07.895 INFO:tasks.workunit.client.0.smithi096.stdout:6/784: dwrite d3/d28/dd/f2b [0,4194304] 0 2024-04-22T15:19:07.898 INFO:tasks.workunit.client.1.smithi196.stdout:9/695: creat db/d11/fea x:0 0 0 2024-04-22T15:19:07.898 INFO:tasks.workunit.client.1.smithi196.stdout:3/722: write d0/d15/d31/dd0/d51/fb7 [247357,31305] 0 2024-04-22T15:19:07.902 INFO:tasks.workunit.client.0.smithi096.stdout:8/909: symlink d4/d10/d25/d4c/d76/l135 0 2024-04-22T15:19:07.905 INFO:tasks.workunit.client.1.smithi196.stdout:5/732: unlink d1/d14/l47 0 2024-04-22T15:19:07.907 INFO:tasks.workunit.client.0.smithi096.stdout:1/952: mkdir d6/dc/d8a/d3f/d62/d7d/ddb/d9b/d12a 0 2024-04-22T15:19:07.909 INFO:tasks.workunit.client.0.smithi096.stdout:9/849: dread d1/d25/f7d [0,4194304] 0 2024-04-22T15:19:07.909 INFO:tasks.workunit.client.1.smithi196.stdout:1/710: creat d1/db/dc/d26/fe5 x:0 0 0 2024-04-22T15:19:07.910 INFO:tasks.workunit.client.0.smithi096.stdout:9/850: dread - d1/f11f zero size 2024-04-22T15:19:07.912 INFO:tasks.workunit.client.1.smithi196.stdout:5/733: dwrite d1/d14/d3e/f4e [4194304,4194304] 0 2024-04-22T15:19:07.912 INFO:tasks.workunit.client.1.smithi196.stdout:8/702: rmdir d2/d5/d2e 39 2024-04-22T15:19:07.922 INFO:tasks.workunit.client.1.smithi196.stdout:4/703: rename d6/d14/c1e to d6/d14/d18/d26/d32/da3/db7/dc9/ce1 0 2024-04-22T15:19:07.924 INFO:tasks.workunit.client.0.smithi096.stdout:2/952: dread d3/db/d30/fd3 [0,4194304] 0 2024-04-22T15:19:07.929 INFO:tasks.workunit.client.0.smithi096.stdout:9/851: dread d1/df/d95/d9d/db5/d43/f9c [0,4194304] 0 2024-04-22T15:19:07.930 INFO:tasks.workunit.client.1.smithi196.stdout:2/715: dread d1/d13/f1e [0,4194304] 0 2024-04-22T15:19:07.931 INFO:tasks.workunit.client.0.smithi096.stdout:9/852: write d1/df/d95/d9d/db5/d43/d57/d5b/d8e/dae/dd9/fee [128461,124580] 0 2024-04-22T15:19:07.936 INFO:tasks.workunit.client.1.smithi196.stdout:3/723: mkdir d0/d15/d31/dd0/d51/ddd/df6 0 2024-04-22T15:19:07.938 INFO:tasks.workunit.client.0.smithi096.stdout:8/910: rename d4/d10/cd5 to d4/d2a/d7e/c136 0 2024-04-22T15:19:07.941 INFO:tasks.workunit.client.0.smithi096.stdout:2/953: creat d3/db/d10d/dfa/f135 x:0 0 0 2024-04-22T15:19:07.941 INFO:tasks.workunit.client.0.smithi096.stdout:1/953: write d6/dc/d8a/d3f/d54/d32/d3b/d3d/f8b [1089178,40739] 0 2024-04-22T15:19:07.942 INFO:tasks.workunit.client.1.smithi196.stdout:3/724: dwrite d0/d15/da7/d4a/d44/d8c/de9/fed [0,4194304] 0 2024-04-22T15:19:07.942 INFO:tasks.workunit.client.1.smithi196.stdout:1/711: mkdir d1/db/d16/d6d/de6 0 2024-04-22T15:19:07.952 INFO:tasks.workunit.client.0.smithi096.stdout:6/785: dwrite f2 [0,4194304] 0 2024-04-22T15:19:07.956 INFO:tasks.workunit.client.1.smithi196.stdout:7/694: rmdir d14/d26/d2a/d2b/dc7/de1 0 2024-04-22T15:19:07.957 INFO:tasks.workunit.client.0.smithi096.stdout:9/853: fdatasync d1/df/d95/d9d/db5/d43/d57/d5b/da1/f100 0 2024-04-22T15:19:07.960 INFO:tasks.workunit.client.0.smithi096.stdout:2/954: write d3/db/df/fde [371032,68085] 0 2024-04-22T15:19:07.960 INFO:tasks.workunit.client.1.smithi196.stdout:6/738: fdatasync dd/d2b/f77 0 2024-04-22T15:19:07.963 INFO:tasks.workunit.client.0.smithi096.stdout:1/954: write d6/dc/d8a/d3f/d54/d32/d3b/d56/ff1 [181247,72451] 0 2024-04-22T15:19:07.964 INFO:tasks.workunit.client.1.smithi196.stdout:9/696: symlink db/dd/d1b/de6/leb 0 2024-04-22T15:19:07.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.962+0000 7f82090d8700 1 -- 172.21.15.96:0/392393552 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82000056c0 con 0x7f82040ffb90 2024-04-22T15:19:07.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.963+0000 7f820c33e700 1 -- 172.21.15.96:0/392393552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 msgr2=0x7f82040fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:07.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.963+0000 7f820c33e700 1 --2- 172.21.15.96:0/392393552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 0x7f82040fffb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8200009910 tx=0x7f8200009c20 comp rx=0 tx=0).stop 2024-04-22T15:19:07.966 INFO:tasks.workunit.client.0.smithi096.stdout:6/786: truncate d3/d28/dd/d11/f1e 1429988 0 2024-04-22T15:19:07.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.963+0000 7f820c33e700 1 -- 172.21.15.96:0/392393552 shutdown_connections 2024-04-22T15:19:07.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.963+0000 7f820c33e700 1 --2- 172.21.15.96:0/392393552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8204100d90 0x7f82041011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.963+0000 7f820c33e700 1 --2- 172.21.15.96:0/392393552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 0x7f82040fffb0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.963+0000 7f820c33e700 1 -- 172.21.15.96:0/392393552 >> 172.21.15.96:0/392393552 conn(0x7f82040fb110 msgr2=0x7f82040fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:07.967 INFO:tasks.workunit.client.0.smithi096.stdout:2/955: dread d3/db/df/d98/f12b [0,4194304] 0 2024-04-22T15:19:07.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.964+0000 7f820c33e700 1 -- 172.21.15.96:0/392393552 shutdown_connections 2024-04-22T15:19:07.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.964+0000 7f820c33e700 1 -- 172.21.15.96:0/392393552 wait complete. 2024-04-22T15:19:07.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f820c33e700 1 Processor -- start 2024-04-22T15:19:07.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f820c33e700 1 -- start start 2024-04-22T15:19:07.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f820c33e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 0x7f82041944c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:07.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f820c33e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8204100d90 0x7f8204194a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:07.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f820c33e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8204195020 con 0x7f82040ffb90 2024-04-22T15:19:07.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f820c33e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8204195160 con 0x7f8204100d90 2024-04-22T15:19:07.969 INFO:tasks.workunit.client.1.smithi196.stdout:8/703: dwrite d2/d35/f92 [0,4194304] 0 2024-04-22T15:19:07.969 INFO:tasks.workunit.client.1.smithi196.stdout:5/734: dwrite d1/d17/d1a/d2f/d30/d6e/fa2 [4194304,4194304] 0 2024-04-22T15:19:07.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f820a0da700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 0x7f82041944c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:07.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f820a0da700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 0x7f82041944c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34882/0 (socket says 172.21.15.96:34882) 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f820a0da700 1 -- 172.21.15.96:0/3715490961 learned_addr learned my addr 172.21.15.96:0/3715490961 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.965+0000 7f82098d9700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8204100d90 0x7f8204194a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.966+0000 7f820a0da700 1 -- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8204100d90 msgr2=0x7f8204194a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.966+0000 7f820a0da700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8204100d90 0x7f8204194a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.966+0000 7f820a0da700 1 -- 172.21.15.96:0/3715490961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82000095f0 con 0x7f82040ffb90 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.966+0000 7f820a0da700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 0x7f82041944c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8200000c00 tx=0x7f8200005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.967+0000 7f81fb7fe700 1 -- 172.21.15.96:0/3715490961 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8200003b60 con 0x7f82040ffb90 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.967+0000 7f81fb7fe700 1 -- 172.21.15.96:0/3715490961 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8200003cc0 con 0x7f82040ffb90 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.967+0000 7f81fb7fe700 1 -- 172.21.15.96:0/3715490961 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f820001e7f0 con 0x7f82040ffb90 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.967+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8204199b60 con 0x7f82040ffb90 2024-04-22T15:19:07.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.967+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820419a050 con 0x7f82040ffb90 2024-04-22T15:19:07.972 INFO:tasks.workunit.client.1.smithi196.stdout:0/731: creat d3/d9/d36/f104 x:0 0 0 2024-04-22T15:19:07.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.969+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820418e620 con 0x7f82040ffb90 2024-04-22T15:19:07.973 INFO:tasks.workunit.client.1.smithi196.stdout:5/735: truncate d1/d17/f79 4279752 0 2024-04-22T15:19:07.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.969+0000 7f81fb7fe700 1 -- 172.21.15.96:0/3715490961 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f8200015c00 con 0x7f82040ffb90 2024-04-22T15:19:07.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.970+0000 7f81fb7fe700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f81f006dc50 0x7f81f0070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:07.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.970+0000 7f81fb7fe700 1 -- 172.21.15.96:0/3715490961 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f8200090ce0 con 0x7f82040ffb90 2024-04-22T15:19:07.974 INFO:tasks.workunit.client.0.smithi096.stdout:9/854: rename d1/df/d95/d9d/db5/d43/d57/d5b/d8e to d1/df/d95/d9d/db5/dd/dba/dc2/d120 0 2024-04-22T15:19:07.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.971+0000 7f82098d9700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f81f006dc50 0x7f81f0070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:07.976 INFO:tasks.workunit.client.1.smithi196.stdout:3/725: creat d0/d15/da7/d85/ff7 x:0 0 0 2024-04-22T15:19:07.976 INFO:tasks.workunit.client.1.smithi196.stdout:9/697: dwrite db/d11/fb2 [0,4194304] 0 2024-04-22T15:19:07.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.973+0000 7f82098d9700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f81f006dc50 0x7f81f0070110 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f81f4004630 tx=0x7f81f400b3c0 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:07.979 INFO:tasks.workunit.client.0.smithi096.stdout:1/955: creat d6/dc/d8a/d3f/d54/d32/d3b/d56/f12b x:0 0 0 2024-04-22T15:19:07.984 INFO:tasks.workunit.client.1.smithi196.stdout:7/695: symlink d14/d26/d5e/le4 0 2024-04-22T15:19:07.985 INFO:tasks.workunit.client.1.smithi196.stdout:7/696: dread - d14/d26/d5a/d5f/fcd zero size 2024-04-22T15:19:07.985 INFO:tasks.workunit.client.0.smithi096.stdout:2/956: chown d3/db/df/d10/f37 1226 1 2024-04-22T15:19:07.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:07.983+0000 7f81fb7fe700 1 -- 172.21.15.96:0/3715490961 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f820005d9d0 con 0x7f82040ffb90 2024-04-22T15:19:07.987 INFO:tasks.workunit.client.0.smithi096.stdout:9/855: truncate d1/f20 290526 0 2024-04-22T15:19:07.988 INFO:tasks.workunit.client.0.smithi096.stdout:8/911: write d4/d71/f77 [668929,116187] 0 2024-04-22T15:19:07.989 INFO:tasks.workunit.client.0.smithi096.stdout:8/912: fdatasync d4/d2a/f36 0 2024-04-22T15:19:07.991 INFO:tasks.workunit.client.1.smithi196.stdout:2/716: mknod d1/d13/c105 0 2024-04-22T15:19:07.992 INFO:tasks.workunit.client.0.smithi096.stdout:1/956: dwrite d6/dc/d8a/d3f/d54/d32/d3b/d56/ff1 [0,4194304] 0 2024-04-22T15:19:07.993 INFO:tasks.workunit.client.1.smithi196.stdout:7/697: read d14/d26/d28/d29/fb3 [7639913,15840] 0 2024-04-22T15:19:07.995 INFO:tasks.workunit.client.0.smithi096.stdout:6/787: mknod d3/d28/dd/c107 0 2024-04-22T15:19:07.996 INFO:tasks.workunit.client.1.smithi196.stdout:7/698: truncate d14/d26/d5a/d5f/faa 1195969 0 2024-04-22T15:19:07.997 INFO:tasks.workunit.client.1.smithi196.stdout:8/704: dread d2/d8/f73 [0,4194304] 0 2024-04-22T15:19:07.997 INFO:tasks.workunit.client.0.smithi096.stdout:2/957: unlink d3/db/df/d10/f3b 0 2024-04-22T15:19:07.998 INFO:tasks.workunit.client.1.smithi196.stdout:8/705: fdatasync d2/d8/d49/f59 0 2024-04-22T15:19:08.000 INFO:tasks.workunit.client.1.smithi196.stdout:0/732: fsync d3/fd 0 2024-04-22T15:19:08.000 INFO:tasks.workunit.client.0.smithi096.stdout:2/958: fdatasync d3/db/df/dce/d46/d69/dd7/f134 0 2024-04-22T15:19:08.001 INFO:tasks.workunit.client.1.smithi196.stdout:0/733: fsync d3/f47 0 2024-04-22T15:19:08.004 INFO:tasks.workunit.client.0.smithi096.stdout:6/788: dwrite d3/d3a/d75/f105 [0,4194304] 0 2024-04-22T15:19:08.006 INFO:tasks.workunit.client.1.smithi196.stdout:0/734: dwrite d3/f47 [0,4194304] 0 2024-04-22T15:19:08.006 INFO:tasks.workunit.client.1.smithi196.stdout:9/698: rmdir db/dd 39 2024-04-22T15:19:08.013 INFO:tasks.workunit.client.1.smithi196.stdout:1/712: write d1/db/d16/d44/faa [155617,81072] 0 2024-04-22T15:19:08.016 INFO:tasks.workunit.client.1.smithi196.stdout:4/704: link d6/d13/d15/l38 d6/d10/le2 0 2024-04-22T15:19:08.016 INFO:tasks.workunit.client.0.smithi096.stdout:3/956: sync 2024-04-22T15:19:08.016 INFO:tasks.workunit.client.0.smithi096.stdout:4/924: sync 2024-04-22T15:19:08.017 INFO:tasks.workunit.client.1.smithi196.stdout:3/726: dread d0/d15/da7/fa8 [0,4194304] 0 2024-04-22T15:19:08.028 INFO:tasks.workunit.client.1.smithi196.stdout:6/739: write dd/d1f/d37/d7b/db6/fd0 [841853,67582] 0 2024-04-22T15:19:08.028 INFO:tasks.workunit.client.1.smithi196.stdout:2/717: rmdir d1/d13/d25/d2a/ddb/dff 39 2024-04-22T15:19:08.029 INFO:tasks.workunit.client.1.smithi196.stdout:2/718: fsync d1/d23/fd1 0 2024-04-22T15:19:08.029 INFO:tasks.workunit.client.1.smithi196.stdout:6/740: truncate dd/d20/d5c/fb7 708984 0 2024-04-22T15:19:08.030 INFO:tasks.workunit.client.1.smithi196.stdout:2/719: chown d1/d23/d27/d2e/d55/fce 4109387 1 2024-04-22T15:19:08.031 INFO:tasks.workunit.client.1.smithi196.stdout:2/720: truncate d1/ffc 18421 0 2024-04-22T15:19:08.031 INFO:tasks.workunit.client.0.smithi096.stdout:3/957: mknod da/db/d9a/def/c13b 0 2024-04-22T15:19:08.032 INFO:tasks.workunit.client.0.smithi096.stdout:3/958: chown da/db/dc/d28/d69 7 1 2024-04-22T15:19:08.033 INFO:tasks.workunit.client.1.smithi196.stdout:7/699: creat d14/d25/d80/fe5 x:0 0 0 2024-04-22T15:19:08.034 INFO:tasks.workunit.client.0.smithi096.stdout:4/925: unlink d5/d7/d1a/d8b/f102 0 2024-04-22T15:19:08.035 INFO:tasks.workunit.client.1.smithi196.stdout:7/700: truncate d14/d26/d5a/d5f/dc2/fda 200828 0 2024-04-22T15:19:08.036 INFO:tasks.workunit.client.0.smithi096.stdout:8/913: write d4/f12 [620494,14935] 0 2024-04-22T15:19:08.037 INFO:tasks.workunit.client.1.smithi196.stdout:5/736: getdents d1/d17/d1a/d2f/d30/d5e/dcb 0 2024-04-22T15:19:08.037 INFO:tasks.workunit.client.0.smithi096.stdout:8/914: dread - d4/d10/d25/d4c/d76/f10f zero size 2024-04-22T15:19:08.040 INFO:tasks.workunit.client.0.smithi096.stdout:9/856: dread d1/df/d95/d9d/db5/d43/d57/f63 [0,4194304] 0 2024-04-22T15:19:08.041 INFO:tasks.workunit.client.0.smithi096.stdout:2/959: write d3/db/df/fcd [1074979,15740] 0 2024-04-22T15:19:08.041 INFO:tasks.workunit.client.1.smithi196.stdout:9/699: chown db/dd/d1b/d2c/cdf 2072 1 2024-04-22T15:19:08.041 INFO:tasks.workunit.client.1.smithi196.stdout:7/701: dwrite d14/d4d/da3/fbc [0,4194304] 0 2024-04-22T15:19:08.042 INFO:tasks.workunit.client.0.smithi096.stdout:6/789: dread d3/f26 [0,4194304] 0 2024-04-22T15:19:08.042 INFO:tasks.workunit.client.1.smithi196.stdout:7/702: chown d14/d26/d5a/d5f/dc2 6911235 1 2024-04-22T15:19:08.043 INFO:tasks.workunit.client.0.smithi096.stdout:2/960: readlink d3/db/df/d24/d58/d5f/d6e/l8c 0 2024-04-22T15:19:08.044 INFO:tasks.workunit.client.0.smithi096.stdout:3/959: rmdir da/db/de/ddc/d68 39 2024-04-22T15:19:08.045 INFO:tasks.workunit.client.1.smithi196.stdout:0/735: creat d3/d6/dc/d8c/d8d/da8/f105 x:0 0 0 2024-04-22T15:19:08.046 INFO:tasks.workunit.client.0.smithi096.stdout:4/926: truncate d5/d7/d2c/d95/dee/f59 4680173 0 2024-04-22T15:19:08.047 INFO:tasks.workunit.client.1.smithi196.stdout:3/727: symlink d0/d15/d4d/d87/lf8 0 2024-04-22T15:19:08.050 INFO:tasks.workunit.client.1.smithi196.stdout:4/705: rename d6/d13/d15/d2c/d62/dd6/fde to d6/d13/d15/d2c/d62/d78/d8b/fe3 0 2024-04-22T15:19:08.052 INFO:tasks.workunit.client.0.smithi096.stdout:1/957: dwrite d6/fea [0,4194304] 0 2024-04-22T15:19:08.052 INFO:tasks.workunit.client.0.smithi096.stdout:6/790: dwrite d3/d28/f38 [0,4194304] 0 2024-04-22T15:19:08.055 INFO:tasks.workunit.client.0.smithi096.stdout:9/857: mknod d1/df/d95/d9d/db5/dd/dba/dc2/d120/dbd/c121 0 2024-04-22T15:19:08.058 INFO:tasks.workunit.client.0.smithi096.stdout:6/791: read d3/d28/dd/f19 [3681364,17804] 0 2024-04-22T15:19:08.058 INFO:tasks.workunit.client.0.smithi096.stdout:9/858: stat d1/d25/d9f 0 2024-04-22T15:19:08.059 INFO:tasks.workunit.client.1.smithi196.stdout:2/721: truncate d1/d13/d1f/f20 4646522 0 2024-04-22T15:19:08.059 INFO:tasks.workunit.client.1.smithi196.stdout:2/722: readlink d1/d23/d27/d73/l86 0 2024-04-22T15:19:08.066 INFO:tasks.workunit.client.0.smithi096.stdout:6/792: dwrite d3/d28/dd/f2b [0,4194304] 0 2024-04-22T15:19:08.069 INFO:tasks.workunit.client.0.smithi096.stdout:1/958: readlink d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/l103 0 2024-04-22T15:19:08.104 INFO:tasks.workunit.client.1.smithi196.stdout:8/706: sync 2024-04-22T15:19:08.114 INFO:tasks.workunit.client.1.smithi196.stdout:3/728: unlink d0/d15/d31/d55/dcf/d8/d70/f97 0 2024-04-22T15:19:08.122 INFO:tasks.workunit.client.1.smithi196.stdout:6/741: mknod dd/d1f/d4e/d53/d7f/da5/cd7 0 2024-04-22T15:19:08.123 INFO:tasks.workunit.client.1.smithi196.stdout:2/723: fdatasync d1/d23/d27/d2e/d55/d5b/f78 0 2024-04-22T15:19:08.125 INFO:tasks.workunit.client.0.smithi096.stdout:6/793: chown d3/d28/dd/d11/d2c/lcb 2 1 2024-04-22T15:19:08.129 INFO:tasks.workunit.client.0.smithi096.stdout:8/915: getdents d4/d10/d25/d4c/db4 0 2024-04-22T15:19:08.130 INFO:tasks.workunit.client.0.smithi096.stdout:2/961: write d3/db/df/d24/d58/d5e/f8b [873407,15007] 0 2024-04-22T15:19:08.137 INFO:tasks.workunit.client.1.smithi196.stdout:8/707: creat d2/d8/fd0 x:0 0 0 2024-04-22T15:19:08.138 INFO:tasks.workunit.client.1.smithi196.stdout:6/742: dread dd/d1f/d37/f31 [0,4194304] 0 2024-04-22T15:19:08.138 INFO:tasks.workunit.client.0.smithi096.stdout:4/927: dwrite d5/d7/d1a/f40 [0,4194304] 0 2024-04-22T15:19:08.138 INFO:tasks.workunit.client.0.smithi096.stdout:3/960: write da/db/de/ddc/ff3 [2516206,110265] 0 2024-04-22T15:19:08.139 INFO:tasks.workunit.client.0.smithi096.stdout:2/962: unlink d3/db/df/d10/la0 0 2024-04-22T15:19:08.145 INFO:tasks.workunit.client.0.smithi096.stdout:6/794: unlink d3/d28/d4d/d82/lf5 0 2024-04-22T15:19:08.146 INFO:tasks.workunit.client.1.smithi196.stdout:4/706: dwrite d6/d14/f87 [0,4194304] 0 2024-04-22T15:19:08.148 INFO:tasks.workunit.client.1.smithi196.stdout:5/737: rename d1/d14/d6a/da7 to d1/d17/d7f/deb 0 2024-04-22T15:19:08.151 INFO:tasks.workunit.client.1.smithi196.stdout:3/729: write d0/d15/d31/d55/dcf/d8/f77 [1189400,107053] 0 2024-04-22T15:19:08.152 INFO:tasks.workunit.client.1.smithi196.stdout:3/730: dread - d0/d15/d31/dd0/d51/fc3 zero size 2024-04-22T15:19:08.155 INFO:tasks.workunit.client.0.smithi096.stdout:6/795: creat d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/f108 x:0 0 0 2024-04-22T15:19:08.156 INFO:tasks.workunit.client.1.smithi196.stdout:2/724: creat d1/d23/d27/d2e/d55/d5b/d93/d97/f106 x:0 0 0 2024-04-22T15:19:08.157 INFO:tasks.workunit.client.0.smithi096.stdout:4/928: truncate d5/d7/fb8 993009 0 2024-04-22T15:19:08.157 INFO:tasks.workunit.client.1.smithi196.stdout:3/731: dwrite d0/d15/da7/d4a/d44/f16 [0,4194304] 0 2024-04-22T15:19:08.160 INFO:tasks.workunit.client.1.smithi196.stdout:3/732: truncate d0/d15/d31/dd0/d51/ddd/fe4 618087 0 2024-04-22T15:19:08.164 INFO:tasks.workunit.client.0.smithi096.stdout:4/929: creat d5/d7/d2c/d95/dee/f140 x:0 0 0 2024-04-22T15:19:08.164 INFO:tasks.workunit.client.1.smithi196.stdout:0/736: rmdir d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8/df7 0 2024-04-22T15:19:08.165 INFO:tasks.workunit.client.1.smithi196.stdout:8/708: chown d2/d3f/l71 724527920 1 2024-04-22T15:19:08.169 INFO:tasks.workunit.client.1.smithi196.stdout:1/713: sync 2024-04-22T15:19:08.172 INFO:tasks.workunit.client.0.smithi096.stdout:4/930: mkdir d5/d7/d2c/d43/d141 0 2024-04-22T15:19:08.174 INFO:tasks.workunit.client.1.smithi196.stdout:4/707: dread d6/d90/f60 [4194304,4194304] 0 2024-04-22T15:19:08.176 INFO:tasks.workunit.client.1.smithi196.stdout:4/708: fsync d6/d14/d18/d26/d32/d67/d9a/f9e 0 2024-04-22T15:19:08.176 INFO:tasks.workunit.client.0.smithi096.stdout:6/796: dread d3/d28/f4f [0,4194304] 0 2024-04-22T15:19:08.177 INFO:tasks.workunit.client.1.smithi196.stdout:1/714: dwrite d1/db/dc/d26/fe5 [0,4194304] 0 2024-04-22T15:19:08.177 INFO:tasks.workunit.client.0.smithi096.stdout:6/797: chown d3/d28/dd/d11/d2c/fb1 54500 1 2024-04-22T15:19:08.178 INFO:tasks.workunit.client.1.smithi196.stdout:4/709: write d6/d13/d15/d2c/d35/fe0 [178269,52418] 0 2024-04-22T15:19:08.183 INFO:tasks.workunit.client.0.smithi096.stdout:4/931: rename d5/d7/fed to d5/d5b/f142 0 2024-04-22T15:19:08.185 INFO:tasks.workunit.client.0.smithi096.stdout:2/963: write d3/db/df/d24/fe0 [331816,76083] 0 2024-04-22T15:19:08.187 INFO:tasks.workunit.client.0.smithi096.stdout:2/964: fsync d3/db/df/f10e 0 2024-04-22T15:19:08.192 INFO:tasks.workunit.client.1.smithi196.stdout:6/743: dread dd/d1f/d4e/f54 [0,4194304] 0 2024-04-22T15:19:08.197 INFO:tasks.workunit.client.1.smithi196.stdout:2/725: rmdir d1/d23/d27 39 2024-04-22T15:19:08.201 INFO:tasks.workunit.client.0.smithi096.stdout:6/798: mknod d3/d28/d4d/d82/d9e/db0/c109 0 2024-04-22T15:19:08.202 INFO:tasks.workunit.client.1.smithi196.stdout:9/700: getdents db/dd/d1b/d2c 0 2024-04-22T15:19:08.203 INFO:tasks.workunit.client.1.smithi196.stdout:9/701: read - db/dd/de/d15/d31/fcf zero size 2024-04-22T15:19:08.204 INFO:tasks.workunit.client.1.smithi196.stdout:9/702: chown db/d11/d29/d7a/d7b/d81/c8a 9067928 1 2024-04-22T15:19:08.204 INFO:tasks.workunit.client.1.smithi196.stdout:7/703: getdents d14/d26/d5a/d5f/dc2 0 2024-04-22T15:19:08.207 INFO:tasks.workunit.client.1.smithi196.stdout:3/733: creat d0/d15/d31/d55/ff9 x:0 0 0 2024-04-22T15:19:08.207 INFO:tasks.workunit.client.1.smithi196.stdout:3/734: readlink d0/d15/d4d/l3f 0 2024-04-22T15:19:08.212 INFO:tasks.workunit.client.0.smithi096.stdout:5/960: sync 2024-04-22T15:19:08.220 INFO:tasks.workunit.client.1.smithi196.stdout:3/735: dread d0/d15/da7/d85/fc7 [0,4194304] 0 2024-04-22T15:19:08.220 INFO:tasks.workunit.client.0.smithi096.stdout:2/965: link d3/f8e d3/db/df/dce/d46/d69/d7f/f136 0 2024-04-22T15:19:08.221 INFO:tasks.workunit.client.0.smithi096.stdout:2/966: stat d3/db/f83 0 2024-04-22T15:19:08.222 INFO:tasks.workunit.client.0.smithi096.stdout:2/967: chown d3/db/df/dce/d46/d59/le8 248467344 1 2024-04-22T15:19:08.225 INFO:tasks.workunit.client.0.smithi096.stdout:6/799: getdents d3/d28/dd/d11/d2c/d6f/dd4/dd6 0 2024-04-22T15:19:08.226 INFO:tasks.workunit.client.0.smithi096.stdout:6/800: chown d3/d28/daf 0 1 2024-04-22T15:19:08.226 INFO:tasks.workunit.client.0.smithi096.stdout:4/932: write f1 [4473021,23063] 0 2024-04-22T15:19:08.231 INFO:tasks.workunit.client.0.smithi096.stdout:4/933: unlink d5/d7/d2c/d3d/d13d/l8a 0 2024-04-22T15:19:08.232 INFO:tasks.workunit.client.0.smithi096.stdout:6/801: mkdir d3/d28/dd/d11/d10a 0 2024-04-22T15:19:08.239 INFO:tasks.workunit.client.0.smithi096.stdout:2/968: getdents d3/db/df/dce/d46/d7e/d84 0 2024-04-22T15:19:08.239 INFO:tasks.workunit.client.0.smithi096.stdout:6/802: symlink d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/l10b 0 2024-04-22T15:19:08.250 INFO:tasks.workunit.client.0.smithi096.stdout:4/934: dread d5/d7/d1f/d84/d6d/d98/f110 [0,4194304] 0 2024-04-22T15:19:08.251 INFO:tasks.workunit.client.0.smithi096.stdout:4/935: fdatasync d5/d7/d2c/d95/df9/f12b 0 2024-04-22T15:19:08.252 INFO:tasks.workunit.client.0.smithi096.stdout:4/936: dread - d5/d7/d2c/d95/dee/f140 zero size 2024-04-22T15:19:08.257 INFO:tasks.workunit.client.1.smithi196.stdout:1/715: write d1/db/dc/de/d5a/d5b/d85/fc8 [75281,71640] 0 2024-04-22T15:19:08.261 INFO:tasks.workunit.client.1.smithi196.stdout:5/738: mkdir d1/d14/dec 0 2024-04-22T15:19:08.266 INFO:tasks.workunit.client.1.smithi196.stdout:9/703: unlink db/dd/d1b/c3b 0 2024-04-22T15:19:08.268 INFO:tasks.workunit.client.1.smithi196.stdout:8/709: dread d2/f3e [0,4194304] 0 2024-04-22T15:19:08.268 INFO:tasks.workunit.client.1.smithi196.stdout:7/704: creat d14/d26/d28/fe6 x:0 0 0 2024-04-22T15:19:08.273 INFO:tasks.workunit.client.1.smithi196.stdout:0/737: getdents d3/de/d1c/dd4 0 2024-04-22T15:19:08.275 INFO:tasks.workunit.client.1.smithi196.stdout:1/716: truncate d1/db/d16/d1e/d49/d7f/fb7 797055 0 2024-04-22T15:19:08.276 INFO:tasks.workunit.client.1.smithi196.stdout:4/710: symlink d6/d14/d18/le4 0 2024-04-22T15:19:08.280 INFO:tasks.workunit.client.1.smithi196.stdout:6/744: rename dd/d1f/d37/dcb/fd4 to dd/d1f/d4e/fd8 0 2024-04-22T15:19:08.282 INFO:tasks.workunit.client.1.smithi196.stdout:2/726: write d1/d23/d27/d73/fda [881965,76111] 0 2024-04-22T15:19:08.284 INFO:tasks.workunit.client.1.smithi196.stdout:9/704: mkdir db/d11/d29/d7a/d7b/d81/dec 0 2024-04-22T15:19:08.288 INFO:tasks.workunit.client.1.smithi196.stdout:7/705: fsync d14/d25/f42 0 2024-04-22T15:19:08.289 INFO:tasks.workunit.client.1.smithi196.stdout:7/706: read - d14/d26/d2a/d2b/da7/dc6/fd4 zero size 2024-04-22T15:19:08.297 INFO:tasks.workunit.client.1.smithi196.stdout:2/727: dread d1/d13/d1f/ddf/f81 [0,4194304] 0 2024-04-22T15:19:08.300 INFO:tasks.workunit.client.1.smithi196.stdout:4/711: creat d6/d14/d18/d40/d85/fe5 x:0 0 0 2024-04-22T15:19:08.303 INFO:tasks.workunit.client.1.smithi196.stdout:5/739: creat d1/d78/dcf/fed x:0 0 0 2024-04-22T15:19:08.305 INFO:tasks.workunit.client.0.smithi096.stdout:2/969: creat d3/db/df/d24/dc8/d103/f137 x:0 0 0 2024-04-22T15:19:08.309 INFO:tasks.workunit.client.1.smithi196.stdout:9/705: truncate fa 1195388 0 2024-04-22T15:19:08.310 INFO:tasks.workunit.client.0.smithi096.stdout:2/970: creat d3/db/df/dce/d46/d7e/d84/f138 x:0 0 0 2024-04-22T15:19:08.315 INFO:tasks.workunit.client.0.smithi096.stdout:2/971: mknod d3/db/df/c139 0 2024-04-22T15:19:08.318 INFO:tasks.workunit.client.1.smithi196.stdout:1/717: mknod d1/db/ce7 0 2024-04-22T15:19:08.319 INFO:tasks.workunit.client.1.smithi196.stdout:1/718: dread - d1/db/dc/dad/d36/fc9 zero size 2024-04-22T15:19:08.320 INFO:tasks.workunit.client.1.smithi196.stdout:1/719: chown d1/db/d16/d44 168411408 1 2024-04-22T15:19:08.321 INFO:tasks.workunit.client.1.smithi196.stdout:4/712: mknod d6/d90/ce6 0 2024-04-22T15:19:08.323 INFO:tasks.workunit.client.0.smithi096.stdout:2/972: link d3/db/df/d10/f11b d3/db/d10d/dfa/f13a 0 2024-04-22T15:19:08.324 INFO:tasks.workunit.client.1.smithi196.stdout:5/740: symlink d1/d17/d1a/d2f/d30/d6e/d7a/lee 0 2024-04-22T15:19:08.324 INFO:tasks.workunit.client.1.smithi196.stdout:5/741: fdatasync d1/d9/f20 0 2024-04-22T15:19:08.326 INFO:tasks.workunit.client.0.smithi096.stdout:2/973: creat d3/db/df/dce/d46/d69/d7f/f13b x:0 0 0 2024-04-22T15:19:08.327 INFO:tasks.workunit.client.1.smithi196.stdout:3/736: symlink d0/d15/d31/d55/d8b/df2/lfa 0 2024-04-22T15:19:08.328 INFO:tasks.workunit.client.0.smithi096.stdout:2/974: readlink d3/db/df/d24/d58/d5e/l81 0 2024-04-22T15:19:08.330 INFO:tasks.workunit.client.1.smithi196.stdout:4/713: truncate d6/d13/d15/d2c/d62/f97 55168 0 2024-04-22T15:19:08.333 INFO:tasks.workunit.client.1.smithi196.stdout:6/745: rename dd/d1f/d37/d7b/db6 to dd/d1f/d38/dd9 0 2024-04-22T15:19:08.334 INFO:tasks.workunit.client.1.smithi196.stdout:9/706: symlink db/dd/de/d98/da9/led 0 2024-04-22T15:19:08.336 INFO:tasks.workunit.client.1.smithi196.stdout:8/710: dwrite d2/d35/fb4 [0,4194304] 0 2024-04-22T15:19:08.337 INFO:tasks.workunit.client.1.smithi196.stdout:1/720: mkdir d1/db/d16/d6d/de6/de8 0 2024-04-22T15:19:08.339 INFO:tasks.workunit.client.0.smithi096.stdout:2/975: getdents d3/db/d30 0 2024-04-22T15:19:08.340 INFO:tasks.workunit.client.1.smithi196.stdout:1/721: chown d1/db/dc/de/ccd 1646027142 1 2024-04-22T15:19:08.340 INFO:tasks.workunit.client.1.smithi196.stdout:7/707: dwrite d14/d25/f42 [0,4194304] 0 2024-04-22T15:19:08.345 INFO:tasks.workunit.client.0.smithi096.stdout:4/937: write d5/d7/d2c/d3d/d13d/fab [1694271,86798] 0 2024-04-22T15:19:08.346 INFO:tasks.workunit.client.0.smithi096.stdout:6/803: dwrite d3/d28/dd/f25 [0,4194304] 0 2024-04-22T15:19:08.349 INFO:tasks.workunit.client.0.smithi096.stdout:6/804: chown d3/d28/daf/c103 204582426 1 2024-04-22T15:19:08.352 INFO:tasks.workunit.client.1.smithi196.stdout:5/742: mknod d1/d78/cef 0 2024-04-22T15:19:08.352 INFO:tasks.workunit.client.1.smithi196.stdout:8/711: mknod d2/d8/d49/d8b/cd1 0 2024-04-22T15:19:08.357 INFO:tasks.workunit.client.0.smithi096.stdout:4/938: mkdir d5/d7/d2c/d3d/d4f/d63/d143 0 2024-04-22T15:19:08.358 INFO:tasks.workunit.client.0.smithi096.stdout:4/939: chown d5/d7/d1f/le0 59750186 1 2024-04-22T15:19:08.359 INFO:tasks.workunit.client.0.smithi096.stdout:6/805: dwrite d3/d28/dd/d11/d2c/fb1 [0,4194304] 0 2024-04-22T15:19:08.361 INFO:tasks.workunit.client.1.smithi196.stdout:3/737: creat d0/d15/d31/dd0/d51/ddd/df6/ffb x:0 0 0 2024-04-22T15:19:08.361 INFO:tasks.workunit.client.1.smithi196.stdout:0/738: truncate d3/de/d1c/f31 3148167 0 2024-04-22T15:19:08.362 INFO:tasks.workunit.client.1.smithi196.stdout:4/714: fdatasync d6/d14/d18/d26/d32/d53/fba 0 2024-04-22T15:19:08.364 INFO:tasks.workunit.client.1.smithi196.stdout:4/715: readlink d6/d14/d18/d26/d32/d53/lb9 0 2024-04-22T15:19:08.367 INFO:tasks.workunit.client.1.smithi196.stdout:2/728: dwrite d1/ff [0,4194304] 0 2024-04-22T15:19:08.369 INFO:tasks.workunit.client.0.smithi096.stdout:6/806: truncate d3/d28/d4d/d82/ffb 700687 0 2024-04-22T15:19:08.373 INFO:tasks.workunit.client.1.smithi196.stdout:8/712: mknod d2/d8/d47/d68/dbc/cd2 0 2024-04-22T15:19:08.374 INFO:tasks.workunit.client.1.smithi196.stdout:9/707: dread f5 [0,4194304] 0 2024-04-22T15:19:08.375 INFO:tasks.workunit.client.1.smithi196.stdout:8/713: chown d2/d8/d47/d68/d81/f88 3 1 2024-04-22T15:19:08.376 INFO:tasks.workunit.client.1.smithi196.stdout:8/714: dread - d2/dc/d45/d98/fba zero size 2024-04-22T15:19:08.377 INFO:tasks.workunit.client.0.smithi096.stdout:6/807: creat d3/d28/dd/d11/d2c/d106/f10c x:0 0 0 2024-04-22T15:19:08.378 INFO:tasks.workunit.client.1.smithi196.stdout:0/739: mknod d3/de/d1c/c106 0 2024-04-22T15:19:08.380 INFO:tasks.workunit.client.1.smithi196.stdout:0/740: write d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8/ffb [375625,81613] 0 2024-04-22T15:19:08.380 INFO:tasks.workunit.client.1.smithi196.stdout:3/738: mkdir d0/d15/d31/d55/dcf/d8/dfc 0 2024-04-22T15:19:08.381 INFO:tasks.workunit.client.1.smithi196.stdout:0/741: write d3/de/d14/f81 [2547498,98179] 0 2024-04-22T15:19:08.381 INFO:tasks.workunit.client.1.smithi196.stdout:3/739: stat d0/d15/d31/d55/dcf/d6f/f81 0 2024-04-22T15:19:08.381 INFO:tasks.workunit.client.1.smithi196.stdout:0/742: chown d3/de/d14 7125 1 2024-04-22T15:19:08.382 INFO:tasks.workunit.client.1.smithi196.stdout:3/740: dread - d0/d15/da7/d4a/f45 zero size 2024-04-22T15:19:08.384 INFO:tasks.workunit.client.1.smithi196.stdout:6/746: creat dd/fda x:0 0 0 2024-04-22T15:19:08.385 INFO:tasks.workunit.client.1.smithi196.stdout:4/716: mkdir d6/d14/d18/d26/d32/d67/de7 0 2024-04-22T15:19:08.389 INFO:tasks.workunit.client.0.smithi096.stdout:2/976: dwrite d3/db/df/d24/d58/d5f/d6e/fac [0,4194304] 0 2024-04-22T15:19:08.392 INFO:tasks.workunit.client.1.smithi196.stdout:9/708: fsync db/dd/de/d15/f1e 0 2024-04-22T15:19:08.395 INFO:tasks.workunit.client.1.smithi196.stdout:7/708: write d14/d25/d31/f63 [695990,6481] 0 2024-04-22T15:19:08.397 INFO:tasks.workunit.client.1.smithi196.stdout:0/743: mkdir d3/d9/d36/d9f/dd1/d65/d107 0 2024-04-22T15:19:08.397 INFO:tasks.workunit.client.0.smithi096.stdout:4/940: dwrite d5/d7/d2c/d3d/d13d/fe6 [0,4194304] 0 2024-04-22T15:19:08.400 INFO:tasks.workunit.client.1.smithi196.stdout:3/741: mkdir d0/d15/da7/d4a/dc5/dfd 0 2024-04-22T15:19:08.400 INFO:tasks.workunit.client.0.smithi096.stdout:4/941: write d5/d7/d1a/f40 [2322682,9845] 0 2024-04-22T15:19:08.406 INFO:tasks.workunit.client.0.smithi096.stdout:2/977: link d3/db/df/d24/d58/lcc d3/d13/db4/dbf/l13c 0 2024-04-22T15:19:08.407 INFO:tasks.workunit.client.0.smithi096.stdout:2/978: dread - d3/db/d30/fe2 zero size 2024-04-22T15:19:08.408 INFO:tasks.workunit.client.0.smithi096.stdout:6/808: dwrite d3/d28/dd/f70 [0,4194304] 0 2024-04-22T15:19:08.409 INFO:tasks.workunit.client.1.smithi196.stdout:1/722: rename d1/db/d16/d1e/d49/cbc to d1/db/ce9 0 2024-04-22T15:19:08.410 INFO:tasks.workunit.client.1.smithi196.stdout:1/723: chown d1/db/dc/de/d5a/l5d 121167 1 2024-04-22T15:19:08.412 INFO:tasks.workunit.client.1.smithi196.stdout:5/743: getdents d1/d4/d92 0 2024-04-22T15:19:08.415 INFO:tasks.workunit.client.0.smithi096.stdout:4/942: mknod d5/d7/c144 0 2024-04-22T15:19:08.416 INFO:tasks.workunit.client.0.smithi096.stdout:6/809: dwrite d3/d28/dd/f25 [0,4194304] 0 2024-04-22T15:19:08.417 INFO:tasks.workunit.client.1.smithi196.stdout:7/709: creat d14/d15/d91/fe7 x:0 0 0 2024-04-22T15:19:08.418 INFO:tasks.workunit.client.1.smithi196.stdout:8/715: dwrite d2/d35/f5f [0,4194304] 0 2024-04-22T15:19:08.420 INFO:tasks.workunit.client.1.smithi196.stdout:3/742: mkdir d0/d15/d31/d55/d6e/dfe 0 2024-04-22T15:19:08.425 INFO:tasks.workunit.client.0.smithi096.stdout:6/810: rmdir d3/d28/d4d 39 2024-04-22T15:19:08.428 INFO:tasks.workunit.client.1.smithi196.stdout:4/717: mkdir d6/de8 0 2024-04-22T15:19:08.430 INFO:tasks.workunit.client.1.smithi196.stdout:3/743: dwrite d0/d15/d31/d55/fb6 [0,4194304] 0 2024-04-22T15:19:08.431 INFO:tasks.workunit.client.1.smithi196.stdout:3/744: readlink d0/d15/leb 0 2024-04-22T15:19:08.432 INFO:tasks.workunit.client.0.smithi096.stdout:6/811: mkdir d3/d28/d54/d68/d6e/d10d 0 2024-04-22T15:19:08.433 INFO:tasks.workunit.client.0.smithi096.stdout:6/812: chown d3/d3a/d75/d83/df7 1309365 1 2024-04-22T15:19:08.437 INFO:tasks.workunit.client.1.smithi196.stdout:5/744: symlink d1/d2/d98/dac/lf0 0 2024-04-22T15:19:08.440 INFO:tasks.workunit.client.0.smithi096.stdout:6/813: rename f1 to d3/d28/daf/d94/f10e 0 2024-04-22T15:19:08.441 INFO:tasks.workunit.client.1.smithi196.stdout:6/747: dwrite dd/d1f/d4e/f73 [0,4194304] 0 2024-04-22T15:19:08.442 INFO:tasks.workunit.client.1.smithi196.stdout:9/709: truncate db/dd/de/f4e 1446174 0 2024-04-22T15:19:08.444 INFO:tasks.workunit.client.0.smithi096.stdout:6/814: mkdir d3/d3a/d75/d83/d10f 0 2024-04-22T15:19:08.445 INFO:tasks.workunit.client.0.smithi096.stdout:6/815: fsync d3/d28/dd/d11/d2c/d3b/f63 0 2024-04-22T15:19:08.446 INFO:tasks.workunit.client.1.smithi196.stdout:7/710: rename d14/d26/d2a/d2b/da7 to d14/d26/d2a/d2b/de8 0 2024-04-22T15:19:08.449 INFO:tasks.workunit.client.0.smithi096.stdout:6/816: truncate d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/f60 2907912 0 2024-04-22T15:19:08.454 INFO:tasks.workunit.client.1.smithi196.stdout:2/729: rename d1/d13/d25/d2a/cb7 to d1/d13/d25/d9f/c107 0 2024-04-22T15:19:08.455 INFO:tasks.workunit.client.1.smithi196.stdout:1/724: mknod d1/db/dc/cea 0 2024-04-22T15:19:08.456 INFO:tasks.workunit.client.0.smithi096.stdout:6/817: symlink d3/d28/dd/d11/d2c/d6f/l110 0 2024-04-22T15:19:08.456 INFO:tasks.workunit.client.1.smithi196.stdout:1/725: chown d1/db/dc/de/d5a/d5b/d9a/cda 92 1 2024-04-22T15:19:08.460 INFO:tasks.workunit.client.0.smithi096.stdout:4/943: truncate d5/d7/d1f/fa6 46449 0 2024-04-22T15:19:08.462 INFO:tasks.workunit.client.1.smithi196.stdout:9/710: rmdir db/dd/de/d15 39 2024-04-22T15:19:08.465 INFO:tasks.workunit.client.1.smithi196.stdout:7/711: mkdir d14/d4d/da3/de9 0 2024-04-22T15:19:08.466 INFO:tasks.workunit.client.1.smithi196.stdout:7/712: stat d14/d4d/ca8 0 2024-04-22T15:19:08.466 INFO:tasks.workunit.client.1.smithi196.stdout:7/713: fsync f10 0 2024-04-22T15:19:08.467 INFO:tasks.workunit.client.0.smithi096.stdout:4/944: creat d5/d7/d2c/d3d/d4f/d63/f145 x:0 0 0 2024-04-22T15:19:08.468 INFO:tasks.workunit.client.0.smithi096.stdout:4/945: write d5/d7/d1f/d84/d6d/f125 [862729,81094] 0 2024-04-22T15:19:08.471 INFO:tasks.workunit.client.1.smithi196.stdout:4/718: write d6/d14/d18/f36 [3839093,112681] 0 2024-04-22T15:19:08.471 INFO:tasks.workunit.client.1.smithi196.stdout:8/716: dread d2/d35/f63 [0,4194304] 0 2024-04-22T15:19:08.474 INFO:tasks.workunit.client.0.smithi096.stdout:4/946: unlink d5/d7/d2c/d95/dee/f94 0 2024-04-22T15:19:08.474 INFO:tasks.workunit.client.0.smithi096.stdout:4/947: chown d5/d7/d1f/d127/l12e 113964821 1 2024-04-22T15:19:08.480 INFO:tasks.workunit.client.0.smithi096.stdout:4/948: creat d5/d7/dcb/f146 x:0 0 0 2024-04-22T15:19:08.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.477+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8204107600 con 0x7f82040ffb90 2024-04-22T15:19:08.481 INFO:tasks.workunit.client.0.smithi096.stdout:6/818: write d3/d28/dd/d11/d2c/f91 [3676559,64263] 0 2024-04-22T15:19:08.481 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:19:08.481 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:19:08.481 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:19:08.481 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:19:08.481 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:19:08.481 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:19:08.481 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:19:08.481 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:19:08.481 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 13, 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:19:08.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.477+0000 7f81fb7fe700 1 -- 172.21.15.96:0/3715490961 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+841 (secure 0 0 0) 0x7f820005d560 con 0x7f82040ffb90 2024-04-22T15:19:08.483 INFO:tasks.workunit.client.1.smithi196.stdout:1/726: symlink d1/db/d16/d6d/leb 0 2024-04-22T15:19:08.483 INFO:tasks.workunit.client.1.smithi196.stdout:1/727: stat d1/db/d16/f40 0 2024-04-22T15:19:08.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.480+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f81f006dc50 msgr2=0x7f81f0070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:08.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.480+0000 7f820c33e700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f81f006dc50 0x7f81f0070110 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f81f4004630 tx=0x7f81f400b3c0 comp rx=0 tx=0).stop 2024-04-22T15:19:08.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.480+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 msgr2=0x7f82041944c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:08.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.480+0000 7f820c33e700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 0x7f82041944c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8200000c00 tx=0x7f8200005050 comp rx=0 tx=0).stop 2024-04-22T15:19:08.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.480+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 shutdown_connections 2024-04-22T15:19:08.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.481+0000 7f820c33e700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f81f006dc50 0x7f81f0070110 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:08.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.481+0000 7f820c33e700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82040ffb90 0x7f82041944c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:08.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.481+0000 7f820c33e700 1 --2- 172.21.15.96:0/3715490961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8204100d90 0x7f8204194a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:08.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.481+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 >> 172.21.15.96:0/3715490961 conn(0x7f82040fb110 msgr2=0x7f82041043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:08.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.481+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 shutdown_connections 2024-04-22T15:19:08.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.481+0000 7f820c33e700 1 -- 172.21.15.96:0/3715490961 wait complete. 2024-04-22T15:19:08.485 INFO:tasks.workunit.client.0.smithi096.stdout:6/819: write d3/d28/fbd [1022120,51121] 0 2024-04-22T15:19:08.486 INFO:tasks.workunit.client.1.smithi196.stdout:6/748: symlink dd/d1f/ldb 0 2024-04-22T15:19:08.486 INFO:tasks.workunit.client.1.smithi196.stdout:3/745: write d0/d15/d4d/db/de6/fee [8240304,30989] 0 2024-04-22T15:19:08.486 INFO:tasks.workunit.client.1.smithi196.stdout:9/711: mknod db/dd/de/dd7/cee 0 2024-04-22T15:19:08.490 INFO:tasks.workunit.client.0.smithi096.stdout:6/820: creat d3/d3a/d75/d83/df7/f111 x:0 0 0 2024-04-22T15:19:08.490 INFO:tasks.workunit.client.1.smithi196.stdout:5/745: truncate d1/d2/d98/fcc 1764046 0 2024-04-22T15:19:08.491 INFO:tasks.workunit.client.1.smithi196.stdout:2/730: dwrite d1/d13/d25/d2a/f3f [0,4194304] 0 2024-04-22T15:19:08.493 INFO:tasks.workunit.client.0.smithi096.stdout:4/949: link d5/d7/d1a/f137 d5/d7/d1a/d8b/d136/f147 0 2024-04-22T15:19:08.495 INFO:tasks.workunit.client.1.smithi196.stdout:7/714: mkdir d14/d4d/dea 0 2024-04-22T15:19:08.497 INFO:tasks.workunit.client.0.smithi096.stdout:6/821: mknod d3/d28/dd/d11/d2c/d6f/dd4/dd6/c112 0 2024-04-22T15:19:08.500 INFO:tasks.workunit.client.1.smithi196.stdout:8/717: symlink d2/d8/d47/d68/d81/ld3 0 2024-04-22T15:19:08.502 INFO:tasks.workunit.client.1.smithi196.stdout:1/728: creat d1/db/d16/d44/fec x:0 0 0 2024-04-22T15:19:08.505 INFO:tasks.workunit.client.1.smithi196.stdout:3/746: creat d0/d15/d31/dd0/d51/ddd/df6/fff x:0 0 0 2024-04-22T15:19:08.506 INFO:tasks.workunit.client.1.smithi196.stdout:3/747: chown d0/d15/d31/dd0/d51/fc3 3 1 2024-04-22T15:19:08.507 INFO:tasks.workunit.client.1.smithi196.stdout:9/712: fdatasync db/d11/d29/d7a/f9c 0 2024-04-22T15:19:08.507 INFO:tasks.workunit.client.1.smithi196.stdout:3/748: read d0/d15/d31/dd0/d51/ddd/fe4 [180724,21768] 0 2024-04-22T15:19:08.509 INFO:tasks.workunit.client.1.smithi196.stdout:5/746: fdatasync d1/d2/f28 0 2024-04-22T15:19:08.515 INFO:tasks.workunit.client.1.smithi196.stdout:0/744: rename d3/d6/dc/d8c/d8d/d98/fca to d3/de/f108 0 2024-04-22T15:19:08.518 INFO:tasks.workunit.client.1.smithi196.stdout:1/729: symlink d1/db/d16/led 0 2024-04-22T15:19:08.518 INFO:tasks.workunit.client.1.smithi196.stdout:1/730: readlink d1/db/d16/d1e/l2f 0 2024-04-22T15:19:08.524 INFO:tasks.workunit.client.1.smithi196.stdout:3/749: mknod d0/d15/d31/dd0/de2/c100 0 2024-04-22T15:19:08.527 INFO:tasks.workunit.client.1.smithi196.stdout:2/731: mknod d1/d23/df5/c108 0 2024-04-22T15:19:08.528 INFO:tasks.workunit.client.0.smithi096.stdout:4/950: dwrite d5/d7/d1a/d8b/fd4 [0,4194304] 0 2024-04-22T15:19:08.530 INFO:tasks.workunit.client.0.smithi096.stdout:4/951: truncate d5/d7/dcb/f146 996088 0 2024-04-22T15:19:08.530 INFO:tasks.workunit.client.1.smithi196.stdout:7/715: rename d14/f3a to d14/d46/feb 0 2024-04-22T15:19:08.531 INFO:tasks.workunit.client.1.smithi196.stdout:7/716: dread - d14/d26/d5e/f7b zero size 2024-04-22T15:19:08.531 INFO:tasks.workunit.client.0.smithi096.stdout:6/822: write d3/d28/d4d/d82/d9e/f92 [1040445,27804] 0 2024-04-22T15:19:08.531 INFO:tasks.workunit.client.0.smithi096.stdout:4/952: readlink d5/d7/d1a/d8b/lb2 0 2024-04-22T15:19:08.534 INFO:tasks.workunit.client.1.smithi196.stdout:0/745: mkdir d3/d109 0 2024-04-22T15:19:08.539 INFO:tasks.workunit.client.1.smithi196.stdout:8/718: write d2/d3f/f4e [2793337,72269] 0 2024-04-22T15:19:08.540 INFO:tasks.workunit.client.0.smithi096.stdout:6/823: write d3/d28/d4d/d82/fe6 [796045,3991] 0 2024-04-22T15:19:08.540 INFO:tasks.workunit.client.1.smithi196.stdout:6/749: link dd/d1f/d37/f31 dd/d1f/d4e/d53/d7f/da5/fdc 0 2024-04-22T15:19:08.540 INFO:tasks.workunit.client.1.smithi196.stdout:6/750: stat dd/d20/c84 0 2024-04-22T15:19:08.545 INFO:tasks.workunit.client.1.smithi196.stdout:8/719: dwrite d2/f4a [0,4194304] 0 2024-04-22T15:19:08.553 INFO:tasks.workunit.client.0.smithi096.stdout:4/953: link d5/d7/c14 d5/d7/d1a/d103/c148 0 2024-04-22T15:19:08.555 INFO:tasks.workunit.client.1.smithi196.stdout:0/746: readlink d3/de/d14/d86/lb2 0 2024-04-22T15:19:08.560 INFO:tasks.workunit.client.1.smithi196.stdout:1/731: creat d1/db/d16/d8e/ddd/fee x:0 0 0 2024-04-22T15:19:08.562 INFO:tasks.workunit.client.0.smithi096.stdout:4/954: symlink d5/d7/d2c/d95/l149 0 2024-04-22T15:19:08.563 INFO:tasks.workunit.client.0.smithi096.stdout:6/824: dwrite d3/d28/d54/d68/d6e/dbb/fbc [4194304,4194304] 0 2024-04-22T15:19:08.564 INFO:tasks.workunit.client.1.smithi196.stdout:6/751: dread dd/f1a [0,4194304] 0 2024-04-22T15:19:08.567 INFO:tasks.workunit.client.1.smithi196.stdout:1/732: dwrite d1/db/dc/de/d5a/d94/dab/fc2 [0,4194304] 0 2024-04-22T15:19:08.571 INFO:tasks.workunit.client.0.smithi096.stdout:6/825: creat d3/d3a/d75/f113 x:0 0 0 2024-04-22T15:19:08.577 INFO:tasks.workunit.client.1.smithi196.stdout:5/747: rename d1/d17/d1a/d2f/d30 to d1/df1 0 2024-04-22T15:19:08.578 INFO:tasks.workunit.client.0.smithi096.stdout:4/955: symlink d5/d7/d1a/l14a 0 2024-04-22T15:19:08.579 INFO:tasks.workunit.client.1.smithi196.stdout:5/748: truncate d1/df1/d5e/fca 1360876 0 2024-04-22T15:19:08.580 INFO:tasks.workunit.client.1.smithi196.stdout:9/713: getdents db/d11/d29/d57/da7 0 2024-04-22T15:19:08.582 INFO:tasks.workunit.client.0.smithi096.stdout:6/826: creat d3/d28/dd/d11/d2c/d6f/dd4/f114 x:0 0 0 2024-04-22T15:19:08.583 INFO:tasks.workunit.client.1.smithi196.stdout:1/733: mkdir d1/db/dc/dad/d36/def 0 2024-04-22T15:19:08.583 INFO:tasks.workunit.client.1.smithi196.stdout:1/734: fdatasync d1/fb2 0 2024-04-22T15:19:08.584 INFO:tasks.workunit.client.1.smithi196.stdout:0/747: dread d3/d6/f48 [0,4194304] 0 2024-04-22T15:19:08.585 INFO:tasks.workunit.client.0.smithi096.stdout:4/956: rename d5/d7/d2c/d3d/d4f/f60 to d5/d7/d2c/d3d/d4f/d9c/f14b 0 2024-04-22T15:19:08.585 INFO:tasks.workunit.client.1.smithi196.stdout:2/732: link d1/d13/d1f/ddf/da1/l83 d1/d23/d27/d2e/d55/d5b/d93/l109 0 2024-04-22T15:19:08.587 INFO:tasks.workunit.client.0.smithi096.stdout:6/827: creat d3/d28/daf/f115 x:0 0 0 2024-04-22T15:19:08.589 INFO:tasks.workunit.client.1.smithi196.stdout:3/750: rename d0/d15/d31/d9d to d0/d15/d31/d55/dcf/d6f/de5/d101 0 2024-04-22T15:19:08.589 INFO:tasks.workunit.client.1.smithi196.stdout:1/735: dwrite d1/db/dc/de/d5a/d5b/f79 [0,4194304] 0 2024-04-22T15:19:08.590 INFO:tasks.workunit.client.1.smithi196.stdout:1/736: chown d1/db/d16/f7e 239 1 2024-04-22T15:19:08.591 INFO:tasks.workunit.client.1.smithi196.stdout:5/749: rmdir d1/d4 39 2024-04-22T15:19:08.593 INFO:tasks.workunit.client.0.smithi096.stdout:4/957: mkdir d5/d7/dcb/d12a/d14c 0 2024-04-22T15:19:08.596 INFO:tasks.workunit.client.0.smithi096.stdout:6/828: creat d3/d3a/d75/d83/df7/f116 x:0 0 0 2024-04-22T15:19:08.598 INFO:tasks.workunit.client.0.smithi096.stdout:4/958: rmdir d5/d7/d2c/d3d/d13d 39 2024-04-22T15:19:08.600 INFO:tasks.workunit.client.0.smithi096.stdout:6/829: mkdir d3/d28/dd/d11/d2c/d6f/dd4/ddd/d117 0 2024-04-22T15:19:08.601 INFO:tasks.workunit.client.0.smithi096.stdout:6/830: fsync d3/d3a/f3e 0 2024-04-22T15:19:08.602 INFO:tasks.workunit.client.1.smithi196.stdout:6/752: dwrite dd/f1a [0,4194304] 0 2024-04-22T15:19:08.603 INFO:tasks.workunit.client.1.smithi196.stdout:6/753: truncate dd/d2b/d8f/fbb 915430 0 2024-04-22T15:19:08.603 INFO:tasks.workunit.client.0.smithi096.stdout:6/831: write d3/d28/d54/d68/d6e/dbb/fbc [7358920,111226] 0 2024-04-22T15:19:08.605 INFO:tasks.workunit.client.1.smithi196.stdout:7/717: rename f10 to d14/d26/d2a/d72/db1/fec 0 2024-04-22T15:19:08.606 INFO:tasks.workunit.client.1.smithi196.stdout:3/751: fdatasync d0/d15/d31/dd0/d51/d5e/f63 0 2024-04-22T15:19:08.610 INFO:tasks.workunit.client.1.smithi196.stdout:3/752: write d0/d15/da7/d4a/d44/ff3 [390229,6060] 0 2024-04-22T15:19:08.611 INFO:tasks.workunit.client.1.smithi196.stdout:1/737: creat d1/db/d16/d44/ff0 x:0 0 0 2024-04-22T15:19:08.615 INFO:tasks.workunit.client.1.smithi196.stdout:5/750: rmdir d1/d17/d7f 39 2024-04-22T15:19:08.620 INFO:tasks.workunit.client.0.smithi096.stdout:4/959: truncate d5/d7/d1f/d84/d6d/d98/f110 719290 0 2024-04-22T15:19:08.623 INFO:tasks.workunit.client.1.smithi196.stdout:6/754: rmdir dd/d1f/d4e/d53/d55/d6d 39 2024-04-22T15:19:08.625 INFO:tasks.workunit.client.1.smithi196.stdout:0/748: dread d3/de/d14/d5a/f5f [0,4194304] 0 2024-04-22T15:19:08.625 INFO:tasks.workunit.client.0.smithi096.stdout:6/832: rename d3/d28/d4d/d82/fd1 to d3/d3a/d75/d83/df7/f118 0 2024-04-22T15:19:08.626 INFO:tasks.workunit.client.1.smithi196.stdout:0/749: chown d3/d6/dc/f92 1291939 1 2024-04-22T15:19:08.628 INFO:tasks.workunit.client.1.smithi196.stdout:9/714: rename db/d11/d29 to db/dd/de/dd7/def 0 2024-04-22T15:19:08.630 INFO:tasks.workunit.client.1.smithi196.stdout:1/738: dread d1/db/d16/d44/f92 [4194304,4194304] 0 2024-04-22T15:19:08.630 INFO:tasks.workunit.client.1.smithi196.stdout:1/739: fdatasync d1/db/dc/de/d43/d5e/d7a/fa0 0 2024-04-22T15:19:08.632 INFO:tasks.workunit.client.1.smithi196.stdout:1/740: chown d1/db/d16/d1e/d65/f76 16 1 2024-04-22T15:19:08.633 INFO:tasks.workunit.client.1.smithi196.stdout:1/741: write d1/d54/d70/f89 [5224718,125459] 0 2024-04-22T15:19:08.634 INFO:tasks.workunit.client.1.smithi196.stdout:2/733: creat d1/d23/f10a x:0 0 0 2024-04-22T15:19:08.635 INFO:tasks.workunit.client.0.smithi096.stdout:6/833: mkdir d3/d28/dd/d11/d119 0 2024-04-22T15:19:08.638 INFO:tasks.workunit.client.0.smithi096.stdout:6/834: fdatasync d3/f26 0 2024-04-22T15:19:08.639 INFO:tasks.workunit.client.1.smithi196.stdout:4/719: sync 2024-04-22T15:19:08.639 INFO:tasks.workunit.client.1.smithi196.stdout:0/750: symlink d3/db4/dc7/l10a 0 2024-04-22T15:19:08.640 INFO:tasks.workunit.client.1.smithi196.stdout:4/720: readlink d6/d14/d18/d26/d32/da3/ld8 0 2024-04-22T15:19:08.640 INFO:tasks.workunit.client.0.smithi096.stdout:9/859: sync 2024-04-22T15:19:08.641 INFO:tasks.workunit.client.1.smithi196.stdout:7/718: rename d14/d26/d28/d29/lba to d14/d26/d2a/d72/db1/led 0 2024-04-22T15:19:08.641 INFO:tasks.workunit.client.1.smithi196.stdout:4/721: truncate d6/d13/d15/fa6 255215 0 2024-04-22T15:19:08.642 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:08 smithi096 ceph-mon[26688]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:08.646 INFO:tasks.workunit.client.0.smithi096.stdout:4/960: dread d5/d7/d1a/f6c [4194304,4194304] 0 2024-04-22T15:19:08.648 INFO:tasks.workunit.client.0.smithi096.stdout:4/961: stat d5/d7/d2c/d3d/d4f/d63/f112 0 2024-04-22T15:19:08.652 INFO:tasks.workunit.client.1.smithi196.stdout:3/753: mkdir d0/d15/d31/d55/d8b/d102 0 2024-04-22T15:19:08.652 INFO:tasks.workunit.client.1.smithi196.stdout:9/715: write db/dd/de/dd7/def/d7a/fbb [4482,18946] 0 2024-04-22T15:19:08.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.650+0000 7f9767fb5700 1 -- 172.21.15.96:0/2314138993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9760101f70 msgr2=0x7f97601023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:08.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.650+0000 7f9767fb5700 1 --2- 172.21.15.96:0/2314138993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9760101f70 0x7f97601023d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9750009910 tx=0x7f9750009c20 comp rx=0 tx=0).stop 2024-04-22T15:19:08.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.650+0000 7f9767fb5700 1 -- 172.21.15.96:0/2314138993 shutdown_connections 2024-04-22T15:19:08.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.650+0000 7f9767fb5700 1 --2- 172.21.15.96:0/2314138993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9760101f70 0x7f97601023d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:08.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.650+0000 7f9767fb5700 1 --2- 172.21.15.96:0/2314138993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97600feef0 0x7f97600ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:08.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.650+0000 7f9767fb5700 1 -- 172.21.15.96:0/2314138993 >> 172.21.15.96:0/2314138993 conn(0x7f97600fa830 msgr2=0x7f97600fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:08.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.650+0000 7f9767fb5700 1 -- 172.21.15.96:0/2314138993 shutdown_connections 2024-04-22T15:19:08.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.650+0000 7f9767fb5700 1 -- 172.21.15.96:0/2314138993 wait complete. 2024-04-22T15:19:08.654 INFO:tasks.workunit.client.1.smithi196.stdout:9/716: write db/d11/f6a [3028110,41894] 0 2024-04-22T15:19:08.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.651+0000 7f9767fb5700 1 Processor -- start 2024-04-22T15:19:08.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.652+0000 7f9767fb5700 1 -- start start 2024-04-22T15:19:08.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.652+0000 7f9767fb5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97600feef0 0x7f9760101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:08.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.652+0000 7f9767fb5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760101f70 0x7f97600fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:08.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.652+0000 7f9767fb5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976019a2c0 con 0x7f97600feef0 2024-04-22T15:19:08.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.652+0000 7f9767fb5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976019a400 con 0x7f9760101f70 2024-04-22T15:19:08.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.653+0000 7f9765550700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760101f70 0x7f97600fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:08.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.653+0000 7f9765d51700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97600feef0 0x7f9760101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:08.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.653+0000 7f9765d51700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97600feef0 0x7f9760101830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34908/0 (socket says 172.21.15.96:34908) 2024-04-22T15:19:08.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.653+0000 7f9765d51700 1 -- 172.21.15.96:0/1929823511 learned_addr learned my addr 172.21.15.96:0/1929823511 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.653+0000 7f9765550700 1 -- 172.21.15.96:0/1929823511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97600feef0 msgr2=0x7f9760101830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.653+0000 7f9765550700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97600feef0 0x7f9760101830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.653+0000 7f9765550700 1 -- 172.21.15.96:0/1929823511 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500095f0 con 0x7f9760101f70 2024-04-22T15:19:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.654+0000 7f9765550700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760101f70 0x7f97600fff20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9750006010 tx=0x7f9750005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.654+0000 7f9756ffd700 1 -- 172.21.15.96:0/1929823511 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975001d070 con 0x7f9760101f70 2024-04-22T15:19:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.654+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9760100460 con 0x7f9760101f70 2024-04-22T15:19:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.654+0000 7f9756ffd700 1 -- 172.21.15.96:0/1929823511 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97500052a0 con 0x7f9760101f70 2024-04-22T15:19:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.654+0000 7f9756ffd700 1 -- 172.21.15.96:0/1929823511 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9750017400 con 0x7f9760101f70 2024-04-22T15:19:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.654+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97601009a0 con 0x7f9760101f70 2024-04-22T15:19:08.658 INFO:tasks.workunit.client.1.smithi196.stdout:2/734: stat d1/d13/d25/d2a/ddb/dff/l103 0 2024-04-22T15:19:08.659 INFO:tasks.workunit.client.0.smithi096.stdout:6/835: creat d3/d28/d54/d68/f11a x:0 0 0 2024-04-22T15:19:08.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.656+0000 7f9756ffd700 1 -- 172.21.15.96:0/1929823511 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 91832+0+0 (secure 0 0 0) 0x7f9750017560 con 0x7f9760101f70 2024-04-22T15:19:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.657+0000 7f9756ffd700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f974c06dc90 0x7f974c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.657+0000 7f9756ffd700 1 -- 172.21.15.96:0/1929823511 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f975008eb10 con 0x7f9760101f70 2024-04-22T15:19:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.657+0000 7f9765d51700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f974c06dc90 0x7f974c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.657+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976010a140 con 0x7f9760101f70 2024-04-22T15:19:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.658+0000 7f9765d51700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f974c06dc90 0x7f974c070150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f975c005950 tx=0x7f975c009260 comp rx=0 tx=0).ready entity=mgr.24359 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:08.662 INFO:tasks.workunit.client.1.smithi196.stdout:0/751: symlink d3/d9/d36/d9f/dd1/d65/d7f/ddb/l10b 0 2024-04-22T15:19:08.662 INFO:tasks.workunit.client.0.smithi096.stdout:4/962: dwrite d5/d5b/f142 [0,4194304] 0 2024-04-22T15:19:08.662 INFO:tasks.workunit.client.0.smithi096.stdout:9/860: dwrite d1/df/d95/d9d/db5/d43/fa3 [0,4194304] 0 2024-04-22T15:19:08.663 INFO:tasks.workunit.client.1.smithi196.stdout:7/719: creat d14/d4d/fee x:0 0 0 2024-04-22T15:19:08.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:08.664+0000 7f9756ffd700 1 -- 172.21.15.96:0/1929823511 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f976010a140 con 0x7f9760101f70 2024-04-22T15:19:08.670 INFO:tasks.workunit.client.1.smithi196.stdout:6/755: dwrite dd/d1f/d37/f31 [4194304,4194304] 0 2024-04-22T15:19:08.670 INFO:tasks.workunit.client.0.smithi096.stdout:4/963: mknod d5/d7/d1a/d103/c14d 0 2024-04-22T15:19:08.673 INFO:tasks.workunit.client.0.smithi096.stdout:9/861: mknod d1/df/d95/d9d/db5/d43/d57/d5b/d73/c122 0 2024-04-22T15:19:08.676 INFO:tasks.workunit.client.0.smithi096.stdout:4/964: dread d5/d7/fbd [0,4194304] 0 2024-04-22T15:19:08.677 INFO:tasks.workunit.client.1.smithi196.stdout:1/742: mkdir d1/d54/de2/df1 0 2024-04-22T15:19:08.678 INFO:tasks.workunit.client.1.smithi196.stdout:0/752: dread - d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/fa9 zero size 2024-04-22T15:19:08.680 INFO:tasks.workunit.client.1.smithi196.stdout:0/753: chown d3/d6/dc/d8c/cda 3148774 1 2024-04-22T15:19:08.681 INFO:tasks.workunit.client.1.smithi196.stdout:4/722: symlink d6/d14/d18/d26/d32/d67/de7/le9 0 2024-04-22T15:19:08.684 INFO:tasks.workunit.client.1.smithi196.stdout:9/717: creat db/dd/de/d15/ff0 x:0 0 0 2024-04-22T15:19:08.687 INFO:tasks.workunit.client.1.smithi196.stdout:0/754: unlink d3/d9/f54 0 2024-04-22T15:19:08.692 INFO:tasks.workunit.client.1.smithi196.stdout:7/720: creat d14/d15/fef x:0 0 0 2024-04-22T15:19:08.693 INFO:tasks.workunit.client.1.smithi196.stdout:0/755: dwrite d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dba/fe0 [0,4194304] 0 2024-04-22T15:19:08.694 INFO:tasks.workunit.client.1.smithi196.stdout:7/721: write d14/d4d/da3/fbc [693810,69167] 0 2024-04-22T15:19:08.694 INFO:tasks.workunit.client.1.smithi196.stdout:8/720: sync 2024-04-22T15:19:08.698 INFO:tasks.workunit.client.1.smithi196.stdout:7/722: mkdir d14/d25/d31/df0 0 2024-04-22T15:19:08.698 INFO:tasks.workunit.client.1.smithi196.stdout:7/723: fsync d14/d25/d80/fbd 0 2024-04-22T15:19:08.699 INFO:tasks.workunit.client.1.smithi196.stdout:7/724: dread - d14/d26/fd0 zero size 2024-04-22T15:19:08.701 INFO:tasks.workunit.client.1.smithi196.stdout:8/721: rename d2/d8/d47/d68/la9 to d2/d8/d47/d68/dbc/da4/ld4 0 2024-04-22T15:19:08.701 INFO:tasks.workunit.client.1.smithi196.stdout:9/718: getdents db/dd/de/d15/d31/d99 0 2024-04-22T15:19:08.702 INFO:tasks.workunit.client.1.smithi196.stdout:8/722: dread d2/d3f/d4b/f6f [0,4194304] 0 2024-04-22T15:19:08.703 INFO:tasks.workunit.client.0.smithi096.stdout:9/862: dread d1/df/d95/d9d/db5/d43/f5f [4194304,4194304] 0 2024-04-22T15:19:08.705 INFO:tasks.workunit.client.1.smithi196.stdout:7/725: mkdir d14/d4d/da3/dc8/df1 0 2024-04-22T15:19:08.707 INFO:tasks.workunit.client.0.smithi096.stdout:6/836: dwrite d3/d3a/d75/d83/f98 [0,4194304] 0 2024-04-22T15:19:08.708 INFO:tasks.workunit.client.0.smithi096.stdout:4/965: dread d5/f13 [4194304,4194304] 0 2024-04-22T15:19:08.708 INFO:tasks.workunit.client.1.smithi196.stdout:3/754: write d0/d15/d31/d55/d8b/fc0 [1861008,106241] 0 2024-04-22T15:19:08.709 INFO:tasks.workunit.client.0.smithi096.stdout:9/863: read - d1/df/d95/d9d/db5/d43/d57/d5b/da1/fc9 zero size 2024-04-22T15:19:08.709 INFO:tasks.workunit.client.0.smithi096.stdout:4/966: chown d5/d7/d2c/d3d/d4f 3 1 2024-04-22T15:19:08.711 INFO:tasks.workunit.client.1.smithi196.stdout:1/743: write d1/db/d16/d1e/d49/f42 [264167,39864] 0 2024-04-22T15:19:08.712 INFO:tasks.workunit.client.1.smithi196.stdout:6/756: dwrite dd/d20/d5c/d6a/f82 [0,4194304] 0 2024-04-22T15:19:08.713 INFO:tasks.workunit.client.1.smithi196.stdout:1/744: fsync d1/db/d16/d8e/fbd 0 2024-04-22T15:19:08.723 INFO:tasks.workunit.client.0.smithi096.stdout:9/864: rename d1/df/d95/d9d/db5/d43/d57/d5b/da1/dfc to d1/df/d95/d123 0 2024-04-22T15:19:08.728 INFO:tasks.workunit.client.0.smithi096.stdout:6/837: write d3/d28/dd/fa5 [1304209,80594] 0 2024-04-22T15:19:08.728 INFO:tasks.workunit.client.0.smithi096.stdout:4/967: truncate d5/d7/f8e 1349855 0 2024-04-22T15:19:08.731 INFO:tasks.workunit.client.0.smithi096.stdout:9/865: creat d1/df/d95/d9d/db5/d43/d57/d5b/d73/de3/f124 x:0 0 0 2024-04-22T15:19:08.733 INFO:tasks.workunit.client.0.smithi096.stdout:4/968: mknod d5/d7/d1f/c14e 0 2024-04-22T15:19:08.734 INFO:tasks.workunit.client.1.smithi196.stdout:4/723: write d6/d13/f68 [2238555,107058] 0 2024-04-22T15:19:08.735 INFO:tasks.workunit.client.0.smithi096.stdout:6/838: dwrite d3/d28/dd/d11/d2c/f91 [0,4194304] 0 2024-04-22T15:19:08.735 INFO:tasks.workunit.client.1.smithi196.stdout:0/756: write d3/de/d14/d5a/f68 [143823,52811] 0 2024-04-22T15:19:08.736 INFO:tasks.workunit.client.0.smithi096.stdout:6/839: read - d3/d28/dd/fba zero size 2024-04-22T15:19:08.739 INFO:tasks.workunit.client.1.smithi196.stdout:0/757: readlink d3/de/d14/d5a/l80 0 2024-04-22T15:19:08.740 INFO:tasks.workunit.client.1.smithi196.stdout:2/735: dwrite d1/d13/d1f/f3a [0,4194304] 0 2024-04-22T15:19:08.745 INFO:tasks.workunit.client.0.smithi096.stdout:6/840: chown d3/d28/d54/d68/d6e/cdb 79887 1 2024-04-22T15:19:08.746 INFO:tasks.workunit.client.1.smithi196.stdout:9/719: creat db/dd/de/ff1 x:0 0 0 2024-04-22T15:19:08.746 INFO:tasks.workunit.client.0.smithi096.stdout:9/866: write d1/df/d95/d9d/db5/d43/f5c [153612,115905] 0 2024-04-22T15:19:08.748 INFO:tasks.workunit.client.1.smithi196.stdout:8/723: dwrite d2/d6a/f8c [0,4194304] 0 2024-04-22T15:19:08.753 INFO:tasks.workunit.client.0.smithi096.stdout:4/969: dwrite d5/d7/d1f/fec [0,4194304] 0 2024-04-22T15:19:08.754 INFO:tasks.workunit.client.0.smithi096.stdout:1/959: sync 2024-04-22T15:19:08.754 INFO:tasks.workunit.client.0.smithi096.stdout:5/961: sync 2024-04-22T15:19:08.754 INFO:tasks.workunit.client.0.smithi096.stdout:8/916: sync 2024-04-22T15:19:08.754 INFO:tasks.workunit.client.0.smithi096.stdout:2/979: sync 2024-04-22T15:19:08.754 INFO:tasks.workunit.client.0.smithi096.stdout:3/961: sync 2024-04-22T15:19:08.757 INFO:tasks.workunit.client.0.smithi096.stdout:1/960: readlink d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/dc3/l117 0 2024-04-22T15:19:08.758 INFO:tasks.workunit.client.1.smithi196.stdout:2/736: dread - d1/d23/d27/d89/fb0 zero size 2024-04-22T15:19:08.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:08 smithi196 ceph-mon[34717]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:08.761 INFO:tasks.workunit.client.0.smithi096.stdout:1/961: readlink d6/dc/d8a/d3f/d62/d7d/ddb/d9b/lf3 0 2024-04-22T15:19:08.762 INFO:tasks.workunit.client.0.smithi096.stdout:8/917: chown d4/d1c/d40/d48/lfa 7938430 1 2024-04-22T15:19:08.764 INFO:tasks.workunit.client.0.smithi096.stdout:3/962: mkdir da/db/de/ddc/d26/db2/d13c 0 2024-04-22T15:19:08.764 INFO:tasks.workunit.client.0.smithi096.stdout:8/918: write d4/d10/d25/d4c/d76/d7f/f11d [378022,103513] 0 2024-04-22T15:19:08.765 INFO:tasks.workunit.client.0.smithi096.stdout:8/919: chown d4/d2a/c30 1352250870 1 2024-04-22T15:19:08.766 INFO:tasks.workunit.client.1.smithi196.stdout:3/755: creat d0/d15/d31/d55/d8b/dbf/f103 x:0 0 0 2024-04-22T15:19:08.770 INFO:tasks.workunit.client.0.smithi096.stdout:4/970: truncate d5/d7/d2c/f65 1713296 0 2024-04-22T15:19:08.770 INFO:tasks.workunit.client.0.smithi096.stdout:2/980: chown d3/db/df/d24/d58/d5f/dfe/d115/d2f/cbe 131120469 1 2024-04-22T15:19:08.771 INFO:tasks.workunit.client.0.smithi096.stdout:5/962: readlink d4/ddb/d3d/d4d/d59/d5a/d5c/l149 0 2024-04-22T15:19:08.775 INFO:tasks.workunit.client.1.smithi196.stdout:6/757: dread dd/d1f/d37/f8e [0,4194304] 0 2024-04-22T15:19:08.777 INFO:tasks.workunit.client.0.smithi096.stdout:5/963: chown d4/ddb/d3d/d4d/d99/cf3 6892288 1 2024-04-22T15:19:08.777 INFO:tasks.workunit.client.1.smithi196.stdout:4/724: dwrite d6/d14/d18/d26/f6a [0,4194304] 0 2024-04-22T15:19:08.778 INFO:tasks.workunit.client.0.smithi096.stdout:5/964: stat d4/ddb/d3d/d4d/d59/d5a/d5c/c73 0 2024-04-22T15:19:08.780 INFO:tasks.workunit.client.1.smithi196.stdout:3/756: rename d0/d15/d31/d55/fb6 to d0/d15/d31/d55/d8b/df2/f104 0 2024-04-22T15:19:08.781 INFO:tasks.workunit.client.1.smithi196.stdout:0/758: link d3/d9/d25/l83 d3/d6/dc/l10c 0 2024-04-22T15:19:08.781 INFO:tasks.workunit.client.1.smithi196.stdout:5/751: sync 2024-04-22T15:19:08.783 INFO:tasks.workunit.client.1.smithi196.stdout:1/745: getdents d1/db/d16/d8e/ddd 0 2024-04-22T15:19:08.789 INFO:tasks.workunit.client.0.smithi096.stdout:6/841: dread d3/d28/d54/d68/f76 [0,4194304] 0 2024-04-22T15:19:08.790 INFO:tasks.workunit.client.0.smithi096.stdout:5/965: creat d4/ddb/d3d/ded/f14e x:0 0 0 2024-04-22T15:19:08.792 INFO:tasks.workunit.client.1.smithi196.stdout:6/758: dread dd/d1f/d4e/d53/d55/f87 [0,4194304] 0 2024-04-22T15:19:08.794 INFO:tasks.workunit.client.0.smithi096.stdout:9/867: dwrite d1/d85/df4/ffa [0,4194304] 0 2024-04-22T15:19:08.795 INFO:tasks.workunit.client.0.smithi096.stdout:6/842: chown d3/d28/dd/d11/d2c/d6f/dd4/ddd/d117 126060123 1 2024-04-22T15:19:08.795 INFO:tasks.workunit.client.0.smithi096.stdout:2/981: creat d3/db/df/dce/d46/d7e/d84/d12e/f13d x:0 0 0 2024-04-22T15:19:08.799 INFO:tasks.workunit.client.1.smithi196.stdout:4/725: creat d6/d14/d18/d26/d32/da3/fea x:0 0 0 2024-04-22T15:19:08.800 INFO:tasks.workunit.client.1.smithi196.stdout:9/720: dwrite fa [0,4194304] 0 2024-04-22T15:19:08.801 INFO:tasks.workunit.client.0.smithi096.stdout:9/868: read d1/d25/db2/d98/fd0 [954334,97390] 0 2024-04-22T15:19:08.805 INFO:tasks.workunit.client.1.smithi196.stdout:7/726: sync 2024-04-22T15:19:08.807 INFO:tasks.workunit.client.1.smithi196.stdout:1/746: rename d1/db/d16/f73 to d1/db/d16/d8e/ff2 0 2024-04-22T15:19:08.808 INFO:tasks.workunit.client.0.smithi096.stdout:1/962: dwrite d6/dc/d8a/d3f/d62/dc2/ff4 [0,4194304] 0 2024-04-22T15:19:08.810 INFO:tasks.workunit.client.0.smithi096.stdout:4/971: dread d5/d7/d1f/f42 [0,4194304] 0 2024-04-22T15:19:08.810 INFO:tasks.workunit.client.0.smithi096.stdout:5/966: symlink d4/ddb/d3d/d4d/d59/db0/d126/l14f 0 2024-04-22T15:19:08.812 INFO:tasks.workunit.client.1.smithi196.stdout:2/737: link d1/d1d/f7a d1/d76/f10b 0 2024-04-22T15:19:08.812 INFO:tasks.workunit.client.1.smithi196.stdout:9/721: dread db/ff [4194304,4194304] 0 2024-04-22T15:19:08.813 INFO:tasks.workunit.client.1.smithi196.stdout:8/724: getdents d2/d8/d47/d68/dbc/da4 0 2024-04-22T15:19:08.816 INFO:tasks.workunit.client.1.smithi196.stdout:6/759: dread - dd/d1f/d38/f92 zero size 2024-04-22T15:19:08.819 INFO:tasks.workunit.client.0.smithi096.stdout:4/972: dwrite d5/d7/d1a/d8b/fd4 [0,4194304] 0 2024-04-22T15:19:08.820 INFO:tasks.workunit.client.1.smithi196.stdout:8/725: dread d2/d35/f63 [0,4194304] 0 2024-04-22T15:19:08.820 INFO:tasks.workunit.client.0.smithi096.stdout:8/920: link d4/c61 d4/d10/d4d/deb/c137 0 2024-04-22T15:19:08.821 INFO:tasks.workunit.client.1.smithi196.stdout:2/738: dwrite d1/fa2 [0,4194304] 0 2024-04-22T15:19:08.822 INFO:tasks.workunit.client.0.smithi096.stdout:3/963: getdents da/db/de/ddc 0 2024-04-22T15:19:08.827 INFO:tasks.workunit.client.0.smithi096.stdout:5/967: read - d4/ddb/d3d/d4d/d59/fc0 zero size 2024-04-22T15:19:08.828 INFO:tasks.workunit.client.1.smithi196.stdout:8/726: dwrite d2/d3f/d4b/f72 [0,4194304] 0 2024-04-22T15:19:08.834 INFO:tasks.workunit.client.1.smithi196.stdout:3/757: dwrite d0/d15/da7/d85/f9c [0,4194304] 0 2024-04-22T15:19:08.835 INFO:tasks.workunit.client.1.smithi196.stdout:5/752: dread d1/d17/f24 [0,4194304] 0 2024-04-22T15:19:08.836 INFO:tasks.workunit.client.1.smithi196.stdout:3/758: fdatasync d0/d15/d31/dd0/d51/ddd/df6/fff 0 2024-04-22T15:19:08.839 INFO:tasks.workunit.client.0.smithi096.stdout:1/963: fsync d6/dc/d8a/d3f/d54/d32/d3b/d56/fb9 0 2024-04-22T15:19:08.842 INFO:tasks.workunit.client.0.smithi096.stdout:2/982: mknod d3/db/df/c13e 0 2024-04-22T15:19:08.842 INFO:tasks.workunit.client.1.smithi196.stdout:9/722: read db/dd/de/dd7/def/f38 [968108,6204] 0 2024-04-22T15:19:08.843 INFO:tasks.workunit.client.1.smithi196.stdout:0/759: truncate d3/de/d14/f81 3299983 0 2024-04-22T15:19:08.849 INFO:tasks.workunit.client.0.smithi096.stdout:2/983: dread d3/db/df/d24/d58/d5f/dfe/d115/f7a [0,4194304] 0 2024-04-22T15:19:08.851 INFO:tasks.workunit.client.0.smithi096.stdout:4/973: stat d5/d7/d2c/d3d/d4f/d9c/c117 0 2024-04-22T15:19:08.851 INFO:tasks.workunit.client.0.smithi096.stdout:8/921: rmdir d4/d1c/d40/da7 39 2024-04-22T15:19:08.853 INFO:tasks.workunit.client.0.smithi096.stdout:1/964: dread d6/dc/d8a/d3f/d62/d7d/dd3/fd6 [0,4194304] 0 2024-04-22T15:19:08.853 INFO:tasks.workunit.client.1.smithi196.stdout:2/739: creat d1/d13/d1f/ddf/da1/f10c x:0 0 0 2024-04-22T15:19:08.853 INFO:tasks.workunit.client.0.smithi096.stdout:1/965: chown d6/dc/d8a 653769739 1 2024-04-22T15:19:08.854 INFO:tasks.workunit.client.0.smithi096.stdout:8/922: dread - d4/d10/d4d/f12a zero size 2024-04-22T15:19:08.861 INFO:tasks.workunit.client.0.smithi096.stdout:4/974: dread d5/fc2 [0,4194304] 0 2024-04-22T15:19:08.862 INFO:tasks.workunit.client.0.smithi096.stdout:6/843: dread d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/fc0 [0,4194304] 0 2024-04-22T15:19:08.862 INFO:tasks.workunit.client.1.smithi196.stdout:5/753: stat d1/d9/d68/l7c 0 2024-04-22T15:19:08.862 INFO:tasks.workunit.client.0.smithi096.stdout:2/984: dread - d3/db/df/d10/db5/fd9 zero size 2024-04-22T15:19:08.864 INFO:tasks.workunit.client.1.smithi196.stdout:3/759: symlink d0/d15/d31/d55/d5d/dc1/l105 0 2024-04-22T15:19:08.867 INFO:tasks.workunit.client.1.smithi196.stdout:8/727: write d2/d3f/d4b/f6f [77490,88174] 0 2024-04-22T15:19:08.868 INFO:tasks.workunit.client.0.smithi096.stdout:5/968: write d4/ddb/d3d/d4d/d59/d98/db9/ff0 [490222,121013] 0 2024-04-22T15:19:08.872 INFO:tasks.workunit.client.1.smithi196.stdout:0/760: rmdir d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8 39 2024-04-22T15:19:08.876 INFO:tasks.workunit.client.1.smithi196.stdout:3/760: dread d0/d15/d31/d55/dcf/d8/d6b/fd6 [0,4194304] 0 2024-04-22T15:19:08.877 INFO:tasks.workunit.client.1.smithi196.stdout:4/726: creat d6/d13/d15/feb x:0 0 0 2024-04-22T15:19:08.878 INFO:tasks.workunit.client.1.smithi196.stdout:2/740: dread - d1/d13/d25/d2a/f95 zero size 2024-04-22T15:19:08.878 INFO:tasks.workunit.client.1.smithi196.stdout:4/727: write d6/d13/d15/d2c/d8c/db3/fdf [959962,124212] 0 2024-04-22T15:19:08.882 INFO:tasks.workunit.client.1.smithi196.stdout:9/723: dwrite db/dd/d1b/d2c/f58 [0,4194304] 0 2024-04-22T15:19:08.882 INFO:tasks.workunit.client.0.smithi096.stdout:6/844: stat d3/d28/dd/d11/d2c/f56 0 2024-04-22T15:19:08.884 INFO:tasks.workunit.client.0.smithi096.stdout:6/845: write d3/d28/f61 [2808599,59754] 0 2024-04-22T15:19:08.886 INFO:tasks.workunit.client.1.smithi196.stdout:8/728: chown d2/d5/d2e/f67 141 1 2024-04-22T15:19:08.887 INFO:tasks.workunit.client.0.smithi096.stdout:9/869: link d1/df/d95/d9d/db5/d43/d57/d5b/da1/cda d1/df/d95/d9d/db5/d59/c125 0 2024-04-22T15:19:08.888 INFO:tasks.workunit.client.1.smithi196.stdout:0/761: mkdir d3/d6/dc/d8c/d8d/d98/d10d 0 2024-04-22T15:19:08.890 INFO:tasks.workunit.client.0.smithi096.stdout:4/975: dwrite d5/d7/f9 [4194304,4194304] 0 2024-04-22T15:19:08.893 INFO:tasks.workunit.client.1.smithi196.stdout:6/760: rmdir dd/d1f/d37/d3c/d89 0 2024-04-22T15:19:08.893 INFO:tasks.workunit.client.0.smithi096.stdout:5/969: rename d4/ddb/d3d/d4d/d59/d98/fd0 to d4/ddb/d3d/d35/d57/dbe/d113/d14a/f150 0 2024-04-22T15:19:08.894 INFO:tasks.workunit.client.1.smithi196.stdout:5/754: dread d1/d4/f9c [0,4194304] 0 2024-04-22T15:19:08.898 INFO:tasks.workunit.client.0.smithi096.stdout:1/966: unlink d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/dbc/fdd 0 2024-04-22T15:19:08.899 INFO:tasks.workunit.client.1.smithi196.stdout:3/761: mknod d0/d15/d31/dd0/d51/d7f/c106 0 2024-04-22T15:19:08.901 INFO:tasks.workunit.client.1.smithi196.stdout:7/727: getdents d14 0 2024-04-22T15:19:08.902 INFO:tasks.workunit.client.1.smithi196.stdout:6/761: dwrite dd/d20/d5c/fb7 [0,4194304] 0 2024-04-22T15:19:08.902 INFO:tasks.workunit.client.1.smithi196.stdout:3/762: stat d0/d15/f7a 0 2024-04-22T15:19:08.907 INFO:tasks.workunit.client.1.smithi196.stdout:9/724: fdatasync db/dd/d1b/f45 0 2024-04-22T15:19:08.909 INFO:tasks.workunit.client.1.smithi196.stdout:3/763: dwrite d0/d15/da7/d4a/d44/ff3 [0,4194304] 0 2024-04-22T15:19:08.910 INFO:tasks.workunit.client.1.smithi196.stdout:8/729: mkdir d2/d5/d93/dd5 0 2024-04-22T15:19:08.912 INFO:tasks.workunit.client.0.smithi096.stdout:5/970: creat d4/d2c/d2f/d31/d82/def/f151 x:0 0 0 2024-04-22T15:19:08.913 INFO:tasks.workunit.client.0.smithi096.stdout:1/967: rmdir d6/dc/d8a/d29 39 2024-04-22T15:19:08.914 INFO:tasks.workunit.client.1.smithi196.stdout:5/755: creat d1/df1/d7d/ff2 x:0 0 0 2024-04-22T15:19:08.915 INFO:tasks.workunit.client.0.smithi096.stdout:1/968: write d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/f108 [1257526,39589] 0 2024-04-22T15:19:08.916 INFO:tasks.workunit.client.1.smithi196.stdout:2/741: mknod d1/d13/dbc/c10d 0 2024-04-22T15:19:08.921 INFO:tasks.workunit.client.1.smithi196.stdout:9/725: dread db/d11/f6a [0,4194304] 0 2024-04-22T15:19:08.925 INFO:tasks.workunit.client.0.smithi096.stdout:8/923: write d4/d1c/d40/da7/ffe [879897,11114] 0 2024-04-22T15:19:08.929 INFO:tasks.workunit.client.0.smithi096.stdout:2/985: getdents d3/db/df 0 2024-04-22T15:19:08.931 INFO:tasks.workunit.client.1.smithi196.stdout:4/728: link d6/d90/f60 d6/d14/d18/d40/d74/fec 0 2024-04-22T15:19:08.933 INFO:tasks.workunit.client.0.smithi096.stdout:5/971: mknod d4/ddb/d3d/d4d/d59/db0/d126/c152 0 2024-04-22T15:19:08.934 INFO:tasks.workunit.client.0.smithi096.stdout:5/972: write d4/f1c [1386048,107734] 0 2024-04-22T15:19:08.934 INFO:tasks.workunit.client.0.smithi096.stdout:9/870: write d1/d85/f94 [1108871,83276] 0 2024-04-22T15:19:08.935 INFO:tasks.workunit.client.0.smithi096.stdout:1/969: dread - d6/dc/d8a/d3f/d54/d32/d3b/d56/f95 zero size 2024-04-22T15:19:08.936 INFO:tasks.workunit.client.0.smithi096.stdout:9/871: fdatasync d1/df/d95/d9d/db5/dd/dba/dc2/f11e 0 2024-04-22T15:19:08.939 INFO:tasks.workunit.client.0.smithi096.stdout:1/970: chown d6/dc/d8a/d3f/d54/d32/d8c/d45/da9/leb 339493 1 2024-04-22T15:19:08.940 INFO:tasks.workunit.client.0.smithi096.stdout:5/973: dread d4/d2c/d2f/d31/f8f [0,4194304] 0 2024-04-22T15:19:08.941 INFO:tasks.workunit.client.0.smithi096.stdout:6/846: rename d3/d28/l97 to d3/d28/d54/d68/l11b 0 2024-04-22T15:19:08.942 INFO:tasks.workunit.client.0.smithi096.stdout:6/847: stat d3/d28/d4d/ff9 0 2024-04-22T15:19:08.943 INFO:tasks.workunit.client.1.smithi196.stdout:8/730: truncate d2/dc/fd 2006281 0 2024-04-22T15:19:08.943 INFO:tasks.workunit.client.0.smithi096.stdout:6/848: chown d3/d3a/d75/d83/ca7 5 1 2024-04-22T15:19:08.945 INFO:tasks.workunit.client.1.smithi196.stdout:3/764: truncate d0/d15/da7/d4a/d44/f20 3123982 0 2024-04-22T15:19:08.945 INFO:tasks.workunit.client.1.smithi196.stdout:1/747: sync 2024-04-22T15:19:08.953 INFO:tasks.workunit.client.0.smithi096.stdout:4/976: dwrite d5/d7/f8e [0,4194304] 0 2024-04-22T15:19:08.954 INFO:tasks.workunit.client.1.smithi196.stdout:9/726: fdatasync db/dd/de/fe9 0 2024-04-22T15:19:08.958 INFO:tasks.workunit.client.1.smithi196.stdout:4/729: creat d6/d13/d15/d2c/d8c/db3/fed x:0 0 0 2024-04-22T15:19:08.959 INFO:tasks.workunit.client.0.smithi096.stdout:2/986: rename d3/db/d10d/f92 to d3/db/df/d24/d58/d5f/d11d/f13f 0 2024-04-22T15:19:08.961 INFO:tasks.workunit.client.1.smithi196.stdout:9/727: dwrite db/dd/dd0/fe5 [0,4194304] 0 2024-04-22T15:19:08.969 INFO:tasks.workunit.client.1.smithi196.stdout:1/748: unlink d1/db/dc/d26/f28 0 2024-04-22T15:19:08.971 INFO:tasks.workunit.client.1.smithi196.stdout:5/756: mknod d1/df1/d6e/db2/cf3 0 2024-04-22T15:19:08.972 INFO:tasks.workunit.client.1.smithi196.stdout:1/749: read d1/db/d16/f5c [1254193,97373] 0 2024-04-22T15:19:08.975 INFO:tasks.workunit.client.1.smithi196.stdout:2/742: unlink d1/d13/f19 0 2024-04-22T15:19:08.976 INFO:tasks.workunit.client.0.smithi096.stdout:9/872: dwrite d1/df/d95/d9d/db5/dd/dba/dc2/dc4/fd7 [0,4194304] 0 2024-04-22T15:19:08.980 INFO:tasks.workunit.client.1.smithi196.stdout:7/728: creat d14/d25/ff2 x:0 0 0 2024-04-22T15:19:08.980 INFO:tasks.workunit.client.0.smithi096.stdout:4/977: read d5/f39 [4165727,34927] 0 2024-04-22T15:19:08.983 INFO:tasks.workunit.client.1.smithi196.stdout:4/730: dread d6/d13/f25 [0,4194304] 0 2024-04-22T15:19:08.984 INFO:tasks.workunit.client.0.smithi096.stdout:5/974: symlink d4/d2c/l153 0 2024-04-22T15:19:08.989 INFO:tasks.workunit.client.1.smithi196.stdout:8/731: dwrite d2/d5/d2e/f43 [0,4194304] 0 2024-04-22T15:19:08.990 INFO:tasks.workunit.client.0.smithi096.stdout:6/849: fsync d3/d28/d4d/d82/fb9 0 2024-04-22T15:19:08.995 INFO:tasks.workunit.client.1.smithi196.stdout:0/762: sync 2024-04-22T15:19:08.995 INFO:tasks.workunit.client.0.smithi096.stdout:9/873: creat d1/df/d95/d9d/db5/dd/dba/dc2/f126 x:0 0 0 2024-04-22T15:19:08.999 INFO:tasks.workunit.client.1.smithi196.stdout:1/750: symlink d1/d54/lf3 0 2024-04-22T15:19:09.001 INFO:tasks.workunit.client.0.smithi096.stdout:5/975: mkdir d4/ddb/d3d/d4d/d59/db0/d126/d154 0 2024-04-22T15:19:09.004 INFO:tasks.workunit.client.0.smithi096.stdout:8/924: getdents d4/d10/dcc 0 2024-04-22T15:19:09.006 INFO:tasks.workunit.client.1.smithi196.stdout:2/743: dwrite d1/fd [4194304,4194304] 0 2024-04-22T15:19:09.008 INFO:tasks.workunit.client.0.smithi096.stdout:1/971: getdents d6/dc/d8a 0 2024-04-22T15:19:09.012 INFO:tasks.workunit.client.1.smithi196.stdout:6/762: mkdir dd/d1f/d4e/d53/d55/d6d/dd1/dca/ddd 0 2024-04-22T15:19:09.013 INFO:tasks.workunit.client.1.smithi196.stdout:6/763: truncate dd/d1f/d38/f94 5106765 0 2024-04-22T15:19:09.014 INFO:tasks.workunit.client.0.smithi096.stdout:4/978: symlink d5/d7/d2c/d43/l14f 0 2024-04-22T15:19:09.017 INFO:tasks.workunit.client.0.smithi096.stdout:5/976: mknod d4/ddb/d3d/d4d/d59/d98/dee/c155 0 2024-04-22T15:19:09.019 INFO:tasks.workunit.client.0.smithi096.stdout:2/987: creat d3/db/df/d98/f140 x:0 0 0 2024-04-22T15:19:09.022 INFO:tasks.workunit.client.1.smithi196.stdout:4/731: dwrite d6/d13/d15/d2c/fa0 [0,4194304] 0 2024-04-22T15:19:09.023 INFO:tasks.workunit.client.1.smithi196.stdout:3/765: link d0/d15/d31/d55/d5d/fa0 d0/d15/d31/d55/dcf/d8/f107 0 2024-04-22T15:19:09.025 INFO:tasks.workunit.client.0.smithi096.stdout:6/850: dwrite d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/fb8 [0,4194304] 0 2024-04-22T15:19:09.026 INFO:tasks.workunit.client.0.smithi096.stdout:1/972: creat d6/dc/d8a/d3f/d62/d7d/dd3/f12c x:0 0 0 2024-04-22T15:19:09.027 INFO:tasks.workunit.client.0.smithi096.stdout:6/851: write d3/d28/dd/d11/d2c/fb1 [4734338,20384] 0 2024-04-22T15:19:09.029 INFO:tasks.workunit.client.1.smithi196.stdout:1/751: rename d1/db/d16/d44/dcc to d1/d54/de2/df1/df4 0 2024-04-22T15:19:09.030 INFO:tasks.workunit.client.1.smithi196.stdout:1/752: readlink d1/db/d16/l8a 0 2024-04-22T15:19:09.032 INFO:tasks.workunit.client.1.smithi196.stdout:4/732: dwrite d6/d13/d15/d2c/d62/d78/d8b/fd2 [0,4194304] 0 2024-04-22T15:19:09.032 INFO:tasks.workunit.client.1.smithi196.stdout:7/729: fdatasync d14/d26/f83 0 2024-04-22T15:19:09.032 INFO:tasks.workunit.client.0.smithi096.stdout:4/979: rmdir d5/d7/d2c/d95/dee 39 2024-04-22T15:19:09.035 INFO:tasks.workunit.client.0.smithi096.stdout:5/977: symlink d4/ddb/d3d/d4d/d59/d8a/d121/l156 0 2024-04-22T15:19:09.035 INFO:tasks.workunit.client.1.smithi196.stdout:2/744: fdatasync d1/d23/d27/d2e/f5d 0 2024-04-22T15:19:09.035 INFO:tasks.workunit.client.1.smithi196.stdout:2/745: chown d1/d13/d1f/ddf 7889713 1 2024-04-22T15:19:09.041 INFO:tasks.workunit.client.0.smithi096.stdout:2/988: rmdir d3/db/df/dce/d46/d7e/d84/d12e 39 2024-04-22T15:19:09.042 INFO:tasks.workunit.client.1.smithi196.stdout:9/728: creat db/dd/de/ff2 x:0 0 0 2024-04-22T15:19:09.042 INFO:tasks.workunit.client.1.smithi196.stdout:9/729: truncate db/d11/fea 964069 0 2024-04-22T15:19:09.043 INFO:tasks.workunit.client.1.smithi196.stdout:8/732: stat d2/d8/d49/d6c/lce 0 2024-04-22T15:19:09.044 INFO:tasks.workunit.client.1.smithi196.stdout:8/733: readlink d2/d35/l65 0 2024-04-22T15:19:09.046 INFO:tasks.workunit.client.1.smithi196.stdout:5/757: creat d1/d14/ff4 x:0 0 0 2024-04-22T15:19:09.048 INFO:tasks.workunit.client.1.smithi196.stdout:5/758: chown d1/df1/f54 1 1 2024-04-22T15:19:09.048 INFO:tasks.workunit.client.1.smithi196.stdout:9/730: dwrite db/dd/de/d15/f1e [4194304,4194304] 0 2024-04-22T15:19:09.049 INFO:tasks.workunit.client.1.smithi196.stdout:5/759: write d1/d17/d1a/fe5 [789781,9970] 0 2024-04-22T15:19:09.051 INFO:tasks.workunit.client.1.smithi196.stdout:0/763: symlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/dd0/de3/l10e 0 2024-04-22T15:19:09.053 INFO:tasks.workunit.client.1.smithi196.stdout:0/764: chown d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9 175578 1 2024-04-22T15:19:09.054 INFO:tasks.workunit.client.0.smithi096.stdout:5/978: unlink d4/ddb/l6d 0 2024-04-22T15:19:09.055 INFO:tasks.workunit.client.1.smithi196.stdout:6/764: dread dd/fe [0,4194304] 0 2024-04-22T15:19:09.058 INFO:tasks.workunit.client.0.smithi096.stdout:2/989: read d3/db/df/d24/d58/fd4 [802184,30506] 0 2024-04-22T15:19:09.060 INFO:tasks.workunit.client.0.smithi096.stdout:2/990: chown d3/db/df/d24/d58/d5f/l7d 1574866 1 2024-04-22T15:19:09.063 INFO:tasks.workunit.client.0.smithi096.stdout:9/874: write d1/df/d95/d9d/db5/f12 [42586,81546] 0 2024-04-22T15:19:09.065 INFO:tasks.workunit.client.1.smithi196.stdout:0/765: dwrite d3/d9/d36/d9f/dd1/d65/d7f/d30/fdd [0,4194304] 0 2024-04-22T15:19:09.065 INFO:tasks.workunit.client.0.smithi096.stdout:9/875: write d1/df/d95/d9d/db5/dd/dba/dc2/dc4/fef [1400272,81617] 0 2024-04-22T15:19:09.066 INFO:tasks.workunit.client.0.smithi096.stdout:5/979: dread - d4/ddb/d3d/d4d/d59/db0/fd9 zero size 2024-04-22T15:19:09.067 INFO:tasks.workunit.client.0.smithi096.stdout:8/925: write d4/d1c/d40/da7/db6/f3e [2924293,124868] 0 2024-04-22T15:19:09.069 INFO:tasks.workunit.client.1.smithi196.stdout:2/746: mkdir d1/d23/d27/d2e/d55/d5b/d93/d97/d10e 0 2024-04-22T15:19:09.071 INFO:tasks.workunit.client.0.smithi096.stdout:6/852: creat d3/d28/d4d/d82/f11c x:0 0 0 2024-04-22T15:19:09.073 INFO:tasks.workunit.client.1.smithi196.stdout:8/734: creat d2/d8/d49/d8b/fd6 x:0 0 0 2024-04-22T15:19:09.073 INFO:tasks.workunit.client.1.smithi196.stdout:8/735: fsync d2/d8/d47/d68/fa3 0 2024-04-22T15:19:09.078 INFO:tasks.workunit.client.1.smithi196.stdout:3/766: truncate d0/d15/da7/d4a/d44/f20 598723 0 2024-04-22T15:19:09.078 INFO:tasks.workunit.client.1.smithi196.stdout:9/731: dread db/d11/f6a [4194304,4194304] 0 2024-04-22T15:19:09.079 INFO:tasks.workunit.client.1.smithi196.stdout:6/765: fsync dd/d20/d5c/d5e/f64 0 2024-04-22T15:19:09.082 INFO:tasks.workunit.client.0.smithi096.stdout:1/973: dwrite d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fdf [0,4194304] 0 2024-04-22T15:19:09.084 INFO:tasks.workunit.client.1.smithi196.stdout:3/767: dwrite d0/d15/d31/d55/dcf/d8/f77 [0,4194304] 0 2024-04-22T15:19:09.085 INFO:tasks.workunit.client.0.smithi096.stdout:5/980: rename d4/ddb/d3d/d4d/d59/d5a/cc1 to d4/d2c/d2f/d31/d82/d129/c157 0 2024-04-22T15:19:09.085 INFO:tasks.workunit.client.1.smithi196.stdout:3/768: dread - d0/d15/d31/d55/dcf/d8/f107 zero size 2024-04-22T15:19:09.086 INFO:tasks.workunit.client.1.smithi196.stdout:7/730: chown d14/d15/d91/d9a/l9e 4996581 1 2024-04-22T15:19:09.089 INFO:tasks.workunit.client.1.smithi196.stdout:1/753: link d1/db/dc/de/d43/d5e/d7a/fcf d1/db/dc/de/d43/ff5 0 2024-04-22T15:19:09.091 INFO:tasks.workunit.client.0.smithi096.stdout:8/926: mkdir d4/d10/d25/d38/d8f/d9c/d138 0 2024-04-22T15:19:09.094 INFO:tasks.workunit.client.0.smithi096.stdout:1/974: fdatasync d6/dc/d8a/d3f/fda 0 2024-04-22T15:19:09.099 INFO:tasks.workunit.client.0.smithi096.stdout:4/980: dwrite d5/d5b/f80 [0,4194304] 0 2024-04-22T15:19:09.100 INFO:tasks.workunit.client.0.smithi096.stdout:3/964: sync 2024-04-22T15:19:09.101 INFO:tasks.workunit.client.0.smithi096.stdout:3/965: chown da/db/de/ddc/d68/db0/ce0 18754 1 2024-04-22T15:19:09.102 INFO:tasks.workunit.client.0.smithi096.stdout:6/853: mknod d3/d28/d4d/c11d 0 2024-04-22T15:19:09.104 INFO:tasks.workunit.client.0.smithi096.stdout:6/854: write d3/d3a/d75/d83/df7/f111 [53277,64494] 0 2024-04-22T15:19:09.105 INFO:tasks.workunit.client.0.smithi096.stdout:5/981: mknod d4/ddb/d3d/d35/d57/c158 0 2024-04-22T15:19:09.106 INFO:tasks.workunit.client.0.smithi096.stdout:2/991: write d3/db/df/d98/db3/fbb [4837703,61938] 0 2024-04-22T15:19:09.107 INFO:tasks.workunit.client.0.smithi096.stdout:5/982: stat d4/ddb/d3d/d35/d57/dbe/d113/d11c 0 2024-04-22T15:19:09.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.111+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 --> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f976010a140 con 0x7f974c06dc90 2024-04-22T15:19:09.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.113+0000 7f9756ffd700 1 -- 172.21.15.96:0/1929823511 <== mgr.24359 v2:172.21.15.196:6828/2093434004 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f976010a140 con 0x7f974c06dc90 2024-04-22T15:19:09.116 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:19:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:19:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:19:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:19:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [], 2024-04-22T15:19:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "", 2024-04-22T15:19:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mgr daemons", 2024-04-22T15:19:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:19:09.117 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.116+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f974c06dc90 msgr2=0x7f974c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.116+0000 7f9767fb5700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f974c06dc90 0x7f974c070150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f975c005950 tx=0x7f975c009260 comp rx=0 tx=0).stop 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.116+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760101f70 msgr2=0x7f97600fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.116+0000 7f9767fb5700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760101f70 0x7f97600fff20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9750006010 tx=0x7f9750005f00 comp rx=0 tx=0).stop 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.116+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 shutdown_connections 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.116+0000 7f9767fb5700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:6828/2093434004,v1:172.21.15.196:6829/2093434004] conn(0x7f974c06dc90 0x7f974c070150 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.116+0000 7f9767fb5700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97600feef0 0x7f9760101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.116+0000 7f9767fb5700 1 --2- 172.21.15.96:0/1929823511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760101f70 0x7f97600fff20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.116+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 >> 172.21.15.96:0/1929823511 conn(0x7f97600fa830 msgr2=0x7f9760105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.117+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 shutdown_connections 2024-04-22T15:19:09.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:09.117+0000 7f9767fb5700 1 -- 172.21.15.96:0/1929823511 wait complete. 2024-04-22T15:19:09.121 INFO:tasks.workunit.client.0.smithi096.stdout:9/876: dwrite d1/df/d95/d9d/db5/d43/d57/d5b/da1/fd3 [0,4194304] 0 2024-04-22T15:19:09.122 INFO:tasks.workunit.client.1.smithi196.stdout:3/769: mknod d0/d15/d31/d55/dcf/d6f/c108 0 2024-04-22T15:19:09.124 INFO:tasks.workunit.client.0.smithi096.stdout:3/966: rmdir da/db/dc 39 2024-04-22T15:19:09.127 INFO:tasks.workunit.client.1.smithi196.stdout:1/754: mkdir d1/db/d16/d8e/ddd/df6 0 2024-04-22T15:19:09.129 INFO:tasks.workunit.client.0.smithi096.stdout:2/992: rename d3/db/df/dce/ld6 to d3/d13/db4/l141 0 2024-04-22T15:19:09.130 INFO:tasks.workunit.client.1.smithi196.stdout:3/770: dwrite d0/d15/da7/d85/f91 [0,4194304] 0 2024-04-22T15:19:09.130 INFO:tasks.workunit.client.1.smithi196.stdout:5/760: sync 2024-04-22T15:19:09.131 INFO:tasks.workunit.client.1.smithi196.stdout:0/766: mkdir d3/d9/d36/d9f/dd1/d65/d7f/d10f 0 2024-04-22T15:19:09.131 INFO:tasks.workunit.client.1.smithi196.stdout:5/761: fsync d1/df1/fdd 0 2024-04-22T15:19:09.132 INFO:tasks.workunit.client.1.smithi196.stdout:0/767: chown d3/de/d14/d86/dd5 505209 1 2024-04-22T15:19:09.132 INFO:tasks.workunit.client.0.smithi096.stdout:5/983: creat d4/ddb/d3d/d35/d57/dbe/d113/d14a/f159 x:0 0 0 2024-04-22T15:19:09.133 INFO:tasks.workunit.client.0.smithi096.stdout:5/984: chown d4/ddb/d3d/d35/fec 1853 1 2024-04-22T15:19:09.135 INFO:tasks.workunit.client.0.smithi096.stdout:1/975: fdatasync d6/dc/d8a/d29/f79 0 2024-04-22T15:19:09.136 INFO:tasks.workunit.client.0.smithi096.stdout:1/976: chown d6/dc/d8a/d3f/d54/d32/d8c 127001 1 2024-04-22T15:19:09.143 INFO:tasks.workunit.client.0.smithi096.stdout:8/927: dread d4/d1c/d40/d46/d58/fb5 [0,4194304] 0 2024-04-22T15:19:09.145 INFO:tasks.workunit.client.0.smithi096.stdout:4/981: mkdir d5/d7/d2c/d95/d150 0 2024-04-22T15:19:09.145 INFO:tasks.workunit.client.0.smithi096.stdout:5/985: dwrite d4/ddb/d3d/d35/fc9 [0,4194304] 0 2024-04-22T15:19:09.146 INFO:tasks.workunit.client.1.smithi196.stdout:6/766: mkdir dd/d1f/d4e/d53/d55/d6d/dde 0 2024-04-22T15:19:09.149 INFO:tasks.workunit.client.0.smithi096.stdout:1/977: dwrite d6/dc/dbd/fc9 [0,4194304] 0 2024-04-22T15:19:09.150 INFO:tasks.workunit.client.1.smithi196.stdout:4/733: getdents d6/d14/d18/d26/d32/d67 0 2024-04-22T15:19:09.153 INFO:tasks.workunit.client.0.smithi096.stdout:3/967: write da/db/dc/d28/d69/f10f [11630,73791] 0 2024-04-22T15:19:09.154 INFO:tasks.workunit.client.1.smithi196.stdout:7/731: creat d14/d4d/da3/de9/ff3 x:0 0 0 2024-04-22T15:19:09.157 INFO:tasks.workunit.client.0.smithi096.stdout:6/855: symlink d3/d28/dd/d11/l11e 0 2024-04-22T15:19:09.168 INFO:tasks.workunit.client.1.smithi196.stdout:0/768: mknod d3/de/d1c/d27/ddc/c110 0 2024-04-22T15:19:09.169 INFO:tasks.workunit.client.1.smithi196.stdout:2/747: creat d1/d23/d27/d2e/d55/d5b/d93/f10f x:0 0 0 2024-04-22T15:19:09.170 INFO:tasks.workunit.client.1.smithi196.stdout:3/771: dread d0/d15/da7/d4a/d44/d8c/de9/fed [0,4194304] 0 2024-04-22T15:19:09.174 INFO:tasks.workunit.client.0.smithi096.stdout:4/982: rmdir d5/d7 39 2024-04-22T15:19:09.176 INFO:tasks.workunit.client.0.smithi096.stdout:8/928: mkdir d4/d10/d25/d4c/db4/dd0/d139 0 2024-04-22T15:19:09.179 INFO:tasks.workunit.client.0.smithi096.stdout:9/877: write d1/df/d95/d9d/db5/f52 [4858066,54592] 0 2024-04-22T15:19:09.181 INFO:tasks.workunit.client.0.smithi096.stdout:1/978: mknod d6/dc/d8a/d3f/d62/d7d/ddb/d9b/c12d 0 2024-04-22T15:19:09.182 INFO:tasks.workunit.client.0.smithi096.stdout:1/979: chown d6/dc/d8a/d3f/d54/d32/d3b/d94/lb0 50105845 1 2024-04-22T15:19:09.183 INFO:tasks.workunit.client.0.smithi096.stdout:3/968: fdatasync da/db/de/ddc/d26/db2/fc2 0 2024-04-22T15:19:09.186 INFO:tasks.workunit.client.1.smithi196.stdout:7/732: mkdir d14/d26/d2a/d2b/dc7/df4 0 2024-04-22T15:19:09.187 INFO:tasks.workunit.client.0.smithi096.stdout:2/993: symlink d3/db/df/d98/l142 0 2024-04-22T15:19:09.188 INFO:tasks.workunit.client.0.smithi096.stdout:2/994: chown d3/db/df/d24/d58/d5f/dfe/d115/d2f 42611089 1 2024-04-22T15:19:09.189 INFO:tasks.workunit.client.1.smithi196.stdout:1/755: mkdir d1/db/d16/d1e/df7 0 2024-04-22T15:19:09.191 INFO:tasks.workunit.client.0.smithi096.stdout:5/986: mkdir d4/d1f/d15a 0 2024-04-22T15:19:09.194 INFO:tasks.workunit.client.0.smithi096.stdout:8/929: creat d4/d113/f13a x:0 0 0 2024-04-22T15:19:09.203 INFO:tasks.workunit.client.0.smithi096.stdout:5/987: fdatasync f2 0 2024-04-22T15:19:09.206 INFO:tasks.workunit.client.1.smithi196.stdout:4/734: write d6/d14/d18/d40/d85/fbf [7775790,53706] 0 2024-04-22T15:19:09.207 INFO:tasks.workunit.client.0.smithi096.stdout:6/856: write d3/d28/d4d/d82/fb9 [267369,84274] 0 2024-04-22T15:19:09.211 INFO:tasks.workunit.client.0.smithi096.stdout:9/878: rename d1/df/c89 to d1/df/d95/d9d/db5/dd/dba/dc2/c127 0 2024-04-22T15:19:09.213 INFO:tasks.workunit.client.0.smithi096.stdout:3/969: symlink da/db/de/d19/l13d 0 2024-04-22T15:19:09.220 INFO:tasks.workunit.client.0.smithi096.stdout:5/988: dread d4/ddb/f62 [0,4194304] 0 2024-04-22T15:19:09.222 INFO:tasks.workunit.client.0.smithi096.stdout:1/980: rename d6/dc/d8a/d3f/d62/d7d/ddb/d9b/l111 to d6/dc/d8a/d29/l12e 0 2024-04-22T15:19:09.225 INFO:tasks.workunit.client.0.smithi096.stdout:2/995: creat d3/db/df/dce/d46/d7e/d84/f143 x:0 0 0 2024-04-22T15:19:09.226 INFO:tasks.workunit.client.0.smithi096.stdout:6/857: link d3/d28/dd/d11/d2c/d3b/f79 d3/d28/dd/d11/d119/f11f 0 2024-04-22T15:19:09.228 INFO:tasks.workunit.client.0.smithi096.stdout:5/989: rmdir d4/ddb/d3d/d4d/d59/d5a/d5c 39 2024-04-22T15:19:09.229 INFO:tasks.workunit.client.0.smithi096.stdout:8/930: dwrite d4/d71/fff [0,4194304] 0 2024-04-22T15:19:09.231 INFO:tasks.workunit.client.0.smithi096.stdout:4/983: rename d5/d7/d1a/d103 to d5/d7/d2c/d3d/d4f/d63/d143/d151 0 2024-04-22T15:19:09.236 INFO:tasks.workunit.client.0.smithi096.stdout:6/858: mknod d3/d28/daf/c120 0 2024-04-22T15:19:09.242 INFO:tasks.workunit.client.0.smithi096.stdout:5/990: symlink d4/ddb/d3d/d4d/d59/d98/dee/l15b 0 2024-04-22T15:19:09.243 INFO:tasks.workunit.client.0.smithi096.stdout:3/970: creat da/db/de/f13e x:0 0 0 2024-04-22T15:19:09.244 INFO:tasks.workunit.client.0.smithi096.stdout:6/859: dread d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/fc0 [0,4194304] 0 2024-04-22T15:19:09.246 INFO:tasks.workunit.client.0.smithi096.stdout:1/981: creat d6/dc/d8a/d3f/d54/d32/d3b/def/f12f x:0 0 0 2024-04-22T15:19:09.247 INFO:tasks.workunit.client.0.smithi096.stdout:4/984: rmdir d5/d7 39 2024-04-22T15:19:09.250 INFO:tasks.workunit.client.0.smithi096.stdout:6/860: dwrite d3/d3a/d75/f105 [4194304,4194304] 0 2024-04-22T15:19:09.251 INFO:tasks.workunit.client.0.smithi096.stdout:9/879: rename d1/df/d95/d9d/db5/dd/dba/dc2/f126 to d1/df/d95/d9d/db5/dd/f128 0 2024-04-22T15:19:09.253 INFO:tasks.workunit.client.0.smithi096.stdout:6/861: dread d3/d28/d4d/d82/ffb [0,4194304] 0 2024-04-22T15:19:09.254 INFO:tasks.workunit.client.0.smithi096.stdout:8/931: truncate d4/d1c/d62/f6b 331475 0 2024-04-22T15:19:09.267 INFO:tasks.workunit.client.0.smithi096.stdout:8/932: dread d4/d1c/d40/d46/d58/f7a [0,4194304] 0 2024-04-22T15:19:09.267 INFO:tasks.workunit.client.0.smithi096.stdout:8/933: chown d4/d1c/lbd 203089026 1 2024-04-22T15:19:09.273 INFO:tasks.workunit.client.1.smithi196.stdout:7/733: sync 2024-04-22T15:19:09.274 INFO:tasks.workunit.client.1.smithi196.stdout:7/734: dread - d14/d15/d91/fe7 zero size 2024-04-22T15:19:09.282 INFO:tasks.workunit.client.1.smithi196.stdout:3/772: rename d0/d15/d31/d55/dcf/d8/l30 to d0/d15/da7/l109 0 2024-04-22T15:19:09.283 INFO:tasks.workunit.client.1.smithi196.stdout:0/769: truncate d3/d9/d36/d9f/dd1/d65/d7f/d51/f53 1098118 0 2024-04-22T15:19:09.283 INFO:tasks.workunit.client.1.smithi196.stdout:8/736: getdents d2/d5/d1a/dcc 0 2024-04-22T15:19:09.285 INFO:tasks.workunit.client.1.smithi196.stdout:9/732: getdents db/dd/de/dd7/def 0 2024-04-22T15:19:09.286 INFO:tasks.workunit.client.1.smithi196.stdout:6/767: mkdir dd/ddf 0 2024-04-22T15:19:09.287 INFO:tasks.workunit.client.1.smithi196.stdout:1/756: creat d1/db/d16/d1e/d49/d90/ff8 x:0 0 0 2024-04-22T15:19:09.289 INFO:tasks.workunit.client.1.smithi196.stdout:4/735: dread d6/d13/d15/d2c/d62/d78/f7b [0,4194304] 0 2024-04-22T15:19:09.289 INFO:tasks.workunit.client.1.smithi196.stdout:7/735: creat d14/d26/d2a/d72/ff5 x:0 0 0 2024-04-22T15:19:09.290 INFO:tasks.workunit.client.1.smithi196.stdout:7/736: fsync d14/d26/d95/fd3 0 2024-04-22T15:19:09.291 INFO:tasks.workunit.client.1.smithi196.stdout:2/748: mkdir d1/d23/d27/d2e/d55/d5b/d93/d110 0 2024-04-22T15:19:09.296 INFO:tasks.workunit.client.1.smithi196.stdout:9/733: unlink db/dd/de/dd7/def/f48 0 2024-04-22T15:19:09.301 INFO:tasks.workunit.client.1.smithi196.stdout:5/762: getdents d1/d17 0 2024-04-22T15:19:09.302 INFO:tasks.workunit.client.1.smithi196.stdout:3/773: dread d0/d15/d31/d55/dcf/f38 [0,4194304] 0 2024-04-22T15:19:09.308 INFO:tasks.workunit.client.1.smithi196.stdout:0/770: creat d3/de/d1c/df1/f111 x:0 0 0 2024-04-22T15:19:09.309 INFO:tasks.workunit.client.1.smithi196.stdout:0/771: dread - d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/fe5 zero size 2024-04-22T15:19:09.325 INFO:tasks.workunit.client.0.smithi096.stdout:5/991: unlink d4/ddb/d3d/d4d/d59/db0/d126/c152 0 2024-04-22T15:19:09.327 INFO:tasks.workunit.client.0.smithi096.stdout:3/971: mkdir da/db/d9a/d132/d13f 0 2024-04-22T15:19:09.331 INFO:tasks.workunit.client.1.smithi196.stdout:0/772: creat d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/f112 x:0 0 0 2024-04-22T15:19:09.336 INFO:tasks.workunit.client.1.smithi196.stdout:2/749: truncate d1/d13/d25/d2a/f3f 301625 0 2024-04-22T15:19:09.339 INFO:tasks.workunit.client.0.smithi096.stdout:2/996: rename d3/db/df/d24/dc8/d103/d110/l11f to d3/db/df/dce/d46/d7e/d84/d12e/l144 0 2024-04-22T15:19:09.340 INFO:tasks.workunit.client.1.smithi196.stdout:5/763: mknod d1/d17/d1a/d2f/d65/cf5 0 2024-04-22T15:19:09.343 INFO:tasks.workunit.client.0.smithi096.stdout:9/880: mknod d1/d25/db2/d98/d9b/c129 0 2024-04-22T15:19:09.344 INFO:tasks.workunit.client.1.smithi196.stdout:5/764: dread d1/d17/d1a/fe5 [0,4194304] 0 2024-04-22T15:19:09.344 INFO:tasks.workunit.client.1.smithi196.stdout:0/773: rmdir d3/de 39 2024-04-22T15:19:09.346 INFO:tasks.workunit.client.0.smithi096.stdout:2/997: dwrite d3/db/d30/f11c [0,4194304] 0 2024-04-22T15:19:09.349 INFO:tasks.workunit.client.0.smithi096.stdout:8/934: creat d4/d1c/d40/d48/f13b x:0 0 0 2024-04-22T15:19:09.350 INFO:tasks.workunit.client.1.smithi196.stdout:8/737: getdents d2/d6a 0 2024-04-22T15:19:09.361 INFO:tasks.workunit.client.0.smithi096.stdout:4/985: rename d5/d7/d2c/d3d/d4f/d88/fe1 to d5/d7/dcb/d12a/d14c/f152 0 2024-04-22T15:19:09.362 INFO:tasks.workunit.client.1.smithi196.stdout:5/765: creat d1/d17/d1a/d2f/d65/ff6 x:0 0 0 2024-04-22T15:19:09.368 INFO:tasks.workunit.client.0.smithi096.stdout:9/881: fdatasync d1/d4/f42 0 2024-04-22T15:19:09.370 INFO:tasks.workunit.client.1.smithi196.stdout:3/774: getdents d0/d15/d4d/d87 0 2024-04-22T15:19:09.373 INFO:tasks.workunit.client.1.smithi196.stdout:0/774: dread d3/d9/f73 [0,4194304] 0 2024-04-22T15:19:09.376 INFO:tasks.workunit.client.0.smithi096.stdout:8/935: fdatasync d4/d1c/d40/d46/d58/fb5 0 2024-04-22T15:19:09.377 INFO:tasks.workunit.client.0.smithi096.stdout:2/998: dread d3/db/df/d24/d58/d5f/dfe/d115/f56 [0,4194304] 0 2024-04-22T15:19:09.378 INFO:tasks.workunit.client.1.smithi196.stdout:0/775: dwrite d3/d6/ff9 [0,4194304] 0 2024-04-22T15:19:09.379 INFO:tasks.workunit.client.1.smithi196.stdout:3/775: rename d0/d15/d31/dd0/d51 to d0/d15/d31/d55/dcf/d6f/d10a 0 2024-04-22T15:19:09.382 INFO:tasks.workunit.client.1.smithi196.stdout:1/757: sync 2024-04-22T15:19:09.384 INFO:tasks.workunit.client.0.smithi096.stdout:1/982: link d6/dc/d8a/l17 d6/dc/d8a/d3f/d54/d32/dcd/l130 0 2024-04-22T15:19:09.386 INFO:tasks.workunit.client.1.smithi196.stdout:0/776: creat d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dba/f113 x:0 0 0 2024-04-22T15:19:09.387 INFO:tasks.workunit.client.1.smithi196.stdout:1/758: mkdir d1/db/dc/d93/df9 0 2024-04-22T15:19:09.389 INFO:tasks.workunit.client.1.smithi196.stdout:5/766: dread d1/d17/d7f/fe0 [0,4194304] 0 2024-04-22T15:19:09.389 INFO:tasks.workunit.client.1.smithi196.stdout:3/776: dwrite d0/d15/d31/dd0/fae [0,4194304] 0 2024-04-22T15:19:09.389 INFO:tasks.workunit.client.1.smithi196.stdout:3/777: chown d0 526434074 1 2024-04-22T15:19:09.389 INFO:tasks.workunit.client.1.smithi196.stdout:1/759: write d1/db/ff [3954367,75524] 0 2024-04-22T15:19:09.396 INFO:tasks.workunit.client.0.smithi096.stdout:4/986: dwrite d5/d7/d2c/d3d/d4f/d63/d143/d151/f10d [0,4194304] 0 2024-04-22T15:19:09.398 INFO:tasks.workunit.client.0.smithi096.stdout:9/882: creat d1/d85/f12a x:0 0 0 2024-04-22T15:19:09.399 INFO:tasks.workunit.client.1.smithi196.stdout:5/767: fdatasync d1/d9/f75 0 2024-04-22T15:19:09.401 INFO:tasks.workunit.client.0.smithi096.stdout:6/862: creat d3/d28/d4d/f121 x:0 0 0 2024-04-22T15:19:09.402 INFO:tasks.workunit.client.1.smithi196.stdout:6/768: write dd/d2b/f77 [1666602,104203] 0 2024-04-22T15:19:09.403 INFO:tasks.workunit.client.1.smithi196.stdout:3/778: readlink d0/d15/d31/l56 0 2024-04-22T15:19:09.406 INFO:tasks.workunit.client.1.smithi196.stdout:1/760: creat d1/db/dc/de/d96/ffa x:0 0 0 2024-04-22T15:19:09.407 INFO:tasks.workunit.client.1.smithi196.stdout:1/761: write d1/db/d16/d44/fb8 [583003,89570] 0 2024-04-22T15:19:09.409 INFO:tasks.workunit.client.1.smithi196.stdout:4/736: write d6/d14/d18/d26/d32/f7d [1011071,43668] 0 2024-04-22T15:19:09.409 INFO:tasks.workunit.client.1.smithi196.stdout:4/737: fdatasync d6/d14/d18/d26/d32/d67/d9a/f9e 0 2024-04-22T15:19:09.411 INFO:tasks.workunit.client.0.smithi096.stdout:2/999: symlink d3/db/df/dce/d46/d59/l145 0 2024-04-22T15:19:09.412 INFO:tasks.workunit.client.1.smithi196.stdout:8/738: sync 2024-04-22T15:19:09.412 INFO:tasks.workunit.client.1.smithi196.stdout:7/737: write d14/d26/d28/d29/fb3 [3550796,75674] 0 2024-04-22T15:19:09.413 INFO:tasks.workunit.client.1.smithi196.stdout:4/738: dread d6/d13/d15/d2c/d62/d78/d8b/fd2 [0,4194304] 0 2024-04-22T15:19:09.414 INFO:tasks.workunit.client.1.smithi196.stdout:4/739: fsync d6/d14/d18/d26/f6a 0 2024-04-22T15:19:09.416 INFO:tasks.workunit.client.1.smithi196.stdout:5/768: rmdir d1/d2 39 2024-04-22T15:19:09.418 INFO:tasks.workunit.client.0.smithi096.stdout:1/983: symlink d6/d97/l131 0 2024-04-22T15:19:09.424 INFO:tasks.workunit.client.1.smithi196.stdout:3/779: unlink d0/d15/d31/d55/dcf/d8/f77 0 2024-04-22T15:19:09.426 INFO:tasks.workunit.client.1.smithi196.stdout:1/762: creat d1/db/d16/ffb x:0 0 0 2024-04-22T15:19:09.427 INFO:tasks.workunit.client.1.smithi196.stdout:1/763: write d1/db/d16/d1e/d49/f42 [197931,91663] 0 2024-04-22T15:19:09.427 INFO:tasks.workunit.client.1.smithi196.stdout:1/764: stat d1/db/dc/c80 0 2024-04-22T15:19:09.431 INFO:tasks.workunit.client.0.smithi096.stdout:3/972: creat da/db/de/d19/d2d/f140 x:0 0 0 2024-04-22T15:19:09.433 INFO:tasks.workunit.client.1.smithi196.stdout:7/738: dread d14/d15/f1a [0,4194304] 0 2024-04-22T15:19:09.434 INFO:tasks.workunit.client.0.smithi096.stdout:1/984: rmdir d6/d97 39 2024-04-22T15:19:09.435 INFO:tasks.workunit.client.0.smithi096.stdout:1/985: chown d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fa1 1 1 2024-04-22T15:19:09.436 INFO:tasks.workunit.client.1.smithi196.stdout:4/740: symlink d6/d14/d18/d26/d32/d67/d9a/lee 0 2024-04-22T15:19:09.437 INFO:tasks.workunit.client.1.smithi196.stdout:4/741: dread - d6/d14/d18/d26/d32/da3/db7/dc9/fd9 zero size 2024-04-22T15:19:09.438 INFO:tasks.workunit.client.0.smithi096.stdout:4/987: symlink d5/d7/l153 0 2024-04-22T15:19:09.439 INFO:tasks.workunit.client.1.smithi196.stdout:2/750: write d1/d13/d41/fa8 [3184735,64976] 0 2024-04-22T15:19:09.440 INFO:tasks.workunit.client.1.smithi196.stdout:9/734: mknod db/dd/de/dd7/def/cf3 0 2024-04-22T15:19:09.441 INFO:tasks.workunit.client.0.smithi096.stdout:9/883: mkdir d1/df/d95/d9d/db5/dd/dba/dc2/d120/d12b 0 2024-04-22T15:19:09.446 INFO:tasks.workunit.client.0.smithi096.stdout:5/992: dwrite d4/ddb/d3d/d35/fb8 [0,4194304] 0 2024-04-22T15:19:09.447 INFO:tasks.workunit.client.1.smithi196.stdout:3/780: fdatasync d0/d15/f92 0 2024-04-22T15:19:09.453 INFO:tasks.workunit.client.1.smithi196.stdout:1/765: mkdir d1/db/d16/d6d/de6/dfc 0 2024-04-22T15:19:09.459 INFO:tasks.workunit.client.0.smithi096.stdout:9/884: mkdir d1/df/d95/d123/d12c 0 2024-04-22T15:19:09.461 INFO:tasks.workunit.client.0.smithi096.stdout:6/863: rmdir d3/d28/d4d/d82/dfe 0 2024-04-22T15:19:09.466 INFO:tasks.workunit.client.1.smithi196.stdout:7/739: creat d14/d26/d5e/ff6 x:0 0 0 2024-04-22T15:19:09.468 INFO:tasks.workunit.client.1.smithi196.stdout:5/769: symlink d1/d14/d6a/lf7 0 2024-04-22T15:19:09.470 INFO:tasks.workunit.client.1.smithi196.stdout:0/777: write d3/d6/dc/f92 [69872,42976] 0 2024-04-22T15:19:09.473 INFO:tasks.workunit.client.1.smithi196.stdout:4/742: rename d6/d13/d15/d2c/d62/d78/d8b/f8e to d6/d14/d18/d26/d32/da3/db7/fef 0 2024-04-22T15:19:09.474 INFO:tasks.workunit.client.1.smithi196.stdout:4/743: dread - d6/d13/d15/d2c/d8c/fcf zero size 2024-04-22T15:19:09.474 INFO:tasks.workunit.client.1.smithi196.stdout:2/751: readlink d1/d23/d27/d2e/d55/d5b/d93/l109 0 2024-04-22T15:19:09.475 INFO:tasks.workunit.client.1.smithi196.stdout:2/752: chown d1/d23/d27/d89/fb0 10539456 1 2024-04-22T15:19:09.475 INFO:tasks.workunit.client.0.smithi096.stdout:6/864: mkdir d3/d28/dc9/d122 0 2024-04-22T15:19:09.477 INFO:tasks.workunit.client.0.smithi096.stdout:1/986: link d6/dc/d8a/d3f/d54/d32/d3b/d3d/l61 d6/dc/d8a/l132 0 2024-04-22T15:19:09.478 INFO:tasks.workunit.client.0.smithi096.stdout:8/936: write d4/d10/d25/d4c/db4/f117 [1168441,112361] 0 2024-04-22T15:19:09.479 INFO:tasks.workunit.client.1.smithi196.stdout:3/781: write d0/d15/d31/d55/dcf/f7d [1538886,75239] 0 2024-04-22T15:19:09.480 INFO:tasks.workunit.client.1.smithi196.stdout:3/782: fsync d0/d15/d31/d55/d6e/fa1 0 2024-04-22T15:19:09.480 INFO:tasks.workunit.client.0.smithi096.stdout:6/865: mknod d3/d28/dd/d11/d2c/d6f/dd4/ddd/c123 0 2024-04-22T15:19:09.481 INFO:tasks.workunit.client.1.smithi196.stdout:1/766: mkdir d1/db/d97/dd7/dfd 0 2024-04-22T15:19:09.482 INFO:tasks.workunit.client.0.smithi096.stdout:6/866: write d3/d28/d4d/d82/d9e/f92 [550831,86004] 0 2024-04-22T15:19:09.486 INFO:tasks.workunit.client.1.smithi196.stdout:3/783: dwrite d0/d15/d31/d55/ff9 [0,4194304] 0 2024-04-22T15:19:09.489 INFO:tasks.workunit.client.1.smithi196.stdout:6/769: dwrite f9 [0,4194304] 0 2024-04-22T15:19:09.490 INFO:tasks.workunit.client.1.smithi196.stdout:8/739: write d2/d8/d47/d68/fb3 [661432,108287] 0 2024-04-22T15:19:09.497 INFO:tasks.workunit.client.1.smithi196.stdout:2/753: mknod d1/d13/d25/c111 0 2024-04-22T15:19:09.498 INFO:tasks.workunit.client.1.smithi196.stdout:2/754: read - d1/d13/d25/fd0 zero size 2024-04-22T15:19:09.498 INFO:tasks.workunit.client.1.smithi196.stdout:4/744: read d6/d10/f63 [1202892,5276] 0 2024-04-22T15:19:09.504 INFO:tasks.workunit.client.1.smithi196.stdout:3/784: symlink d0/d15/d31/d55/dcf/d6f/db5/dd7/l10b 0 2024-04-22T15:19:09.506 INFO:tasks.workunit.client.1.smithi196.stdout:7/740: unlink d14/d46/feb 0 2024-04-22T15:19:09.507 INFO:tasks.workunit.client.1.smithi196.stdout:7/741: write d14/d15/d91/fe7 [279398,59596] 0 2024-04-22T15:19:09.512 INFO:tasks.workunit.client.1.smithi196.stdout:5/770: rename d1/df1/d7d/l7e to d1/d4/lf8 0 2024-04-22T15:19:09.513 INFO:tasks.workunit.client.1.smithi196.stdout:5/771: dread - d1/d17/d1a/d2f/d65/ff6 zero size 2024-04-22T15:19:09.514 INFO:tasks.workunit.client.1.smithi196.stdout:0/778: truncate d3/de/d1c/d27/f38 2809683 0 2024-04-22T15:19:09.520 INFO:tasks.workunit.client.0.smithi096.stdout:5/993: write d4/ddb/d25/ff6 [390272,81648] 0 2024-04-22T15:19:09.521 INFO:tasks.workunit.client.0.smithi096.stdout:3/973: write da/db/de/ddc/d26/ffe [1734638,88777] 0 2024-04-22T15:19:09.524 INFO:tasks.workunit.client.1.smithi196.stdout:3/785: rmdir d0/d15/d31/dd0/de2 39 2024-04-22T15:19:09.529 INFO:tasks.workunit.client.1.smithi196.stdout:5/772: dread d1/d9/f20 [0,4194304] 0 2024-04-22T15:19:09.533 INFO:tasks.workunit.client.1.smithi196.stdout:7/742: rename d14/d25/f7a to d14/d25/d31/ff7 0 2024-04-22T15:19:09.533 INFO:tasks.workunit.client.1.smithi196.stdout:7/743: chown d14/d26/d95/l97 628619659 1 2024-04-22T15:19:09.537 INFO:tasks.workunit.client.1.smithi196.stdout:2/755: symlink d1/d23/d27/d2e/d55/d5b/d93/l112 0 2024-04-22T15:19:09.539 INFO:tasks.workunit.client.1.smithi196.stdout:3/786: truncate d0/d15/da7/d4a/d44/d8c/faa 5146401 0 2024-04-22T15:19:09.540 INFO:tasks.workunit.client.1.smithi196.stdout:1/767: dwrite d1/db/d97/fd3 [4194304,4194304] 0 2024-04-22T15:19:09.540 INFO:tasks.workunit.client.1.smithi196.stdout:5/773: read d1/d4/f9c [1780955,5559] 0 2024-04-22T15:19:09.541 INFO:tasks.workunit.client.1.smithi196.stdout:3/787: dread - d0/d15/da7/d4a/d44/d8c/ff0 zero size 2024-04-22T15:19:09.541 INFO:tasks.workunit.client.1.smithi196.stdout:5/774: readlink d1/d4/d92/db5/lc5 0 2024-04-22T15:19:09.542 INFO:tasks.workunit.client.1.smithi196.stdout:3/788: stat d0/d15/d31/dd0/l7c 0 2024-04-22T15:19:09.543 INFO:tasks.workunit.client.1.smithi196.stdout:0/779: mkdir d3/d9/d36/d114 0 2024-04-22T15:19:09.545 INFO:tasks.workunit.client.1.smithi196.stdout:9/735: getdents db/dd/de/dd7/def/d7a/d7b/d81 0 2024-04-22T15:19:09.549 INFO:tasks.workunit.client.1.smithi196.stdout:6/770: dwrite dd/d1f/d38/f9d [0,4194304] 0 2024-04-22T15:19:09.550 INFO:tasks.workunit.client.1.smithi196.stdout:2/756: dread d1/d23/d27/d2e/f85 [0,4194304] 0 2024-04-22T15:19:09.551 INFO:tasks.workunit.client.1.smithi196.stdout:8/740: dwrite d2/d5/faf [0,4194304] 0 2024-04-22T15:19:09.552 INFO:tasks.workunit.client.1.smithi196.stdout:7/744: link d14/d26/d5a/d5f/faa d14/d26/d2a/d72/db1/ff8 0 2024-04-22T15:19:09.553 INFO:tasks.workunit.client.1.smithi196.stdout:7/745: readlink d14/d25/d31/l53 0 2024-04-22T15:19:09.553 INFO:tasks.workunit.client.1.smithi196.stdout:4/745: write d6/f42 [303130,25705] 0 2024-04-22T15:19:09.556 INFO:tasks.workunit.client.1.smithi196.stdout:7/746: chown d14/d26/fd2 3122433 1 2024-04-22T15:19:09.559 INFO:tasks.workunit.client.1.smithi196.stdout:3/789: unlink d0/d15/d31/d55/dcf/d6f/d10a/d7f/lb9 0 2024-04-22T15:19:09.560 INFO:tasks.workunit.client.1.smithi196.stdout:0/780: creat d3/d6/dc/d8c/d8d/d98/dbd/f115 x:0 0 0 2024-04-22T15:19:09.561 INFO:tasks.workunit.client.1.smithi196.stdout:8/741: dwrite d2/d6a/d7b/fc5 [0,4194304] 0 2024-04-22T15:19:09.563 INFO:tasks.workunit.client.1.smithi196.stdout:9/736: write db/dd/de/d15/d31/d99/fa1 [3456319,24093] 0 2024-04-22T15:19:09.564 INFO:tasks.workunit.client.1.smithi196.stdout:4/746: dread d6/d13/d15/d2c/d62/d78/fd1 [0,4194304] 0 2024-04-22T15:19:09.566 INFO:tasks.workunit.client.1.smithi196.stdout:2/757: unlink d1/d23/d27/d2e/d55/l4a 0 2024-04-22T15:19:09.568 INFO:tasks.workunit.client.1.smithi196.stdout:5/775: mkdir d1/d2/dbb/df9 0 2024-04-22T15:19:09.570 INFO:tasks.workunit.client.1.smithi196.stdout:7/747: symlink d14/d26/d2a/lf9 0 2024-04-22T15:19:09.572 INFO:tasks.workunit.client.1.smithi196.stdout:0/781: chown d3/de/d14/d86/f94 30 1 2024-04-22T15:19:09.574 INFO:tasks.workunit.client.1.smithi196.stdout:8/742: creat d2/d8/d49/d8b/fd7 x:0 0 0 2024-04-22T15:19:09.575 INFO:tasks.workunit.client.1.smithi196.stdout:6/771: mknod dd/ce0 0 2024-04-22T15:19:09.577 INFO:tasks.workunit.client.1.smithi196.stdout:6/772: write dd/d1f/d4e/d53/d55/f8b [2863726,62801] 0 2024-04-22T15:19:09.577 INFO:tasks.workunit.client.1.smithi196.stdout:9/737: mknod db/dd/de/d15/d31/d64/dab/cf4 0 2024-04-22T15:19:09.578 INFO:tasks.workunit.client.1.smithi196.stdout:6/773: fsync dd/d1f/d37/d7b/fb4 0 2024-04-22T15:19:09.579 INFO:tasks.workunit.client.1.smithi196.stdout:6/774: read - dd/d1f/f47 zero size 2024-04-22T15:19:09.579 INFO:tasks.workunit.client.1.smithi196.stdout:8/743: dwrite d2/dc/fe [8388608,4194304] 0 2024-04-22T15:19:09.582 INFO:tasks.workunit.client.1.smithi196.stdout:7/748: stat d14/d25/d31/ff7 0 2024-04-22T15:19:09.582 INFO:tasks.workunit.client.1.smithi196.stdout:7/749: dread - d14/d26/d28/fa1 zero size 2024-04-22T15:19:09.583 INFO:tasks.workunit.client.1.smithi196.stdout:7/750: chown d14/d26/d28/d29 258357102 1 2024-04-22T15:19:09.586 INFO:tasks.workunit.client.1.smithi196.stdout:0/782: unlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9/leb 0 2024-04-22T15:19:09.595 INFO:tasks.workunit.client.0.smithi096.stdout:3/974: mknod da/db/d12/c141 0 2024-04-22T15:19:09.596 INFO:tasks.workunit.client.0.smithi096.stdout:8/937: write d4/d1c/d40/d46/d58/fde [115009,86381] 0 2024-04-22T15:19:09.596 INFO:tasks.workunit.client.0.smithi096.stdout:4/988: dwrite d5/d7/d2c/d3d/d4f/d63/f112 [0,4194304] 0 2024-04-22T15:19:09.598 INFO:tasks.workunit.client.0.smithi096.stdout:3/975: readlink da/db/d12/laf 0 2024-04-22T15:19:09.598 INFO:tasks.workunit.client.1.smithi196.stdout:5/776: dread d1/df1/d6e/fa2 [0,4194304] 0 2024-04-22T15:19:09.599 INFO:tasks.workunit.client.1.smithi196.stdout:1/768: getdents d1/db/dc/de/d43 0 2024-04-22T15:19:09.600 INFO:tasks.workunit.client.0.smithi096.stdout:4/989: dread - d5/d7/dcb/d12a/f13f zero size 2024-04-22T15:19:09.603 INFO:tasks.workunit.client.0.smithi096.stdout:5/994: write d4/ddb/d3d/d35/f10e [470671,50826] 0 2024-04-22T15:19:09.604 INFO:tasks.workunit.client.0.smithi096.stdout:1/987: mknod d6/dc/d8a/d3f/d54/d32/d3b/d94/c133 0 2024-04-22T15:19:09.605 INFO:tasks.workunit.client.1.smithi196.stdout:3/790: truncate d0/d15/d31/d55/ff9 2158302 0 2024-04-22T15:19:09.606 INFO:tasks.workunit.client.0.smithi096.stdout:4/990: dread - d5/d7/dcb/f139 zero size 2024-04-22T15:19:09.606 INFO:tasks.workunit.client.1.smithi196.stdout:3/791: fsync d0/d15/d31/d55/dcf/d8/d70/f7e 0 2024-04-22T15:19:09.609 INFO:tasks.workunit.client.0.smithi096.stdout:4/991: chown d5/d7/d2c/d3d/def/l101 74 1 2024-04-22T15:19:09.611 INFO:tasks.workunit.client.0.smithi096.stdout:6/867: dread d3/d28/f1d [0,4194304] 0 2024-04-22T15:19:09.612 INFO:tasks.workunit.client.0.smithi096.stdout:8/938: unlink d4/d10/c9d 0 2024-04-22T15:19:09.615 INFO:tasks.workunit.client.0.smithi096.stdout:6/868: read d3/d28/dd/d11/d2c/f91 [559394,34680] 0 2024-04-22T15:19:09.615 INFO:tasks.workunit.client.0.smithi096.stdout:3/976: dwrite da/db/d12/d9b/fd6 [8388608,4194304] 0 2024-04-22T15:19:09.616 INFO:tasks.workunit.client.1.smithi196.stdout:2/758: dwrite d1/d13/d25/ff9 [0,4194304] 0 2024-04-22T15:19:09.617 INFO:tasks.workunit.client.0.smithi096.stdout:3/977: chown da/d3f 0 1 2024-04-22T15:19:09.617 INFO:tasks.workunit.client.1.smithi196.stdout:2/759: fdatasync d1/f7e 0 2024-04-22T15:19:09.621 INFO:tasks.workunit.client.0.smithi096.stdout:1/988: dread d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/fdf [0,4194304] 0 2024-04-22T15:19:09.622 INFO:tasks.workunit.client.1.smithi196.stdout:9/738: symlink db/dd/d1b/lf5 0 2024-04-22T15:19:09.624 INFO:tasks.workunit.client.1.smithi196.stdout:4/747: link d6/d13/d15/d2c/d35/f9d d6/d14/d18/d26/d32/da3/ff0 0 2024-04-22T15:19:09.625 INFO:tasks.workunit.client.0.smithi096.stdout:5/995: write d4/ddb/d3d/d4d/d59/d98/ff1 [2608983,97754] 0 2024-04-22T15:19:09.630 INFO:tasks.workunit.client.1.smithi196.stdout:5/777: creat d1/df1/d5e/ffa x:0 0 0 2024-04-22T15:19:09.633 INFO:tasks.workunit.client.0.smithi096.stdout:6/869: creat d3/d28/dd/d11/d2c/d3b/f124 x:0 0 0 2024-04-22T15:19:09.634 INFO:tasks.workunit.client.0.smithi096.stdout:3/978: symlink da/db/dc/d28/d69/d89/l142 0 2024-04-22T15:19:09.635 INFO:tasks.workunit.client.1.smithi196.stdout:6/775: fsync dd/d2b/d8d/fb2 0 2024-04-22T15:19:09.637 INFO:tasks.workunit.client.0.smithi096.stdout:4/992: write d5/d7/d2c/f9d [480692,108377] 0 2024-04-22T15:19:09.642 INFO:tasks.workunit.client.1.smithi196.stdout:3/792: truncate d0/d15/d31/d55/d5d/fa0 606837 0 2024-04-22T15:19:09.643 INFO:tasks.workunit.client.1.smithi196.stdout:3/793: write d0/d15/d4d/f5a [4642647,69104] 0 2024-04-22T15:19:09.643 INFO:tasks.workunit.client.0.smithi096.stdout:6/870: dwrite d3/d3a/d75/d83/df7/fef [0,4194304] 0 2024-04-22T15:19:09.646 INFO:tasks.workunit.client.0.smithi096.stdout:5/996: truncate d4/ddb/d3d/d4d/d59/d5a/fe8 940780 0 2024-04-22T15:19:09.650 INFO:tasks.workunit.client.0.smithi096.stdout:5/997: truncate d4/ddb/d3d/d35/f10e 1066581 0 2024-04-22T15:19:09.653 INFO:tasks.workunit.client.0.smithi096.stdout:5/998: readlink d4/ddb/d3d/d4d/d99/la2 0 2024-04-22T15:19:09.653 INFO:tasks.workunit.client.0.smithi096.stdout:6/871: dwrite f2 [0,4194304] 0 2024-04-22T15:19:09.656 INFO:tasks.workunit.client.0.smithi096.stdout:3/979: dread - da/db/de/d87/fcf zero size 2024-04-22T15:19:09.661 INFO:tasks.workunit.client.0.smithi096.stdout:8/939: dwrite d4/d1c/d62/dc3/fda [0,4194304] 0 2024-04-22T15:19:09.663 INFO:tasks.workunit.client.1.smithi196.stdout:8/744: dwrite d2/d8/f27 [0,4194304] 0 2024-04-22T15:19:09.665 INFO:tasks.workunit.client.0.smithi096.stdout:4/993: symlink d5/d7/d2c/d3d/d4f/d63/d143/l154 0 2024-04-22T15:19:09.665 INFO:tasks.workunit.client.1.smithi196.stdout:6/776: rename dd/d1f/d37/l80 to dd/d1f/d38/d40/dd3/le1 0 2024-04-22T15:19:09.667 INFO:tasks.workunit.client.1.smithi196.stdout:6/777: readlink dd/d1f/d4e/d53/d55/d6d/dd1/lac 0 2024-04-22T15:19:09.667 INFO:tasks.workunit.client.0.smithi096.stdout:6/872: creat d3/d28/daf/d94/f125 x:0 0 0 2024-04-22T15:19:09.668 INFO:tasks.workunit.client.1.smithi196.stdout:3/794: mkdir d0/d15/d31/d55/d10c 0 2024-04-22T15:19:09.669 INFO:tasks.workunit.client.0.smithi096.stdout:1/989: dwrite d6/f1d [0,4194304] 0 2024-04-22T15:19:09.671 INFO:tasks.workunit.client.0.smithi096.stdout:5/999: symlink d4/ddb/d3d/ded/l15c 0 2024-04-22T15:19:09.673 INFO:tasks.workunit.client.0.smithi096.stdout:3/980: rename da/db/cb4 to da/db/d9a/d132/db5/c143 0 2024-04-22T15:19:09.673 INFO:tasks.workunit.client.1.smithi196.stdout:2/760: symlink d1/d23/d27/d2e/db5/d102/l113 0 2024-04-22T15:19:09.678 INFO:tasks.workunit.client.1.smithi196.stdout:6/778: dwrite dd/d1f/d37/d7b/fb4 [0,4194304] 0 2024-04-22T15:19:09.679 INFO:tasks.workunit.client.1.smithi196.stdout:6/779: chown dd/d1f/d4e/d53/d55/fc8 20078 1 2024-04-22T15:19:09.680 INFO:tasks.workunit.client.0.smithi096.stdout:3/981: dread - da/db/de/f13e zero size 2024-04-22T15:19:09.683 INFO:tasks.workunit.client.0.smithi096.stdout:3/982: chown da/db/dc/d28/d69/d89 110075508 1 2024-04-22T15:19:09.685 INFO:tasks.workunit.client.1.smithi196.stdout:5/778: dread d1/d17/d1a/f43 [0,4194304] 0 2024-04-22T15:19:09.687 INFO:tasks.workunit.client.0.smithi096.stdout:3/983: read da/db/de/d19/d3e/fde [814181,52278] 0 2024-04-22T15:19:09.687 INFO:tasks.workunit.client.0.smithi096.stdout:4/994: dwrite d5/d7/d2c/d3d/d4f/d63/f145 [0,4194304] 0 2024-04-22T15:19:09.688 INFO:tasks.workunit.client.1.smithi196.stdout:7/751: getdents d14/d25/d80 0 2024-04-22T15:19:09.690 INFO:tasks.workunit.client.0.smithi096.stdout:1/990: rmdir d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a 39 2024-04-22T15:19:09.692 INFO:tasks.workunit.client.1.smithi196.stdout:2/761: symlink d1/d13/d25/d2a/ddb/dff/l114 0 2024-04-22T15:19:09.695 INFO:tasks.workunit.client.0.smithi096.stdout:6/873: creat d3/d28/dd/d11/d2c/f126 x:0 0 0 2024-04-22T15:19:09.697 INFO:tasks.workunit.client.1.smithi196.stdout:6/780: symlink dd/d1f/d4e/d53/d7f/le2 0 2024-04-22T15:19:09.698 INFO:tasks.workunit.client.1.smithi196.stdout:4/748: write d6/d14/d18/d26/d32/d67/d9a/da5/fac [925352,2997] 0 2024-04-22T15:19:09.699 INFO:tasks.workunit.client.1.smithi196.stdout:6/781: fdatasync dd/d1f/d4e/d53/d7f/da5/fb1 0 2024-04-22T15:19:09.699 INFO:tasks.workunit.client.1.smithi196.stdout:5/779: creat d1/d17/d1a/d2f/ffb x:0 0 0 2024-04-22T15:19:09.701 INFO:tasks.workunit.client.1.smithi196.stdout:3/795: fdatasync d0/d15/da7/d4a/d44/f20 0 2024-04-22T15:19:09.702 INFO:tasks.workunit.client.0.smithi096.stdout:1/991: chown d6/dc/d8a/d29 9326 1 2024-04-22T15:19:09.703 INFO:tasks.workunit.client.1.smithi196.stdout:9/739: dwrite db/dd/de/d15/f6e [0,4194304] 0 2024-04-22T15:19:09.703 INFO:tasks.workunit.client.1.smithi196.stdout:2/762: mknod d1/d13/dbc/c115 0 2024-04-22T15:19:09.708 INFO:tasks.workunit.client.0.smithi096.stdout:3/984: mknod da/db/dc/d28/c144 0 2024-04-22T15:19:09.708 INFO:tasks.workunit.client.1.smithi196.stdout:7/752: dwrite d14/d15/fcc [0,4194304] 0 2024-04-22T15:19:09.714 INFO:tasks.workunit.client.0.smithi096.stdout:6/874: symlink d3/d28/d54/d68/d6e/d10d/l127 0 2024-04-22T15:19:09.716 INFO:tasks.workunit.client.0.smithi096.stdout:4/995: creat d5/d7/f155 x:0 0 0 2024-04-22T15:19:09.717 INFO:tasks.workunit.client.1.smithi196.stdout:9/740: dread - db/dd/fc6 zero size 2024-04-22T15:19:09.720 INFO:tasks.workunit.client.0.smithi096.stdout:6/875: read d3/d28/d54/d68/d6e/fdf [2564451,63203] 0 2024-04-22T15:19:09.724 INFO:tasks.workunit.client.1.smithi196.stdout:0/783: sync 2024-04-22T15:19:09.725 INFO:tasks.workunit.client.1.smithi196.stdout:7/753: truncate d14/d26/d28/f93 1803554 0 2024-04-22T15:19:09.726 INFO:tasks.workunit.client.1.smithi196.stdout:7/754: chown d14/d15/f74 0 1 2024-04-22T15:19:09.728 INFO:tasks.workunit.client.1.smithi196.stdout:3/796: unlink d0/d15/d31/d55/dcf/f5b 0 2024-04-22T15:19:09.729 INFO:tasks.workunit.client.1.smithi196.stdout:9/741: rmdir db/dd/de/dd7/def/d7a 39 2024-04-22T15:19:09.731 INFO:tasks.workunit.client.1.smithi196.stdout:7/755: dwrite d14/d26/d5a/d5f/dc2/fda [0,4194304] 0 2024-04-22T15:19:09.733 INFO:tasks.workunit.client.0.smithi096.stdout:6/876: rename d3/d28/dd/d11/d2c/d6f/dd4/f114 to d3/d3a/d75/f128 0 2024-04-22T15:19:09.735 INFO:tasks.workunit.client.1.smithi196.stdout:2/763: rename d1/d13/c69 to d1/d23/d27/d100/c116 0 2024-04-22T15:19:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:09 smithi096 ceph-mon[26688]: pgmap v41: 65 pgs: 65 active+clean; 3.3 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 104 MiB/s rd, 227 MiB/s wr, 513 op/s 2024-04-22T15:19:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:09 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3715490961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:19:09.736 INFO:tasks.workunit.client.1.smithi196.stdout:2/764: chown d1/d1d 1739919 1 2024-04-22T15:19:09.737 INFO:tasks.workunit.client.1.smithi196.stdout:2/765: write d1/ff [6586823,31237] 0 2024-04-22T15:19:09.738 INFO:tasks.workunit.client.1.smithi196.stdout:0/784: mkdir d3/de/d1c/df1/d116 0 2024-04-22T15:19:09.738 INFO:tasks.workunit.client.1.smithi196.stdout:7/756: dwrite d14/d25/d31/f8e [0,4194304] 0 2024-04-22T15:19:09.740 INFO:tasks.workunit.client.0.smithi096.stdout:6/877: creat d3/d28/dd/d11/d2c/d6f/dd4/ddd/f129 x:0 0 0 2024-04-22T15:19:09.740 INFO:tasks.workunit.client.1.smithi196.stdout:7/757: fsync d14/d26/d2a/d2b/dc7/fe0 0 2024-04-22T15:19:09.741 INFO:tasks.workunit.client.0.smithi096.stdout:6/878: chown d3/d28/f1c 64 1 2024-04-22T15:19:09.742 INFO:tasks.workunit.client.0.smithi096.stdout:4/996: creat d5/d7/d2c/d43/f156 x:0 0 0 2024-04-22T15:19:09.745 INFO:tasks.workunit.client.0.smithi096.stdout:6/879: creat d3/d28/d54/d68/d6e/d10d/f12a x:0 0 0 2024-04-22T15:19:09.746 INFO:tasks.workunit.client.1.smithi196.stdout:3/797: symlink d0/d15/d31/d55/dcf/d6f/db5/dd7/l10d 0 2024-04-22T15:19:09.750 INFO:tasks.workunit.client.0.smithi096.stdout:4/997: dwrite d5/d7/d2c/d3d/d4f/d63/f7e [0,4194304] 0 2024-04-22T15:19:09.751 INFO:tasks.workunit.client.1.smithi196.stdout:3/798: read - d0/d15/d4d/d87/da6/fbd zero size 2024-04-22T15:19:09.752 INFO:tasks.workunit.client.0.smithi096.stdout:4/998: chown d5/d7/dcb/l10a 439760645 1 2024-04-22T15:19:09.752 INFO:tasks.workunit.client.1.smithi196.stdout:9/742: mkdir db/dd/de/d15/d31/df6 0 2024-04-22T15:19:09.753 INFO:tasks.workunit.client.0.smithi096.stdout:4/999: chown d5/d7/d2c/d3d/d4f/d63/ce7 55718 1 2024-04-22T15:19:09.754 INFO:tasks.workunit.client.1.smithi196.stdout:2/766: creat d1/f117 x:0 0 0 2024-04-22T15:19:09.755 INFO:tasks.workunit.client.1.smithi196.stdout:3/799: dread - d0/d15/d31/d55/dcf/d6f/de5/d101/d9e/fc2 zero size 2024-04-22T15:19:09.756 INFO:tasks.workunit.client.1.smithi196.stdout:7/758: dwrite f2 [0,4194304] 0 2024-04-22T15:19:09.759 INFO:tasks.workunit.client.1.smithi196.stdout:9/743: truncate db/dd/d1b/d2c/f7c 374853 0 2024-04-22T15:19:09.759 INFO:tasks.workunit.client.1.smithi196.stdout:2/767: chown d1/d23/d27/d2e/d55/lf3 11686 1 2024-04-22T15:19:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:09 smithi196 ceph-mon[34717]: pgmap v41: 65 pgs: 65 active+clean; 3.3 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 104 MiB/s rd, 227 MiB/s wr, 513 op/s 2024-04-22T15:19:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:09 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3715490961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:19:09.760 INFO:tasks.workunit.client.1.smithi196.stdout:1/769: write d1/db/dc/d26/d2e/f3e [3928422,81819] 0 2024-04-22T15:19:09.762 INFO:tasks.workunit.client.1.smithi196.stdout:7/759: read d14/d26/d2a/d72/db1/fec [8322366,31434] 0 2024-04-22T15:19:09.762 INFO:tasks.workunit.client.1.smithi196.stdout:2/768: creat d1/d13/d25/d9f/f118 x:0 0 0 2024-04-22T15:19:09.763 INFO:tasks.workunit.client.1.smithi196.stdout:9/744: creat db/dd/de/dd7/def/d57/da7/ff7 x:0 0 0 2024-04-22T15:19:09.767 INFO:tasks.workunit.client.1.smithi196.stdout:7/760: creat d14/d15/d91/ffa x:0 0 0 2024-04-22T15:19:09.768 INFO:tasks.workunit.client.1.smithi196.stdout:2/769: mknod d1/d23/df5/c119 0 2024-04-22T15:19:09.770 INFO:tasks.workunit.client.1.smithi196.stdout:7/761: symlink d14/d26/d28/d29/lfb 0 2024-04-22T15:19:09.772 INFO:tasks.workunit.client.1.smithi196.stdout:9/745: creat db/dd/de/dd7/def/d7a/d7b/ff8 x:0 0 0 2024-04-22T15:19:09.772 INFO:tasks.workunit.client.1.smithi196.stdout:0/785: dread d3/d6/dc/f92 [0,4194304] 0 2024-04-22T15:19:09.772 INFO:tasks.workunit.client.1.smithi196.stdout:2/770: truncate d1/f7e 3752010 0 2024-04-22T15:19:09.773 INFO:tasks.workunit.client.1.smithi196.stdout:9/746: readlink db/dd/de/dd7/def/d57/da7/lc7 0 2024-04-22T15:19:09.773 INFO:tasks.workunit.client.1.smithi196.stdout:7/762: symlink d14/d26/d5a/d5f/dc2/lfc 0 2024-04-22T15:19:09.773 INFO:tasks.workunit.client.1.smithi196.stdout:0/786: chown d3/de/d14/l17 116 1 2024-04-22T15:19:09.773 INFO:tasks.workunit.client.1.smithi196.stdout:3/800: dread d0/d15/d31/d55/dcf/d6f/d10a/d5e/f63 [0,4194304] 0 2024-04-22T15:19:09.774 INFO:tasks.workunit.client.1.smithi196.stdout:0/787: readlink d3/d9/l2e 0 2024-04-22T15:19:09.775 INFO:tasks.workunit.client.1.smithi196.stdout:7/763: creat d14/d4d/ffd x:0 0 0 2024-04-22T15:19:09.775 INFO:tasks.workunit.client.1.smithi196.stdout:2/771: symlink d1/d23/d27/d2e/d55/d5b/d93/d97/l11a 0 2024-04-22T15:19:09.776 INFO:tasks.workunit.client.1.smithi196.stdout:3/801: stat d0/d15/d31/d55/dcf/d6f/db5/dd7/l10d 0 2024-04-22T15:19:09.777 INFO:tasks.workunit.client.1.smithi196.stdout:9/747: unlink db/dd/de/d15/d31/d64/dab/lc0 0 2024-04-22T15:19:09.777 INFO:tasks.workunit.client.1.smithi196.stdout:2/772: chown d1/d23/d27/d2e/d55/lcb 88324 1 2024-04-22T15:19:09.778 INFO:tasks.workunit.client.1.smithi196.stdout:7/764: mkdir d14/d4d/da3/de9/dfe 0 2024-04-22T15:19:09.778 INFO:tasks.workunit.client.1.smithi196.stdout:3/802: stat d0/d15/d31/d55/d8b/fc0 0 2024-04-22T15:19:09.779 INFO:tasks.workunit.client.1.smithi196.stdout:9/748: write db/dd/de/d75/faa [273784,72493] 0 2024-04-22T15:19:09.785 INFO:tasks.workunit.client.1.smithi196.stdout:7/765: creat d14/d26/fff x:0 0 0 2024-04-22T15:19:09.786 INFO:tasks.workunit.client.1.smithi196.stdout:0/788: dwrite d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8/ffb [0,4194304] 0 2024-04-22T15:19:09.787 INFO:tasks.workunit.client.1.smithi196.stdout:7/766: fdatasync d14/d26/d2a/d72/db1/fce 0 2024-04-22T15:19:09.787 INFO:tasks.workunit.client.1.smithi196.stdout:9/749: creat db/dd/de/dd7/def/d57/ff9 x:0 0 0 2024-04-22T15:19:09.788 INFO:tasks.workunit.client.1.smithi196.stdout:3/803: getdents d0/d15/d31/d55/d6e 0 2024-04-22T15:19:09.789 INFO:tasks.workunit.client.1.smithi196.stdout:3/804: chown d0/d15/da7/fa8 44 1 2024-04-22T15:19:09.791 INFO:tasks.workunit.client.1.smithi196.stdout:7/767: creat d14/d26/d2a/d2b/de8/f100 x:0 0 0 2024-04-22T15:19:09.791 INFO:tasks.workunit.client.1.smithi196.stdout:9/750: creat db/dd/de/d15/d31/d64/ffa x:0 0 0 2024-04-22T15:19:09.794 INFO:tasks.workunit.client.1.smithi196.stdout:9/751: chown db/dd/de/d15/d31/d64/f62 703 1 2024-04-22T15:19:09.795 INFO:tasks.workunit.client.1.smithi196.stdout:7/768: creat d14/d26/d2a/d2b/f101 x:0 0 0 2024-04-22T15:19:09.795 INFO:tasks.workunit.client.1.smithi196.stdout:3/805: getdents d0/d15/da7/d4a/d44/d96/dd8 0 2024-04-22T15:19:09.796 INFO:tasks.workunit.client.1.smithi196.stdout:0/789: dwrite d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/fe5 [0,4194304] 0 2024-04-22T15:19:09.797 INFO:tasks.workunit.client.1.smithi196.stdout:9/752: unlink db/dd/c1d 0 2024-04-22T15:19:09.798 INFO:tasks.workunit.client.1.smithi196.stdout:9/753: stat db/dd/d1b/d2c/l77 0 2024-04-22T15:19:09.798 INFO:tasks.workunit.client.1.smithi196.stdout:3/806: creat d0/d15/d31/d55/dcf/d6f/de5/f10e x:0 0 0 2024-04-22T15:19:09.799 INFO:tasks.workunit.client.1.smithi196.stdout:7/769: truncate d14/d15/f59 3345556 0 2024-04-22T15:19:09.800 INFO:tasks.workunit.client.1.smithi196.stdout:9/754: truncate db/d11/fea 1949798 0 2024-04-22T15:19:09.801 INFO:tasks.workunit.client.1.smithi196.stdout:7/770: fsync d14/f33 0 2024-04-22T15:19:09.804 INFO:tasks.workunit.client.1.smithi196.stdout:9/755: truncate db/dd/de/fe9 3978006 0 2024-04-22T15:19:09.807 INFO:tasks.workunit.client.1.smithi196.stdout:9/756: symlink db/dd/de/d75/lfb 0 2024-04-22T15:19:09.809 INFO:tasks.workunit.client.1.smithi196.stdout:7/771: rename d14/d4a/c9b to d14/d26/d5a/d5f/dc2/c102 0 2024-04-22T15:19:09.813 INFO:tasks.workunit.client.1.smithi196.stdout:7/772: symlink d14/d25/l103 0 2024-04-22T15:19:09.817 INFO:tasks.workunit.client.1.smithi196.stdout:9/757: link db/dd/l5e db/dd/de/d15/d31/d99/lfc 0 2024-04-22T15:19:09.821 INFO:tasks.workunit.client.1.smithi196.stdout:7/773: rmdir d14/d46 0 2024-04-22T15:19:09.825 INFO:tasks.workunit.client.0.smithi096.stdout:8/940: dwrite d4/d2a/d7e/f9a [0,4194304] 0 2024-04-22T15:19:09.825 INFO:tasks.workunit.client.1.smithi196.stdout:9/758: dwrite db/dd/de/d15/f73 [0,4194304] 0 2024-04-22T15:19:09.826 INFO:tasks.workunit.client.1.smithi196.stdout:3/807: dread d0/d15/d4d/db/de6/fee [0,4194304] 0 2024-04-22T15:19:09.827 INFO:tasks.workunit.client.1.smithi196.stdout:6/782: write dd/f16 [799438,102171] 0 2024-04-22T15:19:09.827 INFO:tasks.workunit.client.0.smithi096.stdout:8/941: mknod d4/d1c/d40/d48/c13c 0 2024-04-22T15:19:09.828 INFO:tasks.workunit.client.1.smithi196.stdout:8/745: dwrite d2/d5/d4f/f6b [0,4194304] 0 2024-04-22T15:19:09.829 INFO:tasks.workunit.client.1.smithi196.stdout:6/783: stat dd/d1f/d4e/d53/fa4 0 2024-04-22T15:19:09.831 INFO:tasks.workunit.client.1.smithi196.stdout:5/780: write d1/d2/faf [393449,74880] 0 2024-04-22T15:19:09.832 INFO:tasks.workunit.client.0.smithi096.stdout:8/942: getdents d4/d71 0 2024-04-22T15:19:09.833 INFO:tasks.workunit.client.0.smithi096.stdout:1/992: write d6/dc/d8a/d3f/d62/d7d/ddb/f78 [1700246,5814] 0 2024-04-22T15:19:09.836 INFO:tasks.workunit.client.1.smithi196.stdout:6/784: fdatasync dd/d1f/d4e/d53/d7f/f83 0 2024-04-22T15:19:09.837 INFO:tasks.workunit.client.0.smithi096.stdout:3/985: truncate da/db/de/ddc/d68/dbc/ff1 1107948 0 2024-04-22T15:19:09.841 INFO:tasks.workunit.client.1.smithi196.stdout:8/746: symlink d2/d8/d47/d68/dbc/da4/ld8 0 2024-04-22T15:19:09.841 INFO:tasks.workunit.client.1.smithi196.stdout:4/749: write d6/d13/d15/d2c/d62/f88 [1047932,1537] 0 2024-04-22T15:19:09.842 INFO:tasks.workunit.client.1.smithi196.stdout:8/747: readlink d2/d3f/l71 0 2024-04-22T15:19:09.846 INFO:tasks.workunit.client.1.smithi196.stdout:3/808: dwrite d0/d15/da7/d4a/d44/d8c/fbc [0,4194304] 0 2024-04-22T15:19:09.847 INFO:tasks.workunit.client.0.smithi096.stdout:3/986: creat da/db/de/ddc/d68/db0/f145 x:0 0 0 2024-04-22T15:19:09.848 INFO:tasks.workunit.client.0.smithi096.stdout:3/987: chown da/db/de/d19/f30 4037 1 2024-04-22T15:19:09.852 INFO:tasks.workunit.client.1.smithi196.stdout:6/785: mkdir dd/d1f/d4e/d53/d55/d6d/dd1/de3 0 2024-04-22T15:19:09.853 INFO:tasks.workunit.client.1.smithi196.stdout:6/786: truncate dd/d1f/d37/d58/fce 2649 0 2024-04-22T15:19:09.855 INFO:tasks.workunit.client.1.smithi196.stdout:4/750: unlink c3 0 2024-04-22T15:19:09.855 INFO:tasks.workunit.client.1.smithi196.stdout:4/751: stat d6/d13/d15/d2c 0 2024-04-22T15:19:09.862 INFO:tasks.workunit.client.0.smithi096.stdout:8/943: getdents d4/d10/d4d/deb 0 2024-04-22T15:19:09.866 INFO:tasks.workunit.client.0.smithi096.stdout:1/993: link d6/dc/d8a/d3f/d54/d32/d3b/d3d/l91 d6/dc/d8a/d3f/d54/d32/d8c/d45/d6a/l134 0 2024-04-22T15:19:09.867 INFO:tasks.workunit.client.1.smithi196.stdout:6/787: mkdir dd/d1f/d4e/d53/d55/d57/de4 0 2024-04-22T15:19:09.868 INFO:tasks.workunit.client.0.smithi096.stdout:1/994: dread d6/dc/d8a/d3f/d54/f21 [0,4194304] 0 2024-04-22T15:19:09.869 INFO:tasks.workunit.client.0.smithi096.stdout:8/944: fsync d4/d10/f21 0 2024-04-22T15:19:09.870 INFO:tasks.workunit.client.1.smithi196.stdout:1/770: truncate d1/db/dc/d26/fe5 1336427 0 2024-04-22T15:19:09.871 INFO:tasks.workunit.client.1.smithi196.stdout:4/752: creat d6/d13/d15/d2c/d62/d78/ff1 x:0 0 0 2024-04-22T15:19:09.873 INFO:tasks.workunit.client.0.smithi096.stdout:1/995: mknod d6/dc/d8a/d3f/d62/dc2/dd0/c135 0 2024-04-22T15:19:09.873 INFO:tasks.workunit.client.0.smithi096.stdout:8/945: dread d4/d1c/d40/d46/d58/f7a [0,4194304] 0 2024-04-22T15:19:09.879 INFO:tasks.workunit.client.1.smithi196.stdout:5/781: getdents d1/d4/d7b 0 2024-04-22T15:19:09.880 INFO:tasks.workunit.client.1.smithi196.stdout:2/773: write d1/f10 [2981523,77328] 0 2024-04-22T15:19:09.881 INFO:tasks.workunit.client.1.smithi196.stdout:6/788: unlink dd/d1f/d38/f79 0 2024-04-22T15:19:09.884 INFO:tasks.workunit.client.0.smithi096.stdout:8/946: dread d4/d10/d25/d4c/f5d [0,4194304] 0 2024-04-22T15:19:09.885 INFO:tasks.workunit.client.0.smithi096.stdout:1/996: creat d6/f136 x:0 0 0 2024-04-22T15:19:09.886 INFO:tasks.workunit.client.0.smithi096.stdout:1/997: chown d6/dc/d8a/d3f/fda 37829 1 2024-04-22T15:19:09.887 INFO:tasks.workunit.client.0.smithi096.stdout:1/998: dread d6/f60 [0,4194304] 0 2024-04-22T15:19:09.889 INFO:tasks.workunit.client.0.smithi096.stdout:8/947: fdatasync d4/d10/d25/f37 0 2024-04-22T15:19:09.893 INFO:tasks.workunit.client.0.smithi096.stdout:1/999: creat d6/dc/d8a/d3f/d62/d7d/dd3/f137 x:0 0 0 2024-04-22T15:19:09.894 INFO:tasks.workunit.client.1.smithi196.stdout:0/790: write d3/d9/d36/d9f/dd1/d65/f6b [2203958,73289] 0 2024-04-22T15:19:09.898 INFO:tasks.workunit.client.0.smithi096.stdout:8/948: mknod d4/d1c/d40/d46/d120/c13d 0 2024-04-22T15:19:09.903 INFO:tasks.workunit.client.0.smithi096.stdout:8/949: symlink d4/d10/d4d/deb/l13e 0 2024-04-22T15:19:09.905 INFO:tasks.workunit.client.1.smithi196.stdout:4/753: getdents d6/d13/d15/d2c/d8c/db3 0 2024-04-22T15:19:09.907 INFO:tasks.workunit.client.1.smithi196.stdout:5/782: getdents d1/d17/d1a/d2f/d65 0 2024-04-22T15:19:09.908 INFO:tasks.workunit.client.1.smithi196.stdout:7/774: dread - d14/d26/d2a/d2b/de8/dc6/fd4 zero size 2024-04-22T15:19:09.909 INFO:tasks.workunit.client.0.smithi096.stdout:8/950: mkdir d4/d10/d25/d38/d13f 0 2024-04-22T15:19:09.911 INFO:tasks.workunit.client.1.smithi196.stdout:4/754: creat d6/d14/d18/d26/d32/d53/ff2 x:0 0 0 2024-04-22T15:19:09.912 INFO:tasks.workunit.client.1.smithi196.stdout:8/748: rename d2/d8/d47 to d2/d5/d1a/db7/dd9 0 2024-04-22T15:19:09.912 INFO:tasks.workunit.client.1.smithi196.stdout:0/791: rename d3/d9/d36 to d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9/d117 22 2024-04-22T15:19:09.916 INFO:tasks.workunit.client.0.smithi096.stdout:8/951: mknod d4/d10/d25/d4c/db4/dd0/d139/c140 0 2024-04-22T15:19:09.918 INFO:tasks.workunit.client.0.smithi096.stdout:3/988: write da/db/de/d19/d3e/fde [561262,93934] 0 2024-04-22T15:19:09.919 INFO:tasks.workunit.client.1.smithi196.stdout:0/792: unlink d3/db4/lf2 0 2024-04-22T15:19:09.923 INFO:tasks.workunit.client.1.smithi196.stdout:8/749: rename d2/d5/d1a/db7/dd9/d68/d81/f89 to d2/dc/d45/fda 0 2024-04-22T15:19:09.924 INFO:tasks.workunit.client.0.smithi096.stdout:3/989: mkdir da/db/de/d38/d114/d146 0 2024-04-22T15:19:09.928 INFO:tasks.workunit.client.1.smithi196.stdout:3/809: dwrite d0/d15/d31/d55/dcf/d6f/d10a/d5e/f63 [0,4194304] 0 2024-04-22T15:19:09.933 INFO:tasks.workunit.client.1.smithi196.stdout:3/810: dread d0/d15/da7/d4a/d44/ff3 [0,4194304] 0 2024-04-22T15:19:09.940 INFO:tasks.workunit.client.1.smithi196.stdout:1/771: sync 2024-04-22T15:19:09.940 INFO:tasks.workunit.client.1.smithi196.stdout:9/759: sync 2024-04-22T15:19:10.073 INFO:tasks.workunit.client.0.smithi096.stdout:8/952: creat d4/d1c/d62/dc3/df9/f141 x:0 0 0 2024-04-22T15:19:10.079 INFO:tasks.workunit.client.0.smithi096.stdout:3/990: symlink da/db/d12/d9b/l147 0 2024-04-22T15:19:10.084 INFO:tasks.workunit.client.1.smithi196.stdout:2/774: write d1/d13/d25/d9f/faf [469500,64790] 0 2024-04-22T15:19:10.085 INFO:tasks.workunit.client.1.smithi196.stdout:2/775: chown d1/d23/d27/d2e/d55/d5b/ca0 38200 1 2024-04-22T15:19:10.086 INFO:tasks.workunit.client.0.smithi096.stdout:8/953: creat d4/d10/d25/d38/d8f/d9c/f142 x:0 0 0 2024-04-22T15:19:10.086 INFO:tasks.workunit.client.1.smithi196.stdout:2/776: chown d1/d23/d27/d89/cfd 1709945 1 2024-04-22T15:19:10.087 INFO:tasks.workunit.client.1.smithi196.stdout:6/789: write dd/d2b/d8d/fb2 [158599,117794] 0 2024-04-22T15:19:10.088 INFO:tasks.workunit.client.1.smithi196.stdout:0/793: fsync d3/d9/d36/d9f/dd1/d65/d7f/d30/fe7 0 2024-04-22T15:19:10.093 INFO:tasks.workunit.client.0.smithi096.stdout:8/954: truncate d4/d10/d25/fe1 134198 0 2024-04-22T15:19:10.094 INFO:tasks.workunit.client.0.smithi096.stdout:8/955: write d4/dc0/d123/f128 [517156,67229] 0 2024-04-22T15:19:10.101 INFO:tasks.workunit.client.0.smithi096.stdout:8/956: fdatasync d4/d1c/d40/f10e 0 2024-04-22T15:19:10.103 INFO:tasks.workunit.client.1.smithi196.stdout:3/811: chown d0/d15/d31/d55/d8b/c93 1294227 1 2024-04-22T15:19:10.105 INFO:tasks.workunit.client.1.smithi196.stdout:9/760: creat db/dd/de/d15/d31/d99/ffd x:0 0 0 2024-04-22T15:19:10.106 INFO:tasks.workunit.client.1.smithi196.stdout:9/761: chown db/dd/de/dd7/def/d57/c2d 357489 1 2024-04-22T15:19:10.108 INFO:tasks.workunit.client.0.smithi096.stdout:8/957: dwrite d4/d10/d25/d4c/d76/f10f [0,4194304] 0 2024-04-22T15:19:10.113 INFO:tasks.workunit.client.1.smithi196.stdout:5/783: dwrite d1/d14/d6a/fa8 [0,4194304] 0 2024-04-22T15:19:10.114 INFO:tasks.workunit.client.1.smithi196.stdout:5/784: read - d1/d14/d6a/fe2 zero size 2024-04-22T15:19:10.123 INFO:tasks.workunit.client.1.smithi196.stdout:5/785: dwrite d1/d78/fd2 [0,4194304] 0 2024-04-22T15:19:10.125 INFO:tasks.workunit.client.1.smithi196.stdout:5/786: read d1/d14/d6a/fa8 [532841,37958] 0 2024-04-22T15:19:10.127 INFO:tasks.workunit.client.1.smithi196.stdout:0/794: symlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/dd0/de3/l118 0 2024-04-22T15:19:10.136 INFO:tasks.workunit.client.1.smithi196.stdout:3/812: fdatasync d0/d15/d31/d55/dcf/f38 0 2024-04-22T15:19:10.274 INFO:tasks.workunit.client.1.smithi196.stdout:5/787: chown d1/d17/d1a/d2f/f99 2164 1 2024-04-22T15:19:10.277 INFO:tasks.workunit.client.1.smithi196.stdout:5/788: dread d1/d9/f20 [0,4194304] 0 2024-04-22T15:19:10.327 INFO:tasks.workunit.client.0.smithi096.stdout:8/958: symlink d4/d10/d25/d4c/db4/l143 0 2024-04-22T15:19:10.337 INFO:tasks.workunit.client.0.smithi096.stdout:9/885: sync 2024-04-22T15:19:10.341 INFO:tasks.workunit.client.0.smithi096.stdout:9/886: mkdir d1/d4/d7f/d12d 0 2024-04-22T15:19:10.342 INFO:tasks.workunit.client.0.smithi096.stdout:9/887: chown d1/df/d95/d9d/db5/d59 14434466 1 2024-04-22T15:19:10.348 INFO:tasks.workunit.client.0.smithi096.stdout:8/959: dread d4/d1c/f3a [4194304,4194304] 0 2024-04-22T15:19:10.351 INFO:tasks.workunit.client.1.smithi196.stdout:0/795: mkdir d3/d9/d36/d114/d119 0 2024-04-22T15:19:10.351 INFO:tasks.workunit.client.1.smithi196.stdout:0/796: write d3/db4/fe9 [1458555,74115] 0 2024-04-22T15:19:10.352 INFO:tasks.workunit.client.1.smithi196.stdout:0/797: chown d3/fd 79493029 1 2024-04-22T15:19:10.355 INFO:tasks.workunit.client.1.smithi196.stdout:5/789: unlink d1/d17/d1a/d2f/d65/cf5 0 2024-04-22T15:19:10.357 INFO:tasks.workunit.client.0.smithi096.stdout:8/960: symlink d4/d1c/df8/l144 0 2024-04-22T15:19:10.358 INFO:tasks.workunit.client.0.smithi096.stdout:8/961: readlink d4/d10/d25/d4c/lac 0 2024-04-22T15:19:10.360 INFO:tasks.workunit.client.1.smithi196.stdout:2/777: link d1/d1d/l8a d1/d23/d27/d2e/d55/d5b/d7c/l11b 0 2024-04-22T15:19:10.368 INFO:tasks.workunit.client.0.smithi096.stdout:8/962: creat d4/d10/d25/d38/d13f/f145 x:0 0 0 2024-04-22T15:19:10.371 INFO:tasks.workunit.client.1.smithi196.stdout:3/813: creat d0/f10f x:0 0 0 2024-04-22T15:19:10.375 INFO:tasks.workunit.client.1.smithi196.stdout:3/814: read d0/d15/d31/d55/dcf/d8/f107 [334036,109079] 0 2024-04-22T15:19:10.376 INFO:tasks.workunit.client.1.smithi196.stdout:5/790: link d1/d14/d6a/lf7 d1/df1/d6e/lfc 0 2024-04-22T15:19:10.379 INFO:tasks.workunit.client.1.smithi196.stdout:3/815: symlink d0/d15/d31/d55/dcf/l110 0 2024-04-22T15:19:10.381 INFO:tasks.workunit.client.1.smithi196.stdout:5/791: mknod d1/d9/d68/cfd 0 2024-04-22T15:19:10.383 INFO:tasks.workunit.client.1.smithi196.stdout:3/816: creat d0/d15/d4d/d87/f111 x:0 0 0 2024-04-22T15:19:10.385 INFO:tasks.workunit.client.1.smithi196.stdout:5/792: write d1/d14/d6a/fa8 [2689109,119108] 0 2024-04-22T15:19:10.388 INFO:tasks.workunit.client.1.smithi196.stdout:3/817: mknod d0/d15/d4d/d87/da6/c112 0 2024-04-22T15:19:10.389 INFO:tasks.workunit.client.1.smithi196.stdout:3/818: chown d0/d15/da7/d18/cdb 177433 1 2024-04-22T15:19:10.390 INFO:tasks.workunit.client.1.smithi196.stdout:5/793: creat d1/d17/d1c/d67/ffe x:0 0 0 2024-04-22T15:19:10.390 INFO:tasks.workunit.client.1.smithi196.stdout:3/819: truncate d0/d15/da7/fdf 609680 0 2024-04-22T15:19:10.391 INFO:tasks.workunit.client.1.smithi196.stdout:2/778: dread d1/f18 [0,4194304] 0 2024-04-22T15:19:10.392 INFO:tasks.workunit.client.1.smithi196.stdout:5/794: fsync d1/d17/d1c/f88 0 2024-04-22T15:19:10.395 INFO:tasks.workunit.client.1.smithi196.stdout:2/779: dread - d1/d23/d27/d2e/d55/d5b/d93/fbf zero size 2024-04-22T15:19:10.398 INFO:tasks.workunit.client.1.smithi196.stdout:5/795: dwrite d1/d17/d1a/d2f/ffb [0,4194304] 0 2024-04-22T15:19:10.400 INFO:tasks.workunit.client.1.smithi196.stdout:5/796: write d1/d14/d6a/fe8 [196619,116792] 0 2024-04-22T15:19:10.401 INFO:tasks.workunit.client.1.smithi196.stdout:5/797: truncate d1/df1/d7d/fd7 695467 0 2024-04-22T15:19:10.403 INFO:tasks.workunit.client.1.smithi196.stdout:5/798: write d1/df1/d6e/d7a/fdb [953672,106878] 0 2024-04-22T15:19:10.404 INFO:tasks.workunit.client.1.smithi196.stdout:5/799: stat d1/d4/fc2 0 2024-04-22T15:19:10.406 INFO:tasks.workunit.client.1.smithi196.stdout:5/800: fdatasync d1/d17/d1a/fe5 0 2024-04-22T15:19:10.408 INFO:tasks.workunit.client.0.smithi096.stdout:3/991: sync 2024-04-22T15:19:10.408 INFO:tasks.workunit.client.0.smithi096.stdout:6/880: sync 2024-04-22T15:19:10.409 INFO:tasks.workunit.client.0.smithi096.stdout:3/992: readlink da/db/d9a/d132/l139 0 2024-04-22T15:19:10.410 INFO:tasks.workunit.client.0.smithi096.stdout:6/881: write d3/d28/daf/d94/ffa [823113,122052] 0 2024-04-22T15:19:10.413 INFO:tasks.workunit.client.1.smithi196.stdout:5/801: fsync d1/d17/d1c/f52 0 2024-04-22T15:19:10.415 INFO:tasks.workunit.client.0.smithi096.stdout:6/882: symlink d3/d28/d4d/d82/d9e/db0/l12b 0 2024-04-22T15:19:10.418 INFO:tasks.workunit.client.0.smithi096.stdout:3/993: dwrite da/db/de/d38/d119/f12c [0,4194304] 0 2024-04-22T15:19:10.418 INFO:tasks.workunit.client.1.smithi196.stdout:2/780: dread d1/d1d/f3c [0,4194304] 0 2024-04-22T15:19:10.419 INFO:tasks.workunit.client.1.smithi196.stdout:2/781: write d1/d13/d25/ff9 [2042138,58180] 0 2024-04-22T15:19:10.421 INFO:tasks.workunit.client.1.smithi196.stdout:2/782: readlink d1/d23/d27/d2e/d55/d5b/d93/d97/l11a 0 2024-04-22T15:19:10.427 INFO:tasks.workunit.client.1.smithi196.stdout:4/755: write d6/d14/d18/d26/d32/fcb [648607,59035] 0 2024-04-22T15:19:10.428 INFO:tasks.workunit.client.1.smithi196.stdout:4/756: chown d6/d10/ld 3023272 1 2024-04-22T15:19:10.429 INFO:tasks.workunit.client.1.smithi196.stdout:4/757: chown d6/d14/d18/d26/d32/da3/ld8 61 1 2024-04-22T15:19:10.432 INFO:tasks.workunit.client.1.smithi196.stdout:8/750: write d2/dc/f28 [73122,5402] 0 2024-04-22T15:19:10.453 INFO:tasks.workunit.client.1.smithi196.stdout:1/772: truncate d1/db/d16/d1e/d49/d90/fb4 3054311 0 2024-04-22T15:19:10.456 INFO:tasks.workunit.client.1.smithi196.stdout:9/762: dwrite db/dd/de/f10 [0,4194304] 0 2024-04-22T15:19:10.456 INFO:tasks.workunit.client.1.smithi196.stdout:3/820: mknod d0/d15/c113 0 2024-04-22T15:19:10.460 INFO:tasks.workunit.client.1.smithi196.stdout:1/773: dwrite d1/db/d16/f1d [0,4194304] 0 2024-04-22T15:19:10.462 INFO:tasks.workunit.client.0.smithi096.stdout:9/888: rename d1/df/d95/d9d/db5/d43/fe0 to d1/d25/f12e 0 2024-04-22T15:19:10.463 INFO:tasks.workunit.client.0.smithi096.stdout:9/889: readlink d1/df/d95/d9d/db5/dd/l6d 0 2024-04-22T15:19:10.463 INFO:tasks.workunit.client.1.smithi196.stdout:9/763: mkdir db/dd/de/dd7/def/d7a/dfe 0 2024-04-22T15:19:10.466 INFO:tasks.workunit.client.1.smithi196.stdout:1/774: mkdir d1/db/d16/d6d/dfe 0 2024-04-22T15:19:10.467 INFO:tasks.workunit.client.0.smithi096.stdout:8/963: rename d4/d10/d4d/deb/c130 to d4/d1c/d62/dc3/c146 0 2024-04-22T15:19:10.471 INFO:tasks.workunit.client.0.smithi096.stdout:9/890: dwrite d1/d25/f110 [0,4194304] 0 2024-04-22T15:19:10.472 INFO:tasks.workunit.client.0.smithi096.stdout:9/891: dread - d1/d4/d7f/f111 zero size 2024-04-22T15:19:10.472 INFO:tasks.workunit.client.1.smithi196.stdout:9/764: unlink db/dd/de/f42 0 2024-04-22T15:19:10.476 INFO:tasks.workunit.client.0.smithi096.stdout:9/892: truncate d1/df/d95/d9d/db5/dd/dba/dc2/d120/dae/dd9/fee 1284846 0 2024-04-22T15:19:10.477 INFO:tasks.workunit.client.0.smithi096.stdout:6/883: rename d3/d3a/d75/d83/lc1 to d3/d28/d54/d68/d6e/dbb/l12c 0 2024-04-22T15:19:10.478 INFO:tasks.workunit.client.1.smithi196.stdout:1/775: mknod d1/db/d97/dd7/cff 0 2024-04-22T15:19:10.481 INFO:tasks.workunit.client.0.smithi096.stdout:9/893: creat d1/d25/db2/dca/f12f x:0 0 0 2024-04-22T15:19:10.481 INFO:tasks.workunit.client.0.smithi096.stdout:9/894: chown d1/d85/f12a 3 1 2024-04-22T15:19:10.484 INFO:tasks.workunit.client.1.smithi196.stdout:1/776: fdatasync d1/db/d16/d1e/d65/f76 0 2024-04-22T15:19:10.488 INFO:tasks.workunit.client.0.smithi096.stdout:6/884: creat d3/d28/dc9/d122/f12d x:0 0 0 2024-04-22T15:19:10.490 INFO:tasks.workunit.client.0.smithi096.stdout:6/885: dread - d3/d28/dd/fba zero size 2024-04-22T15:19:10.492 INFO:tasks.workunit.client.0.smithi096.stdout:9/895: dwrite d1/d25/db2/d98/fd0 [0,4194304] 0 2024-04-22T15:19:10.496 INFO:tasks.workunit.client.0.smithi096.stdout:3/994: symlink da/db/de/ddc/l148 0 2024-04-22T15:19:10.497 INFO:tasks.workunit.client.0.smithi096.stdout:9/896: symlink d1/df/d95/d9d/db5/dd/dba/dc2/d120/dbd/l130 0 2024-04-22T15:19:10.499 INFO:tasks.workunit.client.0.smithi096.stdout:6/886: dwrite d3/d28/daf/d94/ffd [0,4194304] 0 2024-04-22T15:19:10.502 INFO:tasks.workunit.client.1.smithi196.stdout:9/765: dread db/dd/de/d15/d31/fb7 [0,4194304] 0 2024-04-22T15:19:10.507 INFO:tasks.workunit.client.1.smithi196.stdout:9/766: creat db/dd/de/dd7/def/d57/da7/fff x:0 0 0 2024-04-22T15:19:10.512 INFO:tasks.workunit.client.0.smithi096.stdout:6/887: getdents d3/d28/d4d/d82/d9e/da3 0 2024-04-22T15:19:10.513 INFO:tasks.workunit.client.0.smithi096.stdout:6/888: fdatasync d3/d28/d54/d68/f11a 0 2024-04-22T15:19:10.514 INFO:tasks.workunit.client.1.smithi196.stdout:9/767: truncate db/dd/de/d15/d31/d64/fad 2494430 0 2024-04-22T15:19:10.514 INFO:tasks.workunit.client.1.smithi196.stdout:9/768: read - db/dd/de/d15/d31/d99/ffd zero size 2024-04-22T15:19:10.515 INFO:tasks.workunit.client.0.smithi096.stdout:3/995: getdents da/db/dc/d59 0 2024-04-22T15:19:10.516 INFO:tasks.workunit.client.0.smithi096.stdout:3/996: write da/db/f3d [1331038,52124] 0 2024-04-22T15:19:10.518 INFO:tasks.workunit.client.0.smithi096.stdout:6/889: rename d3/d28/dd/d11/d2c/lcb to d3/d28/d4d/d82/dd8/l12e 0 2024-04-22T15:19:10.519 INFO:tasks.workunit.client.1.smithi196.stdout:9/769: dwrite fa [0,4194304] 0 2024-04-22T15:19:10.521 INFO:tasks.workunit.client.0.smithi096.stdout:3/997: rmdir da/d3f 39 2024-04-22T15:19:10.524 INFO:tasks.workunit.client.1.smithi196.stdout:9/770: mkdir db/dd/dd0/d100 0 2024-04-22T15:19:10.526 INFO:tasks.workunit.client.0.smithi096.stdout:6/890: mkdir d3/d28/dd/d11/d2c/d12f 0 2024-04-22T15:19:10.534 INFO:tasks.workunit.client.0.smithi096.stdout:3/998: dwrite da/db/de/d38/d119/f12c [0,4194304] 0 2024-04-22T15:19:10.535 INFO:tasks.workunit.client.0.smithi096.stdout:3/999: stat da/db/d10/f33 0 2024-04-22T15:19:10.536 INFO:tasks.workunit.client.0.smithi096.stdout:6/891: creat d3/d28/d54/d68/f130 x:0 0 0 2024-04-22T15:19:10.556 INFO:tasks.workunit.client.1.smithi196.stdout:5/802: truncate d1/d17/d1a/fe5 179005 0 2024-04-22T15:19:10.559 INFO:tasks.workunit.client.1.smithi196.stdout:5/803: getdents d1/d17/d7f/deb 0 2024-04-22T15:19:10.560 INFO:tasks.workunit.client.1.smithi196.stdout:5/804: write d1/d2/faf [385865,122134] 0 2024-04-22T15:19:10.562 INFO:tasks.workunit.client.1.smithi196.stdout:5/805: readlink d1/d14/d6a/lf7 0 2024-04-22T15:19:10.564 INFO:tasks.workunit.client.1.smithi196.stdout:7/775: rename d14/d15/c2c to d14/d25/d31/df0/c104 0 2024-04-22T15:19:10.564 INFO:tasks.workunit.client.1.smithi196.stdout:5/806: chown d1/df1/d6e/d7a/lee 13 1 2024-04-22T15:19:10.565 INFO:tasks.workunit.client.1.smithi196.stdout:2/783: dwrite d1/f5 [4194304,4194304] 0 2024-04-22T15:19:10.567 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:10 smithi096 ceph-mon[26688]: from='client.24437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:10.569 INFO:tasks.workunit.client.1.smithi196.stdout:6/790: rename dd/d1f/d37/d3c/fcc to dd/d1f/d4e/fe5 0 2024-04-22T15:19:10.569 INFO:tasks.workunit.client.1.smithi196.stdout:8/751: dwrite d2/dc/f20 [0,4194304] 0 2024-04-22T15:19:10.570 INFO:tasks.workunit.client.1.smithi196.stdout:5/807: mkdir d1/d78/dcf/dff 0 2024-04-22T15:19:10.571 INFO:tasks.workunit.client.1.smithi196.stdout:7/776: symlink d14/d4d/da3/l105 0 2024-04-22T15:19:10.581 INFO:tasks.workunit.client.1.smithi196.stdout:6/791: symlink dd/d1f/d37/d7b/le6 0 2024-04-22T15:19:10.583 INFO:tasks.workunit.client.1.smithi196.stdout:5/808: mkdir d1/d17/d1c/d100 0 2024-04-22T15:19:10.587 INFO:tasks.workunit.client.0.smithi096.stdout:8/964: getdents d4/d1c/d62/dc3 0 2024-04-22T15:19:10.588 INFO:tasks.workunit.client.1.smithi196.stdout:3/821: dwrite d0/d15/da7/d4a/d44/f6c [0,4194304] 0 2024-04-22T15:19:10.591 INFO:tasks.workunit.client.1.smithi196.stdout:0/798: rename d3/d6/dc/d8c/d8d/da8/f105 to d3/de/f11a 0 2024-04-22T15:19:10.591 INFO:tasks.workunit.client.1.smithi196.stdout:4/758: rename d6 to d6/d90/df3 22 2024-04-22T15:19:10.592 INFO:tasks.workunit.client.0.smithi096.stdout:8/965: symlink d4/d2a/l147 0 2024-04-22T15:19:10.594 INFO:tasks.workunit.client.1.smithi196.stdout:6/792: creat dd/d1f/d4e/d53/d55/d6d/dde/fe7 x:0 0 0 2024-04-22T15:19:10.599 INFO:tasks.workunit.client.1.smithi196.stdout:7/777: creat d14/d15/f106 x:0 0 0 2024-04-22T15:19:10.601 INFO:tasks.workunit.client.0.smithi096.stdout:8/966: mkdir d4/d1c/d40/d48/d148 0 2024-04-22T15:19:10.602 INFO:tasks.workunit.client.1.smithi196.stdout:5/809: mkdir d1/d78/dcf/dff/d101 0 2024-04-22T15:19:10.604 INFO:tasks.workunit.client.1.smithi196.stdout:3/822: creat d0/d15/d4d/de7/f114 x:0 0 0 2024-04-22T15:19:10.605 INFO:tasks.workunit.client.1.smithi196.stdout:3/823: write d0/f10f [850756,52626] 0 2024-04-22T15:19:10.606 INFO:tasks.workunit.client.1.smithi196.stdout:3/824: dread - d0/d15/d4d/d87/da6/fbd zero size 2024-04-22T15:19:10.607 INFO:tasks.workunit.client.0.smithi096.stdout:8/967: mknod d4/d1c/d40/da7/c149 0 2024-04-22T15:19:10.607 INFO:tasks.workunit.client.1.smithi196.stdout:5/810: dwrite d1/d17/f79 [4194304,4194304] 0 2024-04-22T15:19:10.608 INFO:tasks.workunit.client.1.smithi196.stdout:4/759: mknod d6/d13/d15/d2c/d62/d78/cf4 0 2024-04-22T15:19:10.609 INFO:tasks.workunit.client.1.smithi196.stdout:4/760: dread - d6/d14/d18/d26/d32/d53/db6/fd5 zero size 2024-04-22T15:19:10.610 INFO:tasks.workunit.client.0.smithi096.stdout:8/968: dread d4/d1c/de2/f114 [0,4194304] 0 2024-04-22T15:19:10.611 INFO:tasks.workunit.client.1.smithi196.stdout:0/799: mkdir d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/db7/d11b 0 2024-04-22T15:19:10.614 INFO:tasks.workunit.client.1.smithi196.stdout:1/777: dwrite d1/db/d16/f40 [0,4194304] 0 2024-04-22T15:19:10.619 INFO:tasks.workunit.client.1.smithi196.stdout:3/825: fsync d0/d15/d31/d55/d6e/fb2 0 2024-04-22T15:19:10.621 INFO:tasks.workunit.client.0.smithi096.stdout:8/969: mkdir d4/d14a 0 2024-04-22T15:19:10.622 INFO:tasks.workunit.client.1.smithi196.stdout:5/811: symlink d1/d17/d7f/deb/l102 0 2024-04-22T15:19:10.626 INFO:tasks.workunit.client.1.smithi196.stdout:3/826: dread d0/d15/da7/f1c [0,4194304] 0 2024-04-22T15:19:10.626 INFO:tasks.workunit.client.1.smithi196.stdout:4/761: symlink d6/d14/d18/d26/d32/da3/db7/lf5 0 2024-04-22T15:19:10.627 INFO:tasks.workunit.client.1.smithi196.stdout:6/793: mkdir dd/d1f/d4e/d53/d55/d6d/dd1/de3/de8 0 2024-04-22T15:19:10.629 INFO:tasks.workunit.client.0.smithi096.stdout:8/970: rmdir d4/d10 39 2024-04-22T15:19:10.631 INFO:tasks.workunit.client.1.smithi196.stdout:0/800: fdatasync d3/d9/d36/d9f/dd1/d65/d7f/f9a 0 2024-04-22T15:19:10.631 INFO:tasks.workunit.client.0.smithi096.stdout:9/897: dwrite d1/f6 [4194304,4194304] 0 2024-04-22T15:19:10.632 INFO:tasks.workunit.client.1.smithi196.stdout:0/801: fdatasync d3/d6/dc/f58 0 2024-04-22T15:19:10.634 INFO:tasks.workunit.client.1.smithi196.stdout:7/778: fsync d14/d4a/f71 0 2024-04-22T15:19:10.635 INFO:tasks.workunit.client.1.smithi196.stdout:9/771: truncate db/dd/de/d15/f6e 447812 0 2024-04-22T15:19:10.636 INFO:tasks.workunit.client.1.smithi196.stdout:1/778: fsync d1/db/dc/de/d43/f7b 0 2024-04-22T15:19:10.641 INFO:tasks.workunit.client.0.smithi096.stdout:9/898: unlink d1/df/d95/d9d/db5/d43/d57/d5b/da1/fc9 0 2024-04-22T15:19:10.645 INFO:tasks.workunit.client.0.smithi096.stdout:9/899: chown d1/df/d95/d9d/db5/d43/d57/f63 17 1 2024-04-22T15:19:10.646 INFO:tasks.workunit.client.0.smithi096.stdout:6/892: dwrite d3/d28/dd/d11/d2c/f4e [0,4194304] 0 2024-04-22T15:19:10.651 INFO:tasks.workunit.client.1.smithi196.stdout:3/827: mkdir d0/d15/d31/d55/dcf/d6f/d10a/d5e/d115 0 2024-04-22T15:19:10.654 INFO:tasks.workunit.client.1.smithi196.stdout:4/762: mkdir d6/d13/d15/d2c/d62/d78/df6 0 2024-04-22T15:19:10.657 INFO:tasks.workunit.client.1.smithi196.stdout:6/794: creat dd/d1f/d4e/d53/d55/d57/fe9 x:0 0 0 2024-04-22T15:19:10.657 INFO:tasks.workunit.client.1.smithi196.stdout:2/784: write d1/d13/d1f/f60 [340941,120519] 0 2024-04-22T15:19:10.658 INFO:tasks.workunit.client.0.smithi096.stdout:6/893: symlink d3/d28/dd/d11/d2c/d6f/l131 0 2024-04-22T15:19:10.661 INFO:tasks.workunit.client.1.smithi196.stdout:1/779: dread d1/db/d16/d44/f92 [0,4194304] 0 2024-04-22T15:19:10.663 INFO:tasks.workunit.client.0.smithi096.stdout:6/894: fsync d3/d28/dd/d11/d2c/d6f/dd4/ddd/de9/fe5 0 2024-04-22T15:19:10.665 INFO:tasks.workunit.client.1.smithi196.stdout:8/752: dwrite d2/d3f/d4b/f9b [0,4194304] 0 2024-04-22T15:19:10.666 INFO:tasks.workunit.client.1.smithi196.stdout:8/753: write d2/d5/d2e/f43 [360783,58758] 0 2024-04-22T15:19:10.670 INFO:tasks.workunit.client.1.smithi196.stdout:9/772: rmdir db/dd/de/dd7 39 2024-04-22T15:19:10.674 INFO:tasks.workunit.client.0.smithi096.stdout:6/895: mkdir d3/d28/d132 0 2024-04-22T15:19:10.675 INFO:tasks.workunit.client.0.smithi096.stdout:6/896: chown d3/d28/dc9/fec 1262136 1 2024-04-22T15:19:10.676 INFO:tasks.workunit.client.1.smithi196.stdout:9/773: dwrite db/dd/de/d15/fe0 [0,4194304] 0 2024-04-22T15:19:10.676 INFO:tasks.workunit.client.0.smithi096.stdout:6/897: fdatasync d3/d3a/fe3 0 2024-04-22T15:19:10.676 INFO:tasks.workunit.client.0.smithi096.stdout:6/898: chown d3/f49 675820118 1 2024-04-22T15:19:10.678 INFO:tasks.workunit.client.1.smithi196.stdout:5/812: mknod d1/df1/d5e/dcb/c103 0 2024-04-22T15:19:10.679 INFO:tasks.workunit.client.0.smithi096.stdout:6/899: readlink d3/l45 0 2024-04-22T15:19:10.680 INFO:tasks.workunit.client.0.smithi096.stdout:6/900: chown d3/d28/dd/d11/d2c/d6f/dd4/dd6 249 1 2024-04-22T15:19:10.681 INFO:tasks.workunit.client.1.smithi196.stdout:2/785: chown d1/d23/lae 554 1 2024-04-22T15:19:10.683 INFO:tasks.workunit.client.1.smithi196.stdout:6/795: mkdir dd/d1f/d4e/d53/d7f/dea 0 2024-04-22T15:19:10.688 INFO:tasks.workunit.client.0.smithi096.stdout:6/901: link d3/d3a/d75/d83/ff2 d3/d28/d4d/d82/d9e/db0/f133 0 2024-04-22T15:19:10.692 INFO:tasks.workunit.client.0.smithi096.stdout:6/902: symlink d3/d3a/l134 0 2024-04-22T15:19:10.694 INFO:tasks.workunit.client.1.smithi196.stdout:7/779: fdatasync d14/f1e 0 2024-04-22T15:19:10.698 INFO:tasks.workunit.client.1.smithi196.stdout:7/780: dwrite f7 [0,4194304] 0 2024-04-22T15:19:10.699 INFO:tasks.workunit.client.1.smithi196.stdout:9/774: mkdir db/dd/de/d98/da9/d101 0 2024-04-22T15:19:10.700 INFO:tasks.workunit.client.1.smithi196.stdout:3/828: symlink d0/d15/d31/dcb/l116 0 2024-04-22T15:19:10.702 INFO:tasks.workunit.client.0.smithi096.stdout:8/971: dwrite d4/d10/d25/d4c/d76/f89 [0,4194304] 0 2024-04-22T15:19:10.703 INFO:tasks.workunit.client.1.smithi196.stdout:4/763: link d6/d14/d18/d26/d32/d67/d9a/da5/fac d6/d14/d18/d26/d32/d53/ff7 0 2024-04-22T15:19:10.704 INFO:tasks.workunit.client.1.smithi196.stdout:4/764: chown d6/d13/d15/d2c/d8c/db3/dce 10 1 2024-04-22T15:19:10.713 INFO:tasks.workunit.client.0.smithi096.stdout:8/972: creat d4/d10/d25/d38/d8f/d9c/d138/f14b x:0 0 0 2024-04-22T15:19:10.713 INFO:tasks.workunit.client.1.smithi196.stdout:1/780: getdents d1/db/d16/d8e/ddd/df6 0 2024-04-22T15:19:10.714 INFO:tasks.workunit.client.1.smithi196.stdout:0/802: link d3/d9/d36/d9f/dd1/d65/d7f/l9d d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9/l11c 0 2024-04-22T15:19:10.715 INFO:tasks.workunit.client.0.smithi096.stdout:9/900: dwrite d1/df/d95/d9d/db5/dd/dba/dc2/d120/dbd/fd6 [0,4194304] 0 2024-04-22T15:19:10.715 INFO:tasks.workunit.client.1.smithi196.stdout:8/754: truncate d2/dc/fd 2850069 0 2024-04-22T15:19:10.721 INFO:tasks.workunit.client.0.smithi096.stdout:8/973: truncate d4/d1c/de2/f11e 268795 0 2024-04-22T15:19:10.722 INFO:tasks.workunit.client.0.smithi096.stdout:8/974: dread - d4/d1c/d40/f12f zero size 2024-04-22T15:19:10.730 INFO:tasks.workunit.client.0.smithi096.stdout:9/901: mknod d1/d25/db2/d98/c131 0 2024-04-22T15:19:10.731 INFO:tasks.workunit.client.1.smithi196.stdout:6/796: link dd/d1f/d4e/d53/d7f/da5/fdc dd/d1f/d4e/d53/d7f/dea/feb 0 2024-04-22T15:19:10.733 INFO:tasks.workunit.client.1.smithi196.stdout:1/781: rmdir d1/db/dc/de/d43/d5e/d7a 39 2024-04-22T15:19:10.734 INFO:tasks.workunit.client.0.smithi096.stdout:9/902: creat d1/d4/d7b/f132 x:0 0 0 2024-04-22T15:19:10.735 INFO:tasks.workunit.client.0.smithi096.stdout:6/903: write d3/d28/dd/d11/f1e [653064,104657] 0 2024-04-22T15:19:10.736 INFO:tasks.workunit.client.1.smithi196.stdout:1/782: read d1/db/dc/de/d43/ff5 [3740585,112771] 0 2024-04-22T15:19:10.736 INFO:tasks.workunit.client.1.smithi196.stdout:8/755: creat d2/d3f/d4b/fdb x:0 0 0 2024-04-22T15:19:10.738 INFO:tasks.workunit.client.1.smithi196.stdout:6/797: dwrite dd/d2b/d8d/fb2 [0,4194304] 0 2024-04-22T15:19:10.742 INFO:tasks.workunit.client.1.smithi196.stdout:9/775: creat db/dd/de/dd7/f102 x:0 0 0 2024-04-22T15:19:10.744 INFO:tasks.workunit.client.0.smithi096.stdout:6/904: symlink d3/d28/dd/d11/d2c/l135 0 2024-04-22T15:19:10.747 INFO:tasks.workunit.client.0.smithi096.stdout:6/905: stat d3/d28/d54/d68/l11b 0 2024-04-22T15:19:10.747 INFO:tasks.workunit.client.1.smithi196.stdout:1/783: dwrite d1/db/dc/d26/d2e/f34 [4194304,4194304] 0 2024-04-22T15:19:10.751 INFO:tasks.workunit.client.0.smithi096.stdout:6/906: unlink d3/d28/daf/d94/cd2 0 2024-04-22T15:19:10.752 INFO:tasks.workunit.client.0.smithi096.stdout:8/975: write d4/d1c/de2/ff3 [580703,18779] 0 2024-04-22T15:19:10.753 INFO:tasks.workunit.client.0.smithi096.stdout:6/907: write d3/d28/d4d/ff9 [954737,125486] 0 2024-04-22T15:19:10.754 INFO:tasks.workunit.client.1.smithi196.stdout:8/756: dread d2/f3e [0,4194304] 0 2024-04-22T15:19:10.754 INFO:tasks.workunit.client.1.smithi196.stdout:2/786: dwrite d1/d23/d27/d2e/d55/d5b/fb3 [0,4194304] 0 2024-04-22T15:19:10.754 INFO:tasks.workunit.client.1.smithi196.stdout:3/829: write d0/d15/da7/d4a/d44/f3c [764823,25308] 0 2024-04-22T15:19:10.754 INFO:tasks.workunit.client.1.smithi196.stdout:3/830: fsync d0/d15/d4d/de7/f114 0 2024-04-22T15:19:10.754 INFO:tasks.workunit.client.1.smithi196.stdout:2/787: dread d1/d23/d27/d2e/f85 [0,4194304] 0 2024-04-22T15:19:10.755 INFO:tasks.workunit.client.1.smithi196.stdout:7/781: rmdir d14/d26/d2a/d2b/dc7/df4 0 2024-04-22T15:19:10.756 INFO:tasks.workunit.client.1.smithi196.stdout:6/798: fsync dd/d1f/f60 0 2024-04-22T15:19:10.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:10 smithi196 ceph-mon[34717]: from='client.24437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:10.759 INFO:tasks.workunit.client.0.smithi096.stdout:6/908: symlink d3/d3a/d75/d83/l136 0 2024-04-22T15:19:10.763 INFO:tasks.workunit.client.0.smithi096.stdout:8/976: mknod d4/d1c/d40/c14c 0 2024-04-22T15:19:10.763 INFO:tasks.workunit.client.0.smithi096.stdout:6/909: write d3/d28/d4d/d82/dd8/fe1 [517408,40964] 0 2024-04-22T15:19:10.765 INFO:tasks.workunit.client.1.smithi196.stdout:9/776: truncate db/dd/de/dd7/def/d57/f5a 566212 0 2024-04-22T15:19:10.765 INFO:tasks.workunit.client.0.smithi096.stdout:9/903: dwrite d1/f4f [0,4194304] 0 2024-04-22T15:19:10.767 INFO:tasks.workunit.client.1.smithi196.stdout:0/803: rename d3/d9/d36/d9f/dd1/d65/d7f/d30/f46 to d3/d9/d36/f11d 0 2024-04-22T15:19:10.770 INFO:tasks.workunit.client.0.smithi096.stdout:9/904: mkdir d1/df/d95/d9d/db5/dd/dba/dc2/de5/d133 0 2024-04-22T15:19:10.772 INFO:tasks.workunit.client.1.smithi196.stdout:8/757: rmdir d2 39 2024-04-22T15:19:10.774 INFO:tasks.workunit.client.0.smithi096.stdout:9/905: unlink d1/df/d95/d9d/db5/dd/dba/dc2/d120/faa 0 2024-04-22T15:19:10.778 INFO:tasks.workunit.client.1.smithi196.stdout:3/831: dwrite d0/d15/d31/d55/dcf/d8/d6b/fd6 [0,4194304] 0 2024-04-22T15:19:10.778 INFO:tasks.workunit.client.1.smithi196.stdout:1/784: mkdir d1/db/d16/d1e/d100 0 2024-04-22T15:19:10.780 INFO:tasks.workunit.client.1.smithi196.stdout:1/785: fsync d1/db/d16/d8e/ddd/fee 0 2024-04-22T15:19:10.785 INFO:tasks.workunit.client.1.smithi196.stdout:6/799: dread dd/d1f/d38/d40/fcd [0,4194304] 0 2024-04-22T15:19:10.787 INFO:tasks.workunit.client.1.smithi196.stdout:2/788: rename d1/da4/ld9 to d1/d23/d27/d2e/db5/l11c 0 2024-04-22T15:19:10.788 INFO:tasks.workunit.client.1.smithi196.stdout:3/832: mkdir d0/d15/d31/d55/dcf/d6f/de5/d101/d9e/d117 0 2024-04-22T15:19:10.790 INFO:tasks.workunit.client.1.smithi196.stdout:0/804: symlink d3/d9/l11e 0 2024-04-22T15:19:10.791 INFO:tasks.workunit.client.1.smithi196.stdout:1/786: truncate d1/db/dc/de/d43/d5e/d7a/fa1 912736 0 2024-04-22T15:19:10.792 INFO:tasks.workunit.client.1.smithi196.stdout:8/758: unlink d2/d8/d49/d8b/fd6 0 2024-04-22T15:19:10.793 INFO:tasks.workunit.client.1.smithi196.stdout:8/759: chown d2/d8/f27 0 1 2024-04-22T15:19:10.797 INFO:tasks.workunit.client.1.smithi196.stdout:1/787: creat d1/db/dc/d93/f101 x:0 0 0 2024-04-22T15:19:10.800 INFO:tasks.workunit.client.1.smithi196.stdout:8/760: symlink d2/d5/d1a/ldc 0 2024-04-22T15:19:10.802 INFO:tasks.workunit.client.1.smithi196.stdout:3/833: symlink d0/d15/d31/d55/dcf/d6f/de5/d101/l118 0 2024-04-22T15:19:10.802 INFO:tasks.workunit.client.0.smithi096.stdout:8/977: dread d4/d2a/d7e/fa6 [0,4194304] 0 2024-04-22T15:19:10.803 INFO:tasks.workunit.client.1.smithi196.stdout:3/834: write d0/d15/d31/d55/f74 [4347037,47800] 0 2024-04-22T15:19:10.804 INFO:tasks.workunit.client.0.smithi096.stdout:6/910: dwrite d3/d28/dd/d11/d2c/f91 [4194304,4194304] 0 2024-04-22T15:19:10.804 INFO:tasks.workunit.client.1.smithi196.stdout:3/835: fsync d0/d15/d31/d55/dcf/d8/d70/f7e 0 2024-04-22T15:19:10.805 INFO:tasks.workunit.client.1.smithi196.stdout:1/788: symlink d1/db/dc/de/d43/l102 0 2024-04-22T15:19:10.805 INFO:tasks.workunit.client.0.smithi096.stdout:6/911: chown d3/d28/dd/d11/d2c/d106 52181265 1 2024-04-22T15:19:10.807 INFO:tasks.workunit.client.1.smithi196.stdout:6/800: creat dd/d20/fec x:0 0 0 2024-04-22T15:19:10.807 INFO:tasks.workunit.client.0.smithi096.stdout:9/906: write d1/df/d95/d9d/db5/d43/f5f [6928599,36682] 0 2024-04-22T15:19:10.808 INFO:tasks.workunit.client.1.smithi196.stdout:9/777: truncate db/d11/fea 822914 0 2024-04-22T15:19:10.811 INFO:tasks.workunit.client.1.smithi196.stdout:0/805: truncate d3/d9/d36/f11d 514573 0 2024-04-22T15:19:10.813 INFO:tasks.workunit.client.1.smithi196.stdout:1/789: dwrite d1/db/dc/de/d5a/d94/dab/fd0 [0,4194304] 0 2024-04-22T15:19:10.814 INFO:tasks.workunit.client.1.smithi196.stdout:1/790: readlink d1/db/dc/de/d5a/d5b/lc5 0 2024-04-22T15:19:10.819 INFO:tasks.workunit.client.1.smithi196.stdout:3/836: unlink d0/d15/d31/d55/d8b/c93 0 2024-04-22T15:19:10.819 INFO:tasks.workunit.client.1.smithi196.stdout:2/789: dwrite d1/d13/d41/f6c [0,4194304] 0 2024-04-22T15:19:10.821 INFO:tasks.workunit.client.0.smithi096.stdout:6/912: dwrite d3/d3a/d75/f128 [0,4194304] 0 2024-04-22T15:19:10.830 INFO:tasks.workunit.client.1.smithi196.stdout:0/806: mknod d3/d9/d36/d9f/dd1/d65/d7f/d51/c11f 0 2024-04-22T15:19:10.835 INFO:tasks.workunit.client.1.smithi196.stdout:3/837: symlink d0/d15/d31/d55/dcf/d6f/d10a/ddd/l119 0 2024-04-22T15:19:10.835 INFO:tasks.workunit.client.0.smithi096.stdout:8/978: dwrite d4/d10/d25/d38/f50 [0,4194304] 0 2024-04-22T15:19:10.841 INFO:tasks.workunit.client.0.smithi096.stdout:6/913: rename d3/d28/dd to d3/d28/d4d/d82/d9e/d137 0 2024-04-22T15:19:10.843 INFO:tasks.workunit.client.1.smithi196.stdout:2/790: rename d1/d23/d27/d2e/d55/lcb to d1/d76/l11d 0 2024-04-22T15:19:10.847 INFO:tasks.workunit.client.0.smithi096.stdout:6/914: unlink d3/d28/daf/d94/f10e 0 2024-04-22T15:19:10.849 INFO:tasks.workunit.client.1.smithi196.stdout:8/761: creat d2/dc/fdd x:0 0 0 2024-04-22T15:19:10.850 INFO:tasks.workunit.client.0.smithi096.stdout:6/915: chown d3/d3a/l65 419916866 1 2024-04-22T15:19:10.851 INFO:tasks.workunit.client.0.smithi096.stdout:9/907: symlink d1/df/d95/d9d/db5/d43/d57/l134 0 2024-04-22T15:19:10.853 INFO:tasks.workunit.client.1.smithi196.stdout:3/838: creat d0/d15/da7/f11a x:0 0 0 2024-04-22T15:19:10.854 INFO:tasks.workunit.client.1.smithi196.stdout:2/791: fdatasync d1/d23/d27/d2e/d55/fa6 0 2024-04-22T15:19:10.856 INFO:tasks.workunit.client.1.smithi196.stdout:1/791: write d1/db/dc/de/d43/d5e/fa7 [366407,124829] 0 2024-04-22T15:19:10.858 INFO:tasks.workunit.client.1.smithi196.stdout:6/801: rmdir dd/d1f/dc6 0 2024-04-22T15:19:10.858 INFO:tasks.workunit.client.1.smithi196.stdout:5/813: sync 2024-04-22T15:19:10.859 INFO:tasks.workunit.client.1.smithi196.stdout:5/814: fsync d1/d9/f11 0 2024-04-22T15:19:10.859 INFO:tasks.workunit.client.1.smithi196.stdout:6/802: chown dd/d1f/d38/dd9/fd0 6515068 1 2024-04-22T15:19:10.860 INFO:tasks.workunit.client.0.smithi096.stdout:6/916: creat d3/d28/d4d/d82/d9e/d137/f138 x:0 0 0 2024-04-22T15:19:10.863 INFO:tasks.workunit.client.1.smithi196.stdout:9/778: rename db/dd/de/d15/d31/lc2 to db/dd/de/d15/d31/l103 0 2024-04-22T15:19:10.863 INFO:tasks.workunit.client.0.smithi096.stdout:9/908: symlink d1/d4/d7f/d12d/l135 0 2024-04-22T15:19:10.864 INFO:tasks.workunit.client.1.smithi196.stdout:8/762: mkdir d2/d8/d49/d6c/dde 0 2024-04-22T15:19:10.866 INFO:tasks.workunit.client.1.smithi196.stdout:6/803: dwrite dd/f86 [0,4194304] 0 2024-04-22T15:19:10.866 INFO:tasks.workunit.client.1.smithi196.stdout:5/815: dwrite d1/d2/d98/fb6 [0,4194304] 0 2024-04-22T15:19:10.868 INFO:tasks.workunit.client.1.smithi196.stdout:0/807: truncate d3/f47 1351895 0 2024-04-22T15:19:10.871 INFO:tasks.workunit.client.1.smithi196.stdout:3/839: unlink d0/f1 0 2024-04-22T15:19:10.873 INFO:tasks.workunit.client.1.smithi196.stdout:3/840: read d0/d15/f7a [2725759,19297] 0 2024-04-22T15:19:10.876 INFO:tasks.workunit.client.0.smithi096.stdout:9/909: creat d1/f136 x:0 0 0 2024-04-22T15:19:10.877 INFO:tasks.workunit.client.1.smithi196.stdout:2/792: readlink d1/d13/d25/d2a/ddb/lfb 0 2024-04-22T15:19:10.881 INFO:tasks.workunit.client.0.smithi096.stdout:9/910: symlink d1/df/d95/d9d/db5/dd/dba/dc2/d120/l137 0 2024-04-22T15:19:10.885 INFO:tasks.workunit.client.0.smithi096.stdout:9/911: mknod d1/df/d95/d9d/db5/dd/dba/dc2/d120/dae/c138 0 2024-04-22T15:19:10.892 INFO:tasks.workunit.client.0.smithi096.stdout:9/912: unlink d1/df/d95/d9d/db5/dd/dba/dc2/d120/dbd/fd6 0 2024-04-22T15:19:10.893 INFO:tasks.workunit.client.0.smithi096.stdout:9/913: fsync d1/df/d95/d9d/db5/dd/f23 0 2024-04-22T15:19:10.894 INFO:tasks.workunit.client.0.smithi096.stdout:9/914: dread - d1/d25/db2/dca/f12f zero size 2024-04-22T15:19:10.908 INFO:tasks.workunit.client.0.smithi096.stdout:9/915: dread d1/d4/d7b/fec [0,4194304] 0 2024-04-22T15:19:10.909 INFO:tasks.workunit.client.0.smithi096.stdout:9/916: creat d1/d25/db2/dca/f139 x:0 0 0 2024-04-22T15:19:10.913 INFO:tasks.workunit.client.0.smithi096.stdout:9/917: dread d1/d25/f4c [0,4194304] 0 2024-04-22T15:19:10.915 INFO:tasks.workunit.client.0.smithi096.stdout:9/918: mknod d1/df/d95/d9d/db5/dd/dba/dc2/d120/dae/c13a 0 2024-04-22T15:19:10.916 INFO:tasks.workunit.client.0.smithi096.stdout:9/919: chown d1/d4/d7b/f132 12062957 1 2024-04-22T15:19:10.917 INFO:tasks.workunit.client.1.smithi196.stdout:7/782: sync 2024-04-22T15:19:10.918 INFO:tasks.workunit.client.1.smithi196.stdout:4/765: sync 2024-04-22T15:19:10.918 INFO:tasks.workunit.client.0.smithi096.stdout:9/920: truncate d1/df/d95/d9d/db5/d43/d57/d5b/d73/ff2 134867 0 2024-04-22T15:19:10.920 INFO:tasks.workunit.client.0.smithi096.stdout:9/921: mknod d1/d25/db2/d98/c13b 0 2024-04-22T15:19:10.920 INFO:tasks.workunit.client.0.smithi096.stdout:9/922: write d1/df/d95/d9d/db5/f116 [345514,120117] 0 2024-04-22T15:19:10.922 INFO:tasks.workunit.client.0.smithi096.stdout:9/923: read d1/df/d95/d9d/db5/f12 [1540579,20433] 0 2024-04-22T15:19:10.928 INFO:tasks.workunit.client.0.smithi096.stdout:9/924: write d1/f8a [2351963,103948] 0 2024-04-22T15:19:10.929 INFO:tasks.workunit.client.1.smithi196.stdout:5/816: rename d1/d78/fd2 to d1/df1/d5e/dcb/f104 0 2024-04-22T15:19:10.930 INFO:tasks.workunit.client.1.smithi196.stdout:7/783: readlink d14/d26/d28/d29/lfb 0 2024-04-22T15:19:10.931 INFO:tasks.workunit.client.1.smithi196.stdout:1/792: write d1/db/dc/de/d43/f7b [1658542,84636] 0 2024-04-22T15:19:10.932 INFO:tasks.workunit.client.1.smithi196.stdout:0/808: symlink d3/d9/d36/d9f/dd1/d65/d107/l120 0 2024-04-22T15:19:10.934 INFO:tasks.workunit.client.1.smithi196.stdout:2/793: mkdir d1/d23/d27/d2e/d55/d5b/d93/d97/d11e 0 2024-04-22T15:19:10.935 INFO:tasks.workunit.client.0.smithi096.stdout:9/925: dwrite d1/df/d95/d9d/db5/f116 [0,4194304] 0 2024-04-22T15:19:10.935 INFO:tasks.workunit.client.1.smithi196.stdout:4/766: stat d6/d13/d15/d2c/d62/f5a 0 2024-04-22T15:19:10.937 INFO:tasks.workunit.client.1.smithi196.stdout:3/841: mkdir d0/d15/d31/d55/dcf/d6f/d10a/d5e/d115/d11b 0 2024-04-22T15:19:10.940 INFO:tasks.workunit.client.0.smithi096.stdout:9/926: readlink d1/df/d95/d9d/db5/d43/l7c 0 2024-04-22T15:19:10.941 INFO:tasks.workunit.client.1.smithi196.stdout:7/784: rmdir d14/d15/d91 39 2024-04-22T15:19:10.943 INFO:tasks.workunit.client.1.smithi196.stdout:4/767: dwrite d6/d13/d15/fa6 [0,4194304] 0 2024-04-22T15:19:10.944 INFO:tasks.workunit.client.1.smithi196.stdout:9/779: creat db/dd/de/dd7/def/f104 x:0 0 0 2024-04-22T15:19:10.946 INFO:tasks.workunit.client.1.smithi196.stdout:4/768: read d6/d14/d18/d26/d32/da3/db7/fef [850479,14286] 0 2024-04-22T15:19:10.948 INFO:tasks.workunit.client.1.smithi196.stdout:2/794: creat d1/d23/d27/d2e/d55/f11f x:0 0 0 2024-04-22T15:19:10.955 INFO:tasks.workunit.client.0.smithi096.stdout:9/927: link d1/df/d95/d9d/db5/d43/d57/l134 d1/df/d95/d9d/db5/dd/dba/dc2/d120/l13c 0 2024-04-22T15:19:10.955 INFO:tasks.workunit.client.0.smithi096.stdout:9/928: chown d1/f4f 997512 1 2024-04-22T15:19:10.957 INFO:tasks.workunit.client.0.smithi096.stdout:9/929: chown d1/df/d95/d9d/db5/d43/d57/f113 6182 1 2024-04-22T15:19:10.957 INFO:tasks.workunit.client.1.smithi196.stdout:4/769: creat d6/d14/d18/d40/ff8 x:0 0 0 2024-04-22T15:19:10.958 INFO:tasks.workunit.client.1.smithi196.stdout:0/809: creat d3/d109/f121 x:0 0 0 2024-04-22T15:19:10.960 INFO:tasks.workunit.client.1.smithi196.stdout:7/785: unlink d14/l4e 0 2024-04-22T15:19:10.960 INFO:tasks.workunit.client.1.smithi196.stdout:5/817: dwrite d1/df1/f54 [0,4194304] 0 2024-04-22T15:19:10.961 INFO:tasks.workunit.client.0.smithi096.stdout:9/930: rename d1/df/d95/d9d/db5/dd/dba/dc2/d120/dae/lbe to d1/df/d95/d9d/db5/dd/dba/dc2/d120/dae/dd9/l13d 0 2024-04-22T15:19:10.962 INFO:tasks.workunit.client.1.smithi196.stdout:1/793: dwrite d1/db/dc/de/d43/f48 [0,4194304] 0 2024-04-22T15:19:10.963 INFO:tasks.workunit.client.0.smithi096.stdout:9/931: write d1/df/f65 [1188284,34760] 0 2024-04-22T15:19:10.963 INFO:tasks.workunit.client.1.smithi196.stdout:2/795: mknod d1/d23/d27/d2e/d55/d5b/d93/d97/d10e/c120 0 2024-04-22T15:19:10.967 INFO:tasks.workunit.client.1.smithi196.stdout:9/780: creat db/dd/de/dd7/def/f105 x:0 0 0 2024-04-22T15:19:10.972 INFO:tasks.workunit.client.1.smithi196.stdout:4/770: rename d6/d14/d18/d26/d32/d53/db6 to d6/d90/df9 0 2024-04-22T15:19:10.975 INFO:tasks.workunit.client.1.smithi196.stdout:7/786: symlink d14/d26/d2a/d2b/dc7/l107 0 2024-04-22T15:19:10.975 INFO:tasks.workunit.client.1.smithi196.stdout:3/842: dread d0/d15/d31/d55/dcf/d6f/d10a/fb7 [0,4194304] 0 2024-04-22T15:19:10.975 INFO:tasks.workunit.client.1.smithi196.stdout:1/794: dwrite d1/fb2 [0,4194304] 0 2024-04-22T15:19:10.977 INFO:tasks.workunit.client.1.smithi196.stdout:1/795: write d1/db/dc/dad/d36/fc9 [681463,85768] 0 2024-04-22T15:19:10.979 INFO:tasks.workunit.client.1.smithi196.stdout:5/818: rename d1/d9 to d1/d2/d98/dac/d105 0 2024-04-22T15:19:10.981 INFO:tasks.workunit.client.1.smithi196.stdout:2/796: unlink d1/d23/d27/d2e/d55/d5b/le7 0 2024-04-22T15:19:10.991 INFO:tasks.workunit.client.1.smithi196.stdout:3/843: unlink d0/d15/da7/d4a/d44/d8c/faa 0 2024-04-22T15:19:10.991 INFO:tasks.workunit.client.0.smithi096.stdout:9/932: write d1/d4/f11 [1432277,118181] 0 2024-04-22T15:19:10.993 INFO:tasks.workunit.client.0.smithi096.stdout:9/933: chown d1/df/d95/d9d/db5/dd/dba/dc2/d120/ce2 1088 1 2024-04-22T15:19:10.995 INFO:tasks.workunit.client.1.smithi196.stdout:0/810: dread d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/fa3 [0,4194304] 0 2024-04-22T15:19:10.996 INFO:tasks.workunit.client.1.smithi196.stdout:4/771: getdents d6/d13 0 2024-04-22T15:19:10.998 INFO:tasks.workunit.client.1.smithi196.stdout:4/772: dread - d6/d13/d15/d2c/d62/d78/d8d/fbd zero size 2024-04-22T15:19:10.998 INFO:tasks.workunit.client.1.smithi196.stdout:2/797: mkdir d1/d23/d27/d73/d121 0 2024-04-22T15:19:10.999 INFO:tasks.workunit.client.1.smithi196.stdout:3/844: mkdir d0/d15/d4d/db/d11c 0 2024-04-22T15:19:11.000 INFO:tasks.workunit.client.1.smithi196.stdout:9/781: dread db/dd/de/dd7/def/d57/f23 [0,4194304] 0 2024-04-22T15:19:11.001 INFO:tasks.workunit.client.1.smithi196.stdout:2/798: mkdir d1/d1d/d122 0 2024-04-22T15:19:11.004 INFO:tasks.workunit.client.1.smithi196.stdout:0/811: mknod d3/de/d1c/dd4/c122 0 2024-04-22T15:19:11.006 INFO:tasks.workunit.client.1.smithi196.stdout:9/782: mknod db/dd/de/dd7/def/d7a/d7b/c106 0 2024-04-22T15:19:11.007 INFO:tasks.workunit.client.1.smithi196.stdout:2/799: fdatasync d1/d23/d27/d89/f8c 0 2024-04-22T15:19:11.009 INFO:tasks.workunit.client.1.smithi196.stdout:7/787: dwrite d14/d26/d2a/f66 [4194304,4194304] 0 2024-04-22T15:19:11.011 INFO:tasks.workunit.client.0.smithi096.stdout:9/934: dwrite d1/df/d95/fd4 [0,4194304] 0 2024-04-22T15:19:11.011 INFO:tasks.workunit.client.1.smithi196.stdout:3/845: creat d0/d15/f11d x:0 0 0 2024-04-22T15:19:11.013 INFO:tasks.workunit.client.1.smithi196.stdout:2/800: fdatasync d1/d13/d1f/ddf/da1/ff7 0 2024-04-22T15:19:11.014 INFO:tasks.workunit.client.1.smithi196.stdout:5/819: dwrite d1/df1/d6e/fa2 [0,4194304] 0 2024-04-22T15:19:11.015 INFO:tasks.workunit.client.1.smithi196.stdout:8/763: sync 2024-04-22T15:19:11.015 INFO:tasks.workunit.client.1.smithi196.stdout:1/796: dwrite d1/db/dc/dad/f9b [4194304,4194304] 0 2024-04-22T15:19:11.025 INFO:tasks.workunit.client.1.smithi196.stdout:4/773: dwrite d6/d14/d18/d26/d32/d53/ff7 [0,4194304] 0 2024-04-22T15:19:11.025 INFO:tasks.workunit.client.1.smithi196.stdout:9/783: creat db/dd/f107 x:0 0 0 2024-04-22T15:19:11.026 INFO:tasks.workunit.client.0.smithi096.stdout:9/935: write d1/df/d95/d9d/db5/dd/dba/dc2/fc6 [659224,87037] 0 2024-04-22T15:19:11.032 INFO:tasks.workunit.client.1.smithi196.stdout:7/788: write d14/d26/d2a/d72/db1/fd9 [19911,70080] 0 2024-04-22T15:19:11.035 INFO:tasks.workunit.client.1.smithi196.stdout:3/846: write d0/d15/da7/f1c [508823,35860] 0 2024-04-22T15:19:11.038 INFO:tasks.workunit.client.1.smithi196.stdout:1/797: dwrite d1/db/dc/de/d96/d9d/fe0 [0,4194304] 0 2024-04-22T15:19:11.053 INFO:tasks.workunit.client.1.smithi196.stdout:6/804: sync 2024-04-22T15:19:11.157 INFO:tasks.workunit.client.1.smithi196.stdout:0/812: rename d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/dd0/de3/l10e to d3/de/d1c/l123 0 2024-04-22T15:19:11.158 INFO:tasks.workunit.client.1.smithi196.stdout:0/813: chown d3/d9/d36/d9f/dd1/d65/d7f/d87/l3e 8 1 2024-04-22T15:19:11.159 INFO:tasks.workunit.client.1.smithi196.stdout:8/764: creat d2/d5/d93/dd5/fdf x:0 0 0 2024-04-22T15:19:11.159 INFO:tasks.workunit.client.1.smithi196.stdout:4/774: readlink d6/d14/d18/d40/d85/la7 0 2024-04-22T15:19:11.163 INFO:tasks.workunit.client.1.smithi196.stdout:2/801: fsync d1/d23/d27/d73/fda 0 2024-04-22T15:19:11.163 INFO:tasks.workunit.client.1.smithi196.stdout:5/820: truncate d1/d4/f56 982762 0 2024-04-22T15:19:11.166 INFO:tasks.workunit.client.1.smithi196.stdout:6/805: unlink dd/d1f/d37/f34 0 2024-04-22T15:19:11.166 INFO:tasks.workunit.client.1.smithi196.stdout:8/765: dwrite d2/d3f/d4b/fdb [0,4194304] 0 2024-04-22T15:19:11.167 INFO:tasks.workunit.client.0.smithi096.stdout:9/936: dwrite d1/df/d95/d9d/db5/d43/d57/d5b/fb4 [0,4194304] 0 2024-04-22T15:19:11.167 INFO:tasks.workunit.client.1.smithi196.stdout:9/784: rename db/dd/de/d15/ld9 to db/d11/l108 0 2024-04-22T15:19:11.168 INFO:tasks.workunit.client.1.smithi196.stdout:8/766: stat d2/d5/d1a/c2b 0 2024-04-22T15:19:11.170 INFO:tasks.workunit.client.1.smithi196.stdout:7/789: dread d14/d4a/f8c [0,4194304] 0 2024-04-22T15:19:11.172 INFO:tasks.workunit.client.1.smithi196.stdout:5/821: mknod d1/df1/d5e/c106 0 2024-04-22T15:19:11.178 INFO:tasks.workunit.client.1.smithi196.stdout:2/802: truncate d1/d1d/f3c 4673086 0 2024-04-22T15:19:11.179 INFO:tasks.workunit.client.1.smithi196.stdout:4/775: symlink d6/d13/d15/d2c/d62/d78/ddc/lfa 0 2024-04-22T15:19:11.181 INFO:tasks.workunit.client.1.smithi196.stdout:1/798: write d1/db/d16/d44/f92 [9372695,91852] 0 2024-04-22T15:19:11.181 INFO:tasks.workunit.client.1.smithi196.stdout:9/785: dread db/dd/de/dd7/def/d57/f23 [0,4194304] 0 2024-04-22T15:19:11.182 INFO:tasks.workunit.client.1.smithi196.stdout:1/799: chown d1/db/d16/d1e/d49/c33 127064006 1 2024-04-22T15:19:11.187 INFO:tasks.workunit.client.0.smithi096.stdout:9/937: write d1/df/d95/fe4 [541163,62550] 0 2024-04-22T15:19:11.188 INFO:tasks.workunit.client.0.smithi096.stdout:9/938: chown d1/d4/d7f/lbb 45952969 1 2024-04-22T15:19:11.196 INFO:tasks.workunit.client.1.smithi196.stdout:8/767: dwrite d2/d5/f23 [0,4194304] 0 2024-04-22T15:19:11.198 INFO:tasks.workunit.client.1.smithi196.stdout:4/776: dread d6/d14/f87 [0,4194304] 0 2024-04-22T15:19:11.202 INFO:tasks.workunit.client.1.smithi196.stdout:0/814: rmdir d3/d9/d36/d9f/dd1/d65/d7f/d10f 0 2024-04-22T15:19:11.205 INFO:tasks.workunit.client.1.smithi196.stdout:2/803: creat d1/d23/d27/d2e/d55/f123 x:0 0 0 2024-04-22T15:19:11.205 INFO:tasks.workunit.client.1.smithi196.stdout:9/786: truncate db/dd/d1b/f45 3011852 0 2024-04-22T15:19:11.210 INFO:tasks.workunit.client.1.smithi196.stdout:3/847: rename d0/d15/d4d/de7/lf4 to d0/d15/d4d/db/l11e 0 2024-04-22T15:19:11.211 INFO:tasks.workunit.client.1.smithi196.stdout:3/848: readlink d0/d15/d31/d55/dcf/d6f/db5/dd7/l10b 0 2024-04-22T15:19:11.212 INFO:tasks.workunit.client.1.smithi196.stdout:3/849: chown d0/d15/d31/d55/dcf/d6f/d10a/d5e 6827 1 2024-04-22T15:19:11.215 INFO:tasks.workunit.client.1.smithi196.stdout:6/806: rename dd/f26 to dd/d2b/d8f/fed 0 2024-04-22T15:19:11.216 INFO:tasks.workunit.client.1.smithi196.stdout:2/804: mknod d1/d13/d1f/ddf/c124 0 2024-04-22T15:19:11.221 INFO:tasks.workunit.client.1.smithi196.stdout:9/787: creat db/f109 x:0 0 0 2024-04-22T15:19:11.221 INFO:tasks.workunit.client.1.smithi196.stdout:6/807: mkdir dd/d1f/d4e/d53/dee 0 2024-04-22T15:19:11.222 INFO:tasks.workunit.client.1.smithi196.stdout:4/777: getdents d6/d13/d15/d2c/d62/d78/ddc 0 2024-04-22T15:19:11.223 INFO:tasks.workunit.client.1.smithi196.stdout:1/800: dread d1/db/d16/f7e [0,4194304] 0 2024-04-22T15:19:11.225 INFO:tasks.workunit.client.1.smithi196.stdout:2/805: dwrite d1/d13/d1f/ddf/da1/ff7 [0,4194304] 0 2024-04-22T15:19:11.226 INFO:tasks.workunit.client.1.smithi196.stdout:5/822: getdents d1/d17 0 2024-04-22T15:19:11.227 INFO:tasks.workunit.client.1.smithi196.stdout:4/778: write d6/d13/d15/d2c/d62/d78/ff1 [363177,130190] 0 2024-04-22T15:19:11.228 INFO:tasks.workunit.client.1.smithi196.stdout:9/788: symlink db/dd/d1b/l10a 0 2024-04-22T15:19:11.229 INFO:tasks.workunit.client.1.smithi196.stdout:8/768: dread d2/d5/d1a/db7/dd9/d68/dbc/f80 [0,4194304] 0 2024-04-22T15:19:11.229 INFO:tasks.workunit.client.1.smithi196.stdout:4/779: chown d6/d10/fe 257840 1 2024-04-22T15:19:11.230 INFO:tasks.workunit.client.1.smithi196.stdout:8/769: chown d2/d5/cc4 1134749 1 2024-04-22T15:19:11.230 INFO:tasks.workunit.client.1.smithi196.stdout:9/789: chown db/dd/de/d15/d31/d64/cd1 15 1 2024-04-22T15:19:11.231 INFO:tasks.workunit.client.1.smithi196.stdout:7/790: rename d14/cbf to d14/d25/c108 0 2024-04-22T15:19:11.233 INFO:tasks.workunit.client.1.smithi196.stdout:2/806: creat d1/d23/d27/d100/f125 x:0 0 0 2024-04-22T15:19:11.233 INFO:tasks.workunit.client.1.smithi196.stdout:5/823: dread - d1/d2/d98/dac/fd3 zero size 2024-04-22T15:19:11.234 INFO:tasks.workunit.client.1.smithi196.stdout:5/824: chown d1/d14/f1e 15604177 1 2024-04-22T15:19:11.237 INFO:tasks.workunit.client.1.smithi196.stdout:4/780: creat d6/d14/d18/d26/d32/da3/db7/ffb x:0 0 0 2024-04-22T15:19:11.237 INFO:tasks.workunit.client.1.smithi196.stdout:7/791: dwrite d14/d26/d28/d29/fb3 [0,4194304] 0 2024-04-22T15:19:11.239 INFO:tasks.workunit.client.1.smithi196.stdout:7/792: dread - d14/d26/d95/fd3 zero size 2024-04-22T15:19:11.240 INFO:tasks.workunit.client.1.smithi196.stdout:7/793: readlink d14/d26/d2a/lf9 0 2024-04-22T15:19:11.241 INFO:tasks.workunit.client.1.smithi196.stdout:6/808: getdents dd/d1f/d38/d40 0 2024-04-22T15:19:11.242 INFO:tasks.workunit.client.1.smithi196.stdout:5/825: creat d1/d17/f107 x:0 0 0 2024-04-22T15:19:11.243 INFO:tasks.workunit.client.1.smithi196.stdout:8/770: read d2/d5/d1a/db7/dd9/f62 [189091,91464] 0 2024-04-22T15:19:11.243 INFO:tasks.workunit.client.1.smithi196.stdout:2/807: mkdir d1/d23/d126 0 2024-04-22T15:19:11.244 INFO:tasks.workunit.client.1.smithi196.stdout:5/826: chown d1/d14/d3e/f44 13049 1 2024-04-22T15:19:11.248 INFO:tasks.workunit.client.1.smithi196.stdout:0/815: dwrite d3/de/f7d [4194304,4194304] 0 2024-04-22T15:19:11.251 INFO:tasks.workunit.client.1.smithi196.stdout:9/790: dread db/fc [0,4194304] 0 2024-04-22T15:19:11.252 INFO:tasks.workunit.client.1.smithi196.stdout:5/827: symlink d1/d78/db0/da0/l108 0 2024-04-22T15:19:11.253 INFO:tasks.workunit.client.1.smithi196.stdout:8/771: read - d2/d5/d1a/db7/dd9/d68/d81/f88 zero size 2024-04-22T15:19:11.254 INFO:tasks.workunit.client.1.smithi196.stdout:3/850: dwrite d0/d15/d31/d55/d6e/fb2 [0,4194304] 0 2024-04-22T15:19:11.256 INFO:tasks.workunit.client.1.smithi196.stdout:3/851: write d0/d15/da7/f1c [437973,88070] 0 2024-04-22T15:19:11.257 INFO:tasks.workunit.client.1.smithi196.stdout:1/801: write d1/db/dc/de/d43/d5e/fbf [860148,67493] 0 2024-04-22T15:19:11.262 INFO:tasks.workunit.client.1.smithi196.stdout:5/828: dread d1/d14/d6a/fa8 [0,4194304] 0 2024-04-22T15:19:11.262 INFO:tasks.workunit.client.1.smithi196.stdout:3/852: chown d0/d15/d4d/c1f 2237622 1 2024-04-22T15:19:11.263 INFO:tasks.workunit.client.1.smithi196.stdout:0/816: mkdir d3/de/d14/d86/d124 0 2024-04-22T15:19:11.264 INFO:tasks.workunit.client.1.smithi196.stdout:2/808: mkdir d1/d23/d126/d127 0 2024-04-22T15:19:11.265 INFO:tasks.workunit.client.1.smithi196.stdout:0/817: chown d3/de/d1c/c106 154 1 2024-04-22T15:19:11.267 INFO:tasks.workunit.client.1.smithi196.stdout:1/802: truncate d1/db/dc/dad/d36/fd8 952487 0 2024-04-22T15:19:11.271 INFO:tasks.workunit.client.1.smithi196.stdout:2/809: dwrite d1/d13/d25/d9f/f118 [0,4194304] 0 2024-04-22T15:19:11.271 INFO:tasks.workunit.client.1.smithi196.stdout:8/772: creat d2/d5/d93/dd5/fe0 x:0 0 0 2024-04-22T15:19:11.272 INFO:tasks.workunit.client.1.smithi196.stdout:8/773: dread - d2/d8/d49/d8b/fd7 zero size 2024-04-22T15:19:11.274 INFO:tasks.workunit.client.1.smithi196.stdout:8/774: chown d2/d5/d4f/f6b 71896 1 2024-04-22T15:19:11.279 INFO:tasks.workunit.client.1.smithi196.stdout:5/829: creat d1/d17/f109 x:0 0 0 2024-04-22T15:19:11.280 INFO:tasks.workunit.client.1.smithi196.stdout:6/809: dread dd/f18 [0,4194304] 0 2024-04-22T15:19:11.281 INFO:tasks.workunit.client.1.smithi196.stdout:8/775: dwrite d2/f4a [0,4194304] 0 2024-04-22T15:19:11.285 INFO:tasks.workunit.client.1.smithi196.stdout:0/818: write d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/fa3 [4580777,60612] 0 2024-04-22T15:19:11.287 INFO:tasks.workunit.client.1.smithi196.stdout:5/830: read d1/d4/f9c [3016694,67412] 0 2024-04-22T15:19:11.291 INFO:tasks.workunit.client.1.smithi196.stdout:9/791: dwrite db/dd/de/d75/fbf [0,4194304] 0 2024-04-22T15:19:11.291 INFO:tasks.workunit.client.1.smithi196.stdout:2/810: rmdir d1/d23/df5 39 2024-04-22T15:19:11.293 INFO:tasks.workunit.client.1.smithi196.stdout:2/811: chown d1/d23/d27/d73/d121 0 1 2024-04-22T15:19:11.296 INFO:tasks.workunit.client.1.smithi196.stdout:9/792: fsync db/dd/de/dd7/def/d57/da7/ff7 0 2024-04-22T15:19:11.297 INFO:tasks.workunit.client.1.smithi196.stdout:9/793: fsync db/dd/de/d15/d31/fcf 0 2024-04-22T15:19:11.298 INFO:tasks.workunit.client.1.smithi196.stdout:9/794: chown db/dd/de/dd7/def/d57/da7/lc7 665 1 2024-04-22T15:19:11.310 INFO:tasks.workunit.client.1.smithi196.stdout:7/794: getdents d14/d4d 0 2024-04-22T15:19:11.311 INFO:tasks.workunit.client.1.smithi196.stdout:7/795: readlink d14/d25/d31/l41 0 2024-04-22T15:19:11.311 INFO:tasks.workunit.client.1.smithi196.stdout:7/796: fsync d14/d26/d2a/d2b/f101 0 2024-04-22T15:19:11.316 INFO:tasks.workunit.client.1.smithi196.stdout:5/831: dread d1/d17/f21 [4194304,4194304] 0 2024-04-22T15:19:11.317 INFO:tasks.workunit.client.1.smithi196.stdout:3/853: write d0/d15/da7/d18/f1b [539762,32504] 0 2024-04-22T15:19:11.318 INFO:tasks.workunit.client.1.smithi196.stdout:7/797: read d14/d4d/f7e [646687,92956] 0 2024-04-22T15:19:11.319 INFO:tasks.workunit.client.1.smithi196.stdout:3/854: dread - d0/d15/d4d/d87/f111 zero size 2024-04-22T15:19:11.319 INFO:tasks.workunit.client.1.smithi196.stdout:7/798: chown d14/d26/d28/d29/lb4 6585562 1 2024-04-22T15:19:11.322 INFO:tasks.workunit.client.1.smithi196.stdout:9/795: truncate db/dd/de/dd7/def/f38 22851 0 2024-04-22T15:19:11.323 INFO:tasks.workunit.client.1.smithi196.stdout:3/855: dread d0/d15/d31/d55/d6e/fb2 [0,4194304] 0 2024-04-22T15:19:11.324 INFO:tasks.workunit.client.1.smithi196.stdout:1/803: getdents d1/db/d16/d6d 0 2024-04-22T15:19:11.325 INFO:tasks.workunit.client.1.smithi196.stdout:1/804: write d1/db/dc/de/d5a/d5b/f79 [2094387,120680] 0 2024-04-22T15:19:11.326 INFO:tasks.workunit.client.1.smithi196.stdout:7/799: creat d14/d4d/f109 x:0 0 0 2024-04-22T15:19:11.331 INFO:tasks.workunit.client.1.smithi196.stdout:3/856: fsync d0/d15/da7/d4a/d44/f20 0 2024-04-22T15:19:11.332 INFO:tasks.workunit.client.1.smithi196.stdout:7/800: dwrite d14/d26/d2a/d72/db1/fce [0,4194304] 0 2024-04-22T15:19:11.334 INFO:tasks.workunit.client.1.smithi196.stdout:7/801: write d14/d25/f42 [4875660,8552] 0 2024-04-22T15:19:11.334 INFO:tasks.workunit.client.1.smithi196.stdout:1/805: fdatasync d1/db/fc0 0 2024-04-22T15:19:11.336 INFO:tasks.workunit.client.1.smithi196.stdout:9/796: symlink db/dd/l10b 0 2024-04-22T15:19:11.337 INFO:tasks.workunit.client.1.smithi196.stdout:8/776: write d2/d5/d1a/db7/dd9/f62 [227057,50034] 0 2024-04-22T15:19:11.342 INFO:tasks.workunit.client.1.smithi196.stdout:6/810: dwrite dd/d1f/d4e/d53/d7f/f83 [0,4194304] 0 2024-04-22T15:19:11.343 INFO:tasks.workunit.client.1.smithi196.stdout:9/797: dread fa [0,4194304] 0 2024-04-22T15:19:11.344 INFO:tasks.workunit.client.1.smithi196.stdout:9/798: read - db/dd/de/ff1 zero size 2024-04-22T15:19:11.344 INFO:tasks.workunit.client.1.smithi196.stdout:1/806: mkdir d1/db/dc/de/d5a/d5b/d9a/d103 0 2024-04-22T15:19:11.346 INFO:tasks.workunit.client.1.smithi196.stdout:8/777: rmdir d2/d8/d49 39 2024-04-22T15:19:11.347 INFO:tasks.workunit.client.1.smithi196.stdout:1/807: readlink d1/db/dc/de/d52/l91 0 2024-04-22T15:19:11.348 INFO:tasks.workunit.client.1.smithi196.stdout:3/857: rmdir d0/d15/d31/dd0/de2 39 2024-04-22T15:19:11.350 INFO:tasks.workunit.client.1.smithi196.stdout:7/802: mkdir d14/ddd/d10a 0 2024-04-22T15:19:11.352 INFO:tasks.workunit.client.1.smithi196.stdout:5/832: write d1/d4/f22 [3220648,115493] 0 2024-04-22T15:19:11.353 INFO:tasks.workunit.client.1.smithi196.stdout:6/811: creat dd/ddf/fef x:0 0 0 2024-04-22T15:19:11.354 INFO:tasks.workunit.client.1.smithi196.stdout:0/819: dwrite d3/de/d14/f81 [0,4194304] 0 2024-04-22T15:19:11.355 INFO:tasks.workunit.client.1.smithi196.stdout:1/808: rename d1/db/d16/d1e/d65/f66 to d1/db/d16/d6d/de6/f104 0 2024-04-22T15:19:11.356 INFO:tasks.workunit.client.1.smithi196.stdout:9/799: creat db/dd/de/d15/d31/df6/f10c x:0 0 0 2024-04-22T15:19:11.356 INFO:tasks.workunit.client.1.smithi196.stdout:3/858: truncate d0/d15/da7/d4a/f52 1654253 0 2024-04-22T15:19:11.357 INFO:tasks.workunit.client.1.smithi196.stdout:7/803: truncate d14/d26/fd2 775865 0 2024-04-22T15:19:11.359 INFO:tasks.workunit.client.1.smithi196.stdout:5/833: fdatasync d1/d14/d3e/fd1 0 2024-04-22T15:19:11.359 INFO:tasks.workunit.client.1.smithi196.stdout:7/804: stat d14/d4d/da3/l105 0 2024-04-22T15:19:11.360 INFO:tasks.workunit.client.1.smithi196.stdout:0/820: mknod d3/d6/dc/d8c/d8d/c125 0 2024-04-22T15:19:11.364 INFO:tasks.workunit.client.1.smithi196.stdout:5/834: mknod d1/d2/d98/c10a 0 2024-04-22T15:19:11.367 INFO:tasks.workunit.client.1.smithi196.stdout:9/800: truncate db/dd/de/d15/d31/d64/f62 950536 0 2024-04-22T15:19:11.368 INFO:tasks.workunit.client.1.smithi196.stdout:3/859: mknod d0/d15/d31/d55/d10c/c11f 0 2024-04-22T15:19:11.368 INFO:tasks.workunit.client.1.smithi196.stdout:9/801: dread - db/dd/de/dd7/def/d57/fbd zero size 2024-04-22T15:19:11.369 INFO:tasks.workunit.client.1.smithi196.stdout:0/821: creat d3/d6/dc/d8c/d8d/d98/dbd/f126 x:0 0 0 2024-04-22T15:19:11.373 INFO:tasks.workunit.client.1.smithi196.stdout:3/860: unlink d0/d15/da7/fdf 0 2024-04-22T15:19:11.374 INFO:tasks.workunit.client.1.smithi196.stdout:6/812: getdents dd/d20/d4d 0 2024-04-22T15:19:11.375 INFO:tasks.workunit.client.1.smithi196.stdout:0/822: rename d3/d9/d36/d9f/dd1/l8a to d3/de/d1c/d27/ddc/l127 0 2024-04-22T15:19:11.377 INFO:tasks.workunit.client.1.smithi196.stdout:9/802: dwrite db/dd/de/dd7/def/d57/fbd [0,4194304] 0 2024-04-22T15:19:11.380 INFO:tasks.workunit.client.1.smithi196.stdout:4/781: sync 2024-04-22T15:19:11.387 INFO:tasks.workunit.client.1.smithi196.stdout:1/809: dread d1/db/dc/de/d43/ff5 [4194304,4194304] 0 2024-04-22T15:19:11.387 INFO:tasks.workunit.client.1.smithi196.stdout:4/782: unlink d6/d14/f12 0 2024-04-22T15:19:11.395 INFO:tasks.workunit.client.1.smithi196.stdout:1/810: creat d1/db/d16/d1e/d46/f105 x:0 0 0 2024-04-22T15:19:11.395 INFO:tasks.workunit.client.1.smithi196.stdout:9/803: link db/c17 db/dd/de/dd7/c10d 0 2024-04-22T15:19:11.396 INFO:tasks.workunit.client.1.smithi196.stdout:6/813: link dd/d2b/f3e dd/d1f/d4e/d53/d7f/da5/ff0 0 2024-04-22T15:19:11.396 INFO:tasks.workunit.client.1.smithi196.stdout:7/805: dwrite d14/d26/d5a/dbb/f8d [0,4194304] 0 2024-04-22T15:19:11.401 INFO:tasks.workunit.client.1.smithi196.stdout:1/811: creat d1/db/dc/de/f106 x:0 0 0 2024-04-22T15:19:11.402 INFO:tasks.workunit.client.1.smithi196.stdout:5/835: dwrite d1/df1/f6b [0,4194304] 0 2024-04-22T15:19:11.403 INFO:tasks.workunit.client.1.smithi196.stdout:0/823: dread d3/f5 [0,4194304] 0 2024-04-22T15:19:11.405 INFO:tasks.workunit.client.1.smithi196.stdout:9/804: creat db/dd/de/dd7/def/d7a/d7b/d81/f10e x:0 0 0 2024-04-22T15:19:11.406 INFO:tasks.workunit.client.1.smithi196.stdout:4/783: getdents d6/d90 0 2024-04-22T15:19:11.408 INFO:tasks.workunit.client.1.smithi196.stdout:5/836: rmdir d1/d2/d98/dac 39 2024-04-22T15:19:11.410 INFO:tasks.workunit.client.1.smithi196.stdout:1/812: mknod d1/db/d97/dd7/c107 0 2024-04-22T15:19:11.415 INFO:tasks.workunit.client.1.smithi196.stdout:3/861: dwrite d0/d15/d4d/d87/da6/fbd [0,4194304] 0 2024-04-22T15:19:11.415 INFO:tasks.workunit.client.1.smithi196.stdout:0/824: dread d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/fc0 [0,4194304] 0 2024-04-22T15:19:11.419 INFO:tasks.workunit.client.1.smithi196.stdout:9/805: dwrite db/dd/de/dd7/fe7 [0,4194304] 0 2024-04-22T15:19:11.420 INFO:tasks.workunit.client.1.smithi196.stdout:9/806: fdatasync db/dd/de/d15/f73 0 2024-04-22T15:19:11.423 INFO:tasks.workunit.client.1.smithi196.stdout:3/862: dread d0/d15/d31/d55/dcf/d8/d6b/fd6 [0,4194304] 0 2024-04-22T15:19:11.426 INFO:tasks.workunit.client.1.smithi196.stdout:3/863: read d0/d15/d31/d55/dcf/d6f/db5/dd7/fda [2225442,24988] 0 2024-04-22T15:19:11.480 INFO:tasks.workunit.client.1.smithi196.stdout:4/784: mknod d6/d13/d15/d2c/d62/d78/df6/cfc 0 2024-04-22T15:19:11.480 INFO:tasks.workunit.client.1.smithi196.stdout:6/814: dread dd/f17 [0,4194304] 0 2024-04-22T15:19:11.481 INFO:tasks.workunit.client.1.smithi196.stdout:0/825: rmdir d3 39 2024-04-22T15:19:11.485 INFO:tasks.workunit.client.1.smithi196.stdout:9/807: creat db/dd/de/d15/d31/d64/dab/f10f x:0 0 0 2024-04-22T15:19:11.488 INFO:tasks.workunit.client.1.smithi196.stdout:9/808: truncate db/dd/de/dd7/def/d57/da7/fff 735935 0 2024-04-22T15:19:11.488 INFO:tasks.workunit.client.1.smithi196.stdout:5/837: creat d1/d2/f10b x:0 0 0 2024-04-22T15:19:11.489 INFO:tasks.workunit.client.1.smithi196.stdout:6/815: mknod dd/d70/cf1 0 2024-04-22T15:19:11.490 INFO:tasks.workunit.client.1.smithi196.stdout:0/826: readlink d3/d6/l8 0 2024-04-22T15:19:11.491 INFO:tasks.workunit.client.1.smithi196.stdout:9/809: chown db/dd/de/dd7/def/d57/da7/lcd 7 1 2024-04-22T15:19:11.492 INFO:tasks.workunit.client.1.smithi196.stdout:4/785: rename d6/d13/d15/d2c/d8c/db3/fed to d6/d14/d18/d26/d32/d53/ffd 0 2024-04-22T15:19:11.493 INFO:tasks.workunit.client.1.smithi196.stdout:3/864: dwrite d0/f60 [0,4194304] 0 2024-04-22T15:19:11.494 INFO:tasks.workunit.client.1.smithi196.stdout:9/810: truncate db/dd/de/d15/fe0 5227886 0 2024-04-22T15:19:11.495 INFO:tasks.workunit.client.1.smithi196.stdout:3/865: chown d0/d15/d31/dcb/l116 1109 1 2024-04-22T15:19:11.497 INFO:tasks.workunit.client.1.smithi196.stdout:9/811: dread - db/dd/f107 zero size 2024-04-22T15:19:11.498 INFO:tasks.workunit.client.1.smithi196.stdout:4/786: symlink d6/d14/d18/d40/lfe 0 2024-04-22T15:19:11.499 INFO:tasks.workunit.client.1.smithi196.stdout:3/866: write d0/d15/da7/d18/f1b [9186712,43549] 0 2024-04-22T15:19:11.500 INFO:tasks.workunit.client.1.smithi196.stdout:6/816: creat dd/d1f/d4e/d53/d55/d6d/dd1/de3/de8/ff2 x:0 0 0 2024-04-22T15:19:11.501 INFO:tasks.workunit.client.1.smithi196.stdout:9/812: chown db/dd/de/dd7/def/d7a/d7b/l90 450744916 1 2024-04-22T15:19:11.502 INFO:tasks.workunit.client.1.smithi196.stdout:9/813: chown fa 1 1 2024-04-22T15:19:11.506 INFO:tasks.workunit.client.1.smithi196.stdout:2/812: sync 2024-04-22T15:19:11.507 INFO:tasks.workunit.client.1.smithi196.stdout:4/787: mkdir d6/d13/d15/d2c/d62/d78/d8d/dff 0 2024-04-22T15:19:11.510 INFO:tasks.workunit.client.1.smithi196.stdout:6/817: rename dd/d20/d5c/d5e/d76/f85 to dd/d1f/d4e/d53/d55/d6d/dd1/ff3 0 2024-04-22T15:19:11.511 INFO:tasks.workunit.client.1.smithi196.stdout:2/813: truncate d1/d13/d25/fd0 76622 0 2024-04-22T15:19:11.513 INFO:tasks.workunit.client.1.smithi196.stdout:9/814: mknod db/dd/c110 0 2024-04-22T15:19:11.514 INFO:tasks.workunit.client.1.smithi196.stdout:4/788: creat d6/d14/d18/d40/d85/f100 x:0 0 0 2024-04-22T15:19:11.514 INFO:tasks.workunit.client.1.smithi196.stdout:6/818: creat dd/d20/d5c/d6a/ff4 x:0 0 0 2024-04-22T15:19:11.517 INFO:tasks.workunit.client.1.smithi196.stdout:4/789: fdatasync d6/f93 0 2024-04-22T15:19:11.518 INFO:tasks.workunit.client.1.smithi196.stdout:0/827: dwrite d3/d6/dc/d8c/d8d/fc1 [0,4194304] 0 2024-04-22T15:19:11.518 INFO:tasks.workunit.client.1.smithi196.stdout:5/838: dread d1/d17/f2e [0,4194304] 0 2024-04-22T15:19:11.518 INFO:tasks.workunit.client.1.smithi196.stdout:9/815: creat db/f111 x:0 0 0 2024-04-22T15:19:11.519 INFO:tasks.workunit.client.1.smithi196.stdout:6/819: chown dd/d1f/d38/d40/f43 16016067 1 2024-04-22T15:19:11.524 INFO:tasks.workunit.client.1.smithi196.stdout:9/816: stat db/dd/de/dd7/def/d7a/d7b/d81/f10e 0 2024-04-22T15:19:11.526 INFO:tasks.workunit.client.1.smithi196.stdout:3/867: dwrite d0/d15/f92 [0,4194304] 0 2024-04-22T15:19:11.527 INFO:tasks.workunit.client.1.smithi196.stdout:3/868: write d0/d15/d31/d55/d6e/fa1 [2653148,32219] 0 2024-04-22T15:19:11.528 INFO:tasks.workunit.client.1.smithi196.stdout:4/790: rmdir d6/d13/d15/d2c/d62/d78/d8d/dd4 0 2024-04-22T15:19:11.529 INFO:tasks.workunit.client.1.smithi196.stdout:4/791: readlink d6/d14/d18/d26/l51 0 2024-04-22T15:19:11.531 INFO:tasks.workunit.client.1.smithi196.stdout:4/792: chown d6/d13/d15/d2c/d8c/db3/fdf 661360047 1 2024-04-22T15:19:11.532 INFO:tasks.workunit.client.1.smithi196.stdout:2/814: dread d1/d13/d25/d2a/f3f [0,4194304] 0 2024-04-22T15:19:11.536 INFO:tasks.workunit.client.1.smithi196.stdout:2/815: mknod d1/d23/d27/d2e/d55/d5b/d7c/c128 0 2024-04-22T15:19:11.538 INFO:tasks.workunit.client.1.smithi196.stdout:2/816: mknod d1/d13/c129 0 2024-04-22T15:19:11.542 INFO:tasks.workunit.client.1.smithi196.stdout:2/817: link d1/f117 d1/d76/f12a 0 2024-04-22T15:19:11.543 INFO:tasks.workunit.client.1.smithi196.stdout:5/839: dread d1/d17/f8b [0,4194304] 0 2024-04-22T15:19:11.545 INFO:tasks.workunit.client.1.smithi196.stdout:2/818: stat d1/d13/d25/fd0 0 2024-04-22T15:19:11.549 INFO:tasks.workunit.client.1.smithi196.stdout:5/840: getdents d1/d78/db0 0 2024-04-22T15:19:11.553 INFO:tasks.workunit.client.1.smithi196.stdout:5/841: fsync d1/df1/d6e/fc1 0 2024-04-22T15:19:11.554 INFO:tasks.workunit.client.1.smithi196.stdout:9/817: dwrite db/dd/de/dd7/def/d57/f5a [0,4194304] 0 2024-04-22T15:19:11.555 INFO:tasks.workunit.client.1.smithi196.stdout:5/842: readlink d1/d17/d1c/l27 0 2024-04-22T15:19:11.555 INFO:tasks.workunit.client.1.smithi196.stdout:0/828: dread d3/d9/d36/d9f/dd1/d65/d7f/f9a [0,4194304] 0 2024-04-22T15:19:11.558 INFO:tasks.workunit.client.1.smithi196.stdout:2/819: write d1/f16 [3375312,89233] 0 2024-04-22T15:19:11.559 INFO:tasks.workunit.client.1.smithi196.stdout:0/829: truncate d3/de/d1c/df1/f111 997070 0 2024-04-22T15:19:11.561 INFO:tasks.workunit.client.1.smithi196.stdout:3/869: dread d0/d15/d31/d55/f74 [0,4194304] 0 2024-04-22T15:19:11.563 INFO:tasks.workunit.client.1.smithi196.stdout:4/793: dread d6/d13/d15/fb2 [0,4194304] 0 2024-04-22T15:19:11.563 INFO:tasks.workunit.client.1.smithi196.stdout:2/820: rename d1/l1a to d1/d23/d27/d2e/d55/d5b/d7c/l12b 0 2024-04-22T15:19:11.564 INFO:tasks.workunit.client.1.smithi196.stdout:5/843: link d1/d17/d7f/deb/l102 d1/d78/dcf/dff/l10c 0 2024-04-22T15:19:11.566 INFO:tasks.workunit.client.1.smithi196.stdout:4/794: readlink d6/d10/le2 0 2024-04-22T15:19:11.567 INFO:tasks.workunit.client.1.smithi196.stdout:2/821: symlink d1/d23/d27/d2e/d55/l12c 0 2024-04-22T15:19:11.568 INFO:tasks.workunit.client.1.smithi196.stdout:2/822: dread - d1/d13/dbc/ff1 zero size 2024-04-22T15:19:11.569 INFO:tasks.workunit.client.1.smithi196.stdout:3/870: link d0/d15/da7/d4a/f50 d0/d15/d31/d55/d5d/f120 0 2024-04-22T15:19:11.571 INFO:tasks.workunit.client.1.smithi196.stdout:5/844: dwrite d1/d2/faf [0,4194304] 0 2024-04-22T15:19:11.572 INFO:tasks.workunit.client.1.smithi196.stdout:2/823: unlink d1/d13/d25/c70 0 2024-04-22T15:19:11.573 INFO:tasks.workunit.client.1.smithi196.stdout:3/871: creat d0/d15/da7/d4a/d44/d8c/de9/f121 x:0 0 0 2024-04-22T15:19:11.574 INFO:tasks.workunit.client.1.smithi196.stdout:5/845: fdatasync d1/d17/d50/f6d 0 2024-04-22T15:19:11.575 INFO:tasks.workunit.client.1.smithi196.stdout:2/824: chown d1/d23/d27/d2e/d55/d5b/d93/d97/f106 558695234 1 2024-04-22T15:19:11.576 INFO:tasks.workunit.client.1.smithi196.stdout:3/872: chown d0/d15/d31/d55/dcf/d6f/d10a/f54 1074 1 2024-04-22T15:19:11.577 INFO:tasks.workunit.client.1.smithi196.stdout:5/846: mknod d1/df1/d5e/c10d 0 2024-04-22T15:19:11.583 INFO:tasks.workunit.client.1.smithi196.stdout:3/873: link d0/d15/da7/d4a/dc5/cd5 d0/d15/d4d/db/d11c/c122 0 2024-04-22T15:19:11.584 INFO:tasks.workunit.client.1.smithi196.stdout:9/818: dread db/dd/de/d15/d31/f63 [0,4194304] 0 2024-04-22T15:19:11.585 INFO:tasks.workunit.client.1.smithi196.stdout:9/819: chown db/dd/de/d15/ff0 2618 1 2024-04-22T15:19:11.588 INFO:tasks.workunit.client.1.smithi196.stdout:3/874: truncate d0/d15/da7/d18/f84 55937 0 2024-04-22T15:19:11.588 INFO:tasks.workunit.client.1.smithi196.stdout:4/795: dread d6/d13/d15/d2c/d62/d78/d8d/f9c [0,4194304] 0 2024-04-22T15:19:11.589 INFO:tasks.workunit.client.1.smithi196.stdout:9/820: chown db/dd/de/d15/d31/d64/dab/cdd 20773 1 2024-04-22T15:19:11.589 INFO:tasks.workunit.client.1.smithi196.stdout:3/875: chown d0/d15/d4d/de7/f114 29010 1 2024-04-22T15:19:11.592 INFO:tasks.workunit.client.1.smithi196.stdout:9/821: unlink db/dd/de/dd7/f102 0 2024-04-22T15:19:11.595 INFO:tasks.workunit.client.1.smithi196.stdout:4/796: write d6/d13/d15/d2c/d62/f5a [502722,107373] 0 2024-04-22T15:19:11.595 INFO:tasks.workunit.client.1.smithi196.stdout:2/825: dwrite d1/f117 [0,4194304] 0 2024-04-22T15:19:11.597 INFO:tasks.workunit.client.1.smithi196.stdout:2/826: readlink d1/d23/d27/d2e/d55/d5b/d93/lb1 0 2024-04-22T15:19:11.598 INFO:tasks.workunit.client.1.smithi196.stdout:4/797: mknod d6/d13/d15/d2c/d62/d78/c101 0 2024-04-22T15:19:11.600 INFO:tasks.workunit.client.1.smithi196.stdout:4/798: chown d6/d14/d18/d40/d85/fbf 67 1 2024-04-22T15:19:11.658 INFO:tasks.workunit.client.1.smithi196.stdout:2/827: creat d1/d23/d27/d73/f12d x:0 0 0 2024-04-22T15:19:11.659 INFO:tasks.workunit.client.1.smithi196.stdout:2/828: readlink d1/d13/d1f/lb6 0 2024-04-22T15:19:11.660 INFO:tasks.workunit.client.1.smithi196.stdout:4/799: symlink d6/d14/l102 0 2024-04-22T15:19:11.660 INFO:tasks.workunit.client.1.smithi196.stdout:8/778: sync 2024-04-22T15:19:11.661 INFO:tasks.workunit.client.1.smithi196.stdout:8/779: stat d2/d5/d2e/l33 0 2024-04-22T15:19:11.662 INFO:tasks.workunit.client.1.smithi196.stdout:9/822: dread db/dd/de/d75/f86 [0,4194304] 0 2024-04-22T15:19:11.663 INFO:tasks.workunit.client.1.smithi196.stdout:2/829: truncate d1/d13/d41/fd3 451438 0 2024-04-22T15:19:11.663 INFO:tasks.workunit.client.1.smithi196.stdout:8/780: chown d2/d5/d1a/dcc 1234 1 2024-04-22T15:19:11.664 INFO:tasks.workunit.client.1.smithi196.stdout:3/876: dwrite d0/d15/da7/d4a/f50 [8388608,4194304] 0 2024-04-22T15:19:11.664 INFO:tasks.workunit.client.1.smithi196.stdout:4/800: mkdir d6/d14/d18/d26/d32/d67/de7/d103 0 2024-04-22T15:19:11.666 INFO:tasks.workunit.client.1.smithi196.stdout:2/830: dread d1/d23/d27/d2e/f85 [0,4194304] 0 2024-04-22T15:19:11.667 INFO:tasks.workunit.client.1.smithi196.stdout:9/823: symlink db/dd/de/dd7/def/d57/da7/l112 0 2024-04-22T15:19:11.670 INFO:tasks.workunit.client.1.smithi196.stdout:3/877: mkdir d0/d15/da7/d4a/d44/d96/dd8/d123 0 2024-04-22T15:19:11.673 INFO:tasks.workunit.client.1.smithi196.stdout:9/824: creat db/dd/de/dd7/def/d7a/f113 x:0 0 0 2024-04-22T15:19:11.674 INFO:tasks.workunit.client.1.smithi196.stdout:3/878: read - d0/d15/d31/d55/dcf/d6f/d10a/fc3 zero size 2024-04-22T15:19:11.675 INFO:tasks.workunit.client.1.smithi196.stdout:8/781: dwrite d2/d5/faf [4194304,4194304] 0 2024-04-22T15:19:11.676 INFO:tasks.workunit.client.1.smithi196.stdout:9/825: creat db/dd/de/d98/f114 x:0 0 0 2024-04-22T15:19:11.681 INFO:tasks.workunit.client.1.smithi196.stdout:2/831: write d1/d13/d1f/f9d [81551,61543] 0 2024-04-22T15:19:11.682 INFO:tasks.workunit.client.1.smithi196.stdout:8/782: mkdir d2/d5/d1a/db7/dd9/de1 0 2024-04-22T15:19:11.682 INFO:tasks.workunit.client.1.smithi196.stdout:9/826: write db/dd/de/dd7/def/d57/da7/fff [1015898,111734] 0 2024-04-22T15:19:11.685 INFO:tasks.workunit.client.1.smithi196.stdout:9/827: fdatasync db/dd/de/dd7/def/d57/ff9 0 2024-04-22T15:19:11.685 INFO:tasks.workunit.client.1.smithi196.stdout:3/879: write d0/d15/d31/d55/dcf/d6f/db5/dd7/fda [2811491,35246] 0 2024-04-22T15:19:11.688 INFO:tasks.workunit.client.1.smithi196.stdout:9/828: write db/dd/de/d75/fbf [189110,109746] 0 2024-04-22T15:19:11.691 INFO:tasks.workunit.client.1.smithi196.stdout:3/880: write d0/d15/d31/d55/dcf/d6f/d10a/fc3 [457008,45850] 0 2024-04-22T15:19:11.692 INFO:tasks.workunit.client.1.smithi196.stdout:8/783: dwrite d2/d5/faf [0,4194304] 0 2024-04-22T15:19:11.694 INFO:tasks.workunit.client.1.smithi196.stdout:4/801: dread d6/d13/d15/d2c/f75 [4194304,4194304] 0 2024-04-22T15:19:11.696 INFO:tasks.workunit.client.1.smithi196.stdout:9/829: write db/dd/de/dd7/def/f9b [525992,119187] 0 2024-04-22T15:19:11.696 INFO:tasks.workunit.client.1.smithi196.stdout:4/802: chown d6/d13/f25 6436973 1 2024-04-22T15:19:11.698 INFO:tasks.workunit.client.1.smithi196.stdout:8/784: symlink d2/d5/d1a/db7/dd9/d68/le2 0 2024-04-22T15:19:11.699 INFO:tasks.workunit.client.1.smithi196.stdout:8/785: chown d2/d5/f23 14644 1 2024-04-22T15:19:11.699 INFO:tasks.workunit.client.1.smithi196.stdout:3/881: creat d0/d15/da7/d4a/d44/d96/dd8/d123/f124 x:0 0 0 2024-04-22T15:19:11.700 INFO:tasks.workunit.client.1.smithi196.stdout:2/832: dwrite d1/d76/f10b [4194304,4194304] 0 2024-04-22T15:19:11.701 INFO:tasks.workunit.client.1.smithi196.stdout:4/803: mknod d6/d13/d15/d2c/d62/c104 0 2024-04-22T15:19:11.702 INFO:tasks.workunit.client.1.smithi196.stdout:9/830: mknod db/dd/de/dd7/def/d57/c115 0 2024-04-22T15:19:11.702 INFO:tasks.workunit.client.1.smithi196.stdout:4/804: chown d6/d14/d18/d26/d32/c9b 0 1 2024-04-22T15:19:11.706 INFO:tasks.workunit.client.1.smithi196.stdout:9/831: mkdir db/dd/de/dd7/def/d7a/d116 0 2024-04-22T15:19:11.706 INFO:tasks.workunit.client.1.smithi196.stdout:4/805: fdatasync d6/d14/d18/d26/f3a 0 2024-04-22T15:19:11.708 INFO:tasks.workunit.client.1.smithi196.stdout:9/832: mkdir db/dd/de/d98/da9/d117 0 2024-04-22T15:19:11.708 INFO:tasks.workunit.client.1.smithi196.stdout:8/786: getdents d2/d6a 0 2024-04-22T15:19:11.713 INFO:tasks.workunit.client.1.smithi196.stdout:8/787: rename d2/d5/d93/fc8 to d2/fe3 0 2024-04-22T15:19:11.715 INFO:tasks.workunit.client.1.smithi196.stdout:8/788: creat d2/d5/d1a/db7/dd9/d68/d81/db8/fe4 x:0 0 0 2024-04-22T15:19:11.719 INFO:tasks.workunit.client.1.smithi196.stdout:8/789: dread d2/d5/faf [0,4194304] 0 2024-04-22T15:19:11.719 INFO:tasks.workunit.client.1.smithi196.stdout:3/882: write d0/d15/f7a [5180072,123655] 0 2024-04-22T15:19:11.720 INFO:tasks.workunit.client.1.smithi196.stdout:8/790: fsync d2/d8/fd0 0 2024-04-22T15:19:11.720 INFO:tasks.workunit.client.1.smithi196.stdout:9/833: write db/dd/d1b/d2c/f37 [4197552,54430] 0 2024-04-22T15:19:11.723 INFO:tasks.workunit.client.1.smithi196.stdout:3/883: truncate d0/d15/f8e 721694 0 2024-04-22T15:19:11.726 INFO:tasks.workunit.client.1.smithi196.stdout:4/806: dwrite d6/d14/f98 [0,4194304] 0 2024-04-22T15:19:11.727 INFO:tasks.workunit.client.1.smithi196.stdout:9/834: creat db/dd/de/dd7/def/d57/f118 x:0 0 0 2024-04-22T15:19:11.728 INFO:tasks.workunit.client.1.smithi196.stdout:3/884: symlink d0/d15/d31/d55/dcf/d6f/d10a/l125 0 2024-04-22T15:19:11.730 INFO:tasks.workunit.client.1.smithi196.stdout:4/807: mkdir d6/d10/d105 0 2024-04-22T15:19:11.732 INFO:tasks.workunit.client.1.smithi196.stdout:4/808: write d6/d14/d18/d26/d32/fcb [556927,104684] 0 2024-04-22T15:19:11.733 INFO:tasks.workunit.client.1.smithi196.stdout:3/885: symlink d0/d15/d31/d55/dcf/d6f/l126 0 2024-04-22T15:19:11.734 INFO:tasks.workunit.client.1.smithi196.stdout:4/809: stat d6/d14/d18/d26/d32/d67/d9a/lee 0 2024-04-22T15:19:11.734 INFO:tasks.workunit.client.1.smithi196.stdout:8/791: truncate d2/dc/fe 3463512 0 2024-04-22T15:19:11.735 INFO:tasks.workunit.client.1.smithi196.stdout:2/833: dread d1/d76/f7d [0,4194304] 0 2024-04-22T15:19:11.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:11 smithi096 ceph-mon[26688]: pgmap v42: 65 pgs: 65 active+clean; 3.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 90 MiB/s rd, 173 MiB/s wr, 439 op/s 2024-04-22T15:19:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:11 smithi096 ceph-mon[26688]: Standby manager daemon smithi096.hhkvum restarted 2024-04-22T15:19:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:11 smithi096 ceph-mon[26688]: Standby manager daemon smithi096.hhkvum started 2024-04-22T15:19:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:11 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi096.hhkvum/crt"}]: dispatch 2024-04-22T15:19:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:11 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:19:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:11 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi096.hhkvum/key"}]: dispatch 2024-04-22T15:19:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:11 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:19:11.737 INFO:tasks.workunit.client.1.smithi196.stdout:4/810: fdatasync d6/d10/fb1 0 2024-04-22T15:19:11.738 INFO:tasks.workunit.client.1.smithi196.stdout:9/835: write db/dd/de/d75/f86 [2220841,116854] 0 2024-04-22T15:19:11.739 INFO:tasks.workunit.client.1.smithi196.stdout:8/792: creat d2/d5/d1a/db7/fe5 x:0 0 0 2024-04-22T15:19:11.740 INFO:tasks.workunit.client.1.smithi196.stdout:9/836: readlink db/dd/d1b/de6/leb 0 2024-04-22T15:19:11.740 INFO:tasks.workunit.client.1.smithi196.stdout:4/811: chown d6/d13/d15/d2c/d62/d78/d8b/fd2 2 1 2024-04-22T15:19:11.747 INFO:tasks.workunit.client.1.smithi196.stdout:2/834: dwrite d1/d1d/f57 [0,4194304] 0 2024-04-22T15:19:11.747 INFO:tasks.workunit.client.1.smithi196.stdout:9/837: creat db/dd/de/d98/da9/d101/f119 x:0 0 0 2024-04-22T15:19:11.748 INFO:tasks.workunit.client.1.smithi196.stdout:3/886: dwrite d0/d15/da7/d4a/d44/ff3 [0,4194304] 0 2024-04-22T15:19:11.748 INFO:tasks.workunit.client.1.smithi196.stdout:2/835: rename d1 to d1/d13/d25/d9f/d12e 22 2024-04-22T15:19:11.749 INFO:tasks.workunit.client.1.smithi196.stdout:4/812: mkdir d6/d13/d15/d2c/d62/d78/d8d/dff/d106 0 2024-04-22T15:19:11.749 INFO:tasks.workunit.client.1.smithi196.stdout:8/793: getdents d2/d35/d96 0 2024-04-22T15:19:11.750 INFO:tasks.workunit.client.1.smithi196.stdout:9/838: dread - db/dd/de/dd7/def/d7a/fce zero size 2024-04-22T15:19:11.751 INFO:tasks.workunit.client.1.smithi196.stdout:7/806: sync 2024-04-22T15:19:11.751 INFO:tasks.workunit.client.1.smithi196.stdout:1/813: sync 2024-04-22T15:19:11.751 INFO:tasks.workunit.client.1.smithi196.stdout:3/887: write d0/d15/d31/d55/dcf/d8/d70/f7e [1185682,30370] 0 2024-04-22T15:19:11.755 INFO:tasks.workunit.client.1.smithi196.stdout:1/814: stat d1/db/dc/de/d5a/d5b/d6e 0 2024-04-22T15:19:11.756 INFO:tasks.workunit.client.1.smithi196.stdout:8/794: creat d2/d5/d1a/db7/dd9/d68/dbc/dae/fe6 x:0 0 0 2024-04-22T15:19:11.756 INFO:tasks.workunit.client.1.smithi196.stdout:9/839: chown db/dd/de/dd7/def/d57/da7/ff7 11 1 2024-04-22T15:19:11.757 INFO:tasks.workunit.client.1.smithi196.stdout:7/807: mkdir d14/d26/db7/d10b 0 2024-04-22T15:19:11.758 INFO:tasks.workunit.client.1.smithi196.stdout:3/888: mkdir d0/d15/d31/d55/dcf/d6f/de5/d101/d127 0 2024-04-22T15:19:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:11 smithi196 ceph-mon[34717]: pgmap v42: 65 pgs: 65 active+clean; 3.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 90 MiB/s rd, 173 MiB/s wr, 439 op/s 2024-04-22T15:19:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:11 smithi196 ceph-mon[34717]: Standby manager daemon smithi096.hhkvum restarted 2024-04-22T15:19:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:11 smithi196 ceph-mon[34717]: Standby manager daemon smithi096.hhkvum started 2024-04-22T15:19:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:11 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi096.hhkvum/crt"}]: dispatch 2024-04-22T15:19:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:11 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:19:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:11 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi096.hhkvum/key"}]: dispatch 2024-04-22T15:19:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:11 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:19:11.760 INFO:tasks.workunit.client.1.smithi196.stdout:4/813: link d6/d14/d18/d26/d32/d67/fc8 d6/d14/d18/d26/d32/d67/de7/f107 0 2024-04-22T15:19:11.760 INFO:tasks.workunit.client.1.smithi196.stdout:3/889: write d0/d15/d31/dd0/fae [3143353,58204] 0 2024-04-22T15:19:11.761 INFO:tasks.workunit.client.1.smithi196.stdout:1/815: symlink d1/db/d16/d6d/de6/l108 0 2024-04-22T15:19:11.761 INFO:tasks.workunit.client.1.smithi196.stdout:8/795: mknod d2/d3f/ce7 0 2024-04-22T15:19:11.764 INFO:tasks.workunit.client.1.smithi196.stdout:4/814: dread - d6/d14/d18/d26/d32/da3/db7/dc9/fd9 zero size 2024-04-22T15:19:11.764 INFO:tasks.workunit.client.1.smithi196.stdout:8/796: readlink d2/d5/d1a/ldc 0 2024-04-22T15:19:11.764 INFO:tasks.workunit.client.1.smithi196.stdout:9/840: truncate db/dd/de/dd7/def/d57/fa2 983803 0 2024-04-22T15:19:11.765 INFO:tasks.workunit.client.1.smithi196.stdout:1/816: write d1/db/d16/d8e/ddd/fee [196319,130986] 0 2024-04-22T15:19:11.765 INFO:tasks.workunit.client.1.smithi196.stdout:7/808: fsync d14/d15/f59 0 2024-04-22T15:19:11.768 INFO:tasks.workunit.client.1.smithi196.stdout:9/841: mknod db/dd/de/d15/d31/d64/dab/c11a 0 2024-04-22T15:19:11.768 INFO:tasks.workunit.client.1.smithi196.stdout:4/815: dread d6/d10/f50 [0,4194304] 0 2024-04-22T15:19:11.769 INFO:tasks.workunit.client.1.smithi196.stdout:1/817: unlink d1/db/dc/de/c1a 0 2024-04-22T15:19:11.771 INFO:tasks.workunit.client.1.smithi196.stdout:1/818: write d1/db/d16/d44/f92 [3108624,128414] 0 2024-04-22T15:19:11.771 INFO:tasks.workunit.client.1.smithi196.stdout:3/890: rename d0/d15/d31/d55/dcf/d8/f19 to d0/d15/d31/d55/dcf/d6f/de5/f128 0 2024-04-22T15:19:11.771 INFO:tasks.workunit.client.1.smithi196.stdout:7/809: fsync d14/d26/fa2 0 2024-04-22T15:19:11.773 INFO:tasks.workunit.client.1.smithi196.stdout:4/816: mknod d6/d14/d18/d26/d32/da3/c108 0 2024-04-22T15:19:11.773 INFO:tasks.workunit.client.1.smithi196.stdout:9/842: write db/dd/de/dd7/def/d57/fbc [2939752,95077] 0 2024-04-22T15:19:11.775 INFO:tasks.workunit.client.1.smithi196.stdout:9/843: read db/d11/fb2 [4142563,92464] 0 2024-04-22T15:19:11.776 INFO:tasks.workunit.client.0.smithi096.stdout:8/979: sync 2024-04-22T15:19:11.776 INFO:tasks.workunit.client.1.smithi196.stdout:7/810: mknod d14/d4d/da3/de9/c10c 0 2024-04-22T15:19:11.777 INFO:tasks.workunit.client.1.smithi196.stdout:2/836: write d1/d13/d25/d2a/f95 [194755,124272] 0 2024-04-22T15:19:11.779 INFO:tasks.workunit.client.1.smithi196.stdout:2/837: dread d1/d13/d1f/ddf/f9c [0,4194304] 0 2024-04-22T15:19:11.780 INFO:tasks.workunit.client.1.smithi196.stdout:4/817: mkdir d6/de8/d109 0 2024-04-22T15:19:11.781 INFO:tasks.workunit.client.0.smithi096.stdout:8/980: mknod d4/d1c/c14d 0 2024-04-22T15:19:11.783 INFO:tasks.workunit.client.0.smithi096.stdout:8/981: chown d4/d10/d25/d38/d8f/d9c/d138/f14b 498 1 2024-04-22T15:19:11.783 INFO:tasks.workunit.client.1.smithi196.stdout:9/844: chown db/dd/de/dd7/def/l8b 2 1 2024-04-22T15:19:11.784 INFO:tasks.workunit.client.1.smithi196.stdout:7/811: write d14/d26/fdf [590017,37336] 0 2024-04-22T15:19:11.785 INFO:tasks.workunit.client.0.smithi096.stdout:8/982: creat d4/d1c/de2/f14e x:0 0 0 2024-04-22T15:19:11.786 INFO:tasks.workunit.client.1.smithi196.stdout:3/891: getdents d0/d15/d31/d55/dcf/d6f/de5/d101/d9e/d117 0 2024-04-22T15:19:11.786 INFO:tasks.workunit.client.0.smithi096.stdout:8/983: write d4/d113/f13a [384822,71831] 0 2024-04-22T15:19:11.789 INFO:tasks.workunit.client.1.smithi196.stdout:7/812: mknod d14/d26/d5a/dbb/c10d 0 2024-04-22T15:19:11.790 INFO:tasks.workunit.client.0.smithi096.stdout:8/984: symlink d4/d1c/de2/d125/l14f 0 2024-04-22T15:19:11.793 INFO:tasks.workunit.client.1.smithi196.stdout:8/797: dwrite d2/dc/f82 [4194304,4194304] 0 2024-04-22T15:19:11.793 INFO:tasks.workunit.client.1.smithi196.stdout:1/819: dread d1/db/dc/dad/f9b [4194304,4194304] 0 2024-04-22T15:19:11.793 INFO:tasks.workunit.client.0.smithi096.stdout:8/985: getdents d4/d1c/d40/d48/d101/d122 0 2024-04-22T15:19:11.793 INFO:tasks.workunit.client.1.smithi196.stdout:3/892: mknod d0/d15/d31/d55/dcf/d6f/de5/d101/d9e/d117/c129 0 2024-04-22T15:19:11.794 INFO:tasks.workunit.client.1.smithi196.stdout:9/845: rmdir db/dd/de/dd7/def/d7a/dfe 0 2024-04-22T15:19:11.795 INFO:tasks.workunit.client.0.smithi096.stdout:8/986: fsync d4/d10/d25/d38/f79 0 2024-04-22T15:19:11.796 INFO:tasks.workunit.client.1.smithi196.stdout:1/820: write d1/db/d16/d8e/ddd/fee [385127,24086] 0 2024-04-22T15:19:11.798 INFO:tasks.workunit.client.1.smithi196.stdout:2/838: getdents d1/d23/d84 0 2024-04-22T15:19:11.799 INFO:tasks.workunit.client.1.smithi196.stdout:8/798: unlink d2/d5/c90 0 2024-04-22T15:19:11.800 INFO:tasks.workunit.client.0.smithi096.stdout:8/987: truncate d4/d2a/fbf 971661 0 2024-04-22T15:19:11.801 INFO:tasks.workunit.client.1.smithi196.stdout:8/799: readlink d2/d6a/l8f 0 2024-04-22T15:19:11.801 INFO:tasks.workunit.client.1.smithi196.stdout:7/813: dwrite d14/d15/d91/fe7 [0,4194304] 0 2024-04-22T15:19:11.801 INFO:tasks.workunit.client.1.smithi196.stdout:2/839: chown d1/d23/d27/d73/d121 85248 1 2024-04-22T15:19:11.802 INFO:tasks.workunit.client.1.smithi196.stdout:3/893: mkdir d0/d15/d31/d55/dcf/d8/d6b/d12a 0 2024-04-22T15:19:11.803 INFO:tasks.workunit.client.1.smithi196.stdout:1/821: write d1/db/dc/de/d43/d5e/fbf [201524,38648] 0 2024-04-22T15:19:11.805 INFO:tasks.workunit.client.1.smithi196.stdout:1/822: write d1/db/d16/d44/fb8 [1119355,1097] 0 2024-04-22T15:19:11.805 INFO:tasks.workunit.client.1.smithi196.stdout:3/894: fsync d0/d15/d31/d55/dcf/d6f/d10a/fc3 0 2024-04-22T15:19:11.807 INFO:tasks.workunit.client.1.smithi196.stdout:0/830: sync 2024-04-22T15:19:11.807 INFO:tasks.workunit.client.1.smithi196.stdout:5/847: sync 2024-04-22T15:19:11.807 INFO:tasks.workunit.client.1.smithi196.stdout:6/820: sync 2024-04-22T15:19:11.807 INFO:tasks.workunit.client.1.smithi196.stdout:7/814: truncate d14/d26/d2a/d72/db1/fd9 1129299 0 2024-04-22T15:19:11.807 INFO:tasks.workunit.client.0.smithi096.stdout:8/988: creat d4/d1c/d40/d46/d58/dcb/f150 x:0 0 0 2024-04-22T15:19:11.811 INFO:tasks.workunit.client.1.smithi196.stdout:9/846: getdents db/dd/de/dd7/def/d7a/d116 0 2024-04-22T15:19:11.811 INFO:tasks.workunit.client.1.smithi196.stdout:0/831: readlink d3/de/d14/d86/dea/l100 0 2024-04-22T15:19:11.811 INFO:tasks.workunit.client.1.smithi196.stdout:6/821: rename dd/d1f/d37/dcb to dd/d1f/d37/dcb/df5 22 2024-04-22T15:19:11.812 INFO:tasks.workunit.client.0.smithi096.stdout:8/989: creat d4/d10/d25/d38/d8f/d9c/d138/f151 x:0 0 0 2024-04-22T15:19:11.813 INFO:tasks.workunit.client.1.smithi196.stdout:7/815: write d14/d26/d5a/dbb/f8d [2853101,70347] 0 2024-04-22T15:19:11.813 INFO:tasks.workunit.client.1.smithi196.stdout:0/832: fdatasync d3/de/f24 0 2024-04-22T15:19:11.815 INFO:tasks.workunit.client.0.smithi096.stdout:8/990: mknod d4/d10/d25/d4c/db4/df4/c152 0 2024-04-22T15:19:11.815 INFO:tasks.workunit.client.1.smithi196.stdout:9/847: chown db/dd/de/ff1 1 1 2024-04-22T15:19:11.816 INFO:tasks.workunit.client.1.smithi196.stdout:8/800: dwrite d2/dc/d45/f5b [0,4194304] 0 2024-04-22T15:19:11.817 INFO:tasks.workunit.client.1.smithi196.stdout:1/823: truncate d1/db/dc/de/d43/ff5 5981865 0 2024-04-22T15:19:11.819 INFO:tasks.workunit.client.1.smithi196.stdout:6/822: unlink dd/d1f/d37/d3c/c44 0 2024-04-22T15:19:11.820 INFO:tasks.workunit.client.0.smithi096.stdout:8/991: truncate d4/d1c/d40/d48/f68 4286467 0 2024-04-22T15:19:11.821 INFO:tasks.workunit.client.1.smithi196.stdout:3/895: symlink d0/d15/d31/d55/dcf/d6f/d10a/l12b 0 2024-04-22T15:19:11.825 INFO:tasks.workunit.client.1.smithi196.stdout:9/848: mkdir db/dd/de/d15/d31/d64/d11b 0 2024-04-22T15:19:11.827 INFO:tasks.workunit.client.1.smithi196.stdout:9/849: read db/dd/de/d75/faa [759169,103378] 0 2024-04-22T15:19:11.828 INFO:tasks.workunit.client.1.smithi196.stdout:1/824: creat d1/db/d16/d6d/f109 x:0 0 0 2024-04-22T15:19:11.829 INFO:tasks.workunit.client.1.smithi196.stdout:1/825: chown d1/db/d16/d1e/d49/d90 12700 1 2024-04-22T15:19:11.830 INFO:tasks.workunit.client.1.smithi196.stdout:6/823: fdatasync fc 0 2024-04-22T15:19:11.831 INFO:tasks.workunit.client.1.smithi196.stdout:6/824: readlink dd/d2b/l2c 0 2024-04-22T15:19:11.831 INFO:tasks.workunit.client.1.smithi196.stdout:7/816: creat d14/d4d/dea/f10e x:0 0 0 2024-04-22T15:19:11.831 INFO:tasks.workunit.client.1.smithi196.stdout:3/896: symlink d0/d15/da7/d4a/dc5/l12c 0 2024-04-22T15:19:11.832 INFO:tasks.workunit.client.1.smithi196.stdout:7/817: stat d14/d26/db7/d10b 0 2024-04-22T15:19:11.833 INFO:tasks.workunit.client.1.smithi196.stdout:2/840: dread d1/f15 [0,4194304] 0 2024-04-22T15:19:11.834 INFO:tasks.workunit.client.1.smithi196.stdout:9/850: mknod db/dd/d1b/de6/c11c 0 2024-04-22T15:19:11.834 INFO:tasks.workunit.client.1.smithi196.stdout:7/818: write d14/d25/ff2 [496566,73968] 0 2024-04-22T15:19:11.836 INFO:tasks.workunit.client.1.smithi196.stdout:8/801: creat d2/d8/fe8 x:0 0 0 2024-04-22T15:19:11.837 INFO:tasks.workunit.client.1.smithi196.stdout:7/819: write d14/d25/d31/fd5 [977860,94105] 0 2024-04-22T15:19:11.840 INFO:tasks.workunit.client.1.smithi196.stdout:3/897: creat d0/d15/d31/d55/dcf/d6f/de5/f12d x:0 0 0 2024-04-22T15:19:11.842 INFO:tasks.workunit.client.1.smithi196.stdout:3/898: fdatasync d0/d15/da7/d4a/d44/d96/dd8/d123/f124 0 2024-04-22T15:19:11.842 INFO:tasks.workunit.client.1.smithi196.stdout:2/841: dwrite d1/d23/d27/d2e/f88 [0,4194304] 0 2024-04-22T15:19:11.844 INFO:tasks.workunit.client.1.smithi196.stdout:6/825: mkdir dd/d1f/d37/df6 0 2024-04-22T15:19:11.845 INFO:tasks.workunit.client.1.smithi196.stdout:9/851: creat db/dd/de/dd7/def/d7a/d7b/d81/f11d x:0 0 0 2024-04-22T15:19:11.846 INFO:tasks.workunit.client.0.smithi096.stdout:8/992: truncate d4/d71/fff 868905 0 2024-04-22T15:19:11.847 INFO:tasks.workunit.client.1.smithi196.stdout:8/802: chown d2/d8/d49/d6c/dde 2976685 1 2024-04-22T15:19:11.848 INFO:tasks.workunit.client.1.smithi196.stdout:7/820: dread d14/fa5 [0,4194304] 0 2024-04-22T15:19:11.849 INFO:tasks.workunit.client.1.smithi196.stdout:0/833: dwrite d3/d9/d36/f4c [0,4194304] 0 2024-04-22T15:19:11.851 INFO:tasks.workunit.client.1.smithi196.stdout:3/899: symlink d0/d15/d31/d55/d6e/l12e 0 2024-04-22T15:19:11.852 INFO:tasks.workunit.client.0.smithi096.stdout:8/993: symlink d4/d1c/d40/da7/db6/l153 0 2024-04-22T15:19:11.855 INFO:tasks.workunit.client.1.smithi196.stdout:6/826: mkdir dd/d1f/d4e/d53/d55/d6d/dd1/de3/df7 0 2024-04-22T15:19:11.857 INFO:tasks.workunit.client.1.smithi196.stdout:0/834: mknod d3/de/d14/d86/c128 0 2024-04-22T15:19:11.859 INFO:tasks.workunit.client.0.smithi096.stdout:8/994: dwrite d4/d113/f12b [0,4194304] 0 2024-04-22T15:19:11.861 INFO:tasks.workunit.client.1.smithi196.stdout:7/821: dwrite d14/d26/d5a/d5f/fcd [0,4194304] 0 2024-04-22T15:19:11.861 INFO:tasks.workunit.client.0.smithi096.stdout:8/995: write d4/d1c/d40/d46/d58/f132 [467959,94991] 0 2024-04-22T15:19:11.861 INFO:tasks.workunit.client.1.smithi196.stdout:0/835: chown d3/de/d14 0 1 2024-04-22T15:19:11.865 INFO:tasks.workunit.client.0.smithi096.stdout:9/939: sync 2024-04-22T15:19:11.865 INFO:tasks.workunit.client.0.smithi096.stdout:6/917: sync 2024-04-22T15:19:11.867 INFO:tasks.workunit.client.1.smithi196.stdout:0/836: readlink d3/de/d14/d5a/l80 0 2024-04-22T15:19:11.871 INFO:tasks.workunit.client.0.smithi096.stdout:8/996: dwrite d4/d10/d25/d38/d13f/f145 [0,4194304] 0 2024-04-22T15:19:11.872 INFO:tasks.workunit.client.0.smithi096.stdout:9/940: rmdir d1/df/d95/d9d/db5/dd/dba/dc2/de5 39 2024-04-22T15:19:11.873 INFO:tasks.workunit.client.1.smithi196.stdout:3/900: dwrite d0/d15/da7/d4a/d44/d8c/fbc [0,4194304] 0 2024-04-22T15:19:11.874 INFO:tasks.workunit.client.0.smithi096.stdout:6/918: chown d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/ddd/d117 251 1 2024-04-22T15:19:11.877 INFO:tasks.workunit.client.1.smithi196.stdout:1/826: dwrite d1/db/dc/de/d43/ff5 [0,4194304] 0 2024-04-22T15:19:11.878 INFO:tasks.workunit.client.0.smithi096.stdout:8/997: symlink d4/d1c/d40/d46/d120/l154 0 2024-04-22T15:19:11.878 INFO:tasks.workunit.client.0.smithi096.stdout:8/998: chown d4/d2a/d7e/f9a 10440 1 2024-04-22T15:19:11.879 INFO:tasks.workunit.client.1.smithi196.stdout:7/822: dwrite d14/d25/d80/fbd [0,4194304] 0 2024-04-22T15:19:11.879 INFO:tasks.workunit.client.1.smithi196.stdout:2/842: write d1/d23/d27/d2e/d55/d5b/d7c/fe4 [683466,57303] 0 2024-04-22T15:19:11.880 INFO:tasks.workunit.client.0.smithi096.stdout:8/999: chown d4/d10/f99 3 1 2024-04-22T15:19:11.881 INFO:tasks.workunit.client.1.smithi196.stdout:3/901: dread - d0/d15/d31/d55/dcf/d6f/de5/f12d zero size 2024-04-22T15:19:11.883 INFO:tasks.workunit.client.0.smithi096.stdout:9/941: mkdir d1/df/d95/d9d/db5/dd/dba/dc2/de5/d13e 0 2024-04-22T15:19:11.884 INFO:tasks.workunit.client.0.smithi096.stdout:6/919: symlink d3/d28/d4d/d82/d9e/d137/d11/d2c/d106/l139 0 2024-04-22T15:19:11.884 INFO:tasks.workunit.client.0.smithi096.stdout:6/920: readlink d3/l2a 0 2024-04-22T15:19:11.887 INFO:tasks.workunit.client.1.smithi196.stdout:7/823: symlink d14/d4d/da3/l10f 0 2024-04-22T15:19:11.888 INFO:tasks.workunit.client.1.smithi196.stdout:9/852: truncate db/dd/de/dd7/def/d57/da7/fff 662371 0 2024-04-22T15:19:11.889 INFO:tasks.workunit.client.0.smithi096.stdout:6/921: dread - d3/d28/d4d/d82/d9e/d137/d11/d2c/d3b/f124 zero size 2024-04-22T15:19:11.890 INFO:tasks.workunit.client.1.smithi196.stdout:9/853: fdatasync db/dd/dd0/fe5 0 2024-04-22T15:19:11.891 INFO:tasks.workunit.client.1.smithi196.stdout:1/827: creat d1/d54/d70/f10a x:0 0 0 2024-04-22T15:19:11.893 INFO:tasks.workunit.client.1.smithi196.stdout:7/824: mknod d14/d26/d28/d29/c110 0 2024-04-22T15:19:11.895 INFO:tasks.workunit.client.0.smithi096.stdout:6/922: dread d3/d3a/d75/f105 [4194304,4194304] 0 2024-04-22T15:19:11.895 INFO:tasks.workunit.client.1.smithi196.stdout:3/902: dread d0/d15/da7/fa8 [0,4194304] 0 2024-04-22T15:19:11.897 INFO:tasks.workunit.client.0.smithi096.stdout:9/942: link d1/df/c37 d1/df/d95/d9d/db5/dd/dba/dc2/dc4/c13f 0 2024-04-22T15:19:11.898 INFO:tasks.workunit.client.1.smithi196.stdout:7/825: dread d14/d26/d5a/d5f/fcd [0,4194304] 0 2024-04-22T15:19:11.900 INFO:tasks.workunit.client.1.smithi196.stdout:3/903: rename d0/d15/da7/l2c to d0/d15/d31/l12f 0 2024-04-22T15:19:11.902 INFO:tasks.workunit.client.0.smithi096.stdout:9/943: mknod d1/c140 0 2024-04-22T15:19:11.902 INFO:tasks.workunit.client.1.smithi196.stdout:6/827: write dd/d1f/d4e/d53/d7f/da5/fdc [7548573,1461] 0 2024-04-22T15:19:11.904 INFO:tasks.workunit.client.1.smithi196.stdout:6/828: dread dd/f18 [8388608,4194304] 0 2024-04-22T15:19:11.905 INFO:tasks.workunit.client.1.smithi196.stdout:7/826: rename d14/d15/f1a to d14/d26/d5a/dbb/f111 0 2024-04-22T15:19:11.905 INFO:tasks.workunit.client.1.smithi196.stdout:3/904: rmdir d0/d15/d31/d55/dcf/d6f/d10a/d5e 39 2024-04-22T15:19:11.907 INFO:tasks.workunit.client.1.smithi196.stdout:6/829: fdatasync dd/d1f/d4e/d53/d55/f9a 0 2024-04-22T15:19:11.908 INFO:tasks.workunit.client.1.smithi196.stdout:7/827: truncate d14/d26/d2a/d72/f99 4904633 0 2024-04-22T15:19:11.912 INFO:tasks.workunit.client.1.smithi196.stdout:7/828: mknod d14/d4d/da3/c112 0 2024-04-22T15:19:11.913 INFO:tasks.workunit.client.0.smithi096.stdout:6/923: dread d3/f22 [0,4194304] 0 2024-04-22T15:19:11.914 INFO:tasks.workunit.client.1.smithi196.stdout:3/905: rename d0/d15/d31/d55/c90 to d0/d15/d31/d55/dcf/d8/d6b/d12a/c130 0 2024-04-22T15:19:11.914 INFO:tasks.workunit.client.1.smithi196.stdout:0/837: dread d3/de/d1c/f31 [0,4194304] 0 2024-04-22T15:19:11.915 INFO:tasks.workunit.client.1.smithi196.stdout:0/838: stat d3/d6/dc/d8c/d8d/fc1 0 2024-04-22T15:19:11.915 INFO:tasks.workunit.client.1.smithi196.stdout:6/830: symlink dd/d1f/d4e/d53/d7f/dea/lf8 0 2024-04-22T15:19:11.916 INFO:tasks.workunit.client.1.smithi196.stdout:6/831: read dd/d1f/d38/d40/f5f [238062,54844] 0 2024-04-22T15:19:11.918 INFO:tasks.workunit.client.1.smithi196.stdout:7/829: creat d14/d26/d2a/d72/db1/f113 x:0 0 0 2024-04-22T15:19:11.922 INFO:tasks.workunit.client.1.smithi196.stdout:2/843: write d1/d76/f7d [2199679,40694] 0 2024-04-22T15:19:11.924 INFO:tasks.workunit.client.1.smithi196.stdout:3/906: getdents d0/d15/d31/d55/d8b/d102 0 2024-04-22T15:19:11.925 INFO:tasks.workunit.client.1.smithi196.stdout:9/854: write db/dd/d1b/f68 [13092,52215] 0 2024-04-22T15:19:11.926 INFO:tasks.workunit.client.1.smithi196.stdout:2/844: unlink d1/d76/f10b 0 2024-04-22T15:19:11.930 INFO:tasks.workunit.client.1.smithi196.stdout:1/828: dwrite d1/db/d16/d1e/f60 [0,4194304] 0 2024-04-22T15:19:11.931 INFO:tasks.workunit.client.1.smithi196.stdout:9/855: mkdir db/dd/de/d15/d31/d99/d11e 0 2024-04-22T15:19:11.934 INFO:tasks.workunit.client.1.smithi196.stdout:7/830: creat d14/d15/f114 x:0 0 0 2024-04-22T15:19:11.934 INFO:tasks.workunit.client.1.smithi196.stdout:0/839: getdents d3/d9/d36/d114 0 2024-04-22T15:19:11.938 INFO:tasks.workunit.client.1.smithi196.stdout:9/856: mkdir db/dd/d1b/de6/d11f 0 2024-04-22T15:19:11.939 INFO:tasks.workunit.client.1.smithi196.stdout:9/857: dread - db/dd/de/dd7/def/f105 zero size 2024-04-22T15:19:11.941 INFO:tasks.workunit.client.0.smithi096.stdout:9/944: dwrite d1/df/d95/d9d/db5/dd/dba/dc2/dc4/fe9 [0,4194304] 0 2024-04-22T15:19:11.942 INFO:tasks.workunit.client.0.smithi096.stdout:9/945: dread - d1/df/d95/d9d/db5/dd/f128 zero size 2024-04-22T15:19:11.943 INFO:tasks.workunit.client.1.smithi196.stdout:0/840: symlink d3/de/d14/d5a/l129 0 2024-04-22T15:19:11.945 INFO:tasks.workunit.client.0.smithi096.stdout:6/924: truncate d3/f49 2778676 0 2024-04-22T15:19:11.946 INFO:tasks.workunit.client.1.smithi196.stdout:3/907: link d0/d15/d31/d55/d6e/f8d d0/d15/d31/d55/dcf/d8/dfc/f131 0 2024-04-22T15:19:11.947 INFO:tasks.workunit.client.0.smithi096.stdout:9/946: rmdir d1/d4/d7f/d12d 39 2024-04-22T15:19:11.947 INFO:tasks.workunit.client.1.smithi196.stdout:3/908: truncate d0/d15/d31/d55/dcf/d8/fea 310054 0 2024-04-22T15:19:11.948 INFO:tasks.workunit.client.0.smithi096.stdout:9/947: chown d1/d25/db2/d98/d9b 3951761 1 2024-04-22T15:19:11.949 INFO:tasks.workunit.client.0.smithi096.stdout:9/948: chown d1/df/ff5 98364843 1 2024-04-22T15:19:11.949 INFO:tasks.workunit.client.0.smithi096.stdout:9/949: chown d1/d4/d7f/f83 8285590 1 2024-04-22T15:19:11.954 INFO:tasks.workunit.client.1.smithi196.stdout:7/831: mknod d14/c115 0 2024-04-22T15:19:11.955 INFO:tasks.workunit.client.0.smithi096.stdout:6/925: creat d3/d28/f13a x:0 0 0 2024-04-22T15:19:11.955 INFO:tasks.workunit.client.0.smithi096.stdout:9/950: read d1/df/d95/d9d/db5/f24 [1072765,32532] 0 2024-04-22T15:19:11.956 INFO:tasks.workunit.client.1.smithi196.stdout:9/858: dwrite db/dd/de/dd7/def/fde [0,4194304] 0 2024-04-22T15:19:11.959 INFO:tasks.workunit.client.1.smithi196.stdout:1/829: rename d1/db/dc/de/cc3 to d1/db/dc/de/d5a/c10b 0 2024-04-22T15:19:11.959 INFO:tasks.workunit.client.0.smithi096.stdout:6/926: dread d3/d3a/d75/f105 [4194304,4194304] 0 2024-04-22T15:19:11.961 INFO:tasks.workunit.client.0.smithi096.stdout:9/951: mknod d1/df/d95/d123/c141 0 2024-04-22T15:19:11.961 INFO:tasks.workunit.client.1.smithi196.stdout:1/830: write d1/db/dc/de/d43/d5e/fbf [203990,49212] 0 2024-04-22T15:19:11.963 INFO:tasks.workunit.client.1.smithi196.stdout:2/845: link d1/d13/d25/c3b d1/d23/d126/c12f 0 2024-04-22T15:19:11.966 INFO:tasks.workunit.client.1.smithi196.stdout:7/832: rmdir d14/d26/db7 39 2024-04-22T15:19:11.967 INFO:tasks.workunit.client.1.smithi196.stdout:7/833: readlink d14/d26/d5a/d5f/dc2/lfc 0 2024-04-22T15:19:11.969 INFO:tasks.workunit.client.0.smithi096.stdout:6/927: creat d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/ddd/d117/f13b x:0 0 0 2024-04-22T15:19:11.969 INFO:tasks.workunit.client.1.smithi196.stdout:9/859: stat db/dd/de/dd7/def/f2b 0 2024-04-22T15:19:11.971 INFO:tasks.workunit.client.0.smithi096.stdout:9/952: getdents d1/d4/d7f 0 2024-04-22T15:19:11.972 INFO:tasks.workunit.client.1.smithi196.stdout:0/841: dwrite d3/d6/dc/d8c/d8d/d98/fa6 [0,4194304] 0 2024-04-22T15:19:11.972 INFO:tasks.workunit.client.1.smithi196.stdout:1/831: fsync d1/db/fc0 0 2024-04-22T15:19:11.974 INFO:tasks.workunit.client.1.smithi196.stdout:1/832: fsync d1/db/dc/de/d43/f6b 0 2024-04-22T15:19:11.974 INFO:tasks.workunit.client.1.smithi196.stdout:2/846: readlink d1/d23/d27/d73/l86 0 2024-04-22T15:19:11.974 INFO:tasks.workunit.client.0.smithi096.stdout:9/953: fdatasync d1/df/ff5 0 2024-04-22T15:19:11.976 INFO:tasks.workunit.client.1.smithi196.stdout:2/847: read d1/d13/d25/d2a/f95 [99632,112394] 0 2024-04-22T15:19:11.977 INFO:tasks.workunit.client.0.smithi096.stdout:6/928: symlink d3/d28/d4d/d82/d9e/l13c 0 2024-04-22T15:19:11.979 INFO:tasks.workunit.client.0.smithi096.stdout:6/929: write d3/d28/daf/f115 [429967,58249] 0 2024-04-22T15:19:11.981 INFO:tasks.workunit.client.1.smithi196.stdout:2/848: dwrite d1/d13/d1f/f9d [0,4194304] 0 2024-04-22T15:19:11.984 INFO:tasks.workunit.client.1.smithi196.stdout:7/834: rename d14/d26/d5e/ca9 to d14/d4d/da3/dc8/c116 0 2024-04-22T15:19:11.985 INFO:tasks.workunit.client.1.smithi196.stdout:9/860: fdatasync db/d11/fb2 0 2024-04-22T15:19:11.986 INFO:tasks.workunit.client.0.smithi096.stdout:9/954: chown d1/df/d95/d9d/db5/d43/fa3 5156 1 2024-04-22T15:19:11.987 INFO:tasks.workunit.client.0.smithi096.stdout:6/930: truncate d3/d28/d4d/d82/d9e/d137/f16 922371 0 2024-04-22T15:19:11.990 INFO:tasks.workunit.client.1.smithi196.stdout:0/842: unlink d3/d9/d36/d9f/dd1/d65/d7f/d51/f6a 0 2024-04-22T15:19:11.991 INFO:tasks.workunit.client.1.smithi196.stdout:3/909: getdents d0/d15/d31/d55 0 2024-04-22T15:19:11.993 INFO:tasks.workunit.client.0.smithi096.stdout:9/955: dwrite d1/f136 [0,4194304] 0 2024-04-22T15:19:11.994 INFO:tasks.workunit.client.0.smithi096.stdout:6/931: creat d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/ddd/de9/f13d x:0 0 0 2024-04-22T15:19:11.995 INFO:tasks.workunit.client.1.smithi196.stdout:7/835: fdatasync d14/d25/f90 0 2024-04-22T15:19:11.999 INFO:tasks.workunit.client.1.smithi196.stdout:9/861: mkdir db/dd/d1b/de6/d120 0 2024-04-22T15:19:12.000 INFO:tasks.workunit.client.0.smithi096.stdout:9/956: mkdir d1/df/d95/d9d/db5/dd/dba/dc2/d120/d142 0 2024-04-22T15:19:12.003 INFO:tasks.workunit.client.1.smithi196.stdout:1/833: mkdir d1/db/dc/d26/dc4/d10c 0 2024-04-22T15:19:12.005 INFO:tasks.workunit.client.1.smithi196.stdout:0/843: rename d3/de/d14/d86/l90 to d3/de/d1c/dd4/l12a 0 2024-04-22T15:19:12.008 INFO:tasks.workunit.client.0.smithi096.stdout:9/957: creat d1/df/d95/d9d/db5/dd/dba/dc2/d120/dbd/f143 x:0 0 0 2024-04-22T15:19:12.009 INFO:tasks.workunit.client.1.smithi196.stdout:3/910: mknod d0/d15/d31/dcb/c132 0 2024-04-22T15:19:12.011 INFO:tasks.workunit.client.0.smithi096.stdout:9/958: fdatasync d1/d4/f7a 0 2024-04-22T15:19:12.013 INFO:tasks.workunit.client.1.smithi196.stdout:7/836: creat d14/d4d/da3/de9/f117 x:0 0 0 2024-04-22T15:19:12.015 INFO:tasks.workunit.client.0.smithi096.stdout:6/932: dread d3/d28/f38 [0,4194304] 0 2024-04-22T15:19:12.017 INFO:tasks.workunit.client.1.smithi196.stdout:1/834: creat d1/db/d16/d6d/de6/f10d x:0 0 0 2024-04-22T15:19:12.017 INFO:tasks.workunit.client.1.smithi196.stdout:0/844: chown d3/de/d1c/l123 6 1 2024-04-22T15:19:12.020 INFO:tasks.workunit.client.1.smithi196.stdout:2/849: creat d1/d23/d27/d2e/d55/d5b/d93/f130 x:0 0 0 2024-04-22T15:19:12.021 INFO:tasks.workunit.client.1.smithi196.stdout:3/911: mknod d0/d15/da7/d4a/d44/d96/c133 0 2024-04-22T15:19:12.022 INFO:tasks.workunit.client.1.smithi196.stdout:3/912: truncate d0/d15/d4d/de7/f114 181529 0 2024-04-22T15:19:12.023 INFO:tasks.workunit.client.1.smithi196.stdout:9/862: mknod db/dd/de/dd7/def/d7a/d116/c121 0 2024-04-22T15:19:12.024 INFO:tasks.workunit.client.1.smithi196.stdout:9/863: stat db/dd/de/f54 0 2024-04-22T15:19:12.025 INFO:tasks.workunit.client.1.smithi196.stdout:3/913: write d0/d15/d31/d55/dcf/d6f/d10a/fc3 [441636,112055] 0 2024-04-22T15:19:12.026 INFO:tasks.workunit.client.1.smithi196.stdout:0/845: unlink d3/d9/d36/f104 0 2024-04-22T15:19:12.027 INFO:tasks.workunit.client.1.smithi196.stdout:2/850: dwrite d1/d23/d27/d2e/f88 [0,4194304] 0 2024-04-22T15:19:12.028 INFO:tasks.workunit.client.1.smithi196.stdout:0/846: chown d3/de/d14/d86/c128 4329900 1 2024-04-22T15:19:12.028 INFO:tasks.workunit.client.1.smithi196.stdout:7/837: fdatasync d14/f6b 0 2024-04-22T15:19:12.030 INFO:tasks.workunit.client.0.smithi096.stdout:9/959: write d1/df/d95/d9d/db5/dd/f41 [2557446,29718] 0 2024-04-22T15:19:12.031 INFO:tasks.workunit.client.1.smithi196.stdout:2/851: fdatasync d1/f15 0 2024-04-22T15:19:12.033 INFO:tasks.workunit.client.1.smithi196.stdout:2/852: write d1/d23/d27/d2e/d55/d5b/d7c/fe4 [1442886,109971] 0 2024-04-22T15:19:12.037 INFO:tasks.workunit.client.1.smithi196.stdout:7/838: rmdir d14/d26/d5e 39 2024-04-22T15:19:12.039 INFO:tasks.workunit.client.0.smithi096.stdout:9/960: dwrite d1/df/d95/d9d/db5/d43/d57/d5b/fb4 [0,4194304] 0 2024-04-22T15:19:12.041 INFO:tasks.workunit.client.0.smithi096.stdout:9/961: chown d1/d4/d7f/f83 1763 1 2024-04-22T15:19:12.041 INFO:tasks.workunit.client.1.smithi196.stdout:3/914: rename d0/d15/d31/d55/dcf/d6f/de5/f128 to d0/d15/d31/d55/d8b/df2/f134 0 2024-04-22T15:19:12.042 INFO:tasks.workunit.client.1.smithi196.stdout:1/835: dwrite d1/db/d16/d6d/f7c [0,4194304] 0 2024-04-22T15:19:12.043 INFO:tasks.workunit.client.1.smithi196.stdout:9/864: symlink db/dd/d1b/l122 0 2024-04-22T15:19:12.048 INFO:tasks.workunit.client.0.smithi096.stdout:9/962: mkdir d1/df/d95/d9d/db5/dd/dba/dc2/d144 0 2024-04-22T15:19:12.050 INFO:tasks.workunit.client.0.smithi096.stdout:9/963: mknod d1/d85/df4/c145 0 2024-04-22T15:19:12.052 INFO:tasks.workunit.client.0.smithi096.stdout:9/964: write d1/d25/d9f/fde [1201082,34311] 0 2024-04-22T15:19:12.053 INFO:tasks.workunit.client.1.smithi196.stdout:9/865: creat db/dd/de/dd7/f123 x:0 0 0 2024-04-22T15:19:12.054 INFO:tasks.workunit.client.1.smithi196.stdout:1/836: truncate d1/db/dc/de/f14 3411300 0 2024-04-22T15:19:12.057 INFO:tasks.workunit.client.1.smithi196.stdout:0/847: dread d3/de/d1c/d27/f38 [0,4194304] 0 2024-04-22T15:19:12.057 INFO:tasks.workunit.client.1.smithi196.stdout:9/866: mkdir db/dd/de/d75/d124 0 2024-04-22T15:19:12.058 INFO:tasks.workunit.client.1.smithi196.stdout:0/848: chown d3/d6/dc/d8c/daf 4 1 2024-04-22T15:19:12.059 INFO:tasks.workunit.client.1.smithi196.stdout:5/848: sync 2024-04-22T15:19:12.059 INFO:tasks.workunit.client.1.smithi196.stdout:4/818: sync 2024-04-22T15:19:12.059 INFO:tasks.workunit.client.0.smithi096.stdout:9/965: rmdir d1/df/d95/d123/d12c 0 2024-04-22T15:19:12.062 INFO:tasks.workunit.client.1.smithi196.stdout:1/837: rmdir d1/db/d16 39 2024-04-22T15:19:12.065 INFO:tasks.workunit.client.0.smithi096.stdout:9/966: fsync d1/d25/db2/f96 0 2024-04-22T15:19:12.065 INFO:tasks.workunit.client.1.smithi196.stdout:2/853: link d1/d23/d84/l8f d1/d23/d27/d2e/d55/d5b/d93/l131 0 2024-04-22T15:19:12.067 INFO:tasks.workunit.client.1.smithi196.stdout:0/849: creat d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9/f12b x:0 0 0 2024-04-22T15:19:12.069 INFO:tasks.workunit.client.1.smithi196.stdout:5/849: stat d1/d2/d98/dac/d105/l72 0 2024-04-22T15:19:12.071 INFO:tasks.workunit.client.1.smithi196.stdout:4/819: truncate d6/d10/fb1 926691 0 2024-04-22T15:19:12.071 INFO:tasks.workunit.client.1.smithi196.stdout:5/850: read d1/d17/f8b [3306604,21987] 0 2024-04-22T15:19:12.072 INFO:tasks.workunit.client.1.smithi196.stdout:3/915: write d0/d15/da7/d18/f4c [2124003,40213] 0 2024-04-22T15:19:12.073 INFO:tasks.workunit.client.1.smithi196.stdout:5/851: readlink d1/d17/d7f/deb/lc7 0 2024-04-22T15:19:12.075 INFO:tasks.workunit.client.1.smithi196.stdout:7/839: dwrite d14/d26/d28/f89 [0,4194304] 0 2024-04-22T15:19:12.076 INFO:tasks.workunit.client.0.smithi096.stdout:9/967: creat d1/df/d95/d9d/db5/f146 x:0 0 0 2024-04-22T15:19:12.077 INFO:tasks.workunit.client.1.smithi196.stdout:0/850: creat d3/db4/f12c x:0 0 0 2024-04-22T15:19:12.082 INFO:tasks.workunit.client.1.smithi196.stdout:1/838: dread d1/db/d16/d1e/f60 [0,4194304] 0 2024-04-22T15:19:12.082 INFO:tasks.workunit.client.1.smithi196.stdout:4/820: fsync d6/d14/d18/d26/d32/da3/db7/fef 0 2024-04-22T15:19:12.082 INFO:tasks.workunit.client.1.smithi196.stdout:9/867: write db/dd/de/d15/d31/f32 [841577,47085] 0 2024-04-22T15:19:12.082 INFO:tasks.workunit.client.0.smithi096.stdout:9/968: mknod d1/df/d95/d9d/db5/c147 0 2024-04-22T15:19:12.083 INFO:tasks.workunit.client.1.smithi196.stdout:1/839: chown d1/d54/d70/f10a 44 1 2024-04-22T15:19:12.085 INFO:tasks.workunit.client.1.smithi196.stdout:5/852: truncate d1/fbf 985595 0 2024-04-22T15:19:12.085 INFO:tasks.workunit.client.1.smithi196.stdout:9/868: truncate db/f61 4937120 0 2024-04-22T15:19:12.086 INFO:tasks.workunit.client.1.smithi196.stdout:7/840: dwrite d14/d26/d2a/d72/db1/f113 [0,4194304] 0 2024-04-22T15:19:12.086 INFO:tasks.workunit.client.1.smithi196.stdout:2/854: write d1/d76/fb9 [399420,43596] 0 2024-04-22T15:19:12.087 INFO:tasks.workunit.client.1.smithi196.stdout:4/821: chown d6/d13/d15/d2c/d62/d78/ddc/lfa 75 1 2024-04-22T15:19:12.088 INFO:tasks.workunit.client.1.smithi196.stdout:9/869: chown db/dd/de/dd7/def/d7a/d7b/l9a 326333 1 2024-04-22T15:19:12.090 INFO:tasks.workunit.client.1.smithi196.stdout:9/870: chown db/dd/d1b/d2c/f58 10791 1 2024-04-22T15:19:12.090 INFO:tasks.workunit.client.1.smithi196.stdout:1/840: rmdir d1/db/dc/dad 39 2024-04-22T15:19:12.095 INFO:tasks.workunit.client.0.smithi096.stdout:9/969: dwrite d1/df/d95/d9d/db5/d43/f5f [8388608,4194304] 0 2024-04-22T15:19:12.096 INFO:tasks.workunit.client.0.smithi096.stdout:9/970: stat d1/df/d95/d9d/db5/d43/d57/d5b/d73/l11d 0 2024-04-22T15:19:12.099 INFO:tasks.workunit.client.0.smithi096.stdout:9/971: truncate d1/d25/fd5 4254569 0 2024-04-22T15:19:12.111 INFO:tasks.workunit.client.0.smithi096.stdout:9/972: dwrite d1/d4/d79/f86 [0,4194304] 0 2024-04-22T15:19:12.112 INFO:tasks.workunit.client.1.smithi196.stdout:2/855: unlink d1/d13/d41/fa8 0 2024-04-22T15:19:12.112 INFO:tasks.workunit.client.1.smithi196.stdout:1/841: stat d1/db/dc/de/d5a/d94/dab/fd0 0 2024-04-22T15:19:12.113 INFO:tasks.workunit.client.1.smithi196.stdout:3/916: write d0/d15/da7/d4a/f86 [1018654,25687] 0 2024-04-22T15:19:12.114 INFO:tasks.workunit.client.1.smithi196.stdout:0/851: dwrite d3/f5 [0,4194304] 0 2024-04-22T15:19:12.116 INFO:tasks.workunit.client.1.smithi196.stdout:4/822: creat d6/d13/d15/d2c/d8c/db3/dce/f10a x:0 0 0 2024-04-22T15:19:12.116 INFO:tasks.workunit.client.1.smithi196.stdout:5/853: creat d1/d2/d98/f10e x:0 0 0 2024-04-22T15:19:12.120 INFO:tasks.workunit.client.1.smithi196.stdout:1/842: rmdir d1/db/d16/d1e/d49/d7f 39 2024-04-22T15:19:12.122 INFO:tasks.workunit.client.0.smithi096.stdout:9/973: write d1/df/d95/d9d/db5/d43/d57/d5b/f66 [334580,23988] 0 2024-04-22T15:19:12.124 INFO:tasks.workunit.client.1.smithi196.stdout:9/871: link db/dd/de/dd7/def/d57/da7/lcd db/dd/d1b/de6/d120/l125 0 2024-04-22T15:19:12.126 INFO:tasks.workunit.client.1.smithi196.stdout:9/872: chown db/dd/d1b/f51 8003 1 2024-04-22T15:19:12.127 INFO:tasks.workunit.client.1.smithi196.stdout:2/856: dwrite d1/d23/d27/d100/f125 [0,4194304] 0 2024-04-22T15:19:12.127 INFO:tasks.workunit.client.1.smithi196.stdout:3/917: fsync d0/d15/da7/d4a/f52 0 2024-04-22T15:19:12.128 INFO:tasks.workunit.client.1.smithi196.stdout:5/854: fdatasync d1/d2/d98/dac/fd3 0 2024-04-22T15:19:12.129 INFO:tasks.workunit.client.1.smithi196.stdout:9/873: write db/dd/de/d75/fbf [1950085,48897] 0 2024-04-22T15:19:12.132 INFO:tasks.workunit.client.1.smithi196.stdout:0/852: mknod d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/db7/d11b/c12d 0 2024-04-22T15:19:12.133 INFO:tasks.workunit.client.1.smithi196.stdout:5/855: symlink d1/d17/d1c/l10f 0 2024-04-22T15:19:12.135 INFO:tasks.workunit.client.1.smithi196.stdout:9/874: dread - db/dd/de/dd7/def/f84 zero size 2024-04-22T15:19:12.137 INFO:tasks.workunit.client.1.smithi196.stdout:2/857: mknod d1/d1d/c132 0 2024-04-22T15:19:12.137 INFO:tasks.workunit.client.1.smithi196.stdout:4/823: write d6/d14/d18/d26/f6a [5059137,36734] 0 2024-04-22T15:19:12.138 INFO:tasks.workunit.client.0.smithi096.stdout:9/974: write d1/f38 [3803751,2216] 0 2024-04-22T15:19:12.139 INFO:tasks.workunit.client.1.smithi196.stdout:5/856: readlink d1/d17/d1c/d67/lb7 0 2024-04-22T15:19:12.140 INFO:tasks.workunit.client.1.smithi196.stdout:9/875: fsync db/dd/d1b/f51 0 2024-04-22T15:19:12.142 INFO:tasks.workunit.client.0.smithi096.stdout:9/975: readlink d1/df/d95/d9d/db5/d43/l4b 0 2024-04-22T15:19:12.143 INFO:tasks.workunit.client.1.smithi196.stdout:0/853: mkdir d3/d9/d36/d12e 0 2024-04-22T15:19:12.145 INFO:tasks.workunit.client.1.smithi196.stdout:2/858: rename d1/d13/d25/d2a/ddb/lfb to d1/d13/d25/d9f/ddd/l133 0 2024-04-22T15:19:12.146 INFO:tasks.workunit.client.0.smithi096.stdout:9/976: read - d1/d4/d7b/fa6 zero size 2024-04-22T15:19:12.149 INFO:tasks.workunit.client.0.smithi096.stdout:9/977: mkdir d1/df/d95/d9d/db5/dd/dba/dc2/de5/d148 0 2024-04-22T15:19:12.150 INFO:tasks.workunit.client.1.smithi196.stdout:0/854: dread d3/de/d14/d5a/f68 [0,4194304] 0 2024-04-22T15:19:12.152 INFO:tasks.workunit.client.1.smithi196.stdout:4/824: rename d6/d14/d18/d26/d32/d53/f54 to d6/d13/d15/d2c/d62/d78/d8d/f10b 0 2024-04-22T15:19:12.152 INFO:tasks.workunit.client.1.smithi196.stdout:1/843: write d1/db/d16/d44/f9e [1131898,18958] 0 2024-04-22T15:19:12.153 INFO:tasks.workunit.client.1.smithi196.stdout:2/859: symlink d1/d23/d27/d2e/d55/l134 0 2024-04-22T15:19:12.156 INFO:tasks.workunit.client.0.smithi096.stdout:9/978: mknod d1/df/d95/d9d/db5/c149 0 2024-04-22T15:19:12.157 INFO:tasks.workunit.client.1.smithi196.stdout:0/855: mknod d3/d6/dc/d8c/d8d/d98/dbd/c12f 0 2024-04-22T15:19:12.158 INFO:tasks.workunit.client.1.smithi196.stdout:6/832: sync 2024-04-22T15:19:12.158 INFO:tasks.workunit.client.1.smithi196.stdout:8/803: sync 2024-04-22T15:19:12.158 INFO:tasks.workunit.client.1.smithi196.stdout:0/856: readlink d3/d6/dc/l1b 0 2024-04-22T15:19:12.159 INFO:tasks.workunit.client.1.smithi196.stdout:5/857: rename d1/d4/cd8 to d1/d4/d92/db5/c110 0 2024-04-22T15:19:12.160 INFO:tasks.workunit.client.1.smithi196.stdout:5/858: write d1/d17/f8b [1338010,74609] 0 2024-04-22T15:19:12.161 INFO:tasks.workunit.client.1.smithi196.stdout:3/918: truncate d0/d15/da7/d4a/d44/ff3 2329817 0 2024-04-22T15:19:12.161 INFO:tasks.workunit.client.1.smithi196.stdout:5/859: write d1/d2/d98/fcc [4382526,84524] 0 2024-04-22T15:19:12.162 INFO:tasks.workunit.client.1.smithi196.stdout:1/844: mkdir d1/db/d97/dd7/d10e 0 2024-04-22T15:19:12.165 INFO:tasks.workunit.client.1.smithi196.stdout:4/825: truncate d6/d13/d15/f5e 1252209 0 2024-04-22T15:19:12.167 INFO:tasks.workunit.client.1.smithi196.stdout:8/804: symlink d2/d8/d49/d8b/le9 0 2024-04-22T15:19:12.168 INFO:tasks.workunit.client.0.smithi096.stdout:9/979: getdents d1/df/d95/d9d/db5/dd/dba/dc2/d120/dae 0 2024-04-22T15:19:12.169 INFO:tasks.workunit.client.1.smithi196.stdout:9/876: write db/dd/d1b/fc1 [3936,77901] 0 2024-04-22T15:19:12.170 INFO:tasks.workunit.client.0.smithi096.stdout:9/980: readlink d1/df/d95/d9d/db5/dd/dba/dc2/d120/dae/dd9/l101 0 2024-04-22T15:19:12.171 INFO:tasks.workunit.client.1.smithi196.stdout:3/919: creat d0/d15/d31/d55/f135 x:0 0 0 2024-04-22T15:19:12.174 INFO:tasks.workunit.client.1.smithi196.stdout:2/860: truncate d1/d13/d1f/f20 4273143 0 2024-04-22T15:19:12.176 INFO:tasks.workunit.client.0.smithi096.stdout:9/981: mkdir d1/d25/db2/d98/d14a 0 2024-04-22T15:19:12.181 INFO:tasks.workunit.client.0.smithi096.stdout:9/982: symlink d1/df/d95/d9d/db5/d43/d57/d5b/d73/l14b 0 2024-04-22T15:19:12.184 INFO:tasks.workunit.client.1.smithi196.stdout:9/877: fsync db/dd/de/dd7/def/d7a/fce 0 2024-04-22T15:19:12.184 INFO:tasks.workunit.client.1.smithi196.stdout:7/841: sync 2024-04-22T15:19:12.185 INFO:tasks.workunit.client.1.smithi196.stdout:5/860: mkdir d1/d78/dcf/dff/d101/d111 0 2024-04-22T15:19:12.188 INFO:tasks.workunit.client.0.smithi096.stdout:9/983: rename d1/df/d95/d9d/db5/dd/dba/dc2/d120/dbd/l117 to d1/df/d95/d9d/db5/dd/dba/dc2/de5/d133/l14c 0 2024-04-22T15:19:12.190 INFO:tasks.workunit.client.0.smithi096.stdout:9/984: write d1/d4/d79/f86 [1405853,72000] 0 2024-04-22T15:19:12.190 INFO:tasks.workunit.client.1.smithi196.stdout:6/833: dwrite dd/d20/d5c/d5e/f9b [0,4194304] 0 2024-04-22T15:19:12.192 INFO:tasks.workunit.client.1.smithi196.stdout:8/805: write d2/d5/d1a/fa7 [452372,117038] 0 2024-04-22T15:19:12.192 INFO:tasks.workunit.client.0.smithi096.stdout:9/985: chown d1/df/d95/d9d/db5/d43/d57/d5b/d73/de3/f124 3 1 2024-04-22T15:19:12.193 INFO:tasks.workunit.client.0.smithi096.stdout:9/986: stat d1/d25/d9f/f104 0 2024-04-22T15:19:12.197 INFO:tasks.workunit.client.0.smithi096.stdout:9/987: truncate d1/df/d95/d9d/db5/d43/d57/d5b/d73/de3/f124 809678 0 2024-04-22T15:19:12.207 INFO:tasks.workunit.client.1.smithi196.stdout:3/920: dread d0/d15/da7/f47 [0,4194304] 0 2024-04-22T15:19:12.207 INFO:tasks.workunit.client.1.smithi196.stdout:5/861: dread d1/d17/f21 [0,4194304] 0 2024-04-22T15:19:12.213 INFO:tasks.workunit.client.0.smithi096.stdout:9/988: truncate d1/f6 2852317 0 2024-04-22T15:19:12.235 INFO:tasks.workunit.client.1.smithi196.stdout:6/834: truncate dd/d1f/d38/d40/f49 387593 0 2024-04-22T15:19:12.235 INFO:tasks.workunit.client.0.smithi096.stdout:9/989: dwrite d1/d4/d7b/fdf [0,4194304] 0 2024-04-22T15:19:12.237 INFO:tasks.workunit.client.0.smithi096.stdout:9/990: readlink d1/df/d95/d9d/db5/d43/l4b 0 2024-04-22T15:19:12.240 INFO:tasks.workunit.client.1.smithi196.stdout:0/857: getdents d3/d6/dc/d8c/d8d/da8 0 2024-04-22T15:19:12.242 INFO:tasks.workunit.client.1.smithi196.stdout:9/878: mknod db/dd/c126 0 2024-04-22T15:19:12.244 INFO:tasks.workunit.client.1.smithi196.stdout:9/879: readlink db/dd/d1b/d2c/l8f 0 2024-04-22T15:19:12.244 INFO:tasks.workunit.client.1.smithi196.stdout:1/845: getdents d1/db/d16/d1e/d46 0 2024-04-22T15:19:12.246 INFO:tasks.workunit.client.0.smithi096.stdout:9/991: creat d1/df/d95/d9d/db5/dd/dba/dc2/de5/d13e/f14d x:0 0 0 2024-04-22T15:19:12.249 INFO:tasks.workunit.client.1.smithi196.stdout:3/921: dread d0/d15/d4d/db/f2e [0,4194304] 0 2024-04-22T15:19:12.250 INFO:tasks.workunit.client.1.smithi196.stdout:4/826: dwrite d6/d13/d15/f5e [0,4194304] 0 2024-04-22T15:19:12.252 INFO:tasks.workunit.client.1.smithi196.stdout:0/858: truncate d3/d6/dc/d8c/daf/f7c 1149472 0 2024-04-22T15:19:12.255 INFO:tasks.workunit.client.1.smithi196.stdout:0/859: write d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9/f12b [973142,66831] 0 2024-04-22T15:19:12.256 INFO:tasks.workunit.client.1.smithi196.stdout:9/880: creat db/dd/de/dd7/def/d57/da7/f127 x:0 0 0 2024-04-22T15:19:12.259 INFO:tasks.workunit.client.1.smithi196.stdout:8/806: dread d2/d8/d49/d6c/fc1 [0,4194304] 0 2024-04-22T15:19:12.260 INFO:tasks.workunit.client.1.smithi196.stdout:7/842: rename d14/d4d/da3/dc8 to d14/d26/d5a/d118 0 2024-04-22T15:19:12.261 INFO:tasks.workunit.client.1.smithi196.stdout:5/862: write d1/d17/d1a/fe5 [832270,123560] 0 2024-04-22T15:19:12.266 INFO:tasks.workunit.client.1.smithi196.stdout:2/861: dwrite d1/d13/d1f/f20 [0,4194304] 0 2024-04-22T15:19:12.268 INFO:tasks.workunit.client.1.smithi196.stdout:0/860: creat d3/d9/d36/d114/f130 x:0 0 0 2024-04-22T15:19:12.270 INFO:tasks.workunit.client.1.smithi196.stdout:2/862: truncate d1/d23/d27/d2e/d55/d5b/d93/d97/f106 497180 0 2024-04-22T15:19:12.277 INFO:tasks.workunit.client.1.smithi196.stdout:3/922: rename d0/d15/d31/d55/dcf/d6f/de5/d101/d127 to d0/d15/d31/d55/dcf/d8/d6b/d12a/d136 0 2024-04-22T15:19:12.277 INFO:tasks.workunit.client.1.smithi196.stdout:8/807: dread d2/d5/d1a/db7/dd9/d68/f7c [0,4194304] 0 2024-04-22T15:19:12.279 INFO:tasks.workunit.client.1.smithi196.stdout:3/923: read d0/d15/da7/d4a/d44/f16 [555850,29673] 0 2024-04-22T15:19:12.281 INFO:tasks.workunit.client.1.smithi196.stdout:7/843: read d14/d26/d5a/f8f [954949,41951] 0 2024-04-22T15:19:12.283 INFO:tasks.workunit.client.1.smithi196.stdout:8/808: dwrite d2/d8/d49/f59 [0,4194304] 0 2024-04-22T15:19:12.292 INFO:tasks.workunit.client.1.smithi196.stdout:5/863: dread d1/d17/d7f/fe0 [0,4194304] 0 2024-04-22T15:19:12.292 INFO:tasks.workunit.client.1.smithi196.stdout:2/863: rmdir d1/d23/d27/d2e/d55 39 2024-04-22T15:19:12.305 INFO:tasks.workunit.client.1.smithi196.stdout:8/809: fsync d2/d6a/d7b/f94 0 2024-04-22T15:19:12.306 INFO:tasks.workunit.client.1.smithi196.stdout:8/810: chown d2/d5/d1a/cb0 388734218 1 2024-04-22T15:19:12.306 INFO:tasks.workunit.client.1.smithi196.stdout:8/811: write d2/dc/f82 [4142016,46551] 0 2024-04-22T15:19:12.310 INFO:tasks.workunit.client.1.smithi196.stdout:1/846: getdents d1/db/d16/d1e/d49/d90 0 2024-04-22T15:19:12.311 INFO:tasks.workunit.client.1.smithi196.stdout:1/847: write d1/db/dc/d26/d2e/f34 [3937375,81613] 0 2024-04-22T15:19:12.315 INFO:tasks.workunit.client.1.smithi196.stdout:4/827: getdents d6/d14/d18/d26/d32/d67/de7 0 2024-04-22T15:19:12.319 INFO:tasks.workunit.client.1.smithi196.stdout:9/881: getdents db/dd 0 2024-04-22T15:19:12.327 INFO:tasks.workunit.client.1.smithi196.stdout:1/848: mknod d1/db/d16/d6d/c10f 0 2024-04-22T15:19:12.330 INFO:tasks.workunit.client.1.smithi196.stdout:4/828: rmdir d6/de8 39 2024-04-22T15:19:12.330 INFO:tasks.workunit.client.1.smithi196.stdout:3/924: dwrite d0/d15/da7/d85/f94 [0,4194304] 0 2024-04-22T15:19:12.331 INFO:tasks.workunit.client.1.smithi196.stdout:3/925: stat d0/d15/d31/cca 0 2024-04-22T15:19:12.333 INFO:tasks.workunit.client.1.smithi196.stdout:0/861: link d3/de/d1c/d27/ddc/c110 d3/d9/d36/c131 0 2024-04-22T15:19:12.334 INFO:tasks.workunit.client.1.smithi196.stdout:8/812: dread d2/dc/d45/fda [0,4194304] 0 2024-04-22T15:19:12.336 INFO:tasks.workunit.client.1.smithi196.stdout:3/926: dwrite d0/d15/f92 [0,4194304] 0 2024-04-22T15:19:12.341 INFO:tasks.workunit.client.1.smithi196.stdout:9/882: mkdir db/dd/de/dd7/def/d57/d128 0 2024-04-22T15:19:12.342 INFO:tasks.workunit.client.1.smithi196.stdout:9/883: fsync db/dd/de/dd7/def/d7a/f7d 0 2024-04-22T15:19:12.342 INFO:tasks.workunit.client.1.smithi196.stdout:4/829: rename d6/d90/c72 to d6/d14/d18/d40/c10c 0 2024-04-22T15:19:12.345 INFO:tasks.workunit.client.0.smithi096.stdout:9/992: creat d1/df/d95/d9d/db5/dd/dba/dc2/f14e x:0 0 0 2024-04-22T15:19:12.346 INFO:tasks.workunit.client.1.smithi196.stdout:4/830: dread - d6/d13/d15/d2c/d62/d78/d8d/fbd zero size 2024-04-22T15:19:12.346 INFO:tasks.workunit.client.1.smithi196.stdout:0/862: fdatasync d3/d9/f73 0 2024-04-22T15:19:12.349 INFO:tasks.workunit.client.1.smithi196.stdout:3/927: mkdir d0/d15/d31/d55/d10c/d137 0 2024-04-22T15:19:12.350 INFO:tasks.workunit.client.1.smithi196.stdout:0/863: write d3/d9/d36/d114/f130 [344657,28501] 0 2024-04-22T15:19:12.350 INFO:tasks.workunit.client.1.smithi196.stdout:7/844: link d14/d25/d31/df0/c104 d14/d26/db7/d10b/c119 0 2024-04-22T15:19:12.350 INFO:tasks.workunit.client.1.smithi196.stdout:5/864: write d1/d4/f26 [4753450,70280] 0 2024-04-22T15:19:12.352 INFO:tasks.workunit.client.0.smithi096.stdout:9/993: mknod d1/df/d95/d9d/db5/dd/dba/dc2/d120/c14f 0 2024-04-22T15:19:12.354 INFO:tasks.workunit.client.1.smithi196.stdout:9/884: dwrite db/dd/de/d15/d31/fe8 [0,4194304] 0 2024-04-22T15:19:12.356 INFO:tasks.workunit.client.1.smithi196.stdout:6/835: sync 2024-04-22T15:19:12.359 INFO:tasks.workunit.client.1.smithi196.stdout:4/831: dwrite d6/d13/d15/d2c/d62/f88 [0,4194304] 0 2024-04-22T15:19:12.362 INFO:tasks.workunit.client.1.smithi196.stdout:7/845: mknod d14/c11a 0 2024-04-22T15:19:12.364 INFO:tasks.workunit.client.1.smithi196.stdout:5/865: rename d1/d2/d98/dac/d105/f20 to d1/d17/d1c/f112 0 2024-04-22T15:19:12.364 INFO:tasks.workunit.client.0.smithi096.stdout:9/994: link d1/df/d95/d9d/db5/dd/dba/dc2/d120/dae/c10f d1/d25/d9f/d112/c150 0 2024-04-22T15:19:12.366 INFO:tasks.workunit.client.1.smithi196.stdout:5/866: stat d1/d2/d98/dac/d105/l72 0 2024-04-22T15:19:12.367 INFO:tasks.workunit.client.1.smithi196.stdout:9/885: dwrite db/dd/de/dd7/def/d57/fbd [0,4194304] 0 2024-04-22T15:19:12.367 INFO:tasks.workunit.client.1.smithi196.stdout:3/928: symlink d0/d15/d31/d55/dcf/d6f/d10a/d5e/d115/l138 0 2024-04-22T15:19:12.369 INFO:tasks.workunit.client.0.smithi096.stdout:9/995: symlink d1/df/d95/d9d/db5/dd/dba/dc2/de5/d13e/l151 0 2024-04-22T15:19:12.374 INFO:tasks.workunit.client.1.smithi196.stdout:8/813: write d2/d5/f46 [3301389,45170] 0 2024-04-22T15:19:12.374 INFO:tasks.workunit.client.0.smithi096.stdout:9/996: creat d1/df/d95/d9d/db5/dd/dba/dc2/d120/dbd/f152 x:0 0 0 2024-04-22T15:19:12.374 INFO:tasks.workunit.client.1.smithi196.stdout:7/846: mknod d14/d4d/da3/de9/c11b 0 2024-04-22T15:19:12.376 INFO:tasks.workunit.client.1.smithi196.stdout:0/864: link d3/d6/dc/d8c/d8d/d98/dbd/f126 d3/d6/dc/d8c/d8d/dab/f132 0 2024-04-22T15:19:12.378 INFO:tasks.workunit.client.1.smithi196.stdout:6/836: mkdir dd/d1f/d4e/d53/d55/d6d/dd1/de3/df7/df9 0 2024-04-22T15:19:12.378 INFO:tasks.workunit.client.1.smithi196.stdout:3/929: dread d0/d15/d31/d55/d8b/fc0 [0,4194304] 0 2024-04-22T15:19:12.378 INFO:tasks.workunit.client.1.smithi196.stdout:6/837: chown dd/d2b/d8d 51 1 2024-04-22T15:19:12.379 INFO:tasks.workunit.client.0.smithi096.stdout:9/997: symlink d1/d25/db2/dca/l153 0 2024-04-22T15:19:12.379 INFO:tasks.workunit.client.1.smithi196.stdout:3/930: read d0/d15/d31/d55/d8b/fc0 [1668716,42491] 0 2024-04-22T15:19:12.382 INFO:tasks.workunit.client.1.smithi196.stdout:4/832: rename d6/d90/ca9 to d6/d13/d15/d2c/d62/d78/df6/c10d 0 2024-04-22T15:19:12.384 INFO:tasks.workunit.client.1.smithi196.stdout:7/847: mknod d14/d26/d95/c11c 0 2024-04-22T15:19:12.385 INFO:tasks.workunit.client.0.smithi096.stdout:9/998: creat d1/df/d95/d9d/db5/dd/dba/dc2/d120/d12b/f154 x:0 0 0 2024-04-22T15:19:12.386 INFO:tasks.workunit.client.1.smithi196.stdout:6/838: dread - dd/d1f/d4e/fab zero size 2024-04-22T15:19:12.388 INFO:tasks.workunit.client.1.smithi196.stdout:3/931: truncate d0/f10f 1771346 0 2024-04-22T15:19:12.388 INFO:tasks.workunit.client.1.smithi196.stdout:6/839: stat dd/d2b/d8f/fbb 0 2024-04-22T15:19:12.390 INFO:tasks.workunit.client.1.smithi196.stdout:9/886: rename db/dd/de/dd7/def/d7a/d7b/ff8 to db/dd/de/d98/da9/d101/f129 0 2024-04-22T15:19:12.390 INFO:tasks.workunit.client.0.smithi096.stdout:9/999: dwrite d1/d4/d7b/fdf [0,4194304] 0 2024-04-22T15:19:12.391 INFO:tasks.workunit.client.1.smithi196.stdout:4/833: symlink d6/d13/d15/d2c/d8c/db3/dce/l10e 0 2024-04-22T15:19:12.391 INFO:tasks.workunit.client.1.smithi196.stdout:7/848: dwrite d14/d15/fcc [0,4194304] 0 2024-04-22T15:19:12.393 INFO:tasks.workunit.client.0.smithi096.stdout:6/933: sync 2024-04-22T15:19:12.393 INFO:tasks.workunit.client.0.smithi096.stdout:6/934: chown d3/l72 13 1 2024-04-22T15:19:12.396 INFO:tasks.workunit.client.1.smithi196.stdout:3/932: mknod d0/d15/d31/d55/d8b/df2/c139 0 2024-04-22T15:19:12.398 INFO:tasks.workunit.client.0.smithi096.stdout:6/935: creat d3/d28/d4d/d82/d9e/da3/f13e x:0 0 0 2024-04-22T15:19:12.401 INFO:tasks.workunit.client.1.smithi196.stdout:9/887: creat db/dd/de/dd7/def/d7a/f12a x:0 0 0 2024-04-22T15:19:12.401 INFO:tasks.workunit.client.1.smithi196.stdout:6/840: rename dd/ddf/fef to dd/d1f/d4e/d53/d55/d6d/dd1/dca/ddd/ffa 0 2024-04-22T15:19:12.402 INFO:tasks.workunit.client.1.smithi196.stdout:5/867: dread d1/d17/f2e [0,4194304] 0 2024-04-22T15:19:12.403 INFO:tasks.workunit.client.1.smithi196.stdout:2/864: sync 2024-04-22T15:19:12.403 INFO:tasks.workunit.client.1.smithi196.stdout:7/849: read d14/d26/d28/d29/f68 [2863200,53058] 0 2024-04-22T15:19:12.404 INFO:tasks.workunit.client.1.smithi196.stdout:4/834: symlink d6/d13/d15/d2c/d62/d78/ddc/l10f 0 2024-04-22T15:19:12.406 INFO:tasks.workunit.client.1.smithi196.stdout:8/814: getdents d2/d8/d49/d8b 0 2024-04-22T15:19:12.406 INFO:tasks.workunit.client.1.smithi196.stdout:6/841: write dd/d20/d4d/f65 [1733837,102626] 0 2024-04-22T15:19:12.408 INFO:tasks.workunit.client.1.smithi196.stdout:8/815: chown d2/d5 1080145 1 2024-04-22T15:19:12.411 INFO:tasks.workunit.client.1.smithi196.stdout:7/850: fsync d14/d4a/f71 0 2024-04-22T15:19:12.411 INFO:tasks.workunit.client.1.smithi196.stdout:6/842: dread - dd/d2b/d8d/fb9 zero size 2024-04-22T15:19:12.413 INFO:tasks.workunit.client.1.smithi196.stdout:2/865: mkdir d1/d23/d27/d2e/db5/d102/d135 0 2024-04-22T15:19:12.414 INFO:tasks.workunit.client.1.smithi196.stdout:5/868: getdents d1/d2/dbb/df9 0 2024-04-22T15:19:12.415 INFO:tasks.workunit.client.1.smithi196.stdout:3/933: dwrite d0/d15/d31/d55/d8b/dbf/f103 [0,4194304] 0 2024-04-22T15:19:12.418 INFO:tasks.workunit.client.1.smithi196.stdout:1/849: sync 2024-04-22T15:19:12.419 INFO:tasks.workunit.client.1.smithi196.stdout:7/851: mknod d14/d26/d28/d29/c11d 0 2024-04-22T15:19:12.422 INFO:tasks.workunit.client.1.smithi196.stdout:8/816: getdents d2/d8/d49 0 2024-04-22T15:19:12.425 INFO:tasks.workunit.client.1.smithi196.stdout:4/835: dwrite d6/d14/d18/d40/ff8 [0,4194304] 0 2024-04-22T15:19:12.426 INFO:tasks.workunit.client.1.smithi196.stdout:9/888: dread db/dd/de/d15/d31/fb7 [0,4194304] 0 2024-04-22T15:19:12.426 INFO:tasks.workunit.client.1.smithi196.stdout:2/866: chown d1/d23/d27/d73/l7b 3644 1 2024-04-22T15:19:12.426 INFO:tasks.workunit.client.1.smithi196.stdout:3/934: write d0/d15/da7/d85/f9c [4391009,71999] 0 2024-04-22T15:19:12.427 INFO:tasks.workunit.client.1.smithi196.stdout:7/852: readlink d14/d26/d28/d29/lb4 0 2024-04-22T15:19:12.428 INFO:tasks.workunit.client.1.smithi196.stdout:0/865: dwrite d3/d9/d36/f11d [0,4194304] 0 2024-04-22T15:19:12.430 INFO:tasks.workunit.client.1.smithi196.stdout:1/850: symlink d1/db/l110 0 2024-04-22T15:19:12.431 INFO:tasks.workunit.client.1.smithi196.stdout:4/836: readlink d6/d13/d15/d2c/d8c/db3/lc2 0 2024-04-22T15:19:12.435 INFO:tasks.workunit.client.1.smithi196.stdout:7/853: truncate d14/d15/fef 1047227 0 2024-04-22T15:19:12.435 INFO:tasks.workunit.client.1.smithi196.stdout:9/889: fsync db/dd/d1b/f51 0 2024-04-22T15:19:12.436 INFO:tasks.workunit.client.1.smithi196.stdout:3/935: dread d0/d15/da7/fa8 [0,4194304] 0 2024-04-22T15:19:12.439 INFO:tasks.workunit.client.1.smithi196.stdout:0/866: stat d3/d6/dc/d8c/d8d/d98/dbd/c12f 0 2024-04-22T15:19:12.439 INFO:tasks.workunit.client.1.smithi196.stdout:8/817: getdents d2/d5/d1a/dcc 0 2024-04-22T15:19:12.440 INFO:tasks.workunit.client.0.smithi096.stdout:6/936: dread d3/d3a/d75/d83/df7/f118 [0,4194304] 0 2024-04-22T15:19:12.441 INFO:tasks.workunit.client.1.smithi196.stdout:6/843: dwrite dd/d2b/f2f [0,4194304] 0 2024-04-22T15:19:12.441 INFO:tasks.workunit.client.1.smithi196.stdout:2/867: creat d1/d1d/f136 x:0 0 0 2024-04-22T15:19:12.443 INFO:tasks.workunit.client.1.smithi196.stdout:6/844: readlink dd/d1f/d37/d58/l72 0 2024-04-22T15:19:12.443 INFO:tasks.workunit.client.1.smithi196.stdout:7/854: unlink d14/d26/d5a/dbb/l78 0 2024-04-22T15:19:12.445 INFO:tasks.workunit.client.1.smithi196.stdout:1/851: mkdir d1/db/dc/de/d52/d111 0 2024-04-22T15:19:12.447 INFO:tasks.workunit.client.1.smithi196.stdout:9/890: mknod db/dd/de/d15/d31/d64/c12b 0 2024-04-22T15:19:12.447 INFO:tasks.workunit.client.1.smithi196.stdout:3/936: fsync d0/d15/d31/d55/f74 0 2024-04-22T15:19:12.448 INFO:tasks.workunit.client.1.smithi196.stdout:3/937: fsync d0/d15/d4d/d87/f111 0 2024-04-22T15:19:12.449 INFO:tasks.workunit.client.1.smithi196.stdout:8/818: mkdir d2/d5/d1a/db7/dd9/d68/dea 0 2024-04-22T15:19:12.452 INFO:tasks.workunit.client.1.smithi196.stdout:4/837: creat d6/d14/d18/f110 x:0 0 0 2024-04-22T15:19:12.454 INFO:tasks.workunit.client.1.smithi196.stdout:7/855: creat d14/d4d/da3/f11e x:0 0 0 2024-04-22T15:19:12.455 INFO:tasks.workunit.client.1.smithi196.stdout:6/845: read dd/d1f/d4e/d53/d7f/da5/ff0 [1234872,99080] 0 2024-04-22T15:19:12.456 INFO:tasks.workunit.client.1.smithi196.stdout:3/938: symlink d0/d15/d31/d55/d8b/dbf/l13a 0 2024-04-22T15:19:12.456 INFO:tasks.workunit.client.0.smithi096.stdout:6/937: dwrite d3/d28/f21 [0,4194304] 0 2024-04-22T15:19:12.457 INFO:tasks.workunit.client.1.smithi196.stdout:8/819: dread - d2/d35/fb5 zero size 2024-04-22T15:19:12.457 INFO:tasks.workunit.client.0.smithi096.stdout:6/938: readlink d3/lb 0 2024-04-22T15:19:12.457 INFO:tasks.workunit.client.1.smithi196.stdout:8/820: fdatasync d2/d3f/f53 0 2024-04-22T15:19:12.458 INFO:tasks.workunit.client.0.smithi096.stdout:6/939: write d3/d28/daf/d94/ffd [2972173,97947] 0 2024-04-22T15:19:12.458 INFO:tasks.workunit.client.1.smithi196.stdout:4/838: mkdir d6/d13/d15/d2c/d62/d78/df6/d111 0 2024-04-22T15:19:12.461 INFO:tasks.workunit.client.1.smithi196.stdout:9/891: dread db/dd/d1b/f91 [0,4194304] 0 2024-04-22T15:19:12.463 INFO:tasks.workunit.client.1.smithi196.stdout:1/852: truncate d1/db/d16/d1e/f88 975156 0 2024-04-22T15:19:12.465 INFO:tasks.workunit.client.1.smithi196.stdout:9/892: write db/ff [5215553,126567] 0 2024-04-22T15:19:12.466 INFO:tasks.workunit.client.1.smithi196.stdout:0/867: write d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/fbf [1232339,101688] 0 2024-04-22T15:19:12.470 INFO:tasks.workunit.client.1.smithi196.stdout:8/821: dwrite d2/d5/d1a/db7/dd9/d68/fb3 [0,4194304] 0 2024-04-22T15:19:12.470 INFO:tasks.workunit.client.0.smithi096.stdout:6/940: getdents d3/d28/d54/d68/d6e 0 2024-04-22T15:19:12.470 INFO:tasks.workunit.client.1.smithi196.stdout:9/893: chown db/dd/de/d98/da9/d101/f119 205 1 2024-04-22T15:19:12.471 INFO:tasks.workunit.client.1.smithi196.stdout:2/868: dread d1/d13/d25/ff9 [0,4194304] 0 2024-04-22T15:19:12.471 INFO:tasks.workunit.client.0.smithi096.stdout:6/941: chown d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/ddd/de9/f8b 1379 1 2024-04-22T15:19:12.474 INFO:tasks.workunit.client.1.smithi196.stdout:3/939: rename d0/d15/da7/d85 to d0/d15/d31/dd0/de2/d13b 0 2024-04-22T15:19:12.475 INFO:tasks.workunit.client.1.smithi196.stdout:7/856: rmdir d14/d26/d5a/d118/df1 0 2024-04-22T15:19:12.476 INFO:tasks.workunit.client.1.smithi196.stdout:7/857: chown d14/d25/ff2 350967734 1 2024-04-22T15:19:12.478 INFO:tasks.workunit.client.1.smithi196.stdout:6/846: write dd/d1f/d38/d40/f43 [4882434,80502] 0 2024-04-22T15:19:12.480 INFO:tasks.workunit.client.1.smithi196.stdout:1/853: mknod d1/db/d16/d1e/c112 0 2024-04-22T15:19:12.481 INFO:tasks.workunit.client.0.smithi096.stdout:6/942: creat d3/d28/d4d/d82/d9e/d137/d11/d2c/d12f/f13f x:0 0 0 2024-04-22T15:19:12.482 INFO:tasks.workunit.client.1.smithi196.stdout:9/894: mknod db/dd/de/dd7/def/d7a/d116/c12c 0 2024-04-22T15:19:12.484 INFO:tasks.workunit.client.1.smithi196.stdout:9/895: truncate db/dd/de/dd7/def/d7a/fbb 992048 0 2024-04-22T15:19:12.486 INFO:tasks.workunit.client.0.smithi096.stdout:6/943: rename d3/d28/d4d/d82/c9b to d3/d3a/c140 0 2024-04-22T15:19:12.487 INFO:tasks.workunit.client.1.smithi196.stdout:6/847: truncate dd/d70/fd2 864027 0 2024-04-22T15:19:12.489 INFO:tasks.workunit.client.1.smithi196.stdout:1/854: rename d1/db/dc/d93/df9 to d1/db/d97/dd7/d10e/d113 0 2024-04-22T15:19:12.491 INFO:tasks.workunit.client.0.smithi096.stdout:6/944: write d3/d28/d4d/d82/d9e/d137/d11/d2c/f39 [4561561,85425] 0 2024-04-22T15:19:12.493 INFO:tasks.workunit.client.1.smithi196.stdout:4/839: getdents d6/d13/d15/d2c/d8c/db3/dce 0 2024-04-22T15:19:12.494 INFO:tasks.workunit.client.1.smithi196.stdout:9/896: creat db/dd/f12d x:0 0 0 2024-04-22T15:19:12.494 INFO:tasks.workunit.client.0.smithi096.stdout:6/945: truncate d3/d28/d4d/d82/d9e/d137/fba 614511 0 2024-04-22T15:19:12.495 INFO:tasks.workunit.client.1.smithi196.stdout:0/868: getdents d3/d6/dae 0 2024-04-22T15:19:12.495 INFO:tasks.workunit.client.1.smithi196.stdout:3/940: dread d0/d15/da7/d18/f1b [8388608,4194304] 0 2024-04-22T15:19:12.495 INFO:tasks.workunit.client.0.smithi096.stdout:6/946: chown d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/ddd/de9/l10b 48 1 2024-04-22T15:19:12.496 INFO:tasks.workunit.client.1.smithi196.stdout:1/855: mkdir d1/db/d97/dd7/d114 0 2024-04-22T15:19:12.498 INFO:tasks.workunit.client.1.smithi196.stdout:7/858: dread d14/d25/d31/f8e [0,4194304] 0 2024-04-22T15:19:12.500 INFO:tasks.workunit.client.1.smithi196.stdout:3/941: creat d0/d15/da7/d4a/d44/d96/f13c x:0 0 0 2024-04-22T15:19:12.501 INFO:tasks.workunit.client.1.smithi196.stdout:0/869: mknod d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/dc9/c133 0 2024-04-22T15:19:12.502 INFO:tasks.workunit.client.0.smithi096.stdout:6/947: dwrite d3/d3a/d75/f128 [4194304,4194304] 0 2024-04-22T15:19:12.502 INFO:tasks.workunit.client.1.smithi196.stdout:8/822: dread d2/d5/d4f/f6e [4194304,4194304] 0 2024-04-22T15:19:12.503 INFO:tasks.workunit.client.1.smithi196.stdout:4/840: creat d6/de8/f112 x:0 0 0 2024-04-22T15:19:12.505 INFO:tasks.workunit.client.1.smithi196.stdout:2/869: getdents d1/d23/d27/d2e/d55/d5b 0 2024-04-22T15:19:12.511 INFO:tasks.workunit.client.1.smithi196.stdout:6/848: truncate dd/f18 2669159 0 2024-04-22T15:19:12.511 INFO:tasks.workunit.client.1.smithi196.stdout:8/823: dwrite d2/dc/d45/f5b [0,4194304] 0 2024-04-22T15:19:12.512 INFO:tasks.workunit.client.1.smithi196.stdout:5/869: sync 2024-04-22T15:19:12.513 INFO:tasks.workunit.client.0.smithi096.stdout:6/948: creat d3/d28/d4d/d82/d9e/d137/d11/d2c/f141 x:0 0 0 2024-04-22T15:19:12.514 INFO:tasks.workunit.client.0.smithi096.stdout:6/949: chown d3/d28/d4d/d82/d9e/d137/d11/d2c/d3b/f63 6529 1 2024-04-22T15:19:12.515 INFO:tasks.workunit.client.1.smithi196.stdout:5/870: write d1/df1/f54 [2837486,117731] 0 2024-04-22T15:19:12.519 INFO:tasks.workunit.client.1.smithi196.stdout:5/871: dread - d1/d14/d3e/fd1 zero size 2024-04-22T15:19:12.520 INFO:tasks.workunit.client.1.smithi196.stdout:6/849: creat dd/d1f/d37/d3c/ffb x:0 0 0 2024-04-22T15:19:12.521 INFO:tasks.workunit.client.1.smithi196.stdout:8/824: truncate d2/d8/d49/d6c/f86 337137 0 2024-04-22T15:19:12.524 INFO:tasks.workunit.client.1.smithi196.stdout:3/942: rename d0/d15/d31/d55/dcf/d6f/d10a/f8a to d0/d15/d31/d55/dcf/d6f/de5/d101/f13d 0 2024-04-22T15:19:12.525 INFO:tasks.workunit.client.1.smithi196.stdout:1/856: dwrite d1/db/dc/dad/f9b [4194304,4194304] 0 2024-04-22T15:19:12.527 INFO:tasks.workunit.client.1.smithi196.stdout:4/841: creat d6/d13/d15/d2c/d35/f113 x:0 0 0 2024-04-22T15:19:12.528 INFO:tasks.workunit.client.1.smithi196.stdout:0/870: getdents d3/de/d14/d86 0 2024-04-22T15:19:12.528 INFO:tasks.workunit.client.1.smithi196.stdout:3/943: write d0/d15/da7/d4a/d44/f6c [4217330,74317] 0 2024-04-22T15:19:12.529 INFO:tasks.workunit.client.1.smithi196.stdout:9/897: symlink db/dd/de/dd7/def/d57/l12e 0 2024-04-22T15:19:12.532 INFO:tasks.workunit.client.1.smithi196.stdout:8/825: creat d2/d6a/d7b/feb x:0 0 0 2024-04-22T15:19:12.532 INFO:tasks.workunit.client.1.smithi196.stdout:6/850: mkdir dd/d1f/d4e/d53/d55/d6d/dd1/dfc 0 2024-04-22T15:19:12.535 INFO:tasks.workunit.client.1.smithi196.stdout:1/857: fdatasync d1/db/d16/d1e/d49/d90/fb4 0 2024-04-22T15:19:12.535 INFO:tasks.workunit.client.1.smithi196.stdout:5/872: write d1/d17/d1c/f88 [86982,72422] 0 2024-04-22T15:19:12.536 INFO:tasks.workunit.client.1.smithi196.stdout:3/944: rmdir d0/d15/da7/d4a 39 2024-04-22T15:19:12.537 INFO:tasks.workunit.client.1.smithi196.stdout:7/859: dread d14/d26/d2a/d72/f99 [0,4194304] 0 2024-04-22T15:19:12.539 INFO:tasks.workunit.client.1.smithi196.stdout:6/851: dread - dd/d1f/d4e/d53/d55/d6d/dd1/de3/de8/ff2 zero size 2024-04-22T15:19:12.540 INFO:tasks.workunit.client.1.smithi196.stdout:2/870: rename d1/d1d/c4d to d1/d13/d25/d9f/c137 0 2024-04-22T15:19:12.541 INFO:tasks.workunit.client.1.smithi196.stdout:1/858: read d1/db/d16/d8e/fbd [781552,13371] 0 2024-04-22T15:19:12.543 INFO:tasks.workunit.client.1.smithi196.stdout:9/898: dwrite db/dd/de/dd7/def/f104 [0,4194304] 0 2024-04-22T15:19:12.545 INFO:tasks.workunit.client.1.smithi196.stdout:8/826: dwrite d2/f4a [0,4194304] 0 2024-04-22T15:19:12.545 INFO:tasks.workunit.client.1.smithi196.stdout:6/852: dread - dd/d1f/d4e/d53/d55/d6d/dde/fe7 zero size 2024-04-22T15:19:12.546 INFO:tasks.workunit.client.1.smithi196.stdout:5/873: creat d1/d17/d1a/d2f/d65/f113 x:0 0 0 2024-04-22T15:19:12.549 INFO:tasks.workunit.client.1.smithi196.stdout:0/871: link d3/d9/d36/d9f/dd1/d65/c85 d3/de/d14/c134 0 2024-04-22T15:19:12.549 INFO:tasks.workunit.client.1.smithi196.stdout:7/860: symlink d14/d15/d91/l11f 0 2024-04-22T15:19:12.555 INFO:tasks.workunit.client.1.smithi196.stdout:2/871: rename d1/d13/d1f/ddf to d1/d13/d25/d2a/ddb/dff/d138 0 2024-04-22T15:19:12.555 INFO:tasks.workunit.client.1.smithi196.stdout:8/827: creat d2/dc/d45/d98/fec x:0 0 0 2024-04-22T15:19:12.557 INFO:tasks.workunit.client.1.smithi196.stdout:1/859: creat d1/db/d16/d1e/df7/f115 x:0 0 0 2024-04-22T15:19:12.557 INFO:tasks.workunit.client.1.smithi196.stdout:5/874: mkdir d1/d17/d1a/d2f/d114 0 2024-04-22T15:19:12.558 INFO:tasks.workunit.client.1.smithi196.stdout:3/945: write d0/d15/da7/d4a/d44/f20 [58967,31280] 0 2024-04-22T15:19:12.558 INFO:tasks.workunit.client.1.smithi196.stdout:4/842: dwrite d6/d14/d18/d40/d74/f7a [0,4194304] 0 2024-04-22T15:19:12.560 INFO:tasks.workunit.client.1.smithi196.stdout:6/853: mkdir dd/d1f/d4e/d53/d55/dfd 0 2024-04-22T15:19:12.566 INFO:tasks.workunit.client.1.smithi196.stdout:6/854: write dd/d20/d5c/d5e/f9b [2282023,91036] 0 2024-04-22T15:19:12.566 INFO:tasks.workunit.client.1.smithi196.stdout:2/872: read d1/d1d/f7a [2845400,108737] 0 2024-04-22T15:19:12.588 INFO:tasks.workunit.client.1.smithi196.stdout:9/899: dwrite db/dd/d1b/d2c/f58 [0,4194304] 0 2024-04-22T15:19:12.588 INFO:tasks.workunit.client.1.smithi196.stdout:8/828: creat d2/d5/d1a/db7/dd9/d68/dbc/fed x:0 0 0 2024-04-22T15:19:12.591 INFO:tasks.workunit.client.1.smithi196.stdout:0/872: truncate d3/db4/fe9 146507 0 2024-04-22T15:19:12.592 INFO:tasks.workunit.client.1.smithi196.stdout:2/873: truncate d1/d1d/f136 238223 0 2024-04-22T15:19:12.592 INFO:tasks.workunit.client.1.smithi196.stdout:4/843: symlink d6/d10/d105/l114 0 2024-04-22T15:19:12.593 INFO:tasks.workunit.client.1.smithi196.stdout:5/875: mkdir d1/d2/d98/d115 0 2024-04-22T15:19:12.595 INFO:tasks.workunit.client.1.smithi196.stdout:8/829: creat d2/d5/d4f/fee x:0 0 0 2024-04-22T15:19:12.596 INFO:tasks.workunit.client.1.smithi196.stdout:9/900: mkdir db/dd/de/d98/d12f 0 2024-04-22T15:19:12.597 INFO:tasks.workunit.client.1.smithi196.stdout:1/860: dwrite d1/db/d16/d44/fec [0,4194304] 0 2024-04-22T15:19:12.605 INFO:tasks.workunit.client.1.smithi196.stdout:7/861: dwrite d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:19:12.605 INFO:tasks.workunit.client.1.smithi196.stdout:2/874: unlink d1/d13/d1f/c90 0 2024-04-22T15:19:12.606 INFO:tasks.workunit.client.1.smithi196.stdout:6/855: dwrite dd/d1f/d4e/d53/d55/d6d/dde/fe7 [0,4194304] 0 2024-04-22T15:19:12.609 INFO:tasks.workunit.client.1.smithi196.stdout:5/876: truncate d1/d17/f5d 924348 0 2024-04-22T15:19:12.612 INFO:tasks.workunit.client.1.smithi196.stdout:7/862: symlink d14/d26/d5a/d5f/dc2/l120 0 2024-04-22T15:19:12.615 INFO:tasks.workunit.client.1.smithi196.stdout:9/901: mknod db/dd/de/d75/d124/c130 0 2024-04-22T15:19:12.618 INFO:tasks.workunit.client.1.smithi196.stdout:6/856: symlink dd/d1f/d4e/d53/d7f/da5/lfe 0 2024-04-22T15:19:12.620 INFO:tasks.workunit.client.1.smithi196.stdout:0/873: dwrite d3/d9/fb1 [0,4194304] 0 2024-04-22T15:19:12.620 INFO:tasks.workunit.client.1.smithi196.stdout:1/861: symlink d1/db/dc/dad/d36/def/l116 0 2024-04-22T15:19:12.621 INFO:tasks.workunit.client.1.smithi196.stdout:1/862: fdatasync d1/db/dc/de/d43/f48 0 2024-04-22T15:19:12.624 INFO:tasks.workunit.client.1.smithi196.stdout:1/863: write d1/db/dc/de/d43/d5e/fbf [1606264,48996] 0 2024-04-22T15:19:12.628 INFO:tasks.workunit.client.1.smithi196.stdout:1/864: creat d1/db/dc/dad/f117 x:0 0 0 2024-04-22T15:19:12.631 INFO:tasks.workunit.client.1.smithi196.stdout:5/877: rename d1/df1/d5e/fca to d1/d17/d1a/f116 0 2024-04-22T15:19:12.632 INFO:tasks.workunit.client.1.smithi196.stdout:0/874: truncate d3/f47 1805366 0 2024-04-22T15:19:12.633 INFO:tasks.workunit.client.1.smithi196.stdout:8/830: dwrite d2/d8/d49/d6c/fc6 [0,4194304] 0 2024-04-22T15:19:12.636 INFO:tasks.workunit.client.1.smithi196.stdout:7/863: rename d14/d4d/da3/l10f to d14/d26/d2a/d72/db1/l121 0 2024-04-22T15:19:12.638 INFO:tasks.workunit.client.1.smithi196.stdout:5/878: symlink d1/d4/d92/l117 0 2024-04-22T15:19:12.640 INFO:tasks.workunit.client.1.smithi196.stdout:5/879: stat d1/df1/d7d/ff2 0 2024-04-22T15:19:12.642 INFO:tasks.workunit.client.1.smithi196.stdout:1/865: dwrite d1/db/d16/d8e/ddd/fee [0,4194304] 0 2024-04-22T15:19:12.642 INFO:tasks.workunit.client.1.smithi196.stdout:6/857: rename dd/f17 to dd/d1f/d4e/d53/d7f/fff 0 2024-04-22T15:19:12.647 INFO:tasks.workunit.client.1.smithi196.stdout:0/875: symlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/db7/l135 0 2024-04-22T15:19:12.647 INFO:tasks.workunit.client.1.smithi196.stdout:1/866: truncate d1/d54/d70/f10a 586942 0 2024-04-22T15:19:12.648 INFO:tasks.workunit.client.1.smithi196.stdout:7/864: dread d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:19:12.648 INFO:tasks.workunit.client.1.smithi196.stdout:4/844: dread d6/d13/d15/d2c/d62/d78/d8b/fd2 [0,4194304] 0 2024-04-22T15:19:12.653 INFO:tasks.workunit.client.1.smithi196.stdout:3/946: sync 2024-04-22T15:19:12.654 INFO:tasks.workunit.client.1.smithi196.stdout:9/902: write fa [3194308,98402] 0 2024-04-22T15:19:12.655 INFO:tasks.workunit.client.1.smithi196.stdout:1/867: mkdir d1/db/d16/d6d/d118 0 2024-04-22T15:19:12.656 INFO:tasks.workunit.client.1.smithi196.stdout:3/947: write d0/d15/d31/d55/d6e/fa1 [2304497,73170] 0 2024-04-22T15:19:12.659 INFO:tasks.workunit.client.1.smithi196.stdout:3/948: dread - d0/d15/da7/d4a/d44/d96/f13c zero size 2024-04-22T15:19:12.660 INFO:tasks.workunit.client.1.smithi196.stdout:0/876: fdatasync d3/d6/dc/d8c/daf/f7c 0 2024-04-22T15:19:12.664 INFO:tasks.workunit.client.1.smithi196.stdout:7/865: dwrite d14/d25/d31/f8e [0,4194304] 0 2024-04-22T15:19:12.664 INFO:tasks.workunit.client.1.smithi196.stdout:2/875: dread d1/d76/f7d [0,4194304] 0 2024-04-22T15:19:12.665 INFO:tasks.workunit.client.1.smithi196.stdout:6/858: link fc dd/d1f/d38/d40/dd3/f100 0 2024-04-22T15:19:12.666 INFO:tasks.workunit.client.1.smithi196.stdout:0/877: read d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/fa3 [4560279,26763] 0 2024-04-22T15:19:12.670 INFO:tasks.workunit.client.1.smithi196.stdout:4/845: creat d6/d14/d18/d26/d32/f115 x:0 0 0 2024-04-22T15:19:12.670 INFO:tasks.workunit.client.1.smithi196.stdout:2/876: read d1/d13/d25/d2a/ddb/dff/d138/f81 [502330,96299] 0 2024-04-22T15:19:12.672 INFO:tasks.workunit.client.1.smithi196.stdout:3/949: creat d0/d15/da7/d4a/d44/d96/dd8/d123/f13e x:0 0 0 2024-04-22T15:19:12.675 INFO:tasks.workunit.client.1.smithi196.stdout:9/903: dwrite db/d11/f4d [0,4194304] 0 2024-04-22T15:19:12.676 INFO:tasks.workunit.client.1.smithi196.stdout:5/880: write d1/d4/d7b/fe1 [143409,45346] 0 2024-04-22T15:19:12.679 INFO:tasks.workunit.client.1.smithi196.stdout:0/878: unlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/db7/cd6 0 2024-04-22T15:19:12.680 INFO:tasks.workunit.client.1.smithi196.stdout:5/881: dread - d1/d17/f109 zero size 2024-04-22T15:19:12.680 INFO:tasks.workunit.client.1.smithi196.stdout:8/831: dwrite d2/dc/fe [0,4194304] 0 2024-04-22T15:19:12.682 INFO:tasks.workunit.client.1.smithi196.stdout:7/866: creat d14/f122 x:0 0 0 2024-04-22T15:19:12.682 INFO:tasks.workunit.client.1.smithi196.stdout:9/904: dread db/dd/d1b/fc1 [0,4194304] 0 2024-04-22T15:19:12.683 INFO:tasks.workunit.client.1.smithi196.stdout:6/859: rename dd/d2b/d8f/fed to dd/d1f/d4e/d53/d55/d6d/dd1/de3/df7/df9/f101 0 2024-04-22T15:19:12.684 INFO:tasks.workunit.client.1.smithi196.stdout:5/882: fdatasync d1/df1/d7d/ff2 0 2024-04-22T15:19:12.684 INFO:tasks.workunit.client.1.smithi196.stdout:3/950: mknod d0/d15/d31/d55/d6e/dfe/c13f 0 2024-04-22T15:19:12.685 INFO:tasks.workunit.client.1.smithi196.stdout:0/879: mknod d3/db4/dc7/c136 0 2024-04-22T15:19:12.686 INFO:tasks.workunit.client.1.smithi196.stdout:7/867: write d14/d4d/da3/f11e [317532,104970] 0 2024-04-22T15:19:12.686 INFO:tasks.workunit.client.1.smithi196.stdout:8/832: creat d2/d5/d2e/fef x:0 0 0 2024-04-22T15:19:12.687 INFO:tasks.workunit.client.1.smithi196.stdout:4/846: creat d6/d13/d15/d2c/f116 x:0 0 0 2024-04-22T15:19:12.690 INFO:tasks.workunit.client.1.smithi196.stdout:9/905: chown db/dd/l5e 578596637 1 2024-04-22T15:19:12.692 INFO:tasks.workunit.client.1.smithi196.stdout:1/868: dread d1/db/dc/de/f14 [0,4194304] 0 2024-04-22T15:19:12.692 INFO:tasks.workunit.client.1.smithi196.stdout:5/883: creat d1/d17/d1c/f118 x:0 0 0 2024-04-22T15:19:12.693 INFO:tasks.workunit.client.1.smithi196.stdout:9/906: dread - db/dd/de/dd7/def/d7a/f12a zero size 2024-04-22T15:19:12.693 INFO:tasks.workunit.client.1.smithi196.stdout:8/833: mknod d2/d3f/d4b/cf0 0 2024-04-22T15:19:12.698 INFO:tasks.workunit.client.1.smithi196.stdout:4/847: rename d6/d13/d15/fa6 to d6/d10/f117 0 2024-04-22T15:19:12.698 INFO:tasks.workunit.client.1.smithi196.stdout:5/884: dread - d1/d14/d6a/fe2 zero size 2024-04-22T15:19:12.698 INFO:tasks.workunit.client.1.smithi196.stdout:4/848: readlink d6/d14/l102 0 2024-04-22T15:19:12.698 INFO:tasks.workunit.client.1.smithi196.stdout:8/834: chown d2/d5/d1a/cb0 450 1 2024-04-22T15:19:12.699 INFO:tasks.workunit.client.1.smithi196.stdout:8/835: stat d2/dc/d45/fda 0 2024-04-22T15:19:12.699 INFO:tasks.workunit.client.1.smithi196.stdout:5/885: rmdir d1/d4/d7b 39 2024-04-22T15:19:12.699 INFO:tasks.workunit.client.1.smithi196.stdout:1/869: write d1/db/d16/d1e/d49/d7f/fb7 [1736300,69814] 0 2024-04-22T15:19:12.700 INFO:tasks.workunit.client.1.smithi196.stdout:5/886: chown d1/d17/d7f/deb 885850 1 2024-04-22T15:19:12.700 INFO:tasks.workunit.client.1.smithi196.stdout:9/907: creat db/dd/de/dd7/def/d7a/d7b/d81/f131 x:0 0 0 2024-04-22T15:19:12.703 INFO:tasks.workunit.client.1.smithi196.stdout:8/836: unlink d2/d8/d49/c5a 0 2024-04-22T15:19:12.704 INFO:tasks.workunit.client.1.smithi196.stdout:5/887: fsync d1/d4/d7b/fe1 0 2024-04-22T15:19:12.706 INFO:tasks.workunit.client.1.smithi196.stdout:2/877: dread d1/d13/d1f/f9d [0,4194304] 0 2024-04-22T15:19:12.710 INFO:tasks.workunit.client.1.smithi196.stdout:2/878: symlink d1/d23/d27/d89/l139 0 2024-04-22T15:19:12.710 INFO:tasks.workunit.client.1.smithi196.stdout:0/880: write d3/de/d1c/d27/f93 [4901331,4721] 0 2024-04-22T15:19:12.712 INFO:tasks.workunit.client.1.smithi196.stdout:7/868: truncate d14/d26/d28/f89 1732009 0 2024-04-22T15:19:12.716 INFO:tasks.workunit.client.1.smithi196.stdout:1/870: dwrite d1/db/d16/d6d/de6/f104 [0,4194304] 0 2024-04-22T15:19:12.716 INFO:tasks.workunit.client.1.smithi196.stdout:4/849: write d6/d13/d15/d2c/d35/fb0 [40912,84008] 0 2024-04-22T15:19:12.720 INFO:tasks.workunit.client.1.smithi196.stdout:0/881: fdatasync d3/d9/d25/f63 0 2024-04-22T15:19:12.721 INFO:tasks.workunit.client.1.smithi196.stdout:7/869: creat d14/d26/d5a/d5f/f123 x:0 0 0 2024-04-22T15:19:12.722 INFO:tasks.workunit.client.1.smithi196.stdout:9/908: dwrite db/dd/de/d98/da9/d101/f129 [0,4194304] 0 2024-04-22T15:19:12.726 INFO:tasks.workunit.client.1.smithi196.stdout:6/860: dwrite dd/d1f/d4e/d53/fa4 [0,4194304] 0 2024-04-22T15:19:12.729 INFO:tasks.workunit.client.1.smithi196.stdout:2/879: dread d1/d13/d25/d9f/faf [0,4194304] 0 2024-04-22T15:19:12.729 INFO:tasks.workunit.client.1.smithi196.stdout:8/837: fsync d2/d8/d49/d8b/f99 0 2024-04-22T15:19:12.730 INFO:tasks.workunit.client.1.smithi196.stdout:4/850: truncate d6/d10/f50 104293 0 2024-04-22T15:19:12.730 INFO:tasks.workunit.client.1.smithi196.stdout:7/870: write d14/d26/d5a/dbb/f8d [3063585,19094] 0 2024-04-22T15:19:12.732 INFO:tasks.workunit.client.1.smithi196.stdout:9/909: symlink db/dd/de/d98/da9/l132 0 2024-04-22T15:19:12.733 INFO:tasks.workunit.client.1.smithi196.stdout:5/888: dread d1/df1/fdd [0,4194304] 0 2024-04-22T15:19:12.733 INFO:tasks.workunit.client.1.smithi196.stdout:0/882: dread d3/db4/fe9 [0,4194304] 0 2024-04-22T15:19:12.734 INFO:tasks.workunit.client.1.smithi196.stdout:9/910: fdatasync db/dd/de/ff2 0 2024-04-22T15:19:12.736 INFO:tasks.workunit.client.1.smithi196.stdout:1/871: rename d1/db/dc/de/d43/ff5 to d1/db/dc/de/d5a/d5b/d85/f119 0 2024-04-22T15:19:12.738 INFO:tasks.workunit.client.1.smithi196.stdout:6/861: unlink dd/d1f/d37/d58/l72 0 2024-04-22T15:19:12.740 INFO:tasks.workunit.client.1.smithi196.stdout:8/838: creat d2/d35/d96/ff1 x:0 0 0 2024-04-22T15:19:12.741 INFO:tasks.workunit.client.1.smithi196.stdout:4/851: dwrite d6/d13/d15/d2c/d35/f113 [0,4194304] 0 2024-04-22T15:19:12.745 INFO:tasks.workunit.client.1.smithi196.stdout:5/889: truncate d1/d4/f9d 2128098 0 2024-04-22T15:19:12.748 INFO:tasks.workunit.client.1.smithi196.stdout:6/862: rmdir dd/d20/d5c/d6a 39 2024-04-22T15:19:12.749 INFO:tasks.workunit.client.1.smithi196.stdout:1/872: mkdir d1/db/d16/d8e/ddd/df6/d11a 0 2024-04-22T15:19:12.751 INFO:tasks.workunit.client.1.smithi196.stdout:7/871: dwrite d14/d25/d31/f55 [0,4194304] 0 2024-04-22T15:19:12.751 INFO:tasks.workunit.client.1.smithi196.stdout:1/873: chown d1/d54/de2/df1/df4 2562 1 2024-04-22T15:19:12.753 INFO:tasks.workunit.client.0.smithi096.stdout:6/950: sync 2024-04-22T15:19:12.754 INFO:tasks.workunit.client.1.smithi196.stdout:8/839: mkdir d2/d8/df2 0 2024-04-22T15:19:12.754 INFO:tasks.workunit.client.1.smithi196.stdout:9/911: rmdir db/dd/de/d98 39 2024-04-22T15:19:12.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:12 smithi196 ceph-mon[34717]: mgrmap e26: smithi196.gdaegc(active, since 79s), standbys: smithi096.hhkvum 2024-04-22T15:19:12.763 INFO:tasks.workunit.client.1.smithi196.stdout:7/872: creat d14/d26/d2a/d2b/dc7/f124 x:0 0 0 2024-04-22T15:19:12.764 INFO:tasks.workunit.client.1.smithi196.stdout:1/874: write d1/db/dc/de/d5a/d5b/d85/f119 [6028664,71932] 0 2024-04-22T15:19:12.764 INFO:tasks.workunit.client.0.smithi096.stdout:6/951: creat d3/d28/d4d/d82/d9e/d137/d11/f142 x:0 0 0 2024-04-22T15:19:12.765 INFO:tasks.workunit.client.0.smithi096.stdout:6/952: fsync d3/d28/d4d/f121 0 2024-04-22T15:19:12.766 INFO:tasks.workunit.client.1.smithi196.stdout:9/912: dread - db/dd/de/dd7/def/fd8 zero size 2024-04-22T15:19:12.767 INFO:tasks.workunit.client.1.smithi196.stdout:2/880: link d1/d23/d27/d73/l86 d1/d13/l13a 0 2024-04-22T15:19:12.769 INFO:tasks.workunit.client.1.smithi196.stdout:0/883: dwrite d3/d9/d36/d9f/dd1/d65/d7f/ddb/ffa [0,4194304] 0 2024-04-22T15:19:12.770 INFO:tasks.workunit.client.1.smithi196.stdout:3/951: sync 2024-04-22T15:19:12.771 INFO:tasks.workunit.client.1.smithi196.stdout:7/873: mkdir d14/d4d/d125 0 2024-04-22T15:19:12.772 INFO:tasks.workunit.client.0.smithi096.stdout:6/953: dwrite d3/d28/d4d/d82/d9e/d137/d11/f1e [0,4194304] 0 2024-04-22T15:19:12.773 INFO:tasks.workunit.client.1.smithi196.stdout:8/840: mknod d2/d8/d49/cf3 0 2024-04-22T15:19:12.776 INFO:tasks.workunit.client.0.smithi096.stdout:6/954: symlink d3/d28/daf/d94/l143 0 2024-04-22T15:19:12.776 INFO:tasks.workunit.client.1.smithi196.stdout:1/875: unlink d1/db/dc/de/d5a/d5b/lc5 0 2024-04-22T15:19:12.776 INFO:tasks.workunit.client.1.smithi196.stdout:6/863: write dd/d20/f7d [824690,91983] 0 2024-04-22T15:19:12.779 INFO:tasks.workunit.client.1.smithi196.stdout:5/890: dwrite d1/df1/d6e/fb9 [0,4194304] 0 2024-04-22T15:19:12.779 INFO:tasks.workunit.client.1.smithi196.stdout:2/881: symlink d1/d23/d126/l13b 0 2024-04-22T15:19:12.783 INFO:tasks.workunit.client.0.smithi096.stdout:6/955: dwrite d3/d3a/d75/d83/f98 [0,4194304] 0 2024-04-22T15:19:12.785 INFO:tasks.workunit.client.1.smithi196.stdout:6/864: mkdir dd/d2b/d8d/d102 0 2024-04-22T15:19:12.786 INFO:tasks.workunit.client.1.smithi196.stdout:9/913: dwrite db/dd/de/dd7/def/d57/f23 [0,4194304] 0 2024-04-22T15:19:12.786 INFO:tasks.workunit.client.1.smithi196.stdout:4/852: dread d6/d13/f68 [0,4194304] 0 2024-04-22T15:19:12.786 INFO:tasks.workunit.client.1.smithi196.stdout:8/841: creat d2/d5/d1a/db7/ff4 x:0 0 0 2024-04-22T15:19:12.790 INFO:tasks.workunit.client.1.smithi196.stdout:1/876: unlink d1/db/d16/d1e/d49/f42 0 2024-04-22T15:19:12.790 INFO:tasks.workunit.client.1.smithi196.stdout:0/884: getdents d3/d9/d36/d12e 0 2024-04-22T15:19:12.790 INFO:tasks.workunit.client.1.smithi196.stdout:9/914: stat db/dd/de/dd7/f123 0 2024-04-22T15:19:12.792 INFO:tasks.workunit.client.1.smithi196.stdout:7/874: rename d14/d26/d2a/d72/db1/led to d14/d4d/da3/l126 0 2024-04-22T15:19:12.793 INFO:tasks.workunit.client.1.smithi196.stdout:3/952: creat d0/d15/d4d/db/f140 x:0 0 0 2024-04-22T15:19:12.795 INFO:tasks.workunit.client.1.smithi196.stdout:9/915: chown db/dd/de/dd7/def/f2b 470 1 2024-04-22T15:19:12.798 INFO:tasks.workunit.client.1.smithi196.stdout:8/842: mkdir d2/d5/d1a/db7/dd9/d68/dbc/dae/df5 0 2024-04-22T15:19:12.798 INFO:tasks.workunit.client.1.smithi196.stdout:6/865: dwrite dd/d2b/f2f [8388608,4194304] 0 2024-04-22T15:19:12.800 INFO:tasks.workunit.client.1.smithi196.stdout:1/877: write d1/db/d16/f5c [2613056,118557] 0 2024-04-22T15:19:12.801 INFO:tasks.workunit.client.1.smithi196.stdout:0/885: dread - d3/d6/dc/d8c/d8d/d98/dbd/fe6 zero size 2024-04-22T15:19:12.804 INFO:tasks.workunit.client.1.smithi196.stdout:9/916: creat db/dd/de/dd7/f133 x:0 0 0 2024-04-22T15:19:12.806 INFO:tasks.workunit.client.1.smithi196.stdout:3/953: fdatasync d0/d15/d31/dd0/de2/d13b/fc7 0 2024-04-22T15:19:12.807 INFO:tasks.workunit.client.1.smithi196.stdout:1/878: truncate d1/db/dc/dad/fb9 375414 0 2024-04-22T15:19:12.808 INFO:tasks.workunit.client.1.smithi196.stdout:0/886: rmdir d3/d9/d36/d9f/dd1/d65/d7f/ddb 39 2024-04-22T15:19:12.809 INFO:tasks.workunit.client.1.smithi196.stdout:6/866: read dd/d1f/d38/d40/f4f [4155138,29077] 0 2024-04-22T15:19:12.809 INFO:tasks.workunit.client.1.smithi196.stdout:0/887: write d3/db4/f12c [59280,84642] 0 2024-04-22T15:19:12.810 INFO:tasks.workunit.client.1.smithi196.stdout:5/891: getdents d1 0 2024-04-22T15:19:12.811 INFO:tasks.workunit.client.1.smithi196.stdout:5/892: stat d1/d17/c40 0 2024-04-22T15:19:12.811 INFO:tasks.workunit.client.1.smithi196.stdout:6/867: write dd/d1f/d37/f31 [2979068,25049] 0 2024-04-22T15:19:12.811 INFO:tasks.workunit.client.1.smithi196.stdout:8/843: symlink d2/d8/d49/lf6 0 2024-04-22T15:19:12.813 INFO:tasks.workunit.client.1.smithi196.stdout:0/888: truncate d3/d6/dc/d8c/d8d/d98/dbd/f126 902905 0 2024-04-22T15:19:12.813 INFO:tasks.workunit.client.1.smithi196.stdout:1/879: creat d1/d54/de2/df1/f11b x:0 0 0 2024-04-22T15:19:12.814 INFO:tasks.workunit.client.1.smithi196.stdout:1/880: stat d1/db/dc/d26/l84 0 2024-04-22T15:19:12.815 INFO:tasks.workunit.client.1.smithi196.stdout:3/954: creat d0/d15/d31/d55/dcf/d6f/de5/d101/d9e/d117/f141 x:0 0 0 2024-04-22T15:19:12.816 INFO:tasks.workunit.client.0.smithi096.stdout:6/956: write d3/d3a/d75/fd3 [1924950,69382] 0 2024-04-22T15:19:12.817 INFO:tasks.workunit.client.1.smithi196.stdout:7/875: getdents d14/d15/d91 0 2024-04-22T15:19:12.817 INFO:tasks.workunit.client.1.smithi196.stdout:3/955: chown d0/d15/da7/d4a/d44/f6c 2 1 2024-04-22T15:19:12.820 INFO:tasks.workunit.client.1.smithi196.stdout:7/876: dread - d14/d26/d2a/d2b/de8/dc6/fd4 zero size 2024-04-22T15:19:12.822 INFO:tasks.workunit.client.1.smithi196.stdout:0/889: mkdir d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/d137 0 2024-04-22T15:19:12.822 INFO:tasks.workunit.client.0.smithi096.stdout:6/957: dwrite d3/d28/d4d/d82/d9e/d137/fa5 [0,4194304] 0 2024-04-22T15:19:12.824 INFO:tasks.workunit.client.1.smithi196.stdout:2/882: dwrite d1/d13/dbc/ff1 [0,4194304] 0 2024-04-22T15:19:12.824 INFO:tasks.workunit.client.1.smithi196.stdout:7/877: write d14/d4d/dea/f10e [777302,35032] 0 2024-04-22T15:19:12.827 INFO:tasks.workunit.client.1.smithi196.stdout:8/844: symlink d2/d8/d49/lf7 0 2024-04-22T15:19:12.828 INFO:tasks.workunit.client.1.smithi196.stdout:1/881: read d1/db/dc/de/d5a/d94/fa4 [1130113,58492] 0 2024-04-22T15:19:12.831 INFO:tasks.workunit.client.1.smithi196.stdout:9/917: getdents db/dd/de/d15/d31/d64 0 2024-04-22T15:19:12.833 INFO:tasks.workunit.client.1.smithi196.stdout:7/878: rmdir d14/d26 39 2024-04-22T15:19:12.836 INFO:tasks.workunit.client.0.smithi096.stdout:6/958: write d3/d28/f38 [706599,46662] 0 2024-04-22T15:19:12.839 INFO:tasks.workunit.client.0.smithi096.stdout:6/959: mknod d3/d28/d4d/d82/d9e/db0/c144 0 2024-04-22T15:19:12.840 INFO:tasks.workunit.client.1.smithi196.stdout:2/883: dread d1/d23/d27/d2e/d55/d5b/d7c/f7f [0,4194304] 0 2024-04-22T15:19:12.840 INFO:tasks.workunit.client.0.smithi096.stdout:6/960: fdatasync d3/d28/d4d/d82/d9e/d137/d11/d2c/d12f/f13f 0 2024-04-22T15:19:12.842 INFO:tasks.workunit.client.0.smithi096.stdout:6/961: rename d3/d28/d4d/d82/d9e to d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/ddd/d117/d145 22 2024-04-22T15:19:12.842 INFO:tasks.workunit.client.1.smithi196.stdout:3/956: write d0/d15/da7/d4a/f45 [653277,32909] 0 2024-04-22T15:19:12.843 INFO:tasks.workunit.client.1.smithi196.stdout:1/882: creat d1/db/d16/d1e/d65/f11c x:0 0 0 2024-04-22T15:19:12.844 INFO:tasks.workunit.client.1.smithi196.stdout:8/845: creat d2/d5/d1a/db7/dd9/d68/dbc/dae/ff8 x:0 0 0 2024-04-22T15:19:12.846 INFO:tasks.workunit.client.1.smithi196.stdout:7/879: rename d14/d4d/da3/de9/dfe to d14/d25/d31/d127 0 2024-04-22T15:19:12.846 INFO:tasks.workunit.client.1.smithi196.stdout:4/853: dwrite d6/d14/d18/d26/d32/da3/ff0 [0,4194304] 0 2024-04-22T15:19:12.847 INFO:tasks.workunit.client.1.smithi196.stdout:0/890: dwrite d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/f112 [0,4194304] 0 2024-04-22T15:19:12.848 INFO:tasks.workunit.client.0.smithi096.stdout:6/962: dwrite f2 [0,4194304] 0 2024-04-22T15:19:12.851 INFO:tasks.workunit.client.1.smithi196.stdout:5/893: dread d1/d4/f9c [0,4194304] 0 2024-04-22T15:19:12.851 INFO:tasks.workunit.client.0.smithi096.stdout:6/963: mkdir d3/d28/d4d/d82/d9e/d137/d11/d2c/d12f/d146 0 2024-04-22T15:19:12.851 INFO:tasks.workunit.client.0.smithi096.stdout:6/964: readlink d3/d28/d54/d68/d6e/le7 0 2024-04-22T15:19:12.856 INFO:tasks.workunit.client.1.smithi196.stdout:6/868: dwrite dd/d1f/f47 [0,4194304] 0 2024-04-22T15:19:12.859 INFO:tasks.workunit.client.1.smithi196.stdout:1/883: stat d1/db/dc/de/d5a/c10b 0 2024-04-22T15:19:12.860 INFO:tasks.workunit.client.1.smithi196.stdout:6/869: fsync dd/d1f/d4e/d53/d7f/dea/feb 0 2024-04-22T15:19:12.861 INFO:tasks.workunit.client.0.smithi096.stdout:6/965: creat d3/d28/d4d/f147 x:0 0 0 2024-04-22T15:19:12.863 INFO:tasks.workunit.client.1.smithi196.stdout:2/884: unlink d1/d23/d27/d73/fc6 0 2024-04-22T15:19:12.863 INFO:tasks.workunit.client.1.smithi196.stdout:1/884: truncate d1/db/d16/ffb 307911 0 2024-04-22T15:19:12.865 INFO:tasks.workunit.client.0.smithi096.stdout:6/966: rmdir d3/d28/dc9 39 2024-04-22T15:19:12.866 INFO:tasks.workunit.client.1.smithi196.stdout:0/891: dwrite d3/de/d1c/d27/f93 [0,4194304] 0 2024-04-22T15:19:12.867 INFO:tasks.workunit.client.1.smithi196.stdout:7/880: mknod d14/d26/d5e/c128 0 2024-04-22T15:19:12.868 INFO:tasks.workunit.client.1.smithi196.stdout:6/870: readlink dd/d1f/la7 0 2024-04-22T15:19:12.869 INFO:tasks.workunit.client.0.smithi096.stdout:6/967: truncate d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/ddd/de9/f13d 951109 0 2024-04-22T15:19:12.870 INFO:tasks.workunit.client.0.smithi096.stdout:6/968: chown d3/d28/d4d/d82/d9e/db0/l12b 53 1 2024-04-22T15:19:12.871 INFO:tasks.workunit.client.1.smithi196.stdout:1/885: rename d1/db/d16/d1e/d65/f76 to d1/db/dc/de/d96/f11d 0 2024-04-22T15:19:12.875 INFO:tasks.workunit.client.1.smithi196.stdout:7/881: creat d14/d26/d2a/d2b/f129 x:0 0 0 2024-04-22T15:19:12.876 INFO:tasks.workunit.client.1.smithi196.stdout:6/871: mknod dd/d1f/d4e/d53/d7f/da5/c103 0 2024-04-22T15:19:12.878 INFO:tasks.workunit.client.1.smithi196.stdout:2/885: mknod d1/d23/d27/d2e/d55/d5b/d93/d97/d11e/c13c 0 2024-04-22T15:19:12.879 INFO:tasks.workunit.client.1.smithi196.stdout:3/957: getdents d0/d15/d31/dd0/de2/d13b 0 2024-04-22T15:19:12.879 INFO:tasks.workunit.client.0.smithi096.stdout:6/969: getdents d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4 0 2024-04-22T15:19:12.880 INFO:tasks.workunit.client.1.smithi196.stdout:7/882: creat d14/d26/d2a/d2b/f12a x:0 0 0 2024-04-22T15:19:12.880 INFO:tasks.workunit.client.0.smithi096.stdout:6/970: chown d3/d3a/d75/d83/f86 0 1 2024-04-22T15:19:12.881 INFO:tasks.workunit.client.1.smithi196.stdout:1/886: dwrite d1/db/dc/de/d96/d9d/f50 [0,4194304] 0 2024-04-22T15:19:12.881 INFO:tasks.workunit.client.1.smithi196.stdout:0/892: rmdir d3/d9/d36/d12e 0 2024-04-22T15:19:12.882 INFO:tasks.workunit.client.1.smithi196.stdout:0/893: readlink d3/d6/dc/l41 0 2024-04-22T15:19:12.884 INFO:tasks.workunit.client.1.smithi196.stdout:0/894: readlink d3/d9/l3b 0 2024-04-22T15:19:12.884 INFO:tasks.workunit.client.1.smithi196.stdout:1/887: read - d1/db/d16/d44/ff0 zero size 2024-04-22T15:19:12.885 INFO:tasks.workunit.client.1.smithi196.stdout:2/886: symlink d1/d23/d126/d127/l13d 0 2024-04-22T15:19:12.886 INFO:tasks.workunit.client.1.smithi196.stdout:0/895: write d3/d9/d36/d9f/dd1/d65/f6b [3175838,3319] 0 2024-04-22T15:19:12.888 INFO:tasks.workunit.client.1.smithi196.stdout:3/958: unlink d0/d15/d31/l12f 0 2024-04-22T15:19:12.891 INFO:tasks.workunit.client.1.smithi196.stdout:2/887: mknod d1/d23/d27/d89/c13e 0 2024-04-22T15:19:12.891 INFO:tasks.workunit.client.1.smithi196.stdout:4/854: dread d6/d13/d15/f5e [0,4194304] 0 2024-04-22T15:19:12.893 INFO:tasks.workunit.client.1.smithi196.stdout:1/888: rename d1/db/dc/d26/d2e/f3e to d1/db/d97/dd7/f11e 0 2024-04-22T15:19:12.894 INFO:tasks.workunit.client.1.smithi196.stdout:0/896: rmdir d3/de 39 2024-04-22T15:19:12.895 INFO:tasks.workunit.client.1.smithi196.stdout:4/855: write d6/d13/f24 [1167006,13572] 0 2024-04-22T15:19:12.896 INFO:tasks.workunit.client.1.smithi196.stdout:6/872: getdents dd/d1f/d4e/d53/d7f 0 2024-04-22T15:19:12.897 INFO:tasks.workunit.client.1.smithi196.stdout:3/959: creat d0/d15/d31/dcb/f142 x:0 0 0 2024-04-22T15:19:12.898 INFO:tasks.workunit.client.1.smithi196.stdout:8/846: dwrite d2/dc/f3a [0,4194304] 0 2024-04-22T15:19:12.900 INFO:tasks.workunit.client.1.smithi196.stdout:6/873: chown dd/f12 346 1 2024-04-22T15:19:12.901 INFO:tasks.workunit.client.1.smithi196.stdout:0/897: symlink d3/d6/dc/d8c/d8d/l138 0 2024-04-22T15:19:12.901 INFO:tasks.workunit.client.1.smithi196.stdout:8/847: chown d2/d5/d1a/db7/dd9/d68/dbc/c9a 585202 1 2024-04-22T15:19:12.901 INFO:tasks.workunit.client.1.smithi196.stdout:3/960: write d0/d15/da7/d4a/d44/d96/dd8/d123/f13e [829914,101260] 0 2024-04-22T15:19:12.902 INFO:tasks.workunit.client.1.smithi196.stdout:1/889: mkdir d1/db/d97/dd7/d10e/d11f 0 2024-04-22T15:19:12.902 INFO:tasks.workunit.client.1.smithi196.stdout:4/856: mkdir d6/d13/d15/d2c/d62/d118 0 2024-04-22T15:19:12.907 INFO:tasks.workunit.client.1.smithi196.stdout:7/883: dread fc [0,4194304] 0 2024-04-22T15:19:12.908 INFO:tasks.workunit.client.1.smithi196.stdout:2/888: link d1/d23/d27/d2e/d55/d5b/d7c/fe4 d1/d13/d25/d2a/ddb/dff/f13f 0 2024-04-22T15:19:12.908 INFO:tasks.workunit.client.0.smithi096.stdout:6/971: dwrite d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/f99 [4194304,4194304] 0 2024-04-22T15:19:12.910 INFO:tasks.workunit.client.1.smithi196.stdout:3/961: fdatasync d0/d15/d31/d55/dcf/d6f/de5/d101/d9e/fab 0 2024-04-22T15:19:12.910 INFO:tasks.workunit.client.1.smithi196.stdout:7/884: readlink d14/d26/db7/lc9 0 2024-04-22T15:19:12.910 INFO:tasks.workunit.client.1.smithi196.stdout:2/889: chown d1/d13/c129 3 1 2024-04-22T15:19:12.914 INFO:tasks.workunit.client.1.smithi196.stdout:6/874: dread dd/d20/d5c/d6a/faa [0,4194304] 0 2024-04-22T15:19:12.915 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:12 smithi096 ceph-mon[26688]: mgrmap e26: smithi196.gdaegc(active, since 79s), standbys: smithi096.hhkvum 2024-04-22T15:19:12.916 INFO:tasks.workunit.client.1.smithi196.stdout:4/857: link d6/d14/d18/d26/d32/da3/fea d6/d10/d105/f119 0 2024-04-22T15:19:12.916 INFO:tasks.workunit.client.1.smithi196.stdout:1/890: dread d1/db/d16/d1e/f88 [0,4194304] 0 2024-04-22T15:19:12.919 INFO:tasks.workunit.client.1.smithi196.stdout:7/885: mkdir d14/d26/db7/d10b/d12b 0 2024-04-22T15:19:12.919 INFO:tasks.workunit.client.1.smithi196.stdout:8/848: link d2/d5/d1a/db7/dd9/d68/c78 d2/d6a/d7b/cf9 0 2024-04-22T15:19:12.919 INFO:tasks.workunit.client.1.smithi196.stdout:1/891: fsync d1/db/dc/dad/d36/fc9 0 2024-04-22T15:19:12.920 INFO:tasks.workunit.client.1.smithi196.stdout:0/898: dread d3/d6/dc/d8c/d8d/da8/fbe [0,4194304] 0 2024-04-22T15:19:12.920 INFO:tasks.workunit.client.1.smithi196.stdout:7/886: write d14/d25/d31/f65 [458379,82576] 0 2024-04-22T15:19:12.921 INFO:tasks.workunit.client.1.smithi196.stdout:1/892: write d1/db/d16/d44/ff0 [15943,92400] 0 2024-04-22T15:19:12.921 INFO:tasks.workunit.client.1.smithi196.stdout:1/893: readlink d1/db/dc/l1f 0 2024-04-22T15:19:12.925 INFO:tasks.workunit.client.1.smithi196.stdout:2/890: rename d1/d23/d27/d2e/d55/d5b/d7c/f7f to d1/d13/d41/f140 0 2024-04-22T15:19:12.926 INFO:tasks.workunit.client.1.smithi196.stdout:6/875: dwrite dd/d1f/d38/d40/fcd [0,4194304] 0 2024-04-22T15:19:12.926 INFO:tasks.workunit.client.1.smithi196.stdout:4/858: creat d6/d10/d105/f11a x:0 0 0 2024-04-22T15:19:12.931 INFO:tasks.workunit.client.1.smithi196.stdout:3/962: creat d0/d15/d31/d55/dcf/d8/dfc/f143 x:0 0 0 2024-04-22T15:19:12.935 INFO:tasks.workunit.client.1.smithi196.stdout:4/859: mkdir d6/d13/d15/d2c/d62/dd6/d11b 0 2024-04-22T15:19:12.936 INFO:tasks.workunit.client.1.smithi196.stdout:8/849: mknod d2/d8/cfa 0 2024-04-22T15:19:12.937 INFO:tasks.workunit.client.1.smithi196.stdout:0/899: fdatasync d3/de/d14/f4a 0 2024-04-22T15:19:12.943 INFO:tasks.workunit.client.1.smithi196.stdout:8/850: dwrite d2/dc/d45/d98/fba [0,4194304] 0 2024-04-22T15:19:12.944 INFO:tasks.workunit.client.1.smithi196.stdout:8/851: fdatasync d2/d5/d1a/db7/dd9/d68/fa3 0 2024-04-22T15:19:12.944 INFO:tasks.workunit.client.0.smithi096.stdout:6/972: dwrite d3/d28/d54/ffc [0,4194304] 0 2024-04-22T15:19:12.946 INFO:tasks.workunit.client.1.smithi196.stdout:8/852: write d2/d5/d1a/db7/dd9/d68/dbc/dae/fe6 [486670,14541] 0 2024-04-22T15:19:12.948 INFO:tasks.workunit.client.1.smithi196.stdout:8/853: chown d2/d5/d1a/db7/dd9/d68/dbc/da4/ld4 28 1 2024-04-22T15:19:12.951 INFO:tasks.workunit.client.0.smithi096.stdout:6/973: symlink d3/d28/daf/l148 0 2024-04-22T15:19:12.952 INFO:tasks.workunit.client.1.smithi196.stdout:7/887: creat d14/d26/d5a/f12c x:0 0 0 2024-04-22T15:19:12.955 INFO:tasks.workunit.client.1.smithi196.stdout:3/963: rmdir d0/d15/d31/d55/d8b/d102 0 2024-04-22T15:19:12.957 INFO:tasks.workunit.client.0.smithi096.stdout:6/974: rename d3/d28/d4d/f147 to d3/d28/d4d/d82/d9e/d137/d11/d2c/d3b/f149 0 2024-04-22T15:19:12.958 INFO:tasks.workunit.client.1.smithi196.stdout:1/894: dwrite d1/db/dc/d93/fb6 [0,4194304] 0 2024-04-22T15:19:12.958 INFO:tasks.workunit.client.1.smithi196.stdout:0/900: mknod d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/c139 0 2024-04-22T15:19:12.960 INFO:tasks.workunit.client.0.smithi096.stdout:6/975: creat d3/d3a/d75/f14a x:0 0 0 2024-04-22T15:19:12.960 INFO:tasks.workunit.client.1.smithi196.stdout:8/854: creat d2/d6a/d7b/ffb x:0 0 0 2024-04-22T15:19:12.960 INFO:tasks.workunit.client.1.smithi196.stdout:6/876: dread dd/d1f/d4e/d53/d7f/da5/fdc [4194304,4194304] 0 2024-04-22T15:19:12.961 INFO:tasks.workunit.client.1.smithi196.stdout:2/891: getdents d1/d23/d27/d2e/db5 0 2024-04-22T15:19:12.962 INFO:tasks.workunit.client.1.smithi196.stdout:6/877: stat dd/d2b/f77 0 2024-04-22T15:19:12.964 INFO:tasks.workunit.client.1.smithi196.stdout:7/888: write d14/d26/d2a/d2b/f70 [2671109,111845] 0 2024-04-22T15:19:12.964 INFO:tasks.workunit.client.1.smithi196.stdout:9/918: sync 2024-04-22T15:19:12.965 INFO:tasks.workunit.client.1.smithi196.stdout:2/892: chown d1/d23/d27/d2e/d55/d5b/d93/f130 6 1 2024-04-22T15:19:12.966 INFO:tasks.workunit.client.1.smithi196.stdout:4/860: getdents d6/d13/d15/d2c/d62/d78/d8d/dff 0 2024-04-22T15:19:12.966 INFO:tasks.workunit.client.1.smithi196.stdout:1/895: symlink d1/db/d16/d1e/d46/ddc/l120 0 2024-04-22T15:19:12.967 INFO:tasks.workunit.client.1.smithi196.stdout:3/964: dwrite d0/d15/d31/d55/dcf/d8/fea [0,4194304] 0 2024-04-22T15:19:12.968 INFO:tasks.workunit.client.0.smithi096.stdout:6/976: dwrite d3/d28/d4d/d82/fb9 [0,4194304] 0 2024-04-22T15:19:12.969 INFO:tasks.workunit.client.1.smithi196.stdout:4/861: stat d6/f9f 0 2024-04-22T15:19:12.971 INFO:tasks.workunit.client.1.smithi196.stdout:7/889: creat d14/d25/d80/f12d x:0 0 0 2024-04-22T15:19:12.972 INFO:tasks.workunit.client.0.smithi096.stdout:6/977: rename d3/d28/d4d/d82/d9e/d137/d11/d10a to d3/d28/daf/d14b 0 2024-04-22T15:19:12.973 INFO:tasks.workunit.client.0.smithi096.stdout:6/978: readlink d3/d28/d54/d68/d6e/d10d/l127 0 2024-04-22T15:19:12.974 INFO:tasks.workunit.client.1.smithi196.stdout:1/896: write d1/db/dc/de/d5a/d5b/f79 [1486147,37106] 0 2024-04-22T15:19:12.974 INFO:tasks.workunit.client.1.smithi196.stdout:2/893: creat d1/d23/d27/d2e/db5/f141 x:0 0 0 2024-04-22T15:19:12.978 INFO:tasks.workunit.client.1.smithi196.stdout:1/897: write d1/db/dc/de/d96/d9d/fe0 [5076133,59480] 0 2024-04-22T15:19:12.979 INFO:tasks.workunit.client.1.smithi196.stdout:3/965: mknod d0/d15/d4d/d87/da6/c144 0 2024-04-22T15:19:12.979 INFO:tasks.workunit.client.0.smithi096.stdout:6/979: write d3/d28/d4d/d82/d9e/d137/d11/d2c/fb1 [3361178,44122] 0 2024-04-22T15:19:12.980 INFO:tasks.workunit.client.1.smithi196.stdout:0/901: dread d3/d9/d36/fd2 [0,4194304] 0 2024-04-22T15:19:12.986 INFO:tasks.workunit.client.1.smithi196.stdout:4/862: dwrite d6/d14/d18/d40/d85/fe5 [0,4194304] 0 2024-04-22T15:19:12.986 INFO:tasks.workunit.client.1.smithi196.stdout:7/890: fdatasync d14/d15/d91/d9a/fc4 0 2024-04-22T15:19:12.987 INFO:tasks.workunit.client.0.smithi096.stdout:6/980: mkdir d3/d28/d4d/d82/d9e/d137/d11/d2c/d14c 0 2024-04-22T15:19:12.988 INFO:tasks.workunit.client.1.smithi196.stdout:4/863: fsync d6/d13/d15/d2c/d62/d78/ff1 0 2024-04-22T15:19:12.989 INFO:tasks.workunit.client.1.smithi196.stdout:3/966: write d0/d15/d4d/f5a [66143,48688] 0 2024-04-22T15:19:12.991 INFO:tasks.workunit.client.0.smithi096.stdout:6/981: mkdir d3/d28/d54/d68/d6e/d10d/d14d 0 2024-04-22T15:19:12.993 INFO:tasks.workunit.client.1.smithi196.stdout:1/898: dwrite d1/db/d16/d6d/de6/f104 [0,4194304] 0 2024-04-22T15:19:12.995 INFO:tasks.workunit.client.1.smithi196.stdout:0/902: rmdir d3/d9/d36/d9f/dd1/d65/d7f 39 2024-04-22T15:19:12.997 INFO:tasks.workunit.client.0.smithi096.stdout:6/982: dread d3/d28/d4d/d82/d9e/d137/d11/f1e [0,4194304] 0 2024-04-22T15:19:12.999 INFO:tasks.workunit.client.1.smithi196.stdout:6/878: dwrite dd/d1f/d4e/d53/d55/f95 [0,4194304] 0 2024-04-22T15:19:13.000 INFO:tasks.workunit.client.1.smithi196.stdout:7/891: rename d14/d4d/da3/l105 to d14/d26/d2a/l12e 0 2024-04-22T15:19:13.001 INFO:tasks.workunit.client.1.smithi196.stdout:7/892: read - d14/d26/fd0 zero size 2024-04-22T15:19:13.003 INFO:tasks.workunit.client.1.smithi196.stdout:9/919: dwrite db/dd/de/dd7/def/f84 [0,4194304] 0 2024-04-22T15:19:13.004 INFO:tasks.workunit.client.1.smithi196.stdout:2/894: write d1/d23/d27/d2e/d55/d5b/d93/fbf [747140,55356] 0 2024-04-22T15:19:13.004 INFO:tasks.workunit.client.1.smithi196.stdout:4/864: rmdir d6/d14/d18/d26 39 2024-04-22T15:19:13.008 INFO:tasks.workunit.client.1.smithi196.stdout:8/855: dread d2/d8/f76 [0,4194304] 0 2024-04-22T15:19:13.011 INFO:tasks.workunit.client.1.smithi196.stdout:0/903: dread d3/d6/dc/d8c/daf/f7c [0,4194304] 0 2024-04-22T15:19:13.014 INFO:tasks.workunit.client.1.smithi196.stdout:4/865: fdatasync d6/d13/d15/fb2 0 2024-04-22T15:19:13.015 INFO:tasks.workunit.client.0.smithi096.stdout:6/983: dwrite d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/dd6/fc4 [0,4194304] 0 2024-04-22T15:19:13.016 INFO:tasks.workunit.client.1.smithi196.stdout:4/866: chown d6/d14/d18/d40/d85/f100 0 1 2024-04-22T15:19:13.016 INFO:tasks.workunit.client.1.smithi196.stdout:5/894: sync 2024-04-22T15:19:13.017 INFO:tasks.workunit.client.1.smithi196.stdout:2/895: creat d1/d13/d1f/f142 x:0 0 0 2024-04-22T15:19:13.018 INFO:tasks.workunit.client.1.smithi196.stdout:2/896: write d1/f117 [4823605,8463] 0 2024-04-22T15:19:13.021 INFO:tasks.workunit.client.1.smithi196.stdout:0/904: mknod d3/de/d1c/dd4/c13a 0 2024-04-22T15:19:13.023 INFO:tasks.workunit.client.1.smithi196.stdout:9/920: creat db/dd/de/d98/f134 x:0 0 0 2024-04-22T15:19:13.023 INFO:tasks.workunit.client.1.smithi196.stdout:4/867: creat d6/d13/d15/d2c/d62/d78/df6/f11c x:0 0 0 2024-04-22T15:19:13.024 INFO:tasks.workunit.client.0.smithi096.stdout:6/984: dwrite d3/d3a/d75/fd3 [0,4194304] 0 2024-04-22T15:19:13.024 INFO:tasks.workunit.client.1.smithi196.stdout:9/921: write db/dd/de/dd7/def/d7a/fbb [1810084,25000] 0 2024-04-22T15:19:13.025 INFO:tasks.workunit.client.1.smithi196.stdout:5/895: symlink d1/d78/dcf/dff/d101/l119 0 2024-04-22T15:19:13.026 INFO:tasks.workunit.client.1.smithi196.stdout:9/922: stat db/dd/de/dd7/def/fd8 0 2024-04-22T15:19:13.026 INFO:tasks.workunit.client.1.smithi196.stdout:2/897: fsync d1/f18 0 2024-04-22T15:19:13.027 INFO:tasks.workunit.client.1.smithi196.stdout:8/856: truncate d2/d8/d49/d6c/fc1 1644457 0 2024-04-22T15:19:13.028 INFO:tasks.workunit.client.1.smithi196.stdout:6/879: creat dd/d2b/f104 x:0 0 0 2024-04-22T15:19:13.033 INFO:tasks.workunit.client.1.smithi196.stdout:9/923: read db/dd/de/d15/f73 [953174,47558] 0 2024-04-22T15:19:13.033 INFO:tasks.workunit.client.1.smithi196.stdout:3/967: getdents d0/d15/d31/d55/dcf/d6f/d10a/ddd/df6 0 2024-04-22T15:19:13.034 INFO:tasks.workunit.client.1.smithi196.stdout:5/896: fsync d1/d2/d98/dac/d105/f38 0 2024-04-22T15:19:13.034 INFO:tasks.workunit.client.1.smithi196.stdout:2/898: rmdir d1/d23/d27/d2e/d55 39 2024-04-22T15:19:13.038 INFO:tasks.workunit.client.1.smithi196.stdout:6/880: write dd/d1f/d4e/fd8 [85447,24088] 0 2024-04-22T15:19:13.040 INFO:tasks.workunit.client.1.smithi196.stdout:9/924: creat db/dd/d1b/d2c/f135 x:0 0 0 2024-04-22T15:19:13.042 INFO:tasks.workunit.client.1.smithi196.stdout:6/881: write dd/d2b/f2f [7510772,52429] 0 2024-04-22T15:19:13.043 INFO:tasks.workunit.client.1.smithi196.stdout:1/899: dread d1/db/d16/d1e/f60 [0,4194304] 0 2024-04-22T15:19:13.043 INFO:tasks.workunit.client.1.smithi196.stdout:7/893: dwrite d14/d26/d2a/d2b/f2d [4194304,4194304] 0 2024-04-22T15:19:13.044 INFO:tasks.workunit.client.1.smithi196.stdout:6/882: write dd/f86 [1127594,114573] 0 2024-04-22T15:19:13.046 INFO:tasks.workunit.client.1.smithi196.stdout:8/857: fsync d2/d6a/f7e 0 2024-04-22T15:19:13.046 INFO:tasks.workunit.client.1.smithi196.stdout:5/897: symlink d1/d17/d1c/d100/l11a 0 2024-04-22T15:19:13.049 INFO:tasks.workunit.client.1.smithi196.stdout:4/868: write d6/d13/d15/d2c/d62/f99 [1804162,72510] 0 2024-04-22T15:19:13.052 INFO:tasks.workunit.client.1.smithi196.stdout:0/905: getdents d3/d6/dc/d8c/d8d/d98 0 2024-04-22T15:19:13.052 INFO:tasks.workunit.client.1.smithi196.stdout:2/899: rename d1/d23/d27/d2e/d55/d5b/d93/d97/ff2 to d1/d13/d25/d9f/ddd/f143 0 2024-04-22T15:19:13.054 INFO:tasks.workunit.client.1.smithi196.stdout:2/900: write d1/fa2 [976109,87519] 0 2024-04-22T15:19:13.057 INFO:tasks.workunit.client.1.smithi196.stdout:5/898: creat d1/df1/d6e/f11b x:0 0 0 2024-04-22T15:19:13.058 INFO:tasks.workunit.client.1.smithi196.stdout:7/894: symlink d14/d25/d31/d127/l12f 0 2024-04-22T15:19:13.061 INFO:tasks.workunit.client.1.smithi196.stdout:3/968: dwrite d0/d15/d31/d55/dcf/d6f/d10a/fa9 [0,4194304] 0 2024-04-22T15:19:13.062 INFO:tasks.workunit.client.1.smithi196.stdout:7/895: chown d14/d15/f106 234 1 2024-04-22T15:19:13.066 INFO:tasks.workunit.client.1.smithi196.stdout:0/906: dwrite d3/db4/f12c [0,4194304] 0 2024-04-22T15:19:13.066 INFO:tasks.workunit.client.1.smithi196.stdout:1/900: creat d1/db/dc/de/d5a/d5b/f121 x:0 0 0 2024-04-22T15:19:13.067 INFO:tasks.workunit.client.1.smithi196.stdout:5/899: unlink d1/d2/d98/dac/d105/d68/lb4 0 2024-04-22T15:19:13.068 INFO:tasks.workunit.client.1.smithi196.stdout:3/969: chown d0/d15/da7/d4a/d44/f6d 70267635 1 2024-04-22T15:19:13.069 INFO:tasks.workunit.client.1.smithi196.stdout:7/896: rmdir d14/ddd 39 2024-04-22T15:19:13.069 INFO:tasks.workunit.client.1.smithi196.stdout:4/869: rmdir d6/de8/d109 0 2024-04-22T15:19:13.072 INFO:tasks.workunit.client.1.smithi196.stdout:6/883: getdents dd/d20/d5c 0 2024-04-22T15:19:13.072 INFO:tasks.workunit.client.1.smithi196.stdout:4/870: readlink d6/d14/d18/d40/l48 0 2024-04-22T15:19:13.073 INFO:tasks.workunit.client.1.smithi196.stdout:1/901: symlink d1/db/dc/dad/d36/l122 0 2024-04-22T15:19:13.073 INFO:tasks.workunit.client.1.smithi196.stdout:7/897: chown d14/d15/f106 14 1 2024-04-22T15:19:13.073 INFO:tasks.workunit.client.1.smithi196.stdout:0/907: chown d3/de/c44 3985 1 2024-04-22T15:19:13.074 INFO:tasks.workunit.client.1.smithi196.stdout:5/900: write d1/d4/f22 [8756332,37257] 0 2024-04-22T15:19:13.078 INFO:tasks.workunit.client.1.smithi196.stdout:9/925: dwrite f5 [0,4194304] 0 2024-04-22T15:19:13.078 INFO:tasks.workunit.client.1.smithi196.stdout:7/898: rmdir d14/d26/d2a 39 2024-04-22T15:19:13.079 INFO:tasks.workunit.client.1.smithi196.stdout:0/908: chown d3/d9/d36/d9f/dd1/d65 503921 1 2024-04-22T15:19:13.080 INFO:tasks.workunit.client.1.smithi196.stdout:5/901: creat d1/d2/dbb/f11c x:0 0 0 2024-04-22T15:19:13.081 INFO:tasks.workunit.client.1.smithi196.stdout:4/871: mkdir d6/d14/d18/d26/d32/d67/d9a/da5/d11d 0 2024-04-22T15:19:13.083 INFO:tasks.workunit.client.1.smithi196.stdout:9/926: mkdir db/dd/de/dd7/def/d7a/d7b/d81/d136 0 2024-04-22T15:19:13.085 INFO:tasks.workunit.client.1.smithi196.stdout:0/909: fdatasync d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/f112 0 2024-04-22T15:19:13.087 INFO:tasks.workunit.client.1.smithi196.stdout:0/910: write d3/d9/d36/d9f/dd1/d65/f6b [2170341,105670] 0 2024-04-22T15:19:13.088 INFO:tasks.workunit.client.1.smithi196.stdout:6/884: rename dd/d20/d5c/d5e/f7c to dd/d1f/d38/f105 0 2024-04-22T15:19:13.089 INFO:tasks.workunit.client.1.smithi196.stdout:8/858: truncate d2/d35/fb4 3709867 0 2024-04-22T15:19:13.091 INFO:tasks.workunit.client.1.smithi196.stdout:3/970: link d0/d15/d31/d55/d6e/dfe/c13f d0/d15/d31/dd0/c145 0 2024-04-22T15:19:13.094 INFO:tasks.workunit.client.1.smithi196.stdout:9/927: getdents db/dd/d1b/de6 0 2024-04-22T15:19:13.097 INFO:tasks.workunit.client.1.smithi196.stdout:5/902: dwrite d1/df1/d5e/dcb/f104 [4194304,4194304] 0 2024-04-22T15:19:13.098 INFO:tasks.workunit.client.1.smithi196.stdout:2/901: dwrite d1/d23/d27/d2e/fd8 [0,4194304] 0 2024-04-22T15:19:13.100 INFO:tasks.workunit.client.1.smithi196.stdout:6/885: mkdir dd/d1f/d4e/d53/dee/d106 0 2024-04-22T15:19:13.100 INFO:tasks.workunit.client.1.smithi196.stdout:1/902: write d1/db/fc0 [613129,27838] 0 2024-04-22T15:19:13.106 INFO:tasks.workunit.client.1.smithi196.stdout:1/903: chown d1/d54/de2/df1/df4 72767875 1 2024-04-22T15:19:13.110 INFO:tasks.workunit.client.1.smithi196.stdout:8/859: rename d2/d3f/d4b/f72 to d2/d5/d1a/db7/dd9/d68/dbc/da4/ffc 0 2024-04-22T15:19:13.115 INFO:tasks.workunit.client.1.smithi196.stdout:9/928: fsync db/dd/f88 0 2024-04-22T15:19:13.115 INFO:tasks.workunit.client.1.smithi196.stdout:4/872: dread d6/d90/f60 [0,4194304] 0 2024-04-22T15:19:13.116 INFO:tasks.workunit.client.1.smithi196.stdout:6/886: dread - dd/d1f/d37/d7b/f8a zero size 2024-04-22T15:19:13.117 INFO:tasks.workunit.client.1.smithi196.stdout:7/899: dwrite d14/f1e [0,4194304] 0 2024-04-22T15:19:13.118 INFO:tasks.workunit.client.1.smithi196.stdout:2/902: rename d1/d23/d27/d2e/d55/f123 to d1/d13/d41/f144 0 2024-04-22T15:19:13.118 INFO:tasks.workunit.client.1.smithi196.stdout:6/887: readlink dd/d1f/d4e/d53/d55/d57/l6b 0 2024-04-22T15:19:13.119 INFO:tasks.workunit.client.1.smithi196.stdout:0/911: write d3/d9/d36/d9f/dd1/d65/d7f/fd8 [4976672,122362] 0 2024-04-22T15:19:13.122 INFO:tasks.workunit.client.1.smithi196.stdout:8/860: symlink d2/d5/d1a/db7/dd9/d68/lfd 0 2024-04-22T15:19:13.127 INFO:tasks.workunit.client.1.smithi196.stdout:6/888: dwrite dd/d2b/d8d/fb2 [0,4194304] 0 2024-04-22T15:19:13.128 INFO:tasks.workunit.client.1.smithi196.stdout:1/904: rename d1/db/dc/cea to d1/db/dc/de/d96/d9d/db1/c123 0 2024-04-22T15:19:13.128 INFO:tasks.workunit.client.1.smithi196.stdout:3/971: link d0/d15/f12 d0/d15/d31/d55/d5d/f146 0 2024-04-22T15:19:13.133 INFO:tasks.workunit.client.1.smithi196.stdout:0/912: mkdir d3/de/d14/ded/d13b 0 2024-04-22T15:19:13.134 INFO:tasks.workunit.client.1.smithi196.stdout:5/903: write d1/d14/d3e/fd4 [660282,35063] 0 2024-04-22T15:19:13.136 INFO:tasks.workunit.client.1.smithi196.stdout:8/861: unlink d2/d6a/d7b/f94 0 2024-04-22T15:19:13.138 INFO:tasks.workunit.client.1.smithi196.stdout:6/889: mkdir dd/d2b/d8f/d107 0 2024-04-22T15:19:13.138 INFO:tasks.workunit.client.1.smithi196.stdout:1/905: dread - d1/db/d16/fd1 zero size 2024-04-22T15:19:13.138 INFO:tasks.workunit.client.1.smithi196.stdout:3/972: unlink d0/d15/da7/d4a/f86 0 2024-04-22T15:19:13.139 INFO:tasks.workunit.client.1.smithi196.stdout:5/904: symlink d1/d17/d50/l11d 0 2024-04-22T15:19:13.140 INFO:tasks.workunit.client.1.smithi196.stdout:1/906: dread - d1/db/d16/fd1 zero size 2024-04-22T15:19:13.140 INFO:tasks.workunit.client.1.smithi196.stdout:8/862: readlink d2/d3f/d4b/la1 0 2024-04-22T15:19:13.141 INFO:tasks.workunit.client.1.smithi196.stdout:2/903: truncate d1/d23/d27/d2e/d55/d5b/d7c/f91 375069 0 2024-04-22T15:19:13.141 INFO:tasks.workunit.client.1.smithi196.stdout:5/905: read d1/d17/f21 [2543462,42104] 0 2024-04-22T15:19:13.142 INFO:tasks.workunit.client.1.smithi196.stdout:9/929: write db/d11/f6a [9209630,129026] 0 2024-04-22T15:19:13.143 INFO:tasks.workunit.client.1.smithi196.stdout:8/863: write d2/fe3 [505427,70139] 0 2024-04-22T15:19:13.144 INFO:tasks.workunit.client.1.smithi196.stdout:0/913: symlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/d137/l13c 0 2024-04-22T15:19:13.144 INFO:tasks.workunit.client.1.smithi196.stdout:3/973: creat d0/d15/d31/d55/dcf/df5/f147 x:0 0 0 2024-04-22T15:19:13.146 INFO:tasks.workunit.client.1.smithi196.stdout:5/906: fdatasync d1/d14/d3e/f44 0 2024-04-22T15:19:13.146 INFO:tasks.workunit.client.1.smithi196.stdout:6/890: rename dd/d1f/d4e/d53/d55/d6d/dd1/dca/ddd to dd/d1f/d38/d108 0 2024-04-22T15:19:13.147 INFO:tasks.workunit.client.1.smithi196.stdout:8/864: write d2/d8/fe8 [275130,89813] 0 2024-04-22T15:19:13.149 INFO:tasks.workunit.client.1.smithi196.stdout:9/930: fsync db/dd/de/d15/d31/d64/dab/fba 0 2024-04-22T15:19:13.149 INFO:tasks.workunit.client.1.smithi196.stdout:6/891: readlink dd/d1f/d4e/d53/d7f/dea/lf8 0 2024-04-22T15:19:13.150 INFO:tasks.workunit.client.1.smithi196.stdout:5/907: mkdir d1/df1/d6e/d7a/d11e 0 2024-04-22T15:19:13.150 INFO:tasks.workunit.client.1.smithi196.stdout:3/974: write d0/d15/d31/d55/dcf/d6f/db5/dd7/fda [3954850,67603] 0 2024-04-22T15:19:13.153 INFO:tasks.workunit.client.1.smithi196.stdout:5/908: dread - d1/d17/d1a/d2f/d65/ff6 zero size 2024-04-22T15:19:13.155 INFO:tasks.workunit.client.1.smithi196.stdout:6/892: rmdir dd/d1f/d4e/d53/d7f/dea 39 2024-04-22T15:19:13.157 INFO:tasks.workunit.client.1.smithi196.stdout:1/907: link d1/db/d16/d1e/d49/d90/fb4 d1/db/dc/de/d96/d9d/f124 0 2024-04-22T15:19:13.157 INFO:tasks.workunit.client.1.smithi196.stdout:4/873: dread d6/d13/d15/d2c/d62/f99 [0,4194304] 0 2024-04-22T15:19:13.158 INFO:tasks.workunit.client.1.smithi196.stdout:9/931: creat db/dd/dd0/f137 x:0 0 0 2024-04-22T15:19:13.161 INFO:tasks.workunit.client.1.smithi196.stdout:8/865: dwrite d2/d5/d1a/db7/dd9/d68/dbc/fed [0,4194304] 0 2024-04-22T15:19:13.162 INFO:tasks.workunit.client.1.smithi196.stdout:6/893: creat dd/d20/d5c/f109 x:0 0 0 2024-04-22T15:19:13.162 INFO:tasks.workunit.client.1.smithi196.stdout:5/909: mkdir d1/d2/d98/d11f 0 2024-04-22T15:19:13.166 INFO:tasks.workunit.client.1.smithi196.stdout:6/894: mkdir dd/d70/d10a 0 2024-04-22T15:19:13.169 INFO:tasks.workunit.client.1.smithi196.stdout:5/910: write d1/d17/d7f/fe0 [1900959,105173] 0 2024-04-22T15:19:13.171 INFO:tasks.workunit.client.1.smithi196.stdout:7/900: dwrite d14/d26/d2a/d2b/f77 [0,4194304] 0 2024-04-22T15:19:13.172 INFO:tasks.workunit.client.1.smithi196.stdout:6/895: chown dd/d1f/d4e/d53/d55/d6d/dd1/de3 97911 1 2024-04-22T15:19:13.175 INFO:tasks.workunit.client.1.smithi196.stdout:9/932: dwrite db/dd/de/d15/d31/f32 [4194304,4194304] 0 2024-04-22T15:19:13.177 INFO:tasks.workunit.client.1.smithi196.stdout:1/908: link d1/db/d16/l8a d1/db/dc/de/d43/d5e/d7a/l125 0 2024-04-22T15:19:13.180 INFO:tasks.workunit.client.1.smithi196.stdout:3/975: write d0/d15/d31/d55/d5d/fa0 [562581,117028] 0 2024-04-22T15:19:13.183 INFO:tasks.workunit.client.1.smithi196.stdout:7/901: mkdir d14/d15/d91/d9a/d130 0 2024-04-22T15:19:13.183 INFO:tasks.workunit.client.1.smithi196.stdout:1/909: chown d1/db/d97/dd7/ldf 310 1 2024-04-22T15:19:13.184 INFO:tasks.workunit.client.1.smithi196.stdout:0/914: truncate d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/fc0 2756845 0 2024-04-22T15:19:13.191 INFO:tasks.workunit.client.1.smithi196.stdout:9/933: rename db/dd/de/dd7/def/d57/fa2 to db/dd/d1b/d2c/f138 0 2024-04-22T15:19:13.192 INFO:tasks.workunit.client.1.smithi196.stdout:5/911: dwrite d1/d2/d98/dac/d105/f11 [0,4194304] 0 2024-04-22T15:19:13.196 INFO:tasks.workunit.client.1.smithi196.stdout:8/866: truncate d2/dc/fe 3905708 0 2024-04-22T15:19:13.200 INFO:tasks.workunit.client.1.smithi196.stdout:6/896: dwrite dd/d1f/d4e/d53/d7f/dea/feb [4194304,4194304] 0 2024-04-22T15:19:13.203 INFO:tasks.workunit.client.1.smithi196.stdout:4/874: dwrite d6/d14/d18/d26/d32/da3/db7/dc9/fda [8388608,4194304] 0 2024-04-22T15:19:13.203 INFO:tasks.workunit.client.1.smithi196.stdout:7/902: dwrite d14/d26/d2a/d2b/f2d [4194304,4194304] 0 2024-04-22T15:19:13.222 INFO:tasks.workunit.client.1.smithi196.stdout:0/915: chown d3/d9/d36/cc3 202422803 1 2024-04-22T15:19:13.223 INFO:tasks.workunit.client.1.smithi196.stdout:3/976: creat d0/d15/da7/d4a/d44/d8c/f148 x:0 0 0 2024-04-22T15:19:13.224 INFO:tasks.workunit.client.1.smithi196.stdout:3/977: fdatasync d0/d15/da7/d4a/d44/d8c/fbc 0 2024-04-22T15:19:13.225 INFO:tasks.workunit.client.1.smithi196.stdout:1/910: creat d1/db/dc/d26/d2e/f126 x:0 0 0 2024-04-22T15:19:13.227 INFO:tasks.workunit.client.1.smithi196.stdout:8/867: rename d2/d5/d1a/dcc to d2/d5/d1a/db7/dfe 0 2024-04-22T15:19:13.232 INFO:tasks.workunit.client.1.smithi196.stdout:3/978: mkdir d0/d15/da7/d4a/d44/d8c/de9/d149 0 2024-04-22T15:19:13.233 INFO:tasks.workunit.client.1.smithi196.stdout:6/897: fsync dd/d1f/d38/d40/f49 0 2024-04-22T15:19:13.234 INFO:tasks.workunit.client.1.smithi196.stdout:4/875: mkdir d6/d13/d15/d2c/d62/dd6/d11b/d11e 0 2024-04-22T15:19:13.235 INFO:tasks.workunit.client.1.smithi196.stdout:0/916: symlink d3/d6/l13d 0 2024-04-22T15:19:13.237 INFO:tasks.workunit.client.1.smithi196.stdout:1/911: mkdir d1/db/d16/d6d/de6/dfc/d127 0 2024-04-22T15:19:13.241 INFO:tasks.workunit.client.1.smithi196.stdout:3/979: dwrite d0/d15/d31/d55/dcf/d6f/de5/f12d [0,4194304] 0 2024-04-22T15:19:13.242 INFO:tasks.workunit.client.1.smithi196.stdout:3/980: chown d0/d15/da7/d4a/ld1 112383 1 2024-04-22T15:19:13.243 INFO:tasks.workunit.client.1.smithi196.stdout:0/917: read d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/fbf [1826255,114983] 0 2024-04-22T15:19:13.243 INFO:tasks.workunit.client.1.smithi196.stdout:8/868: creat d2/d8/d49/fff x:0 0 0 2024-04-22T15:19:13.245 INFO:tasks.workunit.client.1.smithi196.stdout:9/934: write db/dd/de/fe9 [910096,14319] 0 2024-04-22T15:19:13.247 INFO:tasks.workunit.client.1.smithi196.stdout:1/912: mkdir d1/db/d16/d1e/d65/d128 0 2024-04-22T15:19:13.248 INFO:tasks.workunit.client.1.smithi196.stdout:5/912: write d1/d4/f9c [1476439,8241] 0 2024-04-22T15:19:13.250 INFO:tasks.workunit.client.1.smithi196.stdout:9/935: symlink db/dd/de/dd7/def/d7a/d116/l139 0 2024-04-22T15:19:13.251 INFO:tasks.workunit.client.1.smithi196.stdout:6/898: dread dd/f12 [0,4194304] 0 2024-04-22T15:19:13.252 INFO:tasks.workunit.client.1.smithi196.stdout:3/981: creat d0/d15/d31/d55/dcf/d8/d70/f14a x:0 0 0 2024-04-22T15:19:13.253 INFO:tasks.workunit.client.1.smithi196.stdout:5/913: read d1/d4/f26 [4447211,115887] 0 2024-04-22T15:19:13.256 INFO:tasks.workunit.client.1.smithi196.stdout:8/869: creat d2/d5/d1a/db7/dd9/f100 x:0 0 0 2024-04-22T15:19:13.258 INFO:tasks.workunit.client.1.smithi196.stdout:4/876: dwrite d6/d14/d18/d26/d32/d67/fc8 [0,4194304] 0 2024-04-22T15:19:13.259 INFO:tasks.workunit.client.1.smithi196.stdout:6/899: creat dd/d1f/d38/d108/f10b x:0 0 0 2024-04-22T15:19:13.261 INFO:tasks.workunit.client.1.smithi196.stdout:6/900: chown dd/lb0 4993342 1 2024-04-22T15:19:13.262 INFO:tasks.workunit.client.1.smithi196.stdout:6/901: write dd/d1f/d4e/d53/fa4 [3676895,75812] 0 2024-04-22T15:19:13.263 INFO:tasks.workunit.client.1.smithi196.stdout:3/982: creat d0/d15/d31/d55/d5d/f14b x:0 0 0 2024-04-22T15:19:13.263 INFO:tasks.workunit.client.1.smithi196.stdout:9/936: dwrite db/dd/de/d15/d31/fcf [0,4194304] 0 2024-04-22T15:19:13.275 INFO:tasks.workunit.client.1.smithi196.stdout:1/913: dwrite d1/db/dc/de/d43/f68 [0,4194304] 0 2024-04-22T15:19:13.279 INFO:tasks.workunit.client.1.smithi196.stdout:9/937: creat db/dd/de/d15/d31/d64/dab/f13a x:0 0 0 2024-04-22T15:19:13.281 INFO:tasks.workunit.client.1.smithi196.stdout:8/870: mkdir d2/d3f/d101 0 2024-04-22T15:19:13.282 INFO:tasks.workunit.client.1.smithi196.stdout:0/918: link d3/de/d14/d5a/f5f d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8/f13e 0 2024-04-22T15:19:13.284 INFO:tasks.workunit.client.1.smithi196.stdout:1/914: creat d1/db/dc/de/d43/d5e/d7a/f129 x:0 0 0 2024-04-22T15:19:13.286 INFO:tasks.workunit.client.1.smithi196.stdout:3/983: getdents d0/d15/d31/d55/dcf/d6f/d10a/d5e/d115/d11b 0 2024-04-22T15:19:13.288 INFO:tasks.workunit.client.1.smithi196.stdout:9/938: dwrite db/dd/de/dd7/def/d7a/d7b/d81/f11d [0,4194304] 0 2024-04-22T15:19:13.288 INFO:tasks.workunit.client.1.smithi196.stdout:1/915: write d1/db/d16/d1e/d46/f105 [567362,20013] 0 2024-04-22T15:19:13.290 INFO:tasks.workunit.client.1.smithi196.stdout:0/919: mknod d3/d109/c13f 0 2024-04-22T15:19:13.290 INFO:tasks.workunit.client.1.smithi196.stdout:3/984: chown d0/d15/f92 133908149 1 2024-04-22T15:19:13.295 INFO:tasks.workunit.client.1.smithi196.stdout:6/902: creat dd/d1f/d4e/d53/d55/d6d/f10c x:0 0 0 2024-04-22T15:19:13.295 INFO:tasks.workunit.client.1.smithi196.stdout:6/903: chown dd/d2b/d8f/d107 6 1 2024-04-22T15:19:13.301 INFO:tasks.workunit.client.1.smithi196.stdout:5/914: dwrite d1/d17/d50/fcd [0,4194304] 0 2024-04-22T15:19:13.307 INFO:tasks.workunit.client.1.smithi196.stdout:0/920: creat d3/d6/dc/d8c/d8d/d98/dbd/f140 x:0 0 0 2024-04-22T15:19:13.309 INFO:tasks.workunit.client.1.smithi196.stdout:6/904: creat dd/d70/f10d x:0 0 0 2024-04-22T15:19:13.312 INFO:tasks.workunit.client.1.smithi196.stdout:1/916: truncate d1/db/dc/de/d96/d9d/f124 969415 0 2024-04-22T15:19:13.314 INFO:tasks.workunit.client.1.smithi196.stdout:5/915: dread d1/d2/d98/dac/d105/f38 [0,4194304] 0 2024-04-22T15:19:13.315 INFO:tasks.workunit.client.1.smithi196.stdout:3/985: symlink d0/d15/d31/d55/dcf/d6f/d10a/l14c 0 2024-04-22T15:19:13.316 INFO:tasks.workunit.client.1.smithi196.stdout:3/986: fsync d0/d15/d31/d55/dcf/d6f/d10a/d5e/f63 0 2024-04-22T15:19:13.317 INFO:tasks.workunit.client.1.smithi196.stdout:3/987: write d0/d15/da7/d18/f4c [2840977,102154] 0 2024-04-22T15:19:13.319 INFO:tasks.workunit.client.1.smithi196.stdout:3/988: dread - d0/d15/d31/d55/f135 zero size 2024-04-22T15:19:13.320 INFO:tasks.workunit.client.1.smithi196.stdout:1/917: creat d1/db/d16/d44/f12a x:0 0 0 2024-04-22T15:19:13.321 INFO:tasks.workunit.client.1.smithi196.stdout:2/904: sync 2024-04-22T15:19:13.322 INFO:tasks.workunit.client.1.smithi196.stdout:9/939: rename db/dd/de/d15/ff0 to db/dd/f13b 0 2024-04-22T15:19:13.326 INFO:tasks.workunit.client.1.smithi196.stdout:3/989: symlink d0/d15/d31/d55/dcf/d6f/de5/d101/d9e/l14d 0 2024-04-22T15:19:13.327 INFO:tasks.workunit.client.1.smithi196.stdout:1/918: mkdir d1/db/dc/de/d5a/d5b/d6e/d12b 0 2024-04-22T15:19:13.327 INFO:tasks.workunit.client.1.smithi196.stdout:2/905: truncate d1/ffc 268797 0 2024-04-22T15:19:13.330 INFO:tasks.workunit.client.1.smithi196.stdout:8/871: dwrite d2/d5/faa [0,4194304] 0 2024-04-22T15:19:13.331 INFO:tasks.workunit.client.1.smithi196.stdout:1/919: write d1/db/d16/d44/f92 [9246034,107056] 0 2024-04-22T15:19:13.331 INFO:tasks.workunit.client.1.smithi196.stdout:9/940: chown db/dd/de/d15/d31/df6/f10c 265 1 2024-04-22T15:19:13.335 INFO:tasks.workunit.client.1.smithi196.stdout:8/872: symlink d2/d6a/d7b/l102 0 2024-04-22T15:19:13.336 INFO:tasks.workunit.client.1.smithi196.stdout:1/920: creat d1/db/d16/d8e/f12c x:0 0 0 2024-04-22T15:19:13.337 INFO:tasks.workunit.client.1.smithi196.stdout:3/990: dwrite d0/d15/d31/d55/d6e/fa1 [0,4194304] 0 2024-04-22T15:19:13.340 INFO:tasks.workunit.client.1.smithi196.stdout:1/921: dread - d1/db/d16/d1e/d65/f11c zero size 2024-04-22T15:19:13.340 INFO:tasks.workunit.client.1.smithi196.stdout:2/906: rename d1/d23/d27/d2e/d55/d5b/d7c/dbd to d1/d1d/d145 0 2024-04-22T15:19:13.341 INFO:tasks.workunit.client.1.smithi196.stdout:6/905: write dd/f3a [1502211,44155] 0 2024-04-22T15:19:13.342 INFO:tasks.workunit.client.1.smithi196.stdout:8/873: stat d2/d8/d49/d8b/f99 0 2024-04-22T15:19:13.345 INFO:tasks.workunit.client.1.smithi196.stdout:2/907: chown d1/d13/dbc/lf4 62761439 1 2024-04-22T15:19:13.346 INFO:tasks.workunit.client.1.smithi196.stdout:3/991: rename d0/d15/da7/d4a/d44/f6d to d0/d15/d31/dd0/de2/d13b/f14e 0 2024-04-22T15:19:13.351 INFO:tasks.workunit.client.1.smithi196.stdout:0/921: dwrite d3/de/d14/d5a/f68 [0,4194304] 0 2024-04-22T15:19:13.351 INFO:tasks.workunit.client.1.smithi196.stdout:9/941: dread db/dd/de/dd7/def/d57/f1f [0,4194304] 0 2024-04-22T15:19:13.353 INFO:tasks.workunit.client.1.smithi196.stdout:8/874: mkdir d2/dc/d103 0 2024-04-22T15:19:13.355 INFO:tasks.workunit.client.1.smithi196.stdout:5/916: dwrite d1/d17/d1c/f3b [0,4194304] 0 2024-04-22T15:19:13.356 INFO:tasks.workunit.client.1.smithi196.stdout:4/877: sync 2024-04-22T15:19:13.356 INFO:tasks.workunit.client.1.smithi196.stdout:7/903: sync 2024-04-22T15:19:13.357 INFO:tasks.workunit.client.1.smithi196.stdout:8/875: fsync d2/d35/f5f 0 2024-04-22T15:19:13.357 INFO:tasks.workunit.client.1.smithi196.stdout:4/878: chown d6 2680 1 2024-04-22T15:19:13.360 INFO:tasks.workunit.client.1.smithi196.stdout:2/908: chown d1/d23/d27/d89/ccd 1 1 2024-04-22T15:19:13.360 INFO:tasks.workunit.client.1.smithi196.stdout:3/992: symlink d0/d15/d31/dcb/l14f 0 2024-04-22T15:19:13.361 INFO:tasks.workunit.client.1.smithi196.stdout:6/906: write dd/d1f/d38/d40/f4f [324580,81797] 0 2024-04-22T15:19:13.362 INFO:tasks.workunit.client.1.smithi196.stdout:4/879: fsync d6/d13/d15/d2c/fa0 0 2024-04-22T15:19:13.363 INFO:tasks.workunit.client.1.smithi196.stdout:7/904: mkdir d14/d15/d131 0 2024-04-22T15:19:13.365 INFO:tasks.workunit.client.1.smithi196.stdout:9/942: rename db/dd/de/d15/f6e to db/dd/de/d98/da9/d101/f13c 0 2024-04-22T15:19:13.366 INFO:tasks.workunit.client.1.smithi196.stdout:9/943: fdatasync db/f109 0 2024-04-22T15:19:13.366 INFO:tasks.workunit.client.1.smithi196.stdout:3/993: write d0/d15/d31/d55/dcf/d6f/de5/f10e [690843,33269] 0 2024-04-22T15:19:13.368 INFO:tasks.workunit.client.1.smithi196.stdout:0/922: link d3/de/d14/l17 d3/de/d14/d86/d124/l141 0 2024-04-22T15:19:13.368 INFO:tasks.workunit.client.1.smithi196.stdout:5/917: truncate d1/d14/d3e/f44 1792222 0 2024-04-22T15:19:13.369 INFO:tasks.workunit.client.1.smithi196.stdout:2/909: creat d1/d1d/d122/f146 x:0 0 0 2024-04-22T15:19:13.372 INFO:tasks.workunit.client.1.smithi196.stdout:6/907: chown dd/d20/d5c/d5e/f64 23 1 2024-04-22T15:19:13.373 INFO:tasks.workunit.client.1.smithi196.stdout:3/994: dread d0/d15/d31/d55/dcf/d6f/d10a/fb7 [0,4194304] 0 2024-04-22T15:19:13.374 INFO:tasks.workunit.client.1.smithi196.stdout:4/880: getdents d6/d14/d18/d26/d32/d67/de7 0 2024-04-22T15:19:13.374 INFO:tasks.workunit.client.1.smithi196.stdout:0/923: chown d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/l9b 0 1 2024-04-22T15:19:13.374 INFO:tasks.workunit.client.1.smithi196.stdout:7/905: getdents d14/d26/d5a/dbb 0 2024-04-22T15:19:13.375 INFO:tasks.workunit.client.1.smithi196.stdout:2/910: mkdir d1/d13/d101/d147 0 2024-04-22T15:19:13.376 INFO:tasks.workunit.client.1.smithi196.stdout:5/918: creat d1/d2/f120 x:0 0 0 2024-04-22T15:19:13.376 INFO:tasks.workunit.client.1.smithi196.stdout:9/944: dwrite db/fc [0,4194304] 0 2024-04-22T15:19:13.378 INFO:tasks.workunit.client.1.smithi196.stdout:6/908: write dd/d2b/d8d/fb9 [240101,79302] 0 2024-04-22T15:19:13.381 INFO:tasks.workunit.client.1.smithi196.stdout:2/911: creat d1/d13/d101/f148 x:0 0 0 2024-04-22T15:19:13.381 INFO:tasks.workunit.client.1.smithi196.stdout:1/922: dread d1/db/d16/d1e/d49/d90/fb4 [0,4194304] 0 2024-04-22T15:19:13.383 INFO:tasks.workunit.client.1.smithi196.stdout:5/919: symlink d1/df1/d6e/d7a/l121 0 2024-04-22T15:19:13.384 INFO:tasks.workunit.client.1.smithi196.stdout:3/995: chown d0/d15/d31/dd0/c145 218 1 2024-04-22T15:19:13.389 INFO:tasks.workunit.client.1.smithi196.stdout:8/876: write d2/d5/d1a/db7/dd9/d68/dbc/f8a [255218,7686] 0 2024-04-22T15:19:13.390 INFO:tasks.workunit.client.1.smithi196.stdout:6/909: write dd/d1f/d4e/d53/d55/d57/fe9 [406430,130845] 0 2024-04-22T15:19:13.390 INFO:tasks.workunit.client.1.smithi196.stdout:7/906: link d14/d25/d31/fd5 d14/ddd/f132 0 2024-04-22T15:19:13.391 INFO:tasks.workunit.client.1.smithi196.stdout:5/920: creat d1/d17/d1c/f122 x:0 0 0 2024-04-22T15:19:13.393 INFO:tasks.workunit.client.1.smithi196.stdout:1/923: symlink d1/db/dc/d26/l12d 0 2024-04-22T15:19:13.397 INFO:tasks.workunit.client.1.smithi196.stdout:4/881: dwrite d6/d90/f60 [4194304,4194304] 0 2024-04-22T15:19:13.400 INFO:tasks.workunit.client.1.smithi196.stdout:9/945: dwrite db/dd/de/d15/d31/d64/ffa [0,4194304] 0 2024-04-22T15:19:13.401 INFO:tasks.workunit.client.1.smithi196.stdout:1/924: mkdir d1/db/dc/dad/d36/def/d12e 0 2024-04-22T15:19:13.402 INFO:tasks.workunit.client.1.smithi196.stdout:2/912: write d1/f1b [281888,72968] 0 2024-04-22T15:19:13.402 INFO:tasks.workunit.client.1.smithi196.stdout:4/882: readlink d6/d14/d18/d40/d85/la7 0 2024-04-22T15:19:13.407 INFO:tasks.workunit.client.1.smithi196.stdout:9/946: chown db/dd/de/dd7/def/f105 0 1 2024-04-22T15:19:13.409 INFO:tasks.workunit.client.1.smithi196.stdout:7/907: rmdir d14/d25/d31 39 2024-04-22T15:19:13.411 INFO:tasks.workunit.client.1.smithi196.stdout:9/947: chown db/dd/de 1499 1 2024-04-22T15:19:13.412 INFO:tasks.workunit.client.1.smithi196.stdout:3/996: truncate d0/d15/d31/d55/dcf/d8/dfc/f131 667391 0 2024-04-22T15:19:13.412 INFO:tasks.workunit.client.1.smithi196.stdout:6/910: write dd/f23 [637704,2932] 0 2024-04-22T15:19:13.414 INFO:tasks.workunit.client.1.smithi196.stdout:0/924: dread d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/f9e [0,4194304] 0 2024-04-22T15:19:13.414 INFO:tasks.workunit.client.1.smithi196.stdout:8/877: link d2/c29 d2/d5/d1a/db7/dfe/c104 0 2024-04-22T15:19:13.415 INFO:tasks.workunit.client.1.smithi196.stdout:4/883: write d6/d13/d15/d2c/d62/d78/d8b/fe3 [3277026,66036] 0 2024-04-22T15:19:13.415 INFO:tasks.workunit.client.1.smithi196.stdout:5/921: dread d1/d17/f21 [4194304,4194304] 0 2024-04-22T15:19:13.418 INFO:tasks.workunit.client.1.smithi196.stdout:6/911: chown dd/d2b/f3e 12481 1 2024-04-22T15:19:13.418 INFO:tasks.workunit.client.1.smithi196.stdout:2/913: write d1/d13/d25/ffa [562552,116869] 0 2024-04-22T15:19:13.420 INFO:tasks.workunit.client.1.smithi196.stdout:0/925: write d3/d9/d36/f11d [4819301,61818] 0 2024-04-22T15:19:13.420 INFO:tasks.workunit.client.1.smithi196.stdout:9/948: creat db/dd/de/dd7/def/d7a/d7b/f13d x:0 0 0 2024-04-22T15:19:13.422 INFO:tasks.workunit.client.1.smithi196.stdout:5/922: read d1/d17/d1c/f3b [1461781,79702] 0 2024-04-22T15:19:13.422 INFO:tasks.workunit.client.1.smithi196.stdout:8/878: creat d2/dc/f105 x:0 0 0 2024-04-22T15:19:13.422 INFO:tasks.workunit.client.1.smithi196.stdout:4/884: stat d6/d13/d15/d2c/d62/dd6/d11b 0 2024-04-22T15:19:13.423 INFO:tasks.workunit.client.1.smithi196.stdout:1/925: dwrite d1/db/d16/d1e/d46/f105 [0,4194304] 0 2024-04-22T15:19:13.423 INFO:tasks.workunit.client.0.smithi096.stdout:6/985: sync 2024-04-22T15:19:13.424 INFO:tasks.workunit.client.1.smithi196.stdout:3/997: creat d0/d15/da7/d4a/dc5/dfd/f150 x:0 0 0 2024-04-22T15:19:13.426 INFO:tasks.workunit.client.1.smithi196.stdout:6/912: dread - dd/d1f/d37/d58/f6e zero size 2024-04-22T15:19:13.427 INFO:tasks.workunit.client.0.smithi096.stdout:6/986: dread - d3/d28/dc9/fec zero size 2024-04-22T15:19:13.427 INFO:tasks.workunit.client.1.smithi196.stdout:2/914: symlink d1/d23/d27/d2e/d55/d5b/l149 0 2024-04-22T15:19:13.427 INFO:tasks.workunit.client.1.smithi196.stdout:9/949: read - db/dd/de/d98/da9/d101/f119 zero size 2024-04-22T15:19:13.428 INFO:tasks.workunit.client.0.smithi096.stdout:6/987: write d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/ddd/f129 [728957,63721] 0 2024-04-22T15:19:13.429 INFO:tasks.workunit.client.1.smithi196.stdout:0/926: creat d3/d9/d36/d9f/dd1/f142 x:0 0 0 2024-04-22T15:19:13.429 INFO:tasks.workunit.client.1.smithi196.stdout:7/908: dread d14/f33 [0,4194304] 0 2024-04-22T15:19:13.430 INFO:tasks.workunit.client.1.smithi196.stdout:1/926: creat d1/db/d16/d1e/d49/d90/f12f x:0 0 0 2024-04-22T15:19:13.431 INFO:tasks.workunit.client.1.smithi196.stdout:8/879: chown d2/dc/fd 1 1 2024-04-22T15:19:13.432 INFO:tasks.workunit.client.1.smithi196.stdout:5/923: rename d1/df1/d7d to d1/d2/d98/dac/d123 0 2024-04-22T15:19:13.433 INFO:tasks.workunit.client.1.smithi196.stdout:8/880: dread - d2/d8/fd0 zero size 2024-04-22T15:19:13.434 INFO:tasks.workunit.client.1.smithi196.stdout:3/998: fsync d0/d15/d31/d55/dcf/d8/fd4 0 2024-04-22T15:19:13.434 INFO:tasks.workunit.client.1.smithi196.stdout:6/913: symlink dd/d1f/d38/d40/dd3/l10e 0 2024-04-22T15:19:13.435 INFO:tasks.workunit.client.1.smithi196.stdout:4/885: dread d6/d13/d15/d2c/d62/f99 [0,4194304] 0 2024-04-22T15:19:13.438 INFO:tasks.workunit.client.0.smithi096.stdout:6/988: getdents d3/d3a 0 2024-04-22T15:19:13.440 INFO:tasks.workunit.client.1.smithi196.stdout:0/927: rename d3/d6/f48 to d3/de/d1c/dd4/f143 0 2024-04-22T15:19:13.441 INFO:tasks.workunit.client.1.smithi196.stdout:5/924: mkdir d1/d2/d98/dac/d124 0 2024-04-22T15:19:13.442 INFO:tasks.workunit.client.0.smithi096.stdout:6/989: rmdir d3/d28/d4d/d82/d9e/da3 39 2024-04-22T15:19:13.443 INFO:tasks.workunit.client.0.smithi096.stdout:6/990: fdatasync d3/d3a/d75/f113 0 2024-04-22T15:19:13.449 INFO:tasks.workunit.client.1.smithi196.stdout:3/999: write d0/d15/d31/d55/dcf/f7d [5250497,30404] 0 2024-04-22T15:19:13.449 INFO:tasks.workunit.client.1.smithi196.stdout:4/886: creat d6/d10/d105/f11f x:0 0 0 2024-04-22T15:19:13.451 INFO:tasks.workunit.client.1.smithi196.stdout:9/950: write db/dd/de/dd7/def/d7a/d7b/d81/fd3 [648134,61868] 0 2024-04-22T15:19:13.454 INFO:tasks.workunit.client.1.smithi196.stdout:4/887: truncate d6/d14/d18/d26/d32/d53/ff2 1031950 0 2024-04-22T15:19:13.455 INFO:tasks.workunit.client.0.smithi096.stdout:6/991: getdents d3/d28/d4d/d82/d9e/d137 0 2024-04-22T15:19:13.456 INFO:tasks.workunit.client.1.smithi196.stdout:1/927: dread d1/db/f29 [0,4194304] 0 2024-04-22T15:19:13.456 INFO:tasks.workunit.client.1.smithi196.stdout:5/925: mkdir d1/df1/d5e/dcb/d125 0 2024-04-22T15:19:13.456 INFO:tasks.workunit.client.1.smithi196.stdout:4/888: read d6/d10/f63 [1124686,89132] 0 2024-04-22T15:19:13.457 INFO:tasks.workunit.client.1.smithi196.stdout:6/914: dread dd/f3a [0,4194304] 0 2024-04-22T15:19:13.458 INFO:tasks.workunit.client.1.smithi196.stdout:0/928: write d3/db4/fe9 [649553,75441] 0 2024-04-22T15:19:13.459 INFO:tasks.workunit.client.1.smithi196.stdout:2/915: rmdir d1/d23/d27/d2e/d55/d5b/d93 39 2024-04-22T15:19:13.459 INFO:tasks.workunit.client.1.smithi196.stdout:7/909: dread d14/d25/d31/f38 [0,4194304] 0 2024-04-22T15:19:13.460 INFO:tasks.workunit.client.1.smithi196.stdout:2/916: chown d1/d76 100968015 1 2024-04-22T15:19:13.461 INFO:tasks.workunit.client.1.smithi196.stdout:5/926: chown d1/d4/f9d 10575460 1 2024-04-22T15:19:13.462 INFO:tasks.workunit.client.1.smithi196.stdout:1/928: creat d1/db/d16/d8e/f130 x:0 0 0 2024-04-22T15:19:13.463 INFO:tasks.workunit.client.1.smithi196.stdout:6/915: mknod dd/d1f/d38/d40/dd3/c10f 0 2024-04-22T15:19:13.465 INFO:tasks.workunit.client.1.smithi196.stdout:4/889: creat d6/d13/d15/d2c/d62/d78/d8d/f120 x:0 0 0 2024-04-22T15:19:13.465 INFO:tasks.workunit.client.0.smithi096.stdout:6/992: dwrite d3/d28/d4d/d82/dd8/fe8 [0,4194304] 0 2024-04-22T15:19:13.468 INFO:tasks.workunit.client.0.smithi096.stdout:6/993: write d3/d28/d4d/d82/d9e/d137/d11/d2c/d6f/dd4/ddd/de9/fb8 [3061499,111064] 0 2024-04-22T15:19:13.468 INFO:tasks.workunit.client.0.smithi096.stdout:6/994: stat d3/d28/d54/d68/d6e/cdb 0 2024-04-22T15:19:13.469 INFO:tasks.workunit.client.1.smithi196.stdout:5/927: readlink d1/d4/d92/l117 0 2024-04-22T15:19:13.472 INFO:tasks.workunit.client.0.smithi096.stdout:6/995: rmdir d3/d28/d54/d68/d6e/d10d 39 2024-04-22T15:19:13.472 INFO:tasks.workunit.client.1.smithi196.stdout:9/951: creat db/dd/f13e x:0 0 0 2024-04-22T15:19:13.476 INFO:tasks.workunit.client.1.smithi196.stdout:1/929: dwrite d1/db/dc/de/d43/f6b [4194304,4194304] 0 2024-04-22T15:19:13.477 INFO:tasks.workunit.client.1.smithi196.stdout:6/916: creat dd/d2b/d8d/d102/f110 x:0 0 0 2024-04-22T15:19:13.479 INFO:tasks.workunit.client.1.smithi196.stdout:5/928: mkdir d1/df1/d6e/d7a/d11e/d126 0 2024-04-22T15:19:13.483 INFO:tasks.workunit.client.1.smithi196.stdout:9/952: rename db/dd/de/ff2 to db/dd/de/d98/d12f/f13f 0 2024-04-22T15:19:13.484 INFO:tasks.workunit.client.1.smithi196.stdout:0/929: dread d3/d6/dc/d8c/d8d/fc1 [0,4194304] 0 2024-04-22T15:19:13.486 INFO:tasks.workunit.client.1.smithi196.stdout:5/929: chown d1/d14/d3e/fd1 141377934 1 2024-04-22T15:19:13.486 INFO:tasks.workunit.client.1.smithi196.stdout:1/930: write d1/db/dc/de/d5a/d5b/f79 [1359914,107780] 0 2024-04-22T15:19:13.487 INFO:tasks.workunit.client.1.smithi196.stdout:2/917: truncate d1/f7e 3443759 0 2024-04-22T15:19:13.487 INFO:tasks.workunit.client.0.smithi096.stdout:6/996: write d3/d28/d4d/d82/d9e/d137/d11/d2c/f56 [173809,46243] 0 2024-04-22T15:19:13.490 INFO:tasks.workunit.client.1.smithi196.stdout:7/910: dwrite d14/d26/d5a/d5f/fc1 [0,4194304] 0 2024-04-22T15:19:13.490 INFO:tasks.workunit.client.1.smithi196.stdout:4/890: dwrite d6/d13/d15/d2c/d62/d78/d8b/fd2 [0,4194304] 0 2024-04-22T15:19:13.491 INFO:tasks.workunit.client.1.smithi196.stdout:0/930: symlink d3/de/d14/l144 0 2024-04-22T15:19:13.491 INFO:tasks.workunit.client.0.smithi096.stdout:6/997: truncate d3/d3a/d75/d83/df7/f118 5797169 0 2024-04-22T15:19:13.492 INFO:tasks.workunit.client.0.smithi096.stdout:6/998: fsync d3/d28/d4d/d82/d9e/d137/d11/f142 0 2024-04-22T15:19:13.494 INFO:tasks.workunit.client.1.smithi196.stdout:2/918: stat d1/d13/d25/d2a/ddb/dff/l114 0 2024-04-22T15:19:13.494 INFO:tasks.workunit.client.1.smithi196.stdout:4/891: dread - d6/d14/d18/d26/d32/d53/ffd zero size 2024-04-22T15:19:13.497 INFO:tasks.workunit.client.1.smithi196.stdout:1/931: readlink d1/db/d16/l8a 0 2024-04-22T15:19:13.497 INFO:tasks.workunit.client.1.smithi196.stdout:5/930: creat d1/d78/dcf/dff/d101/f127 x:0 0 0 2024-04-22T15:19:13.498 INFO:tasks.workunit.client.1.smithi196.stdout:2/919: dread - d1/d23/d27/d89/fb0 zero size 2024-04-22T15:19:13.499 INFO:tasks.workunit.client.1.smithi196.stdout:4/892: chown d6/d14/d18/le4 185 1 2024-04-22T15:19:13.501 INFO:tasks.workunit.client.1.smithi196.stdout:5/931: stat d1/d2/d98/dac 0 2024-04-22T15:19:13.503 INFO:tasks.workunit.client.1.smithi196.stdout:4/893: stat d6/d14/d18/d26/d32/l76 0 2024-04-22T15:19:13.504 INFO:tasks.workunit.client.1.smithi196.stdout:2/920: symlink d1/d13/d1f/l14a 0 2024-04-22T15:19:13.506 INFO:tasks.workunit.client.1.smithi196.stdout:4/894: readlink d6/d90/l5f 0 2024-04-22T15:19:13.508 INFO:tasks.workunit.client.1.smithi196.stdout:5/932: mknod d1/d78/dcf/dff/d101/c128 0 2024-04-22T15:19:13.508 INFO:tasks.workunit.client.1.smithi196.stdout:2/921: dread - d1/d23/f10a zero size 2024-04-22T15:19:13.510 INFO:tasks.workunit.client.1.smithi196.stdout:0/931: write d3/fd [5484748,26253] 0 2024-04-22T15:19:13.510 INFO:tasks.workunit.client.1.smithi196.stdout:7/911: dwrite d14/d26/d2a/d2b/f101 [0,4194304] 0 2024-04-22T15:19:13.510 INFO:tasks.workunit.client.1.smithi196.stdout:6/917: dwrite dd/f11 [0,4194304] 0 2024-04-22T15:19:13.517 INFO:tasks.workunit.client.1.smithi196.stdout:5/933: dread - d1/d17/d1c/f118 zero size 2024-04-22T15:19:13.519 INFO:tasks.workunit.client.1.smithi196.stdout:0/932: dread - d3/de/f11a zero size 2024-04-22T15:19:13.520 INFO:tasks.workunit.client.1.smithi196.stdout:1/932: write d1/db/dc/de/f14 [1111396,106455] 0 2024-04-22T15:19:13.521 INFO:tasks.workunit.client.1.smithi196.stdout:4/895: creat d6/d14/d18/d26/d32/d53/f121 x:0 0 0 2024-04-22T15:19:13.523 INFO:tasks.workunit.client.1.smithi196.stdout:9/953: dwrite db/dd/de/dd7/def/f92 [4194304,4194304] 0 2024-04-22T15:19:13.527 INFO:tasks.workunit.client.1.smithi196.stdout:5/934: truncate d1/df1/d5e/f63 1428274 0 2024-04-22T15:19:13.528 INFO:tasks.workunit.client.1.smithi196.stdout:6/918: link dd/d20/d5c/d5e/d76/l78 dd/d1f/d4e/d53/l111 0 2024-04-22T15:19:13.528 INFO:tasks.workunit.client.1.smithi196.stdout:9/954: readlink db/dd/de/dd7/def/d7a/d7b/d81/la4 0 2024-04-22T15:19:13.529 INFO:tasks.workunit.client.0.smithi096.stdout:6/999: dread d3/d28/f61 [0,4194304] 0 2024-04-22T15:19:13.529 INFO:tasks.workunit.client.1.smithi196.stdout:1/933: chown d1/db/dc/de/d5a/d5b/d69/fca 255983 1 2024-04-22T15:19:13.530 INFO:tasks.workunit.client.1.smithi196.stdout:7/912: dwrite d14/d26/d2a/d72/db1/fec [0,4194304] 0 2024-04-22T15:19:13.534 INFO:tasks.workunit.client.1.smithi196.stdout:1/934: read d1/db/d16/d1e/d46/f105 [2404932,81153] 0 2024-04-22T15:19:13.534 INFO:tasks.workunit.client.1.smithi196.stdout:9/955: read - db/dd/dd0/f137 zero size 2024-04-22T15:19:13.535 INFO:tasks.workunit.client.0.smithi096.stderr:+ rm -rf -- ./tmp.jlIWo8x58o 2024-04-22T15:19:13.535 INFO:tasks.workunit.client.1.smithi196.stdout:6/919: creat dd/d2b/d8d/f112 x:0 0 0 2024-04-22T15:19:13.535 INFO:tasks.workunit.client.1.smithi196.stdout:4/896: dread d6/d14/d18/d40/f65 [0,4194304] 0 2024-04-22T15:19:13.537 INFO:tasks.workunit.client.1.smithi196.stdout:2/922: dwrite d1/d13/d25/f4f [0,4194304] 0 2024-04-22T15:19:13.540 INFO:tasks.workunit.client.1.smithi196.stdout:4/897: fdatasync d6/d14/d18/d40/d85/fe5 0 2024-04-22T15:19:13.541 INFO:tasks.workunit.client.1.smithi196.stdout:2/923: fsync d1/fd 0 2024-04-22T15:19:13.542 INFO:tasks.workunit.client.1.smithi196.stdout:1/935: mknod d1/db/dc/d26/d2e/c131 0 2024-04-22T15:19:13.549 INFO:tasks.workunit.client.1.smithi196.stdout:9/956: rmdir db/dd/de/dd7/def/d7a/d7b/d81/d136 0 2024-04-22T15:19:13.549 INFO:tasks.workunit.client.1.smithi196.stdout:6/920: dwrite dd/d2b/d8d/d102/f110 [0,4194304] 0 2024-04-22T15:19:13.552 INFO:tasks.workunit.client.1.smithi196.stdout:6/921: chown dd/d1f/d4e/d53/d7f/dea 1128902981 1 2024-04-22T15:19:13.554 INFO:tasks.workunit.client.1.smithi196.stdout:2/924: chown d1/d23/d27/d2e/d55/d5b/c82 32079164 1 2024-04-22T15:19:13.557 INFO:tasks.workunit.client.1.smithi196.stdout:2/925: fdatasync d1/d23/f2f 0 2024-04-22T15:19:13.562 INFO:tasks.workunit.client.1.smithi196.stdout:9/957: dwrite db/dd/de/d15/f1e [4194304,4194304] 0 2024-04-22T15:19:13.562 INFO:tasks.workunit.client.1.smithi196.stdout:2/926: write d1/d13/d25/d9f/f118 [4723252,124711] 0 2024-04-22T15:19:13.568 INFO:tasks.workunit.client.1.smithi196.stdout:6/922: dwrite f9 [4194304,4194304] 0 2024-04-22T15:19:13.571 INFO:tasks.workunit.client.1.smithi196.stdout:9/958: symlink db/dd/de/dd7/def/d57/l140 0 2024-04-22T15:19:13.577 INFO:tasks.workunit.client.1.smithi196.stdout:7/913: dwrite d14/d15/f74 [4194304,4194304] 0 2024-04-22T15:19:13.577 INFO:tasks.workunit.client.1.smithi196.stdout:6/923: mkdir dd/d1f/d37/d113 0 2024-04-22T15:19:13.582 INFO:tasks.workunit.client.1.smithi196.stdout:1/936: dread d1/db/dc/d26/f30 [0,4194304] 0 2024-04-22T15:19:13.585 INFO:tasks.workunit.client.1.smithi196.stdout:7/914: mknod d14/c133 0 2024-04-22T15:19:13.586 INFO:tasks.workunit.client.1.smithi196.stdout:9/959: dwrite db/dd/de/d98/f114 [0,4194304] 0 2024-04-22T15:19:13.586 INFO:tasks.workunit.client.1.smithi196.stdout:4/898: dread d6/d13/d15/f5e [0,4194304] 0 2024-04-22T15:19:13.587 INFO:tasks.workunit.client.1.smithi196.stdout:1/937: readlink d1/db/d16/d1e/d46/ddc/l120 0 2024-04-22T15:19:13.588 INFO:tasks.workunit.client.1.smithi196.stdout:6/924: mknod dd/d1f/d4e/d53/d55/c114 0 2024-04-22T15:19:13.591 INFO:tasks.workunit.client.1.smithi196.stdout:9/960: unlink db/dd/dd0/fe5 0 2024-04-22T15:19:13.592 INFO:tasks.workunit.client.1.smithi196.stdout:9/961: stat db/dd/de/dd7/def/d57/fbc 0 2024-04-22T15:19:13.593 INFO:tasks.workunit.client.1.smithi196.stdout:9/962: write db/dd/de/d98/da9/d101/f119 [883343,49966] 0 2024-04-22T15:19:13.595 INFO:tasks.workunit.client.1.smithi196.stdout:7/915: creat d14/d4d/da3/de9/f134 x:0 0 0 2024-04-22T15:19:13.595 INFO:tasks.workunit.client.1.smithi196.stdout:1/938: unlink d1/db/dc/de/d5a/c10b 0 2024-04-22T15:19:13.596 INFO:tasks.workunit.client.1.smithi196.stdout:6/925: symlink dd/d1f/d37/dcb/l115 0 2024-04-22T15:19:13.596 INFO:tasks.workunit.client.1.smithi196.stdout:9/963: write db/dd/de/dd7/def/d7a/d7b/f13d [876825,9073] 0 2024-04-22T15:19:13.596 INFO:tasks.workunit.client.1.smithi196.stdout:1/939: chown d1/db/d16/d1e/d100 24295 1 2024-04-22T15:19:13.598 INFO:tasks.workunit.client.1.smithi196.stdout:4/899: symlink d6/d13/d15/d2c/d35/l122 0 2024-04-22T15:19:13.598 INFO:tasks.workunit.client.1.smithi196.stdout:4/900: dread - d6/d13/d15/feb zero size 2024-04-22T15:19:13.599 INFO:tasks.workunit.client.1.smithi196.stdout:7/916: mknod d14/d4d/da3/de9/c135 0 2024-04-22T15:19:13.599 INFO:tasks.workunit.client.1.smithi196.stdout:4/901: dread - d6/d13/d15/d2c/d62/d78/d8d/f120 zero size 2024-04-22T15:19:13.599 INFO:tasks.workunit.client.1.smithi196.stdout:7/917: chown d14/d25/f42 25716772 1 2024-04-22T15:19:13.600 INFO:tasks.workunit.client.1.smithi196.stdout:9/964: rmdir db/dd/de/d15/d31/d64 39 2024-04-22T15:19:13.601 INFO:tasks.workunit.client.1.smithi196.stdout:7/918: write d14/d26/d2a/d2b/dc7/f124 [601182,105917] 0 2024-04-22T15:19:13.601 INFO:tasks.workunit.client.1.smithi196.stdout:1/940: symlink d1/db/dc/d26/dc4/l132 0 2024-04-22T15:19:13.604 INFO:tasks.workunit.client.1.smithi196.stdout:9/965: readlink db/dd/d1b/d2c/l8f 0 2024-04-22T15:19:13.605 INFO:tasks.workunit.client.1.smithi196.stdout:9/966: stat db/dd/de/dd7/def/d7a/fce 0 2024-04-22T15:19:13.605 INFO:tasks.workunit.client.1.smithi196.stdout:7/919: fsync d14/fa5 0 2024-04-22T15:19:13.606 INFO:tasks.workunit.client.1.smithi196.stdout:2/927: write d1/ffc [28869,113862] 0 2024-04-22T15:19:13.608 INFO:tasks.workunit.client.1.smithi196.stdout:7/920: write d14/d25/f42 [2727444,93194] 0 2024-04-22T15:19:13.609 INFO:tasks.workunit.client.1.smithi196.stdout:9/967: fdatasync db/dd/d1b/d2c/f9f 0 2024-04-22T15:19:13.616 INFO:tasks.workunit.client.1.smithi196.stdout:9/968: unlink db/dd/de/dd7/def/d7a/l85 0 2024-04-22T15:19:13.617 INFO:tasks.workunit.client.1.smithi196.stdout:2/928: dwrite d1/d23/d27/d2e/d55/d5b/d93/f130 [0,4194304] 0 2024-04-22T15:19:13.619 INFO:tasks.workunit.client.1.smithi196.stdout:8/881: sync 2024-04-22T15:19:13.627 INFO:tasks.workunit.client.1.smithi196.stdout:4/902: write d6/d13/d15/d2c/d35/f8f [631014,36744] 0 2024-04-22T15:19:13.632 INFO:tasks.workunit.client.1.smithi196.stdout:1/941: dwrite d1/db/dc/de/d43/d5e/d7a/f83 [0,4194304] 0 2024-04-22T15:19:13.634 INFO:tasks.workunit.client.1.smithi196.stdout:1/942: write d1/db/dc/de/d5a/d94/dab/fc2 [3793638,115486] 0 2024-04-22T15:19:13.637 INFO:tasks.workunit.client.1.smithi196.stdout:2/929: link d1/d13/d41/cb2 d1/d23/d27/d2e/db5/d102/c14b 0 2024-04-22T15:19:13.641 INFO:tasks.workunit.client.1.smithi196.stdout:4/903: getdents d6/d14/d18/d26/d32/d67/d9a/da5/d11d 0 2024-04-22T15:19:13.641 INFO:tasks.workunit.client.1.smithi196.stdout:4/904: stat d6/d14/d18/d26/d32/da3/db7/fef 0 2024-04-22T15:19:13.643 INFO:tasks.workunit.client.1.smithi196.stdout:9/969: link db/dd/de/dd7/ce3 db/c141 0 2024-04-22T15:19:13.692 INFO:tasks.workunit.client.1.smithi196.stdout:8/882: rename d2/f1d to d2/dc/f106 0 2024-04-22T15:19:13.692 INFO:tasks.workunit.client.1.smithi196.stdout:4/905: unlink d6/d90/df9/lc3 0 2024-04-22T15:19:13.693 INFO:tasks.workunit.client.1.smithi196.stdout:4/906: chown d6/f93 295030697 1 2024-04-22T15:19:13.694 INFO:tasks.workunit.client.1.smithi196.stdout:1/943: rmdir d1/db/d16/d6d/dfe 0 2024-04-22T15:19:13.695 INFO:tasks.workunit.client.1.smithi196.stdout:9/970: symlink db/dd/de/dd7/def/d57/d128/l142 0 2024-04-22T15:19:13.698 INFO:tasks.workunit.client.1.smithi196.stdout:8/883: unlink d2/d35/l65 0 2024-04-22T15:19:13.700 INFO:tasks.workunit.client.1.smithi196.stdout:4/907: creat d6/d14/d18/d26/d32/d67/f123 x:0 0 0 2024-04-22T15:19:13.703 INFO:tasks.workunit.client.1.smithi196.stdout:9/971: rmdir db/dd/de/d98 39 2024-04-22T15:19:13.707 INFO:tasks.workunit.client.1.smithi196.stdout:1/944: link d1/db/dc/de/d5a/d5b/d85/f119 d1/db/d16/d8e/ddd/df6/d11a/f133 0 2024-04-22T15:19:13.711 INFO:tasks.workunit.client.1.smithi196.stdout:9/972: dread db/dd/d1b/fc1 [0,4194304] 0 2024-04-22T15:19:13.712 INFO:tasks.workunit.client.1.smithi196.stdout:1/945: dwrite d1/db/dc/de/d5a/d94/dab/fd0 [0,4194304] 0 2024-04-22T15:19:13.713 INFO:tasks.workunit.client.1.smithi196.stdout:1/946: stat d1/db/d16/d1e/d100 0 2024-04-22T15:19:13.714 INFO:tasks.workunit.client.1.smithi196.stdout:1/947: stat d1/db/dc/de/d5a/d5b/d85 0 2024-04-22T15:19:13.716 INFO:tasks.workunit.client.1.smithi196.stdout:8/884: mknod d2/c107 0 2024-04-22T15:19:13.716 INFO:tasks.workunit.client.1.smithi196.stdout:8/885: chown d2/d8 0 1 2024-04-22T15:19:13.717 INFO:tasks.workunit.client.1.smithi196.stdout:0/933: sync 2024-04-22T15:19:13.727 INFO:tasks.workunit.client.1.smithi196.stdout:1/948: rename d1/db/fa8 to d1/db/dc/de/d5a/d5b/d6e/d12b/f134 0 2024-04-22T15:19:13.729 INFO:tasks.workunit.client.1.smithi196.stdout:8/886: rmdir d2/dc/d45/d98 39 2024-04-22T15:19:13.729 INFO:tasks.workunit.client.1.smithi196.stdout:1/949: write d1/db/d16/d44/f92 [643976,61358] 0 2024-04-22T15:19:13.730 INFO:tasks.workunit.client.1.smithi196.stdout:8/887: chown d2/d5/faa 64402060 1 2024-04-22T15:19:13.730 INFO:tasks.workunit.client.1.smithi196.stdout:1/950: chown d1/db/dc/dad/d36 8263 1 2024-04-22T15:19:13.731 INFO:tasks.workunit.client.1.smithi196.stdout:8/888: chown d2/d5/d1a/db7/dd9/d68/dbc/dae/fe6 0 1 2024-04-22T15:19:13.732 INFO:tasks.workunit.client.1.smithi196.stdout:8/889: write d2/d5/d1a/db7/dd9/d68/dbc/f8a [632669,127099] 0 2024-04-22T15:19:13.741 INFO:tasks.workunit.client.1.smithi196.stdout:1/951: dread d1/db/d16/d44/faa [0,4194304] 0 2024-04-22T15:19:13.742 INFO:tasks.workunit.client.1.smithi196.stdout:0/934: mknod d3/d9/d36/d9f/dd1/d65/c145 0 2024-04-22T15:19:13.743 INFO:tasks.workunit.client.1.smithi196.stdout:9/973: dwrite db/dd/de/dd7/def/f38 [0,4194304] 0 2024-04-22T15:19:13.743 INFO:tasks.workunit.client.1.smithi196.stdout:1/952: stat d1/db/d16/d1e/f60 0 2024-04-22T15:19:13.744 INFO:tasks.workunit.client.1.smithi196.stdout:1/953: truncate d1/db/d16/d6d/de6/f10d 117583 0 2024-04-22T15:19:13.752 INFO:tasks.workunit.client.1.smithi196.stdout:1/954: dwrite d1/db/d16/d8e/fbd [0,4194304] 0 2024-04-22T15:19:13.753 INFO:tasks.workunit.client.1.smithi196.stdout:1/955: chown d1/db/dc/de/l12 68 1 2024-04-22T15:19:13.753 INFO:tasks.workunit.client.1.smithi196.stdout:9/974: symlink db/dd/de/d15/l143 0 2024-04-22T15:19:13.764 INFO:tasks.workunit.client.1.smithi196.stdout:0/935: dwrite d3/d9/d36/fd2 [0,4194304] 0 2024-04-22T15:19:13.765 INFO:tasks.workunit.client.1.smithi196.stdout:9/975: creat db/dd/de/dd7/def/d7a/d7b/d81/f144 x:0 0 0 2024-04-22T15:19:13.768 INFO:tasks.workunit.client.1.smithi196.stdout:8/890: write d2/d8/d49/d6c/f86 [3513,24739] 0 2024-04-22T15:19:13.769 INFO:tasks.workunit.client.1.smithi196.stdout:9/976: write db/dd/de/dd7/def/d7a/d7b/d81/f144 [98675,56541] 0 2024-04-22T15:19:13.770 INFO:tasks.workunit.client.1.smithi196.stdout:8/891: dread - d2/d8/d49/d8b/f99 zero size 2024-04-22T15:19:13.771 INFO:tasks.workunit.client.1.smithi196.stdout:0/936: truncate d3/d6/dc/d8c/daf/f7c 872157 0 2024-04-22T15:19:13.772 INFO:tasks.workunit.client.1.smithi196.stdout:1/956: link d1/db/c4f d1/db/d16/d1e/d46/ddc/c135 0 2024-04-22T15:19:13.775 INFO:tasks.workunit.client.1.smithi196.stdout:1/957: truncate d1/db/dc/dad/d36/fc9 1080961 0 2024-04-22T15:19:13.776 INFO:tasks.workunit.client.1.smithi196.stdout:1/958: truncate d1/db/dc/de/d96/ffa 900021 0 2024-04-22T15:19:13.776 INFO:tasks.workunit.client.1.smithi196.stdout:8/892: truncate d2/d5/d2e/fbe 2559 0 2024-04-22T15:19:13.777 INFO:tasks.workunit.client.1.smithi196.stdout:1/959: write d1/db/d16/d1e/d49/d7f/fb7 [793288,47090] 0 2024-04-22T15:19:13.778 INFO:tasks.workunit.client.1.smithi196.stdout:1/960: chown d1/db/dc/de/d5a/d94 29504509 1 2024-04-22T15:19:13.779 INFO:tasks.workunit.client.1.smithi196.stdout:0/937: rename d3/d9/d36/d9f/dd1/d65/d7f/ca5 to d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/c146 0 2024-04-22T15:19:13.780 INFO:tasks.workunit.client.1.smithi196.stdout:0/938: rename d3 to d3/d6/dc/d8c/d8d/d147 22 2024-04-22T15:19:13.780 INFO:tasks.workunit.client.1.smithi196.stdout:9/977: creat db/dd/de/d98/da9/d117/f145 x:0 0 0 2024-04-22T15:19:13.782 INFO:tasks.workunit.client.1.smithi196.stdout:1/961: symlink d1/db/d97/l136 0 2024-04-22T15:19:13.783 INFO:tasks.workunit.client.1.smithi196.stdout:0/939: creat d3/d9/d36/d9f/dd1/d65/d7f/d30/f148 x:0 0 0 2024-04-22T15:19:13.785 INFO:tasks.workunit.client.1.smithi196.stdout:9/978: truncate db/dd/d1b/f51 1929731 0 2024-04-22T15:19:13.789 INFO:tasks.workunit.client.1.smithi196.stdout:8/893: link d2/d5/d1a/db7/dd9/d68/d81/ld3 d2/d8/df2/l108 0 2024-04-22T15:19:13.792 INFO:tasks.workunit.client.1.smithi196.stdout:9/979: creat db/dd/d1b/de6/f146 x:0 0 0 2024-04-22T15:19:13.796 INFO:tasks.workunit.client.1.smithi196.stdout:5/935: sync 2024-04-22T15:19:13.800 INFO:tasks.workunit.client.1.smithi196.stdout:1/962: rename d1/db/ff to d1/db/d16/d1e/d49/f137 0 2024-04-22T15:19:13.800 INFO:tasks.workunit.client.1.smithi196.stdout:0/940: write d3/de/d14/f55 [3987469,39971] 0 2024-04-22T15:19:13.802 INFO:tasks.workunit.client.1.smithi196.stdout:5/936: symlink d1/d78/db0/l129 0 2024-04-22T15:19:13.803 INFO:tasks.workunit.client.1.smithi196.stdout:5/937: chown d1/d2/d98/dac/d105/da9/l87 250 1 2024-04-22T15:19:13.807 INFO:tasks.workunit.client.1.smithi196.stdout:9/980: symlink db/dd/de/l147 0 2024-04-22T15:19:13.808 INFO:tasks.workunit.client.1.smithi196.stdout:9/981: chown db/dd/de/dd7/def/d7a/d7b/d81 30 1 2024-04-22T15:19:13.811 INFO:tasks.workunit.client.1.smithi196.stdout:9/982: read - db/dd/de/dd7/def/fd8 zero size 2024-04-22T15:19:13.814 INFO:tasks.workunit.client.1.smithi196.stdout:8/894: dwrite d2/d5/d93/fad [0,4194304] 0 2024-04-22T15:19:13.814 INFO:tasks.workunit.client.1.smithi196.stdout:0/941: dread d3/db4/f12c [0,4194304] 0 2024-04-22T15:19:13.815 INFO:tasks.workunit.client.1.smithi196.stdout:5/938: mkdir d1/d14/d6a/d12a 0 2024-04-22T15:19:13.818 INFO:tasks.workunit.client.1.smithi196.stdout:9/983: unlink db/dd/de/dd7/def/d57/f3d 0 2024-04-22T15:19:13.819 INFO:tasks.workunit.client.1.smithi196.stdout:8/895: creat d2/d8/d49/d6c/f109 x:0 0 0 2024-04-22T15:19:13.822 INFO:tasks.workunit.client.1.smithi196.stdout:0/942: creat d3/d9/d36/d9f/dd1/d65/d7f/d30/f149 x:0 0 0 2024-04-22T15:19:13.823 INFO:tasks.workunit.client.1.smithi196.stdout:7/921: sync 2024-04-22T15:19:13.823 INFO:tasks.workunit.client.1.smithi196.stdout:2/930: sync 2024-04-22T15:19:13.823 INFO:tasks.workunit.client.1.smithi196.stdout:6/926: sync 2024-04-22T15:19:13.823 INFO:tasks.workunit.client.1.smithi196.stdout:4/908: sync 2024-04-22T15:19:13.825 INFO:tasks.workunit.client.1.smithi196.stdout:4/909: fdatasync d6/d13/d15/d2c/fa0 0 2024-04-22T15:19:13.825 INFO:tasks.workunit.client.1.smithi196.stdout:6/927: write dd/d1f/d38/d40/fcd [4076074,100399] 0 2024-04-22T15:19:13.828 INFO:tasks.workunit.client.1.smithi196.stdout:6/928: write dd/d1f/d38/f9d [2429830,4353] 0 2024-04-22T15:19:13.828 INFO:tasks.workunit.client.1.smithi196.stdout:0/943: mkdir d3/de/d14/d14a 0 2024-04-22T15:19:13.830 INFO:tasks.workunit.client.1.smithi196.stdout:1/963: dwrite d1/db/d16/d1e/d46/fc1 [0,4194304] 0 2024-04-22T15:19:13.831 INFO:tasks.workunit.client.1.smithi196.stdout:1/964: chown d1/db/d16/c87 63 1 2024-04-22T15:19:13.831 INFO:tasks.workunit.client.1.smithi196.stdout:5/939: write d1/d17/f60 [1289150,90029] 0 2024-04-22T15:19:13.833 INFO:tasks.workunit.client.1.smithi196.stdout:8/896: mkdir d2/d10a 0 2024-04-22T15:19:13.834 INFO:tasks.workunit.client.1.smithi196.stdout:1/965: write d1/db/fc0 [346234,82326] 0 2024-04-22T15:19:13.834 INFO:tasks.workunit.client.1.smithi196.stdout:2/931: write d1/f18 [2316697,19210] 0 2024-04-22T15:19:13.836 INFO:tasks.workunit.client.1.smithi196.stdout:1/966: write d1/db/d16/f1d [4060427,42642] 0 2024-04-22T15:19:13.838 INFO:tasks.workunit.client.1.smithi196.stdout:9/984: creat db/dd/f148 x:0 0 0 2024-04-22T15:19:13.841 INFO:tasks.workunit.client.1.smithi196.stdout:5/940: unlink d1/d78/db0/l97 0 2024-04-22T15:19:13.842 INFO:tasks.workunit.client.1.smithi196.stdout:9/985: stat db/dd/de/dd7/def/d7a/d7b/d81/f11d 0 2024-04-22T15:19:13.842 INFO:tasks.workunit.client.1.smithi196.stdout:9/986: fsync fa 0 2024-04-22T15:19:13.843 INFO:tasks.workunit.client.1.smithi196.stdout:5/941: dread - d1/df1/d6e/f11b zero size 2024-04-22T15:19:13.844 INFO:tasks.workunit.client.1.smithi196.stdout:2/932: dwrite d1/d1d/d122/f146 [0,4194304] 0 2024-04-22T15:19:13.844 INFO:tasks.workunit.client.1.smithi196.stdout:8/897: chown d2/d5/d1a/db7/dd9/d68/f7c 699590393 1 2024-04-22T15:19:13.845 INFO:tasks.workunit.client.1.smithi196.stdout:5/942: chown d1/df1/d6e/d7a/fa3 0 1 2024-04-22T15:19:13.847 INFO:tasks.workunit.client.1.smithi196.stdout:1/967: creat d1/db/dc/d26/dc4/f138 x:0 0 0 2024-04-22T15:19:13.847 INFO:tasks.workunit.client.1.smithi196.stdout:4/910: write d6/d13/f6b [973692,45802] 0 2024-04-22T15:19:13.848 INFO:tasks.workunit.client.1.smithi196.stdout:0/944: mknod d3/d6/dc/d8c/d8d/dab/c14b 0 2024-04-22T15:19:13.849 INFO:tasks.workunit.client.1.smithi196.stdout:1/968: write d1/db/d16/d6d/de6/f10d [957846,22665] 0 2024-04-22T15:19:13.850 INFO:tasks.workunit.client.1.smithi196.stdout:4/911: chown d6/d13/d15/d2c/d35/l122 621093605 1 2024-04-22T15:19:13.854 INFO:tasks.workunit.client.1.smithi196.stdout:8/898: mknod d2/d5/d93/dd5/c10b 0 2024-04-22T15:19:13.856 INFO:tasks.workunit.client.1.smithi196.stdout:5/943: read d1/d14/d3e/f44 [1248521,64234] 0 2024-04-22T15:19:13.856 INFO:tasks.workunit.client.1.smithi196.stdout:2/933: rename d1/d13/d25/d2a/ddb/dff/d138/da1/f10c to d1/d23/d126/d127/f14c 0 2024-04-22T15:19:13.859 INFO:tasks.workunit.client.1.smithi196.stdout:0/945: fdatasync d3/de/d1c/f31 0 2024-04-22T15:19:13.861 INFO:tasks.workunit.client.1.smithi196.stdout:6/929: write dd/d20/d5c/d6a/faa [374081,100579] 0 2024-04-22T15:19:13.861 INFO:tasks.workunit.client.1.smithi196.stdout:4/912: truncate d6/d13/d15/d2c/d62/d78/fd1 143224 0 2024-04-22T15:19:13.862 INFO:tasks.workunit.client.1.smithi196.stdout:4/913: chown d6/d13/d15/d2c/d62/d78/d8b/fd2 5282809 1 2024-04-22T15:19:13.864 INFO:tasks.workunit.client.1.smithi196.stdout:8/899: mkdir d2/d5/d2e/d10c 0 2024-04-22T15:19:13.864 INFO:tasks.workunit.client.1.smithi196.stdout:8/900: chown d2/d8/d49/d8b/lc3 5807090 1 2024-04-22T15:19:13.870 INFO:tasks.workunit.client.1.smithi196.stdout:5/944: creat d1/d2/d98/dac/d123/f12b x:0 0 0 2024-04-22T15:19:13.870 INFO:tasks.workunit.client.1.smithi196.stdout:2/934: creat d1/d13/d41/f14d x:0 0 0 2024-04-22T15:19:13.871 INFO:tasks.workunit.client.1.smithi196.stdout:0/946: rmdir d3/d6/dc/d8c/d8d/d98 39 2024-04-22T15:19:13.872 INFO:tasks.workunit.client.1.smithi196.stdout:1/969: dread d1/db/d16/d8e/ff2 [0,4194304] 0 2024-04-22T15:19:13.873 INFO:tasks.workunit.client.1.smithi196.stdout:1/970: fsync d1/db/d16/d6d/f7c 0 2024-04-22T15:19:13.874 INFO:tasks.workunit.client.1.smithi196.stdout:1/971: fdatasync d1/db/dc/de/d5a/d5b/f79 0 2024-04-22T15:19:13.875 INFO:tasks.workunit.client.1.smithi196.stdout:8/901: write d2/d5/faf [6655576,87057] 0 2024-04-22T15:19:13.875 INFO:tasks.workunit.client.1.smithi196.stdout:1/972: chown d1/d54 601 1 2024-04-22T15:19:13.876 INFO:tasks.workunit.client.1.smithi196.stdout:1/973: stat d1/db/d16/d1e/d65 0 2024-04-22T15:19:13.877 INFO:tasks.workunit.client.1.smithi196.stdout:5/945: symlink d1/d17/d1c/d100/l12c 0 2024-04-22T15:19:13.886 INFO:tasks.workunit.client.1.smithi196.stdout:9/987: link db/dd/de/dd7/def/d7a/f9c db/dd/de/dd7/def/d57/f149 0 2024-04-22T15:19:13.888 INFO:tasks.workunit.client.1.smithi196.stdout:6/930: dwrite dd/d2b/f3e [0,4194304] 0 2024-04-22T15:19:13.890 INFO:tasks.workunit.client.1.smithi196.stdout:0/947: dwrite d3/d9/d36/d9f/dd1/d65/d7f/d30/fe7 [0,4194304] 0 2024-04-22T15:19:13.891 INFO:tasks.workunit.client.1.smithi196.stdout:0/948: readlink d3/de/d1c/l5c 0 2024-04-22T15:19:13.894 INFO:tasks.workunit.client.1.smithi196.stdout:2/935: mknod d1/d13/d101/d147/c14e 0 2024-04-22T15:19:13.895 INFO:tasks.workunit.client.1.smithi196.stdout:2/936: fdatasync d1/f5 0 2024-04-22T15:19:13.895 INFO:tasks.workunit.client.1.smithi196.stdout:5/946: truncate d1/d2/d98/dac/fd3 473742 0 2024-04-22T15:19:13.896 INFO:tasks.workunit.client.1.smithi196.stdout:5/947: chown d1/d2/d98/dac/lf0 0 1 2024-04-22T15:19:13.900 INFO:tasks.workunit.client.1.smithi196.stdout:9/988: read db/dd/d1b/d2c/f7c [278961,55285] 0 2024-04-22T15:19:13.907 INFO:tasks.workunit.client.1.smithi196.stdout:8/902: creat d2/dc/d103/f10d x:0 0 0 2024-04-22T15:19:13.910 INFO:tasks.workunit.client.1.smithi196.stdout:0/949: creat d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8/f14c x:0 0 0 2024-04-22T15:19:13.910 INFO:tasks.workunit.client.1.smithi196.stdout:6/931: dread dd/d1f/d4e/d53/d7f/f83 [0,4194304] 0 2024-04-22T15:19:13.912 INFO:tasks.workunit.client.1.smithi196.stdout:2/937: symlink d1/d23/d126/l14f 0 2024-04-22T15:19:13.913 INFO:tasks.workunit.client.1.smithi196.stdout:2/938: write d1/d13/d25/f4f [4592011,9411] 0 2024-04-22T15:19:13.916 INFO:tasks.workunit.client.1.smithi196.stdout:5/948: dread d1/d4/f56 [0,4194304] 0 2024-04-22T15:19:13.919 INFO:tasks.workunit.client.1.smithi196.stdout:4/914: getdents d6 0 2024-04-22T15:19:13.920 INFO:tasks.workunit.client.1.smithi196.stdout:1/974: creat d1/f139 x:0 0 0 2024-04-22T15:19:13.922 INFO:tasks.workunit.client.1.smithi196.stdout:1/975: chown d1/db/dc/de/d43/d5e/d7a/fcf 129486660 1 2024-04-22T15:19:13.922 INFO:tasks.workunit.client.1.smithi196.stdout:8/903: creat d2/d6a/f10e x:0 0 0 2024-04-22T15:19:13.925 INFO:tasks.workunit.client.1.smithi196.stdout:4/915: dwrite d6/d13/d15/d2c/d62/d78/d8b/fe3 [4194304,4194304] 0 2024-04-22T15:19:13.927 INFO:tasks.workunit.client.1.smithi196.stdout:4/916: chown d6/d13/f68 3 1 2024-04-22T15:19:13.927 INFO:tasks.workunit.client.1.smithi196.stdout:9/989: write db/dd/d1b/d2c/fb5 [3664307,26636] 0 2024-04-22T15:19:13.930 INFO:tasks.workunit.client.1.smithi196.stdout:9/990: fdatasync db/dd/de/d15/d31/df6/f10c 0 2024-04-22T15:19:13.930 INFO:tasks.workunit.client.1.smithi196.stdout:2/939: rmdir d1/d13/d25/d9f/ddd 39 2024-04-22T15:19:13.931 INFO:tasks.workunit.client.1.smithi196.stdout:6/932: write dd/d1f/d4e/d53/d55/f9a [966314,79386] 0 2024-04-22T15:19:13.931 INFO:tasks.workunit.client.1.smithi196.stdout:7/922: sync 2024-04-22T15:19:13.934 INFO:tasks.workunit.client.1.smithi196.stdout:0/950: dread d3/d9/fb1 [0,4194304] 0 2024-04-22T15:19:13.935 INFO:tasks.workunit.client.1.smithi196.stdout:0/951: fdatasync d3/de/d1c/d27/f93 0 2024-04-22T15:19:13.936 INFO:tasks.workunit.client.1.smithi196.stdout:1/976: unlink d1/db/dc/cc7 0 2024-04-22T15:19:13.937 INFO:tasks.workunit.client.1.smithi196.stdout:9/991: dread db/dd/de/d98/f114 [0,4194304] 0 2024-04-22T15:19:13.937 INFO:tasks.workunit.client.1.smithi196.stdout:1/977: chown d1/db/dc/dad/c35 0 1 2024-04-22T15:19:13.937 INFO:tasks.workunit.client.1.smithi196.stdout:4/917: mknod d6/c124 0 2024-04-22T15:19:13.939 INFO:tasks.workunit.client.1.smithi196.stdout:1/978: dread - d1/db/d16/d1e/d49/d90/ff8 zero size 2024-04-22T15:19:13.939 INFO:tasks.workunit.client.1.smithi196.stdout:7/923: creat d14/d25/d31/f136 x:0 0 0 2024-04-22T15:19:13.940 INFO:tasks.workunit.client.1.smithi196.stdout:6/933: rename dd/d1f/d37/d58/cd5 to dd/d1f/d37/d58/c116 0 2024-04-22T15:19:13.945 INFO:tasks.workunit.client.1.smithi196.stdout:2/940: creat d1/d23/d27/d2e/d55/d5b/d7c/f150 x:0 0 0 2024-04-22T15:19:13.945 INFO:tasks.workunit.client.1.smithi196.stdout:5/949: write d1/df1/d5e/f77 [1305582,62838] 0 2024-04-22T15:19:13.945 INFO:tasks.workunit.client.1.smithi196.stdout:1/979: dwrite d1/db/dc/de/d96/d9d/f50 [4194304,4194304] 0 2024-04-22T15:19:13.949 INFO:tasks.workunit.client.1.smithi196.stdout:9/992: dread db/dd/de/dd7/def/d7a/f7e [0,4194304] 0 2024-04-22T15:19:13.949 INFO:tasks.workunit.client.1.smithi196.stdout:0/952: symlink d3/de/l14d 0 2024-04-22T15:19:13.950 INFO:tasks.workunit.client.1.smithi196.stdout:0/953: write d3/f5 [4242297,65097] 0 2024-04-22T15:19:13.950 INFO:tasks.workunit.client.1.smithi196.stdout:8/904: rename d2/d35/f63 to d2/f10f 0 2024-04-22T15:19:13.952 INFO:tasks.workunit.client.1.smithi196.stdout:6/934: mknod dd/d2b/d8d/d102/c117 0 2024-04-22T15:19:13.956 INFO:tasks.workunit.client.1.smithi196.stdout:7/924: mkdir d14/d26/d5a/d137 0 2024-04-22T15:19:13.957 INFO:tasks.workunit.client.1.smithi196.stdout:1/980: write d1/db/dc/dad/f9b [5525530,99155] 0 2024-04-22T15:19:13.960 INFO:tasks.workunit.client.1.smithi196.stdout:5/950: rename d1/d2/d98/dac/d123/fd7 to d1/d78/dcf/f12d 0 2024-04-22T15:19:13.961 INFO:tasks.workunit.client.1.smithi196.stdout:0/954: dread - d3/d6/dc/d8c/d8d/d98/dbd/f140 zero size 2024-04-22T15:19:13.962 INFO:tasks.workunit.client.1.smithi196.stdout:6/935: dread dd/d70/fd2 [0,4194304] 0 2024-04-22T15:19:13.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:13 smithi096 ceph-mon[26688]: pgmap v43: 65 pgs: 65 active+clean; 3.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 85 MiB/s rd, 158 MiB/s wr, 393 op/s 2024-04-22T15:19:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:13 smithi196 ceph-mon[34717]: pgmap v43: 65 pgs: 65 active+clean; 3.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 85 MiB/s rd, 158 MiB/s wr, 393 op/s 2024-04-22T15:19:14.046 INFO:tasks.workunit.client.1.smithi196.stdout:2/941: creat d1/d13/f151 x:0 0 0 2024-04-22T15:19:14.046 INFO:tasks.workunit.client.1.smithi196.stdout:9/993: creat db/dd/de/f14a x:0 0 0 2024-04-22T15:19:14.049 INFO:tasks.workunit.client.1.smithi196.stdout:5/951: write d1/d14/f1e [2007561,77409] 0 2024-04-22T15:19:14.051 INFO:tasks.workunit.client.1.smithi196.stdout:8/905: getdents d2/d5/d4f 0 2024-04-22T15:19:14.051 INFO:tasks.workunit.client.1.smithi196.stdout:6/936: fdatasync dd/f1c 0 2024-04-22T15:19:14.052 INFO:tasks.workunit.client.1.smithi196.stdout:0/955: dwrite d3/d9/d36/d9f/dd1/f142 [0,4194304] 0 2024-04-22T15:19:14.055 INFO:tasks.workunit.client.1.smithi196.stdout:0/956: truncate d3/d9/f73 5077426 0 2024-04-22T15:19:14.055 INFO:tasks.workunit.client.1.smithi196.stdout:0/957: readlink d3/db4/dc7/l10a 0 2024-04-22T15:19:14.059 INFO:tasks.workunit.client.1.smithi196.stdout:9/994: mkdir db/dd/de/d15/d31/d64/dab/d14b 0 2024-04-22T15:19:14.061 INFO:tasks.workunit.client.1.smithi196.stdout:6/937: creat dd/d1f/d37/dcb/f118 x:0 0 0 2024-04-22T15:19:14.061 INFO:tasks.workunit.client.1.smithi196.stdout:0/958: mknod d3/d9/d36/d9f/dd1/d65/d7f/d30/c14e 0 2024-04-22T15:19:14.062 INFO:tasks.workunit.client.1.smithi196.stdout:1/981: dread d1/db/d16/d1e/d49/f137 [0,4194304] 0 2024-04-22T15:19:14.063 INFO:tasks.workunit.client.1.smithi196.stdout:6/938: chown dd/d1f/d37/d3c/c7a 1390291 1 2024-04-22T15:19:14.063 INFO:tasks.workunit.client.1.smithi196.stdout:9/995: mknod db/dd/de/d98/c14c 0 2024-04-22T15:19:14.064 INFO:tasks.workunit.client.1.smithi196.stdout:1/982: chown d1/db/d16/fd1 4032 1 2024-04-22T15:19:14.065 INFO:tasks.workunit.client.1.smithi196.stdout:0/959: read d3/f5 [1114151,123851] 0 2024-04-22T15:19:14.068 INFO:tasks.workunit.client.1.smithi196.stdout:2/942: dread d1/d13/d41/fd4 [0,4194304] 0 2024-04-22T15:19:14.068 INFO:tasks.workunit.client.1.smithi196.stdout:0/960: chown d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/c139 29890 1 2024-04-22T15:19:14.069 INFO:tasks.workunit.client.1.smithi196.stdout:6/939: mknod dd/d1f/d37/dcb/c119 0 2024-04-22T15:19:14.070 INFO:tasks.workunit.client.1.smithi196.stdout:1/983: mknod d1/db/dc/de/d5a/d5b/d69/c13a 0 2024-04-22T15:19:14.070 INFO:tasks.workunit.client.1.smithi196.stdout:6/940: chown dd/ddf 48 1 2024-04-22T15:19:14.071 INFO:tasks.workunit.client.1.smithi196.stdout:1/984: write d1/db/dc/de/d43/d5e/d7a/fa0 [500991,53233] 0 2024-04-22T15:19:14.072 INFO:tasks.workunit.client.1.smithi196.stdout:9/996: rename db/dd/de/d75/faa to db/dd/de/d75/d124/f14d 0 2024-04-22T15:19:14.073 INFO:tasks.workunit.client.1.smithi196.stdout:5/952: dread d1/d2/d98/dac/d105/da9/f86 [0,4194304] 0 2024-04-22T15:19:14.073 INFO:tasks.workunit.client.1.smithi196.stdout:9/997: write fa [1758616,106493] 0 2024-04-22T15:19:14.076 INFO:tasks.workunit.client.1.smithi196.stdout:8/906: dread d2/d3f/f53 [0,4194304] 0 2024-04-22T15:19:14.077 INFO:tasks.workunit.client.1.smithi196.stdout:2/943: rename d1/d23/d27/d2e/d55/d5b/d93/lb1 to d1/d23/d126/d127/l152 0 2024-04-22T15:19:14.080 INFO:tasks.workunit.client.1.smithi196.stdout:4/918: sync 2024-04-22T15:19:14.081 INFO:tasks.workunit.client.1.smithi196.stdout:2/944: dwrite d1/f18 [0,4194304] 0 2024-04-22T15:19:14.083 INFO:tasks.workunit.client.1.smithi196.stdout:5/953: symlink d1/df1/d6e/d7a/d11e/l12e 0 2024-04-22T15:19:14.086 INFO:tasks.workunit.client.1.smithi196.stdout:9/998: readlink db/dd/de/dd7/def/d57/da7/lcd 0 2024-04-22T15:19:14.087 INFO:tasks.workunit.client.1.smithi196.stdout:8/907: creat d2/d6a/f110 x:0 0 0 2024-04-22T15:19:14.094 INFO:tasks.workunit.client.1.smithi196.stdout:0/961: rmdir d3/de/d14/d86/dd5 0 2024-04-22T15:19:14.097 INFO:tasks.workunit.client.1.smithi196.stdout:6/941: write dd/d1f/d38/d40/f49 [1204560,51370] 0 2024-04-22T15:19:14.098 INFO:tasks.workunit.client.1.smithi196.stdout:1/985: dread d1/db/d16/d6d/de6/f104 [0,4194304] 0 2024-04-22T15:19:14.098 INFO:tasks.workunit.client.1.smithi196.stdout:5/954: dread - d1/df1/d6e/d7a/fe3 zero size 2024-04-22T15:19:14.101 INFO:tasks.workunit.client.1.smithi196.stdout:5/955: stat d1/d2/dbb 0 2024-04-22T15:19:14.103 INFO:tasks.workunit.client.1.smithi196.stdout:8/908: mkdir d2/d5/d2e/d10c/d111 0 2024-04-22T15:19:14.106 INFO:tasks.workunit.client.1.smithi196.stdout:5/956: fsync d1/d4/f56 0 2024-04-22T15:19:14.107 INFO:tasks.workunit.client.1.smithi196.stdout:2/945: dwrite d1/f7 [4194304,4194304] 0 2024-04-22T15:19:14.108 INFO:tasks.workunit.client.1.smithi196.stdout:7/925: sync 2024-04-22T15:19:14.110 INFO:tasks.workunit.client.1.smithi196.stdout:7/926: chown d14/d26/db7/lc9 684257 1 2024-04-22T15:19:14.111 INFO:tasks.workunit.client.1.smithi196.stdout:1/986: rename d1/db/dc/dad/f117 to d1/db/d97/dd7/f13b 0 2024-04-22T15:19:14.114 INFO:tasks.workunit.client.1.smithi196.stdout:9/999: getdents db/dd/de/dd7 0 2024-04-22T15:19:14.116 INFO:tasks.workunit.client.1.smithi196.stdout:7/927: fdatasync d14/d26/d2a/d72/db1/fec 0 2024-04-22T15:19:14.116 INFO:tasks.workunit.client.1.smithi196.stdout:8/909: dwrite d2/d6a/d7b/feb [0,4194304] 0 2024-04-22T15:19:14.118 INFO:tasks.workunit.client.1.smithi196.stdout:4/919: creat d6/d14/d18/d26/d32/d67/de7/f125 x:0 0 0 2024-04-22T15:19:14.120 INFO:tasks.workunit.client.1.smithi196.stdout:6/942: dread dd/d1f/d4e/fe5 [0,4194304] 0 2024-04-22T15:19:14.121 INFO:tasks.workunit.client.1.smithi196.stdout:5/957: symlink d1/d17/d1a/d2f/l12f 0 2024-04-22T15:19:14.122 INFO:tasks.workunit.client.1.smithi196.stdout:0/962: dwrite d3/de/d14/f39 [0,4194304] 0 2024-04-22T15:19:14.126 INFO:tasks.workunit.client.1.smithi196.stdout:1/987: unlink d1/db/dc/de/d5a/d5b/d85/f119 0 2024-04-22T15:19:14.127 INFO:tasks.workunit.client.1.smithi196.stdout:1/988: fdatasync d1/db/dc/d26/d2e/f34 0 2024-04-22T15:19:14.128 INFO:tasks.workunit.client.1.smithi196.stdout:1/989: fdatasync d1/db/dc/de/d43/d5e/d7a/fa1 0 2024-04-22T15:19:14.130 INFO:tasks.workunit.client.1.smithi196.stdout:4/920: truncate d6/de8/f112 992778 0 2024-04-22T15:19:14.134 INFO:tasks.workunit.client.1.smithi196.stdout:7/928: rmdir d14/d26/d2a/d2b/de8/dc6 39 2024-04-22T15:19:14.138 INFO:tasks.workunit.client.1.smithi196.stdout:6/943: creat dd/ddf/f11a x:0 0 0 2024-04-22T15:19:14.139 INFO:tasks.workunit.client.1.smithi196.stdout:6/944: fsync dd/d1f/d37/f31 0 2024-04-22T15:19:14.140 INFO:tasks.workunit.client.1.smithi196.stdout:5/958: symlink d1/d14/d3e/l130 0 2024-04-22T15:19:14.140 INFO:tasks.workunit.client.1.smithi196.stdout:6/945: write dd/d1f/d37/d7b/fb4 [949721,14399] 0 2024-04-22T15:19:14.141 INFO:tasks.workunit.client.1.smithi196.stdout:0/963: readlink d3/d9/d36/ldf 0 2024-04-22T15:19:14.145 INFO:tasks.workunit.client.1.smithi196.stdout:1/990: fsync d1/db/dc/d26/f2b 0 2024-04-22T15:19:14.147 INFO:tasks.workunit.client.1.smithi196.stdout:4/921: creat d6/d14/d18/d26/d32/d67/f126 x:0 0 0 2024-04-22T15:19:14.150 INFO:tasks.workunit.client.1.smithi196.stdout:8/910: symlink d2/d8/d49/d6c/dde/l112 0 2024-04-22T15:19:14.151 INFO:tasks.workunit.client.1.smithi196.stdout:1/991: dwrite d1/db/d16/d1e/d49/d7f/fb7 [0,4194304] 0 2024-04-22T15:19:14.157 INFO:tasks.workunit.client.1.smithi196.stdout:0/964: write d3/de/d14/ded/ffe [3400020,111285] 0 2024-04-22T15:19:14.159 INFO:tasks.workunit.client.1.smithi196.stdout:0/965: truncate d3/d109/f121 558200 0 2024-04-22T15:19:14.160 INFO:tasks.workunit.client.1.smithi196.stdout:0/966: fdatasync d3/d9/d36/d9f/dd1/d65/d7f/ddb/de8/f14c 0 2024-04-22T15:19:14.160 INFO:tasks.workunit.client.1.smithi196.stdout:0/967: readlink d3/d9/d36/d9f/dd1/d65/lf3 0 2024-04-22T15:19:14.164 INFO:tasks.workunit.client.1.smithi196.stdout:8/911: rmdir d2/d35/d96 39 2024-04-22T15:19:14.165 INFO:tasks.workunit.client.1.smithi196.stdout:8/912: chown d2/d6a 102 1 2024-04-22T15:19:14.171 INFO:tasks.workunit.client.1.smithi196.stdout:6/946: link dd/d1f/d4e/fe5 dd/d1f/d4e/d53/f11b 0 2024-04-22T15:19:14.175 INFO:tasks.workunit.client.1.smithi196.stdout:4/922: chown d6/d10/fb1 0 1 2024-04-22T15:19:14.177 INFO:tasks.workunit.client.1.smithi196.stdout:0/968: write d3/d6/dc/f92 [3452302,35776] 0 2024-04-22T15:19:14.183 INFO:tasks.workunit.client.1.smithi196.stdout:6/947: creat dd/d1f/d4e/d53/d55/d6d/dd1/dca/f11c x:0 0 0 2024-04-22T15:19:14.185 INFO:tasks.workunit.client.1.smithi196.stdout:7/929: getdents d14/d26/db7 0 2024-04-22T15:19:14.188 INFO:tasks.workunit.client.1.smithi196.stdout:8/913: fsync d2/dc/d45/d98/fa6 0 2024-04-22T15:19:14.189 INFO:tasks.workunit.client.1.smithi196.stdout:1/992: rmdir d1/db/d16/d1e/d46/dba 0 2024-04-22T15:19:14.195 INFO:tasks.workunit.client.1.smithi196.stdout:7/930: dwrite d14/d26/fdf [0,4194304] 0 2024-04-22T15:19:14.196 INFO:tasks.workunit.client.1.smithi196.stdout:5/959: getdents d1/d78/dcf/dff 0 2024-04-22T15:19:14.203 INFO:tasks.workunit.client.1.smithi196.stdout:0/969: rename d3/de/d14/lde to d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/l14f 0 2024-04-22T15:19:14.205 INFO:tasks.workunit.client.1.smithi196.stdout:6/948: write dd/d1f/f60 [334477,54998] 0 2024-04-22T15:19:14.206 INFO:tasks.workunit.client.1.smithi196.stdout:4/923: write d6/d10/f37 [42627,39563] 0 2024-04-22T15:19:14.207 INFO:tasks.workunit.client.1.smithi196.stdout:2/946: sync 2024-04-22T15:19:14.208 INFO:tasks.workunit.client.1.smithi196.stdout:6/949: stat dd/d1f/d4e/d53/d55/d6d/dd1/de3/df7/df9/f101 0 2024-04-22T15:19:14.208 INFO:tasks.workunit.client.1.smithi196.stdout:6/950: truncate dd/d2b/f77 4372584 0 2024-04-22T15:19:14.213 INFO:tasks.workunit.client.1.smithi196.stdout:6/951: dwrite dd/f16 [0,4194304] 0 2024-04-22T15:19:14.221 INFO:tasks.workunit.client.1.smithi196.stdout:6/952: dwrite dd/d20/d5c/d5e/f9b [4194304,4194304] 0 2024-04-22T15:19:14.229 INFO:tasks.workunit.client.1.smithi196.stdout:1/993: creat d1/db/dc/de/d43/f13c x:0 0 0 2024-04-22T15:19:14.231 INFO:tasks.workunit.client.1.smithi196.stdout:1/994: truncate d1/db/d16/d1e/d65/f11c 137303 0 2024-04-22T15:19:14.231 INFO:tasks.workunit.client.1.smithi196.stdout:5/960: rename d1/d17/f24 to d1/d2/d98/dac/f131 0 2024-04-22T15:19:14.232 INFO:tasks.workunit.client.1.smithi196.stdout:0/970: creat d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/dd0/f150 x:0 0 0 2024-04-22T15:19:14.232 INFO:tasks.workunit.client.1.smithi196.stdout:1/995: chown d1/db/d16/f1d 930166382 1 2024-04-22T15:19:14.234 INFO:tasks.workunit.client.1.smithi196.stdout:1/996: stat d1/db/dc/dad/d36/def/d12e 0 2024-04-22T15:19:14.235 INFO:tasks.workunit.client.1.smithi196.stdout:1/997: readlink d1/db/dc/dad/l67 0 2024-04-22T15:19:14.236 INFO:tasks.workunit.client.1.smithi196.stdout:1/998: chown d1/db/dc/de/d96/d9d/f124 7 1 2024-04-22T15:19:14.238 INFO:tasks.workunit.client.1.smithi196.stdout:6/953: mknod dd/d1f/d4e/d53/d55/d6d/dd1/de3/de8/c11d 0 2024-04-22T15:19:14.239 INFO:tasks.workunit.client.1.smithi196.stdout:5/961: mkdir d1/d2/d98/dac/d105/da9/d132 0 2024-04-22T15:19:14.240 INFO:tasks.workunit.client.1.smithi196.stdout:8/914: link d2/d5/d1a/cb0 d2/d6a/d7b/c113 0 2024-04-22T15:19:14.240 INFO:tasks.workunit.client.1.smithi196.stdout:8/915: chown d2/d5/d1a/db7/ff4 0 1 2024-04-22T15:19:14.240 INFO:tasks.workunit.client.1.smithi196.stdout:8/916: stat d2/d35 0 2024-04-22T15:19:14.241 INFO:tasks.workunit.client.1.smithi196.stdout:0/971: rename d3/d9/d36/d9f/dd1/f7b to d3/d6/dc/d8c/d8d/d98/f151 0 2024-04-22T15:19:14.243 INFO:tasks.workunit.client.1.smithi196.stdout:6/954: fsync dd/d1f/d38/d40/f5f 0 2024-04-22T15:19:14.245 INFO:tasks.workunit.client.1.smithi196.stdout:5/962: creat d1/df1/d6e/f133 x:0 0 0 2024-04-22T15:19:14.246 INFO:tasks.workunit.client.1.smithi196.stdout:6/955: dread - dd/d2b/f104 zero size 2024-04-22T15:19:14.247 INFO:tasks.workunit.client.1.smithi196.stdout:5/963: write d1/d2/f120 [720964,74266] 0 2024-04-22T15:19:14.249 INFO:tasks.workunit.client.1.smithi196.stdout:0/972: dwrite d3/d9/d36/d9f/dd1/d65/d7f/fd8 [0,4194304] 0 2024-04-22T15:19:14.254 INFO:tasks.workunit.client.1.smithi196.stdout:6/956: dread - dd/d1f/d37/fae zero size 2024-04-22T15:19:14.256 INFO:tasks.workunit.client.1.smithi196.stdout:5/964: truncate d1/d4/f22 9653267 0 2024-04-22T15:19:14.256 INFO:tasks.workunit.client.1.smithi196.stdout:0/973: creat d3/d9/d36/d9f/dd1/f152 x:0 0 0 2024-04-22T15:19:14.256 INFO:tasks.workunit.client.1.smithi196.stdout:4/924: dread d6/d13/d15/d2c/d35/f4c [0,4194304] 0 2024-04-22T15:19:14.258 INFO:tasks.workunit.client.1.smithi196.stdout:6/957: rmdir dd/d20/d5c/d6a 39 2024-04-22T15:19:14.263 INFO:tasks.workunit.client.1.smithi196.stdout:1/999: getdents d1/db/d16/d6d 0 2024-04-22T15:19:14.265 INFO:tasks.workunit.client.1.smithi196.stdout:7/931: write d14/f6b [790068,50588] 0 2024-04-22T15:19:14.266 INFO:tasks.workunit.client.1.smithi196.stdout:7/932: fdatasync d14/d26/d5e/ff6 0 2024-04-22T15:19:14.268 INFO:tasks.workunit.client.1.smithi196.stdout:8/917: rmdir d2/d5 39 2024-04-22T15:19:14.272 INFO:tasks.workunit.client.1.smithi196.stdout:4/925: fsync d6/d13/d15/d2c/d62/f97 0 2024-04-22T15:19:14.273 INFO:tasks.workunit.client.1.smithi196.stdout:2/947: dwrite d1/da4/fe9 [4194304,4194304] 0 2024-04-22T15:19:14.278 INFO:tasks.workunit.client.1.smithi196.stdout:8/918: dread d2/d8/d49/d6c/fc1 [0,4194304] 0 2024-04-22T15:19:14.289 INFO:tasks.workunit.client.1.smithi196.stdout:5/965: write d1/d4/f56 [711876,79159] 0 2024-04-22T15:19:14.294 INFO:tasks.workunit.client.1.smithi196.stdout:7/933: mkdir d14/d25/d138 0 2024-04-22T15:19:14.295 INFO:tasks.workunit.client.1.smithi196.stdout:4/926: dread d6/d14/d18/d26/d32/d67/d9a/da5/fac [0,4194304] 0 2024-04-22T15:19:14.302 INFO:tasks.workunit.client.1.smithi196.stdout:7/934: dwrite d14/d15/d91/fe7 [0,4194304] 0 2024-04-22T15:19:14.317 INFO:tasks.workunit.client.1.smithi196.stdout:8/919: truncate d2/d5/d1a/f37 2454324 0 2024-04-22T15:19:14.318 INFO:tasks.workunit.client.1.smithi196.stdout:6/958: getdents dd/d2b/d8d 0 2024-04-22T15:19:14.320 INFO:tasks.workunit.client.1.smithi196.stdout:0/974: link d3/de/d14/c134 d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/c153 0 2024-04-22T15:19:14.323 INFO:tasks.workunit.client.1.smithi196.stdout:7/935: mkdir d14/d4d/da3/de9/d139 0 2024-04-22T15:19:14.329 INFO:tasks.workunit.client.1.smithi196.stdout:0/975: chown d3/de/l6e 6687337 1 2024-04-22T15:19:14.333 INFO:tasks.workunit.client.1.smithi196.stdout:7/936: mkdir d14/d25/d31/d127/d13a 0 2024-04-22T15:19:14.334 INFO:tasks.workunit.client.1.smithi196.stdout:5/966: write d1/d14/ff4 [630548,32286] 0 2024-04-22T15:19:14.335 INFO:tasks.workunit.client.1.smithi196.stdout:6/959: getdents dd/d1f/d4e/d53/d55/d6d/dd1/dfc 0 2024-04-22T15:19:14.335 INFO:tasks.workunit.client.1.smithi196.stdout:0/976: dwrite d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/f112 [0,4194304] 0 2024-04-22T15:19:14.335 INFO:tasks.workunit.client.1.smithi196.stdout:2/948: write d1/d23/fba [65566,112947] 0 2024-04-22T15:19:14.337 INFO:tasks.workunit.client.1.smithi196.stdout:7/937: write d14/d26/d95/fd3 [300659,130025] 0 2024-04-22T15:19:14.340 INFO:tasks.workunit.client.1.smithi196.stdout:8/920: dread d2/d8/f76 [0,4194304] 0 2024-04-22T15:19:14.340 INFO:tasks.workunit.client.1.smithi196.stdout:4/927: write d6/d14/d18/d26/d32/d67/f77 [5696683,10999] 0 2024-04-22T15:19:14.343 INFO:tasks.workunit.client.1.smithi196.stdout:6/960: mkdir dd/d1f/d37/d58/d11e 0 2024-04-22T15:19:14.343 INFO:tasks.workunit.client.1.smithi196.stdout:8/921: write d2/d5/d1a/db7/dd9/f100 [130530,121427] 0 2024-04-22T15:19:14.345 INFO:tasks.workunit.client.1.smithi196.stdout:8/922: readlink d2/d8/l21 0 2024-04-22T15:19:14.348 INFO:tasks.workunit.client.1.smithi196.stdout:5/967: truncate d1/d2/d98/dac/fd3 394765 0 2024-04-22T15:19:14.349 INFO:tasks.workunit.client.1.smithi196.stdout:0/977: dread d3/d6/dc/d8c/d8d/fc1 [0,4194304] 0 2024-04-22T15:19:14.354 INFO:tasks.workunit.client.1.smithi196.stdout:0/978: dwrite d3/de/d14/d5a/f68 [0,4194304] 0 2024-04-22T15:19:14.356 INFO:tasks.workunit.client.1.smithi196.stdout:0/979: readlink d3/d9/d36/d9f/dd1/d65/lf3 0 2024-04-22T15:19:14.363 INFO:tasks.workunit.client.1.smithi196.stdout:7/938: fsync d14/d26/d28/f93 0 2024-04-22T15:19:14.369 INFO:tasks.workunit.client.1.smithi196.stdout:4/928: creat d6/d14/d18/d26/d32/d67/d9a/da5/d11d/f127 x:0 0 0 2024-04-22T15:19:14.374 INFO:tasks.workunit.client.1.smithi196.stdout:0/980: mkdir d3/d9/d36/d9f/dd1/d65/d7f/d154 0 2024-04-22T15:19:14.376 INFO:tasks.workunit.client.1.smithi196.stdout:4/929: dwrite d6/d14/d18/d26/d32/d67/f123 [0,4194304] 0 2024-04-22T15:19:14.377 INFO:tasks.workunit.client.1.smithi196.stdout:8/923: dread d2/fa [0,4194304] 0 2024-04-22T15:19:14.378 INFO:tasks.workunit.client.1.smithi196.stdout:4/930: write d6/d14/d18/d40/d85/f100 [1043236,45858] 0 2024-04-22T15:19:14.379 INFO:tasks.workunit.client.1.smithi196.stdout:6/961: rename dd/c1e to dd/d1f/c11f 0 2024-04-22T15:19:14.380 INFO:tasks.workunit.client.1.smithi196.stdout:0/981: mknod d3/de/c155 0 2024-04-22T15:19:14.385 INFO:tasks.workunit.client.1.smithi196.stdout:5/968: rename d1/d17/d1a/d2f/d65/f113 to d1/d2/d98/dac/d124/f134 0 2024-04-22T15:19:14.385 INFO:tasks.workunit.client.1.smithi196.stdout:2/949: dwrite d1/f16 [0,4194304] 0 2024-04-22T15:19:14.387 INFO:tasks.workunit.client.1.smithi196.stdout:5/969: write d1/df1/f54 [377327,15594] 0 2024-04-22T15:19:14.387 INFO:tasks.workunit.client.1.smithi196.stdout:2/950: chown d1/d13/d1f/lb6 24375 1 2024-04-22T15:19:14.388 INFO:tasks.workunit.client.1.smithi196.stdout:2/951: fdatasync d1/d23/d27/d2e/f88 0 2024-04-22T15:19:14.389 INFO:tasks.workunit.client.1.smithi196.stdout:4/931: write d6/d14/d18/d40/f65 [5003864,82764] 0 2024-04-22T15:19:14.401 INFO:tasks.workunit.client.1.smithi196.stdout:7/939: dwrite d14/d26/f83 [0,4194304] 0 2024-04-22T15:19:14.403 INFO:tasks.workunit.client.1.smithi196.stdout:2/952: mknod d1/d23/d27/d100/c153 0 2024-04-22T15:19:14.404 INFO:tasks.workunit.client.1.smithi196.stdout:2/953: write d1/d23/d27/d73/f12d [27308,34398] 0 2024-04-22T15:19:14.405 INFO:tasks.workunit.client.1.smithi196.stdout:4/932: mkdir d6/d13/d15/d2c/d62/d78/ddc/d128 0 2024-04-22T15:19:14.405 INFO:tasks.workunit.client.1.smithi196.stdout:2/954: fsync d1/fa2 0 2024-04-22T15:19:14.407 INFO:tasks.workunit.client.1.smithi196.stdout:6/962: write dd/f1c [3017631,51760] 0 2024-04-22T15:19:14.409 INFO:tasks.workunit.client.1.smithi196.stdout:6/963: chown dd/d1f/d38/d40/dd3/c10f 11722 1 2024-04-22T15:19:14.413 INFO:tasks.workunit.client.1.smithi196.stdout:0/982: dwrite d3/d6/dc/d8c/d8d/da8/fbe [0,4194304] 0 2024-04-22T15:19:14.416 INFO:tasks.workunit.client.1.smithi196.stdout:7/940: mknod d14/d4d/d125/c13b 0 2024-04-22T15:19:14.417 INFO:tasks.workunit.client.1.smithi196.stdout:2/955: creat d1/d13/d25/d2a/f154 x:0 0 0 2024-04-22T15:19:14.419 INFO:tasks.workunit.client.1.smithi196.stdout:2/956: fsync d1/d76/f12a 0 2024-04-22T15:19:14.419 INFO:tasks.workunit.client.1.smithi196.stdout:7/941: chown d14/cd7 3 1 2024-04-22T15:19:14.420 INFO:tasks.workunit.client.1.smithi196.stdout:7/942: chown d14/d4d/da3/de9/f117 2 1 2024-04-22T15:19:14.421 INFO:tasks.workunit.client.1.smithi196.stdout:0/983: symlink d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/d137/l156 0 2024-04-22T15:19:14.423 INFO:tasks.workunit.client.1.smithi196.stdout:7/943: dread - d14/d26/d2a/d2b/f12a zero size 2024-04-22T15:19:14.427 INFO:tasks.workunit.client.1.smithi196.stdout:8/924: write d2/f32 [719798,78375] 0 2024-04-22T15:19:14.428 INFO:tasks.workunit.client.1.smithi196.stdout:2/957: dwrite d1/ffc [0,4194304] 0 2024-04-22T15:19:14.429 INFO:tasks.workunit.client.1.smithi196.stdout:5/970: dwrite d1/d2/d98/dac/f131 [0,4194304] 0 2024-04-22T15:19:14.429 INFO:tasks.workunit.client.1.smithi196.stdout:7/944: rename d14/d15/fcc to d14/d26/db7/f13c 0 2024-04-22T15:19:14.439 INFO:tasks.workunit.client.1.smithi196.stdout:2/958: rmdir d1/da4 39 2024-04-22T15:19:14.444 INFO:tasks.workunit.client.1.smithi196.stdout:2/959: write d1/d23/d27/d89/fc1 [425236,69777] 0 2024-04-22T15:19:14.449 INFO:tasks.workunit.client.1.smithi196.stdout:0/984: dread d3/de/d14/f81 [0,4194304] 0 2024-04-22T15:19:14.453 INFO:tasks.workunit.client.1.smithi196.stdout:0/985: rmdir d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c 39 2024-04-22T15:19:14.454 INFO:tasks.workunit.client.1.smithi196.stdout:0/986: chown d3/d9/d36/d9f/dd1/d65/d7f/d87/c3c 1413260 1 2024-04-22T15:19:14.456 INFO:tasks.workunit.client.1.smithi196.stdout:5/971: dread d1/d17/f79 [4194304,4194304] 0 2024-04-22T15:19:14.457 INFO:tasks.workunit.client.1.smithi196.stdout:5/972: write d1/df1/d6e/f133 [552989,127208] 0 2024-04-22T15:19:14.459 INFO:tasks.workunit.client.1.smithi196.stdout:0/987: getdents d3/de/d14 0 2024-04-22T15:19:14.461 INFO:tasks.workunit.client.1.smithi196.stdout:7/945: write d14/d15/f96 [794980,98418] 0 2024-04-22T15:19:14.464 INFO:tasks.workunit.client.1.smithi196.stdout:2/960: truncate d1/d23/d27/d89/fc1 4072532 0 2024-04-22T15:19:14.465 INFO:tasks.workunit.client.1.smithi196.stdout:7/946: rmdir d14/d26/d28 39 2024-04-22T15:19:14.466 INFO:tasks.workunit.client.1.smithi196.stdout:2/961: read d1/f18 [3687073,52272] 0 2024-04-22T15:19:14.467 INFO:tasks.workunit.client.1.smithi196.stdout:8/925: dwrite d2/d5/f87 [0,4194304] 0 2024-04-22T15:19:14.470 INFO:tasks.workunit.client.1.smithi196.stdout:5/973: dwrite d1/d17/d1a/d2f/ffb [4194304,4194304] 0 2024-04-22T15:19:14.472 INFO:tasks.workunit.client.1.smithi196.stdout:0/988: dwrite d3/de/d1c/d27/f93 [0,4194304] 0 2024-04-22T15:19:14.473 INFO:tasks.workunit.client.1.smithi196.stdout:5/974: readlink d1/df1/d6e/d7a/ldc 0 2024-04-22T15:19:14.474 INFO:tasks.workunit.client.1.smithi196.stdout:2/962: symlink d1/d13/d1f/l155 0 2024-04-22T15:19:14.480 INFO:tasks.workunit.client.1.smithi196.stdout:8/926: dwrite d2/d35/d96/ff1 [0,4194304] 0 2024-04-22T15:19:14.481 INFO:tasks.workunit.client.1.smithi196.stdout:8/927: write d2/d5/faf [7282047,32080] 0 2024-04-22T15:19:14.484 INFO:tasks.workunit.client.1.smithi196.stdout:7/947: write d14/d15/f59 [562215,96782] 0 2024-04-22T15:19:14.485 INFO:tasks.workunit.client.1.smithi196.stdout:5/975: symlink d1/d14/d3e/l135 0 2024-04-22T15:19:14.490 INFO:tasks.workunit.client.1.smithi196.stdout:5/976: read d1/d4/f22 [3675234,79816] 0 2024-04-22T15:19:14.495 INFO:tasks.workunit.client.1.smithi196.stdout:8/928: rename d2/d8/d49/lf7 to d2/d5/d1a/db7/dd9/d68/dbc/l114 0 2024-04-22T15:19:14.496 INFO:tasks.workunit.client.1.smithi196.stdout:0/989: mkdir d3/d9/d36/d9f/dd1/d65/d7f/ddb/d157 0 2024-04-22T15:19:14.497 INFO:tasks.workunit.client.1.smithi196.stdout:8/929: truncate d2/dc/f105 823865 0 2024-04-22T15:19:14.498 INFO:tasks.workunit.client.1.smithi196.stdout:5/977: dwrite d1/d78/dcf/dff/d101/f127 [0,4194304] 0 2024-04-22T15:19:14.499 INFO:tasks.workunit.client.1.smithi196.stdout:5/978: stat d1/d17/d1c/c58 0 2024-04-22T15:19:14.500 INFO:tasks.workunit.client.1.smithi196.stdout:7/948: creat d14/d15/dc0/f13d x:0 0 0 2024-04-22T15:19:14.504 INFO:tasks.workunit.client.1.smithi196.stdout:8/930: creat d2/d5/d1a/db7/dd9/d68/d81/f115 x:0 0 0 2024-04-22T15:19:14.506 INFO:tasks.workunit.client.1.smithi196.stdout:2/963: dwrite d1/d13/d41/fd4 [4194304,4194304] 0 2024-04-22T15:19:14.507 INFO:tasks.workunit.client.1.smithi196.stdout:5/979: unlink d1/d17/d1c/f122 0 2024-04-22T15:19:14.509 INFO:tasks.workunit.client.1.smithi196.stdout:0/990: getdents d3/de/d1c/df1/d116 0 2024-04-22T15:19:14.520 INFO:tasks.workunit.client.1.smithi196.stdout:2/964: chown d1/d1d/c21 21598505 1 2024-04-22T15:19:14.532 INFO:tasks.workunit.client.1.smithi196.stdout:2/965: mknod d1/d13/d25/d9f/c156 0 2024-04-22T15:19:14.533 INFO:tasks.workunit.client.1.smithi196.stdout:2/966: chown d1/d23/d27/d73/d121 0 1 2024-04-22T15:19:14.533 INFO:tasks.workunit.client.1.smithi196.stdout:7/949: getdents d14/d26/d2a/d72 0 2024-04-22T15:19:14.534 INFO:tasks.workunit.client.1.smithi196.stdout:5/980: dwrite d1/d4/f22 [4194304,4194304] 0 2024-04-22T15:19:14.535 INFO:tasks.workunit.client.1.smithi196.stdout:8/931: dread d2/d3f/d4b/fcb [0,4194304] 0 2024-04-22T15:19:14.536 INFO:tasks.workunit.client.1.smithi196.stdout:5/981: write d1/df1/d5e/dcb/f104 [1836338,31899] 0 2024-04-22T15:19:14.537 INFO:tasks.workunit.client.1.smithi196.stdout:0/991: truncate d3/de/d14/ded/ffe 2925267 0 2024-04-22T15:19:14.537 INFO:tasks.workunit.client.1.smithi196.stdout:8/932: fdatasync d2/d5/d1a/db7/dd9/d68/d81/db8/fe4 0 2024-04-22T15:19:14.542 INFO:tasks.workunit.client.1.smithi196.stdout:2/967: mkdir d1/d23/d27/d2e/d55/d5b/d93/d157 0 2024-04-22T15:19:14.544 INFO:tasks.workunit.client.1.smithi196.stdout:0/992: dread - d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/fa9 zero size 2024-04-22T15:19:14.545 INFO:tasks.workunit.client.1.smithi196.stdout:8/933: mknod d2/d5/d1a/db7/dd9/d68/dbc/c116 0 2024-04-22T15:19:14.546 INFO:tasks.workunit.client.1.smithi196.stdout:8/934: write d2/d8/f27 [4463933,8206] 0 2024-04-22T15:19:14.546 INFO:tasks.workunit.client.1.smithi196.stdout:5/982: mkdir d1/df1/d6e/d7a/d11e/d126/d136 0 2024-04-22T15:19:14.547 INFO:tasks.workunit.client.1.smithi196.stdout:2/968: creat d1/d13/d41/f158 x:0 0 0 2024-04-22T15:19:14.548 INFO:tasks.workunit.client.1.smithi196.stdout:0/993: creat d3/de/d1c/d27/f158 x:0 0 0 2024-04-22T15:19:14.548 INFO:tasks.workunit.client.1.smithi196.stdout:7/950: getdents d14/d25/dd6 0 2024-04-22T15:19:14.550 INFO:tasks.workunit.client.1.smithi196.stdout:5/983: readlink d1/d4/lce 0 2024-04-22T15:19:14.550 INFO:tasks.workunit.client.1.smithi196.stdout:4/933: sync 2024-04-22T15:19:14.551 INFO:tasks.workunit.client.1.smithi196.stdout:7/951: fdatasync d14/d25/f90 0 2024-04-22T15:19:14.552 INFO:tasks.workunit.client.1.smithi196.stdout:2/969: getdents d1/d23/d27/d2e/d55/d5b/d93/d157 0 2024-04-22T15:19:14.555 INFO:tasks.workunit.client.1.smithi196.stdout:8/935: dwrite d2/d5/d1a/db7/dd9/d68/d81/f115 [0,4194304] 0 2024-04-22T15:19:14.556 INFO:tasks.workunit.client.1.smithi196.stdout:7/952: chown d14/d26/d28 209 1 2024-04-22T15:19:14.557 INFO:tasks.workunit.client.1.smithi196.stdout:4/934: symlink d6/de8/l129 0 2024-04-22T15:19:14.558 INFO:tasks.workunit.client.1.smithi196.stdout:8/936: read d2/d5/d1a/db7/dd9/d68/dbc/fed [2340069,105381] 0 2024-04-22T15:19:14.565 INFO:tasks.workunit.client.1.smithi196.stdout:4/935: dread - d6/d90/df9/fd5 zero size 2024-04-22T15:19:14.567 INFO:tasks.workunit.client.1.smithi196.stdout:7/953: unlink d14/d26/d28/d29/fb3 0 2024-04-22T15:19:14.568 INFO:tasks.workunit.client.1.smithi196.stdout:5/984: mknod d1/d14/d6a/d12a/c137 0 2024-04-22T15:19:14.579 INFO:tasks.workunit.client.1.smithi196.stdout:8/937: rename d2/d3f/c58 to d2/d5/d1a/db7/dd9/d68/c117 0 2024-04-22T15:19:14.580 INFO:tasks.workunit.client.1.smithi196.stdout:2/970: creat d1/f159 x:0 0 0 2024-04-22T15:19:14.585 INFO:tasks.workunit.client.1.smithi196.stdout:4/936: symlink d6/d13/d15/d2c/l12a 0 2024-04-22T15:19:14.585 INFO:tasks.workunit.client.1.smithi196.stdout:4/937: fdatasync d6/d14/d18/d40/d74/fec 0 2024-04-22T15:19:14.590 INFO:tasks.workunit.client.1.smithi196.stdout:5/985: write d1/d2/d98/dac/d105/f75 [3030546,78470] 0 2024-04-22T15:19:14.596 INFO:tasks.workunit.client.1.smithi196.stdout:7/954: getdents d14/d26/d2a/d72 0 2024-04-22T15:19:14.597 INFO:tasks.workunit.client.1.smithi196.stdout:8/938: creat d2/d10a/f118 x:0 0 0 2024-04-22T15:19:14.601 INFO:tasks.workunit.client.1.smithi196.stdout:7/955: symlink d14/d25/d31/l13e 0 2024-04-22T15:19:14.601 INFO:tasks.workunit.client.1.smithi196.stdout:2/971: write d1/f3e [2973088,129869] 0 2024-04-22T15:19:14.603 INFO:tasks.workunit.client.1.smithi196.stdout:4/938: write d6/d13/d15/d2c/d62/d78/d8d/fdb [669048,22153] 0 2024-04-22T15:19:14.607 INFO:tasks.workunit.client.1.smithi196.stdout:5/986: dread d1/d17/d1a/f43 [0,4194304] 0 2024-04-22T15:19:14.609 INFO:tasks.workunit.client.1.smithi196.stdout:8/939: getdents d2/d5/d1a/db7/dd9/d68 0 2024-04-22T15:19:14.609 INFO:tasks.workunit.client.1.smithi196.stdout:7/956: dwrite d14/d25/d80/fbd [0,4194304] 0 2024-04-22T15:19:14.611 INFO:tasks.workunit.client.1.smithi196.stdout:8/940: write d2/d8/d49/d6c/fc6 [1511305,25809] 0 2024-04-22T15:19:14.611 INFO:tasks.workunit.client.1.smithi196.stdout:4/939: truncate d6/d13/d15/d2c/d62/f96 1908166 0 2024-04-22T15:19:14.612 INFO:tasks.workunit.client.1.smithi196.stdout:5/987: unlink d1/df1/d6e/d7a/lee 0 2024-04-22T15:19:14.612 INFO:tasks.workunit.client.1.smithi196.stdout:7/957: write d14/d26/d2a/d72/db1/fec [7378968,117256] 0 2024-04-22T15:19:14.613 INFO:tasks.workunit.client.1.smithi196.stdout:7/958: chown d14/d26/d5e 4115597 1 2024-04-22T15:19:14.615 INFO:tasks.workunit.client.1.smithi196.stdout:5/988: write d1/df1/d6e/fb9 [4819300,34270] 0 2024-04-22T15:19:14.616 INFO:tasks.workunit.client.1.smithi196.stdout:8/941: mknod d2/d5/d1a/db7/dd9/de1/c119 0 2024-04-22T15:19:14.617 INFO:tasks.workunit.client.1.smithi196.stdout:5/989: chown d1/d2/d98/dac/d105/da9/l85 1116 1 2024-04-22T15:19:14.619 INFO:tasks.workunit.client.1.smithi196.stdout:7/959: creat d14/d26/db7/d10b/f13f x:0 0 0 2024-04-22T15:19:14.624 INFO:tasks.workunit.client.1.smithi196.stdout:7/960: creat d14/d4d/f140 x:0 0 0 2024-04-22T15:19:14.625 INFO:tasks.workunit.client.1.smithi196.stdout:2/972: dread d1/d1d/f3c [0,4194304] 0 2024-04-22T15:19:14.626 INFO:tasks.workunit.client.1.smithi196.stdout:7/961: fsync d14/d26/d2a/d72/fde 0 2024-04-22T15:19:14.627 INFO:tasks.workunit.client.1.smithi196.stdout:8/942: rename d2/d5/d1a/db7/dd9/d68/c78 to d2/d5/d1a/db7/dd9/d68/d81/db8/c11a 0 2024-04-22T15:19:14.630 INFO:tasks.workunit.client.1.smithi196.stdout:2/973: mkdir d1/d23/d27/d73/d121/d15a 0 2024-04-22T15:19:14.631 INFO:tasks.workunit.client.1.smithi196.stdout:2/974: dread - d1/d13/d101/f148 zero size 2024-04-22T15:19:14.632 INFO:tasks.workunit.client.1.smithi196.stdout:5/990: write d1/df1/d6e/d7a/fa3 [5002152,91379] 0 2024-04-22T15:19:14.634 INFO:tasks.workunit.client.1.smithi196.stdout:2/975: creat d1/d13/d1f/f15b x:0 0 0 2024-04-22T15:19:14.636 INFO:tasks.workunit.client.1.smithi196.stdout:2/976: read d1/ff [4703706,1290] 0 2024-04-22T15:19:14.637 INFO:tasks.workunit.client.1.smithi196.stdout:5/991: creat d1/df1/d6e/d7a/d11e/d126/f138 x:0 0 0 2024-04-22T15:19:14.637 INFO:tasks.workunit.client.1.smithi196.stdout:4/940: dread d6/d14/f87 [0,4194304] 0 2024-04-22T15:19:14.640 INFO:tasks.workunit.client.1.smithi196.stdout:8/943: write d2/d8/d49/d8b/f99 [306751,8657] 0 2024-04-22T15:19:14.643 INFO:tasks.workunit.client.1.smithi196.stdout:7/962: dwrite d14/d26/d2a/d72/db1/fc5 [0,4194304] 0 2024-04-22T15:19:14.645 INFO:tasks.workunit.client.1.smithi196.stdout:4/941: creat d6/d13/d15/d2c/d8c/db3/dce/f12b x:0 0 0 2024-04-22T15:19:14.647 INFO:tasks.workunit.client.1.smithi196.stdout:4/942: stat d6/d13/d15/d2c/d62/f88 0 2024-04-22T15:19:14.648 INFO:tasks.workunit.client.1.smithi196.stdout:8/944: creat d2/d8/d49/d6c/dde/f11b x:0 0 0 2024-04-22T15:19:14.648 INFO:tasks.workunit.client.1.smithi196.stdout:7/963: creat d14/d25/d31/df0/f141 x:0 0 0 2024-04-22T15:19:14.650 INFO:tasks.workunit.client.1.smithi196.stdout:4/943: creat d6/d13/f12c x:0 0 0 2024-04-22T15:19:14.651 INFO:tasks.workunit.client.1.smithi196.stdout:2/977: getdents d1/d23/d27/d2e/d55/d5b/d93/d97/d11e 0 2024-04-22T15:19:14.655 INFO:tasks.workunit.client.1.smithi196.stdout:7/964: unlink d14/d15/fef 0 2024-04-22T15:19:14.655 INFO:tasks.workunit.client.1.smithi196.stdout:5/992: dread d1/d17/d7f/fe0 [0,4194304] 0 2024-04-22T15:19:14.657 INFO:tasks.workunit.client.1.smithi196.stdout:4/944: dread d6/d14/d18/d26/f6e [0,4194304] 0 2024-04-22T15:19:14.662 INFO:tasks.workunit.client.1.smithi196.stdout:4/945: write d6/d14/d18/d26/d32/d67/f126 [419317,703] 0 2024-04-22T15:19:14.662 INFO:tasks.workunit.client.1.smithi196.stdout:2/978: symlink d1/d13/d25/d2a/ddb/dff/d138/l15c 0 2024-04-22T15:19:14.662 INFO:tasks.workunit.client.1.smithi196.stdout:7/965: rename d14/d15/f114 to d14/d4d/f142 0 2024-04-22T15:19:14.663 INFO:tasks.workunit.client.1.smithi196.stdout:5/993: creat d1/df1/d6e/d7a/d11e/d126/f139 x:0 0 0 2024-04-22T15:19:14.664 INFO:tasks.workunit.client.1.smithi196.stdout:4/946: dwrite d6/d14/d18/d26/d32/fcb [0,4194304] 0 2024-04-22T15:19:14.667 INFO:tasks.workunit.client.1.smithi196.stdout:4/947: write d6/d13/d15/d2c/d8c/db3/dce/f10a [912901,49058] 0 2024-04-22T15:19:14.670 INFO:tasks.workunit.client.1.smithi196.stdout:6/964: sync 2024-04-22T15:19:14.672 INFO:tasks.workunit.client.1.smithi196.stdout:7/966: symlink d14/d26/d2a/l143 0 2024-04-22T15:19:14.674 INFO:tasks.workunit.client.1.smithi196.stdout:4/948: dread d6/d14/d18/d26/d32/d53/ff2 [0,4194304] 0 2024-04-22T15:19:14.676 INFO:tasks.workunit.client.1.smithi196.stdout:5/994: dwrite d1/d2/d98/dac/d123/f12b [0,4194304] 0 2024-04-22T15:19:14.677 INFO:tasks.workunit.client.1.smithi196.stdout:8/945: dwrite d2/dc/fc7 [0,4194304] 0 2024-04-22T15:19:14.681 INFO:tasks.workunit.client.1.smithi196.stdout:7/967: mkdir d14/d25/d31/df0/d144 0 2024-04-22T15:19:14.684 INFO:tasks.workunit.client.1.smithi196.stdout:2/979: dwrite d1/f15 [0,4194304] 0 2024-04-22T15:19:14.685 INFO:tasks.workunit.client.1.smithi196.stdout:7/968: fdatasync d14/d15/d91/fe7 0 2024-04-22T15:19:14.685 INFO:tasks.workunit.client.1.smithi196.stdout:8/946: readlink d2/d3f/l52 0 2024-04-22T15:19:14.686 INFO:tasks.workunit.client.1.smithi196.stdout:4/949: mknod d6/d13/d15/d2c/d62/d78/df6/c12d 0 2024-04-22T15:19:14.687 INFO:tasks.workunit.client.1.smithi196.stdout:6/965: mknod dd/d20/d5c/d6a/c120 0 2024-04-22T15:19:14.690 INFO:tasks.workunit.client.1.smithi196.stdout:2/980: mkdir d1/d13/d41/d15d 0 2024-04-22T15:19:14.690 INFO:tasks.workunit.client.1.smithi196.stdout:4/950: read d6/d14/d18/d26/d32/d67/fc8 [512632,51269] 0 2024-04-22T15:19:14.693 INFO:tasks.workunit.client.1.smithi196.stdout:6/966: chown dd/la1 29124541 1 2024-04-22T15:19:14.695 INFO:tasks.workunit.client.1.smithi196.stdout:7/969: dwrite d14/d26/fa2 [0,4194304] 0 2024-04-22T15:19:14.700 INFO:tasks.workunit.client.1.smithi196.stdout:0/994: sync 2024-04-22T15:19:14.701 INFO:tasks.workunit.client.1.smithi196.stdout:2/981: rmdir d1/d23/d27/d2e/d55/d5b/d93 39 2024-04-22T15:19:14.701 INFO:tasks.workunit.client.1.smithi196.stdout:6/967: dwrite dd/d1f/d4e/d53/d7f/da5/fb1 [0,4194304] 0 2024-04-22T15:19:14.704 INFO:tasks.workunit.client.1.smithi196.stdout:5/995: dwrite d1/d2/fc6 [0,4194304] 0 2024-04-22T15:19:14.710 INFO:tasks.workunit.client.1.smithi196.stdout:8/947: dwrite d2/dc/d45/d98/fa6 [0,4194304] 0 2024-04-22T15:19:14.714 INFO:tasks.workunit.client.1.smithi196.stdout:6/968: mknod dd/d1f/d38/dd9/c121 0 2024-04-22T15:19:14.715 INFO:tasks.workunit.client.1.smithi196.stdout:4/951: dwrite d6/d14/d18/d26/d32/d67/d9a/da5/fac [0,4194304] 0 2024-04-22T15:19:14.717 INFO:tasks.workunit.client.1.smithi196.stdout:2/982: creat d1/d23/d27/d2e/db5/d102/f15e x:0 0 0 2024-04-22T15:19:14.724 INFO:tasks.workunit.client.1.smithi196.stdout:8/948: dread d2/d5/d1a/db7/dd9/d68/dbc/f8a [0,4194304] 0 2024-04-22T15:19:14.724 INFO:tasks.workunit.client.1.smithi196.stdout:5/996: unlink d1/d4/f22 0 2024-04-22T15:19:14.726 INFO:tasks.workunit.client.1.smithi196.stdout:6/969: dwrite dd/d20/fec [0,4194304] 0 2024-04-22T15:19:14.733 INFO:tasks.workunit.client.1.smithi196.stdout:6/970: dwrite dd/d1f/d37/f66 [4194304,4194304] 0 2024-04-22T15:19:14.741 INFO:tasks.workunit.client.1.smithi196.stdout:7/970: write d14/d26/fd2 [1626812,65232] 0 2024-04-22T15:19:14.743 INFO:tasks.workunit.client.1.smithi196.stdout:7/971: write d14/d15/f59 [386206,87445] 0 2024-04-22T15:19:14.744 INFO:tasks.workunit.client.1.smithi196.stdout:4/952: dread d6/d14/d18/d26/d32/da3/db7/dc9/fda [0,4194304] 0 2024-04-22T15:19:14.747 INFO:tasks.workunit.client.1.smithi196.stdout:5/997: symlink d1/d14/d6a/d12a/l13a 0 2024-04-22T15:19:14.748 INFO:tasks.workunit.client.1.smithi196.stdout:8/949: creat d2/d5/d1a/db7/dd9/de1/f11c x:0 0 0 2024-04-22T15:19:14.748 INFO:tasks.workunit.client.1.smithi196.stdout:5/998: read d1/d17/f79 [8066678,38014] 0 2024-04-22T15:19:14.750 INFO:tasks.workunit.client.1.smithi196.stdout:8/950: dread - d2/d5/d1a/db7/dd9/d68/d81/db8/fe4 zero size 2024-04-22T15:19:14.764 INFO:tasks.workunit.client.1.smithi196.stdout:5/999: creat d1/d2/dbb/df9/f13b x:0 0 0 2024-04-22T15:19:14.764 INFO:tasks.workunit.client.1.smithi196.stdout:0/995: getdents d3/d9/d36/d9f/dd1/d65/d7f/d87/d6c/d88/d95/dd0/de3 0 2024-04-22T15:19:14.766 INFO:tasks.workunit.client.1.smithi196.stdout:2/983: getdents d1/d13/d25/d9f/ddd 0 2024-04-22T15:19:14.768 INFO:tasks.workunit.client.1.smithi196.stdout:0/996: truncate d3/de/d1c/d27/f32 722988 0 2024-04-22T15:19:14.770 INFO:tasks.workunit.client.1.smithi196.stdout:8/951: getdents d2/dc 0 2024-04-22T15:19:14.771 INFO:tasks.workunit.client.1.smithi196.stdout:8/952: chown d2/d5/d1a/db7/dd9/d68/dbc/da4/ld4 785535 1 2024-04-22T15:19:14.774 INFO:tasks.workunit.client.1.smithi196.stdout:0/997: dwrite d3/de/f7d [0,4194304] 0 2024-04-22T15:19:14.776 INFO:tasks.workunit.client.1.smithi196.stdout:8/953: creat d2/dc/d45/d98/f11d x:0 0 0 2024-04-22T15:19:14.782 INFO:tasks.workunit.client.1.smithi196.stdout:0/998: symlink d3/d6/dc/d8c/d8d/d98/d10d/l159 0 2024-04-22T15:19:14.784 INFO:tasks.workunit.client.1.smithi196.stdout:4/953: dread d6/d14/d18/d40/f6c [0,4194304] 0 2024-04-22T15:19:14.784 INFO:tasks.workunit.client.1.smithi196.stdout:6/971: dwrite dd/d20/d4d/f8c [0,4194304] 0 2024-04-22T15:19:14.785 INFO:tasks.workunit.client.1.smithi196.stdout:6/972: stat dd/d1f/d38/f94 0 2024-04-22T15:19:14.785 INFO:tasks.workunit.client.1.smithi196.stdout:7/972: dwrite fd [0,4194304] 0 2024-04-22T15:19:14.789 INFO:tasks.workunit.client.1.smithi196.stdout:6/973: chown dd/d1f/d37/d7b/fb4 356110 1 2024-04-22T15:19:14.790 INFO:tasks.workunit.client.1.smithi196.stdout:4/954: symlink d6/d13/d15/d2c/d62/d78/df6/d111/l12e 0 2024-04-22T15:19:14.792 INFO:tasks.workunit.client.1.smithi196.stdout:7/973: rename d14/d4d/f142 to d14/d26/d28/d29/f145 0 2024-04-22T15:19:14.794 INFO:tasks.workunit.client.1.smithi196.stdout:4/955: rmdir d6/d14/d18/d26/d32 39 2024-04-22T15:19:14.796 INFO:tasks.workunit.client.1.smithi196.stdout:6/974: fsync fc 0 2024-04-22T15:19:14.798 INFO:tasks.workunit.client.1.smithi196.stdout:6/975: chown dd/d1f/d4e/d53 1928800 1 2024-04-22T15:19:14.798 INFO:tasks.workunit.client.1.smithi196.stdout:7/974: mknod d14/d15/dc0/c146 0 2024-04-22T15:19:14.799 INFO:tasks.workunit.client.1.smithi196.stdout:6/976: fdatasync dd/d1f/d38/d40/fcd 0 2024-04-22T15:19:14.801 INFO:tasks.workunit.client.1.smithi196.stdout:6/977: chown dd/d1f/d4e/d53/d55/d6d/lc5 368592 1 2024-04-22T15:19:14.802 INFO:tasks.workunit.client.1.smithi196.stdout:6/978: dread - dd/ddf/f11a zero size 2024-04-22T15:19:14.803 INFO:tasks.workunit.client.1.smithi196.stdout:4/956: link d6/d14/d18/d26/d32/da3/db7/dc9/fda d6/d14/d18/d26/d32/d67/d9a/da5/d11d/f12f 0 2024-04-22T15:19:14.804 INFO:tasks.workunit.client.1.smithi196.stdout:6/979: rmdir dd/d1f/d38 39 2024-04-22T15:19:14.804 INFO:tasks.workunit.client.1.smithi196.stdout:6/980: chown f5 937 1 2024-04-22T15:19:14.810 INFO:tasks.workunit.client.1.smithi196.stdout:8/954: dread d2/d6a/d7b/fc5 [0,4194304] 0 2024-04-22T15:19:14.810 INFO:tasks.workunit.client.1.smithi196.stdout:4/957: dwrite d6/d14/d18/d26/d32/d53/f121 [0,4194304] 0 2024-04-22T15:19:14.817 INFO:tasks.workunit.client.1.smithi196.stdout:4/958: dread d6/d14/d18/d26/d32/fcb [0,4194304] 0 2024-04-22T15:19:14.818 INFO:tasks.workunit.client.1.smithi196.stdout:4/959: chown d6/d13/f24 67 1 2024-04-22T15:19:14.819 INFO:tasks.workunit.client.1.smithi196.stdout:8/955: dwrite d2/d35/d96/ff1 [0,4194304] 0 2024-04-22T15:19:14.823 INFO:tasks.workunit.client.1.smithi196.stdout:4/960: read d6/d13/d15/d2c/d62/f96 [513795,16432] 0 2024-04-22T15:19:14.824 INFO:tasks.workunit.client.1.smithi196.stdout:8/956: symlink d2/d5/d1a/db7/l11e 0 2024-04-22T15:19:14.825 INFO:tasks.workunit.client.1.smithi196.stdout:4/961: write d6/d14/d18/d26/d32/da3/ff0 [4904277,122809] 0 2024-04-22T15:19:14.831 INFO:tasks.workunit.client.1.smithi196.stdout:4/962: fdatasync d6/d13/d15/d2c/d35/f9d 0 2024-04-22T15:19:14.832 INFO:tasks.workunit.client.1.smithi196.stdout:6/981: truncate dd/d1f/d37/f66 3746834 0 2024-04-22T15:19:14.836 INFO:tasks.workunit.client.1.smithi196.stdout:4/963: dwrite d6/d13/f6b [0,4194304] 0 2024-04-22T15:19:14.847 INFO:tasks.workunit.client.1.smithi196.stdout:8/957: truncate d2/dc/fc7 1857467 0 2024-04-22T15:19:14.848 INFO:tasks.workunit.client.1.smithi196.stdout:8/958: chown d2/d5/cc4 64461 1 2024-04-22T15:19:14.850 INFO:tasks.workunit.client.1.smithi196.stdout:8/959: fsync d2/d8/f25 0 2024-04-22T15:19:14.852 INFO:tasks.workunit.client.1.smithi196.stdout:6/982: truncate dd/d1f/d4e/d53/d55/d6d/dd1/de3/df7/df9/f101 1044763 0 2024-04-22T15:19:14.861 INFO:tasks.workunit.client.1.smithi196.stdout:6/983: mknod dd/d20/d5c/d5e/c122 0 2024-04-22T15:19:14.863 INFO:tasks.workunit.client.1.smithi196.stdout:8/960: write d2/d5/d1a/db7/dd9/d68/dbc/fed [4070189,98943] 0 2024-04-22T15:19:14.864 INFO:tasks.workunit.client.1.smithi196.stdout:6/984: stat dd/d20/d5c/fb7 0 2024-04-22T15:19:14.870 INFO:tasks.workunit.client.1.smithi196.stdout:8/961: creat d2/d5/d93/dd5/f11f x:0 0 0 2024-04-22T15:19:14.872 INFO:tasks.workunit.client.1.smithi196.stdout:8/962: write d2/d8/d49/d6c/fc6 [2026876,96852] 0 2024-04-22T15:19:14.875 INFO:tasks.workunit.client.1.smithi196.stdout:8/963: creat d2/d5/d2e/d10c/d111/f120 x:0 0 0 2024-04-22T15:19:14.880 INFO:tasks.workunit.client.1.smithi196.stdout:8/964: dwrite d2/d8/d49/fff [0,4194304] 0 2024-04-22T15:19:14.883 INFO:tasks.workunit.client.1.smithi196.stdout:6/985: dwrite dd/d1f/d38/d40/dd3/f100 [0,4194304] 0 2024-04-22T15:19:14.887 INFO:tasks.workunit.client.1.smithi196.stdout:6/986: mkdir dd/d1f/d4e/d53/d55/d6d/dd1/d123 0 2024-04-22T15:19:14.889 INFO:tasks.workunit.client.1.smithi196.stdout:6/987: chown dd/d1f/d4e/d53/d55/d6d/dd1/de3/df7 9188 1 2024-04-22T15:19:14.891 INFO:tasks.workunit.client.1.smithi196.stdout:6/988: write dd/d1f/d4e/d53/d7f/f83 [3228444,45487] 0 2024-04-22T15:19:14.896 INFO:tasks.workunit.client.1.smithi196.stdout:6/989: mkdir dd/d1f/d124 0 2024-04-22T15:19:14.897 INFO:tasks.workunit.client.1.smithi196.stdout:6/990: truncate dd/d1f/d38/d108/ffa 809694 0 2024-04-22T15:19:14.902 INFO:tasks.workunit.client.1.smithi196.stdout:6/991: dwrite dd/d70/f10d [0,4194304] 0 2024-04-22T15:19:14.903 INFO:tasks.workunit.client.1.smithi196.stdout:8/965: dread d2/d8/d49/d6c/fc6 [0,4194304] 0 2024-04-22T15:19:14.905 INFO:tasks.workunit.client.1.smithi196.stdout:6/992: symlink dd/d1f/d4e/l125 0 2024-04-22T15:19:14.906 INFO:tasks.workunit.client.1.smithi196.stdout:2/984: sync 2024-04-22T15:19:14.907 INFO:tasks.workunit.client.1.smithi196.stdout:2/985: readlink d1/d13/d1f/lb6 0 2024-04-22T15:19:14.907 INFO:tasks.workunit.client.1.smithi196.stdout:8/966: chown d2/d5/d1a/db7/dd9/d68/c117 214 1 2024-04-22T15:19:14.914 INFO:tasks.workunit.client.1.smithi196.stdout:6/993: link dd/f16 dd/d2b/d8f/d107/f126 0 2024-04-22T15:19:14.915 INFO:tasks.workunit.client.1.smithi196.stdout:8/967: creat d2/d5/d1a/db7/dd9/d68/dbc/dae/df5/f121 x:0 0 0 2024-04-22T15:19:14.916 INFO:tasks.workunit.client.1.smithi196.stdout:8/968: write d2/f32 [2165295,75799] 0 2024-04-22T15:19:14.916 INFO:tasks.workunit.client.1.smithi196.stdout:6/994: mknod dd/d1f/d4e/d53/d55/d6d/dd1/dca/c127 0 2024-04-22T15:19:14.917 INFO:tasks.workunit.client.1.smithi196.stdout:6/995: chown dd/d1f/d4e/d53/d7f/dea/lf8 12704 1 2024-04-22T15:19:14.921 INFO:tasks.workunit.client.1.smithi196.stdout:6/996: creat dd/d70/d10a/f128 x:0 0 0 2024-04-22T15:19:14.927 INFO:tasks.workunit.client.1.smithi196.stdout:2/986: dread d1/d23/d27/d2e/d55/f36 [0,4194304] 0 2024-04-22T15:19:14.938 INFO:tasks.workunit.client.1.smithi196.stdout:6/997: dread dd/d1f/d38/d40/f4f [0,4194304] 0 2024-04-22T15:19:14.942 INFO:tasks.workunit.client.1.smithi196.stdout:2/987: dread d1/d23/d27/d100/f125 [0,4194304] 0 2024-04-22T15:19:14.949 INFO:tasks.workunit.client.1.smithi196.stdout:2/988: dwrite d1/d13/d41/fd4 [4194304,4194304] 0 2024-04-22T15:19:14.984 INFO:tasks.workunit.client.1.smithi196.stdout:4/964: sync 2024-04-22T15:19:14.985 INFO:tasks.workunit.client.1.smithi196.stdout:7/975: sync 2024-04-22T15:19:14.985 INFO:tasks.workunit.client.1.smithi196.stdout:0/999: sync 2024-04-22T15:19:14.988 INFO:tasks.workunit.client.1.smithi196.stdout:4/965: read d6/d14/d18/d26/d32/da3/ff0 [841095,110955] 0 2024-04-22T15:19:15.029 INFO:tasks.workunit.client.1.smithi196.stdout:7/976: creat d14/d26/d2a/f147 x:0 0 0 2024-04-22T15:19:15.032 INFO:tasks.workunit.client.1.smithi196.stdout:7/977: stat d14/d26/db7/lc9 0 2024-04-22T15:19:15.035 INFO:tasks.workunit.client.1.smithi196.stdout:8/969: dwrite d2/dc/fe [0,4194304] 0 2024-04-22T15:19:15.037 INFO:tasks.workunit.client.1.smithi196.stdout:7/978: unlink d14/d26/d2a/d2b/lb0 0 2024-04-22T15:19:15.038 INFO:tasks.workunit.client.1.smithi196.stdout:8/970: dread - d2/d5/d2e/d10c/d111/f120 zero size 2024-04-22T15:19:15.038 INFO:tasks.workunit.client.1.smithi196.stdout:6/998: dread dd/d1f/d4e/d53/d55/f8b [0,4194304] 0 2024-04-22T15:19:15.051 INFO:tasks.workunit.client.1.smithi196.stdout:4/966: dread d6/f19 [0,4194304] 0 2024-04-22T15:19:15.052 INFO:tasks.workunit.client.1.smithi196.stdout:4/967: stat d6/d14/d18/d26/d32/da3/db7/lf5 0 2024-04-22T15:19:15.111 INFO:tasks.workunit.client.1.smithi196.stdout:4/968: link c2 d6/d14/d18/d40/c130 0 2024-04-22T15:19:15.111 INFO:tasks.workunit.client.1.smithi196.stdout:4/969: readlink d6/d14/d18/d40/lfe 0 2024-04-22T15:19:15.115 INFO:tasks.workunit.client.1.smithi196.stdout:2/989: rename d1/d23/d27/d2e/db5/d102/l113 to d1/d23/d27/d2e/d55/d5b/d7c/l15f 0 2024-04-22T15:19:15.117 INFO:tasks.workunit.client.1.smithi196.stdout:2/990: write d1/d76/fb9 [1618027,63512] 0 2024-04-22T15:19:15.120 INFO:tasks.workunit.client.1.smithi196.stdout:2/991: unlink d1/d1d/f136 0 2024-04-22T15:19:15.128 INFO:tasks.workunit.client.1.smithi196.stdout:2/992: dwrite d1/d23/d27/d2e/d55/f11f [0,4194304] 0 2024-04-22T15:19:15.134 INFO:tasks.workunit.client.1.smithi196.stdout:7/979: truncate d14/d25/d31/f38 1201426 0 2024-04-22T15:19:15.138 INFO:tasks.workunit.client.1.smithi196.stdout:8/971: dwrite d2/d5/d1a/db7/dd9/d68/dbc/f8a [0,4194304] 0 2024-04-22T15:19:15.139 INFO:tasks.workunit.client.1.smithi196.stdout:2/993: dwrite d1/d23/d27/d2e/d55/f87 [0,4194304] 0 2024-04-22T15:19:15.141 INFO:tasks.workunit.client.1.smithi196.stdout:7/980: symlink d14/d26/d5e/l148 0 2024-04-22T15:19:15.147 INFO:tasks.workunit.client.1.smithi196.stdout:2/994: creat d1/d76/f160 x:0 0 0 2024-04-22T15:19:15.150 INFO:tasks.workunit.client.1.smithi196.stdout:8/972: dwrite d2/d5/faf [4194304,4194304] 0 2024-04-22T15:19:15.153 INFO:tasks.workunit.client.1.smithi196.stdout:8/973: mkdir d2/dc/d45/d98/d122 0 2024-04-22T15:19:15.167 INFO:tasks.workunit.client.1.smithi196.stdout:2/995: unlink d1/d23/d27/d2e/d55/l134 0 2024-04-22T15:19:15.168 INFO:tasks.workunit.client.1.smithi196.stdout:6/999: sync 2024-04-22T15:19:15.169 INFO:tasks.workunit.client.1.smithi196.stdout:8/974: creat d2/d5/d1a/db7/dd9/d68/f123 x:0 0 0 2024-04-22T15:19:15.172 INFO:tasks.workunit.client.1.smithi196.stdout:8/975: creat d2/d5/d1a/db7/dd9/d68/dbc/f124 x:0 0 0 2024-04-22T15:19:15.173 INFO:tasks.workunit.client.1.smithi196.stdout:8/976: write d2/d5/d1a/db7/dd9/d68/f123 [483328,118680] 0 2024-04-22T15:19:15.175 INFO:tasks.workunit.client.1.smithi196.stdout:4/970: rename d6/d13/d15/f5e to d6/d14/f131 0 2024-04-22T15:19:15.176 INFO:tasks.workunit.client.1.smithi196.stdout:8/977: mkdir d2/d5/d4f/d125 0 2024-04-22T15:19:15.177 INFO:tasks.workunit.client.1.smithi196.stdout:7/981: rename d14/d26/d2a/d2b/f45 to d14/d26/d5a/d137/f149 0 2024-04-22T15:19:15.178 INFO:tasks.workunit.client.1.smithi196.stdout:8/978: symlink d2/d6a/l126 0 2024-04-22T15:19:15.179 INFO:tasks.workunit.client.1.smithi196.stdout:8/979: truncate d2/fe3 1462836 0 2024-04-22T15:19:15.180 INFO:tasks.workunit.client.1.smithi196.stdout:2/996: rename d1/d23/d27/d73/d121 to d1/d13/d41/d15d/d161 0 2024-04-22T15:19:15.182 INFO:tasks.workunit.client.1.smithi196.stdout:2/997: write d1/d13/d25/ff9 [3147993,112459] 0 2024-04-22T15:19:15.209 INFO:tasks.workunit.client.1.smithi196.stdout:7/982: rename c3 to d14/d15/d91/d9a/d130/c14a 0 2024-04-22T15:19:15.210 INFO:tasks.workunit.client.1.smithi196.stdout:7/983: write d14/d26/fa2 [2598080,79238] 0 2024-04-22T15:19:15.211 INFO:tasks.workunit.client.1.smithi196.stdout:7/984: write d14/d26/d2a/d2b/f2d [325363,47730] 0 2024-04-22T15:19:15.232 INFO:tasks.workunit.client.1.smithi196.stdout:7/985: symlink d14/d26/db7/l14b 0 2024-04-22T15:19:15.233 INFO:tasks.workunit.client.1.smithi196.stdout:7/986: read - d14/d26/db7/d10b/f13f zero size 2024-04-22T15:19:15.238 INFO:tasks.workunit.client.1.smithi196.stdout:8/980: rename d2/d6a/d7b/c113 to d2/dc/d45/c127 0 2024-04-22T15:19:15.239 INFO:tasks.workunit.client.1.smithi196.stdout:7/987: mknod d14/d15/d91/d9a/c14c 0 2024-04-22T15:19:15.242 INFO:tasks.workunit.client.1.smithi196.stdout:2/998: rename d1/d23/f2f to d1/d23/d27/d2e/d55/d5b/d7c/f162 0 2024-04-22T15:19:15.243 INFO:tasks.workunit.client.1.smithi196.stdout:7/988: unlink d14/d25/d31/ff7 0 2024-04-22T15:19:15.245 INFO:tasks.workunit.client.1.smithi196.stdout:2/999: symlink d1/d13/d1f/l163 0 2024-04-22T15:19:15.249 INFO:tasks.workunit.client.1.smithi196.stdout:8/981: dread d2/d5/d1a/db7/dd9/d68/f7c [0,4194304] 0 2024-04-22T15:19:15.249 INFO:tasks.workunit.client.1.smithi196.stdout:8/982: dread - d2/d5/d4f/fee zero size 2024-04-22T15:19:15.251 INFO:tasks.workunit.client.1.smithi196.stdout:7/989: creat d14/d25/f14d x:0 0 0 2024-04-22T15:19:15.252 INFO:tasks.workunit.client.1.smithi196.stdout:8/983: truncate d2/d8/f76 4445757 0 2024-04-22T15:19:15.252 INFO:tasks.workunit.client.1.smithi196.stdout:7/990: read d14/d26/d5a/dbb/f111 [382686,24078] 0 2024-04-22T15:19:15.257 INFO:tasks.workunit.client.1.smithi196.stdout:7/991: dwrite d14/d25/d31/f8e [0,4194304] 0 2024-04-22T15:19:15.258 INFO:tasks.workunit.client.1.smithi196.stdout:7/992: read - d14/d26/d28/d29/f145 zero size 2024-04-22T15:19:15.284 INFO:tasks.workunit.client.1.smithi196.stdout:8/984: write d2/d5/d2e/fbe [933874,130949] 0 2024-04-22T15:19:15.287 INFO:tasks.workunit.client.1.smithi196.stdout:8/985: symlink d2/d5/d1a/db7/dd9/d68/l128 0 2024-04-22T15:19:15.290 INFO:tasks.workunit.client.1.smithi196.stdout:7/993: dwrite d14/ddd/f132 [0,4194304] 0 2024-04-22T15:19:15.294 INFO:tasks.workunit.client.1.smithi196.stdout:4/971: sync 2024-04-22T15:19:15.300 INFO:tasks.workunit.client.1.smithi196.stdout:7/994: dwrite fc [4194304,4194304] 0 2024-04-22T15:19:15.301 INFO:tasks.workunit.client.1.smithi196.stdout:7/995: write d14/d15/dc0/f13d [546981,26160] 0 2024-04-22T15:19:15.306 INFO:tasks.workunit.client.1.smithi196.stdout:7/996: unlink d14/d15/d91/l11f 0 2024-04-22T15:19:15.310 INFO:tasks.workunit.client.1.smithi196.stdout:7/997: fsync d14/d4d/da3/de9/ff3 0 2024-04-22T15:19:15.324 INFO:tasks.workunit.client.1.smithi196.stdout:8/986: write d2/d35/fb4 [733750,76822] 0 2024-04-22T15:19:15.324 INFO:tasks.workunit.client.1.smithi196.stdout:4/972: dread d6/d13/d15/d2c/d62/d78/d8b/fd2 [0,4194304] 0 2024-04-22T15:19:15.325 INFO:tasks.workunit.client.1.smithi196.stdout:4/973: stat d6/d14/d18/d26/d32/d53/ff2 0 2024-04-22T15:19:15.326 INFO:tasks.workunit.client.1.smithi196.stdout:8/987: rmdir d2/d5/d1a/db7 39 2024-04-22T15:19:15.330 INFO:tasks.workunit.client.1.smithi196.stdout:8/988: fdatasync d2/dc/f20 0 2024-04-22T15:19:15.330 INFO:tasks.workunit.client.1.smithi196.stdout:8/989: write d2/d8/fe8 [1378405,77422] 0 2024-04-22T15:19:15.331 INFO:tasks.workunit.client.1.smithi196.stdout:8/990: chown d2/dc/d45/d98/cbb 0 1 2024-04-22T15:19:15.332 INFO:tasks.workunit.client.1.smithi196.stdout:8/991: fdatasync d2/d35/fb4 0 2024-04-22T15:19:15.336 INFO:tasks.workunit.client.1.smithi196.stdout:8/992: fsync d2/d5/d2e/f67 0 2024-04-22T15:19:15.337 INFO:tasks.workunit.client.1.smithi196.stdout:8/993: write d2/d8/fd0 [381661,90111] 0 2024-04-22T15:19:15.343 INFO:tasks.workunit.client.1.smithi196.stdout:8/994: rmdir d2/d5/d1a/db7/dd9/de1 39 2024-04-22T15:19:15.346 INFO:tasks.workunit.client.1.smithi196.stdout:8/995: mknod d2/d35/c129 0 2024-04-22T15:19:15.352 INFO:tasks.workunit.client.1.smithi196.stdout:7/998: write d14/d25/f79 [460073,102417] 0 2024-04-22T15:19:15.353 INFO:tasks.workunit.client.1.smithi196.stdout:8/996: dread d2/d5/d1a/db7/dd9/d68/dbc/dae/fe6 [0,4194304] 0 2024-04-22T15:19:15.357 INFO:tasks.workunit.client.1.smithi196.stdout:7/999: rename d14/d4d to d14/d26/d2a/d14e 0 2024-04-22T15:19:15.361 INFO:tasks.workunit.client.1.smithi196.stdout:8/997: creat d2/dc/d45/f12a x:0 0 0 2024-04-22T15:19:15.366 INFO:tasks.workunit.client.1.smithi196.stdout:8/998: creat d2/d5/d4f/d125/f12b x:0 0 0 2024-04-22T15:19:15.370 INFO:tasks.workunit.client.1.smithi196.stdout:8/999: unlink d2/d5/c15 0 2024-04-22T15:19:15.372 INFO:tasks.workunit.client.1.smithi196.stdout:4/974: dwrite d6/d13/d15/d2c/d62/d78/d8d/f10b [0,4194304] 0 2024-04-22T15:19:15.373 INFO:tasks.workunit.client.1.smithi196.stdout:4/975: readlink d6/d13/d15/l59 0 2024-04-22T15:19:15.381 INFO:tasks.workunit.client.1.smithi196.stdout:4/976: dwrite d6/d14/d18/d26/d32/d67/d9a/f9e [0,4194304] 0 2024-04-22T15:19:15.387 INFO:tasks.workunit.client.1.smithi196.stdout:4/977: mknod d6/d13/d15/d2c/d8c/db3/c132 0 2024-04-22T15:19:15.399 INFO:tasks.workunit.client.1.smithi196.stdout:4/978: write d6/d13/d15/d2c/fd3 [4664659,96930] 0 2024-04-22T15:19:15.404 INFO:tasks.workunit.client.1.smithi196.stdout:4/979: link d6/c7 d6/d13/d15/d2c/d62/dd6/d11b/d11e/c133 0 2024-04-22T15:19:15.405 INFO:tasks.workunit.client.1.smithi196.stdout:4/980: write d6/d14/d18/d40/d85/fbf [8134926,106517] 0 2024-04-22T15:19:15.412 INFO:tasks.workunit.client.1.smithi196.stdout:4/981: dwrite d6/d14/d18/d26/d32/d53/ffd [0,4194304] 0 2024-04-22T15:19:15.418 INFO:tasks.workunit.client.1.smithi196.stdout:4/982: mknod d6/d13/d15/d2c/d35/c134 0 2024-04-22T15:19:15.422 INFO:tasks.workunit.client.1.smithi196.stdout:4/983: creat d6/d13/d15/d2c/d62/d78/f135 x:0 0 0 2024-04-22T15:19:15.544 INFO:tasks.workunit.client.1.smithi196.stdout:4/984: sync 2024-04-22T15:19:15.555 INFO:tasks.workunit.client.1.smithi196.stdout:4/985: write d6/d13/f1b [1758073,81067] 0 2024-04-22T15:19:15.558 INFO:tasks.workunit.client.1.smithi196.stdout:4/986: creat d6/d14/d18/d26/d32/d67/de7/f136 x:0 0 0 2024-04-22T15:19:15.570 INFO:tasks.workunit.client.1.smithi196.stdout:4/987: dwrite d6/d10/f50 [0,4194304] 0 2024-04-22T15:19:15.581 INFO:tasks.workunit.client.1.smithi196.stdout:4/988: truncate d6/d14/d18/d40/d85/fe5 3151735 0 2024-04-22T15:19:15.582 INFO:tasks.workunit.client.1.smithi196.stdout:4/989: write d6/d13/d15/d2c/d35/f8f [1067172,107241] 0 2024-04-22T15:19:15.583 INFO:tasks.workunit.client.1.smithi196.stdout:4/990: read d6/d14/d18/d26/d32/da3/db7/fef [917847,39773] 0 2024-04-22T15:19:15.587 INFO:tasks.workunit.client.1.smithi196.stdout:4/991: symlink d6/d13/d15/d2c/d62/dcd/l137 0 2024-04-22T15:19:15.600 INFO:tasks.workunit.client.1.smithi196.stdout:4/992: dwrite d6/d10/d105/f119 [0,4194304] 0 2024-04-22T15:19:15.605 INFO:tasks.workunit.client.1.smithi196.stdout:4/993: dwrite d6/d14/d18/d26/d32/d53/ffd [0,4194304] 0 2024-04-22T15:19:15.688 INFO:tasks.workunit.client.1.smithi196.stdout:4/994: sync 2024-04-22T15:19:15.689 INFO:tasks.workunit.client.1.smithi196.stdout:4/995: write d6/d14/d18/d26/d32/da3/db7/dc9/fd9 [644234,26694] 0 2024-04-22T15:19:15.694 INFO:tasks.workunit.client.1.smithi196.stdout:4/996: dwrite d6/d10/f50 [4194304,4194304] 0 2024-04-22T15:19:15.698 INFO:tasks.workunit.client.1.smithi196.stdout:4/997: creat d6/d14/d18/d40/d85/f138 x:0 0 0 2024-04-22T15:19:15.712 INFO:tasks.workunit.client.1.smithi196.stdout:4/998: truncate d6/d13/f1b 8231132 0 2024-04-22T15:19:15.716 INFO:tasks.workunit.client.1.smithi196.stdout:4/999: dread d6/d14/d18/d26/d32/d53/f121 [0,4194304] 0 2024-04-22T15:19:15.719 INFO:tasks.workunit.client.1.smithi196.stderr:+ rm -rf -- ./tmp.SOsiBfZkYa 2024-04-22T15:19:15.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:15 smithi096 ceph-mon[26688]: pgmap v44: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 122 MiB/s rd, 256 MiB/s wr, 594 op/s 2024-04-22T15:19:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:15 smithi196 ceph-mon[34717]: pgmap v44: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 122 MiB/s rd, 256 MiB/s wr, 594 op/s 2024-04-22T15:19:17.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:16 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:17.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:16 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:17.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:16 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:17.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:16 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:18.094 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:17 smithi096 ceph-mon[26688]: pgmap v45: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 76 MiB/s rd, 173 MiB/s wr, 392 op/s 2024-04-22T15:19:18.095 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:17 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:18.095 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:17 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:18.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:17 smithi196 ceph-mon[34717]: pgmap v45: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 76 MiB/s rd, 173 MiB/s wr, 392 op/s 2024-04-22T15:19:18.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:17 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:18.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:17 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:19 smithi096 ceph-mon[26688]: pgmap v46: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 90 MiB/s rd, 200 MiB/s wr, 498 op/s 2024-04-22T15:19:20.191 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:19 smithi196 ceph-mon[34717]: pgmap v46: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 90 MiB/s rd, 200 MiB/s wr, 498 op/s 2024-04-22T15:19:21.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: pgmap v47: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 71 MiB/s rd, 163 MiB/s wr, 413 op/s 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr fail", "who": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr fail", "who": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: Activating manager daemon smithi096.hhkvum 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd='[{"prefix": "mgr fail", "who": "smithi196.gdaegc"}]': finished 2024-04-22T15:19:21.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: mgrmap e27: smithi096.hhkvum(active, starting, since 0.00652275s) 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:19:21.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:19:21.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:19:21.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:19:21.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:19:21.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: Manager daemon smithi096.hhkvum is now available 2024-04-22T15:19:21.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:19:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: pgmap v47: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 71 MiB/s rd, 163 MiB/s wr, 413 op/s 2024-04-22T15:19:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.24359 172.21.15.196:0/1964704655' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr fail", "who": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "mgr fail", "who": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: Activating manager daemon smithi096.hhkvum 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.24359 ' entity='mgr.smithi196.gdaegc' cmd='[{"prefix": "mgr fail", "who": "smithi196.gdaegc"}]': finished 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: mgrmap e27: smithi096.hhkvum(active, starting, since 0.00652275s) 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:19:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:19:22.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:19:22.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:19:22.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: Manager daemon smithi096.hhkvum is now available 2024-04-22T15:19:22.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:19:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:22 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:19:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:22 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:19:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:22 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/trash_purge_schedule"}]: dispatch 2024-04-22T15:19:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:22 smithi196 ceph-mon[34717]: mgrmap e28: smithi096.hhkvum(active, since 1.00931s) 2024-04-22T15:19:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:22 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:19:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:22 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:19:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:22 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/trash_purge_schedule"}]: dispatch 2024-04-22T15:19:23.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:22 smithi096 ceph-mon[26688]: mgrmap e28: smithi096.hhkvum(active, since 1.00931s) 2024-04-22T15:19:24.077 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:23 smithi196 ceph-mon[34717]: [22/Apr/2024:15:19:22] ENGINE Bus STARTING 2024-04-22T15:19:24.077 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:23 smithi196 ceph-mon[34717]: [22/Apr/2024:15:19:22] ENGINE Serving on https://172.21.15.96:7150 2024-04-22T15:19:24.077 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:23 smithi196 ceph-mon[34717]: [22/Apr/2024:15:19:22] ENGINE Serving on http://172.21.15.96:8765 2024-04-22T15:19:24.077 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:23 smithi196 ceph-mon[34717]: [22/Apr/2024:15:19:22] ENGINE Bus STARTED 2024-04-22T15:19:24.077 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:23 smithi196 ceph-mon[34717]: pgmap v3: 65 pgs: 65 active+clean; 2.8 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:19:24.077 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:23 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:24.078 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:23 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:24.154 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:23 smithi096 ceph-mon[26688]: [22/Apr/2024:15:19:22] ENGINE Bus STARTING 2024-04-22T15:19:24.154 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:23 smithi096 ceph-mon[26688]: [22/Apr/2024:15:19:22] ENGINE Serving on https://172.21.15.96:7150 2024-04-22T15:19:24.154 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:23 smithi096 ceph-mon[26688]: [22/Apr/2024:15:19:22] ENGINE Serving on http://172.21.15.96:8765 2024-04-22T15:19:24.155 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:23 smithi096 ceph-mon[26688]: [22/Apr/2024:15:19:22] ENGINE Bus STARTED 2024-04-22T15:19:24.155 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:23 smithi096 ceph-mon[26688]: pgmap v3: 65 pgs: 65 active+clean; 2.8 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:19:24.155 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:23 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:24.155 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:23 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:25.218 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:24 smithi196 ceph-mon[34717]: pgmap v4: 65 pgs: 65 active+clean; 2.8 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:19:25.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:24 smithi096 ceph-mon[26688]: pgmap v4: 65 pgs: 65 active+clean; 2.8 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:19:25.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:25 smithi096 ceph-mon[26688]: mgrmap e29: smithi096.hhkvum(active, since 3s) 2024-04-22T15:19:25.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:25 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:25.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:25 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:26.187 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:25 smithi196 ceph-mon[34717]: mgrmap e29: smithi096.hhkvum(active, since 3s) 2024-04-22T15:19:26.187 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:25 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:26.187 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:25 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:26 smithi196 ceph-mon[34717]: pgmap v5: 65 pgs: 65 active+clean; 2.8 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:19:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:26 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:26 smithi096 ceph-mon[26688]: pgmap v5: 65 pgs: 65 active+clean; 2.8 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:19:27.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:26 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:27 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:28.079 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:27 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:29.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:28 smithi096 ceph-mon[26688]: pgmap v6: 65 pgs: 65 active+clean; 2.8 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:19:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:28 smithi196 ceph-mon[34717]: pgmap v6: 65 pgs: 65 active+clean; 2.8 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:19:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:29 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:29 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:29 smithi096 ceph-mon[26688]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:19:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:29 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/274462826' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:19:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:29 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/274462826' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:19:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:29 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/274462826' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:19:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:29 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/274462826' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:19:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:29 smithi096 ceph-mon[26688]: pgmap v7: 65 pgs: 65 active+clean; 1.7 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 884 KiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-22T15:19:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:29 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:29 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:29 smithi196 ceph-mon[34717]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:19:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:29 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/274462826' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:19:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:29 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/274462826' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:19:30.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:29 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/274462826' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:19:30.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:29 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/274462826' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:19:30.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:29 smithi196 ceph-mon[34717]: pgmap v7: 65 pgs: 65 active+clean; 1.7 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 884 KiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-22T15:19:30.776 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T15:19:30.776 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T15:19:31.198 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:30 smithi096 ceph-mon[26688]: mgrmap e30: smithi096.hhkvum(active, since 8s), standbys: smithi196.gdaegc 2024-04-22T15:19:31.198 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:30 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi196.gdaegc", "id": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:19:31.198 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:30 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:31.198 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:30 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:30 smithi196 ceph-mon[34717]: mgrmap e30: smithi096.hhkvum(active, since 8s), standbys: smithi196.gdaegc 2024-04-22T15:19:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:30 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi196.gdaegc", "id": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:19:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:30 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:30 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:32.226 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:31 smithi096 ceph-mon[26688]: pgmap v8: 65 pgs: 65 active+clean; 1.7 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 688 KiB/s rd, 797 KiB/s wr, 116 op/s 2024-04-22T15:19:32.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:31 smithi196 ceph-mon[34717]: pgmap v8: 65 pgs: 65 active+clean; 1.7 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 688 KiB/s rd, 797 KiB/s wr, 116 op/s 2024-04-22T15:19:33.512 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T15:19:33.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-22T15:19:35.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:34 smithi096 ceph-mon[26688]: pgmap v9: 65 pgs: 65 active+clean; 584 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 175 op/s 2024-04-22T15:19:35.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:34 smithi196 ceph-mon[34717]: pgmap v9: 65 pgs: 65 active+clean; 584 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 175 op/s 2024-04-22T15:19:37.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:36 smithi096 ceph-mon[26688]: pgmap v10: 65 pgs: 65 active+clean; 584 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 160 op/s 2024-04-22T15:19:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:36 smithi196 ceph-mon[34717]: pgmap v10: 65 pgs: 65 active+clean; 584 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 160 op/s 2024-04-22T15:19:38.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:37 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:19:38.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:37 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:19:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:38 smithi196 ceph-mon[34717]: pgmap v11: 65 pgs: 65 active+clean; 584 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 160 op/s 2024-04-22T15:19:39.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:38 smithi096 ceph-mon[26688]: pgmap v11: 65 pgs: 65 active+clean; 584 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 160 op/s 2024-04-22T15:19:39.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.279+0000 7f9ac81a0700 1 -- 172.21.15.96:0/1855521096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac0100d90 msgr2=0x7f9ac01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:39.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.279+0000 7f9ac81a0700 1 --2- 172.21.15.96:0/1855521096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac0100d90 0x7f9ac01011f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009910 tx=0x7f9abc009c20 comp rx=0 tx=0).stop 2024-04-22T15:19:39.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.279+0000 7f9ac81a0700 1 -- 172.21.15.96:0/1855521096 shutdown_connections 2024-04-22T15:19:39.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.279+0000 7f9ac81a0700 1 --2- 172.21.15.96:0/1855521096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac0100d90 0x7f9ac01011f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.279+0000 7f9ac81a0700 1 --2- 172.21.15.96:0/1855521096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ac00ffb90 0x7f9ac00fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.279+0000 7f9ac81a0700 1 -- 172.21.15.96:0/1855521096 >> 172.21.15.96:0/1855521096 conn(0x7f9ac00fb110 msgr2=0x7f9ac00fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:39.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.280+0000 7f9ac81a0700 1 -- 172.21.15.96:0/1855521096 shutdown_connections 2024-04-22T15:19:39.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.280+0000 7f9ac81a0700 1 -- 172.21.15.96:0/1855521096 wait complete. 2024-04-22T15:19:39.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.280+0000 7f9ac81a0700 1 Processor -- start 2024-04-22T15:19:39.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.280+0000 7f9ac81a0700 1 -- start start 2024-04-22T15:19:39.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.281+0000 7f9ac81a0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac00ffb90 0x7f9ac01988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:39.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.281+0000 7f9ac81a0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ac0100d90 0x7f9ac0198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:39.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.281+0000 7f9ac81a0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0199410 con 0x7f9ac00ffb90 2024-04-22T15:19:39.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.281+0000 7f9ac81a0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0199550 con 0x7f9ac0100d90 2024-04-22T15:19:39.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.281+0000 7f9ac5f3c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac00ffb90 0x7f9ac01988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:39.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.281+0000 7f9ac5f3c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac00ffb90 0x7f9ac01988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51618/0 (socket says 172.21.15.96:51618) 2024-04-22T15:19:39.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.281+0000 7f9ac5f3c700 1 -- 172.21.15.96:0/2777821484 learned_addr learned my addr 172.21.15.96:0/2777821484 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:39.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.281+0000 7f9ac573b700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ac0100d90 0x7f9ac0198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:39.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.282+0000 7f9ac5f3c700 1 -- 172.21.15.96:0/2777821484 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ac0100d90 msgr2=0x7f9ac0198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:39.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.282+0000 7f9ac5f3c700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ac0100d90 0x7f9ac0198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.282+0000 7f9ac5f3c700 1 -- 172.21.15.96:0/2777821484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc0095f0 con 0x7f9ac00ffb90 2024-04-22T15:19:39.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.282+0000 7f9ac5f3c700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac00ffb90 0x7f9ac01988b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000e8f0 tx=0x7f9ab000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:39.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.282+0000 7f9ab6ffd700 1 -- 172.21.15.96:0/2777821484 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab0009790 con 0x7f9ac00ffb90 2024-04-22T15:19:39.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.282+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac019dfb0 con 0x7f9ac00ffb90 2024-04-22T15:19:39.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.282+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac019e470 con 0x7f9ac00ffb90 2024-04-22T15:19:39.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.283+0000 7f9ab6ffd700 1 -- 172.21.15.96:0/2777821484 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ab00044e0 con 0x7f9ac00ffb90 2024-04-22T15:19:39.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.283+0000 7f9ab6ffd700 1 -- 172.21.15.96:0/2777821484 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab0003e00 con 0x7f9ac00ffb90 2024-04-22T15:19:39.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.284+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa4005320 con 0x7f9ac00ffb90 2024-04-22T15:19:39.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.285+0000 7f9ab6ffd700 1 -- 172.21.15.96:0/2777821484 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7f9ab000cba0 con 0x7f9ac00ffb90 2024-04-22T15:19:39.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.286+0000 7f9ab6ffd700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9aac06ea60 0x7f9aac070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:39.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.286+0000 7f9ab6ffd700 1 -- 172.21.15.96:0/2777821484 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9ab0014070 con 0x7f9ac00ffb90 2024-04-22T15:19:39.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.286+0000 7f9ac573b700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9aac06ea60 0x7f9aac070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:39.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.287+0000 7f9ac573b700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9aac06ea60 0x7f9aac070f20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00b800 tx=0x7f9abc009f90 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:39.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.296+0000 7f9ab6ffd700 1 -- 172.21.15.96:0/2777821484 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f9ab005ba30 con 0x7f9ac00ffb90 2024-04-22T15:19:39.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.595+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9aa4000bf0 con 0x7f9aac06ea60 2024-04-22T15:19:39.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.597+0000 7f9ab6ffd700 1 -- 172.21.15.96:0/2777821484 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f9aa4000bf0 con 0x7f9aac06ea60 2024-04-22T15:19:39.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9aac06ea60 msgr2=0x7f9aac070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:39.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9aac06ea60 0x7f9aac070f20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00b800 tx=0x7f9abc009f90 comp rx=0 tx=0).stop 2024-04-22T15:19:39.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac00ffb90 msgr2=0x7f9ac01988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:39.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac00ffb90 0x7f9ac01988b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000e8f0 tx=0x7f9ab000ecb0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 shutdown_connections 2024-04-22T15:19:39.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9aac06ea60 0x7f9aac070f20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac00ffb90 0x7f9ac01988b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 --2- 172.21.15.96:0/2777821484 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ac0100d90 0x7f9ac0198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 >> 172.21.15.96:0/2777821484 conn(0x7f9ac00fb110 msgr2=0x7f9ac01043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:39.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 shutdown_connections 2024-04-22T15:19:39.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.599+0000 7f9ac81a0700 1 -- 172.21.15.96:0/2777821484 wait complete. 2024-04-22T15:19:39.614 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:19:39.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.747+0000 7f6c81021700 1 -- 172.21.15.96:0/1923610428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c0ff290 msgr2=0x7f6c7c0ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:39.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.747+0000 7f6c81021700 1 --2- 172.21.15.96:0/1923610428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c0ff290 0x7f6c7c0ff6b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00aa20 tx=0x7f6c6c00ad30 comp rx=0 tx=0).stop 2024-04-22T15:19:39.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.747+0000 7f6c81021700 1 -- 172.21.15.96:0/1923610428 shutdown_connections 2024-04-22T15:19:39.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.747+0000 7f6c81021700 1 --2- 172.21.15.96:0/1923610428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c7c100490 0x7f6c7c1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.747+0000 7f6c81021700 1 --2- 172.21.15.96:0/1923610428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c0ff290 0x7f6c7c0ff6b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.747+0000 7f6c81021700 1 -- 172.21.15.96:0/1923610428 >> 172.21.15.96:0/1923610428 conn(0x7f6c7c0fa830 msgr2=0x7f6c7c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:39.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.747+0000 7f6c81021700 1 -- 172.21.15.96:0/1923610428 shutdown_connections 2024-04-22T15:19:39.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.747+0000 7f6c81021700 1 -- 172.21.15.96:0/1923610428 wait complete. 2024-04-22T15:19:39.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.748+0000 7f6c81021700 1 Processor -- start 2024-04-22T15:19:39.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.748+0000 7f6c81021700 1 -- start start 2024-04-22T15:19:39.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.748+0000 7f6c81021700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c7c0ff290 0x7f6c7c196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:39.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.748+0000 7f6c81021700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c100490 0x7f6c7c196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.748+0000 7f6c81021700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c197190 con 0x7f6c7c100490 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.748+0000 7f6c81021700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c1972d0 con 0x7f6c7c0ff290 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c7a59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c100490 0x7f6c7c196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c7a59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c100490 0x7f6c7c196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51634/0 (socket says 172.21.15.96:51634) 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c7a59c700 1 -- 172.21.15.96:0/4120944375 learned_addr learned my addr 172.21.15.96:0/4120944375 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c7ad9d700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c7c0ff290 0x7f6c7c196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c7ad9d700 1 -- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c100490 msgr2=0x7f6c7c196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c7ad9d700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c100490 0x7f6c7c196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c7ad9d700 1 -- 172.21.15.96:0/4120944375 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c00a700 con 0x7f6c7c0ff290 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c7a59c700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c100490 0x7f6c7c196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:19:39.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c7ad9d700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c7c0ff290 0x7f6c7c196630 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c006010 tx=0x7f6c6c005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:39.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.749+0000 7f6c63fff700 1 -- 172.21.15.96:0/4120944375 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c6c005000 con 0x7f6c7c0ff290 2024-04-22T15:19:39.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.750+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c19bc80 con 0x7f6c7c0ff290 2024-04-22T15:19:39.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.750+0000 7f6c63fff700 1 -- 172.21.15.96:0/4120944375 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c6c005160 con 0x7f6c7c0ff290 2024-04-22T15:19:39.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.750+0000 7f6c63fff700 1 -- 172.21.15.96:0/4120944375 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c6c016760 con 0x7f6c7c0ff290 2024-04-22T15:19:39.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.750+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c19c170 con 0x7f6c7c0ff290 2024-04-22T15:19:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.751+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c18e580 con 0x7f6c7c0ff290 2024-04-22T15:19:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.751+0000 7f6c63fff700 1 -- 172.21.15.96:0/4120944375 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7f6c6c00e450 con 0x7f6c7c0ff290 2024-04-22T15:19:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.752+0000 7f6c63fff700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6c6406ea80 0x7f6c64070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.752+0000 7f6c63fff700 1 -- 172.21.15.96:0/4120944375 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f6c6c012070 con 0x7f6c7c0ff290 2024-04-22T15:19:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.752+0000 7f6c7a59c700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6c6406ea80 0x7f6c64070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:39.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.753+0000 7f6c7a59c700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6c6406ea80 0x7f6c64070f40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6c70011550 tx=0x7f6c70009260 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:39.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:39.757+0000 7f6c63fff700 1 -- 172.21.15.96:0/4120944375 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f6c6c05d670 con 0x7f6c7c0ff290 2024-04-22T15:19:40.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.103+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c7c02cf00 con 0x7f6c6406ea80 2024-04-22T15:19:40.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.104+0000 7f6c63fff700 1 -- 172.21.15.96:0/4120944375 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f6c7c02cf00 con 0x7f6c6406ea80 2024-04-22T15:19:40.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6c6406ea80 msgr2=0x7f6c64070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:40.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6c6406ea80 0x7f6c64070f40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6c70011550 tx=0x7f6c70009260 comp rx=0 tx=0).stop 2024-04-22T15:19:40.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c7c0ff290 msgr2=0x7f6c7c196630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:40.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c7c0ff290 0x7f6c7c196630 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c006010 tx=0x7f6c6c005800 comp rx=0 tx=0).stop 2024-04-22T15:19:40.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 shutdown_connections 2024-04-22T15:19:40.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6c6406ea80 0x7f6c64070f40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c7c0ff290 0x7f6c7c196630 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 --2- 172.21.15.96:0/4120944375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c100490 0x7f6c7c196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 >> 172.21.15.96:0/4120944375 conn(0x7f6c7c0fa830 msgr2=0x7f6c7c103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 shutdown_connections 2024-04-22T15:19:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.106+0000 7f6c81021700 1 -- 172.21.15.96:0/4120944375 wait complete. 2024-04-22T15:19:40.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.223+0000 7f28dbd47700 1 -- 172.21.15.96:0/2008272061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d4100d90 msgr2=0x7f28d41011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:40.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.223+0000 7f28dbd47700 1 --2- 172.21.15.96:0/2008272061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d4100d90 0x7f28d41011f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f28c4009910 tx=0x7f28c4009c20 comp rx=0 tx=0).stop 2024-04-22T15:19:40.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.223+0000 7f28dbd47700 1 -- 172.21.15.96:0/2008272061 shutdown_connections 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.223+0000 7f28dbd47700 1 --2- 172.21.15.96:0/2008272061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d4100d90 0x7f28d41011f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.223+0000 7f28dbd47700 1 --2- 172.21.15.96:0/2008272061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d40ffb90 0x7f28d40fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.223+0000 7f28dbd47700 1 -- 172.21.15.96:0/2008272061 >> 172.21.15.96:0/2008272061 conn(0x7f28d40fb110 msgr2=0x7f28d40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.223+0000 7f28dbd47700 1 -- 172.21.15.96:0/2008272061 shutdown_connections 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.223+0000 7f28dbd47700 1 -- 172.21.15.96:0/2008272061 wait complete. 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28dbd47700 1 Processor -- start 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28dbd47700 1 -- start start 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28dbd47700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d40ffb90 0x7f28d40690d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28dbd47700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d4100d90 0x7f28d419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28dbd47700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d41971c0 con 0x7f28d40ffb90 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28dbd47700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d419dd20 con 0x7f28d4100d90 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28d9ae3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d40ffb90 0x7f28d40690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28d9ae3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d40ffb90 0x7f28d40690d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51642/0 (socket says 172.21.15.96:51642) 2024-04-22T15:19:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28d9ae3700 1 -- 172.21.15.96:0/324560375 learned_addr learned my addr 172.21.15.96:0/324560375 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28d92e2700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d4100d90 0x7f28d419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28d9ae3700 1 -- 172.21.15.96:0/324560375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d4100d90 msgr2=0x7f28d419d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28d9ae3700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d4100d90 0x7f28d419d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.224+0000 7f28d9ae3700 1 -- 172.21.15.96:0/324560375 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c40095f0 con 0x7f28d40ffb90 2024-04-22T15:19:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.225+0000 7f28d9ae3700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d40ffb90 0x7f28d40690d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f28d000d930 tx=0x7f28d000dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.225+0000 7f28caffd700 1 -- 172.21.15.96:0/324560375 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28d000b920 con 0x7f28d40ffb90 2024-04-22T15:19:40.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.225+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d419df80 con 0x7f28d40ffb90 2024-04-22T15:19:40.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.225+0000 7f28caffd700 1 -- 172.21.15.96:0/324560375 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28d0010460 con 0x7f28d40ffb90 2024-04-22T15:19:40.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.225+0000 7f28caffd700 1 -- 172.21.15.96:0/324560375 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28d000f590 con 0x7f28d40ffb90 2024-04-22T15:19:40.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.225+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d419e440 con 0x7f28d40ffb90 2024-04-22T15:19:40.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.226+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d418f710 con 0x7f28d40ffb90 2024-04-22T15:19:40.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.227+0000 7f28caffd700 1 -- 172.21.15.96:0/324560375 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7f28d00105d0 con 0x7f28d40ffb90 2024-04-22T15:19:40.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.227+0000 7f28caffd700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f28c006ea80 0x7f28c0070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:40.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.227+0000 7f28caffd700 1 -- 172.21.15.96:0/324560375 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f28d008e790 con 0x7f28d40ffb90 2024-04-22T15:19:40.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.227+0000 7f28d92e2700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f28c006ea80 0x7f28c0070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:40.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.228+0000 7f28d92e2700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f28c006ea80 0x7f28c0070f40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28c4009fd0 tx=0x7f28c4005960 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:40.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.233+0000 7f28caffd700 1 -- 172.21.15.96:0/324560375 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f28d005b320 con 0x7f28d40ffb90 2024-04-22T15:19:40.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:40 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:40.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:40 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:40.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:40 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:19:40.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:40 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:19:40.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:40 smithi096 ceph-mon[26688]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T15:19:40.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:40 smithi096 ceph-mon[26688]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-22T15:19:40.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:40 smithi096 ceph-mon[26688]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:40.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:40 smithi096 ceph-mon[26688]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:19:40.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:40 smithi096 ceph-mon[26688]: pgmap v12: 65 pgs: 65 active+clean; 283 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 214 op/s 2024-04-22T15:19:40.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.549+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f28d402cef0 con 0x7f28c006ea80 2024-04-22T15:19:40.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.560+0000 7f28caffd700 1 -- 172.21.15.96:0/324560375 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f28d402cef0 con 0x7f28c006ea80 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (8m) 11s ago 9m 24.8M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (10m) 11s ago 9m 7990k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (8m) 15s ago 8m 7746k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (9m) 11s ago 9m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (8m) 15s ago 8m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (7m) 11s ago 9m 83.8M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (3m) 11s ago 3m 256M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (4m) 11s ago 4m 274M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:19:40.564 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (4m) 15s ago 4m 24.6M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (3m) 15s ago 3m 25.7M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (44s) 11s ago 11m 479M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (2m) 15s ago 8m 79.5M - 19.0.0-3223-g5883d494 3d983c26e278 8e82f21cd80c 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (11m) 11s ago 11m 53.0M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (8m) 15s ago 8m 42.7M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (9m) 11s ago 9m 14.0M - 1.5.0 0da6a335fe13 852e13a0c299 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (8m) 15s ago 8m 16.2M - 1.5.0 0da6a335fe13 4d02817c7cd0 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (7m) 11s ago 7m 174M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (7m) 11s ago 7m 186M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (6m) 11s ago 6m 150M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (6m) 15s ago 6m 206M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (5m) 15s ago 5m 185M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (5m) 15s ago 5m 184M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:19:40.565 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (74s) 11s ago 9m 65.0M - 2.43.0 a07b618ecd1d db0f9e3ebba7 2024-04-22T15:19:40.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f28c006ea80 msgr2=0x7f28c0070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:40.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f28c006ea80 0x7f28c0070f40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28c4009fd0 tx=0x7f28c4005960 comp rx=0 tx=0).stop 2024-04-22T15:19:40.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d40ffb90 msgr2=0x7f28d40690d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:40.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d40ffb90 0x7f28d40690d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f28d000d930 tx=0x7f28d000dc40 comp rx=0 tx=0).stop 2024-04-22T15:19:40.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 shutdown_connections 2024-04-22T15:19:40.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f28c006ea80 0x7f28c0070f40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28d40ffb90 0x7f28d40690d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 --2- 172.21.15.96:0/324560375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d4100d90 0x7f28d419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 >> 172.21.15.96:0/324560375 conn(0x7f28d40fb110 msgr2=0x7f28d41043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:40.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 shutdown_connections 2024-04-22T15:19:40.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.563+0000 7f28dbd47700 1 -- 172.21.15.96:0/324560375 wait complete. 2024-04-22T15:19:40.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.688+0000 7f8010727700 1 -- 172.21.15.96:0/1185746439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008100c40 msgr2=0x7f8008101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:40.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.688+0000 7f8010727700 1 --2- 172.21.15.96:0/1185746439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008100c40 0x7f8008101060 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8004009910 tx=0x7f8004009c20 comp rx=0 tx=0).stop 2024-04-22T15:19:40.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.689+0000 7f8010727700 1 -- 172.21.15.96:0/1185746439 shutdown_connections 2024-04-22T15:19:40.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.689+0000 7f8010727700 1 --2- 172.21.15.96:0/1185746439 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8008101e40 0x7f80081022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.689+0000 7f8010727700 1 --2- 172.21.15.96:0/1185746439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008100c40 0x7f8008101060 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.689+0000 7f8010727700 1 -- 172.21.15.96:0/1185746439 >> 172.21.15.96:0/1185746439 conn(0x7f80080fc200 msgr2=0x7f80080fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:40.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.689+0000 7f8010727700 1 -- 172.21.15.96:0/1185746439 shutdown_connections 2024-04-22T15:19:40.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.689+0000 7f8010727700 1 -- 172.21.15.96:0/1185746439 wait complete. 2024-04-22T15:19:40.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.689+0000 7f8010727700 1 Processor -- start 2024-04-22T15:19:40.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f8010727700 1 -- start start 2024-04-22T15:19:40.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f8010727700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8008100c40 0x7f8008194480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:40.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f8010727700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008101e40 0x7f80081949c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:40.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f8010727700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8008194fe0 con 0x7f8008101e40 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f8010727700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8008195120 con 0x7f8008100c40 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f800dcc2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008101e40 0x7f80081949c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f800dcc2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008101e40 0x7f80081949c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46936/0 (socket says 172.21.15.96:46936) 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f800dcc2700 1 -- 172.21.15.96:0/3802226685 learned_addr learned my addr 172.21.15.96:0/3802226685 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f800e4c3700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8008100c40 0x7f8008194480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f800dcc2700 1 -- 172.21.15.96:0/3802226685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8008100c40 msgr2=0x7f8008194480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f800dcc2700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8008100c40 0x7f8008194480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.690+0000 7f800dcc2700 1 -- 172.21.15.96:0/3802226685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80040095f0 con 0x7f8008101e40 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.691+0000 7f800dcc2700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008101e40 0x7f80081949c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800d8d0 tx=0x7f7ff800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.691+0000 7f7fff7fe700 1 -- 172.21.15.96:0/3802226685 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ff80098a0 con 0x7f8008101e40 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.691+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8008199b30 con 0x7f8008101e40 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.691+0000 7f7fff7fe700 1 -- 172.21.15.96:0/3802226685 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ff8010460 con 0x7f8008101e40 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.691+0000 7f7fff7fe700 1 -- 172.21.15.96:0/3802226685 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ff800f5f0 con 0x7f8008101e40 2024-04-22T15:19:40.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.691+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800819a100 con 0x7f8008101e40 2024-04-22T15:19:40.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.692+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800804ebb0 con 0x7f8008101e40 2024-04-22T15:19:40.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.693+0000 7f7fff7fe700 1 -- 172.21.15.96:0/3802226685 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7f7ff8009a00 con 0x7f8008101e40 2024-04-22T15:19:40.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.693+0000 7f7fff7fe700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7ff406ea80 0x7f7ff4070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:40.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.693+0000 7f7fff7fe700 1 -- 172.21.15.96:0/3802226685 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7ff808ea50 con 0x7f8008101e40 2024-04-22T15:19:40.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.693+0000 7f800e4c3700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7ff406ea80 0x7f7ff4070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:40.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.694+0000 7f800e4c3700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7ff406ea80 0x7f7ff4070f40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8004000c00 tx=0x7f8004005960 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:40.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:40.698+0000 7f7fff7fe700 1 -- 172.21.15.96:0/3802226685 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f7ff805b5f0 con 0x7f8008101e40 2024-04-22T15:19:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:40 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:40 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:40 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:19:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:40 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:19:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:40 smithi196 ceph-mon[34717]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T15:19:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:40 smithi196 ceph-mon[34717]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-22T15:19:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:40 smithi196 ceph-mon[34717]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:40 smithi196 ceph-mon[34717]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:19:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:40 smithi196 ceph-mon[34717]: pgmap v12: 65 pgs: 65 active+clean; 283 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 214 op/s 2024-04-22T15:19:41.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.142+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8008109890 con 0x7f8008101e40 2024-04-22T15:19:41.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.143+0000 7f7fff7fe700 1 -- 172.21.15.96:0/3802226685 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7f7ff805b180 con 0x7f8008101e40 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:19:41.147 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:19:41.148 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:19:41.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:19:41.148 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:19:41.148 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 12, 2024-04-22T15:19:41.148 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:19:41.148 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:19:41.148 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.145+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7ff406ea80 msgr2=0x7f7ff4070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.145+0000 7f8010727700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7ff406ea80 0x7f7ff4070f40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8004000c00 tx=0x7f8004005960 comp rx=0 tx=0).stop 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.145+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008101e40 msgr2=0x7f80081949c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.145+0000 7f8010727700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008101e40 0x7f80081949c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800d8d0 tx=0x7f7ff800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.145+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 shutdown_connections 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.146+0000 7f8010727700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7ff406ea80 0x7f7ff4070f40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.146+0000 7f8010727700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8008100c40 0x7f8008194480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.146+0000 7f8010727700 1 --2- 172.21.15.96:0/3802226685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8008101e40 0x7f80081949c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.146+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 >> 172.21.15.96:0/3802226685 conn(0x7f80080fc200 msgr2=0x7f8008073490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.146+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 shutdown_connections 2024-04-22T15:19:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.146+0000 7f8010727700 1 -- 172.21.15.96:0/3802226685 wait complete. 2024-04-22T15:19:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.286+0000 7f50720da700 1 -- 172.21.15.96:0/1813600605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 msgr2=0x7f506c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.286+0000 7f50720da700 1 --2- 172.21.15.96:0/1813600605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 0x7f506c0fffb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f505c009910 tx=0x7f505c009c20 comp rx=0 tx=0).stop 2024-04-22T15:19:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.286+0000 7f50720da700 1 -- 172.21.15.96:0/1813600605 shutdown_connections 2024-04-22T15:19:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.286+0000 7f50720da700 1 --2- 172.21.15.96:0/1813600605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f506c100d90 0x7f506c1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.286+0000 7f50720da700 1 --2- 172.21.15.96:0/1813600605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 0x7f506c0fffb0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.286+0000 7f50720da700 1 -- 172.21.15.96:0/1813600605 >> 172.21.15.96:0/1813600605 conn(0x7f506c0fb110 msgr2=0x7f506c0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:41.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.287+0000 7f50720da700 1 -- 172.21.15.96:0/1813600605 shutdown_connections 2024-04-22T15:19:41.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.287+0000 7f50720da700 1 -- 172.21.15.96:0/1813600605 wait complete. 2024-04-22T15:19:41.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.287+0000 7f50720da700 1 Processor -- start 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.287+0000 7f50720da700 1 -- start start 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f50720da700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 0x7f506c103ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f50720da700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f506c100d90 0x7f506c102170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f50720da700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c1026b0 con 0x7f506c0ffb90 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f50720da700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c1027f0 con 0x7f506c100d90 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f506b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 0x7f506c103ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f506affd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f506c100d90 0x7f506c102170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f506b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 0x7f506c103ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46954/0 (socket says 172.21.15.96:46954) 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f506affd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f506c100d90 0x7f506c102170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:41612/0 (socket says 172.21.15.96:41612) 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f506b7fe700 1 -- 172.21.15.96:0/2019708934 learned_addr learned my addr 172.21.15.96:0/2019708934 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:19:41.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f506b7fe700 1 -- 172.21.15.96:0/2019708934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f506c100d90 msgr2=0x7f506c102170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:41.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f506b7fe700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f506c100d90 0x7f506c102170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f506b7fe700 1 -- 172.21.15.96:0/2019708934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c0095f0 con 0x7f506c0ffb90 2024-04-22T15:19:41.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.288+0000 7f506affd700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f506c100d90 0x7f506c102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:19:41.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.289+0000 7f506b7fe700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 0x7f506c103ad0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f505c006010 tx=0x7f505c00b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:41.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.289+0000 7f5068ff9700 1 -- 172.21.15.96:0/2019708934 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f505c01d070 con 0x7f506c0ffb90 2024-04-22T15:19:41.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.289+0000 7f5068ff9700 1 -- 172.21.15.96:0/2019708934 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f505c00fe40 con 0x7f506c0ffb90 2024-04-22T15:19:41.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.289+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506c102a70 con 0x7f506c0ffb90 2024-04-22T15:19:41.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.289+0000 7f5068ff9700 1 -- 172.21.15.96:0/2019708934 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f505c022470 con 0x7f506c0ffb90 2024-04-22T15:19:41.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.289+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c102f60 con 0x7f506c0ffb90 2024-04-22T15:19:41.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.290+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c190720 con 0x7f506c0ffb90 2024-04-22T15:19:41.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.291+0000 7f5068ff9700 1 -- 172.21.15.96:0/2019708934 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7f505c003810 con 0x7f506c0ffb90 2024-04-22T15:19:41.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.292+0000 7f5068ff9700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f505406eb10 0x7f5054070fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:19:41.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.292+0000 7f5068ff9700 1 -- 172.21.15.96:0/2019708934 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f505c090060 con 0x7f506c0ffb90 2024-04-22T15:19:41.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.292+0000 7f506affd700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f505406eb10 0x7f5054070fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:19:41.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.293+0000 7f506affd700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f505406eb10 0x7f5054070fd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5060011490 tx=0x7f5060009260 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:19:41.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.301+0000 7f5068ff9700 1 -- 172.21.15.96:0/2019708934 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f505c05cb60 con 0x7f506c0ffb90 2024-04-22T15:19:41.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.647+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f506c02cdb0 con 0x7f505406eb10 2024-04-22T15:19:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.648+0000 7f5068ff9700 1 -- 172.21.15.96:0/2019708934 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f506c02cdb0 con 0x7f505406eb10 2024-04-22T15:19:41.651 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:19:41.651 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:19:41.652 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:19:41.652 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:19:41.652 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:19:41.652 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:19:41.652 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:19:41.652 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:19:41.652 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "", 2024-04-22T15:19:41.652 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:19:41.652 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:19:41.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f505406eb10 msgr2=0x7f5054070fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:41.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f505406eb10 0x7f5054070fd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5060011490 tx=0x7f5060009260 comp rx=0 tx=0).stop 2024-04-22T15:19:41.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 msgr2=0x7f506c103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:19:41.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 0x7f506c103ad0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f505c006010 tx=0x7f505c00b610 comp rx=0 tx=0).stop 2024-04-22T15:19:41.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 shutdown_connections 2024-04-22T15:19:41.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f505406eb10 0x7f5054070fd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f506c0ffb90 0x7f506c103ad0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 --2- 172.21.15.96:0/2019708934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f506c100d90 0x7f506c102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:19:41.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 >> 172.21.15.96:0/2019708934 conn(0x7f506c0fb110 msgr2=0x7f506c104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:19:41.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 shutdown_connections 2024-04-22T15:19:41.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:19:41.651+0000 7f50720da700 1 -- 172.21.15.96:0/2019708934 wait complete. 2024-04-22T15:19:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:19:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:19:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:19:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:19:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:41.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:41.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:19:41.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3802226685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:19:41.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:41.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:41.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:41 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:19:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:19:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:19:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:19:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:19:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3802226685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:19:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:41 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:19:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:42 smithi096 ceph-mon[26688]: Reconfiguring prometheus.smithi096 (dependencies changed)... 2024-04-22T15:19:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:42 smithi096 ceph-mon[26688]: Reconfiguring daemon prometheus.smithi096 on smithi096 2024-04-22T15:19:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:42 smithi096 ceph-mon[26688]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:42 smithi096 ceph-mon[26688]: pgmap v13: 65 pgs: 65 active+clean; 283 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-22T15:19:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:42 smithi196 ceph-mon[34717]: Reconfiguring prometheus.smithi096 (dependencies changed)... 2024-04-22T15:19:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:42 smithi196 ceph-mon[34717]: Reconfiguring daemon prometheus.smithi096 on smithi096 2024-04-22T15:19:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:42 smithi196 ceph-mon[34717]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:19:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:42 smithi196 ceph-mon[34717]: pgmap v13: 65 pgs: 65 active+clean; 283 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-22T15:19:45.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:44 smithi096 ceph-mon[26688]: pgmap v14: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 170 op/s 2024-04-22T15:19:45.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:44 smithi196 ceph-mon[34717]: pgmap v14: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 170 op/s 2024-04-22T15:19:47.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:46 smithi096 ceph-mon[26688]: pgmap v15: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 97 op/s 2024-04-22T15:19:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:46 smithi196 ceph-mon[34717]: pgmap v15: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 97 op/s 2024-04-22T15:19:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:48 smithi196 ceph-mon[34717]: pgmap v16: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 97 op/s 2024-04-22T15:19:49.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:48 smithi096 ceph-mon[26688]: pgmap v16: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 97 op/s 2024-04-22T15:19:51.182 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:50 smithi096 ceph-mon[26688]: pgmap v17: 65 pgs: 65 active+clean; 296 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 135 op/s 2024-04-22T15:19:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:50 smithi196 ceph-mon[34717]: pgmap v17: 65 pgs: 65 active+clean; 296 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 135 op/s 2024-04-22T15:19:53.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:52 smithi096 ceph-mon[26688]: pgmap v18: 65 pgs: 65 active+clean; 296 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-04-22T15:19:53.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:19:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:52 smithi196 ceph-mon[34717]: pgmap v18: 65 pgs: 65 active+clean; 296 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-04-22T15:19:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:19:55.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:54 smithi096 ceph-mon[26688]: pgmap v19: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 127 op/s 2024-04-22T15:19:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:54 smithi196 ceph-mon[34717]: pgmap v19: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 127 op/s 2024-04-22T15:19:57.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:56 smithi096 ceph-mon[26688]: pgmap v20: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-04-22T15:19:57.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:56 smithi196 ceph-mon[34717]: pgmap v20: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-04-22T15:19:59.105 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:19:58 smithi096 ceph-mon[26688]: pgmap v21: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-04-22T15:19:59.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:19:58 smithi196 ceph-mon[34717]: pgmap v21: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-04-22T15:20:00.891 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:00 smithi096 ceph-mon[26688]: pgmap v22: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 132 op/s 2024-04-22T15:20:00.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:00 smithi096 ceph-mon[26688]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:20:00.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:00 smithi096 ceph-mon[26688]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:20:00.892 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:00 smithi096 ceph-mon[26688]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:20:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:00 smithi196 ceph-mon[34717]: pgmap v22: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 132 op/s 2024-04-22T15:20:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:00 smithi196 ceph-mon[34717]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:20:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:00 smithi196 ceph-mon[34717]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:20:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:00 smithi196 ceph-mon[34717]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:20:02.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:02 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:02.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:02 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:02.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:02 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:20:02.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:02 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:20:02.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:02 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:02.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:02 smithi196 ceph-mon[34717]: pgmap v23: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 94 op/s 2024-04-22T15:20:02.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:02 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:02.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:02 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:02.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:02 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:20:02.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:02 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:20:02.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:02 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:02.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:02 smithi096 ceph-mon[26688]: pgmap v23: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 94 op/s 2024-04-22T15:20:03.462 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:03 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:20:03.462 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:03 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:20:03.462 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:03 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:20:03.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:03 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:20:03.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:03 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:20:03.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:03 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:20:04.476 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:04 smithi196 ceph-mon[34717]: Upgrade: Updating mgr.smithi196.gdaegc 2024-04-22T15:20:04.476 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:04 smithi196 ceph-mon[34717]: Deploying daemon mgr.smithi196.gdaegc on smithi196 2024-04-22T15:20:04.476 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:04 smithi196 ceph-mon[34717]: pgmap v24: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 149 op/s 2024-04-22T15:20:04.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:04 smithi096 ceph-mon[26688]: Upgrade: Updating mgr.smithi196.gdaegc 2024-04-22T15:20:04.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:04 smithi096 ceph-mon[26688]: Deploying daemon mgr.smithi196.gdaegc on smithi196 2024-04-22T15:20:04.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:04 smithi096 ceph-mon[26688]: pgmap v24: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 149 op/s 2024-04-22T15:20:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:06 smithi196 ceph-mon[34717]: pgmap v25: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-22T15:20:07.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:06 smithi096 ceph-mon[26688]: pgmap v25: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-22T15:20:07.993 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:07 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:20:07.993 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:07 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:07.993 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:07 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:07.994 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:07 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:07 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:20:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:07 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:07 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:07 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:09.077 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:08 smithi096 ceph-mon[26688]: pgmap v26: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-22T15:20:09.256 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:08 smithi196 ceph-mon[34717]: pgmap v26: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-22T15:20:10.856 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:10 smithi096 ceph-mon[26688]: pgmap v27: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 152 op/s 2024-04-22T15:20:10.856 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:10.856 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:10 smithi196 ceph-mon[34717]: pgmap v27: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 152 op/s 2024-04-22T15:20:11.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:11.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:11.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.825+0000 7f6b925da700 1 -- 172.21.15.96:0/3614843451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b8c108430 msgr2=0x7f6b8c1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:11.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.825+0000 7f6b925da700 1 --2- 172.21.15.96:0/3614843451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b8c108430 0x7f6b8c1088b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009910 tx=0x7f6b80009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:11.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.826+0000 7f6b925da700 1 -- 172.21.15.96:0/3614843451 shutdown_connections 2024-04-22T15:20:11.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.826+0000 7f6b925da700 1 --2- 172.21.15.96:0/3614843451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b8c108430 0x7f6b8c1088b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:11.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.826+0000 7f6b925da700 1 --2- 172.21.15.96:0/3614843451 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b8c107260 0x7f6b8c107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:11.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.826+0000 7f6b925da700 1 -- 172.21.15.96:0/3614843451 >> 172.21.15.96:0/3614843451 conn(0x7f6b8c0765c0 msgr2=0x7f6b8c078a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:11.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.826+0000 7f6b925da700 1 -- 172.21.15.96:0/3614843451 shutdown_connections 2024-04-22T15:20:11.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.826+0000 7f6b925da700 1 -- 172.21.15.96:0/3614843451 wait complete. 2024-04-22T15:20:11.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.827+0000 7f6b925da700 1 Processor -- start 2024-04-22T15:20:11.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.827+0000 7f6b925da700 1 -- start start 2024-04-22T15:20:11.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.828+0000 7f6b925da700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b8c107260 0x7f6b8c071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:11.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.828+0000 7f6b925da700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b8c108430 0x7f6b8c0722b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:11.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.828+0000 7f6b925da700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c072880 con 0x7f6b8c108430 2024-04-22T15:20:11.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.828+0000 7f6b925da700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c1a66e0 con 0x7f6b8c107260 2024-04-22T15:20:11.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.828+0000 7f6b8bfff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b8c107260 0x7f6b8c071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:11.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.828+0000 7f6b8bfff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b8c107260 0x7f6b8c071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.96:47168/0 (socket says 172.21.15.96:47168) 2024-04-22T15:20:11.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.828+0000 7f6b8bfff700 1 -- 172.21.15.96:0/404206998 learned_addr learned my addr 172.21.15.96:0/404206998 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:11.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.828+0000 7f6b8b7fe700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b8c108430 0x7f6b8c0722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:11.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.829+0000 7f6b8bfff700 1 -- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b8c108430 msgr2=0x7f6b8c0722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:11.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.829+0000 7f6b8bfff700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b8c108430 0x7f6b8c0722b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:11.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.829+0000 7f6b8bfff700 1 -- 172.21.15.96:0/404206998 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800095f0 con 0x7f6b8c107260 2024-04-22T15:20:11.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.830+0000 7f6b8bfff700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b8c107260 0x7f6b8c071d70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00e920 tx=0x7f6b7c00ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:11.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.831+0000 7f6b897fa700 1 -- 172.21.15.96:0/404206998 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b7c00c9d0 con 0x7f6b8c107260 2024-04-22T15:20:11.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.831+0000 7f6b897fa700 1 -- 172.21.15.96:0/404206998 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b7c010430 con 0x7f6b8c107260 2024-04-22T15:20:11.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.831+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8c1a68e0 con 0x7f6b8c107260 2024-04-22T15:20:11.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.831+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8c1a6eb0 con 0x7f6b8c107260 2024-04-22T15:20:11.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.831+0000 7f6b897fa700 1 -- 172.21.15.96:0/404206998 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b7c018650 con 0x7f6b8c107260 2024-04-22T15:20:11.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.833+0000 7f6b897fa700 1 -- 172.21.15.96:0/404206998 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7f6b7c0105a0 con 0x7f6b8c107260 2024-04-22T15:20:11.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.834+0000 7f6b897fa700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6b7406eb90 0x7f6b74071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:11.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.834+0000 7f6b897fa700 1 -- 172.21.15.96:0/404206998 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f6b7c014070 con 0x7f6b8c107260 2024-04-22T15:20:11.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.834+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b78005320 con 0x7f6b8c107260 2024-04-22T15:20:11.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.835+0000 7f6b8b7fe700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6b7406eb90 0x7f6b74071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:11.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.836+0000 7f6b8b7fe700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6b7406eb90 0x7f6b74071050 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c071af0 tx=0x7f6b80000bc0 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:11.844+0000 7f6b897fa700 1 -- 172.21.15.96:0/404206998 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f6b7c0540c0 con 0x7f6b8c107260 2024-04-22T15:20:12.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.169+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b78000bf0 con 0x7f6b7406eb90 2024-04-22T15:20:12.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.171+0000 7f6b897fa700 1 -- 172.21.15.96:0/404206998 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f6b78000bf0 con 0x7f6b7406eb90 2024-04-22T15:20:12.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6b7406eb90 msgr2=0x7f6b74071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:12.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6b7406eb90 0x7f6b74071050 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c071af0 tx=0x7f6b80000bc0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b8c107260 msgr2=0x7f6b8c071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:12.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b8c107260 0x7f6b8c071d70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00e920 tx=0x7f6b7c00ec30 comp rx=0 tx=0).stop 2024-04-22T15:20:12.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 shutdown_connections 2024-04-22T15:20:12.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f6b7406eb90 0x7f6b74071050 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b8c107260 0x7f6b8c071d70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 --2- 172.21.15.96:0/404206998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b8c108430 0x7f6b8c0722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 >> 172.21.15.96:0/404206998 conn(0x7f6b8c0765c0 msgr2=0x7f6b8c10b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:12.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 shutdown_connections 2024-04-22T15:20:12.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.174+0000 7f6b925da700 1 -- 172.21.15.96:0/404206998 wait complete. 2024-04-22T15:20:12.188 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:20:12.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.306+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1055635550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd540ff970 msgr2=0x7fdd540ffd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:12.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.306+0000 7fdd5b1a0700 1 --2- 172.21.15.96:0/1055635550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd540ff970 0x7fdd540ffd90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009910 tx=0x7fdd50009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:12.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.306+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1055635550 shutdown_connections 2024-04-22T15:20:12.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.306+0000 7fdd5b1a0700 1 --2- 172.21.15.96:0/1055635550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd54100b70 0x7fdd54100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.306+0000 7fdd5b1a0700 1 --2- 172.21.15.96:0/1055635550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd540ff970 0x7fdd540ffd90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.306+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1055635550 >> 172.21.15.96:0/1055635550 conn(0x7fdd540faef0 msgr2=0x7fdd540fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:12.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.307+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1055635550 shutdown_connections 2024-04-22T15:20:12.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.307+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1055635550 wait complete. 2024-04-22T15:20:12.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.307+0000 7fdd5b1a0700 1 Processor -- start 2024-04-22T15:20:12.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.307+0000 7fdd5b1a0700 1 -- start start 2024-04-22T15:20:12.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.307+0000 7fdd5b1a0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd540ff970 0x7fdd54198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:12.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.307+0000 7fdd5b1a0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd54100b70 0x7fdd54198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:12.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5b1a0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd541991e0 con 0x7fdd54100b70 2024-04-22T15:20:12.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5b1a0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54199320 con 0x7fdd540ff970 2024-04-22T15:20:12.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5999d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd54100b70 0x7fdd54198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5999d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd54100b70 0x7fdd54198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58736/0 (socket says 172.21.15.96:58736) 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5999d700 1 -- 172.21.15.96:0/1335823649 learned_addr learned my addr 172.21.15.96:0/1335823649 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5a19e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd540ff970 0x7fdd54198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5999d700 1 -- 172.21.15.96:0/1335823649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd540ff970 msgr2=0x7fdd54198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5999d700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd540ff970 0x7fdd54198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5999d700 1 -- 172.21.15.96:0/1335823649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500095f0 con 0x7fdd54100b70 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd5999d700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd54100b70 0x7fdd54198bc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400d8d0 tx=0x7fdd4400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd4b7fe700 1 -- 172.21.15.96:0/1335823649 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd440098a0 con 0x7fdd54100b70 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.308+0000 7fdd4b7fe700 1 -- 172.21.15.96:0/1335823649 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd44010460 con 0x7fdd54100b70 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.309+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5419dd80 con 0x7fdd54100b70 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.309+0000 7fdd4b7fe700 1 -- 172.21.15.96:0/1335823649 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd4400f5f0 con 0x7fdd54100b70 2024-04-22T15:20:12.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.309+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5419e290 con 0x7fdd54100b70 2024-04-22T15:20:12.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.310+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5404f400 con 0x7fdd54100b70 2024-04-22T15:20:12.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.310+0000 7fdd4b7fe700 1 -- 172.21.15.96:0/1335823649 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7fdd44009a00 con 0x7fdd54100b70 2024-04-22T15:20:12.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.311+0000 7fdd4b7fe700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdd4006ead0 0x7fdd40070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:12.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.311+0000 7fdd4b7fe700 1 -- 172.21.15.96:0/1335823649 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fdd4408e9e0 con 0x7fdd54100b70 2024-04-22T15:20:12.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.311+0000 7fdd5a19e700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdd4006ead0 0x7fdd40070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:12.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.311+0000 7fdd5a19e700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdd4006ead0 0x7fdd40070f90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdd50000c00 tx=0x7fdd50005960 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:12.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.316+0000 7fdd4b7fe700 1 -- 172.21.15.96:0/1335823649 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fdd4405b560 con 0x7fdd54100b70 2024-04-22T15:20:12.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.602+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd540fce60 con 0x7fdd4006ead0 2024-04-22T15:20:12.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.603+0000 7fdd4b7fe700 1 -- 172.21.15.96:0/1335823649 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fdd540fce60 con 0x7fdd4006ead0 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.605+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdd4006ead0 msgr2=0x7fdd40070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.605+0000 7fdd5b1a0700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdd4006ead0 0x7fdd40070f90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdd50000c00 tx=0x7fdd50005960 comp rx=0 tx=0).stop 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.605+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd54100b70 msgr2=0x7fdd54198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.605+0000 7fdd5b1a0700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd54100b70 0x7fdd54198bc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400d8d0 tx=0x7fdd4400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.605+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 shutdown_connections 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.605+0000 7fdd5b1a0700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdd4006ead0 0x7fdd40070f90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.605+0000 7fdd5b1a0700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd540ff970 0x7fdd54198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.605+0000 7fdd5b1a0700 1 --2- 172.21.15.96:0/1335823649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd54100b70 0x7fdd54198bc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.605+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 >> 172.21.15.96:0/1335823649 conn(0x7fdd540faef0 msgr2=0x7fdd540fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.606+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 shutdown_connections 2024-04-22T15:20:12.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.606+0000 7fdd5b1a0700 1 -- 172.21.15.96:0/1335823649 wait complete. 2024-04-22T15:20:12.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.716+0000 7efeb2aec700 1 -- 172.21.15.96:0/1225287911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 msgr2=0x7efeac106420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:12.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.716+0000 7efeb2aec700 1 --2- 172.21.15.96:0/1225287911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 0x7efeac106420 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efe98009910 tx=0x7efe98009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:12.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.716+0000 7efeb2aec700 1 -- 172.21.15.96:0/1225287911 shutdown_connections 2024-04-22T15:20:12.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.716+0000 7efeb2aec700 1 --2- 172.21.15.96:0/1225287911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 0x7efeac106420 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.716+0000 7efeb2aec700 1 --2- 172.21.15.96:0/1225287911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeac101700 0x7efeac103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.716+0000 7efeb2aec700 1 -- 172.21.15.96:0/1225287911 >> 172.21.15.96:0/1225287911 conn(0x7efeac0fb0b0 msgr2=0x7efeac0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:12.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.717+0000 7efeb2aec700 1 -- 172.21.15.96:0/1225287911 shutdown_connections 2024-04-22T15:20:12.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.717+0000 7efeb2aec700 1 -- 172.21.15.96:0/1225287911 wait complete. 2024-04-22T15:20:12.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.717+0000 7efeb2aec700 1 Processor -- start 2024-04-22T15:20:12.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.717+0000 7efeb2aec700 1 -- start start 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.717+0000 7efeb2aec700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeac101700 0x7efeac1965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeb2aec700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 0x7efeac196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeb2aec700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac197130 con 0x7efeac104030 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeb2aec700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac197270 con 0x7efeac101700 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeabfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 0x7efeac196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeabfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 0x7efeac196b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58754/0 (socket says 172.21.15.96:58754) 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeabfff700 1 -- 172.21.15.96:0/2613017254 learned_addr learned my addr 172.21.15.96:0/2613017254 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeb0888700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeac101700 0x7efeac1965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeabfff700 1 -- 172.21.15.96:0/2613017254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeac101700 msgr2=0x7efeac1965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeabfff700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeac101700 0x7efeac1965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeabfff700 1 -- 172.21.15.96:0/2613017254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe980095f0 con 0x7efeac104030 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeabfff700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 0x7efeac196b10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efe98005b40 tx=0x7efe9800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:12.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efea9ffb700 1 -- 172.21.15.96:0/2613017254 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe9801d070 con 0x7efeac104030 2024-04-22T15:20:12.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeac19bc70 con 0x7efeac104030 2024-04-22T15:20:12.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.718+0000 7efea9ffb700 1 -- 172.21.15.96:0/2613017254 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe980050d0 con 0x7efeac104030 2024-04-22T15:20:12.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.719+0000 7efea9ffb700 1 -- 172.21.15.96:0/2613017254 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe98022470 con 0x7efeac104030 2024-04-22T15:20:12.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.719+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac19c1e0 con 0x7efeac104030 2024-04-22T15:20:12.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.720+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac04ebb0 con 0x7efeac104030 2024-04-22T15:20:12.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.721+0000 7efea9ffb700 1 -- 172.21.15.96:0/2613017254 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7efe980037f0 con 0x7efeac104030 2024-04-22T15:20:12.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.721+0000 7efea9ffb700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7efe9c06ea70 0x7efe9c070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:12.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.722+0000 7efea9ffb700 1 -- 172.21.15.96:0/2613017254 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7efe98090180 con 0x7efeac104030 2024-04-22T15:20:12.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.722+0000 7efeb0888700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7efe9c06ea70 0x7efe9c070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:12.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.722+0000 7efeb0888700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7efe9c06ea70 0x7efe9c070f30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efea0009c60 tx=0x7efea0009260 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:12.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:12.727+0000 7efea9ffb700 1 -- 172.21.15.96:0/2613017254 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7efe9805cd20 con 0x7efeac104030 2024-04-22T15:20:12.817 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:12 smithi196 ceph-mon[34717]: pgmap v28: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 104 op/s 2024-04-22T15:20:12.817 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:12 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:12.817 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:12 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:12 smithi096 ceph-mon[26688]: pgmap v28: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 104 op/s 2024-04-22T15:20:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:12 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:12 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:13.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.059+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efeac105f10 con 0x7efe9c06ea70 2024-04-22T15:20:13.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.073+0000 7efea9ffb700 1 -- 172.21.15.96:0/2613017254 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7efeac105f10 con 0x7efe9c06ea70 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (8m) 44s ago 10m 24.8M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (10m) 44s ago 10m 7990k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (9m) 2s ago 9m 7805k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (10m) 44s ago 10m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (9m) 2s ago 9m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (8m) 44s ago 9m 83.8M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (4m) 44s ago 4m 256M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (4m) 44s ago 4m 274M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (4m) 2s ago 4m 26.5M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (4m) 2s ago 4m 25.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:20:13.077 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (76s) 44s ago 11m 479M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (6s) 2s ago 9m 62.3M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (11m) 44s ago 11m 53.0M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (9m) 2s ago 9m 42.7M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (10m) 44s ago 10m 14.0M - 1.5.0 0da6a335fe13 852e13a0c299 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (9m) 2s ago 9m 16.6M - 1.5.0 0da6a335fe13 4d02817c7cd0 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (8m) 44s ago 8m 174M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (7m) 44s ago 7m 186M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (7m) 44s ago 7m 150M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (6m) 2s ago 6m 216M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (6m) 2s ago 6m 191M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (6m) 2s ago 6m 173M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:20:13.078 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 starting - - - - 2024-04-22T15:20:13.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7efe9c06ea70 msgr2=0x7efe9c070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:13.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7efe9c06ea70 0x7efe9c070f30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efea0009c60 tx=0x7efea0009260 comp rx=0 tx=0).stop 2024-04-22T15:20:13.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 msgr2=0x7efeac196b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:13.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 0x7efeac196b10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efe98005b40 tx=0x7efe9800f690 comp rx=0 tx=0).stop 2024-04-22T15:20:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 shutdown_connections 2024-04-22T15:20:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7efe9c06ea70 0x7efe9c070f30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeac101700 0x7efeac1965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 --2- 172.21.15.96:0/2613017254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeac104030 0x7efeac196b10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 >> 172.21.15.96:0/2613017254 conn(0x7efeac0fb0b0 msgr2=0x7efeac0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 shutdown_connections 2024-04-22T15:20:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.076+0000 7efeb2aec700 1 -- 172.21.15.96:0/2613017254 wait complete. 2024-04-22T15:20:13.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.190+0000 7f87327c6700 1 -- 172.21.15.96:0/2670653269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 msgr2=0x7f872c102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:13.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.190+0000 7f87327c6700 1 --2- 172.21.15.96:0/2670653269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 0x7f872c102270 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8720009910 tx=0x7f8720009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:13.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.190+0000 7f87327c6700 1 -- 172.21.15.96:0/2670653269 shutdown_connections 2024-04-22T15:20:13.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.190+0000 7f87327c6700 1 --2- 172.21.15.96:0/2670653269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 0x7f872c102270 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.190+0000 7f87327c6700 1 --2- 172.21.15.96:0/2670653269 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f872c100c10 0x7f872c101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.190+0000 7f87327c6700 1 -- 172.21.15.96:0/2670653269 >> 172.21.15.96:0/2670653269 conn(0x7f872c0fc1d0 msgr2=0x7f872c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:13.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.190+0000 7f87327c6700 1 -- 172.21.15.96:0/2670653269 shutdown_connections 2024-04-22T15:20:13.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.190+0000 7f87327c6700 1 -- 172.21.15.96:0/2670653269 wait complete. 2024-04-22T15:20:13.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f87327c6700 1 Processor -- start 2024-04-22T15:20:13.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f87327c6700 1 -- start start 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f87327c6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f872c100c10 0x7f872c196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f87327c6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 0x7f872c196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f87327c6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c197180 con 0x7f872c101e10 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f87327c6700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c1972c0 con 0x7f872c100c10 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f872b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 0x7f872c196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f872b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 0x7f872c196b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58772/0 (socket says 172.21.15.96:58772) 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f872bfff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f872c100c10 0x7f872c196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f872b7fe700 1 -- 172.21.15.96:0/2318938485 learned_addr learned my addr 172.21.15.96:0/2318938485 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f872b7fe700 1 -- 172.21.15.96:0/2318938485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f872c100c10 msgr2=0x7f872c196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:13.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f872b7fe700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f872c100c10 0x7f872c196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f872b7fe700 1 -- 172.21.15.96:0/2318938485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87200095f0 con 0x7f872c101e10 2024-04-22T15:20:13.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.191+0000 7f872bfff700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f872c100c10 0x7f872c196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:20:13.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.192+0000 7f872b7fe700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 0x7f872c196b60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f87200092e0 tx=0x7f872000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:13.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.192+0000 7f87297fa700 1 -- 172.21.15.96:0/2318938485 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872001d070 con 0x7f872c101e10 2024-04-22T15:20:13.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.192+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872c19bc70 con 0x7f872c101e10 2024-04-22T15:20:13.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.192+0000 7f87297fa700 1 -- 172.21.15.96:0/2318938485 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f872000f460 con 0x7f872c101e10 2024-04-22T15:20:13.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.192+0000 7f87297fa700 1 -- 172.21.15.96:0/2318938485 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8720021620 con 0x7f872c101e10 2024-04-22T15:20:13.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.192+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c19c1e0 con 0x7f872c101e10 2024-04-22T15:20:13.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.193+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f872c190790 con 0x7f872c101e10 2024-04-22T15:20:13.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.193+0000 7f87297fa700 1 -- 172.21.15.96:0/2318938485 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7f872000f5d0 con 0x7f872c101e10 2024-04-22T15:20:13.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.194+0000 7f87297fa700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f871406ea80 0x7f8714070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:13.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.194+0000 7f87297fa700 1 -- 172.21.15.96:0/2318938485 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f872008fdf0 con 0x7f872c101e10 2024-04-22T15:20:13.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.194+0000 7f872bfff700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f871406ea80 0x7f8714070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:13.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.194+0000 7f872bfff700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f871406ea80 0x7f8714070f40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f872c101c70 tx=0x7f871c00b410 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:13.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.198+0000 7f87297fa700 1 -- 172.21.15.96:0/2318938485 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f872005c990 con 0x7f872c101e10 2024-04-22T15:20:13.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.550+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f872c109800 con 0x7f872c101e10 2024-04-22T15:20:13.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.550+0000 7f87297fa700 1 -- 172.21.15.96:0/2318938485 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7f872005c520 con 0x7f872c101e10 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:20:13.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:20:13.555 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:20:13.555 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 12, 2024-04-22T15:20:13.555 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:20:13.555 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:20:13.555 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:20:13.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f871406ea80 msgr2=0x7f8714070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:13.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f871406ea80 0x7f8714070f40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f872c101c70 tx=0x7f871c00b410 comp rx=0 tx=0).stop 2024-04-22T15:20:13.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 msgr2=0x7f872c196b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:13.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 0x7f872c196b60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f87200092e0 tx=0x7f872000bd70 comp rx=0 tx=0).stop 2024-04-22T15:20:13.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 shutdown_connections 2024-04-22T15:20:13.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f871406ea80 0x7f8714070f40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f872c100c10 0x7f872c196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 --2- 172.21.15.96:0/2318938485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872c101e10 0x7f872c196b60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 >> 172.21.15.96:0/2318938485 conn(0x7f872c0fc1d0 msgr2=0x7f872c105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:13.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 shutdown_connections 2024-04-22T15:20:13.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.553+0000 7f87327c6700 1 -- 172.21.15.96:0/2318938485 wait complete. 2024-04-22T15:20:13.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.701+0000 7fd375abd700 1 -- 172.21.15.96:0/1897000104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 msgr2=0x7fd370106440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:13.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.701+0000 7fd375abd700 1 --2- 172.21.15.96:0/1897000104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 0x7fd370106440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd364009910 tx=0x7fd364009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:13.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.701+0000 7fd375abd700 1 -- 172.21.15.96:0/1897000104 shutdown_connections 2024-04-22T15:20:13.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.701+0000 7fd375abd700 1 --2- 172.21.15.96:0/1897000104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 0x7fd370106440 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.701+0000 7fd375abd700 1 --2- 172.21.15.96:0/1897000104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd370101720 0x7fd370103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.701+0000 7fd375abd700 1 -- 172.21.15.96:0/1897000104 >> 172.21.15.96:0/1897000104 conn(0x7fd3700fb110 msgr2=0x7fd3700fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:13.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.701+0000 7fd375abd700 1 -- 172.21.15.96:0/1897000104 shutdown_connections 2024-04-22T15:20:13.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.701+0000 7fd375abd700 1 -- 172.21.15.96:0/1897000104 wait complete. 2024-04-22T15:20:13.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd375abd700 1 Processor -- start 2024-04-22T15:20:13.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd375abd700 1 -- start start 2024-04-22T15:20:13.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd375abd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd370101720 0x7fd370101030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd375abd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 0x7fd3700ff6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd375abd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3700ffc10 con 0x7fd370104050 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd375abd700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3700ffd50 con 0x7fd370101720 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd36effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 0x7fd3700ff6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd36effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 0x7fd3700ff6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58786/0 (socket says 172.21.15.96:58786) 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd36effd700 1 -- 172.21.15.96:0/447658068 learned_addr learned my addr 172.21.15.96:0/447658068 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.702+0000 7fd36f7fe700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd370101720 0x7fd370101030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.703+0000 7fd36effd700 1 -- 172.21.15.96:0/447658068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd370101720 msgr2=0x7fd370101030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.703+0000 7fd36effd700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd370101720 0x7fd370101030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.703+0000 7fd36effd700 1 -- 172.21.15.96:0/447658068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3640095f0 con 0x7fd370104050 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.703+0000 7fd36effd700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 0x7fd3700ff6d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd36400b800 tx=0x7fd364005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.703+0000 7fd36cff9700 1 -- 172.21.15.96:0/447658068 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd36401d070 con 0x7fd370104050 2024-04-22T15:20:13.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.703+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3700fffd0 con 0x7fd370104050 2024-04-22T15:20:13.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.703+0000 7fd36cff9700 1 -- 172.21.15.96:0/447658068 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3640052a0 con 0x7fd370104050 2024-04-22T15:20:13.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.703+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd370100400 con 0x7fd370104050 2024-04-22T15:20:13.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.703+0000 7fd36cff9700 1 -- 172.21.15.96:0/447658068 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd364017400 con 0x7fd370104050 2024-04-22T15:20:13.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.704+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37004ebb0 con 0x7fd370104050 2024-04-22T15:20:13.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.705+0000 7fd36cff9700 1 -- 172.21.15.96:0/447658068 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) v1 ==== 92708+0+0 (secure 0 0 0) 0x7fd364003780 con 0x7fd370104050 2024-04-22T15:20:13.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.706+0000 7fd36cff9700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd35806ea80 0x7fd358070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:13.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.706+0000 7fd36cff9700 1 -- 172.21.15.96:0/447658068 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fd36408ff60 con 0x7fd370104050 2024-04-22T15:20:13.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.706+0000 7fd36f7fe700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd35806ea80 0x7fd358070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:13.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.707+0000 7fd36f7fe700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd35806ea80 0x7fd358070f40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd360009c60 tx=0x7fd360009260 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:13.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:13.714+0000 7fd36cff9700 1 -- 172.21.15.96:0/447658068 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fd3640c50c0 con 0x7fd370104050 2024-04-22T15:20:13.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:13 smithi196 ceph-mon[34717]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:13.830 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:13 smithi196 ceph-mon[34717]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:13 smithi096 ceph-mon[26688]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:13 smithi096 ceph-mon[26688]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:13 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2318938485' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:14.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.003+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd370105ee0 con 0x7fd35806ea80 2024-04-22T15:20:14.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.004+0000 7fd36cff9700 1 -- 172.21.15.96:0/447658068 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fd370105ee0 con 0x7fd35806ea80 2024-04-22T15:20:14.007 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:20:14.007 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:20:14.007 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:20:14.007 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:20:14.008 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:20:14.008 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:20:14.008 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:20:14.008 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:20:14.008 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mgr daemons", 2024-04-22T15:20:14.008 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:20:14.008 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:20:14.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd35806ea80 msgr2=0x7fd358070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:14.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd35806ea80 0x7fd358070f40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd360009c60 tx=0x7fd360009260 comp rx=0 tx=0).stop 2024-04-22T15:20:14.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 msgr2=0x7fd3700ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:14.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 0x7fd3700ff6d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd36400b800 tx=0x7fd364005f00 comp rx=0 tx=0).stop 2024-04-22T15:20:14.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 shutdown_connections 2024-04-22T15:20:14.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd35806ea80 0x7fd358070f40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:14.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd370101720 0x7fd370101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:14.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 --2- 172.21.15.96:0/447658068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd370104050 0x7fd3700ff6d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:14.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 >> 172.21.15.96:0/447658068 conn(0x7fd3700fb110 msgr2=0x7fd3700fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:14.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 shutdown_connections 2024-04-22T15:20:14.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:14.007+0000 7fd375abd700 1 -- 172.21.15.96:0/447658068 wait complete. 2024-04-22T15:20:14.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:13 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2318938485' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:14.964 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:14 smithi196 ceph-mon[34717]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:14.965 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:14 smithi196 ceph-mon[34717]: pgmap v29: 65 pgs: 65 active+clean; 306 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 305 op/s 2024-04-22T15:20:15.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:14 smithi096 ceph-mon[26688]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:15.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:14 smithi096 ceph-mon[26688]: pgmap v29: 65 pgs: 65 active+clean; 306 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 305 op/s 2024-04-22T15:20:16.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:15 smithi096 ceph-mon[26688]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:16.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:15 smithi096 ceph-mon[26688]: Standby manager daemon smithi196.gdaegc restarted 2024-04-22T15:20:16.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:15 smithi096 ceph-mon[26688]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:20:16.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:15 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/315392122' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:20:16.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:15 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/315392122' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:20:16.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:15 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/315392122' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:20:16.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:15 smithi096 ceph-mon[26688]: from='mgr.? 172.21.15.196:0/315392122' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:20:16.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:15 smithi196 ceph-mon[34717]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:16.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:15 smithi196 ceph-mon[34717]: Standby manager daemon smithi196.gdaegc restarted 2024-04-22T15:20:16.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:15 smithi196 ceph-mon[34717]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:20:16.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:15 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/315392122' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:20:16.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:15 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/315392122' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:20:16.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:15 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/315392122' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:20:16.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:15 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/315392122' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:20:16.993 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:16 smithi196 ceph-mon[34717]: pgmap v30: 65 pgs: 65 active+clean; 306 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 250 op/s 2024-04-22T15:20:16.994 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:16 smithi196 ceph-mon[34717]: mgrmap e31: smithi096.hhkvum(active, since 54s), standbys: smithi196.gdaegc 2024-04-22T15:20:17.213 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:16 smithi096 ceph-mon[26688]: pgmap v30: 65 pgs: 65 active+clean; 306 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 250 op/s 2024-04-22T15:20:17.213 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:16 smithi096 ceph-mon[26688]: mgrmap e31: smithi096.hhkvum(active, since 54s), standbys: smithi196.gdaegc 2024-04-22T15:20:18.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:18 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:18.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:18 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:18.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:18 smithi096 ceph-mon[26688]: pgmap v31: 65 pgs: 65 active+clean; 306 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 260 op/s 2024-04-22T15:20:18.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:18 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:18.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:18 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:18.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:18 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:20:18.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:18 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:18.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:18 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:18.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:18 smithi196 ceph-mon[34717]: pgmap v31: 65 pgs: 65 active+clean; 306 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 260 op/s 2024-04-22T15:20:18.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:18 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:18.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:18 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:18.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:18 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: Upgrade: Setting container_image for all mgr 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi096.hhkvum"}]: dispatch 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi096.hhkvum"}]': finished 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi196.gdaegc"}]: dispatch 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi196.gdaegc"}]': finished 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: Upgrade: Setting container_image for all rgw 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: Upgrade: Setting container_image for all iscsi 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: Upgrade: Setting container_image for all nfs 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: Upgrade: Setting container_image for all nvmeof 2024-04-22T15:20:19.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:19 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: Upgrade: Setting container_image for all mgr 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi096.hhkvum"}]: dispatch 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi096.hhkvum"}]': finished 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi196.gdaegc"}]: dispatch 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi196.gdaegc"}]': finished 2024-04-22T15:20:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: Upgrade: Setting container_image for all rgw 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: Upgrade: Setting container_image for all iscsi 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: Upgrade: Setting container_image for all nfs 2024-04-22T15:20:19.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:19.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:19.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: Upgrade: Setting container_image for all nvmeof 2024-04-22T15:20:19.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:19 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:20.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:20 smithi096 ceph-mon[26688]: pgmap v32: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 341 op/s 2024-04-22T15:20:20.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:20 smithi196 ceph-mon[34717]: pgmap v32: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 341 op/s 2024-04-22T15:20:21.076 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:21 smithi096 ceph-mon[26688]: Upgrade: Updating node-exporter.smithi096 (1/2) 2024-04-22T15:20:21.076 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:21 smithi096 ceph-mon[26688]: Deploying daemon node-exporter.smithi096 on smithi096 2024-04-22T15:20:21.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:21 smithi196 ceph-mon[34717]: Upgrade: Updating node-exporter.smithi096 (1/2) 2024-04-22T15:20:21.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:21 smithi196 ceph-mon[34717]: Deploying daemon node-exporter.smithi096 on smithi096 2024-04-22T15:20:22.347 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:22 smithi096 ceph-mon[26688]: pgmap v33: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 291 op/s 2024-04-22T15:20:22.348 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:22 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:20:22.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:22 smithi196 ceph-mon[34717]: pgmap v33: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 291 op/s 2024-04-22T15:20:22.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:22 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:20:25.170 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:24 smithi096 ceph-mon[26688]: pgmap v34: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 391 op/s 2024-04-22T15:20:25.170 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:24 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:25.170 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:24 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:25.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:24 smithi196 ceph-mon[34717]: pgmap v34: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 391 op/s 2024-04-22T15:20:25.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:24 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:25.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:24 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:26.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:25 smithi096 ceph-mon[26688]: Upgrade: Updating node-exporter.smithi196 (2/2) 2024-04-22T15:20:26.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:25 smithi096 ceph-mon[26688]: Deploying daemon node-exporter.smithi196 on smithi196 2024-04-22T15:20:26.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:25 smithi096 ceph-mon[26688]: pgmap v35: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 190 op/s 2024-04-22T15:20:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:25 smithi196 ceph-mon[34717]: Upgrade: Updating node-exporter.smithi196 (2/2) 2024-04-22T15:20:26.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:25 smithi196 ceph-mon[34717]: Deploying daemon node-exporter.smithi196 on smithi196 2024-04-22T15:20:26.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:25 smithi196 ceph-mon[34717]: pgmap v35: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 190 op/s 2024-04-22T15:20:29.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:28 smithi096 ceph-mon[26688]: pgmap v36: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 228 op/s 2024-04-22T15:20:29.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:29.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:29.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:29.244 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:28 smithi196 ceph-mon[34717]: pgmap v36: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 228 op/s 2024-04-22T15:20:29.245 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:29.245 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:29.245 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:31.018 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:30 smithi196 ceph-mon[34717]: pgmap v37: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 253 op/s 2024-04-22T15:20:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:30 smithi096 ceph-mon[26688]: pgmap v37: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 253 op/s 2024-04-22T15:20:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:32 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:32 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:32 smithi096 ceph-mon[26688]: pgmap v38: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-22T15:20:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:32 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:32 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:32 smithi196 ceph-mon[34717]: pgmap v38: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-22T15:20:34.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:33 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:34.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:33 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:34.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:33 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:34.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:33 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:35.003 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:34 smithi096 ceph-mon[26688]: pgmap v39: 65 pgs: 65 active+clean; 284 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 261 op/s 2024-04-22T15:20:35.141 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:34 smithi196 ceph-mon[34717]: pgmap v39: 65 pgs: 65 active+clean; 284 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 261 op/s 2024-04-22T15:20:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:36 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:36 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:36 smithi096 ceph-mon[26688]: pgmap v40: 65 pgs: 65 active+clean; 284 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-22T15:20:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:36 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:36 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:36 smithi196 ceph-mon[34717]: pgmap v40: 65 pgs: 65 active+clean; 284 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-22T15:20:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:37 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:37 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:37 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:20:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:37 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:37 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:37 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:37 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:37 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:20:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:37 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:37 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:38 smithi096 ceph-mon[26688]: pgmap v41: 65 pgs: 65 active+clean; 280 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 268 op/s 2024-04-22T15:20:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:38 smithi196 ceph-mon[34717]: pgmap v41: 65 pgs: 65 active+clean; 280 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 268 op/s 2024-04-22T15:20:40.912 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-22T15:20:40.913 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-22T15:20:41.211 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:40 smithi196 ceph-mon[34717]: pgmap v42: 65 pgs: 65 active+clean; 280 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 230 op/s 2024-04-22T15:20:41.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:40 smithi096 ceph-mon[26688]: pgmap v42: 65 pgs: 65 active+clean; 280 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 230 op/s 2024-04-22T15:20:41.263 DEBUG:teuthology.parallel:result is None 2024-04-22T15:20:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:42 smithi096 ceph-mon[26688]: pgmap v43: 65 pgs: 65 active+clean; 280 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 195 op/s 2024-04-22T15:20:43.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:42 smithi196 ceph-mon[34717]: pgmap v43: 65 pgs: 65 active+clean; 280 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 195 op/s 2024-04-22T15:20:44.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.158+0000 7fb371f2c700 1 -- 172.21.15.96:0/3137799750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c100c40 msgr2=0x7fb36c101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.158+0000 7fb371f2c700 1 --2- 172.21.15.96:0/3137799750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c100c40 0x7fb36c101060 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009910 tx=0x7fb35c009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:44.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.158+0000 7fb371f2c700 1 -- 172.21.15.96:0/3137799750 shutdown_connections 2024-04-22T15:20:44.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.158+0000 7fb371f2c700 1 --2- 172.21.15.96:0/3137799750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb36c101e40 0x7fb36c1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.158+0000 7fb371f2c700 1 --2- 172.21.15.96:0/3137799750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c100c40 0x7fb36c101060 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.158+0000 7fb371f2c700 1 -- 172.21.15.96:0/3137799750 >> 172.21.15.96:0/3137799750 conn(0x7fb36c0fc200 msgr2=0x7fb36c0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:44.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.158+0000 7fb371f2c700 1 -- 172.21.15.96:0/3137799750 shutdown_connections 2024-04-22T15:20:44.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.158+0000 7fb371f2c700 1 -- 172.21.15.96:0/3137799750 wait complete. 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.159+0000 7fb371f2c700 1 Processor -- start 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.159+0000 7fb371f2c700 1 -- start start 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.159+0000 7fb371f2c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb36c100c40 0x7fb36c196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.159+0000 7fb371f2c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c101e40 0x7fb36c196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.159+0000 7fb371f2c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c197160 con 0x7fb36c101e40 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.159+0000 7fb371f2c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c1972a0 con 0x7fb36c100c40 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.159+0000 7fb36affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c101e40 0x7fb36c196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.159+0000 7fb36affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c101e40 0x7fb36c196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47862/0 (socket says 172.21.15.96:47862) 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.159+0000 7fb36affd700 1 -- 172.21.15.96:0/632564401 learned_addr learned my addr 172.21.15.96:0/632564401 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb36b7fe700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb36c100c40 0x7fb36c196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:44.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb36affd700 1 -- 172.21.15.96:0/632564401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb36c100c40 msgr2=0x7fb36c196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb36affd700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb36c100c40 0x7fb36c196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb36affd700 1 -- 172.21.15.96:0/632564401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c0095f0 con 0x7fb36c101e40 2024-04-22T15:20:44.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb36b7fe700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb36c100c40 0x7fb36c196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:20:44.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb36affd700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c101e40 0x7fb36c196b40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb36000d900 tx=0x7fb36000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:44.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb368ff9700 1 -- 172.21.15.96:0/632564401 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3600041f0 con 0x7fb36c101e40 2024-04-22T15:20:44.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c19bd00 con 0x7fb36c101e40 2024-04-22T15:20:44.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb368ff9700 1 -- 172.21.15.96:0/632564401 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb360004cf0 con 0x7fb36c101e40 2024-04-22T15:20:44.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb368ff9700 1 -- 172.21.15.96:0/632564401 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36000b750 con 0x7fb36c101e40 2024-04-22T15:20:44.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.160+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c19c250 con 0x7fb36c101e40 2024-04-22T15:20:44.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.162+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36c1907c0 con 0x7fb36c101e40 2024-04-22T15:20:44.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.162+0000 7fb368ff9700 1 -- 172.21.15.96:0/632564401 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 31) v1 ==== 92708+0+0 (secure 0 0 0) 0x7fb360004750 con 0x7fb36c101e40 2024-04-22T15:20:44.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.163+0000 7fb368ff9700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb35406ea80 0x7fb354070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:44.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.163+0000 7fb368ff9700 1 -- 172.21.15.96:0/632564401 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb36008eb10 con 0x7fb36c101e40 2024-04-22T15:20:44.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.163+0000 7fb36b7fe700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb35406ea80 0x7fb354070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:44.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.163+0000 7fb36b7fe700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb35406ea80 0x7fb354070f40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00b5c0 tx=0x7fb35c005960 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:44.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.168+0000 7fb368ff9700 1 -- 172.21.15.96:0/632564401 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fb36005b6b0 con 0x7fb36c101e40 2024-04-22T15:20:44.439 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-22T15:20:44.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-22T15:20:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.470+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb36c02ce80 con 0x7fb35406ea80 2024-04-22T15:20:44.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.471+0000 7fb368ff9700 1 -- 172.21.15.96:0/632564401 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7fb36c02ce80 con 0x7fb35406ea80 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.473+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb35406ea80 msgr2=0x7fb354070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.473+0000 7fb371f2c700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb35406ea80 0x7fb354070f40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00b5c0 tx=0x7fb35c005960 comp rx=0 tx=0).stop 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.473+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c101e40 msgr2=0x7fb36c196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.473+0000 7fb371f2c700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c101e40 0x7fb36c196b40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb36000d900 tx=0x7fb36000dc10 comp rx=0 tx=0).stop 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.473+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 shutdown_connections 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.473+0000 7fb371f2c700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb35406ea80 0x7fb354070f40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.473+0000 7fb371f2c700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb36c100c40 0x7fb36c196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.473+0000 7fb371f2c700 1 --2- 172.21.15.96:0/632564401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb36c101e40 0x7fb36c196b40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.474+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 >> 172.21.15.96:0/632564401 conn(0x7fb36c0fc200 msgr2=0x7fb36c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.474+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 shutdown_connections 2024-04-22T15:20:44.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.474+0000 7fb371f2c700 1 -- 172.21.15.96:0/632564401 wait complete. 2024-04-22T15:20:44.487 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:20:44.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.578+0000 7fe26f414700 1 -- 172.21.15.96:0/1268562353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 msgr2=0x7fe2680fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.578+0000 7fe26f414700 1 --2- 172.21.15.96:0/1268562353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 0x7fe2680fffb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe264009910 tx=0x7fe264009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:44.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.578+0000 7fe26f414700 1 -- 172.21.15.96:0/1268562353 shutdown_connections 2024-04-22T15:20:44.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.578+0000 7fe26f414700 1 --2- 172.21.15.96:0/1268562353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe268100d90 0x7fe2681011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.578+0000 7fe26f414700 1 --2- 172.21.15.96:0/1268562353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 0x7fe2680fffb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.578+0000 7fe26f414700 1 -- 172.21.15.96:0/1268562353 >> 172.21.15.96:0/1268562353 conn(0x7fe2680fb110 msgr2=0x7fe2680fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:44.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.578+0000 7fe26f414700 1 -- 172.21.15.96:0/1268562353 shutdown_connections 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.578+0000 7fe26f414700 1 -- 172.21.15.96:0/1268562353 wait complete. 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.578+0000 7fe26f414700 1 Processor -- start 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26f414700 1 -- start start 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26f414700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 0x7fe268103a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26f414700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe268100d90 0x7fe268102170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26f414700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681026b0 con 0x7fe2680ffb90 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26f414700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681027f0 con 0x7fe268100d90 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26d1b0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 0x7fe268103a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26d1b0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 0x7fe268103a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47886/0 (socket says 172.21.15.96:47886) 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26d1b0700 1 -- 172.21.15.96:0/647213909 learned_addr learned my addr 172.21.15.96:0/647213909 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26c9af700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe268100d90 0x7fe268102170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26d1b0700 1 -- 172.21.15.96:0/647213909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe268100d90 msgr2=0x7fe268102170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26d1b0700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe268100d90 0x7fe268102170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26d1b0700 1 -- 172.21.15.96:0/647213909 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2640095f0 con 0x7fe2680ffb90 2024-04-22T15:20:44.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26d1b0700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 0x7fe268103a80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe2640059d0 tx=0x7fe26400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:44.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe25e7fc700 1 -- 172.21.15.96:0/647213909 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe26401d070 con 0x7fe2680ffb90 2024-04-22T15:20:44.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe25e7fc700 1 -- 172.21.15.96:0/647213909 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe264005dc0 con 0x7fe2680ffb90 2024-04-22T15:20:44.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe268102a70 con 0x7fe2680ffb90 2024-04-22T15:20:44.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.579+0000 7fe25e7fc700 1 -- 172.21.15.96:0/647213909 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe264021410 con 0x7fe2680ffb90 2024-04-22T15:20:44.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.580+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe268102f60 con 0x7fe2680ffb90 2024-04-22T15:20:44.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.580+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26804ebb0 con 0x7fe2680ffb90 2024-04-22T15:20:44.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.581+0000 7fe25e7fc700 1 -- 172.21.15.96:0/647213909 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 31) v1 ==== 92708+0+0 (secure 0 0 0) 0x7fe26400f460 con 0x7fe2680ffb90 2024-04-22T15:20:44.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.581+0000 7fe25e7fc700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe25406ea80 0x7fe254070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:44.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.582+0000 7fe25e7fc700 1 -- 172.21.15.96:0/647213909 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fe26408fc50 con 0x7fe2680ffb90 2024-04-22T15:20:44.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.582+0000 7fe26c9af700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe25406ea80 0x7fe254070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:44.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.582+0000 7fe26c9af700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe25406ea80 0x7fe254070f40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe258005950 tx=0x7fe25800b410 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:44.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.586+0000 7fe25e7fc700 1 -- 172.21.15.96:0/647213909 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fe26405c870 con 0x7fe2680ffb90 2024-04-22T15:20:44.710 DEBUG:teuthology.parallel:result is None 2024-04-22T15:20:44.710 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T15:20:44.742 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T15:20:44.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-22T15:20:44.774 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-22T15:20:44.775 DEBUG:teuthology.parallel:result is None 2024-04-22T15:20:44.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.818+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe268068d40 con 0x7fe25406ea80 2024-04-22T15:20:44.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.819+0000 7fe25e7fc700 1 -- 172.21.15.96:0/647213909 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7fe268068d40 con 0x7fe25406ea80 2024-04-22T15:20:44.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe25406ea80 msgr2=0x7fe254070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe25406ea80 0x7fe254070f40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe258005950 tx=0x7fe25800b410 comp rx=0 tx=0).stop 2024-04-22T15:20:44.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 msgr2=0x7fe268103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 0x7fe268103a80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe2640059d0 tx=0x7fe26400bd70 comp rx=0 tx=0).stop 2024-04-22T15:20:44.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 shutdown_connections 2024-04-22T15:20:44.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe25406ea80 0x7fe254070f40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2680ffb90 0x7fe268103a80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 --2- 172.21.15.96:0/647213909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe268100d90 0x7fe268102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 >> 172.21.15.96:0/647213909 conn(0x7fe2680fb110 msgr2=0x7fe268068630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:44.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 shutdown_connections 2024-04-22T15:20:44.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.821+0000 7fe26f414700 1 -- 172.21.15.96:0/647213909 wait complete. 2024-04-22T15:20:44.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.929+0000 7f5af74f0700 1 -- 172.21.15.96:0/3597548998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 msgr2=0x7f5af00fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.929+0000 7f5af74f0700 1 --2- 172.21.15.96:0/3597548998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 0x7f5af00fff40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009910 tx=0x7f5aec009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:44.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.929+0000 7f5af74f0700 1 -- 172.21.15.96:0/3597548998 shutdown_connections 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.929+0000 7f5af74f0700 1 --2- 172.21.15.96:0/3597548998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5af0100d20 0x7f5af0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 --2- 172.21.15.96:0/3597548998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 0x7f5af00fff40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 -- 172.21.15.96:0/3597548998 >> 172.21.15.96:0/3597548998 conn(0x7f5af00fb0c0 msgr2=0x7f5af00fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 -- 172.21.15.96:0/3597548998 shutdown_connections 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 -- 172.21.15.96:0/3597548998 wait complete. 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 Processor -- start 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 -- start start 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 0x7f5af0103e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5af0100d20 0x7f5af0102520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0102a60 con 0x7f5af00ffb20 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.930+0000 7f5af74f0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0102ba0 con 0x7f5af0100d20 2024-04-22T15:20:44.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af528c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 0x7f5af0103e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af528c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 0x7f5af0103e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47908/0 (socket says 172.21.15.96:47908) 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af528c700 1 -- 172.21.15.96:0/3080673547 learned_addr learned my addr 172.21.15.96:0/3080673547 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af4a8b700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5af0100d20 0x7f5af0102520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af528c700 1 -- 172.21.15.96:0/3080673547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5af0100d20 msgr2=0x7f5af0102520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af528c700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5af0100d20 0x7f5af0102520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af528c700 1 -- 172.21.15.96:0/3080673547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0095f0 con 0x7f5af00ffb20 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af528c700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 0x7f5af0103e80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0059d0 tx=0x7f5aec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5ae67fc700 1 -- 172.21.15.96:0/3080673547 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aec01d070 con 0x7f5af00ffb20 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af0102e20 con 0x7f5af00ffb20 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5ae67fc700 1 -- 172.21.15.96:0/3080673547 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5aec00f460 con 0x7f5af00ffb20 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5ae67fc700 1 -- 172.21.15.96:0/3080673547 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aec021620 con 0x7f5af00ffb20 2024-04-22T15:20:44.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.931+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af0103390 con 0x7f5af00ffb20 2024-04-22T15:20:44.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.932+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af004ebb0 con 0x7f5af00ffb20 2024-04-22T15:20:44.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.933+0000 7f5ae67fc700 1 -- 172.21.15.96:0/3080673547 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 31) v1 ==== 92708+0+0 (secure 0 0 0) 0x7f5aec00fa70 con 0x7f5af00ffb20 2024-04-22T15:20:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.933+0000 7f5ae67fc700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5adc06ea80 0x7f5adc070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.933+0000 7f5ae67fc700 1 -- 172.21.15.96:0/3080673547 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f5aec08fda0 con 0x7f5af00ffb20 2024-04-22T15:20:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.933+0000 7f5af4a8b700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5adc06ea80 0x7f5adc070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.934+0000 7f5af4a8b700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5adc06ea80 0x7f5adc070f40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005fd0 tx=0x7f5ae000a560 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:44.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:44.938+0000 7f5ae67fc700 1 -- 172.21.15.96:0/3080673547 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f5aec05c940 con 0x7f5af00ffb20 2024-04-22T15:20:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:44 smithi096 ceph-mon[26688]: pgmap v44: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 285 op/s 2024-04-22T15:20:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:44 smithi196 ceph-mon[34717]: pgmap v44: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 285 op/s 2024-04-22T15:20:45.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.200+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5af0103390 con 0x7f5adc06ea80 2024-04-22T15:20:45.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.220+0000 7f5ae67fc700 1 -- 172.21.15.96:0/3080673547 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f5af0103390 con 0x7f5adc06ea80 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (9m) 9s ago 10m 25.0M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (11m) 9s ago 11m 8090k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (9m) 13s ago 9m 7864k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (11m) 9s ago 11m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (9m) 13s ago 9m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (9m) 9s ago 10m 86.1M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (5m) 9s ago 5m 202M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (5m) 9s ago 5m 193M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:20:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (5m) 13s ago 5m 26.4M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (4m) 13s ago 4m 25.8M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (108s) 9s ago 12m 499M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (38s) 13s ago 9m 449M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (12m) 9s ago 12m 57.4M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (9m) 13s ago 9m 44.9M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (22s) 9s ago 10m 3426k - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (17s) 13s ago 9m 3602k - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (8m) 9s ago 8m 197M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (8m) 9s ago 8m 210M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (7m) 9s ago 7m 175M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (7m) 13s ago 7m 231M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (7m) 13s ago 7m 208M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (6m) 13s ago 6m 185M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:20:45.225 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (44s) 9s ago 10m 56.9M - 2.43.0 a07b618ecd1d 36521bf72ed8 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5adc06ea80 msgr2=0x7f5adc070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5adc06ea80 0x7f5adc070f40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005fd0 tx=0x7f5ae000a560 comp rx=0 tx=0).stop 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 msgr2=0x7f5af0103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 0x7f5af0103e80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0059d0 tx=0x7f5aec00be20 comp rx=0 tx=0).stop 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 shutdown_connections 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5adc06ea80 0x7f5adc070f40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5af00ffb20 0x7f5af0103e80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 --2- 172.21.15.96:0/3080673547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5af0100d20 0x7f5af0102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 >> 172.21.15.96:0/3080673547 conn(0x7f5af00fb0c0 msgr2=0x7f5af00690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:45.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 shutdown_connections 2024-04-22T15:20:45.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.224+0000 7f5af74f0700 1 -- 172.21.15.96:0/3080673547 wait complete. 2024-04-22T15:20:45.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.330+0000 7fdf89d94700 1 -- 172.21.15.96:0/2419222558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf84100d90 msgr2=0x7fdf841011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.330+0000 7fdf89d94700 1 --2- 172.21.15.96:0/2419222558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf84100d90 0x7fdf841011f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009910 tx=0x7fdf78009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:45.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.330+0000 7fdf89d94700 1 -- 172.21.15.96:0/2419222558 shutdown_connections 2024-04-22T15:20:45.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.330+0000 7fdf89d94700 1 --2- 172.21.15.96:0/2419222558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf84100d90 0x7fdf841011f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.330+0000 7fdf89d94700 1 --2- 172.21.15.96:0/2419222558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf840ffb90 0x7fdf840fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.330+0000 7fdf89d94700 1 -- 172.21.15.96:0/2419222558 >> 172.21.15.96:0/2419222558 conn(0x7fdf840fb110 msgr2=0x7fdf840fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:45.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.330+0000 7fdf89d94700 1 -- 172.21.15.96:0/2419222558 shutdown_connections 2024-04-22T15:20:45.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.330+0000 7fdf89d94700 1 -- 172.21.15.96:0/2419222558 wait complete. 2024-04-22T15:20:45.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.330+0000 7fdf89d94700 1 Processor -- start 2024-04-22T15:20:45.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf89d94700 1 -- start start 2024-04-22T15:20:45.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf89d94700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf840ffb90 0x7fdf84195580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:45.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf89d94700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf84100d90 0x7fdf84195ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:45.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf89d94700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf841960e0 con 0x7fdf840ffb90 2024-04-22T15:20:45.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf89d94700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf84196220 con 0x7fdf84100d90 2024-04-22T15:20:45.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf837fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf840ffb90 0x7fdf84195580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf837fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf840ffb90 0x7fdf84195580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47936/0 (socket says 172.21.15.96:47936) 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf837fe700 1 -- 172.21.15.96:0/4121778674 learned_addr learned my addr 172.21.15.96:0/4121778674 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf82ffd700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf84100d90 0x7fdf84195ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf837fe700 1 -- 172.21.15.96:0/4121778674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf84100d90 msgr2=0x7fdf84195ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf837fe700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf84100d90 0x7fdf84195ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf837fe700 1 -- 172.21.15.96:0/4121778674 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf780095f0 con 0x7fdf840ffb90 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf837fe700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf840ffb90 0x7fdf84195580 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400d900 tx=0x7fdf7400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.331+0000 7fdf80ff9700 1 -- 172.21.15.96:0/4121778674 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf740041f0 con 0x7fdf840ffb90 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.332+0000 7fdf80ff9700 1 -- 172.21.15.96:0/4121778674 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf74004350 con 0x7fdf840ffb90 2024-04-22T15:20:45.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.332+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8419ac10 con 0x7fdf840ffb90 2024-04-22T15:20:45.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.332+0000 7fdf80ff9700 1 -- 172.21.15.96:0/4121778674 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf74003cf0 con 0x7fdf840ffb90 2024-04-22T15:20:45.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.332+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf840688a0 con 0x7fdf840ffb90 2024-04-22T15:20:45.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.333+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8404ebb0 con 0x7fdf840ffb90 2024-04-22T15:20:45.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.333+0000 7fdf80ff9700 1 -- 172.21.15.96:0/4121778674 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 31) v1 ==== 92708+0+0 (secure 0 0 0) 0x7fdf74009730 con 0x7fdf840ffb90 2024-04-22T15:20:45.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.333+0000 7fdf80ff9700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdf6c06ea80 0x7fdf6c070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:45.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.333+0000 7fdf80ff9700 1 -- 172.21.15.96:0/4121778674 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fdf7408eac0 con 0x7fdf840ffb90 2024-04-22T15:20:45.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.333+0000 7fdf82ffd700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdf6c06ea80 0x7fdf6c070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:45.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.334+0000 7fdf82ffd700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdf6c06ea80 0x7fdf6c070f40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdf78005090 tx=0x7fdf78005960 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:45.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.338+0000 7fdf80ff9700 1 -- 172.21.15.96:0/4121778674 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fdf7405b660 con 0x7fdf840ffb90 2024-04-22T15:20:45.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.625+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdf8418f0f0 con 0x7fdf840ffb90 2024-04-22T15:20:45.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.625+0000 7fdf80ff9700 1 -- 172.21.15.96:0/4121778674 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7fdf7405b1f0 con 0x7fdf840ffb90 2024-04-22T15:20:45.629 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:20:45.629 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:20:45.629 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:20:45.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:20:45.629 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:20:45.629 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:20:45.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 12, 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:20:45.630 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:20:45.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.627+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdf6c06ea80 msgr2=0x7fdf6c070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.627+0000 7fdf89d94700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdf6c06ea80 0x7fdf6c070f40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdf78005090 tx=0x7fdf78005960 comp rx=0 tx=0).stop 2024-04-22T15:20:45.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.627+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf840ffb90 msgr2=0x7fdf84195580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.627+0000 7fdf89d94700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf840ffb90 0x7fdf84195580 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400d900 tx=0x7fdf7400dc10 comp rx=0 tx=0).stop 2024-04-22T15:20:45.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.628+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 shutdown_connections 2024-04-22T15:20:45.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.628+0000 7fdf89d94700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fdf6c06ea80 0x7fdf6c070f40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.628+0000 7fdf89d94700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf840ffb90 0x7fdf84195580 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.628+0000 7fdf89d94700 1 --2- 172.21.15.96:0/4121778674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf84100d90 0x7fdf84195ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.628+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 >> 172.21.15.96:0/4121778674 conn(0x7fdf840fb110 msgr2=0x7fdf841043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:45.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.628+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 shutdown_connections 2024-04-22T15:20:45.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.628+0000 7fdf89d94700 1 -- 172.21.15.96:0/4121778674 wait complete. 2024-04-22T15:20:45.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 -- 172.21.15.96:0/2936312382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb800ffd80 msgr2=0x7fbb80100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 --2- 172.21.15.96:0/2936312382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb800ffd80 0x7fbb80100200 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009910 tx=0x7fbb70009c20 comp rx=0 tx=0).stop 2024-04-22T15:20:45.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 -- 172.21.15.96:0/2936312382 shutdown_connections 2024-04-22T15:20:45.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 --2- 172.21.15.96:0/2936312382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb800ffd80 0x7fbb80100200 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 --2- 172.21.15.96:0/2936312382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb80101120 0x7fbb80101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 -- 172.21.15.96:0/2936312382 >> 172.21.15.96:0/2936312382 conn(0x7fbb800fa830 msgr2=0x7fbb800fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:45.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 -- 172.21.15.96:0/2936312382 shutdown_connections 2024-04-22T15:20:45.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 -- 172.21.15.96:0/2936312382 wait complete. 2024-04-22T15:20:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 Processor -- start 2024-04-22T15:20:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.733+0000 7fbb88118700 1 -- start start 2024-04-22T15:20:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb88118700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb800ffd80 0x7fbb80103ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:45.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb88118700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb80101120 0x7fbb80102170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb88118700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb801026b0 con 0x7fbb80101120 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb88118700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb801027f0 con 0x7fbb800ffd80 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb856b3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb80101120 0x7fbb80102170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb856b3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb80101120 0x7fbb80102170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47956/0 (socket says 172.21.15.96:47956) 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb856b3700 1 -- 172.21.15.96:0/165026622 learned_addr learned my addr 172.21.15.96:0/165026622 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb85eb4700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb800ffd80 0x7fbb80103ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb856b3700 1 -- 172.21.15.96:0/165026622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb800ffd80 msgr2=0x7fbb80103ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb856b3700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb800ffd80 0x7fbb80103ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb856b3700 1 -- 172.21.15.96:0/165026622 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb700095f0 con 0x7fbb80101120 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb856b3700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb80101120 0x7fbb80102170 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbb70006010 tx=0x7fbb7000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:45.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb76ffd700 1 -- 172.21.15.96:0/165026622 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb7001d070 con 0x7fbb80101120 2024-04-22T15:20:45.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb80102a70 con 0x7fbb80101120 2024-04-22T15:20:45.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.734+0000 7fbb76ffd700 1 -- 172.21.15.96:0/165026622 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb7000f460 con 0x7fbb80101120 2024-04-22T15:20:45.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.735+0000 7fbb76ffd700 1 -- 172.21.15.96:0/165026622 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb70021640 con 0x7fbb80101120 2024-04-22T15:20:45.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.735+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb80102fe0 con 0x7fbb80101120 2024-04-22T15:20:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.736+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8004ebb0 con 0x7fbb80101120 2024-04-22T15:20:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.736+0000 7fbb76ffd700 1 -- 172.21.15.96:0/165026622 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 31) v1 ==== 92708+0+0 (secure 0 0 0) 0x7fbb7000fab0 con 0x7fbb80101120 2024-04-22T15:20:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.736+0000 7fbb76ffd700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fbb6c06ea80 0x7fbb6c070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:20:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.736+0000 7fbb76ffd700 1 -- 172.21.15.96:0/165026622 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fbb7008fb30 con 0x7fbb80101120 2024-04-22T15:20:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.736+0000 7fbb85eb4700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fbb6c06ea80 0x7fbb6c070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:20:45.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.737+0000 7fbb85eb4700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fbb6c06ea80 0x7fbb6c070f40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005fd0 tx=0x7fbb7c005dd0 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:20:45.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.741+0000 7fbb76ffd700 1 -- 172.21.15.96:0/165026622 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fbb7005c780 con 0x7fbb80101120 2024-04-22T15:20:45.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.964+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb80102fe0 con 0x7fbb6c06ea80 2024-04-22T15:20:45.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:45 smithi096 ceph-mon[26688]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:45.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:45 smithi096 ceph-mon[26688]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:45.968 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:45 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/4121778674' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.966+0000 7fbb76ffd700 1 -- 172.21.15.96:0/165026622 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7fbb80102fe0 con 0x7fbb6c06ea80 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-22T15:20:45.969 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:20:45.970 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:20:45.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fbb6c06ea80 msgr2=0x7fbb6c070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fbb6c06ea80 0x7fbb6c070f40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005fd0 tx=0x7fbb7c005dd0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb80101120 msgr2=0x7fbb80102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:20:45.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb80101120 0x7fbb80102170 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbb70006010 tx=0x7fbb7000be20 comp rx=0 tx=0).stop 2024-04-22T15:20:45.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 shutdown_connections 2024-04-22T15:20:45.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fbb6c06ea80 0x7fbb6c070f40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb800ffd80 0x7fbb80103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 --2- 172.21.15.96:0/165026622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb80101120 0x7fbb80102170 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:20:45.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 >> 172.21.15.96:0/165026622 conn(0x7fbb800fa830 msgr2=0x7fbb800fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:20:45.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 shutdown_connections 2024-04-22T15:20:45.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:20:45.968+0000 7fbb88118700 1 -- 172.21.15.96:0/165026622 wait complete. 2024-04-22T15:20:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:45 smithi196 ceph-mon[34717]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:45 smithi196 ceph-mon[34717]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:45 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/4121778674' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:47.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:46 smithi096 ceph-mon[26688]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:47.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:46 smithi096 ceph-mon[26688]: pgmap v45: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 197 op/s 2024-04-22T15:20:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:46 smithi196 ceph-mon[34717]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:46 smithi196 ceph-mon[34717]: pgmap v45: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 197 op/s 2024-04-22T15:20:48.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:47 smithi096 ceph-mon[26688]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:48.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:47 smithi196 ceph-mon[34717]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:20:49.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:48 smithi096 ceph-mon[26688]: pgmap v46: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 248 op/s 2024-04-22T15:20:49.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:48 smithi196 ceph-mon[34717]: pgmap v46: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 248 op/s 2024-04-22T15:20:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:50 smithi096 ceph-mon[26688]: pgmap v47: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 764 KiB/s rd, 740 KiB/s wr, 141 op/s 2024-04-22T15:20:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:50 smithi196 ceph-mon[34717]: pgmap v47: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 764 KiB/s rd, 740 KiB/s wr, 141 op/s 2024-04-22T15:20:52.168 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:51 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:20:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:51 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:20:53.074 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: pgmap v48: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 764 KiB/s rd, 740 KiB/s wr, 141 op/s 2024-04-22T15:20:53.074 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.075 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.076 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: pgmap v48: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 764 KiB/s rd, 740 KiB/s wr, 141 op/s 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:20:54.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:54 smithi196 ceph-mon[34717]: mgrmap e32: smithi096.hhkvum(active, since 91s), standbys: smithi196.gdaegc 2024-04-22T15:20:54.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:54 smithi196 ceph-mon[34717]: pgmap v49: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 780 KiB/s rd, 756 KiB/s wr, 143 op/s 2024-04-22T15:20:54.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:54 smithi096 ceph-mon[26688]: mgrmap e32: smithi096.hhkvum(active, since 91s), standbys: smithi196.gdaegc 2024-04-22T15:20:54.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:54 smithi096 ceph-mon[26688]: pgmap v49: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 780 KiB/s rd, 756 KiB/s wr, 143 op/s 2024-04-22T15:20:55.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:55 smithi096 ceph-mon[26688]: Upgrade: Updating prometheus.smithi096 2024-04-22T15:20:55.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:55 smithi096 ceph-mon[26688]: Deploying daemon prometheus.smithi096 on smithi096 2024-04-22T15:20:55.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:55 smithi196 ceph-mon[34717]: Upgrade: Updating prometheus.smithi096 2024-04-22T15:20:55.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:55 smithi196 ceph-mon[34717]: Deploying daemon prometheus.smithi096 on smithi096 2024-04-22T15:20:56.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:56 smithi196 ceph-mon[34717]: pgmap v50: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 238 KiB/s rd, 238 KiB/s wr, 53 op/s 2024-04-22T15:20:56.525 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:56 smithi096 ceph-mon[26688]: pgmap v50: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 238 KiB/s rd, 238 KiB/s wr, 53 op/s 2024-04-22T15:20:59.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:20:58 smithi096 ceph-mon[26688]: pgmap v51: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 239 KiB/s rd, 238 KiB/s wr, 55 op/s 2024-04-22T15:20:59.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:20:58 smithi196 ceph-mon[34717]: pgmap v51: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 239 KiB/s rd, 238 KiB/s wr, 55 op/s 2024-04-22T15:21:00.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:00 smithi096 ceph-mon[26688]: pgmap v52: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-22T15:21:00.964 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:00 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:00.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:00 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:00.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:00 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:00.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:00 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:00.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:00 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:01.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:00 smithi196 ceph-mon[34717]: pgmap v52: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-22T15:21:01.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:00 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:01.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:00 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:01.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:00 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:01.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:00 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:01.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:00 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:03.149 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:02 smithi096 ceph-mon[26688]: pgmap v53: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-22T15:21:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:02 smithi196 ceph-mon[34717]: pgmap v53: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-22T15:21:05.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:04 smithi096 ceph-mon[26688]: pgmap v54: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 16 KiB/s wr, 5 op/s 2024-04-22T15:21:05.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:04 smithi196 ceph-mon[34717]: pgmap v54: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 16 KiB/s wr, 5 op/s 2024-04-22T15:21:06.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:06 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:06.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:06 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:06.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:06 smithi096 ceph-mon[26688]: pgmap v55: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:06 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:06 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:06.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:06 smithi196 ceph-mon[34717]: pgmap v55: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:07 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:07 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:07 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:21:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:07 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:07 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:07 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:21:08.676 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:08 smithi196 ceph-mon[34717]: pgmap v56: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:08.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:08 smithi096 ceph-mon[26688]: pgmap v56: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:11.129 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: pgmap v57: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.130 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.131 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.131 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.131 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.131 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:10 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: pgmap v57: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:11.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:10 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:12.199 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:11 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:21:12.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:11 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:21:12.941 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:12 smithi096 ceph-mon[26688]: pgmap v58: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:13.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:12 smithi196 ceph-mon[34717]: pgmap v58: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:14.101 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:13 smithi096 ceph-mon[26688]: Upgrade: Updating alertmanager.smithi096 2024-04-22T15:21:14.102 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:13 smithi096 ceph-mon[26688]: Deploying daemon alertmanager.smithi096 on smithi096 2024-04-22T15:21:14.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:13 smithi196 ceph-mon[34717]: Upgrade: Updating alertmanager.smithi096 2024-04-22T15:21:14.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:13 smithi196 ceph-mon[34717]: Deploying daemon alertmanager.smithi096 on smithi096 2024-04-22T15:21:15.105 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:14 smithi096 ceph-mon[26688]: pgmap v59: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:15.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:14 smithi196 ceph-mon[34717]: pgmap v59: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:16.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 -- 172.21.15.96:0/3684993212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 msgr2=0x7f28840fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 --2- 172.21.15.96:0/3684993212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 0x7f28840fff60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2880009910 tx=0x7f2880009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:16.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 -- 172.21.15.96:0/3684993212 shutdown_connections 2024-04-22T15:21:16.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 --2- 172.21.15.96:0/3684993212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2884100d40 0x7f28841011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 --2- 172.21.15.96:0/3684993212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 0x7f28840fff60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 -- 172.21.15.96:0/3684993212 >> 172.21.15.96:0/3684993212 conn(0x7f28840fb100 msgr2=0x7f28840fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 -- 172.21.15.96:0/3684993212 shutdown_connections 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 -- 172.21.15.96:0/3684993212 wait complete. 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 Processor -- start 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.077+0000 7f288b71b700 1 -- start start 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f288b71b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 0x7f28841967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f288b71b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2884100d40 0x7f2884196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f288b71b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884197350 con 0x7f28840ffb40 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f288b71b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884197490 con 0x7f2884100d40 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f28894b7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 0x7f28841967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f28894b7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 0x7f28841967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50324/0 (socket says 172.21.15.96:50324) 2024-04-22T15:21:16.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f28894b7700 1 -- 172.21.15.96:0/1993104997 learned_addr learned my addr 172.21.15.96:0/1993104997 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f2888cb6700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2884100d40 0x7f2884196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f28894b7700 1 -- 172.21.15.96:0/1993104997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2884100d40 msgr2=0x7f2884196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f28894b7700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2884100d40 0x7f2884196d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f28894b7700 1 -- 172.21.15.96:0/1993104997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28800095f0 con 0x7f28840ffb40 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f28894b7700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 0x7f28841967f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f28800059d0 tx=0x7f288000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f287a7fc700 1 -- 172.21.15.96:0/1993104997 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f288001d070 con 0x7f28840ffb40 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288419be40 con 0x7f28840ffb40 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f287a7fc700 1 -- 172.21.15.96:0/1993104997 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f288000f460 con 0x7f28840ffb40 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f287a7fc700 1 -- 172.21.15.96:0/1993104997 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2880021620 con 0x7f28840ffb40 2024-04-22T15:21:16.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.078+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288419c330 con 0x7f28840ffb40 2024-04-22T15:21:16.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.079+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288404ebb0 con 0x7f28840ffb40 2024-04-22T15:21:16.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.080+0000 7f287a7fc700 1 -- 172.21.15.96:0/1993104997 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f288000fa70 con 0x7f28840ffb40 2024-04-22T15:21:16.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.080+0000 7f287a7fc700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f287006e940 0x7f2870070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:16.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.080+0000 7f287a7fc700 1 -- 172.21.15.96:0/1993104997 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f288008fc80 con 0x7f28840ffb40 2024-04-22T15:21:16.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.080+0000 7f2888cb6700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f287006e940 0x7f2870070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:16.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.081+0000 7f2888cb6700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f287006e940 0x7f2870070e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2874006fd0 tx=0x7f2874008040 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:16.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.085+0000 7f287a7fc700 1 -- 172.21.15.96:0/1993104997 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f288005ca40 con 0x7f28840ffb40 2024-04-22T15:21:16.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.311+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f28840698b0 con 0x7f287006e940 2024-04-22T15:21:16.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.312+0000 7f287a7fc700 1 -- 172.21.15.96:0/1993104997 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f28840698b0 con 0x7f287006e940 2024-04-22T15:21:16.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.314+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f287006e940 msgr2=0x7f2870070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.314+0000 7f288b71b700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f287006e940 0x7f2870070e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2874006fd0 tx=0x7f2874008040 comp rx=0 tx=0).stop 2024-04-22T15:21:16.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.314+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 msgr2=0x7f28841967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.314+0000 7f288b71b700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 0x7f28841967f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f28800059d0 tx=0x7f288000be20 comp rx=0 tx=0).stop 2024-04-22T15:21:16.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.315+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 shutdown_connections 2024-04-22T15:21:16.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.315+0000 7f288b71b700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f287006e940 0x7f2870070e00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.315+0000 7f288b71b700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28840ffb40 0x7f28841967f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.315+0000 7f288b71b700 1 --2- 172.21.15.96:0/1993104997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2884100d40 0x7f2884196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.315+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 >> 172.21.15.96:0/1993104997 conn(0x7f28840fb100 msgr2=0x7f28840691a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:16.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.315+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 shutdown_connections 2024-04-22T15:21:16.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.315+0000 7f288b71b700 1 -- 172.21.15.96:0/1993104997 wait complete. 2024-04-22T15:21:16.330 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.427+0000 7fe12f5fb700 1 -- 172.21.15.96:0/36982363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1280ffb10 msgr2=0x7fe1280fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.427+0000 7fe12f5fb700 1 --2- 172.21.15.96:0/36982363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1280ffb10 0x7fe1280fff30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe120009910 tx=0x7fe120009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.427+0000 7fe12f5fb700 1 -- 172.21.15.96:0/36982363 shutdown_connections 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.427+0000 7fe12f5fb700 1 --2- 172.21.15.96:0/36982363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe128100d10 0x7fe128101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.427+0000 7fe12f5fb700 1 --2- 172.21.15.96:0/36982363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1280ffb10 0x7fe1280fff30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.427+0000 7fe12f5fb700 1 -- 172.21.15.96:0/36982363 >> 172.21.15.96:0/36982363 conn(0x7fe1280fb0b0 msgr2=0x7fe1280fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.427+0000 7fe12f5fb700 1 -- 172.21.15.96:0/36982363 shutdown_connections 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.427+0000 7fe12f5fb700 1 -- 172.21.15.96:0/36982363 wait complete. 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12f5fb700 1 Processor -- start 2024-04-22T15:21:16.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12f5fb700 1 -- start start 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12f5fb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1280ffb10 0x7fe1280699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12f5fb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe128100d10 0x7fe12819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12f5fb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12819b870 con 0x7fe128100d10 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12f5fb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12819b9b0 con 0x7fe1280ffb10 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12cb96700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe128100d10 0x7fe12819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12cb96700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe128100d10 0x7fe12819b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50340/0 (socket says 172.21.15.96:50340) 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12cb96700 1 -- 172.21.15.96:0/1142723435 learned_addr learned my addr 172.21.15.96:0/1142723435 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.428+0000 7fe12d397700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1280ffb10 0x7fe1280699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.429+0000 7fe12cb96700 1 -- 172.21.15.96:0/1142723435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1280ffb10 msgr2=0x7fe1280699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.429+0000 7fe12cb96700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1280ffb10 0x7fe1280699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.429+0000 7fe12cb96700 1 -- 172.21.15.96:0/1142723435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200095f0 con 0x7fe128100d10 2024-04-22T15:21:16.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.429+0000 7fe12cb96700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe128100d10 0x7fe12819b330 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00c930 tx=0x7fe11c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:16.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.429+0000 7fe1267fc700 1 -- 172.21.15.96:0/1142723435 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe11c007ad0 con 0x7fe128100d10 2024-04-22T15:21:16.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.429+0000 7fe1267fc700 1 -- 172.21.15.96:0/1142723435 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe11c00ce80 con 0x7fe128100d10 2024-04-22T15:21:16.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.429+0000 7fe1267fc700 1 -- 172.21.15.96:0/1142723435 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe11c0186c0 con 0x7fe128100d10 2024-04-22T15:21:16.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.429+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12819bc90 con 0x7fe128100d10 2024-04-22T15:21:16.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.429+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12819c260 con 0x7fe128100d10 2024-04-22T15:21:16.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.430+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12818f690 con 0x7fe128100d10 2024-04-22T15:21:16.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.431+0000 7fe1267fc700 1 -- 172.21.15.96:0/1142723435 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7fe11c007c30 con 0x7fe128100d10 2024-04-22T15:21:16.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.431+0000 7fe1267fc700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe11006e940 0x7fe110070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:16.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.431+0000 7fe1267fc700 1 -- 172.21.15.96:0/1142723435 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fe11c08e8b0 con 0x7fe128100d10 2024-04-22T15:21:16.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.431+0000 7fe12d397700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe11006e940 0x7fe110070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:16.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.431+0000 7fe12d397700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe11006e940 0x7fe110070e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe120000c00 tx=0x7fe120005960 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:16.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.436+0000 7fe1267fc700 1 -- 172.21.15.96:0/1142723435 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fe11c05b720 con 0x7fe128100d10 2024-04-22T15:21:16.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.661+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe12802cf00 con 0x7fe11006e940 2024-04-22T15:21:16.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.663+0000 7fe1267fc700 1 -- 172.21.15.96:0/1142723435 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fe12802cf00 con 0x7fe11006e940 2024-04-22T15:21:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe11006e940 msgr2=0x7fe110070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe11006e940 0x7fe110070e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe120000c00 tx=0x7fe120005960 comp rx=0 tx=0).stop 2024-04-22T15:21:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe128100d10 msgr2=0x7fe12819b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe128100d10 0x7fe12819b330 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00c930 tx=0x7fe11c00cc40 comp rx=0 tx=0).stop 2024-04-22T15:21:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 shutdown_connections 2024-04-22T15:21:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe11006e940 0x7fe110070e00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1280ffb10 0x7fe1280699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 --2- 172.21.15.96:0/1142723435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe128100d10 0x7fe12819b330 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 >> 172.21.15.96:0/1142723435 conn(0x7fe1280fb0b0 msgr2=0x7fe1281042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:16.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 shutdown_connections 2024-04-22T15:21:16.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.665+0000 7fe12f5fb700 1 -- 172.21.15.96:0/1142723435 wait complete. 2024-04-22T15:21:16.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.772+0000 7fba16265700 1 -- 172.21.15.96:0/3263775080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10100c40 msgr2=0x7fba10101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.772+0000 7fba16265700 1 --2- 172.21.15.96:0/3263775080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10100c40 0x7fba10101060 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fba04009910 tx=0x7fba04009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:16.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.772+0000 7fba16265700 1 -- 172.21.15.96:0/3263775080 shutdown_connections 2024-04-22T15:21:16.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.772+0000 7fba16265700 1 --2- 172.21.15.96:0/3263775080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba10101e40 0x7fba101022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.772+0000 7fba16265700 1 --2- 172.21.15.96:0/3263775080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10100c40 0x7fba10101060 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.772+0000 7fba16265700 1 -- 172.21.15.96:0/3263775080 >> 172.21.15.96:0/3263775080 conn(0x7fba100fc200 msgr2=0x7fba100fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:16.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.772+0000 7fba16265700 1 -- 172.21.15.96:0/3263775080 shutdown_connections 2024-04-22T15:21:16.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.772+0000 7fba16265700 1 -- 172.21.15.96:0/3263775080 wait complete. 2024-04-22T15:21:16.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.773+0000 7fba16265700 1 Processor -- start 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.773+0000 7fba16265700 1 -- start start 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.773+0000 7fba16265700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba10100c40 0x7fba10074be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.773+0000 7fba16265700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10101e40 0x7fba100732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.773+0000 7fba16265700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10073810 con 0x7fba10101e40 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.773+0000 7fba16265700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10073950 con 0x7fba10100c40 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.773+0000 7fba0f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10101e40 0x7fba100732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.773+0000 7fba0f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10101e40 0x7fba100732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50360/0 (socket says 172.21.15.96:50360) 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.773+0000 7fba0f7fe700 1 -- 172.21.15.96:0/2828334405 learned_addr learned my addr 172.21.15.96:0/2828334405 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba0ffff700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba10100c40 0x7fba10074be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba0f7fe700 1 -- 172.21.15.96:0/2828334405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba10100c40 msgr2=0x7fba10074be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba0f7fe700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba10100c40 0x7fba10074be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:16.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba0f7fe700 1 -- 172.21.15.96:0/2828334405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba040095f0 con 0x7fba10101e40 2024-04-22T15:21:16.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba0ffff700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba10100c40 0x7fba10074be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:21:16.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba0f7fe700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10101e40 0x7fba100732d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fba0000b4b0 tx=0x7fba0000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:16.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba0d7fa700 1 -- 172.21.15.96:0/2828334405 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba00011800 con 0x7fba10101e40 2024-04-22T15:21:16.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba10073c30 con 0x7fba10101e40 2024-04-22T15:21:16.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba0d7fa700 1 -- 172.21.15.96:0/2828334405 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba00011e20 con 0x7fba10101e40 2024-04-22T15:21:16.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba0d7fa700 1 -- 172.21.15.96:0/2828334405 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba0000f2e0 con 0x7fba10101e40 2024-04-22T15:21:16.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.774+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1019e7d0 con 0x7fba10101e40 2024-04-22T15:21:16.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.775+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba10190840 con 0x7fba10101e40 2024-04-22T15:21:16.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.776+0000 7fba0d7fa700 1 -- 172.21.15.96:0/2828334405 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7fba00010380 con 0x7fba10101e40 2024-04-22T15:21:16.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.776+0000 7fba0d7fa700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb9f806e940 0x7fb9f8070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:16.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.776+0000 7fba0d7fa700 1 -- 172.21.15.96:0/2828334405 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fba0008e8e0 con 0x7fba10101e40 2024-04-22T15:21:16.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.776+0000 7fba0ffff700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb9f806e940 0x7fb9f8070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:16.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.777+0000 7fba0ffff700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb9f806e940 0x7fb9f8070e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fba0400b5c0 tx=0x7fba04005180 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:16.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:16.781+0000 7fba0d7fa700 1 -- 172.21.15.96:0/2828334405 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fba0005b720 con 0x7fba10101e40 2024-04-22T15:21:16.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:16 smithi096 ceph-mon[26688]: pgmap v60: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:16 smithi196 ceph-mon[34717]: pgmap v60: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:17.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.007+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fba1002cdb0 con 0x7fb9f806e940 2024-04-22T15:21:17.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.018+0000 7fba0d7fa700 1 -- 172.21.15.96:0/2828334405 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fba1002cdb0 con 0x7fb9f806e940 2024-04-22T15:21:17.021 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:21:17.021 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (9m) 11s ago 11m 25.0M - 0.25.0 c8568f914cd2 1aa0b7928e82 2024-04-22T15:21:17.021 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (11m) 11s ago 11m 8132k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (10m) 45s ago 10m 7864k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (11m) 11s ago 11m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (10m) 45s ago 10m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (9m) 11s ago 10m 86.2M - 9.4.7 954c08fa6188 1f1eb6417649 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (5m) 11s ago 5m 157M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (5m) 11s ago 5m 89.7M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (5m) 45s ago 5m 26.4M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (5m) 45s ago 5m 25.8M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (2m) 11s ago 12m 502M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (70s) 45s ago 10m 449M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (12m) 11s ago 12m 56.5M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (10m) 45s ago 10m 44.9M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:21:17.022 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (53s) 11s ago 11m 7826k - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:21:17.023 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (49s) 45s ago 10m 3602k - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:21:17.023 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (9m) 11s ago 9m 209M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:21:17.023 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (8m) 11s ago 8m 216M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:21:17.023 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (8m) 11s ago 8m 181M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:21:17.023 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (8m) 45s ago 8m 231M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:21:17.023 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (7m) 45s ago 7m 208M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:21:17.023 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (7m) 45s ago 7m 185M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:21:17.023 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (17s) 11s ago 10m 47.5M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.020+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb9f806e940 msgr2=0x7fb9f8070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.020+0000 7fba16265700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb9f806e940 0x7fb9f8070e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fba0400b5c0 tx=0x7fba04005180 comp rx=0 tx=0).stop 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.020+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10101e40 msgr2=0x7fba100732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.020+0000 7fba16265700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10101e40 0x7fba100732d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fba0000b4b0 tx=0x7fba0000b870 comp rx=0 tx=0).stop 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.020+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 shutdown_connections 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.020+0000 7fba16265700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb9f806e940 0x7fb9f8070e00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.020+0000 7fba16265700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba10100c40 0x7fba10074be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.020+0000 7fba16265700 1 --2- 172.21.15.96:0/2828334405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba10101e40 0x7fba100732d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.020+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 >> 172.21.15.96:0/2828334405 conn(0x7fba100fc200 msgr2=0x7fba10105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:17.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.021+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 shutdown_connections 2024-04-22T15:21:17.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.021+0000 7fba16265700 1 -- 172.21.15.96:0/2828334405 wait complete. 2024-04-22T15:21:17.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.131+0000 7f9b860b4700 1 -- 172.21.15.96:0/278410120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 msgr2=0x7f9b80103aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.131+0000 7f9b860b4700 1 --2- 172.21.15.96:0/278410120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 0x7f9b80103aa0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009910 tx=0x7f9b68009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:17.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.131+0000 7f9b860b4700 1 -- 172.21.15.96:0/278410120 shutdown_connections 2024-04-22T15:21:17.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.131+0000 7f9b860b4700 1 --2- 172.21.15.96:0/278410120 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b80103fe0 0x7f9b801063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.131+0000 7f9b860b4700 1 --2- 172.21.15.96:0/278410120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 0x7f9b80103aa0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.132+0000 7f9b860b4700 1 -- 172.21.15.96:0/278410120 >> 172.21.15.96:0/278410120 conn(0x7f9b800fb0e0 msgr2=0x7f9b800fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:17.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.132+0000 7f9b860b4700 1 -- 172.21.15.96:0/278410120 shutdown_connections 2024-04-22T15:21:17.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.132+0000 7f9b860b4700 1 -- 172.21.15.96:0/278410120 wait complete. 2024-04-22T15:21:17.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.132+0000 7f9b860b4700 1 Processor -- start 2024-04-22T15:21:17.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.132+0000 7f9b860b4700 1 -- start start 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.132+0000 7f9b860b4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 0x7f9b80196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.132+0000 7f9b860b4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b80103fe0 0x7f9b80196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.132+0000 7f9b860b4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b801972c0 con 0x7f9b801016b0 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.132+0000 7f9b860b4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b80197400 con 0x7f9b80103fe0 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.133+0000 7f9b7f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 0x7f9b80196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.133+0000 7f9b7f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 0x7f9b80196760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50386/0 (socket says 172.21.15.96:50386) 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.133+0000 7f9b7f7fe700 1 -- 172.21.15.96:0/1914107876 learned_addr learned my addr 172.21.15.96:0/1914107876 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.133+0000 7f9b7effd700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b80103fe0 0x7f9b80196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.133+0000 7f9b7f7fe700 1 -- 172.21.15.96:0/1914107876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b80103fe0 msgr2=0x7f9b80196ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.133+0000 7f9b7f7fe700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b80103fe0 0x7f9b80196ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.133+0000 7f9b7f7fe700 1 -- 172.21.15.96:0/1914107876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b680095f0 con 0x7f9b801016b0 2024-04-22T15:21:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.133+0000 7f9b7f7fe700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 0x7f9b80196760 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9b68006010 tx=0x7f9b6800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.133+0000 7f9b7cff9700 1 -- 172.21.15.96:0/1914107876 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b6801d070 con 0x7f9b801016b0 2024-04-22T15:21:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.134+0000 7f9b7cff9700 1 -- 172.21.15.96:0/1914107876 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b6800f460 con 0x7f9b801016b0 2024-04-22T15:21:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.134+0000 7f9b7cff9700 1 -- 172.21.15.96:0/1914107876 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b68021620 con 0x7f9b801016b0 2024-04-22T15:21:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.134+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8019bdb0 con 0x7f9b801016b0 2024-04-22T15:21:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.134+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8019c2c0 con 0x7f9b801016b0 2024-04-22T15:21:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.135+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8004ec20 con 0x7f9b801016b0 2024-04-22T15:21:17.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.135+0000 7f9b7cff9700 1 -- 172.21.15.96:0/1914107876 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f9b6800f5d0 con 0x7f9b801016b0 2024-04-22T15:21:17.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.135+0000 7f9b7cff9700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b6c06e940 0x7f9b6c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:17.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.136+0000 7f9b7cff9700 1 -- 172.21.15.96:0/1914107876 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9b6808fba0 con 0x7f9b801016b0 2024-04-22T15:21:17.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.136+0000 7f9b7effd700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b6c06e940 0x7f9b6c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:17.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.136+0000 7f9b7effd700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b6c06e940 0x7f9b6c070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009a10 tx=0x7f9b70009190 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:17.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.140+0000 7f9b7cff9700 1 -- 172.21.15.96:0/1914107876 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f9b8004ec20 con 0x7f9b801016b0 2024-04-22T15:21:17.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.472+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9b80109800 con 0x7f9b801016b0 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.472+0000 7f9b7cff9700 1 -- 172.21.15.96:0/1914107876 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7f9b6805c9e0 con 0x7f9b801016b0 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:21:17.476 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:21:17.477 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:21:17.477 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:21:17.477 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:21:17.477 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:21:17.477 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:21:17.477 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 12, 2024-04-22T15:21:17.477 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:21:17.477 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:21:17.477 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:21:17.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.474+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b6c06e940 msgr2=0x7f9b6c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b6c06e940 0x7f9b6c070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009a10 tx=0x7f9b70009190 comp rx=0 tx=0).stop 2024-04-22T15:21:17.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 msgr2=0x7f9b80196760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 0x7f9b80196760 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9b68006010 tx=0x7f9b6800bd70 comp rx=0 tx=0).stop 2024-04-22T15:21:17.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 shutdown_connections 2024-04-22T15:21:17.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b6c06e940 0x7f9b6c070e00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b801016b0 0x7f9b80196760 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 --2- 172.21.15.96:0/1914107876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b80103fe0 0x7f9b80196ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 >> 172.21.15.96:0/1914107876 conn(0x7f9b800fb0e0 msgr2=0x7f9b800fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:17.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 shutdown_connections 2024-04-22T15:21:17.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.475+0000 7f9b860b4700 1 -- 172.21.15.96:0/1914107876 wait complete. 2024-04-22T15:21:17.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.582+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2678738448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc8100d20 msgr2=0x7f9cc8101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.582+0000 7f9cceb9a700 1 --2- 172.21.15.96:0/2678738448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc8100d20 0x7f9cc8101180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009910 tx=0x7f9cb4009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:17.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.583+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2678738448 shutdown_connections 2024-04-22T15:21:17.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.583+0000 7f9cceb9a700 1 --2- 172.21.15.96:0/2678738448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc8100d20 0x7f9cc8101180 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.583+0000 7f9cceb9a700 1 --2- 172.21.15.96:0/2678738448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cc80ffb20 0x7f9cc80fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.583+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2678738448 >> 172.21.15.96:0/2678738448 conn(0x7f9cc80fb0c0 msgr2=0x7f9cc80fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:17.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.583+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2678738448 shutdown_connections 2024-04-22T15:21:17.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.583+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2678738448 wait complete. 2024-04-22T15:21:17.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.583+0000 7f9cceb9a700 1 Processor -- start 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.583+0000 7f9cceb9a700 1 -- start start 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9cceb9a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc80ffb20 0x7f9cc8069c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9cceb9a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cc8100d20 0x7f9cc819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9cceb9a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc819b8a0 con 0x7f9cc80ffb20 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9cceb9a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc819b9e0 con 0x7f9cc8100d20 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9ccc936700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc80ffb20 0x7f9cc8069c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9ccc936700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc80ffb20 0x7f9cc8069c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50418/0 (socket says 172.21.15.96:50418) 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9ccc936700 1 -- 172.21.15.96:0/2064485667 learned_addr learned my addr 172.21.15.96:0/2064485667 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9cc7fff700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cc8100d20 0x7f9cc819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9ccc936700 1 -- 172.21.15.96:0/2064485667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cc8100d20 msgr2=0x7f9cc819b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9ccc936700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cc8100d20 0x7f9cc819b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9ccc936700 1 -- 172.21.15.96:0/2064485667 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb40095f0 con 0x7f9cc80ffb20 2024-04-22T15:21:17.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9ccc936700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc80ffb20 0x7f9cc8069c10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00c8f0 tx=0x7f9cbc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:17.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9cc5ffb700 1 -- 172.21.15.96:0/2064485667 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cbc007a70 con 0x7f9cc80ffb20 2024-04-22T15:21:17.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.584+0000 7f9cc5ffb700 1 -- 172.21.15.96:0/2064485667 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cbc007bd0 con 0x7f9cc80ffb20 2024-04-22T15:21:17.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.585+0000 7f9cc5ffb700 1 -- 172.21.15.96:0/2064485667 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cbc018630 con 0x7f9cc80ffb20 2024-04-22T15:21:17.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.585+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc819bc90 con 0x7f9cc80ffb20 2024-04-22T15:21:17.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.585+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc819c140 con 0x7f9cc80ffb20 2024-04-22T15:21:17.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.586+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc818f6a0 con 0x7f9cc80ffb20 2024-04-22T15:21:17.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.586+0000 7f9cc5ffb700 1 -- 172.21.15.96:0/2064485667 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f9cbc00f450 con 0x7f9cc80ffb20 2024-04-22T15:21:17.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.586+0000 7f9cc5ffb700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9cb806e940 0x7f9cb8070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:17.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.587+0000 7f9cc5ffb700 1 -- 172.21.15.96:0/2064485667 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9cbc08e8f0 con 0x7f9cc80ffb20 2024-04-22T15:21:17.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.587+0000 7f9cc7fff700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9cb806e940 0x7f9cb8070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:17.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.587+0000 7f9cc7fff700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9cb806e940 0x7f9cb8070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400b5c0 tx=0x7f9cb4005160 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:17.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.591+0000 7f9cc5ffb700 1 -- 172.21.15.96:0/2064485667 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f9cbc05b6b0 con 0x7f9cc80ffb20 2024-04-22T15:21:17.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.829+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cc802ceb0 con 0x7f9cb806e940 2024-04-22T15:21:17.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.830+0000 7f9cc5ffb700 1 -- 172.21.15.96:0/2064485667 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f9cc802ceb0 con 0x7f9cb806e940 2024-04-22T15:21:17.833 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:21:17.833 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:21:17.833 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:21:17.833 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:21:17.833 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:21:17.833 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:21:17.834 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:21:17.834 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:21:17.834 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-22T15:21:17.834 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:21:17.834 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:21:17.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9cb806e940 msgr2=0x7f9cb8070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9cb806e940 0x7f9cb8070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400b5c0 tx=0x7f9cb4005160 comp rx=0 tx=0).stop 2024-04-22T15:21:17.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc80ffb20 msgr2=0x7f9cc8069c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:17.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc80ffb20 0x7f9cc8069c10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00c8f0 tx=0x7f9cbc00ccb0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 shutdown_connections 2024-04-22T15:21:17.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9cb806e940 0x7f9cb8070e00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc80ffb20 0x7f9cc8069c10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 --2- 172.21.15.96:0/2064485667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cc8100d20 0x7f9cc819b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 >> 172.21.15.96:0/2064485667 conn(0x7f9cc80fb0c0 msgr2=0x7f9cc81042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 shutdown_connections 2024-04-22T15:21:17.837 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:17 smithi096 ceph-mon[26688]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:17.837 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:17 smithi096 ceph-mon[26688]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:17.837 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:17 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/1914107876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:17.833+0000 7f9cceb9a700 1 -- 172.21.15.96:0/2064485667 wait complete. 2024-04-22T15:21:18.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:17 smithi196 ceph-mon[34717]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:18.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:17 smithi196 ceph-mon[34717]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:18.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:17 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/1914107876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:18 smithi096 ceph-mon[26688]: from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:18 smithi096 ceph-mon[26688]: pgmap v61: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:18 smithi096 ceph-mon[26688]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:18 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:18 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:18 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:19.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:18 smithi196 ceph-mon[34717]: from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:19.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:18 smithi196 ceph-mon[34717]: pgmap v61: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:19.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:18 smithi196 ceph-mon[34717]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:19.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:18 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:19.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:18 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:19.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:18 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:21.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:20 smithi096 ceph-mon[26688]: pgmap v62: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:21.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:20 smithi196 ceph-mon[34717]: pgmap v62: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:21:22.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:21:22.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:22 smithi096 ceph-mon[26688]: pgmap v63: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:23.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:22 smithi196 ceph-mon[34717]: pgmap v63: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:24 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:24 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:24 smithi096 ceph-mon[26688]: pgmap v64: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:24 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:24 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:24.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:24 smithi196 ceph-mon[34717]: pgmap v64: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:25 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:25 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:25 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:25 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:26.694 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:26 smithi196 ceph-mon[34717]: pgmap v65: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:26.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:26 smithi096 ceph-mon[26688]: pgmap v65: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:29.122 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: pgmap v66: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.123 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.124 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.124 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.124 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.124 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.124 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.124 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:28 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: pgmap v66: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:29.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:28 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:30.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:29 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:21:30.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:29 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:21:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:30 smithi096 ceph-mon[26688]: pgmap v67: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:30 smithi096 ceph-mon[26688]: Upgrade: Updating grafana.smithi096 2024-04-22T15:21:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:30 smithi096 ceph-mon[26688]: Deploying daemon grafana.smithi096 on smithi096 2024-04-22T15:21:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:30 smithi196 ceph-mon[34717]: pgmap v67: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:30 smithi196 ceph-mon[34717]: Upgrade: Updating grafana.smithi096 2024-04-22T15:21:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:30 smithi196 ceph-mon[34717]: Deploying daemon grafana.smithi096 on smithi096 2024-04-22T15:21:33.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:32 smithi096 ceph-mon[26688]: pgmap v68: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:33.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:32 smithi196 ceph-mon[34717]: pgmap v68: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:35.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:34 smithi096 ceph-mon[26688]: pgmap v69: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:35.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:34 smithi196 ceph-mon[34717]: pgmap v69: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:37.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:36 smithi196 ceph-mon[34717]: pgmap v70: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:37.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:36 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:21:37.201 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:36 smithi096 ceph-mon[26688]: pgmap v70: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:37.202 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:36 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:21:39.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:38 smithi096 ceph-mon[26688]: pgmap v71: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:39.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:38 smithi196 ceph-mon[34717]: pgmap v71: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:41.174 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:40 smithi096 ceph-mon[26688]: pgmap v72: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:41.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:40 smithi196 ceph-mon[34717]: pgmap v72: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:42 smithi096 ceph-mon[26688]: pgmap v73: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:42 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:42 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:42 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:43.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:42 smithi196 ceph-mon[34717]: pgmap v73: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:43.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:42 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:43.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:42 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:43.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:42 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:45.148 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:44 smithi096 ceph-mon[26688]: pgmap v74: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:45.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:44 smithi196 ceph-mon[34717]: pgmap v74: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:46.866 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:46 smithi096 ceph-mon[26688]: pgmap v75: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:46 smithi196 ceph-mon[34717]: pgmap v75: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.944+0000 7f7a61222700 1 -- 172.21.15.96:0/2868028054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c0ffb90 msgr2=0x7f7a5c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.944+0000 7f7a61222700 1 --2- 172.21.15.96:0/2868028054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c0ffb90 0x7f7a5c0fffb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009910 tx=0x7f7a4c009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.944+0000 7f7a61222700 1 -- 172.21.15.96:0/2868028054 shutdown_connections 2024-04-22T15:21:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.944+0000 7f7a61222700 1 --2- 172.21.15.96:0/2868028054 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a5c100d90 0x7f7a5c1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.944+0000 7f7a61222700 1 --2- 172.21.15.96:0/2868028054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c0ffb90 0x7f7a5c0fffb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.944+0000 7f7a61222700 1 -- 172.21.15.96:0/2868028054 >> 172.21.15.96:0/2868028054 conn(0x7f7a5c0fb110 msgr2=0x7f7a5c0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.944+0000 7f7a61222700 1 -- 172.21.15.96:0/2868028054 shutdown_connections 2024-04-22T15:21:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.944+0000 7f7a61222700 1 -- 172.21.15.96:0/2868028054 wait complete. 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a61222700 1 Processor -- start 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a61222700 1 -- start start 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a61222700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a5c0ffb90 0x7f7a5c0690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a61222700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c100d90 0x7f7a5c19b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a61222700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c19b980 con 0x7f7a5c100d90 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a61222700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c19bac0 con 0x7f7a5c0ffb90 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a5259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c100d90 0x7f7a5c19b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a5259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c100d90 0x7f7a5c19b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42936/0 (socket says 172.21.15.96:42936) 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a5259c700 1 -- 172.21.15.96:0/3688043329 learned_addr learned my addr 172.21.15.96:0/3688043329 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.945+0000 7f7a5ad9d700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a5c0ffb90 0x7f7a5c0690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:47.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.946+0000 7f7a5259c700 1 -- 172.21.15.96:0/3688043329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a5c0ffb90 msgr2=0x7f7a5c0690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:47.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.946+0000 7f7a5259c700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a5c0ffb90 0x7f7a5c0690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:47.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.946+0000 7f7a5259c700 1 -- 172.21.15.96:0/3688043329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c0095f0 con 0x7f7a5c100d90 2024-04-22T15:21:47.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.946+0000 7f7a5259c700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c100d90 0x7f7a5c19b440 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400c930 tx=0x7f7a4400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:47.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.946+0000 7f7a58d99700 1 -- 172.21.15.96:0/3688043329 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a44007ad0 con 0x7f7a5c100d90 2024-04-22T15:21:47.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.946+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5c19bd20 con 0x7f7a5c100d90 2024-04-22T15:21:47.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.946+0000 7f7a58d99700 1 -- 172.21.15.96:0/3688043329 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a4400ce80 con 0x7f7a5c100d90 2024-04-22T15:21:47.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.946+0000 7f7a58d99700 1 -- 172.21.15.96:0/3688043329 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a44018630 con 0x7f7a5c100d90 2024-04-22T15:21:47.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.946+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5c19c270 con 0x7f7a5c100d90 2024-04-22T15:21:47.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.947+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5c18f710 con 0x7f7a5c100d90 2024-04-22T15:21:47.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.948+0000 7f7a58d99700 1 -- 172.21.15.96:0/3688043329 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f7a44007c30 con 0x7f7a5c100d90 2024-04-22T15:21:47.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.948+0000 7f7a58d99700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7a4806e940 0x7f7a48070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:47.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.948+0000 7f7a58d99700 1 -- 172.21.15.96:0/3688043329 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7a4408e950 con 0x7f7a5c100d90 2024-04-22T15:21:47.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.948+0000 7f7a5ad9d700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7a4806e940 0x7f7a48070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:47.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.948+0000 7f7a5ad9d700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7a4806e940 0x7f7a48070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00b5c0 tx=0x7f7a4c005960 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:47.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:47.953+0000 7f7a58d99700 1 -- 172.21.15.96:0/3688043329 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f7a4405b710 con 0x7f7a5c100d90 2024-04-22T15:21:48.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.182+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a5c02cea0 con 0x7f7a4806e940 2024-04-22T15:21:48.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.183+0000 7f7a58d99700 1 -- 172.21.15.96:0/3688043329 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f7a5c02cea0 con 0x7f7a4806e940 2024-04-22T15:21:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7a4806e940 msgr2=0x7f7a48070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7a4806e940 0x7f7a48070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00b5c0 tx=0x7f7a4c005960 comp rx=0 tx=0).stop 2024-04-22T15:21:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c100d90 msgr2=0x7f7a5c19b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c100d90 0x7f7a5c19b440 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400c930 tx=0x7f7a4400cc40 comp rx=0 tx=0).stop 2024-04-22T15:21:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 shutdown_connections 2024-04-22T15:21:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f7a4806e940 0x7f7a48070e00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a5c0ffb90 0x7f7a5c0690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 --2- 172.21.15.96:0/3688043329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a5c100d90 0x7f7a5c19b440 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 >> 172.21.15.96:0/3688043329 conn(0x7f7a5c0fb110 msgr2=0x7f7a5c1043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:48.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 shutdown_connections 2024-04-22T15:21:48.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.186+0000 7f7a61222700 1 -- 172.21.15.96:0/3688043329 wait complete. 2024-04-22T15:21:48.200 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:21:48.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.294+0000 7fb7600d7700 1 -- 172.21.15.96:0/2779888430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb758102210 msgr2=0x7fb758102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.294+0000 7fb7600d7700 1 --2- 172.21.15.96:0/2779888430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb758102210 0x7fb758102630 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb754009910 tx=0x7fb754009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:48.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.294+0000 7fb7600d7700 1 -- 172.21.15.96:0/2779888430 shutdown_connections 2024-04-22T15:21:48.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.294+0000 7fb7600d7700 1 --2- 172.21.15.96:0/2779888430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7580fec90 0x7fb7580ff110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.294+0000 7fb7600d7700 1 --2- 172.21.15.96:0/2779888430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb758102210 0x7fb758102630 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.294+0000 7fb7600d7700 1 -- 172.21.15.96:0/2779888430 >> 172.21.15.96:0/2779888430 conn(0x7fb7580fa830 msgr2=0x7fb7580fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:48.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.294+0000 7fb7600d7700 1 -- 172.21.15.96:0/2779888430 shutdown_connections 2024-04-22T15:21:48.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.294+0000 7fb7600d7700 1 -- 172.21.15.96:0/2779888430 wait complete. 2024-04-22T15:21:48.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb7600d7700 1 Processor -- start 2024-04-22T15:21:48.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb7600d7700 1 -- start start 2024-04-22T15:21:48.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb7600d7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7580fec90 0x7fb758196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:48.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb7600d7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb758102210 0x7fb758196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:48.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb7600d7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb758197260 con 0x7fb7580fec90 2024-04-22T15:21:48.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb7600d7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7581973a0 con 0x7fb758102210 2024-04-22T15:21:48.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb75de73700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7580fec90 0x7fb758196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:48.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb75de73700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7580fec90 0x7fb758196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42958/0 (socket says 172.21.15.96:42958) 2024-04-22T15:21:48.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb75de73700 1 -- 172.21.15.96:0/2658414167 learned_addr learned my addr 172.21.15.96:0/2658414167 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.295+0000 7fb75d672700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb758102210 0x7fb758196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb75de73700 1 -- 172.21.15.96:0/2658414167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb758102210 msgr2=0x7fb758196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb75de73700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb758102210 0x7fb758196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb75de73700 1 -- 172.21.15.96:0/2658414167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7540095f0 con 0x7fb7580fec90 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb75d672700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb758102210 0x7fb758196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb75de73700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7580fec90 0x7fb758196700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb7540059d0 tx=0x7fb75400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb74effd700 1 -- 172.21.15.96:0/2658414167 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb75401d070 con 0x7fb7580fec90 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75819bda0 con 0x7fb7580fec90 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb74effd700 1 -- 172.21.15.96:0/2658414167 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb75400f460 con 0x7fb7580fec90 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb74effd700 1 -- 172.21.15.96:0/2658414167 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb754021620 con 0x7fb7580fec90 2024-04-22T15:21:48.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.296+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75819c2b0 con 0x7fb7580fec90 2024-04-22T15:21:48.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.297+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb758190850 con 0x7fb7580fec90 2024-04-22T15:21:48.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.298+0000 7fb74effd700 1 -- 172.21.15.96:0/2658414167 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7fb75400fa70 con 0x7fb7580fec90 2024-04-22T15:21:48.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.299+0000 7fb74effd700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb74406e940 0x7fb744070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:48.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.299+0000 7fb74effd700 1 -- 172.21.15.96:0/2658414167 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb75408fc30 con 0x7fb7580fec90 2024-04-22T15:21:48.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.299+0000 7fb75d672700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb74406e940 0x7fb744070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:48.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.299+0000 7fb75d672700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb74406e940 0x7fb744070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb758068530 tx=0x7fb748008040 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:48.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.303+0000 7fb74effd700 1 -- 172.21.15.96:0/2658414167 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fb75405c9f0 con 0x7fb7580fec90 2024-04-22T15:21:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:48 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:48 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:48 smithi096 ceph-mon[26688]: pgmap v76: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.534+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb75802ced0 con 0x7fb74406e940 2024-04-22T15:21:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.536+0000 7fb74effd700 1 -- 172.21.15.96:0/2658414167 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fb75802ced0 con 0x7fb74406e940 2024-04-22T15:21:48.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb74406e940 msgr2=0x7fb744070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb74406e940 0x7fb744070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb758068530 tx=0x7fb748008040 comp rx=0 tx=0).stop 2024-04-22T15:21:48.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7580fec90 msgr2=0x7fb758196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7580fec90 0x7fb758196700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb7540059d0 tx=0x7fb75400be20 comp rx=0 tx=0).stop 2024-04-22T15:21:48.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 shutdown_connections 2024-04-22T15:21:48.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb74406e940 0x7fb744070e00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7580fec90 0x7fb758196700 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 --2- 172.21.15.96:0/2658414167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb758102210 0x7fb758196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 >> 172.21.15.96:0/2658414167 conn(0x7fb7580fa830 msgr2=0x7fb7581055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:48.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 shutdown_connections 2024-04-22T15:21:48.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.538+0000 7fb7600d7700 1 -- 172.21.15.96:0/2658414167 wait complete. 2024-04-22T15:21:48.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.647+0000 7f5aacb12700 1 -- 172.21.15.96:0/2590282629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa8100d90 msgr2=0x7f5aa81011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.647+0000 7f5aacb12700 1 --2- 172.21.15.96:0/2590282629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa8100d90 0x7f5aa81011f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009910 tx=0x7f5a98009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:48.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.647+0000 7f5aacb12700 1 -- 172.21.15.96:0/2590282629 shutdown_connections 2024-04-22T15:21:48.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.647+0000 7f5aacb12700 1 --2- 172.21.15.96:0/2590282629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa8100d90 0x7f5aa81011f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.647+0000 7f5aacb12700 1 --2- 172.21.15.96:0/2590282629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa80ffb90 0x7f5aa80fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.647+0000 7f5aacb12700 1 -- 172.21.15.96:0/2590282629 >> 172.21.15.96:0/2590282629 conn(0x7f5aa80fb110 msgr2=0x7f5aa80fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:48.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.647+0000 7f5aacb12700 1 -- 172.21.15.96:0/2590282629 shutdown_connections 2024-04-22T15:21:48.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.647+0000 7f5aacb12700 1 -- 172.21.15.96:0/2590282629 wait complete. 2024-04-22T15:21:48.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.647+0000 7f5aacb12700 1 Processor -- start 2024-04-22T15:21:48.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aacb12700 1 -- start start 2024-04-22T15:21:48.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aacb12700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa80ffb90 0x7f5aa8103ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:48.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aacb12700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8100d90 0x7f5aa8102170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:48.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aacb12700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa8104060 con 0x7f5aa80ffb90 2024-04-22T15:21:48.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aacb12700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81026b0 con 0x7f5aa8100d90 2024-04-22T15:21:48.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aa659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa80ffb90 0x7f5aa8103ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:48.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aa659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa80ffb90 0x7f5aa8103ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42970/0 (socket says 172.21.15.96:42970) 2024-04-22T15:21:48.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aa659c700 1 -- 172.21.15.96:0/2236364719 learned_addr learned my addr 172.21.15.96:0/2236364719 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:48.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aa5d9b700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8100d90 0x7f5aa8102170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:48.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aa659c700 1 -- 172.21.15.96:0/2236364719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8100d90 msgr2=0x7f5aa8102170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aa659c700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8100d90 0x7f5aa8102170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aa659c700 1 -- 172.21.15.96:0/2236364719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a980095f0 con 0x7f5aa80ffb90 2024-04-22T15:21:48.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aa5d9b700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8100d90 0x7f5aa8102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:21:48.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5aa659c700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa80ffb90 0x7f5aa8103ad0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00c910 tx=0x7f5a9c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:48.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.648+0000 7f5a977fe700 1 -- 172.21.15.96:0/2236364719 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a9c0041f0 con 0x7f5aa80ffb90 2024-04-22T15:21:48.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.649+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa8102990 con 0x7f5aa80ffb90 2024-04-22T15:21:48.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.649+0000 7f5a977fe700 1 -- 172.21.15.96:0/2236364719 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a9c007dc0 con 0x7f5aa80ffb90 2024-04-22T15:21:48.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.649+0000 7f5a977fe700 1 -- 172.21.15.96:0/2236364719 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a9c003e70 con 0x7f5aa80ffb90 2024-04-22T15:21:48.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.649+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa8102ee0 con 0x7f5aa80ffb90 2024-04-22T15:21:48.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.650+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa804ebb0 con 0x7f5aa80ffb90 2024-04-22T15:21:48.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.651+0000 7f5a977fe700 1 -- 172.21.15.96:0/2236364719 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f5a9c004350 con 0x7f5aa80ffb90 2024-04-22T15:21:48.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.651+0000 7f5a977fe700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5a9006e940 0x7f5a90070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:48.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.651+0000 7f5a977fe700 1 -- 172.21.15.96:0/2236364719 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f5a9c08e530 con 0x7f5aa80ffb90 2024-04-22T15:21:48.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.651+0000 7f5aa5d9b700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5a9006e940 0x7f5a90070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:48.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.652+0000 7f5aa5d9b700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5a9006e940 0x7f5a90070e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800b5c0 tx=0x7f5a98005180 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:48.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.656+0000 7f5a977fe700 1 -- 172.21.15.96:0/2236364719 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f5a9c05b370 con 0x7f5aa80ffb90 2024-04-22T15:21:48.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:48 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:48.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:48 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:48.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:48 smithi196 ceph-mon[34717]: pgmap v76: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.879+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5aa8068d40 con 0x7f5a9006e940 2024-04-22T15:21:48.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.889+0000 7f5a977fe700 1 -- 172.21.15.96:0/2236364719 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f5aa8068d40 con 0x7f5a9006e940 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (31s) 1s ago 11m 17.0M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (12m) 1s ago 12m 8182k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (10m) 77s ago 10m 7864k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (12m) 1s ago 12m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (10m) 77s ago 10m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (7s) 1s ago 11m 55.1M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (6m) 1s ago 6m 156M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (6m) 1s ago 6m 89.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (6m) 77s ago 6m 26.4M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (6m) 77s ago 6m 25.8M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:21:48.893 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (2m) 1s ago 13m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (102s) 77s ago 10m 449M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (13m) 1s ago 13m 57.7M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (10m) 77s ago 10m 44.9M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (85s) 1s ago 11m 16.7M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (81s) 77s ago 10m 3602k - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (9m) 1s ago 9m 209M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (9m) 1s ago 9m 215M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (8m) 1s ago 8m 181M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (8m) 77s ago 8m 231M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (8m) 77s ago 8m 208M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (7m) 77s ago 7m 185M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:21:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (49s) 1s ago 11m 58.6M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:21:48.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.891+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5a9006e940 msgr2=0x7f5a90070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5a9006e940 0x7f5a90070e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800b5c0 tx=0x7f5a98005180 comp rx=0 tx=0).stop 2024-04-22T15:21:48.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa80ffb90 msgr2=0x7f5aa8103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:48.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa80ffb90 0x7f5aa8103ad0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00c910 tx=0x7f5a9c00ccd0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 shutdown_connections 2024-04-22T15:21:48.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f5a9006e940 0x7f5a90070e00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5aa80ffb90 0x7f5aa8103ad0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 --2- 172.21.15.96:0/2236364719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8100d90 0x7f5aa8102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:48.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 >> 172.21.15.96:0/2236364719 conn(0x7f5aa80fb110 msgr2=0x7f5aa8068630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:48.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 shutdown_connections 2024-04-22T15:21:48.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:48.892+0000 7f5aacb12700 1 -- 172.21.15.96:0/2236364719 wait complete. 2024-04-22T15:21:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.012+0000 7fa519c66700 1 -- 172.21.15.96:0/479529376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 msgr2=0x7fa5141063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.012+0000 7fa519c66700 1 --2- 172.21.15.96:0/479529376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 0x7fa5141063d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa508009910 tx=0x7fa508009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.013+0000 7fa519c66700 1 -- 172.21.15.96:0/479529376 shutdown_connections 2024-04-22T15:21:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.013+0000 7fa519c66700 1 --2- 172.21.15.96:0/479529376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 0x7fa5141063d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.013+0000 7fa519c66700 1 --2- 172.21.15.96:0/479529376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5141016b0 0x7fa514103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.013+0000 7fa519c66700 1 -- 172.21.15.96:0/479529376 >> 172.21.15.96:0/479529376 conn(0x7fa5140fb0c0 msgr2=0x7fa5140fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.013+0000 7fa519c66700 1 -- 172.21.15.96:0/479529376 shutdown_connections 2024-04-22T15:21:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.013+0000 7fa519c66700 1 -- 172.21.15.96:0/479529376 wait complete. 2024-04-22T15:21:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.014+0000 7fa519c66700 1 Processor -- start 2024-04-22T15:21:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.014+0000 7fa519c66700 1 -- start start 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.014+0000 7fa519c66700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5141016b0 0x7fa514196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.014+0000 7fa519c66700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 0x7fa514196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.014+0000 7fa519c66700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa514197180 con 0x7fa514103fe0 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.014+0000 7fa519c66700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5141972c0 con 0x7fa5141016b0 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.014+0000 7fa512ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 0x7fa514196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.014+0000 7fa512ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 0x7fa514196b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42986/0 (socket says 172.21.15.96:42986) 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.014+0000 7fa512ffd700 1 -- 172.21.15.96:0/211333897 learned_addr learned my addr 172.21.15.96:0/211333897 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa5137fe700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5141016b0 0x7fa514196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa512ffd700 1 -- 172.21.15.96:0/211333897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5141016b0 msgr2=0x7fa514196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa512ffd700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5141016b0 0x7fa514196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa512ffd700 1 -- 172.21.15.96:0/211333897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5080095f0 con 0x7fa514103fe0 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa5137fe700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5141016b0 0x7fa514196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa512ffd700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 0x7fa514196b60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa508006010 tx=0x7fa50800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa510ff9700 1 -- 172.21.15.96:0/211333897 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50801d070 con 0x7fa514103fe0 2024-04-22T15:21:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51419bcc0 con 0x7fa514103fe0 2024-04-22T15:21:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa510ff9700 1 -- 172.21.15.96:0/211333897 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa50800f460 con 0x7fa514103fe0 2024-04-22T15:21:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa510ff9700 1 -- 172.21.15.96:0/211333897 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa508021640 con 0x7fa514103fe0 2024-04-22T15:21:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.015+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51419c230 con 0x7fa514103fe0 2024-04-22T15:21:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.016+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51404ebb0 con 0x7fa514103fe0 2024-04-22T15:21:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.017+0000 7fa510ff9700 1 -- 172.21.15.96:0/211333897 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7fa50800fab0 con 0x7fa514103fe0 2024-04-22T15:21:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.017+0000 7fa510ff9700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa4fc06e940 0x7fa4fc070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.017+0000 7fa510ff9700 1 -- 172.21.15.96:0/211333897 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa50808fc20 con 0x7fa514103fe0 2024-04-22T15:21:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.018+0000 7fa5137fe700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa4fc06e940 0x7fa4fc070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.018+0000 7fa5137fe700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa4fc06e940 0x7fa4fc070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa504005950 tx=0x7fa5040058e0 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:49.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.022+0000 7fa510ff9700 1 -- 172.21.15.96:0/211333897 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fa50805c9e0 con 0x7fa514103fe0 2024-04-22T15:21:49.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.327+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa514109780 con 0x7fa514103fe0 2024-04-22T15:21:49.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.327+0000 7fa510ff9700 1 -- 172.21.15.96:0/211333897 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7fa50805c570 con 0x7fa514103fe0 2024-04-22T15:21:49.331 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:21:49.331 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:21:49.331 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:21:49.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:21:49.331 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 12, 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:21:49.332 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:21:49.333 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:21:49.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa4fc06e940 msgr2=0x7fa4fc070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa4fc06e940 0x7fa4fc070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa504005950 tx=0x7fa5040058e0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 msgr2=0x7fa514196b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 0x7fa514196b60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa508006010 tx=0x7fa50800be20 comp rx=0 tx=0).stop 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 shutdown_connections 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa4fc06e940 0x7fa4fc070e00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5141016b0 0x7fa514196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 --2- 172.21.15.96:0/211333897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa514103fe0 0x7fa514196b60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 >> 172.21.15.96:0/211333897 conn(0x7fa5140fb0c0 msgr2=0x7fa5140fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 shutdown_connections 2024-04-22T15:21:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.330+0000 7fa519c66700 1 -- 172.21.15.96:0/211333897 wait complete. 2024-04-22T15:21:49.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.434+0000 7f986a56e700 1 -- 172.21.15.96:0/2973257533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98641016b0 msgr2=0x7f9864103aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:49.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.434+0000 7f986a56e700 1 --2- 172.21.15.96:0/2973257533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98641016b0 0x7f9864103aa0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9858009910 tx=0x7f9858009c20 comp rx=0 tx=0).stop 2024-04-22T15:21:49.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.434+0000 7f986a56e700 1 -- 172.21.15.96:0/2973257533 shutdown_connections 2024-04-22T15:21:49.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.434+0000 7f986a56e700 1 --2- 172.21.15.96:0/2973257533 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9864103fe0 0x7f98641063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.434+0000 7f986a56e700 1 --2- 172.21.15.96:0/2973257533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98641016b0 0x7f9864103aa0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.434+0000 7f986a56e700 1 -- 172.21.15.96:0/2973257533 >> 172.21.15.96:0/2973257533 conn(0x7f98640fb0c0 msgr2=0x7f98640fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:49.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.434+0000 7f986a56e700 1 -- 172.21.15.96:0/2973257533 shutdown_connections 2024-04-22T15:21:49.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.435+0000 7f986a56e700 1 -- 172.21.15.96:0/2973257533 wait complete. 2024-04-22T15:21:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.435+0000 7f986a56e700 1 Processor -- start 2024-04-22T15:21:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.435+0000 7f986a56e700 1 -- start start 2024-04-22T15:21:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f986a56e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98641016b0 0x7f9864196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f986a56e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9864103fe0 0x7f9864196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f986a56e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9864197280 con 0x7f9864103fe0 2024-04-22T15:21:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f986a56e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98641973c0 con 0x7f98641016b0 2024-04-22T15:21:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f98637fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9864103fe0 0x7f9864196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f98637fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9864103fe0 0x7f9864196c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43008/0 (socket says 172.21.15.96:43008) 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f98637fe700 1 -- 172.21.15.96:0/1945184444 learned_addr learned my addr 172.21.15.96:0/1945184444 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f98637fe700 1 -- 172.21.15.96:0/1945184444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98641016b0 msgr2=0x7f9864196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f9863fff700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98641016b0 0x7f9864196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f98637fe700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98641016b0 0x7f9864196720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f98637fe700 1 -- 172.21.15.96:0/1945184444 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98580095f0 con 0x7f9864103fe0 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f9863fff700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98641016b0 0x7f9864196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.436+0000 7f98637fe700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9864103fe0 0x7f9864196c60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f985400b4b0 tx=0x7f985400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.437+0000 7f98617fa700 1 -- 172.21.15.96:0/1945184444 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9854011800 con 0x7f9864103fe0 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.437+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986419bdd0 con 0x7f9864103fe0 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.437+0000 7f98617fa700 1 -- 172.21.15.96:0/1945184444 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9854011e20 con 0x7f9864103fe0 2024-04-22T15:21:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.437+0000 7f98617fa700 1 -- 172.21.15.96:0/1945184444 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f985400f2e0 con 0x7f9864103fe0 2024-04-22T15:21:49.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.437+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986419c370 con 0x7f9864103fe0 2024-04-22T15:21:49.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.438+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986404ebb0 con 0x7f9864103fe0 2024-04-22T15:21:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.438+0000 7f98617fa700 1 -- 172.21.15.96:0/1945184444 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f9854010380 con 0x7f9864103fe0 2024-04-22T15:21:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.439+0000 7f98617fa700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f984c06e940 0x7f984c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:21:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.439+0000 7f98617fa700 1 -- 172.21.15.96:0/1945184444 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f985408e4e0 con 0x7f9864103fe0 2024-04-22T15:21:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.439+0000 7f9863fff700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f984c06e940 0x7f984c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:21:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.439+0000 7f9863fff700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f984c06e940 0x7f984c070e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f98580059d0 tx=0x7f9858005960 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:21:49.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.443+0000 7f98617fa700 1 -- 172.21.15.96:0/1945184444 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f985405b2a0 con 0x7f9864103fe0 2024-04-22T15:21:49.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.668+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9864104a00 con 0x7f984c06e940 2024-04-22T15:21:49.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.669+0000 7f98617fa700 1 -- 172.21.15.96:0/1945184444 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f9864104a00 con 0x7f984c06e940 2024-04-22T15:21:49.672 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:21:49.672 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:21:49.672 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:21:49.672 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:21:49.673 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:21:49.673 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:21:49.673 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:21:49.673 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:21:49.673 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading grafana daemons", 2024-04-22T15:21:49.673 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:21:49.673 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:21:49.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f984c06e940 msgr2=0x7f984c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:49.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f984c06e940 0x7f984c070e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f98580059d0 tx=0x7f9858005960 comp rx=0 tx=0).stop 2024-04-22T15:21:49.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9864103fe0 msgr2=0x7f9864196c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:21:49.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9864103fe0 0x7f9864196c60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f985400b4b0 tx=0x7f985400b870 comp rx=0 tx=0).stop 2024-04-22T15:21:49.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 shutdown_connections 2024-04-22T15:21:49.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f984c06e940 0x7f984c070e00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98641016b0 0x7f9864196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 --2- 172.21.15.96:0/1945184444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9864103fe0 0x7f9864196c60 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:21:49.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 >> 172.21.15.96:0/1945184444 conn(0x7f98640fb0c0 msgr2=0x7f98640fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:21:49.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 shutdown_connections 2024-04-22T15:21:49.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:21:49.671+0000 7f986a56e700 1 -- 172.21.15.96:0/1945184444 wait complete. 2024-04-22T15:21:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:49 smithi096 ceph-mon[26688]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:49 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:49 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:49 smithi096 ceph-mon[26688]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:49 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/211333897' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:49 smithi196 ceph-mon[34717]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:49 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:49 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:49 smithi196 ceph-mon[34717]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:49 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/211333897' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:50.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:50 smithi096 ceph-mon[26688]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:50.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:50 smithi096 ceph-mon[26688]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:50.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:50 smithi096 ceph-mon[26688]: pgmap v77: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:50.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:50 smithi196 ceph-mon[34717]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:50.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:50 smithi196 ceph-mon[34717]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:21:50.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:50 smithi196 ceph-mon[34717]: pgmap v77: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: pgmap v78: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:21:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:21:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T15:21:52.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:52.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:52.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:52 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: pgmap v78: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:21:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:53.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T15:21:53.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:53.263 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:21:53.264 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:52 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:53 smithi096 ceph-mon[26688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:21:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:53 smithi096 ceph-mon[26688]: Upgrade: Finalizing container_image settings 2024-04-22T15:21:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:53 smithi096 ceph-mon[26688]: Upgrade: Complete! 2024-04-22T15:21:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:53 smithi196 ceph-mon[34717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:21:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:53 smithi196 ceph-mon[34717]: Upgrade: Finalizing container_image settings 2024-04-22T15:21:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:53 smithi196 ceph-mon[34717]: Upgrade: Complete! 2024-04-22T15:21:54.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:54 smithi096 ceph-mon[26688]: pgmap v79: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:54 smithi196 ceph-mon[34717]: pgmap v79: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:57.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:56 smithi096 ceph-mon[26688]: pgmap v80: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:57.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:56 smithi196 ceph-mon[34717]: pgmap v80: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:21:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:57 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:58.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:57 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:21:59.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:21:58 smithi096 ceph-mon[26688]: pgmap v81: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:21:59.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:21:58 smithi196 ceph-mon[34717]: pgmap v81: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:01.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:00 smithi096 ceph-mon[26688]: pgmap v82: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:01.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:00 smithi196 ceph-mon[34717]: pgmap v82: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:03.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:02 smithi096 ceph-mon[26688]: pgmap v83: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:02 smithi196 ceph-mon[34717]: pgmap v83: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:05.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:04 smithi096 ceph-mon[26688]: pgmap v84: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:05.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:04 smithi196 ceph-mon[34717]: pgmap v84: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:07.124 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:06 smithi196 ceph-mon[34717]: pgmap v85: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:07.125 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:06 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:22:07.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:06 smithi096 ceph-mon[26688]: pgmap v85: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:07.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:06 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:22:09.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:08 smithi096 ceph-mon[26688]: pgmap v86: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:08 smithi196 ceph-mon[34717]: pgmap v86: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:10 smithi096 ceph-mon[26688]: pgmap v87: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:11.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:10 smithi196 ceph-mon[34717]: pgmap v87: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:13.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:12 smithi096 ceph-mon[26688]: pgmap v88: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:13.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:12 smithi196 ceph-mon[34717]: pgmap v88: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:15.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:14 smithi096 ceph-mon[26688]: pgmap v89: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:15.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:14 smithi196 ceph-mon[34717]: pgmap v89: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:17.231 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:16 smithi196 ceph-mon[34717]: pgmap v90: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:17.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:16 smithi096 ceph-mon[26688]: pgmap v90: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:19.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:18 smithi096 ceph-mon[26688]: pgmap v91: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:19.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:18 smithi196 ceph-mon[34717]: pgmap v91: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.789+0000 7fb04da53700 1 -- 172.21.15.96:0/2009428691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 msgr2=0x7fb048073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.789+0000 7fb04da53700 1 --2- 172.21.15.96:0/2009428691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 0x7fb048073d40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb030009910 tx=0x7fb030009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.789+0000 7fb04da53700 1 -- 172.21.15.96:0/2009428691 shutdown_connections 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.789+0000 7fb04da53700 1 --2- 172.21.15.96:0/2009428691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 0x7fb048073d40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.789+0000 7fb04da53700 1 --2- 172.21.15.96:0/2009428691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb048074e90 0x7fb0480732f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.789+0000 7fb04da53700 1 -- 172.21.15.96:0/2009428691 >> 172.21.15.96:0/2009428691 conn(0x7fb0480fc1d0 msgr2=0x7fb0480fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.789+0000 7fb04da53700 1 -- 172.21.15.96:0/2009428691 shutdown_connections 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.789+0000 7fb04da53700 1 -- 172.21.15.96:0/2009428691 wait complete. 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.790+0000 7fb04da53700 1 Processor -- start 2024-04-22T15:22:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.790+0000 7fb04da53700 1 -- start start 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.790+0000 7fb04da53700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 0x7fb0481988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.790+0000 7fb04da53700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb048074e90 0x7fb048198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.790+0000 7fb04da53700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb048199420 con 0x7fb0480738c0 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.790+0000 7fb04da53700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb048199560 con 0x7fb048074e90 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb046ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 0x7fb0481988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb046ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 0x7fb0481988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41948/0 (socket says 172.21.15.96:41948) 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb046ffd700 1 -- 172.21.15.96:0/2007780485 learned_addr learned my addr 172.21.15.96:0/2007780485 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb0467fc700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb048074e90 0x7fb048198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb046ffd700 1 -- 172.21.15.96:0/2007780485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb048074e90 msgr2=0x7fb048198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb046ffd700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb048074e90 0x7fb048198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb046ffd700 1 -- 172.21.15.96:0/2007780485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0300095f0 con 0x7fb0480738c0 2024-04-22T15:22:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb046ffd700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 0x7fb0481988c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb03800ba50 tx=0x7fb03800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb04ca51700 1 -- 172.21.15.96:0/2007780485 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb03800ca00 con 0x7fb0480738c0 2024-04-22T15:22:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb04ca51700 1 -- 172.21.15.96:0/2007780485 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb03800cb60 con 0x7fb0480738c0 2024-04-22T15:22:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04819dfc0 con 0x7fb0480738c0 2024-04-22T15:22:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb04ca51700 1 -- 172.21.15.96:0/2007780485 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0380075a0 con 0x7fb0480738c0 2024-04-22T15:22:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.791+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04819e500 con 0x7fb0480738c0 2024-04-22T15:22:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.792+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb048192a20 con 0x7fb0480738c0 2024-04-22T15:22:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.793+0000 7fb04ca51700 1 -- 172.21.15.96:0/2007780485 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7fb03800ccd0 con 0x7fb0480738c0 2024-04-22T15:22:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.793+0000 7fb04ca51700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb03406e950 0x7fb034070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.793+0000 7fb04ca51700 1 -- 172.21.15.96:0/2007780485 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb038006e40 con 0x7fb0480738c0 2024-04-22T15:22:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.793+0000 7fb0467fc700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb03406e950 0x7fb034070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.794+0000 7fb0467fc700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb03406e950 0x7fb034070e10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb0300051c0 tx=0x7fb03000b540 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:19.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:19.798+0000 7fb04ca51700 1 -- 172.21.15.96:0/2007780485 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fb03805aed0 con 0x7fb0480738c0 2024-04-22T15:22:20.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.016+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb04802ceb0 con 0x7fb03406e950 2024-04-22T15:22:20.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.017+0000 7fb04ca51700 1 -- 172.21.15.96:0/2007780485 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fb04802ceb0 con 0x7fb03406e950 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.019+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb03406e950 msgr2=0x7fb034070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.019+0000 7fb04da53700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb03406e950 0x7fb034070e10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb0300051c0 tx=0x7fb03000b540 comp rx=0 tx=0).stop 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.019+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 msgr2=0x7fb0481988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.019+0000 7fb04da53700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 0x7fb0481988c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb03800ba50 tx=0x7fb03800be10 comp rx=0 tx=0).stop 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.019+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 shutdown_connections 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.019+0000 7fb04da53700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fb03406e950 0x7fb034070e10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.019+0000 7fb04da53700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0480738c0 0x7fb0481988c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.019+0000 7fb04da53700 1 --2- 172.21.15.96:0/2007780485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb048074e90 0x7fb048198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.019+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 >> 172.21.15.96:0/2007780485 conn(0x7fb0480fc1d0 msgr2=0x7fb048106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.020+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 shutdown_connections 2024-04-22T15:22:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:20.020+0000 7fb04da53700 1 -- 172.21.15.96:0/2007780485 wait complete. 2024-04-22T15:22:20.780 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-22T15:22:21.029 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:21.151 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:20 smithi096 ceph-mon[26688]: pgmap v92: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:21.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:20 smithi196 ceph-mon[34717]: pgmap v92: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:22.182 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:21 smithi096 ceph-mon[26688]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:22.182 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:21 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:22:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:21 smithi196 ceph-mon[34717]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:21 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:22:22.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.699+0000 7f1579787700 1 -- 172.21.15.96:0/731133394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574102680 msgr2=0x7f157410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:22.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.699+0000 7f1579787700 1 --2- 172.21.15.96:0/731133394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574102680 0x7f157410ab70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1568009910 tx=0x7f1568009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:22.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.699+0000 7f1579787700 1 -- 172.21.15.96:0/731133394 shutdown_connections 2024-04-22T15:22:22.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.699+0000 7f1579787700 1 --2- 172.21.15.96:0/731133394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574102680 0x7f157410ab70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:22.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.699+0000 7f1579787700 1 --2- 172.21.15.96:0/731133394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1574101d60 0x7f1574102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:22.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.699+0000 7f1579787700 1 -- 172.21.15.96:0/731133394 >> 172.21.15.96:0/731133394 conn(0x7f1574076740 msgr2=0x7f15740fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:22.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.699+0000 7f1579787700 1 -- 172.21.15.96:0/731133394 shutdown_connections 2024-04-22T15:22:22.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.699+0000 7f1579787700 1 -- 172.21.15.96:0/731133394 wait complete. 2024-04-22T15:22:22.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.699+0000 7f1579787700 1 Processor -- start 2024-04-22T15:22:22.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1579787700 1 -- start start 2024-04-22T15:22:22.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1579787700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574101d60 0x7f15740ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:22.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1579787700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1574102680 0x7f15740fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:22.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1579787700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1574100440 con 0x7f1574101d60 2024-04-22T15:22:22.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1579787700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15741005b0 con 0x7f1574102680 2024-04-22T15:22:22.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1572ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574101d60 0x7f15740ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:22.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1572ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574101d60 0x7f15740ff9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35008/0 (socket says 172.21.15.96:35008) 2024-04-22T15:22:22.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1572ffd700 1 -- 172.21.15.96:0/4167128183 learned_addr learned my addr 172.21.15.96:0/4167128183 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:22.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f15727fc700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1574102680 0x7f15740fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:22.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1572ffd700 1 -- 172.21.15.96:0/4167128183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1574102680 msgr2=0x7f15740fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:22.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1572ffd700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1574102680 0x7f15740fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:22.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1572ffd700 1 -- 172.21.15.96:0/4167128183 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15680095f0 con 0x7f1574101d60 2024-04-22T15:22:22.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.700+0000 7f1572ffd700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574101d60 0x7f15740ff9c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f156400b4b0 tx=0x7f156400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:22.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.701+0000 7f155bfff700 1 -- 172.21.15.96:0/4167128183 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1564011800 con 0x7f1574101d60 2024-04-22T15:22:22.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.701+0000 7f155bfff700 1 -- 172.21.15.96:0/4167128183 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1564011e20 con 0x7f1574101d60 2024-04-22T15:22:22.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.701+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15741a2e20 con 0x7f1574101d60 2024-04-22T15:22:22.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.701+0000 7f155bfff700 1 -- 172.21.15.96:0/4167128183 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f156400f2e0 con 0x7f1574101d60 2024-04-22T15:22:22.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.701+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15741a31d0 con 0x7f1574101d60 2024-04-22T15:22:22.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.702+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1574068ea0 con 0x7f1574101d60 2024-04-22T15:22:22.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.702+0000 7f155bfff700 1 -- 172.21.15.96:0/4167128183 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f1564010380 con 0x7f1574101d60 2024-04-22T15:22:22.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.703+0000 7f155bfff700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f155c06e940 0x7f155c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:22.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.703+0000 7f155bfff700 1 -- 172.21.15.96:0/4167128183 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f156408e490 con 0x7f1574101d60 2024-04-22T15:22:22.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.703+0000 7f15727fc700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f155c06e940 0x7f155c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:22.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.703+0000 7f15727fc700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f155c06e940 0x7f155c070e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1574101470 tx=0x7f156801a040 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:22.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.707+0000 7f155bfff700 1 -- 172.21.15.96:0/4167128183 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f1564057720 con 0x7f1574101d60 2024-04-22T15:22:23.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.999+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1574100770 con 0x7f1574101d60 2024-04-22T15:22:23.002 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:22 smithi096 ceph-mon[26688]: pgmap v93: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:23.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:22.999+0000 7f155bfff700 1 -- 172.21.15.96:0/4167128183 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7f156405ade0 con 0x7f1574101d60 2024-04-22T15:22:23.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f155c06e940 msgr2=0x7f155c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:23.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f155c06e940 0x7f155c070e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1574101470 tx=0x7f156801a040 comp rx=0 tx=0).stop 2024-04-22T15:22:23.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574101d60 msgr2=0x7f15740ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:23.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574101d60 0x7f15740ff9c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f156400b4b0 tx=0x7f156400b870 comp rx=0 tx=0).stop 2024-04-22T15:22:23.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 shutdown_connections 2024-04-22T15:22:23.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f155c06e940 0x7f155c070e00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:23.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1574101d60 0x7f15740ff9c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:23.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 --2- 172.21.15.96:0/4167128183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1574102680 0x7f15740fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:23.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 >> 172.21.15.96:0/4167128183 conn(0x7f1574076740 msgr2=0x7f15741030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:23.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 shutdown_connections 2024-04-22T15:22:23.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:23.002+0000 7f1579787700 1 -- 172.21.15.96:0/4167128183 wait complete. 2024-04-22T15:22:23.016 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:22:23.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:22 smithi196 ceph-mon[34717]: pgmap v93: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:23.655 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-22T15:22:23.873 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:23 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/4167128183' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:23 smithi096 ceph-mon[26688]: pgmap v94: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:23 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/4167128183' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:23 smithi196 ceph-mon[34717]: pgmap v94: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.601+0000 7f9b58114700 1 -- 172.21.15.96:0/3312264869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 msgr2=0x7f9b500fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.601+0000 7f9b58114700 1 --2- 172.21.15.96:0/3312264869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 0x7f9b500fffd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009910 tx=0x7f9b40009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.601+0000 7f9b58114700 1 -- 172.21.15.96:0/3312264869 shutdown_connections 2024-04-22T15:22:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.601+0000 7f9b58114700 1 --2- 172.21.15.96:0/3312264869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 0x7f9b500fffd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.601+0000 7f9b58114700 1 --2- 172.21.15.96:0/3312264869 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b50105c00 0x7f9b50105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.601+0000 7f9b58114700 1 -- 172.21.15.96:0/3312264869 >> 172.21.15.96:0/3312264869 conn(0x7f9b500fb680 msgr2=0x7f9b500fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.601+0000 7f9b58114700 1 -- 172.21.15.96:0/3312264869 shutdown_connections 2024-04-22T15:22:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.601+0000 7f9b58114700 1 -- 172.21.15.96:0/3312264869 wait complete. 2024-04-22T15:22:25.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b58114700 1 Processor -- start 2024-04-22T15:22:25.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b58114700 1 -- start start 2024-04-22T15:22:25.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b58114700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 0x7f9b50195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:25.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b58114700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b50105c00 0x7f9b50195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:25.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b58114700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5006a070 con 0x7f9b500ffb50 2024-04-22T15:22:25.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b58114700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b50196290 con 0x7f9b50105c00 2024-04-22T15:22:25.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b55eb0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 0x7f9b50195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:25.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b55eb0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 0x7f9b50195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35022/0 (socket says 172.21.15.96:35022) 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b55eb0700 1 -- 172.21.15.96:0/2954441999 learned_addr learned my addr 172.21.15.96:0/2954441999 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.602+0000 7f9b556af700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b50105c00 0x7f9b50195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.603+0000 7f9b55eb0700 1 -- 172.21.15.96:0/2954441999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b50105c00 msgr2=0x7f9b50195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.603+0000 7f9b55eb0700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b50105c00 0x7f9b50195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.603+0000 7f9b55eb0700 1 -- 172.21.15.96:0/2954441999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b400095f0 con 0x7f9b500ffb50 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.603+0000 7f9b55eb0700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 0x7f9b50195810 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00b4b0 tx=0x7f9b4c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.603+0000 7f9b46ffd700 1 -- 172.21.15.96:0/2954441999 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b4c011800 con 0x7f9b500ffb50 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.603+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b50196510 con 0x7f9b500ffb50 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.603+0000 7f9b46ffd700 1 -- 172.21.15.96:0/2954441999 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b4c011e20 con 0x7f9b500ffb50 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.603+0000 7f9b46ffd700 1 -- 172.21.15.96:0/2954441999 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b4c00f2e0 con 0x7f9b500ffb50 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.603+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b501a3120 con 0x7f9b500ffb50 2024-04-22T15:22:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.604+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b50108030 con 0x7f9b500ffb50 2024-04-22T15:22:25.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.604+0000 7f9b46ffd700 1 -- 172.21.15.96:0/2954441999 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f9b4c010380 con 0x7f9b500ffb50 2024-04-22T15:22:25.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.605+0000 7f9b46ffd700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b3c06e950 0x7f9b3c070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:25.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.605+0000 7f9b46ffd700 1 -- 172.21.15.96:0/2954441999 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9b4c08e4d0 con 0x7f9b500ffb50 2024-04-22T15:22:25.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.605+0000 7f9b556af700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b3c06e950 0x7f9b3c070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:25.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.605+0000 7f9b556af700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b3c06e950 0x7f9b3c070e10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9b50069980 tx=0x7f9b4001a040 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:25.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.610+0000 7f9b46ffd700 1 -- 172.21.15.96:0/2954441999 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f9b4c05b290 con 0x7f9b500ffb50 2024-04-22T15:22:25.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.896+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9b5010cdc0 con 0x7f9b500ffb50 2024-04-22T15:22:25.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.896+0000 7f9b46ffd700 1 -- 172.21.15.96:0/2954441999 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7f9b4c05ae20 con 0x7f9b500ffb50 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.898+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b3c06e950 msgr2=0x7f9b3c070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.898+0000 7f9b58114700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b3c06e950 0x7f9b3c070e10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9b50069980 tx=0x7f9b4001a040 comp rx=0 tx=0).stop 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.898+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 msgr2=0x7f9b50195810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.898+0000 7f9b58114700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 0x7f9b50195810 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00b4b0 tx=0x7f9b4c00b870 comp rx=0 tx=0).stop 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.899+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 shutdown_connections 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.899+0000 7f9b58114700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f9b3c06e950 0x7f9b3c070e10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.899+0000 7f9b58114700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b500ffb50 0x7f9b50195810 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.899+0000 7f9b58114700 1 --2- 172.21.15.96:0/2954441999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b50105c00 0x7f9b50195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.899+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 >> 172.21.15.96:0/2954441999 conn(0x7f9b500fb680 msgr2=0x7f9b50104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:25.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.899+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 shutdown_connections 2024-04-22T15:22:25.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:25.899+0000 7f9b58114700 1 -- 172.21.15.96:0/2954441999 wait complete. 2024-04-22T15:22:25.911 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)" 2024-04-22T15:22:26.554 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-22T15:22:26.760 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:27.100 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:26 smithi096 ceph-mon[26688]: pgmap v95: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:27.100 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:26 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2954441999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:26 smithi196 ceph-mon[34717]: pgmap v95: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:26 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2954441999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:28.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.370+0000 7fe51ae2f700 1 -- 172.21.15.96:0/2040320573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe514105b60 msgr2=0x7fe514105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:28.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.370+0000 7fe51ae2f700 1 --2- 172.21.15.96:0/2040320573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe514105b60 0x7fe514105f40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe508009910 tx=0x7fe508009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:28.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.370+0000 7fe51ae2f700 1 -- 172.21.15.96:0/2040320573 shutdown_connections 2024-04-22T15:22:28.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.370+0000 7fe51ae2f700 1 --2- 172.21.15.96:0/2040320573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5140ffad0 0x7fe5140fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.370+0000 7fe51ae2f700 1 --2- 172.21.15.96:0/2040320573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe514105b60 0x7fe514105f40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.370+0000 7fe51ae2f700 1 -- 172.21.15.96:0/2040320573 >> 172.21.15.96:0/2040320573 conn(0x7fe5140fb5e0 msgr2=0x7fe5140fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.370+0000 7fe51ae2f700 1 -- 172.21.15.96:0/2040320573 shutdown_connections 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.370+0000 7fe51ae2f700 1 -- 172.21.15.96:0/2040320573 wait complete. 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.370+0000 7fe51ae2f700 1 Processor -- start 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe51ae2f700 1 -- start start 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe51ae2f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5140ffad0 0x7fe514197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe51ae2f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe514105b60 0x7fe514197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe51ae2f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe514198560 con 0x7fe5140ffad0 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe51ae2f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5141986a0 con 0x7fe514105b60 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe518bcb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5140ffad0 0x7fe514197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe518bcb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5140ffad0 0x7fe514197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35038/0 (socket says 172.21.15.96:35038) 2024-04-22T15:22:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe518bcb700 1 -- 172.21.15.96:0/3074357291 learned_addr learned my addr 172.21.15.96:0/3074357291 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe518bcb700 1 -- 172.21.15.96:0/3074357291 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe514105b60 msgr2=0x7fe514197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe518bcb700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe514105b60 0x7fe514197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe518bcb700 1 -- 172.21.15.96:0/3074357291 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5080095f0 con 0x7fe5140ffad0 2024-04-22T15:22:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe518bcb700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5140ffad0 0x7fe514197a50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe508006010 tx=0x7fe50800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.371+0000 7fe511ffb700 1 -- 172.21.15.96:0/3074357291 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe50801d070 con 0x7fe5140ffad0 2024-04-22T15:22:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.372+0000 7fe511ffb700 1 -- 172.21.15.96:0/3074357291 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe50800f460 con 0x7fe5140ffad0 2024-04-22T15:22:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.372+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5141988c0 con 0x7fe5140ffad0 2024-04-22T15:22:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.372+0000 7fe511ffb700 1 -- 172.21.15.96:0/3074357291 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe508017630 con 0x7fe5140ffad0 2024-04-22T15:22:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.372+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5141a0e20 con 0x7fe5140ffad0 2024-04-22T15:22:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.373+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5141080d0 con 0x7fe5140ffad0 2024-04-22T15:22:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.373+0000 7fe511ffb700 1 -- 172.21.15.96:0/3074357291 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7fe50800f5d0 con 0x7fe5140ffad0 2024-04-22T15:22:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.374+0000 7fe511ffb700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe50406e950 0x7fe504070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.374+0000 7fe511ffb700 1 -- 172.21.15.96:0/3074357291 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fe50808fc90 con 0x7fe5140ffad0 2024-04-22T15:22:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.374+0000 7fe513fff700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe50406e950 0x7fe504070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.374+0000 7fe513fff700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe50406e950 0x7fe504070e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe514069920 tx=0x7fe500006d20 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.378+0000 7fe511ffb700 1 -- 172.21.15.96:0/3074357291 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fe508058fe0 con 0x7fe5140ffad0 2024-04-22T15:22:28.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.663+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe5141988c0 con 0x7fe5140ffad0 2024-04-22T15:22:28.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.664+0000 7fe511ffb700 1 -- 172.21.15.96:0/3074357291 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7fe5141988c0 con 0x7fe5140ffad0 2024-04-22T15:22:28.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe50406e950 msgr2=0x7fe504070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:28.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe50406e950 0x7fe504070e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe514069920 tx=0x7fe500006d20 comp rx=0 tx=0).stop 2024-04-22T15:22:28.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5140ffad0 msgr2=0x7fe514197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:28.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5140ffad0 0x7fe514197a50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe508006010 tx=0x7fe50800be20 comp rx=0 tx=0).stop 2024-04-22T15:22:28.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 shutdown_connections 2024-04-22T15:22:28.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fe50406e950 0x7fe504070e10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:28.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5140ffad0 0x7fe514197a50 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:28.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 --2- 172.21.15.96:0/3074357291 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe514105b60 0x7fe514197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:28.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 >> 172.21.15.96:0/3074357291 conn(0x7fe5140fb5e0 msgr2=0x7fe5140fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:28.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 shutdown_connections 2024-04-22T15:22:28.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:28.666+0000 7fe51ae2f700 1 -- 172.21.15.96:0/3074357291 wait complete. 2024-04-22T15:22:28.679 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:22:28.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:28 smithi096 ceph-mon[26688]: pgmap v96: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:28.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:28 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3074357291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:28 smithi196 ceph-mon[34717]: pgmap v96: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:28 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3074357291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:29.277 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-22T15:22:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:30 smithi096 ceph-mon[26688]: pgmap v97: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:30 smithi196 ceph-mon[34717]: pgmap v97: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:31.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.267+0000 7fba7d606700 1 -- 172.21.15.96:0/3728461102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 msgr2=0x7fba78102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:31.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.267+0000 7fba7d606700 1 --2- 172.21.15.96:0/3728461102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 0x7fba78102140 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fba68009910 tx=0x7fba68009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:31.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.267+0000 7fba7d606700 1 -- 172.21.15.96:0/3728461102 shutdown_connections 2024-04-22T15:22:31.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.267+0000 7fba7d606700 1 --2- 172.21.15.96:0/3728461102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba78102680 0x7fba7810ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:31.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.267+0000 7fba7d606700 1 --2- 172.21.15.96:0/3728461102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 0x7fba78102140 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:31.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.267+0000 7fba7d606700 1 -- 172.21.15.96:0/3728461102 >> 172.21.15.96:0/3728461102 conn(0x7fba78076740 msgr2=0x7fba780fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:31.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.267+0000 7fba7d606700 1 -- 172.21.15.96:0/3728461102 shutdown_connections 2024-04-22T15:22:31.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.267+0000 7fba7d606700 1 -- 172.21.15.96:0/3728461102 wait complete. 2024-04-22T15:22:31.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.267+0000 7fba7d606700 1 Processor -- start 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba7d606700 1 -- start start 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba7d606700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 0x7fba7819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba7d606700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba78102680 0x7fba7819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba7d606700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7819d440 con 0x7fba78101d60 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba7d606700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba78196a20 con 0x7fba78102680 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba76ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 0x7fba7819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba76ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 0x7fba7819c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54630/0 (socket says 172.21.15.96:54630) 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba76ffd700 1 -- 172.21.15.96:0/479997110 learned_addr learned my addr 172.21.15.96:0/479997110 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba6e7fc700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba78102680 0x7fba7819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba76ffd700 1 -- 172.21.15.96:0/479997110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba78102680 msgr2=0x7fba7819ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba76ffd700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba78102680 0x7fba7819ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba76ffd700 1 -- 172.21.15.96:0/479997110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba680095f0 con 0x7fba78101d60 2024-04-22T15:22:31.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba6e7fc700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba78102680 0x7fba7819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:22:31.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.268+0000 7fba76ffd700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 0x7fba7819c8e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fba6800bd70 tx=0x7fba6800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:31.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.269+0000 7fba74ff9700 1 -- 172.21.15.96:0/479997110 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba6801d070 con 0x7fba78101d60 2024-04-22T15:22:31.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.269+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba78196ca0 con 0x7fba78101d60 2024-04-22T15:22:31.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.269+0000 7fba74ff9700 1 -- 172.21.15.96:0/479997110 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba6800f460 con 0x7fba78101d60 2024-04-22T15:22:31.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.269+0000 7fba74ff9700 1 -- 172.21.15.96:0/479997110 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba68017630 con 0x7fba78101d60 2024-04-22T15:22:31.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.269+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba78197190 con 0x7fba78101d60 2024-04-22T15:22:31.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.270+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba781054f0 con 0x7fba78101d60 2024-04-22T15:22:31.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.270+0000 7fba74ff9700 1 -- 172.21.15.96:0/479997110 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7fba6800f5d0 con 0x7fba78101d60 2024-04-22T15:22:31.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.270+0000 7fba74ff9700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fba6406e940 0x7fba64070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:31.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.271+0000 7fba74ff9700 1 -- 172.21.15.96:0/479997110 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fba6808fe30 con 0x7fba78101d60 2024-04-22T15:22:31.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.271+0000 7fba6e7fc700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fba6406e940 0x7fba64070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:31.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.271+0000 7fba6e7fc700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fba6406e940 0x7fba64070e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fba78197be0 tx=0x7fba60008040 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:31.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.275+0000 7fba74ff9700 1 -- 172.21.15.96:0/479997110 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fba6805cbf0 con 0x7fba78101d60 2024-04-22T15:22:31.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:31.497+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}) v1 -- 0x7fba78197f80 con 0x7fba6406e940 2024-04-22T15:22:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:32 smithi096 ceph-mon[26688]: from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:32 smithi096 ceph-mon[26688]: pgmap v98: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:33.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:32 smithi196 ceph-mon[34717]: from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:33.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:32 smithi196 ceph-mon[34717]: pgmap v98: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:33.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.567+0000 7fba74ff9700 1 -- 172.21.15.96:0/479997110 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5756 (secure 0 0 0) 0x7fba78197f80 con 0x7fba6406e940 2024-04-22T15:22:33.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fba6406e940 msgr2=0x7fba64070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:33.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fba6406e940 0x7fba64070e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fba78197be0 tx=0x7fba60008040 comp rx=0 tx=0).stop 2024-04-22T15:22:33.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 msgr2=0x7fba7819c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:33.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 0x7fba7819c8e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fba6800bd70 tx=0x7fba6800be50 comp rx=0 tx=0).stop 2024-04-22T15:22:33.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 shutdown_connections 2024-04-22T15:22:33.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fba6406e940 0x7fba64070e00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:33.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba78101d60 0x7fba7819c8e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:33.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 --2- 172.21.15.96:0/479997110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba78102680 0x7fba7819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:33.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 >> 172.21.15.96:0/479997110 conn(0x7fba78076740 msgr2=0x7fba78102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:33.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 shutdown_connections 2024-04-22T15:22:33.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:33.570+0000 7fba7d606700 1 -- 172.21.15.96:0/479997110 wait complete. 2024-04-22T15:22:33.583 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:22:34.202 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch ps' 2024-04-22T15:22:34.409 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:35.186 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:34 smithi096 ceph-mon[26688]: pgmap v99: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:35.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:34 smithi196 ceph-mon[34717]: pgmap v99: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:36.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.013+0000 7fd61ecff700 1 -- 172.21.15.96:0/3972595150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6180ffaa0 msgr2=0x7fd6180fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:36.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.013+0000 7fd61ecff700 1 --2- 172.21.15.96:0/3972595150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6180ffaa0 0x7fd6180fff20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd608009910 tx=0x7fd608009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:36.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.013+0000 7fd61ecff700 1 -- 172.21.15.96:0/3972595150 shutdown_connections 2024-04-22T15:22:36.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.013+0000 7fd61ecff700 1 --2- 172.21.15.96:0/3972595150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6180ffaa0 0x7fd6180fff20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.013+0000 7fd61ecff700 1 --2- 172.21.15.96:0/3972595150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd618105aa0 0x7fd618105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.013+0000 7fd61ecff700 1 -- 172.21.15.96:0/3972595150 >> 172.21.15.96:0/3972595150 conn(0x7fd618076740 msgr2=0x7fd6180fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.013+0000 7fd61ecff700 1 -- 172.21.15.96:0/3972595150 shutdown_connections 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.013+0000 7fd61ecff700 1 -- 172.21.15.96:0/3972595150 wait complete. 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd61ecff700 1 Processor -- start 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd61ecff700 1 -- start start 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd61ecff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6180ffaa0 0x7fd618195710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd61ecff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd618105aa0 0x7fd618195c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd61ecff700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd618069550 con 0x7fd618105aa0 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd61ecff700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd618196190 con 0x7fd6180ffaa0 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd617fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd618105aa0 0x7fd618195c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd617fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd618105aa0 0x7fd618195c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54646/0 (socket says 172.21.15.96:54646) 2024-04-22T15:22:36.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd617fff700 1 -- 172.21.15.96:0/3917397341 learned_addr learned my addr 172.21.15.96:0/3917397341 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd61ca9b700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6180ffaa0 0x7fd618195710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd617fff700 1 -- 172.21.15.96:0/3917397341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6180ffaa0 msgr2=0x7fd618195710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd617fff700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6180ffaa0 0x7fd618195710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd617fff700 1 -- 172.21.15.96:0/3917397341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6080095f0 con 0x7fd618105aa0 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.014+0000 7fd617fff700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd618105aa0 0x7fd618195c50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd60800bd70 tx=0x7fd60800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.015+0000 7fd615ffb700 1 -- 172.21.15.96:0/3917397341 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60801d070 con 0x7fd618105aa0 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.015+0000 7fd615ffb700 1 -- 172.21.15.96:0/3917397341 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6080056e0 con 0x7fd618105aa0 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.015+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd618196410 con 0x7fd618105aa0 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.015+0000 7fd615ffb700 1 -- 172.21.15.96:0/3917397341 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60800fe70 con 0x7fd618105aa0 2024-04-22T15:22:36.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.015+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6181a2f50 con 0x7fd618105aa0 2024-04-22T15:22:36.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.016+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6181080c0 con 0x7fd618105aa0 2024-04-22T15:22:36.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.016+0000 7fd615ffb700 1 -- 172.21.15.96:0/3917397341 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7fd608003a60 con 0x7fd618105aa0 2024-04-22T15:22:36.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.017+0000 7fd615ffb700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd60406e940 0x7fd604070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:36.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.017+0000 7fd615ffb700 1 -- 172.21.15.96:0/3917397341 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fd60808fc40 con 0x7fd618105aa0 2024-04-22T15:22:36.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.017+0000 7fd61ca9b700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd60406e940 0x7fd604070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:36.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.017+0000 7fd61ca9b700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd60406e940 0x7fd604070e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd610009550 tx=0x7fd6100091b0 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:36.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.021+0000 7fd615ffb700 1 -- 172.21.15.96:0/3917397341 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fd60805ca00 con 0x7fd618105aa0 2024-04-22T15:22:36.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.231+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd61810cca0 con 0x7fd60406e940 2024-04-22T15:22:36.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.242+0000 7fd615ffb700 1 -- 172.21.15.96:0/3917397341 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd61810cca0 con 0x7fd60406e940 2024-04-22T15:22:36.245 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:22:36.245 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (79s) 48s ago 12m 17.0M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:22:36.245 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (12m) 48s ago 12m 8182k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:22:36.245 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (11m) 2m ago 11m 7864k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:22:36.245 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (12m) 48s ago 12m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (11m) 2m ago 11m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (54s) 48s ago 12m 55.1M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (6m) 48s ago 6m 156M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (6m) 48s ago 6m 89.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (6m) 2m ago 6m 26.4M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (6m) 2m ago 6m 25.8M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (3m) 48s ago 14m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (2m) 2m ago 11m 449M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (14m) 48s ago 14m 57.7M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (11m) 2m ago 11m 44.9M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (2m) 48s ago 12m 16.7M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (2m) 2m ago 11m 3602k - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:22:36.246 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (10m) 48s ago 10m 209M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:22:36.247 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (10m) 48s ago 10m 215M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:22:36.247 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (9m) 48s ago 9m 181M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:22:36.247 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (9m) 2m ago 9m 231M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:22:36.247 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (8m) 2m ago 8m 208M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:22:36.247 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (8m) 2m ago 8m 185M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:22:36.247 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (96s) 48s ago 12m 58.6M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd60406e940 msgr2=0x7fd604070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd60406e940 0x7fd604070e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd610009550 tx=0x7fd6100091b0 comp rx=0 tx=0).stop 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd618105aa0 msgr2=0x7fd618195c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd618105aa0 0x7fd618195c50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd60800bd70 tx=0x7fd60800bda0 comp rx=0 tx=0).stop 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 shutdown_connections 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fd60406e940 0x7fd604070e00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6180ffaa0 0x7fd618195710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 --2- 172.21.15.96:0/3917397341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd618105aa0 0x7fd618195c50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 >> 172.21.15.96:0/3917397341 conn(0x7fd618076740 msgr2=0x7fd6180fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:36.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 shutdown_connections 2024-04-22T15:22:36.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:36.244+0000 7fd61ecff700 1 -- 172.21.15.96:0/3917397341 wait complete. 2024-04-22T15:22:36.883 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T15:22:36.884 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:22:36.885 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-22T15:22:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:36 smithi096 ceph-mon[26688]: pgmap v100: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:36 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:22:37.092 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:36 smithi196 ceph-mon[34717]: pgmap v100: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:36 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:22:38.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:37 smithi096 ceph-mon[26688]: from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:38.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:37 smithi196 ceph-mon[34717]: from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:38.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.901+0000 7f6592fee700 1 -- 172.21.15.96:0/3636842613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c101de0 msgr2=0x7f658c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.901+0000 7f6592fee700 1 --2- 172.21.15.96:0/3636842613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c101de0 0x7f658c1021c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f657c009940 tx=0x7f657c009c50 comp rx=0 tx=0).stop 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.901+0000 7f6592fee700 1 -- 172.21.15.96:0/3636842613 shutdown_connections 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.901+0000 7f6592fee700 1 --2- 172.21.15.96:0/3636842613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f658c102700 0x7f658c10abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.901+0000 7f6592fee700 1 --2- 172.21.15.96:0/3636842613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c101de0 0x7f658c1021c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.901+0000 7f6592fee700 1 -- 172.21.15.96:0/3636842613 >> 172.21.15.96:0/3636842613 conn(0x7f658c076740 msgr2=0x7f658c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.901+0000 7f6592fee700 1 -- 172.21.15.96:0/3636842613 shutdown_connections 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.901+0000 7f6592fee700 1 -- 172.21.15.96:0/3636842613 wait complete. 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f6592fee700 1 Processor -- start 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f6592fee700 1 -- start start 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f6592fee700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f658c101de0 0x7f658c0ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f6592fee700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c102700 0x7f658c0fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:38.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f6592fee700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c1004c0 con 0x7f658c102700 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f6592fee700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c100600 con 0x7f658c101de0 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f658bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c102700 0x7f658c0fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f658bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c102700 0x7f658c0fff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54660/0 (socket says 172.21.15.96:54660) 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f658bfff700 1 -- 172.21.15.96:0/3988480348 learned_addr learned my addr 172.21.15.96:0/3988480348 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.902+0000 7f6590d8a700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f658c101de0 0x7f658c0ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.903+0000 7f658bfff700 1 -- 172.21.15.96:0/3988480348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f658c101de0 msgr2=0x7f658c0ffa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.903+0000 7f658bfff700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f658c101de0 0x7f658c0ffa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.903+0000 7f658bfff700 1 -- 172.21.15.96:0/3988480348 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c0095f0 con 0x7f658c102700 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.903+0000 7f658bfff700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c102700 0x7f658c0fff80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f658000b480 tx=0x7f658000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.903+0000 7f6589ffb700 1 -- 172.21.15.96:0/3988480348 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65800118c0 con 0x7f658c102700 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.903+0000 7f6589ffb700 1 -- 172.21.15.96:0/3988480348 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6580010380 con 0x7f658c102700 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.903+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c1a2dc0 con 0x7f658c102700 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.903+0000 7f6589ffb700 1 -- 172.21.15.96:0/3988480348 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f658000f4f0 con 0x7f658c102700 2024-04-22T15:22:38.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.903+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c1a3120 con 0x7f658c102700 2024-04-22T15:22:38.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.904+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658c068e20 con 0x7f658c102700 2024-04-22T15:22:38.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.904+0000 7f6589ffb700 1 -- 172.21.15.96:0/3988480348 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f65800104f0 con 0x7f658c102700 2024-04-22T15:22:38.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.905+0000 7f6589ffb700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f657406e940 0x7f6574070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:38.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.905+0000 7f6589ffb700 1 -- 172.21.15.96:0/3988480348 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f658008e550 con 0x7f658c102700 2024-04-22T15:22:38.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.905+0000 7f6590d8a700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f657406e940 0x7f6574070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:38.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.905+0000 7f6590d8a700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f657406e940 0x7f6574070e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f657c00b870 tx=0x7f657c005c00 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:38.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:38.909+0000 7f6589ffb700 1 -- 172.21.15.96:0/3988480348 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f65800577e0 con 0x7f658c102700 2024-04-22T15:22:39.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.107+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f658c100f20 con 0x7f658c102700 2024-04-22T15:22:39.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:38 smithi096 ceph-mon[26688]: pgmap v101: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:39.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.112+0000 7f6589ffb700 1 -- 172.21.15.96:0/3988480348 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v45)=0 v45) v1 ==== 125+0+0 (secure 0 0 0) 0x7f658005aea0 con 0x7f658c102700 2024-04-22T15:22:39.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f657406e940 msgr2=0x7f6574070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:39.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f657406e940 0x7f6574070e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f657c00b870 tx=0x7f657c005c00 comp rx=0 tx=0).stop 2024-04-22T15:22:39.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c102700 msgr2=0x7f658c0fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:39.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c102700 0x7f658c0fff80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f658000b480 tx=0x7f658000b840 comp rx=0 tx=0).stop 2024-04-22T15:22:39.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 shutdown_connections 2024-04-22T15:22:39.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f657406e940 0x7f6574070e00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:39.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f658c101de0 0x7f658c0ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:39.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 --2- 172.21.15.96:0/3988480348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f658c102700 0x7f658c0fff80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:39.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 >> 172.21.15.96:0/3988480348 conn(0x7f658c076740 msgr2=0x7f658c1030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:39.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 shutdown_connections 2024-04-22T15:22:39.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:39.114+0000 7f6592fee700 1 -- 172.21.15.96:0/3988480348 wait complete. 2024-04-22T15:22:39.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:38 smithi196 ceph-mon[34717]: pgmap v101: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:39.767 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T15:22:39.767 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:22:39.767 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T15:22:39.974 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:40.440 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:40 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3988480348' entity='client.admin' 2024-04-22T15:22:40.440 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:40 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:22:40.440 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:40 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:22:40.440 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:40 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:22:40.441 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:40 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:40.441 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:40 smithi096 ceph-mon[26688]: pgmap v102: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:40.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:40 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3988480348' entity='client.admin' 2024-04-22T15:22:40.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:40 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:22:40.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:40 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:22:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:40 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:22:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:40 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:40 smithi196 ceph-mon[34717]: pgmap v102: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:41.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.620+0000 7f4fad460700 1 -- 172.21.15.96:0/2039818395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 msgr2=0x7f4fa81011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:41.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.620+0000 7f4fad460700 1 --2- 172.21.15.96:0/2039818395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 0x7f4fa81011f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009910 tx=0x7f4f98009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:41.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 -- 172.21.15.96:0/2039818395 shutdown_connections 2024-04-22T15:22:41.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 --2- 172.21.15.96:0/2039818395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fa81017c0 0x7f4fa8105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:41.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 --2- 172.21.15.96:0/2039818395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 0x7f4fa81011f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:41.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 -- 172.21.15.96:0/2039818395 >> 172.21.15.96:0/2039818395 conn(0x7f4fa80fc6e0 msgr2=0x7f4fa80feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:41.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 -- 172.21.15.96:0/2039818395 shutdown_connections 2024-04-22T15:22:41.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 -- 172.21.15.96:0/2039818395 wait complete. 2024-04-22T15:22:41.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 Processor -- start 2024-04-22T15:22:41.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 -- start start 2024-04-22T15:22:41.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 0x7f4fa819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:41.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fa81017c0 0x7f4fa819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:41.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa819d360 con 0x7f4fa8100e10 2024-04-22T15:22:41.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.621+0000 7f4fad460700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa8196940 con 0x7f4fa81017c0 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fa6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 0x7f4fa819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fa6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 0x7f4fa819c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53460/0 (socket says 172.21.15.96:53460) 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fa6ffd700 1 -- 172.21.15.96:0/85252153 learned_addr learned my addr 172.21.15.96:0/85252153 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fa67fc700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fa81017c0 0x7f4fa819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fa6ffd700 1 -- 172.21.15.96:0/85252153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fa81017c0 msgr2=0x7f4fa819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fa6ffd700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fa81017c0 0x7f4fa819cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fa6ffd700 1 -- 172.21.15.96:0/85252153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f980095f0 con 0x7f4fa8100e10 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fa67fc700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fa81017c0 0x7f4fa819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fa6ffd700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 0x7f4fa819c800 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4f98006010 tx=0x7f4f9800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4f8ffff700 1 -- 172.21.15.96:0/85252153 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f9801d070 con 0x7f4fa8100e10 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4f8ffff700 1 -- 172.21.15.96:0/85252153 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f980056e0 con 0x7f4fa8100e10 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa8196bc0 con 0x7f4fa8100e10 2024-04-22T15:22:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4f8ffff700 1 -- 172.21.15.96:0/85252153 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f98017400 con 0x7f4fa8100e10 2024-04-22T15:22:41.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.622+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa81970b0 con 0x7f4fa8100e10 2024-04-22T15:22:41.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.623+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa81091b0 con 0x7f4fa8100e10 2024-04-22T15:22:41.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.624+0000 7f4f8ffff700 1 -- 172.21.15.96:0/85252153 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f4f9800f6f0 con 0x7f4fa8100e10 2024-04-22T15:22:41.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.624+0000 7f4f8ffff700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f4f9006e940 0x7f4f90070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.624+0000 7f4f8ffff700 1 -- 172.21.15.96:0/85252153 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f4f9808fce0 con 0x7f4fa8100e10 2024-04-22T15:22:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.624+0000 7f4fa67fc700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f4f9006e940 0x7f4f90070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.625+0000 7f4fa67fc700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f4f9006e940 0x7f4f90070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c005d90 tx=0x7f4f9c005d00 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.629+0000 7f4f8ffff700 1 -- 172.21.15.96:0/85252153 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f4f9805caa0 con 0x7f4fa8100e10 2024-04-22T15:22:41.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.832+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f4fa8197e00 con 0x7f4fa8100e10 2024-04-22T15:22:41.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.832+0000 7f4f8ffff700 1 -- 172.21.15.96:0/85252153 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v45)=0 v45) v1 ==== 155+0+0 (secure 0 0 0) 0x7f4f9805c630 con 0x7f4fa8100e10 2024-04-22T15:22:41.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f4f9006e940 msgr2=0x7f4f90070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:41.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f4f9006e940 0x7f4f90070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c005d90 tx=0x7f4f9c005d00 comp rx=0 tx=0).stop 2024-04-22T15:22:41.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 msgr2=0x7f4fa819c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:41.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 0x7f4fa819c800 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4f98006010 tx=0x7f4f9800be20 comp rx=0 tx=0).stop 2024-04-22T15:22:41.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 shutdown_connections 2024-04-22T15:22:41.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f4f9006e940 0x7f4f90070e00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:41.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa8100e10 0x7f4fa819c800 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:41.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 --2- 172.21.15.96:0/85252153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fa81017c0 0x7f4fa819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:41.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 >> 172.21.15.96:0/85252153 conn(0x7f4fa80fc6e0 msgr2=0x7f4fa80fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:41.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 shutdown_connections 2024-04-22T15:22:41.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:41.835+0000 7f4fad460700 1 -- 172.21.15.96:0/85252153 wait complete. 2024-04-22T15:22:42.389 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T15:22:42.594 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:42.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:42 smithi096 ceph-mon[26688]: pgmap v103: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:43.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:42 smithi196 ceph-mon[34717]: pgmap v103: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 -- 172.21.15.96:0/3571432809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 msgr2=0x7f0a2c105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 --2- 172.21.15.96:0/3571432809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 0x7f0a2c105f40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009910 tx=0x7f0a1c009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 -- 172.21.15.96:0/3571432809 shutdown_connections 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 --2- 172.21.15.96:0/3571432809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a2c0ffad0 0x7f0a2c0fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 --2- 172.21.15.96:0/3571432809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 0x7f0a2c105f40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 -- 172.21.15.96:0/3571432809 >> 172.21.15.96:0/3571432809 conn(0x7f0a2c0fb5e0 msgr2=0x7f0a2c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 -- 172.21.15.96:0/3571432809 shutdown_connections 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 -- 172.21.15.96:0/3571432809 wait complete. 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 Processor -- start 2024-04-22T15:22:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.339+0000 7f0a327dd700 1 -- start start 2024-04-22T15:22:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a327dd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a2c0ffad0 0x7f0a2c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a327dd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 0x7f0a2c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a327dd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c1984c0 con 0x7f0a2c105b60 2024-04-22T15:22:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a327dd700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c198600 con 0x7f0a2c0ffad0 2024-04-22T15:22:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a2b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 0x7f0a2c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a2b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 0x7f0a2c197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53472/0 (socket says 172.21.15.96:53472) 2024-04-22T15:22:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a2b7fe700 1 -- 172.21.15.96:0/2165259942 learned_addr learned my addr 172.21.15.96:0/2165259942 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a2bfff700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a2c0ffad0 0x7f0a2c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a2b7fe700 1 -- 172.21.15.96:0/2165259942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a2c0ffad0 msgr2=0x7f0a2c1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a2b7fe700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a2c0ffad0 0x7f0a2c1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a2b7fe700 1 -- 172.21.15.96:0/2165259942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c0095f0 con 0x7f0a2c105b60 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a2bfff700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a2c0ffad0 0x7f0a2c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a2b7fe700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 0x7f0a2c197ef0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000c8f0 tx=0x7f0a2000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a297fa700 1 -- 172.21.15.96:0/2165259942 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a20007a70 con 0x7f0a2c105b60 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a297fa700 1 -- 172.21.15.96:0/2165259942 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a20004cf0 con 0x7f0a2c105b60 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2c1a0a20 con 0x7f0a2c105b60 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a297fa700 1 -- 172.21.15.96:0/2165259942 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a20005610 con 0x7f0a2c105b60 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.340+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2c1a0e30 con 0x7f0a2c105b60 2024-04-22T15:22:44.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.342+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2c1080d0 con 0x7f0a2c105b60 2024-04-22T15:22:44.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.342+0000 7f0a297fa700 1 -- 172.21.15.96:0/2165259942 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f0a20004830 con 0x7f0a2c105b60 2024-04-22T15:22:44.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.342+0000 7f0a297fa700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0a1406e940 0x7f0a14070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:44.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.343+0000 7f0a297fa700 1 -- 172.21.15.96:0/2165259942 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f0a20013070 con 0x7f0a2c105b60 2024-04-22T15:22:44.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.343+0000 7f0a2bfff700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0a1406e940 0x7f0a14070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:44.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.343+0000 7f0a2bfff700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0a1406e940 0x7f0a14070e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c006010 tx=0x7f0a1c00b540 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:44.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.347+0000 7f0a297fa700 1 -- 172.21.15.96:0/2165259942 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f0a20057aa0 con 0x7f0a2c105b60 2024-04-22T15:22:44.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.547+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f0a2c1987c0 con 0x7f0a2c105b60 2024-04-22T15:22:44.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.548+0000 7f0a297fa700 1 -- 172.21.15.96:0/2165259942 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v45)=0 v45) v1 ==== 163+0+0 (secure 0 0 0) 0x7f0a2005b160 con 0x7f0a2c105b60 2024-04-22T15:22:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0a1406e940 msgr2=0x7f0a14070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0a1406e940 0x7f0a14070e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c006010 tx=0x7f0a1c00b540 comp rx=0 tx=0).stop 2024-04-22T15:22:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 msgr2=0x7f0a2c197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 0x7f0a2c197ef0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000c8f0 tx=0x7f0a2000ccb0 comp rx=0 tx=0).stop 2024-04-22T15:22:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 shutdown_connections 2024-04-22T15:22:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0a1406e940 0x7f0a14070e00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a2c0ffad0 0x7f0a2c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 --2- 172.21.15.96:0/2165259942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a2c105b60 0x7f0a2c197ef0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 >> 172.21.15.96:0/2165259942 conn(0x7f0a2c0fb5e0 msgr2=0x7f0a2c0fc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 shutdown_connections 2024-04-22T15:22:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:44.550+0000 7f0a327dd700 1 -- 172.21.15.96:0/2165259942 wait complete. 2024-04-22T15:22:45.191 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:44 smithi096 ceph-mon[26688]: pgmap v104: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:45.215 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T15:22:45.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:44 smithi196 ceph-mon[34717]: pgmap v104: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:45.420 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:47.094 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:46 smithi096 ceph-mon[26688]: pgmap v105: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:47.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.117+0000 7f738bb08700 1 -- 172.21.15.96:0/2405375743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 msgr2=0x7f73840fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:47.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.117+0000 7f738bb08700 1 --2- 172.21.15.96:0/2405375743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 0x7f73840fff30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7374009910 tx=0x7f7374009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:47.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.117+0000 7f738bb08700 1 -- 172.21.15.96:0/2405375743 shutdown_connections 2024-04-22T15:22:47.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.117+0000 7f738bb08700 1 --2- 172.21.15.96:0/2405375743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 0x7f73840fff30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:47.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.117+0000 7f738bb08700 1 --2- 172.21.15.96:0/2405375743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7384105ab0 0x7f7384105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:47.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.117+0000 7f738bb08700 1 -- 172.21.15.96:0/2405375743 >> 172.21.15.96:0/2405375743 conn(0x7f73840fb5e0 msgr2=0x7f73840fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:47.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.117+0000 7f738bb08700 1 -- 172.21.15.96:0/2405375743 shutdown_connections 2024-04-22T15:22:47.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.117+0000 7f738bb08700 1 -- 172.21.15.96:0/2405375743 wait complete. 2024-04-22T15:22:47.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f738bb08700 1 Processor -- start 2024-04-22T15:22:47.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f738bb08700 1 -- start start 2024-04-22T15:22:47.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f738bb08700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 0x7f738419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:47.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f738bb08700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7384105ab0 0x7f738419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:47.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f738bb08700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738419d390 con 0x7f73840ffab0 2024-04-22T15:22:47.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f738bb08700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7384196970 con 0x7f7384105ab0 2024-04-22T15:22:47.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f73898a4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 0x7f738419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:47.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f73898a4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 0x7f738419c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53486/0 (socket says 172.21.15.96:53486) 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f73898a4700 1 -- 172.21.15.96:0/729608679 learned_addr learned my addr 172.21.15.96:0/729608679 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.118+0000 7f73890a3700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7384105ab0 0x7f738419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.119+0000 7f73898a4700 1 -- 172.21.15.96:0/729608679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7384105ab0 msgr2=0x7f738419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.119+0000 7f73898a4700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7384105ab0 0x7f738419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.119+0000 7f73898a4700 1 -- 172.21.15.96:0/729608679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73740095f0 con 0x7f73840ffab0 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.119+0000 7f73898a4700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 0x7f738419c830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f737c00c8a0 tx=0x7f737c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.119+0000 7f737affd700 1 -- 172.21.15.96:0/729608679 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f737c0078e0 con 0x7f73840ffab0 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.119+0000 7f737affd700 1 -- 172.21.15.96:0/729608679 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f737c00f450 con 0x7f73840ffab0 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.119+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7384196c50 con 0x7f73840ffab0 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.119+0000 7f737affd700 1 -- 172.21.15.96:0/729608679 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f737c00e5e0 con 0x7f73840ffab0 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.119+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73841971a0 con 0x7f73840ffab0 2024-04-22T15:22:47.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.120+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73841080d0 con 0x7f73840ffab0 2024-04-22T15:22:47.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.121+0000 7f737affd700 1 -- 172.21.15.96:0/729608679 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f737c00f5c0 con 0x7f73840ffab0 2024-04-22T15:22:47.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.121+0000 7f737affd700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f737006e940 0x7f7370070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:47.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.121+0000 7f73890a3700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f737006e940 0x7f7370070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:47.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.121+0000 7f737affd700 1 -- 172.21.15.96:0/729608679 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f737c08e750 con 0x7f73840ffab0 2024-04-22T15:22:47.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.122+0000 7f73890a3700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f737006e940 0x7f7370070e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7384197b30 tx=0x7f737401a040 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:47.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.126+0000 7f737affd700 1 -- 172.21.15.96:0/729608679 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f737c05b590 con 0x7f73840ffab0 2024-04-22T15:22:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:46 smithi196 ceph-mon[34717]: pgmap v105: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:47.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.325+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f73841011f0 con 0x7f73840ffab0 2024-04-22T15:22:47.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.325+0000 7f737affd700 1 -- 172.21.15.96:0/729608679 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v45)=0 v45) v1 ==== 135+0+0 (secure 0 0 0) 0x7f737c05b120 con 0x7f73840ffab0 2024-04-22T15:22:47.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f737006e940 msgr2=0x7f7370070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:47.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f737006e940 0x7f7370070e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7384197b30 tx=0x7f737401a040 comp rx=0 tx=0).stop 2024-04-22T15:22:47.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 msgr2=0x7f738419c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:47.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 0x7f738419c830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f737c00c8a0 tx=0x7f737c00cbb0 comp rx=0 tx=0).stop 2024-04-22T15:22:47.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 shutdown_connections 2024-04-22T15:22:47.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f737006e940 0x7f7370070e00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:47.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73840ffab0 0x7f738419c830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:47.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 --2- 172.21.15.96:0/729608679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7384105ab0 0x7f738419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:47.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 >> 172.21.15.96:0/729608679 conn(0x7f73840fb5e0 msgr2=0x7f73840fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:47.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 shutdown_connections 2024-04-22T15:22:47.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:47.327+0000 7f738bb08700 1 -- 172.21.15.96:0/729608679 wait complete. 2024-04-22T15:22:48.038 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-22T15:22:48.244 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:48.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:48 smithi096 ceph-mon[26688]: pgmap v106: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:49.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:48 smithi196 ceph-mon[34717]: pgmap v106: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:49.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.886+0000 7fa1f269d700 1 -- 172.21.15.96:0/1950876395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec102680 msgr2=0x7fa1ec10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:49.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.886+0000 7fa1f269d700 1 --2- 172.21.15.96:0/1950876395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec102680 0x7fa1ec10ab70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009910 tx=0x7fa1e0009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:49.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.886+0000 7fa1f269d700 1 -- 172.21.15.96:0/1950876395 shutdown_connections 2024-04-22T15:22:49.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.886+0000 7fa1f269d700 1 --2- 172.21.15.96:0/1950876395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec102680 0x7fa1ec10ab70 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:49.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.886+0000 7fa1f269d700 1 --2- 172.21.15.96:0/1950876395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1ec101d60 0x7fa1ec102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:49.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.886+0000 7fa1f269d700 1 -- 172.21.15.96:0/1950876395 >> 172.21.15.96:0/1950876395 conn(0x7fa1ec076740 msgr2=0x7fa1ec0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:49.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.886+0000 7fa1f269d700 1 -- 172.21.15.96:0/1950876395 shutdown_connections 2024-04-22T15:22:49.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.886+0000 7fa1f269d700 1 -- 172.21.15.96:0/1950876395 wait complete. 2024-04-22T15:22:49.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1f269d700 1 Processor -- start 2024-04-22T15:22:49.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1f269d700 1 -- start start 2024-04-22T15:22:49.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1f269d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec101d60 0x7fa1ec19c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:49.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1f269d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1ec102680 0x7fa1ec19cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:49.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1f269d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec196a80 con 0x7fa1ec101d60 2024-04-22T15:22:49.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1f269d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec196bf0 con 0x7fa1ec102680 2024-04-22T15:22:49.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1ebfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec101d60 0x7fa1ec19c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:49.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1ebfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec101d60 0x7fa1ec19c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53494/0 (socket says 172.21.15.96:53494) 2024-04-22T15:22:49.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1ebfff700 1 -- 172.21.15.96:0/967670502 learned_addr learned my addr 172.21.15.96:0/967670502 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:49.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.887+0000 7fa1eb7fe700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1ec102680 0x7fa1ec19cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:49.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.888+0000 7fa1ebfff700 1 -- 172.21.15.96:0/967670502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1ec102680 msgr2=0x7fa1ec19cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:49.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.888+0000 7fa1ebfff700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1ec102680 0x7fa1ec19cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:49.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.888+0000 7fa1ebfff700 1 -- 172.21.15.96:0/967670502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e00095f0 con 0x7fa1ec101d60 2024-04-22T15:22:49.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.888+0000 7fa1ebfff700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec101d60 0x7fa1ec19c9d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00b4b0 tx=0x7fa1dc00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:49.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.888+0000 7fa1e97fa700 1 -- 172.21.15.96:0/967670502 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc011800 con 0x7fa1ec101d60 2024-04-22T15:22:49.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.888+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec196ed0 con 0x7fa1ec101d60 2024-04-22T15:22:49.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.888+0000 7fa1e97fa700 1 -- 172.21.15.96:0/967670502 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1dc011e20 con 0x7fa1ec101d60 2024-04-22T15:22:49.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.888+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec1973f0 con 0x7fa1ec101d60 2024-04-22T15:22:49.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.888+0000 7fa1e97fa700 1 -- 172.21.15.96:0/967670502 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc00f2e0 con 0x7fa1ec101d60 2024-04-22T15:22:49.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.889+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec068ea0 con 0x7fa1ec101d60 2024-04-22T15:22:49.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.890+0000 7fa1e97fa700 1 -- 172.21.15.96:0/967670502 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7fa1dc011960 con 0x7fa1ec101d60 2024-04-22T15:22:49.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.890+0000 7fa1e97fa700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa1d4072d50 0x7fa1d4075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:49.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.890+0000 7fa1e97fa700 1 -- 172.21.15.96:0/967670502 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa1dc08e410 con 0x7fa1ec101d60 2024-04-22T15:22:49.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.890+0000 7fa1eb7fe700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa1d4072d50 0x7fa1d4075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:49.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.890+0000 7fa1eb7fe700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa1d4072d50 0x7fa1d4075210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec198280 tx=0x7fa1e001a040 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:49.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:49.895+0000 7fa1e97fa700 1 -- 172.21.15.96:0/967670502 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7fa1dc0576a0 con 0x7fa1ec101d60 2024-04-22T15:22:50.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.116+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1ec197720 con 0x7fa1d4072d50 2024-04-22T15:22:50.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.123+0000 7fa1e97fa700 1 -- 172.21.15.96:0/967670502 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fa1ec197720 con 0x7fa1d4072d50 2024-04-22T15:22:50.127 INFO:teuthology.orchestra.run.smithi096.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:22:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa1d4072d50 msgr2=0x7fa1d4075210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa1d4072d50 0x7fa1d4075210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec198280 tx=0x7fa1e001a040 comp rx=0 tx=0).stop 2024-04-22T15:22:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec101d60 msgr2=0x7fa1ec19c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec101d60 0x7fa1ec19c9d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00b4b0 tx=0x7fa1dc00b7c0 comp rx=0 tx=0).stop 2024-04-22T15:22:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 shutdown_connections 2024-04-22T15:22:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7fa1d4072d50 0x7fa1d4075210 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1ec101d60 0x7fa1ec19c9d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 --2- 172.21.15.96:0/967670502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1ec102680 0x7fa1ec19cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 >> 172.21.15.96:0/967670502 conn(0x7fa1ec076740 msgr2=0x7fa1ec1030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 shutdown_connections 2024-04-22T15:22:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:50.126+0000 7fa1f269d700 1 -- 172.21.15.96:0/967670502 wait complete. 2024-04-22T15:22:50.802 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T15:22:50.803 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T15:22:50.803 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-22T15:22:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:50 smithi096 ceph-mon[26688]: pgmap v107: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:50 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:50 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:22:51.019 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:22:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:50 smithi196 ceph-mon[34717]: pgmap v107: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:50 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:51.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:50 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:22:52.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:51 smithi096 ceph-mon[26688]: from='client.14872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:52.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:51 smithi096 ceph-mon[26688]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:22:52.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:51 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:22:52.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:51 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:22:52.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:51 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:52.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:51 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:22:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:51 smithi196 ceph-mon[34717]: from='client.14872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:51 smithi196 ceph-mon[34717]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:22:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:51 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:22:52.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:51 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:22:52.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:51 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:52.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:51 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:22:52.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.828+0000 7fafaafaa700 1 -- 172.21.15.96:0/2235310991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 msgr2=0x7fafa40fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:52.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.828+0000 7fafaafaa700 1 --2- 172.21.15.96:0/2235310991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 0x7fafa40fff30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7faf90009910 tx=0x7faf90009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:52.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.828+0000 7fafaafaa700 1 -- 172.21.15.96:0/2235310991 shutdown_connections 2024-04-22T15:22:52.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.828+0000 7fafaafaa700 1 --2- 172.21.15.96:0/2235310991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 0x7fafa40fff30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:52.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.828+0000 7fafaafaa700 1 --2- 172.21.15.96:0/2235310991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafa4105ab0 0x7fafa4105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:52.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.828+0000 7fafaafaa700 1 -- 172.21.15.96:0/2235310991 >> 172.21.15.96:0/2235310991 conn(0x7fafa40fb5e0 msgr2=0x7fafa40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:52.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.829+0000 7fafaafaa700 1 -- 172.21.15.96:0/2235310991 shutdown_connections 2024-04-22T15:22:52.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.829+0000 7fafaafaa700 1 -- 172.21.15.96:0/2235310991 wait complete. 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.829+0000 7fafaafaa700 1 Processor -- start 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.829+0000 7fafaafaa700 1 -- start start 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.829+0000 7fafaafaa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 0x7fafa4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.829+0000 7fafaafaa700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafa4105ab0 0x7fafa4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.829+0000 7fafaafaa700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa4198530 con 0x7fafa40ffab0 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.829+0000 7fafaafaa700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa4198670 con 0x7fafa4105ab0 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa8d46700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 0x7fafa4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa8d46700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 0x7fafa4197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55438/0 (socket says 172.21.15.96:55438) 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa8d46700 1 -- 172.21.15.96:0/534713711 learned_addr learned my addr 172.21.15.96:0/534713711 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa3fff700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafa4105ab0 0x7fafa4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa8d46700 1 -- 172.21.15.96:0/534713711 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafa4105ab0 msgr2=0x7fafa4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:52.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa8d46700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafa4105ab0 0x7fafa4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:52.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa8d46700 1 -- 172.21.15.96:0/534713711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf900095f0 con 0x7fafa40ffab0 2024-04-22T15:22:52.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa8d46700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 0x7fafa4197ab0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7faf9800b4b0 tx=0x7faf9800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:52.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa1ffb700 1 -- 172.21.15.96:0/534713711 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf98011800 con 0x7fafa40ffab0 2024-04-22T15:22:52.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa1ffb700 1 -- 172.21.15.96:0/534713711 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf98011e20 con 0x7fafa40ffab0 2024-04-22T15:22:52.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa41a0b20 con 0x7fafa40ffab0 2024-04-22T15:22:52.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.830+0000 7fafa1ffb700 1 -- 172.21.15.96:0/534713711 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf9800f2e0 con 0x7fafa40ffab0 2024-04-22T15:22:52.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.831+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa41a0f30 con 0x7fafa40ffab0 2024-04-22T15:22:52.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.832+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa41080d0 con 0x7fafa40ffab0 2024-04-22T15:22:52.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.832+0000 7fafa1ffb700 1 -- 172.21.15.96:0/534713711 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7faf98010380 con 0x7fafa40ffab0 2024-04-22T15:22:52.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.833+0000 7fafa1ffb700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7faf9406e940 0x7faf94070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:52.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.833+0000 7fafa1ffb700 1 -- 172.21.15.96:0/534713711 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7faf9808e3f0 con 0x7fafa40ffab0 2024-04-22T15:22:52.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.833+0000 7fafa3fff700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7faf9406e940 0x7faf94070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:52.837 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:52 smithi096 ceph-mon[26688]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:22:52.837 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:52 smithi096 ceph-mon[26688]: pgmap v108: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:52.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.833+0000 7fafa3fff700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7faf9406e940 0x7faf94070e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fafa40695f0 tx=0x7faf9001a040 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:52.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:52.842+0000 7fafa1ffb700 1 -- 172.21.15.96:0/534713711 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7faf98057700 con 0x7fafa40ffab0 2024-04-22T15:22:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:52 smithi196 ceph-mon[34717]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:22:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:52 smithi196 ceph-mon[34717]: pgmap v108: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:53.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.066+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fafa4069410 con 0x7faf9406e940 2024-04-22T15:22:53.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.067+0000 7fafa1ffb700 1 -- 172.21.15.96:0/534713711 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fafa4069410 con 0x7faf9406e940 2024-04-22T15:22:53.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.069+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7faf9406e940 msgr2=0x7faf94070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.069+0000 7fafaafaa700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7faf9406e940 0x7faf94070e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fafa40695f0 tx=0x7faf9001a040 comp rx=0 tx=0).stop 2024-04-22T15:22:53.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.070+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 msgr2=0x7fafa4197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.070+0000 7fafaafaa700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 0x7fafa4197ab0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7faf9800b4b0 tx=0x7faf9800b870 comp rx=0 tx=0).stop 2024-04-22T15:22:53.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.070+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 shutdown_connections 2024-04-22T15:22:53.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.070+0000 7fafaafaa700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7faf9406e940 0x7faf94070e00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.070+0000 7fafaafaa700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa40ffab0 0x7fafa4197ab0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.070+0000 7fafaafaa700 1 --2- 172.21.15.96:0/534713711 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafa4105ab0 0x7fafa4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.070+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 >> 172.21.15.96:0/534713711 conn(0x7fafa40fb5e0 msgr2=0x7fafa40fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:53.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.070+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 shutdown_connections 2024-04-22T15:22:53.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.070+0000 7fafaafaa700 1 -- 172.21.15.96:0/534713711 wait complete. 2024-04-22T15:22:53.083 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:22:53.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 -- 172.21.15.96:0/347661629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e60101db0 msgr2=0x7f0e60102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 --2- 172.21.15.96:0/347661629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e60101db0 0x7f0e60102190 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009910 tx=0x7f0e5c009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:53.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 -- 172.21.15.96:0/347661629 shutdown_connections 2024-04-22T15:22:53.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 --2- 172.21.15.96:0/347661629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e601026d0 0x7f0e6010abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 --2- 172.21.15.96:0/347661629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e60101db0 0x7f0e60102190 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 -- 172.21.15.96:0/347661629 >> 172.21.15.96:0/347661629 conn(0x7f0e600fb5e0 msgr2=0x7f0e600fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 -- 172.21.15.96:0/347661629 shutdown_connections 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 -- 172.21.15.96:0/347661629 wait complete. 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 Processor -- start 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.179+0000 7f0e6782a700 1 -- start start 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e6782a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e60101db0 0x7f0e6019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e6782a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e601026d0 0x7f0e6019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e6782a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6019d370 con 0x7f0e601026d0 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e6782a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e60196950 con 0x7f0e60101db0 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e64dc5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e601026d0 0x7f0e6019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:53.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e64dc5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e601026d0 0x7f0e6019cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55450/0 (socket says 172.21.15.96:55450) 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e64dc5700 1 -- 172.21.15.96:0/944441382 learned_addr learned my addr 172.21.15.96:0/944441382 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e655c6700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e60101db0 0x7f0e6019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e64dc5700 1 -- 172.21.15.96:0/944441382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e60101db0 msgr2=0x7f0e6019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e64dc5700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e60101db0 0x7f0e6019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e64dc5700 1 -- 172.21.15.96:0/944441382 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c0095f0 con 0x7f0e601026d0 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e655c6700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e60101db0 0x7f0e6019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.180+0000 7f0e64dc5700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e601026d0 0x7f0e6019cd50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000c8f0 tx=0x7f0e5000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.181+0000 7f0e567fc700 1 -- 172.21.15.96:0/944441382 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e50007ad0 con 0x7f0e601026d0 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.181+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e60196c30 con 0x7f0e601026d0 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.181+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e60197180 con 0x7f0e601026d0 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.181+0000 7f0e567fc700 1 -- 172.21.15.96:0/944441382 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e5000ce90 con 0x7f0e601026d0 2024-04-22T15:22:53.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.181+0000 7f0e567fc700 1 -- 172.21.15.96:0/944441382 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e50018630 con 0x7f0e601026d0 2024-04-22T15:22:53.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.182+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e60105460 con 0x7f0e601026d0 2024-04-22T15:22:53.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.182+0000 7f0e567fc700 1 -- 172.21.15.96:0/944441382 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f0e50007c30 con 0x7f0e601026d0 2024-04-22T15:22:53.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.183+0000 7f0e567fc700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0e4c06e940 0x7f0e4c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:53.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.183+0000 7f0e567fc700 1 -- 172.21.15.96:0/944441382 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f0e5008e8d0 con 0x7f0e601026d0 2024-04-22T15:22:53.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.183+0000 7f0e655c6700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0e4c06e940 0x7f0e4c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:53.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.184+0000 7f0e655c6700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0e4c06e940 0x7f0e4c070e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c006010 tx=0x7f0e5c01a040 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:53.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.188+0000 7f0e567fc700 1 -- 172.21.15.96:0/944441382 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f0e5005b690 con 0x7f0e601026d0 2024-04-22T15:22:53.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.419+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e60197e80 con 0x7f0e4c06e940 2024-04-22T15:22:53.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.420+0000 7f0e567fc700 1 -- 172.21.15.96:0/944441382 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f0e60197e80 con 0x7f0e4c06e940 2024-04-22T15:22:53.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0e4c06e940 msgr2=0x7f0e4c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0e4c06e940 0x7f0e4c070e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c006010 tx=0x7f0e5c01a040 comp rx=0 tx=0).stop 2024-04-22T15:22:53.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e601026d0 msgr2=0x7f0e6019cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e601026d0 0x7f0e6019cd50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000c8f0 tx=0x7f0e5000ccb0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 shutdown_connections 2024-04-22T15:22:53.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f0e4c06e940 0x7f0e4c070e00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e60101db0 0x7f0e6019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 --2- 172.21.15.96:0/944441382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e601026d0 0x7f0e6019cd50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 >> 172.21.15.96:0/944441382 conn(0x7f0e600fb5e0 msgr2=0x7f0e60103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:53.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 shutdown_connections 2024-04-22T15:22:53.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.422+0000 7f0e6782a700 1 -- 172.21.15.96:0/944441382 wait complete. 2024-04-22T15:22:53.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.531+0000 7f2b38b9e700 1 -- 172.21.15.96:0/2707572185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b34101dc0 msgr2=0x7f2b341021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.531+0000 7f2b38b9e700 1 --2- 172.21.15.96:0/2707572185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b34101dc0 0x7f2b341021a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009940 tx=0x7f2b24009c50 comp rx=0 tx=0).stop 2024-04-22T15:22:53.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.531+0000 7f2b38b9e700 1 -- 172.21.15.96:0/2707572185 shutdown_connections 2024-04-22T15:22:53.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.531+0000 7f2b38b9e700 1 --2- 172.21.15.96:0/2707572185 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b341026e0 0x7f2b3410abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.531+0000 7f2b38b9e700 1 --2- 172.21.15.96:0/2707572185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b34101dc0 0x7f2b341021a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.531+0000 7f2b38b9e700 1 -- 172.21.15.96:0/2707572185 >> 172.21.15.96:0/2707572185 conn(0x7f2b34076740 msgr2=0x7f2b340fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:53.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.531+0000 7f2b38b9e700 1 -- 172.21.15.96:0/2707572185 shutdown_connections 2024-04-22T15:22:53.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.531+0000 7f2b38b9e700 1 -- 172.21.15.96:0/2707572185 wait complete. 2024-04-22T15:22:53.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.532+0000 7f2b38b9e700 1 Processor -- start 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.532+0000 7f2b38b9e700 1 -- start start 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.532+0000 7f2b38b9e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b34101dc0 0x7f2b3419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.532+0000 7f2b38b9e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b341026e0 0x7f2b3419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.532+0000 7f2b38b9e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3419d380 con 0x7f2b341026e0 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.532+0000 7f2b38b9e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b34196960 con 0x7f2b34101dc0 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.532+0000 7f2b31d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b341026e0 0x7f2b3419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b31d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b341026e0 0x7f2b3419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55456/0 (socket says 172.21.15.96:55456) 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b31d9b700 1 -- 172.21.15.96:0/1363138019 learned_addr learned my addr 172.21.15.96:0/1363138019 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b3259c700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b34101dc0 0x7f2b3419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b31d9b700 1 -- 172.21.15.96:0/1363138019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b34101dc0 msgr2=0x7f2b3419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b31d9b700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b34101dc0 0x7f2b3419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b31d9b700 1 -- 172.21.15.96:0/1363138019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b240095f0 con 0x7f2b341026e0 2024-04-22T15:22:53.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b31d9b700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b341026e0 0x7f2b3419cd60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d900 tx=0x7f2b2800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:53.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b237fe700 1 -- 172.21.15.96:0/1363138019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b28009900 con 0x7f2b341026e0 2024-04-22T15:22:53.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b237fe700 1 -- 172.21.15.96:0/1363138019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b28010460 con 0x7f2b341026e0 2024-04-22T15:22:53.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b237fe700 1 -- 172.21.15.96:0/1363138019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b2800f5f0 con 0x7f2b341026e0 2024-04-22T15:22:53.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b34196c40 con 0x7f2b341026e0 2024-04-22T15:22:53.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.533+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b34197190 con 0x7f2b341026e0 2024-04-22T15:22:53.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.534+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b341054e0 con 0x7f2b341026e0 2024-04-22T15:22:53.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.535+0000 7f2b237fe700 1 -- 172.21.15.96:0/1363138019 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f2b2800f870 con 0x7f2b341026e0 2024-04-22T15:22:53.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.535+0000 7f2b237fe700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f2b1c06e950 0x7f2b1c070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:53.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.535+0000 7f2b237fe700 1 -- 172.21.15.96:0/1363138019 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f2b2808e460 con 0x7f2b341026e0 2024-04-22T15:22:53.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.535+0000 7f2b3259c700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f2b1c06e950 0x7f2b1c070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:53.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.536+0000 7f2b3259c700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f2b1c06e950 0x7f2b1c070e10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b24006010 tx=0x7f2b24005ad0 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:53.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.540+0000 7f2b237fe700 1 -- 172.21.15.96:0/1363138019 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f2b2805b220 con 0x7f2b341026e0 2024-04-22T15:22:53.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.756+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b34197ec0 con 0x7f2b1c06e950 2024-04-22T15:22:53.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.766+0000 7f2b237fe700 1 -- 172.21.15.96:0/1363138019 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f2b34197ec0 con 0x7f2b1c06e950 2024-04-22T15:22:53.770 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:22:53.770 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (96s) 66s ago 12m 17.0M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:22:53.770 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (13m) 66s ago 13m 8182k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:22:53.770 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (12m) 2m ago 12m 7864k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:22:53.770 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (13m) 66s ago 13m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:22:53.770 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (11m) 2m ago 11m 6660k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:22:53.770 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (72s) 66s ago 12m 55.1M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:22:53.770 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (7m) 66s ago 7m 156M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (7m) 66s ago 7m 89.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (7m) 2m ago 7m 26.4M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (7m) 2m ago 7m 25.8M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (3m) 66s ago 14m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (2m) 2m ago 11m 449M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (14m) 66s ago 14m 57.7M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 3f2d97a8db3a 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (11m) 2m ago 11m 44.9M 2048M 18.2.2-1237-g646b8c7d 9900ed1a6b81 79938ec69af6 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (2m) 66s ago 13m 16.7M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (2m) 2m ago 11m 3602k - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (10m) 66s ago 10m 209M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (10m) 66s ago 10m 215M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (10m) 66s ago 10m 181M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:22:53.771 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (9m) 2m ago 9m 231M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:22:53.772 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (9m) 2m ago 9m 208M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:22:53.772 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (8m) 2m ago 8m 185M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:22:53.772 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (114s) 66s ago 12m 58.6M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f2b1c06e950 msgr2=0x7f2b1c070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f2b1c06e950 0x7f2b1c070e10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b24006010 tx=0x7f2b24005ad0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b341026e0 msgr2=0x7f2b3419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b341026e0 0x7f2b3419cd60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d900 tx=0x7f2b2800dc10 comp rx=0 tx=0).stop 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 shutdown_connections 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f2b1c06e950 0x7f2b1c070e10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b34101dc0 0x7f2b3419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 --2- 172.21.15.96:0/1363138019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b341026e0 0x7f2b3419cd60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 >> 172.21.15.96:0/1363138019 conn(0x7f2b34076740 msgr2=0x7f2b341030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 shutdown_connections 2024-04-22T15:22:53.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.769+0000 7f2b38b9e700 1 -- 172.21.15.96:0/1363138019 wait complete. 2024-04-22T15:22:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 -- 172.21.15.96:0/2944060066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 msgr2=0x7f835c10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 --2- 172.21.15.96:0/2944060066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 0x7f835c10ab70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f834c009910 tx=0x7f834c009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 -- 172.21.15.96:0/2944060066 shutdown_connections 2024-04-22T15:22:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 --2- 172.21.15.96:0/2944060066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 0x7f835c10ab70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 --2- 172.21.15.96:0/2944060066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f835c101d60 0x7f835c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 -- 172.21.15.96:0/2944060066 >> 172.21.15.96:0/2944060066 conn(0x7f835c076740 msgr2=0x7f835c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 -- 172.21.15.96:0/2944060066 shutdown_connections 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 -- 172.21.15.96:0/2944060066 wait complete. 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 Processor -- start 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.876+0000 7f83647b3700 1 -- start start 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f83647b3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f835c101d60 0x7f835c198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f83647b3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 0x7f835c199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f83647b3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c199a20 con 0x7f835c102680 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f83647b3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c19d6f0 con 0x7f835c101d60 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f8361d4e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 0x7f835c199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f8361d4e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 0x7f835c199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55488/0 (socket says 172.21.15.96:55488) 2024-04-22T15:22:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f8361d4e700 1 -- 172.21.15.96:0/3621985036 learned_addr learned my addr 172.21.15.96:0/3621985036 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f836254f700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f835c101d60 0x7f835c198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f8361d4e700 1 -- 172.21.15.96:0/3621985036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f835c101d60 msgr2=0x7f835c198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f8361d4e700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f835c101d60 0x7f835c198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f8361d4e700 1 -- 172.21.15.96:0/3621985036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c0095f0 con 0x7f835c102680 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f8361d4e700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 0x7f835c199340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f834c00bb40 tx=0x7f834c00bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f83537fe700 1 -- 172.21.15.96:0/3621985036 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f834c01d070 con 0x7f835c102680 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835c19d970 con 0x7f835c102680 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f83537fe700 1 -- 172.21.15.96:0/3621985036 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f834c003d10 con 0x7f835c102680 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.877+0000 7f83537fe700 1 -- 172.21.15.96:0/3621985036 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f834c00fde0 con 0x7f835c102680 2024-04-22T15:22:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.878+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835c19de60 con 0x7f835c102680 2024-04-22T15:22:53.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.878+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835c10a6d0 con 0x7f835c102680 2024-04-22T15:22:53.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.880+0000 7f83537fe700 1 -- 172.21.15.96:0/3621985036 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f834c003850 con 0x7f835c102680 2024-04-22T15:22:53.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.880+0000 7f83537fe700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f834806e940 0x7f8348070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:53.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.880+0000 7f83537fe700 1 -- 172.21.15.96:0/3621985036 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f834c08fc00 con 0x7f835c102680 2024-04-22T15:22:53.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.880+0000 7f836254f700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f834806e940 0x7f8348070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:53.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.880+0000 7f836254f700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f834806e940 0x7f8348070e00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8358009a10 tx=0x7f8358009190 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:53.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:53.889+0000 7f83537fe700 1 -- 172.21.15.96:0/3621985036 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f834c095020 con 0x7f835c102680 2024-04-22T15:22:54.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.189+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f835c19a110 con 0x7f835c102680 2024-04-22T15:22:54.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.189+0000 7f83537fe700 1 -- 172.21.15.96:0/3621985036 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7f834c058e90 con 0x7f835c102680 2024-04-22T15:22:54.193 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:22:54.194 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:22:54.194 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2 2024-04-22T15:22:54.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:22:54.194 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:22:54.194 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:22:54.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:22:54.195 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:22:54.195 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:22:54.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:22:54.195 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:22:54.195 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:22:54.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:22:54.196 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:22:54.196 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 12, 2024-04-22T15:22:54.196 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:22:54.196 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:22:54.197 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:22:54.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.191+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f834806e940 msgr2=0x7f8348070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.191+0000 7f83647b3700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f834806e940 0x7f8348070e00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8358009a10 tx=0x7f8358009190 comp rx=0 tx=0).stop 2024-04-22T15:22:54.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.191+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 msgr2=0x7f835c199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.191+0000 7f83647b3700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 0x7f835c199340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f834c00bb40 tx=0x7f834c00bb70 comp rx=0 tx=0).stop 2024-04-22T15:22:54.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.191+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 shutdown_connections 2024-04-22T15:22:54.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.191+0000 7f83647b3700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f834806e940 0x7f8348070e00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.191+0000 7f83647b3700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f835c101d60 0x7f835c198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.191+0000 7f83647b3700 1 --2- 172.21.15.96:0/3621985036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f835c102680 0x7f835c199340 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.191+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 >> 172.21.15.96:0/3621985036 conn(0x7f835c076740 msgr2=0x7f835c101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:54.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.192+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 shutdown_connections 2024-04-22T15:22:54.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.192+0000 7f83647b3700 1 -- 172.21.15.96:0/3621985036 wait complete. 2024-04-22T15:22:54.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 -- 172.21.15.96:0/2399799075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 msgr2=0x7f0300103f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 --2- 172.21.15.96:0/2399799075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 0x7f0300103f20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009910 tx=0x7f02f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:54.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 -- 172.21.15.96:0/2399799075 shutdown_connections 2024-04-22T15:22:54.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 --2- 172.21.15.96:0/2399799075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 0x7f0300103f20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 --2- 172.21.15.96:0/2399799075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03000ff520 0x7f03000ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 -- 172.21.15.96:0/2399799075 >> 172.21.15.96:0/2399799075 conn(0x7f0300075f00 msgr2=0x7f03000fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:54.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 -- 172.21.15.96:0/2399799075 shutdown_connections 2024-04-22T15:22:54.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 -- 172.21.15.96:0/2399799075 wait complete. 2024-04-22T15:22:54.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 Processor -- start 2024-04-22T15:22:54.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.309+0000 7f030875a700 1 -- start start 2024-04-22T15:22:54.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f030875a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03000ff520 0x7f030019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:54.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f030875a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 0x7f030019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:54.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f030875a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030019d3b0 con 0x7f03000ffed0 2024-04-22T15:22:54.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f030875a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001969e0 con 0x7f03000ff520 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f0305cf5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 0x7f030019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f0305cf5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 0x7f030019cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55510/0 (socket says 172.21.15.96:55510) 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f0305cf5700 1 -- 172.21.15.96:0/2151567609 learned_addr learned my addr 172.21.15.96:0/2151567609 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f03064f6700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03000ff520 0x7f030019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f0305cf5700 1 -- 172.21.15.96:0/2151567609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03000ff520 msgr2=0x7f030019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f0305cf5700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03000ff520 0x7f030019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f0305cf5700 1 -- 172.21.15.96:0/2151567609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f00095f0 con 0x7f03000ffed0 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f03064f6700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03000ff520 0x7f030019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.310+0000 7f0305cf5700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 0x7f030019cd90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f02f000bd70 tx=0x7f02f000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.311+0000 7f02f77fe700 1 -- 172.21.15.96:0/2151567609 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02f001d070 con 0x7f03000ffed0 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.311+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0300196c60 con 0x7f03000ffed0 2024-04-22T15:22:54.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.311+0000 7f02f77fe700 1 -- 172.21.15.96:0/2151567609 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02f00056e0 con 0x7f03000ffed0 2024-04-22T15:22:54.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.311+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0300197150 con 0x7f03000ffed0 2024-04-22T15:22:54.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.312+0000 7f02f77fe700 1 -- 172.21.15.96:0/2151567609 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02f0017400 con 0x7f03000ffed0 2024-04-22T15:22:54.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.312+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0300107920 con 0x7f03000ffed0 2024-04-22T15:22:54.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.313+0000 7f02f77fe700 1 -- 172.21.15.96:0/2151567609 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f02f0017620 con 0x7f03000ffed0 2024-04-22T15:22:54.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.313+0000 7f02f77fe700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f02ec06e940 0x7f02ec070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:54.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.313+0000 7f02f77fe700 1 -- 172.21.15.96:0/2151567609 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f02f008fc10 con 0x7f03000ffed0 2024-04-22T15:22:54.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.313+0000 7f03064f6700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f02ec06e940 0x7f02ec070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:54.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.314+0000 7f03064f6700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f02ec06e940 0x7f02ec070e00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f02fc005950 tx=0x7f02fc0058e0 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:54.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.322+0000 7f02f77fe700 1 -- 172.21.15.96:0/2151567609 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f02f005cb10 con 0x7f03000ffed0 2024-04-22T15:22:54.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.572+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0300197e50 con 0x7f03000ffed0 2024-04-22T15:22:54.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.573+0000 7f02f77fe700 1 -- 172.21.15.96:0/2151567609 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1869 (secure 0 0 0) 0x7f02f005c6a0 con 0x7f03000ffed0 2024-04-22T15:22:54.576 INFO:teuthology.orchestra.run.smithi096.stdout:e20 2024-04-22T15:22:54.576 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:22:54.577 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:22:54.577 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:22:54.577 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:22:54.577 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:22:54.577 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:22:54.577 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 20 2024-04-22T15:22:54.577 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:22:54.577 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:22:54.577 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:22:54.578 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:22:54.578 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:22:54.578 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:22:54.578 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:22:54.578 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:22:54.578 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:22:54.578 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:22:54.578 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:22:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:22:54.580 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:22:54.580 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:22:54.580 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:22:54.580 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:22:54.580 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:22:54.580 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:22:54.580 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:22:54.580 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:22:54.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f02ec06e940 msgr2=0x7f02ec070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f02ec06e940 0x7f02ec070e00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f02fc005950 tx=0x7f02fc0058e0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 msgr2=0x7f030019cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 0x7f030019cd90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f02f000bd70 tx=0x7f02f000be50 comp rx=0 tx=0).stop 2024-04-22T15:22:54.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 shutdown_connections 2024-04-22T15:22:54.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f02ec06e940 0x7f02ec070e00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03000ff520 0x7f030019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 --2- 172.21.15.96:0/2151567609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03000ffed0 0x7f030019cd90 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 >> 172.21.15.96:0/2151567609 conn(0x7f0300075f00 msgr2=0x7f0300102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:54.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 shutdown_connections 2024-04-22T15:22:54.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.576+0000 7f030875a700 1 -- 172.21.15.96:0/2151567609 wait complete. 2024-04-22T15:22:54.583 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:22:54.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.694+0000 7f1b4ba40700 1 -- 172.21.15.96:0/1119892341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 msgr2=0x7f1b440fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.694+0000 7f1b4ba40700 1 --2- 172.21.15.96:0/1119892341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 0x7f1b440fff30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1b34009910 tx=0x7f1b34009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:54.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.694+0000 7f1b4ba40700 1 -- 172.21.15.96:0/1119892341 shutdown_connections 2024-04-22T15:22:54.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.694+0000 7f1b4ba40700 1 --2- 172.21.15.96:0/1119892341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 0x7f1b440fff30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.694+0000 7f1b4ba40700 1 --2- 172.21.15.96:0/1119892341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b44105ab0 0x7f1b44105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.694+0000 7f1b4ba40700 1 -- 172.21.15.96:0/1119892341 >> 172.21.15.96:0/1119892341 conn(0x7f1b440fb5e0 msgr2=0x7f1b440fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:54.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.695+0000 7f1b4ba40700 1 -- 172.21.15.96:0/1119892341 shutdown_connections 2024-04-22T15:22:54.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.695+0000 7f1b4ba40700 1 -- 172.21.15.96:0/1119892341 wait complete. 2024-04-22T15:22:54.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.695+0000 7f1b4ba40700 1 Processor -- start 2024-04-22T15:22:54.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.695+0000 7f1b4ba40700 1 -- start start 2024-04-22T15:22:54.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b4ba40700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 0x7f1b44197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:54.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b4ba40700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b44105ab0 0x7f1b44197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:54.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b4ba40700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b44198530 con 0x7f1b440ffab0 2024-04-22T15:22:54.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b4ba40700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b44198670 con 0x7f1b44105ab0 2024-04-22T15:22:54.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b497dc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 0x7f1b44197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:54.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b497dc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 0x7f1b44197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55514/0 (socket says 172.21.15.96:55514) 2024-04-22T15:22:54.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b497dc700 1 -- 172.21.15.96:0/106903432 learned_addr learned my addr 172.21.15.96:0/106903432 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:54.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b48fdb700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b44105ab0 0x7f1b44197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:54.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b497dc700 1 -- 172.21.15.96:0/106903432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b44105ab0 msgr2=0x7f1b44197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b497dc700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b44105ab0 0x7f1b44197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b497dc700 1 -- 172.21.15.96:0/106903432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b340095f0 con 0x7f1b440ffab0 2024-04-22T15:22:54.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b48fdb700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b44105ab0 0x7f1b44197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:22:54.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.696+0000 7f1b497dc700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 0x7f1b44197ab0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000b4b0 tx=0x7f1b4000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:54.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.697+0000 7f1b3a7fc700 1 -- 172.21.15.96:0/106903432 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b40011800 con 0x7f1b440ffab0 2024-04-22T15:22:54.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.697+0000 7f1b3a7fc700 1 -- 172.21.15.96:0/106903432 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b40011e20 con 0x7f1b440ffab0 2024-04-22T15:22:54.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.697+0000 7f1b3a7fc700 1 -- 172.21.15.96:0/106903432 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b4000f2e0 con 0x7f1b440ffab0 2024-04-22T15:22:54.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.697+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b441a0b20 con 0x7f1b440ffab0 2024-04-22T15:22:54.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.697+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b441a0e80 con 0x7f1b440ffab0 2024-04-22T15:22:54.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.698+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b441080d0 con 0x7f1b440ffab0 2024-04-22T15:22:54.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.699+0000 7f1b3a7fc700 1 -- 172.21.15.96:0/106903432 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f1b40011960 con 0x7f1b440ffab0 2024-04-22T15:22:54.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.699+0000 7f1b3a7fc700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f1b3006e740 0x7f1b30070c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:54.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.699+0000 7f1b3a7fc700 1 -- 172.21.15.96:0/106903432 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1b4008e1c0 con 0x7f1b440ffab0 2024-04-22T15:22:54.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.699+0000 7f1b48fdb700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f1b3006e740 0x7f1b30070c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:54.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.700+0000 7f1b48fdb700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f1b3006e740 0x7f1b30070c00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1b34005f50 tx=0x7f1b34005dc0 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:54.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.704+0000 7f1b3a7fc700 1 -- 172.21.15.96:0/106903432 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f1b40057450 con 0x7f1b440ffab0 2024-04-22T15:22:54.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.932+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 --> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b44069410 con 0x7f1b3006e740 2024-04-22T15:22:54.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.933+0000 7f1b3a7fc700 1 -- 172.21.15.96:0/106903432 <== mgr.14678 v2:172.21.15.96:6800/3666812526 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f1b44069410 con 0x7f1b3006e740 2024-04-22T15:22:54.937 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:22:54.937 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:22:54.937 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:22:54.937 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:22:54.937 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:22:54.937 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:22:54.937 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:22:54.937 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "2/23 daemons upgraded", 2024-04-22T15:22:54.937 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mon daemons", 2024-04-22T15:22:54.938 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:22:54.938 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:22:54.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f1b3006e740 msgr2=0x7f1b30070c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f1b3006e740 0x7f1b30070c00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1b34005f50 tx=0x7f1b34005dc0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 msgr2=0x7f1b44197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 0x7f1b44197ab0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000b4b0 tx=0x7f1b4000b870 comp rx=0 tx=0).stop 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 shutdown_connections 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f1b3006e740 0x7f1b30070c00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b440ffab0 0x7f1b44197ab0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 --2- 172.21.15.96:0/106903432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b44105ab0 0x7f1b44197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 >> 172.21.15.96:0/106903432 conn(0x7f1b440fb5e0 msgr2=0x7f1b440fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 shutdown_connections 2024-04-22T15:22:54.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:54.935+0000 7f1b4ba40700 1 -- 172.21.15.96:0/106903432 wait complete. 2024-04-22T15:22:55.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 -- 172.21.15.96:0/1189289824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c0ffac0 msgr2=0x7f043c0fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:55.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 --2- 172.21.15.96:0/1189289824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c0ffac0 0x7f043c0fff40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f042c009910 tx=0x7f042c009c20 comp rx=0 tx=0).stop 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 -- 172.21.15.96:0/1189289824 shutdown_connections 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 --2- 172.21.15.96:0/1189289824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c0ffac0 0x7f043c0fff40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 --2- 172.21.15.96:0/1189289824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f043c105ac0 0x7f043c105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 -- 172.21.15.96:0/1189289824 >> 172.21.15.96:0/1189289824 conn(0x7f043c0fb5d0 msgr2=0x7f043c0fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 -- 172.21.15.96:0/1189289824 shutdown_connections 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 -- 172.21.15.96:0/1189289824 wait complete. 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 Processor -- start 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.043+0000 7f0444347700 1 -- start start 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f0444347700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f043c0ffac0 0x7f043c19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f0444347700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c105ac0 0x7f043c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f0444347700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c19d460 con 0x7f043c105ac0 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f0444347700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c196a40 con 0x7f043c0ffac0 2024-04-22T15:22:55.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04418e2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c105ac0 0x7f043c19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04418e2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c105ac0 0x7f043c19ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55536/0 (socket says 172.21.15.96:55536) 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04418e2700 1 -- 172.21.15.96:0/2569979376 learned_addr learned my addr 172.21.15.96:0/2569979376 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04418e2700 1 -- 172.21.15.96:0/2569979376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f043c0ffac0 msgr2=0x7f043c19c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04418e2700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f043c0ffac0 0x7f043c19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04418e2700 1 -- 172.21.15.96:0/2569979376 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c0095f0 con 0x7f043c105ac0 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04418e2700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c105ac0 0x7f043c19ce40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f042c006010 tx=0x7f042c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04337fe700 1 -- 172.21.15.96:0/2569979376 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f042c01d070 con 0x7f043c105ac0 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04337fe700 1 -- 172.21.15.96:0/2569979376 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f042c00f460 con 0x7f043c105ac0 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c196cc0 con 0x7f043c105ac0 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f04337fe700 1 -- 172.21.15.96:0/2569979376 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f042c017630 con 0x7f043c105ac0 2024-04-22T15:22:55.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.044+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c1971b0 con 0x7f043c105ac0 2024-04-22T15:22:55.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.045+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043c1090f0 con 0x7f043c105ac0 2024-04-22T15:22:55.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.046+0000 7f04337fe700 1 -- 172.21.15.96:0/2569979376 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) v1 ==== 92606+0+0 (secure 0 0 0) 0x7f042c00f5d0 con 0x7f043c105ac0 2024-04-22T15:22:55.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.046+0000 7f04337fe700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f042806e940 0x7f0428070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:22:55.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.046+0000 7f04337fe700 1 -- 172.21.15.96:0/2569979376 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f042c08fde0 con 0x7f043c105ac0 2024-04-22T15:22:55.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.046+0000 7f04420e3700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f042806e940 0x7f0428070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:22:55.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.047+0000 7f04420e3700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f042806e940 0x7f0428070e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f04380097b0 tx=0x7f0438006d20 comp rx=0 tx=0).ready entity=mgr.14678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:22:55.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.051+0000 7f04337fe700 1 -- 172.21.15.96:0/2569979376 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188584 (secure 0 0 0) 0x7f042c05cc60 con 0x7f043c105ac0 2024-04-22T15:22:55.140 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='client.14876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='client.14884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: pgmap v109: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/3621985036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:55.141 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:54 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2151567609' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:22:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='client.14876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='client.14884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: pgmap v109: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/3621985036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:22:55.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:54 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2151567609' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:22:55.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.340+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f043c197f00 con 0x7f043c105ac0 2024-04-22T15:22:55.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.340+0000 7f04337fe700 1 -- 172.21.15.96:0/2569979376 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f042c05c7f0 con 0x7f043c105ac0 2024-04-22T15:22:55.344 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:22:55.344 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:22:55.344 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:22:55.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.342+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f042806e940 msgr2=0x7f0428070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:55.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.342+0000 7f0444347700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f042806e940 0x7f0428070e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f04380097b0 tx=0x7f0438006d20 comp rx=0 tx=0).stop 2024-04-22T15:22:55.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.342+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c105ac0 msgr2=0x7f043c19ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:22:55.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.342+0000 7f0444347700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c105ac0 0x7f043c19ce40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f042c006010 tx=0x7f042c00be20 comp rx=0 tx=0).stop 2024-04-22T15:22:55.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.342+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 shutdown_connections 2024-04-22T15:22:55.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.342+0000 7f0444347700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:6800/3666812526,v1:172.21.15.96:6801/3666812526] conn(0x7f042806e940 0x7f0428070e00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:55.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.342+0000 7f0444347700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f043c0ffac0 0x7f043c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:55.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.342+0000 7f0444347700 1 --2- 172.21.15.96:0/2569979376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f043c105ac0 0x7f043c19ce40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:22:55.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.342+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 >> 172.21.15.96:0/2569979376 conn(0x7f043c0fb5d0 msgr2=0x7f043c104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:22:55.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.343+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 shutdown_connections 2024-04-22T15:22:55.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:22:55.343+0000 7f0444347700 1 -- 172.21.15.96:0/2569979376 wait complete. 2024-04-22T15:22:56.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:55 smithi096 ceph-mon[26688]: Upgrade: Target is version 19.0.0-3223-g5883d494 (squid) 2024-04-22T15:22:56.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:55 smithi096 ceph-mon[26688]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb'] 2024-04-22T15:22:56.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:55 smithi096 ceph-mon[26688]: Upgrade: Setting container_image for all mgr 2024-04-22T15:22:56.110 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:55 smithi096 ceph-mon[26688]: from='client.? 172.21.15.96:0/2569979376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:22:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:55 smithi196 ceph-mon[34717]: Upgrade: Target is version 19.0.0-3223-g5883d494 (squid) 2024-04-22T15:22:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:55 smithi196 ceph-mon[34717]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb'] 2024-04-22T15:22:56.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:55 smithi196 ceph-mon[34717]: Upgrade: Setting container_image for all mgr 2024-04-22T15:22:56.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:55 smithi196 ceph-mon[34717]: from='client.? 172.21.15.96:0/2569979376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:22:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:56 smithi096 ceph-mon[26688]: from='client.14896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:56 smithi096 ceph-mon[26688]: pgmap v110: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:56 smithi096 ceph-mon[26688]: Upgrade: Updating mon.smithi096 2024-04-22T15:22:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:56 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:56 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:22:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:56 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:22:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:56 smithi096 ceph-mon[26688]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:22:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:56 smithi096 ceph-mon[26688]: Deploying daemon mon.smithi096 on smithi096 2024-04-22T15:22:57.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:56 smithi196 ceph-mon[34717]: from='client.14896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:22:57.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:56 smithi196 ceph-mon[34717]: pgmap v110: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:22:57.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:56 smithi196 ceph-mon[34717]: Upgrade: Updating mon.smithi096 2024-04-22T15:22:57.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:56 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' 2024-04-22T15:22:57.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:56 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:22:57.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:56 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:22:57.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:56 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:22:57.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:22:56 smithi196 ceph-mon[34717]: Deploying daemon mon.smithi096 on smithi096 2024-04-22T15:22:58.048 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:57 smithi096 systemd[1]: Stopping Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:22:58.357 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:58 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[26684]: 2024-04-22T15:22:58.044+0000 7f9fff897700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi096 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:22:58.357 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:58 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[26684]: 2024-04-22T15:22:58.044+0000 7f9fff897700 -1 mon.smithi096@0(leader) e2 *** Got Signal Terminated *** 2024-04-22T15:22:58.358 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:58 smithi096 podman[112265]: 2024-04-22 15:22:58.127248097 +0000 UTC m=+0.236307189 container died 3f2d97a8db3a149277b16be4539bcd1cb47c5da480083f9827a73733e873f619 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=reef-646b8c7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:22:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:58 smithi096 podman[112265]: 2024-04-22 15:22:58.662222345 +0000 UTC m=+0.771281446 container cleanup 3f2d97a8db3a149277b16be4539bcd1cb47c5da480083f9827a73733e873f619 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, RELEASE=reef-646b8c7, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T15:22:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:58 smithi096 bash[112265]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096 2024-04-22T15:22:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:58 smithi096 podman[112290]: 2024-04-22 15:22:58.871966445 +0000 UTC m=+0.742590445 container remove 3f2d97a8db3a149277b16be4539bcd1cb47c5da480083f9827a73733e873f619 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-646b8c7) 2024-04-22T15:22:59.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096.service: Deactivated successfully. 2024-04-22T15:22:59.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 systemd[1]: Stopped Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:22:59.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096.service: Consumed 13.915s CPU time. 2024-04-22T15:22:59.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 systemd[1]: Starting Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:22:59.878 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 podman[112424]: 2024-04-22 15:22:59.41814112 +0000 UTC m=+0.018429606 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:22:59.878 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 podman[112424]: 2024-04-22 15:22:59.598116137 +0000 UTC m=+0.198404613 container create cc6f8ad67833cb73f09be75789545f37f7b11913b0dfd53503ede9b300676d3d (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD) 2024-04-22T15:22:59.879 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 podman[112424]: 2024-04-22 15:22:59.876141992 +0000 UTC m=+0.476430463 container init cc6f8ad67833cb73f09be75789545f37f7b11913b0dfd53503ede9b300676d3d (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:23:00.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 podman[112424]: 2024-04-22 15:22:59.879305164 +0000 UTC m=+0.479593637 container start cc6f8ad67833cb73f09be75789545f37f7b11913b0dfd53503ede9b300676d3d (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD) 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev), process ceph-mon, pid 2 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: pidfile_write: ignore empty --pid-file 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: load: jerasure load: lrc 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: RocksDB version: 7.9.2 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Git sha 0 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Compile date 2024-04-22 12:55:29 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: DB SUMMARY 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: DB Session ID: KBH06OL7TG4VR8SMQ8G1 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: CURRENT file: CURRENT 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1498 Bytes 2024-04-22T15:23:00.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi096/store.db dir, Total Num: 1, files: 000032.sst 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi096/store.db: 000030.log size: 2271117 ; 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.error_if_exists: 0 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.create_if_missing: 0 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.paranoid_checks: 1 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.env: 0x56161395ccc0 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.info_log: 0x561614d7daa0 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.statistics: (nil) 2024-04-22T15:23:00.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.use_fsync: 0 2024-04-22T15:23:00.239 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_log_file_size: 0 2024-04-22T15:23:00.239 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T15:23:00.239 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T15:23:00.239 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T15:23:00.239 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.allow_fallocate: 1 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.use_direct_reads: 0 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.db_log_dir: 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.wal_dir: 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T15:23:00.240 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.write_buffer_manager: 0x561614d110e0 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T15:23:00.241 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.unordered_write: 0 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.row_cache: None 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.wal_filter: None 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.two_write_queues: 0 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.wal_compression: 0 2024-04-22T15:23:00.242 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.atomic_flush: 0 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.log_readahead_size: 0 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_background_jobs: 2 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_background_compactions: -1 2024-04-22T15:23:00.243 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_subcompactions: 1 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_open_files: -1 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T15:23:00.244 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_background_flushes: -1 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Compression algorithms supported: 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: kZSTD supported: 0 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: kXpressCompression supported: 0 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: kZlibCompression supported: 1 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: kSnappyCompression supported: 1 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: kLZ4Compression supported: 1 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: kBZip2Compression supported: 0 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T15:23:00.245 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T15:23:00.246 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi096/store.db/MANIFEST-000015 2024-04-22T15:23:00.246 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T15:23:00.246 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T15:23:00.246 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.merge_operator: 2024-04-22T15:23:00.246 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_filter: None 2024-04-22T15:23:00.246 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T15:23:00.246 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T15:23:00.246 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561614d7dc80) 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: cache_index_and_filter_blocks: 1 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: pin_top_level_index_and_filter: 1 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: index_type: 0 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: data_block_index_type: 0 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: index_shortening: 1 2024-04-22T15:23:00.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: checksum: 4 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: no_block_cache: 0 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_cache: 0x561614d431f0 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_cache_name: BinnedLRUCache 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_cache_options: 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: capacity : 536870912 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: num_shard_bits : 4 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: strict_capacity_limit : 0 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: high_pri_pool_ratio: 0.000 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_cache_compressed: (nil) 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: persistent_cache: (nil) 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_size: 4096 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_size_deviation: 10 2024-04-22T15:23:00.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_restart_interval: 16 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: index_block_restart_interval: 1 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: metadata_block_size: 4096 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: partition_filters: 0 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: use_delta_encoding: 1 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: filter_policy: bloomfilter 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: whole_key_filtering: 1 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: verify_compression: 0 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: read_amp_bytes_per_bit: 0 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: format_version: 5 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: enable_index_compression: 1 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: block_align: 0 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: max_auto_readahead_size: 262144 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: prepopulate_block_cache: 0 2024-04-22T15:23:00.249 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: initial_auto_readahead_size: 8192 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression: NoCompression 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.num_levels: 7 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T15:23:00.250 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T15:23:00.251 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T15:23:00.252 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T15:23:00.253 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.inplace_update_support: 0 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.bloom_locality: 0 2024-04-22T15:23:00.254 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.max_successive_merges: 0 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.ttl: 2592000 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.enable_blob_files: false 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.min_blob_size: 0 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T15:23:00.255 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T15:23:00.256 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T15:23:00.256 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T15:23:00.256 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T15:23:00.256 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi096/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12001, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 47b06c48-7c12-4243-b0a8-7f79eb4a9fac 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713799379908560, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713799379915688, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 2016537, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12003, "largest_seqno": 13772, "table_properties": {"data_size": 2008254, "index_size": 5257, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1989, "raw_key_size": 17835, "raw_average_key_size": 23, "raw_value_size": 1991659, "raw_average_value_size": 2579, "num_data_blocks": 240, "num_entries": 772, "num_filter_entries": 772, "num_deletions": 6, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713799379, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "47b06c48-7c12-4243-b0a8-7f79eb4a9fac", "db_session_id": "KBH06OL7TG4VR8SMQ8G1", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713799379915803, "job": 1, "event": "recovery_finished"} 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi096/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T15:23:00.257 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561614e58000 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: DB pointer 0x561614e42000 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ** DB Stats ** 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:23:00.258 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ** Compaction Stats [default] ** 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: L0 1/0 1.92 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 554.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: L6 1/0 7.00 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Sum 2/0 8.92 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 554.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 554.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ** Compaction Stats [default] ** 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 554.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:23:00.259 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Cumulative compaction: 0.00 GB write, 194.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Interval compaction: 0.00 GB write, 194.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Block cache BinnedLRUCache@0x561614d431f0#2 capacity: 512.00 MB usage: 8.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T15:23:00.260 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: starting mon.smithi096 rank 0 at public addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] at bind addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi096 fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: mon.smithi096@-1(???) e2 preinit fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: mon.smithi096@-1(???).mds e20 new map 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: mon.smithi096@-1(???).mds e20 print_map 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: e20 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: legacy client fscid: 1 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: Filesystem 'cephfs' (1) 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: fs_name cephfs 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: epoch 20 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:23:00.261 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: tableserver 0 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: root 0 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: session_timeout 60 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: session_autoclose 300 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: max_file_size 1099511627776 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: max_xattr_size 65536 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: required_client_features {} 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: last_failure 0 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: last_failure_osd_epoch 45 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: max_mds 2 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: in 0,1 2024-04-22T15:23:00.262 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: up {0=14494,1=24257} 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: failed 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: damaged 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: stopped 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: data_pools [3] 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: metadata_pool 2 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: inline_data enabled 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: balancer 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: bal_rank_mask -1 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: standby_count_wanted 1 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: qdb_cluster leader: 0 members: 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: [mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: [mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: [mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:00.263 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: [mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:00.264 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:23:00.264 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout: 2024-04-22T15:23:00.264 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: mon.smithi096@-1(???).osd e47 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-22T15:23:00.264 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: mon.smithi096@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:23:00.264 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: mon.smithi096@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:23:00.264 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: mon.smithi096@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:23:00.264 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 ceph-mon[112443]: mon.smithi096@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-04-22T15:23:00.264 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 bash[112424]: cc6f8ad67833cb73f09be75789545f37f7b11913b0dfd53503ede9b300676d3d 2024-04-22T15:23:00.264 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:22:59 smithi096 systemd[1]: Started Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:23:01.713 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: pgmap v112: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: mon.smithi096 calling monitor election 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: mon.smithi096 is new leader, mons smithi096,smithi196 in quorum (ranks 0,1) 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: monmap epoch 2 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: last_changed 2024-04-22T15:11:11.484887+0000 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: created 2024-04-22T15:08:12.449340+0000 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: min_mon_release 18 (reef) 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: election_strategy: 1 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: 0: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.smithi096 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.smithi196 2024-04-22T15:23:01.714 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:23:01.715 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T15:23:01.715 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: mgrmap e32: smithi096.hhkvum(active, since 3m), standbys: smithi196.gdaegc 2024-04-22T15:23:01.715 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:01.715 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:01.715 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:23:01.715 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: from='mgr.14678 ' entity='' 2024-04-22T15:23:01.715 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:01 smithi196 ceph-mon[34717]: mgrmap e33: smithi096.hhkvum(active, since 3m), standbys: smithi196.gdaegc 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: from='mgr.14678 172.21.15.96:0/51494378' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: pgmap v112: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: mon.smithi096 calling monitor election 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: mon.smithi096 is new leader, mons smithi096,smithi196 in quorum (ranks 0,1) 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: monmap epoch 2 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: last_changed 2024-04-22T15:11:11.484887+0000 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: created 2024-04-22T15:08:12.449340+0000 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: min_mon_release 18 (reef) 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: election_strategy: 1 2024-04-22T15:23:01.716 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: 0: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.smithi096 2024-04-22T15:23:01.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.smithi196 2024-04-22T15:23:01.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:23:01.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T15:23:01.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: mgrmap e32: smithi096.hhkvum(active, since 3m), standbys: smithi196.gdaegc 2024-04-22T15:23:01.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:01.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:01.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:23:01.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: from='mgr.14678 ' entity='' 2024-04-22T15:23:01.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:01 smithi096 ceph-mon[112443]: mgrmap e33: smithi096.hhkvum(active, since 3m), standbys: smithi196.gdaegc 2024-04-22T15:23:08.281 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: Active manager daemon smithi096.hhkvum restarted 2024-04-22T15:23:08.281 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: Activating manager daemon smithi096.hhkvum 2024-04-22T15:23:08.282 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T15:23:08.282 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: mgrmap e34: smithi096.hhkvum(active, starting, since 0.00878708s), standbys: smithi196.gdaegc 2024-04-22T15:23:08.282 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:23:08.282 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:23:08.282 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:23:08.282 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:23:08.282 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:23:08.282 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:23:08.282 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi196.gdaegc", "id": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: Manager daemon smithi096.hhkvum is now available 2024-04-22T15:23:08.283 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:08 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:23:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: Active manager daemon smithi096.hhkvum restarted 2024-04-22T15:23:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: Activating manager daemon smithi096.hhkvum 2024-04-22T15:23:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: mgrmap e34: smithi096.hhkvum(active, starting, since 0.00878708s), standbys: smithi196.gdaegc 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi096.hhkvum", "id": "smithi096.hhkvum"}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr metadata", "who": "smithi196.gdaegc", "id": "smithi196.gdaegc"}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:23:08.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:23:08.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:23:08.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:23:08.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:23:08.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:23:08.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:23:08.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: Manager daemon smithi096.hhkvum is now available 2024-04-22T15:23:08.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:23:09.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:23:09.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:23:09.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/trash_purge_schedule"}]: dispatch 2024-04-22T15:23:09.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: Standby manager daemon smithi196.gdaegc restarted 2024-04-22T15:23:09.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:23:09.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: from='mgr.? 172.21.15.196:0/51657619' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:23:09.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: from='mgr.? 172.21.15.196:0/51657619' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:23:09.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: from='mgr.? 172.21.15.196:0/51657619' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:23:09.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: from='mgr.? 172.21.15.196:0/51657619' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:23:09.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:09 smithi096 ceph-mon[112443]: mgrmap e35: smithi096.hhkvum(active, since 1.01063s), standbys: smithi196.gdaegc 2024-04-22T15:23:09.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:23:09.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:23:09.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi096.hhkvum/trash_purge_schedule"}]: dispatch 2024-04-22T15:23:09.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: Standby manager daemon smithi196.gdaegc restarted 2024-04-22T15:23:09.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: Standby manager daemon smithi196.gdaegc started 2024-04-22T15:23:09.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/51657619' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/crt"}]: dispatch 2024-04-22T15:23:09.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/51657619' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:23:09.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/51657619' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi196.gdaegc/key"}]: dispatch 2024-04-22T15:23:09.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: from='mgr.? 172.21.15.196:0/51657619' entity='mgr.smithi196.gdaegc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:23:09.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:09 smithi196 ceph-mon[34717]: mgrmap e35: smithi096.hhkvum(active, since 1.01063s), standbys: smithi196.gdaegc 2024-04-22T15:23:10.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:10 smithi096 ceph-mon[112443]: [22/Apr/2024:15:23:09] ENGINE Bus STARTING 2024-04-22T15:23:10.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:10 smithi196 ceph-mon[34717]: [22/Apr/2024:15:23:09] ENGINE Bus STARTING 2024-04-22T15:23:11.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:11 smithi096 ceph-mon[112443]: [22/Apr/2024:15:23:09] ENGINE Serving on https://172.21.15.96:7150 2024-04-22T15:23:11.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:11 smithi096 ceph-mon[112443]: [22/Apr/2024:15:23:09] ENGINE Serving on http://172.21.15.96:8765 2024-04-22T15:23:11.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:11 smithi096 ceph-mon[112443]: [22/Apr/2024:15:23:09] ENGINE Bus STARTED 2024-04-22T15:23:11.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:11 smithi096 ceph-mon[112443]: pgmap v4: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:23:11.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:11 smithi096 ceph-mon[112443]: mgrmap e36: smithi096.hhkvum(active, since 2s), standbys: smithi196.gdaegc 2024-04-22T15:23:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:11 smithi196 ceph-mon[34717]: [22/Apr/2024:15:23:09] ENGINE Serving on https://172.21.15.96:7150 2024-04-22T15:23:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:11 smithi196 ceph-mon[34717]: [22/Apr/2024:15:23:09] ENGINE Serving on http://172.21.15.96:8765 2024-04-22T15:23:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:11 smithi196 ceph-mon[34717]: [22/Apr/2024:15:23:09] ENGINE Bus STARTED 2024-04-22T15:23:11.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:11 smithi196 ceph-mon[34717]: pgmap v4: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:23:11.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:11 smithi196 ceph-mon[34717]: mgrmap e36: smithi096.hhkvum(active, since 2s), standbys: smithi196.gdaegc 2024-04-22T15:23:12.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:12 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:12.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:12 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:12.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:12 smithi096 ceph-mon[112443]: pgmap v5: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:23:12.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:12 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:12.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:12 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:12.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:12 smithi196 ceph-mon[34717]: pgmap v5: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:23:13.625 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:13 smithi096 ceph-mon[112443]: mgrmap e37: smithi096.hhkvum(active, since 4s), standbys: smithi196.gdaegc 2024-04-22T15:23:13.625 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:13 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:13.625 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:13 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:13 smithi196 ceph-mon[34717]: mgrmap e37: smithi096.hhkvum(active, since 4s), standbys: smithi196.gdaegc 2024-04-22T15:23:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:13 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:13.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:13 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:15 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:15 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:15 smithi096 ceph-mon[112443]: pgmap v6: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-22T15:23:15.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:15 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:15.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:15 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:15.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:15 smithi196 ceph-mon[34717]: pgmap v6: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-22T15:23:16.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:16 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:16.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:16 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:16.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:16 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:16.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:16 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:17.480 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:17 smithi096 ceph-mon[112443]: pgmap v7: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-22T15:23:17.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:17 smithi196 ceph-mon[34717]: pgmap v7: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-22T15:23:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:19 smithi096 ceph-mon[112443]: pgmap v8: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-22T15:23:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:23:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:19 smithi196 ceph-mon[34717]: pgmap v8: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-22T15:23:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:19 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:19 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:19 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:19 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:23:20.593 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T15:23:20.593 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-22T15:23:20.593 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:23:20.593 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:23:20.593 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.593 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.593 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.594 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.594 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.594 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:23:20.594 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:23:20.594 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:20 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T15:23:20.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T15:23:20.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-22T15:23:20.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:23:20.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:23:20.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:20.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:23:20.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:23:20.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T15:23:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:21 smithi196 ceph-mon[34717]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:23:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:21 smithi196 ceph-mon[34717]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:23:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:21 smithi196 ceph-mon[34717]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:23:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:21 smithi196 ceph-mon[34717]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:23:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:21 smithi196 ceph-mon[34717]: pgmap v9: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T15:23:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:21 smithi096 ceph-mon[112443]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:23:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:21 smithi096 ceph-mon[112443]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:23:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:21 smithi096 ceph-mon[112443]: Updating smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:23:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:21 smithi096 ceph-mon[112443]: Updating smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:23:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:21 smithi096 ceph-mon[112443]: pgmap v9: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T15:23:22.941 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:22 smithi196 ceph-mon[34717]: Upgrade: Updating mon.smithi196 2024-04-22T15:23:22.941 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:22 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:22.941 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:22 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:23:22.942 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:22 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:23:22.942 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:22 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:22.942 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:22 smithi196 ceph-mon[34717]: Deploying daemon mon.smithi196 on smithi196 2024-04-22T15:23:22.942 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:22 smithi196 ceph-mon[34717]: pgmap v10: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-22T15:23:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:22 smithi096 ceph-mon[112443]: Upgrade: Updating mon.smithi196 2024-04-22T15:23:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:22 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:22 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:23:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:22 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:23:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:22 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:22.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:22 smithi096 ceph-mon[112443]: Deploying daemon mon.smithi196 on smithi196 2024-04-22T15:23:22.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:22 smithi096 ceph-mon[112443]: pgmap v10: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-22T15:23:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:23 smithi196 systemd[1]: Stopping Ceph mon.smithi196 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:23:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:23 smithi196 ceph-mon[34717]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:23:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:23:24.094 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:23 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196[34688]: 2024-04-22T15:23:23.769+0000 7f3b8b2ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi196 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:23:24.095 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:23 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196[34688]: 2024-04-22T15:23:23.769+0000 7f3b8b2ef700 -1 mon.smithi196@1(peon) e2 *** Got Signal Terminated *** 2024-04-22T15:23:24.095 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:23 smithi196 podman[81647]: 2024-04-22 15:23:23.831024303 +0000 UTC m=+0.192459136 container died 79938ec69af6bc46d2ac54dfab45ec5e32f053fd669366f310fe4033067cc879 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=reef-646b8c7, io.buildah.version=1.33.5) 2024-04-22T15:23:24.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:24 smithi196 podman[81647]: 2024-04-22 15:23:24.258020431 +0000 UTC m=+0.619455260 container cleanup 79938ec69af6bc46d2ac54dfab45ec5e32f053fd669366f310fe4033067cc879 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-646b8c7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-04-22T15:23:24.487 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:24 smithi196 bash[81647]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196 2024-04-22T15:23:24.739 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:24 smithi196 podman[81661]: 2024-04-22 15:23:24.485033913 +0000 UTC m=+0.651083371 container remove 79938ec69af6bc46d2ac54dfab45ec5e32f053fd669366f310fe4033067cc879 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, org.label-schema.vendor=CentOS, RELEASE=reef-646b8c7, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-22T15:23:24.739 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:24 smithi196 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi196.service: Deactivated successfully. 2024-04-22T15:23:24.739 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:24 smithi196 systemd[1]: Stopped Ceph mon.smithi196 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:23:24.739 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:24 smithi196 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi196.service: Consumed 7.496s CPU time. 2024-04-22T15:23:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:24 smithi196 systemd[1]: Starting Ceph mon.smithi196 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:23:25.424 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 podman[81796]: 2024-04-22 15:23:25.046507925 +0000 UTC m=+0.018080053 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:23:25.425 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 podman[81796]: 2024-04-22 15:23:25.177678463 +0000 UTC m=+0.149250580 container create 351725aca606a722c5bdb0f92830f73e97222074e413700c75812b55a9bd3878 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T15:23:25.425 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 podman[81796]: 2024-04-22 15:23:25.422291698 +0000 UTC m=+0.393863818 container init 351725aca606a722c5bdb0f92830f73e97222074e413700c75812b55a9bd3878 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T15:23:25.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.460+0000 7f6be5285700 1 -- 172.21.15.96:0/3388537268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be00ffb50 msgr2=0x7f6be00fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:25.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.461+0000 7f6be5285700 1 --2- 172.21.15.96:0/3388537268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be00ffb50 0x7f6be00fffd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009910 tx=0x7f6bd4009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:25.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.461+0000 7f6be5285700 1 -- 172.21.15.96:0/3388537268 shutdown_connections 2024-04-22T15:23:25.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.461+0000 7f6be5285700 1 --2- 172.21.15.96:0/3388537268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be00ffb50 0x7f6be00fffd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.461+0000 7f6be5285700 1 --2- 172.21.15.96:0/3388537268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6be0105c00 0x7f6be0105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.461+0000 7f6be5285700 1 -- 172.21.15.96:0/3388537268 >> 172.21.15.96:0/3388537268 conn(0x7f6be00fb680 msgr2=0x7f6be00fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.461+0000 7f6be5285700 1 -- 172.21.15.96:0/3388537268 shutdown_connections 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.461+0000 7f6be5285700 1 -- 172.21.15.96:0/3388537268 wait complete. 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.461+0000 7f6be5285700 1 Processor -- start 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6be5285700 1 -- start start 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6be5285700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6be00ffb50 0x7f6be0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6be5285700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be0105c00 0x7f6be0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6be5285700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be0198480 con 0x7f6be0105c00 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6be5285700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be01985c0 con 0x7f6be00ffb50 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bdeffd700 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6be00ffb50 msgr2=0x7f6be0197a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bde7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be0105c00 0x7f6be0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bdeffd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6be00ffb50 0x7f6be0197a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:23:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bde7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be0105c00 0x7f6be0197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58582/0 (socket says 172.21.15.96:58582) 2024-04-22T15:23:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bde7fc700 1 -- 172.21.15.96:0/1242411951 learned_addr learned my addr 172.21.15.96:0/1242411951 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bde7fc700 1 -- 172.21.15.96:0/1242411951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6be00ffb50 msgr2=0x7f6be0197a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:23:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bde7fc700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6be00ffb50 0x7f6be0197a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bde7fc700 1 -- 172.21.15.96:0/1242411951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd40095f0 con 0x7f6be0105c00 2024-04-22T15:23:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bde7fc700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be0105c00 0x7f6be0197f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400bda0 tx=0x7f6bd400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.462+0000 7f6bc7fff700 1 -- 172.21.15.96:0/1242411951 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bd401d070 con 0x7f6be0105c00 2024-04-22T15:23:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.463+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be0198840 con 0x7f6be0105c00 2024-04-22T15:23:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.463+0000 7f6bc7fff700 1 -- 172.21.15.96:0/1242411951 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bd400f460 con 0x7f6be0105c00 2024-04-22T15:23:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.463+0000 7f6bc7fff700 1 -- 172.21.15.96:0/1242411951 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bd4017630 con 0x7f6be0105c00 2024-04-22T15:23:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.463+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be01a0e20 con 0x7f6be0105c00 2024-04-22T15:23:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.464+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be0108030 con 0x7f6be0105c00 2024-04-22T15:23:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.464+0000 7f6bc7fff700 1 -- 172.21.15.96:0/1242411951 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f6bd400f5d0 con 0x7f6be0105c00 2024-04-22T15:23:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.465+0000 7f6bc7fff700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bc806ed40 0x7f6bc8071200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.465+0000 7f6bc7fff700 1 -- 172.21.15.96:0/1242411951 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f6bd4090570 con 0x7f6be0105c00 2024-04-22T15:23:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.465+0000 7f6bdeffd700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bc806ed40 0x7f6bc8071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.465+0000 7f6bdeffd700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bc806ed40 0x7f6bc8071200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0001050 tx=0x7f6bd0006df0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:25.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.470+0000 7f6bc7fff700 1 -- 172.21.15.96:0/1242411951 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6bd4059160 con 0x7f6be0105c00 2024-04-22T15:23:25.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.690+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6be00696f0 con 0x7f6bc806ed40 2024-04-22T15:23:25.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.691+0000 7f6bc7fff700 1 -- 172.21.15.96:0/1242411951 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f6be00696f0 con 0x7f6bc806ed40 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bc806ed40 msgr2=0x7f6bc8071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bc806ed40 0x7f6bc8071200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0001050 tx=0x7f6bd0006df0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be0105c00 msgr2=0x7f6be0197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be0105c00 0x7f6be0197f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400bda0 tx=0x7f6bd400be80 comp rx=0 tx=0).stop 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 shutdown_connections 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bc806ed40 0x7f6bc8071200 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6be00ffb50 0x7f6be0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 --2- 172.21.15.96:0/1242411951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6be0105c00 0x7f6be0197f40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 >> 172.21.15.96:0/1242411951 conn(0x7f6be00fb680 msgr2=0x7f6be0104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 shutdown_connections 2024-04-22T15:23:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.693+0000 7f6be5285700 1 -- 172.21.15.96:0/1242411951 wait complete. 2024-04-22T15:23:25.707 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:23:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 podman[81796]: 2024-04-22 15:23:25.425991148 +0000 UTC m=+0.397563265 container start 351725aca606a722c5bdb0f92830f73e97222074e413700c75812b55a9bd3878 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:23:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T15:23:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev), process ceph-mon, pid 2 2024-04-22T15:23:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: pidfile_write: ignore empty --pid-file 2024-04-22T15:23:25.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: load: jerasure load: lrc 2024-04-22T15:23:25.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: RocksDB version: 7.9.2 2024-04-22T15:23:25.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Git sha 0 2024-04-22T15:23:25.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Compile date 2024-04-22 12:55:29 2024-04-22T15:23:25.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: DB SUMMARY 2024-04-22T15:23:25.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: DB Session ID: K6ES2R05EK5DKD30KLCM 2024-04-22T15:23:25.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: CURRENT file: CURRENT 2024-04-22T15:23:25.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T15:23:25.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1391 Bytes 2024-04-22T15:23:25.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi196/store.db dir, Total Num: 1, files: 000027.sst 2024-04-22T15:23:25.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi196/store.db: 000025.log size: 6089838 ; 2024-04-22T15:23:25.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.error_if_exists: 0 2024-04-22T15:23:25.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.create_if_missing: 0 2024-04-22T15:23:25.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.paranoid_checks: 1 2024-04-22T15:23:25.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T15:23:25.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T15:23:25.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.env: 0x55638eb4ecc0 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.info_log: 0x556390173aa0 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.statistics: (nil) 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.use_fsync: 0 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_log_file_size: 0 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.allow_fallocate: 1 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T15:23:25.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T15:23:25.764 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.use_direct_reads: 0 2024-04-22T15:23:25.764 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T15:23:25.764 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T15:23:25.764 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.db_log_dir: 2024-04-22T15:23:25.764 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.wal_dir: 2024-04-22T15:23:25.764 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T15:23:25.764 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T15:23:25.764 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T15:23:25.764 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.write_buffer_manager: 0x5563901070e0 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T15:23:25.765 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.unordered_write: 0 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.row_cache: None 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.wal_filter: None 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.two_write_queues: 0 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.wal_compression: 0 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.atomic_flush: 0 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T15:23:25.766 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T15:23:25.768 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T15:23:25.768 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.log_readahead_size: 0 2024-04-22T15:23:25.768 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T15:23:25.768 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T15:23:25.768 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T15:23:25.768 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T15:23:25.768 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T15:23:25.768 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T15:23:25.768 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_background_jobs: 2 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_background_compactions: -1 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_subcompactions: 1 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_open_files: -1 2024-04-22T15:23:25.769 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_background_flushes: -1 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Compression algorithms supported: 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: kZSTD supported: 0 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: kXpressCompression supported: 0 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: kZlibCompression supported: 1 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: kSnappyCompression supported: 1 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: kLZ4Compression supported: 1 2024-04-22T15:23:25.770 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: kBZip2Compression supported: 0 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi196/store.db/MANIFEST-000010 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.merge_operator: 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_filter: None 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556390173c80) 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-04-22T15:23:25.771 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: index_type: 0 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: data_block_index_type: 0 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: index_shortening: 1 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: checksum: 4 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: no_block_cache: 0 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_cache: 0x5563901391f0 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_cache_options: 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: capacity : 536870912 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: num_shard_bits : 4 2024-04-22T15:23:25.772 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: strict_capacity_limit : 0 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_cache_compressed: (nil) 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: persistent_cache: (nil) 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_size: 4096 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_size_deviation: 10 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_restart_interval: 16 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: index_block_restart_interval: 1 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: metadata_block_size: 4096 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: partition_filters: 0 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: use_delta_encoding: 1 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: filter_policy: bloomfilter 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: whole_key_filtering: 1 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: verify_compression: 0 2024-04-22T15:23:25.773 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: format_version: 5 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: enable_index_compression: 1 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: block_align: 0 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: max_auto_readahead_size: 262144 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: prepopulate_block_cache: 0 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: initial_auto_readahead_size: 8192 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression: NoCompression 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T15:23:25.774 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.num_levels: 7 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:23:25.775 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T15:23:25.776 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T15:23:25.777 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T15:23:25.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.inplace_update_support: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.bloom_locality: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.max_successive_merges: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.ttl: 2592000 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T15:23:25.779 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.enable_blob_files: false 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.min_blob_size: 0 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T15:23:25.780 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi196/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12127, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 18ff6f26-d666-4b3f-ad0a-b5ae9b444f56 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713799405454765, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713799405470339, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3692722, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12133, "largest_seqno": 14857, "table_properties": {"data_size": 3681718, "index_size": 6821, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3141, "raw_key_size": 27149, "raw_average_key_size": 21, "raw_value_size": 3657304, "raw_average_value_size": 2954, "num_data_blocks": 308, "num_entries": 1238, "num_filter_entries": 1238, "num_deletions": 257, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713799405, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "18ff6f26-d666-4b3f-ad0a-b5ae9b444f56", "db_session_id": "K6ES2R05EK5DKD30KLCM", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713799405470459, "job": 1, "event": "recovery_finished"} 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi196/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55639024e000 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: DB pointer 0x556390238000 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T15:23:25.781 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ** DB Stats ** 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T15:23:25.782 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: L0 1/0 3.52 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 498.0 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: L6 1/0 7.00 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Sum 2/0 10.52 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 498.0 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 498.0 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 498.0 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-22T15:23:25.783 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Cumulative compaction: 0.00 GB write, 193.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Interval compaction: 0.00 GB write, 193.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Block cache BinnedLRUCache@0x5563901391f0#2 capacity: 512.00 MB usage: 11.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,3.28 KB,0.000625849%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: starting mon.smithi196 rank 1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi196 fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: mon.smithi196@-1(???) e2 preinit fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:23:25.784 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: mon.smithi196@-1(???).mds e20 new map 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: mon.smithi196@-1(???).mds e20 print_map 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: e20 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: legacy client fscid: 1 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: Filesystem 'cephfs' (1) 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: fs_name cephfs 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: epoch 20 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: tableserver 0 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: root 0 2024-04-22T15:23:25.785 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: session_timeout 60 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: session_autoclose 300 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: max_file_size 1099511627776 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: max_xattr_size 65536 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: required_client_features {} 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: last_failure 0 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: last_failure_osd_epoch 45 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: max_mds 2 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: in 0,1 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: up {0=14494,1=24257} 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: failed 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: damaged 2024-04-22T15:23:25.786 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: stopped 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: data_pools [3] 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: metadata_pool 2 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: inline_data enabled 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: balancer 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: bal_rank_mask -1 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: standby_count_wanted 1 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: qdb_cluster leader: 0 members: 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: [mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: [mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: [mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: [mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout: 2024-04-22T15:23:25.787 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: mon.smithi196@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-22T15:23:25.788 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: mon.smithi196@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:23:25.788 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: mon.smithi196@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:23:25.788 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: mon.smithi196@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:23:25.788 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 ceph-mon[81815]: mon.smithi196@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-22T15:23:25.788 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 bash[81796]: 351725aca606a722c5bdb0f92830f73e97222074e413700c75812b55a9bd3878 2024-04-22T15:23:25.788 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:25 smithi196 systemd[1]: Started Ceph mon.smithi196 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:23:25.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.802+0000 7f8ba8582700 1 -- 172.21.15.96:0/3305169636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba0105aa0 msgr2=0x7f8ba0105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:25.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.802+0000 7f8ba8582700 1 --2- 172.21.15.96:0/3305169636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba0105aa0 0x7f8ba0105e80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000b0a0 tx=0x7f8b9000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.802+0000 7f8ba8582700 1 -- 172.21.15.96:0/3305169636 shutdown_connections 2024-04-22T15:23:25.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.802+0000 7f8ba8582700 1 --2- 172.21.15.96:0/3305169636 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba00ffaa0 0x7f8ba00fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.802+0000 7f8ba8582700 1 --2- 172.21.15.96:0/3305169636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba0105aa0 0x7f8ba0105e80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.802+0000 7f8ba8582700 1 -- 172.21.15.96:0/3305169636 >> 172.21.15.96:0/3305169636 conn(0x7f8ba0076740 msgr2=0x7f8ba00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:25.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.802+0000 7f8ba8582700 1 -- 172.21.15.96:0/3305169636 shutdown_connections 2024-04-22T15:23:25.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.802+0000 7f8ba8582700 1 -- 172.21.15.96:0/3305169636 wait complete. 2024-04-22T15:23:25.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba8582700 1 Processor -- start 2024-04-22T15:23:25.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba8582700 1 -- start start 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba8582700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba00ffaa0 0x7f8ba019b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba8582700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba0105aa0 0x7f8ba019bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba8582700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba019c2f0 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba8582700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0195920 con 0x7f8ba0105aa0 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba631e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba00ffaa0 0x7f8ba019b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba631e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba00ffaa0 0x7f8ba019b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58606/0 (socket says 172.21.15.96:58606) 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba631e700 1 -- 172.21.15.96:0/484596129 learned_addr learned my addr 172.21.15.96:0/484596129 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba5b1d700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba0105aa0 0x7f8ba019bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba5b1d700 1 -- 172.21.15.96:0/484596129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba0105aa0 msgr2=0x7f8ba019bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T15:23:25.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba5b1d700 1 -- 172.21.15.96:0/484596129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba0105aa0 msgr2=0x7f8ba019bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba5b1d700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba0105aa0 0x7f8ba019bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.803+0000 7f8ba5b1d700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba0105aa0 0x7f8ba019bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.804+0000 7f8ba631e700 1 -- 172.21.15.96:0/484596129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba0105aa0 msgr2=0x7f8ba019bcd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.804+0000 7f8ba631e700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba0105aa0 0x7f8ba019bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.804+0000 7f8ba631e700 1 -- 172.21.15.96:0/484596129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b90009d00 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.804+0000 7f8ba631e700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba00ffaa0 0x7f8ba019b790 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000b070 tx=0x7f8b90007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.804+0000 7f8b9f7fe700 1 -- 172.21.15.96:0/484596129 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b9000e040 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.804+0000 7f8b9f7fe700 1 -- 172.21.15.96:0/484596129 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b90003c10 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.804+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba0195ba0 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.804+0000 7f8b9f7fe700 1 -- 172.21.15.96:0/484596129 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b9001d6b0 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.804+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba0196090 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.805+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba01080a0 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.805+0000 7f8b9f7fe700 1 -- 172.21.15.96:0/484596129 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f8b90004600 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.806+0000 7f8b9f7fe700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b8806edf0 0x7f8b880712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:25.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.806+0000 7f8b9f7fe700 1 -- 172.21.15.96:0/484596129 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f8b90090810 con 0x7f8ba00ffaa0 2024-04-22T15:23:25.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.806+0000 7f8ba5b1d700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b8806edf0 0x7f8b880712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:25.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.806+0000 7f8ba5b1d700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b8806edf0 0x7f8b880712b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0196a90 tx=0x7f8b9400b2c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:25.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:25.811+0000 7f8b9f7fe700 1 -- 172.21.15.96:0/484596129 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8b9005cf30 con 0x7f8ba00ffaa0 2024-04-22T15:23:26.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.033+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ba0196d90 con 0x7f8b8806edf0 2024-04-22T15:23:26.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.034+0000 7f8b9f7fe700 1 -- 172.21.15.96:0/484596129 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f8ba0196d90 con 0x7f8b8806edf0 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b8806edf0 msgr2=0x7f8b880712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b8806edf0 0x7f8b880712b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0196a90 tx=0x7f8b9400b2c0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba00ffaa0 msgr2=0x7f8ba019b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba00ffaa0 0x7f8ba019b790 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000b070 tx=0x7f8b90007bd0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 shutdown_connections 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b8806edf0 0x7f8b880712b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba00ffaa0 0x7f8ba019b790 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 --2- 172.21.15.96:0/484596129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba0105aa0 0x7f8ba019bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 >> 172.21.15.96:0/484596129 conn(0x7f8ba0076740 msgr2=0x7f8ba00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 shutdown_connections 2024-04-22T15:23:26.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.036+0000 7f8ba8582700 1 -- 172.21.15.96:0/484596129 wait complete. 2024-04-22T15:23:26.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.140+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/1736928548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa141017d0 msgr2=0x7ffa14105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.140+0000 7ffa1b9ce700 1 --2- 172.21.15.96:0/1736928548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa141017d0 0x7ffa14105820 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009910 tx=0x7ffa04009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:26.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.140+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/1736928548 shutdown_connections 2024-04-22T15:23:26.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.140+0000 7ffa1b9ce700 1 --2- 172.21.15.96:0/1736928548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa141017d0 0x7ffa14105820 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.140+0000 7ffa1b9ce700 1 --2- 172.21.15.96:0/1736928548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa14100e20 0x7ffa14101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.140+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/1736928548 >> 172.21.15.96:0/1736928548 conn(0x7ffa140fc6d0 msgr2=0x7ffa140feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:26.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.140+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/1736928548 shutdown_connections 2024-04-22T15:23:26.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.140+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/1736928548 wait complete. 2024-04-22T15:23:26.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1b9ce700 1 Processor -- start 2024-04-22T15:23:26.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1b9ce700 1 -- start start 2024-04-22T15:23:26.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1b9ce700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa14100e20 0x7ffa1419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:26.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1b9ce700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa141017d0 0x7ffa1419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:26.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1b9ce700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1419d330 con 0x7ffa14100e20 2024-04-22T15:23:26.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1b9ce700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14196920 con 0x7ffa141017d0 2024-04-22T15:23:26.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1976a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa14100e20 0x7ffa1419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:26.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1976a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa14100e20 0x7ffa1419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58624/0 (socket says 172.21.15.96:58624) 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1976a700 1 -- 172.21.15.96:0/3404272727 learned_addr learned my addr 172.21.15.96:0/3404272727 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa18f69700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa141017d0 0x7ffa1419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa18f69700 1 -- 172.21.15.96:0/3404272727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa141017d0 msgr2=0x7ffa1419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa18f69700 1 -- 172.21.15.96:0/3404272727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa141017d0 msgr2=0x7ffa1419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa18f69700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa141017d0 0x7ffa1419cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa18f69700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa141017d0 0x7ffa1419cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1976a700 1 -- 172.21.15.96:0/3404272727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa141017d0 msgr2=0x7ffa1419cc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1976a700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa141017d0 0x7ffa1419cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.141+0000 7ffa1976a700 1 -- 172.21.15.96:0/3404272727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa040095f0 con 0x7ffa14100e20 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.142+0000 7ffa1976a700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa14100e20 0x7ffa1419c710 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00d8d0 tx=0x7ffa0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.142+0000 7ffa0a7fc700 1 -- 172.21.15.96:0/3404272727 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa0c004a00 con 0x7ffa14100e20 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.142+0000 7ffa0a7fc700 1 -- 172.21.15.96:0/3404272727 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa0c0054e0 con 0x7ffa14100e20 2024-04-22T15:23:26.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.142+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa14196c00 con 0x7ffa14100e20 2024-04-22T15:23:26.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.142+0000 7ffa0a7fc700 1 -- 172.21.15.96:0/3404272727 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa0c009d60 con 0x7ffa14100e20 2024-04-22T15:23:26.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.142+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa14197150 con 0x7ffa14100e20 2024-04-22T15:23:26.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.143+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa14109220 con 0x7ffa14100e20 2024-04-22T15:23:26.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.143+0000 7ffa0a7fc700 1 -- 172.21.15.96:0/3404272727 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7ffa0c005020 con 0x7ffa14100e20 2024-04-22T15:23:26.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.143+0000 7ffa0a7fc700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffa0006edf0 0x7ffa000712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:26.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.144+0000 7ffa0a7fc700 1 -- 172.21.15.96:0/3404272727 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7ffa0c08eb90 con 0x7ffa14100e20 2024-04-22T15:23:26.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.144+0000 7ffa18f69700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffa0006edf0 0x7ffa000712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:26.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.144+0000 7ffa18f69700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffa0006edf0 0x7ffa000712b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffa141058f0 tx=0x7ffa0401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:26.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.148+0000 7ffa0a7fc700 1 -- 172.21.15.96:0/3404272727 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffa0c05b330 con 0x7ffa14100e20 2024-04-22T15:23:26.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.360+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa14197f30 con 0x7ffa0006edf0 2024-04-22T15:23:26.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.372+0000 7ffa0a7fc700 1 -- 172.21.15.96:0/3404272727 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7ffa14197f30 con 0x7ffa0006edf0 2024-04-22T15:23:26.376 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:23:26.376 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (2m) 12s ago 13m 25.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:23:26.376 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (13m) 12s ago 13m 8304k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:23:26.376 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (12m) 14s ago 12m 8078k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:23:26.376 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (13m) 12s ago 13m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:23:26.376 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (12m) 14s ago 12m 6665k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:23:26.376 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (105s) 12s ago 13m 76.2M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:23:26.376 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (7m) 12s ago 7m 157M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (7m) 12s ago 7m 90.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (7m) 14s ago 7m 26.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (7m) 14s ago 7m 25.7M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (4m) 12s ago 14m 473M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (3m) 14s ago 12m 448M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (26s) 12s ago 15m 27.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 starting - - - 2048M 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (3m) 12s ago 13m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (2m) 14s ago 12m 17.2M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (11m) 12s ago 11m 210M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (11m) 12s ago 11m 218M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:23:26.377 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (10m) 12s ago 10m 182M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:23:26.378 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (10m) 14s ago 10m 241M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:23:26.378 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (9m) 14s ago 9m 213M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:23:26.378 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (9m) 14s ago 9m 195M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:23:26.378 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (2m) 12s ago 12m 73.0M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.374+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffa0006edf0 msgr2=0x7ffa000712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.374+0000 7ffa1b9ce700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffa0006edf0 0x7ffa000712b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffa141058f0 tx=0x7ffa0401a040 comp rx=0 tx=0).stop 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.375+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa14100e20 msgr2=0x7ffa1419c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.375+0000 7ffa1b9ce700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa14100e20 0x7ffa1419c710 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00d8d0 tx=0x7ffa0c00dc90 comp rx=0 tx=0).stop 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.375+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 shutdown_connections 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.375+0000 7ffa1b9ce700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffa0006edf0 0x7ffa000712b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.375+0000 7ffa1b9ce700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa14100e20 0x7ffa1419c710 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.375+0000 7ffa1b9ce700 1 --2- 172.21.15.96:0/3404272727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa141017d0 0x7ffa1419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.375+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 >> 172.21.15.96:0/3404272727 conn(0x7ffa140fc6d0 msgr2=0x7ffa140fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:26.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.375+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 shutdown_connections 2024-04-22T15:23:26.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.375+0000 7ffa1b9ce700 1 -- 172.21.15.96:0/3404272727 wait complete. 2024-04-22T15:23:26.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.477+0000 7f4025c51700 1 -- 172.21.15.96:0/2906729707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 msgr2=0x7f4020105830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.477+0000 7f4025c51700 1 --2- 172.21.15.96:0/2906729707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 0x7f4020105830 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4008009910 tx=0x7f4008009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:26.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.477+0000 7f4025c51700 1 -- 172.21.15.96:0/2906729707 shutdown_connections 2024-04-22T15:23:26.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.477+0000 7f4025c51700 1 --2- 172.21.15.96:0/2906729707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 0x7f4020105830 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.477+0000 7f4025c51700 1 --2- 172.21.15.96:0/2906729707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4020100e30 0x7f4020101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.477+0000 7f4025c51700 1 -- 172.21.15.96:0/2906729707 >> 172.21.15.96:0/2906729707 conn(0x7f40200fc700 msgr2=0x7f40200feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.477+0000 7f4025c51700 1 -- 172.21.15.96:0/2906729707 shutdown_connections 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.477+0000 7f4025c51700 1 -- 172.21.15.96:0/2906729707 wait complete. 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f4025c51700 1 Processor -- start 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f4025c51700 1 -- start start 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f4025c51700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4020100e30 0x7f4020198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f4025c51700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 0x7f4020199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f4025c51700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4020199920 con 0x7f40201017e0 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f4025c51700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402019c540 con 0x7f4020100e30 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f401effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 0x7f4020199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f401effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 0x7f4020199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58630/0 (socket says 172.21.15.96:58630) 2024-04-22T15:23:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f401effd700 1 -- 172.21.15.96:0/982537556 learned_addr learned my addr 172.21.15.96:0/982537556 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f401f7fe700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4020100e30 0x7f4020198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f401effd700 1 -- 172.21.15.96:0/982537556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4020100e30 msgr2=0x7f4020198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f401effd700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4020100e30 0x7f4020198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.478+0000 7f401effd700 1 -- 172.21.15.96:0/982537556 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40080095f0 con 0x7f40201017e0 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.479+0000 7f401effd700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 0x7f4020199290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4008005b40 tx=0x7f400800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.479+0000 7f401cff9700 1 -- 172.21.15.96:0/982537556 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f400801d070 con 0x7f40201017e0 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.479+0000 7f401cff9700 1 -- 172.21.15.96:0/982537556 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f400800f460 con 0x7f40201017e0 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.479+0000 7f401cff9700 1 -- 172.21.15.96:0/982537556 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4008017630 con 0x7f40201017e0 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.479+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f402019c7c0 con 0x7f40201017e0 2024-04-22T15:23:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.479+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f402019ccb0 con 0x7f40201017e0 2024-04-22T15:23:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.480+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4020104fb0 con 0x7f40201017e0 2024-04-22T15:23:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.480+0000 7f401cff9700 1 -- 172.21.15.96:0/982537556 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f4008003a60 con 0x7f40201017e0 2024-04-22T15:23:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.481+0000 7f401cff9700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f400c06edf0 0x7f400c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.481+0000 7f401cff9700 1 -- 172.21.15.96:0/982537556 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f40080904e0 con 0x7f40201017e0 2024-04-22T15:23:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.481+0000 7f401f7fe700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f400c06edf0 0x7f400c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.481+0000 7f401f7fe700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f400c06edf0 0x7f400c0712b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f40180060b0 tx=0x7f4018007b80 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:26.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.485+0000 7f401cff9700 1 -- 172.21.15.96:0/982537556 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4008059150 con 0x7f40201017e0 2024-04-22T15:23:26.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.777+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4020199fd0 con 0x7f40201017e0 2024-04-22T15:23:26.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.777+0000 7f401cff9700 1 -- 172.21.15.96:0/982537556 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+733 (secure 0 0 0) 0x7f400805c810 con 0x7f40201017e0 2024-04-22T15:23:26.781 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:23:26.781 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:23:26.781 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:23:26.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:23:26.781 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:23:26.781 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:23:26.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:23:26.781 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 10, 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:23:26.782 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:23:26.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.779+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f400c06edf0 msgr2=0x7f400c0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.779+0000 7f4025c51700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f400c06edf0 0x7f400c0712b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f40180060b0 tx=0x7f4018007b80 comp rx=0 tx=0).stop 2024-04-22T15:23:26.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.779+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 msgr2=0x7f4020199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.779+0000 7f4025c51700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 0x7f4020199290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4008005b40 tx=0x7f400800be20 comp rx=0 tx=0).stop 2024-04-22T15:23:26.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.780+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 shutdown_connections 2024-04-22T15:23:26.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.780+0000 7f4025c51700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f400c06edf0 0x7f400c0712b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.780+0000 7f4025c51700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4020100e30 0x7f4020198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.780+0000 7f4025c51700 1 --2- 172.21.15.96:0/982537556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40201017e0 0x7f4020199290 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.780+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 >> 172.21.15.96:0/982537556 conn(0x7f40200fc700 msgr2=0x7f40200fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:26.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.780+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 shutdown_connections 2024-04-22T15:23:26.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.780+0000 7f4025c51700 1 -- 172.21.15.96:0/982537556 wait complete. 2024-04-22T15:23:26.840 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:26.840 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: mon.smithi096 calling monitor election 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: mon.smithi196 calling monitor election 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: mon.smithi096 is new leader, mons smithi096,smithi196 in quorum (ranks 0,1) 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: monmap epoch 3 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: last_changed 2024-04-22T15:23:25.753369+0000 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: created 2024-04-22T15:08:12.449340+0000 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: min_mon_release 19 (squid) 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: election_strategy: 1 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: 0: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.smithi096 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.smithi196 2024-04-22T15:23:26.841 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: mgrmap e37: smithi096.hhkvum(active, since 17s), standbys: smithi196.gdaegc 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: pgmap v12: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-22T15:23:26.842 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:26 smithi196 ceph-mon[81815]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:26.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.884+0000 7fef4f218700 1 -- 172.21.15.96:0/2690494750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef480ff170 msgr2=0x7fef480ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:26.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.884+0000 7fef4f218700 1 --2- 172.21.15.96:0/2690494750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef480ff170 0x7fef480ff5f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fef38009910 tx=0x7fef38009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:26.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.884+0000 7fef4f218700 1 -- 172.21.15.96:0/2690494750 shutdown_connections 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.884+0000 7fef4f218700 1 --2- 172.21.15.96:0/2690494750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef480ff170 0x7fef480ff5f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.884+0000 7fef4f218700 1 --2- 172.21.15.96:0/2690494750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef48105c40 0x7fef48106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.884+0000 7fef4f218700 1 -- 172.21.15.96:0/2690494750 >> 172.21.15.96:0/2690494750 conn(0x7fef48075f00 msgr2=0x7fef480fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.884+0000 7fef4f218700 1 -- 172.21.15.96:0/2690494750 shutdown_connections 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.884+0000 7fef4f218700 1 -- 172.21.15.96:0/2690494750 wait complete. 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.884+0000 7fef4f218700 1 Processor -- start 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4f218700 1 -- start start 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4f218700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef480ff170 0x7fef4819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4f218700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef48105c40 0x7fef4819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4f218700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4819d420 con 0x7fef48105c40 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4f218700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef48196a00 con 0x7fef480ff170 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef47fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef48105c40 0x7fef4819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef47fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef48105c40 0x7fef4819ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58640/0 (socket says 172.21.15.96:58640) 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef47fff700 1 -- 172.21.15.96:0/1936541703 learned_addr learned my addr 172.21.15.96:0/1936541703 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4cfb4700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef480ff170 0x7fef4819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4cfb4700 1 -- 172.21.15.96:0/1936541703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef480ff170 msgr2=0x7fef4819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4cfb4700 1 -- 172.21.15.96:0/1936541703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef480ff170 msgr2=0x7fef4819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4cfb4700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef480ff170 0x7fef4819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4cfb4700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef480ff170 0x7fef4819c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef47fff700 1 -- 172.21.15.96:0/1936541703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef480ff170 msgr2=0x7fef4819c8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:23:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef47fff700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef480ff170 0x7fef4819c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef47fff700 1 -- 172.21.15.96:0/1936541703 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef380095f0 con 0x7fef48105c40 2024-04-22T15:23:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef47fff700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef48105c40 0x7fef4819ce00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fef38000c00 tx=0x7fef3800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef45ffb700 1 -- 172.21.15.96:0/1936541703 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef3801d070 con 0x7fef48105c40 2024-04-22T15:23:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef45ffb700 1 -- 172.21.15.96:0/1936541703 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef380056e0 con 0x7fef48105c40 2024-04-22T15:23:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.885+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef48196c80 con 0x7fef48105c40 2024-04-22T15:23:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.886+0000 7fef45ffb700 1 -- 172.21.15.96:0/1936541703 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef3800fe80 con 0x7fef48105c40 2024-04-22T15:23:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.886+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef48197170 con 0x7fef48105c40 2024-04-22T15:23:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.887+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef481091e0 con 0x7fef48105c40 2024-04-22T15:23:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.887+0000 7fef45ffb700 1 -- 172.21.15.96:0/1936541703 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fef3800f6f0 con 0x7fef48105c40 2024-04-22T15:23:26.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.887+0000 7fef45ffb700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef3406edf0 0x7fef340712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:26.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.888+0000 7fef45ffb700 1 -- 172.21.15.96:0/1936541703 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fef380903b0 con 0x7fef48105c40 2024-04-22T15:23:26.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.888+0000 7fef4cfb4700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef3406edf0 0x7fef340712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:26.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.888+0000 7fef4cfb4700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef3406edf0 0x7fef340712b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef400098c0 tx=0x7fef40009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:26.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:26.892+0000 7fef45ffb700 1 -- 172.21.15.96:0/1936541703 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fef3805cb50 con 0x7fef48105c40 2024-04-22T15:23:27.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.133+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fef48102490 con 0x7fef48105c40 2024-04-22T15:23:27.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.134+0000 7fef45ffb700 1 -- 172.21.15.96:0/1936541703 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1901 (secure 0 0 0) 0x7fef3805c6e0 con 0x7fef48105c40 2024-04-22T15:23:27.137 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi096"}]: dispatch 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: mon.smithi096 calling monitor election 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: mon.smithi196 calling monitor election 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mon metadata", "id": "smithi196"}]: dispatch 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: mon.smithi096 is new leader, mons smithi096,smithi196 in quorum (ranks 0,1) 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: monmap epoch 3 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: fsid d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: last_changed 2024-04-22T15:23:25.753369+0000 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: created 2024-04-22T15:08:12.449340+0000 2024-04-22T15:23:27.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: min_mon_release 19 (squid) 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: election_strategy: 1 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: 0: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.smithi096 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.smithi196 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2 up:standby-replay 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: mgrmap e37: smithi096.hhkvum(active, since 17s), standbys: smithi196.gdaegc 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:23:27.139 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:27.140 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: pgmap v12: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-22T15:23:27.140 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:26 smithi096 ceph-mon[112443]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:27.140 INFO:teuthology.orchestra.run.smithi096.stdout:e20 2024-04-22T15:23:27.140 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:23:27.140 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:23:27.140 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:23:27.140 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:23:27.140 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:23:27.140 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:23:27.140 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 20 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:23:27.141 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:23:27.142 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:23:27.143 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:23:27.143 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:27.143 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:27.143 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:27.143 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:27.143 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:23:27.143 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:23:27.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef3406edf0 msgr2=0x7fef340712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:27.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef3406edf0 0x7fef340712b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef400098c0 tx=0x7fef40009040 comp rx=0 tx=0).stop 2024-04-22T15:23:27.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef48105c40 msgr2=0x7fef4819ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:27.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef48105c40 0x7fef4819ce00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fef38000c00 tx=0x7fef3800be20 comp rx=0 tx=0).stop 2024-04-22T15:23:27.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 shutdown_connections 2024-04-22T15:23:27.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef3406edf0 0x7fef340712b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef480ff170 0x7fef4819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 --2- 172.21.15.96:0/1936541703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef48105c40 0x7fef4819ce00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 >> 172.21.15.96:0/1936541703 conn(0x7fef48075f00 msgr2=0x7fef480fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:27.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 shutdown_connections 2024-04-22T15:23:27.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.137+0000 7fef4f218700 1 -- 172.21.15.96:0/1936541703 wait complete. 2024-04-22T15:23:27.145 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:23:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.243+0000 7f4fed7ff700 1 -- 172.21.15.96:0/258955427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 msgr2=0x7f4fe8105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.243+0000 7f4fed7ff700 1 --2- 172.21.15.96:0/258955427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 0x7f4fe8105820 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009910 tx=0x7f4fdc009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.243+0000 7f4fed7ff700 1 -- 172.21.15.96:0/258955427 shutdown_connections 2024-04-22T15:23:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.243+0000 7f4fed7ff700 1 --2- 172.21.15.96:0/258955427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 0x7f4fe8105820 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.243+0000 7f4fed7ff700 1 --2- 172.21.15.96:0/258955427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 0x7f4fe8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.243+0000 7f4fed7ff700 1 -- 172.21.15.96:0/258955427 >> 172.21.15.96:0/258955427 conn(0x7f4fe80fc6d0 msgr2=0x7f4fe80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.243+0000 7f4fed7ff700 1 -- 172.21.15.96:0/258955427 shutdown_connections 2024-04-22T15:23:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.243+0000 7f4fed7ff700 1 -- 172.21.15.96:0/258955427 wait complete. 2024-04-22T15:23:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fed7ff700 1 Processor -- start 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fed7ff700 1 -- start start 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fed7ff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 0x7f4fe819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fed7ff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 0x7f4fe819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fed7ff700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe819d360 con 0x7f4fe81017d0 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fed7ff700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe8196940 con 0x7f4fe8100e20 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 0x7f4fe819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 0x7f4fe819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58656/0 (socket says 172.21.15.96:58656) 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe67fc700 1 -- 172.21.15.96:0/2355536341 learned_addr learned my addr 172.21.15.96:0/2355536341 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe6ffd700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 0x7f4fe819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe6ffd700 1 -- 172.21.15.96:0/2355536341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 msgr2=0x7f4fe819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe6ffd700 1 -- 172.21.15.96:0/2355536341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 msgr2=0x7f4fe819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:23:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe6ffd700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 0x7f4fe819c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe6ffd700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 0x7f4fe819c800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe67fc700 1 -- 172.21.15.96:0/2355536341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 msgr2=0x7f4fe819c800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe67fc700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 0x7f4fe819c800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe67fc700 1 -- 172.21.15.96:0/2355536341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc0095f0 con 0x7f4fe81017d0 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.244+0000 7f4fe67fc700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 0x7f4fe819cd40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00bd70 tx=0x7f4fdc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.245+0000 7f4fcffff700 1 -- 172.21.15.96:0/2355536341 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fdc01d070 con 0x7f4fe81017d0 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.245+0000 7f4fcffff700 1 -- 172.21.15.96:0/2355536341 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fdc0056e0 con 0x7f4fe81017d0 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.245+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe8196bc0 con 0x7f4fe81017d0 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.245+0000 7f4fcffff700 1 -- 172.21.15.96:0/2355536341 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fdc017400 con 0x7f4fe81017d0 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.245+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe81970b0 con 0x7f4fe81017d0 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.246+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe8109220 con 0x7f4fe81017d0 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.246+0000 7f4fcffff700 1 -- 172.21.15.96:0/2355536341 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f4fdc003a60 con 0x7f4fe81017d0 2024-04-22T15:23:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.246+0000 7f4fcffff700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd006ee00 0x7f4fd00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:27.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.247+0000 7f4fe6ffd700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd006ee00 0x7f4fd00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:27.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.247+0000 7f4fcffff700 1 -- 172.21.15.96:0/2355536341 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f4fdc0904b0 con 0x7f4fe81017d0 2024-04-22T15:23:27.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.247+0000 7f4fe6ffd700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd006ee00 0x7f4fd00712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8005950 tx=0x7f4fd80058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:27.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.251+0000 7f4fcffff700 1 -- 172.21.15.96:0/2355536341 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4fdc05cc90 con 0x7f4fe81017d0 2024-04-22T15:23:27.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.468+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fe8197e70 con 0x7f4fd006ee00 2024-04-22T15:23:27.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.469+0000 7f4fcffff700 1 -- 172.21.15.96:0/2355536341 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f4fe8197e70 con 0x7f4fd006ee00 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "3/23 daemons upgraded", 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mon daemons", 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:23:27.473 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:23:27.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.472+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd006ee00 msgr2=0x7f4fd00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:27.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.472+0000 7f4fed7ff700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd006ee00 0x7f4fd00712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8005950 tx=0x7f4fd80058e0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.472+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 msgr2=0x7f4fe819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:27.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.472+0000 7f4fed7ff700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 0x7f4fe819cd40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00bd70 tx=0x7f4fdc00be50 comp rx=0 tx=0).stop 2024-04-22T15:23:27.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.472+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 shutdown_connections 2024-04-22T15:23:27.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.472+0000 7f4fed7ff700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd006ee00 0x7f4fd00712c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.472+0000 7f4fed7ff700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fe8100e20 0x7f4fe819c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.472+0000 7f4fed7ff700 1 --2- 172.21.15.96:0/2355536341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fe81017d0 0x7f4fe819cd40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.472+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 >> 172.21.15.96:0/2355536341 conn(0x7f4fe80fc6d0 msgr2=0x7f4fe80fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:27.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.473+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 shutdown_connections 2024-04-22T15:23:27.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.473+0000 7f4fed7ff700 1 -- 172.21.15.96:0/2355536341 wait complete. 2024-04-22T15:23:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.575+0000 7fdb65feb700 1 -- 172.21.15.96:0/1747704369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 msgr2=0x7fdb600735e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.575+0000 7fdb65feb700 1 --2- 172.21.15.96:0/1747704369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 0x7fdb600735e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdb50009910 tx=0x7fdb50009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.575+0000 7fdb65feb700 1 -- 172.21.15.96:0/1747704369 shutdown_connections 2024-04-22T15:23:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.575+0000 7fdb65feb700 1 --2- 172.21.15.96:0/1747704369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 0x7fdb6010cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.575+0000 7fdb65feb700 1 --2- 172.21.15.96:0/1747704369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 0x7fdb600735e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.575+0000 7fdb65feb700 1 -- 172.21.15.96:0/1747704369 >> 172.21.15.96:0/1747704369 conn(0x7fdb600fc6d0 msgr2=0x7fdb600feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:27.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.575+0000 7fdb65feb700 1 -- 172.21.15.96:0/1747704369 shutdown_connections 2024-04-22T15:23:27.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.575+0000 7fdb65feb700 1 -- 172.21.15.96:0/1747704369 wait complete. 2024-04-22T15:23:27.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.576+0000 7fdb65feb700 1 Processor -- start 2024-04-22T15:23:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.576+0000 7fdb65feb700 1 -- start start 2024-04-22T15:23:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.576+0000 7fdb65feb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 0x7fdb60198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.576+0000 7fdb65feb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 0x7fdb60199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.576+0000 7fdb65feb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb601998d0 con 0x7fdb60073200 2024-04-22T15:23:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.576+0000 7fdb65feb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6019c540 con 0x7fdb60073b20 2024-04-22T15:23:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.576+0000 7fdb5f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 0x7fdb60198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 0x7fdb60198d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58682/0 (socket says 172.21.15.96:58682) 2024-04-22T15:23:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5f7fe700 1 -- 172.21.15.96:0/657731967 learned_addr learned my addr 172.21.15.96:0/657731967 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5effd700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 0x7fdb60199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5effd700 1 -- 172.21.15.96:0/657731967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 msgr2=0x7fdb60199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5effd700 1 -- 172.21.15.96:0/657731967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 msgr2=0x7fdb60199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5effd700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 0x7fdb60199240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5effd700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 0x7fdb60199240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5f7fe700 1 -- 172.21.15.96:0/657731967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 msgr2=0x7fdb60199240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5f7fe700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 0x7fdb60199240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5f7fe700 1 -- 172.21.15.96:0/657731967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb500095f0 con 0x7fdb60073200 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5f7fe700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 0x7fdb60198d00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb50000c00 tx=0x7fdb5000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5cff9700 1 -- 172.21.15.96:0/657731967 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb5001d070 con 0x7fdb60073200 2024-04-22T15:23:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5cff9700 1 -- 172.21.15.96:0/657731967 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb500056e0 con 0x7fdb60073200 2024-04-22T15:23:27.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6019c7c0 con 0x7fdb60073200 2024-04-22T15:23:27.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb5cff9700 1 -- 172.21.15.96:0/657731967 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb5000fe80 con 0x7fdb60073200 2024-04-22T15:23:27.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.577+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6019ccb0 con 0x7fdb60073200 2024-04-22T15:23:27.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.578+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb60074940 con 0x7fdb60073200 2024-04-22T15:23:27.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.579+0000 7fdb5cff9700 1 -- 172.21.15.96:0/657731967 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fdb5000f6f0 con 0x7fdb60073200 2024-04-22T15:23:27.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.579+0000 7fdb5cff9700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb4806edf0 0x7fdb480712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:27.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.579+0000 7fdb5cff9700 1 -- 172.21.15.96:0/657731967 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fdb500903e0 con 0x7fdb60073200 2024-04-22T15:23:27.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.579+0000 7fdb5effd700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb4806edf0 0x7fdb480712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:27.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.580+0000 7fdb5effd700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb4806edf0 0x7fdb480712b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdb54005950 tx=0x7fdb540058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:27.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.584+0000 7fdb5cff9700 1 -- 172.21.15.96:0/657731967 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdb50059050 con 0x7fdb60073200 2024-04-22T15:23:27.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.869+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdb601077c0 con 0x7fdb60073200 2024-04-22T15:23:27.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.869+0000 7fdb5cff9700 1 -- 172.21.15.96:0/657731967 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fdb5005c710 con 0x7fdb60073200 2024-04-22T15:23:27.873 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:27.873 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:23:27.873 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.871+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb4806edf0 msgr2=0x7fdb480712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.871+0000 7fdb65feb700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb4806edf0 0x7fdb480712b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdb54005950 tx=0x7fdb540058e0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.871+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 msgr2=0x7fdb60198d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.871+0000 7fdb65feb700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 0x7fdb60198d00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb50000c00 tx=0x7fdb5000be20 comp rx=0 tx=0).stop 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.871+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 shutdown_connections 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.871+0000 7fdb65feb700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb4806edf0 0x7fdb480712b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.871+0000 7fdb65feb700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb60073200 0x7fdb60198d00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.871+0000 7fdb65feb700 1 --2- 172.21.15.96:0/657731967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb60073b20 0x7fdb60199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.871+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 >> 172.21.15.96:0/657731967 conn(0x7fdb600fc6d0 msgr2=0x7fdb60105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:27.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.872+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 shutdown_connections 2024-04-22T15:23:27.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:27.872+0000 7fdb65feb700 1 -- 172.21.15.96:0/657731967 wait complete. 2024-04-22T15:23:28.092 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:27 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/982537556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:23:28.093 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:27 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1936541703' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:23:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:27 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/982537556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:23:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:27 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1936541703' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:23:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:28 smithi096 ceph-mon[112443]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:28 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/657731967' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:23:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:28 smithi096 ceph-mon[112443]: pgmap v13: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:28 smithi196 ceph-mon[81815]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:28 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/657731967' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:23:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:28 smithi196 ceph-mon[81815]: pgmap v13: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:30.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:29 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:30.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:29 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:30.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:29 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:30.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:29 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:30 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:30 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:30 smithi096 ceph-mon[112443]: pgmap v14: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:30 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:30 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:30 smithi196 ceph-mon[81815]: pgmap v14: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:33.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:33 smithi196 ceph-mon[81815]: pgmap v15: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:33.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:33.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:33.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:33.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:23:33.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:33.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:33 smithi096 ceph-mon[112443]: pgmap v15: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:33.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:33.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:33.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:33.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:23:33.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:34.644 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:34 smithi096 ceph-mon[112443]: Reconfiguring mon.smithi096 (monmap changed)... 2024-04-22T15:23:34.644 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:34 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:23:34.644 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:34 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:23:34.644 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:34 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:34.644 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:34 smithi096 ceph-mon[112443]: Reconfiguring daemon mon.smithi096 on smithi096 2024-04-22T15:23:34.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:34 smithi196 ceph-mon[81815]: Reconfiguring mon.smithi096 (monmap changed)... 2024-04-22T15:23:34.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:34 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:23:34.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:34 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:23:34.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:34 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:34.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:34 smithi196 ceph-mon[81815]: Reconfiguring daemon mon.smithi096 on smithi096 2024-04-22T15:23:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:35 smithi096 ceph-mon[112443]: pgmap v16: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi096.hhkvum", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:23:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:23:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:35 smithi196 ceph-mon[81815]: pgmap v16: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi096.hhkvum", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:23:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:23:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:36.585 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:36 smithi096 ceph-mon[112443]: Reconfiguring mgr.smithi096.hhkvum (monmap changed)... 2024-04-22T15:23:36.585 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:36 smithi096 ceph-mon[112443]: Reconfiguring daemon mgr.smithi096.hhkvum on smithi096 2024-04-22T15:23:36.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:36 smithi196 ceph-mon[81815]: Reconfiguring mgr.smithi096.hhkvum (monmap changed)... 2024-04-22T15:23:36.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:36 smithi196 ceph-mon[81815]: Reconfiguring daemon mgr.smithi096.hhkvum on smithi096 2024-04-22T15:23:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:37 smithi096 ceph-mon[112443]: pgmap v17: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:23:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:37 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:37 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:37 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:23:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:37 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:23:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:37 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:23:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:37 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi096"}]: dispatch 2024-04-22T15:23:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:37 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:37.544 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:37 smithi196 ceph-mon[81815]: pgmap v17: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:23:37.545 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:37 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:37.545 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:37 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:37.545 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:37 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:23:37.545 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:37 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:23:37.545 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:37 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:23:37.545 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:37 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi096"}]: dispatch 2024-04-22T15:23:37.545 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:37 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:38 smithi096 ceph-mon[112443]: Reconfiguring ceph-exporter.smithi096 (monmap changed)... 2024-04-22T15:23:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:38 smithi096 ceph-mon[112443]: Unable to update caps for client.ceph-exporter.smithi096 2024-04-22T15:23:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:38 smithi096 ceph-mon[112443]: Reconfiguring daemon ceph-exporter.smithi096 on smithi096 2024-04-22T15:23:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:23:38.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:38 smithi196 ceph-mon[81815]: Reconfiguring ceph-exporter.smithi096 (monmap changed)... 2024-04-22T15:23:38.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:38 smithi196 ceph-mon[81815]: Unable to update caps for client.ceph-exporter.smithi096 2024-04-22T15:23:38.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:38 smithi196 ceph-mon[81815]: Reconfiguring daemon ceph-exporter.smithi096 on smithi096 2024-04-22T15:23:38.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:23:39.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:39 smithi096 ceph-mon[112443]: pgmap v18: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:39.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:39.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:39.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:39 smithi096 ceph-mon[112443]: Reconfiguring crash.smithi096 (monmap changed)... 2024-04-22T15:23:39.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi096", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:23:39.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:39.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:39 smithi096 ceph-mon[112443]: Reconfiguring daemon crash.smithi096 on smithi096 2024-04-22T15:23:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:39 smithi196 ceph-mon[81815]: pgmap v18: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:39 smithi196 ceph-mon[81815]: Reconfiguring crash.smithi096 (monmap changed)... 2024-04-22T15:23:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi096", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:23:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:39 smithi196 ceph-mon[81815]: Reconfiguring daemon crash.smithi096 on smithi096 2024-04-22T15:23:41.423 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:41.423 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:41.423 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:41 smithi096 ceph-mon[112443]: Reconfiguring osd.0 (monmap changed)... 2024-04-22T15:23:41.423 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:23:41.424 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:41.424 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:41 smithi096 ceph-mon[112443]: Reconfiguring daemon osd.0 on smithi096 2024-04-22T15:23:41.424 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:41 smithi096 ceph-mon[112443]: pgmap v19: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:41.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:41 smithi196 ceph-mon[81815]: Reconfiguring osd.0 (monmap changed)... 2024-04-22T15:23:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:23:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:41 smithi196 ceph-mon[81815]: Reconfiguring daemon osd.0 on smithi096 2024-04-22T15:23:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:41 smithi196 ceph-mon[81815]: pgmap v19: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:43.162 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:43.163 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:43.163 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:42 smithi196 ceph-mon[81815]: Reconfiguring osd.1 (monmap changed)... 2024-04-22T15:23:43.163 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:23:43.163 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:43.163 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:42 smithi196 ceph-mon[81815]: Reconfiguring daemon osd.1 on smithi096 2024-04-22T15:23:43.163 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:42 smithi196 ceph-mon[81815]: pgmap v20: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:42 smithi096 ceph-mon[112443]: Reconfiguring osd.1 (monmap changed)... 2024-04-22T15:23:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:23:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:42 smithi096 ceph-mon[112443]: Reconfiguring daemon osd.1 on smithi096 2024-04-22T15:23:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:42 smithi096 ceph-mon[112443]: pgmap v20: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:44 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:44 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:44 smithi096 ceph-mon[112443]: Reconfiguring osd.2 (monmap changed)... 2024-04-22T15:23:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:44 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:23:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:44 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:44.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:44 smithi096 ceph-mon[112443]: Reconfiguring daemon osd.2 on smithi096 2024-04-22T15:23:44.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:44 smithi096 ceph-mon[112443]: pgmap v21: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:44.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:44 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:44.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:44 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:44.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:44 smithi196 ceph-mon[81815]: Reconfiguring osd.2 (monmap changed)... 2024-04-22T15:23:44.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:44 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:23:44.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:44 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:44.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:44 smithi196 ceph-mon[81815]: Reconfiguring daemon osd.2 on smithi096 2024-04-22T15:23:44.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:44 smithi196 ceph-mon[81815]: pgmap v21: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:46.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:46 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:46.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:46 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:46.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:46 smithi096 ceph-mon[112443]: Reconfiguring mds.cephfs.smithi096.dymqlv (monmap changed)... 2024-04-22T15:23:46.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:46 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.dymqlv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:23:46.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:46 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:46.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:46 smithi096 ceph-mon[112443]: Reconfiguring daemon mds.cephfs.smithi096.dymqlv on smithi096 2024-04-22T15:23:46.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:46 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:46.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:46 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:46.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:46 smithi196 ceph-mon[81815]: Reconfiguring mds.cephfs.smithi096.dymqlv (monmap changed)... 2024-04-22T15:23:46.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:46 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.dymqlv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:23:46.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:46 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:46.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:46 smithi196 ceph-mon[81815]: Reconfiguring daemon mds.cephfs.smithi096.dymqlv on smithi096 2024-04-22T15:23:47.616 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:47 smithi096 ceph-mon[112443]: pgmap v22: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:47.616 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:47 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:47.616 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:47 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:47.616 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:47 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.drkkdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:23:47.616 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:47 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:47 smithi196 ceph-mon[81815]: pgmap v22: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:47.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:47 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:47.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:47 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:47.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:47 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.drkkdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:23:47.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:47 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:48.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:48 smithi196 ceph-mon[81815]: Reconfiguring mds.cephfs.smithi096.drkkdi (monmap changed)... 2024-04-22T15:23:48.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:48 smithi196 ceph-mon[81815]: Reconfiguring daemon mds.cephfs.smithi096.drkkdi on smithi096 2024-04-22T15:23:48.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:48 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:48.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:48 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:48.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:48 smithi096 ceph-mon[112443]: Reconfiguring mds.cephfs.smithi096.drkkdi (monmap changed)... 2024-04-22T15:23:48.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:48 smithi096 ceph-mon[112443]: Reconfiguring daemon mds.cephfs.smithi096.drkkdi on smithi096 2024-04-22T15:23:48.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:48 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:48.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:48 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:49.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:49 smithi196 ceph-mon[81815]: pgmap v23: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:49 smithi196 ceph-mon[81815]: Reconfiguring ceph-exporter.smithi196 (monmap changed)... 2024-04-22T15:23:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:49 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:23:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:49 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:23:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:49 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:23:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:49 smithi196 ceph-mon[81815]: Unable to update caps for client.ceph-exporter.smithi196 2024-04-22T15:23:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:49 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi196"}]: dispatch 2024-04-22T15:23:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:49 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:49 smithi196 ceph-mon[81815]: Reconfiguring daemon ceph-exporter.smithi196 on smithi196 2024-04-22T15:23:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:49 smithi096 ceph-mon[112443]: pgmap v23: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:49 smithi096 ceph-mon[112443]: Reconfiguring ceph-exporter.smithi196 (monmap changed)... 2024-04-22T15:23:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:49 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:23:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:49 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:23:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:49 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:23:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:49 smithi096 ceph-mon[112443]: Unable to update caps for client.ceph-exporter.smithi196 2024-04-22T15:23:49.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:49 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi196"}]: dispatch 2024-04-22T15:23:49.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:49 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:49.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:49 smithi096 ceph-mon[112443]: Reconfiguring daemon ceph-exporter.smithi196 on smithi196 2024-04-22T15:23:51.187 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:50 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:51.187 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:50 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:51.187 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:50 smithi196 ceph-mon[81815]: Reconfiguring crash.smithi196 (monmap changed)... 2024-04-22T15:23:51.187 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:50 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi196", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:23:51.187 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:50 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:51.187 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:50 smithi196 ceph-mon[81815]: Reconfiguring daemon crash.smithi196 on smithi196 2024-04-22T15:23:51.188 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:50 smithi196 ceph-mon[81815]: pgmap v24: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:50 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:50 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:50 smithi096 ceph-mon[112443]: Reconfiguring crash.smithi196 (monmap changed)... 2024-04-22T15:23:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:50 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi196", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:23:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:50 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:50 smithi096 ceph-mon[112443]: Reconfiguring daemon crash.smithi196 on smithi196 2024-04-22T15:23:51.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:50 smithi096 ceph-mon[112443]: pgmap v24: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:52.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:52 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:52 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:52 smithi196 ceph-mon[81815]: Reconfiguring mgr.smithi196.gdaegc (monmap changed)... 2024-04-22T15:23:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:52 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:23:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:52 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:23:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:52 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:52 smithi196 ceph-mon[81815]: Reconfiguring daemon mgr.smithi196.gdaegc on smithi196 2024-04-22T15:23:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:52 smithi196 ceph-mon[81815]: pgmap v25: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:52 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:52 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:52 smithi096 ceph-mon[112443]: Reconfiguring mgr.smithi196.gdaegc (monmap changed)... 2024-04-22T15:23:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:52 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi196.gdaegc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:23:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:52 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:23:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:52 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:52 smithi096 ceph-mon[112443]: Reconfiguring daemon mgr.smithi196.gdaegc on smithi196 2024-04-22T15:23:52.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:52 smithi096 ceph-mon[112443]: pgmap v25: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:54 smithi096 ceph-mon[112443]: Reconfiguring mon.smithi196 (monmap changed)... 2024-04-22T15:23:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:23:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:23:54.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:54.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:54 smithi096 ceph-mon[112443]: Reconfiguring daemon mon.smithi196 on smithi196 2024-04-22T15:23:54.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:23:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:54 smithi196 ceph-mon[81815]: Reconfiguring mon.smithi196 (monmap changed)... 2024-04-22T15:23:54.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:23:54.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:23:54.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:54.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:54 smithi196 ceph-mon[81815]: Reconfiguring daemon mon.smithi196 on smithi196 2024-04-22T15:23:54.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:23:55.420 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:55 smithi196 ceph-mon[81815]: pgmap v26: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:55.421 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:55 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:55.421 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:55 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:55.421 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:55 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:23:55.421 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:55 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:55.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:55 smithi096 ceph-mon[112443]: pgmap v26: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:23:55.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:55 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:55.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:55 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:55.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:55 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:23:55.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:55 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:56.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:56 smithi096 ceph-mon[112443]: Reconfiguring osd.3 (monmap changed)... 2024-04-22T15:23:56.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:56 smithi096 ceph-mon[112443]: Reconfiguring daemon osd.3 on smithi196 2024-04-22T15:23:56.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:56 smithi196 ceph-mon[81815]: Reconfiguring osd.3 (monmap changed)... 2024-04-22T15:23:56.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:56 smithi196 ceph-mon[81815]: Reconfiguring daemon osd.3 on smithi196 2024-04-22T15:23:57.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:57 smithi196 ceph-mon[81815]: pgmap v27: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:57.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:57.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:57.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:57 smithi196 ceph-mon[81815]: Reconfiguring osd.4 (monmap changed)... 2024-04-22T15:23:57.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:23:57.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:57.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:57 smithi196 ceph-mon[81815]: Reconfiguring daemon osd.4 on smithi196 2024-04-22T15:23:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:57 smithi096 ceph-mon[112443]: pgmap v27: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:57 smithi096 ceph-mon[112443]: Reconfiguring osd.4 (monmap changed)... 2024-04-22T15:23:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:23:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:57.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:57 smithi096 ceph-mon[112443]: Reconfiguring daemon osd.4 on smithi196 2024-04-22T15:23:57.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.986+0000 7fb01bd79700 1 -- 172.21.15.96:0/3123699915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb014105ab0 msgr2=0x7fb014105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:57.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.986+0000 7fb01bd79700 1 --2- 172.21.15.96:0/3123699915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb014105ab0 0x7fb014105e90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb004009910 tx=0x7fb004009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:57.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.986+0000 7fb01bd79700 1 -- 172.21.15.96:0/3123699915 shutdown_connections 2024-04-22T15:23:57.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.986+0000 7fb01bd79700 1 --2- 172.21.15.96:0/3123699915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0140ffab0 0x7fb0140fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:57.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.986+0000 7fb01bd79700 1 --2- 172.21.15.96:0/3123699915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb014105ab0 0x7fb014105e90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:57.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.986+0000 7fb01bd79700 1 -- 172.21.15.96:0/3123699915 >> 172.21.15.96:0/3123699915 conn(0x7fb0140fb5e0 msgr2=0x7fb0140fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:57.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.986+0000 7fb01bd79700 1 -- 172.21.15.96:0/3123699915 shutdown_connections 2024-04-22T15:23:57.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.986+0000 7fb01bd79700 1 -- 172.21.15.96:0/3123699915 wait complete. 2024-04-22T15:23:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb01bd79700 1 Processor -- start 2024-04-22T15:23:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb01bd79700 1 -- start start 2024-04-22T15:23:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb01bd79700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0140ffab0 0x7fb014195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb01bd79700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb014105ab0 0x7fb014195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb01bd79700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0141965f0 con 0x7fb0140ffab0 2024-04-22T15:23:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb01bd79700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01419a340 con 0x7fb014105ab0 2024-04-22T15:23:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb019b15700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0140ffab0 0x7fb014195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb019b15700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0140ffab0 0x7fb014195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53948/0 (socket says 172.21.15.96:53948) 2024-04-22T15:23:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb019b15700 1 -- 172.21.15.96:0/3060026010 learned_addr learned my addr 172.21.15.96:0/3060026010 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb019314700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb014105ab0 0x7fb014195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb019b15700 1 -- 172.21.15.96:0/3060026010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb014105ab0 msgr2=0x7fb014195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb019b15700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb014105ab0 0x7fb014195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.987+0000 7fb019b15700 1 -- 172.21.15.96:0/3060026010 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0040095f0 con 0x7fb0140ffab0 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.988+0000 7fb019b15700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0140ffab0 0x7fb014195a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb00400bd70 tx=0x7fb00400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.988+0000 7fb00affd700 1 -- 172.21.15.96:0/3060026010 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb00401d070 con 0x7fb0140ffab0 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.988+0000 7fb00affd700 1 -- 172.21.15.96:0/3060026010 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0040056e0 con 0x7fb0140ffab0 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.988+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01419a5c0 con 0x7fb0140ffab0 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.988+0000 7fb00affd700 1 -- 172.21.15.96:0/3060026010 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb00400fe40 con 0x7fb0140ffab0 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.988+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01419aab0 con 0x7fb0140ffab0 2024-04-22T15:23:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.989+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0141080b0 con 0x7fb0140ffab0 2024-04-22T15:23:57.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.989+0000 7fb00affd700 1 -- 172.21.15.96:0/3060026010 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fb00400f680 con 0x7fb0140ffab0 2024-04-22T15:23:57.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.990+0000 7fb00affd700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb00006ed80 0x7fb000071240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:57.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.990+0000 7fb019314700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb00006ed80 0x7fb000071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:57.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.990+0000 7fb00affd700 1 -- 172.21.15.96:0/3060026010 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fb004090470 con 0x7fb0140ffab0 2024-04-22T15:23:57.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.990+0000 7fb019314700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb00006ed80 0x7fb000071240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb014196f80 tx=0x7fb01000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:57.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:57.995+0000 7fb00affd700 1 -- 172.21.15.96:0/3060026010 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb00405cb90 con 0x7fb0140ffab0 2024-04-22T15:23:58.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.215+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb014196d30 con 0x7fb00006ed80 2024-04-22T15:23:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.216+0000 7fb00affd700 1 -- 172.21.15.96:0/3060026010 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fb014196d30 con 0x7fb00006ed80 2024-04-22T15:23:58.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.218+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb00006ed80 msgr2=0x7fb000071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.218+0000 7fb01bd79700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb00006ed80 0x7fb000071240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb014196f80 tx=0x7fb01000b410 comp rx=0 tx=0).stop 2024-04-22T15:23:58.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.218+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0140ffab0 msgr2=0x7fb014195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.218+0000 7fb01bd79700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0140ffab0 0x7fb014195a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb00400bd70 tx=0x7fb00400bda0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.218+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 shutdown_connections 2024-04-22T15:23:58.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.218+0000 7fb01bd79700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb00006ed80 0x7fb000071240 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.218+0000 7fb01bd79700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0140ffab0 0x7fb014195a20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.219+0000 7fb01bd79700 1 --2- 172.21.15.96:0/3060026010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb014105ab0 0x7fb014195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.219+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 >> 172.21.15.96:0/3060026010 conn(0x7fb0140fb5e0 msgr2=0x7fb0140fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:58.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.219+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 shutdown_connections 2024-04-22T15:23:58.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.219+0000 7fb01bd79700 1 -- 172.21.15.96:0/3060026010 wait complete. 2024-04-22T15:23:58.232 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:23:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.323+0000 7f1b14fd3700 1 -- 172.21.15.96:0/3244174309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 msgr2=0x7f1b10105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.323+0000 7f1b14fd3700 1 --2- 172.21.15.96:0/3244174309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 0x7f1b10105f40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b04009910 tx=0x7f1b04009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.323+0000 7f1b14fd3700 1 -- 172.21.15.96:0/3244174309 shutdown_connections 2024-04-22T15:23:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.323+0000 7f1b14fd3700 1 --2- 172.21.15.96:0/3244174309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b100ffad0 0x7f1b100fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.323+0000 7f1b14fd3700 1 --2- 172.21.15.96:0/3244174309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 0x7f1b10105f40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.323+0000 7f1b14fd3700 1 -- 172.21.15.96:0/3244174309 >> 172.21.15.96:0/3244174309 conn(0x7f1b100fb5e0 msgr2=0x7f1b100fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.323+0000 7f1b14fd3700 1 -- 172.21.15.96:0/3244174309 shutdown_connections 2024-04-22T15:23:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.323+0000 7f1b14fd3700 1 -- 172.21.15.96:0/3244174309 wait complete. 2024-04-22T15:23:58.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.324+0000 7f1b14fd3700 1 Processor -- start 2024-04-22T15:23:58.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.324+0000 7f1b14fd3700 1 -- start start 2024-04-22T15:23:58.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.324+0000 7f1b14fd3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b100ffad0 0x7f1b1019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:58.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.324+0000 7f1b14fd3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 0x7f1b1019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:58.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b14fd3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1019d390 con 0x7f1b10105b60 2024-04-22T15:23:58.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b14fd3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b10196970 con 0x7f1b100ffad0 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b0dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 0x7f1b1019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b0dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 0x7f1b1019cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53972/0 (socket says 172.21.15.96:53972) 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b0dd9b700 1 -- 172.21.15.96:0/1593386306 learned_addr learned my addr 172.21.15.96:0/1593386306 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b0e59c700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b100ffad0 0x7f1b1019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b0dd9b700 1 -- 172.21.15.96:0/1593386306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b100ffad0 msgr2=0x7f1b1019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b0dd9b700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b100ffad0 0x7f1b1019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b0dd9b700 1 -- 172.21.15.96:0/1593386306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b040095f0 con 0x7f1b10105b60 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b0dd9b700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 0x7f1b1019cd70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000e920 tx=0x7f1b0000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1aff7fe700 1 -- 172.21.15.96:0/1593386306 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b0000c9d0 con 0x7f1b10105b60 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1aff7fe700 1 -- 172.21.15.96:0/1593386306 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b00010430 con 0x7f1b10105b60 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b10196c50 con 0x7f1b10105b60 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1aff7fe700 1 -- 172.21.15.96:0/1593386306 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b00018630 con 0x7f1b10105b60 2024-04-22T15:23:58.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.325+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b101971a0 con 0x7f1b10105b60 2024-04-22T15:23:58.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.326+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b101090e0 con 0x7f1b10105b60 2024-04-22T15:23:58.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.327+0000 7f1aff7fe700 1 -- 172.21.15.96:0/1593386306 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f1b000105a0 con 0x7f1b10105b60 2024-04-22T15:23:58.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.327+0000 7f1aff7fe700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1af806edf0 0x7f1af80712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:58.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.327+0000 7f1aff7fe700 1 -- 172.21.15.96:0/1593386306 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f1b00014070 con 0x7f1b10105b60 2024-04-22T15:23:58.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.327+0000 7f1b0e59c700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1af806edf0 0x7f1af80712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:58.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.327+0000 7f1b0e59c700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1af806edf0 0x7f1af80712b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400b760 tx=0x7f1b04005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:58.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.332+0000 7f1aff7fe700 1 -- 172.21.15.96:0/1593386306 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1b00054150 con 0x7f1b10105b60 2024-04-22T15:23:58.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.555+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b10197e30 con 0x7f1af806edf0 2024-04-22T15:23:58.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.556+0000 7f1aff7fe700 1 -- 172.21.15.96:0/1593386306 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f1b10197e30 con 0x7f1af806edf0 2024-04-22T15:23:58.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1af806edf0 msgr2=0x7f1af80712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1af806edf0 0x7f1af80712b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400b760 tx=0x7f1b04005a90 comp rx=0 tx=0).stop 2024-04-22T15:23:58.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 msgr2=0x7f1b1019cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 0x7f1b1019cd70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000e920 tx=0x7f1b0000ec30 comp rx=0 tx=0).stop 2024-04-22T15:23:58.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 shutdown_connections 2024-04-22T15:23:58.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1af806edf0 0x7f1af80712b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b100ffad0 0x7f1b1019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 --2- 172.21.15.96:0/1593386306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b10105b60 0x7f1b1019cd70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 >> 172.21.15.96:0/1593386306 conn(0x7f1b100fb5e0 msgr2=0x7f1b10069650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:58.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 shutdown_connections 2024-04-22T15:23:58.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.558+0000 7f1b14fd3700 1 -- 172.21.15.96:0/1593386306 wait complete. 2024-04-22T15:23:58.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.662+0000 7fe285d00700 1 -- 172.21.15.96:0/3472430270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 msgr2=0x7fe2801057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.662+0000 7fe285d00700 1 --2- 172.21.15.96:0/3472430270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 0x7fe2801057f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe270009910 tx=0x7fe270009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:58.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.662+0000 7fe285d00700 1 -- 172.21.15.96:0/3472430270 shutdown_connections 2024-04-22T15:23:58.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.662+0000 7fe285d00700 1 --2- 172.21.15.96:0/3472430270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 0x7fe2801057f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.662+0000 7fe285d00700 1 --2- 172.21.15.96:0/3472430270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe280100df0 0x7fe2801011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.662+0000 7fe285d00700 1 -- 172.21.15.96:0/3472430270 >> 172.21.15.96:0/3472430270 conn(0x7fe2800fc6a0 msgr2=0x7fe2800feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:58.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.662+0000 7fe285d00700 1 -- 172.21.15.96:0/3472430270 shutdown_connections 2024-04-22T15:23:58.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.662+0000 7fe285d00700 1 -- 172.21.15.96:0/3472430270 wait complete. 2024-04-22T15:23:58.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.662+0000 7fe285d00700 1 Processor -- start 2024-04-22T15:23:58.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe285d00700 1 -- start start 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe285d00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe280100df0 0x7fe28019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe285d00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 0x7fe28019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe285d00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28019d370 con 0x7fe2801017a0 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe285d00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe280196950 con 0x7fe280100df0 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe27effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 0x7fe28019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe27effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 0x7fe28019cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53992/0 (socket says 172.21.15.96:53992) 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe27effd700 1 -- 172.21.15.96:0/1875721889 learned_addr learned my addr 172.21.15.96:0/1875721889 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe27f7fe700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe280100df0 0x7fe28019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe27effd700 1 -- 172.21.15.96:0/1875721889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe280100df0 msgr2=0x7fe28019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe27effd700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe280100df0 0x7fe28019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe27effd700 1 -- 172.21.15.96:0/1875721889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2700095f0 con 0x7fe2801017a0 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.663+0000 7fe27f7fe700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe280100df0 0x7fe28019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:23:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.664+0000 7fe27effd700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 0x7fe28019cd50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe27000bd70 tx=0x7fe27000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:58.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.664+0000 7fe27cff9700 1 -- 172.21.15.96:0/1875721889 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe27001d070 con 0x7fe2801017a0 2024-04-22T15:23:58.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.664+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe280196bd0 con 0x7fe2801017a0 2024-04-22T15:23:58.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.664+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2801970c0 con 0x7fe2801017a0 2024-04-22T15:23:58.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.664+0000 7fe27cff9700 1 -- 172.21.15.96:0/1875721889 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2700056e0 con 0x7fe2801017a0 2024-04-22T15:23:58.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.664+0000 7fe27cff9700 1 -- 172.21.15.96:0/1875721889 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe27000fe50 con 0x7fe2801017a0 2024-04-22T15:23:58.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.665+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2600052f0 con 0x7fe2801017a0 2024-04-22T15:23:58.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.665+0000 7fe27cff9700 1 -- 172.21.15.96:0/1875721889 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fe27000f460 con 0x7fe2801017a0 2024-04-22T15:23:58.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.666+0000 7fe27cff9700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe26c06ee00 0x7fe26c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:58.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.666+0000 7fe27cff9700 1 -- 172.21.15.96:0/1875721889 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fe270090400 con 0x7fe2801017a0 2024-04-22T15:23:58.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.666+0000 7fe27f7fe700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe26c06ee00 0x7fe26c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:58.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.666+0000 7fe27f7fe700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe26c06ee00 0x7fe26c0712c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe268005950 tx=0x7fe26800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:58.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.670+0000 7fe27cff9700 1 -- 172.21.15.96:0/1875721889 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe27005cb20 con 0x7fe2801017a0 2024-04-22T15:23:58.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.882+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe260000bc0 con 0x7fe26c06ee00 2024-04-22T15:23:58.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.893+0000 7fe27cff9700 1 -- 172.21.15.96:0/1875721889 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe260000bc0 con 0x7fe26c06ee00 2024-04-22T15:23:58.896 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (2m) 44s ago 13m 25.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (14m) 44s ago 14m 8304k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (13m) 30s ago 13m 8098k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (14m) 44s ago 14m 6656k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 9954127cedf1 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (13m) 30s ago 13m 6665k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 e792c49d674e 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (2m) 44s ago 13m 76.2M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (8m) 44s ago 8m 157M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (8m) 44s ago 8m 90.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (8m) 30s ago 8m 27.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (8m) 30s ago 8m 26.6M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (5m) 44s ago 15m 473M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (3m) 30s ago 12m 448M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:23:58.897 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (59s) 44s ago 15m 27.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (33s) 30s ago 12m 22.6M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (3m) 44s ago 14m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (3m) 30s ago 12m 16.6M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (12m) 44s ago 12m 210M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (11m) 44s ago 11m 218M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (11m) 44s ago 11m 182M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (10m) 30s ago 10m 242M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (10m) 30s ago 10m 213M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (9m) 30s ago 9m 195M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:23:58.898 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (2m) 44s ago 13m 73.0M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:23:58.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe26c06ee00 msgr2=0x7fe26c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe26c06ee00 0x7fe26c0712c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe268005950 tx=0x7fe26800b410 comp rx=0 tx=0).stop 2024-04-22T15:23:58.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 msgr2=0x7fe28019cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:58.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 0x7fe28019cd50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe27000bd70 tx=0x7fe27000bda0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 shutdown_connections 2024-04-22T15:23:58.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe26c06ee00 0x7fe26c0712c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe280100df0 0x7fe28019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 --2- 172.21.15.96:0/1875721889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2801017a0 0x7fe28019cd50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:58.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 >> 172.21.15.96:0/1875721889 conn(0x7fe2800fc6a0 msgr2=0x7fe2800fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:58.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 shutdown_connections 2024-04-22T15:23:58.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.895+0000 7fe285d00700 1 -- 172.21.15.96:0/1875721889 wait complete. 2024-04-22T15:23:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:58 smithi096 ceph-mon[112443]: Reconfiguring osd.5 (monmap changed)... 2024-04-22T15:23:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:23:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:58 smithi096 ceph-mon[112443]: Reconfiguring daemon osd.5 on smithi196 2024-04-22T15:23:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:58 smithi096 ceph-mon[112443]: pgmap v28: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:58.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:58 smithi096 ceph-mon[112443]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:59.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.997+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/3187670128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f80ffaa0 msgr2=0x7fc8f80fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.997+0000 7fc8ff1f3700 1 --2- 172.21.15.96:0/3187670128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f80ffaa0 0x7fc8f80fff20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009910 tx=0x7fc8e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:59.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.997+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/3187670128 shutdown_connections 2024-04-22T15:23:59.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.997+0000 7fc8ff1f3700 1 --2- 172.21.15.96:0/3187670128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f80ffaa0 0x7fc8f80fff20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.997+0000 7fc8ff1f3700 1 --2- 172.21.15.96:0/3187670128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f8105aa0 0x7fc8f8105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.997+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/3187670128 >> 172.21.15.96:0/3187670128 conn(0x7fc8f8076740 msgr2=0x7fc8f80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:59.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.997+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/3187670128 shutdown_connections 2024-04-22T15:23:59.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.997+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/3187670128 wait complete. 2024-04-22T15:23:59.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8ff1f3700 1 Processor -- start 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8ff1f3700 1 -- start start 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8ff1f3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f80ffaa0 0x7fc8f8195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8ff1f3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f8105aa0 0x7fc8f8195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8ff1f3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8069550 con 0x7fc8f8105aa0 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8ff1f3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8196290 con 0x7fc8f80ffaa0 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8f7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f8105aa0 0x7fc8f8195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8f7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f8105aa0 0x7fc8f8195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54012/0 (socket says 172.21.15.96:54012) 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8f7fff700 1 -- 172.21.15.96:0/1685437469 learned_addr learned my addr 172.21.15.96:0/1685437469 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8f7fff700 1 -- 172.21.15.96:0/1685437469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f80ffaa0 msgr2=0x7fc8f8195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8fcf8f700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f80ffaa0 0x7fc8f8195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8f7fff700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f80ffaa0 0x7fc8f8195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8f7fff700 1 -- 172.21.15.96:0/1685437469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e40095f0 con 0x7fc8f8105aa0 2024-04-22T15:23:59.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8f7fff700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f8105aa0 0x7fc8f8195d50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400bd70 tx=0x7fc8e400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:59.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.998+0000 7fc8f5ffb700 1 -- 172.21.15.96:0/1685437469 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e401d070 con 0x7fc8f8105aa0 2024-04-22T15:23:59.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.999+0000 7fc8f5ffb700 1 -- 172.21.15.96:0/1685437469 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8e400f460 con 0x7fc8f8105aa0 2024-04-22T15:23:59.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.999+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f8196510 con 0x7fc8f8105aa0 2024-04-22T15:23:59.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.999+0000 7fc8f5ffb700 1 -- 172.21.15.96:0/1685437469 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e4017630 con 0x7fc8f8105aa0 2024-04-22T15:23:59.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:58.999+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f81a3050 con 0x7fc8f8105aa0 2024-04-22T15:23:59.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.000+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f81080c0 con 0x7fc8f8105aa0 2024-04-22T15:23:59.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.000+0000 7fc8f5ffb700 1 -- 172.21.15.96:0/1685437469 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fc8e400f5d0 con 0x7fc8f8105aa0 2024-04-22T15:23:59.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.001+0000 7fc8f5ffb700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8e806ee00 0x7fc8e80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:59.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.001+0000 7fc8f5ffb700 1 -- 172.21.15.96:0/1685437469 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fc8e4090520 con 0x7fc8f8105aa0 2024-04-22T15:23:59.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.001+0000 7fc8fcf8f700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8e806ee00 0x7fc8e80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:59.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.001+0000 7fc8fcf8f700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8e806ee00 0x7fc8e80712c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec006fd0 tx=0x7fc8ec008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:59.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.006+0000 7fc8f5ffb700 1 -- 172.21.15.96:0/1685437469 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc8e405ccc0 con 0x7fc8f8105aa0 2024-04-22T15:23:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:58 smithi196 ceph-mon[81815]: Reconfiguring osd.5 (monmap changed)... 2024-04-22T15:23:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:23:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:58 smithi196 ceph-mon[81815]: Reconfiguring daemon osd.5 on smithi196 2024-04-22T15:23:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:58 smithi196 ceph-mon[81815]: pgmap v28: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:23:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:58 smithi196 ceph-mon[81815]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:59.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.292+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc8f81011e0 con 0x7fc8f8105aa0 2024-04-22T15:23:59.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.293+0000 7fc8f5ffb700 1 -- 172.21.15.96:0/1685437469 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+733 (secure 0 0 0) 0x7fc8e405c850 con 0x7fc8f8105aa0 2024-04-22T15:23:59.296 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:23:59.296 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 10, 2024-04-22T15:23:59.297 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:23:59.298 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:23:59.298 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.295+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8e806ee00 msgr2=0x7fc8e80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.295+0000 7fc8ff1f3700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8e806ee00 0x7fc8e80712c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec006fd0 tx=0x7fc8ec008040 comp rx=0 tx=0).stop 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.295+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f8105aa0 msgr2=0x7fc8f8195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.295+0000 7fc8ff1f3700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f8105aa0 0x7fc8f8195d50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400bd70 tx=0x7fc8e400be50 comp rx=0 tx=0).stop 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.295+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 shutdown_connections 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.295+0000 7fc8ff1f3700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8e806ee00 0x7fc8e80712c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.295+0000 7fc8ff1f3700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f80ffaa0 0x7fc8f8195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.295+0000 7fc8ff1f3700 1 --2- 172.21.15.96:0/1685437469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f8105aa0 0x7fc8f8195d50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.295+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 >> 172.21.15.96:0/1685437469 conn(0x7fc8f8076740 msgr2=0x7fc8f80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.296+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 shutdown_connections 2024-04-22T15:23:59.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.296+0000 7fc8ff1f3700 1 -- 172.21.15.96:0/1685437469 wait complete. 2024-04-22T15:23:59.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.399+0000 7f8186541700 1 -- 172.21.15.96:0/2994595745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 msgr2=0x7f8180104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.399+0000 7f8186541700 1 --2- 172.21.15.96:0/2994595745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 0x7f8180104770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8170009910 tx=0x7f8170009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:59.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.399+0000 7f8186541700 1 -- 172.21.15.96:0/2994595745 shutdown_connections 2024-04-22T15:23:59.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.399+0000 7f8186541700 1 --2- 172.21.15.96:0/2994595745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 0x7f8180104770 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.399+0000 7f8186541700 1 --2- 172.21.15.96:0/2994595745 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8180100740 0x7f8180100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.399+0000 7f8186541700 1 -- 172.21.15.96:0/2994595745 >> 172.21.15.96:0/2994595745 conn(0x7f8180074d90 msgr2=0x7f81800751a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:59.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.399+0000 7f8186541700 1 -- 172.21.15.96:0/2994595745 shutdown_connections 2024-04-22T15:23:59.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.399+0000 7f8186541700 1 -- 172.21.15.96:0/2994595745 wait complete. 2024-04-22T15:23:59.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f8186541700 1 Processor -- start 2024-04-22T15:23:59.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f8186541700 1 -- start start 2024-04-22T15:23:59.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f8186541700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8180100740 0x7f8180195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:59.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f8186541700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 0x7f8180195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:59.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f8186541700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81801965f0 con 0x7f81801010f0 2024-04-22T15:23:59.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f8186541700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818019a340 con 0x7f8180100740 2024-04-22T15:23:59.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f817f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 0x7f8180195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:59.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f817f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 0x7f8180195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54032/0 (socket says 172.21.15.96:54032) 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f817f7fe700 1 -- 172.21.15.96:0/929735918 learned_addr learned my addr 172.21.15.96:0/929735918 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.400+0000 7f817ffff700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8180100740 0x7f8180195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.401+0000 7f817f7fe700 1 -- 172.21.15.96:0/929735918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8180100740 msgr2=0x7f8180195a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.401+0000 7f817f7fe700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8180100740 0x7f8180195a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.401+0000 7f817f7fe700 1 -- 172.21.15.96:0/929735918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700095f0 con 0x7f81801010f0 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.401+0000 7f817f7fe700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 0x7f8180195f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f817000bd70 tx=0x7f817000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.401+0000 7f817d7fa700 1 -- 172.21.15.96:0/929735918 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f817001d070 con 0x7f81801010f0 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.401+0000 7f817d7fa700 1 -- 172.21.15.96:0/929735918 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81700056e0 con 0x7f81801010f0 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.401+0000 7f817d7fa700 1 -- 172.21.15.96:0/929735918 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f817000fe50 con 0x7f81801010f0 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.401+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818019a5c0 con 0x7f81801010f0 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.401+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818019a970 con 0x7f81801010f0 2024-04-22T15:23:59.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.402+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8180068e30 con 0x7f81801010f0 2024-04-22T15:23:59.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.403+0000 7f817d7fa700 1 -- 172.21.15.96:0/929735918 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f817000f680 con 0x7f81801010f0 2024-04-22T15:23:59.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.403+0000 7f817d7fa700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f816806ee00 0x7f81680712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:59.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.403+0000 7f817d7fa700 1 -- 172.21.15.96:0/929735918 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f81700902f0 con 0x7f81801010f0 2024-04-22T15:23:59.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.403+0000 7f817ffff700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f816806ee00 0x7f81680712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:59.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.403+0000 7f817ffff700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f816806ee00 0x7f81680712c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8174005950 tx=0x7f817400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:59.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.408+0000 7f817d7fa700 1 -- 172.21.15.96:0/929735918 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f817005ca90 con 0x7f81801010f0 2024-04-22T15:23:59.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.657+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f818010cd10 con 0x7f81801010f0 2024-04-22T15:23:59.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.658+0000 7f817d7fa700 1 -- 172.21.15.96:0/929735918 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1901 (secure 0 0 0) 0x7f817005c620 con 0x7f81801010f0 2024-04-22T15:23:59.661 INFO:teuthology.orchestra.run.smithi096.stdout:e20 2024-04-22T15:23:59.661 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:23:59.661 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:23:59.661 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:23:59.661 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 20 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:23:59.662 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:23:59.663 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:23:59.664 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:23:59.664 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:23:59.664 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:23:59.664 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:59.664 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:59.664 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:59.664 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:23:59.664 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:23:59.664 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:23:59.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f816806ee00 msgr2=0x7f81680712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f816806ee00 0x7f81680712c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8174005950 tx=0x7f817400b410 comp rx=0 tx=0).stop 2024-04-22T15:23:59.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 msgr2=0x7f8180195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 0x7f8180195f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f817000bd70 tx=0x7f817000bda0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 shutdown_connections 2024-04-22T15:23:59.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f816806ee00 0x7f81680712c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8180100740 0x7f8180195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 --2- 172.21.15.96:0/929735918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81801010f0 0x7f8180195f60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 >> 172.21.15.96:0/929735918 conn(0x7f8180074d90 msgr2=0x7f81800fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:59.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 shutdown_connections 2024-04-22T15:23:59.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.660+0000 7f8186541700 1 -- 172.21.15.96:0/929735918 wait complete. 2024-04-22T15:23:59.666 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:23:59.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.765+0000 7fdecf36d700 1 -- 172.21.15.96:0/921844820 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec8100df0 msgr2=0x7fdec81011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.765+0000 7fdecf36d700 1 --2- 172.21.15.96:0/921844820 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec8100df0 0x7fdec81011d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdec4009910 tx=0x7fdec4009c20 comp rx=0 tx=0).stop 2024-04-22T15:23:59.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.765+0000 7fdecf36d700 1 -- 172.21.15.96:0/921844820 shutdown_connections 2024-04-22T15:23:59.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.765+0000 7fdecf36d700 1 --2- 172.21.15.96:0/921844820 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdec81017a0 0x7fdec81057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.765+0000 7fdecf36d700 1 --2- 172.21.15.96:0/921844820 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec8100df0 0x7fdec81011d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.765+0000 7fdecf36d700 1 -- 172.21.15.96:0/921844820 >> 172.21.15.96:0/921844820 conn(0x7fdec80fc6a0 msgr2=0x7fdec80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:23:59.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.765+0000 7fdecf36d700 1 -- 172.21.15.96:0/921844820 shutdown_connections 2024-04-22T15:23:59.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecf36d700 1 -- 172.21.15.96:0/921844820 wait complete. 2024-04-22T15:23:59.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecf36d700 1 Processor -- start 2024-04-22T15:23:59.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecf36d700 1 -- start start 2024-04-22T15:23:59.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecf36d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdec8100df0 0x7fdec819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:59.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecf36d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec81017a0 0x7fdec819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:59.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecf36d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec819d350 con 0x7fdec81017a0 2024-04-22T15:23:59.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecf36d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec8196930 con 0x7fdec8100df0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecc908700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec81017a0 0x7fdec819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecc908700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec81017a0 0x7fdec819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54042/0 (socket says 172.21.15.96:54042) 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecc908700 1 -- 172.21.15.96:0/1398973712 learned_addr learned my addr 172.21.15.96:0/1398973712 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.766+0000 7fdecd109700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdec8100df0 0x7fdec819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.767+0000 7fdecc908700 1 -- 172.21.15.96:0/1398973712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdec8100df0 msgr2=0x7fdec819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.767+0000 7fdecc908700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdec8100df0 0x7fdec819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.767+0000 7fdecc908700 1 -- 172.21.15.96:0/1398973712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec40095f0 con 0x7fdec81017a0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.767+0000 7fdecc908700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec81017a0 0x7fdec819cd30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800e8f0 tx=0x7fdeb800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.767+0000 7fdebe7fc700 1 -- 172.21.15.96:0/1398973712 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdeb8009790 con 0x7fdec81017a0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.767+0000 7fdebe7fc700 1 -- 172.21.15.96:0/1398973712 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdeb800c810 con 0x7fdec81017a0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.767+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec8196c10 con 0x7fdec81017a0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.767+0000 7fdebe7fc700 1 -- 172.21.15.96:0/1398973712 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdeb8005580 con 0x7fdec81017a0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.767+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec8197160 con 0x7fdec81017a0 2024-04-22T15:23:59.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.768+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec8109190 con 0x7fdec81017a0 2024-04-22T15:23:59.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.768+0000 7fdebe7fc700 1 -- 172.21.15.96:0/1398973712 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fdeb800c980 con 0x7fdec81017a0 2024-04-22T15:23:59.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.769+0000 7fdebe7fc700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdeb406ee00 0x7fdeb40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:23:59.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.769+0000 7fdecd109700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdeb406ee00 0x7fdeb40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:23:59.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.769+0000 7fdebe7fc700 1 -- 172.21.15.96:0/1398973712 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fdeb8014070 con 0x7fdec81017a0 2024-04-22T15:23:59.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.769+0000 7fdecd109700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdeb406ee00 0x7fdeb40712c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdec40059d0 tx=0x7fdec4005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:23:59.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.774+0000 7fdebe7fc700 1 -- 172.21.15.96:0/1398973712 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdeb805ba60 con 0x7fdec81017a0 2024-04-22T15:23:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:59 smithi096 ceph-mon[112443]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:59 smithi096 ceph-mon[112443]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:23:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:59 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1685437469' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:23:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:59 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:59 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:23:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:59 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.hwpcvg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:23:59.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:59 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:23:59.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:23:59 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/929735918' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:23:59.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.996+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdec8197e60 con 0x7fdeb406ee00 2024-04-22T15:24:00.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.997+0000 7fdebe7fc700 1 -- 172.21.15.96:0/1398973712 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fdec8197e60 con 0x7fdeb406ee00 2024-04-22T15:24:00.000 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:24:00.000 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:24:00.001 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:24:00.001 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:24:00.001 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:24:00.001 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:24:00.001 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:24:00.001 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:24:00.002 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "4/23 daemons upgraded", 2024-04-22T15:24:00.002 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mon daemons", 2024-04-22T15:24:00.002 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:24:00.002 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:24:00.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdeb406ee00 msgr2=0x7fdeb40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdeb406ee00 0x7fdeb40712c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdec40059d0 tx=0x7fdec4005940 comp rx=0 tx=0).stop 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec81017a0 msgr2=0x7fdec819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec81017a0 0x7fdec819cd30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800e8f0 tx=0x7fdeb800ec00 comp rx=0 tx=0).stop 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 shutdown_connections 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdeb406ee00 0x7fdeb40712c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdec8100df0 0x7fdec819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 --2- 172.21.15.96:0/1398973712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdec81017a0 0x7fdec819cd30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 >> 172.21.15.96:0/1398973712 conn(0x7fdec80fc6a0 msgr2=0x7fdec80fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 shutdown_connections 2024-04-22T15:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:23:59.999+0000 7fdecf36d700 1 -- 172.21.15.96:0/1398973712 wait complete. 2024-04-22T15:24:00.103 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:59 smithi196 ceph-mon[81815]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:00.103 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:59 smithi196 ceph-mon[81815]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:00.103 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:59 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1685437469' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:00.103 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:59 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:00.103 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:59 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:00.103 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:59 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.hwpcvg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:24:00.103 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:59 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:00.104 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:23:59 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/929735918' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:24:00.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.102+0000 7ff402d8b700 1 -- 172.21.15.96:0/352982671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc0ff090 msgr2=0x7ff3fc0ff510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:00.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.102+0000 7ff402d8b700 1 --2- 172.21.15.96:0/352982671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc0ff090 0x7ff3fc0ff510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009910 tx=0x7ff3ec009c20 comp rx=0 tx=0).stop 2024-04-22T15:24:00.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.102+0000 7ff402d8b700 1 -- 172.21.15.96:0/352982671 shutdown_connections 2024-04-22T15:24:00.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.102+0000 7ff402d8b700 1 --2- 172.21.15.96:0/352982671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc0ff090 0x7ff3fc0ff510 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.102+0000 7ff402d8b700 1 --2- 172.21.15.96:0/352982671 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3fc105c70 0x7ff3fc106050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.102+0000 7ff402d8b700 1 -- 172.21.15.96:0/352982671 >> 172.21.15.96:0/352982671 conn(0x7ff3fc075f00 msgr2=0x7ff3fc0fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:00.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.102+0000 7ff402d8b700 1 -- 172.21.15.96:0/352982671 shutdown_connections 2024-04-22T15:24:00.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.102+0000 7ff402d8b700 1 -- 172.21.15.96:0/352982671 wait complete. 2024-04-22T15:24:00.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff402d8b700 1 Processor -- start 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff402d8b700 1 -- start start 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff402d8b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3fc0ff090 0x7ff3fc19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff402d8b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc105c70 0x7ff3fc19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff402d8b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc19d490 con 0x7ff3fc105c70 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff402d8b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc196ac0 con 0x7ff3fc0ff090 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff3fbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc105c70 0x7ff3fc19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff3fbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc105c70 0x7ff3fc19ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54060/0 (socket says 172.21.15.96:54060) 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff3fbfff700 1 -- 172.21.15.96:0/58732015 learned_addr learned my addr 172.21.15.96:0/58732015 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff400b27700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3fc0ff090 0x7ff3fc19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff400b27700 1 -- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc105c70 msgr2=0x7ff3fc19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff400b27700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc105c70 0x7ff3fc19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff400b27700 1 -- 172.21.15.96:0/58732015 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0095f0 con 0x7ff3fc0ff090 2024-04-22T15:24:00.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff3fbfff700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc105c70 0x7ff3fc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:24:00.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.103+0000 7ff400b27700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3fc0ff090 0x7ff3fc19c930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400efc0 tx=0x7ff3f400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:00.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.104+0000 7ff3f9ffb700 1 -- 172.21.15.96:0/58732015 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f4018070 con 0x7ff3fc0ff090 2024-04-22T15:24:00.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.104+0000 7ff3f9ffb700 1 -- 172.21.15.96:0/58732015 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3f400f040 con 0x7ff3fc0ff090 2024-04-22T15:24:00.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.104+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc196d40 con 0x7ff3fc0ff090 2024-04-22T15:24:00.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.104+0000 7ff3f9ffb700 1 -- 172.21.15.96:0/58732015 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f4004020 con 0x7ff3fc0ff090 2024-04-22T15:24:00.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.104+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc197290 con 0x7ff3fc0ff090 2024-04-22T15:24:00.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.105+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3fc109260 con 0x7ff3fc0ff090 2024-04-22T15:24:00.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.105+0000 7ff3f9ffb700 1 -- 172.21.15.96:0/58732015 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7ff3f4004ad0 con 0x7ff3fc0ff090 2024-04-22T15:24:00.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.105+0000 7ff3f9ffb700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff3e806ee00 0x7ff3e80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:00.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.106+0000 7ff3f9ffb700 1 -- 172.21.15.96:0/58732015 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6414+0+0 (secure 0 0 0) 0x7ff3f408efe0 con 0x7ff3fc0ff090 2024-04-22T15:24:00.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.106+0000 7ff3fbfff700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff3e806ee00 0x7ff3e80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:00.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.106+0000 7ff3fbfff700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff3e806ee00 0x7ff3e80712c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00b760 tx=0x7ff3ec005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:00.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.111+0000 7ff3f9ffb700 1 -- 172.21.15.96:0/58732015 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff3f405b6a0 con 0x7ff3fc0ff090 2024-04-22T15:24:00.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.395+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff3fc1023b0 con 0x7ff3fc0ff090 2024-04-22T15:24:00.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.395+0000 7ff3f9ffb700 1 -- 172.21.15.96:0/58732015 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7ff3f405b230 con 0x7ff3fc0ff090 2024-04-22T15:24:00.399 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:24:00.399 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:24:00.399 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff3e806ee00 msgr2=0x7ff3e80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff3e806ee00 0x7ff3e80712c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00b760 tx=0x7ff3ec005a90 comp rx=0 tx=0).stop 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3fc0ff090 msgr2=0x7ff3fc19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3fc0ff090 0x7ff3fc19c930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400efc0 tx=0x7ff3f400c3a0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 shutdown_connections 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff3e806ee00 0x7ff3e80712c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3fc0ff090 0x7ff3fc19c930 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 --2- 172.21.15.96:0/58732015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3fc105c70 0x7ff3fc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 >> 172.21.15.96:0/58732015 conn(0x7ff3fc075f00 msgr2=0x7ff3fc0fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:00.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.397+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 shutdown_connections 2024-04-22T15:24:00.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:00.398+0000 7ff402d8b700 1 -- 172.21.15.96:0/58732015 wait complete. 2024-04-22T15:24:01.078 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:00 smithi196 ceph-mon[81815]: Reconfiguring mds.cephfs.smithi196.hwpcvg (monmap changed)... 2024-04-22T15:24:01.078 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:00 smithi196 ceph-mon[81815]: Reconfiguring daemon mds.cephfs.smithi196.hwpcvg on smithi196 2024-04-22T15:24:01.078 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:00 smithi196 ceph-mon[81815]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:01.078 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:00 smithi196 ceph-mon[81815]: pgmap v29: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:01.078 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:00 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/58732015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:24:01.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:00 smithi096 ceph-mon[112443]: Reconfiguring mds.cephfs.smithi196.hwpcvg (monmap changed)... 2024-04-22T15:24:01.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:00 smithi096 ceph-mon[112443]: Reconfiguring daemon mds.cephfs.smithi196.hwpcvg on smithi196 2024-04-22T15:24:01.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:00 smithi096 ceph-mon[112443]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:01.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:00 smithi096 ceph-mon[112443]: pgmap v29: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:01.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:00 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/58732015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:24:02.440 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:02 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:02.440 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:02 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:02.440 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:02 smithi096 ceph-mon[112443]: Reconfiguring mds.cephfs.smithi196.pbvqlq (monmap changed)... 2024-04-22T15:24:02.440 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:02 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.pbvqlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:24:02.440 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:02 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:02.441 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:02 smithi096 ceph-mon[112443]: Reconfiguring daemon mds.cephfs.smithi196.pbvqlq on smithi196 2024-04-22T15:24:02.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:02 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:02.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:02 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:02.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:02 smithi196 ceph-mon[81815]: Reconfiguring mds.cephfs.smithi196.pbvqlq (monmap changed)... 2024-04-22T15:24:02.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:02 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.pbvqlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:24:02.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:02 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:02.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:02 smithi196 ceph-mon[81815]: Reconfiguring daemon mds.cephfs.smithi196.pbvqlq on smithi196 2024-04-22T15:24:03.469 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: pgmap v30: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:03.469 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:03.470 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:03.470 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:03.470 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:03.470 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:03.470 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:03.470 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi096"}]: dispatch 2024-04-22T15:24:03.470 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi096"}]': finished 2024-04-22T15:24:03.470 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi196"}]: dispatch 2024-04-22T15:24:03.470 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi196"}]': finished 2024-04-22T15:24:03.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: pgmap v30: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi096"}]: dispatch 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi096"}]': finished 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi196"}]: dispatch 2024-04-22T15:24:03.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi196"}]': finished 2024-04-22T15:24:04.380 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:04 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all mon 2024-04-22T15:24:04.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:04 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all mon 2024-04-22T15:24:05.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:05 smithi096 ceph-mon[112443]: pgmap v31: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:05.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:05.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi096", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:24:05.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:05.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:05 smithi196 ceph-mon[81815]: pgmap v31: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:05.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:05.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi096", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:24:05.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:06.446 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:06 smithi096 ceph-mon[112443]: Upgrade: Updating crash.smithi096 (1/2) 2024-04-22T15:24:06.446 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:06 smithi096 ceph-mon[112443]: Deploying daemon crash.smithi096 on smithi096 2024-04-22T15:24:06.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:06 smithi196 ceph-mon[81815]: Upgrade: Updating crash.smithi096 (1/2) 2024-04-22T15:24:06.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:06 smithi196 ceph-mon[81815]: Deploying daemon crash.smithi096 on smithi096 2024-04-22T15:24:07.411 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:07 smithi196 ceph-mon[81815]: pgmap v32: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:07.462 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:07 smithi096 ceph-mon[112443]: pgmap v32: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:09.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:09.247 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:09.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:08 smithi096 ceph-mon[112443]: pgmap v33: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:09.248 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:24:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:09.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:09.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:08 smithi196 ceph-mon[81815]: pgmap v33: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:09.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:24:10.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:10 smithi196 ceph-mon[81815]: Upgrade: Updating crash.smithi196 (2/2) 2024-04-22T15:24:10.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:10 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:10 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi196", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:24:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:10 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:10 smithi196 ceph-mon[81815]: Deploying daemon crash.smithi196 on smithi196 2024-04-22T15:24:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:10 smithi196 ceph-mon[81815]: pgmap v34: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:10 smithi096 ceph-mon[112443]: Upgrade: Updating crash.smithi196 (2/2) 2024-04-22T15:24:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:10 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:10 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi196", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:24:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:10 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:10 smithi096 ceph-mon[112443]: Deploying daemon crash.smithi196 on smithi196 2024-04-22T15:24:10.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:10 smithi096 ceph-mon[112443]: pgmap v34: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:13.486 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:13 smithi196 ceph-mon[81815]: pgmap v35: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:13 smithi096 ceph-mon[112443]: pgmap v35: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:14.457 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:14 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:14.458 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:14 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:14.458 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:14 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:14.462 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:14 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:14.462 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:14 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:14.462 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:14 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:15.637 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:15 smithi196 ceph-mon[81815]: pgmap v36: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:15.692 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:15 smithi096 ceph-mon[112443]: pgmap v36: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:17.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:17 smithi096 ceph-mon[112443]: pgmap v37: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:17.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:17 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:17.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:17 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:17 smithi196 ceph-mon[81815]: pgmap v37: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:17 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:17 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:18.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:18 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:18.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:18 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:18.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:18 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:18.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:18 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:19.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:19 smithi096 ceph-mon[112443]: pgmap v38: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:19.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:19.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:19.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:19 smithi196 ceph-mon[81815]: pgmap v38: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:19.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:20.924 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:20 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:20.924 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:20 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:20.924 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:20 smithi196 ceph-mon[81815]: pgmap v39: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:20.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:20.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:20.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:20 smithi096 ceph-mon[112443]: pgmap v39: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:22.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:22.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:22.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:23.179 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:23 smithi096 ceph-mon[112443]: pgmap v40: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:23.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:23 smithi196 ceph-mon[81815]: pgmap v40: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all crash 2024-04-22T15:24:24.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:24.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi096"}]: dispatch 2024-04-22T15:24:24.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi096"}]': finished 2024-04-22T15:24:24.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi196"}]: dispatch 2024-04-22T15:24:24.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi196"}]': finished 2024-04-22T15:24:24.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T15:24:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all crash 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi096"}]: dispatch 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi096"}]': finished 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi196"}]: dispatch 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi196"}]': finished 2024-04-22T15:24:24.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T15:24:25.623 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:25 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T15:24:25.624 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:25 smithi096 ceph-mon[112443]: Upgrade: osd.0 is safe to restart 2024-04-22T15:24:25.624 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:25 smithi096 ceph-mon[112443]: pgmap v41: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:25.624 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:25 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:25.624 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:25 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:24:25.624 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:25 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:25 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T15:24:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:25 smithi196 ceph-mon[81815]: Upgrade: osd.0 is safe to restart 2024-04-22T15:24:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:25 smithi196 ceph-mon[81815]: pgmap v41: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:24:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:25 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:25 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:24:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:25 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:26.635 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:26 smithi096 ceph-mon[112443]: Upgrade: Updating osd.0 2024-04-22T15:24:26.636 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:26 smithi096 ceph-mon[112443]: Deploying daemon osd.0 on smithi096 2024-04-22T15:24:26.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:26 smithi196 ceph-mon[81815]: Upgrade: Updating osd.0 2024-04-22T15:24:26.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:26 smithi196 ceph-mon[81815]: Deploying daemon osd.0 on smithi096 2024-04-22T15:24:27.686 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:27 smithi096 ceph-mon[112443]: pgmap v42: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:27 smithi196 ceph-mon[81815]: pgmap v42: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:28.640 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:28 smithi096 systemd[1]: Stopping Ceph osd.0 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:24:28.986 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:28 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[45698]: 2024-04-22T15:24:28.636+0000 7f414fa34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:24:28.986 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:28 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[45698]: 2024-04-22T15:24:28.636+0000 7f414fa34700 -1 osd.0 48 *** Got signal Terminated *** 2024-04-22T15:24:28.986 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:28 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[45698]: 2024-04-22T15:24:28.636+0000 7f414fa34700 -1 osd.0 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:24:29.651 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:29 smithi096 ceph-mon[112443]: pgmap v43: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:29.651 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:29 smithi096 ceph-mon[112443]: osd.0 marked itself down and dead 2024-04-22T15:24:29.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:29 smithi196 ceph-mon[81815]: pgmap v43: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:29.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:29 smithi196 ceph-mon[81815]: osd.0 marked itself down and dead 2024-04-22T15:24:29.933 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:29 smithi096 podman[123713]: 2024-04-22 15:24:29.648794538 +0000 UTC m=+1.136628602 container died a076589509ddfa27177ccefb2699ea29fbfb6028f5010226f7e81c81dc135b59 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-646b8c7, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-22T15:24:30.236 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:30 smithi096 podman[123713]: 2024-04-22 15:24:30.046231153 +0000 UTC m=+1.534065258 container cleanup a076589509ddfa27177ccefb2699ea29fbfb6028f5010226f7e81c81dc135b59 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , RELEASE=reef-646b8c7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T15:24:30.236 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:30 smithi096 bash[123713]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0 2024-04-22T15:24:30.517 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:30 smithi096 ceph-mon[112443]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:24:30.518 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:30 smithi096 ceph-mon[112443]: osdmap e49: 6 total, 5 up, 6 in 2024-04-22T15:24:30.518 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:30 smithi096 podman[123732]: 2024-04-22 15:24:30.25588943 +0000 UTC m=+0.604952232 container remove a076589509ddfa27177ccefb2699ea29fbfb6028f5010226f7e81c81dc135b59 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-646b8c7, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-22T15:24:30.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.513+0000 7f7c1174c700 1 -- 172.21.15.96:0/134128184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c0ffad0 msgr2=0x7f7c0c0fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:30.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.513+0000 7f7c1174c700 1 --2- 172.21.15.96:0/134128184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c0ffad0 0x7f7c0c0fff50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009910 tx=0x7f7c00009c20 comp rx=0 tx=0).stop 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.513+0000 7f7c1174c700 1 -- 172.21.15.96:0/134128184 shutdown_connections 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.513+0000 7f7c1174c700 1 --2- 172.21.15.96:0/134128184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c0ffad0 0x7f7c0c0fff50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.513+0000 7f7c1174c700 1 --2- 172.21.15.96:0/134128184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c0c105b60 0x7f7c0c105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.513+0000 7f7c1174c700 1 -- 172.21.15.96:0/134128184 >> 172.21.15.96:0/134128184 conn(0x7f7c0c0fb600 msgr2=0x7f7c0c0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.513+0000 7f7c1174c700 1 -- 172.21.15.96:0/134128184 shutdown_connections 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.513+0000 7f7c1174c700 1 -- 172.21.15.96:0/134128184 wait complete. 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.514+0000 7f7c1174c700 1 Processor -- start 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.514+0000 7f7c1174c700 1 -- start start 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.514+0000 7f7c1174c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c0c0ffad0 0x7f7c0c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.514+0000 7f7c1174c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c105b60 0x7f7c0c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.514+0000 7f7c1174c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c198540 con 0x7f7c0c105b60 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.514+0000 7f7c1174c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c198680 con 0x7f7c0c0ffad0 2024-04-22T15:24:30.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c0a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c105b60 0x7f7c0c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c0a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c105b60 0x7f7c0c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54170/0 (socket says 172.21.15.96:54170) 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c0a7fc700 1 -- 172.21.15.96:0/4050813019 learned_addr learned my addr 172.21.15.96:0/4050813019 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c0affd700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c0c0ffad0 0x7f7c0c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c0a7fc700 1 -- 172.21.15.96:0/4050813019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c0c0ffad0 msgr2=0x7f7c0c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c0a7fc700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c0c0ffad0 0x7f7c0c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c0a7fc700 1 -- 172.21.15.96:0/4050813019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c000095f0 con 0x7f7c0c105b60 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c0a7fc700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c105b60 0x7f7c0c197f70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000bd70 tx=0x7f7c0000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7bf3fff700 1 -- 172.21.15.96:0/4050813019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c0001d070 con 0x7f7c0c105b60 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7bf3fff700 1 -- 172.21.15.96:0/4050813019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c00005700 con 0x7f7c0c105b60 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c1a0aa0 con 0x7f7c0c105b60 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7bf3fff700 1 -- 172.21.15.96:0/4050813019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c00017400 con 0x7f7c0c105b60 2024-04-22T15:24:30.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.515+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c1a0e00 con 0x7f7c0c105b60 2024-04-22T15:24:30.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.516+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0c1080f0 con 0x7f7c0c105b60 2024-04-22T15:24:30.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.517+0000 7f7bf3fff700 1 -- 172.21.15.96:0/4050813019 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f7c00003a60 con 0x7f7c0c105b60 2024-04-22T15:24:30.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.518+0000 7f7bf3fff700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7bf406ee00 0x7f7bf40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:30.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.518+0000 7f7bf3fff700 1 -- 172.21.15.96:0/4050813019 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f7c00090420 con 0x7f7c0c105b60 2024-04-22T15:24:30.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.518+0000 7f7c0affd700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7bf406ee00 0x7f7bf40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:30.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.518+0000 7f7c0affd700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7bf406ee00 0x7f7bf40712c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009a10 tx=0x7f7bfc009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:30.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.523+0000 7f7bf3fff700 1 -- 172.21.15.96:0/4050813019 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7c00059090 con 0x7f7c0c105b60 2024-04-22T15:24:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:30 smithi196 ceph-mon[81815]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:24:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:30 smithi196 ceph-mon[81815]: osdmap e49: 6 total, 5 up, 6 in 2024-04-22T15:24:30.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.765+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c0c10cd50 con 0x7f7bf406ee00 2024-04-22T15:24:30.769 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:30 smithi096 podman[123843]: 2024-04-22 15:24:30.595808024 +0000 UTC m=+0.023807787 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:24:30.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.767+0000 7f7bf3fff700 1 -- 172.21.15.96:0/4050813019 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7c0c10cd50 con 0x7f7bf406ee00 2024-04-22T15:24:30.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7bf406ee00 msgr2=0x7f7bf40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:30.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7bf406ee00 0x7f7bf40712c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009a10 tx=0x7f7bfc009190 comp rx=0 tx=0).stop 2024-04-22T15:24:30.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c105b60 msgr2=0x7f7c0c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:30.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c105b60 0x7f7c0c197f70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000bd70 tx=0x7f7c0000bda0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 shutdown_connections 2024-04-22T15:24:30.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7bf406ee00 0x7f7bf40712c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c0c0ffad0 0x7f7c0c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 --2- 172.21.15.96:0/4050813019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c105b60 0x7f7c0c197f70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 >> 172.21.15.96:0/4050813019 conn(0x7f7c0c0fb600 msgr2=0x7f7c0c0fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:30.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 shutdown_connections 2024-04-22T15:24:30.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.769+0000 7f7c1174c700 1 -- 172.21.15.96:0/4050813019 wait complete. 2024-04-22T15:24:30.782 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:24:30.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.877+0000 7f020f6b9700 1 -- 172.21.15.96:0/2276809148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 msgr2=0x7f02080fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:30.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.877+0000 7f020f6b9700 1 --2- 172.21.15.96:0/2276809148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 0x7f02080fff20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0204009910 tx=0x7f0204009c20 comp rx=0 tx=0).stop 2024-04-22T15:24:30.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.877+0000 7f020f6b9700 1 -- 172.21.15.96:0/2276809148 shutdown_connections 2024-04-22T15:24:30.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.877+0000 7f020f6b9700 1 --2- 172.21.15.96:0/2276809148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 0x7f02080fff20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.877+0000 7f020f6b9700 1 --2- 172.21.15.96:0/2276809148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0208105aa0 0x7f0208105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.877+0000 7f020f6b9700 1 -- 172.21.15.96:0/2276809148 >> 172.21.15.96:0/2276809148 conn(0x7f0208076740 msgr2=0x7f02080fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:30.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.877+0000 7f020f6b9700 1 -- 172.21.15.96:0/2276809148 shutdown_connections 2024-04-22T15:24:30.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.877+0000 7f020f6b9700 1 -- 172.21.15.96:0/2276809148 wait complete. 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020f6b9700 1 Processor -- start 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020f6b9700 1 -- start start 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020f6b9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 0x7f02080fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020f6b9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0208105aa0 0x7f02080fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020f6b9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02080fc6a0 con 0x7f02080ffaa0 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020f6b9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02080fc810 con 0x7f0208105aa0 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020d455700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 0x7f02080fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020d455700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 0x7f02080fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54186/0 (socket says 172.21.15.96:54186) 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020d455700 1 -- 172.21.15.96:0/1110376308 learned_addr learned my addr 172.21.15.96:0/1110376308 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020cc54700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0208105aa0 0x7f02080fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020d455700 1 -- 172.21.15.96:0/1110376308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0208105aa0 msgr2=0x7f02080fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020d455700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0208105aa0 0x7f02080fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020d455700 1 -- 172.21.15.96:0/1110376308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02040095f0 con 0x7f02080ffaa0 2024-04-22T15:24:30.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.878+0000 7f020d455700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 0x7f02080fbc20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f01f800c930 tx=0x7f01f800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:30.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.879+0000 7f01fe7fc700 1 -- 172.21.15.96:0/1110376308 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01f8007ad0 con 0x7f02080ffaa0 2024-04-22T15:24:30.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.879+0000 7f01fe7fc700 1 -- 172.21.15.96:0/1110376308 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01f800ce80 con 0x7f02080ffaa0 2024-04-22T15:24:30.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.879+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02081a2c70 con 0x7f02080ffaa0 2024-04-22T15:24:30.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.879+0000 7f01fe7fc700 1 -- 172.21.15.96:0/1110376308 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01f8018630 con 0x7f02080ffaa0 2024-04-22T15:24:30.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.879+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02081a2fd0 con 0x7f02080ffaa0 2024-04-22T15:24:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.880+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02081011e0 con 0x7f02080ffaa0 2024-04-22T15:24:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.880+0000 7f01fe7fc700 1 -- 172.21.15.96:0/1110376308 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f01f8007c30 con 0x7f02080ffaa0 2024-04-22T15:24:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.880+0000 7f01fe7fc700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f01f406ee00 0x7f01f40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.881+0000 7f01fe7fc700 1 -- 172.21.15.96:0/1110376308 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f01f808f0c0 con 0x7f02080ffaa0 2024-04-22T15:24:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.881+0000 7f020cc54700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f01f406ee00 0x7f01f40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.881+0000 7f020cc54700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f01f406ee00 0x7f01f40712c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f02080fd350 tx=0x7f020401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:30.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:30.885+0000 7f01fe7fc700 1 -- 172.21.15.96:0/1110376308 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f01f8057cb0 con 0x7f02080ffaa0 2024-04-22T15:24:31.112 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:30 smithi096 podman[123843]: 2024-04-22 15:24:30.823847481 +0000 UTC m=+0.251847229 container create a93e231f2816da81e7cd50bb590bfec92ce54cfcd8ac2eefcccdb2fa3d0dc00e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T15:24:31.112 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:31 smithi096 podman[123843]: 2024-04-22 15:24:31.109887637 +0000 UTC m=+0.537887387 container init a93e231f2816da81e7cd50bb590bfec92ce54cfcd8ac2eefcccdb2fa3d0dc00e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:24:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.113+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f020810cca0 con 0x7f01f406ee00 2024-04-22T15:24:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.114+0000 7f01fe7fc700 1 -- 172.21.15.96:0/1110376308 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f020810cca0 con 0x7f01f406ee00 2024-04-22T15:24:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f01f406ee00 msgr2=0x7f01f40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f01f406ee00 0x7f01f40712c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f02080fd350 tx=0x7f020401a040 comp rx=0 tx=0).stop 2024-04-22T15:24:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 msgr2=0x7f02080fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 0x7f02080fbc20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f01f800c930 tx=0x7f01f800cc40 comp rx=0 tx=0).stop 2024-04-22T15:24:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 shutdown_connections 2024-04-22T15:24:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f01f406ee00 0x7f01f40712c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02080ffaa0 0x7f02080fbc20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 --2- 172.21.15.96:0/1110376308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0208105aa0 0x7f02080fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 >> 172.21.15.96:0/1110376308 conn(0x7f0208076740 msgr2=0x7f02080687a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 shutdown_connections 2024-04-22T15:24:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.118+0000 7f020f6b9700 1 -- 172.21.15.96:0/1110376308 wait complete. 2024-04-22T15:24:31.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.228+0000 7f84a90df700 1 -- 172.21.15.96:0/224504824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4102680 msgr2=0x7f84a410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.228+0000 7f84a90df700 1 --2- 172.21.15.96:0/224504824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4102680 0x7f84a410ab70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8498009910 tx=0x7f8498009c20 comp rx=0 tx=0).stop 2024-04-22T15:24:31.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.228+0000 7f84a90df700 1 -- 172.21.15.96:0/224504824 shutdown_connections 2024-04-22T15:24:31.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.228+0000 7f84a90df700 1 --2- 172.21.15.96:0/224504824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4102680 0x7f84a410ab70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.228+0000 7f84a90df700 1 --2- 172.21.15.96:0/224504824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84a4101d60 0x7f84a4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.228+0000 7f84a90df700 1 -- 172.21.15.96:0/224504824 >> 172.21.15.96:0/224504824 conn(0x7f84a4076740 msgr2=0x7f84a40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:31.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.228+0000 7f84a90df700 1 -- 172.21.15.96:0/224504824 shutdown_connections 2024-04-22T15:24:31.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.228+0000 7f84a90df700 1 -- 172.21.15.96:0/224504824 wait complete. 2024-04-22T15:24:31.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.229+0000 7f84a90df700 1 Processor -- start 2024-04-22T15:24:31.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a90df700 1 -- start start 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a90df700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4101d60 0x7f84a419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a90df700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84a4102680 0x7f84a419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a90df700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a419d2f0 con 0x7f84a4101d60 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a90df700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4196960 con 0x7f84a4102680 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a2d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4101d60 0x7f84a419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a2d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4101d60 0x7f84a419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54202/0 (socket says 172.21.15.96:54202) 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a2d9d700 1 -- 172.21.15.96:0/2399284500 learned_addr learned my addr 172.21.15.96:0/2399284500 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a259c700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84a4102680 0x7f84a419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a2d9d700 1 -- 172.21.15.96:0/2399284500 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84a4102680 msgr2=0x7f84a419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a2d9d700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84a4102680 0x7f84a419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.230+0000 7f84a2d9d700 1 -- 172.21.15.96:0/2399284500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84980095f0 con 0x7f84a4101d60 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.231+0000 7f84a2d9d700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4101d60 0x7f84a419c820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f849400c930 tx=0x7f849400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:31.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.231+0000 7f848bfff700 1 -- 172.21.15.96:0/2399284500 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8494007ad0 con 0x7f84a4101d60 2024-04-22T15:24:31.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.231+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a4196c40 con 0x7f84a4101d60 2024-04-22T15:24:31.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.231+0000 7f848bfff700 1 -- 172.21.15.96:0/2399284500 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f849400ce80 con 0x7f84a4101d60 2024-04-22T15:24:31.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.231+0000 7f848bfff700 1 -- 172.21.15.96:0/2399284500 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8494018630 con 0x7f84a4101d60 2024-04-22T15:24:31.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.231+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a4197160 con 0x7f84a4101d60 2024-04-22T15:24:31.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.232+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a4105480 con 0x7f84a4101d60 2024-04-22T15:24:31.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.233+0000 7f848bfff700 1 -- 172.21.15.96:0/2399284500 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f8494007c30 con 0x7f84a4101d60 2024-04-22T15:24:31.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.233+0000 7f848bfff700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f848c06edf0 0x7f848c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:31.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.233+0000 7f848bfff700 1 -- 172.21.15.96:0/2399284500 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f849408f190 con 0x7f84a4101d60 2024-04-22T15:24:31.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.233+0000 7f84a259c700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f848c06edf0 0x7f848c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:31.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.233+0000 7f84a259c700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f848c06edf0 0x7f848c0712b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8498009fd0 tx=0x7f849801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:31.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.237+0000 7f848bfff700 1 -- 172.21.15.96:0/2399284500 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f849405b9d0 con 0x7f84a4101d60 2024-04-22T15:24:31.426 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:31 smithi096 ceph-mon[112443]: pgmap v45: 65 pgs: 9 stale+active+clean, 56 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T15:24:31.454 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:31 smithi096 ceph-mon[112443]: osdmap e50: 6 total, 5 up, 6 in 2024-04-22T15:24:31.454 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:31 smithi096 podman[123843]: 2024-04-22 15:24:31.114360822 +0000 UTC m=+0.542360577 container start a93e231f2816da81e7cd50bb590bfec92ce54cfcd8ac2eefcccdb2fa3d0dc00e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:24:31.454 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:31 smithi096 podman[123843]: 2024-04-22 15:24:31.268850466 +0000 UTC m=+0.696850222 container attach a93e231f2816da81e7cd50bb590bfec92ce54cfcd8ac2eefcccdb2fa3d0dc00e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5) 2024-04-22T15:24:31.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.456+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f84a4197e90 con 0x7f848c06edf0 2024-04-22T15:24:31.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.468+0000 7f848bfff700 1 -- 172.21.15.96:0/2399284500 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f84a4197e90 con 0x7f848c06edf0 2024-04-22T15:24:31.471 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:24:31.471 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (3m) 12s ago 14m 25.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:24:31.471 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (14m) 12s ago 14m 8384k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (13m) 15s ago 13m 8166k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (23s) 12s ago 14m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (18s) 15s ago 13m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (2m) 12s ago 14m 83.3M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (8m) 12s ago 8m 155M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (8m) 12s ago 8m 90.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (8m) 15s ago 8m 27.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (8m) 15s ago 8m 26.3M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (5m) 12s ago 15m 500M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (4m) 15s ago 13m 450M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (92s) 12s ago 16m 43.2M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (66s) 15s ago 13m 35.4M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (4m) 12s ago 14m 16.8M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:24:31.472 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (4m) 15s ago 13m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:24:31.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (12m) 12s ago 12m 210M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 a076589509dd 2024-04-22T15:24:31.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (12m) 12s ago 12m 219M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:24:31.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (11m) 12s ago 11m 183M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:24:31.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (11m) 15s ago 11m 242M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:24:31.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (10m) 15s ago 10m 213M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:24:31.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (10m) 15s ago 10m 196M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:24:31.473 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (3m) 12s ago 14m 76.6M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:24:31.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.470+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f848c06edf0 msgr2=0x7f848c0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.470+0000 7f84a90df700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f848c06edf0 0x7f848c0712b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8498009fd0 tx=0x7f849801a040 comp rx=0 tx=0).stop 2024-04-22T15:24:31.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.470+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4101d60 msgr2=0x7f84a419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.470+0000 7f84a90df700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4101d60 0x7f84a419c820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f849400c930 tx=0x7f849400cc40 comp rx=0 tx=0).stop 2024-04-22T15:24:31.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.471+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 shutdown_connections 2024-04-22T15:24:31.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.471+0000 7f84a90df700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f848c06edf0 0x7f848c0712b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.471+0000 7f84a90df700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a4101d60 0x7f84a419c820 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.471+0000 7f84a90df700 1 --2- 172.21.15.96:0/2399284500 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84a4102680 0x7f84a419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.471+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 >> 172.21.15.96:0/2399284500 conn(0x7f84a4076740 msgr2=0x7f84a4103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:31.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.471+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 shutdown_connections 2024-04-22T15:24:31.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.471+0000 7f84a90df700 1 -- 172.21.15.96:0/2399284500 wait complete. 2024-04-22T15:24:31.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.577+0000 7f5308f8b700 1 -- 172.21.15.96:0/2374563431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 msgr2=0x7f53040fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.577+0000 7f5308f8b700 1 --2- 172.21.15.96:0/2374563431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 0x7f53040fff30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009910 tx=0x7f52f8009c20 comp rx=0 tx=0).stop 2024-04-22T15:24:31.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.577+0000 7f5308f8b700 1 -- 172.21.15.96:0/2374563431 shutdown_connections 2024-04-22T15:24:31.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.577+0000 7f5308f8b700 1 --2- 172.21.15.96:0/2374563431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 0x7f53040fff30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.577+0000 7f5308f8b700 1 --2- 172.21.15.96:0/2374563431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5304105ab0 0x7f5304105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.577+0000 7f5308f8b700 1 -- 172.21.15.96:0/2374563431 >> 172.21.15.96:0/2374563431 conn(0x7f53040fb5e0 msgr2=0x7f53040fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:31.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.577+0000 7f5308f8b700 1 -- 172.21.15.96:0/2374563431 shutdown_connections 2024-04-22T15:24:31.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.577+0000 7f5308f8b700 1 -- 172.21.15.96:0/2374563431 wait complete. 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f5308f8b700 1 Processor -- start 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f5308f8b700 1 -- start start 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f5308f8b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 0x7f530419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f5308f8b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5304105ab0 0x7f530419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f5308f8b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530419d440 con 0x7f53040ffab0 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f5308f8b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5304196a20 con 0x7f5304105ab0 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f530259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 0x7f530419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f530259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 0x7f530419c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54226/0 (socket says 172.21.15.96:54226) 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f530259c700 1 -- 172.21.15.96:0/89421243 learned_addr learned my addr 172.21.15.96:0/89421243 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.578+0000 7f5301d9b700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5304105ab0 0x7f530419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.579+0000 7f530259c700 1 -- 172.21.15.96:0/89421243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5304105ab0 msgr2=0x7f530419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.579+0000 7f530259c700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5304105ab0 0x7f530419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.579+0000 7f530259c700 1 -- 172.21.15.96:0/89421243 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f80095f0 con 0x7f53040ffab0 2024-04-22T15:24:31.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.579+0000 7f530259c700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 0x7f530419c8e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f52f400b4b0 tx=0x7f52f400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:31.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.579+0000 7f52f37fe700 1 -- 172.21.15.96:0/89421243 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52f4011800 con 0x7f53040ffab0 2024-04-22T15:24:31.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.579+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5304196d00 con 0x7f53040ffab0 2024-04-22T15:24:31.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.579+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5304197250 con 0x7f53040ffab0 2024-04-22T15:24:31.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.579+0000 7f52f37fe700 1 -- 172.21.15.96:0/89421243 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52f4011e20 con 0x7f53040ffab0 2024-04-22T15:24:31.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.579+0000 7f52f37fe700 1 -- 172.21.15.96:0/89421243 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52f400f2e0 con 0x7f53040ffab0 2024-04-22T15:24:31.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.580+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e40052f0 con 0x7f53040ffab0 2024-04-22T15:24:31.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.581+0000 7f52f37fe700 1 -- 172.21.15.96:0/89421243 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f52f4010380 con 0x7f53040ffab0 2024-04-22T15:24:31.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.581+0000 7f52f37fe700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f52ec06ee00 0x7f52ec0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:31.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.581+0000 7f52f37fe700 1 -- 172.21.15.96:0/89421243 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f52f408e940 con 0x7f53040ffab0 2024-04-22T15:24:31.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.581+0000 7f5301d9b700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f52ec06ee00 0x7f52ec0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:31.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.582+0000 7f5301d9b700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f52ec06ee00 0x7f52ec0712c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5304197be0 tx=0x7f52f801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:31.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.586+0000 7f52f37fe700 1 -- 172.21.15.96:0/89421243 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f52f405b1f0 con 0x7f53040ffab0 2024-04-22T15:24:31.716 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:31 smithi096 podman[123843]: 2024-04-22 15:24:31.423524507 +0000 UTC m=+0.851524265 container died a93e231f2816da81e7cd50bb590bfec92ce54cfcd8ac2eefcccdb2fa3d0dc00e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS) 2024-04-22T15:24:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:31 smithi196 ceph-mon[81815]: pgmap v45: 65 pgs: 9 stale+active+clean, 56 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T15:24:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:31 smithi196 ceph-mon[81815]: osdmap e50: 6 total, 5 up, 6 in 2024-04-22T15:24:31.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.909+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f52e4005870 con 0x7f53040ffab0 2024-04-22T15:24:31.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.909+0000 7f52f37fe700 1 -- 172.21.15.96:0/89421243 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+732 (secure 0 0 0) 0x7f52f405ad80 con 0x7f53040ffab0 2024-04-22T15:24:31.913 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:24:31.913 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:24:31.913 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 5 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 9, 2024-04-22T15:24:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:24:31.915 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:24:31.915 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:24:31.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f52ec06ee00 msgr2=0x7f52ec0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f52ec06ee00 0x7f52ec0712c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5304197be0 tx=0x7f52f801a040 comp rx=0 tx=0).stop 2024-04-22T15:24:31.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 msgr2=0x7f530419c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:31.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 0x7f530419c8e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f52f400b4b0 tx=0x7f52f400b7c0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 shutdown_connections 2024-04-22T15:24:31.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f52ec06ee00 0x7f52ec0712c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53040ffab0 0x7f530419c8e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 --2- 172.21.15.96:0/89421243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5304105ab0 0x7f530419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:31.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 >> 172.21.15.96:0/89421243 conn(0x7f53040fb5e0 msgr2=0x7f5304104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:31.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 shutdown_connections 2024-04-22T15:24:31.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:31.912+0000 7f5308f8b700 1 -- 172.21.15.96:0/89421243 wait complete. 2024-04-22T15:24:32.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.020+0000 7f23b5252700 1 -- 172.21.15.96:0/3340617778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b00ffaa0 msgr2=0x7f23b00fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.020+0000 7f23b5252700 1 --2- 172.21.15.96:0/3340617778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b00ffaa0 0x7f23b00fff20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009910 tx=0x7f23a4009c20 comp rx=0 tx=0).stop 2024-04-22T15:24:32.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.021+0000 7f23b5252700 1 -- 172.21.15.96:0/3340617778 shutdown_connections 2024-04-22T15:24:32.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.021+0000 7f23b5252700 1 --2- 172.21.15.96:0/3340617778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b00ffaa0 0x7f23b00fff20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.021+0000 7f23b5252700 1 --2- 172.21.15.96:0/3340617778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23b0105aa0 0x7f23b0105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.021+0000 7f23b5252700 1 -- 172.21.15.96:0/3340617778 >> 172.21.15.96:0/3340617778 conn(0x7f23b0076740 msgr2=0x7f23b00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:32.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.021+0000 7f23b5252700 1 -- 172.21.15.96:0/3340617778 shutdown_connections 2024-04-22T15:24:32.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.021+0000 7f23b5252700 1 -- 172.21.15.96:0/3340617778 wait complete. 2024-04-22T15:24:32.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.021+0000 7f23b5252700 1 Processor -- start 2024-04-22T15:24:32.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23b5252700 1 -- start start 2024-04-22T15:24:32.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23b5252700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23b00ffaa0 0x7f23b019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:32.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23b5252700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b0105aa0 0x7f23b019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:32.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23b5252700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b019d320 con 0x7f23b0105aa0 2024-04-22T15:24:32.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23b5252700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0196900 con 0x7f23b00ffaa0 2024-04-22T15:24:32.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b0105aa0 0x7f23b019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b0105aa0 0x7f23b019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54258/0 (socket says 172.21.15.96:54258) 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23ae59c700 1 -- 172.21.15.96:0/449532791 learned_addr learned my addr 172.21.15.96:0/449532791 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23ae59c700 1 -- 172.21.15.96:0/449532791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23b00ffaa0 msgr2=0x7f23b019c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23ae59c700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23b00ffaa0 0x7f23b019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23ae59c700 1 -- 172.21.15.96:0/449532791 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40095f0 con 0x7f23b0105aa0 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f23ae59c700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b0105aa0 0x7f23b019cd00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f23a400bd70 tx=0x7f23a400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f2397fff700 1 -- 172.21.15.96:0/449532791 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23a401d070 con 0x7f23b0105aa0 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.022+0000 7f2397fff700 1 -- 172.21.15.96:0/449532791 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23a40056e0 con 0x7f23b0105aa0 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.023+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b0196b80 con 0x7f23b0105aa0 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.023+0000 7f2397fff700 1 -- 172.21.15.96:0/449532791 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23a4017400 con 0x7f23b0105aa0 2024-04-22T15:24:32.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.023+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b0197070 con 0x7f23b0105aa0 2024-04-22T15:24:32.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.024+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b01011e0 con 0x7f23b0105aa0 2024-04-22T15:24:32.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.025+0000 7f2397fff700 1 -- 172.21.15.96:0/449532791 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f23a4003a60 con 0x7f23b0105aa0 2024-04-22T15:24:32.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.025+0000 7f2397fff700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f239806ee00 0x7f23980712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:32.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.025+0000 7f2397fff700 1 -- 172.21.15.96:0/449532791 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f23a4090430 con 0x7f23b0105aa0 2024-04-22T15:24:32.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.025+0000 7f23aed9d700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f239806ee00 0x7f23980712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:32.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.026+0000 7f23aed9d700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f239806ee00 0x7f23980712c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f23a0005950 tx=0x7f23a00058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:32.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.030+0000 7f2397fff700 1 -- 172.21.15.96:0/449532791 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f23a405cbd0 con 0x7f23b0105aa0 2024-04-22T15:24:32.290 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:32 smithi096 podman[123917]: 2024-04-22 15:24:32.132441407 +0000 UTC m=+0.703454936 container remove a93e231f2816da81e7cd50bb590bfec92ce54cfcd8ac2eefcccdb2fa3d0dc00e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T15:24:32.290 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:32 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-22T15:24:32.290 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:32 smithi096 systemd[1]: Stopped Ceph osd.0 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:24:32.290 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:32 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.0.service: Consumed 53.006s CPU time. 2024-04-22T15:24:32.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.299+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f23b004ebb0 con 0x7f23b0105aa0 2024-04-22T15:24:32.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.299+0000 7f2397fff700 1 -- 172.21.15.96:0/449532791 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1901 (secure 0 0 0) 0x7f23a405c760 con 0x7f23b0105aa0 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:e20 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 20 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:24:32.303 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:24:32.304 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:24:32.305 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:24:32.306 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:24:32.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f239806ee00 msgr2=0x7f23980712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f239806ee00 0x7f23980712c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f23a0005950 tx=0x7f23a00058e0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b0105aa0 msgr2=0x7f23b019cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b0105aa0 0x7f23b019cd00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f23a400bd70 tx=0x7f23a400be50 comp rx=0 tx=0).stop 2024-04-22T15:24:32.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 shutdown_connections 2024-04-22T15:24:32.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f239806ee00 0x7f23980712c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23b00ffaa0 0x7f23b019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 --2- 172.21.15.96:0/449532791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23b0105aa0 0x7f23b019cd00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 >> 172.21.15.96:0/449532791 conn(0x7f23b0076740 msgr2=0x7f23b0068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:32.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 shutdown_connections 2024-04-22T15:24:32.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.302+0000 7f23b5252700 1 -- 172.21.15.96:0/449532791 wait complete. 2024-04-22T15:24:32.307 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.416+0000 7f172b9e0700 1 -- 172.21.15.96:0/53278681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 msgr2=0x7f17241021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.416+0000 7f172b9e0700 1 --2- 172.21.15.96:0/53278681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 0x7f17241021e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1720009910 tx=0x7f1720009c20 comp rx=0 tx=0).stop 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.416+0000 7f172b9e0700 1 -- 172.21.15.96:0/53278681 shutdown_connections 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.416+0000 7f172b9e0700 1 --2- 172.21.15.96:0/53278681 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1724102720 0x7f172410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.416+0000 7f172b9e0700 1 --2- 172.21.15.96:0/53278681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 0x7f17241021e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.416+0000 7f172b9e0700 1 -- 172.21.15.96:0/53278681 >> 172.21.15.96:0/53278681 conn(0x7f1724076740 msgr2=0x7f17240fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.417+0000 7f172b9e0700 1 -- 172.21.15.96:0/53278681 shutdown_connections 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.417+0000 7f172b9e0700 1 -- 172.21.15.96:0/53278681 wait complete. 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.417+0000 7f172b9e0700 1 Processor -- start 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.417+0000 7f172b9e0700 1 -- start start 2024-04-22T15:24:32.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.417+0000 7f172b9e0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 0x7f172419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.417+0000 7f172b9e0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1724102720 0x7f172419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172b9e0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172419d2d0 con 0x7f1724101e00 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172b9e0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1724196940 con 0x7f1724102720 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172977c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 0x7f172419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172977c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 0x7f172419c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54266/0 (socket says 172.21.15.96:54266) 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172977c700 1 -- 172.21.15.96:0/884593081 learned_addr learned my addr 172.21.15.96:0/884593081 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f1728f7b700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1724102720 0x7f172419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172977c700 1 -- 172.21.15.96:0/884593081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1724102720 msgr2=0x7f172419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172977c700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1724102720 0x7f172419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172977c700 1 -- 172.21.15.96:0/884593081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17200095f0 con 0x7f1724101e00 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172977c700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 0x7f172419c800 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17200092e0 tx=0x7f172000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f171a7fc700 1 -- 172.21.15.96:0/884593081 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f172001d070 con 0x7f1724101e00 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1724196bc0 con 0x7f1724101e00 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f171a7fc700 1 -- 172.21.15.96:0/884593081 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f172000f460 con 0x7f1724101e00 2024-04-22T15:24:32.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.418+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17241970b0 con 0x7f1724101e00 2024-04-22T15:24:32.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.419+0000 7f171a7fc700 1 -- 172.21.15.96:0/884593081 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1720017630 con 0x7f1724101e00 2024-04-22T15:24:32.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.420+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1724105520 con 0x7f1724101e00 2024-04-22T15:24:32.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.420+0000 7f171a7fc700 1 -- 172.21.15.96:0/884593081 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f1720003a60 con 0x7f1724101e00 2024-04-22T15:24:32.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.420+0000 7f171a7fc700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f171006edf0 0x7f17100712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:32.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.420+0000 7f171a7fc700 1 -- 172.21.15.96:0/884593081 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f17200905d0 con 0x7f1724101e00 2024-04-22T15:24:32.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.421+0000 7f1728f7b700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f171006edf0 0x7f17100712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:32.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.421+0000 7f1728f7b700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f171006edf0 0x7f17100712b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1714005ea0 tx=0x7f1714005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:32.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.426+0000 7f171a7fc700 1 -- 172.21.15.96:0/884593081 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f172005ccf0 con 0x7f1724101e00 2024-04-22T15:24:32.594 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:32 smithi096 systemd[1]: Starting Ceph osd.0 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:24:32.595 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:32 smithi096 ceph-mon[112443]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:32.595 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:32 smithi096 ceph-mon[112443]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:32.595 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:32 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/89421243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:32.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.662+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1724197e00 con 0x7f171006edf0 2024-04-22T15:24:32.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.663+0000 7f171a7fc700 1 -- 172.21.15.96:0/884593081 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1724197e00 con 0x7f171006edf0 2024-04-22T15:24:32.666 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:24:32.666 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:24:32.666 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:24:32.666 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:24:32.666 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:24:32.667 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:24:32.667 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:24:32.667 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:24:32.667 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:24:32.667 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "6/23 daemons upgraded", 2024-04-22T15:24:32.667 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:24:32.667 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:24:32.667 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:24:32.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f171006edf0 msgr2=0x7f17100712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f171006edf0 0x7f17100712b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1714005ea0 tx=0x7f1714005e30 comp rx=0 tx=0).stop 2024-04-22T15:24:32.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 msgr2=0x7f172419c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 0x7f172419c800 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17200092e0 tx=0x7f172000be20 comp rx=0 tx=0).stop 2024-04-22T15:24:32.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 shutdown_connections 2024-04-22T15:24:32.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f171006edf0 0x7f17100712b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1724101e00 0x7f172419c800 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 --2- 172.21.15.96:0/884593081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1724102720 0x7f172419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 >> 172.21.15.96:0/884593081 conn(0x7f1724076740 msgr2=0x7f17241030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:32.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 shutdown_connections 2024-04-22T15:24:32.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.665+0000 7f172b9e0700 1 -- 172.21.15.96:0/884593081 wait complete. 2024-04-22T15:24:32.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:32 smithi196 ceph-mon[81815]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:32 smithi196 ceph-mon[81815]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:32 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/89421243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:32.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.784+0000 7fabf9cca700 1 -- 172.21.15.96:0/2092185535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4101d60 msgr2=0x7fabf4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.784+0000 7fabf9cca700 1 --2- 172.21.15.96:0/2092185535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4101d60 0x7fabf4102140 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009910 tx=0x7fabe4009c20 comp rx=0 tx=0).stop 2024-04-22T15:24:32.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.784+0000 7fabf9cca700 1 -- 172.21.15.96:0/2092185535 shutdown_connections 2024-04-22T15:24:32.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.784+0000 7fabf9cca700 1 --2- 172.21.15.96:0/2092185535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf4102680 0x7fabf410ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.784+0000 7fabf9cca700 1 --2- 172.21.15.96:0/2092185535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4101d60 0x7fabf4102140 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.784+0000 7fabf9cca700 1 -- 172.21.15.96:0/2092185535 >> 172.21.15.96:0/2092185535 conn(0x7fabf4076740 msgr2=0x7fabf40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.784+0000 7fabf9cca700 1 -- 172.21.15.96:0/2092185535 shutdown_connections 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.784+0000 7fabf9cca700 1 -- 172.21.15.96:0/2092185535 wait complete. 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf9cca700 1 Processor -- start 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf9cca700 1 -- start start 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf9cca700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf4101d60 0x7fabf419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf9cca700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4102680 0x7fabf419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf9cca700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf419d3a0 con 0x7fabf4102680 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf9cca700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf4196980 con 0x7fabf4101d60 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4102680 0x7fabf419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4102680 0x7fabf419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54290/0 (socket says 172.21.15.96:54290) 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf2ffd700 1 -- 172.21.15.96:0/3622780136 learned_addr learned my addr 172.21.15.96:0/3622780136 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:24:32.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf37fe700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf4101d60 0x7fabf419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf2ffd700 1 -- 172.21.15.96:0/3622780136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf4101d60 msgr2=0x7fabf419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf2ffd700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf4101d60 0x7fabf419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf2ffd700 1 -- 172.21.15.96:0/3622780136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe40095f0 con 0x7fabf4102680 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.785+0000 7fabf37fe700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf4101d60 0x7fabf419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.786+0000 7fabf2ffd700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4102680 0x7fabf419cd80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fabe800e940 tx=0x7fabe800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.786+0000 7fabf0ff9700 1 -- 172.21.15.96:0/3622780136 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe80096e0 con 0x7fabf4102680 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.786+0000 7fabf0ff9700 1 -- 172.21.15.96:0/3622780136 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabe8004cf0 con 0x7fabf4102680 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.786+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf4196c60 con 0x7fabf4102680 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.786+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf41971b0 con 0x7fabf4102680 2024-04-22T15:24:32.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.786+0000 7fabf0ff9700 1 -- 172.21.15.96:0/3622780136 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe800eee0 con 0x7fabf4102680 2024-04-22T15:24:32.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.787+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf4105480 con 0x7fabf4102680 2024-04-22T15:24:32.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.787+0000 7fabf0ff9700 1 -- 172.21.15.96:0/3622780136 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fabe800cba0 con 0x7fabf4102680 2024-04-22T15:24:32.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.788+0000 7fabf0ff9700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabdc06ee00 0x7fabdc0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:24:32.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.788+0000 7fabf0ff9700 1 -- 172.21.15.96:0/3622780136 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fabe8014070 con 0x7fabf4102680 2024-04-22T15:24:32.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.788+0000 7fabf37fe700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabdc06ee00 0x7fabdc0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:24:32.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.788+0000 7fabf37fe700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabdc06ee00 0x7fabdc0712c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fabe4000c00 tx=0x7fabe4009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:24:32.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:32.792+0000 7fabf0ff9700 1 -- 172.21.15.96:0/3622780136 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fabe805ba60 con 0x7fabf4102680 2024-04-22T15:24:32.906 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:32 smithi096 podman[124094]: 2024-04-22 15:24:32.591802386 +0000 UTC m=+0.105436445 container create 5b7631108ba3738aba16559162296bf6ef0768a38d06e49fbe2235be8a2837d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:24:32.906 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:32 smithi096 podman[124094]: 2024-04-22 15:24:32.510744022 +0000 UTC m=+0.024378073 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:24:33.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.084+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fabf4197e40 con 0x7fabf4102680 2024-04-22T15:24:33.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.084+0000 7fabf0ff9700 1 -- 172.21.15.96:0/3622780136 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+290 (secure 0 0 0) 0x7fabe805b5f0 con 0x7fabf4102680 2024-04-22T15:24:33.088 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down 2024-04-22T15:24:33.088 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:24:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:24:33.089 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-22T15:24:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: osd.0 (root=default,host=smithi096) is down 2024-04-22T15:24:33.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.086+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabdc06ee00 msgr2=0x7fabdc0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:33.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.086+0000 7fabf9cca700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabdc06ee00 0x7fabdc0712c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fabe4000c00 tx=0x7fabe4009f70 comp rx=0 tx=0).stop 2024-04-22T15:24:33.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.086+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4102680 msgr2=0x7fabf419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:24:33.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.086+0000 7fabf9cca700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4102680 0x7fabf419cd80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fabe800e940 tx=0x7fabe800ed00 comp rx=0 tx=0).stop 2024-04-22T15:24:33.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.086+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 shutdown_connections 2024-04-22T15:24:33.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.086+0000 7fabf9cca700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabdc06ee00 0x7fabdc0712c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:33.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.086+0000 7fabf9cca700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf4101d60 0x7fabf419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:33.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.086+0000 7fabf9cca700 1 --2- 172.21.15.96:0/3622780136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf4102680 0x7fabf419cd80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:24:33.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.086+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 >> 172.21.15.96:0/3622780136 conn(0x7fabf4076740 msgr2=0x7fabf4103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:24:33.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.087+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 shutdown_connections 2024-04-22T15:24:33.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:24:33.087+0000 7fabf9cca700 1 -- 172.21.15.96:0/3622780136 wait complete. 2024-04-22T15:24:33.236 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:32 smithi096 podman[124094]: 2024-04-22 15:24:32.903426269 +0000 UTC m=+0.417060318 container init 5b7631108ba3738aba16559162296bf6ef0768a38d06e49fbe2235be8a2837d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:24:33.236 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:32 smithi096 podman[124094]: 2024-04-22 15:24:32.908039593 +0000 UTC m=+0.421673651 container start 5b7631108ba3738aba16559162296bf6ef0768a38d06e49fbe2235be8a2837d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:24:33.236 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 podman[124094]: 2024-04-22 15:24:33.011690527 +0000 UTC m=+0.525324599 container attach 5b7631108ba3738aba16559162296bf6ef0768a38d06e49fbe2235be8a2837d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T15:24:33.651 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-mon[112443]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:33.651 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-mon[112443]: pgmap v47: 65 pgs: 9 stale+active+clean, 56 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:24:33.651 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/449532791' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:24:33.651 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3622780136' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:24:33.651 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate[124113]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:24:33.651 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate[124113]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:24:33.651 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 bash[124094]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 bash[124094]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 bash[124094]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 bash[124094]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 bash[124094]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 bash[124094]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 bash[124094]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 bash[124094]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 bash[124094]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate[124113]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate[124113]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate[124113]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T15:24:33.652 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate[124113]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T15:24:33.653 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate[124113]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-22T15:24:33.653 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate[124113]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T15:24:33.653 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate[124113]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-22T15:24:33.653 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:33 smithi096 podman[124094]: 2024-04-22 15:24:33.411491749 +0000 UTC m=+0.925125801 container died 5b7631108ba3738aba16559162296bf6ef0768a38d06e49fbe2235be8a2837d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:24:33.696 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:33 smithi196 ceph-mon[81815]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:33.696 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:33 smithi196 ceph-mon[81815]: pgmap v47: 65 pgs: 9 stale+active+clean, 56 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:24:33.696 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:33 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/449532791' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:24:33.697 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:33 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3622780136' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:24:34.298 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:34 smithi096 podman[124266]: 2024-04-22 15:24:34.033111228 +0000 UTC m=+0.612900530 container remove 5b7631108ba3738aba16559162296bf6ef0768a38d06e49fbe2235be8a2837d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True) 2024-04-22T15:24:34.298 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:34 smithi096 podman[124332]: 2024-04-22 15:24:34.150255082 +0000 UTC m=+0.018176874 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:24:34.298 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:34 smithi096 podman[124332]: 2024-04-22 15:24:34.276034418 +0000 UTC m=+0.143956197 container create bbf14d18e92119bbc1195752427c96ba91a029ea97c1527dc20efaffdf9f35ef (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:24:34.548 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:34 smithi096 ceph-mon[112443]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:34.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:34 smithi196 ceph-mon[81815]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:24:34.986 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:34 smithi096 podman[124332]: 2024-04-22 15:24:34.546381598 +0000 UTC m=+0.414303380 container init bbf14d18e92119bbc1195752427c96ba91a029ea97c1527dc20efaffdf9f35ef (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:24:34.986 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:34 smithi096 podman[124332]: 2024-04-22 15:24:34.549504119 +0000 UTC m=+0.417425898 container start bbf14d18e92119bbc1195752427c96ba91a029ea97c1527dc20efaffdf9f35ef (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:24:34.986 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:34 smithi096 ceph-osd[124351]: -- 172.21.15.96:0/2170258637 <== mon.0 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562f08429680 con 0x562f083f2c00 2024-04-22T15:24:34.986 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:34 smithi096 bash[124332]: bbf14d18e92119bbc1195752427c96ba91a029ea97c1527dc20efaffdf9f35ef 2024-04-22T15:24:34.986 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:34 smithi096 systemd[1]: Started Ceph osd.0 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:24:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:35 smithi096 ceph-mon[112443]: pgmap v48: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 62/330 objects degraded (18.788%) 2024-04-22T15:24:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:35 smithi096 ceph-mon[112443]: Health check failed: Degraded data redundancy: 62/330 objects degraded (18.788%), 20 pgs degraded (PG_DEGRADED) 2024-04-22T15:24:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:35.487 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:35 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[124347]: 2024-04-22T15:24:35.094+0000 7f68c6a917c0 -1 Falling back to public interface 2024-04-22T15:24:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:35 smithi196 ceph-mon[81815]: pgmap v48: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 62/330 objects degraded (18.788%) 2024-04-22T15:24:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:35 smithi196 ceph-mon[81815]: Health check failed: Degraded data redundancy: 62/330 objects degraded (18.788%), 20 pgs degraded (PG_DEGRADED) 2024-04-22T15:24:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:37 smithi096 ceph-mon[112443]: pgmap v49: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 62/330 objects degraded (18.788%) 2024-04-22T15:24:37.562 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:37 smithi196 ceph-mon[81815]: pgmap v49: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 62/330 objects degraded (18.788%) 2024-04-22T15:24:39.418 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:39 smithi096 ceph-mon[112443]: pgmap v50: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 62/330 objects degraded (18.788%) 2024-04-22T15:24:39.418 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:39.418 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:24:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:39 smithi196 ceph-mon[81815]: pgmap v50: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 62/330 objects degraded (18.788%) 2024-04-22T15:24:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:24:40.236 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:39 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[124347]: 2024-04-22T15:24:39.873+0000 7f68c6a917c0 -1 osd.0 48 log_to_monitors true 2024-04-22T15:24:40.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:40 smithi096 ceph-mon[112443]: from='osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T15:24:40.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:40.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:40 smithi196 ceph-mon[81815]: from='osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T15:24:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:41.486 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 15:24:41 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[124347]: 2024-04-22T15:24:41.217+0000 7f68b3ee1700 -1 osd.0 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:24:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:41 smithi096 ceph-mon[112443]: pgmap v51: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s; 62/330 objects degraded (18.788%) 2024-04-22T15:24:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:41 smithi096 ceph-mon[112443]: from='osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T15:24:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:41 smithi096 ceph-mon[112443]: osdmap e51: 6 total, 5 up, 6 in 2024-04-22T15:24:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:41 smithi096 ceph-mon[112443]: from='osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:24:41.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:41 smithi096 ceph-mon[112443]: mgrmap e38: smithi096.hhkvum(active, since 92s), standbys: smithi196.gdaegc 2024-04-22T15:24:41.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:41 smithi096 ceph-mon[112443]: from='osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544]' entity='osd.0' 2024-04-22T15:24:41.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:41.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:41 smithi196 ceph-mon[81815]: pgmap v51: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s; 62/330 objects degraded (18.788%) 2024-04-22T15:24:41.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:41 smithi196 ceph-mon[81815]: from='osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T15:24:41.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:41 smithi196 ceph-mon[81815]: osdmap e51: 6 total, 5 up, 6 in 2024-04-22T15:24:41.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:41 smithi196 ceph-mon[81815]: from='osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:24:41.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:41 smithi196 ceph-mon[81815]: mgrmap e38: smithi096.hhkvum(active, since 92s), standbys: smithi196.gdaegc 2024-04-22T15:24:41.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:41 smithi196 ceph-mon[81815]: from='osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544]' entity='osd.0' 2024-04-22T15:24:41.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:41.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:42.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:42 smithi096 ceph-mon[112443]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:24:42.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:42 smithi096 ceph-mon[112443]: osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544] boot 2024-04-22T15:24:42.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:42 smithi096 ceph-mon[112443]: osdmap e52: 6 total, 6 up, 6 in 2024-04-22T15:24:42.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:24:42.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:42 smithi196 ceph-mon[81815]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:24:42.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:42 smithi196 ceph-mon[81815]: osd.0 [v2:172.21.15.96:6802/312948544,v1:172.21.15.96:6803/312948544] boot 2024-04-22T15:24:42.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:42 smithi196 ceph-mon[81815]: osdmap e52: 6 total, 6 up, 6 in 2024-04-22T15:24:42.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:24:43.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:43 smithi096 ceph-mon[112443]: pgmap v54: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 639 B/s wr, 3 op/s; 62/330 objects degraded (18.788%) 2024-04-22T15:24:43.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:43 smithi096 ceph-mon[112443]: osdmap e53: 6 total, 6 up, 6 in 2024-04-22T15:24:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:43 smithi196 ceph-mon[81815]: pgmap v54: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 639 B/s wr, 3 op/s; 62/330 objects degraded (18.788%) 2024-04-22T15:24:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:43 smithi196 ceph-mon[81815]: osdmap e53: 6 total, 6 up, 6 in 2024-04-22T15:24:44.636 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:44 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 9/330 objects degraded (2.727%), 3 pgs degraded (PG_DEGRADED) 2024-04-22T15:24:44.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:44 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 9/330 objects degraded (2.727%), 3 pgs degraded (PG_DEGRADED) 2024-04-22T15:24:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: pgmap v56: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 853 B/s wr, 6 op/s; 9/330 objects degraded (2.727%); 23 B/s, 0 objects/s recovering 2024-04-22T15:24:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:45.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:24:45.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:45.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:45.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:45.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:45.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:45.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: pgmap v56: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 853 B/s wr, 6 op/s; 9/330 objects degraded (2.727%); 23 B/s, 0 objects/s recovering 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:24:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:24:46.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:46 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:24:46.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:46 smithi096 ceph-mon[112443]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-22T15:24:46.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:46 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:24:46.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:46 smithi196 ceph-mon[81815]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-22T15:24:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:47 smithi096 ceph-mon[112443]: pgmap v57: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 70 B/s, 0 objects/s recovering 2024-04-22T15:24:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:47 smithi096 ceph-mon[112443]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/330 objects degraded (2.727%), 3 pgs degraded) 2024-04-22T15:24:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:47 smithi196 ceph-mon[81815]: pgmap v57: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 70 B/s, 0 objects/s recovering 2024-04-22T15:24:47.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:47 smithi196 ceph-mon[81815]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/330 objects degraded (2.727%), 3 pgs degraded) 2024-04-22T15:24:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:49 smithi096 ceph-mon[112443]: pgmap v58: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 902 B/s rd, 1 op/s; 53 B/s, 0 objects/s recovering 2024-04-22T15:24:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:49 smithi196 ceph-mon[81815]: pgmap v58: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 902 B/s rd, 1 op/s; 53 B/s, 0 objects/s recovering 2024-04-22T15:24:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:51 smithi096 ceph-mon[112443]: pgmap v59: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s; 52 B/s, 0 objects/s recovering 2024-04-22T15:24:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:51 smithi196 ceph-mon[81815]: pgmap v59: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s; 52 B/s, 0 objects/s recovering 2024-04-22T15:24:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:53 smithi096 ceph-mon[112443]: pgmap v60: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 42 B/s, 0 objects/s recovering 2024-04-22T15:24:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:24:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:53 smithi196 ceph-mon[81815]: pgmap v60: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 42 B/s, 0 objects/s recovering 2024-04-22T15:24:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:24:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:24:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:55 smithi096 ceph-mon[112443]: pgmap v61: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 38 B/s, 0 objects/s recovering 2024-04-22T15:24:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:55 smithi196 ceph-mon[81815]: pgmap v61: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 38 B/s, 0 objects/s recovering 2024-04-22T15:24:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:57 smithi096 ceph-mon[112443]: pgmap v62: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s; 23 B/s, 0 objects/s recovering 2024-04-22T15:24:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:57 smithi196 ceph-mon[81815]: pgmap v62: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s; 23 B/s, 0 objects/s recovering 2024-04-22T15:24:59.694 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:24:59 smithi096 ceph-mon[112443]: pgmap v63: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:24:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:24:59 smithi196 ceph-mon[81815]: pgmap v63: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:25:00.623 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:25:00.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:25:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:01 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:25:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:01 smithi096 ceph-mon[112443]: Upgrade: osd.1 is safe to restart 2024-04-22T15:25:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:01 smithi096 ceph-mon[112443]: pgmap v64: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:25:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:01 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:25:01.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:01 smithi196 ceph-mon[81815]: Upgrade: osd.1 is safe to restart 2024-04-22T15:25:01.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:01 smithi196 ceph-mon[81815]: pgmap v64: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:25:02.728 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:02 smithi096 ceph-mon[112443]: Upgrade: Updating osd.1 2024-04-22T15:25:02.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:02 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:02.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:02 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:25:02.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:02 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:25:02.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:02 smithi096 ceph-mon[112443]: Deploying daemon osd.1 on smithi096 2024-04-22T15:25:02.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:02 smithi096 ceph-mon[112443]: pgmap v65: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:25:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:02 smithi196 ceph-mon[81815]: Upgrade: Updating osd.1 2024-04-22T15:25:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:02 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:03.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:02 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:25:03.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:02 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:25:03.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:02 smithi196 ceph-mon[81815]: Deploying daemon osd.1 on smithi096 2024-04-22T15:25:03.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:02 smithi196 ceph-mon[81815]: pgmap v65: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:25:03.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 -- 172.21.15.96:0/2372580031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 msgr2=0x7fcfe40ff900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:03.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 --2- 172.21.15.96:0/2372580031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 0x7fcfe40ff900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400aa20 tx=0x7fcfd400ad30 comp rx=0 tx=0).stop 2024-04-22T15:25:03.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 -- 172.21.15.96:0/2372580031 shutdown_connections 2024-04-22T15:25:03.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 --2- 172.21.15.96:0/2372580031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe40ffed0 0x7fcfe4103f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 --2- 172.21.15.96:0/2372580031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 0x7fcfe40ff900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 -- 172.21.15.96:0/2372580031 >> 172.21.15.96:0/2372580031 conn(0x7fcfe4075f00 msgr2=0x7fcfe40fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:03.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 -- 172.21.15.96:0/2372580031 shutdown_connections 2024-04-22T15:25:03.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 -- 172.21.15.96:0/2372580031 wait complete. 2024-04-22T15:25:03.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 Processor -- start 2024-04-22T15:25:03.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.198+0000 7fcfea53c700 1 -- start start 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfea53c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 0x7fcfe419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfea53c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe40ffed0 0x7fcfe419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfea53c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe419b260 con 0x7fcfe40ff520 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfea53c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41947f0 con 0x7fcfe40ffed0 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfe3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 0x7fcfe419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfe3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 0x7fcfe419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39796/0 (socket says 172.21.15.96:39796) 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfe3fff700 1 -- 172.21.15.96:0/1706625013 learned_addr learned my addr 172.21.15.96:0/1706625013 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfe3fff700 1 -- 172.21.15.96:0/1706625013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe40ffed0 msgr2=0x7fcfe419ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfe3fff700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe40ffed0 0x7fcfe419ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.199+0000 7fcfe3fff700 1 -- 172.21.15.96:0/1706625013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd400a700 con 0x7fcfe40ff520 2024-04-22T15:25:03.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.200+0000 7fcfe3fff700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 0x7fcfe419a640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400cd00 tx=0x7fcfd4004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:03.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.200+0000 7fcfe17fa700 1 -- 172.21.15.96:0/1706625013 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd4004b30 con 0x7fcfe40ff520 2024-04-22T15:25:03.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.200+0000 7fcfe17fa700 1 -- 172.21.15.96:0/1706625013 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfd4004c90 con 0x7fcfe40ff520 2024-04-22T15:25:03.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.200+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe4194a70 con 0x7fcfe40ff520 2024-04-22T15:25:03.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.200+0000 7fcfe17fa700 1 -- 172.21.15.96:0/1706625013 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd4016780 con 0x7fcfe40ff520 2024-04-22T15:25:03.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.200+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe4194f60 con 0x7fcfe40ff520 2024-04-22T15:25:03.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.201+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe41078c0 con 0x7fcfe40ff520 2024-04-22T15:25:03.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.202+0000 7fcfe17fa700 1 -- 172.21.15.96:0/1706625013 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcfd400e450 con 0x7fcfe40ff520 2024-04-22T15:25:03.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.202+0000 7fcfe17fa700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcfcc0730e0 0x7fcfcc0755a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:03.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.202+0000 7fcfe17fa700 1 -- 172.21.15.96:0/1706625013 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fcfd4012070 con 0x7fcfe40ff520 2024-04-22T15:25:03.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.202+0000 7fcfe37fe700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcfcc0730e0 0x7fcfcc0755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:03.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.203+0000 7fcfe37fe700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcfcc0730e0 0x7fcfcc0755a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4195ed0 tx=0x7fcfd800b3f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:03.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.207+0000 7fcfe17fa700 1 -- 172.21.15.96:0/1706625013 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcfd405d560 con 0x7fcfe40ff520 2024-04-22T15:25:03.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.437+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfe410bc50 con 0x7fcfcc0730e0 2024-04-22T15:25:03.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.438+0000 7fcfe17fa700 1 -- 172.21.15.96:0/1706625013 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fcfe410bc50 con 0x7fcfcc0730e0 2024-04-22T15:25:03.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.440+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcfcc0730e0 msgr2=0x7fcfcc0755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:03.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.440+0000 7fcfea53c700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcfcc0730e0 0x7fcfcc0755a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4195ed0 tx=0x7fcfd800b3f0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.440+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 msgr2=0x7fcfe419a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:03.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.440+0000 7fcfea53c700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 0x7fcfe419a640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400cd00 tx=0x7fcfd4004690 comp rx=0 tx=0).stop 2024-04-22T15:25:03.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.441+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 shutdown_connections 2024-04-22T15:25:03.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.441+0000 7fcfea53c700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcfcc0730e0 0x7fcfcc0755a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.441+0000 7fcfea53c700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe40ff520 0x7fcfe419a640 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.441+0000 7fcfea53c700 1 --2- 172.21.15.96:0/1706625013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe40ffed0 0x7fcfe419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.441+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 >> 172.21.15.96:0/1706625013 conn(0x7fcfe4075f00 msgr2=0x7fcfe4102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:03.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.441+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 shutdown_connections 2024-04-22T15:25:03.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.441+0000 7fcfea53c700 1 -- 172.21.15.96:0/1706625013 wait complete. 2024-04-22T15:25:03.454 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:25:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.554+0000 7facc3913700 1 -- 172.21.15.96:0/415672561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ffcc0 msgr2=0x7facbc105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.554+0000 7facc3913700 1 --2- 172.21.15.96:0/415672561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ffcc0 0x7facbc105860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7facac009910 tx=0x7facac009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.554+0000 7facc3913700 1 -- 172.21.15.96:0/415672561 shutdown_connections 2024-04-22T15:25:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.554+0000 7facc3913700 1 --2- 172.21.15.96:0/415672561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ffcc0 0x7facbc105860 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.554+0000 7facc3913700 1 --2- 172.21.15.96:0/415672561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7facbc0ff310 0x7facbc0ff6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.554+0000 7facc3913700 1 -- 172.21.15.96:0/415672561 >> 172.21.15.96:0/415672561 conn(0x7facbc075f40 msgr2=0x7facbc0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.554+0000 7facc3913700 1 -- 172.21.15.96:0/415672561 shutdown_connections 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.554+0000 7facc3913700 1 -- 172.21.15.96:0/415672561 wait complete. 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.555+0000 7facc3913700 1 Processor -- start 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.555+0000 7facc3913700 1 -- start start 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.555+0000 7facc3913700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ff310 0x7facbc19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.555+0000 7facc3913700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7facbc0ffcc0 0x7facbc19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.555+0000 7facc3913700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc19d490 con 0x7facbc0ff310 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.555+0000 7facc3913700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc196ac0 con 0x7facbc0ffcc0 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.555+0000 7facc16af700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ff310 0x7facbc19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.555+0000 7facc16af700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ff310 0x7facbc19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39806/0 (socket says 172.21.15.96:39806) 2024-04-22T15:25:03.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.555+0000 7facc16af700 1 -- 172.21.15.96:0/1870090950 learned_addr learned my addr 172.21.15.96:0/1870090950 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:03.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facc0eae700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7facbc0ffcc0 0x7facbc19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:03.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facc16af700 1 -- 172.21.15.96:0/1870090950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7facbc0ffcc0 msgr2=0x7facbc19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:03.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facc16af700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7facbc0ffcc0 0x7facbc19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facc16af700 1 -- 172.21.15.96:0/1870090950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac0095f0 con 0x7facbc0ff310 2024-04-22T15:25:03.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facc16af700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ff310 0x7facbc19c930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facb800b4d0 tx=0x7facb800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facb27fc700 1 -- 172.21.15.96:0/1870090950 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facb8011800 con 0x7facbc0ff310 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facb27fc700 1 -- 172.21.15.96:0/1870090950 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facb8011e20 con 0x7facbc0ff310 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc196da0 con 0x7facbc0ff310 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facb27fc700 1 -- 172.21.15.96:0/1870090950 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facb800f2e0 con 0x7facbc0ff310 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.556+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc1972f0 con 0x7facbc0ff310 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.557+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc109260 con 0x7facbc0ff310 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.558+0000 7facb27fc700 1 -- 172.21.15.96:0/1870090950 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7facb8010380 con 0x7facbc0ff310 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.558+0000 7facb27fc700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faca806ecc0 0x7faca8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.558+0000 7facc0eae700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faca806ecc0 0x7faca8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.558+0000 7facb27fc700 1 -- 172.21.15.96:0/1870090950 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7facb808e9e0 con 0x7facbc0ff310 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.558+0000 7facc0eae700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faca806ecc0 0x7faca8071180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7facbc197c30 tx=0x7facac00b580 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:03.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.565+0000 7facb27fc700 1 -- 172.21.15.96:0/1870090950 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7facb805b320 con 0x7facbc0ff310 2024-04-22T15:25:03.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.795+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7facbc197fa0 con 0x7faca806ecc0 2024-04-22T15:25:03.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.796+0000 7facb27fc700 1 -- 172.21.15.96:0/1870090950 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7facbc197fa0 con 0x7faca806ecc0 2024-04-22T15:25:03.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.798+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faca806ecc0 msgr2=0x7faca8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:03.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.798+0000 7facc3913700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faca806ecc0 0x7faca8071180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7facbc197c30 tx=0x7facac00b580 comp rx=0 tx=0).stop 2024-04-22T15:25:03.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.799+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ff310 msgr2=0x7facbc19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:03.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.799+0000 7facc3913700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ff310 0x7facbc19c930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facb800b4d0 tx=0x7facb800b890 comp rx=0 tx=0).stop 2024-04-22T15:25:03.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.799+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 shutdown_connections 2024-04-22T15:25:03.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.799+0000 7facc3913700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faca806ecc0 0x7faca8071180 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.799+0000 7facc3913700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7facbc0ff310 0x7facbc19c930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.799+0000 7facc3913700 1 --2- 172.21.15.96:0/1870090950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7facbc0ffcc0 0x7facbc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.799+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 >> 172.21.15.96:0/1870090950 conn(0x7facbc075f40 msgr2=0x7facbc104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:03.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.799+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 shutdown_connections 2024-04-22T15:25:03.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.799+0000 7facc3913700 1 -- 172.21.15.96:0/1870090950 wait complete. 2024-04-22T15:25:03.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.907+0000 7f51c535f700 1 -- 172.21.15.96:0/3194793540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c00ffaa0 msgr2=0x7f51c00fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:03.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.907+0000 7f51c535f700 1 --2- 172.21.15.96:0/3194793540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c00ffaa0 0x7f51c00fff20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009910 tx=0x7f51b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:03.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.907+0000 7f51c535f700 1 -- 172.21.15.96:0/3194793540 shutdown_connections 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.907+0000 7f51c535f700 1 --2- 172.21.15.96:0/3194793540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c00ffaa0 0x7f51c00fff20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.907+0000 7f51c535f700 1 --2- 172.21.15.96:0/3194793540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51c0105aa0 0x7f51c0105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.907+0000 7f51c535f700 1 -- 172.21.15.96:0/3194793540 >> 172.21.15.96:0/3194793540 conn(0x7f51c0076740 msgr2=0x7f51c00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.907+0000 7f51c535f700 1 -- 172.21.15.96:0/3194793540 shutdown_connections 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.907+0000 7f51c535f700 1 -- 172.21.15.96:0/3194793540 wait complete. 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.907+0000 7f51c535f700 1 Processor -- start 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51c535f700 1 -- start start 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51c535f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51c00ffaa0 0x7f51c019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51c535f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c0105aa0 0x7f51c019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51c535f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c019f5a0 con 0x7f51c0105aa0 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51c535f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c0198b80 con 0x7f51c00ffaa0 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51be7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c0105aa0 0x7f51c019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:03.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51be7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c0105aa0 0x7f51c019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39830/0 (socket says 172.21.15.96:39830) 2024-04-22T15:25:03.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51be7fc700 1 -- 172.21.15.96:0/1775222153 learned_addr learned my addr 172.21.15.96:0/1775222153 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:03.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51beffd700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51c00ffaa0 0x7f51c019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:03.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51be7fc700 1 -- 172.21.15.96:0/1775222153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51c00ffaa0 msgr2=0x7f51c019ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:03.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51be7fc700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51c00ffaa0 0x7f51c019ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:03.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.908+0000 7f51be7fc700 1 -- 172.21.15.96:0/1775222153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b40095f0 con 0x7f51c0105aa0 2024-04-22T15:25:03.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.909+0000 7f51be7fc700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c0105aa0 0x7f51c019ef80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f51b4006010 tx=0x7f51b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.909+0000 7f51a7fff700 1 -- 172.21.15.96:0/1775222153 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51b401d070 con 0x7f51c0105aa0 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.909+0000 7f51a7fff700 1 -- 172.21.15.96:0/1775222153 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51b400f460 con 0x7f51c0105aa0 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.909+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c0198e00 con 0x7f51c0105aa0 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.909+0000 7f51a7fff700 1 -- 172.21.15.96:0/1775222153 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51b4017630 con 0x7f51c0105aa0 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.909+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c01992f0 con 0x7f51c0105aa0 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.910+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c01011e0 con 0x7f51c0105aa0 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.911+0000 7f51a7fff700 1 -- 172.21.15.96:0/1775222153 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f51b400f5d0 con 0x7f51c0105aa0 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.911+0000 7f51a7fff700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f51a806ecd0 0x7f51a8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.911+0000 7f51a7fff700 1 -- 172.21.15.96:0/1775222153 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f51b4090320 con 0x7f51c0105aa0 2024-04-22T15:25:03.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.911+0000 7f51beffd700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f51a806ecd0 0x7f51a8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:03.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.912+0000 7f51beffd700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f51a806ecd0 0x7f51a8071190 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f51b0005fd0 tx=0x7f51b000a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:03.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:03.916+0000 7f51a7fff700 1 -- 172.21.15.96:0/1775222153 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f51b405cd20 con 0x7f51c0105aa0 2024-04-22T15:25:04.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.127+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f51c010cca0 con 0x7f51a806ecd0 2024-04-22T15:25:04.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.138+0000 7f51a7fff700 1 -- 172.21.15.96:0/1775222153 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f51c010cca0 con 0x7f51a806ecd0 2024-04-22T15:25:04.142 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:25:04.142 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (3m) 24s ago 15m 25.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:25:04.142 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (15m) 24s ago 15m 8418k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:25:04.142 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (14m) 47s ago 14m 8166k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:25:04.142 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (56s) 24s ago 15m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:25:04.142 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (51s) 47s ago 14m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:25:04.142 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (3m) 24s ago 14m 81.7M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:25:04.142 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (9m) 24s ago 9m 155M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:25:04.142 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (9m) 24s ago 9m 90.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (9m) 47s ago 9m 27.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (9m) 47s ago 9m 26.3M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (6m) 24s ago 16m 502M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (4m) 47s ago 13m 450M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (2m) 24s ago 16m 44.5M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (99s) 47s ago 13m 35.4M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (4m) 24s ago 15m 17.0M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (4m) 47s ago 14m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (29s) 24s ago 13m 15.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (12m) 24s ago 12m 220M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 086c04ea748c 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (12m) 24s ago 12m 184M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (11m) 47s ago 11m 242M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (11m) 47s ago 11m 213M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:25:04.143 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (10m) 47s ago 10m 196M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:25:04.144 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (4m) 24s ago 14m 76.7M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:25:04.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.140+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f51a806ecd0 msgr2=0x7f51a8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.140+0000 7f51c535f700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f51a806ecd0 0x7f51a8071190 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f51b0005fd0 tx=0x7f51b000a560 comp rx=0 tx=0).stop 2024-04-22T15:25:04.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.140+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c0105aa0 msgr2=0x7f51c019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.140+0000 7f51c535f700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c0105aa0 0x7f51c019ef80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f51b4006010 tx=0x7f51b400be20 comp rx=0 tx=0).stop 2024-04-22T15:25:04.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.141+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 shutdown_connections 2024-04-22T15:25:04.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.141+0000 7f51c535f700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f51a806ecd0 0x7f51a8071190 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.141+0000 7f51c535f700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51c00ffaa0 0x7f51c019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.141+0000 7f51c535f700 1 --2- 172.21.15.96:0/1775222153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51c0105aa0 0x7f51c019ef80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.141+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 >> 172.21.15.96:0/1775222153 conn(0x7f51c0076740 msgr2=0x7f51c01030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:04.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.141+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 shutdown_connections 2024-04-22T15:25:04.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.141+0000 7f51c535f700 1 -- 172.21.15.96:0/1775222153 wait complete. 2024-04-22T15:25:04.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.245+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2392949328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 msgr2=0x7f06a80fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.245+0000 7f06ae9c3700 1 --2- 172.21.15.96:0/2392949328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 0x7f06a80fff20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0698009910 tx=0x7f0698009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:04.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.245+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2392949328 shutdown_connections 2024-04-22T15:25:04.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.245+0000 7f06ae9c3700 1 --2- 172.21.15.96:0/2392949328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 0x7f06a80fff20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.245+0000 7f06ae9c3700 1 --2- 172.21.15.96:0/2392949328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06a8105aa0 0x7f06a8105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.245+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2392949328 >> 172.21.15.96:0/2392949328 conn(0x7f06a8076740 msgr2=0x7f06a80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:04.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.245+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2392949328 shutdown_connections 2024-04-22T15:25:04.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.245+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2392949328 wait complete. 2024-04-22T15:25:04.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.245+0000 7f06ae9c3700 1 Processor -- start 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06ae9c3700 1 -- start start 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06ae9c3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 0x7f06a8197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06ae9c3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06a8105aa0 0x7f06a8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06ae9c3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a8198530 con 0x7f06a80ffaa0 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06ae9c3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a8198670 con 0x7f06a8105aa0 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06a7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 0x7f06a8197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06a7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 0x7f06a8197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39846/0 (socket says 172.21.15.96:39846) 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06a7fff700 1 -- 172.21.15.96:0/2764080917 learned_addr learned my addr 172.21.15.96:0/2764080917 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06a7fff700 1 -- 172.21.15.96:0/2764080917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06a8105aa0 msgr2=0x7f06a8197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06a7fff700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06a8105aa0 0x7f06a8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06a7fff700 1 -- 172.21.15.96:0/2764080917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980095f0 con 0x7f06a80ffaa0 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.246+0000 7f06a7fff700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 0x7f06a8197a20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f069000b4b0 tx=0x7f069000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.247+0000 7f06a57fa700 1 -- 172.21.15.96:0/2764080917 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0690011800 con 0x7f06a80ffaa0 2024-04-22T15:25:04.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.247+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a81a0a90 con 0x7f06a80ffaa0 2024-04-22T15:25:04.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.247+0000 7f06a57fa700 1 -- 172.21.15.96:0/2764080917 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0690011e20 con 0x7f06a80ffaa0 2024-04-22T15:25:04.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.247+0000 7f06a57fa700 1 -- 172.21.15.96:0/2764080917 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f069000f2e0 con 0x7f06a80ffaa0 2024-04-22T15:25:04.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.247+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a81a0ea0 con 0x7f06a80ffaa0 2024-04-22T15:25:04.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.248+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a81080c0 con 0x7f06a80ffaa0 2024-04-22T15:25:04.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.248+0000 7f06a57fa700 1 -- 172.21.15.96:0/2764080917 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0690011960 con 0x7f06a80ffaa0 2024-04-22T15:25:04.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.249+0000 7f06a57fa700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f069406eac0 0x7f0694070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:04.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.249+0000 7f06a57fa700 1 -- 172.21.15.96:0/2764080917 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f069008e640 con 0x7f06a80ffaa0 2024-04-22T15:25:04.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.249+0000 7f06a77fe700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f069406eac0 0x7f0694070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:04.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.249+0000 7f06a77fe700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f069406eac0 0x7f0694070f80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f069800b5c0 tx=0x7f0698005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:04.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.254+0000 7f06a57fa700 1 -- 172.21.15.96:0/2764080917 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0690057450 con 0x7f06a80ffaa0 2024-04-22T15:25:04.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.544+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f06a804ec10 con 0x7f06a80ffaa0 2024-04-22T15:25:04.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.545+0000 7f06a57fa700 1 -- 172.21.15.96:0/2764080917 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+836 (secure 0 0 0) 0x7f069005ab10 con 0x7f06a80ffaa0 2024-04-22T15:25:04.548 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:25:04.548 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:25:04.548 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:25:04.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:25:04.548 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:25:04.548 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:25:04.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 5, 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 9, 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 5 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:25:04.549 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:25:04.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f069406eac0 msgr2=0x7f0694070f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f069406eac0 0x7f0694070f80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f069800b5c0 tx=0x7f0698005e20 comp rx=0 tx=0).stop 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 msgr2=0x7f06a8197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 0x7f06a8197a20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f069000b4b0 tx=0x7f069000b870 comp rx=0 tx=0).stop 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 shutdown_connections 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f069406eac0 0x7f0694070f80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06a80ffaa0 0x7f06a8197a20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 --2- 172.21.15.96:0/2764080917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06a8105aa0 0x7f06a8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 >> 172.21.15.96:0/2764080917 conn(0x7f06a8076740 msgr2=0x7f06a80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 shutdown_connections 2024-04-22T15:25:04.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.547+0000 7f06ae9c3700 1 -- 172.21.15.96:0/2764080917 wait complete. 2024-04-22T15:25:04.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.666+0000 7f88fbf08700 1 -- 172.21.15.96:0/336173852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f4105ac0 msgr2=0x7f88f4105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.666+0000 7f88fbf08700 1 --2- 172.21.15.96:0/336173852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f4105ac0 0x7f88f4105ea0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009910 tx=0x7f88f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:04.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.666+0000 7f88fbf08700 1 -- 172.21.15.96:0/336173852 shutdown_connections 2024-04-22T15:25:04.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.666+0000 7f88fbf08700 1 --2- 172.21.15.96:0/336173852 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88f40ffac0 0x7f88f40fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.666+0000 7f88fbf08700 1 --2- 172.21.15.96:0/336173852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f4105ac0 0x7f88f4105ea0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.666+0000 7f88fbf08700 1 -- 172.21.15.96:0/336173852 >> 172.21.15.96:0/336173852 conn(0x7f88f40fb5d0 msgr2=0x7f88f40fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.666+0000 7f88fbf08700 1 -- 172.21.15.96:0/336173852 shutdown_connections 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.666+0000 7f88fbf08700 1 -- 172.21.15.96:0/336173852 wait complete. 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88fbf08700 1 Processor -- start 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88fbf08700 1 -- start start 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88fbf08700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f40ffac0 0x7f88f419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88fbf08700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88f4105ac0 0x7f88f419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88fbf08700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f419d330 con 0x7f88f40ffac0 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88fbf08700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f4196910 con 0x7f88f4105ac0 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88f9ca4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f40ffac0 0x7f88f419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88f9ca4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f40ffac0 0x7f88f419c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39864/0 (socket says 172.21.15.96:39864) 2024-04-22T15:25:04.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88f9ca4700 1 -- 172.21.15.96:0/4147083478 learned_addr learned my addr 172.21.15.96:0/4147083478 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88f94a3700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88f4105ac0 0x7f88f419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88f9ca4700 1 -- 172.21.15.96:0/4147083478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88f4105ac0 msgr2=0x7f88f419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88f9ca4700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88f4105ac0 0x7f88f419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88f9ca4700 1 -- 172.21.15.96:0/4147083478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f00095f0 con 0x7f88f40ffac0 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88f94a3700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88f4105ac0 0x7f88f419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88f9ca4700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f40ffac0 0x7f88f419c7d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88f000bd70 tx=0x7f88f000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.667+0000 7f88e6ffd700 1 -- 172.21.15.96:0/4147083478 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f001d070 con 0x7f88f40ffac0 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.668+0000 7f88e6ffd700 1 -- 172.21.15.96:0/4147083478 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f88f00056e0 con 0x7f88f40ffac0 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.668+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f4196b90 con 0x7f88f40ffac0 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.668+0000 7f88e6ffd700 1 -- 172.21.15.96:0/4147083478 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f0017400 con 0x7f88f40ffac0 2024-04-22T15:25:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.668+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f4197080 con 0x7f88f40ffac0 2024-04-22T15:25:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.668+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f41090d0 con 0x7f88f40ffac0 2024-04-22T15:25:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.669+0000 7f88e6ffd700 1 -- 172.21.15.96:0/4147083478 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f88f0003a60 con 0x7f88f40ffac0 2024-04-22T15:25:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.669+0000 7f88e6ffd700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f88e006ecc0 0x7f88e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.670+0000 7f88f94a3700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f88e006ecc0 0x7f88e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.670+0000 7f88e6ffd700 1 -- 172.21.15.96:0/4147083478 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f88f00901b0 con 0x7f88f40ffac0 2024-04-22T15:25:04.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.670+0000 7f88f94a3700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f88e006ecc0 0x7f88e0071180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f88e8005fd0 tx=0x7f88e8005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:04.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.674+0000 7f88e6ffd700 1 -- 172.21.15.96:0/4147083478 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f88f005cb70 con 0x7f88f40ffac0 2024-04-22T15:25:04.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.949+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f88f4197dd0 con 0x7f88f40ffac0 2024-04-22T15:25:04.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.949+0000 7f88e6ffd700 1 -- 172.21.15.96:0/4147083478 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1901 (secure 0 0 0) 0x7f88f005c700 con 0x7f88f40ffac0 2024-04-22T15:25:04.953 INFO:teuthology.orchestra.run.smithi096.stdout:e20 2024-04-22T15:25:04.953 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:25:04.953 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:25:04.953 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:25:04.953 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:25:04.953 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:25:04.953 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 20 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:25:04.954 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:25:04.955 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:25:04.956 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:25:04.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f88e006ecc0 msgr2=0x7f88e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f88e006ecc0 0x7f88e0071180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f88e8005fd0 tx=0x7f88e8005ee0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f40ffac0 msgr2=0x7f88f419c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:04.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f40ffac0 0x7f88f419c7d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88f000bd70 tx=0x7f88f000be50 comp rx=0 tx=0).stop 2024-04-22T15:25:04.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 shutdown_connections 2024-04-22T15:25:04.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f88e006ecc0 0x7f88e0071180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f40ffac0 0x7f88f419c7d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 --2- 172.21.15.96:0/4147083478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88f4105ac0 0x7f88f419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:04.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 >> 172.21.15.96:0/4147083478 conn(0x7f88f40fb5d0 msgr2=0x7f88f4069690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:04.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 shutdown_connections 2024-04-22T15:25:04.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:04.952+0000 7f88fbf08700 1 -- 172.21.15.96:0/4147083478 wait complete. 2024-04-22T15:25:04.958 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:25:05.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.086+0000 7fb410802700 1 -- 172.21.15.96:0/3125318903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c0ffab0 msgr2=0x7fb40c0fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:05.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.086+0000 7fb410802700 1 --2- 172.21.15.96:0/3125318903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c0ffab0 0x7fb40c0fff30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009910 tx=0x7fb3fc009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:05.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.086+0000 7fb410802700 1 -- 172.21.15.96:0/3125318903 shutdown_connections 2024-04-22T15:25:05.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.086+0000 7fb410802700 1 --2- 172.21.15.96:0/3125318903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c0ffab0 0x7fb40c0fff30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.086+0000 7fb410802700 1 --2- 172.21.15.96:0/3125318903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40c105ab0 0x7fb40c105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.086+0000 7fb410802700 1 -- 172.21.15.96:0/3125318903 >> 172.21.15.96:0/3125318903 conn(0x7fb40c0fb5e0 msgr2=0x7fb40c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:05.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.086+0000 7fb410802700 1 -- 172.21.15.96:0/3125318903 shutdown_connections 2024-04-22T15:25:05.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.086+0000 7fb410802700 1 -- 172.21.15.96:0/3125318903 wait complete. 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb410802700 1 Processor -- start 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb410802700 1 -- start start 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb410802700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40c0ffab0 0x7fb40c19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb410802700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c105ab0 0x7fb40c19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb410802700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c19d350 con 0x7fb40c105ab0 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb410802700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c196930 con 0x7fb40c0ffab0 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb409d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c105ab0 0x7fb40c19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb409d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c105ab0 0x7fb40c19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39876/0 (socket says 172.21.15.96:39876) 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb409d9b700 1 -- 172.21.15.96:0/2951298692 learned_addr learned my addr 172.21.15.96:0/2951298692 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb40a59c700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40c0ffab0 0x7fb40c19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb409d9b700 1 -- 172.21.15.96:0/2951298692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40c0ffab0 msgr2=0x7fb40c19c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:05.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb409d9b700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40c0ffab0 0x7fb40c19c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb409d9b700 1 -- 172.21.15.96:0/2951298692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc0095f0 con 0x7fb40c105ab0 2024-04-22T15:25:05.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.087+0000 7fb409d9b700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c105ab0 0x7fb40c19cd30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00bd70 tx=0x7fb3fc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:05.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.088+0000 7fb3fb7fe700 1 -- 172.21.15.96:0/2951298692 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3fc01d070 con 0x7fb40c105ab0 2024-04-22T15:25:05.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.088+0000 7fb3fb7fe700 1 -- 172.21.15.96:0/2951298692 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3fc0056e0 con 0x7fb40c105ab0 2024-04-22T15:25:05.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.088+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c196bb0 con 0x7fb40c105ab0 2024-04-22T15:25:05.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.088+0000 7fb3fb7fe700 1 -- 172.21.15.96:0/2951298692 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3fc017400 con 0x7fb40c105ab0 2024-04-22T15:25:05.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.088+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c1970a0 con 0x7fb40c105ab0 2024-04-22T15:25:05.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.089+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c1090e0 con 0x7fb40c105ab0 2024-04-22T15:25:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.089+0000 7fb3fb7fe700 1 -- 172.21.15.96:0/2951298692 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb3fc003a60 con 0x7fb40c105ab0 2024-04-22T15:25:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.090+0000 7fb3fb7fe700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb3f406ecc0 0x7fb3f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.090+0000 7fb3fb7fe700 1 -- 172.21.15.96:0/2951298692 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fb3fc090240 con 0x7fb40c105ab0 2024-04-22T15:25:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.090+0000 7fb40a59c700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb3f406ecc0 0x7fb3f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.090+0000 7fb40a59c700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb3f406ecc0 0x7fb3f4071180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb400005950 tx=0x7fb4000058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:05.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.094+0000 7fb3fb7fe700 1 -- 172.21.15.96:0/2951298692 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb3fc05cb80 con 0x7fb40c105ab0 2024-04-22T15:25:05.236 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:04 smithi096 systemd[1]: Stopping Ceph osd.1 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:25:05.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.331+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb40c197e60 con 0x7fb3f406ecc0 2024-04-22T15:25:05.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.332+0000 7fb3fb7fe700 1 -- 172.21.15.96:0/2951298692 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fb40c197e60 con 0x7fb3f406ecc0 2024-04-22T15:25:05.335 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:25:05.335 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:25:05.335 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:25:05.335 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:25:05.335 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:25:05.336 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:25:05.336 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:25:05.336 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:25:05.336 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:25:05.336 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "7/23 daemons upgraded", 2024-04-22T15:25:05.336 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:25:05.336 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:25:05.336 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:25:05.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb3f406ecc0 msgr2=0x7fb3f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb3f406ecc0 0x7fb3f4071180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb400005950 tx=0x7fb4000058e0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c105ab0 msgr2=0x7fb40c19cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c105ab0 0x7fb40c19cd30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00bd70 tx=0x7fb3fc00be50 comp rx=0 tx=0).stop 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 shutdown_connections 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb3f406ecc0 0x7fb3f4071180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40c0ffab0 0x7fb40c19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 --2- 172.21.15.96:0/2951298692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb40c105ab0 0x7fb40c19cd30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 >> 172.21.15.96:0/2951298692 conn(0x7fb40c0fb5e0 msgr2=0x7fb40c069390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 shutdown_connections 2024-04-22T15:25:05.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.334+0000 7fb410802700 1 -- 172.21.15.96:0/2951298692 wait complete. 2024-04-22T15:25:05.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.438+0000 7f97ae56c700 1 -- 172.21.15.96:0/2645002676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 msgr2=0x7f97a8105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:05.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.438+0000 7f97ae56c700 1 --2- 172.21.15.96:0/2645002676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 0x7f97a8105820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9798009910 tx=0x7f9798009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:05.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.438+0000 7f97ae56c700 1 -- 172.21.15.96:0/2645002676 shutdown_connections 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.438+0000 7f97ae56c700 1 --2- 172.21.15.96:0/2645002676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 0x7f97a8105820 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.438+0000 7f97ae56c700 1 --2- 172.21.15.96:0/2645002676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a8100e20 0x7f97a8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.438+0000 7f97ae56c700 1 -- 172.21.15.96:0/2645002676 >> 172.21.15.96:0/2645002676 conn(0x7f97a80fc6d0 msgr2=0x7f97a80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.438+0000 7f97ae56c700 1 -- 172.21.15.96:0/2645002676 shutdown_connections 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.438+0000 7f97ae56c700 1 -- 172.21.15.96:0/2645002676 wait complete. 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97ae56c700 1 Processor -- start 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97ae56c700 1 -- start start 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97ae56c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a8100e20 0x7f97a819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97ae56c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 0x7f97a819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97ae56c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a819d4c0 con 0x7f97a81017d0 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97ae56c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8196aa0 con 0x7f97a8100e20 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97a77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 0x7f97a819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:05.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97a77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 0x7f97a819cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39888/0 (socket says 172.21.15.96:39888) 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97a77fe700 1 -- 172.21.15.96:0/1871800975 learned_addr learned my addr 172.21.15.96:0/1871800975 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97a7fff700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a8100e20 0x7f97a819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97a77fe700 1 -- 172.21.15.96:0/1871800975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a8100e20 msgr2=0x7f97a819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97a77fe700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a8100e20 0x7f97a819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97a77fe700 1 -- 172.21.15.96:0/1871800975 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980095f0 con 0x7f97a81017d0 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.439+0000 7f97a7fff700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a8100e20 0x7f97a819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.440+0000 7f97a77fe700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 0x7f97a819cea0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f97980092e0 tx=0x7f979800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.440+0000 7f97a57fa700 1 -- 172.21.15.96:0/1871800975 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f979801d070 con 0x7f97a81017d0 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.440+0000 7f97a57fa700 1 -- 172.21.15.96:0/1871800975 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f979800f460 con 0x7f97a81017d0 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.440+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a8196d20 con 0x7f97a81017d0 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.440+0000 7f97a57fa700 1 -- 172.21.15.96:0/1871800975 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9798017610 con 0x7f97a81017d0 2024-04-22T15:25:05.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.440+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a8197210 con 0x7f97a81017d0 2024-04-22T15:25:05.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.441+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a8109170 con 0x7f97a81017d0 2024-04-22T15:25:05.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.441+0000 7f97a57fa700 1 -- 172.21.15.96:0/1871800975 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f979800f5d0 con 0x7f97a81017d0 2024-04-22T15:25:05.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.442+0000 7f97a57fa700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f979006ecd0 0x7f9790071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:05.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.442+0000 7f97a57fa700 1 -- 172.21.15.96:0/1871800975 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f97980902e0 con 0x7f97a81017d0 2024-04-22T15:25:05.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.442+0000 7f97a7fff700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f979006ecd0 0x7f9790071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:05.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.442+0000 7f97a7fff700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f979006ecd0 0x7f9790071190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f979c0099d0 tx=0x7f979c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:05.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.447+0000 7f97a57fa700 1 -- 172.21.15.96:0/1871800975 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f979805cc20 con 0x7f97a81017d0 2024-04-22T15:25:05.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.732+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f97a8102300 con 0x7f97a81017d0 2024-04-22T15:25:05.736 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[53642]: 2024-04-22T15:25:05.262+0000 7fd165822700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:25:05.736 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[53642]: 2024-04-22T15:25:05.262+0000 7fd165822700 -1 osd.1 53 *** Got signal Terminated *** 2024-04-22T15:25:05.736 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[53642]: 2024-04-22T15:25:05.262+0000 7fd165822700 -1 osd.1 53 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:25:05.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-mon[112443]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:05.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-mon[112443]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:05.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-mon[112443]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:05.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-mon[112443]: pgmap v66: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:25:05.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2764080917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:05.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4147083478' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:25:05.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:05 smithi096 ceph-mon[112443]: osd.1 marked itself down and dead 2024-04-22T15:25:05.738 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down 2024-04-22T15:25:05.738 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:25:05.738 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:25:05.738 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-22T15:25:05.738 INFO:teuthology.orchestra.run.smithi096.stdout: osd.1 (root=default,host=smithi096) is down 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.732+0000 7f97a57fa700 1 -- 172.21.15.96:0/1871800975 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+290 (secure 0 0 0) 0x7f979805c7b0 con 0x7f97a81017d0 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.734+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f979006ecd0 msgr2=0x7f9790071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.734+0000 7f97ae56c700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f979006ecd0 0x7f9790071190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f979c0099d0 tx=0x7f979c008040 comp rx=0 tx=0).stop 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.734+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 msgr2=0x7f97a819cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.734+0000 7f97ae56c700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 0x7f97a819cea0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f97980092e0 tx=0x7f979800be20 comp rx=0 tx=0).stop 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.735+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 shutdown_connections 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.735+0000 7f97ae56c700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f979006ecd0 0x7f9790071190 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.735+0000 7f97ae56c700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a8100e20 0x7f97a819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.735+0000 7f97ae56c700 1 --2- 172.21.15.96:0/1871800975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81017d0 0x7f97a819cea0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.735+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 >> 172.21.15.96:0/1871800975 conn(0x7f97a80fc6d0 msgr2=0x7f97a80fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:05.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.735+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 shutdown_connections 2024-04-22T15:25:05.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:05.735+0000 7f97ae56c700 1 -- 172.21.15.96:0/1871800975 wait complete. 2024-04-22T15:25:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:05 smithi196 ceph-mon[81815]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:05.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:05 smithi196 ceph-mon[81815]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:05.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:05 smithi196 ceph-mon[81815]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:05.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:05 smithi196 ceph-mon[81815]: pgmap v66: 65 pgs: 65 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:25:05.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:05 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2764080917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:05.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:05 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4147083478' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:25:05.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:05 smithi196 ceph-mon[81815]: osd.1 marked itself down and dead 2024-04-22T15:25:06.100 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:05 smithi096 podman[129284]: 2024-04-22 15:25:05.812777711 +0000 UTC m=+0.675669184 container died 086c04ea748cfb0fe6c3d185c266819643f459bcd3b32b227e0cb8319b928681 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240416, RELEASE=reef-646b8c7, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T15:25:06.484 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:06 smithi096 podman[129284]: 2024-04-22 15:25:06.196534354 +0000 UTC m=+1.059425821 container cleanup 086c04ea748cfb0fe6c3d185c266819643f459bcd3b32b227e0cb8319b928681 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, RELEASE=reef-646b8c7, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T15:25:06.485 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:06 smithi096 bash[129284]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1 2024-04-22T15:25:06.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:06 smithi096 ceph-mon[112443]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:25:06.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:06 smithi096 ceph-mon[112443]: osdmap e54: 6 total, 5 up, 6 in 2024-04-22T15:25:06.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:06 smithi096 ceph-mon[112443]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:06.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:06 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1871800975' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:25:06.736 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:06 smithi096 podman[129328]: 2024-04-22 15:25:06.482341062 +0000 UTC m=+0.667448211 container remove 086c04ea748cfb0fe6c3d185c266819643f459bcd3b32b227e0cb8319b928681 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=reef-646b8c7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-22T15:25:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:06 smithi196 ceph-mon[81815]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:25:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:06 smithi196 ceph-mon[81815]: osdmap e54: 6 total, 5 up, 6 in 2024-04-22T15:25:06.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:06 smithi196 ceph-mon[81815]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:06.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:06 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1871800975' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:25:07.088 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:06 smithi096 podman[129423]: 2024-04-22 15:25:06.72490722 +0000 UTC m=+0.018221001 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:25:07.089 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:06 smithi096 podman[129423]: 2024-04-22 15:25:06.824547407 +0000 UTC m=+0.117861181 container create 76e8f8707551a177d6c2a1eaf67158b30101dd204bec3dd3375f098a0e7ab64b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-22T15:25:07.089 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:07 smithi096 podman[129423]: 2024-04-22 15:25:07.086100734 +0000 UTC m=+0.379414515 container init 76e8f8707551a177d6c2a1eaf67158b30101dd204bec3dd3375f098a0e7ab64b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T15:25:07.388 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:07 smithi096 ceph-mon[112443]: pgmap v68: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-22T15:25:07.388 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:07 smithi096 ceph-mon[112443]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:25:07.388 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:07 smithi096 ceph-mon[112443]: osdmap e55: 6 total, 5 up, 6 in 2024-04-22T15:25:07.389 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:07 smithi096 podman[129423]: 2024-04-22 15:25:07.089730407 +0000 UTC m=+0.383044184 container start 76e8f8707551a177d6c2a1eaf67158b30101dd204bec3dd3375f098a0e7ab64b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T15:25:07.389 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:07 smithi096 podman[129423]: 2024-04-22 15:25:07.185818482 +0000 UTC m=+0.479132270 container attach 76e8f8707551a177d6c2a1eaf67158b30101dd204bec3dd3375f098a0e7ab64b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-22T15:25:07.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:07 smithi196 ceph-mon[81815]: pgmap v68: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-22T15:25:07.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:07 smithi196 ceph-mon[81815]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:25:07.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:07 smithi196 ceph-mon[81815]: osdmap e55: 6 total, 5 up, 6 in 2024-04-22T15:25:07.736 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:07 smithi096 podman[129423]: 2024-04-22 15:25:07.385666613 +0000 UTC m=+0.678980391 container died 76e8f8707551a177d6c2a1eaf67158b30101dd204bec3dd3375f098a0e7ab64b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:25:08.220 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:07 smithi096 podman[129449]: 2024-04-22 15:25:07.981291289 +0000 UTC m=+0.589749781 container remove 76e8f8707551a177d6c2a1eaf67158b30101dd204bec3dd3375f098a0e7ab64b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:25:08.221 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:08 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-22T15:25:08.221 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:08 smithi096 systemd[1]: Stopped Ceph osd.1 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:25:08.221 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:08 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.1.service: Consumed 59.126s CPU time. 2024-04-22T15:25:08.486 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:08 smithi096 systemd[1]: Starting Ceph osd.1 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:25:08.486 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:08 smithi096 podman[129554]: 2024-04-22 15:25:08.343186219 +0000 UTC m=+0.018530270 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:25:08.486 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:08 smithi096 podman[129554]: 2024-04-22 15:25:08.45751971 +0000 UTC m=+0.132863785 container create 499570dc150fb4488469ded57108729dc5f09631bea81b16a251eb4aae583dec (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T15:25:08.986 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:08 smithi096 podman[129554]: 2024-04-22 15:25:08.652479624 +0000 UTC m=+0.327823662 container init 499570dc150fb4488469ded57108729dc5f09631bea81b16a251eb4aae583dec (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T15:25:08.986 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:08 smithi096 podman[129554]: 2024-04-22 15:25:08.656091529 +0000 UTC m=+0.331435562 container start 499570dc150fb4488469ded57108729dc5f09631bea81b16a251eb4aae583dec (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T15:25:08.986 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:08 smithi096 podman[129554]: 2024-04-22 15:25:08.743735738 +0000 UTC m=+0.419079782 container attach 499570dc150fb4488469ded57108729dc5f09631bea81b16a251eb4aae583dec (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T15:25:09.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-mon[112443]: pgmap v70: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-22T15:25:09.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:09.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:25:09.486 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate[129569]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate[129569]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 bash[129554]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 bash[129554]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 bash[129554]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 bash[129554]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 bash[129554]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 bash[129554]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 bash[129554]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 bash[129554]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 bash[129554]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate[129569]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate[129569]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate[129569]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T15:25:09.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate[129569]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T15:25:09.488 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate[129569]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-22T15:25:09.488 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate[129569]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T15:25:09.488 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate[129569]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-22T15:25:09.488 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 podman[129554]: 2024-04-22 15:25:09.163116741 +0000 UTC m=+0.838460773 container died 499570dc150fb4488469ded57108729dc5f09631bea81b16a251eb4aae583dec (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T15:25:09.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:09 smithi196 ceph-mon[81815]: pgmap v70: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-22T15:25:09.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:09.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:25:09.833 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 podman[129710]: 2024-04-22 15:25:09.756303084 +0000 UTC m=+0.588051029 container remove 499570dc150fb4488469ded57108729dc5f09631bea81b16a251eb4aae583dec (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:25:10.096 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 podman[129776]: 2024-04-22 15:25:09.876019436 +0000 UTC m=+0.019341966 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:25:10.096 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:09 smithi096 podman[129776]: 2024-04-22 15:25:09.97400063 +0000 UTC m=+0.117323153 container create 8ca99058d1fb8c3bb7648d662b6dbc6e55faabe92bcf32d3454fc58fa7b011f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T15:25:10.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:10 smithi096 ceph-mon[112443]: Health check failed: Degraded data redundancy: 43/330 objects degraded (13.030%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:10.486 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:10 smithi096 podman[129776]: 2024-04-22 15:25:10.185251044 +0000 UTC m=+0.328573568 container init 8ca99058d1fb8c3bb7648d662b6dbc6e55faabe92bcf32d3454fc58fa7b011f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T15:25:10.486 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:10 smithi096 podman[129776]: 2024-04-22 15:25:10.188219506 +0000 UTC m=+0.331542048 container start 8ca99058d1fb8c3bb7648d662b6dbc6e55faabe92bcf32d3454fc58fa7b011f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T15:25:10.486 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:10 smithi096 ceph-osd[129795]: -- 172.21.15.96:0/2238229223 <== mon.0 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564f78433680 con 0x564f783fd000 2024-04-22T15:25:10.486 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:10 smithi096 bash[129776]: 8ca99058d1fb8c3bb7648d662b6dbc6e55faabe92bcf32d3454fc58fa7b011f4 2024-04-22T15:25:10.487 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:10 smithi096 systemd[1]: Started Ceph osd.1 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:25:10.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:10 smithi196 ceph-mon[81815]: Health check failed: Degraded data redundancy: 43/330 objects degraded (13.030%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:10.986 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:10 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[129791]: 2024-04-22T15:25:10.731+0000 7f7ff89607c0 -1 Falling back to public interface 2024-04-22T15:25:11.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:11 smithi096 ceph-mon[112443]: pgmap v71: 65 pgs: 13 active+undersized+degraded, 5 peering, 16 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 43/330 objects degraded (13.030%) 2024-04-22T15:25:11.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:11 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:11.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:11 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:11.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:11 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:25:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:11 smithi196 ceph-mon[81815]: pgmap v71: 65 pgs: 13 active+undersized+degraded, 5 peering, 16 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 43/330 objects degraded (13.030%) 2024-04-22T15:25:11.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:11 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:11.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:11 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:11.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:11 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:25:12.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:12 smithi096 ceph-mon[112443]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T15:25:12.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:12 smithi196 ceph-mon[81815]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T15:25:13.722 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:13 smithi096 ceph-mon[112443]: pgmap v72: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 47/330 objects degraded (14.242%) 2024-04-22T15:25:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:13 smithi196 ceph-mon[81815]: pgmap v72: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 47/330 objects degraded (14.242%) 2024-04-22T15:25:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:15 smithi096 ceph-mon[112443]: pgmap v73: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 47/330 objects degraded (14.242%) 2024-04-22T15:25:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:15 smithi196 ceph-mon[81815]: pgmap v73: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 47/330 objects degraded (14.242%) 2024-04-22T15:25:15.986 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:15 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[129791]: 2024-04-22T15:25:15.546+0000 7f7ff89607c0 -1 osd.1 53 log_to_monitors true 2024-04-22T15:25:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:16 smithi096 ceph-mon[112443]: from='osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T15:25:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:16 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:16 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:16.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:16 smithi196 ceph-mon[81815]: from='osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T15:25:16.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:16 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:16.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:16 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:17.671 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 15:25:17 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[129791]: 2024-04-22T15:25:17.326+0000 7f7fe5db0700 -1 osd.1 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:25:17.671 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:17 smithi096 ceph-mon[112443]: pgmap v74: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 47/330 objects degraded (14.242%) 2024-04-22T15:25:17.671 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:17 smithi096 ceph-mon[112443]: from='osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T15:25:17.671 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:17 smithi096 ceph-mon[112443]: osdmap e56: 6 total, 5 up, 6 in 2024-04-22T15:25:17.671 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:17 smithi096 ceph-mon[112443]: from='osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:25:17.671 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:17 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:17.671 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:17 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:17.672 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:17 smithi096 ceph-mon[112443]: from='osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374]' entity='osd.1' 2024-04-22T15:25:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:17 smithi196 ceph-mon[81815]: pgmap v74: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 47/330 objects degraded (14.242%) 2024-04-22T15:25:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:17 smithi196 ceph-mon[81815]: from='osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T15:25:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:17 smithi196 ceph-mon[81815]: osdmap e56: 6 total, 5 up, 6 in 2024-04-22T15:25:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:17 smithi196 ceph-mon[81815]: from='osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:25:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:17 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:17 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:17.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:17 smithi196 ceph-mon[81815]: from='osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374]' entity='osd.1' 2024-04-22T15:25:18.654 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:18 smithi096 ceph-mon[112443]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:25:18.654 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:18 smithi096 ceph-mon[112443]: osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374] boot 2024-04-22T15:25:18.654 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:18 smithi096 ceph-mon[112443]: osdmap e57: 6 total, 6 up, 6 in 2024-04-22T15:25:18.654 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:18 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:25:18.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:18 smithi196 ceph-mon[81815]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:25:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:18 smithi196 ceph-mon[81815]: osd.1 [v2:172.21.15.96:6810/3607865374,v1:172.21.15.96:6811/3607865374] boot 2024-04-22T15:25:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:18 smithi196 ceph-mon[81815]: osdmap e57: 6 total, 6 up, 6 in 2024-04-22T15:25:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:18 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:25:19.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:19 smithi196 ceph-mon[81815]: pgmap v76: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 47/330 objects degraded (14.242%) 2024-04-22T15:25:19.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:19 smithi196 ceph-mon[81815]: osdmap e58: 6 total, 6 up, 6 in 2024-04-22T15:25:19.530 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:19 smithi096 ceph-mon[112443]: pgmap v76: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 47/330 objects degraded (14.242%) 2024-04-22T15:25:19.531 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:19 smithi096 ceph-mon[112443]: osdmap e58: 6 total, 6 up, 6 in 2024-04-22T15:25:20.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:20 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 47/330 objects degraded (14.242%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:20.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:20 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 47/330 objects degraded (14.242%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: pgmap v79: 65 pgs: 20 peering, 7 active+undersized+degraded, 7 active+undersized, 31 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 24/330 objects degraded (7.273%) 2024-04-22T15:25:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:25:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:25:21.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:21.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:25:21.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:21.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:21.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:21.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:25:21.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: pgmap v79: 65 pgs: 20 peering, 7 active+undersized+degraded, 7 active+undersized, 31 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 24/330 objects degraded (7.273%) 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:21.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:25:22.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:22 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:25:22.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:22 smithi096 ceph-mon[112443]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-22T15:25:22.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:22 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:25:22.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:22 smithi196 ceph-mon[81815]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-22T15:25:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:23 smithi096 ceph-mon[112443]: pgmap v80: 65 pgs: 20 peering, 45 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-22T15:25:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:23 smithi096 ceph-mon[112443]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/330 objects degraded (7.273%), 7 pgs degraded) 2024-04-22T15:25:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:25:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:23 smithi196 ceph-mon[81815]: pgmap v80: 65 pgs: 20 peering, 45 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-22T15:25:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:23 smithi196 ceph-mon[81815]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/330 objects degraded (7.273%), 7 pgs degraded) 2024-04-22T15:25:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:25:24.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:24 smithi196 ceph-mon[81815]: pgmap v81: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-22T15:25:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:24 smithi096 ceph-mon[112443]: pgmap v81: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-22T15:25:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:27 smithi096 ceph-mon[112443]: pgmap v82: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-22T15:25:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:27 smithi196 ceph-mon[81815]: pgmap v82: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-22T15:25:29.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:29 smithi096 ceph-mon[112443]: pgmap v83: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-22T15:25:29.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:29 smithi196 ceph-mon[81815]: pgmap v83: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-22T15:25:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:31 smithi096 ceph-mon[112443]: pgmap v84: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-22T15:25:31.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:31 smithi196 ceph-mon[81815]: pgmap v84: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-22T15:25:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:33 smithi096 ceph-mon[112443]: pgmap v85: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:25:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:33 smithi196 ceph-mon[81815]: pgmap v85: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:25:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:35.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:35 smithi096 ceph-mon[112443]: pgmap v86: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:25:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:35 smithi196 ceph-mon[81815]: pgmap v86: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:25:35.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.853+0000 7f24d4e39700 1 -- 172.21.15.96:0/345740130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 msgr2=0x7f24d0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:35.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.853+0000 7f24d4e39700 1 --2- 172.21.15.96:0/345740130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 0x7f24d0101200 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f24c0009910 tx=0x7f24c0009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.853+0000 7f24d4e39700 1 -- 172.21.15.96:0/345740130 shutdown_connections 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.853+0000 7f24d4e39700 1 --2- 172.21.15.96:0/345740130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24d01017d0 0x7f24d0105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.853+0000 7f24d4e39700 1 --2- 172.21.15.96:0/345740130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 0x7f24d0101200 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.853+0000 7f24d4e39700 1 -- 172.21.15.96:0/345740130 >> 172.21.15.96:0/345740130 conn(0x7f24d00fc6d0 msgr2=0x7f24d00feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.853+0000 7f24d4e39700 1 -- 172.21.15.96:0/345740130 shutdown_connections 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.853+0000 7f24d4e39700 1 -- 172.21.15.96:0/345740130 wait complete. 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24d4e39700 1 Processor -- start 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24d4e39700 1 -- start start 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24d4e39700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 0x7f24d019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24d4e39700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24d01017d0 0x7f24d019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24d4e39700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d019f610 con 0x7f24d0100e20 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24d4e39700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d0198b80 con 0x7f24d01017d0 2024-04-22T15:25:35.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24ce59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 0x7f24d019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24ce59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 0x7f24d019ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45574/0 (socket says 172.21.15.96:45574) 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24ce59c700 1 -- 172.21.15.96:0/1698167240 learned_addr learned my addr 172.21.15.96:0/1698167240 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24cdd9b700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24d01017d0 0x7f24d019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24ce59c700 1 -- 172.21.15.96:0/1698167240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24d01017d0 msgr2=0x7f24d019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.854+0000 7f24ce59c700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24d01017d0 0x7f24d019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.855+0000 7f24ce59c700 1 -- 172.21.15.96:0/1698167240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c00095f0 con 0x7f24d0100e20 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.855+0000 7f24ce59c700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 0x7f24d019ea40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f24c000bd70 tx=0x7f24c000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.855+0000 7f24bf7fe700 1 -- 172.21.15.96:0/1698167240 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24c001d070 con 0x7f24d0100e20 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.855+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d0198e00 con 0x7f24d0100e20 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.855+0000 7f24bf7fe700 1 -- 172.21.15.96:0/1698167240 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24c0005700 con 0x7f24d0100e20 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.855+0000 7f24bf7fe700 1 -- 172.21.15.96:0/1698167240 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24c0017400 con 0x7f24d0100e20 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.855+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d01992f0 con 0x7f24d0100e20 2024-04-22T15:25:35.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.856+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d0104fa0 con 0x7f24d0100e20 2024-04-22T15:25:35.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.857+0000 7f24bf7fe700 1 -- 172.21.15.96:0/1698167240 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f24c0003a60 con 0x7f24d0100e20 2024-04-22T15:25:35.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.857+0000 7f24bf7fe700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f24b806ecc0 0x7f24b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:35.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.857+0000 7f24bf7fe700 1 -- 172.21.15.96:0/1698167240 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f24c0090290 con 0x7f24d0100e20 2024-04-22T15:25:35.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.857+0000 7f24cdd9b700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f24b806ecc0 0x7f24b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:35.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.858+0000 7f24cdd9b700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f24b806ecc0 0x7f24b8071180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f24d00fdd00 tx=0x7f24c400a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:35.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:35.862+0000 7f24bf7fe700 1 -- 172.21.15.96:0/1698167240 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f24c005cbd0 con 0x7f24d0100e20 2024-04-22T15:25:36.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.085+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f24d010ddf0 con 0x7f24b806ecc0 2024-04-22T15:25:36.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.087+0000 7f24bf7fe700 1 -- 172.21.15.96:0/1698167240 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f24d010ddf0 con 0x7f24b806ecc0 2024-04-22T15:25:36.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f24b806ecc0 msgr2=0x7f24b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f24b806ecc0 0x7f24b8071180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f24d00fdd00 tx=0x7f24c400a300 comp rx=0 tx=0).stop 2024-04-22T15:25:36.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 msgr2=0x7f24d019ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 0x7f24d019ea40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f24c000bd70 tx=0x7f24c000bda0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 shutdown_connections 2024-04-22T15:25:36.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f24b806ecc0 0x7f24b8071180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d0100e20 0x7f24d019ea40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 --2- 172.21.15.96:0/1698167240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24d01017d0 0x7f24d019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 >> 172.21.15.96:0/1698167240 conn(0x7f24d00fc6d0 msgr2=0x7f24d00fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:36.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 shutdown_connections 2024-04-22T15:25:36.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.089+0000 7f24d4e39700 1 -- 172.21.15.96:0/1698167240 wait complete. 2024-04-22T15:25:36.103 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:25:36.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.199+0000 7f30405ca700 1 -- 172.21.15.96:0/3375197157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ff520 msgr2=0x7f30380ff900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.199+0000 7f30405ca700 1 --2- 172.21.15.96:0/3375197157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ff520 0x7f30380ff900 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3028009910 tx=0x7f3028009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:36.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.199+0000 7f30405ca700 1 -- 172.21.15.96:0/3375197157 shutdown_connections 2024-04-22T15:25:36.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.199+0000 7f30405ca700 1 --2- 172.21.15.96:0/3375197157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30380ffed0 0x7f3038103f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.199+0000 7f30405ca700 1 --2- 172.21.15.96:0/3375197157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ff520 0x7f30380ff900 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.199+0000 7f30405ca700 1 -- 172.21.15.96:0/3375197157 >> 172.21.15.96:0/3375197157 conn(0x7f3038075f00 msgr2=0x7f30380fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:36.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.199+0000 7f30405ca700 1 -- 172.21.15.96:0/3375197157 shutdown_connections 2024-04-22T15:25:36.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.199+0000 7f30405ca700 1 -- 172.21.15.96:0/3375197157 wait complete. 2024-04-22T15:25:36.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.199+0000 7f30405ca700 1 Processor -- start 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f30405ca700 1 -- start start 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f30405ca700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30380ff520 0x7f3038197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f30405ca700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ffed0 0x7f3038198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f30405ca700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3038198560 con 0x7f30380ffed0 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f30405ca700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30381986d0 con 0x7f30380ff520 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f303db65700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ffed0 0x7f3038198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f303db65700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ffed0 0x7f3038198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45606/0 (socket says 172.21.15.96:45606) 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f303db65700 1 -- 172.21.15.96:0/2286617043 learned_addr learned my addr 172.21.15.96:0/2286617043 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f303e366700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30380ff520 0x7f3038197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f303db65700 1 -- 172.21.15.96:0/2286617043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30380ff520 msgr2=0x7f3038197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f303db65700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30380ff520 0x7f3038197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f303db65700 1 -- 172.21.15.96:0/2286617043 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30280095f0 con 0x7f30380ffed0 2024-04-22T15:25:36.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f303db65700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ffed0 0x7f3038198020 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f303400b480 tx=0x7f303400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:36.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.200+0000 7f302f7fe700 1 -- 172.21.15.96:0/2286617043 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30340118c0 con 0x7f30380ffed0 2024-04-22T15:25:36.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.201+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3038198960 con 0x7f30380ffed0 2024-04-22T15:25:36.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.201+0000 7f302f7fe700 1 -- 172.21.15.96:0/2286617043 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3034010380 con 0x7f30380ffed0 2024-04-22T15:25:36.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.201+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30381a0eb0 con 0x7f30380ffed0 2024-04-22T15:25:36.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.201+0000 7f302f7fe700 1 -- 172.21.15.96:0/2286617043 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303400f4f0 con 0x7f30380ffed0 2024-04-22T15:25:36.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.201+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30381078c0 con 0x7f30380ffed0 2024-04-22T15:25:36.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.202+0000 7f302f7fe700 1 -- 172.21.15.96:0/2286617043 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f30340104f0 con 0x7f30380ffed0 2024-04-22T15:25:36.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.202+0000 7f302f7fe700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f302406ecd0 0x7f3024071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:36.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.203+0000 7f302f7fe700 1 -- 172.21.15.96:0/2286617043 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f303408e9b0 con 0x7f30380ffed0 2024-04-22T15:25:36.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.203+0000 7f303e366700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f302406ecd0 0x7f3024071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:36.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.203+0000 7f303e366700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f302406ecd0 0x7f3024071190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f302800b5c0 tx=0x7f302801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:36.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.212+0000 7f302f7fe700 1 -- 172.21.15.96:0/2286617043 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f30340577c0 con 0x7f30380ffed0 2024-04-22T15:25:36.444 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:36 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:25:36.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.440+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f303810cde0 con 0x7f302406ecd0 2024-04-22T15:25:36.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.442+0000 7f302f7fe700 1 -- 172.21.15.96:0/2286617043 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f303810cde0 con 0x7f302406ecd0 2024-04-22T15:25:36.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f302406ecd0 msgr2=0x7f3024071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f302406ecd0 0x7f3024071190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f302800b5c0 tx=0x7f302801a040 comp rx=0 tx=0).stop 2024-04-22T15:25:36.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ffed0 msgr2=0x7f3038198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ffed0 0x7f3038198020 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f303400b480 tx=0x7f303400b840 comp rx=0 tx=0).stop 2024-04-22T15:25:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 shutdown_connections 2024-04-22T15:25:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f302406ecd0 0x7f3024071190 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30380ff520 0x7f3038197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 --2- 172.21.15.96:0/2286617043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30380ffed0 0x7f3038198020 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 >> 172.21.15.96:0/2286617043 conn(0x7f3038075f00 msgr2=0x7f3038102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 shutdown_connections 2024-04-22T15:25:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.445+0000 7f30405ca700 1 -- 172.21.15.96:0/2286617043 wait complete. 2024-04-22T15:25:36.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.558+0000 7fc705322700 1 -- 172.21.15.96:0/2166491443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 msgr2=0x7fc700101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.558+0000 7fc705322700 1 --2- 172.21.15.96:0/2166491443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 0x7fc700101200 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009910 tx=0x7fc6f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:36.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.558+0000 7fc705322700 1 -- 172.21.15.96:0/2166491443 shutdown_connections 2024-04-22T15:25:36.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.558+0000 7fc705322700 1 --2- 172.21.15.96:0/2166491443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc7001017d0 0x7fc700105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.558+0000 7fc705322700 1 --2- 172.21.15.96:0/2166491443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 0x7fc700101200 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.558+0000 7fc705322700 1 -- 172.21.15.96:0/2166491443 >> 172.21.15.96:0/2166491443 conn(0x7fc7000fc6d0 msgr2=0x7fc7000feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:36.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.558+0000 7fc705322700 1 -- 172.21.15.96:0/2166491443 shutdown_connections 2024-04-22T15:25:36.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.558+0000 7fc705322700 1 -- 172.21.15.96:0/2166491443 wait complete. 2024-04-22T15:25:36.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc705322700 1 Processor -- start 2024-04-22T15:25:36.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc705322700 1 -- start start 2024-04-22T15:25:36.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc705322700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 0x7fc70019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:36.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc705322700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc7001017d0 0x7fc70019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc705322700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70019d3b0 con 0x7fc700100e20 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc705322700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc700196990 con 0x7fc7001017d0 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc6feffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 0x7fc70019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc6feffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 0x7fc70019c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45634/0 (socket says 172.21.15.96:45634) 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc6feffd700 1 -- 172.21.15.96:0/2812316286 learned_addr learned my addr 172.21.15.96:0/2812316286 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.559+0000 7fc6fe7fc700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc7001017d0 0x7fc70019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.560+0000 7fc6feffd700 1 -- 172.21.15.96:0/2812316286 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc7001017d0 msgr2=0x7fc70019cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.560+0000 7fc6feffd700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc7001017d0 0x7fc70019cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.560+0000 7fc6feffd700 1 -- 172.21.15.96:0/2812316286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f00095f0 con 0x7fc700100e20 2024-04-22T15:25:36.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.560+0000 7fc6feffd700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 0x7fc70019c850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0006010 tx=0x7fc6f000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:36.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.560+0000 7fc6e7fff700 1 -- 172.21.15.96:0/2812316286 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6f001d070 con 0x7fc700100e20 2024-04-22T15:25:36.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.560+0000 7fc6e7fff700 1 -- 172.21.15.96:0/2812316286 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6f00056e0 con 0x7fc700100e20 2024-04-22T15:25:36.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.560+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc700196c10 con 0x7fc700100e20 2024-04-22T15:25:36.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.560+0000 7fc6e7fff700 1 -- 172.21.15.96:0/2812316286 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6f000fe80 con 0x7fc700100e20 2024-04-22T15:25:36.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.560+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc700197100 con 0x7fc700100e20 2024-04-22T15:25:36.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.561+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7001091c0 con 0x7fc700100e20 2024-04-22T15:25:36.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.561+0000 7fc6e7fff700 1 -- 172.21.15.96:0/2812316286 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc6f000f6f0 con 0x7fc700100e20 2024-04-22T15:25:36.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.562+0000 7fc6e7fff700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc6e806ec60 0x7fc6e8071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:36.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.562+0000 7fc6e7fff700 1 -- 172.21.15.96:0/2812316286 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fc6f0090160 con 0x7fc700100e20 2024-04-22T15:25:36.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.562+0000 7fc6fe7fc700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc6e806ec60 0x7fc6e8071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:36.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.562+0000 7fc6fe7fc700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc6e806ec60 0x7fc6e8071120 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc700197b50 tx=0x7fc6f400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:36.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.567+0000 7fc6e7fff700 1 -- 172.21.15.96:0/2812316286 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc6f005caa0 con 0x7fc700100e20 2024-04-22T15:25:36.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:36 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:25:36.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.786+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc700197e50 con 0x7fc6e806ec60 2024-04-22T15:25:36.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.797+0000 7fc6e7fff700 1 -- 172.21.15.96:0/2812316286 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc700197e50 con 0x7fc6e806ec60 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (4m) 20s ago 15m 25.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (15m) 20s ago 15m 8623k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (14m) 80s ago 14m 8166k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (89s) 20s ago 15m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (84s) 80s ago 14m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (3m) 20s ago 15m 82.2M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (9m) 20s ago 9m 155M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (9m) 20s ago 9m 90.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (9m) 80s ago 9m 27.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (9m) 80s ago 9m 26.3M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:25:36.801 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (6m) 20s ago 17m 504M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (5m) 80s ago 14m 450M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (2m) 20s ago 17m 46.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (2m) 80s ago 14m 35.4M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (5m) 20s ago 15m 17.0M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (5m) 80s ago 14m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (62s) 20s ago 13m 90.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (26s) 20s ago 13m 15.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (12m) 20s ago 12m 183M 1216M 18.2.2-1237-g646b8c7d 9900ed1a6b81 bccf2e539131 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (12m) 80s ago 12m 242M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (11m) 80s ago 11m 213M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (11m) 80s ago 11m 196M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:25:36.802 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (4m) 20s ago 15m 76.7M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:25:36.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.799+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc6e806ec60 msgr2=0x7fc6e8071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.799+0000 7fc705322700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc6e806ec60 0x7fc6e8071120 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc700197b50 tx=0x7fc6f400b410 comp rx=0 tx=0).stop 2024-04-22T15:25:36.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.799+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 msgr2=0x7fc70019c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.799+0000 7fc705322700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 0x7fc70019c850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0006010 tx=0x7fc6f000bd70 comp rx=0 tx=0).stop 2024-04-22T15:25:36.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.799+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 shutdown_connections 2024-04-22T15:25:36.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.799+0000 7fc705322700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc6e806ec60 0x7fc6e8071120 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.799+0000 7fc705322700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc700100e20 0x7fc70019c850 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.799+0000 7fc705322700 1 --2- 172.21.15.96:0/2812316286 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc7001017d0 0x7fc70019cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.799+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 >> 172.21.15.96:0/2812316286 conn(0x7fc7000fc6d0 msgr2=0x7fc7000fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:36.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.800+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 shutdown_connections 2024-04-22T15:25:36.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.800+0000 7fc705322700 1 -- 172.21.15.96:0/2812316286 wait complete. 2024-04-22T15:25:36.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.904+0000 7fe35011d700 1 -- 172.21.15.96:0/4035404864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe348105ab0 msgr2=0x7fe348105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.904+0000 7fe35011d700 1 --2- 172.21.15.96:0/4035404864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe348105ab0 0x7fe348105e90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe344009910 tx=0x7fe344009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.904+0000 7fe35011d700 1 -- 172.21.15.96:0/4035404864 shutdown_connections 2024-04-22T15:25:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.904+0000 7fe35011d700 1 --2- 172.21.15.96:0/4035404864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3480ffab0 0x7fe3480fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.904+0000 7fe35011d700 1 --2- 172.21.15.96:0/4035404864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe348105ab0 0x7fe348105e90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.904+0000 7fe35011d700 1 -- 172.21.15.96:0/4035404864 >> 172.21.15.96:0/4035404864 conn(0x7fe3480fb5e0 msgr2=0x7fe3480fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.904+0000 7fe35011d700 1 -- 172.21.15.96:0/4035404864 shutdown_connections 2024-04-22T15:25:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.904+0000 7fe35011d700 1 -- 172.21.15.96:0/4035404864 wait complete. 2024-04-22T15:25:36.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe35011d700 1 Processor -- start 2024-04-22T15:25:36.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe35011d700 1 -- start start 2024-04-22T15:25:36.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe35011d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3480ffab0 0x7fe34819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe35011d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe348105ab0 0x7fe34819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe35011d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34819d400 con 0x7fe3480ffab0 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe35011d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe348196a70 con 0x7fe348105ab0 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe34deb9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3480ffab0 0x7fe34819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe34deb9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3480ffab0 0x7fe34819c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45652/0 (socket says 172.21.15.96:45652) 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe34deb9700 1 -- 172.21.15.96:0/822186410 learned_addr learned my addr 172.21.15.96:0/822186410 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe34d6b8700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe348105ab0 0x7fe34819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe34deb9700 1 -- 172.21.15.96:0/822186410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe348105ab0 msgr2=0x7fe34819ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe34deb9700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe348105ab0 0x7fe34819ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.905+0000 7fe34deb9700 1 -- 172.21.15.96:0/822186410 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440095f0 con 0x7fe3480ffab0 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.906+0000 7fe34deb9700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3480ffab0 0x7fe34819c930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe344006010 tx=0x7fe34400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.906+0000 7fe33effd700 1 -- 172.21.15.96:0/822186410 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe34401d070 con 0x7fe3480ffab0 2024-04-22T15:25:36.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.906+0000 7fe33effd700 1 -- 172.21.15.96:0/822186410 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe34400f460 con 0x7fe3480ffab0 2024-04-22T15:25:36.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.906+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe348196cf0 con 0x7fe3480ffab0 2024-04-22T15:25:36.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.906+0000 7fe33effd700 1 -- 172.21.15.96:0/822186410 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe344017630 con 0x7fe3480ffab0 2024-04-22T15:25:36.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.906+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3481971e0 con 0x7fe3480ffab0 2024-04-22T15:25:36.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.907+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3481080b0 con 0x7fe3480ffab0 2024-04-22T15:25:36.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.907+0000 7fe33effd700 1 -- 172.21.15.96:0/822186410 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe34400f5d0 con 0x7fe3480ffab0 2024-04-22T15:25:36.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.908+0000 7fe33effd700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe33406ecc0 0x7fe334071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:36.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.908+0000 7fe33effd700 1 -- 172.21.15.96:0/822186410 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fe3440902a0 con 0x7fe3480ffab0 2024-04-22T15:25:36.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.908+0000 7fe34d6b8700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe33406ecc0 0x7fe334071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:36.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.908+0000 7fe34d6b8700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe33406ecc0 0x7fe334071180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe338005ea0 tx=0x7fe338005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:36.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:36.913+0000 7fe33effd700 1 -- 172.21.15.96:0/822186410 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe34405cca0 con 0x7fe3480ffab0 2024-04-22T15:25:37.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.209+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe348197f30 con 0x7fe3480ffab0 2024-04-22T15:25:37.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.210+0000 7fe33effd700 1 -- 172.21.15.96:0/822186410 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+836 (secure 0 0 0) 0x7fe34405c830 con 0x7fe3480ffab0 2024-04-22T15:25:37.213 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:25:37.213 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:25:37.213 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:25:37.213 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:25:37.213 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:25:37.213 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4, 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 8, 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:25:37.214 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:25:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe33406ecc0 msgr2=0x7fe334071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe33406ecc0 0x7fe334071180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe338005ea0 tx=0x7fe338005e30 comp rx=0 tx=0).stop 2024-04-22T15:25:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3480ffab0 msgr2=0x7fe34819c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3480ffab0 0x7fe34819c930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe344006010 tx=0x7fe34400be20 comp rx=0 tx=0).stop 2024-04-22T15:25:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 shutdown_connections 2024-04-22T15:25:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe33406ecc0 0x7fe334071180 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3480ffab0 0x7fe34819c930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 --2- 172.21.15.96:0/822186410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe348105ab0 0x7fe34819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 >> 172.21.15.96:0/822186410 conn(0x7fe3480fb5e0 msgr2=0x7fe348109f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 shutdown_connections 2024-04-22T15:25:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.212+0000 7fe35011d700 1 -- 172.21.15.96:0/822186410 wait complete. 2024-04-22T15:25:37.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.317+0000 7f942b4a8700 1 -- 172.21.15.96:0/1456363562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 msgr2=0x7f9424105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.317+0000 7f942b4a8700 1 --2- 172.21.15.96:0/1456363562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 0x7f9424105810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9414009910 tx=0x7f9414009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:37.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.318+0000 7f942b4a8700 1 -- 172.21.15.96:0/1456363562 shutdown_connections 2024-04-22T15:25:37.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.318+0000 7f942b4a8700 1 --2- 172.21.15.96:0/1456363562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 0x7f9424105810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.318+0000 7f942b4a8700 1 --2- 172.21.15.96:0/1456363562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9424100e10 0x7f94241011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.318+0000 7f942b4a8700 1 -- 172.21.15.96:0/1456363562 >> 172.21.15.96:0/1456363562 conn(0x7f94240fc6e0 msgr2=0x7f94240feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:37.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.318+0000 7f942b4a8700 1 -- 172.21.15.96:0/1456363562 shutdown_connections 2024-04-22T15:25:37.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.318+0000 7f942b4a8700 1 -- 172.21.15.96:0/1456363562 wait complete. 2024-04-22T15:25:37.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.318+0000 7f942b4a8700 1 Processor -- start 2024-04-22T15:25:37.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.318+0000 7f942b4a8700 1 -- start start 2024-04-22T15:25:37.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f942b4a8700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9424100e10 0x7f942419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:37.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f942b4a8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 0x7f942419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:37.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f942b4a8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942419d380 con 0x7f94241017c0 2024-04-22T15:25:37.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f942b4a8700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94241969d0 con 0x7f9424100e10 2024-04-22T15:25:37.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f9428a43700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 0x7f942419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:37.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f9428a43700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 0x7f942419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45664/0 (socket says 172.21.15.96:45664) 2024-04-22T15:25:37.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f9428a43700 1 -- 172.21.15.96:0/496713584 learned_addr learned my addr 172.21.15.96:0/496713584 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f9429244700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9424100e10 0x7f942419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f9428a43700 1 -- 172.21.15.96:0/496713584 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9424100e10 msgr2=0x7f942419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f9428a43700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9424100e10 0x7f942419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f9428a43700 1 -- 172.21.15.96:0/496713584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94140095f0 con 0x7f94241017c0 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f9428a43700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 0x7f942419cd80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9414005b40 tx=0x7f941400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f941a7fc700 1 -- 172.21.15.96:0/496713584 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941401d070 con 0x7f94241017c0 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9424196c50 con 0x7f94241017c0 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f941a7fc700 1 -- 172.21.15.96:0/496713584 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f941400f460 con 0x7f94241017c0 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f941a7fc700 1 -- 172.21.15.96:0/496713584 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9414017630 con 0x7f94241017c0 2024-04-22T15:25:37.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.319+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9424197140 con 0x7f94241017c0 2024-04-22T15:25:37.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.320+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9424109210 con 0x7f94241017c0 2024-04-22T15:25:37.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.321+0000 7f941a7fc700 1 -- 172.21.15.96:0/496713584 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9414003a60 con 0x7f94241017c0 2024-04-22T15:25:37.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.321+0000 7f941a7fc700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f941006ecd0 0x7f9410071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:37.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.321+0000 7f9429244700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f941006ecd0 0x7f9410071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:37.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.321+0000 7f941a7fc700 1 -- 172.21.15.96:0/496713584 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f94140902b0 con 0x7f94241017c0 2024-04-22T15:25:37.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.322+0000 7f9429244700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f941006ecd0 0x7f9410071190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f942000ba60 tx=0x7f9420005d50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:37.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.326+0000 7f941a7fc700 1 -- 172.21.15.96:0/496713584 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f941405cbf0 con 0x7f94241017c0 2024-04-22T15:25:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:37 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:25:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:37 smithi096 ceph-mon[112443]: Upgrade: osd.2 is safe to restart 2024-04-22T15:25:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:37 smithi096 ceph-mon[112443]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:37 smithi096 ceph-mon[112443]: pgmap v87: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:25:37.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:37 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/822186410' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:37.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.575+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9424197e40 con 0x7f94241017c0 2024-04-22T15:25:37.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.575+0000 7f941a7fc700 1 -- 172.21.15.96:0/496713584 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1901 (secure 0 0 0) 0x7f941405c780 con 0x7f94241017c0 2024-04-22T15:25:37.580 INFO:teuthology.orchestra.run.smithi096.stdout:e20 2024-04-22T15:25:37.580 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:25:37.580 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:25:37.580 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:25:37.580 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:25:37.580 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:25:37.580 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:25:37.581 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 20 2024-04-22T15:25:37.581 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:25:37.581 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:25:37.581 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:25:37.581 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:25:37.581 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:25:37.581 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:25:37.581 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:25:37.582 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:25:37.582 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:25:37.582 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:25:37.582 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:25:37.582 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:25:37.582 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:25:37.582 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:25:37.583 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:25:37.583 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:25:37.583 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:25:37.583 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:25:37.583 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:25:37.583 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:25:37.583 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:25:37.584 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:25:37.584 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:25:37.584 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:25:37.584 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:25:37.584 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:25:37.584 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:25:37.585 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:25:37.585 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:25:37.585 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:25:37.585 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:25:37.586 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:25:37.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.577+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f941006ecd0 msgr2=0x7f9410071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.577+0000 7f942b4a8700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f941006ecd0 0x7f9410071190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f942000ba60 tx=0x7f9420005d50 comp rx=0 tx=0).stop 2024-04-22T15:25:37.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.577+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 msgr2=0x7f942419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.577+0000 7f942b4a8700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 0x7f942419cd80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9414005b40 tx=0x7f941400be20 comp rx=0 tx=0).stop 2024-04-22T15:25:37.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.578+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 shutdown_connections 2024-04-22T15:25:37.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.578+0000 7f942b4a8700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f941006ecd0 0x7f9410071190 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.578+0000 7f942b4a8700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9424100e10 0x7f942419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.578+0000 7f942b4a8700 1 --2- 172.21.15.96:0/496713584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94241017c0 0x7f942419cd80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.578+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 >> 172.21.15.96:0/496713584 conn(0x7f94240fc6e0 msgr2=0x7f94240fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:37.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.578+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 shutdown_connections 2024-04-22T15:25:37.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.578+0000 7f942b4a8700 1 -- 172.21.15.96:0/496713584 wait complete. 2024-04-22T15:25:37.588 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:25:37.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:37 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:25:37.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:37 smithi196 ceph-mon[81815]: Upgrade: osd.2 is safe to restart 2024-04-22T15:25:37.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:37 smithi196 ceph-mon[81815]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:37.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:37 smithi196 ceph-mon[81815]: pgmap v87: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:25:37.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:37 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/822186410' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.683+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/3408858101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 msgr2=0x7fbc781011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.683+0000 7fbc7d0d8700 1 --2- 172.21.15.96:0/3408858101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 0x7fbc781011d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009910 tx=0x7fbc68009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.683+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/3408858101 shutdown_connections 2024-04-22T15:25:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.683+0000 7fbc7d0d8700 1 --2- 172.21.15.96:0/3408858101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc781017a0 0x7fbc781057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.683+0000 7fbc7d0d8700 1 --2- 172.21.15.96:0/3408858101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 0x7fbc781011d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.683+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/3408858101 >> 172.21.15.96:0/3408858101 conn(0x7fbc780fc6a0 msgr2=0x7fbc780feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.683+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/3408858101 shutdown_connections 2024-04-22T15:25:37.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.683+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/3408858101 wait complete. 2024-04-22T15:25:37.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.684+0000 7fbc7d0d8700 1 Processor -- start 2024-04-22T15:25:37.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.684+0000 7fbc7d0d8700 1 -- start start 2024-04-22T15:25:37.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc7d0d8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 0x7fbc780731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:37.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc7d0d8700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc781017a0 0x7fbc78073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:37.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc7d0d8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78073c60 con 0x7fbc78100df0 2024-04-22T15:25:37.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc7d0d8700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78073da0 con 0x7fbc781017a0 2024-04-22T15:25:37.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc76d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 0x7fbc780731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc76d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 0x7fbc780731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45678/0 (socket says 172.21.15.96:45678) 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc76d9d700 1 -- 172.21.15.96:0/2983229068 learned_addr learned my addr 172.21.15.96:0/2983229068 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc7659c700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc781017a0 0x7fbc78073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc76d9d700 1 -- 172.21.15.96:0/2983229068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc781017a0 msgr2=0x7fbc78073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc76d9d700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc781017a0 0x7fbc78073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc76d9d700 1 -- 172.21.15.96:0/2983229068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc680095f0 con 0x7fbc78100df0 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc76d9d700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 0x7fbc780731e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800bd70 tx=0x7fbc6800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc5ffff700 1 -- 172.21.15.96:0/2983229068 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc6801d070 con 0x7fbc78100df0 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc78074020 con 0x7fbc78100df0 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc5ffff700 1 -- 172.21.15.96:0/2983229068 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc680056e0 con 0x7fbc78100df0 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc5ffff700 1 -- 172.21.15.96:0/2983229068 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc6800fe40 con 0x7fbc78100df0 2024-04-22T15:25:37.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.685+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc781a2f30 con 0x7fbc78100df0 2024-04-22T15:25:37.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.686+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc78109190 con 0x7fbc78100df0 2024-04-22T15:25:37.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.687+0000 7fbc5ffff700 1 -- 172.21.15.96:0/2983229068 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbc6800f680 con 0x7fbc78100df0 2024-04-22T15:25:37.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.687+0000 7fbc5ffff700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc6006ec60 0x7fbc60071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:37.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.687+0000 7fbc5ffff700 1 -- 172.21.15.96:0/2983229068 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fbc680901e0 con 0x7fbc78100df0 2024-04-22T15:25:37.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.687+0000 7fbc7659c700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc6006ec60 0x7fbc60071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:37.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.688+0000 7fbc7659c700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc6006ec60 0x7fbc60071120 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbc78074910 tx=0x7fbc6c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:37.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.692+0000 7fbc5ffff700 1 -- 172.21.15.96:0/2983229068 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbc68058ff0 con 0x7fbc78100df0 2024-04-22T15:25:37.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.920+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc7810ddb0 con 0x7fbc6006ec60 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.922+0000 7fbc5ffff700 1 -- 172.21.15.96:0/2983229068 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fbc7810ddb0 con 0x7fbc6006ec60 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:25:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:25:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "8/23 daemons upgraded", 2024-04-22T15:25:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:25:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:25:37.926 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc6006ec60 msgr2=0x7fbc60071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc6006ec60 0x7fbc60071120 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbc78074910 tx=0x7fbc6c00b410 comp rx=0 tx=0).stop 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 msgr2=0x7fbc780731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 0x7fbc780731e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800bd70 tx=0x7fbc6800bda0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 shutdown_connections 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc6006ec60 0x7fbc60071120 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc78100df0 0x7fbc780731e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 --2- 172.21.15.96:0/2983229068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc781017a0 0x7fbc78073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 >> 172.21.15.96:0/2983229068 conn(0x7fbc780fc6a0 msgr2=0x7fbc780fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 shutdown_connections 2024-04-22T15:25:37.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:37.924+0000 7fbc7d0d8700 1 -- 172.21.15.96:0/2983229068 wait complete. 2024-04-22T15:25:38.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 -- 172.21.15.96:0/472383687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 msgr2=0x7f7a14105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:38.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 --2- 172.21.15.96:0/472383687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 0x7f7a14105e80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009910 tx=0x7f7a08009c20 comp rx=0 tx=0).stop 2024-04-22T15:25:38.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 -- 172.21.15.96:0/472383687 shutdown_connections 2024-04-22T15:25:38.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 --2- 172.21.15.96:0/472383687 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a140ffaa0 0x7f7a140fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:38.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 --2- 172.21.15.96:0/472383687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 0x7f7a14105e80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:38.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 -- 172.21.15.96:0/472383687 >> 172.21.15.96:0/472383687 conn(0x7f7a14076740 msgr2=0x7f7a140fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:38.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 -- 172.21.15.96:0/472383687 shutdown_connections 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 -- 172.21.15.96:0/472383687 wait complete. 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 Processor -- start 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.032+0000 7f7a1b070700 1 -- start start 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a1b070700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a140ffaa0 0x7f7a14195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a1b070700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 0x7f7a14195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a1b070700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14196620 con 0x7f7a14105aa0 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a1b070700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1419a370 con 0x7f7a140ffaa0 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a13fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 0x7f7a14195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a13fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 0x7f7a14195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45700/0 (socket says 172.21.15.96:45700) 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a13fff700 1 -- 172.21.15.96:0/2657051712 learned_addr learned my addr 172.21.15.96:0/2657051712 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:25:38.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a18e0c700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a140ffaa0 0x7f7a14195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:38.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a13fff700 1 -- 172.21.15.96:0/2657051712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a140ffaa0 msgr2=0x7f7a14195a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:38.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a13fff700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a140ffaa0 0x7f7a14195a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a13fff700 1 -- 172.21.15.96:0/2657051712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a080095f0 con 0x7f7a14105aa0 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a18e0c700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a140ffaa0 0x7f7a14195a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a13fff700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 0x7f7a14195f90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000e920 tx=0x7f7a0000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.033+0000 7f7a11ffb700 1 -- 172.21.15.96:0/2657051712 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a0000c9d0 con 0x7f7a14105aa0 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.034+0000 7f7a11ffb700 1 -- 172.21.15.96:0/2657051712 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a00010430 con 0x7f7a14105aa0 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.034+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1419a650 con 0x7f7a14105aa0 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.034+0000 7f7a11ffb700 1 -- 172.21.15.96:0/2657051712 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a00018630 con 0x7f7a14105aa0 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.034+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1419aba0 con 0x7f7a14105aa0 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.035+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a141080a0 con 0x7f7a14105aa0 2024-04-22T15:25:38.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.035+0000 7f7a11ffb700 1 -- 172.21.15.96:0/2657051712 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7a000105a0 con 0x7f7a14105aa0 2024-04-22T15:25:38.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.036+0000 7f7a11ffb700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7a0406ecd0 0x7f7a04071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:25:38.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.036+0000 7f7a11ffb700 1 -- 172.21.15.96:0/2657051712 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f7a00014070 con 0x7f7a14105aa0 2024-04-22T15:25:38.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.036+0000 7f7a18e0c700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7a0406ecd0 0x7f7a04071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:25:38.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.036+0000 7f7a18e0c700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7a0406ecd0 0x7f7a04071190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7a08005b40 tx=0x7f7a08005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:25:38.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.040+0000 7f7a11ffb700 1 -- 172.21.15.96:0/2657051712 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7a00054150 con 0x7f7a14105aa0 2024-04-22T15:25:38.343 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:25:38.343 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:25:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:25:38.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.339+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7a141011e0 con 0x7f7a14105aa0 2024-04-22T15:25:38.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.340+0000 7f7a11ffb700 1 -- 172.21.15.96:0/2657051712 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f7a0005b9b0 con 0x7f7a14105aa0 2024-04-22T15:25:38.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7a0406ecd0 msgr2=0x7f7a04071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:38.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7a0406ecd0 0x7f7a04071190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7a08005b40 tx=0x7f7a08005a90 comp rx=0 tx=0).stop 2024-04-22T15:25:38.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 msgr2=0x7f7a14195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:25:38.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 0x7f7a14195f90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000e920 tx=0x7f7a0000ec30 comp rx=0 tx=0).stop 2024-04-22T15:25:38.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 shutdown_connections 2024-04-22T15:25:38.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7a0406ecd0 0x7f7a04071190 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:38.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a140ffaa0 0x7f7a14195a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:38.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 --2- 172.21.15.96:0/2657051712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a14105aa0 0x7f7a14195f90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:25:38.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 >> 172.21.15.96:0/2657051712 conn(0x7f7a14076740 msgr2=0x7f7a140fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:25:38.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 shutdown_connections 2024-04-22T15:25:38.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:25:38.342+0000 7f7a1b070700 1 -- 172.21.15.96:0/2657051712 wait complete. 2024-04-22T15:25:38.564 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:38 smithi096 ceph-mon[112443]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:38.564 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:38 smithi096 ceph-mon[112443]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:38.564 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:38 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/496713584' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:25:38.564 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:38.565 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:25:38.565 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:25:38.565 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:25:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:38 smithi196 ceph-mon[81815]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:38 smithi196 ceph-mon[81815]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:38.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:38 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/496713584' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:25:38.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:38.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:25:38.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:25:38.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:25:39.669 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:39 smithi096 ceph-mon[112443]: Upgrade: Updating osd.2 2024-04-22T15:25:39.669 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:39 smithi096 ceph-mon[112443]: Deploying daemon osd.2 on smithi096 2024-04-22T15:25:39.669 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:39 smithi096 ceph-mon[112443]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:39.669 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:39 smithi096 ceph-mon[112443]: pgmap v88: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:25:39.669 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:39 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2657051712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:25:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:39 smithi196 ceph-mon[81815]: Upgrade: Updating osd.2 2024-04-22T15:25:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:39 smithi196 ceph-mon[81815]: Deploying daemon osd.2 on smithi096 2024-04-22T15:25:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:39 smithi196 ceph-mon[81815]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:25:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:39 smithi196 ceph-mon[81815]: pgmap v88: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:25:39.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:39 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2657051712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:25:41.318 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:41 smithi096 systemd[1]: Stopping Ceph osd.2 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:25:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:41 smithi096 ceph-mon[112443]: pgmap v89: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:25:41.736 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:41 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[61334]: 2024-04-22T15:25:41.323+0000 7fe1edb25700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:25:41.736 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:41 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[61334]: 2024-04-22T15:25:41.323+0000 7fe1edb25700 -1 osd.2 58 *** Got signal Terminated *** 2024-04-22T15:25:41.736 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:41 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[61334]: 2024-04-22T15:25:41.323+0000 7fe1edb25700 -1 osd.2 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:25:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:41 smithi196 ceph-mon[81815]: pgmap v89: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:25:42.727 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:42 smithi096 ceph-mon[112443]: osd.2 marked itself down and dead 2024-04-22T15:25:42.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:42 smithi196 ceph-mon[81815]: osd.2 marked itself down and dead 2024-04-22T15:25:42.986 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:42 smithi096 podman[135167]: 2024-04-22 15:25:42.725139948 +0000 UTC m=+1.557440926 container died bccf2e539131aa9036fdf69a96cd3ee8db2a2efc50fb0da59b0672fe7355cd5a (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=reef-646b8c7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:25:43.320 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:43 smithi096 podman[135167]: 2024-04-22 15:25:43.174257261 +0000 UTC m=+2.006558235 container cleanup bccf2e539131aa9036fdf69a96cd3ee8db2a2efc50fb0da59b0672fe7355cd5a (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=reef-646b8c7, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T15:25:43.321 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:43 smithi096 bash[135167]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2 2024-04-22T15:25:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:43 smithi096 ceph-mon[112443]: pgmap v90: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:25:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:43 smithi096 ceph-mon[112443]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:25:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:43 smithi096 ceph-mon[112443]: osdmap e59: 6 total, 5 up, 6 in 2024-04-22T15:25:43.736 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:43 smithi096 podman[135185]: 2024-04-22 15:25:43.459500846 +0000 UTC m=+0.736937208 container remove bccf2e539131aa9036fdf69a96cd3ee8db2a2efc50fb0da59b0672fe7355cd5a (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=reef-646b8c7, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-22T15:25:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:43 smithi196 ceph-mon[81815]: pgmap v90: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:25:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:43 smithi196 ceph-mon[81815]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:25:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:43 smithi196 ceph-mon[81815]: osdmap e59: 6 total, 5 up, 6 in 2024-04-22T15:25:44.083 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:43 smithi096 podman[135278]: 2024-04-22 15:25:43.699198593 +0000 UTC m=+0.018923823 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:25:44.083 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:43 smithi096 podman[135278]: 2024-04-22 15:25:43.802214899 +0000 UTC m=+0.121940113 container create 6bd4adb8bcdc79c4fe3af33057d1c0a73deb26bd27e5d490ac164adf5f32ee7c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:25:44.380 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:44 smithi096 ceph-mon[112443]: osdmap e60: 6 total, 5 up, 6 in 2024-04-22T15:25:44.381 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:44 smithi096 podman[135278]: 2024-04-22 15:25:44.080533013 +0000 UTC m=+0.400258227 container init 6bd4adb8bcdc79c4fe3af33057d1c0a73deb26bd27e5d490ac164adf5f32ee7c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, ceph=True) 2024-04-22T15:25:44.381 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:44 smithi096 podman[135278]: 2024-04-22 15:25:44.083909478 +0000 UTC m=+0.403634704 container start 6bd4adb8bcdc79c4fe3af33057d1c0a73deb26bd27e5d490ac164adf5f32ee7c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-22T15:25:44.381 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:44 smithi096 podman[135278]: 2024-04-22 15:25:44.155358143 +0000 UTC m=+0.475083367 container attach 6bd4adb8bcdc79c4fe3af33057d1c0a73deb26bd27e5d490ac164adf5f32ee7c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T15:25:44.736 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:44 smithi096 podman[135278]: 2024-04-22 15:25:44.379070547 +0000 UTC m=+0.698795777 container died 6bd4adb8bcdc79c4fe3af33057d1c0a73deb26bd27e5d490ac164adf5f32ee7c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-22T15:25:44.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:44 smithi196 ceph-mon[81815]: osdmap e60: 6 total, 5 up, 6 in 2024-04-22T15:25:45.230 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:44 smithi096 podman[135304]: 2024-04-22 15:25:44.950966014 +0000 UTC m=+0.565931936 container remove 6bd4adb8bcdc79c4fe3af33057d1c0a73deb26bd27e5d490ac164adf5f32ee7c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:25:45.230 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:44 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-22T15:25:45.231 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:44 smithi096 systemd[1]: Stopped Ceph osd.2 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:25:45.231 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:44 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.2.service: Consumed 48.470s CPU time. 2024-04-22T15:25:45.231 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:45 smithi096 systemd[1]: Starting Ceph osd.2 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:25:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:45 smithi096 ceph-mon[112443]: pgmap v93: 65 pgs: 6 active+undersized+degraded, 7 stale+active+clean, 6 active+undersized, 46 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 3 op/s; 17/330 objects degraded (5.152%) 2024-04-22T15:25:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:45 smithi096 ceph-mon[112443]: Health check failed: Degraded data redundancy: 17/330 objects degraded (5.152%), 6 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:45.486 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:45 smithi096 podman[135410]: 2024-04-22 15:25:45.309037353 +0000 UTC m=+0.018838519 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:25:45.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:45 smithi096 podman[135410]: 2024-04-22 15:25:45.418937445 +0000 UTC m=+0.128738576 container create d6179c31e1dd01afb3a46fb0898d8c8cc25c6497e494ea4491b9c72f96ccf307 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:25:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:45 smithi196 ceph-mon[81815]: pgmap v93: 65 pgs: 6 active+undersized+degraded, 7 stale+active+clean, 6 active+undersized, 46 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 3 op/s; 17/330 objects degraded (5.152%) 2024-04-22T15:25:45.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:45 smithi196 ceph-mon[81815]: Health check failed: Degraded data redundancy: 17/330 objects degraded (5.152%), 6 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:45.986 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:45 smithi096 podman[135410]: 2024-04-22 15:25:45.688680245 +0000 UTC m=+0.398481379 container init d6179c31e1dd01afb3a46fb0898d8c8cc25c6497e494ea4491b9c72f96ccf307 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0) 2024-04-22T15:25:45.986 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:45 smithi096 podman[135410]: 2024-04-22 15:25:45.692497763 +0000 UTC m=+0.402298895 container start d6179c31e1dd01afb3a46fb0898d8c8cc25c6497e494ea4491b9c72f96ccf307 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T15:25:45.986 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:45 smithi096 podman[135410]: 2024-04-22 15:25:45.755321339 +0000 UTC m=+0.465122472 container attach d6179c31e1dd01afb3a46fb0898d8c8cc25c6497e494ea4491b9c72f96ccf307 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:25:46.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate[135424]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:25:46.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate[135424]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:25:46.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 bash[135410]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:25:46.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 bash[135410]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:25:46.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 bash[135410]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T15:25:46.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 bash[135410]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 bash[135410]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 bash[135410]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 bash[135410]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 bash[135410]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 bash[135410]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate[135424]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate[135424]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate[135424]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate[135424]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T15:25:46.488 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate[135424]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-22T15:25:46.489 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate[135424]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T15:25:46.489 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate[135424]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-22T15:25:46.489 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 podman[135410]: 2024-04-22 15:25:46.188836432 +0000 UTC m=+0.898637577 container died d6179c31e1dd01afb3a46fb0898d8c8cc25c6497e494ea4491b9c72f96ccf307 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T15:25:47.099 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 podman[135556]: 2024-04-22 15:25:46.78448887 +0000 UTC m=+0.590129029 container remove d6179c31e1dd01afb3a46fb0898d8c8cc25c6497e494ea4491b9c72f96ccf307 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T15:25:47.099 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:46 smithi096 podman[135621]: 2024-04-22 15:25:46.99402743 +0000 UTC m=+0.108958883 container create afcdcb7020b9ebcd860d256700bc844dab36b5874f90645a2526b8944e362901 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-04-22T15:25:47.099 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:47 smithi096 podman[135621]: 2024-04-22 15:25:46.903813339 +0000 UTC m=+0.018744763 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:25:47.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:47 smithi096 ceph-mon[112443]: pgmap v94: 65 pgs: 10 active+undersized+degraded, 4 stale+active+clean, 10 active+undersized, 41 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s; 29/330 objects degraded (8.788%) 2024-04-22T15:25:47.486 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:47 smithi096 podman[135621]: 2024-04-22 15:25:47.188890076 +0000 UTC m=+0.303821496 container init afcdcb7020b9ebcd860d256700bc844dab36b5874f90645a2526b8944e362901 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-22T15:25:47.486 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:47 smithi096 podman[135621]: 2024-04-22 15:25:47.192196145 +0000 UTC m=+0.307127553 container start afcdcb7020b9ebcd860d256700bc844dab36b5874f90645a2526b8944e362901 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T15:25:47.486 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:47 smithi096 ceph-osd[135641]: -- 172.21.15.96:0/764105995 <== mon.0 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b6580e1680 con 0x55b6580aac00 2024-04-22T15:25:47.486 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:47 smithi096 bash[135621]: afcdcb7020b9ebcd860d256700bc844dab36b5874f90645a2526b8944e362901 2024-04-22T15:25:47.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:47 smithi096 systemd[1]: Started Ceph osd.2 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:25:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:47 smithi196 ceph-mon[81815]: pgmap v94: 65 pgs: 10 active+undersized+degraded, 4 stale+active+clean, 10 active+undersized, 41 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s; 29/330 objects degraded (8.788%) 2024-04-22T15:25:48.138 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:47 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[135637]: 2024-04-22T15:25:47.735+0000 7f4f8387c7c0 -1 Falling back to public interface 2024-04-22T15:25:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:48 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:48 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:48.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:48 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:25:48.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:48 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:48.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:48 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:48.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:48 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:25:49.608 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:49 smithi096 ceph-mon[112443]: pgmap v95: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 44/330 objects degraded (13.333%) 2024-04-22T15:25:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:49 smithi196 ceph-mon[81815]: pgmap v95: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 44/330 objects degraded (13.333%) 2024-04-22T15:25:50.571 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:50 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 44/330 objects degraded (13.333%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:50.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:50 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 44/330 objects degraded (13.333%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:51.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:51 smithi096 ceph-mon[112443]: pgmap v96: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 44/330 objects degraded (13.333%) 2024-04-22T15:25:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:51 smithi196 ceph-mon[81815]: pgmap v96: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 44/330 objects degraded (13.333%) 2024-04-22T15:25:52.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:52 smithi096 ceph-mon[112443]: from='osd.2 [v2:172.21.15.96:6818/2071585004,v1:172.21.15.96:6819/2071585004]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T15:25:52.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:52 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[135637]: 2024-04-22T15:25:52.184+0000 7f4f8387c7c0 -1 osd.2 58 log_to_monitors true 2024-04-22T15:25:52.487 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 15:25:52 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[135637]: 2024-04-22T15:25:52.382+0000 7f4f74cd4700 -1 osd.2 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:25:52.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:52 smithi196 ceph-mon[81815]: from='osd.2 [v2:172.21.15.96:6818/2071585004,v1:172.21.15.96:6819/2071585004]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T15:25:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:53 smithi096 ceph-mon[112443]: pgmap v97: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 929 B/s rd, 2 op/s; 44/330 objects degraded (13.333%) 2024-04-22T15:25:53.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:53 smithi096 ceph-mon[112443]: from='osd.2 [v2:172.21.15.96:6818/2071585004,v1:172.21.15.96:6819/2071585004]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T15:25:53.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:53 smithi096 ceph-mon[112443]: osdmap e61: 6 total, 5 up, 6 in 2024-04-22T15:25:53.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:53 smithi096 ceph-mon[112443]: from='osd.2 [v2:172.21.15.96:6818/2071585004,v1:172.21.15.96:6819/2071585004]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:25:53.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:53.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:53.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:53.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:25:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:53 smithi196 ceph-mon[81815]: pgmap v97: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 929 B/s rd, 2 op/s; 44/330 objects degraded (13.333%) 2024-04-22T15:25:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:53 smithi196 ceph-mon[81815]: from='osd.2 [v2:172.21.15.96:6818/2071585004,v1:172.21.15.96:6819/2071585004]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T15:25:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:53 smithi196 ceph-mon[81815]: osdmap e61: 6 total, 5 up, 6 in 2024-04-22T15:25:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:53 smithi196 ceph-mon[81815]: from='osd.2 [v2:172.21.15.96:6818/2071585004,v1:172.21.15.96:6819/2071585004]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T15:25:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:53.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:53.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:53.761 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:25:54.559 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:54 smithi096 ceph-mon[112443]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:25:54.560 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:54 smithi096 ceph-mon[112443]: osd.2 [v2:172.21.15.96:6818/2071585004,v1:172.21.15.96:6819/2071585004] boot 2024-04-22T15:25:54.560 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:54 smithi096 ceph-mon[112443]: osdmap e62: 6 total, 6 up, 6 in 2024-04-22T15:25:54.560 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:25:54.560 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:54.560 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:54.560 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:54 smithi096 ceph-mon[112443]: osdmap e63: 6 total, 6 up, 6 in 2024-04-22T15:25:54.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:54 smithi196 ceph-mon[81815]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:25:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:54 smithi196 ceph-mon[81815]: osd.2 [v2:172.21.15.96:6818/2071585004,v1:172.21.15.96:6819/2071585004] boot 2024-04-22T15:25:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:54 smithi196 ceph-mon[81815]: osdmap e62: 6 total, 6 up, 6 in 2024-04-22T15:25:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:25:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:54 smithi196 ceph-mon[81815]: osdmap e63: 6 total, 6 up, 6 in 2024-04-22T15:25:55.590 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:55 smithi096 ceph-mon[112443]: pgmap v100: 65 pgs: 10 peering, 8 active+undersized+degraded, 9 active+undersized, 38 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s; 28/330 objects degraded (8.485%) 2024-04-22T15:25:55.590 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:55 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 28/330 objects degraded (8.485%), 8 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:55 smithi196 ceph-mon[81815]: pgmap v100: 65 pgs: 10 peering, 8 active+undersized+degraded, 9 active+undersized, 38 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s; 28/330 objects degraded (8.485%) 2024-04-22T15:25:55.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:55 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 28/330 objects degraded (8.485%), 8 pgs degraded (PG_DEGRADED) 2024-04-22T15:25:57.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:57 smithi096 ceph-mon[112443]: pgmap v102: 65 pgs: 10 peering, 6 active+undersized+degraded, 6 active+undersized, 43 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 22/330 objects degraded (6.667%) 2024-04-22T15:25:57.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:57.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:57.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:25:57.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:25:57.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:57.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:57 smithi196 ceph-mon[81815]: pgmap v102: 65 pgs: 10 peering, 6 active+undersized+degraded, 6 active+undersized, 43 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 22/330 objects degraded (6.667%) 2024-04-22T15:25:57.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:57.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:57.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:25:57.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:25:57.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:25:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:25:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:25:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:58 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:25:58.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:58 smithi096 ceph-mon[112443]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-22T15:25:58.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:25:58.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:58.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:58.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:25:58.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:25:58.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:58 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:25:58.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:58 smithi196 ceph-mon[81815]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-22T15:25:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:59 smithi096 ceph-mon[112443]: pgmap v103: 65 pgs: 10 peering, 55 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-22T15:25:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:25:59 smithi096 ceph-mon[112443]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/330 objects degraded (6.667%), 6 pgs degraded) 2024-04-22T15:25:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:59 smithi196 ceph-mon[81815]: pgmap v103: 65 pgs: 10 peering, 55 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-22T15:25:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:25:59 smithi196 ceph-mon[81815]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/330 objects degraded (6.667%), 6 pgs degraded) 2024-04-22T15:26:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:01 smithi096 ceph-mon[112443]: pgmap v104: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 3 op/s 2024-04-22T15:26:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:01 smithi196 ceph-mon[81815]: pgmap v104: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 3 op/s 2024-04-22T15:26:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:03 smithi096 ceph-mon[112443]: pgmap v105: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:26:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:03 smithi196 ceph-mon[81815]: pgmap v105: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:26:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:05 smithi096 ceph-mon[112443]: pgmap v106: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T15:26:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:05 smithi196 ceph-mon[81815]: pgmap v106: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T15:26:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:07 smithi096 ceph-mon[112443]: pgmap v107: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:07 smithi196 ceph-mon[81815]: pgmap v107: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.458+0000 7f5eef14a700 1 -- 172.21.15.96:0/3357520361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee8105ab0 msgr2=0x7f5ee8105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.458+0000 7f5eef14a700 1 --2- 172.21.15.96:0/3357520361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee8105ab0 0x7f5ee8105e90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009910 tx=0x7f5ed8009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.458+0000 7f5eef14a700 1 -- 172.21.15.96:0/3357520361 shutdown_connections 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.458+0000 7f5eef14a700 1 --2- 172.21.15.96:0/3357520361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ee80ffab0 0x7f5ee80fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.458+0000 7f5eef14a700 1 --2- 172.21.15.96:0/3357520361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee8105ab0 0x7f5ee8105e90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.458+0000 7f5eef14a700 1 -- 172.21.15.96:0/3357520361 >> 172.21.15.96:0/3357520361 conn(0x7f5ee80fb5e0 msgr2=0x7f5ee80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.458+0000 7f5eef14a700 1 -- 172.21.15.96:0/3357520361 shutdown_connections 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.458+0000 7f5eef14a700 1 -- 172.21.15.96:0/3357520361 wait complete. 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.459+0000 7f5eef14a700 1 Processor -- start 2024-04-22T15:26:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.459+0000 7f5eef14a700 1 -- start start 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.459+0000 7f5eef14a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee80ffab0 0x7f5ee819b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.459+0000 7f5eef14a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ee8105ab0 0x7f5ee819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.459+0000 7f5eef14a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee819c370 con 0x7f5ee80ffab0 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.459+0000 7f5eef14a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee8195950 con 0x7f5ee8105ab0 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.459+0000 7f5eecee6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee80ffab0 0x7f5ee819b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.459+0000 7f5eecee6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee80ffab0 0x7f5ee819b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41888/0 (socket says 172.21.15.96:41888) 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.459+0000 7f5eecee6700 1 -- 172.21.15.96:0/2276556268 learned_addr learned my addr 172.21.15.96:0/2276556268 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5ee7fff700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ee8105ab0 0x7f5ee819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5eecee6700 1 -- 172.21.15.96:0/2276556268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ee8105ab0 msgr2=0x7f5ee819bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5eecee6700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ee8105ab0 0x7f5ee819bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5eecee6700 1 -- 172.21.15.96:0/2276556268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed80095f0 con 0x7f5ee80ffab0 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5ee7fff700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ee8105ab0 0x7f5ee819bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:26:08.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5eecee6700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee80ffab0 0x7f5ee819b810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8006010 tx=0x7f5ed800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5ee5ffb700 1 -- 172.21.15.96:0/2276556268 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ed801d070 con 0x7f5ee80ffab0 2024-04-22T15:26:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee8195bd0 con 0x7f5ee80ffab0 2024-04-22T15:26:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5ee5ffb700 1 -- 172.21.15.96:0/2276556268 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ed800f460 con 0x7f5ee80ffab0 2024-04-22T15:26:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5ee5ffb700 1 -- 172.21.15.96:0/2276556268 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ed8017630 con 0x7f5ee80ffab0 2024-04-22T15:26:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.460+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee81960c0 con 0x7f5ee80ffab0 2024-04-22T15:26:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.461+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee81080b0 con 0x7f5ee80ffab0 2024-04-22T15:26:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.462+0000 7f5ee5ffb700 1 -- 172.21.15.96:0/2276556268 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5ed800f5d0 con 0x7f5ee80ffab0 2024-04-22T15:26:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.462+0000 7f5ee5ffb700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ed006ecc0 0x7f5ed0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.462+0000 7f5ee5ffb700 1 -- 172.21.15.96:0/2276556268 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f5ed80902f0 con 0x7f5ee80ffab0 2024-04-22T15:26:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.462+0000 7f5ee7fff700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ed006ecc0 0x7f5ed0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.463+0000 7f5ee7fff700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ed006ecc0 0x7f5ed0071180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8196b10 tx=0x7f5edc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:08.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.467+0000 7f5ee5ffb700 1 -- 172.21.15.96:0/2276556268 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5ed805ccf0 con 0x7f5ee80ffab0 2024-04-22T15:26:08.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.686+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ee8196e80 con 0x7f5ed006ecc0 2024-04-22T15:26:08.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.687+0000 7f5ee5ffb700 1 -- 172.21.15.96:0/2276556268 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f5ee8196e80 con 0x7f5ed006ecc0 2024-04-22T15:26:08.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ed006ecc0 msgr2=0x7f5ed0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:08.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ed006ecc0 0x7f5ed0071180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8196b10 tx=0x7f5edc008040 comp rx=0 tx=0).stop 2024-04-22T15:26:08.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee80ffab0 msgr2=0x7f5ee819b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:08.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee80ffab0 0x7f5ee819b810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8006010 tx=0x7f5ed800be20 comp rx=0 tx=0).stop 2024-04-22T15:26:08.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 shutdown_connections 2024-04-22T15:26:08.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ed006ecc0 0x7f5ed0071180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:08.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee80ffab0 0x7f5ee819b810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:08.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 --2- 172.21.15.96:0/2276556268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ee8105ab0 0x7f5ee819bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:08.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 >> 172.21.15.96:0/2276556268 conn(0x7f5ee80fb5e0 msgr2=0x7f5ee80fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:08.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 shutdown_connections 2024-04-22T15:26:08.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.689+0000 7f5eef14a700 1 -- 172.21.15.96:0/2276556268 wait complete. 2024-04-22T15:26:08.736 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:26:08.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.795+0000 7f7da07d1700 1 -- 172.21.15.96:0/10505402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 msgr2=0x7f7d981011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:08.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.795+0000 7f7da07d1700 1 --2- 172.21.15.96:0/10505402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 0x7f7d981011f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009910 tx=0x7f7d88009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:08.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.795+0000 7f7da07d1700 1 -- 172.21.15.96:0/10505402 shutdown_connections 2024-04-22T15:26:08.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.795+0000 7f7da07d1700 1 --2- 172.21.15.96:0/10505402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d981017c0 0x7f7d98105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:08.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.795+0000 7f7da07d1700 1 --2- 172.21.15.96:0/10505402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 0x7f7d981011f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:08.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.795+0000 7f7da07d1700 1 -- 172.21.15.96:0/10505402 >> 172.21.15.96:0/10505402 conn(0x7f7d980fc6e0 msgr2=0x7f7d980feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:08.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.795+0000 7f7da07d1700 1 -- 172.21.15.96:0/10505402 shutdown_connections 2024-04-22T15:26:08.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.795+0000 7f7da07d1700 1 -- 172.21.15.96:0/10505402 wait complete. 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7da07d1700 1 Processor -- start 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7da07d1700 1 -- start start 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7da07d1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 0x7f7d98073200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7da07d1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d981017c0 0x7f7d98073740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7da07d1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d980751e0 con 0x7f7d98100e10 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7da07d1700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98073c80 con 0x7f7d981017c0 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7d9e56d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 0x7f7d98073200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7d9e56d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 0x7f7d98073200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41904/0 (socket says 172.21.15.96:41904) 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7d9e56d700 1 -- 172.21.15.96:0/3951647282 learned_addr learned my addr 172.21.15.96:0/3951647282 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:08.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7d9dd6c700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d981017c0 0x7f7d98073740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:08.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7d9e56d700 1 -- 172.21.15.96:0/3951647282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d981017c0 msgr2=0x7f7d98073740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:08.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7d9e56d700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d981017c0 0x7f7d98073740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:08.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7d9e56d700 1 -- 172.21.15.96:0/3951647282 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880095f0 con 0x7f7d98100e10 2024-04-22T15:26:08.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.796+0000 7f7d9e56d700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 0x7f7d98073200 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800bd70 tx=0x7f7d8800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:08.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.797+0000 7f7d8f7fe700 1 -- 172.21.15.96:0/3951647282 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8801d070 con 0x7f7d98100e10 2024-04-22T15:26:08.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.797+0000 7f7d8f7fe700 1 -- 172.21.15.96:0/3951647282 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d880056e0 con 0x7f7d98100e10 2024-04-22T15:26:08.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.797+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d98073f00 con 0x7f7d98100e10 2024-04-22T15:26:08.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.797+0000 7f7d8f7fe700 1 -- 172.21.15.96:0/3951647282 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d88017400 con 0x7f7d98100e10 2024-04-22T15:26:08.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.797+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d981a2f40 con 0x7f7d98100e10 2024-04-22T15:26:08.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.797+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d981091b0 con 0x7f7d98100e10 2024-04-22T15:26:08.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.798+0000 7f7d8f7fe700 1 -- 172.21.15.96:0/3951647282 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7d88003a60 con 0x7f7d98100e10 2024-04-22T15:26:08.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.798+0000 7f7d8f7fe700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8406ecc0 0x7f7d84071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:08.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.799+0000 7f7d8f7fe700 1 -- 172.21.15.96:0/3951647282 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f7d88090230 con 0x7f7d98100e10 2024-04-22T15:26:08.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.799+0000 7f7d9dd6c700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8406ecc0 0x7f7d84071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:08.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.799+0000 7f7d9dd6c700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8406ecc0 0x7f7d84071180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d94005950 tx=0x7f7d940058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:08.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:08.803+0000 7f7d8f7fe700 1 -- 172.21.15.96:0/3951647282 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7d88059040 con 0x7f7d98100e10 2024-04-22T15:26:09.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.021+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d9810dde0 con 0x7f7d8406ecc0 2024-04-22T15:26:09.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.023+0000 7f7d8f7fe700 1 -- 172.21.15.96:0/3951647282 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7d9810dde0 con 0x7f7d8406ecc0 2024-04-22T15:26:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8406ecc0 msgr2=0x7f7d84071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8406ecc0 0x7f7d84071180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d94005950 tx=0x7f7d940058e0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 msgr2=0x7f7d98073200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 0x7f7d98073200 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800bd70 tx=0x7f7d8800be50 comp rx=0 tx=0).stop 2024-04-22T15:26:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 shutdown_connections 2024-04-22T15:26:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8406ecc0 0x7f7d84071180 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d98100e10 0x7f7d98073200 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 --2- 172.21.15.96:0/3951647282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d981017c0 0x7f7d98073740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 >> 172.21.15.96:0/3951647282 conn(0x7f7d980fc6e0 msgr2=0x7f7d980fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 shutdown_connections 2024-04-22T15:26:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.025+0000 7f7da07d1700 1 -- 172.21.15.96:0/3951647282 wait complete. 2024-04-22T15:26:09.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.129+0000 7ff091563700 1 -- 172.21.15.96:0/3610437478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 msgr2=0x7ff08c105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.129+0000 7ff091563700 1 --2- 172.21.15.96:0/3610437478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 0x7ff08c105810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff074009910 tx=0x7ff074009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:09.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.129+0000 7ff091563700 1 -- 172.21.15.96:0/3610437478 shutdown_connections 2024-04-22T15:26:09.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.129+0000 7ff091563700 1 --2- 172.21.15.96:0/3610437478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 0x7ff08c105810 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.129+0000 7ff091563700 1 --2- 172.21.15.96:0/3610437478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff08c100e10 0x7ff08c1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.129+0000 7ff091563700 1 -- 172.21.15.96:0/3610437478 >> 172.21.15.96:0/3610437478 conn(0x7ff08c0fc6e0 msgr2=0x7ff08c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:09.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.129+0000 7ff091563700 1 -- 172.21.15.96:0/3610437478 shutdown_connections 2024-04-22T15:26:09.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.129+0000 7ff091563700 1 -- 172.21.15.96:0/3610437478 wait complete. 2024-04-22T15:26:09.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff091563700 1 Processor -- start 2024-04-22T15:26:09.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff091563700 1 -- start start 2024-04-22T15:26:09.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff091563700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff08c100e10 0x7ff08c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:09.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff091563700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 0x7ff08c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:09.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff091563700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c073c60 con 0x7ff08c1017c0 2024-04-22T15:26:09.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff091563700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c073da0 con 0x7ff08c100e10 2024-04-22T15:26:09.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff0827fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 0x7ff08c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:09.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff0827fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 0x7ff08c073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41930/0 (socket says 172.21.15.96:41930) 2024-04-22T15:26:09.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff0827fc700 1 -- 172.21.15.96:0/1732572422 learned_addr learned my addr 172.21.15.96:0/1732572422 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:09.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff08affd700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff08c100e10 0x7ff08c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff0827fc700 1 -- 172.21.15.96:0/1732572422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff08c100e10 msgr2=0x7ff08c0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff0827fc700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff08c100e10 0x7ff08c0731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff0827fc700 1 -- 172.21.15.96:0/1732572422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0740095f0 con 0x7ff08c1017c0 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.130+0000 7ff08affd700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff08c100e10 0x7ff08c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.131+0000 7ff0827fc700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 0x7ff08c073720 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff074009fd0 tx=0x7ff07400b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.131+0000 7ff088ff9700 1 -- 172.21.15.96:0/1732572422 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff07401d070 con 0x7ff08c1017c0 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.131+0000 7ff088ff9700 1 -- 172.21.15.96:0/1732572422 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0740052a0 con 0x7ff08c1017c0 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.131+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c074020 con 0x7ff08c1017c0 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.131+0000 7ff088ff9700 1 -- 172.21.15.96:0/1732572422 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff074022470 con 0x7ff08c1017c0 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.131+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c1a2f60 con 0x7ff08c1017c0 2024-04-22T15:26:09.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.132+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c109210 con 0x7ff08c1017c0 2024-04-22T15:26:09.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.133+0000 7ff088ff9700 1 -- 172.21.15.96:0/1732572422 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff07400f770 con 0x7ff08c1017c0 2024-04-22T15:26:09.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.133+0000 7ff088ff9700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff07806ecc0 0x7ff078071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:09.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.133+0000 7ff088ff9700 1 -- 172.21.15.96:0/1732572422 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6414+0+0 (secure 0 0 0) 0x7ff0740901e0 con 0x7ff08c1017c0 2024-04-22T15:26:09.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.133+0000 7ff08affd700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff07806ecc0 0x7ff078071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:09.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.133+0000 7ff08affd700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff07806ecc0 0x7ff078071180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009cb0 tx=0x7ff07c009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:09.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.138+0000 7ff088ff9700 1 -- 172.21.15.96:0/1732572422 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff074058ff0 con 0x7ff08c1017c0 2024-04-22T15:26:09.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.347+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff08c10ddb0 con 0x7ff07806ecc0 2024-04-22T15:26:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.358+0000 7ff088ff9700 1 -- 172.21.15.96:0/1732572422 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7ff08c10ddb0 con 0x7ff07806ecc0 2024-04-22T15:26:09.361 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (4m) 16s ago 16m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (16m) 16s ago 16m 8870k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (15m) 113s ago 15m 8166k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (2m) 16s ago 16m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (116s) 113s ago 15m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (4m) 16s ago 15m 80.7M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (10m) 16s ago 10m 155M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (10m) 16s ago 10m 90.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (10m) 113s ago 10m 27.0M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (10m) 113s ago 10m 26.3M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (7m) 16s ago 17m 505M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (6m) 113s ago 15m 450M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:26:09.362 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (3m) 16s ago 17m 46.9M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (2m) 113s ago 14m 35.4M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (5m) 16s ago 16m 17.3M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (5m) 113s ago 15m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (95s) 16s ago 14m 92.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (59s) 16s ago 13m 93.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (22s) 16s ago 13m 11.9M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (12m) 113s ago 12m 242M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 d057260f11a3 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (12m) 113s ago 12m 213M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (12m) 113s ago 12m 196M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:26:09.363 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (5m) 16s ago 15m 77.0M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff07806ecc0 msgr2=0x7ff078071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff07806ecc0 0x7ff078071180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009cb0 tx=0x7ff07c009260 comp rx=0 tx=0).stop 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 msgr2=0x7ff08c073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 0x7ff08c073720 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff074009fd0 tx=0x7ff07400b650 comp rx=0 tx=0).stop 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 shutdown_connections 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff07806ecc0 0x7ff078071180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff08c100e10 0x7ff08c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 --2- 172.21.15.96:0/1732572422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff08c1017c0 0x7ff08c073720 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 >> 172.21.15.96:0/1732572422 conn(0x7ff08c0fc6e0 msgr2=0x7ff08c0fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 shutdown_connections 2024-04-22T15:26:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.360+0000 7ff091563700 1 -- 172.21.15.96:0/1732572422 wait complete. 2024-04-22T15:26:09.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.464+0000 7f657b94d700 1 -- 172.21.15.96:0/470124836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65740ffad0 msgr2=0x7f65740fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.464+0000 7f657b94d700 1 --2- 172.21.15.96:0/470124836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65740ffad0 0x7f65740fff50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6568009910 tx=0x7f6568009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:09.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.464+0000 7f657b94d700 1 -- 172.21.15.96:0/470124836 shutdown_connections 2024-04-22T15:26:09.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.464+0000 7f657b94d700 1 --2- 172.21.15.96:0/470124836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65740ffad0 0x7f65740fff50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.464+0000 7f657b94d700 1 --2- 172.21.15.96:0/470124836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6574105b60 0x7f6574105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.464+0000 7f657b94d700 1 -- 172.21.15.96:0/470124836 >> 172.21.15.96:0/470124836 conn(0x7f65740fb600 msgr2=0x7f65740fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:09.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:09 smithi096 ceph-mon[112443]: pgmap v108: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:09.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:09.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:26:09.468 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:09.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.464+0000 7f657b94d700 1 -- 172.21.15.96:0/470124836 shutdown_connections 2024-04-22T15:26:09.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.464+0000 7f657b94d700 1 -- 172.21.15.96:0/470124836 wait complete. 2024-04-22T15:26:09.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.465+0000 7f657b94d700 1 Processor -- start 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.465+0000 7f657b94d700 1 -- start start 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.465+0000 7f657b94d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65740ffad0 0x7f6574197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.465+0000 7f657b94d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6574105b60 0x7f6574197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.465+0000 7f657b94d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65741983b0 con 0x7f6574105b60 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.465+0000 7f657b94d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65741984f0 con 0x7f65740ffad0 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f6578ee8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6574105b60 0x7f6574197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f6578ee8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6574105b60 0x7f6574197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41950/0 (socket says 172.21.15.96:41950) 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f6578ee8700 1 -- 172.21.15.96:0/906370892 learned_addr learned my addr 172.21.15.96:0/906370892 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f65796e9700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65740ffad0 0x7f6574197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f6578ee8700 1 -- 172.21.15.96:0/906370892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65740ffad0 msgr2=0x7f6574197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f6578ee8700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65740ffad0 0x7f6574197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f6578ee8700 1 -- 172.21.15.96:0/906370892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680095f0 con 0x7f6574105b60 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f6578ee8700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6574105b60 0x7f6574197e70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6568006010 tx=0x7f656800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:09.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f65727fc700 1 -- 172.21.15.96:0/906370892 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f656801d070 con 0x7f6574105b60 2024-04-22T15:26:09.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f65727fc700 1 -- 172.21.15.96:0/906370892 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f656800f460 con 0x7f6574105b60 2024-04-22T15:26:09.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6574198710 con 0x7f6574105b60 2024-04-22T15:26:09.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f65727fc700 1 -- 172.21.15.96:0/906370892 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6568017630 con 0x7f6574105b60 2024-04-22T15:26:09.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.466+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65741a0c80 con 0x7f6574105b60 2024-04-22T15:26:09.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.467+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65741080f0 con 0x7f6574105b60 2024-04-22T15:26:09.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.468+0000 7f65727fc700 1 -- 172.21.15.96:0/906370892 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f656800f5d0 con 0x7f6574105b60 2024-04-22T15:26:09.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.469+0000 7f65727fc700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f655c06ecc0 0x7f655c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:09.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.469+0000 7f65796e9700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f655c06ecc0 0x7f655c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:09.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.469+0000 7f65727fc700 1 -- 172.21.15.96:0/906370892 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f65680903e0 con 0x7f6574105b60 2024-04-22T15:26:09.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.469+0000 7f65796e9700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f655c06ecc0 0x7f655c071180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f656c00e270 tx=0x7f656c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:09.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.473+0000 7f65727fc700 1 -- 172.21.15.96:0/906370892 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f65680593d0 con 0x7f6574105b60 2024-04-22T15:26:09.726 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:09 smithi196 ceph-mon[81815]: pgmap v108: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:09.726 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:09.726 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:26:09.726 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:09.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.759+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f65741011d0 con 0x7f6574105b60 2024-04-22T15:26:09.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.760+0000 7f65727fc700 1 -- 172.21.15.96:0/906370892 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+836 (secure 0 0 0) 0x7f65680163b0 con 0x7f6574105b60 2024-04-22T15:26:09.763 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:26:09.763 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:26:09.763 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:26:09.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:26:09.763 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:26:09.763 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 3, 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 7, 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 7 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:26:09.764 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:26:09.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f655c06ecc0 msgr2=0x7f655c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f655c06ecc0 0x7f655c071180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f656c00e270 tx=0x7f656c009040 comp rx=0 tx=0).stop 2024-04-22T15:26:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6574105b60 msgr2=0x7f6574197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6574105b60 0x7f6574197e70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6568006010 tx=0x7f656800be20 comp rx=0 tx=0).stop 2024-04-22T15:26:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 shutdown_connections 2024-04-22T15:26:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f655c06ecc0 0x7f655c071180 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65740ffad0 0x7f6574197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 --2- 172.21.15.96:0/906370892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6574105b60 0x7f6574197e70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 >> 172.21.15.96:0/906370892 conn(0x7f65740fb600 msgr2=0x7f65740fc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 shutdown_connections 2024-04-22T15:26:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.762+0000 7f657b94d700 1 -- 172.21.15.96:0/906370892 wait complete. 2024-04-22T15:26:09.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.866+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/3875084391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 msgr2=0x7fdf18105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.866+0000 7fdf1ddcd700 1 --2- 172.21.15.96:0/3875084391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 0x7fdf18105810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009910 tx=0x7fdf00009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:09.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.866+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/3875084391 shutdown_connections 2024-04-22T15:26:09.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.866+0000 7fdf1ddcd700 1 --2- 172.21.15.96:0/3875084391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 0x7fdf18105810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.866+0000 7fdf1ddcd700 1 --2- 172.21.15.96:0/3875084391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf18100e10 0x7fdf181011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.866+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/3875084391 >> 172.21.15.96:0/3875084391 conn(0x7fdf180fc6e0 msgr2=0x7fdf180feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:09.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.866+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/3875084391 shutdown_connections 2024-04-22T15:26:09.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.866+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/3875084391 wait complete. 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.866+0000 7fdf1ddcd700 1 Processor -- start 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf1ddcd700 1 -- start start 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf1ddcd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf18100e10 0x7fdf1819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf1ddcd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 0x7fdf1819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf1ddcd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1819d380 con 0x7fdf181017c0 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf1ddcd700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf181969d0 con 0x7fdf18100e10 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf0effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 0x7fdf1819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf0effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 0x7fdf1819cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41964/0 (socket says 172.21.15.96:41964) 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf0effd700 1 -- 172.21.15.96:0/1561725814 learned_addr learned my addr 172.21.15.96:0/1561725814 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf177fe700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf18100e10 0x7fdf1819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf0effd700 1 -- 172.21.15.96:0/1561725814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf18100e10 msgr2=0x7fdf1819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:09.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf0effd700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf18100e10 0x7fdf1819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:09.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf0effd700 1 -- 172.21.15.96:0/1561725814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf000095f0 con 0x7fdf181017c0 2024-04-22T15:26:09.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf177fe700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf18100e10 0x7fdf1819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:26:09.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.867+0000 7fdf0effd700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 0x7fdf1819cd80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdf000092e0 tx=0x7fdf0000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:09.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.868+0000 7fdf157fa700 1 -- 172.21.15.96:0/1561725814 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf0001d070 con 0x7fdf181017c0 2024-04-22T15:26:09.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.868+0000 7fdf157fa700 1 -- 172.21.15.96:0/1561725814 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf0000f460 con 0x7fdf181017c0 2024-04-22T15:26:09.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.868+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf18196c50 con 0x7fdf181017c0 2024-04-22T15:26:09.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.868+0000 7fdf157fa700 1 -- 172.21.15.96:0/1561725814 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf00017630 con 0x7fdf181017c0 2024-04-22T15:26:09.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.868+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf18197140 con 0x7fdf181017c0 2024-04-22T15:26:09.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.869+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf18109210 con 0x7fdf181017c0 2024-04-22T15:26:09.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.869+0000 7fdf157fa700 1 -- 172.21.15.96:0/1561725814 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdf0000f5d0 con 0x7fdf181017c0 2024-04-22T15:26:09.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.869+0000 7fdf157fa700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf0406ecc0 0x7fdf04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:09.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.870+0000 7fdf157fa700 1 -- 172.21.15.96:0/1561725814 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fdf00090280 con 0x7fdf181017c0 2024-04-22T15:26:09.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.870+0000 7fdf177fe700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf0406ecc0 0x7fdf04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:09.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.870+0000 7fdf177fe700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf0406ecc0 0x7fdf04071180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdf08005fd0 tx=0x7fdf08005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:09.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:09.874+0000 7fdf157fa700 1 -- 172.21.15.96:0/1561725814 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdf0005cbc0 con 0x7fdf181017c0 2024-04-22T15:26:10.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.121+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdf18197e40 con 0x7fdf181017c0 2024-04-22T15:26:10.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.122+0000 7fdf157fa700 1 -- 172.21.15.96:0/1561725814 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1901 (secure 0 0 0) 0x7fdf00005740 con 0x7fdf181017c0 2024-04-22T15:26:10.125 INFO:teuthology.orchestra.run.smithi096.stdout:e20 2024-04-22T15:26:10.125 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 20 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:26:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:26:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:26:10.128 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:26:10.128 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:26:10.128 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:26:10.128 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:26:10.128 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:26:10.128 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:26:10.128 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:26:10.128 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:26:10.128 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf0406ecc0 msgr2=0x7fdf04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf0406ecc0 0x7fdf04071180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdf08005fd0 tx=0x7fdf08005e20 comp rx=0 tx=0).stop 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 msgr2=0x7fdf1819cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 0x7fdf1819cd80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdf000092e0 tx=0x7fdf0000be20 comp rx=0 tx=0).stop 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 shutdown_connections 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf0406ecc0 0x7fdf04071180 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf18100e10 0x7fdf1819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 --2- 172.21.15.96:0/1561725814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf181017c0 0x7fdf1819cd80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 >> 172.21.15.96:0/1561725814 conn(0x7fdf180fc6e0 msgr2=0x7fdf180fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 shutdown_connections 2024-04-22T15:26:10.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.125+0000 7fdf1ddcd700 1 -- 172.21.15.96:0/1561725814 wait complete. 2024-04-22T15:26:10.130 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.232+0000 7fa24f84f700 1 -- 172.21.15.96:0/1166114926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2481026d0 msgr2=0x7fa24810abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.232+0000 7fa24f84f700 1 --2- 172.21.15.96:0/1166114926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2481026d0 0x7fa24810abc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa238009910 tx=0x7fa238009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.232+0000 7fa24f84f700 1 -- 172.21.15.96:0/1166114926 shutdown_connections 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.232+0000 7fa24f84f700 1 --2- 172.21.15.96:0/1166114926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2481026d0 0x7fa24810abc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.232+0000 7fa24f84f700 1 --2- 172.21.15.96:0/1166114926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa248101db0 0x7fa248102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.232+0000 7fa24f84f700 1 -- 172.21.15.96:0/1166114926 >> 172.21.15.96:0/1166114926 conn(0x7fa2480fb640 msgr2=0x7fa2480fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.232+0000 7fa24f84f700 1 -- 172.21.15.96:0/1166114926 shutdown_connections 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.232+0000 7fa24f84f700 1 -- 172.21.15.96:0/1166114926 wait complete. 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24f84f700 1 Processor -- start 2024-04-22T15:26:10.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24f84f700 1 -- start start 2024-04-22T15:26:10.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24f84f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa248101db0 0x7fa24819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:10.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24f84f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2481026d0 0x7fa24819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:10.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24f84f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24819d3d0 con 0x7fa248101db0 2024-04-22T15:26:10.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24f84f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2481969b0 con 0x7fa2481026d0 2024-04-22T15:26:10.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24d5eb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa248101db0 0x7fa24819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24d5eb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa248101db0 0x7fa24819c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41980/0 (socket says 172.21.15.96:41980) 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24d5eb700 1 -- 172.21.15.96:0/2796976695 learned_addr learned my addr 172.21.15.96:0/2796976695 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24cdea700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2481026d0 0x7fa24819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24d5eb700 1 -- 172.21.15.96:0/2796976695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2481026d0 msgr2=0x7fa24819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24d5eb700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2481026d0 0x7fa24819cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.233+0000 7fa24d5eb700 1 -- 172.21.15.96:0/2796976695 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2380095f0 con 0x7fa248101db0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.234+0000 7fa24d5eb700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa248101db0 0x7fa24819c870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa24400d8d0 tx=0x7fa24400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.234+0000 7fa23e7fc700 1 -- 172.21.15.96:0/2796976695 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2440098a0 con 0x7fa248101db0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.234+0000 7fa23e7fc700 1 -- 172.21.15.96:0/2796976695 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa244010460 con 0x7fa248101db0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.234+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa248196c90 con 0x7fa248101db0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.234+0000 7fa23e7fc700 1 -- 172.21.15.96:0/2796976695 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa24400f5f0 con 0x7fa248101db0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.234+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2481971b0 con 0x7fa248101db0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.235+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2481054d0 con 0x7fa248101db0 2024-04-22T15:26:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.235+0000 7fa23e7fc700 1 -- 172.21.15.96:0/2796976695 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa244009a00 con 0x7fa248101db0 2024-04-22T15:26:10.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.236+0000 7fa23e7fc700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa23406ecd0 0x7fa234071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:10.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.236+0000 7fa23e7fc700 1 -- 172.21.15.96:0/2796976695 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fa24408ecd0 con 0x7fa248101db0 2024-04-22T15:26:10.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.236+0000 7fa24cdea700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa23406ecd0 0x7fa234071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:10.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.236+0000 7fa24cdea700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa23406ecd0 0x7fa234071190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa238009fd0 tx=0x7fa23801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:10.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.241+0000 7fa23e7fc700 1 -- 172.21.15.96:0/2796976695 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa24405b610 con 0x7fa248101db0 2024-04-22T15:26:10.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.465+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa248197ee0 con 0x7fa23406ecd0 2024-04-22T15:26:10.469 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:10 smithi096 ceph-mon[112443]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:10.469 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:10 smithi096 ceph-mon[112443]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:10.469 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:10 smithi096 ceph-mon[112443]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:10.469 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:10 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/906370892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:10.469 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:10 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1561725814' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:26:10.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.466+0000 7fa23e7fc700 1 -- 172.21.15.96:0/2796976695 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fa248197ee0 con 0x7fa23406ecd0 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "9/23 daemons upgraded", 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:26:10.470 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:26:10.471 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:26:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa23406ecd0 msgr2=0x7fa234071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa23406ecd0 0x7fa234071190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa238009fd0 tx=0x7fa23801a040 comp rx=0 tx=0).stop 2024-04-22T15:26:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa248101db0 msgr2=0x7fa24819c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa248101db0 0x7fa24819c870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa24400d8d0 tx=0x7fa24400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 shutdown_connections 2024-04-22T15:26:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa23406ecd0 0x7fa234071190 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa248101db0 0x7fa24819c870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 --2- 172.21.15.96:0/2796976695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2481026d0 0x7fa24819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 >> 172.21.15.96:0/2796976695 conn(0x7fa2480fb640 msgr2=0x7fa2481030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:10.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 shutdown_connections 2024-04-22T15:26:10.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.469+0000 7fa24f84f700 1 -- 172.21.15.96:0/2796976695 wait complete. 2024-04-22T15:26:10.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.576+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/2330212628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 msgr2=0x7fe6b8100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.576+0000 7fe6bf2c7700 1 --2- 172.21.15.96:0/2330212628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 0x7fe6b8100760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8009910 tx=0x7fe6a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:10.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.576+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/2330212628 shutdown_connections 2024-04-22T15:26:10.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.576+0000 7fe6bf2c7700 1 --2- 172.21.15.96:0/2330212628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 0x7fe6b8100760 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.576+0000 7fe6bf2c7700 1 --2- 172.21.15.96:0/2330212628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6b8104b30 0x7fe6b8104f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.576+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/2330212628 >> 172.21.15.96:0/2330212628 conn(0x7fe6b8075f00 msgr2=0x7fe6b80fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:10.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.577+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/2330212628 shutdown_connections 2024-04-22T15:26:10.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.577+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/2330212628 wait complete. 2024-04-22T15:26:10.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.577+0000 7fe6bf2c7700 1 Processor -- start 2024-04-22T15:26:10.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.577+0000 7fe6bf2c7700 1 -- start start 2024-04-22T15:26:10.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.577+0000 7fe6bf2c7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 0x7fe6b8197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:10.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.577+0000 7fe6bf2c7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6b8104b30 0x7fe6b8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:10.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.577+0000 7fe6bf2c7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8198530 con 0x7fe6b81002e0 2024-04-22T15:26:10.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.577+0000 7fe6bf2c7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8198670 con 0x7fe6b8104b30 2024-04-22T15:26:10.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bd063700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 0x7fe6b8197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:10.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bd063700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 0x7fe6b8197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55632/0 (socket says 172.21.15.96:55632) 2024-04-22T15:26:10.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bd063700 1 -- 172.21.15.96:0/539483458 learned_addr learned my addr 172.21.15.96:0/539483458 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bc862700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6b8104b30 0x7fe6b8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bd063700 1 -- 172.21.15.96:0/539483458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6b8104b30 msgr2=0x7fe6b8197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bd063700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6b8104b30 0x7fe6b8197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bd063700 1 -- 172.21.15.96:0/539483458 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a80095f0 con 0x7fe6b81002e0 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bc862700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6b8104b30 0x7fe6b8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bd063700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 0x7fe6b8197a20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400d8d0 tx=0x7fe6b400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6ae7fc700 1 -- 172.21.15.96:0/539483458 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6b4009940 con 0x7fe6b81002e0 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b81a0a90 con 0x7fe6b81002e0 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6ae7fc700 1 -- 172.21.15.96:0/539483458 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6b4010460 con 0x7fe6b81002e0 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6ae7fc700 1 -- 172.21.15.96:0/539483458 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6b400f5f0 con 0x7fe6b81002e0 2024-04-22T15:26:10.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.578+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b81a0f00 con 0x7fe6b81002e0 2024-04-22T15:26:10.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.579+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b8108050 con 0x7fe6b81002e0 2024-04-22T15:26:10.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.580+0000 7fe6ae7fc700 1 -- 172.21.15.96:0/539483458 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe6b40105d0 con 0x7fe6b81002e0 2024-04-22T15:26:10.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.580+0000 7fe6ae7fc700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe6a406ecc0 0x7fe6a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:10.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.580+0000 7fe6ae7fc700 1 -- 172.21.15.96:0/539483458 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fe6b408ee10 con 0x7fe6b81002e0 2024-04-22T15:26:10.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.580+0000 7fe6bc862700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe6a406ecc0 0x7fe6a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:10.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.581+0000 7fe6bc862700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe6a406ecc0 0x7fe6a4071180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8006010 tx=0x7fe6a800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:10.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.585+0000 7fe6ae7fc700 1 -- 172.21.15.96:0/539483458 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe6b4057c20 con 0x7fe6b81002e0 2024-04-22T15:26:10.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:10 smithi196 ceph-mon[81815]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:10 smithi196 ceph-mon[81815]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:10 smithi196 ceph-mon[81815]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:10 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/906370892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:10 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1561725814' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:26:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.869+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe6b81012f0 con 0x7fe6b81002e0 2024-04-22T15:26:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.869+0000 7fe6ae7fc700 1 -- 172.21.15.96:0/539483458 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fe6b405b2e0 con 0x7fe6b81002e0 2024-04-22T15:26:10.873 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:26:10.873 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:26:10.873 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:26:10.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe6a406ecc0 msgr2=0x7fe6a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe6a406ecc0 0x7fe6a4071180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8006010 tx=0x7fe6a800b540 comp rx=0 tx=0).stop 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 msgr2=0x7fe6b8197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 0x7fe6b8197a20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400d8d0 tx=0x7fe6b400dc90 comp rx=0 tx=0).stop 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 shutdown_connections 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe6a406ecc0 0x7fe6a4071180 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6b81002e0 0x7fe6b8197a20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 --2- 172.21.15.96:0/539483458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6b8104b30 0x7fe6b8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 >> 172.21.15.96:0/539483458 conn(0x7fe6b8075f00 msgr2=0x7fe6b80fb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 shutdown_connections 2024-04-22T15:26:10.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:10.871+0000 7fe6bf2c7700 1 -- 172.21.15.96:0/539483458 wait complete. 2024-04-22T15:26:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:11 smithi096 ceph-mon[112443]: pgmap v109: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:26:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:11 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/539483458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:26:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:11 smithi196 ceph-mon[81815]: pgmap v109: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:26:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:11 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/539483458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:26:12.693 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:12 smithi096 ceph-mon[112443]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:12.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:12 smithi196 ceph-mon[81815]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:13 smithi096 ceph-mon[112443]: pgmap v110: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:26:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:13 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:26:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:13 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:26:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:13 smithi096 ceph-mon[112443]: Upgrade: osd.3 is safe to restart 2024-04-22T15:26:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:13 smithi196 ceph-mon[81815]: pgmap v110: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:26:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:13 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:26:13.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:13 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:26:13.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:13 smithi196 ceph-mon[81815]: Upgrade: osd.3 is safe to restart 2024-04-22T15:26:15.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:15 smithi196 ceph-mon[81815]: Upgrade: Updating osd.3 2024-04-22T15:26:15.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:15 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:15.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:15 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:26:15.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:15 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:26:15.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:15 smithi196 ceph-mon[81815]: Deploying daemon osd.3 on smithi196 2024-04-22T15:26:15.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:15 smithi196 ceph-mon[81815]: pgmap v111: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:26:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:15 smithi096 ceph-mon[112443]: Upgrade: Updating osd.3 2024-04-22T15:26:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:15 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:15 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:26:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:15 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:26:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:15 smithi096 ceph-mon[112443]: Deploying daemon osd.3 on smithi196 2024-04-22T15:26:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:15 smithi096 ceph-mon[112443]: pgmap v111: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:26:17.541 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:17 smithi196 ceph-mon[81815]: pgmap v112: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:17.541 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:17 smithi196 systemd[1]: Stopping Ceph osd.3 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:26:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:17 smithi096 ceph-mon[112443]: pgmap v112: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:18.009 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:17 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[40667]: 2024-04-22T15:26:17.537+0000 7f93fb5e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:26:18.010 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:17 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[40667]: 2024-04-22T15:26:17.537+0000 7f93fb5e9700 -1 osd.3 63 *** Got signal Terminated *** 2024-04-22T15:26:18.010 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:17 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[40667]: 2024-04-22T15:26:17.537+0000 7f93fb5e9700 -1 osd.3 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:26:18.716 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:18 smithi196 ceph-mon[81815]: osd.3 marked itself down and dead 2024-04-22T15:26:18.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:18 smithi096 ceph-mon[112443]: osd.3 marked itself down and dead 2024-04-22T15:26:19.009 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:18 smithi196 podman[90927]: 2024-04-22 15:26:18.713533927 +0000 UTC m=+1.303918860 container died d057260f11a392bcfb6e8973d4dcb09087df5788805f0bd306ff27487205c02f (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=reef-646b8c7, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:26:19.308 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:19 smithi196 podman[90927]: 2024-04-22 15:26:19.154271588 +0000 UTC m=+1.744656521 container cleanup d057260f11a392bcfb6e8973d4dcb09087df5788805f0bd306ff27487205c02f (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=reef-646b8c7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-22T15:26:19.308 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:19 smithi196 bash[90927]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3 2024-04-22T15:26:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:19 smithi096 ceph-mon[112443]: pgmap v113: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:19 smithi096 ceph-mon[112443]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:26:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:19 smithi096 ceph-mon[112443]: osdmap e64: 6 total, 5 up, 6 in 2024-04-22T15:26:19.759 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:19 smithi196 podman[90946]: 2024-04-22 15:26:19.397508805 +0000 UTC m=+0.680393096 container remove d057260f11a392bcfb6e8973d4dcb09087df5788805f0bd306ff27487205c02f (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-646b8c7, GIT_BRANCH=HEAD) 2024-04-22T15:26:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:19 smithi196 ceph-mon[81815]: pgmap v113: 65 pgs: 65 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:19 smithi196 ceph-mon[81815]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:26:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:19 smithi196 ceph-mon[81815]: osdmap e64: 6 total, 5 up, 6 in 2024-04-22T15:26:20.120 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:19 smithi196 podman[91036]: 2024-04-22 15:26:19.730187901 +0000 UTC m=+0.018486329 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:26:20.120 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:19 smithi196 podman[91036]: 2024-04-22 15:26:19.831526602 +0000 UTC m=+0.119825031 container create 81d4da4307cfa952b693b23d83256b95ad9c274ffe3fe2296f511c1bcc45279a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T15:26:20.120 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:20 smithi196 podman[91036]: 2024-04-22 15:26:20.050891351 +0000 UTC m=+0.339189779 container init 81d4da4307cfa952b693b23d83256b95ad9c274ffe3fe2296f511c1bcc45279a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-22T15:26:20.120 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:20 smithi196 podman[91036]: 2024-04-22 15:26:20.054116782 +0000 UTC m=+0.342415215 container start 81d4da4307cfa952b693b23d83256b95ad9c274ffe3fe2296f511c1bcc45279a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:26:20.121 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:20 smithi196 podman[91036]: 2024-04-22 15:26:20.117408637 +0000 UTC m=+0.405707085 container attach 81d4da4307cfa952b693b23d83256b95ad9c274ffe3fe2296f511c1bcc45279a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T15:26:20.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:20 smithi196 ceph-mon[81815]: osdmap e65: 6 total, 5 up, 6 in 2024-04-22T15:26:20.510 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:20 smithi196 podman[91036]: 2024-04-22 15:26:20.34417818 +0000 UTC m=+0.632476602 container died 81d4da4307cfa952b693b23d83256b95ad9c274ffe3fe2296f511c1bcc45279a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:26:20.657 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:20 smithi096 ceph-mon[112443]: osdmap e65: 6 total, 5 up, 6 in 2024-04-22T15:26:21.260 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:20 smithi196 podman[91062]: 2024-04-22 15:26:20.97100943 +0000 UTC m=+0.620899785 container remove 81d4da4307cfa952b693b23d83256b95ad9c274ffe3fe2296f511c1bcc45279a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T15:26:21.260 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:20 smithi196 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-22T15:26:21.260 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:20 smithi196 systemd[1]: Stopped Ceph osd.3 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:26:21.260 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:20 smithi196 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.3.service: Consumed 1min 4.076s CPU time. 2024-04-22T15:26:21.260 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:21 smithi196 systemd[1]: Starting Ceph osd.3 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:26:21.668 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:21 smithi196 ceph-mon[81815]: pgmap v116: 65 pgs: 18 peering, 6 stale+active+clean, 41 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:26:21.668 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:21 smithi196 ceph-mon[81815]: Health check failed: Reduced data availability: 10 pgs peering (PG_AVAILABILITY) 2024-04-22T15:26:21.669 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:21 smithi196 podman[91167]: 2024-04-22 15:26:21.310903935 +0000 UTC m=+0.018082473 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:26:21.669 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:21 smithi196 podman[91167]: 2024-04-22 15:26:21.430368185 +0000 UTC m=+0.137546712 container create b33ee4d375f5ecbdf86cc07f46a53461ce9889c8a86fc804702c0f62f7f5e5f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-22T15:26:21.669 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:21 smithi196 podman[91167]: 2024-04-22 15:26:21.66625863 +0000 UTC m=+0.373437166 container init b33ee4d375f5ecbdf86cc07f46a53461ce9889c8a86fc804702c0f62f7f5e5f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T15:26:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:21 smithi096 ceph-mon[112443]: pgmap v116: 65 pgs: 18 peering, 6 stale+active+clean, 41 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:26:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:21 smithi096 ceph-mon[112443]: Health check failed: Reduced data availability: 10 pgs peering (PG_AVAILABILITY) 2024-04-22T15:26:22.009 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:21 smithi196 podman[91167]: 2024-04-22 15:26:21.669677733 +0000 UTC m=+0.376856269 container start b33ee4d375f5ecbdf86cc07f46a53461ce9889c8a86fc804702c0f62f7f5e5f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T15:26:22.010 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:21 smithi196 podman[91167]: 2024-04-22 15:26:21.749222745 +0000 UTC m=+0.456401282 container attach b33ee4d375f5ecbdf86cc07f46a53461ce9889c8a86fc804702c0f62f7f5e5f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T15:26:22.438 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-mon[81815]: Health check failed: Degraded data redundancy: 29/330 objects degraded (8.788%), 9 pgs degraded (PG_DEGRADED) 2024-04-22T15:26:22.438 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate[91181]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:26:22.438 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate[91181]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 bash[91167]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 bash[91167]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 bash[91167]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 bash[91167]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 bash[91167]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 bash[91167]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 bash[91167]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 bash[91167]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 bash[91167]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate[91181]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate[91181]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate[91181]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate[91181]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate[91181]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-22T15:26:22.439 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate[91181]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T15:26:22.440 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate[91181]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-22T15:26:22.440 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 podman[91167]: 2024-04-22 15:26:22.168124329 +0000 UTC m=+0.875302847 container died b33ee4d375f5ecbdf86cc07f46a53461ce9889c8a86fc804702c0f62f7f5e5f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:26:22.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:22 smithi096 ceph-mon[112443]: Health check failed: Degraded data redundancy: 29/330 objects degraded (8.788%), 9 pgs degraded (PG_DEGRADED) 2024-04-22T15:26:23.010 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 podman[91308]: 2024-04-22 15:26:22.74459211 +0000 UTC m=+0.570201354 container remove b33ee4d375f5ecbdf86cc07f46a53461ce9889c8a86fc804702c0f62f7f5e5f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T15:26:23.010 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 podman[91374]: 2024-04-22 15:26:22.861049061 +0000 UTC m=+0.017924384 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:26:23.010 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:22 smithi196 podman[91374]: 2024-04-22 15:26:22.978862433 +0000 UTC m=+0.135737751 container create 984101ebadb9324e0344bd08b4e2290f32f16d9e998abbaefdf663ba2e0e1dd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T15:26:23.378 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:23 smithi196 podman[91374]: 2024-04-22 15:26:23.181070816 +0000 UTC m=+0.337946131 container init 984101ebadb9324e0344bd08b4e2290f32f16d9e998abbaefdf663ba2e0e1dd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T15:26:23.378 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:23 smithi196 podman[91374]: 2024-04-22 15:26:23.1840725 +0000 UTC m=+0.340947821 container start 984101ebadb9324e0344bd08b4e2290f32f16d9e998abbaefdf663ba2e0e1dd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T15:26:23.378 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:23 smithi196 ceph-osd[91393]: -- 172.21.15.196:0/470256342 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bf7c17d680 con 0x55bf7c146c00 2024-04-22T15:26:23.379 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:23 smithi196 bash[91374]: 984101ebadb9324e0344bd08b4e2290f32f16d9e998abbaefdf663ba2e0e1dd7 2024-04-22T15:26:23.379 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:23 smithi196 systemd[1]: Started Ceph osd.3 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:26:23.733 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:23 smithi196 ceph-mon[81815]: pgmap v117: 65 pgs: 9 active+undersized+degraded, 18 peering, 1 stale+active+clean, 7 active+undersized, 30 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 29/330 objects degraded (8.788%) 2024-04-22T15:26:23.733 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:23.733 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:26:23.733 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:23.733 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:23.733 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:26:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:23 smithi096 ceph-mon[112443]: pgmap v117: 65 pgs: 9 active+undersized+degraded, 18 peering, 1 stale+active+clean, 7 active+undersized, 30 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 29/330 objects degraded (8.788%) 2024-04-22T15:26:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:26:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:26:24.009 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:23 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[91389]: 2024-04-22T15:26:23.730+0000 7f1f641497c0 -1 Falling back to public interface 2024-04-22T15:26:24.528 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:24 smithi196 ceph-mon[81815]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs peering) 2024-04-22T15:26:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:24 smithi096 ceph-mon[112443]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs peering) 2024-04-22T15:26:25.431 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:25 smithi196 ceph-mon[81815]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/330 objects degraded (21.212%) 2024-04-22T15:26:25.719 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:25 smithi096 ceph-mon[112443]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/330 objects degraded (21.212%) 2024-04-22T15:26:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:27 smithi096 ceph-mon[112443]: pgmap v119: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/330 objects degraded (21.212%) 2024-04-22T15:26:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:27 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:27 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:27 smithi196 ceph-mon[81815]: pgmap v119: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/330 objects degraded (21.212%) 2024-04-22T15:26:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:27 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:27 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:28 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:28 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:28 smithi096 ceph-mon[112443]: pgmap v120: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/330 objects degraded (21.212%) 2024-04-22T15:26:29.010 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:28 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[91389]: 2024-04-22T15:26:28.708+0000 7f1f641497c0 -1 osd.3 63 log_to_monitors true 2024-04-22T15:26:29.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:28 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:29.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:28 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:29.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:28 smithi196 ceph-mon[81815]: pgmap v120: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/330 objects degraded (21.212%) 2024-04-22T15:26:29.862 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:29 smithi196 ceph-mon[81815]: from='osd.3 [v2:172.21.15.196:6800/2743487260,v1:172.21.15.196:6801/2743487260]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:26:29.862 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:29 smithi196 ceph-mon[81815]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:26:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:29 smithi096 ceph-mon[112443]: from='osd.3 [v2:172.21.15.196:6800/2743487260,v1:172.21.15.196:6801/2743487260]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:26:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:29 smithi096 ceph-mon[112443]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:26:30.974 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 15:26:30 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[91389]: 2024-04-22T15:26:30.570+0000 7f1f51599700 -1 osd.3 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:26:30.977 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:30 smithi196 ceph-mon[81815]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T15:26:30.977 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:30 smithi196 ceph-mon[81815]: osdmap e66: 6 total, 5 up, 6 in 2024-04-22T15:26:30.978 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:30 smithi196 ceph-mon[81815]: from='osd.3 [v2:172.21.15.196:6800/2743487260,v1:172.21.15.196:6801/2743487260]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:26:30.978 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:30 smithi196 ceph-mon[81815]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:26:30.978 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:30 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 70/330 objects degraded (21.212%), 19 pgs degraded (PG_DEGRADED) 2024-04-22T15:26:30.978 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:30 smithi196 ceph-mon[81815]: pgmap v122: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 818 B/s rd, 2 op/s; 70/330 objects degraded (21.212%) 2024-04-22T15:26:30.978 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:30 smithi196 ceph-mon[81815]: from='osd.3 ' entity='osd.3' 2024-04-22T15:26:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:30 smithi096 ceph-mon[112443]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T15:26:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:30 smithi096 ceph-mon[112443]: osdmap e66: 6 total, 5 up, 6 in 2024-04-22T15:26:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:30 smithi096 ceph-mon[112443]: from='osd.3 [v2:172.21.15.196:6800/2743487260,v1:172.21.15.196:6801/2743487260]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:26:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:30 smithi096 ceph-mon[112443]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:26:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:30 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 70/330 objects degraded (21.212%), 19 pgs degraded (PG_DEGRADED) 2024-04-22T15:26:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:30 smithi096 ceph-mon[112443]: pgmap v122: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 818 B/s rd, 2 op/s; 70/330 objects degraded (21.212%) 2024-04-22T15:26:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:30 smithi096 ceph-mon[112443]: from='osd.3 ' entity='osd.3' 2024-04-22T15:26:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:26:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:26:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:26:32.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:32.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:32.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:32.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:26:32.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:26:32.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: osd.3 [v2:172.21.15.196:6800/2743487260,v1:172.21.15.196:6801/2743487260] boot 2024-04-22T15:26:32.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: osdmap e67: 6 total, 6 up, 6 in 2024-04-22T15:26:32.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:26:32.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:32.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:32.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: osd.3 [v2:172.21.15.196:6800/2743487260,v1:172.21.15.196:6801/2743487260] boot 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: osdmap e67: 6 total, 6 up, 6 in 2024-04-22T15:26:32.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:26:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:33 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:26:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:33 smithi096 ceph-mon[112443]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-22T15:26:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:33 smithi096 ceph-mon[112443]: pgmap v124: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s; 70/330 objects degraded (21.212%) 2024-04-22T15:26:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:33 smithi096 ceph-mon[112443]: osdmap e68: 6 total, 6 up, 6 in 2024-04-22T15:26:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:33 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:26:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:33 smithi196 ceph-mon[81815]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-22T15:26:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:33 smithi196 ceph-mon[81815]: pgmap v124: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s; 70/330 objects degraded (21.212%) 2024-04-22T15:26:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:33 smithi196 ceph-mon[81815]: osdmap e68: 6 total, 6 up, 6 in 2024-04-22T15:26:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:35 smithi096 ceph-mon[112443]: pgmap v126: 65 pgs: 3 peering, 15 active+undersized+degraded, 12 active+undersized, 35 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:26:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:35 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 56/330 objects degraded (16.970%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:26:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:35 smithi196 ceph-mon[81815]: pgmap v126: 65 pgs: 3 peering, 15 active+undersized+degraded, 12 active+undersized, 35 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:26:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:35 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 56/330 objects degraded (16.970%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:26:37.562 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:37 smithi196 ceph-mon[81815]: pgmap v127: 65 pgs: 3 peering, 4 active+undersized+degraded, 3 active+undersized, 55 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 3 op/s; 16/330 objects degraded (4.848%) 2024-04-22T15:26:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:37 smithi096 ceph-mon[112443]: pgmap v127: 65 pgs: 3 peering, 4 active+undersized+degraded, 3 active+undersized, 55 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 3 op/s; 16/330 objects degraded (4.848%) 2024-04-22T15:26:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:39 smithi096 ceph-mon[112443]: pgmap v128: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-22T15:26:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:26:39.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:39 smithi096 ceph-mon[112443]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/330 objects degraded (4.848%), 4 pgs degraded) 2024-04-22T15:26:39.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:39 smithi196 ceph-mon[81815]: pgmap v128: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-22T15:26:39.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:39.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:26:39.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:39 smithi196 ceph-mon[81815]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/330 objects degraded (4.848%), 4 pgs degraded) 2024-04-22T15:26:41.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.005+0000 7f253789b700 1 -- 172.21.15.96:0/690641055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25301017d0 msgr2=0x7f2530105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.005+0000 7f253789b700 1 --2- 172.21.15.96:0/690641055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25301017d0 0x7f2530105820 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2520009910 tx=0x7f2520009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:41.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.005+0000 7f253789b700 1 -- 172.21.15.96:0/690641055 shutdown_connections 2024-04-22T15:26:41.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.005+0000 7f253789b700 1 --2- 172.21.15.96:0/690641055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25301017d0 0x7f2530105820 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.005+0000 7f253789b700 1 --2- 172.21.15.96:0/690641055 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2530100e20 0x7f2530101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.005+0000 7f253789b700 1 -- 172.21.15.96:0/690641055 >> 172.21.15.96:0/690641055 conn(0x7f25300fc6d0 msgr2=0x7f25300feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:41.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.005+0000 7f253789b700 1 -- 172.21.15.96:0/690641055 shutdown_connections 2024-04-22T15:26:41.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.005+0000 7f253789b700 1 -- 172.21.15.96:0/690641055 wait complete. 2024-04-22T15:26:41.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f253789b700 1 Processor -- start 2024-04-22T15:26:41.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f253789b700 1 -- start start 2024-04-22T15:26:41.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f253789b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2530100e20 0x7f253019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:41.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f253789b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25301017d0 0x7f253019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:41.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f253789b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253019d4c0 con 0x7f2530100e20 2024-04-22T15:26:41.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f253789b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2530196aa0 con 0x7f25301017d0 2024-04-22T15:26:41.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f2535637700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2530100e20 0x7f253019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:41.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f2535637700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2530100e20 0x7f253019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35022/0 (socket says 172.21.15.96:35022) 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f2535637700 1 -- 172.21.15.96:0/2900736587 learned_addr learned my addr 172.21.15.96:0/2900736587 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f2534e36700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25301017d0 0x7f253019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f2535637700 1 -- 172.21.15.96:0/2900736587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25301017d0 msgr2=0x7f253019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f2535637700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25301017d0 0x7f253019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f2535637700 1 -- 172.21.15.96:0/2900736587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25200095f0 con 0x7f2530100e20 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.006+0000 7f2534e36700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25301017d0 0x7f253019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.007+0000 7f2535637700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2530100e20 0x7f253019c960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f252c00b4b0 tx=0x7f252c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.007+0000 7f25267fc700 1 -- 172.21.15.96:0/2900736587 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f252c011800 con 0x7f2530100e20 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.007+0000 7f25267fc700 1 -- 172.21.15.96:0/2900736587 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f252c011e20 con 0x7f2530100e20 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.007+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2530196d80 con 0x7f2530100e20 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.007+0000 7f25267fc700 1 -- 172.21.15.96:0/2900736587 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f252c00f2e0 con 0x7f2530100e20 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.007+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25301972d0 con 0x7f2530100e20 2024-04-22T15:26:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.008+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2530109150 con 0x7f2530100e20 2024-04-22T15:26:41.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.008+0000 7f25267fc700 1 -- 172.21.15.96:0/2900736587 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f252c010380 con 0x7f2530100e20 2024-04-22T15:26:41.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.009+0000 7f25267fc700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f251c06ecd0 0x7f251c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:41.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.009+0000 7f25267fc700 1 -- 172.21.15.96:0/2900736587 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f252c08e9a0 con 0x7f2530100e20 2024-04-22T15:26:41.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.009+0000 7f2534e36700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f251c06ecd0 0x7f251c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:41.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.009+0000 7f2534e36700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f251c06ecd0 0x7f251c071190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2530197c60 tx=0x7f252000b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:41.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.014+0000 7f25267fc700 1 -- 172.21.15.96:0/2900736587 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f252c05b2e0 con 0x7f2530100e20 2024-04-22T15:26:41.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.232+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2530197fd0 con 0x7f251c06ecd0 2024-04-22T15:26:41.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.233+0000 7f25267fc700 1 -- 172.21.15.96:0/2900736587 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f2530197fd0 con 0x7f251c06ecd0 2024-04-22T15:26:41.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f251c06ecd0 msgr2=0x7f251c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f251c06ecd0 0x7f251c071190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2530197c60 tx=0x7f252000b560 comp rx=0 tx=0).stop 2024-04-22T15:26:41.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2530100e20 msgr2=0x7f253019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2530100e20 0x7f253019c960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f252c00b4b0 tx=0x7f252c00b870 comp rx=0 tx=0).stop 2024-04-22T15:26:41.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 shutdown_connections 2024-04-22T15:26:41.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f251c06ecd0 0x7f251c071190 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2530100e20 0x7f253019c960 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 --2- 172.21.15.96:0/2900736587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25301017d0 0x7f253019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 >> 172.21.15.96:0/2900736587 conn(0x7f25300fc6d0 msgr2=0x7f25300fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:41.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 shutdown_connections 2024-04-22T15:26:41.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.236+0000 7f253789b700 1 -- 172.21.15.96:0/2900736587 wait complete. 2024-04-22T15:26:41.250 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:26:41.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.341+0000 7f2735958700 1 -- 172.21.15.96:0/2171176141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 msgr2=0x7f2730101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.341+0000 7f2735958700 1 --2- 172.21.15.96:0/2171176141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 0x7f2730101200 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f272800b0a0 tx=0x7f272800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.341+0000 7f2735958700 1 -- 172.21.15.96:0/2171176141 shutdown_connections 2024-04-22T15:26:41.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.341+0000 7f2735958700 1 --2- 172.21.15.96:0/2171176141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27301017d0 0x7f2730105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.341+0000 7f2735958700 1 --2- 172.21.15.96:0/2171176141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 0x7f2730101200 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.341+0000 7f2735958700 1 -- 172.21.15.96:0/2171176141 >> 172.21.15.96:0/2171176141 conn(0x7f27300fc6d0 msgr2=0x7f27300feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.341+0000 7f2735958700 1 -- 172.21.15.96:0/2171176141 shutdown_connections 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.341+0000 7f2735958700 1 -- 172.21.15.96:0/2171176141 wait complete. 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f2735958700 1 Processor -- start 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f2735958700 1 -- start start 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f2735958700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 0x7f273019c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f2735958700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27301017d0 0x7f273019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f2735958700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273019d520 con 0x7f2730100e20 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f2735958700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2730196b00 con 0x7f27301017d0 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f272effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 0x7f273019c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f272effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 0x7f273019c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35042/0 (socket says 172.21.15.96:35042) 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f272effd700 1 -- 172.21.15.96:0/2637956163 learned_addr learned my addr 172.21.15.96:0/2637956163 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f272e7fc700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27301017d0 0x7f273019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:41.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f272effd700 1 -- 172.21.15.96:0/2637956163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27301017d0 msgr2=0x7f273019cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.342+0000 7f272effd700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27301017d0 0x7f273019cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.343+0000 7f272effd700 1 -- 172.21.15.96:0/2637956163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2728009d00 con 0x7f2730100e20 2024-04-22T15:26:41.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.343+0000 7f272e7fc700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27301017d0 0x7f273019cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:26:41.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.343+0000 7f272effd700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 0x7f273019c9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f272800b070 tx=0x7f2728007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:41.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.343+0000 7f2734956700 1 -- 172.21.15.96:0/2637956163 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f272800e040 con 0x7f2730100e20 2024-04-22T15:26:41.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.343+0000 7f2734956700 1 -- 172.21.15.96:0/2637956163 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2728003de0 con 0x7f2730100e20 2024-04-22T15:26:41.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.343+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2730196d80 con 0x7f2730100e20 2024-04-22T15:26:41.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.343+0000 7f2734956700 1 -- 172.21.15.96:0/2637956163 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f272801d6b0 con 0x7f2730100e20 2024-04-22T15:26:41.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.343+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2730197270 con 0x7f2730100e20 2024-04-22T15:26:41.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.344+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2730102300 con 0x7f2730100e20 2024-04-22T15:26:41.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.344+0000 7f2734956700 1 -- 172.21.15.96:0/2637956163 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2728004040 con 0x7f2730100e20 2024-04-22T15:26:41.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.345+0000 7f2734956700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f271806ecd0 0x7f2718071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:41.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.345+0000 7f2734956700 1 -- 172.21.15.96:0/2637956163 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f2728090600 con 0x7f2730100e20 2024-04-22T15:26:41.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.345+0000 7f272e7fc700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f271806ecd0 0x7f2718071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:41.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.345+0000 7f272e7fc700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f271806ecd0 0x7f2718071190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2730197cc0 tx=0x7f2724008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:41.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.349+0000 7f2734956700 1 -- 172.21.15.96:0/2637956163 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f272805cf40 con 0x7f2730100e20 2024-04-22T15:26:41.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.567+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f273010de60 con 0x7f271806ecd0 2024-04-22T15:26:41.571 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:41 smithi096 ceph-mon[112443]: pgmap v129: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-22T15:26:41.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.569+0000 7f2734956700 1 -- 172.21.15.96:0/2637956163 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f273010de60 con 0x7f271806ecd0 2024-04-22T15:26:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f271806ecd0 msgr2=0x7f2718071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f271806ecd0 0x7f2718071190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2730197cc0 tx=0x7f2724008040 comp rx=0 tx=0).stop 2024-04-22T15:26:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 msgr2=0x7f273019c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 0x7f273019c9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f272800b070 tx=0x7f2728007bd0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 shutdown_connections 2024-04-22T15:26:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f271806ecd0 0x7f2718071190 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2730100e20 0x7f273019c9c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 --2- 172.21.15.96:0/2637956163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27301017d0 0x7f273019cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 >> 172.21.15.96:0/2637956163 conn(0x7f27300fc6d0 msgr2=0x7f27300fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:41.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 shutdown_connections 2024-04-22T15:26:41.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.571+0000 7f2735958700 1 -- 172.21.15.96:0/2637956163 wait complete. 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 -- 172.21.15.96:0/2490143344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 msgr2=0x7f92f40fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 --2- 172.21.15.96:0/2490143344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 0x7f92f40fff40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f92e4009910 tx=0x7f92e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 -- 172.21.15.96:0/2490143344 shutdown_connections 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 --2- 172.21.15.96:0/2490143344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 0x7f92f40fff40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 --2- 172.21.15.96:0/2490143344 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92f4105ac0 0x7f92f4105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 -- 172.21.15.96:0/2490143344 >> 172.21.15.96:0/2490143344 conn(0x7f92f40fb610 msgr2=0x7f92f40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 -- 172.21.15.96:0/2490143344 shutdown_connections 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 -- 172.21.15.96:0/2490143344 wait complete. 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 Processor -- start 2024-04-22T15:26:41.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.675+0000 7f92fc4ed700 1 -- start start 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fc4ed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 0x7f92f4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fc4ed700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92f4105ac0 0x7f92f4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fc4ed700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4198570 con 0x7f92f40ffac0 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fc4ed700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f41986b0 con 0x7f92f4105ac0 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fa289700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 0x7f92f4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fa289700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 0x7f92f4197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35064/0 (socket says 172.21.15.96:35064) 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fa289700 1 -- 172.21.15.96:0/325714022 learned_addr learned my addr 172.21.15.96:0/325714022 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fa289700 1 -- 172.21.15.96:0/325714022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92f4105ac0 msgr2=0x7f92f4197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fa289700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92f4105ac0 0x7f92f4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fa289700 1 -- 172.21.15.96:0/325714022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e40095f0 con 0x7f92f40ffac0 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fa289700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 0x7f92f4197a60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f92f000b4b0 tx=0x7f92f000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92eb7fe700 1 -- 172.21.15.96:0/325714022 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92f0011800 con 0x7f92f40ffac0 2024-04-22T15:26:41.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92eb7fe700 1 -- 172.21.15.96:0/325714022 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92f0011e20 con 0x7f92f40ffac0 2024-04-22T15:26:41.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92eb7fe700 1 -- 172.21.15.96:0/325714022 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92f000f2e0 con 0x7f92f40ffac0 2024-04-22T15:26:41.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f41a0ad0 con 0x7f92f40ffac0 2024-04-22T15:26:41.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.676+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f41a0e90 con 0x7f92f40ffac0 2024-04-22T15:26:41.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.677+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f41080e0 con 0x7f92f40ffac0 2024-04-22T15:26:41.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.678+0000 7f92eb7fe700 1 -- 172.21.15.96:0/325714022 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f92f0010380 con 0x7f92f40ffac0 2024-04-22T15:26:41.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.678+0000 7f92eb7fe700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f92e006ecd0 0x7f92e0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:41.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.678+0000 7f92eb7fe700 1 -- 172.21.15.96:0/325714022 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f92f008e9a0 con 0x7f92f40ffac0 2024-04-22T15:26:41.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.678+0000 7f92f9a88700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f92e006ecd0 0x7f92e0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:41.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.679+0000 7f92f9a88700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f92e006ecd0 0x7f92e0071190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92e4006010 tx=0x7f92e400b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:41.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.683+0000 7f92eb7fe700 1 -- 172.21.15.96:0/325714022 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f92f00577b0 con 0x7f92f40ffac0 2024-04-22T15:26:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:41 smithi196 ceph-mon[81815]: pgmap v129: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-22T15:26:41.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.891+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f92f410cd40 con 0x7f92e006ecd0 2024-04-22T15:26:41.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.902+0000 7f92eb7fe700 1 -- 172.21.15.96:0/325714022 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f92f410cd40 con 0x7f92e006ecd0 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (5m) 49s ago 16m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (17m) 49s ago 17m 8870k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (15m) 15s ago 15m 8334k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (2m) 49s ago 16m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (2m) 15s ago 15m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (5m) 49s ago 16m 80.7M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (10m) 49s ago 10m 155M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (11m) 49s ago 11m 90.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (11m) 15s ago 11m 26.7M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (10m) 15s ago 10m 26.6M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:26:41.906 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (7m) 49s ago 18m 505M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (6m) 15s ago 15m 451M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (3m) 49s ago 18m 46.9M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (3m) 15s ago 15m 38.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (6m) 49s ago 16m 17.3M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (6m) 15s ago 15m 16.9M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (2m) 49s ago 14m 92.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (92s) 49s ago 14m 93.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (54s) 49s ago 13m 11.9M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (19s) 15s ago 13m 12.6M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (12m) 15s ago 12m 215M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 301cfbadc06a 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (12m) 15s ago 12m 196M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:26:41.907 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (5m) 49s ago 16m 77.0M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:26:41.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f92e006ecd0 msgr2=0x7f92e0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f92e006ecd0 0x7f92e0071190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92e4006010 tx=0x7f92e400b540 comp rx=0 tx=0).stop 2024-04-22T15:26:41.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 msgr2=0x7f92f4197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:41.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 0x7f92f4197a60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f92f000b4b0 tx=0x7f92f000b870 comp rx=0 tx=0).stop 2024-04-22T15:26:41.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 shutdown_connections 2024-04-22T15:26:41.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f92e006ecd0 0x7f92e0071190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f40ffac0 0x7f92f4197a60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 --2- 172.21.15.96:0/325714022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92f4105ac0 0x7f92f4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:41.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 >> 172.21.15.96:0/325714022 conn(0x7f92f40fb610 msgr2=0x7f92f40fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:41.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 shutdown_connections 2024-04-22T15:26:41.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:41.904+0000 7f92fc4ed700 1 -- 172.21.15.96:0/325714022 wait complete. 2024-04-22T15:26:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.012+0000 7f35b4a08700 1 -- 172.21.15.96:0/247116255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b0105ab0 msgr2=0x7f35b0105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.012+0000 7f35b4a08700 1 --2- 172.21.15.96:0/247116255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b0105ab0 0x7f35b0105e90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f35a800b0a0 tx=0x7f35a800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.012+0000 7f35b4a08700 1 -- 172.21.15.96:0/247116255 shutdown_connections 2024-04-22T15:26:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.012+0000 7f35b4a08700 1 --2- 172.21.15.96:0/247116255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35b00ffab0 0x7f35b00fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.012+0000 7f35b4a08700 1 --2- 172.21.15.96:0/247116255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b0105ab0 0x7f35b0105e90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.012+0000 7f35b4a08700 1 -- 172.21.15.96:0/247116255 >> 172.21.15.96:0/247116255 conn(0x7f35b00fb5e0 msgr2=0x7f35b00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.012+0000 7f35b4a08700 1 -- 172.21.15.96:0/247116255 shutdown_connections 2024-04-22T15:26:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.012+0000 7f35b4a08700 1 -- 172.21.15.96:0/247116255 wait complete. 2024-04-22T15:26:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35b4a08700 1 Processor -- start 2024-04-22T15:26:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35b4a08700 1 -- start start 2024-04-22T15:26:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35b4a08700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b00ffab0 0x7f35b0195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35b4a08700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35b0105ab0 0x7f35b0195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35b4a08700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b0069e00 con 0x7f35b00ffab0 2024-04-22T15:26:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35b4a08700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b01962c0 con 0x7f35b0105ab0 2024-04-22T15:26:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b00ffab0 0x7f35b0195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b00ffab0 0x7f35b0195840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35084/0 (socket says 172.21.15.96:35084) 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35ae59c700 1 -- 172.21.15.96:0/1408826163 learned_addr learned my addr 172.21.15.96:0/1408826163 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35ae59c700 1 -- 172.21.15.96:0/1408826163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35b0105ab0 msgr2=0x7f35b0195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35ae59c700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35b0105ab0 0x7f35b0195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35ae59c700 1 -- 172.21.15.96:0/1408826163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a8009d00 con 0x7f35b00ffab0 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.013+0000 7f35ae59c700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b00ffab0 0x7f35b0195840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f35a8000f80 tx=0x7f35a8007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.014+0000 7f359f7fe700 1 -- 172.21.15.96:0/1408826163 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35a800e040 con 0x7f35b00ffab0 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.014+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b0196540 con 0x7f35b00ffab0 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.014+0000 7f359f7fe700 1 -- 172.21.15.96:0/1408826163 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35a8003de0 con 0x7f35b00ffab0 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.014+0000 7f359f7fe700 1 -- 172.21.15.96:0/1408826163 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35a801d720 con 0x7f35b00ffab0 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.014+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b01a30d0 con 0x7f35b00ffab0 2024-04-22T15:26:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.015+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b01080b0 con 0x7f35b00ffab0 2024-04-22T15:26:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.015+0000 7f359f7fe700 1 -- 172.21.15.96:0/1408826163 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f35a8004690 con 0x7f35b00ffab0 2024-04-22T15:26:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.016+0000 7f359f7fe700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f359806ecc0 0x7f3598071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.016+0000 7f35add9b700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f359806ecc0 0x7f3598071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.016+0000 7f359f7fe700 1 -- 172.21.15.96:0/1408826163 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f35a8090620 con 0x7f35b00ffab0 2024-04-22T15:26:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.016+0000 7f35add9b700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f359806ecc0 0x7f3598071180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f35b0069710 tx=0x7f35a4006d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:42.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.020+0000 7f359f7fe700 1 -- 172.21.15.96:0/1408826163 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f35a805cf60 con 0x7f35b00ffab0 2024-04-22T15:26:42.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.308+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f35b0069440 con 0x7f35b00ffab0 2024-04-22T15:26:42.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.308+0000 7f359f7fe700 1 -- 172.21.15.96:0/1408826163 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+836 (secure 0 0 0) 0x7f35a805caf0 con 0x7f35b00ffab0 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2, 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:26:42.312 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:26:42.313 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:26:42.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:26:42.313 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:26:42.313 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 6, 2024-04-22T15:26:42.313 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 8 2024-04-22T15:26:42.313 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:26:42.313 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:26:42.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f359806ecc0 msgr2=0x7f3598071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f359806ecc0 0x7f3598071180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f35b0069710 tx=0x7f35a4006d20 comp rx=0 tx=0).stop 2024-04-22T15:26:42.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b00ffab0 msgr2=0x7f35b0195840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b00ffab0 0x7f35b0195840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f35a8000f80 tx=0x7f35a8007bd0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 shutdown_connections 2024-04-22T15:26:42.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f359806ecc0 0x7f3598071180 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35b00ffab0 0x7f35b0195840 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 --2- 172.21.15.96:0/1408826163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35b0105ab0 0x7f35b0195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 >> 172.21.15.96:0/1408826163 conn(0x7f35b00fb5e0 msgr2=0x7f35b00fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:42.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 shutdown_connections 2024-04-22T15:26:42.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.311+0000 7f35b4a08700 1 -- 172.21.15.96:0/1408826163 wait complete. 2024-04-22T15:26:42.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 -- 172.21.15.96:0/3930195399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc060105bd0 msgr2=0x7fc060105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 --2- 172.21.15.96:0/3930195399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc060105bd0 0x7fc060105fb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc05c009910 tx=0x7fc05c009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:42.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 -- 172.21.15.96:0/3930195399 shutdown_connections 2024-04-22T15:26:42.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 --2- 172.21.15.96:0/3930195399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0600ff3d0 0x7fc0600ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 --2- 172.21.15.96:0/3930195399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc060105bd0 0x7fc060105fb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 -- 172.21.15.96:0/3930195399 >> 172.21.15.96:0/3930195399 conn(0x7fc060074db0 msgr2=0x7fc0600751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:42.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 -- 172.21.15.96:0/3930195399 shutdown_connections 2024-04-22T15:26:42.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 -- 172.21.15.96:0/3930195399 wait complete. 2024-04-22T15:26:42.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 Processor -- start 2024-04-22T15:26:42.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.417+0000 7fc0683a5700 1 -- start start 2024-04-22T15:26:42.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc0683a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0600ff3d0 0x7fc06019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc0683a5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc060105bd0 0x7fc06019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc0683a5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06019d410 con 0x7fc0600ff3d0 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc0683a5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc060196a40 con 0x7fc060105bd0 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc066141700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0600ff3d0 0x7fc06019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc066141700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0600ff3d0 0x7fc06019c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35106/0 (socket says 172.21.15.96:35106) 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc066141700 1 -- 172.21.15.96:0/321507652 learned_addr learned my addr 172.21.15.96:0/321507652 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc066141700 1 -- 172.21.15.96:0/321507652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc060105bd0 msgr2=0x7fc06019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc066141700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc060105bd0 0x7fc06019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc066141700 1 -- 172.21.15.96:0/321507652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c0095f0 con 0x7fc0600ff3d0 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc066141700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0600ff3d0 0x7fc06019c8b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0092e0 tx=0x7fc05c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc0577fe700 1 -- 172.21.15.96:0/321507652 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc05c01d070 con 0x7fc0600ff3d0 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc0577fe700 1 -- 172.21.15.96:0/321507652 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc05c00f460 con 0x7fc0600ff3d0 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc060196cc0 con 0x7fc0600ff3d0 2024-04-22T15:26:42.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc0577fe700 1 -- 172.21.15.96:0/321507652 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc05c017610 con 0x7fc0600ff3d0 2024-04-22T15:26:42.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.418+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc060197230 con 0x7fc0600ff3d0 2024-04-22T15:26:42.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.419+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc060109210 con 0x7fc0600ff3d0 2024-04-22T15:26:42.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.420+0000 7fc0577fe700 1 -- 172.21.15.96:0/321507652 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc05c00f5d0 con 0x7fc0600ff3d0 2024-04-22T15:26:42.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.420+0000 7fc0577fe700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc04c06ecd0 0x7fc04c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:42.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.420+0000 7fc0577fe700 1 -- 172.21.15.96:0/321507652 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fc05c0902e0 con 0x7fc0600ff3d0 2024-04-22T15:26:42.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.420+0000 7fc065940700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc04c06ecd0 0x7fc04c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:42.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.421+0000 7fc065940700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc04c06ecd0 0x7fc04c071190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc060197bb0 tx=0x7fc050006d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:42.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.425+0000 7fc0577fe700 1 -- 172.21.15.96:0/321507652 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc05c05cc20 con 0x7fc0600ff3d0 2024-04-22T15:26:42.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.669+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc060197eb0 con 0x7fc0600ff3d0 2024-04-22T15:26:42.672 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:42 smithi096 ceph-mon[112443]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:42.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.669+0000 7fc0577fe700 1 -- 172.21.15.96:0/321507652 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1901 (secure 0 0 0) 0x7fc05c05c7b0 con 0x7fc0600ff3d0 2024-04-22T15:26:42.673 INFO:teuthology.orchestra.run.smithi096.stdout:e20 2024-04-22T15:26:42.673 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:26:42.673 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:26:42.673 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:26:42.673 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:26:42.673 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:26:42.673 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 20 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:26:42.674 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:26:42.675 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:26:42.676 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:26:42.676 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:26:42.676 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:26:42.676 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:26:42.676 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc04c06ecd0 msgr2=0x7fc04c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc04c06ecd0 0x7fc04c071190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc060197bb0 tx=0x7fc050006d20 comp rx=0 tx=0).stop 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0600ff3d0 msgr2=0x7fc06019c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0600ff3d0 0x7fc06019c8b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0092e0 tx=0x7fc05c00be20 comp rx=0 tx=0).stop 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 shutdown_connections 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc04c06ecd0 0x7fc04c071190 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0600ff3d0 0x7fc06019c8b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 --2- 172.21.15.96:0/321507652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc060105bd0 0x7fc06019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 >> 172.21.15.96:0/321507652 conn(0x7fc060074db0 msgr2=0x7fc0600fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 shutdown_connections 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.671+0000 7fc0683a5700 1 -- 172.21.15.96:0/321507652 wait complete. 2024-04-22T15:26:42.677 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:26:42.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:42 smithi196 ceph-mon[81815]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:42.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.776+0000 7febeda4a700 1 -- 172.21.15.96:0/2043958842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 msgr2=0x7febe80fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.776+0000 7febeda4a700 1 --2- 172.21.15.96:0/2043958842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 0x7febe80fff30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7febdc009910 tx=0x7febdc009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:42.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.776+0000 7febeda4a700 1 -- 172.21.15.96:0/2043958842 shutdown_connections 2024-04-22T15:26:42.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.776+0000 7febeda4a700 1 --2- 172.21.15.96:0/2043958842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 0x7febe80fff30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.776+0000 7febeda4a700 1 --2- 172.21.15.96:0/2043958842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febe8105ab0 0x7febe8105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.776+0000 7febeda4a700 1 -- 172.21.15.96:0/2043958842 >> 172.21.15.96:0/2043958842 conn(0x7febe80fb5e0 msgr2=0x7febe80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:42.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.776+0000 7febeda4a700 1 -- 172.21.15.96:0/2043958842 shutdown_connections 2024-04-22T15:26:42.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.776+0000 7febeda4a700 1 -- 172.21.15.96:0/2043958842 wait complete. 2024-04-22T15:26:42.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.777+0000 7febeda4a700 1 Processor -- start 2024-04-22T15:26:42.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.777+0000 7febeda4a700 1 -- start start 2024-04-22T15:26:42.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.777+0000 7febeda4a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 0x7febe819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:42.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.777+0000 7febeda4a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febe8105ab0 0x7febe819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:42.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.777+0000 7febeda4a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe819d2e0 con 0x7febe80ffab0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.777+0000 7febeda4a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe8196950 con 0x7febe8105ab0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.777+0000 7febe6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 0x7febe819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febe6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 0x7febe819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35116/0 (socket says 172.21.15.96:35116) 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febe6ffd700 1 -- 172.21.15.96:0/2089040813 learned_addr learned my addr 172.21.15.96:0/2089040813 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febe6ffd700 1 -- 172.21.15.96:0/2089040813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febe8105ab0 msgr2=0x7febe819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febe6ffd700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febe8105ab0 0x7febe819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febe6ffd700 1 -- 172.21.15.96:0/2089040813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc0095f0 con 0x7febe80ffab0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febe6ffd700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 0x7febe819c810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7febd800c930 tx=0x7febd800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febeca48700 1 -- 172.21.15.96:0/2089040813 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd8007ad0 con 0x7febe80ffab0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe8196c30 con 0x7febe80ffab0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febeca48700 1 -- 172.21.15.96:0/2089040813 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febd8007c30 con 0x7febe80ffab0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febeca48700 1 -- 172.21.15.96:0/2089040813 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd8018610 con 0x7febe80ffab0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.778+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe8197180 con 0x7febe80ffab0 2024-04-22T15:26:42.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.779+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe81080d0 con 0x7febe80ffab0 2024-04-22T15:26:42.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.780+0000 7febeca48700 1 -- 172.21.15.96:0/2089040813 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7febd800f450 con 0x7febe80ffab0 2024-04-22T15:26:42.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.780+0000 7febeca48700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7febd006ecd0 0x7febd0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:42.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.780+0000 7febeca48700 1 -- 172.21.15.96:0/2089040813 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6414+0+0 (secure 0 0 0) 0x7febd808f040 con 0x7febe80ffab0 2024-04-22T15:26:42.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.780+0000 7febe67fc700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7febd006ecd0 0x7febd0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:42.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.780+0000 7febe67fc700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7febd006ecd0 0x7febd0071190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7febdc006010 tx=0x7febdc00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:42.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:42.785+0000 7febeca48700 1 -- 172.21.15.96:0/2089040813 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7febd805b980 con 0x7febe80ffab0 2024-04-22T15:26:43.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.001+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7febe8197e10 con 0x7febd006ecd0 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.002+0000 7febeca48700 1 -- 172.21.15.96:0/2089040813 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7febe8197e10 con 0x7febd006ecd0 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:26:43.006 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "10/23 daemons upgraded", 2024-04-22T15:26:43.007 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:26:43.007 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:26:43.007 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:26:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.004+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7febd006ecd0 msgr2=0x7febd0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.004+0000 7febeda4a700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7febd006ecd0 0x7febd0071190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7febdc006010 tx=0x7febdc00b540 comp rx=0 tx=0).stop 2024-04-22T15:26:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.004+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 msgr2=0x7febe819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.004+0000 7febeda4a700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 0x7febe819c810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7febd800c930 tx=0x7febd800ccf0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.005+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 shutdown_connections 2024-04-22T15:26:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.005+0000 7febeda4a700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7febd006ecd0 0x7febd0071190 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.005+0000 7febeda4a700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe80ffab0 0x7febe819c810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.005+0000 7febeda4a700 1 --2- 172.21.15.96:0/2089040813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febe8105ab0 0x7febe819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.005+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 >> 172.21.15.96:0/2089040813 conn(0x7febe80fb5e0 msgr2=0x7febe80fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:43.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.005+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 shutdown_connections 2024-04-22T15:26:43.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.005+0000 7febeda4a700 1 -- 172.21.15.96:0/2089040813 wait complete. 2024-04-22T15:26:43.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.108+0000 7f79f49e0700 1 -- 172.21.15.96:0/3974184354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 msgr2=0x7f79f0105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:43.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.108+0000 7f79f49e0700 1 --2- 172.21.15.96:0/3974184354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 0x7f79f0105ea0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009910 tx=0x7f79e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:26:43.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.108+0000 7f79f49e0700 1 -- 172.21.15.96:0/3974184354 shutdown_connections 2024-04-22T15:26:43.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.108+0000 7f79f49e0700 1 --2- 172.21.15.96:0/3974184354 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79f00ffac0 0x7f79f00fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.108+0000 7f79f49e0700 1 --2- 172.21.15.96:0/3974184354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 0x7f79f0105ea0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.108+0000 7f79f49e0700 1 -- 172.21.15.96:0/3974184354 >> 172.21.15.96:0/3974184354 conn(0x7f79f00fb5d0 msgr2=0x7f79f00fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:43.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.108+0000 7f79f49e0700 1 -- 172.21.15.96:0/3974184354 shutdown_connections 2024-04-22T15:26:43.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.108+0000 7f79f49e0700 1 -- 172.21.15.96:0/3974184354 wait complete. 2024-04-22T15:26:43.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.109+0000 7f79f49e0700 1 Processor -- start 2024-04-22T15:26:43.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.109+0000 7f79f49e0700 1 -- start start 2024-04-22T15:26:43.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.109+0000 7f79f49e0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79f00ffac0 0x7f79f0197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:43.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.109+0000 7f79f49e0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 0x7f79f0197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:43.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.109+0000 7f79f49e0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f0198430 con 0x7f79f0105ac0 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.109+0000 7f79f49e0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f0198570 con 0x7f79f00ffac0 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.109+0000 7f79edd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 0x7f79f0197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.109+0000 7f79edd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 0x7f79f0197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35140/0 (socket says 172.21.15.96:35140) 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.109+0000 7f79edd9b700 1 -- 172.21.15.96:0/2098934916 learned_addr learned my addr 172.21.15.96:0/2098934916 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79ee59c700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79f00ffac0 0x7f79f0197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79edd9b700 1 -- 172.21.15.96:0/2098934916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79f00ffac0 msgr2=0x7f79f0197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79edd9b700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79f00ffac0 0x7f79f0197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79edd9b700 1 -- 172.21.15.96:0/2098934916 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e40095f0 con 0x7f79f0105ac0 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79edd9b700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 0x7f79f0197e60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f79e000d8d0 tx=0x7f79e000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79df7fe700 1 -- 172.21.15.96:0/2098934916 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e00098a0 con 0x7f79f0105ac0 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f01a0990 con 0x7f79f0105ac0 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79df7fe700 1 -- 172.21.15.96:0/2098934916 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79e0010460 con 0x7f79f0105ac0 2024-04-22T15:26:43.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79df7fe700 1 -- 172.21.15.96:0/2098934916 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e000f5f0 con 0x7f79f0105ac0 2024-04-22T15:26:43.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.110+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f01a0da0 con 0x7f79f0105ac0 2024-04-22T15:26:43.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.111+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f01080c0 con 0x7f79f0105ac0 2024-04-22T15:26:43.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.112+0000 7f79df7fe700 1 -- 172.21.15.96:0/2098934916 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f79e0009a00 con 0x7f79f0105ac0 2024-04-22T15:26:43.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.112+0000 7f79df7fe700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f79d806ecc0 0x7f79d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:26:43.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.112+0000 7f79ee59c700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f79d806ecc0 0x7f79d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:26:43.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.112+0000 7f79df7fe700 1 -- 172.21.15.96:0/2098934916 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f79e008ecd0 con 0x7f79f0105ac0 2024-04-22T15:26:43.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.113+0000 7f79ee59c700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f79d806ecc0 0x7f79d8071180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000c00 tx=0x7f79e401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:26:43.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.117+0000 7f79df7fe700 1 -- 172.21.15.96:0/2098934916 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f79e0057ae0 con 0x7f79f0105ac0 2024-04-22T15:26:43.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.399+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f79f0101200 con 0x7f79f0105ac0 2024-04-22T15:26:43.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.400+0000 7f79df7fe700 1 -- 172.21.15.96:0/2098934916 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f79e005b1a0 con 0x7f79f0105ac0 2024-04-22T15:26:43.403 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:26:43.403 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:26:43.403 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:26:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f79d806ecc0 msgr2=0x7f79d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f79d806ecc0 0x7f79d8071180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000c00 tx=0x7f79e401a040 comp rx=0 tx=0).stop 2024-04-22T15:26:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 msgr2=0x7f79f0197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:26:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 0x7f79f0197e60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f79e000d8d0 tx=0x7f79e000dbe0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 shutdown_connections 2024-04-22T15:26:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f79d806ecc0 0x7f79d8071180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79f00ffac0 0x7f79f0197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 --2- 172.21.15.96:0/2098934916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f0105ac0 0x7f79f0197e60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:26:43.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 >> 172.21.15.96:0/2098934916 conn(0x7f79f00fb5d0 msgr2=0x7f79f00fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:26:43.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 shutdown_connections 2024-04-22T15:26:43.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:26:43.402+0000 7f79f49e0700 1 -- 172.21.15.96:0/2098934916 wait complete. 2024-04-22T15:26:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:43 smithi096 ceph-mon[112443]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:43 smithi096 ceph-mon[112443]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:43 smithi096 ceph-mon[112443]: pgmap v130: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-22T15:26:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:43 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1408826163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:43 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/321507652' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:26:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:43 smithi196 ceph-mon[81815]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:43 smithi196 ceph-mon[81815]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:43 smithi196 ceph-mon[81815]: pgmap v130: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-22T15:26:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:43 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1408826163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:26:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:43 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/321507652' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:26:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:44 smithi096 ceph-mon[112443]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:44 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:44 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2098934916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:26:44.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:44 smithi196 ceph-mon[81815]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:26:44.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:44 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:44.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:44 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2098934916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:26:45.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:45 smithi196 ceph-mon[81815]: pgmap v131: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:26:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:45 smithi096 ceph-mon[112443]: pgmap v131: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:26:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:47 smithi096 ceph-mon[112443]: pgmap v132: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-22T15:26:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:47 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:26:47.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:47 smithi196 ceph-mon[81815]: pgmap v132: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-22T15:26:47.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:47 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:26:48.688 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:48 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:26:48.688 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:48 smithi196 ceph-mon[81815]: Upgrade: osd.4 is safe to restart 2024-04-22T15:26:48.688 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:48 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:48.688 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:48 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:26:48.688 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:48 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:26:48.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:48 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:26:48.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:48 smithi096 ceph-mon[112443]: Upgrade: osd.4 is safe to restart 2024-04-22T15:26:48.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:48 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:48.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:48 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:26:48.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:48 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:26:49.662 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:49 smithi196 ceph-mon[81815]: Upgrade: Updating osd.4 2024-04-22T15:26:49.662 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:49 smithi196 ceph-mon[81815]: Deploying daemon osd.4 on smithi196 2024-04-22T15:26:49.662 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:49 smithi196 ceph-mon[81815]: pgmap v133: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:49 smithi096 ceph-mon[112443]: Upgrade: Updating osd.4 2024-04-22T15:26:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:49 smithi096 ceph-mon[112443]: Deploying daemon osd.4 on smithi196 2024-04-22T15:26:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:49 smithi096 ceph-mon[112443]: pgmap v133: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:26:51.566 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:51 smithi196 ceph-mon[81815]: pgmap v134: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:26:51.566 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:51 smithi196 systemd[1]: Stopping Ceph osd.4 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:26:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:51 smithi096 ceph-mon[112443]: pgmap v134: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:26:52.009 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:51 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[47277]: 2024-04-22T15:26:51.562+0000 7f39b393a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:26:52.010 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:51 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[47277]: 2024-04-22T15:26:51.562+0000 7f39b393a700 -1 osd.4 68 *** Got signal Terminated *** 2024-04-22T15:26:52.010 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:51 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[47277]: 2024-04-22T15:26:51.562+0000 7f39b393a700 -1 osd.4 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:26:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:52 smithi096 ceph-mon[112443]: osd.4 marked itself down and dead 2024-04-22T15:26:52.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:52 smithi196 ceph-mon[81815]: osd.4 marked itself down and dead 2024-04-22T15:26:53.181 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:52 smithi196 podman[96129]: 2024-04-22 15:26:52.892055507 +0000 UTC m=+1.452106969 container died 301cfbadc06a06df2f7daa8cb8dc7f5a5635dc9eb83813384551eac9cbfc146a (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4, RELEASE=reef-646b8c7, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T15:26:53.181 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:53 smithi196 podman[96129]: 2024-04-22 15:26:53.178807773 +0000 UTC m=+1.738859243 container cleanup 301cfbadc06a06df2f7daa8cb8dc7f5a5635dc9eb83813384551eac9cbfc146a (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-646b8c7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T15:26:53.474 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:53 smithi196 ceph-mon[81815]: pgmap v135: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:26:53.475 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:53 smithi196 ceph-mon[81815]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:26:53.475 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:53 smithi196 ceph-mon[81815]: osdmap e69: 6 total, 5 up, 6 in 2024-04-22T15:26:53.475 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:53.475 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:26:53.475 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:53 smithi196 bash[96129]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4 2024-04-22T15:26:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:53 smithi096 ceph-mon[112443]: pgmap v135: 65 pgs: 65 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:26:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:53 smithi096 ceph-mon[112443]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:26:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:53 smithi096 ceph-mon[112443]: osdmap e69: 6 total, 5 up, 6 in 2024-04-22T15:26:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:26:53.759 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:53 smithi196 podman[96148]: 2024-04-22 15:26:53.472044714 +0000 UTC m=+0.578970981 container remove 301cfbadc06a06df2f7daa8cb8dc7f5a5635dc9eb83813384551eac9cbfc146a (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=reef-646b8c7, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:26:54.077 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:53 smithi196 podman[96240]: 2024-04-22 15:26:53.798002925 +0000 UTC m=+0.114861193 container create 7bdf939fa025256d11bcc25216baf40fe64a5cb94cf65611ff1b94b58329e5f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:26:54.077 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:53 smithi196 podman[96240]: 2024-04-22 15:26:53.700882629 +0000 UTC m=+0.017740896 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:26:54.077 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:54 smithi196 podman[96240]: 2024-04-22 15:26:54.000293033 +0000 UTC m=+0.317151308 container init 7bdf939fa025256d11bcc25216baf40fe64a5cb94cf65611ff1b94b58329e5f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:26:54.078 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:54 smithi196 podman[96240]: 2024-04-22 15:26:54.004291224 +0000 UTC m=+0.321149491 container start 7bdf939fa025256d11bcc25216baf40fe64a5cb94cf65611ff1b94b58329e5f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:26:54.078 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:54 smithi196 podman[96240]: 2024-04-22 15:26:54.075104731 +0000 UTC m=+0.391963015 container attach 7bdf939fa025256d11bcc25216baf40fe64a5cb94cf65611ff1b94b58329e5f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T15:26:54.345 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:54 smithi196 podman[96240]: 2024-04-22 15:26:54.297065322 +0000 UTC m=+0.613923582 container died 7bdf939fa025256d11bcc25216baf40fe64a5cb94cf65611ff1b94b58329e5f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T15:26:54.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:54 smithi096 ceph-mon[112443]: osdmap e70: 6 total, 5 up, 6 in 2024-04-22T15:26:54.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:54 smithi196 ceph-mon[81815]: osdmap e70: 6 total, 5 up, 6 in 2024-04-22T15:26:55.208 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:54 smithi196 podman[96266]: 2024-04-22 15:26:54.870245786 +0000 UTC m=+0.565036643 container remove 7bdf939fa025256d11bcc25216baf40fe64a5cb94cf65611ff1b94b58329e5f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-22T15:26:55.208 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:54 smithi196 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-22T15:26:55.208 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:54 smithi196 systemd[1]: Stopped Ceph osd.4 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:26:55.208 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:54 smithi196 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.4.service: Consumed 54.130s CPU time. 2024-04-22T15:26:55.208 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:55 smithi196 systemd[1]: Starting Ceph osd.4 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:26:55.474 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:55 smithi196 ceph-mon[81815]: pgmap v138: 65 pgs: 2 active+undersized+degraded, 2 peering, 4 stale+active+clean, 3 active+undersized, 54 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 7/330 objects degraded (2.121%) 2024-04-22T15:26:55.474 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:55 smithi196 ceph-mon[81815]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-22T15:26:55.475 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:55 smithi196 ceph-mon[81815]: Health check failed: Degraded data redundancy: 7/330 objects degraded (2.121%), 2 pgs degraded (PG_DEGRADED) 2024-04-22T15:26:55.475 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:55 smithi196 podman[96366]: 2024-04-22 15:26:55.202974152 +0000 UTC m=+0.017958066 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:26:55.475 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:55 smithi196 podman[96366]: 2024-04-22 15:26:55.329793618 +0000 UTC m=+0.144777506 container create b17be9b1a1dc92fc704c8df12400df951aa5f21be65aa01c6799cf47bc668897 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:26:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:55 smithi096 ceph-mon[112443]: pgmap v138: 65 pgs: 2 active+undersized+degraded, 2 peering, 4 stale+active+clean, 3 active+undersized, 54 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 7/330 objects degraded (2.121%) 2024-04-22T15:26:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:55 smithi096 ceph-mon[112443]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-22T15:26:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:55 smithi096 ceph-mon[112443]: Health check failed: Degraded data redundancy: 7/330 objects degraded (2.121%), 2 pgs degraded (PG_DEGRADED) 2024-04-22T15:26:56.010 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:55 smithi196 podman[96366]: 2024-04-22 15:26:55.573869586 +0000 UTC m=+0.388853476 container init b17be9b1a1dc92fc704c8df12400df951aa5f21be65aa01c6799cf47bc668897 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T15:26:56.010 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:55 smithi196 podman[96366]: 2024-04-22 15:26:55.577006616 +0000 UTC m=+0.391990505 container start b17be9b1a1dc92fc704c8df12400df951aa5f21be65aa01c6799cf47bc668897 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:26:56.010 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:55 smithi196 podman[96366]: 2024-04-22 15:26:55.65711656 +0000 UTC m=+0.472100462 container attach b17be9b1a1dc92fc704c8df12400df951aa5f21be65aa01c6799cf47bc668897 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate[96380]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate[96380]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 bash[96366]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 bash[96366]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 bash[96366]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 bash[96366]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 bash[96366]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 bash[96366]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 bash[96366]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 bash[96366]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T15:26:56.455 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 bash[96366]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-22T15:26:56.456 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate[96380]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T15:26:56.456 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate[96380]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T15:26:56.456 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate[96380]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T15:26:56.456 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate[96380]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T15:26:56.456 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate[96380]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-22T15:26:56.456 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate[96380]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T15:26:56.456 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate[96380]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-22T15:26:56.456 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 podman[96366]: 2024-04-22 15:26:56.086218409 +0000 UTC m=+0.901202303 container died b17be9b1a1dc92fc704c8df12400df951aa5f21be65aa01c6799cf47bc668897 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-22T15:26:57.009 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 podman[96522]: 2024-04-22 15:26:56.750490303 +0000 UTC m=+0.655266360 container remove b17be9b1a1dc92fc704c8df12400df951aa5f21be65aa01c6799cf47bc668897 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T15:26:57.010 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 podman[96581]: 2024-04-22 15:26:56.945005313 +0000 UTC m=+0.095638189 container create ad74967cb5b79696a1fee88446eecba3bef328257e437dd6ad7f581f2425568e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-22T15:26:57.010 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:56 smithi196 podman[96581]: 2024-04-22 15:26:56.867320485 +0000 UTC m=+0.017953361 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:26:57.351 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:57 smithi196 podman[96581]: 2024-04-22 15:26:57.164197013 +0000 UTC m=+0.314829889 container init ad74967cb5b79696a1fee88446eecba3bef328257e437dd6ad7f581f2425568e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:26:57.351 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:57 smithi196 podman[96581]: 2024-04-22 15:26:57.166995847 +0000 UTC m=+0.317628729 container start ad74967cb5b79696a1fee88446eecba3bef328257e437dd6ad7f581f2425568e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:26:57.351 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:57 smithi196 bash[96581]: ad74967cb5b79696a1fee88446eecba3bef328257e437dd6ad7f581f2425568e 2024-04-22T15:26:57.351 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:57 smithi196 systemd[1]: Started Ceph osd.4 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:26:57.716 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:57 smithi196 ceph-mon[81815]: pgmap v139: 65 pgs: 11 active+undersized+degraded, 2 peering, 1 stale+active+clean, 13 active+undersized, 38 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s; 46/330 objects degraded (13.939%) 2024-04-22T15:26:57.717 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:57.717 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:57.717 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:26:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:57 smithi096 ceph-mon[112443]: pgmap v139: 65 pgs: 11 active+undersized+degraded, 2 peering, 1 stale+active+clean, 13 active+undersized, 38 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s; 46/330 objects degraded (13.939%) 2024-04-22T15:26:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:26:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:26:58.011 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:26:57 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T15:26:57.713+0000 7f751a2917c0 -1 Falling back to public interface 2024-04-22T15:26:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:58 smithi096 ceph-mon[112443]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T15:26:58.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:58 smithi196 ceph-mon[81815]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T15:26:59.731 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:26:59 smithi196 ceph-mon[81815]: pgmap v140: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 2 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:26:59.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:26:59 smithi096 ceph-mon[112443]: pgmap v140: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 2 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:27:00.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:00 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 56/330 objects degraded (16.970%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:00.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:00.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:00 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 56/330 objects degraded (16.970%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:01.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:01 smithi096 ceph-mon[112443]: pgmap v141: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:27:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:01 smithi196 ceph-mon[81815]: pgmap v141: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:27:02.753 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:02 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:02.753 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:02 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:02.753 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:02 smithi096 ceph-mon[112443]: pgmap v142: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:27:02.759 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:27:02 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T15:27:02.616+0000 7f751a2917c0 -1 osd.4 68 log_to_monitors true 2024-04-22T15:27:02.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:02 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:02.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:02 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:02.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:02 smithi196 ceph-mon[81815]: pgmap v142: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:27:03.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:03 smithi196 ceph-mon[81815]: from='osd.4 [v2:172.21.15.196:6808/2228895659,v1:172.21.15.196:6809/2228895659]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:27:03.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:03 smithi196 ceph-mon[81815]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:27:03.760 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 15:27:03 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T15:27:03.592+0000 7f750b6e9700 -1 osd.4 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:27:03.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:03 smithi096 ceph-mon[112443]: from='osd.4 [v2:172.21.15.196:6808/2228895659,v1:172.21.15.196:6809/2228895659]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:27:03.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:03 smithi096 ceph-mon[112443]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:27:04.907 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:04 smithi196 ceph-mon[81815]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T15:27:04.907 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:04 smithi196 ceph-mon[81815]: osdmap e71: 6 total, 5 up, 6 in 2024-04-22T15:27:04.907 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:04 smithi196 ceph-mon[81815]: from='osd.4 [v2:172.21.15.196:6808/2228895659,v1:172.21.15.196:6809/2228895659]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:27:04.907 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:04 smithi196 ceph-mon[81815]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:27:04.908 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:04 smithi196 ceph-mon[81815]: pgmap v144: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:27:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:04 smithi096 ceph-mon[112443]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T15:27:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:04 smithi096 ceph-mon[112443]: osdmap e71: 6 total, 5 up, 6 in 2024-04-22T15:27:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:04 smithi096 ceph-mon[112443]: from='osd.4 [v2:172.21.15.196:6808/2228895659,v1:172.21.15.196:6809/2228895659]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:27:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:04 smithi096 ceph-mon[112443]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:27:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:04 smithi096 ceph-mon[112443]: pgmap v144: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 56/330 objects degraded (16.970%) 2024-04-22T15:27:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:27:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: osd.4 [v2:172.21.15.196:6808/2228895659,v1:172.21.15.196:6809/2228895659] boot 2024-04-22T15:27:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: osdmap e72: 6 total, 6 up, 6 in 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:27:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:27:05.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:05 smithi096 ceph-mon[112443]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: osd.4 [v2:172.21.15.196:6808/2228895659,v1:172.21.15.196:6809/2228895659] boot 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: osdmap e72: 6 total, 6 up, 6 in 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:06.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:06.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:06.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:27:06.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:27:06.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:05 smithi196 ceph-mon[81815]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-22T15:27:06.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:06 smithi096 ceph-mon[112443]: osdmap e73: 6 total, 6 up, 6 in 2024-04-22T15:27:06.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:06 smithi096 ceph-mon[112443]: pgmap v147: 65 pgs: 16 peering, 7 active+undersized+degraded, 11 active+undersized, 31 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 20/330 objects degraded (6.061%) 2024-04-22T15:27:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:06 smithi196 ceph-mon[81815]: osdmap e73: 6 total, 6 up, 6 in 2024-04-22T15:27:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:06 smithi196 ceph-mon[81815]: pgmap v147: 65 pgs: 16 peering, 7 active+undersized+degraded, 11 active+undersized, 31 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 20/330 objects degraded (6.061%) 2024-04-22T15:27:07.911 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:07 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 20/330 objects degraded (6.061%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:07 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 20/330 objects degraded (6.061%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:09.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:09 smithi196 ceph-mon[81815]: pgmap v148: 65 pgs: 16 peering, 3 active+undersized+degraded, 7 active+undersized, 39 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 10/330 objects degraded (3.030%) 2024-04-22T15:27:09.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:09.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:27:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:09 smithi096 ceph-mon[112443]: pgmap v148: 65 pgs: 16 peering, 3 active+undersized+degraded, 7 active+undersized, 39 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 10/330 objects degraded (3.030%) 2024-04-22T15:27:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:27:10.375 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:10 smithi196 ceph-mon[81815]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/330 objects degraded (3.030%), 3 pgs degraded) 2024-04-22T15:27:10.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:10 smithi096 ceph-mon[112443]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/330 objects degraded (3.030%), 3 pgs degraded) 2024-04-22T15:27:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:11 smithi096 ceph-mon[112443]: pgmap v149: 65 pgs: 6 peering, 59 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-22T15:27:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:11 smithi196 ceph-mon[81815]: pgmap v149: 65 pgs: 6 peering, 59 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-22T15:27:13.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:13 smithi096 ceph-mon[112443]: pgmap v150: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:27:13.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.522+0000 7f74d0428700 1 -- 172.21.15.96:0/1181632247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 msgr2=0x7f74c8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:13.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.522+0000 7f74d0428700 1 --2- 172.21.15.96:0/1181632247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 0x7f74c8101200 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f74c000b0a0 tx=0x7f74c000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.522+0000 7f74d0428700 1 -- 172.21.15.96:0/1181632247 shutdown_connections 2024-04-22T15:27:13.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.522+0000 7f74d0428700 1 --2- 172.21.15.96:0/1181632247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74c81017d0 0x7f74c8105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.522+0000 7f74d0428700 1 --2- 172.21.15.96:0/1181632247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 0x7f74c8101200 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.522+0000 7f74d0428700 1 -- 172.21.15.96:0/1181632247 >> 172.21.15.96:0/1181632247 conn(0x7f74c80fc6d0 msgr2=0x7f74c80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.523+0000 7f74d0428700 1 -- 172.21.15.96:0/1181632247 shutdown_connections 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.523+0000 7f74d0428700 1 -- 172.21.15.96:0/1181632247 wait complete. 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.523+0000 7f74d0428700 1 Processor -- start 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.523+0000 7f74d0428700 1 -- start start 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.523+0000 7f74d0428700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 0x7f74c80731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.523+0000 7f74d0428700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74c81017d0 0x7f74c8073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.523+0000 7f74d0428700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8073c60 con 0x7f74c8100e20 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.523+0000 7f74d0428700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8073da0 con 0x7f74c81017d0 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74ce1c4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 0x7f74c80731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74ce1c4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 0x7f74c80731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50250/0 (socket says 172.21.15.96:50250) 2024-04-22T15:27:13.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74ce1c4700 1 -- 172.21.15.96:0/4053604952 learned_addr learned my addr 172.21.15.96:0/4053604952 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74cd9c3700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74c81017d0 0x7f74c8073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74ce1c4700 1 -- 172.21.15.96:0/4053604952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74c81017d0 msgr2=0x7f74c8073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74ce1c4700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74c81017d0 0x7f74c8073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74ce1c4700 1 -- 172.21.15.96:0/4053604952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c0009d00 con 0x7f74c8100e20 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74ce1c4700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 0x7f74c80731e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f74c000b070 tx=0x7f74c0007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74c77fe700 1 -- 172.21.15.96:0/4053604952 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c000e040 con 0x7f74c8100e20 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74c77fe700 1 -- 172.21.15.96:0/4053604952 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74c00096e0 con 0x7f74c8100e20 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c8074020 con 0x7f74c8100e20 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74c77fe700 1 -- 172.21.15.96:0/4053604952 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c001d6b0 con 0x7f74c8100e20 2024-04-22T15:27:13.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.524+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c81a2fd0 con 0x7f74c8100e20 2024-04-22T15:27:13.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.525+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c81091c0 con 0x7f74c8100e20 2024-04-22T15:27:13.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.526+0000 7f74c77fe700 1 -- 172.21.15.96:0/4053604952 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f74c0003c10 con 0x7f74c8100e20 2024-04-22T15:27:13.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.526+0000 7f74c77fe700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74b006ecd0 0x7f74b0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:13.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.526+0000 7f74c77fe700 1 -- 172.21.15.96:0/4053604952 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f74c0090490 con 0x7f74c8100e20 2024-04-22T15:27:13.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.526+0000 7f74cd9c3700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74b006ecd0 0x7f74b0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:13.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.527+0000 7f74cd9c3700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74b006ecd0 0x7f74b0071190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f74c8074910 tx=0x7f74bc00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:13.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.531+0000 7f74c77fe700 1 -- 172.21.15.96:0/4053604952 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f74c00592a0 con 0x7f74c8100e20 2024-04-22T15:27:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:13 smithi196 ceph-mon[81815]: pgmap v150: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:27:13.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.757+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f74c810ddf0 con 0x7f74b006ecd0 2024-04-22T15:27:13.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.758+0000 7f74c77fe700 1 -- 172.21.15.96:0/4053604952 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f74c810ddf0 con 0x7f74b006ecd0 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.760+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74b006ecd0 msgr2=0x7f74b0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.760+0000 7f74d0428700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74b006ecd0 0x7f74b0071190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f74c8074910 tx=0x7f74bc00b410 comp rx=0 tx=0).stop 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.760+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 msgr2=0x7f74c80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.760+0000 7f74d0428700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 0x7f74c80731e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f74c000b070 tx=0x7f74c0007b20 comp rx=0 tx=0).stop 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.760+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 shutdown_connections 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.760+0000 7f74d0428700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74b006ecd0 0x7f74b0071190 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.760+0000 7f74d0428700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74c8100e20 0x7f74c80731e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.760+0000 7f74d0428700 1 --2- 172.21.15.96:0/4053604952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74c81017d0 0x7f74c8073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.760+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 >> 172.21.15.96:0/4053604952 conn(0x7f74c80fc6d0 msgr2=0x7f74c80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.761+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 shutdown_connections 2024-04-22T15:27:13.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.761+0000 7f74d0428700 1 -- 172.21.15.96:0/4053604952 wait complete. 2024-04-22T15:27:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:27:13.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.865+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/4052282058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc0ffab0 msgr2=0x7f8adc0fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.865+0000 7f8ae3a6d700 1 --2- 172.21.15.96:0/4052282058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc0ffab0 0x7f8adc0fff30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009910 tx=0x7f8acc009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.865+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/4052282058 shutdown_connections 2024-04-22T15:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.865+0000 7f8ae3a6d700 1 --2- 172.21.15.96:0/4052282058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc0ffab0 0x7f8adc0fff30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.865+0000 7f8ae3a6d700 1 --2- 172.21.15.96:0/4052282058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8adc105ab0 0x7f8adc105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.865+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/4052282058 >> 172.21.15.96:0/4052282058 conn(0x7f8adc0fb5e0 msgr2=0x7f8adc0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.865+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/4052282058 shutdown_connections 2024-04-22T15:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.865+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/4052282058 wait complete. 2024-04-22T15:27:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae3a6d700 1 Processor -- start 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae3a6d700 1 -- start start 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae3a6d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8adc0ffab0 0x7f8adc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae3a6d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc105ab0 0x7f8adc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae3a6d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc198560 con 0x7f8adc105ab0 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae3a6d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc1986a0 con 0x7f8adc0ffab0 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae1008700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc105ab0 0x7f8adc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae1008700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc105ab0 0x7f8adc197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50270/0 (socket says 172.21.15.96:50270) 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae1008700 1 -- 172.21.15.96:0/2084255661 learned_addr learned my addr 172.21.15.96:0/2084255661 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae1809700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8adc0ffab0 0x7f8adc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae1008700 1 -- 172.21.15.96:0/2084255661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8adc0ffab0 msgr2=0x7f8adc197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae1008700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8adc0ffab0 0x7f8adc197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae1008700 1 -- 172.21.15.96:0/2084255661 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc0095f0 con 0x7f8adc105ab0 2024-04-22T15:27:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.866+0000 7f8ae1809700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8adc0ffab0 0x7f8adc197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:27:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.867+0000 7f8ae1008700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc105ab0 0x7f8adc197f90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8acc006010 tx=0x7f8acc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.867+0000 7f8ad2ffd700 1 -- 172.21.15.96:0/2084255661 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8acc01d070 con 0x7f8adc105ab0 2024-04-22T15:27:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.867+0000 7f8ad2ffd700 1 -- 172.21.15.96:0/2084255661 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8acc00f460 con 0x7f8adc105ab0 2024-04-22T15:27:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.867+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8adc1a0ac0 con 0x7f8adc105ab0 2024-04-22T15:27:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.867+0000 7f8ad2ffd700 1 -- 172.21.15.96:0/2084255661 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8acc017630 con 0x7f8adc105ab0 2024-04-22T15:27:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.867+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc1a0e20 con 0x7f8adc105ab0 2024-04-22T15:27:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.868+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc1080d0 con 0x7f8adc105ab0 2024-04-22T15:27:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.868+0000 7f8ad2ffd700 1 -- 172.21.15.96:0/2084255661 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8acc00f5d0 con 0x7f8adc105ab0 2024-04-22T15:27:13.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.869+0000 7f8ad2ffd700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ac806ecc0 0x7f8ac8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:13.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.869+0000 7f8ad2ffd700 1 -- 172.21.15.96:0/2084255661 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f8acc0902f0 con 0x7f8adc105ab0 2024-04-22T15:27:13.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.869+0000 7f8ae1809700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ac806ecc0 0x7f8ac8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:13.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.869+0000 7f8ae1809700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ac806ecc0 0x7f8ac8071180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8007900 tx=0x7f8ad8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:13.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:13.873+0000 7f8ad2ffd700 1 -- 172.21.15.96:0/2084255661 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8acc0591e0 con 0x7f8adc105ab0 2024-04-22T15:27:14.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.091+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8adc10cd30 con 0x7f8ac806ecc0 2024-04-22T15:27:14.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.092+0000 7f8ad2ffd700 1 -- 172.21.15.96:0/2084255661 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f8adc10cd30 con 0x7f8ac806ecc0 2024-04-22T15:27:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ac806ecc0 msgr2=0x7f8ac8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ac806ecc0 0x7f8ac8071180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8007900 tx=0x7f8ad8008040 comp rx=0 tx=0).stop 2024-04-22T15:27:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc105ab0 msgr2=0x7f8adc197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc105ab0 0x7f8adc197f90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8acc006010 tx=0x7f8acc00be20 comp rx=0 tx=0).stop 2024-04-22T15:27:14.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 shutdown_connections 2024-04-22T15:27:14.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ac806ecc0 0x7f8ac8071180 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8adc0ffab0 0x7f8adc197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 --2- 172.21.15.96:0/2084255661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8adc105ab0 0x7f8adc197f90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 >> 172.21.15.96:0/2084255661 conn(0x7f8adc0fb5e0 msgr2=0x7f8adc0fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:14.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 shutdown_connections 2024-04-22T15:27:14.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.095+0000 7f8ae3a6d700 1 -- 172.21.15.96:0/2084255661 wait complete. 2024-04-22T15:27:14.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.198+0000 7f4125b8e700 1 -- 172.21.15.96:0/52540852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41200ffaa0 msgr2=0x7f41200fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.198+0000 7f4125b8e700 1 --2- 172.21.15.96:0/52540852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41200ffaa0 0x7f41200fff20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4110009910 tx=0x7f4110009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:14.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.198+0000 7f4125b8e700 1 -- 172.21.15.96:0/52540852 shutdown_connections 2024-04-22T15:27:14.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.198+0000 7f4125b8e700 1 --2- 172.21.15.96:0/52540852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41200ffaa0 0x7f41200fff20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.198+0000 7f4125b8e700 1 --2- 172.21.15.96:0/52540852 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4120105aa0 0x7f4120105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.198+0000 7f4125b8e700 1 -- 172.21.15.96:0/52540852 >> 172.21.15.96:0/52540852 conn(0x7f4120076740 msgr2=0x7f41200fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:14.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.198+0000 7f4125b8e700 1 -- 172.21.15.96:0/52540852 shutdown_connections 2024-04-22T15:27:14.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.198+0000 7f4125b8e700 1 -- 172.21.15.96:0/52540852 wait complete. 2024-04-22T15:27:14.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f4125b8e700 1 Processor -- start 2024-04-22T15:27:14.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f4125b8e700 1 -- start start 2024-04-22T15:27:14.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f4125b8e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41200ffaa0 0x7f412019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:14.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f4125b8e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4120105aa0 0x7f412019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f4125b8e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41201968d0 con 0x7f4120105aa0 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f4125b8e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4120196a40 con 0x7f41200ffaa0 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f411effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4120105aa0 0x7f412019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f411effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4120105aa0 0x7f412019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50296/0 (socket says 172.21.15.96:50296) 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f411effd700 1 -- 172.21.15.96:0/3814825415 learned_addr learned my addr 172.21.15.96:0/3814825415 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f411f7fe700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41200ffaa0 0x7f412019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.199+0000 7f411effd700 1 -- 172.21.15.96:0/3814825415 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41200ffaa0 msgr2=0x7f412019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.200+0000 7f411effd700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41200ffaa0 0x7f412019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.200+0000 7f411effd700 1 -- 172.21.15.96:0/3814825415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41100095f0 con 0x7f4120105aa0 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.200+0000 7f411effd700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4120105aa0 0x7f412019cd60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4110006010 tx=0x7f411000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.200+0000 7f411cff9700 1 -- 172.21.15.96:0/3814825415 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f411001d070 con 0x7f4120105aa0 2024-04-22T15:27:14.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.200+0000 7f411cff9700 1 -- 172.21.15.96:0/3814825415 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f411000f460 con 0x7f4120105aa0 2024-04-22T15:27:14.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.200+0000 7f411cff9700 1 -- 172.21.15.96:0/3814825415 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4110017630 con 0x7f4120105aa0 2024-04-22T15:27:14.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.200+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4120196cc0 con 0x7f4120105aa0 2024-04-22T15:27:14.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.200+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41201971b0 con 0x7f4120105aa0 2024-04-22T15:27:14.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.201+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41201011e0 con 0x7f4120105aa0 2024-04-22T15:27:14.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.201+0000 7f411cff9700 1 -- 172.21.15.96:0/3814825415 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f411000f5d0 con 0x7f4120105aa0 2024-04-22T15:27:14.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.202+0000 7f411cff9700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f410806ecd0 0x7f4108071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:14.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.202+0000 7f411cff9700 1 -- 172.21.15.96:0/3814825415 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f4110090260 con 0x7f4120105aa0 2024-04-22T15:27:14.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.202+0000 7f411f7fe700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f410806ecd0 0x7f4108071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:14.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.202+0000 7f411f7fe700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f410806ecd0 0x7f4108071190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4114005950 tx=0x7f411400a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:14.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.207+0000 7f411cff9700 1 -- 172.21.15.96:0/3814825415 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4110059130 con 0x7f4120105aa0 2024-04-22T15:27:14.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.416+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4120069660 con 0x7f410806ecd0 2024-04-22T15:27:14.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.427+0000 7f411cff9700 1 -- 172.21.15.96:0/3814825415 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4120069660 con 0x7f410806ecd0 2024-04-22T15:27:14.430 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:27:14.430 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (5m) 81s ago 17m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:27:14.430 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (17m) 81s ago 17m 8870k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:27:14.430 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (16m) 14s ago 16m 8581k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:27:14.431 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (3m) 81s ago 17m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:27:14.431 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (3m) 14s ago 16m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:27:14.432 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (5m) 81s ago 16m 80.7M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:27:14.432 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (11m) 81s ago 11m 155M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:27:14.432 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (11m) 81s ago 11m 90.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:27:14.432 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (11m) 14s ago 11m 26.8M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:27:14.432 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (11m) 14s ago 11m 26.6M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:27:14.432 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (8m) 81s ago 18m 505M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:27:14.432 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (7m) 14s ago 16m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:27:14.432 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (4m) 81s ago 18m 46.9M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:27:14.432 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (3m) 14s ago 16m 38.2M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:27:14.433 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (6m) 81s ago 17m 17.3M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:27:14.433 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (6m) 14s ago 16m 16.6M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:27:14.433 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (2m) 81s ago 15m 92.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:27:14.433 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (2m) 81s ago 14m 93.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:27:14.433 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (87s) 81s ago 14m 11.9M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:27:14.433 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (51s) 14s ago 13m 97.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:27:14.434 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (17s) 14s ago 13m 12.6M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:27:14.434 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (13m) 14s ago 13m 196M 2240M 18.2.2-1237-g646b8c7d 9900ed1a6b81 628bbf81d595 2024-04-22T15:27:14.434 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (6m) 81s ago 16m 77.0M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:27:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.429+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f410806ecd0 msgr2=0x7f4108071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.429+0000 7f4125b8e700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f410806ecd0 0x7f4108071190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4114005950 tx=0x7f411400a300 comp rx=0 tx=0).stop 2024-04-22T15:27:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.429+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4120105aa0 msgr2=0x7f412019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.429+0000 7f4125b8e700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4120105aa0 0x7f412019cd60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4110006010 tx=0x7f411000bd70 comp rx=0 tx=0).stop 2024-04-22T15:27:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.430+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 shutdown_connections 2024-04-22T15:27:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.430+0000 7f4125b8e700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f410806ecd0 0x7f4108071190 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.430+0000 7f4125b8e700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41200ffaa0 0x7f412019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.430+0000 7f4125b8e700 1 --2- 172.21.15.96:0/3814825415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4120105aa0 0x7f412019cd60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.430+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 >> 172.21.15.96:0/3814825415 conn(0x7f4120076740 msgr2=0x7f41200687a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:14.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.430+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 shutdown_connections 2024-04-22T15:27:14.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.430+0000 7f4125b8e700 1 -- 172.21.15.96:0/3814825415 wait complete. 2024-04-22T15:27:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.533+0000 7f290cdca700 1 -- 172.21.15.96:0/3940695114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ff3b0 msgr2=0x7f29080ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.533+0000 7f290cdca700 1 --2- 172.21.15.96:0/3940695114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ff3b0 0x7f29080ff790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f28f8009910 tx=0x7f28f8009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.533+0000 7f290cdca700 1 -- 172.21.15.96:0/3940695114 shutdown_connections 2024-04-22T15:27:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.533+0000 7f290cdca700 1 --2- 172.21.15.96:0/3940695114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29080ffd60 0x7f29081058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.533+0000 7f290cdca700 1 --2- 172.21.15.96:0/3940695114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ff3b0 0x7f29080ff790 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.533+0000 7f290cdca700 1 -- 172.21.15.96:0/3940695114 >> 172.21.15.96:0/3940695114 conn(0x7f2908075f00 msgr2=0x7f29080fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.533+0000 7f290cdca700 1 -- 172.21.15.96:0/3940695114 shutdown_connections 2024-04-22T15:27:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.533+0000 7f290cdca700 1 -- 172.21.15.96:0/3940695114 wait complete. 2024-04-22T15:27:14.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.534+0000 7f290cdca700 1 Processor -- start 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.534+0000 7f290cdca700 1 -- start start 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.534+0000 7f290cdca700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29080ff3b0 0x7f2908100330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.534+0000 7f290cdca700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ffd60 0x7f2908100870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.534+0000 7f290cdca700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2908100db0 con 0x7f29080ffd60 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.534+0000 7f290cdca700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2908100ef0 con 0x7f29080ff3b0 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f2905d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ffd60 0x7f2908100870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f2905d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ffd60 0x7f2908100870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50316/0 (socket says 172.21.15.96:50316) 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f2905d9b700 1 -- 172.21.15.96:0/511840497 learned_addr learned my addr 172.21.15.96:0/511840497 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f290659c700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29080ff3b0 0x7f2908100330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f2905d9b700 1 -- 172.21.15.96:0/511840497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29080ff3b0 msgr2=0x7f2908100330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f2905d9b700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29080ff3b0 0x7f2908100330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f2905d9b700 1 -- 172.21.15.96:0/511840497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f80095f0 con 0x7f29080ffd60 2024-04-22T15:27:14.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f2905d9b700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ffd60 0x7f2908100870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00d900 tx=0x7f28fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:14.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f28f77fe700 1 -- 172.21.15.96:0/511840497 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28fc009900 con 0x7f29080ffd60 2024-04-22T15:27:14.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f28f77fe700 1 -- 172.21.15.96:0/511840497 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28fc010460 con 0x7f29080ffd60 2024-04-22T15:27:14.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29081a2d30 con 0x7f29080ffd60 2024-04-22T15:27:14.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f28f77fe700 1 -- 172.21.15.96:0/511840497 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28fc00f5f0 con 0x7f29080ffd60 2024-04-22T15:27:14.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.535+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29081a3090 con 0x7f29080ffd60 2024-04-22T15:27:14.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.536+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2908109240 con 0x7f29080ffd60 2024-04-22T15:27:14.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.537+0000 7f28f77fe700 1 -- 172.21.15.96:0/511840497 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f28fc0105d0 con 0x7f29080ffd60 2024-04-22T15:27:14.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.537+0000 7f28f77fe700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28f006ecc0 0x7f28f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:14.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.537+0000 7f290659c700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28f006ecc0 0x7f28f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:14.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.537+0000 7f28f77fe700 1 -- 172.21.15.96:0/511840497 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f28fc08e8c0 con 0x7f29080ffd60 2024-04-22T15:27:14.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.538+0000 7f290659c700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28f006ecc0 0x7f28f0071180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f28f8000c00 tx=0x7f28f801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:14.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.542+0000 7f28f77fe700 1 -- 172.21.15.96:0/511840497 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f28fc057750 con 0x7f29080ffd60 2024-04-22T15:27:14.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.822+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2908102440 con 0x7f29080ffd60 2024-04-22T15:27:14.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.823+0000 7f28f77fe700 1 -- 172.21.15.96:0/511840497 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+836 (secure 0 0 0) 0x7f28fc05ae10 con 0x7f29080ffd60 2024-04-22T15:27:14.826 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 5 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 4 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:27:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 5, 2024-04-22T15:27:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 9 2024-04-22T15:27:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:27:14.828 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:27:14.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.825+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28f006ecc0 msgr2=0x7f28f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.825+0000 7f290cdca700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28f006ecc0 0x7f28f0071180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f28f8000c00 tx=0x7f28f801a040 comp rx=0 tx=0).stop 2024-04-22T15:27:14.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.825+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ffd60 msgr2=0x7f2908100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.825+0000 7f290cdca700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ffd60 0x7f2908100870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00d900 tx=0x7f28fc00dc10 comp rx=0 tx=0).stop 2024-04-22T15:27:14.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.825+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 shutdown_connections 2024-04-22T15:27:14.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.825+0000 7f290cdca700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28f006ecc0 0x7f28f0071180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.825+0000 7f290cdca700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29080ff3b0 0x7f2908100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.826+0000 7f290cdca700 1 --2- 172.21.15.96:0/511840497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29080ffd60 0x7f2908100870 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.826+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 >> 172.21.15.96:0/511840497 conn(0x7f2908075f00 msgr2=0x7f2908104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:14.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.826+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 shutdown_connections 2024-04-22T15:27:14.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.826+0000 7f290cdca700 1 -- 172.21.15.96:0/511840497 wait complete. 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.930+0000 7f4645c6d700 1 -- 172.21.15.96:0/3285921234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4640101db0 msgr2=0x7f4640102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.930+0000 7f4645c6d700 1 --2- 172.21.15.96:0/3285921234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4640101db0 0x7f4640102190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4630009910 tx=0x7f4630009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.930+0000 7f4645c6d700 1 -- 172.21.15.96:0/3285921234 shutdown_connections 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.930+0000 7f4645c6d700 1 --2- 172.21.15.96:0/3285921234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46401026d0 0x7f464010abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.930+0000 7f4645c6d700 1 --2- 172.21.15.96:0/3285921234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4640101db0 0x7f4640102190 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.930+0000 7f4645c6d700 1 -- 172.21.15.96:0/3285921234 >> 172.21.15.96:0/3285921234 conn(0x7f46400fb640 msgr2=0x7f46400fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.930+0000 7f4645c6d700 1 -- 172.21.15.96:0/3285921234 shutdown_connections 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.930+0000 7f4645c6d700 1 -- 172.21.15.96:0/3285921234 wait complete. 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.930+0000 7f4645c6d700 1 Processor -- start 2024-04-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f4645c6d700 1 -- start start 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f4645c6d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4640101db0 0x7f464019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f4645c6d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46401026d0 0x7f464019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f4645c6d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464019d340 con 0x7f46401026d0 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f4645c6d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46401969b0 con 0x7f4640101db0 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f463effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46401026d0 0x7f464019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f463effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46401026d0 0x7f464019cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50342/0 (socket says 172.21.15.96:50342) 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f463effd700 1 -- 172.21.15.96:0/993845547 learned_addr learned my addr 172.21.15.96:0/993845547 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f463f7fe700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4640101db0 0x7f464019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f463effd700 1 -- 172.21.15.96:0/993845547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4640101db0 msgr2=0x7f464019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f463effd700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4640101db0 0x7f464019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f463effd700 1 -- 172.21.15.96:0/993845547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300095f0 con 0x7f46401026d0 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.931+0000 7f463effd700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46401026d0 0x7f464019cdb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f463400c8f0 tx=0x7f463400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.932+0000 7f463cff9700 1 -- 172.21.15.96:0/993845547 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46340078a0 con 0x7f46401026d0 2024-04-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.932+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4640196c90 con 0x7f46401026d0 2024-04-22T15:27:14.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.932+0000 7f463cff9700 1 -- 172.21.15.96:0/993845547 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f463400f450 con 0x7f46401026d0 2024-04-22T15:27:14.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.932+0000 7f463cff9700 1 -- 172.21.15.96:0/993845547 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4634018630 con 0x7f46401026d0 2024-04-22T15:27:14.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.932+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46401971e0 con 0x7f46401026d0 2024-04-22T15:27:14.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.933+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46401054d0 con 0x7f46401026d0 2024-04-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.933+0000 7f463cff9700 1 -- 172.21.15.96:0/993845547 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4634007a00 con 0x7f46401026d0 2024-04-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.933+0000 7f463cff9700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f462806ecc0 0x7f4628071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.934+0000 7f463f7fe700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f462806ecc0 0x7f4628071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.934+0000 7f463cff9700 1 -- 172.21.15.96:0/993845547 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f463408ece0 con 0x7f46401026d0 2024-04-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.934+0000 7f463f7fe700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f462806ecc0 0x7f4628071180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4630009fd0 tx=0x7f463001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:14.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:14.938+0000 7f463cff9700 1 -- 172.21.15.96:0/993845547 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f463405b620 con 0x7f46401026d0 2024-04-22T15:27:15.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.182+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f46400699f0 con 0x7f46401026d0 2024-04-22T15:27:15.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.183+0000 7f463cff9700 1 -- 172.21.15.96:0/993845547 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1901 (secure 0 0 0) 0x7f463405b1b0 con 0x7f46401026d0 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:e20 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 20 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:15:53.847723+0000 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:27:15.187 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 45 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=14494,1=24257} 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:27:15.188 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{0:14494} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.96:6828/3534778299,v1:172.21.15.96:6829/3534778299] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:14504} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/231416323,v1:172.21.15.96:6827/231416323] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.196:6824/3248990491,v1:172.21.15.196:6825/3248990491] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:24267} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.196:6826/3238575297,v1:172.21.15.196:6827/3238575297] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:27:15.189 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:27:15.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f462806ecc0 msgr2=0x7f4628071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f462806ecc0 0x7f4628071180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4630009fd0 tx=0x7f463001a040 comp rx=0 tx=0).stop 2024-04-22T15:27:15.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46401026d0 msgr2=0x7f464019cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46401026d0 0x7f464019cdb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f463400c8f0 tx=0x7f463400cc00 comp rx=0 tx=0).stop 2024-04-22T15:27:15.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 shutdown_connections 2024-04-22T15:27:15.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f462806ecc0 0x7f4628071180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4640101db0 0x7f464019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 --2- 172.21.15.96:0/993845547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46401026d0 0x7f464019cdb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 >> 172.21.15.96:0/993845547 conn(0x7f46400fb640 msgr2=0x7f46401030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:15.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.186+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 shutdown_connections 2024-04-22T15:27:15.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.187+0000 7f4645c6d700 1 -- 172.21.15.96:0/993845547 wait complete. 2024-04-22T15:27:15.191 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 20 2024-04-22T15:27:15.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 -- 172.21.15.96:0/329558896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 msgr2=0x7fb3140ff050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 --2- 172.21.15.96:0/329558896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 0x7fb3140ff050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb308009910 tx=0x7fb308009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 -- 172.21.15.96:0/329558896 shutdown_connections 2024-04-22T15:27:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 --2- 172.21.15.96:0/329558896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb314101f20 0x7fb3140ff590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 --2- 172.21.15.96:0/329558896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 0x7fb3140ff050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 -- 172.21.15.96:0/329558896 >> 172.21.15.96:0/329558896 conn(0x7fb314075f40 msgr2=0x7fb3140fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 -- 172.21.15.96:0/329558896 shutdown_connections 2024-04-22T15:27:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 -- 172.21.15.96:0/329558896 wait complete. 2024-04-22T15:27:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 Processor -- start 2024-04-22T15:27:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 -- start start 2024-04-22T15:27:15.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.292+0000 7fb31af80700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 0x7fb314198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:15.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb31af80700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb314101f20 0x7fb314199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:15.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb31af80700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3141998f0 con 0x7fb314101b40 2024-04-22T15:27:15.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb31af80700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31419c560 con 0x7fb314101f20 2024-04-22T15:27:15.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb318d1c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 0x7fb314198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:15.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb318d1c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 0x7fb314198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50350/0 (socket says 172.21.15.96:50350) 2024-04-22T15:27:15.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb318d1c700 1 -- 172.21.15.96:0/2918404944 learned_addr learned my addr 172.21.15.96:0/2918404944 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:15.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb318d1c700 1 -- 172.21.15.96:0/2918404944 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb314101f20 msgr2=0x7fb314199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb313fff700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb314101f20 0x7fb314199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb318d1c700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb314101f20 0x7fb314199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb318d1c700 1 -- 172.21.15.96:0/2918404944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080095f0 con 0x7fb314101b40 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb318d1c700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 0x7fb314198d20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb30800bd70 tx=0x7fb30800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb311ffb700 1 -- 172.21.15.96:0/2918404944 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb30801d070 con 0x7fb314101b40 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb311ffb700 1 -- 172.21.15.96:0/2918404944 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3080056e0 con 0x7fb314101b40 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31419c7e0 con 0x7fb314101b40 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.293+0000 7fb311ffb700 1 -- 172.21.15.96:0/2918404944 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb308017400 con 0x7fb314101b40 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.294+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31419cd50 con 0x7fb314101b40 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.294+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31410af40 con 0x7fb314101b40 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.295+0000 7fb311ffb700 1 -- 172.21.15.96:0/2918404944 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb308003a60 con 0x7fb314101b40 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.295+0000 7fb311ffb700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb30406ecc0 0x7fb304071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.295+0000 7fb311ffb700 1 -- 172.21.15.96:0/2918404944 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fb3080902a0 con 0x7fb314101b40 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.295+0000 7fb313fff700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb30406ecc0 0x7fb304071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:15.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.296+0000 7fb313fff700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb30406ecc0 0x7fb304071180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb300005950 tx=0x7fb3000058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:15.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.300+0000 7fb311ffb700 1 -- 172.21.15.96:0/2918404944 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb308059050 con 0x7fb314101b40 2024-04-22T15:27:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:15 smithi096 ceph-mon[112443]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:15 smithi096 ceph-mon[112443]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:15 smithi096 ceph-mon[112443]: pgmap v151: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T15:27:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:15 smithi096 ceph-mon[112443]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:15 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/511840497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:15.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:15 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/993845547' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:27:15.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.520+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb314110050 con 0x7fb30406ecc0 2024-04-22T15:27:15.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.521+0000 7fb311ffb700 1 -- 172.21.15.96:0/2918404944 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fb314110050 con 0x7fb30406ecc0 2024-04-22T15:27:15.524 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:27:15.524 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:27:15.524 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:27:15.524 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:27:15.524 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:27:15.525 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:27:15.525 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:27:15.525 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr" 2024-04-22T15:27:15.525 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:27:15.525 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "11/23 daemons upgraded", 2024-04-22T15:27:15.525 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:27:15.525 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:27:15.525 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:27:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb30406ecc0 msgr2=0x7fb304071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb30406ecc0 0x7fb304071180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb300005950 tx=0x7fb3000058e0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 msgr2=0x7fb314198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 0x7fb314198d20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb30800bd70 tx=0x7fb30800be50 comp rx=0 tx=0).stop 2024-04-22T15:27:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 shutdown_connections 2024-04-22T15:27:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb30406ecc0 0x7fb304071180 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb314101b40 0x7fb314198d20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 --2- 172.21.15.96:0/2918404944 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb314101f20 0x7fb314199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 >> 172.21.15.96:0/2918404944 conn(0x7fb314075f40 msgr2=0x7fb3140fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 shutdown_connections 2024-04-22T15:27:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.523+0000 7fb31af80700 1 -- 172.21.15.96:0/2918404944 wait complete. 2024-04-22T15:27:15.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 -- 172.21.15.96:0/3944574176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c106d70 msgr2=0x7fdf3c107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 --2- 172.21.15.96:0/3944574176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c106d70 0x7fdf3c107150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009910 tx=0x7fdf2c009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:15.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 -- 172.21.15.96:0/3944574176 shutdown_connections 2024-04-22T15:27:15.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 --2- 172.21.15.96:0/3944574176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf3c0ff3d0 0x7fdf3c0ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 --2- 172.21.15.96:0/3944574176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c106d70 0x7fdf3c107150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 -- 172.21.15.96:0/3944574176 >> 172.21.15.96:0/3944574176 conn(0x7fdf3c074d90 msgr2=0x7fdf3c0751a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:15.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 -- 172.21.15.96:0/3944574176 shutdown_connections 2024-04-22T15:27:15.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 -- 172.21.15.96:0/3944574176 wait complete. 2024-04-22T15:27:15.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 Processor -- start 2024-04-22T15:27:15.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.627+0000 7fdf4152e700 1 -- start start 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf4152e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c0ff3d0 0x7fdf3c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf4152e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf3c106d70 0x7fdf3c19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf4152e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c19d4e0 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf4152e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c196ac0 con 0x7fdf3c106d70 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf3affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c0ff3d0 0x7fdf3c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf3affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c0ff3d0 0x7fdf3c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50370/0 (socket says 172.21.15.96:50370) 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf3affd700 1 -- 172.21.15.96:0/1861930943 learned_addr learned my addr 172.21.15.96:0/1861930943 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf327fc700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf3c106d70 0x7fdf3c19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf3affd700 1 -- 172.21.15.96:0/1861930943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf3c106d70 msgr2=0x7fdf3c19cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf3affd700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf3c106d70 0x7fdf3c19cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf3affd700 1 -- 172.21.15.96:0/1861930943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c0095f0 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf3affd700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c0ff3d0 0x7fdf3c19c980 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00bd70 tx=0x7fdf2c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:15.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf38ff9700 1 -- 172.21.15.96:0/1861930943 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf2c01d070 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c196d40 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf38ff9700 1 -- 172.21.15.96:0/1861930943 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf2c00f460 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf38ff9700 1 -- 172.21.15.96:0/1861930943 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf2c017630 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.628+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c197230 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.629+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3c1092c0 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.630+0000 7fdf38ff9700 1 -- 172.21.15.96:0/1861930943 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdf2c00f5d0 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.630+0000 7fdf38ff9700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf2806ecc0 0x7fdf28071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:15.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.630+0000 7fdf38ff9700 1 -- 172.21.15.96:0/1861930943 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fdf2c090320 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.630+0000 7fdf327fc700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf2806ecc0 0x7fdf28071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:15.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.631+0000 7fdf327fc700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf2806ecc0 0x7fdf28071180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdf24006fd0 tx=0x7fdf24008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:15.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.635+0000 7fdf38ff9700 1 -- 172.21.15.96:0/1861930943 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdf2c05cc60 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:15 smithi196 ceph-mon[81815]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:15.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:15 smithi196 ceph-mon[81815]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:15.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:15 smithi196 ceph-mon[81815]: pgmap v151: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T15:27:15.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:15 smithi196 ceph-mon[81815]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:15.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:15 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/511840497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:15.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:15 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/993845547' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:27:15.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.927+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdf3c197f80 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.927+0000 7fdf38ff9700 1 -- 172.21.15.96:0/1861930943 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fdf2c05c7f0 con 0x7fdf3c0ff3d0 2024-04-22T15:27:15.931 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:27:15.931 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:27:15.931 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.929+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf2806ecc0 msgr2=0x7fdf28071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.929+0000 7fdf4152e700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf2806ecc0 0x7fdf28071180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdf24006fd0 tx=0x7fdf24008040 comp rx=0 tx=0).stop 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.929+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c0ff3d0 msgr2=0x7fdf3c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.929+0000 7fdf4152e700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c0ff3d0 0x7fdf3c19c980 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00bd70 tx=0x7fdf2c00be50 comp rx=0 tx=0).stop 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.929+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 shutdown_connections 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.929+0000 7fdf4152e700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf2806ecc0 0x7fdf28071180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.929+0000 7fdf4152e700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf3c0ff3d0 0x7fdf3c19c980 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.929+0000 7fdf4152e700 1 --2- 172.21.15.96:0/1861930943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf3c106d70 0x7fdf3c19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.929+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 >> 172.21.15.96:0/1861930943 conn(0x7fdf3c074d90 msgr2=0x7fdf3c0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:15.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.930+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 shutdown_connections 2024-04-22T15:27:15.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:15.930+0000 7fdf4152e700 1 -- 172.21.15.96:0/1861930943 wait complete. 2024-04-22T15:27:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:16 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1861930943' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:27:16.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:16 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1861930943' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:27:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:17 smithi096 ceph-mon[112443]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:17 smithi096 ceph-mon[112443]: pgmap v152: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-22T15:27:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:17 smithi196 ceph-mon[81815]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:17 smithi196 ceph-mon[81815]: pgmap v152: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-22T15:27:19.602 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:19 smithi196 ceph-mon[81815]: pgmap v153: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:27:19.602 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:19 smithi096 ceph-mon[112443]: pgmap v153: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:27:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:20.488 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:20 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:27:20.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:20 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:27:21.613 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:21 smithi196 ceph-mon[81815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:27:21.613 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:21 smithi196 ceph-mon[81815]: Upgrade: osd.5 is safe to restart 2024-04-22T15:27:21.613 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:21 smithi196 ceph-mon[81815]: pgmap v154: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:27:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:21 smithi096 ceph-mon[112443]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:27:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:21 smithi096 ceph-mon[112443]: Upgrade: osd.5 is safe to restart 2024-04-22T15:27:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:21 smithi096 ceph-mon[112443]: pgmap v154: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:27:22.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:22 smithi196 ceph-mon[81815]: Upgrade: Updating osd.5 2024-04-22T15:27:22.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:22 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:22.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:22 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:27:22.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:22 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:22.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:22 smithi196 ceph-mon[81815]: Deploying daemon osd.5 on smithi196 2024-04-22T15:27:22.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:22 smithi196 ceph-mon[81815]: pgmap v155: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:27:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:22 smithi096 ceph-mon[112443]: Upgrade: Updating osd.5 2024-04-22T15:27:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:22 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:22 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:27:22.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:22 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:22.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:22 smithi096 ceph-mon[112443]: Deploying daemon osd.5 on smithi196 2024-04-22T15:27:22.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:22 smithi096 ceph-mon[112443]: pgmap v155: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T15:27:24.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:24.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:24 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:27:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:24 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:27:25.162 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:24 smithi196 systemd[1]: Stopping Ceph osd.5 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:27:25.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:25 smithi196 ceph-mon[81815]: pgmap v156: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:27:25.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:25 smithi196 ceph-mon[81815]: osd.5 marked itself down and dead 2024-04-22T15:27:25.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:25 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[54127]: 2024-04-22T15:27:25.168+0000 7f0fbd73d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:27:25.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:25 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[54127]: 2024-04-22T15:27:25.168+0000 7f0fbd73d700 -1 osd.5 73 *** Got signal Terminated *** 2024-04-22T15:27:25.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:25 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[54127]: 2024-04-22T15:27:25.168+0000 7f0fbd73d700 -1 osd.5 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:27:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:25 smithi096 ceph-mon[112443]: pgmap v156: 65 pgs: 65 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:27:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:25 smithi096 ceph-mon[112443]: osd.5 marked itself down and dead 2024-04-22T15:27:25.948 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:25 smithi196 podman[101307]: 2024-04-22 15:27:25.660130343 +0000 UTC m=+0.599635316 container died 628bbf81d595398d6d9d7d6f70e706104aee44245bd2a5f3e6d8785ec5138406 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5, RELEASE=reef-646b8c7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T15:27:26.260 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:26 smithi196 podman[101307]: 2024-04-22 15:27:26.036469045 +0000 UTC m=+0.975974025 container cleanup 628bbf81d595398d6d9d7d6f70e706104aee44245bd2a5f3e6d8785ec5138406 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-646b8c7, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T15:27:26.260 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:26 smithi196 bash[101307]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5 2024-04-22T15:27:26.477 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:26 smithi096 ceph-mon[112443]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:27:26.477 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:26 smithi096 ceph-mon[112443]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-22T15:27:26.477 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:26 smithi096 ceph-mon[112443]: osdmap e74: 6 total, 5 up, 6 in 2024-04-22T15:27:26.657 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:26 smithi196 ceph-mon[81815]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:27:26.657 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:26 smithi196 ceph-mon[81815]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-22T15:27:26.657 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:26 smithi196 ceph-mon[81815]: osdmap e74: 6 total, 5 up, 6 in 2024-04-22T15:27:26.657 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:26 smithi196 podman[101334]: 2024-04-22 15:27:26.304396617 +0000 UTC m=+0.635133310 container remove 628bbf81d595398d6d9d7d6f70e706104aee44245bd2a5f3e6d8785ec5138406 (image=quay.ceph.io/ceph-ci/ceph@sha256:417b28a7fda24d3105eeb9bae56a845b9ffa154499a85601880a5fd035847210, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=reef-646b8c7, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:27:26.935 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:26 smithi196 podman[101422]: 2024-04-22 15:27:26.557547783 +0000 UTC m=+0.017873276 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:27:26.936 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:26 smithi196 podman[101422]: 2024-04-22 15:27:26.688704942 +0000 UTC m=+0.149030410 container create 149e1809d3e7b8675ed59d079beae2e363347d1bbe2675a00d2deffd0b6fb42c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-22T15:27:26.936 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:26 smithi196 podman[101422]: 2024-04-22 15:27:26.933237206 +0000 UTC m=+0.393562688 container init 149e1809d3e7b8675ed59d079beae2e363347d1bbe2675a00d2deffd0b6fb42c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T15:27:27.228 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:26 smithi196 podman[101422]: 2024-04-22 15:27:26.936895477 +0000 UTC m=+0.397220956 container start 149e1809d3e7b8675ed59d079beae2e363347d1bbe2675a00d2deffd0b6fb42c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-22T15:27:27.228 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:27 smithi196 podman[101422]: 2024-04-22 15:27:27.00766463 +0000 UTC m=+0.467990116 container attach 149e1809d3e7b8675ed59d079beae2e363347d1bbe2675a00d2deffd0b6fb42c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:27:27.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:27 smithi196 ceph-mon[81815]: pgmap v158: 65 pgs: 3 peering, 11 stale+active+clean, 51 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T15:27:27.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:27 smithi196 ceph-mon[81815]: osdmap e75: 6 total, 5 up, 6 in 2024-04-22T15:27:27.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:27 smithi196 ceph-mon[81815]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:27:27.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:27 smithi196 podman[101422]: 2024-04-22 15:27:27.2268318 +0000 UTC m=+0.687157275 container died 149e1809d3e7b8675ed59d079beae2e363347d1bbe2675a00d2deffd0b6fb42c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:27:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:27 smithi096 ceph-mon[112443]: pgmap v158: 65 pgs: 3 peering, 11 stale+active+clean, 51 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T15:27:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:27 smithi096 ceph-mon[112443]: osdmap e75: 6 total, 5 up, 6 in 2024-04-22T15:27:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:27 smithi096 ceph-mon[112443]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:27:28.009 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:27 smithi196 podman[101449]: 2024-04-22 15:27:27.861596708 +0000 UTC m=+0.629214756 container remove 149e1809d3e7b8675ed59d079beae2e363347d1bbe2675a00d2deffd0b6fb42c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-04-22T15:27:28.010 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:27 smithi196 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-22T15:27:28.010 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:27 smithi196 systemd[1]: Stopped Ceph osd.5 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:27:28.010 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:27 smithi196 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.5.service: Consumed 54.749s CPU time. 2024-04-22T15:27:28.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:28 smithi196 ceph-mon[81815]: Health check failed: Degraded data redundancy: 13/330 objects degraded (3.939%), 5 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:28.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:28 smithi196 systemd[1]: Starting Ceph osd.5 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T15:27:28.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:28 smithi196 podman[101551]: 2024-04-22 15:27:28.275472478 +0000 UTC m=+0.018407346 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:27:28.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:28 smithi196 podman[101551]: 2024-04-22 15:27:28.395883841 +0000 UTC m=+0.138818681 container create 6537d137e8966dce3f0ddde383903338702efc6adf04d50b0ca713d252c5a40e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:27:28.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:28 smithi096 ceph-mon[112443]: Health check failed: Degraded data redundancy: 13/330 objects degraded (3.939%), 5 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:29.009 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:28 smithi196 podman[101551]: 2024-04-22 15:27:28.656458832 +0000 UTC m=+0.399393672 container init 6537d137e8966dce3f0ddde383903338702efc6adf04d50b0ca713d252c5a40e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T15:27:29.010 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:28 smithi196 podman[101551]: 2024-04-22 15:27:28.6595149 +0000 UTC m=+0.402449742 container start 6537d137e8966dce3f0ddde383903338702efc6adf04d50b0ca713d252c5a40e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-22T15:27:29.010 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:28 smithi196 podman[101551]: 2024-04-22 15:27:28.739859943 +0000 UTC m=+0.482794804 container attach 6537d137e8966dce3f0ddde383903338702efc6adf04d50b0ca713d252c5a40e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-22T15:27:29.462 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-mon[81815]: pgmap v160: 65 pgs: 5 active+undersized+degraded, 3 peering, 7 stale+active+clean, 3 active+undersized, 47 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 13/330 objects degraded (3.939%) 2024-04-22T15:27:29.462 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate[101567]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:27:29.462 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate[101567]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 bash[101551]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 bash[101551]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 bash[101551]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 bash[101551]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 bash[101551]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 bash[101551]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 bash[101551]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 bash[101551]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 bash[101551]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate[101567]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate[101567]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate[101567]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate[101567]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T15:27:29.463 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate[101567]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-22T15:27:29.464 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate[101567]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T15:27:29.464 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate[101567]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-22T15:27:29.464 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 podman[101551]: 2024-04-22 15:27:29.164081411 +0000 UTC m=+0.907016275 container died 6537d137e8966dce3f0ddde383903338702efc6adf04d50b0ca713d252c5a40e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-22T15:27:29.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:29 smithi096 ceph-mon[112443]: pgmap v160: 65 pgs: 5 active+undersized+degraded, 3 peering, 7 stale+active+clean, 3 active+undersized, 47 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 13/330 objects degraded (3.939%) 2024-04-22T15:27:30.112 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 podman[101709]: 2024-04-22 15:27:29.785546707 +0000 UTC m=+0.612521496 container remove 6537d137e8966dce3f0ddde383903338702efc6adf04d50b0ca713d252c5a40e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-22T15:27:30.112 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:29 smithi196 podman[101772]: 2024-04-22 15:27:29.994599035 +0000 UTC m=+0.102637003 container create 1628335216db531c67d036812dd6b1f2867c168b2d7ebd24c6fc3d9e5b819476 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T15:27:30.112 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:30 smithi196 podman[101772]: 2024-04-22 15:27:29.910395062 +0000 UTC m=+0.018433043 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:27:30.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:30 smithi196 podman[101772]: 2024-04-22 15:27:30.222465734 +0000 UTC m=+0.330503700 container init 1628335216db531c67d036812dd6b1f2867c168b2d7ebd24c6fc3d9e5b819476 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:27:30.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:30 smithi196 podman[101772]: 2024-04-22 15:27:30.22562378 +0000 UTC m=+0.333661746 container start 1628335216db531c67d036812dd6b1f2867c168b2d7ebd24c6fc3d9e5b819476 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T15:27:30.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:30 smithi196 ceph-osd[101792]: -- 172.21.15.196:0/2687463440 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563762b0f680 con 0x563762ad9000 2024-04-22T15:27:30.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:30 smithi196 bash[101772]: 1628335216db531c67d036812dd6b1f2867c168b2d7ebd24c6fc3d9e5b819476 2024-04-22T15:27:30.510 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:30 smithi196 systemd[1]: Started Ceph osd.5 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T15:27:31.009 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:30 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T15:27:30.771+0000 7f09b61297c0 -1 Falling back to public interface 2024-04-22T15:27:31.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:31 smithi196 ceph-mon[81815]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 3 peering, 11 active+undersized, 37 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-04-22T15:27:31.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:31 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:31.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:31 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:31.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:31 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:31 smithi096 ceph-mon[112443]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 3 peering, 11 active+undersized, 37 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-04-22T15:27:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:31 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:31 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:31 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:32.688 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:32 smithi196 ceph-mon[81815]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T15:27:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:32 smithi096 ceph-mon[112443]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T15:27:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:33 smithi096 ceph-mon[112443]: pgmap v162: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-04-22T15:27:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:33 smithi196 ceph-mon[81815]: pgmap v162: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-04-22T15:27:33.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:33.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:35.420 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:35 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T15:27:35.242+0000 7f09b61297c0 -1 osd.5 73 log_to_monitors true 2024-04-22T15:27:35.679 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:35 smithi196 ceph-mon[81815]: pgmap v163: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-04-22T15:27:35.679 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:35.679 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:35 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:35.679 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:35 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 51/330 objects degraded (15.455%), 14 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:35.679 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:35 smithi196 ceph-mon[81815]: from='osd.5 [v2:172.21.15.196:6816/288820489,v1:172.21.15.196:6817/288820489]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:27:35.679 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:35 smithi196 ceph-mon[81815]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:27:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:35 smithi096 ceph-mon[112443]: pgmap v163: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 260 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-04-22T15:27:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:35 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:35 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 51/330 objects degraded (15.455%), 14 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:35 smithi096 ceph-mon[112443]: from='osd.5 [v2:172.21.15.196:6816/288820489,v1:172.21.15.196:6817/288820489]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:27:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:35 smithi096 ceph-mon[112443]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:27:36.712 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:36 smithi196 ceph-mon[81815]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T15:27:36.712 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:36 smithi196 ceph-mon[81815]: osdmap e76: 6 total, 5 up, 6 in 2024-04-22T15:27:36.712 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:36 smithi196 ceph-mon[81815]: from='osd.5 [v2:172.21.15.196:6816/288820489,v1:172.21.15.196:6817/288820489]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:27:36.712 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:36 smithi196 ceph-mon[81815]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:27:36.712 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:36 smithi196 ceph-mon[81815]: pgmap v165: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-04-22T15:27:36.712 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:36 smithi196 ceph-mon[81815]: from='osd.5 ' entity='osd.5' 2024-04-22T15:27:36.713 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 15:27:36 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T15:27:36.414+0000 7f09a3579700 -1 osd.5 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:27:36.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:36 smithi096 ceph-mon[112443]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T15:27:36.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:36 smithi096 ceph-mon[112443]: osdmap e76: 6 total, 5 up, 6 in 2024-04-22T15:27:36.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:36 smithi096 ceph-mon[112443]: from='osd.5 [v2:172.21.15.196:6816/288820489,v1:172.21.15.196:6817/288820489]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:27:36.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:36 smithi096 ceph-mon[112443]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-22T15:27:36.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:36 smithi096 ceph-mon[112443]: pgmap v165: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-04-22T15:27:36.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:36 smithi096 ceph-mon[112443]: from='osd.5 ' entity='osd.5' 2024-04-22T15:27:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:37 smithi096 ceph-mon[112443]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:27:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:37 smithi096 ceph-mon[112443]: osd.5 [v2:172.21.15.196:6816/288820489,v1:172.21.15.196:6817/288820489] boot 2024-04-22T15:27:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:37 smithi096 ceph-mon[112443]: osdmap e77: 6 total, 6 up, 6 in 2024-04-22T15:27:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:37 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:27:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:37 smithi196 ceph-mon[81815]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:27:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:37 smithi196 ceph-mon[81815]: osd.5 [v2:172.21.15.196:6816/288820489,v1:172.21.15.196:6817/288820489] boot 2024-04-22T15:27:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:37 smithi196 ceph-mon[81815]: osdmap e77: 6 total, 6 up, 6 in 2024-04-22T15:27:37.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:37 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:27:39.467 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.467 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.467 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:39.467 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all osd 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-22T15:27:39.468 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: pgmap v167: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s; 51/330 objects degraded (15.455%) 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: Upgrade: Setting require_osd_release to 19 squid 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: osdmap e78: 6 total, 6 up, 6 in 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.469 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:39.472 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all osd 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: pgmap v167: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s; 51/330 objects degraded (15.455%) 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:27:39.473 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: Upgrade: Setting require_osd_release to 19 squid 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: osdmap e78: 6 total, 6 up, 6 in 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:39.474 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-22T15:27:39.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:39 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:27:39.580+0000 7f5dd82c5700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:27:40.419 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: osdmap e79: 6 total, 6 up, 6 in 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:40.420 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: osdmap e80: 6 total, 6 up, 6 in 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] up:boot 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6826/1411150341,v1:172.21.15.96:6827/1411150341] up:boot 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:27:40.421 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:27:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: osdmap e79: 6 total, 6 up, 6 in 2024-04-22T15:27:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-22T15:27:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-22T15:27:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: fsmap cephfs:2 {0=cephfs.smithi096.drkkdi=up:active,1=cephfs.smithi196.hwpcvg=up:active} 2024-04-22T15:27:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:27:40.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: osdmap e80: 6 total, 6 up, 6 in 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] up:boot 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6826/1411150341,v1:172.21.15.96:6827/1411150341] up:boot 2024-04-22T15:27:40.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-22T15:27:40.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:27:40.512 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:27:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:41 smithi096 ceph-mon[112443]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-22T15:27:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:41 smithi096 ceph-mon[112443]: pgmap v171: 65 pgs: 7 active+undersized+degraded, 11 active+undersized, 47 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s; 23/330 objects degraded (6.970%) 2024-04-22T15:27:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:41 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.196:6824/4222304137,v1:172.21.15.196:6825/4222304137] up:boot 2024-04-22T15:27:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:41 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6828/1619485870,v1:172.21.15.96:6829/1619485870] up:boot 2024-04-22T15:27:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:41 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:27:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:27:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:27:41.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:41 smithi196 ceph-mon[81815]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-22T15:27:41.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:41 smithi196 ceph-mon[81815]: pgmap v171: 65 pgs: 7 active+undersized+degraded, 11 active+undersized, 47 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s; 23/330 objects degraded (6.970%) 2024-04-22T15:27:41.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:41 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.196:6824/4222304137,v1:172.21.15.196:6825/4222304137] up:boot 2024-04-22T15:27:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:41 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6828/1619485870,v1:172.21.15.96:6829/1619485870] up:boot 2024-04-22T15:27:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:41 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:27:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:27:41.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:27:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:42 smithi096 ceph-mon[112443]: Health check update: Degraded data redundancy: 23/330 objects degraded (6.970%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:42 smithi096 ceph-mon[112443]: Upgrade: Updating mds.cephfs.smithi096.dymqlv 2024-04-22T15:27:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:42.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.dymqlv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:27:42.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:42.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:42 smithi096 ceph-mon[112443]: Deploying daemon mds.cephfs.smithi096.dymqlv on smithi096 2024-04-22T15:27:42.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:42 smithi196 ceph-mon[81815]: Health check update: Degraded data redundancy: 23/330 objects degraded (6.970%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T15:27:42.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:42 smithi196 ceph-mon[81815]: Upgrade: Updating mds.cephfs.smithi096.dymqlv 2024-04-22T15:27:42.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:42.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.dymqlv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:27:42.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:27:42.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:42 smithi196 ceph-mon[81815]: Deploying daemon mds.cephfs.smithi096.dymqlv on smithi096 2024-04-22T15:27:43.154 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:43 smithi096 ceph-mon[112443]: pgmap v172: 65 pgs: 1 active+undersized+degraded, 1 active+undersized, 63 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 3/330 objects degraded (0.909%) 2024-04-22T15:27:43.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:43 smithi196 ceph-mon[81815]: pgmap v172: 65 pgs: 1 active+undersized+degraded, 1 active+undersized, 63 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 3/330 objects degraded (0.909%) 2024-04-22T15:27:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:45 smithi096 ceph-mon[112443]: osdmap e81: 6 total, 6 up, 6 in 2024-04-22T15:27:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:45 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T15:27:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:45 smithi096 ceph-mon[112443]: pgmap v174: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:45.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:45 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:27:45.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:45 smithi196 ceph-mon[81815]: osdmap e81: 6 total, 6 up, 6 in 2024-04-22T15:27:45.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:45 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T15:27:45.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:45 smithi196 ceph-mon[81815]: pgmap v174: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:45.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:45 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.049+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/938756543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 msgr2=0x7f5eb80fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.049+0000 7f5ebd5fb700 1 --2- 172.21.15.96:0/938756543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 0x7f5eb80fff30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009910 tx=0x7f5ea0009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.049+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/938756543 shutdown_connections 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.049+0000 7f5ebd5fb700 1 --2- 172.21.15.96:0/938756543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 0x7f5eb80fff30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.049+0000 7f5ebd5fb700 1 --2- 172.21.15.96:0/938756543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb8105ab0 0x7f5eb8105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.049+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/938756543 >> 172.21.15.96:0/938756543 conn(0x7f5eb80fb5e0 msgr2=0x7f5eb80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.049+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/938756543 shutdown_connections 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.049+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/938756543 wait complete. 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5ebd5fb700 1 Processor -- start 2024-04-22T15:27:46.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5ebd5fb700 1 -- start start 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5ebd5fb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 0x7f5eb819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5ebd5fb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb8105ab0 0x7f5eb819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5ebd5fb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb819d440 con 0x7f5eb80ffab0 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5ebd5fb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8196a20 con 0x7f5eb8105ab0 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5eb6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 0x7f5eb819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5eb6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 0x7f5eb819c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35800/0 (socket says 172.21.15.96:35800) 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5eb6ffd700 1 -- 172.21.15.96:0/3393362062 learned_addr learned my addr 172.21.15.96:0/3393362062 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.050+0000 7f5eae7fc700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb8105ab0 0x7f5eb819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5eb6ffd700 1 -- 172.21.15.96:0/3393362062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb8105ab0 msgr2=0x7f5eb819ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5eb6ffd700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb8105ab0 0x7f5eb819ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5eb6ffd700 1 -- 172.21.15.96:0/3393362062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea00095f0 con 0x7f5eb80ffab0 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5eae7fc700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb8105ab0 0x7f5eb819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:27:46.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5eb6ffd700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 0x7f5eb819c8e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800b4b0 tx=0x7f5ea800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:46.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5eb4ff9700 1 -- 172.21.15.96:0/3393362062 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ea8011800 con 0x7f5eb80ffab0 2024-04-22T15:27:46.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5eb4ff9700 1 -- 172.21.15.96:0/3393362062 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ea8011e20 con 0x7f5eb80ffab0 2024-04-22T15:27:46.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb8196d00 con 0x7f5eb80ffab0 2024-04-22T15:27:46.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5eb4ff9700 1 -- 172.21.15.96:0/3393362062 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ea800f2e0 con 0x7f5eb80ffab0 2024-04-22T15:27:46.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.051+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb8197250 con 0x7f5eb80ffab0 2024-04-22T15:27:46.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.052+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb81090e0 con 0x7f5eb80ffab0 2024-04-22T15:27:46.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.053+0000 7f5eb4ff9700 1 -- 172.21.15.96:0/3393362062 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5ea8010380 con 0x7f5eb80ffab0 2024-04-22T15:27:46.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.053+0000 7f5eb4ff9700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ea406ecc0 0x7f5ea4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:46.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.053+0000 7f5eb4ff9700 1 -- 172.21.15.96:0/3393362062 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5ea808ea50 con 0x7f5eb80ffab0 2024-04-22T15:27:46.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.053+0000 7f5eae7fc700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ea406ecc0 0x7f5ea4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:46.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.054+0000 7f5eae7fc700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ea406ecc0 0x7f5ea4071180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8197be0 tx=0x7f5ea001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:46.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.058+0000 7f5eb4ff9700 1 -- 172.21.15.96:0/3393362062 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5ea805b2e0 con 0x7f5eb80ffab0 2024-04-22T15:27:46.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.279+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5eb8197f50 con 0x7f5ea406ecc0 2024-04-22T15:27:46.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.280+0000 7f5eb4ff9700 1 -- 172.21.15.96:0/3393362062 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5eb8197f50 con 0x7f5ea406ecc0 2024-04-22T15:27:46.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ea406ecc0 msgr2=0x7f5ea4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ea406ecc0 0x7f5ea4071180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8197be0 tx=0x7f5ea001a040 comp rx=0 tx=0).stop 2024-04-22T15:27:46.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 msgr2=0x7f5eb819c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 0x7f5eb819c8e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800b4b0 tx=0x7f5ea800b870 comp rx=0 tx=0).stop 2024-04-22T15:27:46.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 shutdown_connections 2024-04-22T15:27:46.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5ea406ecc0 0x7f5ea4071180 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5eb80ffab0 0x7f5eb819c8e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 --2- 172.21.15.96:0/3393362062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb8105ab0 0x7f5eb819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 >> 172.21.15.96:0/3393362062 conn(0x7f5eb80fb5e0 msgr2=0x7f5eb8104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:46.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 shutdown_connections 2024-04-22T15:27:46.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.282+0000 7f5ebd5fb700 1 -- 172.21.15.96:0/3393362062 wait complete. 2024-04-22T15:27:46.296 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:27:46.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.389+0000 7fbd1fd99700 1 -- 172.21.15.96:0/1924172749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd180ff090 msgr2=0x7fbd180ff510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.389+0000 7fbd1fd99700 1 --2- 172.21.15.96:0/1924172749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd180ff090 0x7fbd180ff510 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009910 tx=0x7fbd08009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:46.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.389+0000 7fbd1fd99700 1 -- 172.21.15.96:0/1924172749 shutdown_connections 2024-04-22T15:27:46.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.389+0000 7fbd1fd99700 1 --2- 172.21.15.96:0/1924172749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd180ff090 0x7fbd180ff510 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.389+0000 7fbd1fd99700 1 --2- 172.21.15.96:0/1924172749 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd18105c70 0x7fbd18106050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.389+0000 7fbd1fd99700 1 -- 172.21.15.96:0/1924172749 >> 172.21.15.96:0/1924172749 conn(0x7fbd18075f00 msgr2=0x7fbd180fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:46.393 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:46 smithi096 ceph-mon[112443]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/330 objects degraded (0.909%), 1 pg degraded) 2024-04-22T15:27:46.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.389+0000 7fbd1fd99700 1 -- 172.21.15.96:0/1924172749 shutdown_connections 2024-04-22T15:27:46.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.389+0000 7fbd1fd99700 1 -- 172.21.15.96:0/1924172749 wait complete. 2024-04-22T15:27:46.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.389+0000 7fbd1fd99700 1 Processor -- start 2024-04-22T15:27:46.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1fd99700 1 -- start start 2024-04-22T15:27:46.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1fd99700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd180ff090 0x7fbd18100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:46.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1fd99700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18105c70 0x7fbd181007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:46.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1fd99700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18100cf0 con 0x7fbd18105c70 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1fd99700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18100e60 con 0x7fbd180ff090 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1d334700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18105c70 0x7fbd181007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1d334700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18105c70 0x7fbd181007b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35810/0 (socket says 172.21.15.96:35810) 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1d334700 1 -- 172.21.15.96:0/930753490 learned_addr learned my addr 172.21.15.96:0/930753490 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1db35700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd180ff090 0x7fbd18100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1d334700 1 -- 172.21.15.96:0/930753490 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd180ff090 msgr2=0x7fbd18100270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1d334700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd180ff090 0x7fbd18100270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1d334700 1 -- 172.21.15.96:0/930753490 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd080095f0 con 0x7fbd18105c70 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd1d334700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18105c70 0x7fbd181007b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbd08006010 tx=0x7fbd0800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.390+0000 7fbd0effd700 1 -- 172.21.15.96:0/930753490 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd0801d070 con 0x7fbd18105c70 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.391+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd181010e0 con 0x7fbd18105c70 2024-04-22T15:27:46.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.391+0000 7fbd0effd700 1 -- 172.21.15.96:0/930753490 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd0800f460 con 0x7fbd18105c70 2024-04-22T15:27:46.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.391+0000 7fbd0effd700 1 -- 172.21.15.96:0/930753490 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd08017630 con 0x7fbd18105c70 2024-04-22T15:27:46.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.391+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd181a30a0 con 0x7fbd18105c70 2024-04-22T15:27:46.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.391+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd181091f0 con 0x7fbd18105c70 2024-04-22T15:27:46.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.392+0000 7fbd0effd700 1 -- 172.21.15.96:0/930753490 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbd0800f5d0 con 0x7fbd18105c70 2024-04-22T15:27:46.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.393+0000 7fbd0effd700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbd0406ecc0 0x7fbd04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:46.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.393+0000 7fbd1db35700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbd0406ecc0 0x7fbd04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:46.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.393+0000 7fbd0effd700 1 -- 172.21.15.96:0/930753490 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbd08090180 con 0x7fbd18105c70 2024-04-22T15:27:46.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.393+0000 7fbd1db35700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbd0406ecc0 0x7fbd04071180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbd14005ea0 tx=0x7fbd14005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:46.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.397+0000 7fbd0effd700 1 -- 172.21.15.96:0/930753490 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbd08058fa0 con 0x7fbd18105c70 2024-04-22T15:27:46.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:46 smithi196 ceph-mon[81815]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/330 objects degraded (0.909%), 1 pg degraded) 2024-04-22T15:27:46.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.619+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd18102370 con 0x7fbd0406ecc0 2024-04-22T15:27:46.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.621+0000 7fbd0effd700 1 -- 172.21.15.96:0/930753490 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbd1400a890 con 0x7fbd0406ecc0 2024-04-22T15:27:46.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbd0406ecc0 msgr2=0x7fbd04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbd0406ecc0 0x7fbd04071180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbd14005ea0 tx=0x7fbd14005e30 comp rx=0 tx=0).stop 2024-04-22T15:27:46.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18105c70 msgr2=0x7fbd181007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18105c70 0x7fbd181007b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbd08006010 tx=0x7fbd0800be20 comp rx=0 tx=0).stop 2024-04-22T15:27:46.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 shutdown_connections 2024-04-22T15:27:46.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbd0406ecc0 0x7fbd04071180 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd180ff090 0x7fbd18100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 --2- 172.21.15.96:0/930753490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18105c70 0x7fbd181007b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 >> 172.21.15.96:0/930753490 conn(0x7fbd18075f00 msgr2=0x7fbd180fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:46.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 shutdown_connections 2024-04-22T15:27:46.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.623+0000 7fbd1fd99700 1 -- 172.21.15.96:0/930753490 wait complete. 2024-04-22T15:27:46.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.728+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/1830370245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c40ffab0 msgr2=0x7fe9c40fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.728+0000 7fe9cc4e3700 1 --2- 172.21.15.96:0/1830370245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c40ffab0 0x7fe9c40fff30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009910 tx=0x7fe9b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:46.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.729+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/1830370245 shutdown_connections 2024-04-22T15:27:46.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.729+0000 7fe9cc4e3700 1 --2- 172.21.15.96:0/1830370245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c40ffab0 0x7fe9c40fff30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.729+0000 7fe9cc4e3700 1 --2- 172.21.15.96:0/1830370245 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9c4105ab0 0x7fe9c4105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.729+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/1830370245 >> 172.21.15.96:0/1830370245 conn(0x7fe9c40fb5e0 msgr2=0x7fe9c40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:46.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.729+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/1830370245 shutdown_connections 2024-04-22T15:27:46.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.729+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/1830370245 wait complete. 2024-04-22T15:27:46.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.729+0000 7fe9cc4e3700 1 Processor -- start 2024-04-22T15:27:46.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.729+0000 7fe9cc4e3700 1 -- start start 2024-04-22T15:27:46.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9cc4e3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9c40ffab0 0x7fe9c4197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9cc4e3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c4105ab0 0x7fe9c4197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9cc4e3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c4198420 con 0x7fe9c4105ab0 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9cc4e3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c4198560 con 0x7fe9c40ffab0 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9c9a7e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c4105ab0 0x7fe9c4197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9c9a7e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c4105ab0 0x7fe9c4197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35820/0 (socket says 172.21.15.96:35820) 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9c9a7e700 1 -- 172.21.15.96:0/2525000478 learned_addr learned my addr 172.21.15.96:0/2525000478 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9ca27f700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9c40ffab0 0x7fe9c4197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9c9a7e700 1 -- 172.21.15.96:0/2525000478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9c40ffab0 msgr2=0x7fe9c4197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9c9a7e700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9c40ffab0 0x7fe9c4197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9c9a7e700 1 -- 172.21.15.96:0/2525000478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b40095f0 con 0x7fe9c4105ab0 2024-04-22T15:27:46.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9ca27f700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9c40ffab0 0x7fe9c4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:27:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9c9a7e700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c4105ab0 0x7fe9c4197e50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4005f50 tx=0x7fe9b400b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9bb7fe700 1 -- 172.21.15.96:0/2525000478 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9b401d070 con 0x7fe9c4105ab0 2024-04-22T15:27:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9bb7fe700 1 -- 172.21.15.96:0/2525000478 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9b40052a0 con 0x7fe9c4105ab0 2024-04-22T15:27:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9bb7fe700 1 -- 172.21.15.96:0/2525000478 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9b4017400 con 0x7fe9c4105ab0 2024-04-22T15:27:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c41a0980 con 0x7fe9c4105ab0 2024-04-22T15:27:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.730+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c41a0d30 con 0x7fe9c4105ab0 2024-04-22T15:27:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.731+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c41080d0 con 0x7fe9c4105ab0 2024-04-22T15:27:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.732+0000 7fe9bb7fe700 1 -- 172.21.15.96:0/2525000478 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe9b400f7a0 con 0x7fe9c4105ab0 2024-04-22T15:27:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.732+0000 7fe9bb7fe700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9b006ecd0 0x7fe9b0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.732+0000 7fe9bb7fe700 1 -- 172.21.15.96:0/2525000478 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe9b4090330 con 0x7fe9c4105ab0 2024-04-22T15:27:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.733+0000 7fe9ca27f700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9b006ecd0 0x7fe9b0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.733+0000 7fe9ca27f700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9b006ecd0 0x7fe9b0071190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009be0 tx=0x7fe9c0009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:46.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.737+0000 7fe9bb7fe700 1 -- 172.21.15.96:0/2525000478 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe9b4059090 con 0x7fe9c4105ab0 2024-04-22T15:27:46.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.948+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9c410ccb0 con 0x7fe9b006ecd0 2024-04-22T15:27:46.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.959+0000 7fe9bb7fe700 1 -- 172.21.15.96:0/2525000478 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe9c410ccb0 con 0x7fe9b006ecd0 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (6m) 114s ago 17m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (18m) 114s ago 18m 8870k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (16m) 13s ago 16m 8816k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (3m) 114s ago 18m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (3m) 13s ago 16m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (6m) 114s ago 17m 80.7M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (12m) 114s ago 12m 155M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 b12eb04fdea7 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (12m) 114s ago 12m 90.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 94855d397c4e 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (12m) 13s ago 12m 27.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (11m) 13s ago 11m 26.6M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (8m) 114s ago 19m 505M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (7m) 13s ago 16m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:27:46.963 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (4m) 114s ago 19m 46.9M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (4m) 13s ago 16m 39.2M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (7m) 114s ago 17m 17.3M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (7m) 13s ago 16m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (3m) 114s ago 15m 92.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (2m) 114s ago 15m 93.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (2m) 114s ago 14m 11.9M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (84s) 13s ago 14m 100M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (50s) 13s ago 14m 86.5M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (17s) 13s ago 13m 12.1M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:27:46.964 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (6m) 114s ago 17m 77.0M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:27:46.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9b006ecd0 msgr2=0x7fe9b0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9b006ecd0 0x7fe9b0071190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009be0 tx=0x7fe9c0009260 comp rx=0 tx=0).stop 2024-04-22T15:27:46.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c4105ab0 msgr2=0x7fe9c4197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:46.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c4105ab0 0x7fe9c4197e50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4005f50 tx=0x7fe9b400b650 comp rx=0 tx=0).stop 2024-04-22T15:27:46.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 shutdown_connections 2024-04-22T15:27:46.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9b006ecd0 0x7fe9b0071190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9c40ffab0 0x7fe9c4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 --2- 172.21.15.96:0/2525000478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9c4105ab0 0x7fe9c4197e50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:46.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 >> 172.21.15.96:0/2525000478 conn(0x7fe9c40fb5e0 msgr2=0x7fe9c40fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:46.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 shutdown_connections 2024-04-22T15:27:46.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:46.961+0000 7fe9cc4e3700 1 -- 172.21.15.96:0/2525000478 wait complete. 2024-04-22T15:27:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.064+0000 7fefca7d2700 1 -- 172.21.15.96:0/3567756459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 msgr2=0x7fefc410aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.064+0000 7fefca7d2700 1 --2- 172.21.15.96:0/3567756459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 0x7fefc410aba0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009910 tx=0x7fefb4009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.064+0000 7fefca7d2700 1 -- 172.21.15.96:0/3567756459 shutdown_connections 2024-04-22T15:27:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.064+0000 7fefca7d2700 1 --2- 172.21.15.96:0/3567756459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 0x7fefc410aba0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.064+0000 7fefca7d2700 1 --2- 172.21.15.96:0/3567756459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fefc4101d90 0x7fefc4102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.064+0000 7fefca7d2700 1 -- 172.21.15.96:0/3567756459 >> 172.21.15.96:0/3567756459 conn(0x7fefc40fb5e0 msgr2=0x7fefc40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.064+0000 7fefca7d2700 1 -- 172.21.15.96:0/3567756459 shutdown_connections 2024-04-22T15:27:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.064+0000 7fefca7d2700 1 -- 172.21.15.96:0/3567756459 wait complete. 2024-04-22T15:27:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.064+0000 7fefca7d2700 1 Processor -- start 2024-04-22T15:27:47.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefca7d2700 1 -- start start 2024-04-22T15:27:47.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefca7d2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fefc4101d90 0x7fefc419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:47.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefca7d2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 0x7fefc419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:47.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefca7d2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc419d340 con 0x7fefc41026b0 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefca7d2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc41969b0 con 0x7fefc4101d90 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefc37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 0x7fefc419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefc37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 0x7fefc419cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35842/0 (socket says 172.21.15.96:35842) 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefc37fe700 1 -- 172.21.15.96:0/2443044717 learned_addr learned my addr 172.21.15.96:0/2443044717 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefc3fff700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fefc4101d90 0x7fefc419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefc37fe700 1 -- 172.21.15.96:0/2443044717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fefc4101d90 msgr2=0x7fefc419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefc37fe700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fefc4101d90 0x7fefc419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefc37fe700 1 -- 172.21.15.96:0/2443044717 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb40095f0 con 0x7fefc41026b0 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefc3fff700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fefc4101d90 0x7fefc419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.065+0000 7fefc37fe700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 0x7fefc419cdb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fefb40092e0 tx=0x7fefb400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.066+0000 7fefc17fa700 1 -- 172.21.15.96:0/2443044717 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefb401d070 con 0x7fefc41026b0 2024-04-22T15:27:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.066+0000 7fefc17fa700 1 -- 172.21.15.96:0/2443044717 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefb400f460 con 0x7fefc41026b0 2024-04-22T15:27:47.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.066+0000 7fefc17fa700 1 -- 172.21.15.96:0/2443044717 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefb4017630 con 0x7fefc41026b0 2024-04-22T15:27:47.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.066+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc4196c30 con 0x7fefc41026b0 2024-04-22T15:27:47.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.066+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc4197120 con 0x7fefc41026b0 2024-04-22T15:27:47.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.067+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc41054b0 con 0x7fefc41026b0 2024-04-22T15:27:47.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.067+0000 7fefc17fa700 1 -- 172.21.15.96:0/2443044717 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fefb400f5d0 con 0x7fefc41026b0 2024-04-22T15:27:47.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.068+0000 7fefc17fa700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fefb006ecc0 0x7fefb0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:47.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.068+0000 7fefc17fa700 1 -- 172.21.15.96:0/2443044717 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fefb4090370 con 0x7fefc41026b0 2024-04-22T15:27:47.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.068+0000 7fefc3fff700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fefb006ecc0 0x7fefb0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:47.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.068+0000 7fefc3fff700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fefb006ecc0 0x7fefb0071180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fefac005fd0 tx=0x7fefac005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:47.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.072+0000 7fefc17fa700 1 -- 172.21.15.96:0/2443044717 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fefb405cc00 con 0x7fefc41026b0 2024-04-22T15:27:47.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:47 smithi096 ceph-mon[112443]: pgmap v175: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:47.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:47 smithi096 ceph-mon[112443]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:47.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.360+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fefc4197e70 con 0x7fefc41026b0 2024-04-22T15:27:47.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.361+0000 7fefc17fa700 1 -- 172.21.15.96:0/2443044717 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+731 (secure 0 0 0) 0x7fefb4005740 con 0x7fefc41026b0 2024-04-22T15:27:47.364 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:27:47.364 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:27:47.364 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 3 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 3, 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 10 2024-04-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fefb006ecc0 msgr2=0x7fefb0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fefb006ecc0 0x7fefb0071180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fefac005fd0 tx=0x7fefac005e20 comp rx=0 tx=0).stop 2024-04-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 msgr2=0x7fefc419cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 0x7fefc419cdb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fefb40092e0 tx=0x7fefb400be20 comp rx=0 tx=0).stop 2024-04-22T15:27:47.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 shutdown_connections 2024-04-22T15:27:47.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fefb006ecc0 0x7fefb0071180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fefc4101d90 0x7fefc419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 --2- 172.21.15.96:0/2443044717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefc41026b0 0x7fefc419cdb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 >> 172.21.15.96:0/2443044717 conn(0x7fefc40fb5e0 msgr2=0x7fefc4103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:47.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 shutdown_connections 2024-04-22T15:27:47.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.363+0000 7fefca7d2700 1 -- 172.21.15.96:0/2443044717 wait complete. 2024-04-22T15:27:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.470+0000 7fbc0782c700 1 -- 172.21.15.96:0/1066779414 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc00100e30 msgr2=0x7fbc00101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.470+0000 7fbc0782c700 1 --2- 172.21.15.96:0/1066779414 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc00100e30 0x7fbc00101210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009910 tx=0x7fbbf0009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.470+0000 7fbc0782c700 1 -- 172.21.15.96:0/1066779414 shutdown_connections 2024-04-22T15:27:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.470+0000 7fbc0782c700 1 --2- 172.21.15.96:0/1066779414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc001017e0 0x7fbc00105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.470+0000 7fbc0782c700 1 --2- 172.21.15.96:0/1066779414 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc00100e30 0x7fbc00101210 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.470+0000 7fbc0782c700 1 -- 172.21.15.96:0/1066779414 >> 172.21.15.96:0/1066779414 conn(0x7fbc000fc700 msgr2=0x7fbc000feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.470+0000 7fbc0782c700 1 -- 172.21.15.96:0/1066779414 shutdown_connections 2024-04-22T15:27:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.470+0000 7fbc0782c700 1 -- 172.21.15.96:0/1066779414 wait complete. 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc0782c700 1 Processor -- start 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc0782c700 1 -- start start 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc0782c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc00100e30 0x7fbc0019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc0782c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc001017e0 0x7fbc0019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc0782c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0019d490 con 0x7fbc001017e0 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc0782c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00196a00 con 0x7fbc00100e30 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc04dc7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc001017e0 0x7fbc0019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc04dc7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc001017e0 0x7fbc0019ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35870/0 (socket says 172.21.15.96:35870) 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc04dc7700 1 -- 172.21.15.96:0/3253094550 learned_addr learned my addr 172.21.15.96:0/3253094550 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc055c8700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc00100e30 0x7fbc0019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc04dc7700 1 -- 172.21.15.96:0/3253094550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc00100e30 msgr2=0x7fbc0019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc04dc7700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc00100e30 0x7fbc0019c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.471+0000 7fbc04dc7700 1 -- 172.21.15.96:0/3253094550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf00095f0 con 0x7fbc001017e0 2024-04-22T15:27:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.472+0000 7fbc04dc7700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc001017e0 0x7fbc0019ce00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00d900 tx=0x7fbbfc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.472+0000 7fbbf67fc700 1 -- 172.21.15.96:0/3253094550 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbfc009900 con 0x7fbc001017e0 2024-04-22T15:27:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.472+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc00196ce0 con 0x7fbc001017e0 2024-04-22T15:27:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.472+0000 7fbbf67fc700 1 -- 172.21.15.96:0/3253094550 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbfc010460 con 0x7fbc001017e0 2024-04-22T15:27:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.472+0000 7fbbf67fc700 1 -- 172.21.15.96:0/3253094550 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbfc00f5f0 con 0x7fbc001017e0 2024-04-22T15:27:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.472+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc00197230 con 0x7fbc001017e0 2024-04-22T15:27:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.473+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc001091d0 con 0x7fbc001017e0 2024-04-22T15:27:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.473+0000 7fbbf67fc700 1 -- 172.21.15.96:0/3253094550 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbbfc00fa80 con 0x7fbc001017e0 2024-04-22T15:27:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.473+0000 7fbbf67fc700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbec06ecc0 0x7fbbec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.474+0000 7fbbf67fc700 1 -- 172.21.15.96:0/3253094550 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbbfc08ecf0 con 0x7fbc001017e0 2024-04-22T15:27:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.474+0000 7fbc055c8700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbec06ecc0 0x7fbbec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.474+0000 7fbc055c8700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbec06ecc0 0x7fbbec071180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0005b40 tx=0x7fbbf001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:47.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.478+0000 7fbbf67fc700 1 -- 172.21.15.96:0/3253094550 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbbfc05b580 con 0x7fbc001017e0 2024-04-22T15:27:47.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:47 smithi196 ceph-mon[81815]: pgmap v175: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:47.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:47 smithi196 ceph-mon[81815]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:47.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.728+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbc00197ec0 con 0x7fbc001017e0 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.728+0000 7fbbf67fc700 1 -- 172.21.15.96:0/3253094550 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1696 (secure 0 0 0) 0x7fbbfc05b110 con 0x7fbc001017e0 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout:e24 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:27:47.732 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:27:47.733 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:27:47.734 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:27:47.735 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:27:47.735 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/1619485870,v1:172.21.15.96:6829/1619485870] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:27:47.735 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:27:47.735 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44233} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4222304137,v1:172.21.15.196:6825/4222304137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:27:47.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.730+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbec06ecc0 msgr2=0x7fbbec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.730+0000 7fbc0782c700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbec06ecc0 0x7fbbec071180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0005b40 tx=0x7fbbf001a040 comp rx=0 tx=0).stop 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.730+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc001017e0 msgr2=0x7fbc0019ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.730+0000 7fbc0782c700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc001017e0 0x7fbc0019ce00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00d900 tx=0x7fbbfc00dcc0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.730+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 shutdown_connections 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.730+0000 7fbc0782c700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbec06ecc0 0x7fbbec071180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.730+0000 7fbc0782c700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc00100e30 0x7fbc0019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.730+0000 7fbc0782c700 1 --2- 172.21.15.96:0/3253094550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc001017e0 0x7fbc0019ce00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.730+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 >> 172.21.15.96:0/3253094550 conn(0x7fbc000fc700 msgr2=0x7fbc000fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.731+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 shutdown_connections 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.731+0000 7fbc0782c700 1 -- 172.21.15.96:0/3253094550 wait complete. 2024-04-22T15:27:47.736 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 24 2024-04-22T15:27:47.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 -- 172.21.15.96:0/1318330904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 msgr2=0x7f9fbc105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 --2- 172.21.15.96:0/1318330904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 0x7f9fbc105fe0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009910 tx=0x7f9fb8009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:47.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 -- 172.21.15.96:0/1318330904 shutdown_connections 2024-04-22T15:27:47.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 --2- 172.21.15.96:0/1318330904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9fbc0ffb50 0x7f9fbc0fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 --2- 172.21.15.96:0/1318330904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 0x7f9fbc105fe0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 -- 172.21.15.96:0/1318330904 >> 172.21.15.96:0/1318330904 conn(0x7f9fbc0fb680 msgr2=0x7f9fbc0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 -- 172.21.15.96:0/1318330904 shutdown_connections 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 -- 172.21.15.96:0/1318330904 wait complete. 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 Processor -- start 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.837+0000 7f9fc43e9700 1 -- start start 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc43e9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9fbc0ffb50 0x7f9fbc195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc43e9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 0x7f9fbc195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc43e9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc06a070 con 0x7f9fbc105c00 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc43e9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc1962d0 con 0x7f9fbc0ffb50 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc1984700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 0x7f9fbc195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc1984700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 0x7f9fbc195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35884/0 (socket says 172.21.15.96:35884) 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc1984700 1 -- 172.21.15.96:0/4220618781 learned_addr learned my addr 172.21.15.96:0/4220618781 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc2185700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9fbc0ffb50 0x7f9fbc195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:47.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc1984700 1 -- 172.21.15.96:0/4220618781 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9fbc0ffb50 msgr2=0x7f9fbc195850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:47.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc1984700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9fbc0ffb50 0x7f9fbc195850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:47.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc1984700 1 -- 172.21.15.96:0/4220618781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb80095f0 con 0x7f9fbc105c00 2024-04-22T15:27:47.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc1984700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 0x7f9fbc195d90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00b4b0 tx=0x7f9fac00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:47.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fb37fe700 1 -- 172.21.15.96:0/4220618781 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fac011800 con 0x7f9fbc105c00 2024-04-22T15:27:47.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc196550 con 0x7f9fbc105c00 2024-04-22T15:27:47.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fb37fe700 1 -- 172.21.15.96:0/4220618781 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9fac011e20 con 0x7f9fbc105c00 2024-04-22T15:27:47.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fb37fe700 1 -- 172.21.15.96:0/4220618781 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fac00f2e0 con 0x7f9fbc105c00 2024-04-22T15:27:47.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.838+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc1a3160 con 0x7f9fbc105c00 2024-04-22T15:27:47.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.839+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc108030 con 0x7f9fbc105c00 2024-04-22T15:27:47.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.840+0000 7f9fb37fe700 1 -- 172.21.15.96:0/4220618781 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9fac010380 con 0x7f9fbc105c00 2024-04-22T15:27:47.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.840+0000 7f9fb37fe700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9fa806ecd0 0x7f9fa8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:47.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.840+0000 7f9fc2185700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9fa806ecd0 0x7f9fa8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:47.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.840+0000 7f9fb37fe700 1 -- 172.21.15.96:0/4220618781 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9fac08ea90 con 0x7f9fbc105c00 2024-04-22T15:27:47.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.841+0000 7f9fc2185700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9fa806ecd0 0x7f9fa8071190 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8000c00 tx=0x7f9fb801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:47.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:47.845+0000 7f9fb37fe700 1 -- 172.21.15.96:0/4220618781 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9fac05b320 con 0x7f9fbc105c00 2024-04-22T15:27:48.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.072+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fbc069680 con 0x7f9fa806ecd0 2024-04-22T15:27:48.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.074+0000 7f9fb37fe700 1 -- 172.21.15.96:0/4220618781 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9fbc069680 con 0x7f9fa806ecd0 2024-04-22T15:27:48.077 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:27:48.077 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:27:48.077 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:27:48.077 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:27:48.077 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:27:48.078 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:27:48.078 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:27:48.078 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:27:48.078 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:27:48.078 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:27:48.078 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "12/23 daemons upgraded", 2024-04-22T15:27:48.078 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:27:48.078 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:27:48.078 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9fa806ecd0 msgr2=0x7f9fa8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9fa806ecd0 0x7f9fa8071190 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8000c00 tx=0x7f9fb801a040 comp rx=0 tx=0).stop 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 msgr2=0x7f9fbc195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 0x7f9fbc195d90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00b4b0 tx=0x7f9fac00b870 comp rx=0 tx=0).stop 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 shutdown_connections 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9fa806ecd0 0x7f9fa8071190 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9fbc0ffb50 0x7f9fbc195850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 --2- 172.21.15.96:0/4220618781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc105c00 0x7f9fbc195d90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 >> 172.21.15.96:0/4220618781 conn(0x7f9fbc0fb680 msgr2=0x7f9fbc104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 shutdown_connections 2024-04-22T15:27:48.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.076+0000 7f9fc43e9700 1 -- 172.21.15.96:0/4220618781 wait complete. 2024-04-22T15:27:48.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.182+0000 7fcc426e0700 1 -- 172.21.15.96:0/3049117839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c104b30 msgr2=0x7fcc3c104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.182+0000 7fcc426e0700 1 --2- 172.21.15.96:0/3049117839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c104b30 0x7fcc3c104f10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009910 tx=0x7fcc2c009c20 comp rx=0 tx=0).stop 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.182+0000 7fcc426e0700 1 -- 172.21.15.96:0/3049117839 shutdown_connections 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.182+0000 7fcc426e0700 1 --2- 172.21.15.96:0/3049117839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc3c100220 0x7fcc3c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.182+0000 7fcc426e0700 1 --2- 172.21.15.96:0/3049117839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c104b30 0x7fcc3c104f10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.182+0000 7fcc426e0700 1 -- 172.21.15.96:0/3049117839 >> 172.21.15.96:0/3049117839 conn(0x7fcc3c075f40 msgr2=0x7fcc3c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.182+0000 7fcc426e0700 1 -- 172.21.15.96:0/3049117839 shutdown_connections 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.182+0000 7fcc426e0700 1 -- 172.21.15.96:0/3049117839 wait complete. 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.183+0000 7fcc426e0700 1 Processor -- start 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.183+0000 7fcc426e0700 1 -- start start 2024-04-22T15:27:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.183+0000 7fcc426e0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c100220 0x7fcc3c197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.183+0000 7fcc426e0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc3c104b30 0x7fcc3c197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.183+0000 7fcc426e0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c198390 con 0x7fcc3c100220 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.183+0000 7fcc426e0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c1984d0 con 0x7fcc3c104b30 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.183+0000 7fcc3bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c100220 0x7fcc3c197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc3bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c100220 0x7fcc3c197880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35912/0 (socket says 172.21.15.96:35912) 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc3bfff700 1 -- 172.21.15.96:0/1648738639 learned_addr learned my addr 172.21.15.96:0/1648738639 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc3b7fe700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc3c104b30 0x7fcc3c197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc3bfff700 1 -- 172.21.15.96:0/1648738639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc3c104b30 msgr2=0x7fcc3c197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc3bfff700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc3c104b30 0x7fcc3c197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc3bfff700 1 -- 172.21.15.96:0/1648738639 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0095f0 con 0x7fcc3c100220 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc3b7fe700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc3c104b30 0x7fcc3c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:27:48.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc3bfff700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c100220 0x7fcc3c197880 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00bd70 tx=0x7fcc2c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:48.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc397fa700 1 -- 172.21.15.96:0/1648738639 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc2c01d070 con 0x7fcc3c100220 2024-04-22T15:27:48.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc397fa700 1 -- 172.21.15.96:0/1648738639 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc2c00f460 con 0x7fcc3c100220 2024-04-22T15:27:48.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc397fa700 1 -- 172.21.15.96:0/1648738639 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc2c017630 con 0x7fcc3c100220 2024-04-22T15:27:48.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c1a08f0 con 0x7fcc3c100220 2024-04-22T15:27:48.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.184+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c1a0d20 con 0x7fcc3c100220 2024-04-22T15:27:48.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.185+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c108100 con 0x7fcc3c100220 2024-04-22T15:27:48.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.186+0000 7fcc397fa700 1 -- 172.21.15.96:0/1648738639 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc2c00f5d0 con 0x7fcc3c100220 2024-04-22T15:27:48.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.186+0000 7fcc397fa700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc2406ecc0 0x7fcc24071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:27:48.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.186+0000 7fcc397fa700 1 -- 172.21.15.96:0/1648738639 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcc2c090400 con 0x7fcc3c100220 2024-04-22T15:27:48.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.186+0000 7fcc3b7fe700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc2406ecc0 0x7fcc24071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:27:48.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.187+0000 7fcc3b7fe700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc2406ecc0 0x7fcc24071180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c0ff960 tx=0x7fcc30005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:27:48.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.192+0000 7fcc397fa700 1 -- 172.21.15.96:0/1648738639 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcc2c059160 con 0x7fcc3c100220 2024-04-22T15:27:48.330 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:48 smithi096 ceph-mon[112443]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:48.330 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:48 smithi096 ceph-mon[112443]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:48.330 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:48 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2443044717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:48.330 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:48 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3253094550' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:27:48.504 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:27:48.504 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:27:48.504 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:27:48.504 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:27:48.504 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:27:48.504 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:27:48.504 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:27:48.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.499+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcc3c101440 con 0x7fcc3c100220 2024-04-22T15:27:48.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.500+0000 7fcc397fa700 1 -- 172.21.15.96:0/1648738639 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fcc2c05c820 con 0x7fcc3c100220 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc2406ecc0 msgr2=0x7fcc24071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc2406ecc0 0x7fcc24071180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c0ff960 tx=0x7fcc30005f50 comp rx=0 tx=0).stop 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c100220 msgr2=0x7fcc3c197880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c100220 0x7fcc3c197880 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00bd70 tx=0x7fcc2c00be50 comp rx=0 tx=0).stop 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 shutdown_connections 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc2406ecc0 0x7fcc24071180 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc3c100220 0x7fcc3c197880 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 --2- 172.21.15.96:0/1648738639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc3c104b30 0x7fcc3c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 >> 172.21.15.96:0/1648738639 conn(0x7fcc3c075f40 msgr2=0x7fcc3c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 shutdown_connections 2024-04-22T15:27:48.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:27:48.502+0000 7fcc426e0700 1 -- 172.21.15.96:0/1648738639 wait complete. 2024-04-22T15:27:48.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:48 smithi196 ceph-mon[81815]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:48.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:48 smithi196 ceph-mon[81815]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:48.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:48 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2443044717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:27:48.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:48 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3253094550' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:27:49.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:49 smithi096 ceph-mon[112443]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:49.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:49 smithi096 ceph-mon[112443]: pgmap v176: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:49.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:49 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1648738639' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:27:49.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:49 smithi196 ceph-mon[81815]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:27:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:49 smithi196 ceph-mon[81815]: pgmap v176: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:49.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:49 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1648738639' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:27:50.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:50 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:boot 2024-04-22T15:27:50.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:50 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:27:50.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:50 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:27:50.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:50 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:50.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:50 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:50.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:50 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:50.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:50 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:boot 2024-04-22T15:27:50.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:50 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:27:50.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:50 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:27:50.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:50 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:50.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:50 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:50.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:50 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:27:51.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:51 smithi096 ceph-mon[112443]: pgmap v177: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:51.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:51 smithi196 ceph-mon[81815]: pgmap v177: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:53.689 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:53 smithi096 ceph-mon[112443]: pgmap v178: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:53.689 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:53.689 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:27:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:53 smithi196 ceph-mon[81815]: pgmap v178: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:27:54.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:54.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:55 smithi096 ceph-mon[112443]: pgmap v179: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:55 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:55 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:55 smithi196 ceph-mon[81815]: pgmap v179: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:55 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:55 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:56 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:56 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:56 smithi096 ceph-mon[112443]: pgmap v180: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:56 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:56 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:27:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:56 smithi196 ceph-mon[81815]: pgmap v180: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:27:59 smithi096 ceph-mon[112443]: pgmap v181: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:27:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:27:59 smithi196 ceph-mon[81815]: pgmap v181: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:28:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:28:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:00.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:28:00.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:00.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:00.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:00.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:00 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:00.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:00.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:28:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:28:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:28:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:00 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:01.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:01 smithi096 ceph-mon[112443]: pgmap v182: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:01.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:01 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:01.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:01 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.drkkdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:28:01.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:01 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:28:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:01 smithi196 ceph-mon[81815]: pgmap v182: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:01.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:01 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:01.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:01 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi096.drkkdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:28:01.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:01 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:28:02.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:02 smithi096 ceph-mon[112443]: Upgrade: Updating mds.cephfs.smithi096.drkkdi 2024-04-22T15:28:02.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:02 smithi096 ceph-mon[112443]: Deploying daemon mds.cephfs.smithi096.drkkdi on smithi096 2024-04-22T15:28:02.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:02 smithi196 ceph-mon[81815]: Upgrade: Updating mds.cephfs.smithi096.drkkdi 2024-04-22T15:28:02.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:02 smithi196 ceph-mon[81815]: Deploying daemon mds.cephfs.smithi096.drkkdi on smithi096 2024-04-22T15:28:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:03 smithi096 ceph-mon[112443]: pgmap v183: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:03 smithi196 ceph-mon[81815]: pgmap v183: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:04.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:04 smithi096 ceph-mon[112443]: osdmap e82: 6 total, 6 up, 6 in 2024-04-22T15:28:04.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:04 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T15:28:04.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:04 smithi196 ceph-mon[81815]: osdmap e82: 6 total, 6 up, 6 in 2024-04-22T15:28:04.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:04 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T15:28:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:05 smithi096 ceph-mon[112443]: pgmap v185: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:05 smithi196 ceph-mon[81815]: pgmap v185: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:28:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:06 smithi096 ceph-mon[112443]: pgmap v186: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:28:06.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:06 smithi196 ceph-mon[81815]: pgmap v186: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:08.580 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:28:08.669 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:28:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:09 smithi096 ceph-mon[112443]: pgmap v187: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:28:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:09 smithi196 ceph-mon[81815]: pgmap v187: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:28:10.616 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:10 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:boot 2024-04-22T15:28:10.616 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:10 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:28:10.616 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:10 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:28:10.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:10 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:boot 2024-04-22T15:28:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:10 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:28:10.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:10 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:28:11.603 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:11 smithi096 ceph-mon[112443]: pgmap v188: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:11 smithi196 ceph-mon[81815]: pgmap v188: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:13.390 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:13 smithi096 ceph-mon[112443]: pgmap v189: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:13 smithi196 ceph-mon[81815]: pgmap v189: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:15 smithi096 ceph-mon[112443]: pgmap v190: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:15 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:15 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:15 smithi196 ceph-mon[81815]: pgmap v190: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:15 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:15 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:16 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:16 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:16 smithi096 ceph-mon[112443]: pgmap v191: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:16.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:16 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:16.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:16 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:16.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:16 smithi196 ceph-mon[81815]: pgmap v191: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 -- 172.21.15.96:0/466883369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8101e70 msgr2=0x7fc3a8102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 --2- 172.21.15.96:0/466883369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8101e70 0x7fc3a8102250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009940 tx=0x7fc39c009c50 comp rx=0 tx=0).stop 2024-04-22T15:28:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 -- 172.21.15.96:0/466883369 shutdown_connections 2024-04-22T15:28:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 --2- 172.21.15.96:0/466883369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a8102790 0x7fc3a810ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 --2- 172.21.15.96:0/466883369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8101e70 0x7fc3a8102250 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 -- 172.21.15.96:0/466883369 >> 172.21.15.96:0/466883369 conn(0x7fc3a80fb680 msgr2=0x7fc3a80fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 -- 172.21.15.96:0/466883369 shutdown_connections 2024-04-22T15:28:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 -- 172.21.15.96:0/466883369 wait complete. 2024-04-22T15:28:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 Processor -- start 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.612+0000 7fc3aed60700 1 -- start start 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3aed60700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a8101e70 0x7fc3a80ffad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3aed60700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8102790 0x7fc3a8100010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3aed60700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8100550 con 0x7fc3a8102790 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3aed60700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8100690 con 0x7fc3a8101e70 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8102790 0x7fc3a8100010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8102790 0x7fc3a8100010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39202/0 (socket says 172.21.15.96:39202) 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a7fff700 1 -- 172.21.15.96:0/2039115070 learned_addr learned my addr 172.21.15.96:0/2039115070 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3acafc700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a8101e70 0x7fc3a80ffad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a7fff700 1 -- 172.21.15.96:0/2039115070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a8101e70 msgr2=0x7fc3a80ffad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a7fff700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a8101e70 0x7fc3a80ffad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a7fff700 1 -- 172.21.15.96:0/2039115070 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c0095f0 con 0x7fc3a8102790 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a7fff700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8102790 0x7fc3a8100010 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc39400d900 tx=0x7fc39400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:18.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a5ffb700 1 -- 172.21.15.96:0/2039115070 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc394009900 con 0x7fc3a8102790 2024-04-22T15:28:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a5ffb700 1 -- 172.21.15.96:0/2039115070 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc394010460 con 0x7fc3a8102790 2024-04-22T15:28:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a81a2d70 con 0x7fc3a8102790 2024-04-22T15:28:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.613+0000 7fc3a5ffb700 1 -- 172.21.15.96:0/2039115070 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc39400f5f0 con 0x7fc3a8102790 2024-04-22T15:28:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.614+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a81a30d0 con 0x7fc3a8102790 2024-04-22T15:28:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.615+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a8069990 con 0x7fc3a8102790 2024-04-22T15:28:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.616+0000 7fc3a5ffb700 1 -- 172.21.15.96:0/2039115070 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc394009a60 con 0x7fc3a8102790 2024-04-22T15:28:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.616+0000 7fc3a5ffb700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39806ecd0 0x7fc398071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.616+0000 7fc3a5ffb700 1 -- 172.21.15.96:0/2039115070 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc39408ea20 con 0x7fc3a8102790 2024-04-22T15:28:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.616+0000 7fc3acafc700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39806ecd0 0x7fc398071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.616+0000 7fc3acafc700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39806ecd0 0x7fc398071190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc39c006010 tx=0x7fc39c005ad0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:18.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.621+0000 7fc3a5ffb700 1 -- 172.21.15.96:0/2039115070 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc394057780 con 0x7fc3a8102790 2024-04-22T15:28:18.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.845+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3a81055c0 con 0x7fc39806ecd0 2024-04-22T15:28:18.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.846+0000 7fc3a5ffb700 1 -- 172.21.15.96:0/2039115070 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc39c01e040 con 0x7fc39806ecd0 2024-04-22T15:28:18.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.848+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39806ecd0 msgr2=0x7fc398071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:18.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.848+0000 7fc3aed60700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39806ecd0 0x7fc398071190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc39c006010 tx=0x7fc39c005ad0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.848+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8102790 msgr2=0x7fc3a8100010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:18.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.848+0000 7fc3aed60700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8102790 0x7fc3a8100010 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc39400d900 tx=0x7fc39400dc10 comp rx=0 tx=0).stop 2024-04-22T15:28:18.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.849+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 shutdown_connections 2024-04-22T15:28:18.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.849+0000 7fc3aed60700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39806ecd0 0x7fc398071190 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.849+0000 7fc3aed60700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a8101e70 0x7fc3a80ffad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.849+0000 7fc3aed60700 1 --2- 172.21.15.96:0/2039115070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a8102790 0x7fc3a8100010 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.849+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 >> 172.21.15.96:0/2039115070 conn(0x7fc3a80fb680 msgr2=0x7fc3a8103160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:18.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.849+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 shutdown_connections 2024-04-22T15:28:18.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.849+0000 7fc3aed60700 1 -- 172.21.15.96:0/2039115070 wait complete. 2024-04-22T15:28:18.866 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.955+0000 7fcf28506700 1 -- 172.21.15.96:0/4072640886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf20105ac0 msgr2=0x7fcf20105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.955+0000 7fcf28506700 1 --2- 172.21.15.96:0/4072640886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf20105ac0 0x7fcf20105ea0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009910 tx=0x7fcf10009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.955+0000 7fcf28506700 1 -- 172.21.15.96:0/4072640886 shutdown_connections 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.955+0000 7fcf28506700 1 --2- 172.21.15.96:0/4072640886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf200ffac0 0x7fcf200fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.955+0000 7fcf28506700 1 --2- 172.21.15.96:0/4072640886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf20105ac0 0x7fcf20105ea0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.955+0000 7fcf28506700 1 -- 172.21.15.96:0/4072640886 >> 172.21.15.96:0/4072640886 conn(0x7fcf200fb5d0 msgr2=0x7fcf200fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.955+0000 7fcf28506700 1 -- 172.21.15.96:0/4072640886 shutdown_connections 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.955+0000 7fcf28506700 1 -- 172.21.15.96:0/4072640886 wait complete. 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf28506700 1 Processor -- start 2024-04-22T15:28:18.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf28506700 1 -- start start 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf28506700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf200ffac0 0x7fcf20198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf28506700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf20105ac0 0x7fcf20199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf28506700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf201999f0 con 0x7fcf200ffac0 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf28506700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2019d6c0 con 0x7fcf20105ac0 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf262a2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf200ffac0 0x7fcf20198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf262a2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf200ffac0 0x7fcf20198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39216/0 (socket says 172.21.15.96:39216) 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf262a2700 1 -- 172.21.15.96:0/1778705517 learned_addr learned my addr 172.21.15.96:0/1778705517 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf25aa1700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf20105ac0 0x7fcf20199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf262a2700 1 -- 172.21.15.96:0/1778705517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf20105ac0 msgr2=0x7fcf20199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf262a2700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf20105ac0 0x7fcf20199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf262a2700 1 -- 172.21.15.96:0/1778705517 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf100095f0 con 0x7fcf200ffac0 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.956+0000 7fcf262a2700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf200ffac0 0x7fcf20198dd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf10006010 tx=0x7fcf1000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.957+0000 7fcf177fe700 1 -- 172.21.15.96:0/1778705517 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf1001d070 con 0x7fcf200ffac0 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.957+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2019d940 con 0x7fcf200ffac0 2024-04-22T15:28:18.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.957+0000 7fcf177fe700 1 -- 172.21.15.96:0/1778705517 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf1000f460 con 0x7fcf200ffac0 2024-04-22T15:28:18.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.957+0000 7fcf177fe700 1 -- 172.21.15.96:0/1778705517 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf10017630 con 0x7fcf200ffac0 2024-04-22T15:28:18.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.957+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2019de30 con 0x7fcf200ffac0 2024-04-22T15:28:18.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.958+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf20101200 con 0x7fcf200ffac0 2024-04-22T15:28:18.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.958+0000 7fcf177fe700 1 -- 172.21.15.96:0/1778705517 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcf1000f5d0 con 0x7fcf200ffac0 2024-04-22T15:28:18.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.958+0000 7fcf177fe700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf0c06ecc0 0x7fcf0c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:18.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.959+0000 7fcf177fe700 1 -- 172.21.15.96:0/1778705517 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcf100903a0 con 0x7fcf200ffac0 2024-04-22T15:28:18.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.959+0000 7fcf25aa1700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf0c06ecc0 0x7fcf0c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:18.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.959+0000 7fcf25aa1700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf0c06ecc0 0x7fcf0c071180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c006fd0 tx=0x7fcf1c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:18.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:18.963+0000 7fcf177fe700 1 -- 172.21.15.96:0/1778705517 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcf100591c0 con 0x7fcf200ffac0 2024-04-22T15:28:19.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.209+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf2010cdb0 con 0x7fcf0c06ecc0 2024-04-22T15:28:19.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.210+0000 7fcf177fe700 1 -- 172.21.15.96:0/1778705517 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcf2010cdb0 con 0x7fcf0c06ecc0 2024-04-22T15:28:19.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf0c06ecc0 msgr2=0x7fcf0c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf0c06ecc0 0x7fcf0c071180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c006fd0 tx=0x7fcf1c008040 comp rx=0 tx=0).stop 2024-04-22T15:28:19.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf200ffac0 msgr2=0x7fcf20198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf200ffac0 0x7fcf20198dd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf10006010 tx=0x7fcf1000be20 comp rx=0 tx=0).stop 2024-04-22T15:28:19.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 shutdown_connections 2024-04-22T15:28:19.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf0c06ecc0 0x7fcf0c071180 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf200ffac0 0x7fcf20198dd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 --2- 172.21.15.96:0/1778705517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf20105ac0 0x7fcf20199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 >> 172.21.15.96:0/1778705517 conn(0x7fcf200fb5d0 msgr2=0x7fcf200fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:19.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 shutdown_connections 2024-04-22T15:28:19.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.212+0000 7fcf28506700 1 -- 172.21.15.96:0/1778705517 wait complete. 2024-04-22T15:28:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.315+0000 7fe18923f700 1 -- 172.21.15.96:0/1498561108 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 msgr2=0x7fe184107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.315+0000 7fe18923f700 1 --2- 172.21.15.96:0/1498561108 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 0x7fe184107150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe174009910 tx=0x7fe174009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.315+0000 7fe18923f700 1 -- 172.21.15.96:0/1498561108 shutdown_connections 2024-04-22T15:28:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.315+0000 7fe18923f700 1 --2- 172.21.15.96:0/1498561108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1840ff3d0 0x7fe1840ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.315+0000 7fe18923f700 1 --2- 172.21.15.96:0/1498561108 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 0x7fe184107150 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.315+0000 7fe18923f700 1 -- 172.21.15.96:0/1498561108 >> 172.21.15.96:0/1498561108 conn(0x7fe184074d90 msgr2=0x7fe1840751a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.315+0000 7fe18923f700 1 -- 172.21.15.96:0/1498561108 shutdown_connections 2024-04-22T15:28:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.315+0000 7fe18923f700 1 -- 172.21.15.96:0/1498561108 wait complete. 2024-04-22T15:28:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.316+0000 7fe18923f700 1 Processor -- start 2024-04-22T15:28:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.316+0000 7fe18923f700 1 -- start start 2024-04-22T15:28:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.316+0000 7fe18923f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1840ff3d0 0x7fe1841005b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.316+0000 7fe18923f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 0x7fe184100af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.316+0000 7fe18923f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe184101030 con 0x7fe184106d70 2024-04-22T15:28:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.316+0000 7fe18923f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1841011a0 con 0x7fe1840ff3d0 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe18259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 0x7fe184100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe18259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 0x7fe184100af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39234/0 (socket says 172.21.15.96:39234) 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe18259c700 1 -- 172.21.15.96:0/2803870739 learned_addr learned my addr 172.21.15.96:0/2803870739 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe182d9d700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1840ff3d0 0x7fe1841005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe18259c700 1 -- 172.21.15.96:0/2803870739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1840ff3d0 msgr2=0x7fe1841005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe18259c700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1840ff3d0 0x7fe1841005b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe18259c700 1 -- 172.21.15.96:0/2803870739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1740095f0 con 0x7fe184106d70 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe18259c700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 0x7fe184100af0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe17800d8d0 tx=0x7fe17800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe16bfff700 1 -- 172.21.15.96:0/2803870739 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1780098a0 con 0x7fe184106d70 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe16bfff700 1 -- 172.21.15.96:0/2803870739 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe178010460 con 0x7fe184106d70 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1841a2dc0 con 0x7fe184106d70 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe16bfff700 1 -- 172.21.15.96:0/2803870739 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe17800f5f0 con 0x7fe184106d70 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.317+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1841a3120 con 0x7fe184106d70 2024-04-22T15:28:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.318+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1841092c0 con 0x7fe184106d70 2024-04-22T15:28:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.319+0000 7fe16bfff700 1 -- 172.21.15.96:0/2803870739 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe178009a00 con 0x7fe184106d70 2024-04-22T15:28:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.319+0000 7fe16bfff700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe16c06ecd0 0x7fe16c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.319+0000 7fe16bfff700 1 -- 172.21.15.96:0/2803870739 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe17808ed80 con 0x7fe184106d70 2024-04-22T15:28:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.319+0000 7fe182d9d700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe16c06ecd0 0x7fe16c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.320+0000 7fe182d9d700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe16c06ecd0 0x7fe16c071190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe174006010 tx=0x7fe17401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:19.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.324+0000 7fe16bfff700 1 -- 172.21.15.96:0/2803870739 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe178057ae0 con 0x7fe184106d70 2024-04-22T15:28:19.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: pgmap v192: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:19.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:19.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:19.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:28:19.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:28:19.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:19.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:28:19.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:19.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:19.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:19.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:19 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:19.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.532+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe184101b30 con 0x7fe16c06ecd0 2024-04-22T15:28:19.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.543+0000 7fe16bfff700 1 -- 172.21.15.96:0/2803870739 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fe184101b30 con 0x7fe16c06ecd0 2024-04-22T15:28:19.546 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (7m) 5s ago 18m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (18m) 5s ago 18m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (17m) 46s ago 17m 8816k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (4m) 5s ago 18m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (4m) 46s ago 17m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (6m) 5s ago 18m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (10s) 5s ago 12m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (30s) 5s ago 12m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (12m) 46s ago 12m 27.1M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 bc83eb943a44 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (12m) 46s ago 12m 26.6M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (9m) 5s ago 19m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (8m) 46s ago 17m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:28:19.547 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (5m) 5s ago 19m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (4m) 46s ago 17m 39.2M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (7m) 5s ago 18m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (7m) 46s ago 17m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (3m) 5s ago 16m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (3m) 5s ago 15m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (2m) 5s ago 15m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (116s) 46s ago 15m 100M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (82s) 46s ago 14m 86.5M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (49s) 46s ago 14m 12.1M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:28:19.548 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (7m) 5s ago 17m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:28:19.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.545+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe16c06ecd0 msgr2=0x7fe16c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.545+0000 7fe18923f700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe16c06ecd0 0x7fe16c071190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe174006010 tx=0x7fe17401a040 comp rx=0 tx=0).stop 2024-04-22T15:28:19.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.546+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 msgr2=0x7fe184100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.546+0000 7fe18923f700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 0x7fe184100af0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe17800d8d0 tx=0x7fe17800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.546+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 shutdown_connections 2024-04-22T15:28:19.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.546+0000 7fe18923f700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe16c06ecd0 0x7fe16c071190 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.546+0000 7fe18923f700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1840ff3d0 0x7fe1841005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.546+0000 7fe18923f700 1 --2- 172.21.15.96:0/2803870739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe184106d70 0x7fe184100af0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.546+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 >> 172.21.15.96:0/2803870739 conn(0x7fe184074d90 msgr2=0x7fe1840fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:19.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.546+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 shutdown_connections 2024-04-22T15:28:19.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.546+0000 7fe18923f700 1 -- 172.21.15.96:0/2803870739 wait complete. 2024-04-22T15:28:19.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.648+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/572788348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 msgr2=0x7f9dc4105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.648+0000 7f9dc9ffe700 1 --2- 172.21.15.96:0/572788348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 0x7f9dc4105e80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9db4009910 tx=0x7f9db4009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:19.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.649+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/572788348 shutdown_connections 2024-04-22T15:28:19.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.649+0000 7f9dc9ffe700 1 --2- 172.21.15.96:0/572788348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9dc40ffaa0 0x7f9dc40fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.649+0000 7f9dc9ffe700 1 --2- 172.21.15.96:0/572788348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 0x7f9dc4105e80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.649+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/572788348 >> 172.21.15.96:0/572788348 conn(0x7f9dc4076740 msgr2=0x7f9dc40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:19.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.649+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/572788348 shutdown_connections 2024-04-22T15:28:19.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.649+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/572788348 wait complete. 2024-04-22T15:28:19.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.649+0000 7f9dc9ffe700 1 Processor -- start 2024-04-22T15:28:19.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.649+0000 7f9dc9ffe700 1 -- start start 2024-04-22T15:28:19.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc9ffe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9dc40ffaa0 0x7f9dc419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:19.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc9ffe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 0x7f9dc419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:19.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc9ffe700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc419d330 con 0x7f9dc4105aa0 2024-04-22T15:28:19.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc9ffe700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4196910 con 0x7f9dc40ffaa0 2024-04-22T15:28:19.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 0x7f9dc419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 0x7f9dc419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39250/0 (socket says 172.21.15.96:39250) 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc2ffd700 1 -- 172.21.15.96:0/1468981036 learned_addr learned my addr 172.21.15.96:0/1468981036 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc37fe700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9dc40ffaa0 0x7f9dc419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc2ffd700 1 -- 172.21.15.96:0/1468981036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9dc40ffaa0 msgr2=0x7f9dc419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc2ffd700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9dc40ffaa0 0x7f9dc419c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc2ffd700 1 -- 172.21.15.96:0/1468981036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db40095f0 con 0x7f9dc4105aa0 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc2ffd700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 0x7f9dc419cd10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9db800d8d0 tx=0x7f9db800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc0ff9700 1 -- 172.21.15.96:0/1468981036 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9db80098a0 con 0x7f9dc4105aa0 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc4196bf0 con 0x7f9dc4105aa0 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc0ff9700 1 -- 172.21.15.96:0/1468981036 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9db8010460 con 0x7f9dc4105aa0 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc0ff9700 1 -- 172.21.15.96:0/1468981036 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9db800f5f0 con 0x7f9dc4105aa0 2024-04-22T15:28:19.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.650+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc4197140 con 0x7f9dc4105aa0 2024-04-22T15:28:19.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.651+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc41011e0 con 0x7f9dc4105aa0 2024-04-22T15:28:19.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.652+0000 7f9dc0ff9700 1 -- 172.21.15.96:0/1468981036 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9db8009a00 con 0x7f9dc4105aa0 2024-04-22T15:28:19.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.653+0000 7f9dc0ff9700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dac06ecc0 0x7f9dac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:19.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.653+0000 7f9dc37fe700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dac06ecc0 0x7f9dac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:19.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.653+0000 7f9dc0ff9700 1 -- 172.21.15.96:0/1468981036 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9db808ed80 con 0x7f9dc4105aa0 2024-04-22T15:28:19.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.653+0000 7f9dc37fe700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dac06ecc0 0x7f9dac071180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9db4000c00 tx=0x7f9db401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:19.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.657+0000 7f9dc0ff9700 1 -- 172.21.15.96:0/1468981036 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9db805b610 con 0x7f9dc4105aa0 2024-04-22T15:28:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: pgmap v192: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:19 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:19.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.940+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9dc410cca0 con 0x7f9dc4105aa0 2024-04-22T15:28:19.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.940+0000 7f9dc0ff9700 1 -- 172.21.15.96:0/1468981036 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f9db805b1a0 con 0x7f9dc4105aa0 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2, 2024-04-22T15:28:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:28:19.945 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:28:19.945 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:28:19.945 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 2, 2024-04-22T15:28:19.945 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 12 2024-04-22T15:28:19.945 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:28:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dac06ecc0 msgr2=0x7f9dac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dac06ecc0 0x7f9dac071180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9db4000c00 tx=0x7f9db401a040 comp rx=0 tx=0).stop 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 msgr2=0x7f9dc419cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 0x7f9dc419cd10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9db800d8d0 tx=0x7f9db800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 shutdown_connections 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dac06ecc0 0x7f9dac071180 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9dc40ffaa0 0x7f9dc419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 --2- 172.21.15.96:0/1468981036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dc4105aa0 0x7f9dc419cd10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 >> 172.21.15.96:0/1468981036 conn(0x7f9dc4076740 msgr2=0x7f9dc40fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 shutdown_connections 2024-04-22T15:28:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:19.942+0000 7f9dc9ffe700 1 -- 172.21.15.96:0/1468981036 wait complete. 2024-04-22T15:28:20.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.047+0000 7f548a260700 1 -- 172.21.15.96:0/2578905924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 msgr2=0x7f5484101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.047+0000 7f548a260700 1 --2- 172.21.15.96:0/2578905924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 0x7f5484101200 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5474009910 tx=0x7f5474009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:20.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.047+0000 7f548a260700 1 -- 172.21.15.96:0/2578905924 shutdown_connections 2024-04-22T15:28:20.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.047+0000 7f548a260700 1 --2- 172.21.15.96:0/2578905924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54841017d0 0x7f5484105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.047+0000 7f548a260700 1 --2- 172.21.15.96:0/2578905924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 0x7f5484101200 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.047+0000 7f548a260700 1 -- 172.21.15.96:0/2578905924 >> 172.21.15.96:0/2578905924 conn(0x7f54840fc6d0 msgr2=0x7f54840feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:20.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.047+0000 7f548a260700 1 -- 172.21.15.96:0/2578905924 shutdown_connections 2024-04-22T15:28:20.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.047+0000 7f548a260700 1 -- 172.21.15.96:0/2578905924 wait complete. 2024-04-22T15:28:20.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f548a260700 1 Processor -- start 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f548a260700 1 -- start start 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f548a260700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 0x7f54840731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f548a260700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54841017d0 0x7f5484073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f548a260700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5484073c60 con 0x7f5484100e20 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f548a260700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5484073da0 con 0x7f54841017d0 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f54837fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 0x7f54840731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f54837fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 0x7f54840731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39270/0 (socket says 172.21.15.96:39270) 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f54837fe700 1 -- 172.21.15.96:0/4178975200 learned_addr learned my addr 172.21.15.96:0/4178975200 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f5482ffd700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54841017d0 0x7f5484073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f54837fe700 1 -- 172.21.15.96:0/4178975200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54841017d0 msgr2=0x7f5484073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f54837fe700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54841017d0 0x7f5484073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.048+0000 7f54837fe700 1 -- 172.21.15.96:0/4178975200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54740095f0 con 0x7f5484100e20 2024-04-22T15:28:20.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.049+0000 7f54837fe700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 0x7f54840731e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f547400bd70 tx=0x7f547400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:20.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.049+0000 7f5480ff9700 1 -- 172.21.15.96:0/4178975200 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f547401d070 con 0x7f5484100e20 2024-04-22T15:28:20.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.049+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5484074020 con 0x7f5484100e20 2024-04-22T15:28:20.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.049+0000 7f5480ff9700 1 -- 172.21.15.96:0/4178975200 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54740056e0 con 0x7f5484100e20 2024-04-22T15:28:20.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.049+0000 7f5480ff9700 1 -- 172.21.15.96:0/4178975200 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5474017400 con 0x7f5484100e20 2024-04-22T15:28:20.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.049+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54841a2fd0 con 0x7f5484100e20 2024-04-22T15:28:20.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.050+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54841091c0 con 0x7f5484100e20 2024-04-22T15:28:20.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.050+0000 7f5480ff9700 1 -- 172.21.15.96:0/4178975200 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f547400f6f0 con 0x7f5484100e20 2024-04-22T15:28:20.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.051+0000 7f5480ff9700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f546c06ecd0 0x7f546c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:20.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.051+0000 7f5480ff9700 1 -- 172.21.15.96:0/4178975200 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f54740902b0 con 0x7f5484100e20 2024-04-22T15:28:20.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.051+0000 7f5482ffd700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f546c06ecd0 0x7f546c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:20.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.051+0000 7f5482ffd700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f546c06ecd0 0x7f546c071190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5484074910 tx=0x7f547800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.056+0000 7f5480ff9700 1 -- 172.21.15.96:0/4178975200 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f54740590d0 con 0x7f5484100e20 2024-04-22T15:28:20.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.301+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f54840746c0 con 0x7f5484100e20 2024-04-22T15:28:20.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.302+0000 7f5480ff9700 1 -- 172.21.15.96:0/4178975200 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f547405c790 con 0x7f5484100e20 2024-04-22T15:28:20.305 INFO:teuthology.orchestra.run.smithi096.stdout:e27 2024-04-22T15:28:20.305 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:28:20.305 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:28:20.306 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:28:20.307 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:28:20.308 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44233} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4222304137,v1:172.21.15.196:6825/4222304137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:28:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f546c06ecd0 msgr2=0x7f546c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f546c06ecd0 0x7f546c071190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5484074910 tx=0x7f547800b410 comp rx=0 tx=0).stop 2024-04-22T15:28:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 msgr2=0x7f54840731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 0x7f54840731e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f547400bd70 tx=0x7f547400bda0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 shutdown_connections 2024-04-22T15:28:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f546c06ecd0 0x7f546c071190 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5484100e20 0x7f54840731e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 --2- 172.21.15.96:0/4178975200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54841017d0 0x7f5484073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 >> 172.21.15.96:0/4178975200 conn(0x7f54840fc6d0 msgr2=0x7f54840fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 shutdown_connections 2024-04-22T15:28:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.304+0000 7f548a260700 1 -- 172.21.15.96:0/4178975200 wait complete. 2024-04-22T15:28:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 27 2024-04-22T15:28:20.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 -- 172.21.15.96:0/3277185219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 msgr2=0x7f3edc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 --2- 172.21.15.96:0/3277185219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 0x7f3edc101200 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009910 tx=0x7f3ecc009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:20.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 -- 172.21.15.96:0/3277185219 shutdown_connections 2024-04-22T15:28:20.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 --2- 172.21.15.96:0/3277185219 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3edc1017d0 0x7f3edc105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 --2- 172.21.15.96:0/3277185219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 0x7f3edc101200 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 -- 172.21.15.96:0/3277185219 >> 172.21.15.96:0/3277185219 conn(0x7f3edc0fc6d0 msgr2=0x7f3edc0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 -- 172.21.15.96:0/3277185219 shutdown_connections 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 -- 172.21.15.96:0/3277185219 wait complete. 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 Processor -- start 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.411+0000 7f3ee2518700 1 -- start start 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3ee2518700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 0x7f3edc19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3ee2518700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3edc1017d0 0x7f3edc19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3ee2518700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc19d3a0 con 0x7f3edc100e20 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3ee2518700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc1969d0 con 0x7f3edc1017d0 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3edbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 0x7f3edc19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3edbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 0x7f3edc19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39292/0 (socket says 172.21.15.96:39292) 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3edbfff700 1 -- 172.21.15.96:0/1683809466 learned_addr learned my addr 172.21.15.96:0/1683809466 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:20.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3edb7fe700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3edc1017d0 0x7f3edc19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3edbfff700 1 -- 172.21.15.96:0/1683809466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3edc1017d0 msgr2=0x7f3edc19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3edbfff700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3edc1017d0 0x7f3edc19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3edbfff700 1 -- 172.21.15.96:0/1683809466 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc0095f0 con 0x7f3edc100e20 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3edbfff700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 0x7f3edc19c840 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc000c00 tx=0x7f3ecc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.412+0000 7f3ed97fa700 1 -- 172.21.15.96:0/1683809466 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ecc01d070 con 0x7f3edc100e20 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.413+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc196c50 con 0x7f3edc100e20 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.413+0000 7f3ed97fa700 1 -- 172.21.15.96:0/1683809466 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ecc0056e0 con 0x7f3edc100e20 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.413+0000 7f3ed97fa700 1 -- 172.21.15.96:0/1683809466 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ecc00fe40 con 0x7f3edc100e20 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.413+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc197140 con 0x7f3edc100e20 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.413+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc1091c0 con 0x7f3edc100e20 2024-04-22T15:28:20.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.414+0000 7f3ed97fa700 1 -- 172.21.15.96:0/1683809466 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3ecc00f680 con 0x7f3edc100e20 2024-04-22T15:28:20.420 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:20 smithi196 ceph-mon[81815]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:20.420 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:20 smithi196 ceph-mon[81815]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:20.420 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:20 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1468981036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:20.420 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:20 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4178975200' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:28:20.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.414+0000 7f3ed97fa700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ec406ec70 0x7f3ec4071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:20.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.415+0000 7f3ed97fa700 1 -- 172.21.15.96:0/1683809466 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3ecc090250 con 0x7f3edc100e20 2024-04-22T15:28:20.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.415+0000 7f3edb7fe700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ec406ec70 0x7f3ec4071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:20.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.415+0000 7f3edb7fe700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ec406ec70 0x7f3ec4071130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3edc197b40 tx=0x7f3ed000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:20.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.419+0000 7f3ed97fa700 1 -- 172.21.15.96:0/1683809466 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3ecc05cae0 con 0x7f3edc100e20 2024-04-22T15:28:20.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.637+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3edc10ddf0 con 0x7f3ec406ec70 2024-04-22T15:28:20.641 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:20 smithi096 ceph-mon[112443]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:20.641 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:20 smithi096 ceph-mon[112443]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:20.641 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:20 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1468981036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:20.641 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:20 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4178975200' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:28:20.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.639+0000 7f3ed97fa700 1 -- 172.21.15.96:0/1683809466 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3edc10ddf0 con 0x7f3ec406ec70 2024-04-22T15:28:20.642 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:28:20.642 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:28:20.642 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:28:20.642 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:28:20.642 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:28:20.642 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:28:20.642 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:28:20.642 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:28:20.643 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:28:20.643 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:28:20.643 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "14/23 daemons upgraded", 2024-04-22T15:28:20.643 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:28:20.643 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:28:20.643 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:28:20.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ec406ec70 msgr2=0x7f3ec4071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ec406ec70 0x7f3ec4071130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3edc197b40 tx=0x7f3ed000b410 comp rx=0 tx=0).stop 2024-04-22T15:28:20.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 msgr2=0x7f3edc19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 0x7f3edc19c840 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc000c00 tx=0x7f3ecc00bd70 comp rx=0 tx=0).stop 2024-04-22T15:28:20.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 shutdown_connections 2024-04-22T15:28:20.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ec406ec70 0x7f3ec4071130 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3edc100e20 0x7f3edc19c840 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 --2- 172.21.15.96:0/1683809466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3edc1017d0 0x7f3edc19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 >> 172.21.15.96:0/1683809466 conn(0x7f3edc0fc6d0 msgr2=0x7f3edc0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:20.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 shutdown_connections 2024-04-22T15:28:20.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.641+0000 7f3ee2518700 1 -- 172.21.15.96:0/1683809466 wait complete. 2024-04-22T15:28:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 -- 172.21.15.96:0/1326144104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 msgr2=0x7f7ba4102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 --2- 172.21.15.96:0/1326144104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 0x7f7ba4102190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009910 tx=0x7f7b98009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 -- 172.21.15.96:0/1326144104 shutdown_connections 2024-04-22T15:28:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 --2- 172.21.15.96:0/1326144104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ba41026d0 0x7f7ba410abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 --2- 172.21.15.96:0/1326144104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 0x7f7ba4102190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 -- 172.21.15.96:0/1326144104 >> 172.21.15.96:0/1326144104 conn(0x7f7ba40fb5e0 msgr2=0x7f7ba40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 -- 172.21.15.96:0/1326144104 shutdown_connections 2024-04-22T15:28:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 -- 172.21.15.96:0/1326144104 wait complete. 2024-04-22T15:28:20.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 Processor -- start 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.745+0000 7f7baadda700 1 -- start start 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7baadda700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 0x7f7ba419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7baadda700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ba41026d0 0x7f7ba419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7baadda700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba419d4b0 con 0x7f7ba4101db0 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7baadda700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba41969d0 con 0x7f7ba41026d0 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba8b76700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 0x7f7ba419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba8b76700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 0x7f7ba419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37672/0 (socket says 172.21.15.96:37672) 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba8b76700 1 -- 172.21.15.96:0/1531473007 learned_addr learned my addr 172.21.15.96:0/1531473007 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba3fff700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ba41026d0 0x7f7ba419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba8b76700 1 -- 172.21.15.96:0/1531473007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ba41026d0 msgr2=0x7f7ba419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba8b76700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ba41026d0 0x7f7ba419ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:20.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba8b76700 1 -- 172.21.15.96:0/1531473007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980095f0 con 0x7f7ba4101db0 2024-04-22T15:28:20.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba8b76700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 0x7f7ba419c920 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b98006010 tx=0x7f7b9800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:20.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba1ffb700 1 -- 172.21.15.96:0/1531473007 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9801d070 con 0x7f7ba4101db0 2024-04-22T15:28:20.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba1ffb700 1 -- 172.21.15.96:0/1531473007 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b9800f460 con 0x7f7ba4101db0 2024-04-22T15:28:20.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba4196c50 con 0x7f7ba4101db0 2024-04-22T15:28:20.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.746+0000 7f7ba1ffb700 1 -- 172.21.15.96:0/1531473007 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b98017630 con 0x7f7ba4101db0 2024-04-22T15:28:20.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.747+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba4197140 con 0x7f7ba4101db0 2024-04-22T15:28:20.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.748+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba40699d0 con 0x7f7ba4101db0 2024-04-22T15:28:20.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.748+0000 7f7ba1ffb700 1 -- 172.21.15.96:0/1531473007 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7b9800f5d0 con 0x7f7ba4101db0 2024-04-22T15:28:20.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.748+0000 7f7ba1ffb700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7b9406ecc0 0x7f7b94071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:20.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.748+0000 7f7ba1ffb700 1 -- 172.21.15.96:0/1531473007 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7b98090330 con 0x7f7ba4101db0 2024-04-22T15:28:20.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.748+0000 7f7ba3fff700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7b9406ecc0 0x7f7b94071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:20.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.749+0000 7f7ba3fff700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7b9406ecc0 0x7f7b94071180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4198140 tx=0x7f7b90005d70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:20.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:20.753+0000 7f7ba1ffb700 1 -- 172.21.15.96:0/1531473007 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7b98059150 con 0x7f7ba4101db0 2024-04-22T15:28:21.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.040+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7ba410ddf0 con 0x7f7ba4101db0 2024-04-22T15:28:21.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.040+0000 7f7ba1ffb700 1 -- 172.21.15.96:0/1531473007 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f7b9805c810 con 0x7f7ba4101db0 2024-04-22T15:28:21.044 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:28:21.044 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:28:21.044 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:28:21.044 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:28:21.044 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:28:21.044 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:28:21.044 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:28:21.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.042+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7b9406ecc0 msgr2=0x7f7b94071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:21.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.042+0000 7f7baadda700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7b9406ecc0 0x7f7b94071180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4198140 tx=0x7f7b90005d70 comp rx=0 tx=0).stop 2024-04-22T15:28:21.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.042+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 msgr2=0x7f7ba419c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:21.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.042+0000 7f7baadda700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 0x7f7ba419c920 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b98006010 tx=0x7f7b9800be20 comp rx=0 tx=0).stop 2024-04-22T15:28:21.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.042+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 shutdown_connections 2024-04-22T15:28:21.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.042+0000 7f7baadda700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7b9406ecc0 0x7f7b94071180 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:21.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.042+0000 7f7baadda700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ba4101db0 0x7f7ba419c920 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:21.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.042+0000 7f7baadda700 1 --2- 172.21.15.96:0/1531473007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ba41026d0 0x7f7ba419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:21.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.042+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 >> 172.21.15.96:0/1531473007 conn(0x7f7ba40fb5e0 msgr2=0x7f7ba4103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:21.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.043+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 shutdown_connections 2024-04-22T15:28:21.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:21.043+0000 7f7baadda700 1 -- 172.21.15.96:0/1531473007 wait complete. 2024-04-22T15:28:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:21 smithi196 ceph-mon[81815]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:21 smithi196 ceph-mon[81815]: pgmap v193: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.hwpcvg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:28:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:21 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:28:21.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:21 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1531473007' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:28:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:21 smithi096 ceph-mon[112443]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:21 smithi096 ceph-mon[112443]: pgmap v193: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.hwpcvg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:28:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:21 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:28:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:21 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1531473007' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:28:22.718 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:22 smithi196 ceph-mon[81815]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:22.718 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:22 smithi196 ceph-mon[81815]: Upgrade: Updating mds.cephfs.smithi196.hwpcvg 2024-04-22T15:28:22.718 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:22 smithi196 ceph-mon[81815]: Deploying daemon mds.cephfs.smithi196.hwpcvg on smithi196 2024-04-22T15:28:22.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:22 smithi096 ceph-mon[112443]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:22.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:22 smithi096 ceph-mon[112443]: Upgrade: Updating mds.cephfs.smithi196.hwpcvg 2024-04-22T15:28:22.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:22 smithi096 ceph-mon[112443]: Deploying daemon mds.cephfs.smithi196.hwpcvg on smithi196 2024-04-22T15:28:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:23 smithi096 ceph-mon[112443]: pgmap v194: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:28:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:23 smithi196 ceph-mon[81815]: pgmap v194: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:28:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:24 smithi096 ceph-mon[112443]: osdmap e83: 6 total, 6 up, 6 in 2024-04-22T15:28:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:24 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T15:28:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:24 smithi196 ceph-mon[81815]: osdmap e83: 6 total, 6 up, 6 in 2024-04-22T15:28:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:24 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T15:28:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:25 smithi096 ceph-mon[112443]: pgmap v196: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:25 smithi196 ceph-mon[81815]: pgmap v196: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:26.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:26 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:28:26.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:26 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:28:27.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:27 smithi096 ceph-mon[112443]: pgmap v197: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:27.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:27 smithi196 ceph-mon[81815]: pgmap v197: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:29.696 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:29 smithi196 ceph-mon[81815]: pgmap v198: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:29.696 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:29 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:29.697 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:29 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:29.697 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:29 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:28:29.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:29 smithi096 ceph-mon[112443]: pgmap v198: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:29.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:29 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:29.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:29 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:29.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:29 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:28:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:30 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] up:boot 2024-04-22T15:28:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:30 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:28:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:30 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:28:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:30 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] up:boot 2024-04-22T15:28:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:30 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:28:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:30 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:28:31.656 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:31 smithi096 ceph-mon[112443]: pgmap v199: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:31 smithi196 ceph-mon[81815]: pgmap v199: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:32 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:32 smithi096 ceph-mon[112443]: pgmap v200: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:32 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:32 smithi196 ceph-mon[81815]: pgmap v200: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:33.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:33.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:33 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:34.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:34.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:33 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:28:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:34 smithi096 ceph-mon[112443]: pgmap v201: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:34 smithi196 ceph-mon[81815]: pgmap v201: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:37 smithi096 ceph-mon[112443]: pgmap v202: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:37 smithi196 ceph-mon[81815]: pgmap v202: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:38.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:28:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:28:39.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:39 smithi096 ceph-mon[112443]: pgmap v203: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:39 smithi196 ceph-mon[81815]: pgmap v203: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:41 smithi096 ceph-mon[112443]: pgmap v204: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:41 smithi196 ceph-mon[81815]: pgmap v204: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:43 smithi096 ceph-mon[112443]: pgmap v205: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:43 smithi196 ceph-mon[81815]: pgmap v205: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:45 smithi096 ceph-mon[112443]: pgmap v206: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:45.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:45 smithi196 ceph-mon[81815]: pgmap v206: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:47 smithi096 ceph-mon[112443]: pgmap v207: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:47 smithi196 ceph-mon[81815]: pgmap v207: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:49 smithi096 ceph-mon[112443]: pgmap v208: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:49 smithi196 ceph-mon[81815]: pgmap v208: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:51.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.161+0000 7f0d10bbb700 1 -- 172.21.15.96:0/1570167026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c1017d0 msgr2=0x7f0d0c105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.161+0000 7f0d10bbb700 1 --2- 172.21.15.96:0/1570167026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c1017d0 0x7f0d0c105820 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d00009910 tx=0x7f0d00009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:51.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.161+0000 7f0d10bbb700 1 -- 172.21.15.96:0/1570167026 shutdown_connections 2024-04-22T15:28:51.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.161+0000 7f0d10bbb700 1 --2- 172.21.15.96:0/1570167026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c1017d0 0x7f0d0c105820 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.161+0000 7f0d10bbb700 1 --2- 172.21.15.96:0/1570167026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d0c100e20 0x7f0d0c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.161+0000 7f0d10bbb700 1 -- 172.21.15.96:0/1570167026 >> 172.21.15.96:0/1570167026 conn(0x7f0d0c0fc6d0 msgr2=0x7f0d0c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.161+0000 7f0d10bbb700 1 -- 172.21.15.96:0/1570167026 shutdown_connections 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.161+0000 7f0d10bbb700 1 -- 172.21.15.96:0/1570167026 wait complete. 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d10bbb700 1 Processor -- start 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d10bbb700 1 -- start start 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d10bbb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c100e20 0x7f0d0c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d10bbb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d0c1017d0 0x7f0d0c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d10bbb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c073c60 con 0x7f0d0c100e20 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d10bbb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c073da0 con 0x7f0d0c1017d0 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d0a59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c100e20 0x7f0d0c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d0a59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c100e20 0x7f0d0c0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35738/0 (socket says 172.21.15.96:35738) 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d0a59c700 1 -- 172.21.15.96:0/2773496308 learned_addr learned my addr 172.21.15.96:0/2773496308 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:51.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d09d9b700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d0c1017d0 0x7f0d0c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d0a59c700 1 -- 172.21.15.96:0/2773496308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d0c1017d0 msgr2=0x7f0d0c073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d0a59c700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d0c1017d0 0x7f0d0c073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.162+0000 7f0d0a59c700 1 -- 172.21.15.96:0/2773496308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d000095f0 con 0x7f0d0c100e20 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.163+0000 7f0d0a59c700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c100e20 0x7f0d0c0731e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00d8d0 tx=0x7f0cfc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.163+0000 7f0cfb7fe700 1 -- 172.21.15.96:0/2773496308 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cfc0098a0 con 0x7f0d0c100e20 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.163+0000 7f0cfb7fe700 1 -- 172.21.15.96:0/2773496308 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cfc010460 con 0x7f0d0c100e20 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.163+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c1a2ce0 con 0x7f0d0c100e20 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.163+0000 7f0cfb7fe700 1 -- 172.21.15.96:0/2773496308 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cfc00f5f0 con 0x7f0d0c100e20 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.163+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c1a3040 con 0x7f0d0c100e20 2024-04-22T15:28:51.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.164+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c109220 con 0x7f0d0c100e20 2024-04-22T15:28:51.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.164+0000 7f0cfb7fe700 1 -- 172.21.15.96:0/2773496308 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0cfc009a00 con 0x7f0d0c100e20 2024-04-22T15:28:51.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.165+0000 7f0cfb7fe700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cf406ecc0 0x7f0cf4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:51.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.165+0000 7f0cfb7fe700 1 -- 172.21.15.96:0/2773496308 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0cfc08ec60 con 0x7f0d0c100e20 2024-04-22T15:28:51.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.165+0000 7f0d09d9b700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cf406ecc0 0x7f0cf4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:51.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.165+0000 7f0d09d9b700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cf406ecc0 0x7f0cf4071180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c074910 tx=0x7f0d0001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:51.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.170+0000 7f0cfb7fe700 1 -- 172.21.15.96:0/2773496308 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0cfc057a40 con 0x7f0d0c100e20 2024-04-22T15:28:51.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.391+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d0c10ddc0 con 0x7f0cf406ecc0 2024-04-22T15:28:51.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.392+0000 7f0cfb7fe700 1 -- 172.21.15.96:0/2773496308 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0d0c10ddc0 con 0x7f0cf406ecc0 2024-04-22T15:28:51.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cf406ecc0 msgr2=0x7f0cf4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cf406ecc0 0x7f0cf4071180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c074910 tx=0x7f0d0001a040 comp rx=0 tx=0).stop 2024-04-22T15:28:51.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c100e20 msgr2=0x7f0d0c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c100e20 0x7f0d0c0731e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00d8d0 tx=0x7f0cfc00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 shutdown_connections 2024-04-22T15:28:51.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cf406ecc0 0x7f0cf4071180 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d0c100e20 0x7f0d0c0731e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 --2- 172.21.15.96:0/2773496308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d0c1017d0 0x7f0d0c073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 >> 172.21.15.96:0/2773496308 conn(0x7f0d0c0fc6d0 msgr2=0x7f0d0c0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:51.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 shutdown_connections 2024-04-22T15:28:51.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.395+0000 7f0d10bbb700 1 -- 172.21.15.96:0/2773496308 wait complete. 2024-04-22T15:28:51.409 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:28:51.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:51 smithi096 ceph-mon[112443]: pgmap v209: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.502+0000 7f97585cb700 1 -- 172.21.15.96:0/2887445873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 msgr2=0x7f97500fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.502+0000 7f97585cb700 1 --2- 172.21.15.96:0/2887445873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 0x7f97500fff30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f974c009910 tx=0x7f974c009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.502+0000 7f97585cb700 1 -- 172.21.15.96:0/2887445873 shutdown_connections 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.502+0000 7f97585cb700 1 --2- 172.21.15.96:0/2887445873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 0x7f97500fff30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.502+0000 7f97585cb700 1 --2- 172.21.15.96:0/2887445873 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9750105ab0 0x7f9750105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.502+0000 7f97585cb700 1 -- 172.21.15.96:0/2887445873 >> 172.21.15.96:0/2887445873 conn(0x7f97500fb5e0 msgr2=0x7f97500fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.502+0000 7f97585cb700 1 -- 172.21.15.96:0/2887445873 shutdown_connections 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.502+0000 7f97585cb700 1 -- 172.21.15.96:0/2887445873 wait complete. 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f97585cb700 1 Processor -- start 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f97585cb700 1 -- start start 2024-04-22T15:28:51.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f97585cb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 0x7f975019b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f97585cb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9750105ab0 0x7f975019bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f97585cb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975019c2e0 con 0x7f97500ffab0 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f97585cb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750195850 con 0x7f9750105ab0 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f9756367700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 0x7f975019b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f9756367700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 0x7f975019b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35748/0 (socket says 172.21.15.96:35748) 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f9756367700 1 -- 172.21.15.96:0/3602776092 learned_addr learned my addr 172.21.15.96:0/3602776092 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f9755b66700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9750105ab0 0x7f975019bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f9756367700 1 -- 172.21.15.96:0/3602776092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9750105ab0 msgr2=0x7f975019bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f9756367700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9750105ab0 0x7f975019bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.503+0000 7f9756367700 1 -- 172.21.15.96:0/3602776092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0095f0 con 0x7f97500ffab0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.504+0000 7f9756367700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 0x7f975019b710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f974000d900 tx=0x7f974000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.504+0000 7f97477fe700 1 -- 172.21.15.96:0/3602776092 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97400041f0 con 0x7f97500ffab0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.504+0000 7f97477fe700 1 -- 172.21.15.96:0/3602776092 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9740004cf0 con 0x7f97500ffab0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.504+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9750195b30 con 0x7f97500ffab0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.504+0000 7f97477fe700 1 -- 172.21.15.96:0/3602776092 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f974000b750 con 0x7f97500ffab0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.504+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9750196080 con 0x7f97500ffab0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.505+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97501080d0 con 0x7f97500ffab0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.505+0000 7f97477fe700 1 -- 172.21.15.96:0/3602776092 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9740004750 con 0x7f97500ffab0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.506+0000 7f97477fe700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f973c06ecc0 0x7f973c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.506+0000 7f97477fe700 1 -- 172.21.15.96:0/3602776092 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f974008eca0 con 0x7f97500ffab0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.506+0000 7f9755b66700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f973c06ecc0 0x7f973c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:51.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.506+0000 7f9755b66700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f973c06ecc0 0x7f973c071180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9750196c70 tx=0x7f974c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:51.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.511+0000 7f97477fe700 1 -- 172.21.15.96:0/3602776092 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9740093070 con 0x7f97500ffab0 2024-04-22T15:28:51.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.731+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f975010ccb0 con 0x7f973c06ecc0 2024-04-22T15:28:51.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.732+0000 7f97477fe700 1 -- 172.21.15.96:0/3602776092 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f975010ccb0 con 0x7f973c06ecc0 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f973c06ecc0 msgr2=0x7f973c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f973c06ecc0 0x7f973c071180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9750196c70 tx=0x7f974c01a040 comp rx=0 tx=0).stop 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 msgr2=0x7f975019b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 0x7f975019b710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f974000d900 tx=0x7f974000dc10 comp rx=0 tx=0).stop 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 shutdown_connections 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f973c06ecc0 0x7f973c071180 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97500ffab0 0x7f975019b710 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 --2- 172.21.15.96:0/3602776092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9750105ab0 0x7f975019bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 >> 172.21.15.96:0/3602776092 conn(0x7f97500fb5e0 msgr2=0x7f97500fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 shutdown_connections 2024-04-22T15:28:51.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.734+0000 7f97585cb700 1 -- 172.21.15.96:0/3602776092 wait complete. 2024-04-22T15:28:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:51 smithi196 ceph-mon[81815]: pgmap v209: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:51.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.838+0000 7fdf57515700 1 -- 172.21.15.96:0/803615838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 msgr2=0x7fdf500fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.838+0000 7fdf57515700 1 --2- 172.21.15.96:0/803615838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 0x7fdf500fff20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009910 tx=0x7fdf4c009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:51.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.838+0000 7fdf57515700 1 -- 172.21.15.96:0/803615838 shutdown_connections 2024-04-22T15:28:51.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.838+0000 7fdf57515700 1 --2- 172.21.15.96:0/803615838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 0x7fdf500fff20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.838+0000 7fdf57515700 1 --2- 172.21.15.96:0/803615838 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf50105aa0 0x7fdf50105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.838+0000 7fdf57515700 1 -- 172.21.15.96:0/803615838 >> 172.21.15.96:0/803615838 conn(0x7fdf50076740 msgr2=0x7fdf500fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:51.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.838+0000 7fdf57515700 1 -- 172.21.15.96:0/803615838 shutdown_connections 2024-04-22T15:28:51.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.838+0000 7fdf57515700 1 -- 172.21.15.96:0/803615838 wait complete. 2024-04-22T15:28:51.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf57515700 1 Processor -- start 2024-04-22T15:28:51.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf57515700 1 -- start start 2024-04-22T15:28:51.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf57515700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 0x7fdf500fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:51.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf57515700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf50105aa0 0x7fdf500fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:51.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf57515700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf500fdbf0 con 0x7fdf500ffaa0 2024-04-22T15:28:51.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf57515700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf500fc6a0 con 0x7fdf50105aa0 2024-04-22T15:28:51.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf552b1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 0x7fdf500fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:51.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf552b1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 0x7fdf500fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35768/0 (socket says 172.21.15.96:35768) 2024-04-22T15:28:51.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf552b1700 1 -- 172.21.15.96:0/618666707 learned_addr learned my addr 172.21.15.96:0/618666707 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:51.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf54ab0700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf50105aa0 0x7fdf500fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:51.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf552b1700 1 -- 172.21.15.96:0/618666707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf50105aa0 msgr2=0x7fdf500fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:51.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf552b1700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf50105aa0 0x7fdf500fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:51.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.839+0000 7fdf552b1700 1 -- 172.21.15.96:0/618666707 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c0095f0 con 0x7fdf500ffaa0 2024-04-22T15:28:51.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.840+0000 7fdf552b1700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 0x7fdf500fbc20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000c930 tx=0x7fdf4000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:51.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.840+0000 7fdf467fc700 1 -- 172.21.15.96:0/618666707 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf40007ad0 con 0x7fdf500ffaa0 2024-04-22T15:28:51.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.840+0000 7fdf467fc700 1 -- 172.21.15.96:0/618666707 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf4000ce80 con 0x7fdf500ffaa0 2024-04-22T15:28:51.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.840+0000 7fdf467fc700 1 -- 172.21.15.96:0/618666707 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf40018630 con 0x7fdf500ffaa0 2024-04-22T15:28:51.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.840+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf500fc950 con 0x7fdf500ffaa0 2024-04-22T15:28:51.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.840+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf501a2f40 con 0x7fdf500ffaa0 2024-04-22T15:28:51.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.841+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf501011e0 con 0x7fdf500ffaa0 2024-04-22T15:28:51.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.841+0000 7fdf467fc700 1 -- 172.21.15.96:0/618666707 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdf40007c30 con 0x7fdf500ffaa0 2024-04-22T15:28:51.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.842+0000 7fdf467fc700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3c06ecc0 0x7fdf3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:51.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.842+0000 7fdf467fc700 1 -- 172.21.15.96:0/618666707 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fdf4008eef0 con 0x7fdf500ffaa0 2024-04-22T15:28:51.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.842+0000 7fdf54ab0700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3c06ecc0 0x7fdf3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:51.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.842+0000 7fdf54ab0700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3c06ecc0 0x7fdf3c071180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf500fd350 tx=0x7fdf4c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:51.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:51.846+0000 7fdf467fc700 1 -- 172.21.15.96:0/618666707 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdf40057c50 con 0x7fdf500ffaa0 2024-04-22T15:28:52.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.056+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf5010cd60 con 0x7fdf3c06ecc0 2024-04-22T15:28:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.066+0000 7fdf467fc700 1 -- 172.21.15.96:0/618666707 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fdf5010cd60 con 0x7fdf3c06ecc0 2024-04-22T15:28:52.070 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:28:52.070 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (7m) 37s ago 18m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:28:52.070 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (19m) 37s ago 19m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:28:52.070 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (18m) 20s ago 18m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:28:52.070 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (4m) 37s ago 19m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:28:52.070 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (4m) 20s ago 17m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:28:52.070 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (7m) 37s ago 18m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:28:52.070 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (43s) 37s ago 13m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (63s) 37s ago 13m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (23s) 20s ago 13m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (13m) 20s ago 13m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (9m) 37s ago 20m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (8m) 20s ago 17m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (5m) 37s ago 20m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (5m) 20s ago 17m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (8m) 37s ago 19m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (8m) 20s ago 17m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (4m) 37s ago 16m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (3m) 37s ago 16m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (3m) 37s ago 15m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (2m) 20s ago 15m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:28:52.071 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (115s) 20s ago 15m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:28:52.072 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (82s) 20s ago 14m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:28:52.072 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (7m) 37s ago 18m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:28:52.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.068+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3c06ecc0 msgr2=0x7fdf3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.068+0000 7fdf57515700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3c06ecc0 0x7fdf3c071180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf500fd350 tx=0x7fdf4c01a040 comp rx=0 tx=0).stop 2024-04-22T15:28:52.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.069+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 msgr2=0x7fdf500fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.069+0000 7fdf57515700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 0x7fdf500fbc20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000c930 tx=0x7fdf4000cc40 comp rx=0 tx=0).stop 2024-04-22T15:28:52.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.069+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 shutdown_connections 2024-04-22T15:28:52.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.069+0000 7fdf57515700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3c06ecc0 0x7fdf3c071180 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.069+0000 7fdf57515700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf500ffaa0 0x7fdf500fbc20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.069+0000 7fdf57515700 1 --2- 172.21.15.96:0/618666707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf50105aa0 0x7fdf500fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.069+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 >> 172.21.15.96:0/618666707 conn(0x7fdf50076740 msgr2=0x7fdf50104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:52.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.069+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 shutdown_connections 2024-04-22T15:28:52.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.069+0000 7fdf57515700 1 -- 172.21.15.96:0/618666707 wait complete. 2024-04-22T15:28:52.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.170+0000 7f847422f700 1 -- 172.21.15.96:0/2261861107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c102690 msgr2=0x7f846c10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.170+0000 7f847422f700 1 --2- 172.21.15.96:0/2261861107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c102690 0x7f846c10ab80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f845c009910 tx=0x7f845c009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.170+0000 7f847422f700 1 -- 172.21.15.96:0/2261861107 shutdown_connections 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.170+0000 7f847422f700 1 --2- 172.21.15.96:0/2261861107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c102690 0x7f846c10ab80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.170+0000 7f847422f700 1 --2- 172.21.15.96:0/2261861107 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f846c101ce0 0x7f846c1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.170+0000 7f847422f700 1 -- 172.21.15.96:0/2261861107 >> 172.21.15.96:0/2261861107 conn(0x7f846c0fb5e0 msgr2=0x7f846c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.170+0000 7f847422f700 1 -- 172.21.15.96:0/2261861107 shutdown_connections 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.170+0000 7f847422f700 1 -- 172.21.15.96:0/2261861107 wait complete. 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.171+0000 7f847422f700 1 Processor -- start 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.171+0000 7f847422f700 1 -- start start 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.171+0000 7f847422f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101ce0 0x7f846c0ff9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.171+0000 7f847422f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f846c102690 0x7f846c0fff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:52.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.171+0000 7f847422f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c100450 con 0x7f846c101ce0 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.171+0000 7f847422f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c100590 con 0x7f846c102690 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.171+0000 7f8471fcb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101ce0 0x7f846c0ff9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.171+0000 7f8471fcb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101ce0 0x7f846c0ff9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35794/0 (socket says 172.21.15.96:35794) 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.171+0000 7f8471fcb700 1 -- 172.21.15.96:0/3839956692 learned_addr learned my addr 172.21.15.96:0/3839956692 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f84717ca700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f846c102690 0x7f846c0fff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f8471fcb700 1 -- 172.21.15.96:0/3839956692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f846c102690 msgr2=0x7f846c0fff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f8471fcb700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f846c102690 0x7f846c0fff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f8471fcb700 1 -- 172.21.15.96:0/3839956692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c0095f0 con 0x7f846c101ce0 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f8471fcb700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101ce0 0x7f846c0ff9d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f846400d8d0 tx=0x7f846400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f8462ffd700 1 -- 172.21.15.96:0/3839956692 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8464009940 con 0x7f846c101ce0 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c1a2cb0 con 0x7f846c101ce0 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f8462ffd700 1 -- 172.21.15.96:0/3839956692 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8464010460 con 0x7f846c101ce0 2024-04-22T15:28:52.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f8462ffd700 1 -- 172.21.15.96:0/3839956692 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f846400f5f0 con 0x7f846c101ce0 2024-04-22T15:28:52.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.172+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846c1a3010 con 0x7f846c101ce0 2024-04-22T15:28:52.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.173+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846c069730 con 0x7f846c101ce0 2024-04-22T15:28:52.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.174+0000 7f8462ffd700 1 -- 172.21.15.96:0/3839956692 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f84640105d0 con 0x7f846c101ce0 2024-04-22T15:28:52.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.175+0000 7f8462ffd700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f845806ecc0 0x7f8458071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:52.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.175+0000 7f84717ca700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f845806ecc0 0x7f8458071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:52.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.175+0000 7f8462ffd700 1 -- 172.21.15.96:0/3839956692 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f846408eee0 con 0x7f846c101ce0 2024-04-22T15:28:52.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.175+0000 7f84717ca700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f845806ecc0 0x7f8458071180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f846c101100 tx=0x7f845c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:52.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.179+0000 7f8462ffd700 1 -- 172.21.15.96:0/3839956692 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8464057d70 con 0x7f846c101ce0 2024-04-22T15:28:52.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.475+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f846c04ebb0 con 0x7f846c101ce0 2024-04-22T15:28:52.479 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:52 smithi096 ceph-mon[112443]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:52.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.476+0000 7f8462ffd700 1 -- 172.21.15.96:0/3839956692 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f846405b430 con 0x7f846c101ce0 2024-04-22T15:28:52.479 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:28:52.480 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:28:52.481 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:28:52.481 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:28:52.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f845806ecc0 msgr2=0x7f8458071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f845806ecc0 0x7f8458071180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f846c101100 tx=0x7f845c01a040 comp rx=0 tx=0).stop 2024-04-22T15:28:52.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101ce0 msgr2=0x7f846c0ff9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101ce0 0x7f846c0ff9d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f846400d8d0 tx=0x7f846400dc90 comp rx=0 tx=0).stop 2024-04-22T15:28:52.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 shutdown_connections 2024-04-22T15:28:52.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f845806ecc0 0x7f8458071180 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101ce0 0x7f846c0ff9d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 --2- 172.21.15.96:0/3839956692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f846c102690 0x7f846c0fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 >> 172.21.15.96:0/3839956692 conn(0x7f846c0fb5e0 msgr2=0x7f846c103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:52.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 shutdown_connections 2024-04-22T15:28:52.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.478+0000 7f847422f700 1 -- 172.21.15.96:0/3839956692 wait complete. 2024-04-22T15:28:52.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.582+0000 7f74c135c700 1 -- 172.21.15.96:0/1013758960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 msgr2=0x7f74bc105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.582+0000 7f74c135c700 1 --2- 172.21.15.96:0/1013758960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 0x7f74bc105820 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009910 tx=0x7f74ac009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:52.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.582+0000 7f74c135c700 1 -- 172.21.15.96:0/1013758960 shutdown_connections 2024-04-22T15:28:52.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.582+0000 7f74c135c700 1 --2- 172.21.15.96:0/1013758960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 0x7f74bc105820 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.582+0000 7f74c135c700 1 --2- 172.21.15.96:0/1013758960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74bc100e20 0x7f74bc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.582+0000 7f74c135c700 1 -- 172.21.15.96:0/1013758960 >> 172.21.15.96:0/1013758960 conn(0x7f74bc0fc6d0 msgr2=0x7f74bc0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:52.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.582+0000 7f74c135c700 1 -- 172.21.15.96:0/1013758960 shutdown_connections 2024-04-22T15:28:52.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.582+0000 7f74c135c700 1 -- 172.21.15.96:0/1013758960 wait complete. 2024-04-22T15:28:52.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74c135c700 1 Processor -- start 2024-04-22T15:28:52.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74c135c700 1 -- start start 2024-04-22T15:28:52.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74c135c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74bc100e20 0x7f74bc19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74c135c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 0x7f74bc19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74c135c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc19d380 con 0x7f74bc1017d0 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74c135c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc196960 con 0x7f74bc100e20 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74ba7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 0x7f74bc19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74ba7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 0x7f74bc19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35808/0 (socket says 172.21.15.96:35808) 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74ba7fc700 1 -- 172.21.15.96:0/2764333856 learned_addr learned my addr 172.21.15.96:0/2764333856 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.583+0000 7f74baffd700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74bc100e20 0x7f74bc19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.584+0000 7f74ba7fc700 1 -- 172.21.15.96:0/2764333856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74bc100e20 msgr2=0x7f74bc19c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.584+0000 7f74ba7fc700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74bc100e20 0x7f74bc19c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.584+0000 7f74ba7fc700 1 -- 172.21.15.96:0/2764333856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac0095f0 con 0x7f74bc1017d0 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.584+0000 7f74ba7fc700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 0x7f74bc19cd60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00bd70 tx=0x7f74ac00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.584+0000 7f74a3fff700 1 -- 172.21.15.96:0/2764333856 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74ac01d070 con 0x7f74bc1017d0 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.584+0000 7f74a3fff700 1 -- 172.21.15.96:0/2764333856 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74ac0056e0 con 0x7f74bc1017d0 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.584+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74bc196be0 con 0x7f74bc1017d0 2024-04-22T15:28:52.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.584+0000 7f74a3fff700 1 -- 172.21.15.96:0/2764333856 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74ac00fe50 con 0x7f74bc1017d0 2024-04-22T15:28:52.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.584+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74bc1970d0 con 0x7f74bc1017d0 2024-04-22T15:28:52.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.585+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74bc109220 con 0x7f74bc1017d0 2024-04-22T15:28:52.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.585+0000 7f74a3fff700 1 -- 172.21.15.96:0/2764333856 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f74ac003a60 con 0x7f74bc1017d0 2024-04-22T15:28:52.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.586+0000 7f74a3fff700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74a406ecc0 0x7f74a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:52.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.586+0000 7f74baffd700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74a406ecc0 0x7f74a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:52.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.586+0000 7f74a3fff700 1 -- 172.21.15.96:0/2764333856 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f74ac090140 con 0x7f74bc1017d0 2024-04-22T15:28:52.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.586+0000 7f74baffd700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74a406ecc0 0x7f74a4071180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f74b0005950 tx=0x7f74b000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:52.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.591+0000 7f74a3fff700 1 -- 172.21.15.96:0/2764333856 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f74ac05c9d0 con 0x7f74bc1017d0 2024-04-22T15:28:52.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:52 smithi196 ceph-mon[81815]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:52.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.835+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f74bc102300 con 0x7f74bc1017d0 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.835+0000 7f74a3fff700 1 -- 172.21.15.96:0/2764333856 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f74ac05c560 con 0x7f74bc1017d0 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:28:52.839 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:28:52.841 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:28:52.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.837+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74a406ecc0 msgr2=0x7f74a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.837+0000 7f74c135c700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74a406ecc0 0x7f74a4071180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f74b0005950 tx=0x7f74b000b410 comp rx=0 tx=0).stop 2024-04-22T15:28:52.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.837+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 msgr2=0x7f74bc19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.837+0000 7f74c135c700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 0x7f74bc19cd60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00bd70 tx=0x7f74ac00bda0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.837+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 shutdown_connections 2024-04-22T15:28:52.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.837+0000 7f74c135c700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74a406ecc0 0x7f74a4071180 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.837+0000 7f74c135c700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74bc100e20 0x7f74bc19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.838+0000 7f74c135c700 1 --2- 172.21.15.96:0/2764333856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74bc1017d0 0x7f74bc19cd60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.838+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 >> 172.21.15.96:0/2764333856 conn(0x7f74bc0fc6d0 msgr2=0x7f74bc0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:52.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.838+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 shutdown_connections 2024-04-22T15:28:52.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.838+0000 7f74c135c700 1 -- 172.21.15.96:0/2764333856 wait complete. 2024-04-22T15:28:52.843 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:28:52.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.942+0000 7f631286d700 1 -- 172.21.15.96:0/987509132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 msgr2=0x7f630c0ff8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.942+0000 7f631286d700 1 --2- 172.21.15.96:0/987509132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 0x7f630c0ff8c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00aa20 tx=0x7f62fc00ad30 comp rx=0 tx=0).stop 2024-04-22T15:28:52.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.942+0000 7f631286d700 1 -- 172.21.15.96:0/987509132 shutdown_connections 2024-04-22T15:28:52.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.942+0000 7f631286d700 1 --2- 172.21.15.96:0/987509132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f630c0ffe90 0x7f630c103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.942+0000 7f631286d700 1 --2- 172.21.15.96:0/987509132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 0x7f630c0ff8c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.942+0000 7f631286d700 1 -- 172.21.15.96:0/987509132 >> 172.21.15.96:0/987509132 conn(0x7f630c075f40 msgr2=0x7f630c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:52.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.942+0000 7f631286d700 1 -- 172.21.15.96:0/987509132 shutdown_connections 2024-04-22T15:28:52.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.942+0000 7f631286d700 1 -- 172.21.15.96:0/987509132 wait complete. 2024-04-22T15:28:52.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.943+0000 7f631286d700 1 Processor -- start 2024-04-22T15:28:52.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.943+0000 7f631286d700 1 -- start start 2024-04-22T15:28:52.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.943+0000 7f631286d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 0x7f630c19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:52.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.943+0000 7f631286d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f630c0ffe90 0x7f630c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:52.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.943+0000 7f631286d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c19b280 con 0x7f630c0ff4e0 2024-04-22T15:28:52.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.943+0000 7f631286d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c1947f0 con 0x7f630c0ffe90 2024-04-22T15:28:52.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f630bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 0x7f630c19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:52.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f630bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 0x7f630c19a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35822/0 (socket says 172.21.15.96:35822) 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f630bfff700 1 -- 172.21.15.96:0/739135857 learned_addr learned my addr 172.21.15.96:0/739135857 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f630b7fe700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f630c0ffe90 0x7f630c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f630bfff700 1 -- 172.21.15.96:0/739135857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f630c0ffe90 msgr2=0x7f630c19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f630bfff700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f630c0ffe90 0x7f630c19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f630bfff700 1 -- 172.21.15.96:0/739135857 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc00a700 con 0x7f630c0ff4e0 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f630b7fe700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f630c0ffe90 0x7f630c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f630bfff700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 0x7f630c19a6b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f62fc006010 tx=0x7f62fc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f63097fa700 1 -- 172.21.15.96:0/739135857 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62fc004ca0 con 0x7f630c0ff4e0 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f63097fa700 1 -- 172.21.15.96:0/739135857 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62fc004e00 con 0x7f630c0ff4e0 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630c194a70 con 0x7f630c0ff4e0 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f63097fa700 1 -- 172.21.15.96:0/739135857 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62fc016890 con 0x7f630c0ff4e0 2024-04-22T15:28:52.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.944+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630c194f60 con 0x7f630c0ff4e0 2024-04-22T15:28:52.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.945+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c107880 con 0x7f630c0ff4e0 2024-04-22T15:28:52.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.946+0000 7f63097fa700 1 -- 172.21.15.96:0/739135857 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f62fc016ac0 con 0x7f630c0ff4e0 2024-04-22T15:28:52.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.946+0000 7f63097fa700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62f406ecc0 0x7f62f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:52.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.946+0000 7f63097fa700 1 -- 172.21.15.96:0/739135857 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f62fc012070 con 0x7f630c0ff4e0 2024-04-22T15:28:52.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.946+0000 7f630b7fe700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62f406ecc0 0x7f62f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:52.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.946+0000 7f630b7fe700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62f406ecc0 0x7f62f4071180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f630c195c10 tx=0x7f6300008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:52.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:52.951+0000 7f63097fa700 1 -- 172.21.15.96:0/739135857 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f62fc05e880 con 0x7f630c0ff4e0 2024-04-22T15:28:53.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.168+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f630c195cb0 con 0x7f62f406ecc0 2024-04-22T15:28:53.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.169+0000 7f63097fa700 1 -- 172.21.15.96:0/739135857 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f630c195cb0 con 0x7f62f406ecc0 2024-04-22T15:28:53.172 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:28:53.172 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:28:53.172 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:28:53.172 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:28:53.173 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62f406ecc0 msgr2=0x7f62f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62f406ecc0 0x7f62f4071180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f630c195c10 tx=0x7f6300008040 comp rx=0 tx=0).stop 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 msgr2=0x7f630c19a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 0x7f630c19a6b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f62fc006010 tx=0x7f62fc004740 comp rx=0 tx=0).stop 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 shutdown_connections 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62f406ecc0 0x7f62f4071180 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630c0ff4e0 0x7f630c19a6b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 --2- 172.21.15.96:0/739135857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f630c0ffe90 0x7f630c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 >> 172.21.15.96:0/739135857 conn(0x7f630c075f40 msgr2=0x7f630c102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 shutdown_connections 2024-04-22T15:28:53.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.171+0000 7f631286d700 1 -- 172.21.15.96:0/739135857 wait complete. 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.275+0000 7fe5a6351700 1 -- 172.21.15.96:0/2650970139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 msgr2=0x7fe5a010ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.275+0000 7fe5a6351700 1 --2- 172.21.15.96:0/2650970139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 0x7fe5a010ac10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe590009910 tx=0x7fe590009c20 comp rx=0 tx=0).stop 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.276+0000 7fe5a6351700 1 -- 172.21.15.96:0/2650970139 shutdown_connections 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.276+0000 7fe5a6351700 1 --2- 172.21.15.96:0/2650970139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 0x7fe5a010ac10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.276+0000 7fe5a6351700 1 --2- 172.21.15.96:0/2650970139 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a0101e00 0x7fe5a01021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.276+0000 7fe5a6351700 1 -- 172.21.15.96:0/2650970139 >> 172.21.15.96:0/2650970139 conn(0x7fe5a00fb670 msgr2=0x7fe5a00fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.276+0000 7fe5a6351700 1 -- 172.21.15.96:0/2650970139 shutdown_connections 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.276+0000 7fe5a6351700 1 -- 172.21.15.96:0/2650970139 wait complete. 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.276+0000 7fe5a6351700 1 Processor -- start 2024-04-22T15:28:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.276+0000 7fe5a6351700 1 -- start start 2024-04-22T15:28:53.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe5a6351700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a0101e00 0x7fe5a00ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:53.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe5a6351700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 0x7fe5a00fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:53.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe5a6351700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a0101a30 con 0x7fe5a0102720 2024-04-22T15:28:53.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe5a6351700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a01004e0 con 0x7fe5a0101e00 2024-04-22T15:28:53.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe59f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 0x7fe5a00fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:53.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe59f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 0x7fe5a00fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35826/0 (socket says 172.21.15.96:35826) 2024-04-22T15:28:53.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe59f7fe700 1 -- 172.21.15.96:0/4120638591 learned_addr learned my addr 172.21.15.96:0/4120638591 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe59ffff700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a0101e00 0x7fe5a00ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe59f7fe700 1 -- 172.21.15.96:0/4120638591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a0101e00 msgr2=0x7fe5a00ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe59f7fe700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a0101e00 0x7fe5a00ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.277+0000 7fe59f7fe700 1 -- 172.21.15.96:0/4120638591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5900095f0 con 0x7fe5a0102720 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.278+0000 7fe59f7fe700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 0x7fe5a00fffa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5900092e0 tx=0x7fe59000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.278+0000 7fe59d7fa700 1 -- 172.21.15.96:0/4120638591 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe59001d070 con 0x7fe5a0102720 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.278+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a0100730 con 0x7fe5a0102720 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.278+0000 7fe59d7fa700 1 -- 172.21.15.96:0/4120638591 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe590003d50 con 0x7fe5a0102720 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.278+0000 7fe59d7fa700 1 -- 172.21.15.96:0/4120638591 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe590017400 con 0x7fe5a0102720 2024-04-22T15:28:53.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.278+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a01a2fb0 con 0x7fe5a0102720 2024-04-22T15:28:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.279+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a00699b0 con 0x7fe5a0102720 2024-04-22T15:28:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.279+0000 7fe59d7fa700 1 -- 172.21.15.96:0/4120638591 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe590003890 con 0x7fe5a0102720 2024-04-22T15:28:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.280+0000 7fe59d7fa700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe58806ecc0 0x7fe588071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:28:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.280+0000 7fe59d7fa700 1 -- 172.21.15.96:0/4120638591 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe590090270 con 0x7fe5a0102720 2024-04-22T15:28:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.280+0000 7fe59ffff700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe58806ecc0 0x7fe588071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:28:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.280+0000 7fe59ffff700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe58806ecc0 0x7fe588071180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe594005ea0 tx=0x7fe594005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:28:53.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.285+0000 7fe59d7fa700 1 -- 172.21.15.96:0/4120638591 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe590058fd0 con 0x7fe5a0102720 2024-04-22T15:28:53.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:53 smithi096 ceph-mon[112443]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:53.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:53 smithi096 ceph-mon[112443]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:53.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:53 smithi096 ceph-mon[112443]: pgmap v210: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:53.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3839956692' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:53.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2764333856' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:28:53.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:28:53.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.570+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe5a0105510 con 0x7fe5a0102720 2024-04-22T15:28:53.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.571+0000 7fe59d7fa700 1 -- 172.21.15.96:0/4120638591 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fe59007be60 con 0x7fe5a0102720 2024-04-22T15:28:53.574 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:28:53.574 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:28:53.574 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:28:53.574 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:28:53.574 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:28:53.574 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:28:53.574 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:28:53.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe58806ecc0 msgr2=0x7fe588071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:53.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe58806ecc0 0x7fe588071180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe594005ea0 tx=0x7fe594005e30 comp rx=0 tx=0).stop 2024-04-22T15:28:53.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 msgr2=0x7fe5a00fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:28:53.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 0x7fe5a00fffa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5900092e0 tx=0x7fe59000be20 comp rx=0 tx=0).stop 2024-04-22T15:28:53.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 shutdown_connections 2024-04-22T15:28:53.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe58806ecc0 0x7fe588071180 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:53.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a0101e00 0x7fe5a00ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:53.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 --2- 172.21.15.96:0/4120638591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5a0102720 0x7fe5a00fffa0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:28:53.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 >> 172.21.15.96:0/4120638591 conn(0x7fe5a00fb670 msgr2=0x7fe5a01030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:28:53.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 shutdown_connections 2024-04-22T15:28:53.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:28:53.573+0000 7fe5a6351700 1 -- 172.21.15.96:0/4120638591 wait complete. 2024-04-22T15:28:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:53 smithi196 ceph-mon[81815]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:53 smithi196 ceph-mon[81815]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:53 smithi196 ceph-mon[81815]: pgmap v210: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3839956692' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:28:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2764333856' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:28:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:28:54.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:54 smithi096 ceph-mon[112443]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:54.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:54 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4120638591' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:28:54.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:54 smithi196 ceph-mon[81815]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:28:54.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:54 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4120638591' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:28:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:55 smithi096 ceph-mon[112443]: pgmap v211: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:55 smithi196 ceph-mon[81815]: pgmap v211: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:57 smithi096 ceph-mon[112443]: pgmap v212: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:57 smithi196 ceph-mon[81815]: pgmap v212: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:28:59 smithi096 ceph-mon[112443]: pgmap v213: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:28:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:28:59 smithi196 ceph-mon[81815]: pgmap v213: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:01.739 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:01 smithi096 ceph-mon[112443]: pgmap v214: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:01.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:01 smithi196 ceph-mon[81815]: pgmap v214: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:03 smithi096 ceph-mon[112443]: pgmap v215: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:03 smithi196 ceph-mon[81815]: pgmap v215: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:05 smithi096 ceph-mon[112443]: pgmap v216: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:05 smithi196 ceph-mon[81815]: pgmap v216: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:07 smithi096 ceph-mon[112443]: pgmap v217: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:07 smithi196 ceph-mon[81815]: pgmap v217: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:08.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:29:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:29:09.674 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:09 smithi196 ceph-mon[81815]: pgmap v218: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:09 smithi096 ceph-mon[112443]: pgmap v218: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:11.705 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:11 smithi196 ceph-mon[81815]: pgmap v219: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:11 smithi096 ceph-mon[112443]: pgmap v219: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:13 smithi096 ceph-mon[112443]: pgmap v220: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:13 smithi196 ceph-mon[81815]: pgmap v220: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:15 smithi096 ceph-mon[112443]: pgmap v221: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:15 smithi196 ceph-mon[81815]: pgmap v221: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:17.675 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:17 smithi096 ceph-mon[112443]: pgmap v222: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:17 smithi196 ceph-mon[81815]: pgmap v222: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:19 smithi096 ceph-mon[112443]: pgmap v223: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:19 smithi196 ceph-mon[81815]: pgmap v223: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:21 smithi096 ceph-mon[112443]: pgmap v224: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:21.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:21 smithi196 ceph-mon[81815]: pgmap v224: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:23.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.690+0000 7f99689fe700 1 -- 172.21.15.96:0/1252609542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 msgr2=0x7f9964105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:23.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.690+0000 7f99689fe700 1 --2- 172.21.15.96:0/1252609542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 0x7f9964105f40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9954009910 tx=0x7f9954009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:23.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.690+0000 7f99689fe700 1 -- 172.21.15.96:0/1252609542 shutdown_connections 2024-04-22T15:29:23.694 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:23 smithi096 ceph-mon[112443]: pgmap v225: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:23.694 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:29:23.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.690+0000 7f99689fe700 1 --2- 172.21.15.96:0/1252609542 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99640ffad0 0x7f99640fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:23.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.690+0000 7f99689fe700 1 --2- 172.21.15.96:0/1252609542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 0x7f9964105f40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:23.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.690+0000 7f99689fe700 1 -- 172.21.15.96:0/1252609542 >> 172.21.15.96:0/1252609542 conn(0x7f99640fb5e0 msgr2=0x7f99640fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.690+0000 7f99689fe700 1 -- 172.21.15.96:0/1252609542 shutdown_connections 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.690+0000 7f99689fe700 1 -- 172.21.15.96:0/1252609542 wait complete. 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f99689fe700 1 Processor -- start 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f99689fe700 1 -- start start 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f99689fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99640ffad0 0x7f996419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f99689fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 0x7f996419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f99689fe700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996419d300 con 0x7f9964105b60 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f99689fe700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964196970 con 0x7f99640ffad0 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f9961d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 0x7f996419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f9961d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 0x7f996419cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47546/0 (socket says 172.21.15.96:47546) 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f9961d9b700 1 -- 172.21.15.96:0/2793161203 learned_addr learned my addr 172.21.15.96:0/2793161203 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.691+0000 7f996259c700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99640ffad0 0x7f996419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.692+0000 7f9961d9b700 1 -- 172.21.15.96:0/2793161203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99640ffad0 msgr2=0x7f996419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.692+0000 7f9961d9b700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99640ffad0 0x7f996419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.692+0000 7f9961d9b700 1 -- 172.21.15.96:0/2793161203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99540095f0 con 0x7f9964105b60 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.692+0000 7f9961d9b700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 0x7f996419cd70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f995800c930 tx=0x7f995800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.692+0000 7f99537fe700 1 -- 172.21.15.96:0/2793161203 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9958007ad0 con 0x7f9964105b60 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.692+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9964196c50 con 0x7f9964105b60 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.692+0000 7f99537fe700 1 -- 172.21.15.96:0/2793161203 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f995800ce80 con 0x7f9964105b60 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.692+0000 7f99537fe700 1 -- 172.21.15.96:0/2793161203 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9958018630 con 0x7f9964105b60 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.692+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99641971a0 con 0x7f9964105b60 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.693+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99641090e0 con 0x7f9964105b60 2024-04-22T15:29:23.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.693+0000 7f99537fe700 1 -- 172.21.15.96:0/2793161203 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9958007c30 con 0x7f9964105b60 2024-04-22T15:29:23.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.694+0000 7f99537fe700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f994c06ecd0 0x7f994c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:23.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.694+0000 7f99537fe700 1 -- 172.21.15.96:0/2793161203 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f995808eef0 con 0x7f9964105b60 2024-04-22T15:29:23.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.694+0000 7f996259c700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f994c06ecd0 0x7f994c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:23.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.694+0000 7f996259c700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f994c06ecd0 0x7f994c071190 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9954006010 tx=0x7f995401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.699+0000 7f99537fe700 1 -- 172.21.15.96:0/2793161203 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f995805b780 con 0x7f9964105b60 2024-04-22T15:29:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:23 smithi196 ceph-mon[81815]: pgmap v225: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:29:23.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.916+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9964197ea0 con 0x7f994c06ecd0 2024-04-22T15:29:23.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.917+0000 7f99537fe700 1 -- 172.21.15.96:0/2793161203 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9964197ea0 con 0x7f994c06ecd0 2024-04-22T15:29:23.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f994c06ecd0 msgr2=0x7f994c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f994c06ecd0 0x7f994c071190 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9954006010 tx=0x7f995401a040 comp rx=0 tx=0).stop 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 msgr2=0x7f996419cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 0x7f996419cd70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f995800c930 tx=0x7f995800cc40 comp rx=0 tx=0).stop 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 shutdown_connections 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f994c06ecd0 0x7f994c071190 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99640ffad0 0x7f996419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 --2- 172.21.15.96:0/2793161203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9964105b60 0x7f996419cd70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 >> 172.21.15.96:0/2793161203 conn(0x7f99640fb5e0 msgr2=0x7f9964069650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 shutdown_connections 2024-04-22T15:29:23.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:23.919+0000 7f99689fe700 1 -- 172.21.15.96:0/2793161203 wait complete. 2024-04-22T15:29:23.933 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:29:24.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.024+0000 7f249775b700 1 -- 172.21.15.96:0/2944356032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 msgr2=0x7f24901057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.024+0000 7f249775b700 1 --2- 172.21.15.96:0/2944356032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 0x7f24901057f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2480009910 tx=0x7f2480009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:24.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.024+0000 7f249775b700 1 -- 172.21.15.96:0/2944356032 shutdown_connections 2024-04-22T15:29:24.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.024+0000 7f249775b700 1 --2- 172.21.15.96:0/2944356032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 0x7f24901057f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.024+0000 7f249775b700 1 --2- 172.21.15.96:0/2944356032 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2490100df0 0x7f24901011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.024+0000 7f249775b700 1 -- 172.21.15.96:0/2944356032 >> 172.21.15.96:0/2944356032 conn(0x7f24900fc6a0 msgr2=0x7f24900feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:24.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f249775b700 1 -- 172.21.15.96:0/2944356032 shutdown_connections 2024-04-22T15:29:24.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f249775b700 1 -- 172.21.15.96:0/2944356032 wait complete. 2024-04-22T15:29:24.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f249775b700 1 Processor -- start 2024-04-22T15:29:24.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f249775b700 1 -- start start 2024-04-22T15:29:24.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f249775b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2490100df0 0x7f249019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:24.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f249775b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 0x7f249019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:24.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f249775b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249019d450 con 0x7f24901017a0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f249775b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2490196a30 con 0x7f2490100df0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f2494cf6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 0x7f249019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f2494cf6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 0x7f249019ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47550/0 (socket says 172.21.15.96:47550) 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.025+0000 7f2494cf6700 1 -- 172.21.15.96:0/1942594427 learned_addr learned my addr 172.21.15.96:0/1942594427 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f24954f7700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2490100df0 0x7f249019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f2494cf6700 1 -- 172.21.15.96:0/1942594427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2490100df0 msgr2=0x7f249019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f2494cf6700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2490100df0 0x7f249019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f2494cf6700 1 -- 172.21.15.96:0/1942594427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24800095f0 con 0x7f24901017a0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f2494cf6700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 0x7f249019ce30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f24800092e0 tx=0x7f248000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f24867fc700 1 -- 172.21.15.96:0/1942594427 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f248001d070 con 0x7f24901017a0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f24867fc700 1 -- 172.21.15.96:0/1942594427 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f248000f460 con 0x7f24901017a0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2490196cb0 con 0x7f24901017a0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f24867fc700 1 -- 172.21.15.96:0/1942594427 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2480017630 con 0x7f24901017a0 2024-04-22T15:29:24.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.026+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24901971a0 con 0x7f24901017a0 2024-04-22T15:29:24.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.027+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2490109140 con 0x7f24901017a0 2024-04-22T15:29:24.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.028+0000 7f24867fc700 1 -- 172.21.15.96:0/1942594427 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f248000f5d0 con 0x7f24901017a0 2024-04-22T15:29:24.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.028+0000 7f24867fc700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f247c06ecc0 0x7f247c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:24.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.028+0000 7f24867fc700 1 -- 172.21.15.96:0/1942594427 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2480090240 con 0x7f24901017a0 2024-04-22T15:29:24.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.028+0000 7f24954f7700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f247c06ecc0 0x7f247c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:24.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.028+0000 7f24954f7700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f247c06ecc0 0x7f247c071180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f248c006fd0 tx=0x7f248c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:24.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.033+0000 7f24867fc700 1 -- 172.21.15.96:0/1942594427 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f248005cb50 con 0x7f24901017a0 2024-04-22T15:29:24.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.249+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2490197f60 con 0x7f247c06ecc0 2024-04-22T15:29:24.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.250+0000 7f24867fc700 1 -- 172.21.15.96:0/1942594427 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2490197f60 con 0x7f247c06ecc0 2024-04-22T15:29:24.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f247c06ecc0 msgr2=0x7f247c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f247c06ecc0 0x7f247c071180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f248c006fd0 tx=0x7f248c008040 comp rx=0 tx=0).stop 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 msgr2=0x7f249019ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 0x7f249019ce30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f24800092e0 tx=0x7f248000be20 comp rx=0 tx=0).stop 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 shutdown_connections 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f247c06ecc0 0x7f247c071180 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2490100df0 0x7f249019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 --2- 172.21.15.96:0/1942594427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24901017a0 0x7f249019ce30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 >> 172.21.15.96:0/1942594427 conn(0x7f24900fc6a0 msgr2=0x7f24900fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 shutdown_connections 2024-04-22T15:29:24.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.252+0000 7f249775b700 1 -- 172.21.15.96:0/1942594427 wait complete. 2024-04-22T15:29:24.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.357+0000 7f4b77aca700 1 -- 172.21.15.96:0/1074468706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b70100e40 msgr2=0x7f4b70101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.357+0000 7f4b77aca700 1 --2- 172.21.15.96:0/1074468706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b70100e40 0x7f4b70101220 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009910 tx=0x7f4b60009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:24.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.357+0000 7f4b77aca700 1 -- 172.21.15.96:0/1074468706 shutdown_connections 2024-04-22T15:29:24.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.357+0000 7f4b77aca700 1 --2- 172.21.15.96:0/1074468706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b701017f0 0x7f4b70105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.357+0000 7f4b77aca700 1 --2- 172.21.15.96:0/1074468706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b70100e40 0x7f4b70101220 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.357+0000 7f4b77aca700 1 -- 172.21.15.96:0/1074468706 >> 172.21.15.96:0/1074468706 conn(0x7f4b700fc710 msgr2=0x7f4b700feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:24.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.357+0000 7f4b77aca700 1 -- 172.21.15.96:0/1074468706 shutdown_connections 2024-04-22T15:29:24.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.357+0000 7f4b77aca700 1 -- 172.21.15.96:0/1074468706 wait complete. 2024-04-22T15:29:24.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b77aca700 1 Processor -- start 2024-04-22T15:29:24.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b77aca700 1 -- start start 2024-04-22T15:29:24.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b77aca700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b70100e40 0x7f4b7019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:24.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b77aca700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b701017f0 0x7f4b7019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b77aca700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7019d3a0 con 0x7f4b701017f0 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b77aca700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701969d0 con 0x7f4b70100e40 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b75065700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b701017f0 0x7f4b7019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b75065700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b701017f0 0x7f4b7019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47560/0 (socket says 172.21.15.96:47560) 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b75065700 1 -- 172.21.15.96:0/2688603359 learned_addr learned my addr 172.21.15.96:0/2688603359 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.358+0000 7f4b75866700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b70100e40 0x7f4b7019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.359+0000 7f4b75065700 1 -- 172.21.15.96:0/2688603359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b70100e40 msgr2=0x7f4b7019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.359+0000 7f4b75065700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b70100e40 0x7f4b7019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.359+0000 7f4b75065700 1 -- 172.21.15.96:0/2688603359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b600095f0 con 0x7f4b701017f0 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.359+0000 7f4b75065700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b701017f0 0x7f4b7019cd80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00d900 tx=0x7f4b6c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.359+0000 7f4b66ffd700 1 -- 172.21.15.96:0/2688603359 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b6c009900 con 0x7f4b701017f0 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.359+0000 7f4b66ffd700 1 -- 172.21.15.96:0/2688603359 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b6c010460 con 0x7f4b701017f0 2024-04-22T15:29:24.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.359+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b70196cb0 con 0x7f4b701017f0 2024-04-22T15:29:24.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.359+0000 7f4b66ffd700 1 -- 172.21.15.96:0/2688603359 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b6c00f5f0 con 0x7f4b701017f0 2024-04-22T15:29:24.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.359+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b70197200 con 0x7f4b701017f0 2024-04-22T15:29:24.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.360+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b701091e0 con 0x7f4b701017f0 2024-04-22T15:29:24.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.361+0000 7f4b66ffd700 1 -- 172.21.15.96:0/2688603359 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4b6c0105d0 con 0x7f4b701017f0 2024-04-22T15:29:24.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.361+0000 7f4b66ffd700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b5c06ecc0 0x7f4b5c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:24.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.361+0000 7f4b75866700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b5c06ecc0 0x7f4b5c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:24.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.361+0000 7f4b66ffd700 1 -- 172.21.15.96:0/2688603359 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4b6c08e9b0 con 0x7f4b701017f0 2024-04-22T15:29:24.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.362+0000 7f4b75866700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b5c06ecc0 0x7f4b5c071180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b60000c00 tx=0x7f4b6001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:24.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.366+0000 7f4b66ffd700 1 -- 172.21.15.96:0/2688603359 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4b6c05b360 con 0x7f4b701017f0 2024-04-22T15:29:24.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.575+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b7010de10 con 0x7f4b5c06ecc0 2024-04-22T15:29:24.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.586+0000 7f4b66ffd700 1 -- 172.21.15.96:0/2688603359 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f4b7010de10 con 0x7f4b5c06ecc0 2024-04-22T15:29:24.589 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (8m) 70s ago 19m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (19m) 70s ago 19m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (18m) 52s ago 18m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (5m) 70s ago 19m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (5m) 52s ago 18m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (7m) 70s ago 19m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (76s) 70s ago 13m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (95s) 70s ago 13m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (56s) 52s ago 13m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (13m) 52s ago 13m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:29:24.590 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (10m) 70s ago 20m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (9m) 52s ago 18m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (6m) 70s ago 21m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (5m) 52s ago 18m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (9m) 70s ago 19m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (8m) 52s ago 18m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (4m) 70s ago 17m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (4m) 70s ago 17m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (3m) 70s ago 16m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (3m) 52s ago 16m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (2m) 52s ago 15m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (114s) 52s ago 15m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:29:24.591 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (8m) 70s ago 18m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:29:24.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b5c06ecc0 msgr2=0x7f4b5c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b5c06ecc0 0x7f4b5c071180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b60000c00 tx=0x7f4b6001a040 comp rx=0 tx=0).stop 2024-04-22T15:29:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b701017f0 msgr2=0x7f4b7019cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b701017f0 0x7f4b7019cd80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00d900 tx=0x7f4b6c00dc10 comp rx=0 tx=0).stop 2024-04-22T15:29:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 shutdown_connections 2024-04-22T15:29:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b5c06ecc0 0x7f4b5c071180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b70100e40 0x7f4b7019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 --2- 172.21.15.96:0/2688603359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b701017f0 0x7f4b7019cd80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 >> 172.21.15.96:0/2688603359 conn(0x7f4b700fc710 msgr2=0x7f4b700fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 shutdown_connections 2024-04-22T15:29:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.588+0000 7f4b77aca700 1 -- 172.21.15.96:0/2688603359 wait complete. 2024-04-22T15:29:24.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.693+0000 7fbc6e29a700 1 -- 172.21.15.96:0/1671928281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68101ce0 msgr2=0x7fbc681020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.693+0000 7fbc6e29a700 1 --2- 172.21.15.96:0/1671928281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68101ce0 0x7fbc681020c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009910 tx=0x7fbc5c009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:24.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.693+0000 7fbc6e29a700 1 -- 172.21.15.96:0/1671928281 shutdown_connections 2024-04-22T15:29:24.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.693+0000 7fbc6e29a700 1 --2- 172.21.15.96:0/1671928281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc68102690 0x7fbc6810ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.693+0000 7fbc6e29a700 1 --2- 172.21.15.96:0/1671928281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68101ce0 0x7fbc681020c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.693+0000 7fbc6e29a700 1 -- 172.21.15.96:0/1671928281 >> 172.21.15.96:0/1671928281 conn(0x7fbc680fb5e0 msgr2=0x7fbc680fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:24.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.693+0000 7fbc6e29a700 1 -- 172.21.15.96:0/1671928281 shutdown_connections 2024-04-22T15:29:24.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.693+0000 7fbc6e29a700 1 -- 172.21.15.96:0/1671928281 wait complete. 2024-04-22T15:29:24.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc6e29a700 1 Processor -- start 2024-04-22T15:29:24.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc6e29a700 1 -- start start 2024-04-22T15:29:24.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc6e29a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc68101ce0 0x7fbc6819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:24.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc6e29a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68102690 0x7fbc6819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:24.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc6e29a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6819d2f0 con 0x7fbc68102690 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc6e29a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc681969b0 con 0x7fbc68101ce0 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc677fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68102690 0x7fbc6819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc677fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68102690 0x7fbc6819cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47568/0 (socket says 172.21.15.96:47568) 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc677fe700 1 -- 172.21.15.96:0/919420242 learned_addr learned my addr 172.21.15.96:0/919420242 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc67fff700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc68101ce0 0x7fbc6819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc677fe700 1 -- 172.21.15.96:0/919420242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc68101ce0 msgr2=0x7fbc6819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc677fe700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc68101ce0 0x7fbc6819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.694+0000 7fbc677fe700 1 -- 172.21.15.96:0/919420242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c0095f0 con 0x7fbc68102690 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.695+0000 7fbc677fe700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68102690 0x7fbc6819cd60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c960 tx=0x7fbc5800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.695+0000 7fbc657fa700 1 -- 172.21.15.96:0/919420242 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc58007a70 con 0x7fbc68102690 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.695+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc68196c90 con 0x7fbc68102690 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.695+0000 7fbc657fa700 1 -- 172.21.15.96:0/919420242 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc58007bd0 con 0x7fbc68102690 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.695+0000 7fbc657fa700 1 -- 172.21.15.96:0/919420242 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc580186c0 con 0x7fbc68102690 2024-04-22T15:29:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.695+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc681971e0 con 0x7fbc68102690 2024-04-22T15:29:24.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.696+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc68105500 con 0x7fbc68102690 2024-04-22T15:29:24.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.696+0000 7fbc657fa700 1 -- 172.21.15.96:0/919420242 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbc5800faa0 con 0x7fbc68102690 2024-04-22T15:29:24.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.697+0000 7fbc657fa700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc5006ecc0 0x7fbc50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:24.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.697+0000 7fbc67fff700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc5006ecc0 0x7fbc50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:24.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.697+0000 7fbc657fa700 1 -- 172.21.15.96:0/919420242 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbc5808eec0 con 0x7fbc68102690 2024-04-22T15:29:24.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.698+0000 7fbc67fff700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc5006ecc0 0x7fbc50071180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c005f50 tx=0x7fbc5c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:24.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.702+0000 7fbc657fa700 1 -- 172.21.15.96:0/919420242 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbc5805b4c0 con 0x7fbc68102690 2024-04-22T15:29:24.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.985+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbc68069730 con 0x7fbc68102690 2024-04-22T15:29:24.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.986+0000 7fbc657fa700 1 -- 172.21.15.96:0/919420242 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7fbc5805b2e0 con 0x7fbc68102690 2024-04-22T15:29:24.989 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:29:24.989 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:29:24.990 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:29:24.991 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:29:24.991 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:29:24.991 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:29:24.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc5006ecc0 msgr2=0x7fbc50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc5006ecc0 0x7fbc50071180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c005f50 tx=0x7fbc5c01a040 comp rx=0 tx=0).stop 2024-04-22T15:29:24.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68102690 msgr2=0x7fbc6819cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68102690 0x7fbc6819cd60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c960 tx=0x7fbc5800cd20 comp rx=0 tx=0).stop 2024-04-22T15:29:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 shutdown_connections 2024-04-22T15:29:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbc5006ecc0 0x7fbc50071180 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbc68101ce0 0x7fbc6819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 --2- 172.21.15.96:0/919420242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc68102690 0x7fbc6819cd60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 >> 172.21.15.96:0/919420242 conn(0x7fbc680fb5e0 msgr2=0x7fbc68103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 shutdown_connections 2024-04-22T15:29:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:24.988+0000 7fbc6e29a700 1 -- 172.21.15.96:0/919420242 wait complete. 2024-04-22T15:29:25.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 -- 172.21.15.96:0/759752232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67441017d0 msgr2=0x7f6744105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 --2- 172.21.15.96:0/759752232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67441017d0 0x7f6744105820 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f672c009910 tx=0x7f672c009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:25.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 -- 172.21.15.96:0/759752232 shutdown_connections 2024-04-22T15:29:25.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 --2- 172.21.15.96:0/759752232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67441017d0 0x7f6744105820 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 --2- 172.21.15.96:0/759752232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6744100e20 0x7f6744101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 -- 172.21.15.96:0/759752232 >> 172.21.15.96:0/759752232 conn(0x7f67440fc6d0 msgr2=0x7f67440feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:25.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 -- 172.21.15.96:0/759752232 shutdown_connections 2024-04-22T15:29:25.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 -- 172.21.15.96:0/759752232 wait complete. 2024-04-22T15:29:25.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 Processor -- start 2024-04-22T15:29:25.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.091+0000 7f67496ad700 1 -- start start 2024-04-22T15:29:25.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f67496ad700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6744100e20 0x7f674419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:25.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f67496ad700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67441017d0 0x7f674419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:25.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f67496ad700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674419d380 con 0x7f6744100e20 2024-04-22T15:29:25.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f67496ad700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6744196960 con 0x7f67441017d0 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6742ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6744100e20 0x7f674419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6742ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6744100e20 0x7f674419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47574/0 (socket says 172.21.15.96:47574) 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6742ffd700 1 -- 172.21.15.96:0/533236804 learned_addr learned my addr 172.21.15.96:0/533236804 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f673a7fc700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67441017d0 0x7f674419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6742ffd700 1 -- 172.21.15.96:0/533236804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67441017d0 msgr2=0x7f674419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6742ffd700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67441017d0 0x7f674419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6742ffd700 1 -- 172.21.15.96:0/533236804 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c0095f0 con 0x7f6744100e20 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f673a7fc700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67441017d0 0x7f674419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6742ffd700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6744100e20 0x7f674419c820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f673400d8d0 tx=0x7f673400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6740ff9700 1 -- 172.21.15.96:0/533236804 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6734009940 con 0x7f6744100e20 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6744196c40 con 0x7f6744100e20 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6740ff9700 1 -- 172.21.15.96:0/533236804 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6734010460 con 0x7f6744100e20 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.092+0000 7f6740ff9700 1 -- 172.21.15.96:0/533236804 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673400f5f0 con 0x7f6744100e20 2024-04-22T15:29:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.093+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6744197190 con 0x7f6744100e20 2024-04-22T15:29:25.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.093+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6744109220 con 0x7f6744100e20 2024-04-22T15:29:25.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.094+0000 7f6740ff9700 1 -- 172.21.15.96:0/533236804 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f67340105d0 con 0x7f6744100e20 2024-04-22T15:29:25.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.094+0000 7f6740ff9700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673006ecc0 0x7f6730071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:25.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.095+0000 7f6740ff9700 1 -- 172.21.15.96:0/533236804 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f673408eed0 con 0x7f6744100e20 2024-04-22T15:29:25.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.095+0000 7f673a7fc700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673006ecc0 0x7f6730071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:25.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.095+0000 7f673a7fc700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673006ecc0 0x7f6730071180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6744197b20 tx=0x7f672c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:25.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.099+0000 7f6740ff9700 1 -- 172.21.15.96:0/533236804 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f673405b760 con 0x7f6744100e20 2024-04-22T15:29:25.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.341+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f674410ddc0 con 0x7f6744100e20 2024-04-22T15:29:25.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.341+0000 7f6740ff9700 1 -- 172.21.15.96:0/533236804 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f673405b2f0 con 0x7f6744100e20 2024-04-22T15:29:25.345 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:29:25.345 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:29:25.345 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:29:25.345 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:29:25.345 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:29:25.345 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:29:25.345 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:29:25.345 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:29:25.346 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:29:25.347 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:29:25.348 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:29:25.348 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:29:25.348 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:29:25.348 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:29:25.348 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:29:25.348 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:29:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673006ecc0 msgr2=0x7f6730071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673006ecc0 0x7f6730071180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6744197b20 tx=0x7f672c01a040 comp rx=0 tx=0).stop 2024-04-22T15:29:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6744100e20 msgr2=0x7f674419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6744100e20 0x7f674419c820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f673400d8d0 tx=0x7f673400dc90 comp rx=0 tx=0).stop 2024-04-22T15:29:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 shutdown_connections 2024-04-22T15:29:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673006ecc0 0x7f6730071180 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6744100e20 0x7f674419c820 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 --2- 172.21.15.96:0/533236804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67441017d0 0x7f674419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 >> 172.21.15.96:0/533236804 conn(0x7f67440fc6d0 msgr2=0x7f67440fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:25.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 shutdown_connections 2024-04-22T15:29:25.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.344+0000 7f67496ad700 1 -- 172.21.15.96:0/533236804 wait complete. 2024-04-22T15:29:25.350 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:29:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.451+0000 7f0a80c6c700 1 -- 172.21.15.96:0/1440596074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ff3b0 msgr2=0x7f0a7c0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.451+0000 7f0a80c6c700 1 --2- 172.21.15.96:0/1440596074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ff3b0 0x7f0a7c0ff790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00b0a0 tx=0x7f0a6c00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.451+0000 7f0a80c6c700 1 -- 172.21.15.96:0/1440596074 shutdown_connections 2024-04-22T15:29:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.451+0000 7f0a80c6c700 1 --2- 172.21.15.96:0/1440596074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a7c0ffd60 0x7f0a7c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.451+0000 7f0a80c6c700 1 --2- 172.21.15.96:0/1440596074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ff3b0 0x7f0a7c0ff790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.451+0000 7f0a80c6c700 1 -- 172.21.15.96:0/1440596074 >> 172.21.15.96:0/1440596074 conn(0x7f0a7c075f00 msgr2=0x7f0a7c0fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.451+0000 7f0a80c6c700 1 -- 172.21.15.96:0/1440596074 shutdown_connections 2024-04-22T15:29:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.451+0000 7f0a80c6c700 1 -- 172.21.15.96:0/1440596074 wait complete. 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a80c6c700 1 Processor -- start 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a80c6c700 1 -- start start 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a80c6c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a7c0ff3b0 0x7f0a7c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a80c6c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ffd60 0x7f0a7c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a80c6c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c19d3d0 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a80c6c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c1969b0 con 0x7f0a7c0ff3b0 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a79d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ffd60 0x7f0a7c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a79d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ffd60 0x7f0a7c19cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47596/0 (socket says 172.21.15.96:47596) 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a79d9b700 1 -- 172.21.15.96:0/3918729754 learned_addr learned my addr 172.21.15.96:0/3918729754 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a7a59c700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a7c0ff3b0 0x7f0a7c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a79d9b700 1 -- 172.21.15.96:0/3918729754 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a7c0ff3b0 msgr2=0x7f0a7c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.452+0000 7f0a79d9b700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a7c0ff3b0 0x7f0a7c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.453+0000 7f0a79d9b700 1 -- 172.21.15.96:0/3918729754 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c009d00 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.453+0000 7f0a79d9b700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ffd60 0x7f0a7c19cdb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400e8f0 tx=0x7f0a6400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.453+0000 7f0a737fe700 1 -- 172.21.15.96:0/3918729754 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a64009790 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.453+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c196c90 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.453+0000 7f0a737fe700 1 -- 172.21.15.96:0/3918729754 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a64004cf0 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.453+0000 7f0a737fe700 1 -- 172.21.15.96:0/3918729754 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a64005690 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.453+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c1971e0 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.454+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c109240 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.454+0000 7f0a737fe700 1 -- 172.21.15.96:0/3918729754 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0a64004750 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.455+0000 7f0a737fe700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a6806ecc0 0x7f0a68071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:25.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.455+0000 7f0a737fe700 1 -- 172.21.15.96:0/3918729754 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0a64014070 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.455+0000 7f0a7a59c700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a6806ecc0 0x7f0a68071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:25.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.455+0000 7f0a7a59c700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a6806ecc0 0x7f0a68071180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0062a0 tx=0x7f0a6c0061f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.459+0000 7f0a737fe700 1 -- 172.21.15.96:0/3918729754 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0a6405bb50 con 0x7f0a7c0ffd60 2024-04-22T15:29:25.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.678+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a7c197e70 con 0x7f0a6806ecc0 2024-04-22T15:29:25.682 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:25 smithi096 ceph-mon[112443]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:25.682 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:25 smithi096 ceph-mon[112443]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:25.682 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:25 smithi096 ceph-mon[112443]: pgmap v226: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:25.682 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:25 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/919420242' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:29:25.682 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:25 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/533236804' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:29:25.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.679+0000 7f0a737fe700 1 -- 172.21.15.96:0/3918729754 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0a7c197e70 con 0x7f0a6806ecc0 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:29:25.683 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:29:25.684 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:29:25.684 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:29:25.684 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:29:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a6806ecc0 msgr2=0x7f0a68071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a6806ecc0 0x7f0a68071180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0062a0 tx=0x7f0a6c0061f0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ffd60 msgr2=0x7f0a7c19cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ffd60 0x7f0a7c19cdb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400e8f0 tx=0x7f0a6400ecb0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 shutdown_connections 2024-04-22T15:29:25.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a6806ecc0 0x7f0a68071180 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a7c0ff3b0 0x7f0a7c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 --2- 172.21.15.96:0/3918729754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a7c0ffd60 0x7f0a7c19cdb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 >> 172.21.15.96:0/3918729754 conn(0x7f0a7c075f00 msgr2=0x7f0a7c104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:25.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 shutdown_connections 2024-04-22T15:29:25.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.682+0000 7f0a80c6c700 1 -- 172.21.15.96:0/3918729754 wait complete. 2024-04-22T15:29:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:25 smithi196 ceph-mon[81815]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:25 smithi196 ceph-mon[81815]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:25 smithi196 ceph-mon[81815]: pgmap v226: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:25 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/919420242' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:29:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:25 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/533236804' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:29:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.785+0000 7f4dbee35700 1 -- 172.21.15.96:0/2913006127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db8100df0 msgr2=0x7f4db81011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.785+0000 7f4dbee35700 1 --2- 172.21.15.96:0/2913006127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db8100df0 0x7f4db81011d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009910 tx=0x7f4dac009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.786+0000 7f4dbee35700 1 -- 172.21.15.96:0/2913006127 shutdown_connections 2024-04-22T15:29:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.786+0000 7f4dbee35700 1 --2- 172.21.15.96:0/2913006127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4db81017a0 0x7f4db81057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.786+0000 7f4dbee35700 1 --2- 172.21.15.96:0/2913006127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db8100df0 0x7f4db81011d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.786+0000 7f4dbee35700 1 -- 172.21.15.96:0/2913006127 >> 172.21.15.96:0/2913006127 conn(0x7f4db80fc6a0 msgr2=0x7f4db80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.786+0000 7f4dbee35700 1 -- 172.21.15.96:0/2913006127 shutdown_connections 2024-04-22T15:29:25.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.786+0000 7f4dbee35700 1 -- 172.21.15.96:0/2913006127 wait complete. 2024-04-22T15:29:25.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.786+0000 7f4dbee35700 1 Processor -- start 2024-04-22T15:29:25.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.786+0000 7f4dbee35700 1 -- start start 2024-04-22T15:29:25.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4dbee35700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4db8100df0 0x7f4db819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:25.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4dbee35700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db81017a0 0x7f4db819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4dbee35700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db819d350 con 0x7f4db81017a0 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4dbee35700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8196930 con 0x7f4db8100df0 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db81017a0 0x7f4db819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db81017a0 0x7f4db819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47616/0 (socket says 172.21.15.96:47616) 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db7fff700 1 -- 172.21.15.96:0/1319218431 learned_addr learned my addr 172.21.15.96:0/1319218431 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4dbcbd1700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4db8100df0 0x7f4db819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db7fff700 1 -- 172.21.15.96:0/1319218431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4db8100df0 msgr2=0x7f4db819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db7fff700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4db8100df0 0x7f4db819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db7fff700 1 -- 172.21.15.96:0/1319218431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0095f0 con 0x7f4db81017a0 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4dbcbd1700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4db8100df0 0x7f4db819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db7fff700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db81017a0 0x7f4db819cd30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4da400c8f0 tx=0x7f4da400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db5ffb700 1 -- 172.21.15.96:0/1319218431 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da4007a50 con 0x7f4db81017a0 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db5ffb700 1 -- 172.21.15.96:0/1319218431 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4da4007bb0 con 0x7f4db81017a0 2024-04-22T15:29:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db8196c10 con 0x7f4db81017a0 2024-04-22T15:29:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.787+0000 7f4db5ffb700 1 -- 172.21.15.96:0/1319218431 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da4018630 con 0x7f4db81017a0 2024-04-22T15:29:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.788+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db8197160 con 0x7f4db81017a0 2024-04-22T15:29:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.788+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db8109190 con 0x7f4db81017a0 2024-04-22T15:29:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.789+0000 7f4db5ffb700 1 -- 172.21.15.96:0/1319218431 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4da4007d20 con 0x7f4db81017a0 2024-04-22T15:29:25.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.789+0000 7f4db5ffb700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4da806ecc0 0x7f4da8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:25.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.789+0000 7f4db5ffb700 1 -- 172.21.15.96:0/1319218431 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4da408eea0 con 0x7f4db81017a0 2024-04-22T15:29:25.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.789+0000 7f4dbcbd1700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4da806ecc0 0x7f4da8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:25.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.790+0000 7f4dbcbd1700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4da806ecc0 0x7f4da8071180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4dac006010 tx=0x7f4dac00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:25.794+0000 7f4db5ffb700 1 -- 172.21.15.96:0/1319218431 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4da405b730 con 0x7f4db81017a0 2024-04-22T15:29:26.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.076+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4db81022d0 con 0x7f4db81017a0 2024-04-22T15:29:26.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.077+0000 7f4db5ffb700 1 -- 172.21.15.96:0/1319218431 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f4da405b2c0 con 0x7f4db81017a0 2024-04-22T15:29:26.080 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:29:26.080 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:29:26.080 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:29:26.080 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:29:26.080 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:29:26.080 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:29:26.080 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:29:26.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4da806ecc0 msgr2=0x7f4da8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:26.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4da806ecc0 0x7f4da8071180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4dac006010 tx=0x7f4dac00b540 comp rx=0 tx=0).stop 2024-04-22T15:29:26.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db81017a0 msgr2=0x7f4db819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:26.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db81017a0 0x7f4db819cd30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4da400c8f0 tx=0x7f4da400ccb0 comp rx=0 tx=0).stop 2024-04-22T15:29:26.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 shutdown_connections 2024-04-22T15:29:26.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4da806ecc0 0x7f4da8071180 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:26.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4db8100df0 0x7f4db819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:26.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 --2- 172.21.15.96:0/1319218431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4db81017a0 0x7f4db819cd30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:26.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 >> 172.21.15.96:0/1319218431 conn(0x7f4db80fc6a0 msgr2=0x7f4db80fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:26.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 shutdown_connections 2024-04-22T15:29:26.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:26.079+0000 7f4dbee35700 1 -- 172.21.15.96:0/1319218431 wait complete. 2024-04-22T15:29:26.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:26 smithi096 ceph-mon[112443]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:26.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:26 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1319218431' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:29:26.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:26 smithi196 ceph-mon[81815]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:26.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:26 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1319218431' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:29:27.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:27 smithi096 ceph-mon[112443]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:27.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:27 smithi096 ceph-mon[112443]: pgmap v227: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:27 smithi196 ceph-mon[81815]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:27 smithi196 ceph-mon[81815]: pgmap v227: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:29.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:29 smithi096 ceph-mon[112443]: pgmap v228: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:29.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:29 smithi196 ceph-mon[81815]: pgmap v228: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:31 smithi096 ceph-mon[112443]: pgmap v229: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:31 smithi196 ceph-mon[81815]: pgmap v229: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:33 smithi096 ceph-mon[112443]: pgmap v230: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:33 smithi196 ceph-mon[81815]: pgmap v230: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:35 smithi096 ceph-mon[112443]: pgmap v231: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:35 smithi196 ceph-mon[81815]: pgmap v231: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:37 smithi096 ceph-mon[112443]: pgmap v232: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:37 smithi196 ceph-mon[81815]: pgmap v232: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:29:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:29:39.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:39 smithi096 ceph-mon[112443]: pgmap v233: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:39 smithi196 ceph-mon[81815]: pgmap v233: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:41 smithi096 ceph-mon[112443]: pgmap v234: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:41 smithi196 ceph-mon[81815]: pgmap v234: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:43 smithi096 ceph-mon[112443]: pgmap v235: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:43.820 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:43 smithi196 ceph-mon[81815]: pgmap v235: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:45 smithi096 ceph-mon[112443]: pgmap v236: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:45.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:45 smithi196 ceph-mon[81815]: pgmap v236: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:47 smithi096 ceph-mon[112443]: pgmap v237: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:47 smithi196 ceph-mon[81815]: pgmap v237: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:49 smithi096 ceph-mon[112443]: pgmap v238: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:49 smithi196 ceph-mon[81815]: pgmap v238: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:51 smithi096 ceph-mon[112443]: pgmap v239: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:51 smithi196 ceph-mon[81815]: pgmap v239: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:53 smithi096 ceph-mon[112443]: pgmap v240: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:29:53.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:53 smithi196 ceph-mon[81815]: pgmap v240: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:53.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:29:54.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:54 smithi096 ceph-mon[112443]: pgmap v241: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:54.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:54 smithi196 ceph-mon[81815]: pgmap v241: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.197+0000 7f10fc257700 1 -- 172.21.15.96:0/208300387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 msgr2=0x7f10f4105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.197+0000 7f10fc257700 1 --2- 172.21.15.96:0/208300387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 0x7f10f4105fe0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009910 tx=0x7f10f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.197+0000 7f10fc257700 1 -- 172.21.15.96:0/208300387 shutdown_connections 2024-04-22T15:29:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.197+0000 7f10fc257700 1 --2- 172.21.15.96:0/208300387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f40ffb50 0x7f10f40fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.197+0000 7f10fc257700 1 --2- 172.21.15.96:0/208300387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 0x7f10f4105fe0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.197+0000 7f10fc257700 1 -- 172.21.15.96:0/208300387 >> 172.21.15.96:0/208300387 conn(0x7f10f40fb680 msgr2=0x7f10f40fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.197+0000 7f10fc257700 1 -- 172.21.15.96:0/208300387 shutdown_connections 2024-04-22T15:29:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.197+0000 7f10fc257700 1 -- 172.21.15.96:0/208300387 wait complete. 2024-04-22T15:29:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.197+0000 7f10fc257700 1 Processor -- start 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10fc257700 1 -- start start 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10fc257700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f40ffb50 0x7f10f419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10fc257700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 0x7f10f419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10fc257700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f419d320 con 0x7f10f4105c00 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10fc257700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f4196900 con 0x7f10f40ffb50 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10f97f2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 0x7f10f419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10f97f2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 0x7f10f419cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44088/0 (socket says 172.21.15.96:44088) 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10f97f2700 1 -- 172.21.15.96:0/1119428393 learned_addr learned my addr 172.21.15.96:0/1119428393 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10f9ff3700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f40ffb50 0x7f10f419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10f97f2700 1 -- 172.21.15.96:0/1119428393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f40ffb50 msgr2=0x7f10f419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10f97f2700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f40ffb50 0x7f10f419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10f97f2700 1 -- 172.21.15.96:0/1119428393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00095f0 con 0x7f10f4105c00 2024-04-22T15:29:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10f9ff3700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f40ffb50 0x7f10f419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:29:56.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10f97f2700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 0x7f10f419cd00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f10e400d8d0 tx=0x7f10e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:56.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10eaffd700 1 -- 172.21.15.96:0/1119428393 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e4009940 con 0x7f10f4105c00 2024-04-22T15:29:56.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10eaffd700 1 -- 172.21.15.96:0/1119428393 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10e4010460 con 0x7f10f4105c00 2024-04-22T15:29:56.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f4196be0 con 0x7f10f4105c00 2024-04-22T15:29:56.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.198+0000 7f10eaffd700 1 -- 172.21.15.96:0/1119428393 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e400f5f0 con 0x7f10f4105c00 2024-04-22T15:29:56.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.199+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f4197130 con 0x7f10f4105c00 2024-04-22T15:29:56.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.200+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f4101200 con 0x7f10f4105c00 2024-04-22T15:29:56.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.200+0000 7f10eaffd700 1 -- 172.21.15.96:0/1119428393 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f10e40105d0 con 0x7f10f4105c00 2024-04-22T15:29:56.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.200+0000 7f10eaffd700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10e006ecc0 0x7f10e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:56.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.201+0000 7f10eaffd700 1 -- 172.21.15.96:0/1119428393 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f10e408edc0 con 0x7f10f4105c00 2024-04-22T15:29:56.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.201+0000 7f10f9ff3700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10e006ecc0 0x7f10e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:56.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.201+0000 7f10f9ff3700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10e006ecc0 0x7f10e0071180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f10f00059d0 tx=0x7f10f001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:56.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.205+0000 7f10eaffd700 1 -- 172.21.15.96:0/1119428393 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f10e405b6d0 con 0x7f10f4105c00 2024-04-22T15:29:56.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.424+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f10f410cdb0 con 0x7f10e006ecc0 2024-04-22T15:29:56.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.426+0000 7f10eaffd700 1 -- 172.21.15.96:0/1119428393 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f10f410cdb0 con 0x7f10e006ecc0 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10e006ecc0 msgr2=0x7f10e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10e006ecc0 0x7f10e0071180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f10f00059d0 tx=0x7f10f001a040 comp rx=0 tx=0).stop 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 msgr2=0x7f10f419cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 0x7f10f419cd00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f10e400d8d0 tx=0x7f10e400dc90 comp rx=0 tx=0).stop 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 shutdown_connections 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10e006ecc0 0x7f10e0071180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f40ffb50 0x7f10f419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 --2- 172.21.15.96:0/1119428393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f4105c00 0x7f10f419cd00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 >> 172.21.15.96:0/1119428393 conn(0x7f10f40fb680 msgr2=0x7f10f4069370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 shutdown_connections 2024-04-22T15:29:56.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.428+0000 7f10fc257700 1 -- 172.21.15.96:0/1119428393 wait complete. 2024-04-22T15:29:56.442 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:29:56.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.534+0000 7fc23ffd2700 1 -- 172.21.15.96:0/1949502217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc238105b80 msgr2=0x7fc238105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.534+0000 7fc23ffd2700 1 --2- 172.21.15.96:0/1949502217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc238105b80 0x7fc238105f60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc230009910 tx=0x7fc230009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:56.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.534+0000 7fc23ffd2700 1 -- 172.21.15.96:0/1949502217 shutdown_connections 2024-04-22T15:29:56.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.534+0000 7fc23ffd2700 1 --2- 172.21.15.96:0/1949502217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2380ffaf0 0x7fc2380fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.534+0000 7fc23ffd2700 1 --2- 172.21.15.96:0/1949502217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc238105b80 0x7fc238105f60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.534+0000 7fc23ffd2700 1 -- 172.21.15.96:0/1949502217 >> 172.21.15.96:0/1949502217 conn(0x7fc2380fb640 msgr2=0x7fc2380fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:56.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.534+0000 7fc23ffd2700 1 -- 172.21.15.96:0/1949502217 shutdown_connections 2024-04-22T15:29:56.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.534+0000 7fc23ffd2700 1 -- 172.21.15.96:0/1949502217 wait complete. 2024-04-22T15:29:56.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23ffd2700 1 Processor -- start 2024-04-22T15:29:56.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23ffd2700 1 -- start start 2024-04-22T15:29:56.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23ffd2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2380ffaf0 0x7fc2380fbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:56.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23ffd2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc238105b80 0x7fc2380fc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:56.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23ffd2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2380fc990 con 0x7fc2380ffaf0 2024-04-22T15:29:56.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23ffd2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2380fcad0 con 0x7fc238105b80 2024-04-22T15:29:56.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23dd6e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2380ffaf0 0x7fc2380fbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23dd6e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2380ffaf0 0x7fc2380fbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44104/0 (socket says 172.21.15.96:44104) 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23dd6e700 1 -- 172.21.15.96:0/612045598 learned_addr learned my addr 172.21.15.96:0/612045598 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23dd6e700 1 -- 172.21.15.96:0/612045598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc238105b80 msgr2=0x7fc2380fc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23d56d700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc238105b80 0x7fc2380fc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23dd6e700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc238105b80 0x7fc2380fc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23dd6e700 1 -- 172.21.15.96:0/612045598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2300095f0 con 0x7fc2380ffaf0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.535+0000 7fc23dd6e700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2380ffaf0 0x7fc2380fbf10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc23000bd70 tx=0x7fc23000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.536+0000 7fc22effd700 1 -- 172.21.15.96:0/612045598 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc23001d070 con 0x7fc2380ffaf0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.536+0000 7fc22effd700 1 -- 172.21.15.96:0/612045598 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc23000f460 con 0x7fc2380ffaf0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.536+0000 7fc22effd700 1 -- 172.21.15.96:0/612045598 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc230017630 con 0x7fc2380ffaf0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.536+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2380fcd50 con 0x7fc2380ffaf0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.536+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2381a3050 con 0x7fc2380ffaf0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.536+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc238109100 con 0x7fc2380ffaf0 2024-04-22T15:29:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.537+0000 7fc22effd700 1 -- 172.21.15.96:0/612045598 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc23000f5d0 con 0x7fc2380ffaf0 2024-04-22T15:29:56.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.537+0000 7fc22effd700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc22406ecc0 0x7fc224071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:56.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.538+0000 7fc23d56d700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc22406ecc0 0x7fc224071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:56.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.538+0000 7fc22effd700 1 -- 172.21.15.96:0/612045598 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc2300902b0 con 0x7fc2380ffaf0 2024-04-22T15:29:56.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.538+0000 7fc23d56d700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc22406ecc0 0x7fc224071180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc228006fd0 tx=0x7fc228008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:56.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.542+0000 7fc22effd700 1 -- 172.21.15.96:0/612045598 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc230059090 con 0x7fc2380ffaf0 2024-04-22T15:29:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.763+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2380fd460 con 0x7fc22406ecc0 2024-04-22T15:29:56.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.764+0000 7fc22effd700 1 -- 172.21.15.96:0/612045598 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc2380fd460 con 0x7fc22406ecc0 2024-04-22T15:29:56.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc22406ecc0 msgr2=0x7fc224071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc22406ecc0 0x7fc224071180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc228006fd0 tx=0x7fc228008040 comp rx=0 tx=0).stop 2024-04-22T15:29:56.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2380ffaf0 msgr2=0x7fc2380fbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2380ffaf0 0x7fc2380fbf10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc23000bd70 tx=0x7fc23000be50 comp rx=0 tx=0).stop 2024-04-22T15:29:56.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 shutdown_connections 2024-04-22T15:29:56.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc22406ecc0 0x7fc224071180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2380ffaf0 0x7fc2380fbf10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 --2- 172.21.15.96:0/612045598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc238105b80 0x7fc2380fc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 >> 172.21.15.96:0/612045598 conn(0x7fc2380fb640 msgr2=0x7fc238104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:56.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 shutdown_connections 2024-04-22T15:29:56.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.767+0000 7fc23ffd2700 1 -- 172.21.15.96:0/612045598 wait complete. 2024-04-22T15:29:56.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.871+0000 7fcc20409700 1 -- 172.21.15.96:0/3509774755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc181017d0 msgr2=0x7fcc18105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.871+0000 7fcc20409700 1 --2- 172.21.15.96:0/3509774755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc181017d0 0x7fcc18105820 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcc14009910 tx=0x7fcc14009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:56.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.871+0000 7fcc20409700 1 -- 172.21.15.96:0/3509774755 shutdown_connections 2024-04-22T15:29:56.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.871+0000 7fcc20409700 1 --2- 172.21.15.96:0/3509774755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc181017d0 0x7fcc18105820 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.871+0000 7fcc20409700 1 --2- 172.21.15.96:0/3509774755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc18100e20 0x7fcc18101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.871+0000 7fcc20409700 1 -- 172.21.15.96:0/3509774755 >> 172.21.15.96:0/3509774755 conn(0x7fcc180fc6d0 msgr2=0x7fcc180feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:56.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.871+0000 7fcc20409700 1 -- 172.21.15.96:0/3509774755 shutdown_connections 2024-04-22T15:29:56.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.871+0000 7fcc20409700 1 -- 172.21.15.96:0/3509774755 wait complete. 2024-04-22T15:29:56.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc20409700 1 Processor -- start 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc20409700 1 -- start start 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc20409700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc18100e20 0x7fcc1819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc20409700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc181017d0 0x7fcc1819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc20409700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1819d380 con 0x7fcc18100e20 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc20409700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc18196960 con 0x7fcc181017d0 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc1e1a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc18100e20 0x7fcc1819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc1e1a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc18100e20 0x7fcc1819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44128/0 (socket says 172.21.15.96:44128) 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc1e1a5700 1 -- 172.21.15.96:0/2779985345 learned_addr learned my addr 172.21.15.96:0/2779985345 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc1d9a4700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc181017d0 0x7fcc1819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc1e1a5700 1 -- 172.21.15.96:0/2779985345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc181017d0 msgr2=0x7fcc1819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:56.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc1e1a5700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc181017d0 0x7fcc1819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc1e1a5700 1 -- 172.21.15.96:0/2779985345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc140095f0 con 0x7fcc18100e20 2024-04-22T15:29:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.872+0000 7fcc1e1a5700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc18100e20 0x7fcc1819c820 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800d8d0 tx=0x7fcc0800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.873+0000 7fcc0f7fe700 1 -- 172.21.15.96:0/2779985345 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc08009940 con 0x7fcc18100e20 2024-04-22T15:29:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.873+0000 7fcc0f7fe700 1 -- 172.21.15.96:0/2779985345 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc08010460 con 0x7fcc18100e20 2024-04-22T15:29:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.873+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc18196c40 con 0x7fcc18100e20 2024-04-22T15:29:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.873+0000 7fcc0f7fe700 1 -- 172.21.15.96:0/2779985345 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc0800f5f0 con 0x7fcc18100e20 2024-04-22T15:29:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.873+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc18197190 con 0x7fcc18100e20 2024-04-22T15:29:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.874+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc18109220 con 0x7fcc18100e20 2024-04-22T15:29:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.874+0000 7fcc0f7fe700 1 -- 172.21.15.96:0/2779985345 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc080105d0 con 0x7fcc18100e20 2024-04-22T15:29:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.874+0000 7fcc0f7fe700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc0406ecc0 0x7fcc04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.875+0000 7fcc0f7fe700 1 -- 172.21.15.96:0/2779985345 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcc0808eed0 con 0x7fcc18100e20 2024-04-22T15:29:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.875+0000 7fcc1d9a4700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc0406ecc0 0x7fcc04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:56.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.875+0000 7fcc1d9a4700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc0406ecc0 0x7fcc04071180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcc18197b20 tx=0x7fcc1401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:56.879+0000 7fcc0f7fe700 1 -- 172.21.15.96:0/2779985345 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcc0805b760 con 0x7fcc18100e20 2024-04-22T15:29:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.087+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc18197e20 con 0x7fcc0406ecc0 2024-04-22T15:29:57.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.098+0000 7fcc0f7fe700 1 -- 172.21.15.96:0/2779985345 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fcc18197e20 con 0x7fcc0406ecc0 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (8m) 102s ago 19m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (20m) 102s ago 20m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (19m) 85s ago 19m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (5m) 102s ago 20m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (5m) 85s ago 19m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (8m) 102s ago 19m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (108s) 102s ago 14m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (2m) 102s ago 14m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:29:57.102 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (88s) 85s ago 14m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (14m) 85s ago 14m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (11m) 102s ago 21m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (9m) 85s ago 18m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (6m) 102s ago 21m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (6m) 85s ago 18m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (9m) 102s ago 20m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (9m) 85s ago 18m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (5m) 102s ago 18m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (4m) 102s ago 17m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (4m) 102s ago 17m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (3m) 85s ago 16m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (3m) 85s ago 16m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:29:57.103 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (2m) 85s ago 15m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:29:57.104 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (8m) 102s ago 19m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:29:57.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.100+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc0406ecc0 msgr2=0x7fcc04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.100+0000 7fcc20409700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc0406ecc0 0x7fcc04071180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcc18197b20 tx=0x7fcc1401a040 comp rx=0 tx=0).stop 2024-04-22T15:29:57.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.100+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc18100e20 msgr2=0x7fcc1819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.100+0000 7fcc20409700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc18100e20 0x7fcc1819c820 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800d8d0 tx=0x7fcc0800dc90 comp rx=0 tx=0).stop 2024-04-22T15:29:57.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.100+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 shutdown_connections 2024-04-22T15:29:57.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.100+0000 7fcc20409700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc0406ecc0 0x7fcc04071180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.100+0000 7fcc20409700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc18100e20 0x7fcc1819c820 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.100+0000 7fcc20409700 1 --2- 172.21.15.96:0/2779985345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc181017d0 0x7fcc1819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.100+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 >> 172.21.15.96:0/2779985345 conn(0x7fcc180fc6d0 msgr2=0x7fcc180fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:57.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.101+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 shutdown_connections 2024-04-22T15:29:57.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.101+0000 7fcc20409700 1 -- 172.21.15.96:0/2779985345 wait complete. 2024-04-22T15:29:57.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.204+0000 7f894289d700 1 -- 172.21.15.96:0/843434543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 msgr2=0x7f893c0fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.204+0000 7f894289d700 1 --2- 172.21.15.96:0/843434543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 0x7f893c0fff30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8930009910 tx=0x7f8930009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:57.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.204+0000 7f894289d700 1 -- 172.21.15.96:0/843434543 shutdown_connections 2024-04-22T15:29:57.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.204+0000 7f894289d700 1 --2- 172.21.15.96:0/843434543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 0x7f893c0fff30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.204+0000 7f894289d700 1 --2- 172.21.15.96:0/843434543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f893c105ab0 0x7f893c105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.204+0000 7f894289d700 1 -- 172.21.15.96:0/843434543 >> 172.21.15.96:0/843434543 conn(0x7f893c0fb5e0 msgr2=0x7f893c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:57.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.204+0000 7f894289d700 1 -- 172.21.15.96:0/843434543 shutdown_connections 2024-04-22T15:29:57.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.204+0000 7f894289d700 1 -- 172.21.15.96:0/843434543 wait complete. 2024-04-22T15:29:57.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f894289d700 1 Processor -- start 2024-04-22T15:29:57.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f894289d700 1 -- start start 2024-04-22T15:29:57.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f894289d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 0x7f893c197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f894289d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f893c105ab0 0x7f893c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f894289d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c198480 con 0x7f893c0ffab0 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f894289d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1985c0 con 0x7f893c105ab0 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f893bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 0x7f893c197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f893bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 0x7f893c197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44142/0 (socket says 172.21.15.96:44142) 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f893bfff700 1 -- 172.21.15.96:0/172450832 learned_addr learned my addr 172.21.15.96:0/172450832 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f893b7fe700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f893c105ab0 0x7f893c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f893bfff700 1 -- 172.21.15.96:0/172450832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f893c105ab0 msgr2=0x7f893c197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f893bfff700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f893c105ab0 0x7f893c197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.205+0000 7f893bfff700 1 -- 172.21.15.96:0/172450832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300095f0 con 0x7f893c0ffab0 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.206+0000 7f893bfff700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 0x7f893c197970 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f892c00d8d0 tx=0x7f892c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:57.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.206+0000 7f89397fa700 1 -- 172.21.15.96:0/172450832 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f892c0098a0 con 0x7f893c0ffab0 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.206+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c1a09e0 con 0x7f893c0ffab0 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.206+0000 7f89397fa700 1 -- 172.21.15.96:0/172450832 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f892c010460 con 0x7f893c0ffab0 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.206+0000 7f89397fa700 1 -- 172.21.15.96:0/172450832 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f892c00f5f0 con 0x7f893c0ffab0 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.206+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c1a0df0 con 0x7f893c0ffab0 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.207+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893c1080d0 con 0x7f893c0ffab0 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.207+0000 7f89397fa700 1 -- 172.21.15.96:0/172450832 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f892c009a00 con 0x7f893c0ffab0 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.207+0000 7f89397fa700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f892406ecc0 0x7f8924071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.208+0000 7f89397fa700 1 -- 172.21.15.96:0/172450832 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f892c08edc0 con 0x7f893c0ffab0 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.208+0000 7f893b7fe700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f892406ecc0 0x7f8924071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.208+0000 7f893b7fe700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f892406ecc0 0x7f8924071180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f893c069640 tx=0x7f893001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:57.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.212+0000 7f89397fa700 1 -- 172.21.15.96:0/172450832 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f892c057b20 con 0x7f893c0ffab0 2024-04-22T15:29:57.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.497+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f893c1011f0 con 0x7f893c0ffab0 2024-04-22T15:29:57.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.497+0000 7f89397fa700 1 -- 172.21.15.96:0/172450832 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f892c05b1e0 con 0x7f893c0ffab0 2024-04-22T15:29:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:29:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:29:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:29:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:29:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:29:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:29:57.503 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:29:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.499+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f892406ecc0 msgr2=0x7f8924071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.499+0000 7f894289d700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f892406ecc0 0x7f8924071180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f893c069640 tx=0x7f893001a040 comp rx=0 tx=0).stop 2024-04-22T15:29:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.500+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 msgr2=0x7f893c197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.500+0000 7f894289d700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 0x7f893c197970 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f892c00d8d0 tx=0x7f892c00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.500+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 shutdown_connections 2024-04-22T15:29:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.500+0000 7f894289d700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f892406ecc0 0x7f8924071180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.500+0000 7f894289d700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f893c0ffab0 0x7f893c197970 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.500+0000 7f894289d700 1 --2- 172.21.15.96:0/172450832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f893c105ab0 0x7f893c197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.500+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 >> 172.21.15.96:0/172450832 conn(0x7f893c0fb5e0 msgr2=0x7f893c0fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.500+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 shutdown_connections 2024-04-22T15:29:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.500+0000 7f894289d700 1 -- 172.21.15.96:0/172450832 wait complete. 2024-04-22T15:29:57.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:57 smithi196 ceph-mon[81815]: pgmap v242: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:57.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:57 smithi196 ceph-mon[81815]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.603+0000 7f2e8df5a700 1 -- 172.21.15.96:0/3412799473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e88105ab0 msgr2=0x7f2e88105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.603+0000 7f2e8df5a700 1 --2- 172.21.15.96:0/3412799473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e88105ab0 0x7f2e88105e90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009910 tx=0x7f2e78009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.603+0000 7f2e8df5a700 1 -- 172.21.15.96:0/3412799473 shutdown_connections 2024-04-22T15:29:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.603+0000 7f2e8df5a700 1 --2- 172.21.15.96:0/3412799473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e880ffab0 0x7f2e880fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.603+0000 7f2e8df5a700 1 --2- 172.21.15.96:0/3412799473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e88105ab0 0x7f2e88105e90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.603+0000 7f2e8df5a700 1 -- 172.21.15.96:0/3412799473 >> 172.21.15.96:0/3412799473 conn(0x7f2e880fb5e0 msgr2=0x7f2e880fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.603+0000 7f2e8df5a700 1 -- 172.21.15.96:0/3412799473 shutdown_connections 2024-04-22T15:29:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.603+0000 7f2e8df5a700 1 -- 172.21.15.96:0/3412799473 wait complete. 2024-04-22T15:29:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.603+0000 7f2e8df5a700 1 Processor -- start 2024-04-22T15:29:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e8df5a700 1 -- start start 2024-04-22T15:29:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e8df5a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e880ffab0 0x7f2e8819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e8df5a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e88105ab0 0x7f2e8819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e8df5a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8819d380 con 0x7f2e880ffab0 2024-04-22T15:29:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e8df5a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e881969f0 con 0x7f2e88105ab0 2024-04-22T15:29:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e877fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e880ffab0 0x7f2e8819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e877fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e880ffab0 0x7f2e8819c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44154/0 (socket says 172.21.15.96:44154) 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e877fe700 1 -- 172.21.15.96:0/2349276992 learned_addr learned my addr 172.21.15.96:0/2349276992 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e86ffd700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e88105ab0 0x7f2e8819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e877fe700 1 -- 172.21.15.96:0/2349276992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e88105ab0 msgr2=0x7f2e8819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e877fe700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e88105ab0 0x7f2e8819cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e877fe700 1 -- 172.21.15.96:0/2349276992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e780095f0 con 0x7f2e880ffab0 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.604+0000 7f2e877fe700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e880ffab0 0x7f2e8819c8b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006010 tx=0x7f2e7800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.605+0000 7f2e84ff9700 1 -- 172.21.15.96:0/2349276992 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e7801d070 con 0x7f2e880ffab0 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.605+0000 7f2e84ff9700 1 -- 172.21.15.96:0/2349276992 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e7800f460 con 0x7f2e880ffab0 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.605+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e88196c70 con 0x7f2e880ffab0 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.605+0000 7f2e84ff9700 1 -- 172.21.15.96:0/2349276992 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e780175f0 con 0x7f2e880ffab0 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.605+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e88197160 con 0x7f2e880ffab0 2024-04-22T15:29:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.606+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e881080b0 con 0x7f2e880ffab0 2024-04-22T15:29:57.611 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:57 smithi096 ceph-mon[112443]: pgmap v242: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:57.611 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:57 smithi096 ceph-mon[112443]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:57.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.606+0000 7f2e84ff9700 1 -- 172.21.15.96:0/2349276992 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2e7800f5d0 con 0x7f2e880ffab0 2024-04-22T15:29:57.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.607+0000 7f2e84ff9700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e7006ecc0 0x7f2e70071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:57.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.607+0000 7f2e84ff9700 1 -- 172.21.15.96:0/2349276992 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2e78090350 con 0x7f2e880ffab0 2024-04-22T15:29:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.607+0000 7f2e86ffd700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e7006ecc0 0x7f2e70071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.607+0000 7f2e86ffd700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e7006ecc0 0x7f2e70071180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2e88197bb0 tx=0x7f2e7c00a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:57.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.614+0000 7f2e84ff9700 1 -- 172.21.15.96:0/2349276992 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2e7805cbe0 con 0x7f2e880ffab0 2024-04-22T15:29:57.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.866+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2e88197eb0 con 0x7f2e880ffab0 2024-04-22T15:29:57.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.867+0000 7f2e84ff9700 1 -- 172.21.15.96:0/2349276992 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f2e78005740 con 0x7f2e880ffab0 2024-04-22T15:29:57.870 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:29:57.871 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:29:57.872 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:29:57.873 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:29:57.874 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:29:57.874 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:29:57.874 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:29:57.874 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:29:57.874 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e7006ecc0 msgr2=0x7f2e70071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e7006ecc0 0x7f2e70071180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2e88197bb0 tx=0x7f2e7c00a380 comp rx=0 tx=0).stop 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e880ffab0 msgr2=0x7f2e8819c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e880ffab0 0x7f2e8819c8b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006010 tx=0x7f2e7800bd70 comp rx=0 tx=0).stop 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 shutdown_connections 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e7006ecc0 0x7f2e70071180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e880ffab0 0x7f2e8819c8b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 --2- 172.21.15.96:0/2349276992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e88105ab0 0x7f2e8819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 >> 172.21.15.96:0/2349276992 conn(0x7f2e880fb5e0 msgr2=0x7f2e88109ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 shutdown_connections 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.869+0000 7f2e8df5a700 1 -- 172.21.15.96:0/2349276992 wait complete. 2024-04-22T15:29:57.875 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:29:57.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.973+0000 7f7091a6f700 1 -- 172.21.15.96:0/1605883825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c104af0 msgr2=0x7f708c104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.973+0000 7f7091a6f700 1 --2- 172.21.15.96:0/1605883825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c104af0 0x7f708c104ed0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f707c009910 tx=0x7f707c009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:57.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.973+0000 7f7091a6f700 1 -- 172.21.15.96:0/1605883825 shutdown_connections 2024-04-22T15:29:57.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.973+0000 7f7091a6f700 1 --2- 172.21.15.96:0/1605883825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f708c1002a0 0x7f708c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.973+0000 7f7091a6f700 1 --2- 172.21.15.96:0/1605883825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c104af0 0x7f708c104ed0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.973+0000 7f7091a6f700 1 -- 172.21.15.96:0/1605883825 >> 172.21.15.96:0/1605883825 conn(0x7f708c075f40 msgr2=0x7f708c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.973+0000 7f7091a6f700 1 -- 172.21.15.96:0/1605883825 shutdown_connections 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.973+0000 7f7091a6f700 1 -- 172.21.15.96:0/1605883825 wait complete. 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.974+0000 7f7091a6f700 1 Processor -- start 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.974+0000 7f7091a6f700 1 -- start start 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.974+0000 7f7091a6f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c1002a0 0x7f708c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.974+0000 7f7091a6f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f708c104af0 0x7f708c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.974+0000 7f7091a6f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c198580 con 0x7f708c1002a0 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.974+0000 7f7091a6f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1986c0 con 0x7f708c104af0 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.974+0000 7f708b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c1002a0 0x7f708c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.974+0000 7f708b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c1002a0 0x7f708c197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44166/0 (socket says 172.21.15.96:44166) 2024-04-22T15:29:57.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.974+0000 7f708b7fe700 1 -- 172.21.15.96:0/548107138 learned_addr learned my addr 172.21.15.96:0/548107138 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f7083fff700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f708c104af0 0x7f708c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f708b7fe700 1 -- 172.21.15.96:0/548107138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f708c104af0 msgr2=0x7f708c197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f708b7fe700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f708c104af0 0x7f708c197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f708b7fe700 1 -- 172.21.15.96:0/548107138 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c0095f0 con 0x7f708c1002a0 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f708b7fe700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c1002a0 0x7f708c197a70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f707c00bd70 tx=0x7f707c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f70897fa700 1 -- 172.21.15.96:0/548107138 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f707c01d070 con 0x7f708c1002a0 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f70897fa700 1 -- 172.21.15.96:0/548107138 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f707c0056e0 con 0x7f708c1002a0 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708c1a0ae0 con 0x7f708c1002a0 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f70897fa700 1 -- 172.21.15.96:0/548107138 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f707c017400 con 0x7f708c1002a0 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.975+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708c1a0e90 con 0x7f708c1002a0 2024-04-22T15:29:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.976+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708c1080a0 con 0x7f708c1002a0 2024-04-22T15:29:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.976+0000 7f70897fa700 1 -- 172.21.15.96:0/548107138 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f707c003a60 con 0x7f708c1002a0 2024-04-22T15:29:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.977+0000 7f70897fa700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f707806ecc0 0x7f7078071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.977+0000 7f70897fa700 1 -- 172.21.15.96:0/548107138 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f707c090260 con 0x7f708c1002a0 2024-04-22T15:29:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.977+0000 7f7083fff700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f707806ecc0 0x7f7078071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.977+0000 7f7083fff700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f707806ecc0 0x7f7078071180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7074005950 tx=0x7f70740058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:57.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:57.982+0000 7f70897fa700 1 -- 172.21.15.96:0/548107138 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f707c059040 con 0x7f708c1002a0 2024-04-22T15:29:58.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.211+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f708c0ff730 con 0x7f707806ecc0 2024-04-22T15:29:58.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.212+0000 7f70897fa700 1 -- 172.21.15.96:0/548107138 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f708c0ff730 con 0x7f707806ecc0 2024-04-22T15:29:58.216 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:29:58.216 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:29:58.216 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:29:58.216 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:29:58.216 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:29:58.216 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:29:58.216 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:29:58.216 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:29:58.217 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:29:58.217 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:29:58.217 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:29:58.217 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:29:58.217 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:29:58.217 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f707806ecc0 msgr2=0x7f7078071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f707806ecc0 0x7f7078071180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7074005950 tx=0x7f70740058e0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c1002a0 msgr2=0x7f708c197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c1002a0 0x7f708c197a70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f707c00bd70 tx=0x7f707c00be50 comp rx=0 tx=0).stop 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 shutdown_connections 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f707806ecc0 0x7f7078071180 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708c1002a0 0x7f708c197a70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 --2- 172.21.15.96:0/548107138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f708c104af0 0x7f708c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 >> 172.21.15.96:0/548107138 conn(0x7f708c075f40 msgr2=0x7f708c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 shutdown_connections 2024-04-22T15:29:58.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.215+0000 7f7091a6f700 1 -- 172.21.15.96:0/548107138 wait complete. 2024-04-22T15:29:58.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.319+0000 7f1dad94c700 1 -- 172.21.15.96:0/1949312470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8100e60 msgr2=0x7f1da8101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:58.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.319+0000 7f1dad94c700 1 --2- 172.21.15.96:0/1949312470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8100e60 0x7f1da8101240 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009910 tx=0x7f1d9c009c20 comp rx=0 tx=0).stop 2024-04-22T15:29:58.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.319+0000 7f1dad94c700 1 -- 172.21.15.96:0/1949312470 shutdown_connections 2024-04-22T15:29:58.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.319+0000 7f1dad94c700 1 --2- 172.21.15.96:0/1949312470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da8101810 0x7f1da8105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.319+0000 7f1dad94c700 1 --2- 172.21.15.96:0/1949312470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8100e60 0x7f1da8101240 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.319+0000 7f1dad94c700 1 -- 172.21.15.96:0/1949312470 >> 172.21.15.96:0/1949312470 conn(0x7f1da80fc710 msgr2=0x7f1da80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:58.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.319+0000 7f1dad94c700 1 -- 172.21.15.96:0/1949312470 shutdown_connections 2024-04-22T15:29:58.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.319+0000 7f1dad94c700 1 -- 172.21.15.96:0/1949312470 wait complete. 2024-04-22T15:29:58.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.320+0000 7f1dad94c700 1 Processor -- start 2024-04-22T15:29:58.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.320+0000 7f1dad94c700 1 -- start start 2024-04-22T15:29:58.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.320+0000 7f1dad94c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da8100e60 0x7f1da819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:58.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.320+0000 7f1dad94c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8101810 0x7f1da819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:58.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.320+0000 7f1dad94c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da819d4b0 con 0x7f1da8101810 2024-04-22T15:29:58.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1dad94c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da8196aa0 con 0x7f1da8100e60 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1da67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8101810 0x7f1da819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1da67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8101810 0x7f1da819cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44192/0 (socket says 172.21.15.96:44192) 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1da67fc700 1 -- 172.21.15.96:0/2506799800 learned_addr learned my addr 172.21.15.96:0/2506799800 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1da67fc700 1 -- 172.21.15.96:0/2506799800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da8100e60 msgr2=0x7f1da819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1da67fc700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da8100e60 0x7f1da819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1da67fc700 1 -- 172.21.15.96:0/2506799800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c0095f0 con 0x7f1da8101810 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1da67fc700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8101810 0x7f1da819cdd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800b480 tx=0x7f1d9800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1dac94a700 1 -- 172.21.15.96:0/2506799800 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d980118c0 con 0x7f1da8101810 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1dac94a700 1 -- 172.21.15.96:0/2506799800 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d98010380 con 0x7f1da8101810 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da8196d80 con 0x7f1da8101810 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1dac94a700 1 -- 172.21.15.96:0/2506799800 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d9800f4f0 con 0x7f1da8101810 2024-04-22T15:29:58.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.321+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da81972d0 con 0x7f1da8101810 2024-04-22T15:29:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.322+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da8109200 con 0x7f1da8101810 2024-04-22T15:29:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.323+0000 7f1dac94a700 1 -- 172.21.15.96:0/2506799800 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1d9800f7b0 con 0x7f1da8101810 2024-04-22T15:29:58.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.323+0000 7f1dac94a700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d9006ecc0 0x7f1d90071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:29:58.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.323+0000 7f1dac94a700 1 -- 172.21.15.96:0/2506799800 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1d9808ebb0 con 0x7f1da8101810 2024-04-22T15:29:58.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.323+0000 7f1da6ffd700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d9006ecc0 0x7f1d90071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:29:58.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.323+0000 7f1da6ffd700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d9006ecc0 0x7f1d90071180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00b5c0 tx=0x7f1d9c005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:29:58.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.328+0000 7f1dac94a700 1 -- 172.21.15.96:0/2506799800 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1d9805b410 con 0x7f1da8101810 2024-04-22T15:29:58.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:58 smithi096 ceph-mon[112443]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:58.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:58 smithi096 ceph-mon[112443]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:58.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:58 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/172450832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:29:58.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:58 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2349276992' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:29:58.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.612+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1da8102340 con 0x7f1da8101810 2024-04-22T15:29:58.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.612+0000 7f1dac94a700 1 -- 172.21.15.96:0/2506799800 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f1d9805afa0 con 0x7f1da8101810 2024-04-22T15:29:58.616 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:29:58.616 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:29:58.616 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:29:58.616 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:29:58.616 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:29:58.616 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:29:58.616 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:29:58.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d9006ecc0 msgr2=0x7f1d90071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d9006ecc0 0x7f1d90071180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00b5c0 tx=0x7f1d9c005960 comp rx=0 tx=0).stop 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8101810 msgr2=0x7f1da819cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8101810 0x7f1da819cdd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800b480 tx=0x7f1d9800b840 comp rx=0 tx=0).stop 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 shutdown_connections 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d9006ecc0 0x7f1d90071180 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da8100e60 0x7f1da819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 --2- 172.21.15.96:0/2506799800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da8101810 0x7f1da819cdd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 >> 172.21.15.96:0/2506799800 conn(0x7f1da80fc710 msgr2=0x7f1da80fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 shutdown_connections 2024-04-22T15:29:58.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:29:58.614+0000 7f1dad94c700 1 -- 172.21.15.96:0/2506799800 wait complete. 2024-04-22T15:29:58.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:58 smithi196 ceph-mon[81815]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:58.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:58 smithi196 ceph-mon[81815]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:58.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:58 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/172450832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:29:58.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:58 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2349276992' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:29:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:59 smithi096 ceph-mon[112443]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:59 smithi096 ceph-mon[112443]: pgmap v243: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:29:59 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2506799800' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:29:59.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:59 smithi196 ceph-mon[81815]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:29:59.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:59 smithi196 ceph-mon[81815]: pgmap v243: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:29:59.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:29:59 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2506799800' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:30:00.370 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:30:00.000+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:30:00.370 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:30:00.000+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:30:00.371 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:30:00.000+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-22T15:30:00.371 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:30:00.000+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:30:00.371 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:30:00.000+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:30:00.371 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:30:00.000+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:30:00.371 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:30:00.000+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-22T15:30:00.371 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-mon[112443]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:30:00.371 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-mon[112443]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:30:00.371 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-mon[112443]: fs cephfs is degraded 2024-04-22T15:30:00.371 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-mon[112443]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:30:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-mon[112443]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:30:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-mon[112443]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:30:00.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:00 smithi096 ceph-mon[112443]: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:30:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:00 smithi196 ceph-mon[81815]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:30:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:00 smithi196 ceph-mon[81815]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:30:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:00 smithi196 ceph-mon[81815]: fs cephfs is degraded 2024-04-22T15:30:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:00 smithi196 ceph-mon[81815]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:30:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:00 smithi196 ceph-mon[81815]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:30:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:00 smithi196 ceph-mon[81815]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:30:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:00 smithi196 ceph-mon[81815]: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:30:01.637 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:01 smithi096 ceph-mon[112443]: pgmap v244: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:01 smithi196 ceph-mon[81815]: pgmap v244: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:03.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:03 smithi096 ceph-mon[112443]: pgmap v245: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:03 smithi196 ceph-mon[81815]: pgmap v245: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:05 smithi096 ceph-mon[112443]: pgmap v246: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:05 smithi196 ceph-mon[81815]: pgmap v246: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:07 smithi096 ceph-mon[112443]: pgmap v247: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:07 smithi196 ceph-mon[81815]: pgmap v247: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:08.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:30:08.661 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:30:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:09 smithi096 ceph-mon[112443]: pgmap v248: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:09 smithi196 ceph-mon[81815]: pgmap v248: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:11 smithi096 ceph-mon[112443]: pgmap v249: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:11 smithi196 ceph-mon[81815]: pgmap v249: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:13 smithi096 ceph-mon[112443]: pgmap v250: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:13 smithi196 ceph-mon[81815]: pgmap v250: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:15 smithi096 ceph-mon[112443]: pgmap v251: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:15 smithi196 ceph-mon[81815]: pgmap v251: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:17.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:17 smithi096 ceph-mon[112443]: pgmap v252: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:17 smithi196 ceph-mon[81815]: pgmap v252: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:19 smithi096 ceph-mon[112443]: pgmap v253: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:19 smithi196 ceph-mon[81815]: pgmap v253: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:21 smithi096 ceph-mon[112443]: pgmap v254: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:21.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:21 smithi196 ceph-mon[81815]: pgmap v254: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:23 smithi096 ceph-mon[112443]: pgmap v255: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:30:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:23 smithi196 ceph-mon[81815]: pgmap v255: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:30:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:25 smithi096 ceph-mon[112443]: pgmap v256: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:25 smithi196 ceph-mon[81815]: pgmap v256: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:27 smithi096 ceph-mon[112443]: pgmap v257: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:27 smithi196 ceph-mon[81815]: pgmap v257: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:28.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.729+0000 7fa578c0c700 1 -- 172.21.15.96:0/2573524883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574102680 msgr2=0x7fa57410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:28.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.729+0000 7fa578c0c700 1 --2- 172.21.15.96:0/2573524883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574102680 0x7fa57410ab70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009910 tx=0x7fa55c009c20 comp rx=0 tx=0).stop 2024-04-22T15:30:28.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.729+0000 7fa578c0c700 1 -- 172.21.15.96:0/2573524883 shutdown_connections 2024-04-22T15:30:28.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.729+0000 7fa578c0c700 1 --2- 172.21.15.96:0/2573524883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574102680 0x7fa57410ab70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:28.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.729+0000 7fa578c0c700 1 --2- 172.21.15.96:0/2573524883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa574101d60 0x7fa574102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:28.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.729+0000 7fa578c0c700 1 -- 172.21.15.96:0/2573524883 >> 172.21.15.96:0/2573524883 conn(0x7fa574076740 msgr2=0x7fa5740fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:28.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.729+0000 7fa578c0c700 1 -- 172.21.15.96:0/2573524883 shutdown_connections 2024-04-22T15:30:28.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.729+0000 7fa578c0c700 1 -- 172.21.15.96:0/2573524883 wait complete. 2024-04-22T15:30:28.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.730+0000 7fa578c0c700 1 Processor -- start 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.730+0000 7fa578c0c700 1 -- start start 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.730+0000 7fa578c0c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574101d60 0x7fa57419c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.730+0000 7fa578c0c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa574102680 0x7fa57419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.730+0000 7fa578c0c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa574196a80 con 0x7fa574101d60 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.730+0000 7fa578c0c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa574196bf0 con 0x7fa574102680 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.730+0000 7fa57259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574101d60 0x7fa57419c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.730+0000 7fa57259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574101d60 0x7fa57419c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52518/0 (socket says 172.21.15.96:52518) 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.730+0000 7fa57259c700 1 -- 172.21.15.96:0/2901911009 learned_addr learned my addr 172.21.15.96:0/2901911009 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa571d9b700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa574102680 0x7fa57419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa57259c700 1 -- 172.21.15.96:0/2901911009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa574102680 msgr2=0x7fa57419cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa57259c700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa574102680 0x7fa57419cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa57259c700 1 -- 172.21.15.96:0/2901911009 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c0095f0 con 0x7fa574101d60 2024-04-22T15:30:28.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa571d9b700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa574102680 0x7fa57419cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:30:28.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa57259c700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574101d60 0x7fa57419c9d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa56400ba50 tx=0x7fa56400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:28.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa56b7fe700 1 -- 172.21.15.96:0/2901911009 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa56400ca00 con 0x7fa574101d60 2024-04-22T15:30:28.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa574196ed0 con 0x7fa574101d60 2024-04-22T15:30:28.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa56b7fe700 1 -- 172.21.15.96:0/2901911009 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa56400cb60 con 0x7fa574101d60 2024-04-22T15:30:28.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa56b7fe700 1 -- 172.21.15.96:0/2901911009 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5640075a0 con 0x7fa574101d60 2024-04-22T15:30:28.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.731+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5741973f0 con 0x7fa574101d60 2024-04-22T15:30:28.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.732+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa574068ea0 con 0x7fa574101d60 2024-04-22T15:30:28.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.733+0000 7fa56b7fe700 1 -- 172.21.15.96:0/2901911009 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa56400ccd0 con 0x7fa574101d60 2024-04-22T15:30:28.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.733+0000 7fa56b7fe700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa56006ecc0 0x7fa560071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:28.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.733+0000 7fa56b7fe700 1 -- 172.21.15.96:0/2901911009 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa564006e40 con 0x7fa574101d60 2024-04-22T15:30:28.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.733+0000 7fa571d9b700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa56006ecc0 0x7fa560071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:28.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.734+0000 7fa571d9b700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa56006ecc0 0x7fa560071180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00b5c0 tx=0x7fa55c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:28.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.738+0000 7fa56b7fe700 1 -- 172.21.15.96:0/2901911009 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa5640573f0 con 0x7fa574101d60 2024-04-22T15:30:28.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.955+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa57410de40 con 0x7fa56006ecc0 2024-04-22T15:30:28.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.956+0000 7fa56b7fe700 1 -- 172.21.15.96:0/2901911009 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa57410de40 con 0x7fa56006ecc0 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa56006ecc0 msgr2=0x7fa560071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa56006ecc0 0x7fa560071180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00b5c0 tx=0x7fa55c01a040 comp rx=0 tx=0).stop 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574101d60 msgr2=0x7fa57419c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574101d60 0x7fa57419c9d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa56400ba50 tx=0x7fa56400be10 comp rx=0 tx=0).stop 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 shutdown_connections 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa56006ecc0 0x7fa560071180 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa574101d60 0x7fa57419c9d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 --2- 172.21.15.96:0/2901911009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa574102680 0x7fa57419cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 >> 172.21.15.96:0/2901911009 conn(0x7fa574076740 msgr2=0x7fa5741030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:28.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 shutdown_connections 2024-04-22T15:30:28.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:28.958+0000 7fa578c0c700 1 -- 172.21.15.96:0/2901911009 wait complete. 2024-04-22T15:30:28.972 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:30:29.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.063+0000 7fc9f65f2700 1 -- 172.21.15.96:0/2099536081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 msgr2=0x7fc9f0100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.063+0000 7fc9f65f2700 1 --2- 172.21.15.96:0/2099536081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 0x7fc9f0100720 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009910 tx=0x7fc9e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:30:29.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.063+0000 7fc9f65f2700 1 -- 172.21.15.96:0/2099536081 shutdown_connections 2024-04-22T15:30:29.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.063+0000 7fc9f65f2700 1 --2- 172.21.15.96:0/2099536081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 0x7fc9f0100720 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.063+0000 7fc9f65f2700 1 --2- 172.21.15.96:0/2099536081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f0104af0 0x7fc9f0104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.063+0000 7fc9f65f2700 1 -- 172.21.15.96:0/2099536081 >> 172.21.15.96:0/2099536081 conn(0x7fc9f0075f40 msgr2=0x7fc9f00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:29.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.063+0000 7fc9f65f2700 1 -- 172.21.15.96:0/2099536081 shutdown_connections 2024-04-22T15:30:29.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.063+0000 7fc9f65f2700 1 -- 172.21.15.96:0/2099536081 wait complete. 2024-04-22T15:30:29.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.064+0000 7fc9f65f2700 1 Processor -- start 2024-04-22T15:30:29.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.064+0000 7fc9f65f2700 1 -- start start 2024-04-22T15:30:29.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.064+0000 7fc9f65f2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 0x7fc9f00fb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:29.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.064+0000 7fc9f65f2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f0104af0 0x7fc9f00fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:29.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.064+0000 7fc9f65f2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f00fbe00 con 0x7fc9f01002a0 2024-04-22T15:30:29.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.064+0000 7fc9f65f2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f00fbf40 con 0x7fc9f0104af0 2024-04-22T15:30:29.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9effff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 0x7fc9f00fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:29.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9effff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 0x7fc9f00fb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52534/0 (socket says 172.21.15.96:52534) 2024-04-22T15:30:29.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9effff700 1 -- 172.21.15.96:0/3601945155 learned_addr learned my addr 172.21.15.96:0/3601945155 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:30:29.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9ef7fe700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f0104af0 0x7fc9f00fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9effff700 1 -- 172.21.15.96:0/3601945155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f0104af0 msgr2=0x7fc9f00fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9effff700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f0104af0 0x7fc9f00fb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9effff700 1 -- 172.21.15.96:0/3601945155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e40095f0 con 0x7fc9f01002a0 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9effff700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 0x7fc9f00fb380 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000d8d0 tx=0x7fc9e000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9ed7fa700 1 -- 172.21.15.96:0/3601945155 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9e00098a0 con 0x7fc9f01002a0 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f01a2d70 con 0x7fc9f01002a0 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9ed7fa700 1 -- 172.21.15.96:0/3601945155 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9e0010460 con 0x7fc9f01002a0 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9ed7fa700 1 -- 172.21.15.96:0/3601945155 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9e001b620 con 0x7fc9f01002a0 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.065+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f01a30d0 con 0x7fc9f01002a0 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.066+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f0109010 con 0x7fc9f01002a0 2024-04-22T15:30:29.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.067+0000 7fc9ed7fa700 1 -- 172.21.15.96:0/3601945155 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc9e0009a00 con 0x7fc9f01002a0 2024-04-22T15:30:29.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.067+0000 7fc9ed7fa700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc9d806ecc0 0x7fc9d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:29.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.067+0000 7fc9ed7fa700 1 -- 172.21.15.96:0/3601945155 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc9e008f0f0 con 0x7fc9f01002a0 2024-04-22T15:30:29.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.067+0000 7fc9ef7fe700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc9d806ecc0 0x7fc9d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:29.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.068+0000 7fc9ef7fe700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc9d806ecc0 0x7fc9d8071180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc9f00fca60 tx=0x7fc9e401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:29.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.072+0000 7fc9ed7fa700 1 -- 172.21.15.96:0/3601945155 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc9e0057e50 con 0x7fc9f01002a0 2024-04-22T15:30:29.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.287+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9f00fc810 con 0x7fc9d806ecc0 2024-04-22T15:30:29.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.288+0000 7fc9ed7fa700 1 -- 172.21.15.96:0/3601945155 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc9f00fc810 con 0x7fc9d806ecc0 2024-04-22T15:30:29.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc9d806ecc0 msgr2=0x7fc9d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc9d806ecc0 0x7fc9d8071180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc9f00fca60 tx=0x7fc9e401a040 comp rx=0 tx=0).stop 2024-04-22T15:30:29.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 msgr2=0x7fc9f00fb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 0x7fc9f00fb380 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000d8d0 tx=0x7fc9e000dbe0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 shutdown_connections 2024-04-22T15:30:29.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc9d806ecc0 0x7fc9d8071180 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9f01002a0 0x7fc9f00fb380 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 --2- 172.21.15.96:0/3601945155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f0104af0 0x7fc9f00fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 >> 172.21.15.96:0/3601945155 conn(0x7fc9f0075f40 msgr2=0x7fc9f00fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:29.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 shutdown_connections 2024-04-22T15:30:29.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.291+0000 7fc9f65f2700 1 -- 172.21.15.96:0/3601945155 wait complete. 2024-04-22T15:30:29.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.395+0000 7f5761687700 1 -- 172.21.15.96:0/184977653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 msgr2=0x7f575c10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.395+0000 7f5761687700 1 --2- 172.21.15.96:0/184977653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 0x7f575c10ac10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5744009910 tx=0x7f5744009c20 comp rx=0 tx=0).stop 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.395+0000 7f5761687700 1 -- 172.21.15.96:0/184977653 shutdown_connections 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.395+0000 7f5761687700 1 --2- 172.21.15.96:0/184977653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 0x7f575c10ac10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.395+0000 7f5761687700 1 --2- 172.21.15.96:0/184977653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f575c101e00 0x7f575c1021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.395+0000 7f5761687700 1 -- 172.21.15.96:0/184977653 >> 172.21.15.96:0/184977653 conn(0x7f575c076740 msgr2=0x7f575c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.395+0000 7f5761687700 1 -- 172.21.15.96:0/184977653 shutdown_connections 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.395+0000 7f5761687700 1 -- 172.21.15.96:0/184977653 wait complete. 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.395+0000 7f5761687700 1 Processor -- start 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f5761687700 1 -- start start 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f5761687700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f575c101e00 0x7f575c0ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f5761687700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 0x7f575c0fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:29.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f5761687700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1004e0 con 0x7f575c102720 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f5761687700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c100650 con 0x7f575c101e00 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f57527fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 0x7f575c0fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f57527fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 0x7f575c0fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52550/0 (socket says 172.21.15.96:52550) 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f57527fc700 1 -- 172.21.15.96:0/2336284365 learned_addr learned my addr 172.21.15.96:0/2336284365 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f575affd700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f575c101e00 0x7f575c0ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f57527fc700 1 -- 172.21.15.96:0/2336284365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f575c101e00 msgr2=0x7f575c0ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f57527fc700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f575c101e00 0x7f575c0ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f57527fc700 1 -- 172.21.15.96:0/2336284365 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57440095f0 con 0x7f575c102720 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f575affd700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f575c101e00 0x7f575c0ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.396+0000 7f57527fc700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 0x7f575c0fffa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5744000c00 tx=0x7f574400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.397+0000 7f5758ff9700 1 -- 172.21.15.96:0/2336284365 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f574401d070 con 0x7f575c102720 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.397+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c1008d0 con 0x7f575c102720 2024-04-22T15:30:29.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.397+0000 7f5758ff9700 1 -- 172.21.15.96:0/2336284365 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5744005700 con 0x7f575c102720 2024-04-22T15:30:29.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.397+0000 7f5758ff9700 1 -- 172.21.15.96:0/2336284365 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f574400fe50 con 0x7f575c102720 2024-04-22T15:30:29.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.397+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c1a3060 con 0x7f575c102720 2024-04-22T15:30:29.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.398+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c068e00 con 0x7f575c102720 2024-04-22T15:30:29.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.398+0000 7f5758ff9700 1 -- 172.21.15.96:0/2336284365 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f574400f640 con 0x7f575c102720 2024-04-22T15:30:29.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.399+0000 7f5758ff9700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f574806ecd0 0x7f5748071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:29.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.399+0000 7f5758ff9700 1 -- 172.21.15.96:0/2336284365 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5744090100 con 0x7f575c102720 2024-04-22T15:30:29.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.399+0000 7f575affd700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f574806ecd0 0x7f5748071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:29.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.400+0000 7f575affd700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f574806ecd0 0x7f5748071190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f574c005fd0 tx=0x7f574c005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:29.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.404+0000 7f5758ff9700 1 -- 172.21.15.96:0/2336284365 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5744058e60 con 0x7f575c102720 2024-04-22T15:30:29.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.611+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f575c1008d0 con 0x7f574806ecd0 2024-04-22T15:30:29.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.622+0000 7f5758ff9700 1 -- 172.21.15.96:0/2336284365 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f575c1008d0 con 0x7f574806ecd0 2024-04-22T15:30:29.625 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:30:29.625 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (9m) 2m ago 20m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (20m) 2m ago 20m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (19m) 117s ago 19m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (6m) 2m ago 20m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (6m) 117s ago 19m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (8m) 2m ago 20m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (2m) 2m ago 14m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (2m) 2m ago 14m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (2m) 117s ago 14m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (14m) 117s ago 14m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (11m) 2m ago 21m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (10m) 117s ago 19m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (7m) 2m ago 22m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:30:29.626 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (7m) 117s ago 19m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:30:29.627 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (10m) 2m ago 20m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:30:29.627 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (10m) 117s ago 19m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:30:29.627 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (5m) 2m ago 18m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:30:29.627 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (5m) 2m ago 18m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:30:29.627 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (4m) 2m ago 17m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:30:29.627 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (4m) 117s ago 17m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:30:29.627 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (3m) 117s ago 16m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:30:29.627 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (2m) 117s ago 16m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:30:29.627 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (9m) 2m ago 19m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:30:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f574806ecd0 msgr2=0x7f5748071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f574806ecd0 0x7f5748071190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f574c005fd0 tx=0x7f574c005ee0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 msgr2=0x7f575c0fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 0x7f575c0fffa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5744000c00 tx=0x7f574400be20 comp rx=0 tx=0).stop 2024-04-22T15:30:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 shutdown_connections 2024-04-22T15:30:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f574806ecd0 0x7f5748071190 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f575c101e00 0x7f575c0ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 --2- 172.21.15.96:0/2336284365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f575c102720 0x7f575c0fffa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 >> 172.21.15.96:0/2336284365 conn(0x7f575c076740 msgr2=0x7f575c1030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:29.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 shutdown_connections 2024-04-22T15:30:29.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.624+0000 7f5761687700 1 -- 172.21.15.96:0/2336284365 wait complete. 2024-04-22T15:30:29.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.726+0000 7f7798cf5700 1 -- 172.21.15.96:0/2654260438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7794105a30 msgr2=0x7f7794105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.726+0000 7f7798cf5700 1 --2- 172.21.15.96:0/2654260438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7794105a30 0x7f7794105e10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7784009910 tx=0x7f7784009c20 comp rx=0 tx=0).stop 2024-04-22T15:30:29.730 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:29 smithi096 ceph-mon[112443]: pgmap v258: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:29.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.726+0000 7f7798cf5700 1 -- 172.21.15.96:0/2654260438 shutdown_connections 2024-04-22T15:30:29.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.726+0000 7f7798cf5700 1 --2- 172.21.15.96:0/2654260438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77940ffbc0 0x7f7794100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.726+0000 7f7798cf5700 1 --2- 172.21.15.96:0/2654260438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7794105a30 0x7f7794105e10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.726+0000 7f7798cf5700 1 -- 172.21.15.96:0/2654260438 >> 172.21.15.96:0/2654260438 conn(0x7f7794076790 msgr2=0x7f7794076ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:29.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.726+0000 7f7798cf5700 1 -- 172.21.15.96:0/2654260438 shutdown_connections 2024-04-22T15:30:29.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.726+0000 7f7798cf5700 1 -- 172.21.15.96:0/2654260438 wait complete. 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f7798cf5700 1 Processor -- start 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f7798cf5700 1 -- start start 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f7798cf5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77940ffbc0 0x7f779419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f7798cf5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7794105a30 0x7f779419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f7798cf5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779419d2e0 con 0x7f77940ffbc0 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f7798cf5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7794196850 con 0x7f7794105a30 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f779259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77940ffbc0 0x7f779419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f779259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77940ffbc0 0x7f779419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52574/0 (socket says 172.21.15.96:52574) 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f779259c700 1 -- 172.21.15.96:0/1847136137 learned_addr learned my addr 172.21.15.96:0/1847136137 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f7791d9b700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7794105a30 0x7f779419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f779259c700 1 -- 172.21.15.96:0/1847136137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7794105a30 msgr2=0x7f779419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f779259c700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7794105a30 0x7f779419cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:29.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.727+0000 7f779259c700 1 -- 172.21.15.96:0/1847136137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77840095f0 con 0x7f77940ffbc0 2024-04-22T15:30:29.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.728+0000 7f779259c700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77940ffbc0 0x7f779419c710 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f778400bd70 tx=0x7f778400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:29.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.728+0000 7f77837fe700 1 -- 172.21.15.96:0/1847136137 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f778401d070 con 0x7f77940ffbc0 2024-04-22T15:30:29.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.728+0000 7f77837fe700 1 -- 172.21.15.96:0/1847136137 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77840056e0 con 0x7f77940ffbc0 2024-04-22T15:30:29.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.728+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7794196ad0 con 0x7f77940ffbc0 2024-04-22T15:30:29.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.728+0000 7f77837fe700 1 -- 172.21.15.96:0/1847136137 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f778400fe40 con 0x7f77940ffbc0 2024-04-22T15:30:29.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.728+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7794196fc0 con 0x7f77940ffbc0 2024-04-22T15:30:29.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.729+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7794101170 con 0x7f77940ffbc0 2024-04-22T15:30:29.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.729+0000 7f77837fe700 1 -- 172.21.15.96:0/1847136137 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f778400f680 con 0x7f77940ffbc0 2024-04-22T15:30:29.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.730+0000 7f77837fe700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f777c06ec60 0x7f777c071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:29.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.730+0000 7f77837fe700 1 -- 172.21.15.96:0/1847136137 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f77840902a0 con 0x7f77940ffbc0 2024-04-22T15:30:29.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.730+0000 7f7791d9b700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f777c06ec60 0x7f777c071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:29.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.730+0000 7f7791d9b700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f777c06ec60 0x7f777c071120 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7794197c70 tx=0x7f778800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:29.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:29.735+0000 7f77837fe700 1 -- 172.21.15.96:0/1847136137 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f778405cb30 con 0x7f77940ffbc0 2024-04-22T15:30:29.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:29 smithi196 ceph-mon[81815]: pgmap v258: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:30.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.018+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f779410cc60 con 0x7f77940ffbc0 2024-04-22T15:30:30.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.019+0000 7f77837fe700 1 -- 172.21.15.96:0/1847136137 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f778405c6c0 con 0x7f77940ffbc0 2024-04-22T15:30:30.023 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:30:30.023 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:30:30.023 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:30:30.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:30:30.023 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:30:30.023 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:30:30.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:30:30.023 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:30:30.023 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:30:30.024 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:30:30.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f777c06ec60 msgr2=0x7f777c071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f777c06ec60 0x7f777c071120 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7794197c70 tx=0x7f778800b410 comp rx=0 tx=0).stop 2024-04-22T15:30:30.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77940ffbc0 msgr2=0x7f779419c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77940ffbc0 0x7f779419c710 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f778400bd70 tx=0x7f778400bda0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 shutdown_connections 2024-04-22T15:30:30.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f777c06ec60 0x7f777c071120 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77940ffbc0 0x7f779419c710 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 --2- 172.21.15.96:0/1847136137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7794105a30 0x7f779419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 >> 172.21.15.96:0/1847136137 conn(0x7f7794076790 msgr2=0x7f77940688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:30.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 shutdown_connections 2024-04-22T15:30:30.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.021+0000 7f7798cf5700 1 -- 172.21.15.96:0/1847136137 wait complete. 2024-04-22T15:30:30.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.125+0000 7f1f20691700 1 -- 172.21.15.96:0/1876312833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f181017f0 msgr2=0x7f1f18105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.125+0000 7f1f20691700 1 --2- 172.21.15.96:0/1876312833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f181017f0 0x7f1f18105840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009910 tx=0x7f1f08009c20 comp rx=0 tx=0).stop 2024-04-22T15:30:30.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.125+0000 7f1f20691700 1 -- 172.21.15.96:0/1876312833 shutdown_connections 2024-04-22T15:30:30.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.125+0000 7f1f20691700 1 --2- 172.21.15.96:0/1876312833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f181017f0 0x7f1f18105840 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.125+0000 7f1f20691700 1 --2- 172.21.15.96:0/1876312833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f18100e40 0x7f1f18101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.125+0000 7f1f20691700 1 -- 172.21.15.96:0/1876312833 >> 172.21.15.96:0/1876312833 conn(0x7f1f180fc710 msgr2=0x7f1f180feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:30.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.125+0000 7f1f20691700 1 -- 172.21.15.96:0/1876312833 shutdown_connections 2024-04-22T15:30:30.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.125+0000 7f1f20691700 1 -- 172.21.15.96:0/1876312833 wait complete. 2024-04-22T15:30:30.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.126+0000 7f1f20691700 1 Processor -- start 2024-04-22T15:30:30.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.126+0000 7f1f20691700 1 -- start start 2024-04-22T15:30:30.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.126+0000 7f1f20691700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18100e40 0x7f1f1819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.126+0000 7f1f20691700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f181017f0 0x7f1f1819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.126+0000 7f1f20691700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1819d480 con 0x7f1f18100e40 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.126+0000 7f1f20691700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18196a60 con 0x7f1f181017f0 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.126+0000 7f1f1e42d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18100e40 0x7f1f1819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.126+0000 7f1f1e42d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18100e40 0x7f1f1819c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52596/0 (socket says 172.21.15.96:52596) 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.126+0000 7f1f1e42d700 1 -- 172.21.15.96:0/2494489847 learned_addr learned my addr 172.21.15.96:0/2494489847 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f1dc2c700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f181017f0 0x7f1f1819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f1e42d700 1 -- 172.21.15.96:0/2494489847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f181017f0 msgr2=0x7f1f1819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f1e42d700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f181017f0 0x7f1f1819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f1e42d700 1 -- 172.21.15.96:0/2494489847 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080095f0 con 0x7f1f18100e40 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f1e42d700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18100e40 0x7f1f1819c920 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400b4b0 tx=0x7f1f1400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f0f7fe700 1 -- 172.21.15.96:0/2494489847 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f14011800 con 0x7f1f18100e40 2024-04-22T15:30:30.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f18196d40 con 0x7f1f18100e40 2024-04-22T15:30:30.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f0f7fe700 1 -- 172.21.15.96:0/2494489847 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f14011e20 con 0x7f1f18100e40 2024-04-22T15:30:30.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f0f7fe700 1 -- 172.21.15.96:0/2494489847 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f1400f2e0 con 0x7f1f18100e40 2024-04-22T15:30:30.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.127+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f18197290 con 0x7f1f18100e40 2024-04-22T15:30:30.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.128+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f18109170 con 0x7f1f18100e40 2024-04-22T15:30:30.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.129+0000 7f1f0f7fe700 1 -- 172.21.15.96:0/2494489847 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1f14010380 con 0x7f1f18100e40 2024-04-22T15:30:30.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.129+0000 7f1f0f7fe700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1f0406ecc0 0x7f1f04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:30.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.129+0000 7f1f1dc2c700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1f0406ecc0 0x7f1f04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:30.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.129+0000 7f1f0f7fe700 1 -- 172.21.15.96:0/2494489847 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1f1408e9d0 con 0x7f1f18100e40 2024-04-22T15:30:30.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.130+0000 7f1f1dc2c700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1f0406ecc0 0x7f1f04071180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f18197c20 tx=0x7f1f0801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:30.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.134+0000 7f1f0f7fe700 1 -- 172.21.15.96:0/2494489847 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1f1405b260 con 0x7f1f18100e40 2024-04-22T15:30:30.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.377+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1f18197f20 con 0x7f1f18100e40 2024-04-22T15:30:30.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.377+0000 7f1f0f7fe700 1 -- 172.21.15.96:0/2494489847 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f1f1405adf0 con 0x7f1f18100e40 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:30:30.381 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:30:30.382 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:30:30.383 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:30:30.384 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:30:30.384 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:30:30.384 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:30:30.384 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1f0406ecc0 msgr2=0x7f1f04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1f0406ecc0 0x7f1f04071180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f18197c20 tx=0x7f1f0801a040 comp rx=0 tx=0).stop 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18100e40 msgr2=0x7f1f1819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18100e40 0x7f1f1819c920 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400b4b0 tx=0x7f1f1400b7c0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 shutdown_connections 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1f0406ecc0 0x7f1f04071180 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18100e40 0x7f1f1819c920 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 --2- 172.21.15.96:0/2494489847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f181017f0 0x7f1f1819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 >> 172.21.15.96:0/2494489847 conn(0x7f1f180fc710 msgr2=0x7f1f180fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 shutdown_connections 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.380+0000 7f1f20691700 1 -- 172.21.15.96:0/2494489847 wait complete. 2024-04-22T15:30:30.385 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:30:30.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.486+0000 7f1160060700 1 -- 172.21.15.96:0/613029996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1158100e10 msgr2=0x7f11581011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.486+0000 7f1160060700 1 --2- 172.21.15.96:0/613029996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1158100e10 0x7f11581011f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f114800b0a0 tx=0x7f114800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.486+0000 7f1160060700 1 -- 172.21.15.96:0/613029996 shutdown_connections 2024-04-22T15:30:30.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.486+0000 7f1160060700 1 --2- 172.21.15.96:0/613029996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11581017c0 0x7f1158105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.486+0000 7f1160060700 1 --2- 172.21.15.96:0/613029996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1158100e10 0x7f11581011f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.486+0000 7f1160060700 1 -- 172.21.15.96:0/613029996 >> 172.21.15.96:0/613029996 conn(0x7f11580fc6e0 msgr2=0x7f11580feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:30.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.486+0000 7f1160060700 1 -- 172.21.15.96:0/613029996 shutdown_connections 2024-04-22T15:30:30.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.486+0000 7f1160060700 1 -- 172.21.15.96:0/613029996 wait complete. 2024-04-22T15:30:30.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f1160060700 1 Processor -- start 2024-04-22T15:30:30.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f1160060700 1 -- start start 2024-04-22T15:30:30.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f1160060700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1158100e10 0x7f1158196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:30.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f1160060700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11581017c0 0x7f11581970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:30.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f1160060700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1158197790 con 0x7f11581017c0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f1160060700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115819b3a0 con 0x7f1158100e10 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f115d5fb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11581017c0 0x7f11581970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f115d5fb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11581017c0 0x7f11581970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57584/0 (socket says 172.21.15.96:57584) 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f115d5fb700 1 -- 172.21.15.96:0/2381920150 learned_addr learned my addr 172.21.15.96:0/2381920150 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f115ddfc700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1158100e10 0x7f1158196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f115d5fb700 1 -- 172.21.15.96:0/2381920150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1158100e10 msgr2=0x7f1158196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f115d5fb700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1158100e10 0x7f1158196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.487+0000 7f115d5fb700 1 -- 172.21.15.96:0/2381920150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1148009d00 con 0x7f11581017c0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.488+0000 7f115d5fb700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11581017c0 0x7f11581970b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f114c00d900 tx=0x7f114c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.488+0000 7f1156ffd700 1 -- 172.21.15.96:0/2381920150 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f114c009900 con 0x7f11581017c0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.488+0000 7f1156ffd700 1 -- 172.21.15.96:0/2381920150 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f114c010460 con 0x7f11581017c0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.488+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115819b680 con 0x7f11581017c0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.488+0000 7f1156ffd700 1 -- 172.21.15.96:0/2381920150 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f114c00f5f0 con 0x7f11581017c0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.488+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115819bbd0 con 0x7f11581017c0 2024-04-22T15:30:30.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.489+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11581091b0 con 0x7f11581017c0 2024-04-22T15:30:30.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.489+0000 7f1156ffd700 1 -- 172.21.15.96:0/2381920150 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f114c0105d0 con 0x7f11581017c0 2024-04-22T15:30:30.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.490+0000 7f1156ffd700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f114006ecc0 0x7f1140071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:30.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.490+0000 7f1156ffd700 1 -- 172.21.15.96:0/2381920150 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f114c08e960 con 0x7f11581017c0 2024-04-22T15:30:30.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.490+0000 7f115ddfc700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f114006ecc0 0x7f1140071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:30.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.490+0000 7f115ddfc700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f114006ecc0 0x7f1140071180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1148009fd0 tx=0x7f11480060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:30.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.494+0000 7f1156ffd700 1 -- 172.21.15.96:0/2381920150 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f114c057740 con 0x7f11581017c0 2024-04-22T15:30:30.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.714+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1158197dc0 con 0x7f114006ecc0 2024-04-22T15:30:30.718 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:30 smithi096 ceph-mon[112443]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:30:30.718 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:30 smithi096 ceph-mon[112443]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:30:30.718 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:30 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1847136137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:30:30.718 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:30 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2494489847' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:30:30.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.715+0000 7f1156ffd700 1 -- 172.21.15.96:0/2381920150 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1158197dc0 con 0x7f114006ecc0 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:30:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:30:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:30:30.720 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f114006ecc0 msgr2=0x7f1140071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f114006ecc0 0x7f1140071180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1148009fd0 tx=0x7f11480060c0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11581017c0 msgr2=0x7f11581970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11581017c0 0x7f11581970b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f114c00d900 tx=0x7f114c00dc10 comp rx=0 tx=0).stop 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 shutdown_connections 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f114006ecc0 0x7f1140071180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1158100e10 0x7f1158196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 --2- 172.21.15.96:0/2381920150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11581017c0 0x7f11581970b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 >> 172.21.15.96:0/2381920150 conn(0x7f11580fc6e0 msgr2=0x7f11580fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 shutdown_connections 2024-04-22T15:30:30.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.717+0000 7f1160060700 1 -- 172.21.15.96:0/2381920150 wait complete. 2024-04-22T15:30:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:30 smithi196 ceph-mon[81815]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:30:30.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:30 smithi196 ceph-mon[81815]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:30:30.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:30 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1847136137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:30:30.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:30 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2494489847' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:30:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.820+0000 7fc3ab742700 1 -- 172.21.15.96:0/3221511950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100dd0 msgr2=0x7fc3a41047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.820+0000 7fc3ab742700 1 --2- 172.21.15.96:0/3221511950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100dd0 0x7fc3a41047a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009910 tx=0x7fc3a0009c20 comp rx=0 tx=0).stop 2024-04-22T15:30:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.820+0000 7fc3ab742700 1 -- 172.21.15.96:0/3221511950 shutdown_connections 2024-04-22T15:30:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.820+0000 7fc3ab742700 1 --2- 172.21.15.96:0/3221511950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100dd0 0x7fc3a41047a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.820+0000 7fc3ab742700 1 --2- 172.21.15.96:0/3221511950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a4100420 0x7fc3a4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.820+0000 7fc3ab742700 1 -- 172.21.15.96:0/3221511950 >> 172.21.15.96:0/3221511950 conn(0x7fc3a4075f40 msgr2=0x7fc3a40fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.820+0000 7fc3ab742700 1 -- 172.21.15.96:0/3221511950 shutdown_connections 2024-04-22T15:30:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.820+0000 7fc3ab742700 1 -- 172.21.15.96:0/3221511950 wait complete. 2024-04-22T15:30:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3ab742700 1 Processor -- start 2024-04-22T15:30:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3ab742700 1 -- start start 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3ab742700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100420 0x7fc3a41979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3ab742700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a4100dd0 0x7fc3a4197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3ab742700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a4198440 con 0x7fc3a4100420 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3ab742700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a4198580 con 0x7fc3a4100dd0 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3a94de700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100420 0x7fc3a41979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3a94de700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100420 0x7fc3a41979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57614/0 (socket says 172.21.15.96:57614) 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3a94de700 1 -- 172.21.15.96:0/813566772 learned_addr learned my addr 172.21.15.96:0/813566772 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.821+0000 7fc3a8cdd700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a4100dd0 0x7fc3a4197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.822+0000 7fc3a94de700 1 -- 172.21.15.96:0/813566772 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a4100dd0 msgr2=0x7fc3a4197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.822+0000 7fc3a94de700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a4100dd0 0x7fc3a4197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.822+0000 7fc3a94de700 1 -- 172.21.15.96:0/813566772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a00095f0 con 0x7fc3a4100420 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.822+0000 7fc3a94de700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100420 0x7fc3a41979c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc39400d8d0 tx=0x7fc39400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.822+0000 7fc39a7fc700 1 -- 172.21.15.96:0/813566772 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3940098a0 con 0x7fc3a4100420 2024-04-22T15:30:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.822+0000 7fc39a7fc700 1 -- 172.21.15.96:0/813566772 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc394010460 con 0x7fc3a4100420 2024-04-22T15:30:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.822+0000 7fc39a7fc700 1 -- 172.21.15.96:0/813566772 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc39400f5f0 con 0x7fc3a4100420 2024-04-22T15:30:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.822+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a41a0a30 con 0x7fc3a4100420 2024-04-22T15:30:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.822+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a41a0de0 con 0x7fc3a4100420 2024-04-22T15:30:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.823+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a4108110 con 0x7fc3a4100420 2024-04-22T15:30:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.823+0000 7fc39a7fc700 1 -- 172.21.15.96:0/813566772 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc394009a00 con 0x7fc3a4100420 2024-04-22T15:30:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.824+0000 7fc39a7fc700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39006ecc0 0x7fc390071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:30:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.824+0000 7fc39a7fc700 1 -- 172.21.15.96:0/813566772 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc39408ec80 con 0x7fc3a4100420 2024-04-22T15:30:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.824+0000 7fc3a8cdd700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39006ecc0 0x7fc390071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:30:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.824+0000 7fc3a8cdd700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39006ecc0 0x7fc390071180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0000c00 tx=0x7fc3a001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:30:30.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:30.829+0000 7fc39a7fc700 1 -- 172.21.15.96:0/813566772 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc394057a60 con 0x7fc3a4100420 2024-04-22T15:30:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.109+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc3a40ff6c0 con 0x7fc3a4100420 2024-04-22T15:30:31.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.109+0000 7fc39a7fc700 1 -- 172.21.15.96:0/813566772 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fc3940161c0 con 0x7fc3a4100420 2024-04-22T15:30:31.113 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:30:31.113 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:30:31.113 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:30:31.113 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:30:31.113 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:30:31.113 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:30:31.113 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39006ecc0 msgr2=0x7fc390071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39006ecc0 0x7fc390071180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0000c00 tx=0x7fc3a001a040 comp rx=0 tx=0).stop 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100420 msgr2=0x7fc3a41979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100420 0x7fc3a41979c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc39400d8d0 tx=0x7fc39400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 shutdown_connections 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc39006ecc0 0x7fc390071180 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3a4100420 0x7fc3a41979c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 --2- 172.21.15.96:0/813566772 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3a4100dd0 0x7fc3a4197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 >> 172.21.15.96:0/813566772 conn(0x7fc3a4075f40 msgr2=0x7fc3a41035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 shutdown_connections 2024-04-22T15:30:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:30:31.111+0000 7fc3ab742700 1 -- 172.21.15.96:0/813566772 wait complete. 2024-04-22T15:30:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:31 smithi096 ceph-mon[112443]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:30:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:31 smithi096 ceph-mon[112443]: pgmap v259: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:31 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/813566772' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:30:31.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:31 smithi196 ceph-mon[81815]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:30:31.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:31 smithi196 ceph-mon[81815]: pgmap v259: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:31.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:31 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/813566772' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:30:32.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:32 smithi096 ceph-mon[112443]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:30:32.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:32 smithi196 ceph-mon[81815]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:30:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:33 smithi096 ceph-mon[112443]: pgmap v260: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:33 smithi196 ceph-mon[81815]: pgmap v260: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:35 smithi096 ceph-mon[112443]: pgmap v261: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:35 smithi196 ceph-mon[81815]: pgmap v261: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:37 smithi096 ceph-mon[112443]: pgmap v262: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:37 smithi196 ceph-mon[81815]: pgmap v262: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:30:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:30:39.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:39 smithi096 ceph-mon[112443]: pgmap v263: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:39 smithi196 ceph-mon[81815]: pgmap v263: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:41 smithi096 ceph-mon[112443]: pgmap v264: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:41 smithi196 ceph-mon[81815]: pgmap v264: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:43 smithi096 ceph-mon[112443]: pgmap v265: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:43 smithi196 ceph-mon[81815]: pgmap v265: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:45 smithi096 ceph-mon[112443]: pgmap v266: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:45.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:45 smithi196 ceph-mon[81815]: pgmap v266: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:47.716 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:47 smithi196 ceph-mon[81815]: pgmap v267: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:47 smithi096 ceph-mon[112443]: pgmap v267: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:49 smithi096 ceph-mon[112443]: pgmap v268: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:49 smithi196 ceph-mon[81815]: pgmap v268: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:51 smithi096 ceph-mon[112443]: pgmap v269: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:51 smithi196 ceph-mon[81815]: pgmap v269: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:53 smithi096 ceph-mon[112443]: pgmap v270: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:30:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:53 smithi196 ceph-mon[81815]: pgmap v270: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:30:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:55 smithi096 ceph-mon[112443]: pgmap v271: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:55 smithi196 ceph-mon[81815]: pgmap v271: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:57 smithi096 ceph-mon[112443]: pgmap v272: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:57 smithi196 ceph-mon[81815]: pgmap v272: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:30:59 smithi096 ceph-mon[112443]: pgmap v273: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:30:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:30:59 smithi196 ceph-mon[81815]: pgmap v273: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:01.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.228+0000 7f97a9bb4700 1 -- 172.21.15.96:0/749144353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a40ffae0 msgr2=0x7f97a40fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.228+0000 7f97a9bb4700 1 --2- 172.21.15.96:0/749144353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a40ffae0 0x7f97a40fff60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9798009910 tx=0x7f9798009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.228+0000 7f97a9bb4700 1 -- 172.21.15.96:0/749144353 shutdown_connections 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.228+0000 7f97a9bb4700 1 --2- 172.21.15.96:0/749144353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a40ffae0 0x7f97a40fff60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.228+0000 7f97a9bb4700 1 --2- 172.21.15.96:0/749144353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a4105b70 0x7f97a4105f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.228+0000 7f97a9bb4700 1 -- 172.21.15.96:0/749144353 >> 172.21.15.96:0/749144353 conn(0x7f97a40fb5f0 msgr2=0x7f97a40fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.228+0000 7f97a9bb4700 1 -- 172.21.15.96:0/749144353 shutdown_connections 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.228+0000 7f97a9bb4700 1 -- 172.21.15.96:0/749144353 wait complete. 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.228+0000 7f97a9bb4700 1 Processor -- start 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a9bb4700 1 -- start start 2024-04-22T15:31:01.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a9bb4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a40ffae0 0x7f97a419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a9bb4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a4105b70 0x7f97a419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a9bb4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a419d490 con 0x7f97a4105b70 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a9bb4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a4196a70 con 0x7f97a40ffae0 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a4105b70 0x7f97a419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a4105b70 0x7f97a419ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33590/0 (socket says 172.21.15.96:33590) 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a2ffd700 1 -- 172.21.15.96:0/947315377 learned_addr learned my addr 172.21.15.96:0/947315377 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a37fe700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a40ffae0 0x7f97a419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a2ffd700 1 -- 172.21.15.96:0/947315377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a40ffae0 msgr2=0x7f97a419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a2ffd700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a40ffae0 0x7f97a419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.229+0000 7f97a2ffd700 1 -- 172.21.15.96:0/947315377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980095f0 con 0x7f97a4105b70 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.230+0000 7f97a2ffd700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a4105b70 0x7f97a419ce70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9798006010 tx=0x7f979800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.230+0000 7f97a0ff9700 1 -- 172.21.15.96:0/947315377 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f979801d070 con 0x7f97a4105b70 2024-04-22T15:31:01.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.230+0000 7f97a0ff9700 1 -- 172.21.15.96:0/947315377 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f979800f460 con 0x7f97a4105b70 2024-04-22T15:31:01.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.230+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a4196cf0 con 0x7f97a4105b70 2024-04-22T15:31:01.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.230+0000 7f97a0ff9700 1 -- 172.21.15.96:0/947315377 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9798017630 con 0x7f97a4105b70 2024-04-22T15:31:01.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.230+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a41971e0 con 0x7f97a4105b70 2024-04-22T15:31:01.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.231+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a4109110 con 0x7f97a4105b70 2024-04-22T15:31:01.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.231+0000 7f97a0ff9700 1 -- 172.21.15.96:0/947315377 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f979800f5d0 con 0x7f97a4105b70 2024-04-22T15:31:01.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.232+0000 7f97a0ff9700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f978c06ecd0 0x7f978c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:01.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.232+0000 7f97a0ff9700 1 -- 172.21.15.96:0/947315377 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9798090320 con 0x7f97a4105b70 2024-04-22T15:31:01.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.232+0000 7f97a37fe700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f978c06ecd0 0x7f978c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:01.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.232+0000 7f97a37fe700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f978c06ecd0 0x7f978c071190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9794009a10 tx=0x7f9794009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:01.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.237+0000 7f97a0ff9700 1 -- 172.21.15.96:0/947315377 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f979805cc70 con 0x7f97a4105b70 2024-04-22T15:31:01.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.456+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97a4197fa0 con 0x7f978c06ecd0 2024-04-22T15:31:01.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.457+0000 7f97a0ff9700 1 -- 172.21.15.96:0/947315377 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f97a4197fa0 con 0x7f978c06ecd0 2024-04-22T15:31:01.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.459+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f978c06ecd0 msgr2=0x7f978c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.459+0000 7f97a9bb4700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f978c06ecd0 0x7f978c071190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9794009a10 tx=0x7f9794009190 comp rx=0 tx=0).stop 2024-04-22T15:31:01.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.459+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a4105b70 msgr2=0x7f97a419ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.459+0000 7f97a9bb4700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a4105b70 0x7f97a419ce70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9798006010 tx=0x7f979800bd70 comp rx=0 tx=0).stop 2024-04-22T15:31:01.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.459+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 shutdown_connections 2024-04-22T15:31:01.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.459+0000 7f97a9bb4700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f978c06ecd0 0x7f978c071190 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.459+0000 7f97a9bb4700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97a40ffae0 0x7f97a419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.459+0000 7f97a9bb4700 1 --2- 172.21.15.96:0/947315377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a4105b70 0x7f97a419ce70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.459+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 >> 172.21.15.96:0/947315377 conn(0x7f97a40fb5f0 msgr2=0x7f97a4104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:01.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.460+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 shutdown_connections 2024-04-22T15:31:01.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.460+0000 7f97a9bb4700 1 -- 172.21.15.96:0/947315377 wait complete. 2024-04-22T15:31:01.473 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:31:01.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 -- 172.21.15.96:0/1505046166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 msgr2=0x7f4f3410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 --2- 172.21.15.96:0/1505046166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 0x7f4f3410ab70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009910 tx=0x7f4f28009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:01.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 -- 172.21.15.96:0/1505046166 shutdown_connections 2024-04-22T15:31:01.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 --2- 172.21.15.96:0/1505046166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 0x7f4f3410ab70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 --2- 172.21.15.96:0/1505046166 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f34101d60 0x7f4f34102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 -- 172.21.15.96:0/1505046166 >> 172.21.15.96:0/1505046166 conn(0x7f4f340fb5f0 msgr2=0x7f4f340fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:01.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 -- 172.21.15.96:0/1505046166 shutdown_connections 2024-04-22T15:31:01.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 -- 172.21.15.96:0/1505046166 wait complete. 2024-04-22T15:31:01.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 Processor -- start 2024-04-22T15:31:01.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.566+0000 7f4f389be700 1 -- start start 2024-04-22T15:31:01.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f389be700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f34101d60 0x7f4f3419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:01.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f389be700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 0x7f4f3419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:01.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f389be700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3419d3c0 con 0x7f4f34102680 2024-04-22T15:31:01.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f389be700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f341969a0 con 0x7f4f34101d60 2024-04-22T15:31:01.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f31d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 0x7f4f3419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f31d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 0x7f4f3419cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33610/0 (socket says 172.21.15.96:33610) 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f31d9b700 1 -- 172.21.15.96:0/1084113767 learned_addr learned my addr 172.21.15.96:0/1084113767 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f3259c700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f34101d60 0x7f4f3419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f31d9b700 1 -- 172.21.15.96:0/1084113767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f34101d60 msgr2=0x7f4f3419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f31d9b700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f34101d60 0x7f4f3419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f31d9b700 1 -- 172.21.15.96:0/1084113767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f280095f0 con 0x7f4f34102680 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f31d9b700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 0x7f4f3419cda0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009fd0 tx=0x7f4f2800b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f237fe700 1 -- 172.21.15.96:0/1084113767 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f2801d070 con 0x7f4f34102680 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f237fe700 1 -- 172.21.15.96:0/1084113767 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f280052a0 con 0x7f4f34102680 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.567+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f34196c20 con 0x7f4f34102680 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.568+0000 7f4f237fe700 1 -- 172.21.15.96:0/1084113767 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f28022470 con 0x7f4f34102680 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.568+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f34197110 con 0x7f4f34102680 2024-04-22T15:31:01.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.568+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f34105480 con 0x7f4f34102680 2024-04-22T15:31:01.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.569+0000 7f4f237fe700 1 -- 172.21.15.96:0/1084113767 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4f2800f6f0 con 0x7f4f34102680 2024-04-22T15:31:01.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.569+0000 7f4f237fe700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f1c06ecc0 0x7f4f1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:01.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.570+0000 7f4f237fe700 1 -- 172.21.15.96:0/1084113767 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4f280901a0 con 0x7f4f34102680 2024-04-22T15:31:01.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.570+0000 7f4f3259c700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f1c06ecc0 0x7f4f1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:01.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.570+0000 7f4f3259c700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f1c06ecc0 0x7f4f1c071180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009bf0 tx=0x7f4f24009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:01.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.574+0000 7f4f237fe700 1 -- 172.21.15.96:0/1084113767 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4f2805ca30 con 0x7f4f34102680 2024-04-22T15:31:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:01 smithi096 ceph-mon[112443]: pgmap v274: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:01 smithi196 ceph-mon[81815]: pgmap v274: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:01.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.793+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f34197e60 con 0x7f4f1c06ecc0 2024-04-22T15:31:01.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.794+0000 7f4f237fe700 1 -- 172.21.15.96:0/1084113767 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4f34197e60 con 0x7f4f1c06ecc0 2024-04-22T15:31:01.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f1c06ecc0 msgr2=0x7f4f1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f1c06ecc0 0x7f4f1c071180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009bf0 tx=0x7f4f24009260 comp rx=0 tx=0).stop 2024-04-22T15:31:01.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 msgr2=0x7f4f3419cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 0x7f4f3419cda0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009fd0 tx=0x7f4f2800b5a0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 shutdown_connections 2024-04-22T15:31:01.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f1c06ecc0 0x7f4f1c071180 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f34101d60 0x7f4f3419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 --2- 172.21.15.96:0/1084113767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f34102680 0x7f4f3419cda0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 >> 172.21.15.96:0/1084113767 conn(0x7f4f340fb5f0 msgr2=0x7f4f34103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:01.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 shutdown_connections 2024-04-22T15:31:01.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.797+0000 7f4f389be700 1 -- 172.21.15.96:0/1084113767 wait complete. 2024-04-22T15:31:01.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.900+0000 7fe15218f700 1 -- 172.21.15.96:0/3142381104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c101e30 msgr2=0x7fe14c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.900+0000 7fe15218f700 1 --2- 172.21.15.96:0/3142381104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c101e30 0x7fe14c102210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009910 tx=0x7fe13c009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:01.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.900+0000 7fe15218f700 1 -- 172.21.15.96:0/3142381104 shutdown_connections 2024-04-22T15:31:01.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.900+0000 7fe15218f700 1 --2- 172.21.15.96:0/3142381104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe14c102750 0x7fe14c10ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.900+0000 7fe15218f700 1 --2- 172.21.15.96:0/3142381104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c101e30 0x7fe14c102210 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.900+0000 7fe15218f700 1 -- 172.21.15.96:0/3142381104 >> 172.21.15.96:0/3142381104 conn(0x7fe14c0fb600 msgr2=0x7fe14c0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:01.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.900+0000 7fe15218f700 1 -- 172.21.15.96:0/3142381104 shutdown_connections 2024-04-22T15:31:01.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.900+0000 7fe15218f700 1 -- 172.21.15.96:0/3142381104 wait complete. 2024-04-22T15:31:01.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.901+0000 7fe15218f700 1 Processor -- start 2024-04-22T15:31:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.901+0000 7fe15218f700 1 -- start start 2024-04-22T15:31:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.901+0000 7fe15218f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe14c101e30 0x7fe14c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.901+0000 7fe15218f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c102750 0x7fe14c19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.901+0000 7fe15218f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c19d390 con 0x7fe14c102750 2024-04-22T15:31:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.901+0000 7fe15218f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c196970 con 0x7fe14c101e30 2024-04-22T15:31:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe14affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c102750 0x7fe14c19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe14affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c102750 0x7fe14c19cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33634/0 (socket says 172.21.15.96:33634) 2024-04-22T15:31:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe14affd700 1 -- 172.21.15.96:0/506474730 learned_addr learned my addr 172.21.15.96:0/506474730 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe14b7fe700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe14c101e30 0x7fe14c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe14affd700 1 -- 172.21.15.96:0/506474730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe14c101e30 msgr2=0x7fe14c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe14affd700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe14c101e30 0x7fe14c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe14affd700 1 -- 172.21.15.96:0/506474730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13c0095f0 con 0x7fe14c102750 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe14b7fe700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe14c101e30 0x7fe14c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe14affd700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c102750 0x7fe14c19cd70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe14000c8f0 tx=0x7fe14000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe148ff9700 1 -- 172.21.15.96:0/506474730 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe140007a50 con 0x7fe14c102750 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe148ff9700 1 -- 172.21.15.96:0/506474730 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe140007bb0 con 0x7fe14c102750 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14c196c50 con 0x7fe14c102750 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe148ff9700 1 -- 172.21.15.96:0/506474730 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe140018630 con 0x7fe14c102750 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.902+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14c1971a0 con 0x7fe14c102750 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.903+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14c105550 con 0x7fe14c102750 2024-04-22T15:31:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.904+0000 7fe148ff9700 1 -- 172.21.15.96:0/506474730 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe140007d20 con 0x7fe14c102750 2024-04-22T15:31:01.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.904+0000 7fe148ff9700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe13406ecd0 0x7fe134071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:01.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.904+0000 7fe14b7fe700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe13406ecd0 0x7fe134071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:01.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.904+0000 7fe148ff9700 1 -- 172.21.15.96:0/506474730 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe14008ee60 con 0x7fe14c102750 2024-04-22T15:31:01.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.905+0000 7fe14b7fe700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe13406ecd0 0x7fe134071190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe13c00b5c0 tx=0x7fe13c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:01.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:01.911+0000 7fe148ff9700 1 -- 172.21.15.96:0/506474730 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe14005b6f0 con 0x7fe14c102750 2024-04-22T15:31:02.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.118+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe14c10de10 con 0x7fe13406ecd0 2024-04-22T15:31:02.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.130+0000 7fe148ff9700 1 -- 172.21.15.96:0/506474730 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe14c10de10 con 0x7fe13406ecd0 2024-04-22T15:31:02.133 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (9m) 2m ago 21m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (21m) 2m ago 21m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (20m) 2m ago 20m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (6m) 2m ago 21m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (6m) 2m ago 20m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (9m) 2m ago 20m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (2m) 2m ago 15m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (3m) 2m ago 15m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (2m) 2m ago 15m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (15m) 2m ago 15m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (12m) 2m ago 22m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:31:02.134 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (10m) 2m ago 19m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (8m) 2m ago 22m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (7m) 2m ago 19m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (10m) 2m ago 21m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (10m) 2m ago 19m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (6m) 2m ago 19m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (5m) 2m ago 18m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (5m) 2m ago 18m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (4m) 2m ago 17m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (4m) 2m ago 17m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (3m) 2m ago 16m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:31:02.135 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (10m) 2m ago 20m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:31:02.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe13406ecd0 msgr2=0x7fe134071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe13406ecd0 0x7fe134071190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe13c00b5c0 tx=0x7fe13c01a040 comp rx=0 tx=0).stop 2024-04-22T15:31:02.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c102750 msgr2=0x7fe14c19cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c102750 0x7fe14c19cd70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe14000c8f0 tx=0x7fe14000ccb0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 shutdown_connections 2024-04-22T15:31:02.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe13406ecd0 0x7fe134071190 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe14c101e30 0x7fe14c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 --2- 172.21.15.96:0/506474730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14c102750 0x7fe14c19cd70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 >> 172.21.15.96:0/506474730 conn(0x7fe14c0fb600 msgr2=0x7fe14c1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:02.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 shutdown_connections 2024-04-22T15:31:02.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.132+0000 7fe15218f700 1 -- 172.21.15.96:0/506474730 wait complete. 2024-04-22T15:31:02.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.235+0000 7fe92f158700 1 -- 172.21.15.96:0/1791894068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 msgr2=0x7fe928105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.235+0000 7fe92f158700 1 --2- 172.21.15.96:0/1791894068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 0x7fe928105e90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe92000b0a0 tx=0x7fe92000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.235+0000 7fe92f158700 1 -- 172.21.15.96:0/1791894068 shutdown_connections 2024-04-22T15:31:02.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.235+0000 7fe92f158700 1 --2- 172.21.15.96:0/1791894068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9280ffab0 0x7fe9280fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.235+0000 7fe92f158700 1 --2- 172.21.15.96:0/1791894068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 0x7fe928105e90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.235+0000 7fe92f158700 1 -- 172.21.15.96:0/1791894068 >> 172.21.15.96:0/1791894068 conn(0x7fe9280fb5e0 msgr2=0x7fe9280fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:02.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.235+0000 7fe92f158700 1 -- 172.21.15.96:0/1791894068 shutdown_connections 2024-04-22T15:31:02.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.235+0000 7fe92f158700 1 -- 172.21.15.96:0/1791894068 wait complete. 2024-04-22T15:31:02.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe92f158700 1 Processor -- start 2024-04-22T15:31:02.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe92f158700 1 -- start start 2024-04-22T15:31:02.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe92f158700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9280ffab0 0x7fe928197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:02.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe92f158700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 0x7fe928197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:02.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe92f158700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe928198520 con 0x7fe928105ab0 2024-04-22T15:31:02.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe92f158700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe928198660 con 0x7fe9280ffab0 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe927fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 0x7fe928197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe927fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 0x7fe928197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33650/0 (socket says 172.21.15.96:33650) 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe927fff700 1 -- 172.21.15.96:0/2221476402 learned_addr learned my addr 172.21.15.96:0/2221476402 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe92cef4700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9280ffab0 0x7fe928197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe927fff700 1 -- 172.21.15.96:0/2221476402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9280ffab0 msgr2=0x7fe928197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe927fff700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9280ffab0 0x7fe928197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe927fff700 1 -- 172.21.15.96:0/2221476402 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe920009d00 con 0x7fe928105ab0 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.236+0000 7fe92cef4700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9280ffab0 0x7fe928197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.237+0000 7fe927fff700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 0x7fe928197f50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe91800b4b0 tx=0x7fe91800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.237+0000 7fe925ffb700 1 -- 172.21.15.96:0/2221476402 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe918011800 con 0x7fe928105ab0 2024-04-22T15:31:02.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.237+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9281a0a80 con 0x7fe928105ab0 2024-04-22T15:31:02.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.237+0000 7fe925ffb700 1 -- 172.21.15.96:0/2221476402 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe918011e20 con 0x7fe928105ab0 2024-04-22T15:31:02.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.237+0000 7fe925ffb700 1 -- 172.21.15.96:0/2221476402 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe91800f2e0 con 0x7fe928105ab0 2024-04-22T15:31:02.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.237+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9281a0e90 con 0x7fe928105ab0 2024-04-22T15:31:02.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.238+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9281080b0 con 0x7fe928105ab0 2024-04-22T15:31:02.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.238+0000 7fe925ffb700 1 -- 172.21.15.96:0/2221476402 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe918010380 con 0x7fe928105ab0 2024-04-22T15:31:02.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.239+0000 7fe925ffb700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe91406ecc0 0x7fe914071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:02.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.239+0000 7fe92cef4700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe91406ecc0 0x7fe914071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:02.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.239+0000 7fe925ffb700 1 -- 172.21.15.96:0/2221476402 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe91808eab0 con 0x7fe928105ab0 2024-04-22T15:31:02.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.239+0000 7fe92cef4700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe91406ecc0 0x7fe914071180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe92000b830 tx=0x7fe920006210 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:02.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.243+0000 7fe925ffb700 1 -- 172.21.15.96:0/2221476402 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe9180577b0 con 0x7fe928105ab0 2024-04-22T15:31:02.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.530+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe9281011f0 con 0x7fe928105ab0 2024-04-22T15:31:02.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.531+0000 7fe925ffb700 1 -- 172.21.15.96:0/2221476402 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7fe91805ae70 con 0x7fe928105ab0 2024-04-22T15:31:02.534 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:31:02.535 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:31:02.536 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:31:02.536 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:31:02.536 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:31:02.536 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:31:02.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe91406ecc0 msgr2=0x7fe914071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe91406ecc0 0x7fe914071180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe92000b830 tx=0x7fe920006210 comp rx=0 tx=0).stop 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 msgr2=0x7fe928197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 0x7fe928197f50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe91800b4b0 tx=0x7fe91800b870 comp rx=0 tx=0).stop 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 shutdown_connections 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe91406ecc0 0x7fe914071180 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9280ffab0 0x7fe928197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 --2- 172.21.15.96:0/2221476402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe928105ab0 0x7fe928197f50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 >> 172.21.15.96:0/2221476402 conn(0x7fe9280fb5e0 msgr2=0x7fe9280fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 shutdown_connections 2024-04-22T15:31:02.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.533+0000 7fe92f158700 1 -- 172.21.15.96:0/2221476402 wait complete. 2024-04-22T15:31:02.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.637+0000 7f1df2e7e700 1 -- 172.21.15.96:0/4252318096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 msgr2=0x7f1dec105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.637+0000 7f1df2e7e700 1 --2- 172.21.15.96:0/4252318096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 0x7f1dec105810 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009910 tx=0x7f1ddc009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.637+0000 7f1df2e7e700 1 -- 172.21.15.96:0/4252318096 shutdown_connections 2024-04-22T15:31:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.637+0000 7f1df2e7e700 1 --2- 172.21.15.96:0/4252318096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 0x7f1dec105810 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.637+0000 7f1df2e7e700 1 --2- 172.21.15.96:0/4252318096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dec100e10 0x7f1dec1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.637+0000 7f1df2e7e700 1 -- 172.21.15.96:0/4252318096 >> 172.21.15.96:0/4252318096 conn(0x7f1dec0fc6e0 msgr2=0x7f1dec0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.637+0000 7f1df2e7e700 1 -- 172.21.15.96:0/4252318096 shutdown_connections 2024-04-22T15:31:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.637+0000 7f1df2e7e700 1 -- 172.21.15.96:0/4252318096 wait complete. 2024-04-22T15:31:02.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1df2e7e700 1 Processor -- start 2024-04-22T15:31:02.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1df2e7e700 1 -- start start 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1df2e7e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dec100e10 0x7f1dec0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1df2e7e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 0x7f1dec073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1df2e7e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec073c60 con 0x7f1dec1017c0 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1df2e7e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec073da0 con 0x7f1dec100e10 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1de3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 0x7f1dec073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1de3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 0x7f1dec073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33674/0 (socket says 172.21.15.96:33674) 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1de3fff700 1 -- 172.21.15.96:0/2011303921 learned_addr learned my addr 172.21.15.96:0/2011303921 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1df0c1a700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dec100e10 0x7f1dec0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1de3fff700 1 -- 172.21.15.96:0/2011303921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dec100e10 msgr2=0x7f1dec0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1de3fff700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dec100e10 0x7f1dec0731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1de3fff700 1 -- 172.21.15.96:0/2011303921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc0095f0 con 0x7f1dec1017c0 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.638+0000 7f1de3fff700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 0x7f1dec073720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00bd70 tx=0x7f1ddc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:02.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.639+0000 7f1de1ffb700 1 -- 172.21.15.96:0/2011303921 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ddc01d070 con 0x7f1dec1017c0 2024-04-22T15:31:02.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.639+0000 7f1de1ffb700 1 -- 172.21.15.96:0/2011303921 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ddc00f460 con 0x7f1dec1017c0 2024-04-22T15:31:02.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.639+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec074020 con 0x7f1dec1017c0 2024-04-22T15:31:02.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.639+0000 7f1de1ffb700 1 -- 172.21.15.96:0/2011303921 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ddc017630 con 0x7f1dec1017c0 2024-04-22T15:31:02.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.639+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec1a2f60 con 0x7f1dec1017c0 2024-04-22T15:31:02.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.640+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dec109210 con 0x7f1dec1017c0 2024-04-22T15:31:02.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.640+0000 7f1de1ffb700 1 -- 172.21.15.96:0/2011303921 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1ddc00f5d0 con 0x7f1dec1017c0 2024-04-22T15:31:02.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.641+0000 7f1de1ffb700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1dd406ecc0 0x7f1dd4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:02.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.641+0000 7f1de1ffb700 1 -- 172.21.15.96:0/2011303921 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1ddc090320 con 0x7f1dec1017c0 2024-04-22T15:31:02.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.641+0000 7f1df0c1a700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1dd406ecc0 0x7f1dd4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:02.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.641+0000 7f1df0c1a700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1dd406ecc0 0x7f1dd4071180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1de80060b0 tx=0x7f1de8007b80 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:02.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.645+0000 7f1de1ffb700 1 -- 172.21.15.96:0/2011303921 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1ddc059080 con 0x7f1dec1017c0 2024-04-22T15:31:02.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:02 smithi196 ceph-mon[81815]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:02.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.890+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1dec1022f0 con 0x7f1dec1017c0 2024-04-22T15:31:02.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:02 smithi096 ceph-mon[112443]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:02.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.891+0000 7f1de1ffb700 1 -- 172.21.15.96:0/2011303921 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f1ddc005740 con 0x7f1dec1017c0 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:31:02.895 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:31:02.896 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:31:02.897 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:31:02.898 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:31:02.898 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:31:02.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.893+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1dd406ecc0 msgr2=0x7f1dd4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.893+0000 7f1df2e7e700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1dd406ecc0 0x7f1dd4071180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1de80060b0 tx=0x7f1de8007b80 comp rx=0 tx=0).stop 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.893+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 msgr2=0x7f1dec073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.893+0000 7f1df2e7e700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 0x7f1dec073720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00bd70 tx=0x7f1ddc00be50 comp rx=0 tx=0).stop 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.893+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 shutdown_connections 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.893+0000 7f1df2e7e700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1dd406ecc0 0x7f1dd4071180 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.893+0000 7f1df2e7e700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dec100e10 0x7f1dec0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.893+0000 7f1df2e7e700 1 --2- 172.21.15.96:0/2011303921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1dec1017c0 0x7f1dec073720 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.893+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 >> 172.21.15.96:0/2011303921 conn(0x7f1dec0fc6e0 msgr2=0x7f1dec0fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.894+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 shutdown_connections 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.894+0000 7f1df2e7e700 1 -- 172.21.15.96:0/2011303921 wait complete. 2024-04-22T15:31:02.899 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:31:03.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.998+0000 7f5dc04ca700 1 -- 172.21.15.96:0/1667414561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 msgr2=0x7f5db80fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:03.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.998+0000 7f5dc04ca700 1 --2- 172.21.15.96:0/1667414561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 0x7f5db80fff50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009910 tx=0x7f5db4009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:03.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.998+0000 7f5dc04ca700 1 -- 172.21.15.96:0/1667414561 shutdown_connections 2024-04-22T15:31:03.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.998+0000 7f5dc04ca700 1 --2- 172.21.15.96:0/1667414561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 0x7f5db80fff50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.998+0000 7f5dc04ca700 1 --2- 172.21.15.96:0/1667414561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5db8105b60 0x7f5db8105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.998+0000 7f5dc04ca700 1 -- 172.21.15.96:0/1667414561 >> 172.21.15.96:0/1667414561 conn(0x7f5db80fb5e0 msgr2=0x7f5db80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:03.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.998+0000 7f5dc04ca700 1 -- 172.21.15.96:0/1667414561 shutdown_connections 2024-04-22T15:31:03.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.998+0000 7f5dc04ca700 1 -- 172.21.15.96:0/1667414561 wait complete. 2024-04-22T15:31:03.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.999+0000 7f5dc04ca700 1 Processor -- start 2024-04-22T15:31:03.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.999+0000 7f5dc04ca700 1 -- start start 2024-04-22T15:31:03.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.999+0000 7f5dc04ca700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 0x7f5db8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:03.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.999+0000 7f5dc04ca700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5db8105b60 0x7f5db8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:03.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.999+0000 7f5dc04ca700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db81985a0 con 0x7f5db80ffad0 2024-04-22T15:31:03.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.999+0000 7f5dc04ca700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db81986e0 con 0x7f5db8105b60 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.999+0000 7f5dbe266700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 0x7f5db8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.999+0000 7f5dbe266700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 0x7f5db8197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33690/0 (socket says 172.21.15.96:33690) 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:02.999+0000 7f5dbe266700 1 -- 172.21.15.96:0/88136301 learned_addr learned my addr 172.21.15.96:0/88136301 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.000+0000 7f5dbe266700 1 -- 172.21.15.96:0/88136301 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5db8105b60 msgr2=0x7f5db8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.000+0000 7f5dbe266700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5db8105b60 0x7f5db8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.000+0000 7f5dbe266700 1 -- 172.21.15.96:0/88136301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db40095f0 con 0x7f5db80ffad0 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.000+0000 7f5dbe266700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 0x7f5db8197a90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5da800b4d0 tx=0x7f5da800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.000+0000 7f5daf7fe700 1 -- 172.21.15.96:0/88136301 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5da8011800 con 0x7f5db80ffad0 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.000+0000 7f5daf7fe700 1 -- 172.21.15.96:0/88136301 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5da8011e20 con 0x7f5db80ffad0 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.000+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db81a0b00 con 0x7f5db80ffad0 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.000+0000 7f5daf7fe700 1 -- 172.21.15.96:0/88136301 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5da800f2e0 con 0x7f5db80ffad0 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.000+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db81a0f10 con 0x7f5db80ffad0 2024-04-22T15:31:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.001+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db81080f0 con 0x7f5db80ffad0 2024-04-22T15:31:03.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.001+0000 7f5daf7fe700 1 -- 172.21.15.96:0/88136301 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5da8010380 con 0x7f5db80ffad0 2024-04-22T15:31:03.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.002+0000 7f5daf7fe700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5da406ecc0 0x7f5da4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:03.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.002+0000 7f5dbda65700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5da406ecc0 0x7f5da4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:03.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.002+0000 7f5daf7fe700 1 -- 172.21.15.96:0/88136301 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5da808e970 con 0x7f5db80ffad0 2024-04-22T15:31:03.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.002+0000 7f5dbda65700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5da406ecc0 0x7f5da4071180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5db400b5c0 tx=0x7f5db4005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:03.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.006+0000 7f5daf7fe700 1 -- 172.21.15.96:0/88136301 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5da8057750 con 0x7f5db80ffad0 2024-04-22T15:31:03.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.223+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5db8069760 con 0x7f5da406ecc0 2024-04-22T15:31:03.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.225+0000 7f5daf7fe700 1 -- 172.21.15.96:0/88136301 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5db8069760 con 0x7f5da406ecc0 2024-04-22T15:31:03.228 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:31:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:31:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:31:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:31:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:31:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:31:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:31:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:31:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:31:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:31:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:31:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:31:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:31:03.229 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5da406ecc0 msgr2=0x7f5da4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5da406ecc0 0x7f5da4071180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5db400b5c0 tx=0x7f5db4005fd0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 msgr2=0x7f5db8197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 0x7f5db8197a90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5da800b4d0 tx=0x7f5da800b890 comp rx=0 tx=0).stop 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 shutdown_connections 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5da406ecc0 0x7f5da4071180 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db80ffad0 0x7f5db8197a90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 --2- 172.21.15.96:0/88136301 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5db8105b60 0x7f5db8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 >> 172.21.15.96:0/88136301 conn(0x7f5db80fb5e0 msgr2=0x7f5db80fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 shutdown_connections 2024-04-22T15:31:03.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.227+0000 7f5dc04ca700 1 -- 172.21.15.96:0/88136301 wait complete. 2024-04-22T15:31:03.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.330+0000 7fcb48d77700 1 -- 172.21.15.96:0/76155790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 msgr2=0x7fcb440fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:03.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.330+0000 7fcb48d77700 1 --2- 172.21.15.96:0/76155790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 0x7fcb440fff20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcb38009910 tx=0x7fcb38009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:03.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.330+0000 7fcb48d77700 1 -- 172.21.15.96:0/76155790 shutdown_connections 2024-04-22T15:31:03.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.330+0000 7fcb48d77700 1 --2- 172.21.15.96:0/76155790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 0x7fcb440fff20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.330+0000 7fcb48d77700 1 --2- 172.21.15.96:0/76155790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb44105aa0 0x7fcb44105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.330+0000 7fcb48d77700 1 -- 172.21.15.96:0/76155790 >> 172.21.15.96:0/76155790 conn(0x7fcb44076740 msgr2=0x7fcb440fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:03.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.330+0000 7fcb48d77700 1 -- 172.21.15.96:0/76155790 shutdown_connections 2024-04-22T15:31:03.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.330+0000 7fcb48d77700 1 -- 172.21.15.96:0/76155790 wait complete. 2024-04-22T15:31:03.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb48d77700 1 Processor -- start 2024-04-22T15:31:03.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb48d77700 1 -- start start 2024-04-22T15:31:03.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb48d77700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 0x7fcb4419b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:03.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb48d77700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb44105aa0 0x7fcb4419bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:03.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb48d77700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4419c290 con 0x7fcb440ffaa0 2024-04-22T15:31:03.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb48d77700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44195800 con 0x7fcb44105aa0 2024-04-22T15:31:03.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb4259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 0x7fcb4419b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:03.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb4259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 0x7fcb4419b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33704/0 (socket says 172.21.15.96:33704) 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb4259c700 1 -- 172.21.15.96:0/3616177937 learned_addr learned my addr 172.21.15.96:0/3616177937 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb41d9b700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb44105aa0 0x7fcb4419bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb4259c700 1 -- 172.21.15.96:0/3616177937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb44105aa0 msgr2=0x7fcb4419bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb4259c700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb44105aa0 0x7fcb4419bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.331+0000 7fcb4259c700 1 -- 172.21.15.96:0/3616177937 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb380095f0 con 0x7fcb440ffaa0 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.332+0000 7fcb4259c700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 0x7fcb4419b6c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0060b0 tx=0x7fcb3c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.332+0000 7fcb337fe700 1 -- 172.21.15.96:0/3616177937 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3c00dda0 con 0x7fcb440ffaa0 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.332+0000 7fcb337fe700 1 -- 172.21.15.96:0/3616177937 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb3c00f040 con 0x7fcb440ffaa0 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.332+0000 7fcb337fe700 1 -- 172.21.15.96:0/3616177937 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3c014790 con 0x7fcb440ffaa0 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.332+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb44195ae0 con 0x7fcb440ffaa0 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.332+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb44196030 con 0x7fcb440ffaa0 2024-04-22T15:31:03.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.333+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb441080c0 con 0x7fcb440ffaa0 2024-04-22T15:31:03.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.333+0000 7fcb337fe700 1 -- 172.21.15.96:0/3616177937 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcb3c0148f0 con 0x7fcb440ffaa0 2024-04-22T15:31:03.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.334+0000 7fcb337fe700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcb2c06ecc0 0x7fcb2c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:03.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.334+0000 7fcb337fe700 1 -- 172.21.15.96:0/3616177937 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcb3c08e880 con 0x7fcb440ffaa0 2024-04-22T15:31:03.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.334+0000 7fcb41d9b700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcb2c06ecc0 0x7fcb2c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:03.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.334+0000 7fcb41d9b700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcb2c06ecc0 0x7fcb2c071180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb44196c20 tx=0x7fcb3800b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:03.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.339+0000 7fcb337fe700 1 -- 172.21.15.96:0/3616177937 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcb3c05b240 con 0x7fcb440ffaa0 2024-04-22T15:31:03.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.620+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcb44196d60 con 0x7fcb440ffaa0 2024-04-22T15:31:03.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.621+0000 7fcb337fe700 1 -- 172.21.15.96:0/3616177937 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fcb3c08ce60 con 0x7fcb440ffaa0 2024-04-22T15:31:03.624 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:31:03.625 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:31:03.625 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:31:03.625 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:31:03.625 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:31:03.625 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:31:03.625 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:31:03.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcb2c06ecc0 msgr2=0x7fcb2c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:03.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcb2c06ecc0 0x7fcb2c071180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb44196c20 tx=0x7fcb3800b560 comp rx=0 tx=0).stop 2024-04-22T15:31:03.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 msgr2=0x7fcb4419b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:03.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 0x7fcb4419b6c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0060b0 tx=0x7fcb3c00d3b0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 shutdown_connections 2024-04-22T15:31:03.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcb2c06ecc0 0x7fcb2c071180 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb440ffaa0 0x7fcb4419b6c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 --2- 172.21.15.96:0/3616177937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb44105aa0 0x7fcb4419bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:03.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 >> 172.21.15.96:0/3616177937 conn(0x7fcb44076740 msgr2=0x7fcb440fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:03.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 shutdown_connections 2024-04-22T15:31:03.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:03.623+0000 7fcb48d77700 1 -- 172.21.15.96:0/3616177937 wait complete. 2024-04-22T15:31:03.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:03 smithi196 ceph-mon[81815]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:03.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:03 smithi196 ceph-mon[81815]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:03.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:03 smithi196 ceph-mon[81815]: pgmap v275: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:03.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:03 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2221476402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:31:03.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:03 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2011303921' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:31:03.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:03 smithi196 ceph-mon[81815]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:03.811 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:03 smithi096 ceph-mon[112443]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:03.812 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:03 smithi096 ceph-mon[112443]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:03.812 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:03 smithi096 ceph-mon[112443]: pgmap v275: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:03.812 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:03 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2221476402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:31:03.812 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:03 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2011303921' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:31:03.812 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:03 smithi096 ceph-mon[112443]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:04.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:04 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3616177937' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:31:04.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:04 smithi196 ceph-mon[81815]: pgmap v276: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:04 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3616177937' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:31:04.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:04 smithi096 ceph-mon[112443]: pgmap v276: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:07 smithi096 ceph-mon[112443]: pgmap v277: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:07.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:07 smithi196 ceph-mon[81815]: pgmap v277: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:31:08.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:31:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:09 smithi096 ceph-mon[112443]: pgmap v278: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:09 smithi196 ceph-mon[81815]: pgmap v278: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:11 smithi096 ceph-mon[112443]: pgmap v279: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:11 smithi196 ceph-mon[81815]: pgmap v279: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:13 smithi096 ceph-mon[112443]: pgmap v280: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:13.754 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:13 smithi196 ceph-mon[81815]: pgmap v280: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:15 smithi096 ceph-mon[112443]: pgmap v281: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:15.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:15 smithi196 ceph-mon[81815]: pgmap v281: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:17 smithi096 ceph-mon[112443]: pgmap v282: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:17 smithi196 ceph-mon[81815]: pgmap v282: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:19 smithi096 ceph-mon[112443]: pgmap v283: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:19 smithi196 ceph-mon[81815]: pgmap v283: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:21 smithi096 ceph-mon[112443]: pgmap v284: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:21.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:21 smithi196 ceph-mon[81815]: pgmap v284: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:23 smithi096 ceph-mon[112443]: pgmap v285: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:31:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:23 smithi196 ceph-mon[81815]: pgmap v285: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:31:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:25 smithi096 ceph-mon[112443]: pgmap v286: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:25 smithi196 ceph-mon[81815]: pgmap v286: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:27 smithi096 ceph-mon[112443]: pgmap v287: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:27 smithi196 ceph-mon[81815]: pgmap v287: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:29.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:29 smithi096 ceph-mon[112443]: pgmap v288: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:29.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:29 smithi196 ceph-mon[81815]: pgmap v288: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:31 smithi096 ceph-mon[112443]: pgmap v289: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:31 smithi196 ceph-mon[81815]: pgmap v289: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:33.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:33 smithi196 ceph-mon[81815]: pgmap v290: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:33 smithi096 ceph-mon[112443]: pgmap v290: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:33.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.742+0000 7f96c9694700 1 -- 172.21.15.96:0/170552078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4105aa0 msgr2=0x7f96c4105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:33.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.742+0000 7f96c9694700 1 --2- 172.21.15.96:0/170552078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4105aa0 0x7f96c4105e80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009910 tx=0x7f96b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:33.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.742+0000 7f96c9694700 1 -- 172.21.15.96:0/170552078 shutdown_connections 2024-04-22T15:31:33.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.742+0000 7f96c9694700 1 --2- 172.21.15.96:0/170552078 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c40ffaa0 0x7f96c40fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:33.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.742+0000 7f96c9694700 1 --2- 172.21.15.96:0/170552078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4105aa0 0x7f96c4105e80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:33.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.742+0000 7f96c9694700 1 -- 172.21.15.96:0/170552078 >> 172.21.15.96:0/170552078 conn(0x7f96c4076740 msgr2=0x7f96c40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:33.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.742+0000 7f96c9694700 1 -- 172.21.15.96:0/170552078 shutdown_connections 2024-04-22T15:31:33.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.742+0000 7f96c9694700 1 -- 172.21.15.96:0/170552078 wait complete. 2024-04-22T15:31:33.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c9694700 1 Processor -- start 2024-04-22T15:31:33.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c9694700 1 -- start start 2024-04-22T15:31:33.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c9694700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c40ffaa0 0x7f96c4196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:33.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c9694700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4197290 0x7f96c419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:33.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c9694700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c41978b0 con 0x7f96c4197290 2024-04-22T15:31:33.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c9694700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c41979f0 con 0x7f96c40ffaa0 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4197290 0x7f96c419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4197290 0x7f96c419a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57950/0 (socket says 172.21.15.96:57950) 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c27fc700 1 -- 172.21.15.96:0/723727317 learned_addr learned my addr 172.21.15.96:0/723727317 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.743+0000 7f96c2ffd700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c40ffaa0 0x7f96c4196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.744+0000 7f96c27fc700 1 -- 172.21.15.96:0/723727317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c40ffaa0 msgr2=0x7f96c4196d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.744+0000 7f96c27fc700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c40ffaa0 0x7f96c4196d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.744+0000 7f96c27fc700 1 -- 172.21.15.96:0/723727317 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b40095f0 con 0x7f96c4197290 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.744+0000 7f96c27fc700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4197290 0x7f96c419a540 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f96b800d8d0 tx=0x7f96b800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.744+0000 7f96abfff700 1 -- 172.21.15.96:0/723727317 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b800f860 con 0x7f96c4197290 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.744+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c419ab40 con 0x7f96c4197290 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.744+0000 7f96abfff700 1 -- 172.21.15.96:0/723727317 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96b800fe80 con 0x7f96c4197290 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.744+0000 7f96abfff700 1 -- 172.21.15.96:0/723727317 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b800e5e0 con 0x7f96c4197290 2024-04-22T15:31:33.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.744+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c419b090 con 0x7f96c4197290 2024-04-22T15:31:33.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.745+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c4105150 con 0x7f96c4197290 2024-04-22T15:31:33.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.745+0000 7f96abfff700 1 -- 172.21.15.96:0/723727317 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f96b800f9c0 con 0x7f96c4197290 2024-04-22T15:31:33.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.746+0000 7f96abfff700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96ac06ecc0 0x7f96ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:33.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.746+0000 7f96abfff700 1 -- 172.21.15.96:0/723727317 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f96b808eda0 con 0x7f96c4197290 2024-04-22T15:31:33.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.746+0000 7f96c2ffd700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96ac06ecc0 0x7f96ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:33.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.746+0000 7f96c2ffd700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96ac06ecc0 0x7f96ac071180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f96b4006010 tx=0x7f96b401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:33.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.751+0000 7f96abfff700 1 -- 172.21.15.96:0/723727317 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f96b8057b00 con 0x7f96c4197290 2024-04-22T15:31:33.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.970+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f96c419b380 con 0x7f96ac06ecc0 2024-04-22T15:31:33.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.971+0000 7f96abfff700 1 -- 172.21.15.96:0/723727317 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f96c419b380 con 0x7f96ac06ecc0 2024-04-22T15:31:33.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96ac06ecc0 msgr2=0x7f96ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:33.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96ac06ecc0 0x7f96ac071180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f96b4006010 tx=0x7f96b401a040 comp rx=0 tx=0).stop 2024-04-22T15:31:33.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4197290 msgr2=0x7f96c419a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:33.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4197290 0x7f96c419a540 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f96b800d8d0 tx=0x7f96b800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:31:33.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 shutdown_connections 2024-04-22T15:31:33.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96ac06ecc0 0x7f96ac071180 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:33.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96c40ffaa0 0x7f96c4196d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:33.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 --2- 172.21.15.96:0/723727317 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c4197290 0x7f96c419a540 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:33.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 >> 172.21.15.96:0/723727317 conn(0x7f96c4076740 msgr2=0x7f96c40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:33.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 shutdown_connections 2024-04-22T15:31:33.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:33.974+0000 7f96c9694700 1 -- 172.21.15.96:0/723727317 wait complete. 2024-04-22T15:31:33.988 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:31:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.078+0000 7fda16cf7700 1 -- 172.21.15.96:0/588868285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda101003e0 msgr2=0x7fda101007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.078+0000 7fda16cf7700 1 --2- 172.21.15.96:0/588868285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda101003e0 0x7fda101007c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda04009910 tx=0x7fda04009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.078+0000 7fda16cf7700 1 -- 172.21.15.96:0/588868285 shutdown_connections 2024-04-22T15:31:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.078+0000 7fda16cf7700 1 --2- 172.21.15.96:0/588868285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda10100d90 0x7fda10104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.078+0000 7fda16cf7700 1 --2- 172.21.15.96:0/588868285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda101003e0 0x7fda101007c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.078+0000 7fda16cf7700 1 -- 172.21.15.96:0/588868285 >> 172.21.15.96:0/588868285 conn(0x7fda10075f30 msgr2=0x7fda100fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.078+0000 7fda16cf7700 1 -- 172.21.15.96:0/588868285 shutdown_connections 2024-04-22T15:31:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.078+0000 7fda16cf7700 1 -- 172.21.15.96:0/588868285 wait complete. 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda16cf7700 1 Processor -- start 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda16cf7700 1 -- start start 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda16cf7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda101003e0 0x7fda10197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda16cf7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda10100d90 0x7fda10197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda16cf7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda10198530 con 0x7fda10100d90 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda16cf7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda10198670 con 0x7fda101003e0 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda0ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda10100d90 0x7fda10197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda0ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda10100d90 0x7fda10197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57962/0 (socket says 172.21.15.96:57962) 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda0ffff700 1 -- 172.21.15.96:0/944538572 learned_addr learned my addr 172.21.15.96:0/944538572 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.079+0000 7fda14a93700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda101003e0 0x7fda10197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.080+0000 7fda0ffff700 1 -- 172.21.15.96:0/944538572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda101003e0 msgr2=0x7fda10197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.080+0000 7fda0ffff700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda101003e0 0x7fda10197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.080+0000 7fda0ffff700 1 -- 172.21.15.96:0/944538572 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda040095f0 con 0x7fda10100d90 2024-04-22T15:31:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.080+0000 7fda0ffff700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda10100d90 0x7fda10197f60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00d8d0 tx=0x7fd9fc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.080+0000 7fda0dffb700 1 -- 172.21.15.96:0/944538572 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9fc00f860 con 0x7fda10100d90 2024-04-22T15:31:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.080+0000 7fda0dffb700 1 -- 172.21.15.96:0/944538572 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9fc00fe80 con 0x7fda10100d90 2024-04-22T15:31:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.080+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda101a0a90 con 0x7fda10100d90 2024-04-22T15:31:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.080+0000 7fda0dffb700 1 -- 172.21.15.96:0/944538572 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9fc00e5e0 con 0x7fda10100d90 2024-04-22T15:31:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.080+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda101a0ea0 con 0x7fda10100d90 2024-04-22T15:31:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.081+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda101080f0 con 0x7fda10100d90 2024-04-22T15:31:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.082+0000 7fda0dffb700 1 -- 172.21.15.96:0/944538572 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd9fc010460 con 0x7fda10100d90 2024-04-22T15:31:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.082+0000 7fda0dffb700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda0006ecd0 0x7fda00071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.083+0000 7fda0dffb700 1 -- 172.21.15.96:0/944538572 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd9fc08eab0 con 0x7fda10100d90 2024-04-22T15:31:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.083+0000 7fda14a93700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda0006ecd0 0x7fda00071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:34.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.083+0000 7fda14a93700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda0006ecd0 0x7fda00071190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda04006010 tx=0x7fda0401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:34.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.087+0000 7fda0dffb700 1 -- 172.21.15.96:0/944538572 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd9fc057810 con 0x7fda10100d90 2024-04-22T15:31:34.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.308+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda1010cd20 con 0x7fda0006ecd0 2024-04-22T15:31:34.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.309+0000 7fda0dffb700 1 -- 172.21.15.96:0/944538572 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fda1010cd20 con 0x7fda0006ecd0 2024-04-22T15:31:34.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda0006ecd0 msgr2=0x7fda00071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda0006ecd0 0x7fda00071190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda04006010 tx=0x7fda0401a040 comp rx=0 tx=0).stop 2024-04-22T15:31:34.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda10100d90 msgr2=0x7fda10197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda10100d90 0x7fda10197f60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00d8d0 tx=0x7fd9fc00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 shutdown_connections 2024-04-22T15:31:34.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda0006ecd0 0x7fda00071190 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda101003e0 0x7fda10197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 --2- 172.21.15.96:0/944538572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda10100d90 0x7fda10197f60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 >> 172.21.15.96:0/944538572 conn(0x7fda10075f30 msgr2=0x7fda101034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:34.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.311+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 shutdown_connections 2024-04-22T15:31:34.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.312+0000 7fda16cf7700 1 -- 172.21.15.96:0/944538572 wait complete. 2024-04-22T15:31:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.415+0000 7f28fb72d700 1 -- 172.21.15.96:0/4122512212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f40ffab0 msgr2=0x7f28f40fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.415+0000 7f28fb72d700 1 --2- 172.21.15.96:0/4122512212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f40ffab0 0x7f28f40fff30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009910 tx=0x7f28f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.415+0000 7f28fb72d700 1 -- 172.21.15.96:0/4122512212 shutdown_connections 2024-04-22T15:31:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.415+0000 7f28fb72d700 1 --2- 172.21.15.96:0/4122512212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f40ffab0 0x7f28f40fff30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.415+0000 7f28fb72d700 1 --2- 172.21.15.96:0/4122512212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28f4105ab0 0x7f28f4105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.415+0000 7f28fb72d700 1 -- 172.21.15.96:0/4122512212 >> 172.21.15.96:0/4122512212 conn(0x7f28f40fb5e0 msgr2=0x7f28f40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.415+0000 7f28fb72d700 1 -- 172.21.15.96:0/4122512212 shutdown_connections 2024-04-22T15:31:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.415+0000 7f28fb72d700 1 -- 172.21.15.96:0/4122512212 wait complete. 2024-04-22T15:31:34.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28fb72d700 1 Processor -- start 2024-04-22T15:31:34.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28fb72d700 1 -- start start 2024-04-22T15:31:34.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28fb72d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28f40ffab0 0x7f28f4198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:34.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28fb72d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f4105ab0 0x7f28f4199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:34.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28fb72d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f41998d0 con 0x7f28f4105ab0 2024-04-22T15:31:34.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28fb72d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f419c540 con 0x7f28f40ffab0 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28f8cc8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f4105ab0 0x7f28f4199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28f8cc8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f4105ab0 0x7f28f4199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57982/0 (socket says 172.21.15.96:57982) 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28f8cc8700 1 -- 172.21.15.96:0/2992867062 learned_addr learned my addr 172.21.15.96:0/2992867062 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28f94c9700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28f40ffab0 0x7f28f4198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28f8cc8700 1 -- 172.21.15.96:0/2992867062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28f40ffab0 msgr2=0x7f28f4198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28f8cc8700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28f40ffab0 0x7f28f4198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28f8cc8700 1 -- 172.21.15.96:0/2992867062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f00095f0 con 0x7f28f4105ab0 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.416+0000 7f28f94c9700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28f40ffab0 0x7f28f4198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.417+0000 7f28f8cc8700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f4105ab0 0x7f28f4199240 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f28f000bd70 tx=0x7f28f000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.417+0000 7f28ea7fc700 1 -- 172.21.15.96:0/2992867062 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28f001d070 con 0x7f28f4105ab0 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.417+0000 7f28ea7fc700 1 -- 172.21.15.96:0/2992867062 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28f00056e0 con 0x7f28f4105ab0 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.417+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f419c7c0 con 0x7f28f4105ab0 2024-04-22T15:31:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.417+0000 7f28ea7fc700 1 -- 172.21.15.96:0/2992867062 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28f0017400 con 0x7f28f4105ab0 2024-04-22T15:31:34.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.417+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f419ccb0 con 0x7f28f4105ab0 2024-04-22T15:31:34.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.418+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f41011f0 con 0x7f28f4105ab0 2024-04-22T15:31:34.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.419+0000 7f28ea7fc700 1 -- 172.21.15.96:0/2992867062 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f28f0003a60 con 0x7f28f4105ab0 2024-04-22T15:31:34.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.419+0000 7f28ea7fc700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28e006ecc0 0x7f28e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:34.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.419+0000 7f28ea7fc700 1 -- 172.21.15.96:0/2992867062 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f28f0090280 con 0x7f28f4105ab0 2024-04-22T15:31:34.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.419+0000 7f28f94c9700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28e006ecc0 0x7f28e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:34.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.420+0000 7f28f94c9700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28e006ecc0 0x7f28e0071180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f28e4005950 tx=0x7f28e40058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:34.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.424+0000 7f28ea7fc700 1 -- 172.21.15.96:0/2992867062 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f28f0059060 con 0x7f28f4105ab0 2024-04-22T15:31:34.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.631+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f28f410ccb0 con 0x7f28e006ecc0 2024-04-22T15:31:34.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.642+0000 7f28ea7fc700 1 -- 172.21.15.96:0/2992867062 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f28f410ccb0 con 0x7f28e006ecc0 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (10m) 3m ago 21m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (21m) 3m ago 21m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (20m) 3m ago 20m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (7m) 3m ago 21m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (7m) 3m ago 20m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (9m) 3m ago 21m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (3m) 3m ago 15m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (3m) 3m ago 15m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (3m) 3m ago 15m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:31:34.646 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (15m) 3m ago 15m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (12m) 3m ago 23m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (11m) 3m ago 20m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (8m) 3m ago 23m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (8m) 3m ago 20m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (11m) 3m ago 21m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (11m) 3m ago 20m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (7m) 3m ago 19m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (6m) 3m ago 19m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (5m) 3m ago 18m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (5m) 3m ago 18m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (4m) 3m ago 17m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (4m) 3m ago 17m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:31:34.647 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (10m) 3m ago 21m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:31:34.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28e006ecc0 msgr2=0x7f28e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28e006ecc0 0x7f28e0071180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f28e4005950 tx=0x7f28e40058e0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f4105ab0 msgr2=0x7f28f4199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f4105ab0 0x7f28f4199240 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f28f000bd70 tx=0x7f28f000be50 comp rx=0 tx=0).stop 2024-04-22T15:31:34.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 shutdown_connections 2024-04-22T15:31:34.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f28e006ecc0 0x7f28e0071180 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28f40ffab0 0x7f28f4198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 --2- 172.21.15.96:0/2992867062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28f4105ab0 0x7f28f4199240 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 >> 172.21.15.96:0/2992867062 conn(0x7f28f40fb5e0 msgr2=0x7f28f40691b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:34.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 shutdown_connections 2024-04-22T15:31:34.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.644+0000 7f28fb72d700 1 -- 172.21.15.96:0/2992867062 wait complete. 2024-04-22T15:31:34.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.745+0000 7f2f4c98c700 1 -- 172.21.15.96:0/648126572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 msgr2=0x7f2f481057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.745+0000 7f2f4c98c700 1 --2- 172.21.15.96:0/648126572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 0x7f2f481057f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009910 tx=0x7f2f3c009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:34.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 -- 172.21.15.96:0/648126572 shutdown_connections 2024-04-22T15:31:34.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 --2- 172.21.15.96:0/648126572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 0x7f2f481057f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 --2- 172.21.15.96:0/648126572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f48100df0 0x7f2f481011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 -- 172.21.15.96:0/648126572 >> 172.21.15.96:0/648126572 conn(0x7f2f480fc6a0 msgr2=0x7f2f480feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 -- 172.21.15.96:0/648126572 shutdown_connections 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 -- 172.21.15.96:0/648126572 wait complete. 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 Processor -- start 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 -- start start 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f48100df0 0x7f2f4819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 0x7f2f4819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4819d370 con 0x7f2f481017a0 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.746+0000 7f2f4c98c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f48196950 con 0x7f2f48100df0 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f45d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 0x7f2f4819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f45d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 0x7f2f4819cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57992/0 (socket says 172.21.15.96:57992) 2024-04-22T15:31:34.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f45d9b700 1 -- 172.21.15.96:0/3228993092 learned_addr learned my addr 172.21.15.96:0/3228993092 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f4659c700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f48100df0 0x7f2f4819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f45d9b700 1 -- 172.21.15.96:0/3228993092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f48100df0 msgr2=0x7f2f4819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f45d9b700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f48100df0 0x7f2f4819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f45d9b700 1 -- 172.21.15.96:0/3228993092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c0095f0 con 0x7f2f481017a0 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f4659c700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f48100df0 0x7f2f4819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f45d9b700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 0x7f2f4819cd50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00bd70 tx=0x7f2f3c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f377fe700 1 -- 172.21.15.96:0/3228993092 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3c01d070 con 0x7f2f481017a0 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f377fe700 1 -- 172.21.15.96:0/3228993092 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f3c0056e0 con 0x7f2f481017a0 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f48196bd0 con 0x7f2f481017a0 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f377fe700 1 -- 172.21.15.96:0/3228993092 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3c017400 con 0x7f2f481017a0 2024-04-22T15:31:34.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.747+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f481970c0 con 0x7f2f481017a0 2024-04-22T15:31:34.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.748+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f481091f0 con 0x7f2f481017a0 2024-04-22T15:31:34.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.749+0000 7f2f377fe700 1 -- 172.21.15.96:0/3228993092 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2f3c00f460 con 0x7f2f481017a0 2024-04-22T15:31:34.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.749+0000 7f2f377fe700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f3006ecc0 0x7f2f30071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:34.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.749+0000 7f2f377fe700 1 -- 172.21.15.96:0/3228993092 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2f3c090230 con 0x7f2f481017a0 2024-04-22T15:31:34.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.749+0000 7f2f4659c700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f3006ecc0 0x7f2f30071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:34.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.750+0000 7f2f4659c700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f3006ecc0 0x7f2f30071180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2f38005950 tx=0x7f2f380058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:34.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:34.754+0000 7f2f377fe700 1 -- 172.21.15.96:0/3228993092 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2f3c05cc00 con 0x7f2f481017a0 2024-04-22T15:31:35.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.037+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2f481022d0 con 0x7f2f481017a0 2024-04-22T15:31:35.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.037+0000 7f2f377fe700 1 -- 172.21.15.96:0/3228993092 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f2f3c05c790 con 0x7f2f481017a0 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:31:35.041 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:31:35.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:31:35.042 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:31:35.042 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:31:35.042 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:31:35.042 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:31:35.042 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f3006ecc0 msgr2=0x7f2f30071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f3006ecc0 0x7f2f30071180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2f38005950 tx=0x7f2f380058e0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 msgr2=0x7f2f4819cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 0x7f2f4819cd50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00bd70 tx=0x7f2f3c00be50 comp rx=0 tx=0).stop 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 shutdown_connections 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f3006ecc0 0x7f2f30071180 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f48100df0 0x7f2f4819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 --2- 172.21.15.96:0/3228993092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f481017a0 0x7f2f4819cd50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 >> 172.21.15.96:0/3228993092 conn(0x7f2f480fc6a0 msgr2=0x7f2f480fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 shutdown_connections 2024-04-22T15:31:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.040+0000 7f2f4c98c700 1 -- 172.21.15.96:0/3228993092 wait complete. 2024-04-22T15:31:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.144+0000 7fab5c90f700 1 -- 172.21.15.96:0/581271128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab581017d0 msgr2=0x7fab58105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.144+0000 7fab5c90f700 1 --2- 172.21.15.96:0/581271128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab581017d0 0x7fab58105820 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009910 tx=0x7fab4c009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.145+0000 7fab5c90f700 1 -- 172.21.15.96:0/581271128 shutdown_connections 2024-04-22T15:31:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.145+0000 7fab5c90f700 1 --2- 172.21.15.96:0/581271128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab581017d0 0x7fab58105820 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.145+0000 7fab5c90f700 1 --2- 172.21.15.96:0/581271128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab58100e20 0x7fab58101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.145+0000 7fab5c90f700 1 -- 172.21.15.96:0/581271128 >> 172.21.15.96:0/581271128 conn(0x7fab580fc6d0 msgr2=0x7fab580feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:35.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.145+0000 7fab5c90f700 1 -- 172.21.15.96:0/581271128 shutdown_connections 2024-04-22T15:31:35.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.145+0000 7fab5c90f700 1 -- 172.21.15.96:0/581271128 wait complete. 2024-04-22T15:31:35.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.145+0000 7fab5c90f700 1 Processor -- start 2024-04-22T15:31:35.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.145+0000 7fab5c90f700 1 -- start start 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5c90f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab58100e20 0x7fab58198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5c90f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab581017d0 0x7fab58199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5c90f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab58199b20 con 0x7fab58100e20 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5c90f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5819d7a0 con 0x7fab581017d0 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab58100e20 0x7fab58198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab58100e20 0x7fab58198f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57996/0 (socket says 172.21.15.96:57996) 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5659c700 1 -- 172.21.15.96:0/2956303074 learned_addr learned my addr 172.21.15.96:0/2956303074 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab55d9b700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab581017d0 0x7fab58199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5659c700 1 -- 172.21.15.96:0/2956303074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab581017d0 msgr2=0x7fab58199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5659c700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab581017d0 0x7fab58199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5659c700 1 -- 172.21.15.96:0/2956303074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c0095f0 con 0x7fab58100e20 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5659c700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab58100e20 0x7fab58198f00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fab4800b4b0 tx=0x7fab4800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab477fe700 1 -- 172.21.15.96:0/2956303074 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab48011800 con 0x7fab58100e20 2024-04-22T15:31:35.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5819da80 con 0x7fab58100e20 2024-04-22T15:31:35.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab477fe700 1 -- 172.21.15.96:0/2956303074 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab48011e20 con 0x7fab58100e20 2024-04-22T15:31:35.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab477fe700 1 -- 172.21.15.96:0/2956303074 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab4800f2e0 con 0x7fab58100e20 2024-04-22T15:31:35.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.146+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5819dfd0 con 0x7fab58100e20 2024-04-22T15:31:35.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.147+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab58102300 con 0x7fab58100e20 2024-04-22T15:31:35.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.148+0000 7fab477fe700 1 -- 172.21.15.96:0/2956303074 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fab48010380 con 0x7fab58100e20 2024-04-22T15:31:35.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.148+0000 7fab477fe700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab4006ecc0 0x7fab40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:35.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.148+0000 7fab477fe700 1 -- 172.21.15.96:0/2956303074 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fab4808ea50 con 0x7fab58100e20 2024-04-22T15:31:35.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.148+0000 7fab55d9b700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab4006ecc0 0x7fab40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:35.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.149+0000 7fab55d9b700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab4006ecc0 0x7fab40071180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fab5819a410 tx=0x7fab4c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:35.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.153+0000 7fab477fe700 1 -- 172.21.15.96:0/2956303074 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fab48090e60 con 0x7fab58100e20 2024-04-22T15:31:35.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.395+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fab5810de40 con 0x7fab58100e20 2024-04-22T15:31:35.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.395+0000 7fab477fe700 1 -- 172.21.15.96:0/2956303074 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fab480577b0 con 0x7fab58100e20 2024-04-22T15:31:35.399 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:31:35.399 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:31:35.399 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:31:35.399 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:31:35.399 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:31:35.399 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:31:35.399 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:31:35.399 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:31:35.399 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:31:35.400 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:31:35.400 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:31:35.400 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:31:35.400 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:31:35.400 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:31:35.400 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:31:35.400 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:31:35.400 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:31:35.400 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:31:35.401 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:31:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:31:35.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab4006ecc0 msgr2=0x7fab40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab4006ecc0 0x7fab40071180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fab5819a410 tx=0x7fab4c01a040 comp rx=0 tx=0).stop 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab58100e20 msgr2=0x7fab58198f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab58100e20 0x7fab58198f00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fab4800b4b0 tx=0x7fab4800b870 comp rx=0 tx=0).stop 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 shutdown_connections 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab4006ecc0 0x7fab40071180 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab58100e20 0x7fab58198f00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 --2- 172.21.15.96:0/2956303074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab581017d0 0x7fab58199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 >> 172.21.15.96:0/2956303074 conn(0x7fab580fc6d0 msgr2=0x7fab580745e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 shutdown_connections 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.397+0000 7fab5c90f700 1 -- 172.21.15.96:0/2956303074 wait complete. 2024-04-22T15:31:35.404 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:31:35.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.503+0000 7fc2d1802700 1 -- 172.21.15.96:0/3762086205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc105aa0 msgr2=0x7fc2cc105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.503+0000 7fc2d1802700 1 --2- 172.21.15.96:0/3762086205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc105aa0 0x7fc2cc105e80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009910 tx=0x7fc2bc009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:35.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.503+0000 7fc2d1802700 1 -- 172.21.15.96:0/3762086205 shutdown_connections 2024-04-22T15:31:35.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.503+0000 7fc2d1802700 1 --2- 172.21.15.96:0/3762086205 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2cc0ffaa0 0x7fc2cc0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.503+0000 7fc2d1802700 1 --2- 172.21.15.96:0/3762086205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc105aa0 0x7fc2cc105e80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.503+0000 7fc2d1802700 1 -- 172.21.15.96:0/3762086205 >> 172.21.15.96:0/3762086205 conn(0x7fc2cc076740 msgr2=0x7fc2cc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:35.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.503+0000 7fc2d1802700 1 -- 172.21.15.96:0/3762086205 shutdown_connections 2024-04-22T15:31:35.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.503+0000 7fc2d1802700 1 -- 172.21.15.96:0/3762086205 wait complete. 2024-04-22T15:31:35.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.504+0000 7fc2d1802700 1 Processor -- start 2024-04-22T15:31:35.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.504+0000 7fc2d1802700 1 -- start start 2024-04-22T15:31:35.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.504+0000 7fc2d1802700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc0ffaa0 0x7fc2cc0fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:35.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.504+0000 7fc2d1802700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2cc105aa0 0x7fc2cc0fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:35.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.504+0000 7fc2d1802700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0fc6a0 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.504+0000 7fc2d1802700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0fc810 con 0x7fc2cc105aa0 2024-04-22T15:31:35.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.504+0000 7fc2caffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc0ffaa0 0x7fc2cc0fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:35.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.504+0000 7fc2caffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc0ffaa0 0x7fc2cc0fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58014/0 (socket says 172.21.15.96:58014) 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2caffd700 1 -- 172.21.15.96:0/3467995104 learned_addr learned my addr 172.21.15.96:0/3467995104 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2ca7fc700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2cc105aa0 0x7fc2cc0fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2caffd700 1 -- 172.21.15.96:0/3467995104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2cc105aa0 msgr2=0x7fc2cc0fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2caffd700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2cc105aa0 0x7fc2cc0fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2caffd700 1 -- 172.21.15.96:0/3467995104 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0095f0 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2caffd700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc0ffaa0 0x7fc2cc0fbc20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00bd70 tx=0x7fc2bc00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2d0800700 1 -- 172.21.15.96:0/3467995104 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2bc01d070 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2d0800700 1 -- 172.21.15.96:0/3467995104 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2bc0056e0 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc0fca90 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2d0800700 1 -- 172.21.15.96:0/3467995104 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2bc00fe40 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.505+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc1a2fe0 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.506+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc1011e0 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.507+0000 7fc2d0800700 1 -- 172.21.15.96:0/3467995104 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc2bc00f680 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.507+0000 7fc2d0800700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2b406ec60 0x7fc2b4071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:35.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.507+0000 7fc2ca7fc700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2b406ec60 0x7fc2b4071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:35.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.507+0000 7fc2d0800700 1 -- 172.21.15.96:0/3467995104 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc2bc090120 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.508+0000 7fc2ca7fc700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2b406ec60 0x7fc2b4071120 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0fd300 tx=0x7fc2c000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:35.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.512+0000 7fc2d0800700 1 -- 172.21.15.96:0/3467995104 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc2bc058e80 con 0x7fc2cc0ffaa0 2024-04-22T15:31:35.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.728+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2cc0fca90 con 0x7fc2b406ec60 2024-04-22T15:31:35.731 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:35 smithi096 ceph-mon[112443]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:35.732 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:35 smithi096 ceph-mon[112443]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:35.732 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:35 smithi096 ceph-mon[112443]: pgmap v291: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:35.732 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:35 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3228993092' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:31:35.732 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:35 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2956303074' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:31:35.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.729+0000 7fc2d0800700 1 -- 172.21.15.96:0/3467995104 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc2cc0fca90 con 0x7fc2b406ec60 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:31:35.733 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:31:35.734 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:31:35.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2b406ec60 msgr2=0x7fc2b4071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2b406ec60 0x7fc2b4071120 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0fd300 tx=0x7fc2c000b410 comp rx=0 tx=0).stop 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc0ffaa0 msgr2=0x7fc2cc0fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc0ffaa0 0x7fc2cc0fbc20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00bd70 tx=0x7fc2bc00bda0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 shutdown_connections 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2b406ec60 0x7fc2b4071120 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2cc0ffaa0 0x7fc2cc0fbc20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 --2- 172.21.15.96:0/3467995104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2cc105aa0 0x7fc2cc0fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 >> 172.21.15.96:0/3467995104 conn(0x7fc2cc076740 msgr2=0x7fc2cc068860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 shutdown_connections 2024-04-22T15:31:35.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.731+0000 7fc2d1802700 1 -- 172.21.15.96:0/3467995104 wait complete. 2024-04-22T15:31:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:35 smithi196 ceph-mon[81815]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:35 smithi196 ceph-mon[81815]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:35 smithi196 ceph-mon[81815]: pgmap v291: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:35 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3228993092' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:31:35.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:35 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2956303074' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:31:35.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.834+0000 7fb1147a4700 1 -- 172.21.15.96:0/2329307158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c100d90 msgr2=0x7fb10c104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.834+0000 7fb1147a4700 1 --2- 172.21.15.96:0/2329307158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c100d90 0x7fb10c104760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb108009910 tx=0x7fb108009c20 comp rx=0 tx=0).stop 2024-04-22T15:31:35.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.834+0000 7fb1147a4700 1 -- 172.21.15.96:0/2329307158 shutdown_connections 2024-04-22T15:31:35.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.834+0000 7fb1147a4700 1 --2- 172.21.15.96:0/2329307158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c100d90 0x7fb10c104760 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.834+0000 7fb1147a4700 1 --2- 172.21.15.96:0/2329307158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c1003e0 0x7fb10c1007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.834+0000 7fb1147a4700 1 -- 172.21.15.96:0/2329307158 >> 172.21.15.96:0/2329307158 conn(0x7fb10c075f30 msgr2=0x7fb10c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.834+0000 7fb1147a4700 1 -- 172.21.15.96:0/2329307158 shutdown_connections 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.834+0000 7fb1147a4700 1 -- 172.21.15.96:0/2329307158 wait complete. 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb1147a4700 1 Processor -- start 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb1147a4700 1 -- start start 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb1147a4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c1003e0 0x7fb10c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb1147a4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c100d90 0x7fb10c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb1147a4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c19d380 con 0x7fb10c1003e0 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb1147a4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c196960 con 0x7fb10c100d90 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb112540700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c1003e0 0x7fb10c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb112540700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c1003e0 0x7fb10c19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58018/0 (socket says 172.21.15.96:58018) 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb112540700 1 -- 172.21.15.96:0/2840154326 learned_addr learned my addr 172.21.15.96:0/2840154326 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb111d3f700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c100d90 0x7fb10c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:35.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb112540700 1 -- 172.21.15.96:0/2840154326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c100d90 msgr2=0x7fb10c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:35.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb112540700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c100d90 0x7fb10c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:35.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.835+0000 7fb112540700 1 -- 172.21.15.96:0/2840154326 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1080095f0 con 0x7fb10c1003e0 2024-04-22T15:31:35.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.836+0000 7fb112540700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c1003e0 0x7fb10c19c820 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00d8d0 tx=0x7fb0fc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:35.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.836+0000 7fb1037fe700 1 -- 172.21.15.96:0/2840154326 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0fc0098a0 con 0x7fb10c1003e0 2024-04-22T15:31:35.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.836+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10c196c40 con 0x7fb10c1003e0 2024-04-22T15:31:35.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.836+0000 7fb1037fe700 1 -- 172.21.15.96:0/2840154326 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0fc010460 con 0x7fb10c1003e0 2024-04-22T15:31:35.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.836+0000 7fb1037fe700 1 -- 172.21.15.96:0/2840154326 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0fc00f5f0 con 0x7fb10c1003e0 2024-04-22T15:31:35.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.836+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10c197160 con 0x7fb10c1003e0 2024-04-22T15:31:35.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.837+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10c1013a0 con 0x7fb10c1003e0 2024-04-22T15:31:35.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.837+0000 7fb1037fe700 1 -- 172.21.15.96:0/2840154326 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb0fc009a00 con 0x7fb10c1003e0 2024-04-22T15:31:35.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.838+0000 7fb1037fe700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb0f806ecc0 0x7fb0f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:31:35.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.838+0000 7fb1037fe700 1 -- 172.21.15.96:0/2840154326 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb0fc08ed80 con 0x7fb10c1003e0 2024-04-22T15:31:35.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.838+0000 7fb111d3f700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb0f806ecc0 0x7fb0f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:31:35.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.838+0000 7fb111d3f700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb0f806ecc0 0x7fb0f8071180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb108000c00 tx=0x7fb10801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:31:35.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:35.842+0000 7fb1037fe700 1 -- 172.21.15.96:0/2840154326 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb0fc05b610 con 0x7fb10c1003e0 2024-04-22T15:31:36.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.128+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb10c10cce0 con 0x7fb10c1003e0 2024-04-22T15:31:36.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.129+0000 7fb1037fe700 1 -- 172.21.15.96:0/2840154326 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fb0fc05b1a0 con 0x7fb10c1003e0 2024-04-22T15:31:36.132 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:31:36.132 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:31:36.133 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:31:36.133 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:31:36.133 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:31:36.133 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:31:36.133 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:31:36.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb0f806ecc0 msgr2=0x7fb0f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:36.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb0f806ecc0 0x7fb0f8071180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb108000c00 tx=0x7fb10801a040 comp rx=0 tx=0).stop 2024-04-22T15:31:36.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c1003e0 msgr2=0x7fb10c19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:31:36.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c1003e0 0x7fb10c19c820 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00d8d0 tx=0x7fb0fc00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:31:36.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 shutdown_connections 2024-04-22T15:31:36.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb0f806ecc0 0x7fb0f8071180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:36.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb10c1003e0 0x7fb10c19c820 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:36.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 --2- 172.21.15.96:0/2840154326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c100d90 0x7fb10c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:31:36.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 >> 172.21.15.96:0/2840154326 conn(0x7fb10c075f30 msgr2=0x7fb10c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:31:36.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 shutdown_connections 2024-04-22T15:31:36.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:31:36.131+0000 7fb1147a4700 1 -- 172.21.15.96:0/2840154326 wait complete. 2024-04-22T15:31:36.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:36 smithi096 ceph-mon[112443]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:36.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:36 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2840154326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:31:36.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:36 smithi196 ceph-mon[81815]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:36.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:36 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2840154326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:31:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:37 smithi096 ceph-mon[112443]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:37 smithi096 ceph-mon[112443]: pgmap v292: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:37.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:37 smithi196 ceph-mon[81815]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:31:37.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:37 smithi196 ceph-mon[81815]: pgmap v292: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:38.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:31:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:31:39.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:39 smithi096 ceph-mon[112443]: pgmap v293: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:39 smithi196 ceph-mon[81815]: pgmap v293: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:41 smithi096 ceph-mon[112443]: pgmap v294: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:41 smithi196 ceph-mon[81815]: pgmap v294: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:43 smithi096 ceph-mon[112443]: pgmap v295: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:43 smithi196 ceph-mon[81815]: pgmap v295: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:45 smithi096 ceph-mon[112443]: pgmap v296: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:45.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:45 smithi196 ceph-mon[81815]: pgmap v296: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:47 smithi096 ceph-mon[112443]: pgmap v297: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:47 smithi196 ceph-mon[81815]: pgmap v297: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:49 smithi096 ceph-mon[112443]: pgmap v298: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:49.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:49 smithi196 ceph-mon[81815]: pgmap v298: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:51 smithi096 ceph-mon[112443]: pgmap v299: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:51 smithi196 ceph-mon[81815]: pgmap v299: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:53 smithi096 ceph-mon[112443]: pgmap v300: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:31:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:53 smithi196 ceph-mon[81815]: pgmap v300: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:31:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:55 smithi096 ceph-mon[112443]: pgmap v301: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:55 smithi196 ceph-mon[81815]: pgmap v301: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:57 smithi096 ceph-mon[112443]: pgmap v302: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:57 smithi196 ceph-mon[81815]: pgmap v302: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:31:59 smithi096 ceph-mon[112443]: pgmap v303: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:31:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:31:59 smithi196 ceph-mon[81815]: pgmap v303: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:01 smithi096 ceph-mon[112443]: pgmap v304: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:01 smithi196 ceph-mon[81815]: pgmap v304: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:03 smithi096 ceph-mon[112443]: pgmap v305: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:03 smithi196 ceph-mon[81815]: pgmap v305: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:05 smithi096 ceph-mon[112443]: pgmap v306: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:05 smithi196 ceph-mon[81815]: pgmap v306: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:06.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.248+0000 7fbcf3744700 1 -- 172.21.15.96:0/3255102911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 msgr2=0x7fbcec104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.248+0000 7fbcf3744700 1 --2- 172.21.15.96:0/3255102911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 0x7fbcec104960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009910 tx=0x7fbcdc009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:06.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.248+0000 7fbcf3744700 1 -- 172.21.15.96:0/3255102911 shutdown_connections 2024-04-22T15:32:06.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.248+0000 7fbcf3744700 1 --2- 172.21.15.96:0/3255102911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 0x7fbcec104960 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.248+0000 7fbcf3744700 1 --2- 172.21.15.96:0/3255102911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcec0ffb70 0x7fbcec0fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.248+0000 7fbcf3744700 1 -- 172.21.15.96:0/3255102911 >> 172.21.15.96:0/3255102911 conn(0x7fbcec0fb5d0 msgr2=0x7fbcec0fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:06.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.248+0000 7fbcf3744700 1 -- 172.21.15.96:0/3255102911 shutdown_connections 2024-04-22T15:32:06.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.248+0000 7fbcf3744700 1 -- 172.21.15.96:0/3255102911 wait complete. 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.248+0000 7fbcf3744700 1 Processor -- start 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf3744700 1 -- start start 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf3744700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcec0ffb70 0x7fbcec19d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf3744700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 0x7fbcec19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf3744700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec19dce0 con 0x7fbcec100490 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf3744700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec19de20 con 0x7fbcec0ffb70 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf0cdf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 0x7fbcec19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf0cdf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 0x7fbcec19d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43292/0 (socket says 172.21.15.96:43292) 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf0cdf700 1 -- 172.21.15.96:0/3557112275 learned_addr learned my addr 172.21.15.96:0/3557112275 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf14e0700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcec0ffb70 0x7fbcec19d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf0cdf700 1 -- 172.21.15.96:0/3557112275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcec0ffb70 msgr2=0x7fbcec19d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf0cdf700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcec0ffb70 0x7fbcec19d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf0cdf700 1 -- 172.21.15.96:0/3557112275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc0095f0 con 0x7fbcec100490 2024-04-22T15:32:06.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.249+0000 7fbcf0cdf700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 0x7fbcec19d690 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0092e0 tx=0x7fbcdc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:06.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.250+0000 7fbce27fc700 1 -- 172.21.15.96:0/3557112275 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcdc01d070 con 0x7fbcec100490 2024-04-22T15:32:06.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.250+0000 7fbce27fc700 1 -- 172.21.15.96:0/3557112275 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcdc003d50 con 0x7fbcec100490 2024-04-22T15:32:06.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.250+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec1a1b50 con 0x7fbcec100490 2024-04-22T15:32:06.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.250+0000 7fbce27fc700 1 -- 172.21.15.96:0/3557112275 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcdc00fe70 con 0x7fbcec100490 2024-04-22T15:32:06.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.250+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcec1a2040 con 0x7fbcec100490 2024-04-22T15:32:06.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.251+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcec069a90 con 0x7fbcec100490 2024-04-22T15:32:06.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.251+0000 7fbce27fc700 1 -- 172.21.15.96:0/3557112275 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbcdc003890 con 0x7fbcec100490 2024-04-22T15:32:06.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.252+0000 7fbce27fc700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbcd806ec60 0x7fbcd8071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:06.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.252+0000 7fbce27fc700 1 -- 172.21.15.96:0/3557112275 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbcdc090200 con 0x7fbcec100490 2024-04-22T15:32:06.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.252+0000 7fbcf14e0700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbcd806ec60 0x7fbcd8071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:06.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.252+0000 7fbcf14e0700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbcd806ec60 0x7fbcd8071120 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbce800ac50 tx=0x7fbce800a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:06.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.257+0000 7fbce27fc700 1 -- 172.21.15.96:0/3557112275 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbcdc058f60 con 0x7fbcec100490 2024-04-22T15:32:06.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.477+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcec114450 con 0x7fbcd806ec60 2024-04-22T15:32:06.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.478+0000 7fbce27fc700 1 -- 172.21.15.96:0/3557112275 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbcec114450 con 0x7fbcd806ec60 2024-04-22T15:32:06.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbcd806ec60 msgr2=0x7fbcd8071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbcd806ec60 0x7fbcd8071120 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbce800ac50 tx=0x7fbce800a380 comp rx=0 tx=0).stop 2024-04-22T15:32:06.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 msgr2=0x7fbcec19d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 0x7fbcec19d690 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0092e0 tx=0x7fbcdc00bd70 comp rx=0 tx=0).stop 2024-04-22T15:32:06.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 shutdown_connections 2024-04-22T15:32:06.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbcd806ec60 0x7fbcd8071120 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcec0ffb70 0x7fbcec19d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 --2- 172.21.15.96:0/3557112275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcec100490 0x7fbcec19d690 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 >> 172.21.15.96:0/3557112275 conn(0x7fbcec0fb5d0 msgr2=0x7fbcec10e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:06.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 shutdown_connections 2024-04-22T15:32:06.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.481+0000 7fbcf3744700 1 -- 172.21.15.96:0/3557112275 wait complete. 2024-04-22T15:32:06.495 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.585+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/3411614684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 msgr2=0x7fcdf0105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.585+0000 7fcdf6b7a700 1 --2- 172.21.15.96:0/3411614684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 0x7fcdf0105f40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009910 tx=0x7fcde4009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/3411614684 shutdown_connections 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 --2- 172.21.15.96:0/3411614684 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdf00ffad0 0x7fcdf00fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 --2- 172.21.15.96:0/3411614684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 0x7fcdf0105f40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/3411614684 >> 172.21.15.96:0/3411614684 conn(0x7fcdf00fb5e0 msgr2=0x7fcdf00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/3411614684 shutdown_connections 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/3411614684 wait complete. 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 Processor -- start 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 -- start start 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdf00ffad0 0x7fcdf0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:06.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 0x7fcdf0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf01985a0 con 0x7fcdf0105b60 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdf6b7a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf01986e0 con 0x7fcdf00ffad0 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdeffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 0x7fcdf0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.586+0000 7fcdeffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 0x7fcdf0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43320/0 (socket says 172.21.15.96:43320) 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdeffff700 1 -- 172.21.15.96:0/472208495 learned_addr learned my addr 172.21.15.96:0/472208495 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdf4916700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdf00ffad0 0x7fcdf0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdeffff700 1 -- 172.21.15.96:0/472208495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdf00ffad0 msgr2=0x7fcdf0197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdeffff700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdf00ffad0 0x7fcdf0197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdeffff700 1 -- 172.21.15.96:0/472208495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde40095f0 con 0x7fcdf0105b60 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdeffff700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 0x7fcdf0197fd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00b4b0 tx=0x7fcddc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdedffb700 1 -- 172.21.15.96:0/472208495 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcddc011800 con 0x7fcdf0105b60 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdedffb700 1 -- 172.21.15.96:0/472208495 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcddc011e20 con 0x7fcdf0105b60 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf01a0b00 con 0x7fcdf0105b60 2024-04-22T15:32:06.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdedffb700 1 -- 172.21.15.96:0/472208495 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcddc00f2e0 con 0x7fcdf0105b60 2024-04-22T15:32:06.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.587+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf01a0f10 con 0x7fcdf0105b60 2024-04-22T15:32:06.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.588+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf01080d0 con 0x7fcdf0105b60 2024-04-22T15:32:06.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.589+0000 7fcdedffb700 1 -- 172.21.15.96:0/472208495 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcddc010380 con 0x7fcdf0105b60 2024-04-22T15:32:06.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.589+0000 7fcdedffb700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcde006ecc0 0x7fcde0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:06.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.589+0000 7fcdf4916700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcde006ecc0 0x7fcde0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:06.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.589+0000 7fcdedffb700 1 -- 172.21.15.96:0/472208495 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcddc08e9b0 con 0x7fcdf0105b60 2024-04-22T15:32:06.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.590+0000 7fcdf4916700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcde006ecc0 0x7fcde0071180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcde4005f50 tx=0x7fcde401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:06.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.594+0000 7fcdedffb700 1 -- 172.21.15.96:0/472208495 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcddc057790 con 0x7fcdf0105b60 2024-04-22T15:32:06.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.812+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdf0069780 con 0x7fcde006ecc0 2024-04-22T15:32:06.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.813+0000 7fcdedffb700 1 -- 172.21.15.96:0/472208495 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcdf0069780 con 0x7fcde006ecc0 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcde006ecc0 msgr2=0x7fcde0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcde006ecc0 0x7fcde0071180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcde4005f50 tx=0x7fcde401a040 comp rx=0 tx=0).stop 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 msgr2=0x7fcdf0197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 0x7fcdf0197fd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00b4b0 tx=0x7fcddc00b870 comp rx=0 tx=0).stop 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 shutdown_connections 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcde006ecc0 0x7fcde0071180 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdf00ffad0 0x7fcdf0197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 --2- 172.21.15.96:0/472208495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdf0105b60 0x7fcdf0197fd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 >> 172.21.15.96:0/472208495 conn(0x7fcdf00fb5e0 msgr2=0x7fcdf00fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:06.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 shutdown_connections 2024-04-22T15:32:06.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.815+0000 7fcdf6b7a700 1 -- 172.21.15.96:0/472208495 wait complete. 2024-04-22T15:32:06.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.919+0000 7f7c3a41c700 1 -- 172.21.15.96:0/785235357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 msgr2=0x7f7c34101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.919+0000 7f7c3a41c700 1 --2- 172.21.15.96:0/785235357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 0x7f7c34101da0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009910 tx=0x7f7c28009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:06.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.919+0000 7f7c3a41c700 1 -- 172.21.15.96:0/785235357 shutdown_connections 2024-04-22T15:32:06.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.919+0000 7f7c3a41c700 1 --2- 172.21.15.96:0/785235357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c340ff3d0 0x7f7c340ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.919+0000 7f7c3a41c700 1 --2- 172.21.15.96:0/785235357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 0x7f7c34101da0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.919+0000 7f7c3a41c700 1 -- 172.21.15.96:0/785235357 >> 172.21.15.96:0/785235357 conn(0x7f7c34074d90 msgr2=0x7f7c340751a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:06.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.919+0000 7f7c3a41c700 1 -- 172.21.15.96:0/785235357 shutdown_connections 2024-04-22T15:32:06.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.919+0000 7f7c3a41c700 1 -- 172.21.15.96:0/785235357 wait complete. 2024-04-22T15:32:06.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c3a41c700 1 Processor -- start 2024-04-22T15:32:06.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c3a41c700 1 -- start start 2024-04-22T15:32:06.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c3a41c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c340ff3d0 0x7f7c3419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:06.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c3a41c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 0x7f7c3419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:06.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c3a41c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3419f4e0 con 0x7f7c341019c0 2024-04-22T15:32:06.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c3a41c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34198ac0 con 0x7f7c340ff3d0 2024-04-22T15:32:06.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c337fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 0x7f7c3419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:06.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c337fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 0x7f7c3419eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43348/0 (socket says 172.21.15.96:43348) 2024-04-22T15:32:06.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c337fe700 1 -- 172.21.15.96:0/315444589 learned_addr learned my addr 172.21.15.96:0/315444589 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.920+0000 7f7c33fff700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c340ff3d0 0x7f7c3419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.921+0000 7f7c337fe700 1 -- 172.21.15.96:0/315444589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c340ff3d0 msgr2=0x7f7c3419e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.921+0000 7f7c337fe700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c340ff3d0 0x7f7c3419e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.921+0000 7f7c337fe700 1 -- 172.21.15.96:0/315444589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c280095f0 con 0x7f7c341019c0 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.921+0000 7f7c337fe700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 0x7f7c3419eec0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c8a0 tx=0x7f7c2400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.921+0000 7f7c317fa700 1 -- 172.21.15.96:0/315444589 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c240078e0 con 0x7f7c341019c0 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.921+0000 7f7c317fa700 1 -- 172.21.15.96:0/315444589 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c2400f450 con 0x7f7c341019c0 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.921+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c34198da0 con 0x7f7c341019c0 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.921+0000 7f7c317fa700 1 -- 172.21.15.96:0/315444589 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2400e5e0 con 0x7f7c341019c0 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.921+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c34199370 con 0x7f7c341019c0 2024-04-22T15:32:06.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.922+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3410b3f0 con 0x7f7c341019c0 2024-04-22T15:32:06.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.922+0000 7f7c317fa700 1 -- 172.21.15.96:0/315444589 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7c2400f5c0 con 0x7f7c341019c0 2024-04-22T15:32:06.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.923+0000 7f7c317fa700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c1c06ecc0 0x7f7c1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:06.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.923+0000 7f7c317fa700 1 -- 172.21.15.96:0/315444589 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7c2408ef70 con 0x7f7c341019c0 2024-04-22T15:32:06.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.923+0000 7f7c33fff700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c1c06ecc0 0x7f7c1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:06.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.923+0000 7f7c33fff700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c1c06ecc0 0x7f7c1c071180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c28000c00 tx=0x7f7c2801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:06.927+0000 7f7c317fa700 1 -- 172.21.15.96:0/315444589 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7c2405b760 con 0x7f7c341019c0 2024-04-22T15:32:07.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.136+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c34199ff0 con 0x7f7c1c06ecc0 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.146+0000 7f7c317fa700 1 -- 172.21.15.96:0/315444589 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7c34199ff0 con 0x7f7c1c06ecc0 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (10m) 3m ago 22m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (22m) 3m ago 22m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (21m) 3m ago 21m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (7m) 3m ago 22m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (7m) 3m ago 21m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (10m) 3m ago 21m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (3m) 3m ago 16m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (4m) 3m ago 16m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:32:07.150 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (3m) 3m ago 16m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (16m) 3m ago 16m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (13m) 3m ago 23m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (12m) 3m ago 21m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (9m) 3m ago 23m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (8m) 3m ago 20m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (11m) 3m ago 22m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (11m) 3m ago 21m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (7m) 3m ago 20m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (6m) 3m ago 19m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (6m) 3m ago 19m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (5m) 3m ago 18m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (5m) 3m ago 18m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (4m) 3m ago 17m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:32:07.151 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (11m) 3m ago 21m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:32:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.148+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c1c06ecc0 msgr2=0x7f7c1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.148+0000 7f7c3a41c700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c1c06ecc0 0x7f7c1c071180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c28000c00 tx=0x7f7c2801a040 comp rx=0 tx=0).stop 2024-04-22T15:32:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.148+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 msgr2=0x7f7c3419eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.148+0000 7f7c3a41c700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 0x7f7c3419eec0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c8a0 tx=0x7f7c2400cbb0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.149+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 shutdown_connections 2024-04-22T15:32:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.149+0000 7f7c3a41c700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c1c06ecc0 0x7f7c1c071180 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.149+0000 7f7c3a41c700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c340ff3d0 0x7f7c3419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.149+0000 7f7c3a41c700 1 --2- 172.21.15.96:0/315444589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c341019c0 0x7f7c3419eec0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.149+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 >> 172.21.15.96:0/315444589 conn(0x7f7c34074d90 msgr2=0x7f7c340fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.149+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 shutdown_connections 2024-04-22T15:32:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.149+0000 7f7c3a41c700 1 -- 172.21.15.96:0/315444589 wait complete. 2024-04-22T15:32:07.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.251+0000 7fbf7149a700 1 -- 172.21.15.96:0/1880129741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c100e20 msgr2=0x7fbf6c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.251+0000 7fbf7149a700 1 --2- 172.21.15.96:0/1880129741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c100e20 0x7fbf6c101200 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00b0a0 tx=0x7fbf5c00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.251+0000 7fbf7149a700 1 -- 172.21.15.96:0/1880129741 shutdown_connections 2024-04-22T15:32:07.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.251+0000 7fbf7149a700 1 --2- 172.21.15.96:0/1880129741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf6c1017d0 0x7fbf6c105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.251+0000 7fbf7149a700 1 --2- 172.21.15.96:0/1880129741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c100e20 0x7fbf6c101200 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.251+0000 7fbf7149a700 1 -- 172.21.15.96:0/1880129741 >> 172.21.15.96:0/1880129741 conn(0x7fbf6c0fc6d0 msgr2=0x7fbf6c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:07.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.251+0000 7fbf7149a700 1 -- 172.21.15.96:0/1880129741 shutdown_connections 2024-04-22T15:32:07.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.251+0000 7fbf7149a700 1 -- 172.21.15.96:0/1880129741 wait complete. 2024-04-22T15:32:07.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf7149a700 1 Processor -- start 2024-04-22T15:32:07.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf7149a700 1 -- start start 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf7149a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf6c100e20 0x7fbf6c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf7149a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c1017d0 0x7fbf6c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf7149a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c19d3a0 con 0x7fbf6c1017d0 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf7149a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c1969d0 con 0x7fbf6c100e20 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf6a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c1017d0 0x7fbf6c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf6a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c1017d0 0x7fbf6c19cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43362/0 (socket says 172.21.15.96:43362) 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf6a7fc700 1 -- 172.21.15.96:0/2540768307 learned_addr learned my addr 172.21.15.96:0/2540768307 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf6affd700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf6c100e20 0x7fbf6c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf6a7fc700 1 -- 172.21.15.96:0/2540768307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf6c100e20 msgr2=0x7fbf6c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf6a7fc700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf6c100e20 0x7fbf6c19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf6a7fc700 1 -- 172.21.15.96:0/2540768307 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c009d00 con 0x7fbf6c1017d0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.252+0000 7fbf6affd700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf6c100e20 0x7fbf6c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.253+0000 7fbf6a7fc700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c1017d0 0x7fbf6c19cd80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400d900 tx=0x7fbf5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.253+0000 7fbf63fff700 1 -- 172.21.15.96:0/2540768307 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf54009900 con 0x7fbf6c1017d0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.253+0000 7fbf63fff700 1 -- 172.21.15.96:0/2540768307 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf54010460 con 0x7fbf6c1017d0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.253+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c196cb0 con 0x7fbf6c1017d0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.253+0000 7fbf63fff700 1 -- 172.21.15.96:0/2540768307 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf5400f5f0 con 0x7fbf6c1017d0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.253+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c197200 con 0x7fbf6c1017d0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.254+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c1091c0 con 0x7fbf6c1017d0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.254+0000 7fbf63fff700 1 -- 172.21.15.96:0/2540768307 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbf540105d0 con 0x7fbf6c1017d0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.255+0000 7fbf63fff700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf5806ecc0 0x7fbf58071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.255+0000 7fbf63fff700 1 -- 172.21.15.96:0/2540768307 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbf5408eb10 con 0x7fbf6c1017d0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.255+0000 7fbf6affd700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf5806ecc0 0x7fbf58071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:07.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.255+0000 7fbf6affd700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf5806ecc0 0x7fbf58071180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00b830 tx=0x7fbf5c006210 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:07.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.260+0000 7fbf63fff700 1 -- 172.21.15.96:0/2540768307 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbf5405b3a0 con 0x7fbf6c1017d0 2024-04-22T15:32:07.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.539+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbf6c197e40 con 0x7fbf6c1017d0 2024-04-22T15:32:07.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.540+0000 7fbf63fff700 1 -- 172.21.15.96:0/2540768307 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7fbf5405af30 con 0x7fbf6c1017d0 2024-04-22T15:32:07.544 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:32:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:32:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:32:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:32:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:32:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:32:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:32:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:32:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:32:07.545 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:32:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf5806ecc0 msgr2=0x7fbf58071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf5806ecc0 0x7fbf58071180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00b830 tx=0x7fbf5c006210 comp rx=0 tx=0).stop 2024-04-22T15:32:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c1017d0 msgr2=0x7fbf6c19cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c1017d0 0x7fbf6c19cd80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400d900 tx=0x7fbf5400dcc0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 shutdown_connections 2024-04-22T15:32:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf5806ecc0 0x7fbf58071180 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf6c100e20 0x7fbf6c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 --2- 172.21.15.96:0/2540768307 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf6c1017d0 0x7fbf6c19cd80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 >> 172.21.15.96:0/2540768307 conn(0x7fbf6c0fc6d0 msgr2=0x7fbf6c0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:07.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 shutdown_connections 2024-04-22T15:32:07.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.542+0000 7fbf7149a700 1 -- 172.21.15.96:0/2540768307 wait complete. 2024-04-22T15:32:07.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.646+0000 7f125b714700 1 -- 172.21.15.96:0/1105531966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12541017d0 msgr2=0x7f1254105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.646+0000 7f125b714700 1 --2- 172.21.15.96:0/1105531966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12541017d0 0x7f1254105820 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1244009910 tx=0x7f1244009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.646+0000 7f125b714700 1 -- 172.21.15.96:0/1105531966 shutdown_connections 2024-04-22T15:32:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.646+0000 7f125b714700 1 --2- 172.21.15.96:0/1105531966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12541017d0 0x7f1254105820 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.646+0000 7f125b714700 1 --2- 172.21.15.96:0/1105531966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1254100e20 0x7f1254101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.646+0000 7f125b714700 1 -- 172.21.15.96:0/1105531966 >> 172.21.15.96:0/1105531966 conn(0x7f12540fc6d0 msgr2=0x7f12540feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.646+0000 7f125b714700 1 -- 172.21.15.96:0/1105531966 shutdown_connections 2024-04-22T15:32:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.646+0000 7f125b714700 1 -- 172.21.15.96:0/1105531966 wait complete. 2024-04-22T15:32:07.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f125b714700 1 Processor -- start 2024-04-22T15:32:07.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f125b714700 1 -- start start 2024-04-22T15:32:07.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f125b714700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1254100e20 0x7f12540731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:07.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f125b714700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12541017d0 0x7f1254073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f125b714700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1254073c60 con 0x7f1254100e20 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f125b714700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1254073da0 con 0x7f12541017d0 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f12594b0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1254100e20 0x7f12540731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f12594b0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1254100e20 0x7f12540731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43380/0 (socket says 172.21.15.96:43380) 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f12594b0700 1 -- 172.21.15.96:0/1010799304 learned_addr learned my addr 172.21.15.96:0/1010799304 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f1258caf700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12541017d0 0x7f1254073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f12594b0700 1 -- 172.21.15.96:0/1010799304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12541017d0 msgr2=0x7f1254073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f12594b0700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12541017d0 0x7f1254073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f12594b0700 1 -- 172.21.15.96:0/1010799304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12440095f0 con 0x7f1254100e20 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.647+0000 7f12594b0700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1254100e20 0x7f12540731e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f125000d8d0 tx=0x7f125000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.648+0000 7f124a7fc700 1 -- 172.21.15.96:0/1010799304 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1250009940 con 0x7f1254100e20 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.648+0000 7f124a7fc700 1 -- 172.21.15.96:0/1010799304 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1250010460 con 0x7f1254100e20 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.648+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12541a2ce0 con 0x7f1254100e20 2024-04-22T15:32:07.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.648+0000 7f124a7fc700 1 -- 172.21.15.96:0/1010799304 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f125000f5f0 con 0x7f1254100e20 2024-04-22T15:32:07.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.648+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12541a3040 con 0x7f1254100e20 2024-04-22T15:32:07.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.649+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1254109220 con 0x7f1254100e20 2024-04-22T15:32:07.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.649+0000 7f124a7fc700 1 -- 172.21.15.96:0/1010799304 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f12500105d0 con 0x7f1254100e20 2024-04-22T15:32:07.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.650+0000 7f124a7fc700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f124006ecc0 0x7f1240071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:07.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.650+0000 7f1258caf700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f124006ecc0 0x7f1240071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:07.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.650+0000 7f124a7fc700 1 -- 172.21.15.96:0/1010799304 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f125008eec0 con 0x7f1254100e20 2024-04-22T15:32:07.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.650+0000 7f1258caf700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f124006ecc0 0x7f1240071180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f124400b5c0 tx=0x7f12440052a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:07.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.655+0000 7f124a7fc700 1 -- 172.21.15.96:0/1010799304 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1250057c20 con 0x7f1254100e20 2024-04-22T15:32:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:07 smithi196 ceph-mon[81815]: pgmap v307: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:07 smithi196 ceph-mon[81815]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:07.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.897+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f125410ddc0 con 0x7f1254100e20 2024-04-22T15:32:07.901 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:07 smithi096 ceph-mon[112443]: pgmap v307: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:07.901 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:07 smithi096 ceph-mon[112443]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:07.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.897+0000 7f124a7fc700 1 -- 172.21.15.96:0/1010799304 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f125005b2e0 con 0x7f1254100e20 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:32:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:32:07.903 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:32:07.904 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:32:07.905 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:32:07.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.899+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f124006ecc0 msgr2=0x7f1240071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.899+0000 7f125b714700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f124006ecc0 0x7f1240071180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f124400b5c0 tx=0x7f12440052a0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.899+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1254100e20 msgr2=0x7f12540731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:07.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.899+0000 7f125b714700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1254100e20 0x7f12540731e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f125000d8d0 tx=0x7f125000dc90 comp rx=0 tx=0).stop 2024-04-22T15:32:07.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.899+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 shutdown_connections 2024-04-22T15:32:07.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.899+0000 7f125b714700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f124006ecc0 0x7f1240071180 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.899+0000 7f125b714700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1254100e20 0x7f12540731e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.899+0000 7f125b714700 1 --2- 172.21.15.96:0/1010799304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12541017d0 0x7f1254073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:07.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.899+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 >> 172.21.15.96:0/1010799304 conn(0x7f12540fc6d0 msgr2=0x7f12540fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:07.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.900+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 shutdown_connections 2024-04-22T15:32:07.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:07.900+0000 7f125b714700 1 -- 172.21.15.96:0/1010799304 wait complete. 2024-04-22T15:32:07.906 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:32:08.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 -- 172.21.15.96:0/2064106638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 msgr2=0x7fad4c0ff710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:08.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 --2- 172.21.15.96:0/2064106638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 0x7fad4c0ff710 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fad48009910 tx=0x7fad48009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:08.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 -- 172.21.15.96:0/2064106638 shutdown_connections 2024-04-22T15:32:08.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 --2- 172.21.15.96:0/2064106638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c0ffce0 0x7fad4c105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 --2- 172.21.15.96:0/2064106638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 0x7fad4c0ff710 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 -- 172.21.15.96:0/2064106638 >> 172.21.15.96:0/2064106638 conn(0x7fad4c075f30 msgr2=0x7fad4c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:08.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 -- 172.21.15.96:0/2064106638 shutdown_connections 2024-04-22T15:32:08.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 -- 172.21.15.96:0/2064106638 wait complete. 2024-04-22T15:32:08.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 Processor -- start 2024-04-22T15:32:08.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.005+0000 7fad52b38700 1 -- start start 2024-04-22T15:32:08.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad52b38700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 0x7fad4c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:08.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad52b38700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c0ffce0 0x7fad4c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad52b38700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c19d2b0 con 0x7fad4c0ff330 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad52b38700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c196970 con 0x7fad4c0ffce0 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad508d4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 0x7fad4c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad508d4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 0x7fad4c19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43396/0 (socket says 172.21.15.96:43396) 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad508d4700 1 -- 172.21.15.96:0/1588463961 learned_addr learned my addr 172.21.15.96:0/1588463961 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad43fff700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c0ffce0 0x7fad4c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad508d4700 1 -- 172.21.15.96:0/1588463961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c0ffce0 msgr2=0x7fad4c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad508d4700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c0ffce0 0x7fad4c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad508d4700 1 -- 172.21.15.96:0/1588463961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad480095f0 con 0x7fad4c0ff330 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad43fff700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c0ffce0 0x7fad4c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad508d4700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 0x7fad4c19c7e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fad48006010 tx=0x7fad4800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad41ffb700 1 -- 172.21.15.96:0/1588463961 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad4801d070 con 0x7fad4c0ff330 2024-04-22T15:32:08.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c196bf0 con 0x7fad4c0ff330 2024-04-22T15:32:08.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad41ffb700 1 -- 172.21.15.96:0/1588463961 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad4800f460 con 0x7fad4c0ff330 2024-04-22T15:32:08.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad41ffb700 1 -- 172.21.15.96:0/1588463961 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad48017630 con 0x7fad4c0ff330 2024-04-22T15:32:08.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.006+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c1970e0 con 0x7fad4c0ff330 2024-04-22T15:32:08.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.007+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4c1091c0 con 0x7fad4c0ff330 2024-04-22T15:32:08.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.008+0000 7fad41ffb700 1 -- 172.21.15.96:0/1588463961 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fad48003a60 con 0x7fad4c0ff330 2024-04-22T15:32:08.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.008+0000 7fad41ffb700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fad3406ecc0 0x7fad34071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:08.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.008+0000 7fad41ffb700 1 -- 172.21.15.96:0/1588463961 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fad48090390 con 0x7fad4c0ff330 2024-04-22T15:32:08.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.008+0000 7fad43fff700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fad3406ecc0 0x7fad34071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:08.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.009+0000 7fad43fff700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fad3406ecc0 0x7fad34071180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fad3c005fd0 tx=0x7fad3c005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:08.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.013+0000 7fad41ffb700 1 -- 172.21.15.96:0/1588463961 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fad4805cce0 con 0x7fad4c0ff330 2024-04-22T15:32:08.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.231+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fad4c197de0 con 0x7fad3406ecc0 2024-04-22T15:32:08.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.233+0000 7fad41ffb700 1 -- 172.21.15.96:0/1588463961 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fad4c197de0 con 0x7fad3406ecc0 2024-04-22T15:32:08.236 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:32:08.236 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:32:08.236 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:32:08.236 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:32:08.236 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:32:08.236 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:32:08.236 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:32:08.236 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:32:08.237 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:32:08.237 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:32:08.237 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:32:08.237 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:32:08.237 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:32:08.237 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:32:08.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fad3406ecc0 msgr2=0x7fad34071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:08.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fad3406ecc0 0x7fad34071180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fad3c005fd0 tx=0x7fad3c005e20 comp rx=0 tx=0).stop 2024-04-22T15:32:08.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 msgr2=0x7fad4c19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:08.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 0x7fad4c19c7e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fad48006010 tx=0x7fad4800be20 comp rx=0 tx=0).stop 2024-04-22T15:32:08.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 shutdown_connections 2024-04-22T15:32:08.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fad3406ecc0 0x7fad34071180 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad4c0ff330 0x7fad4c19c7e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 --2- 172.21.15.96:0/1588463961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c0ffce0 0x7fad4c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 >> 172.21.15.96:0/1588463961 conn(0x7fad4c075f30 msgr2=0x7fad4c1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:08.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 shutdown_connections 2024-04-22T15:32:08.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.235+0000 7fad52b38700 1 -- 172.21.15.96:0/1588463961 wait complete. 2024-04-22T15:32:08.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.341+0000 7f1da489b700 1 -- 172.21.15.96:0/1262982233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da01017d0 msgr2=0x7f1da0105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:08.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.341+0000 7f1da489b700 1 --2- 172.21.15.96:0/1262982233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da01017d0 0x7f1da0105820 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009910 tx=0x7f1d88009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:08.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.341+0000 7f1da489b700 1 -- 172.21.15.96:0/1262982233 shutdown_connections 2024-04-22T15:32:08.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.341+0000 7f1da489b700 1 --2- 172.21.15.96:0/1262982233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da01017d0 0x7f1da0105820 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.341+0000 7f1da489b700 1 --2- 172.21.15.96:0/1262982233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da0100e20 0x7f1da0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.341+0000 7f1da489b700 1 -- 172.21.15.96:0/1262982233 >> 172.21.15.96:0/1262982233 conn(0x7f1da00fc6d0 msgr2=0x7f1da00feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:08.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.341+0000 7f1da489b700 1 -- 172.21.15.96:0/1262982233 shutdown_connections 2024-04-22T15:32:08.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.341+0000 7f1da489b700 1 -- 172.21.15.96:0/1262982233 wait complete. 2024-04-22T15:32:08.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1da489b700 1 Processor -- start 2024-04-22T15:32:08.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1da489b700 1 -- start start 2024-04-22T15:32:08.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1da489b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da0100e20 0x7f1da019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:08.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1da489b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da01017d0 0x7f1da019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1da489b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da019d380 con 0x7f1da0100e20 2024-04-22T15:32:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1da489b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da0196960 con 0x7f1da01017d0 2024-04-22T15:32:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1d9e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da0100e20 0x7f1da019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1d9e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da0100e20 0x7f1da019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43404/0 (socket says 172.21.15.96:43404) 2024-04-22T15:32:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1d9e59c700 1 -- 172.21.15.96:0/3347788710 learned_addr learned my addr 172.21.15.96:0/3347788710 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.342+0000 7f1d9dd9b700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da01017d0 0x7f1da019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.343+0000 7f1d9e59c700 1 -- 172.21.15.96:0/3347788710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da01017d0 msgr2=0x7f1da019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.343+0000 7f1d9e59c700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da01017d0 0x7f1da019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.343+0000 7f1d9e59c700 1 -- 172.21.15.96:0/3347788710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d880095f0 con 0x7f1da0100e20 2024-04-22T15:32:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.343+0000 7f1d9e59c700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da0100e20 0x7f1da019c820 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1d98007e90 tx=0x7f1d9800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.343+0000 7f1d977fe700 1 -- 172.21.15.96:0/3347788710 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d98017070 con 0x7f1da0100e20 2024-04-22T15:32:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.343+0000 7f1d977fe700 1 -- 172.21.15.96:0/3347788710 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d9800f040 con 0x7f1da0100e20 2024-04-22T15:32:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.343+0000 7f1d977fe700 1 -- 172.21.15.96:0/3347788710 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d98005890 con 0x7f1da0100e20 2024-04-22T15:32:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.343+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da0196c40 con 0x7f1da0100e20 2024-04-22T15:32:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.343+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da0197190 con 0x7f1da0100e20 2024-04-22T15:32:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.344+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da0109220 con 0x7f1da0100e20 2024-04-22T15:32:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.345+0000 7f1d977fe700 1 -- 172.21.15.96:0/3347788710 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1d98004c40 con 0x7f1da0100e20 2024-04-22T15:32:08.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.345+0000 7f1d977fe700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8c06ecc0 0x7f1d8c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:08.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.345+0000 7f1d977fe700 1 -- 172.21.15.96:0/3347788710 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1d9808e810 con 0x7f1da0100e20 2024-04-22T15:32:08.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.345+0000 7f1d9dd9b700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8c06ecc0 0x7f1d8c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:08.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.345+0000 7f1d9dd9b700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8c06ecc0 0x7f1d8c071180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1da0197b20 tx=0x7f1d8801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:08.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.350+0000 7f1d977fe700 1 -- 172.21.15.96:0/3347788710 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1d9805b120 con 0x7f1da0100e20 2024-04-22T15:32:08.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.642+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1da0197e20 con 0x7f1da0100e20 2024-04-22T15:32:08.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.643+0000 7f1d977fe700 1 -- 172.21.15.96:0/3347788710 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f1d9805acb0 con 0x7f1da0100e20 2024-04-22T15:32:08.646 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:32:08.646 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:32:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:32:08.646 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:32:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:32:08.647 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:32:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:32:08.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8c06ecc0 msgr2=0x7f1d8c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:08.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8c06ecc0 0x7f1d8c071180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1da0197b20 tx=0x7f1d8801a040 comp rx=0 tx=0).stop 2024-04-22T15:32:08.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da0100e20 msgr2=0x7f1da019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:08.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da0100e20 0x7f1da019c820 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1d98007e90 tx=0x7f1d9800d3b0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 shutdown_connections 2024-04-22T15:32:08.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8c06ecc0 0x7f1d8c071180 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1da0100e20 0x7f1da019c820 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 --2- 172.21.15.96:0/3347788710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1da01017d0 0x7f1da019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:08.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 >> 172.21.15.96:0/3347788710 conn(0x7f1da00fc6d0 msgr2=0x7f1da00fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:08.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 shutdown_connections 2024-04-22T15:32:08.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:08.645+0000 7f1da489b700 1 -- 172.21.15.96:0/3347788710 wait complete. 2024-04-22T15:32:08.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:08 smithi196 ceph-mon[81815]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:08 smithi196 ceph-mon[81815]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:08 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2540768307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:32:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:08 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1010799304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:32:08.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:32:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:08 smithi096 ceph-mon[112443]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:08 smithi096 ceph-mon[112443]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:08 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2540768307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:32:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:08 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1010799304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:32:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:32:09.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:09 smithi196 ceph-mon[81815]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:09.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:09 smithi196 ceph-mon[81815]: pgmap v308: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:09.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:09 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3347788710' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:32:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:09 smithi096 ceph-mon[112443]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:09 smithi096 ceph-mon[112443]: pgmap v308: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:09 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3347788710' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:32:11.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:11 smithi196 ceph-mon[81815]: pgmap v309: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:11.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:11 smithi096 ceph-mon[112443]: pgmap v309: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:12 smithi096 ceph-mon[112443]: pgmap v310: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:12 smithi196 ceph-mon[81815]: pgmap v310: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:15 smithi096 ceph-mon[112443]: pgmap v311: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:15 smithi196 ceph-mon[81815]: pgmap v311: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:17.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:17 smithi096 ceph-mon[112443]: pgmap v312: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:17 smithi196 ceph-mon[81815]: pgmap v312: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:19 smithi096 ceph-mon[112443]: pgmap v313: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:19 smithi196 ceph-mon[81815]: pgmap v313: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:21 smithi096 ceph-mon[112443]: pgmap v314: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:21.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:21 smithi196 ceph-mon[81815]: pgmap v314: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:23.727 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:23 smithi196 ceph-mon[81815]: pgmap v315: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:23.727 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:32:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:23 smithi096 ceph-mon[112443]: pgmap v315: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:32:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:25 smithi096 ceph-mon[112443]: pgmap v316: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:25 smithi196 ceph-mon[81815]: pgmap v316: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:27.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:27 smithi096 ceph-mon[112443]: pgmap v317: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:27 smithi196 ceph-mon[81815]: pgmap v317: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:29.720 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:29 smithi096 ceph-mon[112443]: pgmap v318: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:29.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:29 smithi196 ceph-mon[81815]: pgmap v318: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:31 smithi096 ceph-mon[112443]: pgmap v319: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:31 smithi196 ceph-mon[81815]: pgmap v319: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:33 smithi096 ceph-mon[112443]: pgmap v320: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:33 smithi196 ceph-mon[81815]: pgmap v320: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:35 smithi096 ceph-mon[112443]: pgmap v321: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:35 smithi196 ceph-mon[81815]: pgmap v321: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:37 smithi096 ceph-mon[112443]: pgmap v322: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:37 smithi196 ceph-mon[81815]: pgmap v322: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:32:38.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:32:38.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.763+0000 7fb4bf055700 1 -- 172.21.15.96:0/2096978555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 msgr2=0x7fb4b8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:38.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.763+0000 7fb4bf055700 1 --2- 172.21.15.96:0/2096978555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 0x7fb4b8101200 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009910 tx=0x7fb4b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:38.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.763+0000 7fb4bf055700 1 -- 172.21.15.96:0/2096978555 shutdown_connections 2024-04-22T15:32:38.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.763+0000 7fb4bf055700 1 --2- 172.21.15.96:0/2096978555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4b81017d0 0x7fb4b8105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.763+0000 7fb4bf055700 1 --2- 172.21.15.96:0/2096978555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 0x7fb4b8101200 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.763+0000 7fb4bf055700 1 -- 172.21.15.96:0/2096978555 >> 172.21.15.96:0/2096978555 conn(0x7fb4b80fc6d0 msgr2=0x7fb4b80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.763+0000 7fb4bf055700 1 -- 172.21.15.96:0/2096978555 shutdown_connections 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.763+0000 7fb4bf055700 1 -- 172.21.15.96:0/2096978555 wait complete. 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bf055700 1 Processor -- start 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bf055700 1 -- start start 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bf055700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 0x7fb4b80731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bf055700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4b81017d0 0x7fb4b8073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bf055700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8073c60 con 0x7fb4b8100e20 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bf055700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8073da0 con 0x7fb4b81017d0 2024-04-22T15:32:38.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bcdf1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 0x7fb4b80731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bcdf1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 0x7fb4b80731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32898/0 (socket says 172.21.15.96:32898) 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bcdf1700 1 -- 172.21.15.96:0/3001350966 learned_addr learned my addr 172.21.15.96:0/3001350966 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4affff700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4b81017d0 0x7fb4b8073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bcdf1700 1 -- 172.21.15.96:0/3001350966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4b81017d0 msgr2=0x7fb4b8073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bcdf1700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4b81017d0 0x7fb4b8073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bcdf1700 1 -- 172.21.15.96:0/3001350966 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b40095f0 con 0x7fb4b8100e20 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.764+0000 7fb4bcdf1700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 0x7fb4b80731e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400bd70 tx=0x7fb4b400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.765+0000 7fb4adffb700 1 -- 172.21.15.96:0/3001350966 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b401d070 con 0x7fb4b8100e20 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.765+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b8074020 con 0x7fb4b8100e20 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.765+0000 7fb4adffb700 1 -- 172.21.15.96:0/3001350966 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4b40056e0 con 0x7fb4b8100e20 2024-04-22T15:32:38.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.765+0000 7fb4adffb700 1 -- 172.21.15.96:0/3001350966 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b4017400 con 0x7fb4b8100e20 2024-04-22T15:32:38.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.765+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b81a2f80 con 0x7fb4b8100e20 2024-04-22T15:32:38.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.766+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b81091c0 con 0x7fb4b8100e20 2024-04-22T15:32:38.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.766+0000 7fb4adffb700 1 -- 172.21.15.96:0/3001350966 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb4b4003a60 con 0x7fb4b8100e20 2024-04-22T15:32:38.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.767+0000 7fb4adffb700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4a006ecc0 0x7fb4a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:38.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.767+0000 7fb4adffb700 1 -- 172.21.15.96:0/3001350966 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb4b40902e0 con 0x7fb4b8100e20 2024-04-22T15:32:38.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.767+0000 7fb4affff700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4a006ecc0 0x7fb4a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:38.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.767+0000 7fb4affff700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4a006ecc0 0x7fb4a0071180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8005950 tx=0x7fb4a80058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:38.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.772+0000 7fb4adffb700 1 -- 172.21.15.96:0/3001350966 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb4b4059040 con 0x7fb4b8100e20 2024-04-22T15:32:38.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.988+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4b810ddf0 con 0x7fb4a006ecc0 2024-04-22T15:32:38.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.990+0000 7fb4adffb700 1 -- 172.21.15.96:0/3001350966 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb4b810ddf0 con 0x7fb4a006ecc0 2024-04-22T15:32:38.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4a006ecc0 msgr2=0x7fb4a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:38.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4a006ecc0 0x7fb4a0071180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8005950 tx=0x7fb4a80058e0 comp rx=0 tx=0).stop 2024-04-22T15:32:38.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 msgr2=0x7fb4b80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:38.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 0x7fb4b80731e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400bd70 tx=0x7fb4b400be50 comp rx=0 tx=0).stop 2024-04-22T15:32:38.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 shutdown_connections 2024-04-22T15:32:38.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4a006ecc0 0x7fb4a0071180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:38.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b8100e20 0x7fb4b80731e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:38.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 --2- 172.21.15.96:0/3001350966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4b81017d0 0x7fb4b8073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:38.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 >> 172.21.15.96:0/3001350966 conn(0x7fb4b80fc6d0 msgr2=0x7fb4b80fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:38.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 shutdown_connections 2024-04-22T15:32:38.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:38.992+0000 7fb4bf055700 1 -- 172.21.15.96:0/3001350966 wait complete. 2024-04-22T15:32:39.006 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:32:39.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.097+0000 7f59e6089700 1 -- 172.21.15.96:0/472625179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 msgr2=0x7f59e010aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.097+0000 7f59e6089700 1 --2- 172.21.15.96:0/472625179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 0x7f59e010aba0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009910 tx=0x7f59d4009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:39.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.097+0000 7f59e6089700 1 -- 172.21.15.96:0/472625179 shutdown_connections 2024-04-22T15:32:39.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.097+0000 7f59e6089700 1 --2- 172.21.15.96:0/472625179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 0x7f59e010aba0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.097+0000 7f59e6089700 1 --2- 172.21.15.96:0/472625179 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59e0101d90 0x7f59e0102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.097+0000 7f59e6089700 1 -- 172.21.15.96:0/472625179 >> 172.21.15.96:0/472625179 conn(0x7f59e00fb5e0 msgr2=0x7f59e00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:39.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.097+0000 7f59e6089700 1 -- 172.21.15.96:0/472625179 shutdown_connections 2024-04-22T15:32:39.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.097+0000 7f59e6089700 1 -- 172.21.15.96:0/472625179 wait complete. 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.098+0000 7f59e6089700 1 Processor -- start 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.098+0000 7f59e6089700 1 -- start start 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.098+0000 7f59e6089700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59e0101d90 0x7f59e019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.098+0000 7f59e6089700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 0x7f59e019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.098+0000 7f59e6089700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e019d3f0 con 0x7f59e01026b0 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.098+0000 7f59e6089700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e01969d0 con 0x7f59e0101d90 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59deffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 0x7f59e019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59deffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 0x7f59e019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32918/0 (socket says 172.21.15.96:32918) 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59deffd700 1 -- 172.21.15.96:0/1788662902 learned_addr learned my addr 172.21.15.96:0/1788662902 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59df7fe700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59e0101d90 0x7f59e019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59deffd700 1 -- 172.21.15.96:0/1788662902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59e0101d90 msgr2=0x7f59e019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59deffd700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59e0101d90 0x7f59e019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59deffd700 1 -- 172.21.15.96:0/1788662902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d40095f0 con 0x7f59e01026b0 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59df7fe700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59e0101d90 0x7f59e019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59deffd700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 0x7f59e019cdd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f59d4000c00 tx=0x7f59d400bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59dcff9700 1 -- 172.21.15.96:0/1788662902 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d401d070 con 0x7f59e01026b0 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e0196c50 con 0x7f59e01026b0 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59dcff9700 1 -- 172.21.15.96:0/1788662902 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d4005e00 con 0x7f59e01026b0 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59dcff9700 1 -- 172.21.15.96:0/1788662902 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d400fdf0 con 0x7f59e01026b0 2024-04-22T15:32:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.099+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e0197140 con 0x7f59e01026b0 2024-04-22T15:32:39.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.100+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e0105520 con 0x7f59e01026b0 2024-04-22T15:32:39.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.101+0000 7f59dcff9700 1 -- 172.21.15.96:0/1788662902 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f59d400f460 con 0x7f59e01026b0 2024-04-22T15:32:39.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.101+0000 7f59dcff9700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59c806ecd0 0x7f59c8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:39.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.101+0000 7f59dcff9700 1 -- 172.21.15.96:0/1788662902 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f59d4090200 con 0x7f59e01026b0 2024-04-22T15:32:39.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.101+0000 7f59df7fe700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59c806ecd0 0x7f59c8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:39.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.102+0000 7f59df7fe700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59c806ecd0 0x7f59c8071190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f59d0005fd0 tx=0x7f59d0005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:39.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.106+0000 7f59dcff9700 1 -- 172.21.15.96:0/1788662902 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f59d405ca90 con 0x7f59e01026b0 2024-04-22T15:32:39.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.325+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f59e0197f00 con 0x7f59c806ecd0 2024-04-22T15:32:39.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.326+0000 7f59dcff9700 1 -- 172.21.15.96:0/1788662902 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f59e0197f00 con 0x7f59c806ecd0 2024-04-22T15:32:39.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59c806ecd0 msgr2=0x7f59c8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59c806ecd0 0x7f59c8071190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f59d0005fd0 tx=0x7f59d0005ee0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 msgr2=0x7f59e019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 0x7f59e019cdd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f59d4000c00 tx=0x7f59d400bbf0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 shutdown_connections 2024-04-22T15:32:39.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59c806ecd0 0x7f59c8071190 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59e0101d90 0x7f59e019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 --2- 172.21.15.96:0/1788662902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59e01026b0 0x7f59e019cdd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 >> 172.21.15.96:0/1788662902 conn(0x7f59e00fb5e0 msgr2=0x7f59e0103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:39.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 shutdown_connections 2024-04-22T15:32:39.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.328+0000 7f59e6089700 1 -- 172.21.15.96:0/1788662902 wait complete. 2024-04-22T15:32:39.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.432+0000 7fc2674f8700 1 -- 172.21.15.96:0/3639955819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2601017c0 msgr2=0x7fc260105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.432+0000 7fc2674f8700 1 --2- 172.21.15.96:0/3639955819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2601017c0 0x7fc260105810 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009910 tx=0x7fc25c009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:39.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.432+0000 7fc2674f8700 1 -- 172.21.15.96:0/3639955819 shutdown_connections 2024-04-22T15:32:39.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.432+0000 7fc2674f8700 1 --2- 172.21.15.96:0/3639955819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2601017c0 0x7fc260105810 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.432+0000 7fc2674f8700 1 --2- 172.21.15.96:0/3639955819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc260100e10 0x7fc2601011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.432+0000 7fc2674f8700 1 -- 172.21.15.96:0/3639955819 >> 172.21.15.96:0/3639955819 conn(0x7fc2600fc6e0 msgr2=0x7fc2600feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:39.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.432+0000 7fc2674f8700 1 -- 172.21.15.96:0/3639955819 shutdown_connections 2024-04-22T15:32:39.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.432+0000 7fc2674f8700 1 -- 172.21.15.96:0/3639955819 wait complete. 2024-04-22T15:32:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc2674f8700 1 Processor -- start 2024-04-22T15:32:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc2674f8700 1 -- start start 2024-04-22T15:32:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc2674f8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc260100e10 0x7fc26019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc2674f8700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2601017c0 0x7fc26019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc2674f8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26019d380 con 0x7fc260100e10 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc2674f8700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2601969b0 con 0x7fc2601017c0 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc265294700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc260100e10 0x7fc26019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc265294700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc260100e10 0x7fc26019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32930/0 (socket says 172.21.15.96:32930) 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc265294700 1 -- 172.21.15.96:0/3843623505 learned_addr learned my addr 172.21.15.96:0/3843623505 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc264a93700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2601017c0 0x7fc26019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc265294700 1 -- 172.21.15.96:0/3843623505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2601017c0 msgr2=0x7fc26019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc265294700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2601017c0 0x7fc26019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.433+0000 7fc265294700 1 -- 172.21.15.96:0/3843623505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c0095f0 con 0x7fc260100e10 2024-04-22T15:32:39.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.434+0000 7fc265294700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc260100e10 0x7fc26019c820 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc25000d8d0 tx=0x7fc25000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:39.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.436+0000 7fc2567fc700 1 -- 172.21.15.96:0/3843623505 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2500098a0 con 0x7fc260100e10 2024-04-22T15:32:39.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.436+0000 7fc2567fc700 1 -- 172.21.15.96:0/3843623505 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc250010460 con 0x7fc260100e10 2024-04-22T15:32:39.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.436+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc260196c90 con 0x7fc260100e10 2024-04-22T15:32:39.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.436+0000 7fc2567fc700 1 -- 172.21.15.96:0/3843623505 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25000f5f0 con 0x7fc260100e10 2024-04-22T15:32:39.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.436+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2601971e0 con 0x7fc260100e10 2024-04-22T15:32:39.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.437+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc260109210 con 0x7fc260100e10 2024-04-22T15:32:39.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.438+0000 7fc2567fc700 1 -- 172.21.15.96:0/3843623505 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc250009a00 con 0x7fc260100e10 2024-04-22T15:32:39.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.438+0000 7fc2567fc700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc24c06ecc0 0x7fc24c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:39.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.438+0000 7fc2567fc700 1 -- 172.21.15.96:0/3843623505 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc25008ec70 con 0x7fc260100e10 2024-04-22T15:32:39.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.438+0000 7fc264a93700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc24c06ecc0 0x7fc24c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:39.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.438+0000 7fc264a93700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc24c06ecc0 0x7fc24c071180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc260197b20 tx=0x7fc25c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:39.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.443+0000 7fc2567fc700 1 -- 172.21.15.96:0/3843623505 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc25005b580 con 0x7fc260100e10 2024-04-22T15:32:39.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.659+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc260197e90 con 0x7fc24c06ecc0 2024-04-22T15:32:39.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.670+0000 7fc2567fc700 1 -- 172.21.15.96:0/3843623505 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc260197e90 con 0x7fc24c06ecc0 2024-04-22T15:32:39.673 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:32:39.673 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (11m) 4m ago 22m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:32:39.673 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (22m) 4m ago 22m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (21m) 4m ago 21m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (8m) 4m ago 22m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (8m) 4m ago 21m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (10m) 4m ago 22m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (4m) 4m ago 16m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (4m) 4m ago 17m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (4m) 4m ago 16m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (16m) 4m ago 16m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (13m) 4m ago 24m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:32:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (12m) 4m ago 21m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (9m) 4m ago 24m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (9m) 4m ago 21m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (12m) 4m ago 22m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (12m) 4m ago 21m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (8m) 4m ago 20m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (7m) 4m ago 20m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (6m) 4m ago 19m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (6m) 4m ago 19m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (5m) 4m ago 18m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (5m) 4m ago 18m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:32:39.675 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (11m) 4m ago 22m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:32:39.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.672+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc24c06ecc0 msgr2=0x7fc24c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.672+0000 7fc2674f8700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc24c06ecc0 0x7fc24c071180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc260197b20 tx=0x7fc25c01a040 comp rx=0 tx=0).stop 2024-04-22T15:32:39.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.672+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc260100e10 msgr2=0x7fc26019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.672+0000 7fc2674f8700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc260100e10 0x7fc26019c820 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc25000d8d0 tx=0x7fc25000dbe0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.672+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 shutdown_connections 2024-04-22T15:32:39.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.672+0000 7fc2674f8700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc24c06ecc0 0x7fc24c071180 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.672+0000 7fc2674f8700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc260100e10 0x7fc26019c820 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.672+0000 7fc2674f8700 1 --2- 172.21.15.96:0/3843623505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2601017c0 0x7fc26019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.672+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 >> 172.21.15.96:0/3843623505 conn(0x7fc2600fc6e0 msgr2=0x7fc2600fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:39.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.673+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 shutdown_connections 2024-04-22T15:32:39.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.673+0000 7fc2674f8700 1 -- 172.21.15.96:0/3843623505 wait complete. 2024-04-22T15:32:39.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:39 smithi096 ceph-mon[112443]: pgmap v323: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:39 smithi196 ceph-mon[81815]: pgmap v323: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:39.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 -- 172.21.15.96:0/1440564510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 msgr2=0x7f5148105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 --2- 172.21.15.96:0/1440564510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 0x7f5148105820 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5144009910 tx=0x7f5144009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:39.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 -- 172.21.15.96:0/1440564510 shutdown_connections 2024-04-22T15:32:39.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 --2- 172.21.15.96:0/1440564510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 0x7f5148105820 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 --2- 172.21.15.96:0/1440564510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5148100e20 0x7f5148101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 -- 172.21.15.96:0/1440564510 >> 172.21.15.96:0/1440564510 conn(0x7f51480fc6d0 msgr2=0x7f51480feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:39.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 -- 172.21.15.96:0/1440564510 shutdown_connections 2024-04-22T15:32:39.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 -- 172.21.15.96:0/1440564510 wait complete. 2024-04-22T15:32:39.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 Processor -- start 2024-04-22T15:32:39.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 -- start start 2024-04-22T15:32:39.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5148100e20 0x7f514819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:39.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 0x7f514819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:39.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.778+0000 7f514fce0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514819d380 con 0x7f51481017d0 2024-04-22T15:32:39.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514fce0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5148196960 con 0x7f5148100e20 2024-04-22T15:32:39.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514d27b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 0x7f514819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:39.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514d27b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 0x7f514819cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32936/0 (socket says 172.21.15.96:32936) 2024-04-22T15:32:39.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514d27b700 1 -- 172.21.15.96:0/306672822 learned_addr learned my addr 172.21.15.96:0/306672822 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:39.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514da7c700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5148100e20 0x7f514819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514d27b700 1 -- 172.21.15.96:0/306672822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5148100e20 msgr2=0x7f514819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514d27b700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5148100e20 0x7f514819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514d27b700 1 -- 172.21.15.96:0/306672822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51440095f0 con 0x7f51481017d0 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514d27b700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 0x7f514819cd60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f51440092e0 tx=0x7f514400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f513effd700 1 -- 172.21.15.96:0/306672822 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f514401d070 con 0x7f51481017d0 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f513effd700 1 -- 172.21.15.96:0/306672822 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5144005700 con 0x7f51481017d0 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5148196be0 con 0x7f51481017d0 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f513effd700 1 -- 172.21.15.96:0/306672822 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f514400fdb0 con 0x7f51481017d0 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.779+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51481970d0 con 0x7f51481017d0 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.780+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5148109220 con 0x7f51481017d0 2024-04-22T15:32:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.781+0000 7f513effd700 1 -- 172.21.15.96:0/306672822 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f514400f5f0 con 0x7f51481017d0 2024-04-22T15:32:39.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.781+0000 7f513effd700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f513406ecc0 0x7f5134071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:39.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.781+0000 7f513effd700 1 -- 172.21.15.96:0/306672822 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5144090220 con 0x7f51481017d0 2024-04-22T15:32:39.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.781+0000 7f514da7c700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f513406ecc0 0x7f5134071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:39.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.782+0000 7f514da7c700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f513406ecc0 0x7f5134071180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5138007be0 tx=0x7f51380073d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:39.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:39.786+0000 7f513effd700 1 -- 172.21.15.96:0/306672822 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f514405ca10 con 0x7f51481017d0 2024-04-22T15:32:40.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.074+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5148197e20 con 0x7f51481017d0 2024-04-22T15:32:40.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.075+0000 7f513effd700 1 -- 172.21.15.96:0/306672822 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f514405c5a0 con 0x7f51481017d0 2024-04-22T15:32:40.078 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:32:40.079 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:32:40.080 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:32:40.080 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:32:40.080 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:32:40.080 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:32:40.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f513406ecc0 msgr2=0x7f5134071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f513406ecc0 0x7f5134071180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5138007be0 tx=0x7f51380073d0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 msgr2=0x7f514819cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 0x7f514819cd60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f51440092e0 tx=0x7f514400bd70 comp rx=0 tx=0).stop 2024-04-22T15:32:40.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 shutdown_connections 2024-04-22T15:32:40.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f513406ecc0 0x7f5134071180 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5148100e20 0x7f514819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 --2- 172.21.15.96:0/306672822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51481017d0 0x7f514819cd60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 >> 172.21.15.96:0/306672822 conn(0x7f51480fc6d0 msgr2=0x7f51480fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:40.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 shutdown_connections 2024-04-22T15:32:40.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.078+0000 7f514fce0700 1 -- 172.21.15.96:0/306672822 wait complete. 2024-04-22T15:32:40.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.182+0000 7f0bb6772700 1 -- 172.21.15.96:0/831047858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 msgr2=0x7f0bb00ff050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.182+0000 7f0bb6772700 1 --2- 172.21.15.96:0/831047858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 0x7f0bb00ff050 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009910 tx=0x7f0ba0009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:40.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.182+0000 7f0bb6772700 1 -- 172.21.15.96:0/831047858 shutdown_connections 2024-04-22T15:32:40.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.182+0000 7f0bb6772700 1 --2- 172.21.15.96:0/831047858 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0bb0101f20 0x7f0bb00ff590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.182+0000 7f0bb6772700 1 --2- 172.21.15.96:0/831047858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 0x7f0bb00ff050 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.182+0000 7f0bb6772700 1 -- 172.21.15.96:0/831047858 >> 172.21.15.96:0/831047858 conn(0x7f0bb0075f40 msgr2=0x7f0bb00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:40.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.182+0000 7f0bb6772700 1 -- 172.21.15.96:0/831047858 shutdown_connections 2024-04-22T15:32:40.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.182+0000 7f0bb6772700 1 -- 172.21.15.96:0/831047858 wait complete. 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0bb6772700 1 Processor -- start 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0bb6772700 1 -- start start 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0bb6772700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 0x7f0bb0102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0bb6772700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0bb0101f20 0x7f0bb0102980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0bb6772700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb0106510 con 0x7f0bb0101b40 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0bb6772700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb0102ec0 con 0x7f0bb0101f20 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0baffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 0x7f0bb0102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0baffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 0x7f0bb0102440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32956/0 (socket says 172.21.15.96:32956) 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0baffff700 1 -- 172.21.15.96:0/2655935324 learned_addr learned my addr 172.21.15.96:0/2655935324 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0baffff700 1 -- 172.21.15.96:0/2655935324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0bb0101f20 msgr2=0x7f0bb0102980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0baf7fe700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0bb0101f20 0x7f0bb0102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0baffff700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0bb0101f20 0x7f0bb0102980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0baffff700 1 -- 172.21.15.96:0/2655935324 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba00095f0 con 0x7f0bb0101b40 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.183+0000 7f0baf7fe700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0bb0101f20 0x7f0bb0102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.184+0000 7f0baffff700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 0x7f0bb0102440 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0ba000bd70 tx=0x7f0ba000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.184+0000 7f0bad7fa700 1 -- 172.21.15.96:0/2655935324 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ba001d070 con 0x7f0bb0101b40 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.184+0000 7f0bad7fa700 1 -- 172.21.15.96:0/2655935324 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ba00056e0 con 0x7f0bb0101b40 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.184+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb0103140 con 0x7f0bb0101b40 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.184+0000 7f0bad7fa700 1 -- 172.21.15.96:0/2655935324 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ba0017400 con 0x7f0bb0101b40 2024-04-22T15:32:40.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.184+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb01a74a0 con 0x7f0bb0101b40 2024-04-22T15:32:40.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.185+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb010af40 con 0x7f0bb0101b40 2024-04-22T15:32:40.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.185+0000 7f0bad7fa700 1 -- 172.21.15.96:0/2655935324 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0ba0003a60 con 0x7f0bb0101b40 2024-04-22T15:32:40.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.186+0000 7f0bad7fa700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0b9806ecc0 0x7f0b98071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:40.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.186+0000 7f0bad7fa700 1 -- 172.21.15.96:0/2655935324 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0ba0090240 con 0x7f0bb0101b40 2024-04-22T15:32:40.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.186+0000 7f0baf7fe700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0b9806ecc0 0x7f0b98071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:40.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.186+0000 7f0baf7fe700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0b9806ecc0 0x7f0b98071180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4005fd0 tx=0x7f0ba4005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:40.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.190+0000 7f0bad7fa700 1 -- 172.21.15.96:0/2655935324 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0ba0059020 con 0x7f0bb0101b40 2024-04-22T15:32:40.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.432+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0bb01039c0 con 0x7f0bb0101b40 2024-04-22T15:32:40.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.432+0000 7f0bad7fa700 1 -- 172.21.15.96:0/2655935324 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f0ba005c6e0 con 0x7f0bb0101b40 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:32:40.436 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:32:40.437 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:32:40.438 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:32:40.439 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:32:40.439 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:32:40.439 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:32:40.439 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:32:40.439 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:32:40.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0b9806ecc0 msgr2=0x7f0b98071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0b9806ecc0 0x7f0b98071180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4005fd0 tx=0x7f0ba4005ee0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 msgr2=0x7f0bb0102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 0x7f0bb0102440 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0ba000bd70 tx=0x7f0ba000be50 comp rx=0 tx=0).stop 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 shutdown_connections 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0b9806ecc0 0x7f0b98071180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb0101b40 0x7f0bb0102440 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 --2- 172.21.15.96:0/2655935324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0bb0101f20 0x7f0bb0102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 >> 172.21.15.96:0/2655935324 conn(0x7f0bb0075f40 msgr2=0x7f0bb00fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 shutdown_connections 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.435+0000 7f0bb6772700 1 -- 172.21.15.96:0/2655935324 wait complete. 2024-04-22T15:32:40.440 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:32:40.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.541+0000 7f39ba1b9700 1 -- 172.21.15.96:0/3463265022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 msgr2=0x7f39b4104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.541+0000 7f39ba1b9700 1 --2- 172.21.15.96:0/3463265022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 0x7f39b4104f10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009910 tx=0x7f39a4009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:40.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.541+0000 7f39ba1b9700 1 -- 172.21.15.96:0/3463265022 shutdown_connections 2024-04-22T15:32:40.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.541+0000 7f39ba1b9700 1 --2- 172.21.15.96:0/3463265022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b4100260 0x7f39b41006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.541+0000 7f39ba1b9700 1 --2- 172.21.15.96:0/3463265022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 0x7f39b4104f10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.541+0000 7f39ba1b9700 1 -- 172.21.15.96:0/3463265022 >> 172.21.15.96:0/3463265022 conn(0x7f39b4075f40 msgr2=0x7f39b40fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:40.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.541+0000 7f39ba1b9700 1 -- 172.21.15.96:0/3463265022 shutdown_connections 2024-04-22T15:32:40.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.542+0000 7f39ba1b9700 1 -- 172.21.15.96:0/3463265022 wait complete. 2024-04-22T15:32:40.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.542+0000 7f39ba1b9700 1 Processor -- start 2024-04-22T15:32:40.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.542+0000 7f39ba1b9700 1 -- start start 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.542+0000 7f39ba1b9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b4100260 0x7f39b41979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.542+0000 7f39ba1b9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 0x7f39b4197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.542+0000 7f39ba1b9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b41984d0 con 0x7f39b4104b30 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.542+0000 7f39ba1b9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b4198640 con 0x7f39b4100260 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 0x7f39b4197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 0x7f39b4197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49086/0 (socket says 172.21.15.96:49086) 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b2ffd700 1 -- 172.21.15.96:0/1038180127 learned_addr learned my addr 172.21.15.96:0/1038180127 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b37fe700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b4100260 0x7f39b41979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b2ffd700 1 -- 172.21.15.96:0/1038180127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b4100260 msgr2=0x7f39b41979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b2ffd700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b4100260 0x7f39b41979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b2ffd700 1 -- 172.21.15.96:0/1038180127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a40095f0 con 0x7f39b4104b30 2024-04-22T15:32:40.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b37fe700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b4100260 0x7f39b41979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:32:40.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b2ffd700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 0x7f39b4197f00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f39a800d8d0 tx=0x7f39a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:40.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b0ff9700 1 -- 172.21.15.96:0/1038180127 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a8009940 con 0x7f39b4104b30 2024-04-22T15:32:40.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b41a0a30 con 0x7f39b4104b30 2024-04-22T15:32:40.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b0ff9700 1 -- 172.21.15.96:0/1038180127 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39a8010460 con 0x7f39b4104b30 2024-04-22T15:32:40.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39b0ff9700 1 -- 172.21.15.96:0/1038180127 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a800f5f0 con 0x7f39b4104b30 2024-04-22T15:32:40.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.543+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b41a0ea0 con 0x7f39b4104b30 2024-04-22T15:32:40.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.544+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b4108120 con 0x7f39b4104b30 2024-04-22T15:32:40.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.545+0000 7f39b0ff9700 1 -- 172.21.15.96:0/1038180127 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f39a80105d0 con 0x7f39b4104b30 2024-04-22T15:32:40.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.545+0000 7f39b0ff9700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f399c06ecc0 0x7f399c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:40.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.545+0000 7f39b0ff9700 1 -- 172.21.15.96:0/1038180127 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f39a808eed0 con 0x7f39b4104b30 2024-04-22T15:32:40.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.545+0000 7f39b37fe700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f399c06ecc0 0x7f399c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:40.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.545+0000 7f39b37fe700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f399c06ecc0 0x7f399c071180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f39a40059d0 tx=0x7f39a401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:40.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.550+0000 7f39b0ff9700 1 -- 172.21.15.96:0/1038180127 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f39a8057c30 con 0x7f39b4104b30 2024-04-22T15:32:40.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:40 smithi096 ceph-mon[112443]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:40.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:40 smithi096 ceph-mon[112443]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:40.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:40 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/306672822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:32:40.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:40 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2655935324' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:32:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:40 smithi196 ceph-mon[81815]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:40 smithi196 ceph-mon[81815]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:40 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/306672822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:32:40.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:40 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2655935324' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:32:40.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.771+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39b410cd00 con 0x7f399c06ecc0 2024-04-22T15:32:40.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.772+0000 7f39b0ff9700 1 -- 172.21.15.96:0/1038180127 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f39b410cd00 con 0x7f399c06ecc0 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:32:40.776 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:32:40.777 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:32:40.777 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:32:40.777 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:32:40.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.774+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f399c06ecc0 msgr2=0x7f399c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.774+0000 7f39ba1b9700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f399c06ecc0 0x7f399c071180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f39a40059d0 tx=0x7f39a401a040 comp rx=0 tx=0).stop 2024-04-22T15:32:40.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.774+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 msgr2=0x7f39b4197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.774+0000 7f39ba1b9700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 0x7f39b4197f00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f39a800d8d0 tx=0x7f39a800dc90 comp rx=0 tx=0).stop 2024-04-22T15:32:40.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.774+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 shutdown_connections 2024-04-22T15:32:40.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.774+0000 7f39ba1b9700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f399c06ecc0 0x7f399c071180 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.774+0000 7f39ba1b9700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b4100260 0x7f39b41979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.774+0000 7f39ba1b9700 1 --2- 172.21.15.96:0/1038180127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4104b30 0x7f39b4197f00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.774+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 >> 172.21.15.96:0/1038180127 conn(0x7f39b4075f40 msgr2=0x7f39b40fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:40.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.775+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 shutdown_connections 2024-04-22T15:32:40.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.775+0000 7f39ba1b9700 1 -- 172.21.15.96:0/1038180127 wait complete. 2024-04-22T15:32:40.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.877+0000 7f25d3d0e700 1 -- 172.21.15.96:0/1266425189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 msgr2=0x7f25cc107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.877+0000 7f25d3d0e700 1 --2- 172.21.15.96:0/1266425189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 0x7f25cc107150 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009910 tx=0x7f25c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:32:40.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.877+0000 7f25d3d0e700 1 -- 172.21.15.96:0/1266425189 shutdown_connections 2024-04-22T15:32:40.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.877+0000 7f25d3d0e700 1 --2- 172.21.15.96:0/1266425189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25cc0ff3d0 0x7f25cc0ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.877+0000 7f25d3d0e700 1 --2- 172.21.15.96:0/1266425189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 0x7f25cc107150 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.877+0000 7f25d3d0e700 1 -- 172.21.15.96:0/1266425189 >> 172.21.15.96:0/1266425189 conn(0x7f25cc074d90 msgr2=0x7f25cc0751a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:40.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.877+0000 7f25d3d0e700 1 -- 172.21.15.96:0/1266425189 shutdown_connections 2024-04-22T15:32:40.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.877+0000 7f25d3d0e700 1 -- 172.21.15.96:0/1266425189 wait complete. 2024-04-22T15:32:40.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.877+0000 7f25d3d0e700 1 Processor -- start 2024-04-22T15:32:40.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d3d0e700 1 -- start start 2024-04-22T15:32:40.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d3d0e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25cc0ff3d0 0x7f25cc19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:40.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d3d0e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 0x7f25cc19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:40.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d3d0e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc19d460 con 0x7f25cc106d70 2024-04-22T15:32:40.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d3d0e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc196a40 con 0x7f25cc0ff3d0 2024-04-22T15:32:40.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d12a9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 0x7f25cc19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:40.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d12a9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 0x7f25cc19ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49092/0 (socket says 172.21.15.96:49092) 2024-04-22T15:32:40.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d12a9700 1 -- 172.21.15.96:0/2512132712 learned_addr learned my addr 172.21.15.96:0/2512132712 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d1aaa700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25cc0ff3d0 0x7f25cc19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d12a9700 1 -- 172.21.15.96:0/2512132712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25cc0ff3d0 msgr2=0x7f25cc19c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d12a9700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25cc0ff3d0 0x7f25cc19c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d12a9700 1 -- 172.21.15.96:0/2512132712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c80095f0 con 0x7f25cc106d70 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25d12a9700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 0x7f25cc19ce40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00d8d0 tx=0x7f25bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.878+0000 7f25c2ffd700 1 -- 172.21.15.96:0/2512132712 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25bc009940 con 0x7f25cc106d70 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.879+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25cc196d20 con 0x7f25cc106d70 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.879+0000 7f25c2ffd700 1 -- 172.21.15.96:0/2512132712 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25bc010460 con 0x7f25cc106d70 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.879+0000 7f25c2ffd700 1 -- 172.21.15.96:0/2512132712 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25bc00f5f0 con 0x7f25cc106d70 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.879+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25cc197270 con 0x7f25cc106d70 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.879+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25cc1092e0 con 0x7f25cc106d70 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.880+0000 7f25c2ffd700 1 -- 172.21.15.96:0/2512132712 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f25bc0105d0 con 0x7f25cc106d70 2024-04-22T15:32:40.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.880+0000 7f25c2ffd700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25b806ecc0 0x7f25b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:32:40.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.880+0000 7f25c2ffd700 1 -- 172.21.15.96:0/2512132712 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f25bc08eed0 con 0x7f25cc106d70 2024-04-22T15:32:40.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.881+0000 7f25d1aaa700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25b806ecc0 0x7f25b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:32:40.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.881+0000 7f25d1aaa700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25b806ecc0 0x7f25b8071180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b5c0 tx=0x7f25c80052a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:32:40.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:40.885+0000 7f25c2ffd700 1 -- 172.21.15.96:0/2512132712 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f25bc05b760 con 0x7f25cc106d70 2024-04-22T15:32:41.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.167+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f25cc068d70 con 0x7f25cc106d70 2024-04-22T15:32:41.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.168+0000 7f25c2ffd700 1 -- 172.21.15.96:0/2512132712 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f25bc05b2f0 con 0x7f25cc106d70 2024-04-22T15:32:41.171 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:32:41.171 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:32:41.171 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:32:41.171 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:32:41.171 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:32:41.171 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:32:41.172 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:32:41.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25b806ecc0 msgr2=0x7f25b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:41.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25b806ecc0 0x7f25b8071180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b5c0 tx=0x7f25c80052a0 comp rx=0 tx=0).stop 2024-04-22T15:32:41.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 msgr2=0x7f25cc19ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:32:41.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 0x7f25cc19ce40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00d8d0 tx=0x7f25bc00dc90 comp rx=0 tx=0).stop 2024-04-22T15:32:41.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 shutdown_connections 2024-04-22T15:32:41.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25b806ecc0 0x7f25b8071180 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:41.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25cc0ff3d0 0x7f25cc19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:41.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 --2- 172.21.15.96:0/2512132712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25cc106d70 0x7f25cc19ce40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:32:41.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 >> 172.21.15.96:0/2512132712 conn(0x7f25cc074d90 msgr2=0x7f25cc0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:32:41.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 shutdown_connections 2024-04-22T15:32:41.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:32:41.170+0000 7f25d3d0e700 1 -- 172.21.15.96:0/2512132712 wait complete. 2024-04-22T15:32:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:41 smithi096 ceph-mon[112443]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:41 smithi096 ceph-mon[112443]: pgmap v324: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:41 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2512132712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:32:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:41 smithi196 ceph-mon[81815]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:41.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:41 smithi196 ceph-mon[81815]: pgmap v324: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:41.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:41 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2512132712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:32:42.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:42 smithi096 ceph-mon[112443]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:42.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:42 smithi196 ceph-mon[81815]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:32:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:43 smithi096 ceph-mon[112443]: pgmap v325: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:43 smithi196 ceph-mon[81815]: pgmap v325: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:45 smithi096 ceph-mon[112443]: pgmap v326: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:45.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:45 smithi196 ceph-mon[81815]: pgmap v326: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:47 smithi096 ceph-mon[112443]: pgmap v327: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:47 smithi196 ceph-mon[81815]: pgmap v327: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:49 smithi096 ceph-mon[112443]: pgmap v328: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:49 smithi196 ceph-mon[81815]: pgmap v328: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:51 smithi096 ceph-mon[112443]: pgmap v329: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:51 smithi196 ceph-mon[81815]: pgmap v329: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:53 smithi096 ceph-mon[112443]: pgmap v330: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:32:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:53 smithi196 ceph-mon[81815]: pgmap v330: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:32:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:55 smithi096 ceph-mon[112443]: pgmap v331: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:55 smithi196 ceph-mon[81815]: pgmap v331: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:57 smithi096 ceph-mon[112443]: pgmap v332: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:57 smithi196 ceph-mon[81815]: pgmap v332: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:32:59 smithi096 ceph-mon[112443]: pgmap v333: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:32:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:32:59 smithi196 ceph-mon[81815]: pgmap v333: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:01 smithi096 ceph-mon[112443]: pgmap v334: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:01 smithi196 ceph-mon[81815]: pgmap v334: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:03 smithi096 ceph-mon[112443]: pgmap v335: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:03 smithi196 ceph-mon[81815]: pgmap v335: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:05 smithi096 ceph-mon[112443]: pgmap v336: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:05 smithi196 ceph-mon[81815]: pgmap v336: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:07 smithi196 ceph-mon[81815]: pgmap v337: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:07 smithi096 ceph-mon[112443]: pgmap v337: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:08.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:33:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:33:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:09 smithi196 ceph-mon[81815]: pgmap v338: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:09 smithi096 ceph-mon[112443]: pgmap v338: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:11.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.287+0000 7f9574be3700 1 -- 172.21.15.96:0/4213354091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95700ffae0 msgr2=0x7f95700fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.287+0000 7f9574be3700 1 --2- 172.21.15.96:0/4213354091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95700ffae0 0x7f95700fff60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9558009910 tx=0x7f9558009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:11.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.287+0000 7f9574be3700 1 -- 172.21.15.96:0/4213354091 shutdown_connections 2024-04-22T15:33:11.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.287+0000 7f9574be3700 1 --2- 172.21.15.96:0/4213354091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95700ffae0 0x7f95700fff60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.287+0000 7f9574be3700 1 --2- 172.21.15.96:0/4213354091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9570105b70 0x7f9570105f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.287+0000 7f9574be3700 1 -- 172.21.15.96:0/4213354091 >> 172.21.15.96:0/4213354091 conn(0x7f95700fb610 msgr2=0x7f95700fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:11.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.287+0000 7f9574be3700 1 -- 172.21.15.96:0/4213354091 shutdown_connections 2024-04-22T15:33:11.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.287+0000 7f9574be3700 1 -- 172.21.15.96:0/4213354091 wait complete. 2024-04-22T15:33:11.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.288+0000 7f9574be3700 1 Processor -- start 2024-04-22T15:33:11.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.288+0000 7f9574be3700 1 -- start start 2024-04-22T15:33:11.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.288+0000 7f9574be3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95700ffae0 0x7f957019b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:11.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.288+0000 7f9574be3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9570105b70 0x7f957019bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:11.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.288+0000 7f9574be3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957019c2a0 con 0x7f9570105b70 2024-04-22T15:33:11.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.288+0000 7f9574be3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9570195860 con 0x7f95700ffae0 2024-04-22T15:33:11.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.288+0000 7f9565d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9570105b70 0x7f957019bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.288+0000 7f9565d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9570105b70 0x7f957019bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38370/0 (socket says 172.21.15.96:38370) 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9565d9b700 1 -- 172.21.15.96:0/873969522 learned_addr learned my addr 172.21.15.96:0/873969522 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f956e59c700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95700ffae0 0x7f957019b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9565d9b700 1 -- 172.21.15.96:0/873969522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95700ffae0 msgr2=0x7f957019b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9565d9b700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95700ffae0 0x7f957019b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9565d9b700 1 -- 172.21.15.96:0/873969522 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95580095f0 con 0x7f9570105b70 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9565d9b700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9570105b70 0x7f957019bc10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9558000c00 tx=0x7f955800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9567fff700 1 -- 172.21.15.96:0/873969522 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f955801d070 con 0x7f9570105b70 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9570195ae0 con 0x7f9570105b70 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9567fff700 1 -- 172.21.15.96:0/873969522 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95580056e0 con 0x7f9570105b70 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9567fff700 1 -- 172.21.15.96:0/873969522 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f955800fe40 con 0x7f9570105b70 2024-04-22T15:33:11.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.289+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9570195fd0 con 0x7f9570105b70 2024-04-22T15:33:11.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.290+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9570108100 con 0x7f9570105b70 2024-04-22T15:33:11.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.291+0000 7f9567fff700 1 -- 172.21.15.96:0/873969522 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9558017450 con 0x7f9570105b70 2024-04-22T15:33:11.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.291+0000 7f9567fff700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955c06ec70 0x7f955c071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:11.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.291+0000 7f9567fff700 1 -- 172.21.15.96:0/873969522 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9558090240 con 0x7f9570105b70 2024-04-22T15:33:11.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.291+0000 7f956e59c700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955c06ec70 0x7f955c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:11.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.292+0000 7f956e59c700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955c06ec70 0x7f955c071130 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9560005950 tx=0x7f956000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:11.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.296+0000 7f9567fff700 1 -- 172.21.15.96:0/873969522 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f955805cad0 con 0x7f9570105b70 2024-04-22T15:33:11.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.518+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9570196cd0 con 0x7f955c06ec70 2024-04-22T15:33:11.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.519+0000 7f9567fff700 1 -- 172.21.15.96:0/873969522 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9570196cd0 con 0x7f955c06ec70 2024-04-22T15:33:11.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.521+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955c06ec70 msgr2=0x7f955c071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.521+0000 7f9574be3700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955c06ec70 0x7f955c071130 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9560005950 tx=0x7f956000b410 comp rx=0 tx=0).stop 2024-04-22T15:33:11.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.521+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9570105b70 msgr2=0x7f957019bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.521+0000 7f9574be3700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9570105b70 0x7f957019bc10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9558000c00 tx=0x7f955800bd70 comp rx=0 tx=0).stop 2024-04-22T15:33:11.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.522+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 shutdown_connections 2024-04-22T15:33:11.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.522+0000 7f9574be3700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955c06ec70 0x7f955c071130 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.522+0000 7f9574be3700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95700ffae0 0x7f957019b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.522+0000 7f9574be3700 1 --2- 172.21.15.96:0/873969522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9570105b70 0x7f957019bc10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.522+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 >> 172.21.15.96:0/873969522 conn(0x7f95700fb610 msgr2=0x7f95700fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:11.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.522+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 shutdown_connections 2024-04-22T15:33:11.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.522+0000 7f9574be3700 1 -- 172.21.15.96:0/873969522 wait complete. 2024-04-22T15:33:11.535 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:33:11.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.626+0000 7f1d97dcf700 1 -- 172.21.15.96:0/3883609040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d90105aa0 msgr2=0x7f1d90105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.626+0000 7f1d97dcf700 1 --2- 172.21.15.96:0/3883609040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d90105aa0 0x7f1d90105e80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1d80009910 tx=0x7f1d80009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:11.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.626+0000 7f1d97dcf700 1 -- 172.21.15.96:0/3883609040 shutdown_connections 2024-04-22T15:33:11.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.626+0000 7f1d97dcf700 1 --2- 172.21.15.96:0/3883609040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d900ffaa0 0x7f1d900fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.626+0000 7f1d97dcf700 1 --2- 172.21.15.96:0/3883609040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d90105aa0 0x7f1d90105e80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.626+0000 7f1d97dcf700 1 -- 172.21.15.96:0/3883609040 >> 172.21.15.96:0/3883609040 conn(0x7f1d90076740 msgr2=0x7f1d900fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:11.630 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:11 smithi096 ceph-mon[112443]: pgmap v339: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:11.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.626+0000 7f1d97dcf700 1 -- 172.21.15.96:0/3883609040 shutdown_connections 2024-04-22T15:33:11.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.626+0000 7f1d97dcf700 1 -- 172.21.15.96:0/3883609040 wait complete. 2024-04-22T15:33:11.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d97dcf700 1 Processor -- start 2024-04-22T15:33:11.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d97dcf700 1 -- start start 2024-04-22T15:33:11.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d97dcf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d900ffaa0 0x7f1d900fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:11.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d97dcf700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d90105aa0 0x7f1d900fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d97dcf700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d900fc6a0 con 0x7f1d900ffaa0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d97dcf700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d900fc7e0 con 0x7f1d90105aa0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d95b6b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d900ffaa0 0x7f1d900fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d95b6b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d900ffaa0 0x7f1d900fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38382/0 (socket says 172.21.15.96:38382) 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d95b6b700 1 -- 172.21.15.96:0/1464428625 learned_addr learned my addr 172.21.15.96:0/1464428625 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d9536a700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d90105aa0 0x7f1d900fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d95b6b700 1 -- 172.21.15.96:0/1464428625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d90105aa0 msgr2=0x7f1d900fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d95b6b700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d90105aa0 0x7f1d900fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.627+0000 7f1d95b6b700 1 -- 172.21.15.96:0/1464428625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d800095f0 con 0x7f1d900ffaa0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.628+0000 7f1d95b6b700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d900ffaa0 0x7f1d900fbc20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000bd70 tx=0x7f1d8000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.628+0000 7f1d86ffd700 1 -- 172.21.15.96:0/1464428625 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d8001d070 con 0x7f1d900ffaa0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.628+0000 7f1d86ffd700 1 -- 172.21.15.96:0/1464428625 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d800056e0 con 0x7f1d900ffaa0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.628+0000 7f1d86ffd700 1 -- 172.21.15.96:0/1464428625 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d8000fe40 con 0x7f1d900ffaa0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.628+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d900fca60 con 0x7f1d900ffaa0 2024-04-22T15:33:11.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.628+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d901a2f10 con 0x7f1d900ffaa0 2024-04-22T15:33:11.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.629+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d901011e0 con 0x7f1d900ffaa0 2024-04-22T15:33:11.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.629+0000 7f1d86ffd700 1 -- 172.21.15.96:0/1464428625 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1d8000f680 con 0x7f1d900ffaa0 2024-04-22T15:33:11.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.630+0000 7f1d86ffd700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d7c06ec60 0x7f1d7c071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:11.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.630+0000 7f1d9536a700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d7c06ec60 0x7f1d7c071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:11.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.630+0000 7f1d86ffd700 1 -- 172.21.15.96:0/1464428625 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1d800902a0 con 0x7f1d900ffaa0 2024-04-22T15:33:11.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.630+0000 7f1d9536a700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d7c06ec60 0x7f1d7c071120 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1d900fd350 tx=0x7f1d8c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:11.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.635+0000 7f1d86ffd700 1 -- 172.21.15.96:0/1464428625 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1d80059000 con 0x7f1d900ffaa0 2024-04-22T15:33:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:11 smithi196 ceph-mon[81815]: pgmap v339: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:11.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.852+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d9010cca0 con 0x7f1d7c06ec60 2024-04-22T15:33:11.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.853+0000 7f1d86ffd700 1 -- 172.21.15.96:0/1464428625 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1d9010cca0 con 0x7f1d7c06ec60 2024-04-22T15:33:11.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d7c06ec60 msgr2=0x7f1d7c071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d7c06ec60 0x7f1d7c071120 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1d900fd350 tx=0x7f1d8c00b410 comp rx=0 tx=0).stop 2024-04-22T15:33:11.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d900ffaa0 msgr2=0x7f1d900fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d900ffaa0 0x7f1d900fbc20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000bd70 tx=0x7f1d8000bda0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 shutdown_connections 2024-04-22T15:33:11.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d7c06ec60 0x7f1d7c071120 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d900ffaa0 0x7f1d900fbc20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 --2- 172.21.15.96:0/1464428625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d90105aa0 0x7f1d900fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 >> 172.21.15.96:0/1464428625 conn(0x7f1d90076740 msgr2=0x7f1d900fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 shutdown_connections 2024-04-22T15:33:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.855+0000 7f1d97dcf700 1 -- 172.21.15.96:0/1464428625 wait complete. 2024-04-22T15:33:11.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.959+0000 7f7e08295700 1 -- 172.21.15.96:0/1206044993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00106c00 msgr2=0x7f7e00106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.959+0000 7f7e08295700 1 --2- 172.21.15.96:0/1206044993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00106c00 0x7f7e00106fe0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7df800b0a0 tx=0x7f7df800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.959+0000 7f7e08295700 1 -- 172.21.15.96:0/1206044993 shutdown_connections 2024-04-22T15:33:11.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.959+0000 7f7e08295700 1 --2- 172.21.15.96:0/1206044993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e00100b50 0x7f7e00100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.959+0000 7f7e08295700 1 --2- 172.21.15.96:0/1206044993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00106c00 0x7f7e00106fe0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.959+0000 7f7e08295700 1 -- 172.21.15.96:0/1206044993 >> 172.21.15.96:0/1206044993 conn(0x7f7e000fc640 msgr2=0x7f7e000fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.959+0000 7f7e08295700 1 -- 172.21.15.96:0/1206044993 shutdown_connections 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.959+0000 7f7e08295700 1 -- 172.21.15.96:0/1206044993 wait complete. 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e08295700 1 Processor -- start 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e08295700 1 -- start start 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e08295700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00100b50 0x7f7e0019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e08295700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e00106c00 0x7f7e0019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e08295700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0019d350 con 0x7f7e00100b50 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e08295700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00196980 con 0x7f7e00106c00 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e06031700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00100b50 0x7f7e0019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e06031700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00100b50 0x7f7e0019c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38396/0 (socket says 172.21.15.96:38396) 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e06031700 1 -- 172.21.15.96:0/2589463814 learned_addr learned my addr 172.21.15.96:0/2589463814 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e05830700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e00106c00 0x7f7e0019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:11.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e06031700 1 -- 172.21.15.96:0/2589463814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e00106c00 msgr2=0x7f7e0019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:11.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e06031700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e00106c00 0x7f7e0019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:11.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e06031700 1 -- 172.21.15.96:0/2589463814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df8009d00 con 0x7f7e00100b50 2024-04-22T15:33:11.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.960+0000 7f7e05830700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e00106c00 0x7f7e0019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:33:11.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.961+0000 7f7e06031700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00100b50 0x7f7e0019c7f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7df8000f80 tx=0x7f7df8007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:11.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.961+0000 7f7df77fe700 1 -- 172.21.15.96:0/2589463814 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7df800e040 con 0x7f7e00100b50 2024-04-22T15:33:11.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.961+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e00196c00 con 0x7f7e00100b50 2024-04-22T15:33:11.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.961+0000 7f7df77fe700 1 -- 172.21.15.96:0/2589463814 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7df8003de0 con 0x7f7e00100b50 2024-04-22T15:33:11.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.961+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e00197170 con 0x7f7e00100b50 2024-04-22T15:33:11.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.961+0000 7f7df77fe700 1 -- 172.21.15.96:0/2589463814 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7df801d720 con 0x7f7e00100b50 2024-04-22T15:33:11.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.962+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e00109030 con 0x7f7e00100b50 2024-04-22T15:33:11.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.963+0000 7f7df77fe700 1 -- 172.21.15.96:0/2589463814 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7df8004690 con 0x7f7e00100b50 2024-04-22T15:33:11.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.963+0000 7f7df77fe700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7dec06ecc0 0x7f7dec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:11.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.963+0000 7f7df77fe700 1 -- 172.21.15.96:0/2589463814 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7df8090680 con 0x7f7e00100b50 2024-04-22T15:33:11.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.963+0000 7f7e05830700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7dec06ecc0 0x7f7dec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:11.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.964+0000 7f7e05830700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7dec06ecc0 0x7f7dec071180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7e00197af0 tx=0x7f7df0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:11.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:11.968+0000 7f7df77fe700 1 -- 172.21.15.96:0/2589463814 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7df805cf10 con 0x7f7e00100b50 2024-04-22T15:33:12.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.175+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e00197e60 con 0x7f7dec06ecc0 2024-04-22T15:33:12.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.185+0000 7f7df77fe700 1 -- 172.21.15.96:0/2589463814 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7e00197e60 con 0x7f7dec06ecc0 2024-04-22T15:33:12.189 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:33:12.189 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (11m) 4m ago 23m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:33:12.189 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (23m) 4m ago 23m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:33:12.189 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (22m) 4m ago 22m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:33:12.189 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (9m) 4m ago 23m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:33:12.189 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (8m) 4m ago 22m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:33:12.189 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (11m) 4m ago 22m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:33:12.189 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (5m) 4m ago 17m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (5m) 4m ago 17m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (4m) 4m ago 17m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (17m) 4m ago 17m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (14m) 4m ago 24m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (13m) 4m ago 22m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (10m) 4m ago 24m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (9m) 4m ago 22m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (12m) 4m ago 23m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (12m) 4m ago 22m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (8m) 4m ago 21m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (8m) 4m ago 20m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (7m) 4m ago 20m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (6m) 4m ago 19m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:33:12.190 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (6m) 4m ago 19m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:33:12.191 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (5m) 4m ago 19m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:33:12.191 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (12m) 4m ago 22m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:33:12.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7dec06ecc0 msgr2=0x7f7dec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7dec06ecc0 0x7f7dec071180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7e00197af0 tx=0x7f7df0008040 comp rx=0 tx=0).stop 2024-04-22T15:33:12.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00100b50 msgr2=0x7f7e0019c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00100b50 0x7f7e0019c7f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7df8000f80 tx=0x7f7df8007bd0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 shutdown_connections 2024-04-22T15:33:12.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7dec06ecc0 0x7f7dec071180 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e00100b50 0x7f7e0019c7f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 --2- 172.21.15.96:0/2589463814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e00106c00 0x7f7e0019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 >> 172.21.15.96:0/2589463814 conn(0x7f7e000fc640 msgr2=0x7f7e00105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:12.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 shutdown_connections 2024-04-22T15:33:12.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.188+0000 7f7e08295700 1 -- 172.21.15.96:0/2589463814 wait complete. 2024-04-22T15:33:12.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.290+0000 7f19d602d700 1 -- 172.21.15.96:0/2679258121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d00ff920 msgr2=0x7f19d00ffd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.290+0000 7f19d602d700 1 --2- 172.21.15.96:0/2679258121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d00ff920 0x7f19d00ffd00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f19c0009910 tx=0x7f19c0009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:12.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.290+0000 7f19d602d700 1 -- 172.21.15.96:0/2679258121 shutdown_connections 2024-04-22T15:33:12.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.290+0000 7f19d602d700 1 --2- 172.21.15.96:0/2679258121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d01002d0 0x7f19d010cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.290+0000 7f19d602d700 1 --2- 172.21.15.96:0/2679258121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d00ff920 0x7f19d00ffd00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.290+0000 7f19d602d700 1 -- 172.21.15.96:0/2679258121 >> 172.21.15.96:0/2679258121 conn(0x7f19d00fb5e0 msgr2=0x7f19d00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:12.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.290+0000 7f19d602d700 1 -- 172.21.15.96:0/2679258121 shutdown_connections 2024-04-22T15:33:12.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.290+0000 7f19d602d700 1 -- 172.21.15.96:0/2679258121 wait complete. 2024-04-22T15:33:12.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.290+0000 7f19d602d700 1 Processor -- start 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19d602d700 1 -- start start 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19d602d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d00ff920 0x7f19d019e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19d602d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d01002d0 0x7f19d019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19d602d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d019f540 con 0x7f19d01002d0 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19d602d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d0198b70 con 0x7f19d00ff920 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19ceffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d01002d0 0x7f19d019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19ceffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d01002d0 0x7f19d019ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38414/0 (socket says 172.21.15.96:38414) 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19ceffd700 1 -- 172.21.15.96:0/1063108248 learned_addr learned my addr 172.21.15.96:0/1063108248 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19ceffd700 1 -- 172.21.15.96:0/1063108248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d00ff920 msgr2=0x7f19d019e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19ceffd700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d00ff920 0x7f19d019e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19ceffd700 1 -- 172.21.15.96:0/1063108248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c00095f0 con 0x7f19d01002d0 2024-04-22T15:33:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19ceffd700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d01002d0 0x7f19d019ef20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f19c400d900 tx=0x7f19c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:12.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.291+0000 7f19ccff9700 1 -- 172.21.15.96:0/1063108248 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c40041f0 con 0x7f19d01002d0 2024-04-22T15:33:12.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.292+0000 7f19ccff9700 1 -- 172.21.15.96:0/1063108248 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19c4004cf0 con 0x7f19d01002d0 2024-04-22T15:33:12.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.292+0000 7f19ccff9700 1 -- 172.21.15.96:0/1063108248 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c400b750 con 0x7f19d01002d0 2024-04-22T15:33:12.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.292+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d0198e50 con 0x7f19d01002d0 2024-04-22T15:33:12.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.292+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d01993a0 con 0x7f19d01002d0 2024-04-22T15:33:12.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.293+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d010ae10 con 0x7f19d01002d0 2024-04-22T15:33:12.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.293+0000 7f19ccff9700 1 -- 172.21.15.96:0/1063108248 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f19c400bac0 con 0x7f19d01002d0 2024-04-22T15:33:12.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.293+0000 7f19ccff9700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19b806ecc0 0x7f19b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:12.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.294+0000 7f19ccff9700 1 -- 172.21.15.96:0/1063108248 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f19c408ec80 con 0x7f19d01002d0 2024-04-22T15:33:12.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.294+0000 7f19cf7fe700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19b806ecc0 0x7f19b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:12.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.294+0000 7f19cf7fe700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19b806ecc0 0x7f19b8071180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f19c000b5c0 tx=0x7f19c0005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:12.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.298+0000 7f19ccff9700 1 -- 172.21.15.96:0/1063108248 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f19c405b590 con 0x7f19d01002d0 2024-04-22T15:33:12.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.583+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f19d0199fe0 con 0x7f19d01002d0 2024-04-22T15:33:12.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.584+0000 7f19ccff9700 1 -- 172.21.15.96:0/1063108248 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f19c4016360 con 0x7f19d01002d0 2024-04-22T15:33:12.587 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:33:12.587 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:33:12.588 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:33:12.589 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:33:12.589 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:33:12.589 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:33:12.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19b806ecc0 msgr2=0x7f19b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19b806ecc0 0x7f19b8071180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f19c000b5c0 tx=0x7f19c0005960 comp rx=0 tx=0).stop 2024-04-22T15:33:12.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d01002d0 msgr2=0x7f19d019ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d01002d0 0x7f19d019ef20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f19c400d900 tx=0x7f19c400dcc0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 shutdown_connections 2024-04-22T15:33:12.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19b806ecc0 0x7f19b8071180 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d00ff920 0x7f19d019e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 --2- 172.21.15.96:0/1063108248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d01002d0 0x7f19d019ef20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 >> 172.21.15.96:0/1063108248 conn(0x7f19d00fb5e0 msgr2=0x7f19d0107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:12.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 shutdown_connections 2024-04-22T15:33:12.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.586+0000 7f19d602d700 1 -- 172.21.15.96:0/1063108248 wait complete. 2024-04-22T15:33:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.690+0000 7f8387501700 1 -- 172.21.15.96:0/4043464419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 msgr2=0x7f83800fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.690+0000 7f8387501700 1 --2- 172.21.15.96:0/4043464419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 0x7f83800fff20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8370009910 tx=0x7f8370009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.690+0000 7f8387501700 1 -- 172.21.15.96:0/4043464419 shutdown_connections 2024-04-22T15:33:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.690+0000 7f8387501700 1 --2- 172.21.15.96:0/4043464419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 0x7f83800fff20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.690+0000 7f8387501700 1 --2- 172.21.15.96:0/4043464419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8380105aa0 0x7f8380105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.690+0000 7f8387501700 1 -- 172.21.15.96:0/4043464419 >> 172.21.15.96:0/4043464419 conn(0x7f8380076740 msgr2=0x7f83800fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.690+0000 7f8387501700 1 -- 172.21.15.96:0/4043464419 shutdown_connections 2024-04-22T15:33:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.690+0000 7f8387501700 1 -- 172.21.15.96:0/4043464419 wait complete. 2024-04-22T15:33:12.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.691+0000 7f8387501700 1 Processor -- start 2024-04-22T15:33:12.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.691+0000 7f8387501700 1 -- start start 2024-04-22T15:33:12.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.691+0000 7f8387501700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 0x7f8380197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:12.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.691+0000 7f8387501700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8380105aa0 0x7f8380197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.691+0000 7f8387501700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8380198410 con 0x7f83800ffaa0 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.691+0000 7f8387501700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8380198550 con 0x7f8380105aa0 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.691+0000 7f838529d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 0x7f8380197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.691+0000 7f838529d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 0x7f8380197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38442/0 (socket says 172.21.15.96:38442) 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.691+0000 7f838529d700 1 -- 172.21.15.96:0/200839809 learned_addr learned my addr 172.21.15.96:0/200839809 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f838529d700 1 -- 172.21.15.96:0/200839809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8380105aa0 msgr2=0x7f8380197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f8384a9c700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8380105aa0 0x7f8380197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f838529d700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8380105aa0 0x7f8380197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f838529d700 1 -- 172.21.15.96:0/200839809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83700095f0 con 0x7f83800ffaa0 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f8384a9c700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8380105aa0 0x7f8380197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f838529d700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 0x7f8380197900 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f837c00d8d0 tx=0x7f837c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f83767fc700 1 -- 172.21.15.96:0/200839809 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f837c009940 con 0x7f83800ffaa0 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f83767fc700 1 -- 172.21.15.96:0/200839809 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f837c010460 con 0x7f83800ffaa0 2024-04-22T15:33:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83801a0970 con 0x7f83800ffaa0 2024-04-22T15:33:12.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f83767fc700 1 -- 172.21.15.96:0/200839809 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f837c00f5f0 con 0x7f83800ffaa0 2024-04-22T15:33:12.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.692+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83801a0d80 con 0x7f83800ffaa0 2024-04-22T15:33:12.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.693+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83801080c0 con 0x7f83800ffaa0 2024-04-22T15:33:12.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.694+0000 7f83767fc700 1 -- 172.21.15.96:0/200839809 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f837c0105d0 con 0x7f83800ffaa0 2024-04-22T15:33:12.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.694+0000 7f83767fc700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f836c06ecd0 0x7f836c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:12.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.694+0000 7f83767fc700 1 -- 172.21.15.96:0/200839809 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f837c08edc0 con 0x7f83800ffaa0 2024-04-22T15:33:12.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.694+0000 7f8384a9c700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f836c06ecd0 0x7f836c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:12.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.694+0000 7f8384a9c700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f836c06ecd0 0x7f836c071190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8370006010 tx=0x7f837000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:12.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.699+0000 7f83767fc700 1 -- 172.21.15.96:0/200839809 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f837c057ba0 con 0x7f83800ffaa0 2024-04-22T15:33:12.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.942+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f83801011e0 con 0x7f83800ffaa0 2024-04-22T15:33:12.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.943+0000 7f83767fc700 1 -- 172.21.15.96:0/200839809 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f837c016770 con 0x7f83800ffaa0 2024-04-22T15:33:12.946 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:33:12.946 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:33:12.947 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:33:12.948 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:33:12.949 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:33:12.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f836c06ecd0 msgr2=0x7f836c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f836c06ecd0 0x7f836c071190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8370006010 tx=0x7f837000b540 comp rx=0 tx=0).stop 2024-04-22T15:33:12.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 msgr2=0x7f8380197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:12.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 0x7f8380197900 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f837c00d8d0 tx=0x7f837c00dc90 comp rx=0 tx=0).stop 2024-04-22T15:33:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 shutdown_connections 2024-04-22T15:33:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f836c06ecd0 0x7f836c071190 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83800ffaa0 0x7f8380197900 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 --2- 172.21.15.96:0/200839809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8380105aa0 0x7f8380197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 >> 172.21.15.96:0/200839809 conn(0x7f8380076740 msgr2=0x7f83800fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 shutdown_connections 2024-04-22T15:33:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:12.945+0000 7f8387501700 1 -- 172.21.15.96:0/200839809 wait complete. 2024-04-22T15:33:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:33:13.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.049+0000 7f512f99d700 1 -- 172.21.15.96:0/213339530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51280ffad0 msgr2=0x7f51280fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:13.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.049+0000 7f512f99d700 1 --2- 172.21.15.96:0/213339530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51280ffad0 0x7f51280fff50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5118009910 tx=0x7f5118009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:13.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.049+0000 7f512f99d700 1 -- 172.21.15.96:0/213339530 shutdown_connections 2024-04-22T15:33:13.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.049+0000 7f512f99d700 1 --2- 172.21.15.96:0/213339530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51280ffad0 0x7f51280fff50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.049+0000 7f512f99d700 1 --2- 172.21.15.96:0/213339530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5128105b60 0x7f5128105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.049+0000 7f512f99d700 1 -- 172.21.15.96:0/213339530 >> 172.21.15.96:0/213339530 conn(0x7f51280fb5e0 msgr2=0x7f51280fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:13.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512f99d700 1 -- 172.21.15.96:0/213339530 shutdown_connections 2024-04-22T15:33:13.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512f99d700 1 -- 172.21.15.96:0/213339530 wait complete. 2024-04-22T15:33:13.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512f99d700 1 Processor -- start 2024-04-22T15:33:13.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512f99d700 1 -- start start 2024-04-22T15:33:13.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512f99d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51280ffad0 0x7f5128197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:13.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512f99d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5128105b60 0x7f5128197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:13.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512f99d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51281985a0 con 0x7f5128105b60 2024-04-22T15:33:13.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512f99d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51281986e0 con 0x7f51280ffad0 2024-04-22T15:33:13.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512cf38700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5128105b60 0x7f5128197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512cf38700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5128105b60 0x7f5128197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38460/0 (socket says 172.21.15.96:38460) 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.050+0000 7f512cf38700 1 -- 172.21.15.96:0/1126484767 learned_addr learned my addr 172.21.15.96:0/1126484767 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.051+0000 7f512cf38700 1 -- 172.21.15.96:0/1126484767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51280ffad0 msgr2=0x7f5128197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.051+0000 7f512cf38700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51280ffad0 0x7f5128197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.051+0000 7f512cf38700 1 -- 172.21.15.96:0/1126484767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51180095f0 con 0x7f5128105b60 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.051+0000 7f512cf38700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5128105b60 0x7f5128197fd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f511800bd70 tx=0x7f511800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.051+0000 7f511e7fc700 1 -- 172.21.15.96:0/1126484767 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f511801d070 con 0x7f5128105b60 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.051+0000 7f511e7fc700 1 -- 172.21.15.96:0/1126484767 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f511800f460 con 0x7f5128105b60 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.051+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51281a0b00 con 0x7f5128105b60 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.051+0000 7f511e7fc700 1 -- 172.21.15.96:0/1126484767 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5118017630 con 0x7f5128105b60 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.051+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51281a0e60 con 0x7f5128105b60 2024-04-22T15:33:13.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.052+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51281080f0 con 0x7f5128105b60 2024-04-22T15:33:13.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.052+0000 7f511e7fc700 1 -- 172.21.15.96:0/1126484767 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f511800f5d0 con 0x7f5128105b60 2024-04-22T15:33:13.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.053+0000 7f511e7fc700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f511406ecc0 0x7f5114071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:13.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.053+0000 7f511e7fc700 1 -- 172.21.15.96:0/1126484767 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5118090400 con 0x7f5128105b60 2024-04-22T15:33:13.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.053+0000 7f512d739700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f511406ecc0 0x7f5114071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:13.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.053+0000 7f512d739700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f511406ecc0 0x7f5114071180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f51240097b0 tx=0x7f5124006d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:13.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.057+0000 7f511e7fc700 1 -- 172.21.15.96:0/1126484767 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5118059160 con 0x7f5128105b60 2024-04-22T15:33:13.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.277+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f512810cd50 con 0x7f511406ecc0 2024-04-22T15:33:13.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.278+0000 7f511e7fc700 1 -- 172.21.15.96:0/1126484767 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f512810cd50 con 0x7f511406ecc0 2024-04-22T15:33:13.281 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:33:13.281 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:33:13.282 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:33:13.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.280+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f511406ecc0 msgr2=0x7f5114071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:13.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.280+0000 7f512f99d700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f511406ecc0 0x7f5114071180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f51240097b0 tx=0x7f5124006d20 comp rx=0 tx=0).stop 2024-04-22T15:33:13.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.280+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5128105b60 msgr2=0x7f5128197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:13.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.280+0000 7f512f99d700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5128105b60 0x7f5128197fd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f511800bd70 tx=0x7f511800be50 comp rx=0 tx=0).stop 2024-04-22T15:33:13.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.280+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 shutdown_connections 2024-04-22T15:33:13.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.280+0000 7f512f99d700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f511406ecc0 0x7f5114071180 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.280+0000 7f512f99d700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51280ffad0 0x7f5128197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.280+0000 7f512f99d700 1 --2- 172.21.15.96:0/1126484767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5128105b60 0x7f5128197fd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.280+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 >> 172.21.15.96:0/1126484767 conn(0x7f51280fb5e0 msgr2=0x7f51280fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:13.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.281+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 shutdown_connections 2024-04-22T15:33:13.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.281+0000 7f512f99d700 1 -- 172.21.15.96:0/1126484767 wait complete. 2024-04-22T15:33:13.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.383+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3183603792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d081049e0 msgr2=0x7f2d08104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:13.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.383+0000 7f2d0ead9700 1 --2- 172.21.15.96:0/3183603792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d081049e0 0x7f2d08104dc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009910 tx=0x7f2cfc009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:13.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.383+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3183603792 shutdown_connections 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.383+0000 7f2d0ead9700 1 --2- 172.21.15.96:0/3183603792 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d08105390 0x7f2d080ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.383+0000 7f2d0ead9700 1 --2- 172.21.15.96:0/3183603792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d081049e0 0x7f2d08104dc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.383+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3183603792 >> 172.21.15.96:0/3183603792 conn(0x7f2d08074d50 msgr2=0x7f2d08075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.383+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3183603792 shutdown_connections 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.383+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3183603792 wait complete. 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d0ead9700 1 Processor -- start 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d0ead9700 1 -- start start 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d0ead9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d081049e0 0x7f2d0819d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d0ead9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d08105390 0x7f2d0819d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d0ead9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0819ddd0 con 0x7f2d08105390 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d0ead9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081a1aa0 con 0x7f2d081049e0 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d07fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d08105390 0x7f2d0819d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d07fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d08105390 0x7f2d0819d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38472/0 (socket says 172.21.15.96:38472) 2024-04-22T15:33:13.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d07fff700 1 -- 172.21.15.96:0/3567850200 learned_addr learned my addr 172.21.15.96:0/3567850200 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.384+0000 7f2d0c875700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d081049e0 0x7f2d0819d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d07fff700 1 -- 172.21.15.96:0/3567850200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d081049e0 msgr2=0x7f2d0819d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d07fff700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d081049e0 0x7f2d0819d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d07fff700 1 -- 172.21.15.96:0/3567850200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc0095f0 con 0x7f2d08105390 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d0c875700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d081049e0 0x7f2d0819d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d07fff700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d08105390 0x7f2d0819d6f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400c8a0 tx=0x7f2cf400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d05ffb700 1 -- 172.21.15.96:0/3567850200 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cf400cea0 con 0x7f2d08105390 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d05ffb700 1 -- 172.21.15.96:0/3567850200 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cf4004830 con 0x7f2d08105390 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d081a1d80 con 0x7f2d08105390 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d05ffb700 1 -- 172.21.15.96:0/3567850200 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cf40055b0 con 0x7f2d08105390 2024-04-22T15:33:13.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.385+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d081a22d0 con 0x7f2d08105390 2024-04-22T15:33:13.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.386+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d08110b20 con 0x7f2d08105390 2024-04-22T15:33:13.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.387+0000 7f2d05ffb700 1 -- 172.21.15.96:0/3567850200 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2cf40049a0 con 0x7f2d08105390 2024-04-22T15:33:13.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.387+0000 7f2d05ffb700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2cf806ecc0 0x7f2cf8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:13.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.387+0000 7f2d05ffb700 1 -- 172.21.15.96:0/3567850200 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2cf408eaa0 con 0x7f2d08105390 2024-04-22T15:33:13.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.387+0000 7f2d0c875700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2cf806ecc0 0x7f2cf8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:13.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.387+0000 7f2d0c875700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2cf806ecc0 0x7f2cf8071180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0059d0 tx=0x7f2cfc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:13.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.392+0000 7f2d05ffb700 1 -- 172.21.15.96:0/3567850200 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2cf4057880 con 0x7f2d08105390 2024-04-22T15:33:13.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.675+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2d0819e4c0 con 0x7f2d08105390 2024-04-22T15:33:13.679 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:13 smithi096 ceph-mon[112443]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:13.679 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:13 smithi096 ceph-mon[112443]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:13.679 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:13 smithi096 ceph-mon[112443]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:13.679 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:13 smithi096 ceph-mon[112443]: pgmap v340: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:13.679 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:13 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1063108248' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:33:13.679 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:13 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/200839809' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:33:13.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.675+0000 7f2d05ffb700 1 -- 172.21.15.96:0/3567850200 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f2cf401d090 con 0x7f2d08105390 2024-04-22T15:33:13.680 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:33:13.680 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:33:13.680 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:33:13.680 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:33:13.680 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:33:13.680 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:33:13.680 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:33:13.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2cf806ecc0 msgr2=0x7f2cf8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:13.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2cf806ecc0 0x7f2cf8071180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0059d0 tx=0x7f2cfc01a040 comp rx=0 tx=0).stop 2024-04-22T15:33:13.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d08105390 msgr2=0x7f2d0819d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:13.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d08105390 0x7f2d0819d6f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400c8a0 tx=0x7f2cf400cc60 comp rx=0 tx=0).stop 2024-04-22T15:33:13.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 shutdown_connections 2024-04-22T15:33:13.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2cf806ecc0 0x7f2cf8071180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d081049e0 0x7f2d0819d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 --2- 172.21.15.96:0/3567850200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d08105390 0x7f2d0819d6f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:13.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 >> 172.21.15.96:0/3567850200 conn(0x7f2d08074d50 msgr2=0x7f2d08101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:13.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 shutdown_connections 2024-04-22T15:33:13.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:13.678+0000 7f2d0ead9700 1 -- 172.21.15.96:0/3567850200 wait complete. 2024-04-22T15:33:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:13 smithi196 ceph-mon[81815]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:13.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:13 smithi196 ceph-mon[81815]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:13.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:13 smithi196 ceph-mon[81815]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:13.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:13 smithi196 ceph-mon[81815]: pgmap v340: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:13.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:13 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1063108248' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:33:13.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:13 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/200839809' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:33:14.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:14 smithi196 ceph-mon[81815]: from='client.34696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:14.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:14 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3567850200' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:33:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:14 smithi096 ceph-mon[112443]: from='client.34696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:14 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3567850200' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:33:15.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:15 smithi096 ceph-mon[112443]: pgmap v341: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:15 smithi196 ceph-mon[81815]: pgmap v341: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:16.911 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:16 smithi196 ceph-mon[81815]: pgmap v342: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:16.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:16 smithi096 ceph-mon[112443]: pgmap v342: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:19.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:19 smithi096 ceph-mon[112443]: pgmap v343: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:19 smithi196 ceph-mon[81815]: pgmap v343: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:21 smithi096 ceph-mon[112443]: pgmap v344: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:21.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:21 smithi196 ceph-mon[81815]: pgmap v344: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:23 smithi096 ceph-mon[112443]: pgmap v345: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:33:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:23 smithi196 ceph-mon[81815]: pgmap v345: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:33:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:25 smithi096 ceph-mon[112443]: pgmap v346: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:25 smithi196 ceph-mon[81815]: pgmap v346: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:27.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:27 smithi096 ceph-mon[112443]: pgmap v347: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:27 smithi196 ceph-mon[81815]: pgmap v347: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:29.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:29 smithi096 ceph-mon[112443]: pgmap v348: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:29.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:29 smithi196 ceph-mon[81815]: pgmap v348: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:31.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:31 smithi096 ceph-mon[112443]: pgmap v349: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:31 smithi196 ceph-mon[81815]: pgmap v349: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:33 smithi096 ceph-mon[112443]: pgmap v350: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:33 smithi196 ceph-mon[81815]: pgmap v350: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:35.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:35 smithi096 ceph-mon[112443]: pgmap v351: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:35 smithi196 ceph-mon[81815]: pgmap v351: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:37 smithi096 ceph-mon[112443]: pgmap v352: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:37 smithi196 ceph-mon[81815]: pgmap v352: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:33:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:33:39.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:39 smithi096 ceph-mon[112443]: pgmap v353: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:39 smithi196 ceph-mon[81815]: pgmap v353: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:41.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:41 smithi096 ceph-mon[112443]: pgmap v354: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:41.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:41 smithi196 ceph-mon[81815]: pgmap v354: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:43 smithi096 ceph-mon[112443]: pgmap v355: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:43 smithi196 ceph-mon[81815]: pgmap v355: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:43.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.797+0000 7f3a1494f700 1 -- 172.21.15.96:0/2302227891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a10100df0 msgr2=0x7f3a101011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:43.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.797+0000 7f3a1494f700 1 --2- 172.21.15.96:0/2302227891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a10100df0 0x7f3a101011d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009910 tx=0x7f3a00009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:43.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.797+0000 7f3a1494f700 1 -- 172.21.15.96:0/2302227891 shutdown_connections 2024-04-22T15:33:43.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.797+0000 7f3a1494f700 1 --2- 172.21.15.96:0/2302227891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a101017a0 0x7f3a101057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:43.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.797+0000 7f3a1494f700 1 --2- 172.21.15.96:0/2302227891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a10100df0 0x7f3a101011d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:43.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.797+0000 7f3a1494f700 1 -- 172.21.15.96:0/2302227891 >> 172.21.15.96:0/2302227891 conn(0x7f3a100fc6a0 msgr2=0x7f3a100feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:43.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.797+0000 7f3a1494f700 1 -- 172.21.15.96:0/2302227891 shutdown_connections 2024-04-22T15:33:43.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.797+0000 7f3a1494f700 1 -- 172.21.15.96:0/2302227891 wait complete. 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a1494f700 1 Processor -- start 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a1494f700 1 -- start start 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a1494f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a10100df0 0x7f3a1019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a1494f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a101017a0 0x7f3a1019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a1494f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1019d390 con 0x7f3a101017a0 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a1494f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a10196970 con 0x7f3a10100df0 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a0dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a101017a0 0x7f3a1019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a0dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a101017a0 0x7f3a1019cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58306/0 (socket says 172.21.15.96:58306) 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a0dd9b700 1 -- 172.21.15.96:0/3794544188 learned_addr learned my addr 172.21.15.96:0/3794544188 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a0e59c700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a10100df0 0x7f3a1019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:43.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a0dd9b700 1 -- 172.21.15.96:0/3794544188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a10100df0 msgr2=0x7f3a1019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:43.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a0dd9b700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a10100df0 0x7f3a1019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:43.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.798+0000 7f3a0dd9b700 1 -- 172.21.15.96:0/3794544188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a000095f0 con 0x7f3a101017a0 2024-04-22T15:33:43.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.799+0000 7f3a0dd9b700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a101017a0 0x7f3a1019cd70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c8f0 tx=0x7f3a0400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:43.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.799+0000 7f39ff7fe700 1 -- 172.21.15.96:0/3794544188 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a040078a0 con 0x7f3a101017a0 2024-04-22T15:33:43.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.799+0000 7f39ff7fe700 1 -- 172.21.15.96:0/3794544188 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a0400f450 con 0x7f3a101017a0 2024-04-22T15:33:43.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.799+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a10196c50 con 0x7f3a101017a0 2024-04-22T15:33:43.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.799+0000 7f39ff7fe700 1 -- 172.21.15.96:0/3794544188 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a04018610 con 0x7f3a101017a0 2024-04-22T15:33:43.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.799+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a101971a0 con 0x7f3a101017a0 2024-04-22T15:33:43.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.800+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a10109190 con 0x7f3a101017a0 2024-04-22T15:33:43.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.800+0000 7f39ff7fe700 1 -- 172.21.15.96:0/3794544188 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3a0400f5c0 con 0x7f3a101017a0 2024-04-22T15:33:43.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.801+0000 7f39ff7fe700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39f806ecd0 0x7f39f8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:43.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.801+0000 7f3a0e59c700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39f806ecd0 0x7f39f8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:43.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.801+0000 7f39ff7fe700 1 -- 172.21.15.96:0/3794544188 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3a0408ed90 con 0x7f3a101017a0 2024-04-22T15:33:43.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.801+0000 7f3a0e59c700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39f806ecd0 0x7f39f8071190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a00006010 tx=0x7f3a0001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:43.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:43.805+0000 7f39ff7fe700 1 -- 172.21.15.96:0/3794544188 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3a0405b620 con 0x7f3a101017a0 2024-04-22T15:33:44.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.023+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a1010ddd0 con 0x7f39f806ecd0 2024-04-22T15:33:44.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.024+0000 7f39ff7fe700 1 -- 172.21.15.96:0/3794544188 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3a1010ddd0 con 0x7f39f806ecd0 2024-04-22T15:33:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39f806ecd0 msgr2=0x7f39f8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39f806ecd0 0x7f39f8071190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a00006010 tx=0x7f3a0001a040 comp rx=0 tx=0).stop 2024-04-22T15:33:44.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a101017a0 msgr2=0x7f3a1019cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a101017a0 0x7f3a1019cd70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c8f0 tx=0x7f3a0400cc00 comp rx=0 tx=0).stop 2024-04-22T15:33:44.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 shutdown_connections 2024-04-22T15:33:44.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39f806ecd0 0x7f39f8071190 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a10100df0 0x7f3a1019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 --2- 172.21.15.96:0/3794544188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a101017a0 0x7f3a1019cd70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 >> 172.21.15.96:0/3794544188 conn(0x7f3a100fc6a0 msgr2=0x7f3a100fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:44.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 shutdown_connections 2024-04-22T15:33:44.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.026+0000 7f3a1494f700 1 -- 172.21.15.96:0/3794544188 wait complete. 2024-04-22T15:33:44.040 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:33:44.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 -- 172.21.15.96:0/3753434263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2688105c00 msgr2=0x7f2688105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 --2- 172.21.15.96:0/3753434263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2688105c00 0x7f2688105fe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2684009910 tx=0x7f2684009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:44.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 -- 172.21.15.96:0/3753434263 shutdown_connections 2024-04-22T15:33:44.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 --2- 172.21.15.96:0/3753434263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26880ffb50 0x7f26880fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 --2- 172.21.15.96:0/3753434263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2688105c00 0x7f2688105fe0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 -- 172.21.15.96:0/3753434263 >> 172.21.15.96:0/3753434263 conn(0x7f26880fb680 msgr2=0x7f26880fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:44.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 -- 172.21.15.96:0/3753434263 shutdown_connections 2024-04-22T15:33:44.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 -- 172.21.15.96:0/3753434263 wait complete. 2024-04-22T15:33:44.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 Processor -- start 2024-04-22T15:33:44.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 -- start start 2024-04-22T15:33:44.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26880ffb50 0x7f268819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:44.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2688105c00 0x7f268819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2688198b00 con 0x7f26880ffb50 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.132+0000 7f268fd91700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2688198c70 con 0x7f2688105c00 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268db2d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26880ffb50 0x7f268819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268db2d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26880ffb50 0x7f268819ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58318/0 (socket says 172.21.15.96:58318) 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268db2d700 1 -- 172.21.15.96:0/736060394 learned_addr learned my addr 172.21.15.96:0/736060394 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268d32c700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2688105c00 0x7f268819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268db2d700 1 -- 172.21.15.96:0/736060394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2688105c00 msgr2=0x7f268819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268db2d700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2688105c00 0x7f268819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268db2d700 1 -- 172.21.15.96:0/736060394 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26840095f0 con 0x7f26880ffb50 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268d32c700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2688105c00 0x7f268819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268db2d700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26880ffb50 0x7f268819ea50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f268400bd70 tx=0x7f268400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f267effd700 1 -- 172.21.15.96:0/736060394 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268401d070 con 0x7f26880ffb50 2024-04-22T15:33:44.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2688198ef0 con 0x7f26880ffb50 2024-04-22T15:33:44.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.133+0000 7f267effd700 1 -- 172.21.15.96:0/736060394 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f268400f460 con 0x7f26880ffb50 2024-04-22T15:33:44.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.134+0000 7f267effd700 1 -- 172.21.15.96:0/736060394 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2684017630 con 0x7f26880ffb50 2024-04-22T15:33:44.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.134+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26881993e0 con 0x7f26880ffb50 2024-04-22T15:33:44.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.135+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2688101200 con 0x7f26880ffb50 2024-04-22T15:33:44.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.135+0000 7f267effd700 1 -- 172.21.15.96:0/736060394 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2684003a60 con 0x7f26880ffb50 2024-04-22T15:33:44.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.135+0000 7f267effd700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f267406ecc0 0x7f2674071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:44.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.135+0000 7f267effd700 1 -- 172.21.15.96:0/736060394 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f268408fd50 con 0x7f26880ffb50 2024-04-22T15:33:44.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.135+0000 7f268d32c700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f267406ecc0 0x7f2674071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:44.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.136+0000 7f268d32c700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f267406ecc0 0x7f2674071180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f268819a270 tx=0x7f2678005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:44.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.140+0000 7f267effd700 1 -- 172.21.15.96:0/736060394 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2684058a20 con 0x7f26880ffb50 2024-04-22T15:33:44.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.358+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f268810cd40 con 0x7f267406ecc0 2024-04-22T15:33:44.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.359+0000 7f267effd700 1 -- 172.21.15.96:0/736060394 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f268810cd40 con 0x7f267406ecc0 2024-04-22T15:33:44.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f267406ecc0 msgr2=0x7f2674071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f267406ecc0 0x7f2674071180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f268819a270 tx=0x7f2678005f50 comp rx=0 tx=0).stop 2024-04-22T15:33:44.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26880ffb50 msgr2=0x7f268819ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26880ffb50 0x7f268819ea50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f268400bd70 tx=0x7f268400be50 comp rx=0 tx=0).stop 2024-04-22T15:33:44.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 shutdown_connections 2024-04-22T15:33:44.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f267406ecc0 0x7f2674071180 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26880ffb50 0x7f268819ea50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 --2- 172.21.15.96:0/736060394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2688105c00 0x7f268819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 >> 172.21.15.96:0/736060394 conn(0x7f26880fb680 msgr2=0x7f2688069470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:44.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 shutdown_connections 2024-04-22T15:33:44.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.362+0000 7f268fd91700 1 -- 172.21.15.96:0/736060394 wait complete. 2024-04-22T15:33:44.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.466+0000 7fb79776f700 1 -- 172.21.15.96:0/2491551538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 msgr2=0x7fb790102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.466+0000 7fb79776f700 1 --2- 172.21.15.96:0/2491551538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 0x7fb790102270 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009910 tx=0x7fb78c009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:44.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.466+0000 7fb79776f700 1 -- 172.21.15.96:0/2491551538 shutdown_connections 2024-04-22T15:33:44.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.466+0000 7fb79776f700 1 --2- 172.21.15.96:0/2491551538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7901027b0 0x7fb79010aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.466+0000 7fb79776f700 1 --2- 172.21.15.96:0/2491551538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 0x7fb790102270 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.466+0000 7fb79776f700 1 -- 172.21.15.96:0/2491551538 >> 172.21.15.96:0/2491551538 conn(0x7fb7900fb680 msgr2=0x7fb7900fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.466+0000 7fb79776f700 1 -- 172.21.15.96:0/2491551538 shutdown_connections 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.466+0000 7fb79776f700 1 -- 172.21.15.96:0/2491551538 wait complete. 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79776f700 1 Processor -- start 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79776f700 1 -- start start 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79776f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 0x7fb79019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79776f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7901027b0 0x7fb79019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79776f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79019d360 con 0x7fb790101e90 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79776f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7901969d0 con 0x7fb7901027b0 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79550b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 0x7fb79019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79550b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 0x7fb79019c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58338/0 (socket says 172.21.15.96:58338) 2024-04-22T15:33:44.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79550b700 1 -- 172.21.15.96:0/2795896459 learned_addr learned my addr 172.21.15.96:0/2795896459 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb794d0a700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7901027b0 0x7fb79019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79550b700 1 -- 172.21.15.96:0/2795896459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7901027b0 msgr2=0x7fb79019cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79550b700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7901027b0 0x7fb79019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.467+0000 7fb79550b700 1 -- 172.21.15.96:0/2795896459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c0095f0 con 0x7fb790101e90 2024-04-22T15:33:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.468+0000 7fb79550b700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 0x7fb79019c890 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb78c006010 tx=0x7fb78c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.468+0000 7fb7867fc700 1 -- 172.21.15.96:0/2795896459 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78c01d070 con 0x7fb790101e90 2024-04-22T15:33:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.468+0000 7fb7867fc700 1 -- 172.21.15.96:0/2795896459 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb78c00f460 con 0x7fb790101e90 2024-04-22T15:33:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.468+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb790196c50 con 0x7fb790101e90 2024-04-22T15:33:44.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.468+0000 7fb7867fc700 1 -- 172.21.15.96:0/2795896459 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78c017630 con 0x7fb790101e90 2024-04-22T15:33:44.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.468+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb790197140 con 0x7fb790101e90 2024-04-22T15:33:44.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.469+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7901055b0 con 0x7fb790101e90 2024-04-22T15:33:44.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.469+0000 7fb7867fc700 1 -- 172.21.15.96:0/2795896459 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb78c00f5d0 con 0x7fb790101e90 2024-04-22T15:33:44.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.470+0000 7fb7867fc700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77c06ecc0 0x7fb77c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:44.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.470+0000 7fb7867fc700 1 -- 172.21.15.96:0/2795896459 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb78c090350 con 0x7fb790101e90 2024-04-22T15:33:44.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.470+0000 7fb794d0a700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77c06ecc0 0x7fb77c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:44.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.470+0000 7fb794d0a700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77c06ecc0 0x7fb77c071180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb790197b90 tx=0x7fb780005d50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:44.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.474+0000 7fb7867fc700 1 -- 172.21.15.96:0/2795896459 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb78c05cd20 con 0x7fb790101e90 2024-04-22T15:33:44.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.686+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb79004ec20 con 0x7fb77c06ecc0 2024-04-22T15:33:44.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.696+0000 7fb7867fc700 1 -- 172.21.15.96:0/2795896459 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb79004ec20 con 0x7fb77c06ecc0 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (12m) 5m ago 23m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (24m) 5m ago 24m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (22m) 5m ago 22m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (9m) 5m ago 23m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (9m) 5m ago 22m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (12m) 5m ago 23m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (5m) 5m ago 17m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (5m) 5m ago 18m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (5m) 5m ago 18m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:33:44.700 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (17m) 5m ago 17m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (14m) 5m ago 25m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (13m) 5m ago 22m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (10m) 5m ago 25m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (10m) 5m ago 22m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (13m) 5m ago 23m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (13m) 5m ago 22m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (9m) 5m ago 21m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (8m) 5m ago 21m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (7m) 5m ago 20m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (7m) 5m ago 20m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (6m) 5m ago 20m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (6m) 5m ago 19m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:33:44.701 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (12m) 5m ago 23m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:33:44.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.698+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77c06ecc0 msgr2=0x7fb77c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.698+0000 7fb79776f700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77c06ecc0 0x7fb77c071180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb790197b90 tx=0x7fb780005d50 comp rx=0 tx=0).stop 2024-04-22T15:33:44.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.698+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 msgr2=0x7fb79019c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.698+0000 7fb79776f700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 0x7fb79019c890 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb78c006010 tx=0x7fb78c00be20 comp rx=0 tx=0).stop 2024-04-22T15:33:44.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.698+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 shutdown_connections 2024-04-22T15:33:44.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.698+0000 7fb79776f700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77c06ecc0 0x7fb77c071180 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.698+0000 7fb79776f700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790101e90 0x7fb79019c890 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.698+0000 7fb79776f700 1 --2- 172.21.15.96:0/2795896459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7901027b0 0x7fb79019cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.698+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 >> 172.21.15.96:0/2795896459 conn(0x7fb7900fb680 msgr2=0x7fb790103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.699+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 shutdown_connections 2024-04-22T15:33:44.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.699+0000 7fb79776f700 1 -- 172.21.15.96:0/2795896459 wait complete. 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.802+0000 7f131261e700 1 -- 172.21.15.96:0/1011038874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 msgr2=0x7f130c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.802+0000 7f131261e700 1 --2- 172.21.15.96:0/1011038874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 0x7f130c1021d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1300009910 tx=0x7f1300009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.802+0000 7f131261e700 1 -- 172.21.15.96:0/1011038874 shutdown_connections 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.802+0000 7f131261e700 1 --2- 172.21.15.96:0/1011038874 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f130c102710 0x7f130c10ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.802+0000 7f131261e700 1 --2- 172.21.15.96:0/1011038874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 0x7f130c1021d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.802+0000 7f131261e700 1 -- 172.21.15.96:0/1011038874 >> 172.21.15.96:0/1011038874 conn(0x7f130c0fb680 msgr2=0x7f130c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.802+0000 7f131261e700 1 -- 172.21.15.96:0/1011038874 shutdown_connections 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.802+0000 7f131261e700 1 -- 172.21.15.96:0/1011038874 wait complete. 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f131261e700 1 Processor -- start 2024-04-22T15:33:44.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f131261e700 1 -- start start 2024-04-22T15:33:44.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f131261e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 0x7f130c19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:44.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f131261e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f130c102710 0x7f130c19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:44.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f131261e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c19d490 con 0x7f130c101df0 2024-04-22T15:33:44.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f131261e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c196a70 con 0x7f130c102710 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f130bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 0x7f130c19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f130bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 0x7f130c19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58350/0 (socket says 172.21.15.96:58350) 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f130bfff700 1 -- 172.21.15.96:0/1303182563 learned_addr learned my addr 172.21.15.96:0/1303182563 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f130b7fe700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f130c102710 0x7f130c19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f130bfff700 1 -- 172.21.15.96:0/1303182563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f130c102710 msgr2=0x7f130c19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f130bfff700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f130c102710 0x7f130c19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f130bfff700 1 -- 172.21.15.96:0/1303182563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13000095f0 con 0x7f130c101df0 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.803+0000 7f130b7fe700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f130c102710 0x7f130c19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.804+0000 7f130bfff700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 0x7f130c19c930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f130000bd70 tx=0x7f130000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.804+0000 7f13097fa700 1 -- 172.21.15.96:0/1303182563 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f130001d070 con 0x7f130c101df0 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.804+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c196cf0 con 0x7f130c101df0 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.804+0000 7f13097fa700 1 -- 172.21.15.96:0/1303182563 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f130000f460 con 0x7f130c101df0 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.804+0000 7f13097fa700 1 -- 172.21.15.96:0/1303182563 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1300017630 con 0x7f130c101df0 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.804+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c1971e0 con 0x7f130c101df0 2024-04-22T15:33:44.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.805+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c105510 con 0x7f130c101df0 2024-04-22T15:33:44.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.805+0000 7f13097fa700 1 -- 172.21.15.96:0/1303182563 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f130000f5d0 con 0x7f130c101df0 2024-04-22T15:33:44.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.806+0000 7f13097fa700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f12f406ecc0 0x7f12f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:44.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.806+0000 7f130b7fe700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f12f406ecc0 0x7f12f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:44.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.806+0000 7f13097fa700 1 -- 172.21.15.96:0/1303182563 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1300090400 con 0x7f130c101df0 2024-04-22T15:33:44.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.806+0000 7f130b7fe700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f12f406ecc0 0x7f12f4071180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f130c197c30 tx=0x7f12fc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:44.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:44.810+0000 7f13097fa700 1 -- 172.21.15.96:0/1303182563 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f130005cc90 con 0x7f130c101df0 2024-04-22T15:33:45.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.097+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f130c197f30 con 0x7f130c101df0 2024-04-22T15:33:45.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.097+0000 7f13097fa700 1 -- 172.21.15.96:0/1303182563 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f130005c820 con 0x7f130c101df0 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:33:45.101 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:33:45.102 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:33:45.102 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:33:45.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:33:45.102 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:33:45.102 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:33:45.102 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:33:45.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:33:45.102 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:33:45.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.099+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f12f406ecc0 msgr2=0x7f12f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.099+0000 7f131261e700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f12f406ecc0 0x7f12f4071180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f130c197c30 tx=0x7f12fc008040 comp rx=0 tx=0).stop 2024-04-22T15:33:45.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.099+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 msgr2=0x7f130c19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.099+0000 7f131261e700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 0x7f130c19c930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f130000bd70 tx=0x7f130000be50 comp rx=0 tx=0).stop 2024-04-22T15:33:45.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.100+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 shutdown_connections 2024-04-22T15:33:45.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.100+0000 7f131261e700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f12f406ecc0 0x7f12f4071180 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.100+0000 7f131261e700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f130c101df0 0x7f130c19c930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.100+0000 7f131261e700 1 --2- 172.21.15.96:0/1303182563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f130c102710 0x7f130c19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.100+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 >> 172.21.15.96:0/1303182563 conn(0x7f130c0fb680 msgr2=0x7f130c1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:45.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.100+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 shutdown_connections 2024-04-22T15:33:45.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.100+0000 7f131261e700 1 -- 172.21.15.96:0/1303182563 wait complete. 2024-04-22T15:33:45.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.202+0000 7f4e106c0700 1 -- 172.21.15.96:0/2153295107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e081017a0 msgr2=0x7f4e081057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.203+0000 7f4e106c0700 1 --2- 172.21.15.96:0/2153295107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e081017a0 0x7f4e081057f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009910 tx=0x7f4df8009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:45.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.203+0000 7f4e106c0700 1 -- 172.21.15.96:0/2153295107 shutdown_connections 2024-04-22T15:33:45.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.203+0000 7f4e106c0700 1 --2- 172.21.15.96:0/2153295107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e081017a0 0x7f4e081057f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.203+0000 7f4e106c0700 1 --2- 172.21.15.96:0/2153295107 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e08100df0 0x7f4e081011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.203+0000 7f4e106c0700 1 -- 172.21.15.96:0/2153295107 >> 172.21.15.96:0/2153295107 conn(0x7f4e080fc6a0 msgr2=0x7f4e080feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.203+0000 7f4e106c0700 1 -- 172.21.15.96:0/2153295107 shutdown_connections 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.203+0000 7f4e106c0700 1 -- 172.21.15.96:0/2153295107 wait complete. 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.203+0000 7f4e106c0700 1 Processor -- start 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.203+0000 7f4e106c0700 1 -- start start 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e106c0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e08100df0 0x7f4e0819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e106c0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e081017a0 0x7f4e0819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e106c0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0819d370 con 0x7f4e08100df0 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e106c0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e08196950 con 0x7f4e081017a0 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e0e45c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e08100df0 0x7f4e0819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e0e45c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e08100df0 0x7f4e0819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58362/0 (socket says 172.21.15.96:58362) 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e0e45c700 1 -- 172.21.15.96:0/1228557393 learned_addr learned my addr 172.21.15.96:0/1228557393 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e0dc5b700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e081017a0 0x7f4e0819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:45.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e0e45c700 1 -- 172.21.15.96:0/1228557393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e081017a0 msgr2=0x7f4e0819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e0e45c700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e081017a0 0x7f4e0819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e0e45c700 1 -- 172.21.15.96:0/1228557393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df80095f0 con 0x7f4e08100df0 2024-04-22T15:33:45.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e0e45c700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e08100df0 0x7f4e0819c810 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400d8d0 tx=0x7f4e0400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:45.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4dff7fe700 1 -- 172.21.15.96:0/1228557393 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e040098a0 con 0x7f4e08100df0 2024-04-22T15:33:45.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e08196c30 con 0x7f4e08100df0 2024-04-22T15:33:45.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4dff7fe700 1 -- 172.21.15.96:0/1228557393 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e04010460 con 0x7f4e08100df0 2024-04-22T15:33:45.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4dff7fe700 1 -- 172.21.15.96:0/1228557393 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e0400f5f0 con 0x7f4e08100df0 2024-04-22T15:33:45.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.204+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e08197180 con 0x7f4e08100df0 2024-04-22T15:33:45.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.205+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e081091f0 con 0x7f4e08100df0 2024-04-22T15:33:45.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.206+0000 7f4dff7fe700 1 -- 172.21.15.96:0/1228557393 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4e04009a00 con 0x7f4e08100df0 2024-04-22T15:33:45.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.206+0000 7f4dff7fe700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4df406ecc0 0x7f4df4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:45.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.206+0000 7f4dff7fe700 1 -- 172.21.15.96:0/1228557393 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4e0408ed80 con 0x7f4e08100df0 2024-04-22T15:33:45.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.206+0000 7f4e0dc5b700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4df406ecc0 0x7f4df4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:45.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.207+0000 7f4e0dc5b700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4df406ecc0 0x7f4df4071180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4e08197b10 tx=0x7f4df801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:45.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.211+0000 7f4dff7fe700 1 -- 172.21.15.96:0/1228557393 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4e0405b610 con 0x7f4e08100df0 2024-04-22T15:33:45.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.455+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4e08197e10 con 0x7f4e08100df0 2024-04-22T15:33:45.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.455+0000 7f4dff7fe700 1 -- 172.21.15.96:0/1228557393 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f4e0405b1a0 con 0x7f4e08100df0 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:33:45.459 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:33:45.460 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:33:45.461 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:33:45.462 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:33:45.462 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:33:45.462 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:33:45.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4df406ecc0 msgr2=0x7f4df4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4df406ecc0 0x7f4df4071180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4e08197b10 tx=0x7f4df801a040 comp rx=0 tx=0).stop 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e08100df0 msgr2=0x7f4e0819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e08100df0 0x7f4e0819c810 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400d8d0 tx=0x7f4e0400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 shutdown_connections 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4df406ecc0 0x7f4df4071180 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e08100df0 0x7f4e0819c810 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 --2- 172.21.15.96:0/1228557393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e081017a0 0x7f4e0819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 >> 172.21.15.96:0/1228557393 conn(0x7f4e080fc6a0 msgr2=0x7f4e080fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 shutdown_connections 2024-04-22T15:33:45.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.458+0000 7f4e106c0700 1 -- 172.21.15.96:0/1228557393 wait complete. 2024-04-22T15:33:45.464 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:33:45.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.565+0000 7fc0452ff700 1 -- 172.21.15.96:0/3685780799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 msgr2=0x7fc0400fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.565+0000 7fc0452ff700 1 --2- 172.21.15.96:0/3685780799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 0x7fc0400fff40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc034009910 tx=0x7fc034009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:45.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.565+0000 7fc0452ff700 1 -- 172.21.15.96:0/3685780799 shutdown_connections 2024-04-22T15:33:45.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.565+0000 7fc0452ff700 1 --2- 172.21.15.96:0/3685780799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 0x7fc0400fff40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.565+0000 7fc0452ff700 1 --2- 172.21.15.96:0/3685780799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040105ac0 0x7fc040105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.565+0000 7fc0452ff700 1 -- 172.21.15.96:0/3685780799 >> 172.21.15.96:0/3685780799 conn(0x7fc0400fb5d0 msgr2=0x7fc0400fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:45.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.565+0000 7fc0452ff700 1 -- 172.21.15.96:0/3685780799 shutdown_connections 2024-04-22T15:33:45.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.566+0000 7fc0452ff700 1 -- 172.21.15.96:0/3685780799 wait complete. 2024-04-22T15:33:45.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.566+0000 7fc0452ff700 1 Processor -- start 2024-04-22T15:33:45.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.566+0000 7fc0452ff700 1 -- start start 2024-04-22T15:33:45.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc0452ff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 0x7fc040197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:45.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc0452ff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040105ac0 0x7fc040197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:45.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc0452ff700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc040198430 con 0x7fc0400ffac0 2024-04-22T15:33:45.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc0452ff700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc040198570 con 0x7fc040105ac0 2024-04-22T15:33:45.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc03effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 0x7fc040197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:45.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc03effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 0x7fc040197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58382/0 (socket says 172.21.15.96:58382) 2024-04-22T15:33:45.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc03effd700 1 -- 172.21.15.96:0/1792251563 learned_addr learned my addr 172.21.15.96:0/1792251563 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:45.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc03e7fc700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040105ac0 0x7fc040197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc03effd700 1 -- 172.21.15.96:0/1792251563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040105ac0 msgr2=0x7fc040197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc03effd700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040105ac0 0x7fc040197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc03effd700 1 -- 172.21.15.96:0/1792251563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0340095f0 con 0x7fc0400ffac0 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc03effd700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 0x7fc040197920 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc03000d8d0 tx=0x7fc03000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc027fff700 1 -- 172.21.15.96:0/1792251563 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc030009940 con 0x7fc0400ffac0 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc027fff700 1 -- 172.21.15.96:0/1792251563 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc030010460 con 0x7fc0400ffac0 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0401a0990 con 0x7fc0400ffac0 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.567+0000 7fc027fff700 1 -- 172.21.15.96:0/1792251563 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc03000f5f0 con 0x7fc0400ffac0 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.568+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0401a0e00 con 0x7fc0400ffac0 2024-04-22T15:33:45.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.569+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0401080e0 con 0x7fc0400ffac0 2024-04-22T15:33:45.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.569+0000 7fc027fff700 1 -- 172.21.15.96:0/1792251563 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc0300105d0 con 0x7fc0400ffac0 2024-04-22T15:33:45.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.569+0000 7fc027fff700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc02806ecd0 0x7fc028071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:45.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.570+0000 7fc027fff700 1 -- 172.21.15.96:0/1792251563 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc03008eed0 con 0x7fc0400ffac0 2024-04-22T15:33:45.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.570+0000 7fc03e7fc700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc02806ecd0 0x7fc028071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:45.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.570+0000 7fc03e7fc700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc02806ecd0 0x7fc028071190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc034005b40 tx=0x7fc03401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:45.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.574+0000 7fc027fff700 1 -- 172.21.15.96:0/1792251563 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc030057c30 con 0x7fc0400ffac0 2024-04-22T15:33:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:45 smithi096 ceph-mon[112443]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:45 smithi096 ceph-mon[112443]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:45 smithi096 ceph-mon[112443]: pgmap v356: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:45 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1303182563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:33:45.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:45 smithi196 ceph-mon[81815]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:45.762 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:45 smithi196 ceph-mon[81815]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:45.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:45 smithi196 ceph-mon[81815]: pgmap v356: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:45.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:45 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1303182563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:33:45.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.799+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0400697c0 con 0x7fc02806ecd0 2024-04-22T15:33:45.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.800+0000 7fc027fff700 1 -- 172.21.15.96:0/1792251563 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc0400697c0 con 0x7fc02806ecd0 2024-04-22T15:33:45.803 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:33:45.803 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:33:45.803 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:33:45.803 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:33:45.804 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:33:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.802+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc02806ecd0 msgr2=0x7fc028071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.802+0000 7fc0452ff700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc02806ecd0 0x7fc028071190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc034005b40 tx=0x7fc03401a040 comp rx=0 tx=0).stop 2024-04-22T15:33:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.802+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 msgr2=0x7fc040197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.802+0000 7fc0452ff700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 0x7fc040197920 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc03000d8d0 tx=0x7fc03000dc90 comp rx=0 tx=0).stop 2024-04-22T15:33:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.802+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 shutdown_connections 2024-04-22T15:33:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.802+0000 7fc0452ff700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc02806ecd0 0x7fc028071190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.802+0000 7fc0452ff700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0400ffac0 0x7fc040197920 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.802+0000 7fc0452ff700 1 --2- 172.21.15.96:0/1792251563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc040105ac0 0x7fc040197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.802+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 >> 172.21.15.96:0/1792251563 conn(0x7fc0400fb5d0 msgr2=0x7fc0400fc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.803+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 shutdown_connections 2024-04-22T15:33:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.803+0000 7fc0452ff700 1 -- 172.21.15.96:0/1792251563 wait complete. 2024-04-22T15:33:45.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.906+0000 7fa02b155700 1 -- 172.21.15.96:0/682026958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 msgr2=0x7fa024101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.906+0000 7fa02b155700 1 --2- 172.21.15.96:0/682026958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 0x7fa024101200 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa018009910 tx=0x7fa018009c20 comp rx=0 tx=0).stop 2024-04-22T15:33:45.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.906+0000 7fa02b155700 1 -- 172.21.15.96:0/682026958 shutdown_connections 2024-04-22T15:33:45.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.906+0000 7fa02b155700 1 --2- 172.21.15.96:0/682026958 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0241017d0 0x7fa024105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.906+0000 7fa02b155700 1 --2- 172.21.15.96:0/682026958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 0x7fa024101200 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.906+0000 7fa02b155700 1 -- 172.21.15.96:0/682026958 >> 172.21.15.96:0/682026958 conn(0x7fa0240fc6d0 msgr2=0x7fa0240feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.906+0000 7fa02b155700 1 -- 172.21.15.96:0/682026958 shutdown_connections 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.906+0000 7fa02b155700 1 -- 172.21.15.96:0/682026958 wait complete. 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa02b155700 1 Processor -- start 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa02b155700 1 -- start start 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa02b155700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 0x7fa02419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa02b155700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0241017d0 0x7fa02419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa02b155700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02419d3a0 con 0x7fa024100e20 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa02b155700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0241969d0 con 0x7fa0241017d0 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa028ef1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 0x7fa02419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa028ef1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 0x7fa02419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58404/0 (socket says 172.21.15.96:58404) 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa028ef1700 1 -- 172.21.15.96:0/3696932470 learned_addr learned my addr 172.21.15.96:0/3696932470 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:33:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa023fff700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0241017d0 0x7fa02419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa028ef1700 1 -- 172.21.15.96:0/3696932470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0241017d0 msgr2=0x7fa02419cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa028ef1700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0241017d0 0x7fa02419cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa028ef1700 1 -- 172.21.15.96:0/3696932470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0180095f0 con 0x7fa024100e20 2024-04-22T15:33:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.907+0000 7fa028ef1700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 0x7fa02419c840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa01800bd70 tx=0x7fa01800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.908+0000 7fa021ffb700 1 -- 172.21.15.96:0/3696932470 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa01801d070 con 0x7fa024100e20 2024-04-22T15:33:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.908+0000 7fa021ffb700 1 -- 172.21.15.96:0/3696932470 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa01800f460 con 0x7fa024100e20 2024-04-22T15:33:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.908+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa024196c50 con 0x7fa024100e20 2024-04-22T15:33:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.908+0000 7fa021ffb700 1 -- 172.21.15.96:0/3696932470 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa018017630 con 0x7fa024100e20 2024-04-22T15:33:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.908+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa024197140 con 0x7fa024100e20 2024-04-22T15:33:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.909+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0241091c0 con 0x7fa024100e20 2024-04-22T15:33:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.909+0000 7fa021ffb700 1 -- 172.21.15.96:0/3696932470 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa01800f5d0 con 0x7fa024100e20 2024-04-22T15:33:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.910+0000 7fa021ffb700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01406ecc0 0x7fa014071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:33:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.910+0000 7fa023fff700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01406ecc0 0x7fa014071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:33:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.910+0000 7fa021ffb700 1 -- 172.21.15.96:0/3696932470 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa018090400 con 0x7fa024100e20 2024-04-22T15:33:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.910+0000 7fa023fff700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01406ecc0 0x7fa014071180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa010005fd0 tx=0x7fa01000a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:33:45.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:45.915+0000 7fa021ffb700 1 -- 172.21.15.96:0/3696932470 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa01805cc90 con 0x7fa024100e20 2024-04-22T15:33:46.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.197+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa024197e40 con 0x7fa024100e20 2024-04-22T15:33:46.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.197+0000 7fa021ffb700 1 -- 172.21.15.96:0/3696932470 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fa01805c820 con 0x7fa024100e20 2024-04-22T15:33:46.201 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:33:46.201 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:33:46.201 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:33:46.201 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:33:46.201 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:33:46.201 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:33:46.202 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:33:46.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01406ecc0 msgr2=0x7fa014071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:46.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01406ecc0 0x7fa014071180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa010005fd0 tx=0x7fa01000a560 comp rx=0 tx=0).stop 2024-04-22T15:33:46.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 msgr2=0x7fa02419c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:33:46.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 0x7fa02419c840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa01800bd70 tx=0x7fa01800be50 comp rx=0 tx=0).stop 2024-04-22T15:33:46.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 shutdown_connections 2024-04-22T15:33:46.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01406ecc0 0x7fa014071180 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:46.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa024100e20 0x7fa02419c840 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:46.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 --2- 172.21.15.96:0/3696932470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0241017d0 0x7fa02419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:33:46.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 >> 172.21.15.96:0/3696932470 conn(0x7fa0240fc6d0 msgr2=0x7fa0240fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:33:46.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 shutdown_connections 2024-04-22T15:33:46.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:33:46.200+0000 7fa02b155700 1 -- 172.21.15.96:0/3696932470 wait complete. 2024-04-22T15:33:46.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:46 smithi096 ceph-mon[112443]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:46.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:46 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1228557393' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:33:46.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:46 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3696932470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:33:46.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:46 smithi196 ceph-mon[81815]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:46.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:46 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1228557393' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:33:46.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:46 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3696932470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:33:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:47 smithi096 ceph-mon[112443]: from='client.34724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:47 smithi096 ceph-mon[112443]: pgmap v357: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:47 smithi196 ceph-mon[81815]: from='client.34724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:33:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:47 smithi196 ceph-mon[81815]: pgmap v357: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:49 smithi096 ceph-mon[112443]: pgmap v358: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:49 smithi196 ceph-mon[81815]: pgmap v358: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:51 smithi096 ceph-mon[112443]: pgmap v359: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:51 smithi196 ceph-mon[81815]: pgmap v359: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:53 smithi096 ceph-mon[112443]: pgmap v360: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:33:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:53 smithi196 ceph-mon[81815]: pgmap v360: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:33:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:55 smithi096 ceph-mon[112443]: pgmap v361: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:55 smithi196 ceph-mon[81815]: pgmap v361: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:57 smithi096 ceph-mon[112443]: pgmap v362: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:57 smithi196 ceph-mon[81815]: pgmap v362: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:33:59 smithi096 ceph-mon[112443]: pgmap v363: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:33:59.749 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:33:59 smithi196 ceph-mon[81815]: pgmap v363: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:01 smithi096 ceph-mon[112443]: pgmap v364: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:01 smithi196 ceph-mon[81815]: pgmap v364: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:03 smithi096 ceph-mon[112443]: pgmap v365: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:03 smithi196 ceph-mon[81815]: pgmap v365: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:05 smithi096 ceph-mon[112443]: pgmap v366: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:05.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:05 smithi196 ceph-mon[81815]: pgmap v366: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:07 smithi096 ceph-mon[112443]: pgmap v367: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:07 smithi196 ceph-mon[81815]: pgmap v367: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:34:08.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:34:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:09 smithi096 ceph-mon[112443]: pgmap v368: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:09 smithi196 ceph-mon[81815]: pgmap v368: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:11 smithi096 ceph-mon[112443]: pgmap v369: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:11 smithi196 ceph-mon[81815]: pgmap v369: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:13 smithi096 ceph-mon[112443]: pgmap v370: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:13 smithi196 ceph-mon[81815]: pgmap v370: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:15.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:15 smithi096 ceph-mon[112443]: pgmap v371: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:15 smithi196 ceph-mon[81815]: pgmap v371: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.317+0000 7f982ef75700 1 -- 172.21.15.96:0/2608100264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 msgr2=0x7f9828100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.317+0000 7f982ef75700 1 --2- 172.21.15.96:0/2608100264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 0x7f9828100ac0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f981c009910 tx=0x7f981c009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.317+0000 7f982ef75700 1 -- 172.21.15.96:0/2608100264 shutdown_connections 2024-04-22T15:34:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.317+0000 7f982ef75700 1 --2- 172.21.15.96:0/2608100264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9828101090 0x7f9828104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.317+0000 7f982ef75700 1 --2- 172.21.15.96:0/2608100264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 0x7f9828100ac0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.317+0000 7f982ef75700 1 -- 172.21.15.96:0/2608100264 >> 172.21.15.96:0/2608100264 conn(0x7f9828074df0 msgr2=0x7f9828075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.317+0000 7f982ef75700 1 -- 172.21.15.96:0/2608100264 shutdown_connections 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.317+0000 7f982ef75700 1 -- 172.21.15.96:0/2608100264 wait complete. 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.318+0000 7f982ef75700 1 Processor -- start 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.318+0000 7f982ef75700 1 -- start start 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.318+0000 7f982ef75700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 0x7f982819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.318+0000 7f982ef75700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9828101090 0x7f982819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.318+0000 7f982ef75700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982819d470 con 0x7f98281006e0 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.318+0000 7f982ef75700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828196a50 con 0x7f9828101090 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.318+0000 7f982cd11700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 0x7f982819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.318+0000 7f982cd11700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 0x7f982819c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34484/0 (socket says 172.21.15.96:34484) 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.318+0000 7f982cd11700 1 -- 172.21.15.96:0/64113672 learned_addr learned my addr 172.21.15.96:0/64113672 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f9827fff700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9828101090 0x7f982819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:16.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f982cd11700 1 -- 172.21.15.96:0/64113672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9828101090 msgr2=0x7f982819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f982cd11700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9828101090 0x7f982819ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f982cd11700 1 -- 172.21.15.96:0/64113672 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c0095f0 con 0x7f98281006e0 2024-04-22T15:34:16.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f982cd11700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 0x7f982819c910 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f981c00bd70 tx=0x7f981c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:16.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f9825ffb700 1 -- 172.21.15.96:0/64113672 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f981c01d070 con 0x7f98281006e0 2024-04-22T15:34:16.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9828196cd0 con 0x7f98281006e0 2024-04-22T15:34:16.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f9825ffb700 1 -- 172.21.15.96:0/64113672 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f981c00f460 con 0x7f98281006e0 2024-04-22T15:34:16.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f9825ffb700 1 -- 172.21.15.96:0/64113672 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f981c017630 con 0x7f98281006e0 2024-04-22T15:34:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.319+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98281971c0 con 0x7f98281006e0 2024-04-22T15:34:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.320+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98281090c0 con 0x7f98281006e0 2024-04-22T15:34:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.321+0000 7f9825ffb700 1 -- 172.21.15.96:0/64113672 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f981c00f5d0 con 0x7f98281006e0 2024-04-22T15:34:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.321+0000 7f9825ffb700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f981806ecc0 0x7f9818071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.321+0000 7f9825ffb700 1 -- 172.21.15.96:0/64113672 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f981c0903f0 con 0x7f98281006e0 2024-04-22T15:34:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.321+0000 7f9827fff700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f981806ecc0 0x7f9818071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.322+0000 7f9827fff700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f981806ecc0 0x7f9818071180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9814006fd0 tx=0x7f9814008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:16.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.326+0000 7f9825ffb700 1 -- 172.21.15.96:0/64113672 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f981c05cc80 con 0x7f98281006e0 2024-04-22T15:34:16.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.546+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9828197f80 con 0x7f981806ecc0 2024-04-22T15:34:16.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.547+0000 7f9825ffb700 1 -- 172.21.15.96:0/64113672 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9828197f80 con 0x7f981806ecc0 2024-04-22T15:34:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f981806ecc0 msgr2=0x7f9818071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f981806ecc0 0x7f9818071180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9814006fd0 tx=0x7f9814008040 comp rx=0 tx=0).stop 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 msgr2=0x7f982819c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 0x7f982819c910 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f981c00bd70 tx=0x7f981c00be50 comp rx=0 tx=0).stop 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 shutdown_connections 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f981806ecc0 0x7f9818071180 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98281006e0 0x7f982819c910 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 --2- 172.21.15.96:0/64113672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9828101090 0x7f982819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 >> 172.21.15.96:0/64113672 conn(0x7f9828074df0 msgr2=0x7f9828106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 shutdown_connections 2024-04-22T15:34:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.549+0000 7f982ef75700 1 -- 172.21.15.96:0/64113672 wait complete. 2024-04-22T15:34:16.563 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:34:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.655+0000 7ff52d6fb700 1 -- 172.21.15.96:0/1067114440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 msgr2=0x7ff528101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.655+0000 7ff52d6fb700 1 --2- 172.21.15.96:0/1067114440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 0x7ff528101220 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff518009910 tx=0x7ff518009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.656+0000 7ff52d6fb700 1 -- 172.21.15.96:0/1067114440 shutdown_connections 2024-04-22T15:34:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.656+0000 7ff52d6fb700 1 --2- 172.21.15.96:0/1067114440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5281017f0 0x7ff528105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.656+0000 7ff52d6fb700 1 --2- 172.21.15.96:0/1067114440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 0x7ff528101220 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.656+0000 7ff52d6fb700 1 -- 172.21.15.96:0/1067114440 >> 172.21.15.96:0/1067114440 conn(0x7ff5280fc710 msgr2=0x7ff5280feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.656+0000 7ff52d6fb700 1 -- 172.21.15.96:0/1067114440 shutdown_connections 2024-04-22T15:34:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.656+0000 7ff52d6fb700 1 -- 172.21.15.96:0/1067114440 wait complete. 2024-04-22T15:34:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.656+0000 7ff52d6fb700 1 Processor -- start 2024-04-22T15:34:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.656+0000 7ff52d6fb700 1 -- start start 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff52d6fb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 0x7ff5280731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff52d6fb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5281017f0 0x7ff528073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff52d6fb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff528073c60 con 0x7ff528100e40 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff52d6fb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff528073da0 con 0x7ff5281017f0 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff526ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 0x7ff5280731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff526ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 0x7ff5280731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34502/0 (socket says 172.21.15.96:34502) 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff526ffd700 1 -- 172.21.15.96:0/2439325676 learned_addr learned my addr 172.21.15.96:0/2439325676 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff5267fc700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5281017f0 0x7ff528073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff526ffd700 1 -- 172.21.15.96:0/2439325676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5281017f0 msgr2=0x7ff528073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff526ffd700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5281017f0 0x7ff528073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff526ffd700 1 -- 172.21.15.96:0/2439325676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5180095f0 con 0x7ff528100e40 2024-04-22T15:34:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff526ffd700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 0x7ff5280731e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff51800bd70 tx=0x7ff51800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff50ffff700 1 -- 172.21.15.96:0/2439325676 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff51801d070 con 0x7ff528100e40 2024-04-22T15:34:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff528074020 con 0x7ff528100e40 2024-04-22T15:34:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff50ffff700 1 -- 172.21.15.96:0/2439325676 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5180056e0 con 0x7ff528100e40 2024-04-22T15:34:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff50ffff700 1 -- 172.21.15.96:0/2439325676 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff51800fe40 con 0x7ff528100e40 2024-04-22T15:34:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.657+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5281a2f80 con 0x7ff528100e40 2024-04-22T15:34:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.658+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5281091e0 con 0x7ff528100e40 2024-04-22T15:34:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.659+0000 7ff50ffff700 1 -- 172.21.15.96:0/2439325676 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff51800f680 con 0x7ff528100e40 2024-04-22T15:34:16.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.659+0000 7ff50ffff700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff51006ec60 0x7ff510071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:16.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.659+0000 7ff50ffff700 1 -- 172.21.15.96:0/2439325676 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff518090290 con 0x7ff528100e40 2024-04-22T15:34:16.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.659+0000 7ff5267fc700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff51006ec60 0x7ff510071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:16.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.660+0000 7ff5267fc700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff51006ec60 0x7ff510071120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff5280748c0 tx=0x7ff51c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:16.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.664+0000 7ff50ffff700 1 -- 172.21.15.96:0/2439325676 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff518058ff0 con 0x7ff528100e40 2024-04-22T15:34:16.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.881+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff52810de10 con 0x7ff51006ec60 2024-04-22T15:34:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.882+0000 7ff50ffff700 1 -- 172.21.15.96:0/2439325676 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff52810de10 con 0x7ff51006ec60 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff51006ec60 msgr2=0x7ff510071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff51006ec60 0x7ff510071120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff5280748c0 tx=0x7ff51c00b410 comp rx=0 tx=0).stop 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 msgr2=0x7ff5280731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 0x7ff5280731e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff51800bd70 tx=0x7ff51800bda0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 shutdown_connections 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff51006ec60 0x7ff510071120 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff528100e40 0x7ff5280731e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 --2- 172.21.15.96:0/2439325676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5281017f0 0x7ff528073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 >> 172.21.15.96:0/2439325676 conn(0x7ff5280fc710 msgr2=0x7ff5280fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 shutdown_connections 2024-04-22T15:34:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.884+0000 7ff52d6fb700 1 -- 172.21.15.96:0/2439325676 wait complete. 2024-04-22T15:34:16.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.988+0000 7f1121348700 1 -- 172.21.15.96:0/2060413279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 msgr2=0x7f111c105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.988+0000 7f1121348700 1 --2- 172.21.15.96:0/2060413279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 0x7f111c105820 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1110009910 tx=0x7f1110009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:16.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.988+0000 7f1121348700 1 -- 172.21.15.96:0/2060413279 shutdown_connections 2024-04-22T15:34:16.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.988+0000 7f1121348700 1 --2- 172.21.15.96:0/2060413279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 0x7f111c105820 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.988+0000 7f1121348700 1 --2- 172.21.15.96:0/2060413279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f111c100e20 0x7f111c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.988+0000 7f1121348700 1 -- 172.21.15.96:0/2060413279 >> 172.21.15.96:0/2060413279 conn(0x7f111c0fc6d0 msgr2=0x7f111c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:16.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.988+0000 7f1121348700 1 -- 172.21.15.96:0/2060413279 shutdown_connections 2024-04-22T15:34:16.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.988+0000 7f1121348700 1 -- 172.21.15.96:0/2060413279 wait complete. 2024-04-22T15:34:16.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f1121348700 1 Processor -- start 2024-04-22T15:34:16.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f1121348700 1 -- start start 2024-04-22T15:34:16.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f1121348700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f111c100e20 0x7f111c198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:16.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f1121348700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 0x7f111c199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:16.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f1121348700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c199840 con 0x7f111c1017d0 2024-04-22T15:34:16.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f1121348700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c1999b0 con 0x7f111c100e20 2024-04-22T15:34:16.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f111a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 0x7f111c199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:16.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f111a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 0x7f111c199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34512/0 (socket says 172.21.15.96:34512) 2024-04-22T15:34:16.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f111a7fc700 1 -- 172.21.15.96:0/2408269202 learned_addr learned my addr 172.21.15.96:0/2408269202 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f111affd700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f111c100e20 0x7f111c198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f111a7fc700 1 -- 172.21.15.96:0/2408269202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f111c100e20 msgr2=0x7f111c198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f111a7fc700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f111c100e20 0x7f111c198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.989+0000 7f111a7fc700 1 -- 172.21.15.96:0/2408269202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11100095f0 con 0x7f111c1017d0 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.990+0000 7f111a7fc700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 0x7f111c199240 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1110005b40 tx=0x7f111000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.990+0000 7f1103fff700 1 -- 172.21.15.96:0/2408269202 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f111001d070 con 0x7f111c1017d0 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.990+0000 7f1103fff700 1 -- 172.21.15.96:0/2408269202 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f111000f460 con 0x7f111c1017d0 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.990+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c19c630 con 0x7f111c1017d0 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.990+0000 7f1103fff700 1 -- 172.21.15.96:0/2408269202 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1110005720 con 0x7f111c1017d0 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.990+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c19cb20 con 0x7f111c1017d0 2024-04-22T15:34:16.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.991+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c104fa0 con 0x7f111c1017d0 2024-04-22T15:34:16.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.991+0000 7f1103fff700 1 -- 172.21.15.96:0/2408269202 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f111000f5d0 con 0x7f111c1017d0 2024-04-22T15:34:16.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.992+0000 7f1103fff700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f110406ecd0 0x7f1104071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:16.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.992+0000 7f111affd700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f110406ecd0 0x7f1104071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:16.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.992+0000 7f1103fff700 1 -- 172.21.15.96:0/2408269202 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1110090320 con 0x7f111c1017d0 2024-04-22T15:34:16.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.992+0000 7f111affd700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f110406ecd0 0x7f1104071190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f110c00ac50 tx=0x7f110c00a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:17.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:16.997+0000 7f1103fff700 1 -- 172.21.15.96:0/2408269202 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1110059080 con 0x7f111c1017d0 2024-04-22T15:34:17.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.204+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f111c19a040 con 0x7f110406ecd0 2024-04-22T15:34:17.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.216+0000 7f1103fff700 1 -- 172.21.15.96:0/2408269202 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f111c19a040 con 0x7f110406ecd0 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (12m) 6m ago 24m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (24m) 6m ago 24m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (23m) 5m ago 23m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (10m) 6m ago 24m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (10m) 5m ago 23m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (12m) 6m ago 23m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (6m) 6m ago 18m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (6m) 6m ago 18m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (5m) 5m ago 18m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (18m) 5m ago 18m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (15m) 6m ago 25m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:34:17.220 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (14m) 5m ago 23m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (11m) 6m ago 25m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (10m) 5m ago 23m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (13m) 6m ago 24m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (13m) 5m ago 23m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (9m) 6m ago 22m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (9m) 6m ago 21m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (8m) 6m ago 21m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (7m) 5m ago 21m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (7m) 5m ago 20m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (6m) 5m ago 20m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:34:17.221 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (13m) 6m ago 23m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:34:17.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.218+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f110406ecd0 msgr2=0x7f1104071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.218+0000 7f1121348700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f110406ecd0 0x7f1104071190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f110c00ac50 tx=0x7f110c00a380 comp rx=0 tx=0).stop 2024-04-22T15:34:17.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.218+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 msgr2=0x7f111c199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.218+0000 7f1121348700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 0x7f111c199240 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1110005b40 tx=0x7f111000bd70 comp rx=0 tx=0).stop 2024-04-22T15:34:17.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.218+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 shutdown_connections 2024-04-22T15:34:17.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.218+0000 7f1121348700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f110406ecd0 0x7f1104071190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.218+0000 7f1121348700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f111c100e20 0x7f111c198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.218+0000 7f1121348700 1 --2- 172.21.15.96:0/2408269202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f111c1017d0 0x7f111c199240 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.218+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 >> 172.21.15.96:0/2408269202 conn(0x7f111c0fc6d0 msgr2=0x7f111c0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:17.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.219+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 shutdown_connections 2024-04-22T15:34:17.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.219+0000 7f1121348700 1 -- 172.21.15.96:0/2408269202 wait complete. 2024-04-22T15:34:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.321+0000 7fcce98a5700 1 -- 172.21.15.96:0/101786389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 msgr2=0x7fcce4105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.321+0000 7fcce98a5700 1 --2- 172.21.15.96:0/101786389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 0x7fcce4105860 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009910 tx=0x7fccd8009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.321+0000 7fcce98a5700 1 -- 172.21.15.96:0/101786389 shutdown_connections 2024-04-22T15:34:17.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.321+0000 7fcce98a5700 1 --2- 172.21.15.96:0/101786389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 0x7fcce4105860 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.321+0000 7fcce98a5700 1 --2- 172.21.15.96:0/101786389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcce4100e60 0x7fcce4101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.321+0000 7fcce98a5700 1 -- 172.21.15.96:0/101786389 >> 172.21.15.96:0/101786389 conn(0x7fcce40fc710 msgr2=0x7fcce40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:17.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.321+0000 7fcce98a5700 1 -- 172.21.15.96:0/101786389 shutdown_connections 2024-04-22T15:34:17.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.321+0000 7fcce98a5700 1 -- 172.21.15.96:0/101786389 wait complete. 2024-04-22T15:34:17.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce98a5700 1 Processor -- start 2024-04-22T15:34:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce98a5700 1 -- start start 2024-04-22T15:34:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce98a5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcce4100e60 0x7fcce4073200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce98a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 0x7fcce4073740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce98a5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce40751e0 con 0x7fcce4101810 2024-04-22T15:34:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce98a5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce4073c80 con 0x7fcce4100e60 2024-04-22T15:34:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 0x7fcce4073740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 0x7fcce4073740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34526/0 (socket says 172.21.15.96:34526) 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce27fc700 1 -- 172.21.15.96:0/3745648163 learned_addr learned my addr 172.21.15.96:0/3745648163 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.322+0000 7fcce2ffd700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcce4100e60 0x7fcce4073200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.323+0000 7fcce27fc700 1 -- 172.21.15.96:0/3745648163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcce4100e60 msgr2=0x7fcce4073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.323+0000 7fcce27fc700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcce4100e60 0x7fcce4073200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.323+0000 7fcce27fc700 1 -- 172.21.15.96:0/3745648163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd80095f0 con 0x7fcce4101810 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.323+0000 7fcce27fc700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 0x7fcce4073740 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fccd800bd70 tx=0x7fccd800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.323+0000 7fcce88a3700 1 -- 172.21.15.96:0/3745648163 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccd801d070 con 0x7fcce4101810 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.323+0000 7fcce88a3700 1 -- 172.21.15.96:0/3745648163 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccd80056e0 con 0x7fcce4101810 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.323+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce4073f00 con 0x7fcce4101810 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.323+0000 7fcce88a3700 1 -- 172.21.15.96:0/3745648163 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccd800fe50 con 0x7fcce4101810 2024-04-22T15:34:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.323+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce41a2fc0 con 0x7fcce4101810 2024-04-22T15:34:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.324+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce4109260 con 0x7fcce4101810 2024-04-22T15:34:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.324+0000 7fcce88a3700 1 -- 172.21.15.96:0/3745648163 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fccd800f680 con 0x7fcce4101810 2024-04-22T15:34:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.325+0000 7fcce88a3700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcccc06ecc0 0x7fcccc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.325+0000 7fcce88a3700 1 -- 172.21.15.96:0/3745648163 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fccd8090320 con 0x7fcce4101810 2024-04-22T15:34:17.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.325+0000 7fcce2ffd700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcccc06ecc0 0x7fcccc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:17.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.325+0000 7fcce2ffd700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcccc06ecc0 0x7fcccc071180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fccd4005950 tx=0x7fccd400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:17.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.330+0000 7fcce88a3700 1 -- 172.21.15.96:0/3745648163 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fccd8059080 con 0x7fcce4101810 2024-04-22T15:34:17.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.616+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcce4074620 con 0x7fcce4101810 2024-04-22T15:34:17.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.616+0000 7fcce88a3700 1 -- 172.21.15.96:0/3745648163 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7fccd805c740 con 0x7fcce4101810 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:34:17.620 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:34:17.621 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:34:17.621 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:34:17.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:34:17.621 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:34:17.621 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:34:17.621 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:34:17.621 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:34:17.621 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.618+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcccc06ecc0 msgr2=0x7fcccc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.618+0000 7fcce98a5700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcccc06ecc0 0x7fcccc071180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fccd4005950 tx=0x7fccd400b410 comp rx=0 tx=0).stop 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.618+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 msgr2=0x7fcce4073740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.618+0000 7fcce98a5700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 0x7fcce4073740 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fccd800bd70 tx=0x7fccd800bda0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.618+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 shutdown_connections 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.618+0000 7fcce98a5700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcccc06ecc0 0x7fcccc071180 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.618+0000 7fcce98a5700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcce4100e60 0x7fcce4073200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.618+0000 7fcce98a5700 1 --2- 172.21.15.96:0/3745648163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcce4101810 0x7fcce4073740 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.618+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 >> 172.21.15.96:0/3745648163 conn(0x7fcce40fc710 msgr2=0x7fcce40fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.619+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 shutdown_connections 2024-04-22T15:34:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.619+0000 7fcce98a5700 1 -- 172.21.15.96:0/3745648163 wait complete. 2024-04-22T15:34:17.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.722+0000 7fbe589fc700 1 -- 172.21.15.96:0/903529943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 msgr2=0x7fbe54105f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.722+0000 7fbe589fc700 1 --2- 172.21.15.96:0/903529943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 0x7fbe54105f50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00b0a0 tx=0x7fbe4c00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.722+0000 7fbe589fc700 1 -- 172.21.15.96:0/903529943 shutdown_connections 2024-04-22T15:34:17.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.722+0000 7fbe589fc700 1 --2- 172.21.15.96:0/903529943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe540ffae0 0x7fbe540fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.722+0000 7fbe589fc700 1 --2- 172.21.15.96:0/903529943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 0x7fbe54105f50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.722+0000 7fbe589fc700 1 -- 172.21.15.96:0/903529943 >> 172.21.15.96:0/903529943 conn(0x7fbe540fb610 msgr2=0x7fbe540fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.722+0000 7fbe589fc700 1 -- 172.21.15.96:0/903529943 shutdown_connections 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.722+0000 7fbe589fc700 1 -- 172.21.15.96:0/903529943 wait complete. 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe589fc700 1 Processor -- start 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe589fc700 1 -- start start 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe589fc700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe540ffae0 0x7fbe541957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe589fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 0x7fbe54195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe589fc700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe54196270 con 0x7fbe54105b70 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe589fc700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe541963e0 con 0x7fbe540ffae0 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe51d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 0x7fbe54195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:17.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe51d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 0x7fbe54195d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34540/0 (socket says 172.21.15.96:34540) 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe51d9b700 1 -- 172.21.15.96:0/1965695721 learned_addr learned my addr 172.21.15.96:0/1965695721 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe5259c700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe540ffae0 0x7fbe541957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe51d9b700 1 -- 172.21.15.96:0/1965695721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe540ffae0 msgr2=0x7fbe541957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe51d9b700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe540ffae0 0x7fbe541957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe51d9b700 1 -- 172.21.15.96:0/1965695721 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c009d00 con 0x7fbe54105b70 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe5259c700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe540ffae0 0x7fbe541957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.723+0000 7fbe51d9b700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 0x7fbe54195d30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800b4b0 tx=0x7fbe4800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.724+0000 7fbe437fe700 1 -- 172.21.15.96:0/1965695721 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe48011800 con 0x7fbe54105b70 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.724+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe541a2e60 con 0x7fbe54105b70 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.724+0000 7fbe437fe700 1 -- 172.21.15.96:0/1965695721 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe48011e20 con 0x7fbe54105b70 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.724+0000 7fbe437fe700 1 -- 172.21.15.96:0/1965695721 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe4800f2e0 con 0x7fbe54105b70 2024-04-22T15:34:17.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.724+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe541a31c0 con 0x7fbe54105b70 2024-04-22T15:34:17.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.725+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe541080e0 con 0x7fbe54105b70 2024-04-22T15:34:17.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.725+0000 7fbe437fe700 1 -- 172.21.15.96:0/1965695721 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbe48010380 con 0x7fbe54105b70 2024-04-22T15:34:17.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.725+0000 7fbe437fe700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3c06ecc0 0x7fbe3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:17.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.726+0000 7fbe437fe700 1 -- 172.21.15.96:0/1965695721 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbe4808ea20 con 0x7fbe54105b70 2024-04-22T15:34:17.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.726+0000 7fbe5259c700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3c06ecc0 0x7fbe3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:17.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.726+0000 7fbe5259c700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3c06ecc0 0x7fbe3c071180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c009480 tx=0x7fbe4c0097c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:17.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.730+0000 7fbe437fe700 1 -- 172.21.15.96:0/1965695721 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbe4805b2b0 con 0x7fbe54105b70 2024-04-22T15:34:17.734 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:17 smithi096 ceph-mon[112443]: pgmap v372: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:17 smithi196 ceph-mon[81815]: pgmap v372: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:17.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.977+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbe541011e0 con 0x7fbe54105b70 2024-04-22T15:34:17.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.977+0000 7fbe437fe700 1 -- 172.21.15.96:0/1965695721 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fbe4805ae40 con 0x7fbe54105b70 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:34:17.981 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:34:17.982 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:34:17.983 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:34:17.984 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:34:17.984 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:34:17.984 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3c06ecc0 msgr2=0x7fbe3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3c06ecc0 0x7fbe3c071180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c009480 tx=0x7fbe4c0097c0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 msgr2=0x7fbe54195d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 0x7fbe54195d30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800b4b0 tx=0x7fbe4800b870 comp rx=0 tx=0).stop 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 shutdown_connections 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3c06ecc0 0x7fbe3c071180 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe540ffae0 0x7fbe541957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 --2- 172.21.15.96:0/1965695721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe54105b70 0x7fbe54195d30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 >> 172.21.15.96:0/1965695721 conn(0x7fbe540fb610 msgr2=0x7fbe540fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 shutdown_connections 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:17.979+0000 7fbe589fc700 1 -- 172.21.15.96:0/1965695721 wait complete. 2024-04-22T15:34:17.985 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:34:18.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.085+0000 7f6089c89700 1 -- 172.21.15.96:0/4026612000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60841017d0 msgr2=0x7f6084105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:18.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.085+0000 7f6089c89700 1 --2- 172.21.15.96:0/4026612000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60841017d0 0x7f6084105820 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6078009910 tx=0x7f6078009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:18.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.085+0000 7f6089c89700 1 -- 172.21.15.96:0/4026612000 shutdown_connections 2024-04-22T15:34:18.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.085+0000 7f6089c89700 1 --2- 172.21.15.96:0/4026612000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60841017d0 0x7f6084105820 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.085+0000 7f6089c89700 1 --2- 172.21.15.96:0/4026612000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6084100e20 0x7f6084101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.085+0000 7f6089c89700 1 -- 172.21.15.96:0/4026612000 >> 172.21.15.96:0/4026612000 conn(0x7f60840fc6d0 msgr2=0x7f60840feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:18.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.085+0000 7f6089c89700 1 -- 172.21.15.96:0/4026612000 shutdown_connections 2024-04-22T15:34:18.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.085+0000 7f6089c89700 1 -- 172.21.15.96:0/4026612000 wait complete. 2024-04-22T15:34:18.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.086+0000 7f6089c89700 1 Processor -- start 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.086+0000 7f6089c89700 1 -- start start 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.086+0000 7f6089c89700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6084100e20 0x7f60840731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.086+0000 7f6089c89700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60841017d0 0x7f6084073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.086+0000 7f6089c89700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840751b0 con 0x7f6084100e20 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.086+0000 7f6089c89700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6084073c60 con 0x7f60841017d0 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.086+0000 7f60837fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6084100e20 0x7f60840731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f60837fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6084100e20 0x7f60840731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34554/0 (socket says 172.21.15.96:34554) 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f60837fe700 1 -- 172.21.15.96:0/3418720623 learned_addr learned my addr 172.21.15.96:0/3418720623 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f6082ffd700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60841017d0 0x7f6084073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f60837fe700 1 -- 172.21.15.96:0/3418720623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60841017d0 msgr2=0x7f6084073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f60837fe700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60841017d0 0x7f6084073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f60837fe700 1 -- 172.21.15.96:0/3418720623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60780095f0 con 0x7f6084100e20 2024-04-22T15:34:18.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f6082ffd700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60841017d0 0x7f6084073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:34:18.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f60837fe700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6084100e20 0x7f60840731e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f607400c930 tx=0x7f607400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:18.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f6080ff9700 1 -- 172.21.15.96:0/3418720623 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6074007ad0 con 0x7f6084100e20 2024-04-22T15:34:18.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6084073f10 con 0x7f6084100e20 2024-04-22T15:34:18.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f6080ff9700 1 -- 172.21.15.96:0/3418720623 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6074007c30 con 0x7f6084100e20 2024-04-22T15:34:18.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60841a2f60 con 0x7f6084100e20 2024-04-22T15:34:18.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.087+0000 7f6080ff9700 1 -- 172.21.15.96:0/3418720623 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6074018610 con 0x7f6084100e20 2024-04-22T15:34:18.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.088+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6084109220 con 0x7f6084100e20 2024-04-22T15:34:18.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.089+0000 7f6080ff9700 1 -- 172.21.15.96:0/3418720623 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6074018900 con 0x7f6084100e20 2024-04-22T15:34:18.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.089+0000 7f6080ff9700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f606c06ecc0 0x7f606c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:18.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.089+0000 7f6080ff9700 1 -- 172.21.15.96:0/3418720623 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f607408f0c0 con 0x7f6084100e20 2024-04-22T15:34:18.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.089+0000 7f6082ffd700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f606c06ecc0 0x7f606c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:18.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.090+0000 7f6082ffd700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f606c06ecc0 0x7f606c071180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6084074910 tx=0x7f607801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:18.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.094+0000 7f6080ff9700 1 -- 172.21.15.96:0/3418720623 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6074057e90 con 0x7f6084100e20 2024-04-22T15:34:18.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.313+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6084074710 con 0x7f606c06ecc0 2024-04-22T15:34:18.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.315+0000 7f6080ff9700 1 -- 172.21.15.96:0/3418720623 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6084074710 con 0x7f606c06ecc0 2024-04-22T15:34:18.318 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:34:18.318 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:34:18.318 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:34:18.318 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:34:18.318 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:34:18.318 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:34:18.318 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:34:18.318 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:34:18.319 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:34:18.319 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:34:18.319 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:34:18.319 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:34:18.319 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:34:18.319 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:34:18.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f606c06ecc0 msgr2=0x7f606c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:18.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f606c06ecc0 0x7f606c071180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6084074910 tx=0x7f607801a040 comp rx=0 tx=0).stop 2024-04-22T15:34:18.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6084100e20 msgr2=0x7f60840731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:18.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6084100e20 0x7f60840731e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f607400c930 tx=0x7f607400ccf0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 shutdown_connections 2024-04-22T15:34:18.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f606c06ecc0 0x7f606c071180 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6084100e20 0x7f60840731e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 --2- 172.21.15.96:0/3418720623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60841017d0 0x7f6084073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 >> 172.21.15.96:0/3418720623 conn(0x7f60840fc6d0 msgr2=0x7f60840fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:18.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 shutdown_connections 2024-04-22T15:34:18.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.317+0000 7f6089c89700 1 -- 172.21.15.96:0/3418720623 wait complete. 2024-04-22T15:34:18.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.421+0000 7fc21d05f700 1 -- 172.21.15.96:0/1071362399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2180ff090 msgr2=0x7fc2180ff510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:18.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.421+0000 7fc21d05f700 1 --2- 172.21.15.96:0/1071362399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2180ff090 0x7fc2180ff510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc20c009910 tx=0x7fc20c009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:18.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.421+0000 7fc21d05f700 1 -- 172.21.15.96:0/1071362399 shutdown_connections 2024-04-22T15:34:18.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.421+0000 7fc21d05f700 1 --2- 172.21.15.96:0/1071362399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2180ff090 0x7fc2180ff510 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.421+0000 7fc21d05f700 1 --2- 172.21.15.96:0/1071362399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc218105c70 0x7fc218106050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.421+0000 7fc21d05f700 1 -- 172.21.15.96:0/1071362399 >> 172.21.15.96:0/1071362399 conn(0x7fc218075f00 msgr2=0x7fc2180fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:18.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.421+0000 7fc21d05f700 1 -- 172.21.15.96:0/1071362399 shutdown_connections 2024-04-22T15:34:18.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.421+0000 7fc21d05f700 1 -- 172.21.15.96:0/1071362399 wait complete. 2024-04-22T15:34:18.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.422+0000 7fc21d05f700 1 Processor -- start 2024-04-22T15:34:18.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.422+0000 7fc21d05f700 1 -- start start 2024-04-22T15:34:18.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21d05f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2180ff090 0x7fc21819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:18.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21d05f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc218105c70 0x7fc21819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:18.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21d05f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21819d4a0 con 0x7fc218105c70 2024-04-22T15:34:18.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21d05f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc218196a80 con 0x7fc2180ff090 2024-04-22T15:34:18.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc218105c70 0x7fc21819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:18.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc218105c70 0x7fc21819ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34572/0 (socket says 172.21.15.96:34572) 2024-04-22T15:34:18.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21659c700 1 -- 172.21.15.96:0/371145529 learned_addr learned my addr 172.21.15.96:0/371145529 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:18.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc216d9d700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2180ff090 0x7fc21819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:18.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21659c700 1 -- 172.21.15.96:0/371145529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2180ff090 msgr2=0x7fc21819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:18.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21659c700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2180ff090 0x7fc21819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21659c700 1 -- 172.21.15.96:0/371145529 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c0095f0 con 0x7fc218105c70 2024-04-22T15:34:18.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc21659c700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc218105c70 0x7fc21819ce80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00bd70 tx=0x7fc20c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:18.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.423+0000 7fc1fffff700 1 -- 172.21.15.96:0/371145529 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc20c01d070 con 0x7fc218105c70 2024-04-22T15:34:18.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.424+0000 7fc1fffff700 1 -- 172.21.15.96:0/371145529 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc20c005700 con 0x7fc218105c70 2024-04-22T15:34:18.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.424+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc218196d00 con 0x7fc218105c70 2024-04-22T15:34:18.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.424+0000 7fc1fffff700 1 -- 172.21.15.96:0/371145529 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc20c017400 con 0x7fc218105c70 2024-04-22T15:34:18.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.424+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2181971f0 con 0x7fc218105c70 2024-04-22T15:34:18.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.425+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc218109240 con 0x7fc218105c70 2024-04-22T15:34:18.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.425+0000 7fc1fffff700 1 -- 172.21.15.96:0/371145529 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc20c017670 con 0x7fc218105c70 2024-04-22T15:34:18.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.425+0000 7fc1fffff700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc20006ecc0 0x7fc200071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:18.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.426+0000 7fc1fffff700 1 -- 172.21.15.96:0/371145529 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc20c090360 con 0x7fc218105c70 2024-04-22T15:34:18.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.426+0000 7fc216d9d700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc20006ecc0 0x7fc200071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:18.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.426+0000 7fc216d9d700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc20006ecc0 0x7fc200071180 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc208009a10 tx=0x7fc208009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:18.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.430+0000 7fc1fffff700 1 -- 172.21.15.96:0/371145529 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc20c05cc60 con 0x7fc218105c70 2024-04-22T15:34:18.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.718+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc218197f40 con 0x7fc218105c70 2024-04-22T15:34:18.722 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:18 smithi096 ceph-mon[112443]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:18.722 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:18 smithi096 ceph-mon[112443]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:18.722 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:18 smithi096 ceph-mon[112443]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:18.722 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:18 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3745648163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:34:18.723 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:18 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1965695721' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:34:18.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.719+0000 7fc1fffff700 1 -- 172.21.15.96:0/371145529 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fc20c05c7f0 con 0x7fc218105c70 2024-04-22T15:34:18.723 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:34:18.723 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:34:18.723 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:34:18.723 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:34:18.723 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:34:18.724 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:34:18.724 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:34:18.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc20006ecc0 msgr2=0x7fc200071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:18.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc20006ecc0 0x7fc200071180 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc208009a10 tx=0x7fc208009190 comp rx=0 tx=0).stop 2024-04-22T15:34:18.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc218105c70 msgr2=0x7fc21819ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:18.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc218105c70 0x7fc21819ce80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00bd70 tx=0x7fc20c00bda0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 shutdown_connections 2024-04-22T15:34:18.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc20006ecc0 0x7fc200071180 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2180ff090 0x7fc21819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 --2- 172.21.15.96:0/371145529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc218105c70 0x7fc21819ce80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:18.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 >> 172.21.15.96:0/371145529 conn(0x7fc218075f00 msgr2=0x7fc2180fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:18.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 shutdown_connections 2024-04-22T15:34:18.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:18.721+0000 7fc21d05f700 1 -- 172.21.15.96:0/371145529 wait complete. 2024-04-22T15:34:18.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:18 smithi196 ceph-mon[81815]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:18.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:18 smithi196 ceph-mon[81815]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:18 smithi196 ceph-mon[81815]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:18 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3745648163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:34:18.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:18 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1965695721' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:34:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:19 smithi196 ceph-mon[81815]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:19 smithi196 ceph-mon[81815]: pgmap v373: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:19.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:19 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/371145529' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:34:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:19 smithi096 ceph-mon[112443]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:19 smithi096 ceph-mon[112443]: pgmap v373: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:19 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/371145529' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:34:21.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:21 smithi196 ceph-mon[81815]: pgmap v374: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:21.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:21 smithi096 ceph-mon[112443]: pgmap v374: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:23 smithi196 ceph-mon[81815]: pgmap v375: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:34:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:23 smithi096 ceph-mon[112443]: pgmap v375: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:34:25.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:25 smithi196 ceph-mon[81815]: pgmap v376: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:25.952 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:25 smithi096 ceph-mon[112443]: pgmap v376: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:27.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:27 smithi196 ceph-mon[81815]: pgmap v377: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:27.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:27 smithi096 ceph-mon[112443]: pgmap v377: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:29.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:29 smithi196 ceph-mon[81815]: pgmap v378: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:29 smithi096 ceph-mon[112443]: pgmap v378: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:31 smithi196 ceph-mon[81815]: pgmap v379: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:31 smithi096 ceph-mon[112443]: pgmap v379: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:33 smithi196 ceph-mon[81815]: pgmap v380: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:33.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:33 smithi096 ceph-mon[112443]: pgmap v380: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:35.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:35 smithi096 ceph-mon[112443]: pgmap v381: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:35 smithi196 ceph-mon[81815]: pgmap v381: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:37 smithi096 ceph-mon[112443]: pgmap v382: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:37 smithi196 ceph-mon[81815]: pgmap v382: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:38.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:34:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:34:39.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:39 smithi096 ceph-mon[112443]: pgmap v383: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:39 smithi196 ceph-mon[81815]: pgmap v383: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:40 smithi096 ceph-mon[112443]: pgmap v384: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:41.022 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:40 smithi196 ceph-mon[81815]: pgmap v384: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:43 smithi096 ceph-mon[112443]: pgmap v385: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:43 smithi196 ceph-mon[81815]: pgmap v385: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:45.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:45 smithi196 ceph-mon[81815]: pgmap v386: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:45 smithi096 ceph-mon[112443]: pgmap v386: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:47 smithi096 ceph-mon[112443]: pgmap v387: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:47 smithi196 ceph-mon[81815]: pgmap v387: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:48.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.834+0000 7f2db2e26700 1 -- 172.21.15.96:0/617578855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac100df0 msgr2=0x7f2dac1011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:48.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.834+0000 7f2db2e26700 1 --2- 172.21.15.96:0/617578855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac100df0 0x7f2dac1011d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009910 tx=0x7f2da0009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:48.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 -- 172.21.15.96:0/617578855 shutdown_connections 2024-04-22T15:34:48.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 --2- 172.21.15.96:0/617578855 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2dac1017a0 0x7f2dac1057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:48.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 --2- 172.21.15.96:0/617578855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac100df0 0x7f2dac1011d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:48.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 -- 172.21.15.96:0/617578855 >> 172.21.15.96:0/617578855 conn(0x7f2dac0fc6a0 msgr2=0x7f2dac0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:48.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 -- 172.21.15.96:0/617578855 shutdown_connections 2024-04-22T15:34:48.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 -- 172.21.15.96:0/617578855 wait complete. 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 Processor -- start 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 -- start start 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2dac100df0 0x7f2dac19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac1017a0 0x7f2dac19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac19d420 con 0x7f2dac1017a0 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.835+0000 7f2db2e26700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac196a50 con 0x7f2dac100df0 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2dabfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac1017a0 0x7f2dac19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2dabfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac1017a0 0x7f2dac19ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47734/0 (socket says 172.21.15.96:47734) 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2dabfff700 1 -- 172.21.15.96:0/2008086397 learned_addr learned my addr 172.21.15.96:0/2008086397 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2db0bc2700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2dac100df0 0x7f2dac19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2dabfff700 1 -- 172.21.15.96:0/2008086397 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2dac100df0 msgr2=0x7f2dac19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2dabfff700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2dac100df0 0x7f2dac19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:48.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2dabfff700 1 -- 172.21.15.96:0/2008086397 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da00095f0 con 0x7f2dac1017a0 2024-04-22T15:34:48.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2db0bc2700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2dac100df0 0x7f2dac19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:34:48.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2dabfff700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac1017a0 0x7f2dac19ce00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800b480 tx=0x7f2d9800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:48.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2da9ffb700 1 -- 172.21.15.96:0/2008086397 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d980118c0 con 0x7f2dac1017a0 2024-04-22T15:34:48.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac196d30 con 0x7f2dac1017a0 2024-04-22T15:34:48.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2da9ffb700 1 -- 172.21.15.96:0/2008086397 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d98010380 con 0x7f2dac1017a0 2024-04-22T15:34:48.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2da9ffb700 1 -- 172.21.15.96:0/2008086397 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d9800f4f0 con 0x7f2dac1017a0 2024-04-22T15:34:48.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.836+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac197280 con 0x7f2dac1017a0 2024-04-22T15:34:48.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.837+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac109190 con 0x7f2dac1017a0 2024-04-22T15:34:48.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.838+0000 7f2da9ffb700 1 -- 172.21.15.96:0/2008086397 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2d980104f0 con 0x7f2dac1017a0 2024-04-22T15:34:48.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.838+0000 7f2da9ffb700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2d9c06ecc0 0x7f2d9c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:48.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.838+0000 7f2da9ffb700 1 -- 172.21.15.96:0/2008086397 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2d9808eae0 con 0x7f2dac1017a0 2024-04-22T15:34:48.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.838+0000 7f2db0bc2700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2d9c06ecc0 0x7f2d9c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:48.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.839+0000 7f2db0bc2700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2d9c06ecc0 0x7f2d9c071180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2da0006010 tx=0x7f2da000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:48.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:48.843+0000 7f2da9ffb700 1 -- 172.21.15.96:0/2008086397 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2d9805b370 con 0x7f2dac1017a0 2024-04-22T15:34:49.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.060+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2dac197f30 con 0x7f2d9c06ecc0 2024-04-22T15:34:49.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.061+0000 7f2da9ffb700 1 -- 172.21.15.96:0/2008086397 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2dac197f30 con 0x7f2d9c06ecc0 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2d9c06ecc0 msgr2=0x7f2d9c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2d9c06ecc0 0x7f2d9c071180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2da0006010 tx=0x7f2da000b540 comp rx=0 tx=0).stop 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac1017a0 msgr2=0x7f2dac19ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac1017a0 0x7f2dac19ce00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800b480 tx=0x7f2d9800b840 comp rx=0 tx=0).stop 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 shutdown_connections 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2d9c06ecc0 0x7f2d9c071180 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2dac100df0 0x7f2dac19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 --2- 172.21.15.96:0/2008086397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac1017a0 0x7f2dac19ce00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 >> 172.21.15.96:0/2008086397 conn(0x7f2dac0fc6a0 msgr2=0x7f2dac0fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:49.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 shutdown_connections 2024-04-22T15:34:49.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.063+0000 7f2db2e26700 1 -- 172.21.15.96:0/2008086397 wait complete. 2024-04-22T15:34:49.077 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:34:49.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.168+0000 7f78b8c67700 1 -- 172.21.15.96:0/3087879407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 msgr2=0x7f78b40fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.168+0000 7f78b8c67700 1 --2- 172.21.15.96:0/3087879407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 0x7f78b40fff20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009910 tx=0x7f78a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:49.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.168+0000 7f78b8c67700 1 -- 172.21.15.96:0/3087879407 shutdown_connections 2024-04-22T15:34:49.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.168+0000 7f78b8c67700 1 --2- 172.21.15.96:0/3087879407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 0x7f78b40fff20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.168+0000 7f78b8c67700 1 --2- 172.21.15.96:0/3087879407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78b4105aa0 0x7f78b4105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.168+0000 7f78b8c67700 1 -- 172.21.15.96:0/3087879407 >> 172.21.15.96:0/3087879407 conn(0x7f78b4076740 msgr2=0x7f78b40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:49.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.168+0000 7f78b8c67700 1 -- 172.21.15.96:0/3087879407 shutdown_connections 2024-04-22T15:34:49.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.168+0000 7f78b8c67700 1 -- 172.21.15.96:0/3087879407 wait complete. 2024-04-22T15:34:49.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b8c67700 1 Processor -- start 2024-04-22T15:34:49.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b8c67700 1 -- start start 2024-04-22T15:34:49.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b8c67700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 0x7f78b419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:49.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b8c67700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78b4105aa0 0x7f78b419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:49.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b8c67700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b41968d0 con 0x7f78b40ffaa0 2024-04-22T15:34:49.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b8c67700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b4196a40 con 0x7f78b4105aa0 2024-04-22T15:34:49.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 0x7f78b419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 0x7f78b419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47762/0 (socket says 172.21.15.96:47762) 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b259c700 1 -- 172.21.15.96:0/4062101272 learned_addr learned my addr 172.21.15.96:0/4062101272 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.169+0000 7f78b1d9b700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78b4105aa0 0x7f78b419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.170+0000 7f78b259c700 1 -- 172.21.15.96:0/4062101272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78b4105aa0 msgr2=0x7f78b419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.170+0000 7f78b259c700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78b4105aa0 0x7f78b419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.170+0000 7f78b259c700 1 -- 172.21.15.96:0/4062101272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a80095f0 con 0x7f78b40ffaa0 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.170+0000 7f78b259c700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 0x7f78b419c820 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f78a400c8a0 tx=0x7f78a400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.170+0000 7f78a37fe700 1 -- 172.21.15.96:0/4062101272 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a40078e0 con 0x7f78b40ffaa0 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.170+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b4196d20 con 0x7f78b40ffaa0 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.170+0000 7f78a37fe700 1 -- 172.21.15.96:0/4062101272 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78a400f450 con 0x7f78b40ffaa0 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.170+0000 7f78a37fe700 1 -- 172.21.15.96:0/4062101272 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a400e5e0 con 0x7f78b40ffaa0 2024-04-22T15:34:49.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.170+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b4197270 con 0x7f78b40ffaa0 2024-04-22T15:34:49.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.171+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b41011e0 con 0x7f78b40ffaa0 2024-04-22T15:34:49.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.172+0000 7f78a37fe700 1 -- 172.21.15.96:0/4062101272 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f78a400f5c0 con 0x7f78b40ffaa0 2024-04-22T15:34:49.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.172+0000 7f78a37fe700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f789c06ecc0 0x7f789c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:49.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.172+0000 7f78a37fe700 1 -- 172.21.15.96:0/4062101272 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f78a408eed0 con 0x7f78b40ffaa0 2024-04-22T15:34:49.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.172+0000 7f78b1d9b700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f789c06ecc0 0x7f789c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:49.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.173+0000 7f78b1d9b700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f789c06ecc0 0x7f789c071180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f78b4198040 tx=0x7f78a801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:49.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.179+0000 7f78a37fe700 1 -- 172.21.15.96:0/4062101272 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f78a4057c30 con 0x7f78b40ffaa0 2024-04-22T15:34:49.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.396+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f78b4197540 con 0x7f789c06ecc0 2024-04-22T15:34:49.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.397+0000 7f78a37fe700 1 -- 172.21.15.96:0/4062101272 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f78b4197540 con 0x7f789c06ecc0 2024-04-22T15:34:49.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f789c06ecc0 msgr2=0x7f789c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f789c06ecc0 0x7f789c071180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f78b4198040 tx=0x7f78a801a040 comp rx=0 tx=0).stop 2024-04-22T15:34:49.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 msgr2=0x7f78b419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 0x7f78b419c820 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f78a400c8a0 tx=0x7f78a400cbb0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 shutdown_connections 2024-04-22T15:34:49.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f789c06ecc0 0x7f789c071180 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78b40ffaa0 0x7f78b419c820 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 --2- 172.21.15.96:0/4062101272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78b4105aa0 0x7f78b419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 >> 172.21.15.96:0/4062101272 conn(0x7f78b4076740 msgr2=0x7f78b40687a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:49.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 shutdown_connections 2024-04-22T15:34:49.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.399+0000 7f78b8c67700 1 -- 172.21.15.96:0/4062101272 wait complete. 2024-04-22T15:34:49.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.505+0000 7ff080d19700 1 -- 172.21.15.96:0/2531130059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 msgr2=0x7ff07c105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.505+0000 7ff080d19700 1 --2- 172.21.15.96:0/2531130059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 0x7ff07c105f40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009910 tx=0x7ff06c009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.505+0000 7ff080d19700 1 -- 172.21.15.96:0/2531130059 shutdown_connections 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.505+0000 7ff080d19700 1 --2- 172.21.15.96:0/2531130059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff07c0ffad0 0x7ff07c0fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.505+0000 7ff080d19700 1 --2- 172.21.15.96:0/2531130059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 0x7ff07c105f40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.505+0000 7ff080d19700 1 -- 172.21.15.96:0/2531130059 >> 172.21.15.96:0/2531130059 conn(0x7ff07c0fb600 msgr2=0x7ff07c0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.505+0000 7ff080d19700 1 -- 172.21.15.96:0/2531130059 shutdown_connections 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.505+0000 7ff080d19700 1 -- 172.21.15.96:0/2531130059 wait complete. 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.505+0000 7ff080d19700 1 Processor -- start 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff080d19700 1 -- start start 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff080d19700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff07c0ffad0 0x7ff07c101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff080d19700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 0x7ff07c101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff080d19700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c1021b0 con 0x7ff07c105b60 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff080d19700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c1022f0 con 0x7ff07c0ffad0 2024-04-22T15:34:49.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff079d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 0x7ff07c101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff079d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 0x7ff07c101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47770/0 (socket says 172.21.15.96:47770) 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff079d9b700 1 -- 172.21.15.96:0/2656475536 learned_addr learned my addr 172.21.15.96:0/2656475536 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff07a59c700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff07c0ffad0 0x7ff07c101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff079d9b700 1 -- 172.21.15.96:0/2656475536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff07c0ffad0 msgr2=0x7ff07c101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff079d9b700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff07c0ffad0 0x7ff07c101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff079d9b700 1 -- 172.21.15.96:0/2656475536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c0095f0 con 0x7ff07c105b60 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.506+0000 7ff079d9b700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 0x7ff07c101c70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff07000d8d0 tx=0x7ff07000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.507+0000 7ff06b7fe700 1 -- 172.21.15.96:0/2656475536 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0700098a0 con 0x7ff07c105b60 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.507+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c1a2ca0 con 0x7ff07c105b60 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.507+0000 7ff06b7fe700 1 -- 172.21.15.96:0/2656475536 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff070010460 con 0x7ff07c105b60 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.507+0000 7ff06b7fe700 1 -- 172.21.15.96:0/2656475536 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff07000f5f0 con 0x7ff07c105b60 2024-04-22T15:34:49.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.507+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c1a3000 con 0x7ff07c105b60 2024-04-22T15:34:49.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.508+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c1011d0 con 0x7ff07c105b60 2024-04-22T15:34:49.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.508+0000 7ff06b7fe700 1 -- 172.21.15.96:0/2656475536 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff07000f750 con 0x7ff07c105b60 2024-04-22T15:34:49.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.508+0000 7ff06b7fe700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff06406ecc0 0x7ff064071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:49.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.508+0000 7ff06b7fe700 1 -- 172.21.15.96:0/2656475536 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff07008ed10 con 0x7ff07c105b60 2024-04-22T15:34:49.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.508+0000 7ff07a59c700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff06406ecc0 0x7ff064071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:49.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.509+0000 7ff07a59c700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff06406ecc0 0x7ff064071180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006010 tx=0x7ff06c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:49.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.513+0000 7ff06b7fe700 1 -- 172.21.15.96:0/2656475536 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff070057b60 con 0x7ff07c105b60 2024-04-22T15:34:49.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.723+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff07c10cde0 con 0x7ff06406ecc0 2024-04-22T15:34:49.727 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:49 smithi096 ceph-mon[112443]: pgmap v388: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:49.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.734+0000 7ff06b7fe700 1 -- 172.21.15.96:0/2656475536 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff07c10cde0 con 0x7ff06406ecc0 2024-04-22T15:34:49.737 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:34:49.737 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (13m) 6m ago 24m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:34:49.737 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (25m) 6m ago 25m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (23m) 6m ago 23m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (10m) 6m ago 25m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (10m) 6m ago 23m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (13m) 6m ago 24m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (6m) 6m ago 19m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (7m) 6m ago 19m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (6m) 6m ago 19m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (19m) 6m ago 19m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (15m) 6m ago 26m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (14m) 6m ago 23m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (11m) 6m ago 26m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (11m) 6m ago 23m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:34:49.738 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (14m) 6m ago 24m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:34:49.739 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (14m) 6m ago 23m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:34:49.739 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (10m) 6m ago 22m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:34:49.739 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (9m) 6m ago 22m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:34:49.739 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (9m) 6m ago 21m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:34:49.739 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (8m) 6m ago 21m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:34:49.739 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (7m) 6m ago 21m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:34:49.739 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (7m) 6m ago 20m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:34:49.739 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (13m) 6m ago 24m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:34:49.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff06406ecc0 msgr2=0x7ff064071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff06406ecc0 0x7ff064071180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006010 tx=0x7ff06c01a040 comp rx=0 tx=0).stop 2024-04-22T15:34:49.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 msgr2=0x7ff07c101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 0x7ff07c101c70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff07000d8d0 tx=0x7ff07000dbe0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 shutdown_connections 2024-04-22T15:34:49.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff06406ecc0 0x7ff064071180 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff07c0ffad0 0x7ff07c101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 --2- 172.21.15.96:0/2656475536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff07c105b60 0x7ff07c101c70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 >> 172.21.15.96:0/2656475536 conn(0x7ff07c0fb600 msgr2=0x7ff07c103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:49.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 shutdown_connections 2024-04-22T15:34:49.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.737+0000 7ff080d19700 1 -- 172.21.15.96:0/2656475536 wait complete. 2024-04-22T15:34:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:49 smithi196 ceph-mon[81815]: pgmap v388: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.839+0000 7feb1e10f700 1 -- 172.21.15.96:0/3011008823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb181026e0 msgr2=0x7feb1810abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.839+0000 7feb1e10f700 1 --2- 172.21.15.96:0/3011008823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb181026e0 0x7feb1810abd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7feb00009940 tx=0x7feb00009c50 comp rx=0 tx=0).stop 2024-04-22T15:34:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.839+0000 7feb1e10f700 1 -- 172.21.15.96:0/3011008823 shutdown_connections 2024-04-22T15:34:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.839+0000 7feb1e10f700 1 --2- 172.21.15.96:0/3011008823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb181026e0 0x7feb1810abd0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.839+0000 7feb1e10f700 1 --2- 172.21.15.96:0/3011008823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb18101dc0 0x7feb181021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.839+0000 7feb1e10f700 1 -- 172.21.15.96:0/3011008823 >> 172.21.15.96:0/3011008823 conn(0x7feb18076740 msgr2=0x7feb180fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.839+0000 7feb1e10f700 1 -- 172.21.15.96:0/3011008823 shutdown_connections 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.839+0000 7feb1e10f700 1 -- 172.21.15.96:0/3011008823 wait complete. 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb1e10f700 1 Processor -- start 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb1e10f700 1 -- start start 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb1e10f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb18101dc0 0x7feb1819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb1e10f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb181026e0 0x7feb1819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb1e10f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1819d2f0 con 0x7feb18101dc0 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb1e10f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb18196960 con 0x7feb181026e0 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb177fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb18101dc0 0x7feb1819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb177fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb18101dc0 0x7feb1819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47780/0 (socket says 172.21.15.96:47780) 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb177fe700 1 -- 172.21.15.96:0/1075723839 learned_addr learned my addr 172.21.15.96:0/1075723839 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb0ffff700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb181026e0 0x7feb1819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb177fe700 1 -- 172.21.15.96:0/1075723839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb181026e0 msgr2=0x7feb1819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb177fe700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb181026e0 0x7feb1819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:49.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.840+0000 7feb177fe700 1 -- 172.21.15.96:0/1075723839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb000095f0 con 0x7feb18101dc0 2024-04-22T15:34:49.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.841+0000 7feb177fe700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb18101dc0 0x7feb1819c820 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7feb0800c930 tx=0x7feb0800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:49.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.841+0000 7feb157fa700 1 -- 172.21.15.96:0/1075723839 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb08007ad0 con 0x7feb18101dc0 2024-04-22T15:34:49.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.841+0000 7feb157fa700 1 -- 172.21.15.96:0/1075723839 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb08007c30 con 0x7feb18101dc0 2024-04-22T15:34:49.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.841+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb18196c40 con 0x7feb18101dc0 2024-04-22T15:34:49.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.841+0000 7feb157fa700 1 -- 172.21.15.96:0/1075723839 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb08018610 con 0x7feb18101dc0 2024-04-22T15:34:49.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.841+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb18197160 con 0x7feb18101dc0 2024-04-22T15:34:49.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.842+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb181054e0 con 0x7feb18101dc0 2024-04-22T15:34:49.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.842+0000 7feb157fa700 1 -- 172.21.15.96:0/1075723839 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feb0800f450 con 0x7feb18101dc0 2024-04-22T15:34:49.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.843+0000 7feb157fa700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb0406ecc0 0x7feb04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:49.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.843+0000 7feb157fa700 1 -- 172.21.15.96:0/1075723839 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7feb0808f0d0 con 0x7feb18101dc0 2024-04-22T15:34:49.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.843+0000 7feb0ffff700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb0406ecc0 0x7feb04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:49.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.843+0000 7feb0ffff700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb0406ecc0 0x7feb04071180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb00009910 tx=0x7feb00005ea0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:49.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:49.848+0000 7feb157fa700 1 -- 172.21.15.96:0/1075723839 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7feb0805b960 con 0x7feb18101dc0 2024-04-22T15:34:50.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.134+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feb18197e20 con 0x7feb18101dc0 2024-04-22T15:34:50.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.134+0000 7feb157fa700 1 -- 172.21.15.96:0/1075723839 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7feb0805b4f0 con 0x7feb18101dc0 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:34:50.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:34:50.139 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:34:50.139 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:34:50.139 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:34:50.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:34:50.139 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:34:50.139 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:34:50.139 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:34:50.139 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:34:50.139 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:34:50.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.136+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb0406ecc0 msgr2=0x7feb04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.136+0000 7feb1e10f700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb0406ecc0 0x7feb04071180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb00009910 tx=0x7feb00005ea0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.136+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb18101dc0 msgr2=0x7feb1819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.136+0000 7feb1e10f700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb18101dc0 0x7feb1819c820 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7feb0800c930 tx=0x7feb0800ccf0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.136+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 shutdown_connections 2024-04-22T15:34:50.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.136+0000 7feb1e10f700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb0406ecc0 0x7feb04071180 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.136+0000 7feb1e10f700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb18101dc0 0x7feb1819c820 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.136+0000 7feb1e10f700 1 --2- 172.21.15.96:0/1075723839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb181026e0 0x7feb1819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.136+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 >> 172.21.15.96:0/1075723839 conn(0x7feb18076740 msgr2=0x7feb181030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:50.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.137+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 shutdown_connections 2024-04-22T15:34:50.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.137+0000 7feb1e10f700 1 -- 172.21.15.96:0/1075723839 wait complete. 2024-04-22T15:34:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 -- 172.21.15.96:0/278868630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 msgr2=0x7f55f0105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 --2- 172.21.15.96:0/278868630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 0x7f55f0105e90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009910 tx=0x7f55e0009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 -- 172.21.15.96:0/278868630 shutdown_connections 2024-04-22T15:34:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 --2- 172.21.15.96:0/278868630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f00ffab0 0x7f55f00fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 --2- 172.21.15.96:0/278868630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 0x7f55f0105e90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 -- 172.21.15.96:0/278868630 >> 172.21.15.96:0/278868630 conn(0x7f55f00fb5e0 msgr2=0x7f55f00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 -- 172.21.15.96:0/278868630 shutdown_connections 2024-04-22T15:34:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 -- 172.21.15.96:0/278868630 wait complete. 2024-04-22T15:34:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 Processor -- start 2024-04-22T15:34:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.240+0000 7f55f6626700 1 -- start start 2024-04-22T15:34:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55f6626700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f00ffab0 0x7f55f0195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55f6626700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 0x7f55f0195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55f6626700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f01965f0 con 0x7f55f0105ab0 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55f6626700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f019a340 con 0x7f55f00ffab0 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55ef7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 0x7f55f0195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55ef7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 0x7f55f0195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47796/0 (socket says 172.21.15.96:47796) 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55ef7fe700 1 -- 172.21.15.96:0/3086763901 learned_addr learned my addr 172.21.15.96:0/3086763901 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55ef7fe700 1 -- 172.21.15.96:0/3086763901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f00ffab0 msgr2=0x7f55f0195a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55ef7fe700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f00ffab0 0x7f55f0195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55ef7fe700 1 -- 172.21.15.96:0/3086763901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e00095f0 con 0x7f55f0105ab0 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55ef7fe700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 0x7f55f0195f60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f55e400d8d0 tx=0x7f55e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.241+0000 7f55ed7fa700 1 -- 172.21.15.96:0/3086763901 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55e4009940 con 0x7f55f0105ab0 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.242+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f019a620 con 0x7f55f0105ab0 2024-04-22T15:34:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.242+0000 7f55ed7fa700 1 -- 172.21.15.96:0/3086763901 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55e4010460 con 0x7f55f0105ab0 2024-04-22T15:34:50.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.242+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f019ab70 con 0x7f55f0105ab0 2024-04-22T15:34:50.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.242+0000 7f55ed7fa700 1 -- 172.21.15.96:0/3086763901 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55e400f5f0 con 0x7f55f0105ab0 2024-04-22T15:34:50.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.242+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc0052f0 con 0x7f55f0105ab0 2024-04-22T15:34:50.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.243+0000 7f55ed7fa700 1 -- 172.21.15.96:0/3086763901 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f55e4010ad0 con 0x7f55f0105ab0 2024-04-22T15:34:50.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.243+0000 7f55ed7fa700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d806ecd0 0x7f55d8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:50.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.244+0000 7f55effff700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d806ecd0 0x7f55d8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:50.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.244+0000 7f55ed7fa700 1 -- 172.21.15.96:0/3086763901 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f55e408ed10 con 0x7f55f0105ab0 2024-04-22T15:34:50.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.244+0000 7f55effff700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d806ecd0 0x7f55d8071190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f55e0006010 tx=0x7f55e000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:50.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.248+0000 7f55ed7fa700 1 -- 172.21.15.96:0/3086763901 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f55e405b620 con 0x7f55f0105ab0 2024-04-22T15:34:50.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.495+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f55dc005db0 con 0x7f55f0105ab0 2024-04-22T15:34:50.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.495+0000 7f55ed7fa700 1 -- 172.21.15.96:0/3086763901 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f55e4016770 con 0x7f55f0105ab0 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:34:50.499 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:34:50.500 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:34:50.501 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:34:50.502 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:34:50.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d806ecd0 msgr2=0x7f55d8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d806ecd0 0x7f55d8071190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f55e0006010 tx=0x7f55e000b540 comp rx=0 tx=0).stop 2024-04-22T15:34:50.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 msgr2=0x7f55f0195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 0x7f55f0195f60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f55e400d8d0 tx=0x7f55e400dc90 comp rx=0 tx=0).stop 2024-04-22T15:34:50.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 shutdown_connections 2024-04-22T15:34:50.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d806ecd0 0x7f55d8071190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f00ffab0 0x7f55f0195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 --2- 172.21.15.96:0/3086763901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55f0105ab0 0x7f55f0195f60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 >> 172.21.15.96:0/3086763901 conn(0x7f55f00fb5e0 msgr2=0x7f55f00fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:50.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 shutdown_connections 2024-04-22T15:34:50.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.498+0000 7f55f6626700 1 -- 172.21.15.96:0/3086763901 wait complete. 2024-04-22T15:34:50.503 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:34:50.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.603+0000 7fdfde115700 1 -- 172.21.15.96:0/1094475096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 msgr2=0x7fdfd8100880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.603+0000 7fdfde115700 1 --2- 172.21.15.96:0/1094475096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 0x7fdfd8100880 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009910 tx=0x7fdfc8009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:50.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.603+0000 7fdfde115700 1 -- 172.21.15.96:0/1094475096 shutdown_connections 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.603+0000 7fdfde115700 1 --2- 172.21.15.96:0/1094475096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd8100e50 0x7fdfd8104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.603+0000 7fdfde115700 1 --2- 172.21.15.96:0/1094475096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 0x7fdfd8100880 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.603+0000 7fdfde115700 1 -- 172.21.15.96:0/1094475096 >> 172.21.15.96:0/1094475096 conn(0x7fdfd8075f40 msgr2=0x7fdfd80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.603+0000 7fdfde115700 1 -- 172.21.15.96:0/1094475096 shutdown_connections 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.603+0000 7fdfde115700 1 -- 172.21.15.96:0/1094475096 wait complete. 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.604+0000 7fdfde115700 1 Processor -- start 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.604+0000 7fdfde115700 1 -- start start 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.604+0000 7fdfde115700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 0x7fdfd8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.604+0000 7fdfde115700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd8100e50 0x7fdfd81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.604+0000 7fdfde115700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd8198680 con 0x7fdfd81004a0 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.604+0000 7fdfde115700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd81987c0 con 0x7fdfd8100e50 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.604+0000 7fdfd77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 0x7fdfd8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:50.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.604+0000 7fdfd77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 0x7fdfd8197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40314/0 (socket says 172.21.15.96:40314) 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.604+0000 7fdfd77fe700 1 -- 172.21.15.96:0/2081009177 learned_addr learned my addr 172.21.15.96:0/2081009177 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfd6ffd700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd8100e50 0x7fdfd81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfd77fe700 1 -- 172.21.15.96:0/2081009177 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd8100e50 msgr2=0x7fdfd81980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfd77fe700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd8100e50 0x7fdfd81980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfd77fe700 1 -- 172.21.15.96:0/2081009177 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc80095f0 con 0x7fdfd81004a0 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfd77fe700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 0x7fdfd8197b70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800bd70 tx=0x7fdfc800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfd4ff9700 1 -- 172.21.15.96:0/2081009177 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc801d070 con 0x7fdfd81004a0 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd81a0be0 con 0x7fdfd81004a0 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfd4ff9700 1 -- 172.21.15.96:0/2081009177 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfc80056e0 con 0x7fdfd81004a0 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfd4ff9700 1 -- 172.21.15.96:0/2081009177 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc8017400 con 0x7fdfd81004a0 2024-04-22T15:34:50.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.605+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd81a0f40 con 0x7fdfd81004a0 2024-04-22T15:34:50.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.606+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd8108320 con 0x7fdfd81004a0 2024-04-22T15:34:50.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.606+0000 7fdfd4ff9700 1 -- 172.21.15.96:0/2081009177 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdfc8003a60 con 0x7fdfd81004a0 2024-04-22T15:34:50.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.607+0000 7fdfd4ff9700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdfc006ecc0 0x7fdfc0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:50.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.607+0000 7fdfd4ff9700 1 -- 172.21.15.96:0/2081009177 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fdfc80902f0 con 0x7fdfd81004a0 2024-04-22T15:34:50.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.607+0000 7fdfd6ffd700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdfc006ecc0 0x7fdfc0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:50.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.607+0000 7fdfd6ffd700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdfc006ecc0 0x7fdfc0071180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc005950 tx=0x7fdfcc0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:50.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.612+0000 7fdfd4ff9700 1 -- 172.21.15.96:0/2081009177 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdfc8059050 con 0x7fdfd81004a0 2024-04-22T15:34:50.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:50 smithi096 ceph-mon[112443]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:50.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:50 smithi096 ceph-mon[112443]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:50.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:50 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1075723839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:34:50.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:50 smithi196 ceph-mon[81815]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:50.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:50 smithi196 ceph-mon[81815]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:50.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:50 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1075723839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:34:50.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.831+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfd80ff6d0 con 0x7fdfc006ecc0 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.833+0000 7fdfd4ff9700 1 -- 172.21.15.96:0/2081009177 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdfd80ff6d0 con 0x7fdfc006ecc0 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:34:50.836 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:34:50.837 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:34:50.837 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:34:50.837 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:34:50.837 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:34:50.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdfc006ecc0 msgr2=0x7fdfc0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdfc006ecc0 0x7fdfc0071180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc005950 tx=0x7fdfcc0058e0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 msgr2=0x7fdfd8197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 0x7fdfd8197b70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800bd70 tx=0x7fdfc800be50 comp rx=0 tx=0).stop 2024-04-22T15:34:50.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 shutdown_connections 2024-04-22T15:34:50.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdfc006ecc0 0x7fdfc0071180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfd81004a0 0x7fdfd8197b70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 --2- 172.21.15.96:0/2081009177 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd8100e50 0x7fdfd81980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 >> 172.21.15.96:0/2081009177 conn(0x7fdfd8075f40 msgr2=0x7fdfd8103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:50.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 shutdown_connections 2024-04-22T15:34:50.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.835+0000 7fdfde115700 1 -- 172.21.15.96:0/2081009177 wait complete. 2024-04-22T15:34:50.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.938+0000 7f2254362700 1 -- 172.21.15.96:0/2921084688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c0ffac0 msgr2=0x7f224c0fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.938+0000 7f2254362700 1 --2- 172.21.15.96:0/2921084688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c0ffac0 0x7f224c0fff40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f223c009910 tx=0x7f223c009c20 comp rx=0 tx=0).stop 2024-04-22T15:34:50.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.938+0000 7f2254362700 1 -- 172.21.15.96:0/2921084688 shutdown_connections 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.938+0000 7f2254362700 1 --2- 172.21.15.96:0/2921084688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c0ffac0 0x7f224c0fff40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.938+0000 7f2254362700 1 --2- 172.21.15.96:0/2921084688 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f224c105ac0 0x7f224c105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f2254362700 1 -- 172.21.15.96:0/2921084688 >> 172.21.15.96:0/2921084688 conn(0x7f224c0fb5d0 msgr2=0x7f224c0fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f2254362700 1 -- 172.21.15.96:0/2921084688 shutdown_connections 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f2254362700 1 -- 172.21.15.96:0/2921084688 wait complete. 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f2254362700 1 Processor -- start 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f2254362700 1 -- start start 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f2254362700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f224c0ffac0 0x7f224c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f2254362700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c105ac0 0x7f224c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f2254362700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c1985d0 con 0x7f224c105ac0 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f2254362700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c198710 con 0x7f224c0ffac0 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f22518fd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c105ac0 0x7f224c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:50.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.939+0000 7f22518fd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c105ac0 0x7f224c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40318/0 (socket says 172.21.15.96:40318) 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f22518fd700 1 -- 172.21.15.96:0/3319968610 learned_addr learned my addr 172.21.15.96:0/3319968610 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f22520fe700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f224c0ffac0 0x7f224c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f22518fd700 1 -- 172.21.15.96:0/3319968610 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f224c0ffac0 msgr2=0x7f224c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f22518fd700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f224c0ffac0 0x7f224c197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f22518fd700 1 -- 172.21.15.96:0/3319968610 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c0095f0 con 0x7f224c105ac0 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f22518fd700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c105ac0 0x7f224c198000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f223c00bd70 tx=0x7f223c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f22437fe700 1 -- 172.21.15.96:0/3319968610 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f223c01d070 con 0x7f224c105ac0 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f22437fe700 1 -- 172.21.15.96:0/3319968610 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f223c00f460 con 0x7f224c105ac0 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c1a0b30 con 0x7f224c105ac0 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f22437fe700 1 -- 172.21.15.96:0/3319968610 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f223c017630 con 0x7f224c105ac0 2024-04-22T15:34:50.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.940+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c1a0e90 con 0x7f224c105ac0 2024-04-22T15:34:50.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.941+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c1080e0 con 0x7f224c105ac0 2024-04-22T15:34:50.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.942+0000 7f22437fe700 1 -- 172.21.15.96:0/3319968610 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f223c00f5d0 con 0x7f224c105ac0 2024-04-22T15:34:50.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.942+0000 7f22437fe700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f223806ecc0 0x7f2238071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:34:50.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.942+0000 7f22437fe700 1 -- 172.21.15.96:0/3319968610 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f223c090400 con 0x7f224c105ac0 2024-04-22T15:34:50.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.942+0000 7f22520fe700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f223806ecc0 0x7f2238071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:34:50.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.942+0000 7f22520fe700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f223806ecc0 0x7f2238071180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2248006fd0 tx=0x7f2248008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:34:50.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:50.947+0000 7f22437fe700 1 -- 172.21.15.96:0/3319968610 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f223c059160 con 0x7f224c105ac0 2024-04-22T15:34:51.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.230+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f224c10cd40 con 0x7f224c105ac0 2024-04-22T15:34:51.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.230+0000 7f22437fe700 1 -- 172.21.15.96:0/3319968610 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f223c05c820 con 0x7f224c105ac0 2024-04-22T15:34:51.234 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:34:51.234 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:34:51.234 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:34:51.235 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:34:51.235 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:34:51.235 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:34:51.235 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:34:51.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.232+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f223806ecc0 msgr2=0x7f2238071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:51.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.232+0000 7f2254362700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f223806ecc0 0x7f2238071180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2248006fd0 tx=0x7f2248008040 comp rx=0 tx=0).stop 2024-04-22T15:34:51.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.232+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c105ac0 msgr2=0x7f224c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:34:51.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.232+0000 7f2254362700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c105ac0 0x7f224c198000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f223c00bd70 tx=0x7f223c00be50 comp rx=0 tx=0).stop 2024-04-22T15:34:51.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.232+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 shutdown_connections 2024-04-22T15:34:51.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.232+0000 7f2254362700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f223806ecc0 0x7f2238071180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:51.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.232+0000 7f2254362700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f224c0ffac0 0x7f224c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:51.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.232+0000 7f2254362700 1 --2- 172.21.15.96:0/3319968610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f224c105ac0 0x7f224c198000 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:34:51.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.233+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 >> 172.21.15.96:0/3319968610 conn(0x7f224c0fb5d0 msgr2=0x7f224c104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:34:51.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.233+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 shutdown_connections 2024-04-22T15:34:51.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:34:51.233+0000 7f2254362700 1 -- 172.21.15.96:0/3319968610 wait complete. 2024-04-22T15:34:51.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:51 smithi096 ceph-mon[112443]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:51.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:51 smithi096 ceph-mon[112443]: pgmap v389: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:51.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:51 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3086763901' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:34:51.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:51 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3319968610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:34:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:51 smithi196 ceph-mon[81815]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:51.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:51 smithi196 ceph-mon[81815]: pgmap v389: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:51.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:51 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3086763901' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:34:51.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:51 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3319968610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:34:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:52 smithi096 ceph-mon[112443]: from='client.34780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:52.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:52 smithi196 ceph-mon[81815]: from='client.34780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:34:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:53 smithi096 ceph-mon[112443]: pgmap v390: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:34:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:53 smithi196 ceph-mon[81815]: pgmap v390: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:34:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:55 smithi096 ceph-mon[112443]: pgmap v391: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:55 smithi196 ceph-mon[81815]: pgmap v391: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:57 smithi096 ceph-mon[112443]: pgmap v392: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:57 smithi196 ceph-mon[81815]: pgmap v392: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:34:59 smithi096 ceph-mon[112443]: pgmap v393: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:34:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:34:59 smithi196 ceph-mon[81815]: pgmap v393: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:01 smithi096 ceph-mon[112443]: pgmap v394: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:01 smithi196 ceph-mon[81815]: pgmap v394: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:03 smithi096 ceph-mon[112443]: pgmap v395: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:03 smithi196 ceph-mon[81815]: pgmap v395: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:05 smithi096 ceph-mon[112443]: pgmap v396: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:05 smithi196 ceph-mon[81815]: pgmap v396: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:07.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:07 smithi096 ceph-mon[112443]: pgmap v397: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:07 smithi196 ceph-mon[81815]: pgmap v397: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:35:08.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:35:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:09 smithi096 ceph-mon[112443]: pgmap v398: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:09 smithi196 ceph-mon[81815]: pgmap v398: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:11 smithi096 ceph-mon[112443]: pgmap v399: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:11 smithi196 ceph-mon[81815]: pgmap v399: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:13 smithi196 ceph-mon[81815]: pgmap v400: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:13 smithi096 ceph-mon[112443]: pgmap v400: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:15 smithi196 ceph-mon[81815]: pgmap v401: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:15.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:15 smithi096 ceph-mon[112443]: pgmap v401: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:17 smithi196 ceph-mon[81815]: pgmap v402: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:17.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:17 smithi096 ceph-mon[112443]: pgmap v402: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:19 smithi196 ceph-mon[81815]: pgmap v403: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:19 smithi096 ceph-mon[112443]: pgmap v403: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:21.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.350+0000 7f8029ab7700 1 -- 172.21.15.96:0/324963923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8024105c00 msgr2=0x7f8024105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:21.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.350+0000 7f8029ab7700 1 --2- 172.21.15.96:0/324963923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8024105c00 0x7f8024105fe0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f801400b0a0 tx=0x7f801400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.350+0000 7f8029ab7700 1 -- 172.21.15.96:0/324963923 shutdown_connections 2024-04-22T15:35:21.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.350+0000 7f8029ab7700 1 --2- 172.21.15.96:0/324963923 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80240ffb50 0x7f80240fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.350+0000 7f8029ab7700 1 --2- 172.21.15.96:0/324963923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8024105c00 0x7f8024105fe0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.350+0000 7f8029ab7700 1 -- 172.21.15.96:0/324963923 >> 172.21.15.96:0/324963923 conn(0x7f80240fb680 msgr2=0x7f80240fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:21.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.350+0000 7f8029ab7700 1 -- 172.21.15.96:0/324963923 shutdown_connections 2024-04-22T15:35:21.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.350+0000 7f8029ab7700 1 -- 172.21.15.96:0/324963923 wait complete. 2024-04-22T15:35:21.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.350+0000 7f8029ab7700 1 Processor -- start 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f8029ab7700 1 -- start start 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f8029ab7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80240ffb50 0x7f80241978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f8029ab7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8024105c00 0x7f8024197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f8029ab7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80241983f0 con 0x7f80240ffb50 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f8029ab7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8024198530 con 0x7f8024105c00 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f80237fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80240ffb50 0x7f80241978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f80237fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80240ffb50 0x7f80241978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52466/0 (socket says 172.21.15.96:52466) 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f80237fe700 1 -- 172.21.15.96:0/4181149789 learned_addr learned my addr 172.21.15.96:0/4181149789 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f8022ffd700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8024105c00 0x7f8024197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f80237fe700 1 -- 172.21.15.96:0/4181149789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8024105c00 msgr2=0x7f8024197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f80237fe700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8024105c00 0x7f8024197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f80237fe700 1 -- 172.21.15.96:0/4181149789 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8014009d00 con 0x7f80240ffb50 2024-04-22T15:35:21.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.351+0000 7f8022ffd700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8024105c00 0x7f8024197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:35:21.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.352+0000 7f80237fe700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80240ffb50 0x7f80241978e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8014000f80 tx=0x7f8014007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:21.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.352+0000 7f8020ff9700 1 -- 172.21.15.96:0/4181149789 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f801400e040 con 0x7f80240ffb50 2024-04-22T15:35:21.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.352+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80241a0950 con 0x7f80240ffb50 2024-04-22T15:35:21.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.352+0000 7f8020ff9700 1 -- 172.21.15.96:0/4181149789 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8014003de0 con 0x7f80240ffb50 2024-04-22T15:35:21.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.352+0000 7f8020ff9700 1 -- 172.21.15.96:0/4181149789 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f801401d720 con 0x7f80240ffb50 2024-04-22T15:35:21.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.352+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80241a0d00 con 0x7f80240ffb50 2024-04-22T15:35:21.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.353+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8024108030 con 0x7f80240ffb50 2024-04-22T15:35:21.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.353+0000 7f8020ff9700 1 -- 172.21.15.96:0/4181149789 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8014004690 con 0x7f80240ffb50 2024-04-22T15:35:21.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.354+0000 7f8020ff9700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f801006ecc0 0x7f8010071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:21.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.354+0000 7f8020ff9700 1 -- 172.21.15.96:0/4181149789 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8014090680 con 0x7f80240ffb50 2024-04-22T15:35:21.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.354+0000 7f8022ffd700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f801006ecc0 0x7f8010071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:21.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.354+0000 7f8022ffd700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f801006ecc0 0x7f8010071180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8024069980 tx=0x7f800c005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:21.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.359+0000 7f8020ff9700 1 -- 172.21.15.96:0/4181149789 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f80140593e0 con 0x7f80240ffb50 2024-04-22T15:35:21.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.586+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f80240697a0 con 0x7f801006ecc0 2024-04-22T15:35:21.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.587+0000 7f8020ff9700 1 -- 172.21.15.96:0/4181149789 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f80240697a0 con 0x7f801006ecc0 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.589+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f801006ecc0 msgr2=0x7f8010071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.589+0000 7f8029ab7700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f801006ecc0 0x7f8010071180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8024069980 tx=0x7f800c005f50 comp rx=0 tx=0).stop 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.589+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80240ffb50 msgr2=0x7f80241978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.589+0000 7f8029ab7700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80240ffb50 0x7f80241978e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8014000f80 tx=0x7f8014007bd0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.590+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 shutdown_connections 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.590+0000 7f8029ab7700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f801006ecc0 0x7f8010071180 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.590+0000 7f8029ab7700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80240ffb50 0x7f80241978e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.590+0000 7f8029ab7700 1 --2- 172.21.15.96:0/4181149789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8024105c00 0x7f8024197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.590+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 >> 172.21.15.96:0/4181149789 conn(0x7f80240fb680 msgr2=0x7f80240fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:21.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.590+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 shutdown_connections 2024-04-22T15:35:21.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.590+0000 7f8029ab7700 1 -- 172.21.15.96:0/4181149789 wait complete. 2024-04-22T15:35:21.603 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:35:21.604 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:21 smithi096 ceph-mon[112443]: pgmap v404: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:21.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.696+0000 7fba4e404700 1 -- 172.21.15.96:0/4096341755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba481017d0 msgr2=0x7fba48105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:21.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.696+0000 7fba4e404700 1 --2- 172.21.15.96:0/4096341755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba481017d0 0x7fba48105820 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fba38009910 tx=0x7fba38009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:21.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.696+0000 7fba4e404700 1 -- 172.21.15.96:0/4096341755 shutdown_connections 2024-04-22T15:35:21.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.696+0000 7fba4e404700 1 --2- 172.21.15.96:0/4096341755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba481017d0 0x7fba48105820 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.696+0000 7fba4e404700 1 --2- 172.21.15.96:0/4096341755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba48100e20 0x7fba48101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.696+0000 7fba4e404700 1 -- 172.21.15.96:0/4096341755 >> 172.21.15.96:0/4096341755 conn(0x7fba480fc6d0 msgr2=0x7fba480feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:21.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.696+0000 7fba4e404700 1 -- 172.21.15.96:0/4096341755 shutdown_connections 2024-04-22T15:35:21.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.696+0000 7fba4e404700 1 -- 172.21.15.96:0/4096341755 wait complete. 2024-04-22T15:35:21.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba4e404700 1 Processor -- start 2024-04-22T15:35:21.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba4e404700 1 -- start start 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba4e404700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba48100e20 0x7fba4819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba4e404700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba481017d0 0x7fba4819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba4e404700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4819d380 con 0x7fba48100e20 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba4e404700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba48196960 con 0x7fba481017d0 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba47fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba48100e20 0x7fba4819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba47fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba48100e20 0x7fba4819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52476/0 (socket says 172.21.15.96:52476) 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba47fff700 1 -- 172.21.15.96:0/159152014 learned_addr learned my addr 172.21.15.96:0/159152014 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.697+0000 7fba477fe700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba481017d0 0x7fba4819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.698+0000 7fba47fff700 1 -- 172.21.15.96:0/159152014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba481017d0 msgr2=0x7fba4819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.698+0000 7fba47fff700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba481017d0 0x7fba4819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.698+0000 7fba47fff700 1 -- 172.21.15.96:0/159152014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba380095f0 con 0x7fba48100e20 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.698+0000 7fba47fff700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba48100e20 0x7fba4819c820 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00d8d0 tx=0x7fba3c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:21.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.698+0000 7fba457fa700 1 -- 172.21.15.96:0/159152014 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba3c0098a0 con 0x7fba48100e20 2024-04-22T15:35:21.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.698+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba48196c40 con 0x7fba48100e20 2024-04-22T15:35:21.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.698+0000 7fba457fa700 1 -- 172.21.15.96:0/159152014 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba3c010460 con 0x7fba48100e20 2024-04-22T15:35:21.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.698+0000 7fba457fa700 1 -- 172.21.15.96:0/159152014 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba3c00f5f0 con 0x7fba48100e20 2024-04-22T15:35:21.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.698+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba48197190 con 0x7fba48100e20 2024-04-22T15:35:21.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.699+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba48109220 con 0x7fba48100e20 2024-04-22T15:35:21.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.699+0000 7fba457fa700 1 -- 172.21.15.96:0/159152014 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fba3c009a00 con 0x7fba48100e20 2024-04-22T15:35:21.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.700+0000 7fba457fa700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fba3006ecc0 0x7fba30071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:21.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.700+0000 7fba457fa700 1 -- 172.21.15.96:0/159152014 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fba3c08ee20 con 0x7fba48100e20 2024-04-22T15:35:21.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.700+0000 7fba477fe700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fba3006ecc0 0x7fba30071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:21.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.700+0000 7fba477fe700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fba3006ecc0 0x7fba30071180 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fba48197b20 tx=0x7fba3801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:21.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.705+0000 7fba457fa700 1 -- 172.21.15.96:0/159152014 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fba3c05b610 con 0x7fba48100e20 2024-04-22T15:35:21.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:21 smithi196 ceph-mon[81815]: pgmap v404: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:21.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.926+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba48197e20 con 0x7fba3006ecc0 2024-04-22T15:35:21.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.927+0000 7fba457fa700 1 -- 172.21.15.96:0/159152014 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fba48197e20 con 0x7fba3006ecc0 2024-04-22T15:35:21.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fba3006ecc0 msgr2=0x7fba30071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:21.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fba3006ecc0 0x7fba30071180 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fba48197b20 tx=0x7fba3801a040 comp rx=0 tx=0).stop 2024-04-22T15:35:21.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba48100e20 msgr2=0x7fba4819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:21.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba48100e20 0x7fba4819c820 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00d8d0 tx=0x7fba3c00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 shutdown_connections 2024-04-22T15:35:21.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fba3006ecc0 0x7fba30071180 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba48100e20 0x7fba4819c820 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 --2- 172.21.15.96:0/159152014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba481017d0 0x7fba4819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:21.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 >> 172.21.15.96:0/159152014 conn(0x7fba480fc6d0 msgr2=0x7fba480fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:21.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 shutdown_connections 2024-04-22T15:35:21.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:21.929+0000 7fba4e404700 1 -- 172.21.15.96:0/159152014 wait complete. 2024-04-22T15:35:22.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.032+0000 7f769d876700 1 -- 172.21.15.96:0/4151909054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ffe10 msgr2=0x7f7698111320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.032+0000 7f769d876700 1 --2- 172.21.15.96:0/4151909054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ffe10 0x7f7698111320 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f768c009940 tx=0x7f768c009c50 comp rx=0 tx=0).stop 2024-04-22T15:35:22.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.032+0000 7f769d876700 1 -- 172.21.15.96:0/4151909054 shutdown_connections 2024-04-22T15:35:22.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.032+0000 7f769d876700 1 --2- 172.21.15.96:0/4151909054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ffe10 0x7f7698111320 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.032+0000 7f769d876700 1 --2- 172.21.15.96:0/4151909054 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76980ff460 0x7f76980ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.032+0000 7f769d876700 1 -- 172.21.15.96:0/4151909054 >> 172.21.15.96:0/4151909054 conn(0x7f7698074d90 msgr2=0x7f76980751a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:22.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.032+0000 7f769d876700 1 -- 172.21.15.96:0/4151909054 shutdown_connections 2024-04-22T15:35:22.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.032+0000 7f769d876700 1 -- 172.21.15.96:0/4151909054 wait complete. 2024-04-22T15:35:22.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.033+0000 7f769d876700 1 Processor -- start 2024-04-22T15:35:22.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.033+0000 7f769d876700 1 -- start start 2024-04-22T15:35:22.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.033+0000 7f769d876700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ff460 0x7f769819d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:22.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.033+0000 7f769d876700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76980ffe10 0x7f769819d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:22.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.033+0000 7f769d876700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769819ddc0 con 0x7f76980ff460 2024-04-22T15:35:22.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.033+0000 7f769d876700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76981a1a90 con 0x7f76980ffe10 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.033+0000 7f7696ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ff460 0x7f769819d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.033+0000 7f7696ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ff460 0x7f769819d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52496/0 (socket says 172.21.15.96:52496) 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.033+0000 7f7696ffd700 1 -- 172.21.15.96:0/347628477 learned_addr learned my addr 172.21.15.96:0/347628477 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f76967fc700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76980ffe10 0x7f769819d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f7696ffd700 1 -- 172.21.15.96:0/347628477 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76980ffe10 msgr2=0x7f769819d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f7696ffd700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76980ffe10 0x7f769819d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f7696ffd700 1 -- 172.21.15.96:0/347628477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0095f0 con 0x7f76980ff460 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f7696ffd700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ff460 0x7f769819d1a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f768800b4b0 tx=0x7f768800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f769c874700 1 -- 172.21.15.96:0/347628477 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7688011800 con 0x7f76980ff460 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f769c874700 1 -- 172.21.15.96:0/347628477 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7688011e20 con 0x7f76980ff460 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76981a1d70 con 0x7f76980ff460 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f769c874700 1 -- 172.21.15.96:0/347628477 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f768800f2e0 con 0x7f76980ff460 2024-04-22T15:35:22.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.034+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76981a22c0 con 0x7f76980ff460 2024-04-22T15:35:22.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.035+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7698101dd0 con 0x7f76980ff460 2024-04-22T15:35:22.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.035+0000 7f769c874700 1 -- 172.21.15.96:0/347628477 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7688010380 con 0x7f76980ff460 2024-04-22T15:35:22.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.036+0000 7f769c874700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f768006ecd0 0x7f7680071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:22.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.036+0000 7f769c874700 1 -- 172.21.15.96:0/347628477 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f768808e8e0 con 0x7f76980ff460 2024-04-22T15:35:22.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.036+0000 7f76967fc700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f768006ecd0 0x7f7680071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:22.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.036+0000 7f76967fc700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f768006ecd0 0x7f7680071190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f769819e700 tx=0x7f768c00b7d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:22.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.041+0000 7f769c874700 1 -- 172.21.15.96:0/347628477 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f76880576c0 con 0x7f76980ff460 2024-04-22T15:35:22.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.248+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7698114450 con 0x7f768006ecd0 2024-04-22T15:35:22.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.259+0000 7f769c874700 1 -- 172.21.15.96:0/347628477 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7698114450 con 0x7f768006ecd0 2024-04-22T15:35:22.262 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:35:22.262 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (14m) 7m ago 25m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:35:22.262 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (25m) 7m ago 25m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (24m) 6m ago 24m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (11m) 7m ago 25m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (11m) 6m ago 24m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (13m) 7m ago 25m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (7m) 7m ago 19m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (7m) 7m ago 19m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (6m) 6m ago 19m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (19m) 6m ago 19m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (16m) 7m ago 26m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (15m) 6m ago 24m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (12m) 7m ago 26m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (11m) 6m ago 24m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:35:22.263 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (14m) 7m ago 25m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:35:22.264 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (14m) 6m ago 24m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:35:22.264 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (10m) 7m ago 23m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:35:22.264 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (10m) 7m ago 22m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:35:22.264 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (9m) 7m ago 22m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:35:22.264 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (8m) 6m ago 22m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:35:22.264 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (8m) 6m ago 21m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:35:22.264 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (7m) 6m ago 21m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:35:22.264 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (14m) 7m ago 24m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:35:22.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f768006ecd0 msgr2=0x7f7680071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f768006ecd0 0x7f7680071190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f769819e700 tx=0x7f768c00b7d0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ff460 msgr2=0x7f769819d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ff460 0x7f769819d1a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f768800b4b0 tx=0x7f768800b7c0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 shutdown_connections 2024-04-22T15:35:22.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f768006ecd0 0x7f7680071190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76980ff460 0x7f769819d1a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 --2- 172.21.15.96:0/347628477 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76980ffe10 0x7f769819d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 >> 172.21.15.96:0/347628477 conn(0x7f7698074d90 msgr2=0x7f769810f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:22.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 shutdown_connections 2024-04-22T15:35:22.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.261+0000 7f769d876700 1 -- 172.21.15.96:0/347628477 wait complete. 2024-04-22T15:35:22.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.363+0000 7f851cb43700 1 -- 172.21.15.96:0/2113114343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8518105b80 msgr2=0x7f8518105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.363+0000 7f851cb43700 1 --2- 172.21.15.96:0/2113114343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8518105b80 0x7f8518105f60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8508009910 tx=0x7f8508009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:22.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.364+0000 7f851cb43700 1 -- 172.21.15.96:0/2113114343 shutdown_connections 2024-04-22T15:35:22.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.364+0000 7f851cb43700 1 --2- 172.21.15.96:0/2113114343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85180ffaf0 0x7f85180fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.364+0000 7f851cb43700 1 --2- 172.21.15.96:0/2113114343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8518105b80 0x7f8518105f60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.364+0000 7f851cb43700 1 -- 172.21.15.96:0/2113114343 >> 172.21.15.96:0/2113114343 conn(0x7f85180fb5e0 msgr2=0x7f85180fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:22.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.364+0000 7f851cb43700 1 -- 172.21.15.96:0/2113114343 shutdown_connections 2024-04-22T15:35:22.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.364+0000 7f851cb43700 1 -- 172.21.15.96:0/2113114343 wait complete. 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.364+0000 7f851cb43700 1 Processor -- start 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.364+0000 7f851cb43700 1 -- start start 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851cb43700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85180ffaf0 0x7f85181978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851cb43700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8518105b80 0x7f8518197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851cb43700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8518198400 con 0x7f85180ffaf0 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851cb43700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8518198570 con 0x7f8518105b80 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85180ffaf0 0x7f85181978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85180ffaf0 0x7f85181978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52510/0 (socket says 172.21.15.96:52510) 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851659c700 1 -- 172.21.15.96:0/1429799801 learned_addr learned my addr 172.21.15.96:0/1429799801 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:22.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851659c700 1 -- 172.21.15.96:0/1429799801 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8518105b80 msgr2=0x7f8518197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f8515d9b700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8518105b80 0x7f8518197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851659c700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8518105b80 0x7f8518197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851659c700 1 -- 172.21.15.96:0/1429799801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85080095f0 con 0x7f85180ffaf0 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851659c700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85180ffaf0 0x7f85181978f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8508000c00 tx=0x7f850800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f85077fe700 1 -- 172.21.15.96:0/1429799801 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f850801d070 con 0x7f85180ffaf0 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f85077fe700 1 -- 172.21.15.96:0/1429799801 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85080056e0 con 0x7f85180ffaf0 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85181a0960 con 0x7f85180ffaf0 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f85077fe700 1 -- 172.21.15.96:0/1429799801 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f850800fe80 con 0x7f85180ffaf0 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.365+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85181a0d70 con 0x7f85180ffaf0 2024-04-22T15:35:22.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.366+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85181080f0 con 0x7f85180ffaf0 2024-04-22T15:35:22.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.367+0000 7f85077fe700 1 -- 172.21.15.96:0/1429799801 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f850800f6f0 con 0x7f85180ffaf0 2024-04-22T15:35:22.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.367+0000 7f85077fe700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f850006ecd0 0x7f8500071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:22.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.367+0000 7f85077fe700 1 -- 172.21.15.96:0/1429799801 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8508090210 con 0x7f85180ffaf0 2024-04-22T15:35:22.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.367+0000 7f8515d9b700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f850006ecd0 0x7f8500071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:22.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.368+0000 7f8515d9b700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f850006ecd0 0x7f8500071190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f850c005950 tx=0x7f850c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:22.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.372+0000 7f85077fe700 1 -- 172.21.15.96:0/1429799801 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8508058ff0 con 0x7f85180ffaf0 2024-04-22T15:35:22.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.658+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f85180696b0 con 0x7f85180ffaf0 2024-04-22T15:35:22.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.659+0000 7f85077fe700 1 -- 172.21.15.96:0/1429799801 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f850805c6b0 con 0x7f85180ffaf0 2024-04-22T15:35:22.662 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:35:22.663 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:35:22.664 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:35:22.664 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:35:22.664 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:35:22.664 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:35:22.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f850006ecd0 msgr2=0x7f8500071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f850006ecd0 0x7f8500071190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f850c005950 tx=0x7f850c0058e0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85180ffaf0 msgr2=0x7f85181978f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85180ffaf0 0x7f85181978f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8508000c00 tx=0x7f850800be20 comp rx=0 tx=0).stop 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 shutdown_connections 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f850006ecd0 0x7f8500071190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85180ffaf0 0x7f85181978f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 --2- 172.21.15.96:0/1429799801 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8518105b80 0x7f8518197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 >> 172.21.15.96:0/1429799801 conn(0x7f85180fb5e0 msgr2=0x7f85180fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 shutdown_connections 2024-04-22T15:35:22.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.661+0000 7f851cb43700 1 -- 172.21.15.96:0/1429799801 wait complete. 2024-04-22T15:35:22.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 -- 172.21.15.96:0/3692150666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 msgr2=0x7f6760105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 --2- 172.21.15.96:0/3692150666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 0x7f6760105ea0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6750009910 tx=0x7f6750009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:22.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 -- 172.21.15.96:0/3692150666 shutdown_connections 2024-04-22T15:35:22.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 --2- 172.21.15.96:0/3692150666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67600ffac0 0x7f67600fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 --2- 172.21.15.96:0/3692150666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 0x7f6760105ea0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 -- 172.21.15.96:0/3692150666 >> 172.21.15.96:0/3692150666 conn(0x7f67600fb5d0 msgr2=0x7f67600fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 -- 172.21.15.96:0/3692150666 shutdown_connections 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 -- 172.21.15.96:0/3692150666 wait complete. 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 Processor -- start 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.765+0000 7f67663f4700 1 -- start start 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f67663f4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67600ffac0 0x7f676019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f67663f4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 0x7f676019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f67663f4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676019d450 con 0x7f6760105ac0 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f67663f4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6760196a30 con 0x7f67600ffac0 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f675f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 0x7f676019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f675f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 0x7f676019ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52534/0 (socket says 172.21.15.96:52534) 2024-04-22T15:35:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f675f7fe700 1 -- 172.21.15.96:0/566673623 learned_addr learned my addr 172.21.15.96:0/566673623 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f675ffff700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67600ffac0 0x7f676019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f675f7fe700 1 -- 172.21.15.96:0/566673623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67600ffac0 msgr2=0x7f676019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f675f7fe700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67600ffac0 0x7f676019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f675f7fe700 1 -- 172.21.15.96:0/566673623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67500095f0 con 0x7f6760105ac0 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.766+0000 7f675f7fe700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 0x7f676019ce30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f675400b4b0 tx=0x7f675400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.767+0000 7f675d7fa700 1 -- 172.21.15.96:0/566673623 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6754011800 con 0x7f6760105ac0 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.767+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6760196d10 con 0x7f6760105ac0 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.767+0000 7f675d7fa700 1 -- 172.21.15.96:0/566673623 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6754011e20 con 0x7f6760105ac0 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.767+0000 7f675d7fa700 1 -- 172.21.15.96:0/566673623 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f675400f2e0 con 0x7f6760105ac0 2024-04-22T15:35:22.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.767+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6760197260 con 0x7f6760105ac0 2024-04-22T15:35:22.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.768+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67601090d0 con 0x7f6760105ac0 2024-04-22T15:35:22.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.768+0000 7f675d7fa700 1 -- 172.21.15.96:0/566673623 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6754010380 con 0x7f6760105ac0 2024-04-22T15:35:22.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.769+0000 7f675d7fa700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f674806ecc0 0x7f6748071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:22.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.769+0000 7f675d7fa700 1 -- 172.21.15.96:0/566673623 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f675408ea50 con 0x7f6760105ac0 2024-04-22T15:35:22.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.769+0000 7f675ffff700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f674806ecc0 0x7f6748071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:22.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.769+0000 7f675ffff700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f674806ecc0 0x7f6748071180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f675000b5c0 tx=0x7f675001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:22.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:22.773+0000 7f675d7fa700 1 -- 172.21.15.96:0/566673623 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f675405b2e0 con 0x7f6760105ac0 2024-04-22T15:35:23.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.014+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6760197ef0 con 0x7f6760105ac0 2024-04-22T15:35:23.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.014+0000 7f675d7fa700 1 -- 172.21.15.96:0/566673623 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f675405ae70 con 0x7f6760105ac0 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:35:23.019 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:35:23.020 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:35:23.021 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:35:23.022 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:35:23.022 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:35:23.022 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:35:23.022 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:35:23.022 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:35:23.022 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:35:23.022 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:35:23.022 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:35:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.016+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f674806ecc0 msgr2=0x7f6748071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.016+0000 7f67663f4700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f674806ecc0 0x7f6748071180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f675000b5c0 tx=0x7f675001a040 comp rx=0 tx=0).stop 2024-04-22T15:35:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.016+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 msgr2=0x7f676019ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.016+0000 7f67663f4700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 0x7f676019ce30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f675400b4b0 tx=0x7f675400b870 comp rx=0 tx=0).stop 2024-04-22T15:35:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.017+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 shutdown_connections 2024-04-22T15:35:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.017+0000 7f67663f4700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f674806ecc0 0x7f6748071180 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.017+0000 7f67663f4700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67600ffac0 0x7f676019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.017+0000 7f67663f4700 1 --2- 172.21.15.96:0/566673623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6760105ac0 0x7f676019ce30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.017+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 >> 172.21.15.96:0/566673623 conn(0x7f67600fb5d0 msgr2=0x7f6760104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.017+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 shutdown_connections 2024-04-22T15:35:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.017+0000 7f67663f4700 1 -- 172.21.15.96:0/566673623 wait complete. 2024-04-22T15:35:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:35:23.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 -- 172.21.15.96:0/1452687234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4102770 msgr2=0x7ff0f410ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 --2- 172.21.15.96:0/1452687234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4102770 0x7ff0f410ac60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009910 tx=0x7ff0dc009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:23.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 -- 172.21.15.96:0/1452687234 shutdown_connections 2024-04-22T15:35:23.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 --2- 172.21.15.96:0/1452687234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4102770 0x7ff0f410ac60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 --2- 172.21.15.96:0/1452687234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f4101e50 0x7ff0f4102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 -- 172.21.15.96:0/1452687234 >> 172.21.15.96:0/1452687234 conn(0x7ff0f40fb5e0 msgr2=0x7ff0f40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:23.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 -- 172.21.15.96:0/1452687234 shutdown_connections 2024-04-22T15:35:23.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 -- 172.21.15.96:0/1452687234 wait complete. 2024-04-22T15:35:23.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 Processor -- start 2024-04-22T15:35:23.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.122+0000 7ff0f998f700 1 -- start start 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f998f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4101e50 0x7ff0f419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f998f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f4102770 0x7ff0f419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f998f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f419d370 con 0x7ff0f4101e50 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f998f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f4196950 con 0x7ff0f4102770 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4101e50 0x7ff0f419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4101e50 0x7ff0f419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52550/0 (socket says 172.21.15.96:52550) 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f2ffd700 1 -- 172.21.15.96:0/442262174 learned_addr learned my addr 172.21.15.96:0/442262174 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0ea7fc700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f4102770 0x7ff0f419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f2ffd700 1 -- 172.21.15.96:0/442262174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f4102770 msgr2=0x7ff0f419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f2ffd700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f4102770 0x7ff0f419cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f2ffd700 1 -- 172.21.15.96:0/442262174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc0095f0 con 0x7ff0f4101e50 2024-04-22T15:35:23.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f2ffd700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4101e50 0x7ff0f419c810 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400d8d0 tx=0x7ff0e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:23.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.123+0000 7ff0f898d700 1 -- 172.21.15.96:0/442262174 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0e4009940 con 0x7ff0f4101e50 2024-04-22T15:35:23.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.124+0000 7ff0f898d700 1 -- 172.21.15.96:0/442262174 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0e4010460 con 0x7ff0f4101e50 2024-04-22T15:35:23.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.124+0000 7ff0f898d700 1 -- 172.21.15.96:0/442262174 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0e400f5f0 con 0x7ff0f4101e50 2024-04-22T15:35:23.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.124+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f4196c30 con 0x7ff0f4101e50 2024-04-22T15:35:23.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.124+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f4197180 con 0x7ff0f4101e50 2024-04-22T15:35:23.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.124+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f4105500 con 0x7ff0f4101e50 2024-04-22T15:35:23.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.125+0000 7ff0f898d700 1 -- 172.21.15.96:0/442262174 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff0e40105d0 con 0x7ff0f4101e50 2024-04-22T15:35:23.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.125+0000 7ff0f898d700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff0e006ecc0 0x7ff0e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:23.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.126+0000 7ff0ea7fc700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff0e006ecc0 0x7ff0e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:23.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.126+0000 7ff0f898d700 1 -- 172.21.15.96:0/442262174 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff0e408eed0 con 0x7ff0f4101e50 2024-04-22T15:35:23.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.126+0000 7ff0ea7fc700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff0e006ecc0 0x7ff0e0071180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0059d0 tx=0x7ff0dc005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:23.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.130+0000 7ff0f898d700 1 -- 172.21.15.96:0/442262174 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff0e405b760 con 0x7ff0f4101e50 2024-04-22T15:35:23.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.347+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0f4197e80 con 0x7ff0e006ecc0 2024-04-22T15:35:23.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.348+0000 7ff0f898d700 1 -- 172.21.15.96:0/442262174 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff0f4197e80 con 0x7ff0e006ecc0 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:35:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:35:23.353 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:35:23.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff0e006ecc0 msgr2=0x7ff0e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff0e006ecc0 0x7ff0e0071180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0059d0 tx=0x7ff0dc005960 comp rx=0 tx=0).stop 2024-04-22T15:35:23.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4101e50 msgr2=0x7ff0f419c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4101e50 0x7ff0f419c810 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400d8d0 tx=0x7ff0e400dc90 comp rx=0 tx=0).stop 2024-04-22T15:35:23.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 shutdown_connections 2024-04-22T15:35:23.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff0e006ecc0 0x7ff0e0071180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0f4101e50 0x7ff0f419c810 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 --2- 172.21.15.96:0/442262174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f4102770 0x7ff0f419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 >> 172.21.15.96:0/442262174 conn(0x7ff0f40fb5e0 msgr2=0x7ff0f4103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:23.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 shutdown_connections 2024-04-22T15:35:23.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.351+0000 7ff0f998f700 1 -- 172.21.15.96:0/442262174 wait complete. 2024-04-22T15:35:23.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.454+0000 7f6120e39700 1 -- 172.21.15.96:0/3340079765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c100220 msgr2=0x7f611c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.454+0000 7f6120e39700 1 --2- 172.21.15.96:0/3340079765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c100220 0x7f611c1006a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f610c009910 tx=0x7f610c009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:23.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.454+0000 7f6120e39700 1 -- 172.21.15.96:0/3340079765 shutdown_connections 2024-04-22T15:35:23.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.454+0000 7f6120e39700 1 --2- 172.21.15.96:0/3340079765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c100220 0x7f611c1006a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.454+0000 7f6120e39700 1 --2- 172.21.15.96:0/3340079765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f611c104b30 0x7f611c104f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.454+0000 7f6120e39700 1 -- 172.21.15.96:0/3340079765 >> 172.21.15.96:0/3340079765 conn(0x7f611c075f40 msgr2=0x7f611c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:23.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.454+0000 7f6120e39700 1 -- 172.21.15.96:0/3340079765 shutdown_connections 2024-04-22T15:35:23.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.454+0000 7f6120e39700 1 -- 172.21.15.96:0/3340079765 wait complete. 2024-04-22T15:35:23.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f6120e39700 1 Processor -- start 2024-04-22T15:35:23.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f6120e39700 1 -- start start 2024-04-22T15:35:23.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f6120e39700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f611c100220 0x7f611c19b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:23.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f6120e39700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c104b30 0x7f611c19bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:23.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f6120e39700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c19c2e0 con 0x7f611c104b30 2024-04-22T15:35:23.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f6120e39700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c1958c0 con 0x7f611c100220 2024-04-22T15:35:23.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f6119d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c104b30 0x7f611c19bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:23.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f6119d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c104b30 0x7f611c19bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52568/0 (socket says 172.21.15.96:52568) 2024-04-22T15:35:23.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f6119d9b700 1 -- 172.21.15.96:0/4177056032 learned_addr learned my addr 172.21.15.96:0/4177056032 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.455+0000 7f611a59c700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f611c100220 0x7f611c19b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.456+0000 7f6119d9b700 1 -- 172.21.15.96:0/4177056032 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f611c100220 msgr2=0x7f611c19b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.456+0000 7f6119d9b700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f611c100220 0x7f611c19b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.456+0000 7f6119d9b700 1 -- 172.21.15.96:0/4177056032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c0095f0 con 0x7f611c104b30 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.456+0000 7f6119d9b700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c104b30 0x7f611c19bcc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f610c00bd70 tx=0x7f610c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.456+0000 7f610b7fe700 1 -- 172.21.15.96:0/4177056032 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610c01d070 con 0x7f611c104b30 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.456+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c195b40 con 0x7f611c104b30 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.456+0000 7f610b7fe700 1 -- 172.21.15.96:0/4177056032 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f610c005700 con 0x7f611c104b30 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.456+0000 7f610b7fe700 1 -- 172.21.15.96:0/4177056032 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610c017400 con 0x7f611c104b30 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.456+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c1960b0 con 0x7f611c104b30 2024-04-22T15:35:23.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.457+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611c108100 con 0x7f611c104b30 2024-04-22T15:35:23.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.457+0000 7f610b7fe700 1 -- 172.21.15.96:0/4177056032 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f610c003a60 con 0x7f611c104b30 2024-04-22T15:35:23.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.458+0000 7f610b7fe700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f610406ecc0 0x7f6104071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:23.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.458+0000 7f611a59c700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f610406ecc0 0x7f6104071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:23.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.458+0000 7f610b7fe700 1 -- 172.21.15.96:0/4177056032 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f610c090340 con 0x7f611c104b30 2024-04-22T15:35:23.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.458+0000 7f611a59c700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f610406ecc0 0x7f6104071180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6110009a10 tx=0x7f6110009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.463+0000 7f610b7fe700 1 -- 172.21.15.96:0/4177056032 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f610c05cbd0 con 0x7f611c104b30 2024-04-22T15:35:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:23 smithi096 ceph-mon[112443]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:23 smithi096 ceph-mon[112443]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:23 smithi096 ceph-mon[112443]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:23 smithi096 ceph-mon[112443]: pgmap v405: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:23 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1429799801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:35:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:23 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/566673623' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:35:23.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:35:23.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.750+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f611c101440 con 0x7f611c104b30 2024-04-22T15:35:23.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.750+0000 7f610b7fe700 1 -- 172.21.15.96:0/4177056032 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f610c05c760 con 0x7f611c104b30 2024-04-22T15:35:23.753 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:35:23.753 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:35:23.754 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:35:23.754 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:35:23.754 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:35:23.754 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:35:23.754 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f610406ecc0 msgr2=0x7f6104071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f610406ecc0 0x7f6104071180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6110009a10 tx=0x7f6110009190 comp rx=0 tx=0).stop 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c104b30 msgr2=0x7f611c19bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c104b30 0x7f611c19bcc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f610c00bd70 tx=0x7f610c00bda0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 shutdown_connections 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f610406ecc0 0x7f6104071180 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f611c100220 0x7f611c19b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 --2- 172.21.15.96:0/4177056032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f611c104b30 0x7f611c19bcc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 >> 172.21.15.96:0/4177056032 conn(0x7f611c075f40 msgr2=0x7f611c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 shutdown_connections 2024-04-22T15:35:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:23.752+0000 7f6120e39700 1 -- 172.21.15.96:0/4177056032 wait complete. 2024-04-22T15:35:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:23 smithi196 ceph-mon[81815]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:23 smithi196 ceph-mon[81815]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:23 smithi196 ceph-mon[81815]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:23 smithi196 ceph-mon[81815]: pgmap v405: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:23 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1429799801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:35:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:23 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/566673623' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:35:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:35:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:24 smithi196 ceph-mon[81815]: from='client.34808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:24 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4177056032' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:35:24.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:24 smithi096 ceph-mon[112443]: from='client.34808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:24.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:24 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4177056032' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:35:25.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:25 smithi096 ceph-mon[112443]: pgmap v406: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:25 smithi196 ceph-mon[81815]: pgmap v406: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:27.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:27 smithi096 ceph-mon[112443]: pgmap v407: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:27 smithi196 ceph-mon[81815]: pgmap v407: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:29 smithi096 ceph-mon[112443]: pgmap v408: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:29 smithi196 ceph-mon[81815]: pgmap v408: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:31.788 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:31 smithi096 ceph-mon[112443]: pgmap v409: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:32.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:31 smithi196 ceph-mon[81815]: pgmap v409: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:33.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:33 smithi096 ceph-mon[112443]: pgmap v410: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:33 smithi196 ceph-mon[81815]: pgmap v410: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:35.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:35 smithi096 ceph-mon[112443]: pgmap v411: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:35 smithi196 ceph-mon[81815]: pgmap v411: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:37 smithi096 ceph-mon[112443]: pgmap v412: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:37 smithi196 ceph-mon[81815]: pgmap v412: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:38.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:35:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:35:39.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:39 smithi096 ceph-mon[112443]: pgmap v413: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:39 smithi196 ceph-mon[81815]: pgmap v413: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:40.811 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:40 smithi196 ceph-mon[81815]: pgmap v414: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:40 smithi096 ceph-mon[112443]: pgmap v414: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:43 smithi096 ceph-mon[112443]: pgmap v415: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:43 smithi196 ceph-mon[81815]: pgmap v415: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:45 smithi096 ceph-mon[112443]: pgmap v416: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:45.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:45 smithi196 ceph-mon[81815]: pgmap v416: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:47 smithi096 ceph-mon[112443]: pgmap v417: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:47 smithi196 ceph-mon[81815]: pgmap v417: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:49.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:49 smithi096 ceph-mon[112443]: pgmap v418: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:49 smithi196 ceph-mon[81815]: pgmap v418: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:51 smithi096 ceph-mon[112443]: pgmap v419: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:51 smithi196 ceph-mon[81815]: pgmap v419: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:53 smithi096 ceph-mon[112443]: pgmap v420: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:35:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:53 smithi196 ceph-mon[81815]: pgmap v420: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:35:53.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.870+0000 7fda6b83d700 1 -- 172.21.15.96:0/1049227943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 msgr2=0x7fda640fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:53.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.870+0000 7fda6b83d700 1 --2- 172.21.15.96:0/1049227943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 0x7fda640fff50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fda54009910 tx=0x7fda54009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:53.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.870+0000 7fda6b83d700 1 -- 172.21.15.96:0/1049227943 shutdown_connections 2024-04-22T15:35:53.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.870+0000 7fda6b83d700 1 --2- 172.21.15.96:0/1049227943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 0x7fda640fff50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:53.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.870+0000 7fda6b83d700 1 --2- 172.21.15.96:0/1049227943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda64105b60 0x7fda64105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:53.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.870+0000 7fda6b83d700 1 -- 172.21.15.96:0/1049227943 >> 172.21.15.96:0/1049227943 conn(0x7fda640fb600 msgr2=0x7fda640fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:53.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.870+0000 7fda6b83d700 1 -- 172.21.15.96:0/1049227943 shutdown_connections 2024-04-22T15:35:53.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.870+0000 7fda6b83d700 1 -- 172.21.15.96:0/1049227943 wait complete. 2024-04-22T15:35:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.871+0000 7fda6b83d700 1 Processor -- start 2024-04-22T15:35:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.871+0000 7fda6b83d700 1 -- start start 2024-04-22T15:35:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.871+0000 7fda6b83d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 0x7fda64197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.871+0000 7fda6b83d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda64105b60 0x7fda64197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.871+0000 7fda6b83d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda64198520 con 0x7fda640ffad0 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.871+0000 7fda6b83d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda64198660 con 0x7fda64105b60 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.871+0000 7fda695d9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 0x7fda64197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.871+0000 7fda695d9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 0x7fda64197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45406/0 (socket says 172.21.15.96:45406) 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.871+0000 7fda695d9700 1 -- 172.21.15.96:0/747479897 learned_addr learned my addr 172.21.15.96:0/747479897 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda68dd8700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda64105b60 0x7fda64197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda695d9700 1 -- 172.21.15.96:0/747479897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda64105b60 msgr2=0x7fda64197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda695d9700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda64105b60 0x7fda64197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda695d9700 1 -- 172.21.15.96:0/747479897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda540095f0 con 0x7fda640ffad0 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda695d9700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 0x7fda64197a10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fda6000b4b0 tx=0x7fda6000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda5a7fc700 1 -- 172.21.15.96:0/747479897 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda60011800 con 0x7fda640ffad0 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda5a7fc700 1 -- 172.21.15.96:0/747479897 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda60011e20 con 0x7fda640ffad0 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda641a0a80 con 0x7fda640ffad0 2024-04-22T15:35:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda5a7fc700 1 -- 172.21.15.96:0/747479897 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda6000f2e0 con 0x7fda640ffad0 2024-04-22T15:35:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.872+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda641a0e90 con 0x7fda640ffad0 2024-04-22T15:35:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.873+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda641080f0 con 0x7fda640ffad0 2024-04-22T15:35:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.874+0000 7fda5a7fc700 1 -- 172.21.15.96:0/747479897 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fda60010380 con 0x7fda640ffad0 2024-04-22T15:35:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.874+0000 7fda5a7fc700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda5006ecc0 0x7fda50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.874+0000 7fda5a7fc700 1 -- 172.21.15.96:0/747479897 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fda6008ea90 con 0x7fda640ffad0 2024-04-22T15:35:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.874+0000 7fda68dd8700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda5006ecc0 0x7fda50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.875+0000 7fda68dd8700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda5006ecc0 0x7fda50071180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fda540059d0 tx=0x7fda5401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:53.879+0000 7fda5a7fc700 1 -- 172.21.15.96:0/747479897 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fda600577f0 con 0x7fda640ffad0 2024-04-22T15:35:54.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.096+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda640697d0 con 0x7fda5006ecc0 2024-04-22T15:35:54.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.097+0000 7fda5a7fc700 1 -- 172.21.15.96:0/747479897 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fda640697d0 con 0x7fda5006ecc0 2024-04-22T15:35:54.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda5006ecc0 msgr2=0x7fda50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda5006ecc0 0x7fda50071180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fda540059d0 tx=0x7fda5401a040 comp rx=0 tx=0).stop 2024-04-22T15:35:54.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 msgr2=0x7fda64197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 0x7fda64197a10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fda6000b4b0 tx=0x7fda6000b870 comp rx=0 tx=0).stop 2024-04-22T15:35:54.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 shutdown_connections 2024-04-22T15:35:54.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda5006ecc0 0x7fda50071180 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda640ffad0 0x7fda64197a10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 --2- 172.21.15.96:0/747479897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda64105b60 0x7fda64197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 >> 172.21.15.96:0/747479897 conn(0x7fda640fb600 msgr2=0x7fda640fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:54.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 shutdown_connections 2024-04-22T15:35:54.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.099+0000 7fda6b83d700 1 -- 172.21.15.96:0/747479897 wait complete. 2024-04-22T15:35:54.113 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:35:54.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.203+0000 7fd23b0d7700 1 -- 172.21.15.96:0/4214231399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2340ffad0 msgr2=0x7fd2340fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.203+0000 7fd23b0d7700 1 --2- 172.21.15.96:0/4214231399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2340ffad0 0x7fd2340fff50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd220009910 tx=0x7fd220009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:54.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.203+0000 7fd23b0d7700 1 -- 172.21.15.96:0/4214231399 shutdown_connections 2024-04-22T15:35:54.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.203+0000 7fd23b0d7700 1 --2- 172.21.15.96:0/4214231399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2340ffad0 0x7fd2340fff50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.203+0000 7fd23b0d7700 1 --2- 172.21.15.96:0/4214231399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd234105b60 0x7fd234105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.203+0000 7fd23b0d7700 1 -- 172.21.15.96:0/4214231399 >> 172.21.15.96:0/4214231399 conn(0x7fd2340fb5e0 msgr2=0x7fd2340fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:54.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.204+0000 7fd23b0d7700 1 -- 172.21.15.96:0/4214231399 shutdown_connections 2024-04-22T15:35:54.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.204+0000 7fd23b0d7700 1 -- 172.21.15.96:0/4214231399 wait complete. 2024-04-22T15:35:54.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.204+0000 7fd23b0d7700 1 Processor -- start 2024-04-22T15:35:54.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.204+0000 7fd23b0d7700 1 -- start start 2024-04-22T15:35:54.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.204+0000 7fd23b0d7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2340ffad0 0x7fd23419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:54.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.204+0000 7fd23b0d7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd234105b60 0x7fd23419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:54.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.204+0000 7fd23b0d7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23419d2e0 con 0x7fd234105b60 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.204+0000 7fd23b0d7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd234196950 con 0x7fd2340ffad0 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.204+0000 7fd233fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd234105b60 0x7fd23419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd233fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd234105b60 0x7fd23419cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45436/0 (socket says 172.21.15.96:45436) 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd233fff700 1 -- 172.21.15.96:0/1576601075 learned_addr learned my addr 172.21.15.96:0/1576601075 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd238e73700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2340ffad0 0x7fd23419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd233fff700 1 -- 172.21.15.96:0/1576601075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2340ffad0 msgr2=0x7fd23419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd233fff700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2340ffad0 0x7fd23419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd233fff700 1 -- 172.21.15.96:0/1576601075 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2200095f0 con 0x7fd234105b60 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd233fff700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd234105b60 0x7fd23419cd50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd220006010 tx=0x7fd22000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd231ffb700 1 -- 172.21.15.96:0/1576601075 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd22001d070 con 0x7fd234105b60 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd231ffb700 1 -- 172.21.15.96:0/1576601075 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd22000f460 con 0x7fd234105b60 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd234196bd0 con 0x7fd234105b60 2024-04-22T15:35:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd231ffb700 1 -- 172.21.15.96:0/1576601075 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2200175f0 con 0x7fd234105b60 2024-04-22T15:35:54.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.205+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2341970c0 con 0x7fd234105b60 2024-04-22T15:35:54.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.207+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd234109100 con 0x7fd234105b60 2024-04-22T15:35:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.207+0000 7fd231ffb700 1 -- 172.21.15.96:0/1576601075 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd22000f5d0 con 0x7fd234105b60 2024-04-22T15:35:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.207+0000 7fd231ffb700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd22406ecc0 0x7fd224071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.208+0000 7fd231ffb700 1 -- 172.21.15.96:0/1576601075 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd220090280 con 0x7fd234105b60 2024-04-22T15:35:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.208+0000 7fd238e73700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd22406ecc0 0x7fd224071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.208+0000 7fd238e73700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd22406ecc0 0x7fd224071180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd22800ac50 tx=0x7fd22800a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:54.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.212+0000 7fd231ffb700 1 -- 172.21.15.96:0/1576601075 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd22005cb90 con 0x7fd234105b60 2024-04-22T15:35:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.434+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd234197e10 con 0x7fd22406ecc0 2024-04-22T15:35:54.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.435+0000 7fd231ffb700 1 -- 172.21.15.96:0/1576601075 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd234197e10 con 0x7fd22406ecc0 2024-04-22T15:35:54.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd22406ecc0 msgr2=0x7fd224071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd22406ecc0 0x7fd224071180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd22800ac50 tx=0x7fd22800a380 comp rx=0 tx=0).stop 2024-04-22T15:35:54.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd234105b60 msgr2=0x7fd23419cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd234105b60 0x7fd23419cd50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd220006010 tx=0x7fd22000bd70 comp rx=0 tx=0).stop 2024-04-22T15:35:54.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 shutdown_connections 2024-04-22T15:35:54.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd22406ecc0 0x7fd224071180 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2340ffad0 0x7fd23419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 --2- 172.21.15.96:0/1576601075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd234105b60 0x7fd23419cd50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 >> 172.21.15.96:0/1576601075 conn(0x7fd2340fb5e0 msgr2=0x7fd234069690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:54.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 shutdown_connections 2024-04-22T15:35:54.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.437+0000 7fd23b0d7700 1 -- 172.21.15.96:0/1576601075 wait complete. 2024-04-22T15:35:54.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.542+0000 7ff5144de700 1 -- 172.21.15.96:0/3647628924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c100d90 msgr2=0x7ff50c104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.542+0000 7ff5144de700 1 --2- 172.21.15.96:0/3647628924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c100d90 0x7ff50c104760 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff508009910 tx=0x7ff508009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:54.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.542+0000 7ff5144de700 1 -- 172.21.15.96:0/3647628924 shutdown_connections 2024-04-22T15:35:54.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.542+0000 7ff5144de700 1 --2- 172.21.15.96:0/3647628924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c100d90 0x7ff50c104760 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.542+0000 7ff5144de700 1 --2- 172.21.15.96:0/3647628924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff50c1003e0 0x7ff50c1007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.542+0000 7ff5144de700 1 -- 172.21.15.96:0/3647628924 >> 172.21.15.96:0/3647628924 conn(0x7ff50c075f30 msgr2=0x7ff50c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:54.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.542+0000 7ff5144de700 1 -- 172.21.15.96:0/3647628924 shutdown_connections 2024-04-22T15:35:54.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.542+0000 7ff5144de700 1 -- 172.21.15.96:0/3647628924 wait complete. 2024-04-22T15:35:54.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff5144de700 1 Processor -- start 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff5144de700 1 -- start start 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff5144de700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c1003e0 0x7ff50c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff5144de700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff50c100d90 0x7ff50c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff5144de700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c19d380 con 0x7ff50c1003e0 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff5144de700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c1969b0 con 0x7ff50c100d90 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff51227a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c1003e0 0x7ff50c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff51227a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c1003e0 0x7ff50c19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45444/0 (socket says 172.21.15.96:45444) 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff51227a700 1 -- 172.21.15.96:0/1789520535 learned_addr learned my addr 172.21.15.96:0/1789520535 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff511a79700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff50c100d90 0x7ff50c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff51227a700 1 -- 172.21.15.96:0/1789520535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff50c100d90 msgr2=0x7ff50c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff51227a700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff50c100d90 0x7ff50c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff51227a700 1 -- 172.21.15.96:0/1789520535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5080095f0 con 0x7ff50c1003e0 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff511a79700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff50c100d90 0x7ff50c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:35:54.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.543+0000 7ff51227a700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c1003e0 0x7ff50c19c820 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00d8d0 tx=0x7ff4fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:54.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.544+0000 7ff5037fe700 1 -- 172.21.15.96:0/1789520535 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4fc009940 con 0x7ff50c1003e0 2024-04-22T15:35:54.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.544+0000 7ff5037fe700 1 -- 172.21.15.96:0/1789520535 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4fc010460 con 0x7ff50c1003e0 2024-04-22T15:35:54.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.544+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c196c90 con 0x7ff50c1003e0 2024-04-22T15:35:54.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.544+0000 7ff5037fe700 1 -- 172.21.15.96:0/1789520535 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4fc00f5f0 con 0x7ff50c1003e0 2024-04-22T15:35:54.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.544+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c1971b0 con 0x7ff50c1003e0 2024-04-22T15:35:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.544+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50c1013a0 con 0x7ff50c1003e0 2024-04-22T15:35:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.545+0000 7ff5037fe700 1 -- 172.21.15.96:0/1789520535 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff4fc0105d0 con 0x7ff50c1003e0 2024-04-22T15:35:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.545+0000 7ff5037fe700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4f806ecc0 0x7ff4f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.546+0000 7ff511a79700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4f806ecc0 0x7ff4f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.546+0000 7ff5037fe700 1 -- 172.21.15.96:0/1789520535 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff4fc08eec0 con 0x7ff50c1003e0 2024-04-22T15:35:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.546+0000 7ff511a79700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4f806ecc0 0x7ff4f8071180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff508000c00 tx=0x7ff50801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:54.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.550+0000 7ff5037fe700 1 -- 172.21.15.96:0/1789520535 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff4fc05b750 con 0x7ff50c1003e0 2024-04-22T15:35:54.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.767+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff50c10cce0 con 0x7ff4f806ecc0 2024-04-22T15:35:54.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.777+0000 7ff5037fe700 1 -- 172.21.15.96:0/1789520535 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff50c10cce0 con 0x7ff4f806ecc0 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (14m) 7m ago 25m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (26m) 7m ago 26m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (25m) 7m ago 25m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (11m) 7m ago 26m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (11m) 7m ago 24m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (14m) 7m ago 25m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (7m) 7m ago 20m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (8m) 7m ago 20m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (7m) 7m ago 20m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (20m) 7m ago 20m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:35:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (16m) 7m ago 27m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (15m) 7m ago 24m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (12m) 7m ago 27m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (12m) 7m ago 24m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (15m) 7m ago 26m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (15m) 7m ago 24m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (11m) 7m ago 23m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (10m) 7m ago 23m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (10m) 7m ago 23m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (9m) 7m ago 22m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (8m) 7m ago 22m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (8m) 7m ago 21m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:35:54.782 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (14m) 7m ago 25m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:35:54.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4f806ecc0 msgr2=0x7ff4f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4f806ecc0 0x7ff4f8071180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff508000c00 tx=0x7ff50801a040 comp rx=0 tx=0).stop 2024-04-22T15:35:54.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c1003e0 msgr2=0x7ff50c19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c1003e0 0x7ff50c19c820 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00d8d0 tx=0x7ff4fc00dc90 comp rx=0 tx=0).stop 2024-04-22T15:35:54.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 shutdown_connections 2024-04-22T15:35:54.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4f806ecc0 0x7ff4f8071180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff50c1003e0 0x7ff50c19c820 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 --2- 172.21.15.96:0/1789520535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff50c100d90 0x7ff50c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 >> 172.21.15.96:0/1789520535 conn(0x7ff50c075f30 msgr2=0x7ff50c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:54.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 shutdown_connections 2024-04-22T15:35:54.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.780+0000 7ff5144de700 1 -- 172.21.15.96:0/1789520535 wait complete. 2024-04-22T15:35:54.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.882+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2260899449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f041017f0 msgr2=0x7f3f04105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.882+0000 7f3f0b7bc700 1 --2- 172.21.15.96:0/2260899449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f041017f0 0x7f3f04105840 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8009910 tx=0x7f3ef8009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:54.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.882+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2260899449 shutdown_connections 2024-04-22T15:35:54.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.882+0000 7f3f0b7bc700 1 --2- 172.21.15.96:0/2260899449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f041017f0 0x7f3f04105840 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.882+0000 7f3f0b7bc700 1 --2- 172.21.15.96:0/2260899449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f04100e40 0x7f3f04101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.882+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2260899449 >> 172.21.15.96:0/2260899449 conn(0x7f3f040fc710 msgr2=0x7f3f040feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:54.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.883+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2260899449 shutdown_connections 2024-04-22T15:35:54.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.883+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2260899449 wait complete. 2024-04-22T15:35:54.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.883+0000 7f3f0b7bc700 1 Processor -- start 2024-04-22T15:35:54.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.883+0000 7f3f0b7bc700 1 -- start start 2024-04-22T15:35:54.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f0b7bc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f04100e40 0x7f3f0419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:54.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f0b7bc700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f041017f0 0x7f3f0419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:54.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f0b7bc700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0419d380 con 0x7f3f04100e40 2024-04-22T15:35:54.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f0b7bc700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f04196960 con 0x7f3f041017f0 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f09558700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f04100e40 0x7f3f0419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f09558700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f04100e40 0x7f3f0419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45466/0 (socket says 172.21.15.96:45466) 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f09558700 1 -- 172.21.15.96:0/2268096802 learned_addr learned my addr 172.21.15.96:0/2268096802 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f08d57700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f041017f0 0x7f3f0419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f09558700 1 -- 172.21.15.96:0/2268096802 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f041017f0 msgr2=0x7f3f0419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f09558700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f041017f0 0x7f3f0419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f09558700 1 -- 172.21.15.96:0/2268096802 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef80095f0 con 0x7f3f04100e40 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f09558700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f04100e40 0x7f3f0419c820 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3efc0060b0 tx=0x7f3efc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.884+0000 7f3f027fc700 1 -- 172.21.15.96:0/2268096802 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3efc00dda0 con 0x7f3f04100e40 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.885+0000 7f3f027fc700 1 -- 172.21.15.96:0/2268096802 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3efc00f040 con 0x7f3f04100e40 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.885+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f04196c40 con 0x7f3f04100e40 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.885+0000 7f3f027fc700 1 -- 172.21.15.96:0/2268096802 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3efc0146f0 con 0x7f3f04100e40 2024-04-22T15:35:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.885+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f04197190 con 0x7f3f04100e40 2024-04-22T15:35:54.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.886+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f04109240 con 0x7f3f04100e40 2024-04-22T15:35:54.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.886+0000 7f3f027fc700 1 -- 172.21.15.96:0/2268096802 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3efc015400 con 0x7f3f04100e40 2024-04-22T15:35:54.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.886+0000 7f3f027fc700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3eec06ecc0 0x7f3eec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:54.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.887+0000 7f3f027fc700 1 -- 172.21.15.96:0/2268096802 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3efc08e8c0 con 0x7f3f04100e40 2024-04-22T15:35:54.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.887+0000 7f3f08d57700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3eec06ecc0 0x7f3eec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:54.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.887+0000 7f3f08d57700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3eec06ecc0 0x7f3eec071180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3f04197b20 tx=0x7f3ef801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:54.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:54.891+0000 7f3f027fc700 1 -- 172.21.15.96:0/2268096802 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3efc05b200 con 0x7f3f04100e40 2024-04-22T15:35:55.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.171+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3f04197e20 con 0x7f3f04100e40 2024-04-22T15:35:55.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.171+0000 7f3f027fc700 1 -- 172.21.15.96:0/2268096802 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f3efc05ad90 con 0x7f3f04100e40 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:35:55.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:35:55.176 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:35:55.176 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:35:55.176 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:35:55.176 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:35:55.176 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:35:55.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.173+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3eec06ecc0 msgr2=0x7f3eec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.173+0000 7f3f0b7bc700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3eec06ecc0 0x7f3eec071180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3f04197b20 tx=0x7f3ef801a040 comp rx=0 tx=0).stop 2024-04-22T15:35:55.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.173+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f04100e40 msgr2=0x7f3f0419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.173+0000 7f3f0b7bc700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f04100e40 0x7f3f0419c820 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3efc0060b0 tx=0x7f3efc00d3b0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.173+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 shutdown_connections 2024-04-22T15:35:55.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.173+0000 7f3f0b7bc700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3eec06ecc0 0x7f3eec071180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.173+0000 7f3f0b7bc700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f04100e40 0x7f3f0419c820 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.173+0000 7f3f0b7bc700 1 --2- 172.21.15.96:0/2268096802 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f041017f0 0x7f3f0419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.173+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 >> 172.21.15.96:0/2268096802 conn(0x7f3f040fc710 msgr2=0x7f3f040fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:55.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.174+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 shutdown_connections 2024-04-22T15:35:55.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.174+0000 7f3f0b7bc700 1 -- 172.21.15.96:0/2268096802 wait complete. 2024-04-22T15:35:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.276+0000 7f2390e42700 1 -- 172.21.15.96:0/2597271152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 msgr2=0x7f238c0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.276+0000 7f2390e42700 1 --2- 172.21.15.96:0/2597271152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 0x7f238c0fff20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2374009910 tx=0x7f2374009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.276+0000 7f2390e42700 1 -- 172.21.15.96:0/2597271152 shutdown_connections 2024-04-22T15:35:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.276+0000 7f2390e42700 1 --2- 172.21.15.96:0/2597271152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 0x7f238c0fff20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.276+0000 7f2390e42700 1 --2- 172.21.15.96:0/2597271152 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f238c105aa0 0x7f238c105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.276+0000 7f2390e42700 1 -- 172.21.15.96:0/2597271152 >> 172.21.15.96:0/2597271152 conn(0x7f238c076740 msgr2=0x7f238c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:55.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.276+0000 7f2390e42700 1 -- 172.21.15.96:0/2597271152 shutdown_connections 2024-04-22T15:35:55.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.276+0000 7f2390e42700 1 -- 172.21.15.96:0/2597271152 wait complete. 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f2390e42700 1 Processor -- start 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f2390e42700 1 -- start start 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f2390e42700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 0x7f238c197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f2390e42700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f238c105aa0 0x7f238c197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f2390e42700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c198380 con 0x7f238c0ffaa0 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f2390e42700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c1984c0 con 0x7f238c105aa0 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f238a59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 0x7f238c197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f238a59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 0x7f238c197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45484/0 (socket says 172.21.15.96:45484) 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f238a59c700 1 -- 172.21.15.96:0/980770878 learned_addr learned my addr 172.21.15.96:0/980770878 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f2389d9b700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f238c105aa0 0x7f238c197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f238a59c700 1 -- 172.21.15.96:0/980770878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f238c105aa0 msgr2=0x7f238c197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f238a59c700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f238c105aa0 0x7f238c197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.277+0000 7f238a59c700 1 -- 172.21.15.96:0/980770878 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23740095f0 con 0x7f238c0ffaa0 2024-04-22T15:35:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.278+0000 7f238a59c700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 0x7f238c197900 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f238400cfd0 tx=0x7f238400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.278+0000 7f23837fe700 1 -- 172.21.15.96:0/980770878 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f238400ad70 con 0x7f238c0ffaa0 2024-04-22T15:35:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.278+0000 7f23837fe700 1 -- 172.21.15.96:0/980770878 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2384007e70 con 0x7f238c0ffaa0 2024-04-22T15:35:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.278+0000 7f23837fe700 1 -- 172.21.15.96:0/980770878 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2384011810 con 0x7f238c0ffaa0 2024-04-22T15:35:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.278+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c198740 con 0x7f238c0ffaa0 2024-04-22T15:35:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.278+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c1a0d00 con 0x7f238c0ffaa0 2024-04-22T15:35:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.279+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c1080c0 con 0x7f238c0ffaa0 2024-04-22T15:35:55.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.279+0000 7f23837fe700 1 -- 172.21.15.96:0/980770878 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f23840119b0 con 0x7f238c0ffaa0 2024-04-22T15:35:55.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.280+0000 7f23837fe700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f237806ecc0 0x7f2378071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:55.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.280+0000 7f23837fe700 1 -- 172.21.15.96:0/980770878 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f238408eee0 con 0x7f238c0ffaa0 2024-04-22T15:35:55.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.280+0000 7f2389d9b700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f237806ecc0 0x7f2378071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:55.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.280+0000 7f2389d9b700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f237806ecc0 0x7f2378071180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f238c068dd0 tx=0x7f237401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:55.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.285+0000 7f23837fe700 1 -- 172.21.15.96:0/980770878 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2384057c40 con 0x7f238c0ffaa0 2024-04-22T15:35:55.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.530+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f238c068c20 con 0x7f238c0ffaa0 2024-04-22T15:35:55.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.530+0000 7f23837fe700 1 -- 172.21.15.96:0/980770878 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f238405b300 con 0x7f238c0ffaa0 2024-04-22T15:35:55.533 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:35:55.534 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:35:55.535 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:35:55.536 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:35:55.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f237806ecc0 msgr2=0x7f2378071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f237806ecc0 0x7f2378071180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f238c068dd0 tx=0x7f237401a040 comp rx=0 tx=0).stop 2024-04-22T15:35:55.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 msgr2=0x7f238c197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 0x7f238c197900 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f238400cfd0 tx=0x7f238400a340 comp rx=0 tx=0).stop 2024-04-22T15:35:55.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 shutdown_connections 2024-04-22T15:35:55.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f237806ecc0 0x7f2378071180 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0ffaa0 0x7f238c197900 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 --2- 172.21.15.96:0/980770878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f238c105aa0 0x7f238c197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 >> 172.21.15.96:0/980770878 conn(0x7f238c076740 msgr2=0x7f238c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:55.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 shutdown_connections 2024-04-22T15:35:55.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.533+0000 7f2390e42700 1 -- 172.21.15.96:0/980770878 wait complete. 2024-04-22T15:35:55.538 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:35:55.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.637+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/2393563481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 msgr2=0x7fd79c10aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.637+0000 7fd7a3ac6700 1 --2- 172.21.15.96:0/2393563481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 0x7fd79c10aca0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009910 tx=0x7fd78c009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:55.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.637+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/2393563481 shutdown_connections 2024-04-22T15:35:55.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.637+0000 7fd7a3ac6700 1 --2- 172.21.15.96:0/2393563481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 0x7fd79c10aca0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.637+0000 7fd7a3ac6700 1 --2- 172.21.15.96:0/2393563481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd79c101e90 0x7fd79c102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.637+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/2393563481 >> 172.21.15.96:0/2393563481 conn(0x7fd79c0fb680 msgr2=0x7fd79c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:55.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.638+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/2393563481 shutdown_connections 2024-04-22T15:35:55.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.638+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/2393563481 wait complete. 2024-04-22T15:35:55.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.638+0000 7fd7a3ac6700 1 Processor -- start 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.638+0000 7fd7a3ac6700 1 -- start start 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.638+0000 7fd7a3ac6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd79c101e90 0x7fd79c19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.638+0000 7fd7a3ac6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 0x7fd79c19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.638+0000 7fd7a3ac6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c19d380 con 0x7fd79c1027b0 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.638+0000 7fd7a3ac6700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c1969f0 con 0x7fd79c101e90 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.638+0000 7fd7a1061700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 0x7fd79c19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd7a1061700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 0x7fd79c19cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45506/0 (socket says 172.21.15.96:45506) 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd7a1061700 1 -- 172.21.15.96:0/4173874452 learned_addr learned my addr 172.21.15.96:0/4173874452 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd7a1862700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd79c101e90 0x7fd79c19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd7a1061700 1 -- 172.21.15.96:0/4173874452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd79c101e90 msgr2=0x7fd79c19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd7a1061700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd79c101e90 0x7fd79c19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd7a1061700 1 -- 172.21.15.96:0/4173874452 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c0095f0 con 0x7fd79c1027b0 2024-04-22T15:35:55.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd7a1061700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 0x7fd79c19cdf0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0092e0 tx=0x7fd78c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:55.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd792ffd700 1 -- 172.21.15.96:0/4173874452 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd78c01d070 con 0x7fd79c1027b0 2024-04-22T15:35:55.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd792ffd700 1 -- 172.21.15.96:0/4173874452 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd78c00f460 con 0x7fd79c1027b0 2024-04-22T15:35:55.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79c196c70 con 0x7fd79c1027b0 2024-04-22T15:35:55.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd792ffd700 1 -- 172.21.15.96:0/4173874452 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd78c017630 con 0x7fd79c1027b0 2024-04-22T15:35:55.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.639+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79c197160 con 0x7fd79c1027b0 2024-04-22T15:35:55.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.640+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79c1055b0 con 0x7fd79c1027b0 2024-04-22T15:35:55.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.641+0000 7fd792ffd700 1 -- 172.21.15.96:0/4173874452 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd78c00f5d0 con 0x7fd79c1027b0 2024-04-22T15:35:55.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.641+0000 7fd792ffd700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd78806ecc0 0x7fd788071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:55.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.641+0000 7fd792ffd700 1 -- 172.21.15.96:0/4173874452 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd78c0901a0 con 0x7fd79c1027b0 2024-04-22T15:35:55.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.641+0000 7fd7a1862700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd78806ecc0 0x7fd788071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:55.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.642+0000 7fd7a1862700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd78806ecc0 0x7fd788071180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd79800ac50 tx=0x7fd79800a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:55.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.646+0000 7fd792ffd700 1 -- 172.21.15.96:0/4173874452 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd78c05cab0 con 0x7fd79c1027b0 2024-04-22T15:35:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:55 smithi096 ceph-mon[112443]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:55 smithi096 ceph-mon[112443]: pgmap v421: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:55 smithi096 ceph-mon[112443]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:55 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2268096802' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:35:55.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:55 smithi196 ceph-mon[81815]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:55.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:55 smithi196 ceph-mon[81815]: pgmap v421: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:55.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:55 smithi196 ceph-mon[81815]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:55.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:55 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2268096802' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:35:55.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.862+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd79c197eb0 con 0x7fd78806ecc0 2024-04-22T15:35:55.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.864+0000 7fd792ffd700 1 -- 172.21.15.96:0/4173874452 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd79c197eb0 con 0x7fd78806ecc0 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:35:55.867 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:35:55.868 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:35:55.868 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:35:55.868 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:35:55.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd78806ecc0 msgr2=0x7fd788071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd78806ecc0 0x7fd788071180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd79800ac50 tx=0x7fd79800a380 comp rx=0 tx=0).stop 2024-04-22T15:35:55.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 msgr2=0x7fd79c19cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 0x7fd79c19cdf0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0092e0 tx=0x7fd78c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:35:55.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 shutdown_connections 2024-04-22T15:35:55.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd78806ecc0 0x7fd788071180 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd79c101e90 0x7fd79c19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 --2- 172.21.15.96:0/4173874452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1027b0 0x7fd79c19cdf0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 >> 172.21.15.96:0/4173874452 conn(0x7fd79c0fb680 msgr2=0x7fd79c103180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:55.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 shutdown_connections 2024-04-22T15:35:55.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.866+0000 7fd7a3ac6700 1 -- 172.21.15.96:0/4173874452 wait complete. 2024-04-22T15:35:55.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/884341942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 msgr2=0x7fa5c4105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 --2- 172.21.15.96:0/884341942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 0x7fa5c4105820 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009910 tx=0x7fa5b8009c20 comp rx=0 tx=0).stop 2024-04-22T15:35:55.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/884341942 shutdown_connections 2024-04-22T15:35:55.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 --2- 172.21.15.96:0/884341942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 0x7fa5c4105820 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 --2- 172.21.15.96:0/884341942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5c4100e20 0x7fa5c4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/884341942 >> 172.21.15.96:0/884341942 conn(0x7fa5c40fc6d0 msgr2=0x7fa5c40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:55.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/884341942 shutdown_connections 2024-04-22T15:35:55.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/884341942 wait complete. 2024-04-22T15:35:55.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 Processor -- start 2024-04-22T15:35:55.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.969+0000 7fa5c9cb0700 1 -- start start 2024-04-22T15:35:55.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c9cb0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5c4100e20 0x7fa5c40731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:55.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c9cb0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 0x7fa5c4073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:55.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c9cb0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c4073c60 con 0x7fa5c41017d0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c9cb0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c4073da0 con 0x7fa5c4100e20 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 0x7fa5c4073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 0x7fa5c4073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45524/0 (socket says 172.21.15.96:45524) 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c2ffd700 1 -- 172.21.15.96:0/461802377 learned_addr learned my addr 172.21.15.96:0/461802377 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c37fe700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5c4100e20 0x7fa5c40731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c2ffd700 1 -- 172.21.15.96:0/461802377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5c4100e20 msgr2=0x7fa5c40731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c2ffd700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5c4100e20 0x7fa5c40731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c2ffd700 1 -- 172.21.15.96:0/461802377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80095f0 con 0x7fa5c41017d0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c2ffd700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 0x7fa5c4073720 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800bd70 tx=0x7fa5b800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c0ff9700 1 -- 172.21.15.96:0/461802377 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5b801d070 con 0x7fa5c41017d0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c4074020 con 0x7fa5c41017d0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c0ff9700 1 -- 172.21.15.96:0/461802377 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5b80056e0 con 0x7fa5c41017d0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c0ff9700 1 -- 172.21.15.96:0/461802377 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5b800fe50 con 0x7fa5c41017d0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.970+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c41a2f60 con 0x7fa5c41017d0 2024-04-22T15:35:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.971+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c4109220 con 0x7fa5c41017d0 2024-04-22T15:35:55.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.972+0000 7fa5c0ff9700 1 -- 172.21.15.96:0/461802377 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa5b800f680 con 0x7fa5c41017d0 2024-04-22T15:35:55.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.972+0000 7fa5c0ff9700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa5ac06ecc0 0x7fa5ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:35:55.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.972+0000 7fa5c0ff9700 1 -- 172.21.15.96:0/461802377 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa5b8090190 con 0x7fa5c41017d0 2024-04-22T15:35:55.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.972+0000 7fa5c37fe700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa5ac06ecc0 0x7fa5ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:35:55.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.973+0000 7fa5c37fe700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa5ac06ecc0 0x7fa5ac071180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4005950 tx=0x7fa5b400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:35:55.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:55.977+0000 7fa5c0ff9700 1 -- 172.21.15.96:0/461802377 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa5b8058ef0 con 0x7fa5c41017d0 2024-04-22T15:35:56.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.259+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa5c4102300 con 0x7fa5c41017d0 2024-04-22T15:35:56.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.259+0000 7fa5c0ff9700 1 -- 172.21.15.96:0/461802377 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fa5b805c5b0 con 0x7fa5c41017d0 2024-04-22T15:35:56.263 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:35:56.263 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:35:56.263 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:35:56.263 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:35:56.263 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:35:56.263 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:35:56.263 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:35:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.261+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa5ac06ecc0 msgr2=0x7fa5ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa5ac06ecc0 0x7fa5ac071180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4005950 tx=0x7fa5b400b410 comp rx=0 tx=0).stop 2024-04-22T15:35:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 msgr2=0x7fa5c4073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:35:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 0x7fa5c4073720 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800bd70 tx=0x7fa5b800bda0 comp rx=0 tx=0).stop 2024-04-22T15:35:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 shutdown_connections 2024-04-22T15:35:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa5ac06ecc0 0x7fa5ac071180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5c4100e20 0x7fa5c40731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 --2- 172.21.15.96:0/461802377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5c41017d0 0x7fa5c4073720 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:35:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 >> 172.21.15.96:0/461802377 conn(0x7fa5c40fc6d0 msgr2=0x7fa5c40fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:35:56.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 shutdown_connections 2024-04-22T15:35:56.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:35:56.262+0000 7fa5c9cb0700 1 -- 172.21.15.96:0/461802377 wait complete. 2024-04-22T15:35:56.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:56 smithi096 ceph-mon[112443]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:56.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:56 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/980770878' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:35:56.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:56 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/461802377' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:35:56.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:56 smithi196 ceph-mon[81815]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:56.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:56 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/980770878' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:35:56.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:56 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/461802377' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:35:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:57 smithi096 ceph-mon[112443]: from='client.34836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:57 smithi096 ceph-mon[112443]: pgmap v422: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:57 smithi196 ceph-mon[81815]: from='client.34836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:35:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:57 smithi196 ceph-mon[81815]: pgmap v422: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:35:59 smithi096 ceph-mon[112443]: pgmap v423: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:35:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:35:59 smithi196 ceph-mon[81815]: pgmap v423: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:01 smithi096 ceph-mon[112443]: pgmap v424: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:01 smithi196 ceph-mon[81815]: pgmap v424: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:03 smithi096 ceph-mon[112443]: pgmap v425: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:03 smithi196 ceph-mon[81815]: pgmap v425: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:05.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:05 smithi096 ceph-mon[112443]: pgmap v426: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:05 smithi196 ceph-mon[81815]: pgmap v426: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:07 smithi196 ceph-mon[81815]: pgmap v427: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:07 smithi096 ceph-mon[112443]: pgmap v427: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:08.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:36:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:36:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:09 smithi196 ceph-mon[81815]: pgmap v428: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:09 smithi096 ceph-mon[112443]: pgmap v428: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:11 smithi196 ceph-mon[81815]: pgmap v429: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:11.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:11 smithi096 ceph-mon[112443]: pgmap v429: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:13 smithi196 ceph-mon[81815]: pgmap v430: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:13 smithi096 ceph-mon[112443]: pgmap v430: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:15 smithi196 ceph-mon[81815]: pgmap v431: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:15.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:15 smithi096 ceph-mon[112443]: pgmap v431: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:17 smithi196 ceph-mon[81815]: pgmap v432: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:17.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:17 smithi096 ceph-mon[112443]: pgmap v432: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:19 smithi196 ceph-mon[81815]: pgmap v433: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:19 smithi096 ceph-mon[112443]: pgmap v433: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:21.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:21 smithi096 ceph-mon[112443]: pgmap v434: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:21 smithi196 ceph-mon[81815]: pgmap v434: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:23 smithi096 ceph-mon[112443]: pgmap v435: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:36:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:23 smithi196 ceph-mon[81815]: pgmap v435: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:36:25.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:25 smithi096 ceph-mon[112443]: pgmap v436: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:25 smithi196 ceph-mon[81815]: pgmap v436: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:26.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.377+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/2780199716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 msgr2=0x7fc3c01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.377+0000 7fc3c7c5e700 1 --2- 172.21.15.96:0/2780199716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 0x7fc3c01011f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc009910 tx=0x7fc3bc009c20 comp rx=0 tx=0).stop 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.377+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/2780199716 shutdown_connections 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.377+0000 7fc3c7c5e700 1 --2- 172.21.15.96:0/2780199716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3c01017c0 0x7fc3c0105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.377+0000 7fc3c7c5e700 1 --2- 172.21.15.96:0/2780199716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 0x7fc3c01011f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.377+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/2780199716 >> 172.21.15.96:0/2780199716 conn(0x7fc3c00fc6e0 msgr2=0x7fc3c00feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/2780199716 shutdown_connections 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/2780199716 wait complete. 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c7c5e700 1 Processor -- start 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c7c5e700 1 -- start start 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c7c5e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 0x7fc3c019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c7c5e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3c01017c0 0x7fc3c019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:26.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c7c5e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c019f550 con 0x7fc3c0100e10 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c7c5e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c0198b30 con 0x7fc3c01017c0 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c59fa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 0x7fc3c019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c59fa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 0x7fc3c019e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52830/0 (socket says 172.21.15.96:52830) 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.378+0000 7fc3c59fa700 1 -- 172.21.15.96:0/3952744401 learned_addr learned my addr 172.21.15.96:0/3952744401 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3c51f9700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3c01017c0 0x7fc3c019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3c59fa700 1 -- 172.21.15.96:0/3952744401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3c01017c0 msgr2=0x7fc3c019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3c59fa700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3c01017c0 0x7fc3c019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3c59fa700 1 -- 172.21.15.96:0/3952744401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc0095f0 con 0x7fc3c0100e10 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3c51f9700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3c01017c0 0x7fc3c019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3c59fa700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 0x7fc3c019e9f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00bd70 tx=0x7fc3bc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3b6ffd700 1 -- 172.21.15.96:0/3952744401 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3bc01d070 con 0x7fc3c0100e10 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3b6ffd700 1 -- 172.21.15.96:0/3952744401 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3bc0056e0 con 0x7fc3c0100e10 2024-04-22T15:36:26.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c0198db0 con 0x7fc3c0100e10 2024-04-22T15:36:26.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3b6ffd700 1 -- 172.21.15.96:0/3952744401 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3bc017400 con 0x7fc3c0100e10 2024-04-22T15:36:26.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.379+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c01992a0 con 0x7fc3c0100e10 2024-04-22T15:36:26.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.380+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c0104f90 con 0x7fc3c0100e10 2024-04-22T15:36:26.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.381+0000 7fc3b6ffd700 1 -- 172.21.15.96:0/3952744401 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc3bc003a60 con 0x7fc3c0100e10 2024-04-22T15:36:26.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.381+0000 7fc3b6ffd700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc3ac06ecc0 0x7fc3ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:26.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.381+0000 7fc3b6ffd700 1 -- 172.21.15.96:0/3952744401 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc3bc090260 con 0x7fc3c0100e10 2024-04-22T15:36:26.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.381+0000 7fc3c51f9700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc3ac06ecc0 0x7fc3ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:26.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.382+0000 7fc3c51f9700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc3ac06ecc0 0x7fc3ac071180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0005950 tx=0x7fc3b00058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:26.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.386+0000 7fc3b6ffd700 1 -- 172.21.15.96:0/3952744401 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc3bc05cb70 con 0x7fc3c0100e10 2024-04-22T15:36:26.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.609+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3c019a060 con 0x7fc3ac06ecc0 2024-04-22T15:36:26.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.610+0000 7fc3b6ffd700 1 -- 172.21.15.96:0/3952744401 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc3c019a060 con 0x7fc3ac06ecc0 2024-04-22T15:36:26.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.612+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc3ac06ecc0 msgr2=0x7fc3ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:26.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.612+0000 7fc3c7c5e700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc3ac06ecc0 0x7fc3ac071180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0005950 tx=0x7fc3b00058e0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.613+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 msgr2=0x7fc3c019e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:26.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.613+0000 7fc3c7c5e700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 0x7fc3c019e9f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00bd70 tx=0x7fc3bc00be50 comp rx=0 tx=0).stop 2024-04-22T15:36:26.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.613+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 shutdown_connections 2024-04-22T15:36:26.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.613+0000 7fc3c7c5e700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc3ac06ecc0 0x7fc3ac071180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.613+0000 7fc3c7c5e700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3c0100e10 0x7fc3c019e9f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.613+0000 7fc3c7c5e700 1 --2- 172.21.15.96:0/3952744401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3c01017c0 0x7fc3c019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.613+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 >> 172.21.15.96:0/3952744401 conn(0x7fc3c00fc6e0 msgr2=0x7fc3c00fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:26.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.613+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 shutdown_connections 2024-04-22T15:36:26.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.613+0000 7fc3c7c5e700 1 -- 172.21.15.96:0/3952744401 wait complete. 2024-04-22T15:36:26.626 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:36:26.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.717+0000 7f635b7d3700 1 -- 172.21.15.96:0/847040526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ff520 msgr2=0x7f63540ff900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:26.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.717+0000 7f635b7d3700 1 --2- 172.21.15.96:0/847040526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ff520 0x7f63540ff900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6350009910 tx=0x7f6350009c20 comp rx=0 tx=0).stop 2024-04-22T15:36:26.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.717+0000 7f635b7d3700 1 -- 172.21.15.96:0/847040526 shutdown_connections 2024-04-22T15:36:26.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.717+0000 7f635b7d3700 1 --2- 172.21.15.96:0/847040526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63540ffed0 0x7f6354103f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.717+0000 7f635b7d3700 1 --2- 172.21.15.96:0/847040526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ff520 0x7f63540ff900 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.717+0000 7f635b7d3700 1 -- 172.21.15.96:0/847040526 >> 172.21.15.96:0/847040526 conn(0x7f6354075f00 msgr2=0x7f63540fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:26.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.717+0000 7f635b7d3700 1 -- 172.21.15.96:0/847040526 shutdown_connections 2024-04-22T15:36:26.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.717+0000 7f635b7d3700 1 -- 172.21.15.96:0/847040526 wait complete. 2024-04-22T15:36:26.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f635b7d3700 1 Processor -- start 2024-04-22T15:36:26.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f635b7d3700 1 -- start start 2024-04-22T15:36:26.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f635b7d3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63540ff520 0x7f635419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f635b7d3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ffed0 0x7f635419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f635b7d3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635419d340 con 0x7f63540ffed0 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f635b7d3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541969b0 con 0x7f63540ff520 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f6358d6e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ffed0 0x7f635419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f6358d6e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ffed0 0x7f635419cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52848/0 (socket says 172.21.15.96:52848) 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f6358d6e700 1 -- 172.21.15.96:0/2108841633 learned_addr learned my addr 172.21.15.96:0/2108841633 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f635956f700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63540ff520 0x7f635419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f6358d6e700 1 -- 172.21.15.96:0/2108841633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63540ff520 msgr2=0x7f635419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f6358d6e700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63540ff520 0x7f635419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.718+0000 7f6358d6e700 1 -- 172.21.15.96:0/2108841633 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63500095f0 con 0x7f63540ffed0 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.719+0000 7f6358d6e700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ffed0 0x7f635419cdb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f634400c960 tx=0x7f634400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:26.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.719+0000 7f634a7fc700 1 -- 172.21.15.96:0/2108841633 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6344007a70 con 0x7f63540ffed0 2024-04-22T15:36:26.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.719+0000 7f634a7fc700 1 -- 172.21.15.96:0/2108841633 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6344007bd0 con 0x7f63540ffed0 2024-04-22T15:36:26.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.719+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6354196c90 con 0x7f63540ffed0 2024-04-22T15:36:26.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.719+0000 7f634a7fc700 1 -- 172.21.15.96:0/2108841633 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6344018630 con 0x7f63540ffed0 2024-04-22T15:36:26.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.719+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63541971e0 con 0x7f63540ffed0 2024-04-22T15:36:26.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.720+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6354100a00 con 0x7f63540ffed0 2024-04-22T15:36:26.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.720+0000 7f634a7fc700 1 -- 172.21.15.96:0/2108841633 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f634400f450 con 0x7f63540ffed0 2024-04-22T15:36:26.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.721+0000 7f634a7fc700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f634006ecc0 0x7f6340071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:26.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.721+0000 7f634a7fc700 1 -- 172.21.15.96:0/2108841633 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f634408ee10 con 0x7f63540ffed0 2024-04-22T15:36:26.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.721+0000 7f635956f700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f634006ecc0 0x7f6340071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:26.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.721+0000 7f635956f700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f634006ecc0 0x7f6340071180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6350006010 tx=0x7f635001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:26.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.725+0000 7f634a7fc700 1 -- 172.21.15.96:0/2108841633 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f634405b6a0 con 0x7f63540ffed0 2024-04-22T15:36:26.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.946+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6354197ee0 con 0x7f634006ecc0 2024-04-22T15:36:26.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.947+0000 7f634a7fc700 1 -- 172.21.15.96:0/2108841633 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6354197ee0 con 0x7f634006ecc0 2024-04-22T15:36:26.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f634006ecc0 msgr2=0x7f6340071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f634006ecc0 0x7f6340071180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6350006010 tx=0x7f635001a040 comp rx=0 tx=0).stop 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ffed0 msgr2=0x7f635419cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ffed0 0x7f635419cdb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f634400c960 tx=0x7f634400cc70 comp rx=0 tx=0).stop 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 shutdown_connections 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f634006ecc0 0x7f6340071180 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63540ff520 0x7f635419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 --2- 172.21.15.96:0/2108841633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63540ffed0 0x7f635419cdb0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 >> 172.21.15.96:0/2108841633 conn(0x7f6354075f00 msgr2=0x7f63540fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 shutdown_connections 2024-04-22T15:36:26.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:26.949+0000 7f635b7d3700 1 -- 172.21.15.96:0/2108841633 wait complete. 2024-04-22T15:36:27.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.053+0000 7faa21a1b700 1 -- 172.21.15.96:0/1367263421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c0687b0 msgr2=0x7faa1c068b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.053+0000 7faa21a1b700 1 --2- 172.21.15.96:0/1367263421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c0687b0 0x7faa1c068b90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009910 tx=0x7faa0c009c20 comp rx=0 tx=0).stop 2024-04-22T15:36:27.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.053+0000 7faa21a1b700 1 -- 172.21.15.96:0/1367263421 shutdown_connections 2024-04-22T15:36:27.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.053+0000 7faa21a1b700 1 --2- 172.21.15.96:0/1367263421 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa1c069160 0x7faa1c105780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.053+0000 7faa21a1b700 1 --2- 172.21.15.96:0/1367263421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c0687b0 0x7faa1c068b90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.053+0000 7faa21a1b700 1 -- 172.21.15.96:0/1367263421 >> 172.21.15.96:0/1367263421 conn(0x7faa1c076790 msgr2=0x7faa1c076ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:27.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.053+0000 7faa21a1b700 1 -- 172.21.15.96:0/1367263421 shutdown_connections 2024-04-22T15:36:27.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.053+0000 7faa21a1b700 1 -- 172.21.15.96:0/1367263421 wait complete. 2024-04-22T15:36:27.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.053+0000 7faa21a1b700 1 Processor -- start 2024-04-22T15:36:27.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa21a1b700 1 -- start start 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa21a1b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa1c0687b0 0x7faa1c196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa21a1b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c069160 0x7faa1c196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa21a1b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c1975e0 con 0x7faa1c069160 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa21a1b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c197750 con 0x7faa1c0687b0 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa127fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c069160 0x7faa1c196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa127fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c069160 0x7faa1c196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52874/0 (socket says 172.21.15.96:52874) 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa127fc700 1 -- 172.21.15.96:0/941735618 learned_addr learned my addr 172.21.15.96:0/941735618 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa1affd700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa1c0687b0 0x7faa1c196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa127fc700 1 -- 172.21.15.96:0/941735618 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa1c0687b0 msgr2=0x7faa1c196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa127fc700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa1c0687b0 0x7faa1c196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa127fc700 1 -- 172.21.15.96:0/941735618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c0095f0 con 0x7faa1c069160 2024-04-22T15:36:27.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.054+0000 7faa127fc700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c069160 0x7faa1c196fe0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7faa0400c960 tx=0x7faa0400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:27.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.055+0000 7faa20a19700 1 -- 172.21.15.96:0/941735618 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa04007a70 con 0x7faa1c069160 2024-04-22T15:36:27.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.055+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c19b4c0 con 0x7faa1c069160 2024-04-22T15:36:27.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.055+0000 7faa20a19700 1 -- 172.21.15.96:0/941735618 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa04007bd0 con 0x7faa1c069160 2024-04-22T15:36:27.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.055+0000 7faa20a19700 1 -- 172.21.15.96:0/941735618 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa040186c0 con 0x7faa1c069160 2024-04-22T15:36:27.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.055+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c19ba10 con 0x7faa1c069160 2024-04-22T15:36:27.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.056+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c109120 con 0x7faa1c069160 2024-04-22T15:36:27.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.056+0000 7faa20a19700 1 -- 172.21.15.96:0/941735618 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faa0400faa0 con 0x7faa1c069160 2024-04-22T15:36:27.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.057+0000 7faa20a19700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa0806ecd0 0x7faa08071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:27.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.057+0000 7faa20a19700 1 -- 172.21.15.96:0/941735618 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7faa0408ee50 con 0x7faa1c069160 2024-04-22T15:36:27.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.057+0000 7faa1affd700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa0806ecd0 0x7faa08071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:27.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.057+0000 7faa1affd700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa0806ecd0 0x7faa08071190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00b5c0 tx=0x7faa0c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:27.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.061+0000 7faa20a19700 1 -- 172.21.15.96:0/941735618 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faa0405b4d0 con 0x7faa1c069160 2024-04-22T15:36:27.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.270+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faa1c10dd70 con 0x7faa0806ecd0 2024-04-22T15:36:27.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.287+0000 7faa20a19700 1 -- 172.21.15.96:0/941735618 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7faa1c10dd70 con 0x7faa0806ecd0 2024-04-22T15:36:27.290 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (15m) 8m ago 26m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (26m) 8m ago 26m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (25m) 7m ago 25m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (12m) 8m ago 26m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (12m) 7m ago 25m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (14m) 8m ago 26m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (8m) 8m ago 20m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (8m) 8m ago 20m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (7m) 7m ago 20m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (20m) 7m ago 20m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (17m) 8m ago 27m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:36:27.291 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (16m) 7m ago 25m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (13m) 8m ago 28m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (13m) 7m ago 25m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (16m) 8m ago 26m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (15m) 7m ago 25m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (11m) 8m ago 24m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (11m) 8m ago 24m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (10m) 8m ago 23m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (10m) 7m ago 23m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (9m) 7m ago 22m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (8m) 7m ago 22m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:36:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (15m) 8m ago 25m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:36:27.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.289+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa0806ecd0 msgr2=0x7faa08071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.289+0000 7faa21a1b700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa0806ecd0 0x7faa08071190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00b5c0 tx=0x7faa0c01a040 comp rx=0 tx=0).stop 2024-04-22T15:36:27.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.289+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c069160 msgr2=0x7faa1c196fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.289+0000 7faa21a1b700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c069160 0x7faa1c196fe0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7faa0400c960 tx=0x7faa0400cd20 comp rx=0 tx=0).stop 2024-04-22T15:36:27.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.290+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 shutdown_connections 2024-04-22T15:36:27.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.290+0000 7faa21a1b700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa0806ecd0 0x7faa08071190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.290+0000 7faa21a1b700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa1c0687b0 0x7faa1c196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.290+0000 7faa21a1b700 1 --2- 172.21.15.96:0/941735618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c069160 0x7faa1c196fe0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.290+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 >> 172.21.15.96:0/941735618 conn(0x7faa1c076790 msgr2=0x7faa1c1070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:27.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.290+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 shutdown_connections 2024-04-22T15:36:27.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.290+0000 7faa21a1b700 1 -- 172.21.15.96:0/941735618 wait complete. 2024-04-22T15:36:27.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.392+0000 7f8306223700 1 -- 172.21.15.96:0/3567033482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83000ffaa0 msgr2=0x7f83000fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.392+0000 7f8306223700 1 --2- 172.21.15.96:0/3567033482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83000ffaa0 0x7f83000fff20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009910 tx=0x7f82e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:36:27.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.392+0000 7f8306223700 1 -- 172.21.15.96:0/3567033482 shutdown_connections 2024-04-22T15:36:27.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.392+0000 7f8306223700 1 --2- 172.21.15.96:0/3567033482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83000ffaa0 0x7f83000fff20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.392+0000 7f8306223700 1 --2- 172.21.15.96:0/3567033482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8300105aa0 0x7f8300105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.392+0000 7f8306223700 1 -- 172.21.15.96:0/3567033482 >> 172.21.15.96:0/3567033482 conn(0x7f8300076740 msgr2=0x7f83000fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:27.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.392+0000 7f8306223700 1 -- 172.21.15.96:0/3567033482 shutdown_connections 2024-04-22T15:36:27.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.392+0000 7f8306223700 1 -- 172.21.15.96:0/3567033482 wait complete. 2024-04-22T15:36:27.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f8306223700 1 Processor -- start 2024-04-22T15:36:27.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f8306223700 1 -- start start 2024-04-22T15:36:27.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f8306223700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83000ffaa0 0x7f83000fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:27.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f8306223700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8300105aa0 0x7f83000fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:27.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f8306223700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83000fc6a0 con 0x7f8300105aa0 2024-04-22T15:36:27.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f8306223700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83000fc810 con 0x7f83000ffaa0 2024-04-22T15:36:27.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f82feffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8300105aa0 0x7f83000fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f82feffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8300105aa0 0x7f83000fc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52892/0 (socket says 172.21.15.96:52892) 2024-04-22T15:36:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f82feffd700 1 -- 172.21.15.96:0/2390152360 learned_addr learned my addr 172.21.15.96:0/2390152360 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.393+0000 7f82ff7fe700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83000ffaa0 0x7f83000fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.394+0000 7f82feffd700 1 -- 172.21.15.96:0/2390152360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83000ffaa0 msgr2=0x7f83000fbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.394+0000 7f82feffd700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83000ffaa0 0x7f83000fbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.394+0000 7f82feffd700 1 -- 172.21.15.96:0/2390152360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e80095f0 con 0x7f8300105aa0 2024-04-22T15:36:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.394+0000 7f82feffd700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8300105aa0 0x7f83000fc160 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f82e8000c00 tx=0x7f82e800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.394+0000 7f82fcff9700 1 -- 172.21.15.96:0/2390152360 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e801d070 con 0x7f8300105aa0 2024-04-22T15:36:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.394+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83000fca90 con 0x7f8300105aa0 2024-04-22T15:36:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.394+0000 7f82fcff9700 1 -- 172.21.15.96:0/2390152360 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82e80056e0 con 0x7f8300105aa0 2024-04-22T15:36:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.394+0000 7f82fcff9700 1 -- 172.21.15.96:0/2390152360 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e800fe40 con 0x7f8300105aa0 2024-04-22T15:36:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.394+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83001a2f90 con 0x7f8300105aa0 2024-04-22T15:36:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.395+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83001011e0 con 0x7f8300105aa0 2024-04-22T15:36:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.396+0000 7f82fcff9700 1 -- 172.21.15.96:0/2390152360 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f82e800f680 con 0x7f8300105aa0 2024-04-22T15:36:27.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.396+0000 7f82fcff9700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82ec06ec70 0x7f82ec071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:27.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.397+0000 7f82fcff9700 1 -- 172.21.15.96:0/2390152360 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f82e8090070 con 0x7f8300105aa0 2024-04-22T15:36:27.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.397+0000 7f82ff7fe700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82ec06ec70 0x7f82ec071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:27.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.397+0000 7f82ff7fe700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82ec06ec70 0x7f82ec071130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009550 tx=0x7f82f80091b0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:27.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.401+0000 7f82fcff9700 1 -- 172.21.15.96:0/2390152360 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f82e8058dd0 con 0x7f8300105aa0 2024-04-22T15:36:27.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.689+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f830004ebb0 con 0x7f8300105aa0 2024-04-22T15:36:27.693 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:27 smithi096 ceph-mon[112443]: pgmap v437: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:27.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.690+0000 7f82fcff9700 1 -- 172.21.15.96:0/2390152360 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f82e805c490 con 0x7f8300105aa0 2024-04-22T15:36:27.693 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:36:27.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:36:27.695 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:36:27.695 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:36:27.695 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:36:27.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:36:27.695 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:36:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82ec06ec70 msgr2=0x7f82ec071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82ec06ec70 0x7f82ec071130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009550 tx=0x7f82f80091b0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8300105aa0 msgr2=0x7f83000fc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8300105aa0 0x7f83000fc160 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f82e8000c00 tx=0x7f82e800bd70 comp rx=0 tx=0).stop 2024-04-22T15:36:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 shutdown_connections 2024-04-22T15:36:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82ec06ec70 0x7f82ec071130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83000ffaa0 0x7f83000fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 --2- 172.21.15.96:0/2390152360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8300105aa0 0x7f83000fc160 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 >> 172.21.15.96:0/2390152360 conn(0x7f8300076740 msgr2=0x7f83000687c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:27.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 shutdown_connections 2024-04-22T15:36:27.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.692+0000 7f8306223700 1 -- 172.21.15.96:0/2390152360 wait complete. 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.796+0000 7f0e09693700 1 -- 172.21.15.96:0/111420598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e041026f0 msgr2=0x7f0e0410abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.796+0000 7f0e09693700 1 --2- 172.21.15.96:0/111420598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e041026f0 0x7f0e0410abe0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009940 tx=0x7f0df8009c50 comp rx=0 tx=0).stop 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.796+0000 7f0e09693700 1 -- 172.21.15.96:0/111420598 shutdown_connections 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.796+0000 7f0e09693700 1 --2- 172.21.15.96:0/111420598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e041026f0 0x7f0e0410abe0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.796+0000 7f0e09693700 1 --2- 172.21.15.96:0/111420598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e04101dd0 0x7f0e041021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.796+0000 7f0e09693700 1 -- 172.21.15.96:0/111420598 >> 172.21.15.96:0/111420598 conn(0x7f0e040fb5e0 msgr2=0x7f0e040fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.796+0000 7f0e09693700 1 -- 172.21.15.96:0/111420598 shutdown_connections 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.796+0000 7f0e09693700 1 -- 172.21.15.96:0/111420598 wait complete. 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e09693700 1 Processor -- start 2024-04-22T15:36:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e09693700 1 -- start start 2024-04-22T15:36:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e09693700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e04101dd0 0x7f0e0419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e09693700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e041026f0 0x7f0e0419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e09693700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0419d340 con 0x7f0e04101dd0 2024-04-22T15:36:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e09693700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e041969b0 con 0x7f0e041026f0 2024-04-22T15:36:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e02ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e04101dd0 0x7f0e0419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e02ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e04101dd0 0x7f0e0419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52900/0 (socket says 172.21.15.96:52900) 2024-04-22T15:36:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e02ffd700 1 -- 172.21.15.96:0/681442401 learned_addr learned my addr 172.21.15.96:0/681442401 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e027fc700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e041026f0 0x7f0e0419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e02ffd700 1 -- 172.21.15.96:0/681442401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e041026f0 msgr2=0x7f0e0419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:27.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e02ffd700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e041026f0 0x7f0e0419cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:27.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.797+0000 7f0e02ffd700 1 -- 172.21.15.96:0/681442401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df80095f0 con 0x7f0e04101dd0 2024-04-22T15:36:27.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.798+0000 7f0e02ffd700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e04101dd0 0x7f0e0419c870 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0df400c930 tx=0x7f0df400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:27.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.798+0000 7f0debfff700 1 -- 172.21.15.96:0/681442401 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0df4007ad0 con 0x7f0e04101dd0 2024-04-22T15:36:27.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.798+0000 7f0debfff700 1 -- 172.21.15.96:0/681442401 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0df400ce80 con 0x7f0e04101dd0 2024-04-22T15:36:27.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.798+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e04196c90 con 0x7f0e04101dd0 2024-04-22T15:36:27.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.798+0000 7f0debfff700 1 -- 172.21.15.96:0/681442401 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0df4018630 con 0x7f0e04101dd0 2024-04-22T15:36:27.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.798+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e041971b0 con 0x7f0e04101dd0 2024-04-22T15:36:27.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.799+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e041054f0 con 0x7f0e04101dd0 2024-04-22T15:36:27.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.799+0000 7f0debfff700 1 -- 172.21.15.96:0/681442401 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0df4007c30 con 0x7f0e04101dd0 2024-04-22T15:36:27.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.800+0000 7f0debfff700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dec06ecc0 0x7f0dec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:27.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.800+0000 7f0e027fc700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dec06ecc0 0x7f0dec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:27.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.800+0000 7f0debfff700 1 -- 172.21.15.96:0/681442401 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0df408eef0 con 0x7f0e04101dd0 2024-04-22T15:36:27.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.800+0000 7f0e027fc700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dec06ecc0 0x7f0dec071180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0df800b580 tx=0x7f0df800b8d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:27.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:27.805+0000 7f0debfff700 1 -- 172.21.15.96:0/681442401 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0df405b780 con 0x7f0e04101dd0 2024-04-22T15:36:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:27 smithi196 ceph-mon[81815]: pgmap v437: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:28.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.052+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0e04197e70 con 0x7f0e04101dd0 2024-04-22T15:36:28.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.052+0000 7f0debfff700 1 -- 172.21.15.96:0/681442401 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f0df405b310 con 0x7f0e04101dd0 2024-04-22T15:36:28.056 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:36:28.056 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:36:28.056 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:36:28.056 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:36:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:36:28.056 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:36:28.056 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:36:28.056 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:36:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:36:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:36:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:36:28.059 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:36:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:36:28.059 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:36:28.059 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:36:28.059 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:36:28.059 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:36:28.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dec06ecc0 msgr2=0x7f0dec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dec06ecc0 0x7f0dec071180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0df800b580 tx=0x7f0df800b8d0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e04101dd0 msgr2=0x7f0e0419c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e04101dd0 0x7f0e0419c870 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0df400c930 tx=0x7f0df400cc40 comp rx=0 tx=0).stop 2024-04-22T15:36:28.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 shutdown_connections 2024-04-22T15:36:28.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dec06ecc0 0x7f0dec071180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e04101dd0 0x7f0e0419c870 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 --2- 172.21.15.96:0/681442401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e041026f0 0x7f0e0419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 >> 172.21.15.96:0/681442401 conn(0x7f0e040fb5e0 msgr2=0x7f0e041030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:28.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 shutdown_connections 2024-04-22T15:36:28.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.055+0000 7f0e09693700 1 -- 172.21.15.96:0/681442401 wait complete. 2024-04-22T15:36:28.061 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:36:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.159+0000 7fe319233700 1 -- 172.21.15.96:0/431596346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 msgr2=0x7fe314105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.159+0000 7fe319233700 1 --2- 172.21.15.96:0/431596346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 0x7fe314105f40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe30400b0a0 tx=0x7fe30400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.159+0000 7fe319233700 1 -- 172.21.15.96:0/431596346 shutdown_connections 2024-04-22T15:36:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.159+0000 7fe319233700 1 --2- 172.21.15.96:0/431596346 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3140ffad0 0x7fe3140fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.159+0000 7fe319233700 1 --2- 172.21.15.96:0/431596346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 0x7fe314105f40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.159+0000 7fe319233700 1 -- 172.21.15.96:0/431596346 >> 172.21.15.96:0/431596346 conn(0x7fe3140fb600 msgr2=0x7fe3140fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.159+0000 7fe319233700 1 -- 172.21.15.96:0/431596346 shutdown_connections 2024-04-22T15:36:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.160+0000 7fe319233700 1 -- 172.21.15.96:0/431596346 wait complete. 2024-04-22T15:36:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.160+0000 7fe319233700 1 Processor -- start 2024-04-22T15:36:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.160+0000 7fe319233700 1 -- start start 2024-04-22T15:36:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe319233700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3140ffad0 0x7fe3140fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe319233700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 0x7fe3140fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe319233700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140fc810 con 0x7fe314105b60 2024-04-22T15:36:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe319233700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140fc950 con 0x7fe3140ffad0 2024-04-22T15:36:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe31259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 0x7fe3140fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe31259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 0x7fe3140fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52918/0 (socket says 172.21.15.96:52918) 2024-04-22T15:36:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe31259c700 1 -- 172.21.15.96:0/1119860799 learned_addr learned my addr 172.21.15.96:0/1119860799 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe312d9d700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3140ffad0 0x7fe3140fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe31259c700 1 -- 172.21.15.96:0/1119860799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3140ffad0 msgr2=0x7fe3140fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe31259c700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3140ffad0 0x7fe3140fbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe31259c700 1 -- 172.21.15.96:0/1119860799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304009d00 con 0x7fe314105b60 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe31259c700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 0x7fe3140fc2d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe30800d8d0 tx=0x7fe30800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe2fbfff700 1 -- 172.21.15.96:0/1119860799 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3080098a0 con 0x7fe314105b60 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3141a2c90 con 0x7fe314105b60 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe2fbfff700 1 -- 172.21.15.96:0/1119860799 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe308010460 con 0x7fe314105b60 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe2fbfff700 1 -- 172.21.15.96:0/1119860799 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe30800f5f0 con 0x7fe314105b60 2024-04-22T15:36:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.161+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3141a2ff0 con 0x7fe314105b60 2024-04-22T15:36:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.162+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3141090e0 con 0x7fe314105b60 2024-04-22T15:36:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.163+0000 7fe2fbfff700 1 -- 172.21.15.96:0/1119860799 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe308009a00 con 0x7fe314105b60 2024-04-22T15:36:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.163+0000 7fe2fbfff700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe2fc06ecc0 0x7fe2fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.163+0000 7fe2fbfff700 1 -- 172.21.15.96:0/1119860799 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe30808edc0 con 0x7fe314105b60 2024-04-22T15:36:28.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.163+0000 7fe312d9d700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe2fc06ecc0 0x7fe2fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:28.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.164+0000 7fe312d9d700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe2fc06ecc0 0x7fe2fc071180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe304000f80 tx=0x7fe3040060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:28.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.168+0000 7fe2fbfff700 1 -- 172.21.15.96:0/1119860799 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe308057b20 con 0x7fe314105b60 2024-04-22T15:36:28.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.384+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3140fd270 con 0x7fe2fc06ecc0 2024-04-22T15:36:28.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.386+0000 7fe2fbfff700 1 -- 172.21.15.96:0/1119860799 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe3140fd270 con 0x7fe2fc06ecc0 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:36:28.389 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:36:28.390 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:36:28.390 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:36:28.390 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:36:28.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe2fc06ecc0 msgr2=0x7fe2fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe2fc06ecc0 0x7fe2fc071180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe304000f80 tx=0x7fe3040060c0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 msgr2=0x7fe3140fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 0x7fe3140fc2d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe30800d8d0 tx=0x7fe30800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 shutdown_connections 2024-04-22T15:36:28.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe2fc06ecc0 0x7fe2fc071180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3140ffad0 0x7fe3140fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 --2- 172.21.15.96:0/1119860799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe314105b60 0x7fe3140fc2d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 >> 172.21.15.96:0/1119860799 conn(0x7fe3140fb600 msgr2=0x7fe314069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:28.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 shutdown_connections 2024-04-22T15:36:28.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.388+0000 7fe319233700 1 -- 172.21.15.96:0/1119860799 wait complete. 2024-04-22T15:36:28.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.492+0000 7f6ea4763700 1 -- 172.21.15.96:0/2769169246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c1017d0 msgr2=0x7f6e9c105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.492+0000 7f6ea4763700 1 --2- 172.21.15.96:0/2769169246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c1017d0 0x7f6e9c105820 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c009910 tx=0x7f6e8c009c20 comp rx=0 tx=0).stop 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 -- 172.21.15.96:0/2769169246 shutdown_connections 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 --2- 172.21.15.96:0/2769169246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c1017d0 0x7f6e9c105820 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 --2- 172.21.15.96:0/2769169246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e9c100e20 0x7f6e9c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 -- 172.21.15.96:0/2769169246 >> 172.21.15.96:0/2769169246 conn(0x7f6e9c0fc6d0 msgr2=0x7f6e9c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 -- 172.21.15.96:0/2769169246 shutdown_connections 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 -- 172.21.15.96:0/2769169246 wait complete. 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 Processor -- start 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 -- start start 2024-04-22T15:36:28.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c100e20 0x7f6e9c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e9c1017d0 0x7f6e9c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c073c60 con 0x7f6e9c100e20 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea4763700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c073da0 con 0x7f6e9c1017d0 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.493+0000 7f6ea24ff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c100e20 0x7f6e9c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6ea24ff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c100e20 0x7f6e9c0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52938/0 (socket says 172.21.15.96:52938) 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6ea24ff700 1 -- 172.21.15.96:0/2589879230 learned_addr learned my addr 172.21.15.96:0/2589879230 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6ea1cfe700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e9c1017d0 0x7f6e9c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6ea24ff700 1 -- 172.21.15.96:0/2589879230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e9c1017d0 msgr2=0x7f6e9c073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6ea24ff700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e9c1017d0 0x7f6e9c073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6ea24ff700 1 -- 172.21.15.96:0/2589879230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c0095f0 con 0x7f6e9c100e20 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6ea24ff700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c100e20 0x7f6e9c0731e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800d900 tx=0x7f6e9800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6e937fe700 1 -- 172.21.15.96:0/2589879230 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e980041f0 con 0x7f6e9c100e20 2024-04-22T15:36:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c1a2ce0 con 0x7f6e9c100e20 2024-04-22T15:36:28.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6e937fe700 1 -- 172.21.15.96:0/2589879230 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e98004cf0 con 0x7f6e9c100e20 2024-04-22T15:36:28.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6e937fe700 1 -- 172.21.15.96:0/2589879230 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9800b750 con 0x7f6e9c100e20 2024-04-22T15:36:28.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.494+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c1a3040 con 0x7f6e9c100e20 2024-04-22T15:36:28.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.495+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c109220 con 0x7f6e9c100e20 2024-04-22T15:36:28.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.496+0000 7f6e937fe700 1 -- 172.21.15.96:0/2589879230 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6e98004750 con 0x7f6e9c100e20 2024-04-22T15:36:28.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.496+0000 7f6e937fe700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e8806ecc0 0x7f6e88071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:28.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.496+0000 7f6e937fe700 1 -- 172.21.15.96:0/2589879230 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6e9808ed70 con 0x7f6e9c100e20 2024-04-22T15:36:28.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.496+0000 7f6ea1cfe700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e8806ecc0 0x7f6e88071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:28.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.496+0000 7f6ea1cfe700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e8806ecc0 0x7f6e88071180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c074910 tx=0x7f6e8c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:28.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.501+0000 7f6e937fe700 1 -- 172.21.15.96:0/2589879230 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6e98057ad0 con 0x7f6e9c100e20 2024-04-22T15:36:28.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:28 smithi096 ceph-mon[112443]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:36:28.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:28 smithi096 ceph-mon[112443]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:36:28.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:28 smithi096 ceph-mon[112443]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:36:28.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:28 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2390152360' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:36:28.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:28 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/681442401' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:36:28.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.784+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6e9c073f60 con 0x7f6e9c100e20 2024-04-22T15:36:28.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.784+0000 7f6e937fe700 1 -- 172.21.15.96:0/2589879230 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f6e980161c0 con 0x7f6e9c100e20 2024-04-22T15:36:28.788 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:36:28.788 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:36:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:36:28.788 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:36:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:36:28.788 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:36:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e8806ecc0 msgr2=0x7f6e88071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e8806ecc0 0x7f6e88071180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c074910 tx=0x7f6e8c01a040 comp rx=0 tx=0).stop 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c100e20 msgr2=0x7f6e9c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c100e20 0x7f6e9c0731e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800d900 tx=0x7f6e9800dc10 comp rx=0 tx=0).stop 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 shutdown_connections 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e8806ecc0 0x7f6e88071180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9c100e20 0x7f6e9c0731e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 --2- 172.21.15.96:0/2589879230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e9c1017d0 0x7f6e9c073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 >> 172.21.15.96:0/2589879230 conn(0x7f6e9c0fc6d0 msgr2=0x7f6e9c0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:28.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 shutdown_connections 2024-04-22T15:36:28.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:28.786+0000 7f6ea4763700 1 -- 172.21.15.96:0/2589879230 wait complete. 2024-04-22T15:36:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:28 smithi196 ceph-mon[81815]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:36:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:28 smithi196 ceph-mon[81815]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:36:29.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:28 smithi196 ceph-mon[81815]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:36:29.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:28 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2390152360' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:36:29.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:28 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/681442401' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:36:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:29 smithi096 ceph-mon[112443]: from='client.34864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:36:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:29 smithi096 ceph-mon[112443]: pgmap v438: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:29 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2589879230' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:36:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:29 smithi196 ceph-mon[81815]: from='client.34864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:36:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:29 smithi196 ceph-mon[81815]: pgmap v438: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:30.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:29 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2589879230' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:36:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:31 smithi196 ceph-mon[81815]: pgmap v439: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:31.788 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:31 smithi096 ceph-mon[112443]: pgmap v439: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:33.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:33 smithi096 ceph-mon[112443]: pgmap v440: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:33 smithi196 ceph-mon[81815]: pgmap v440: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:35.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:35 smithi096 ceph-mon[112443]: pgmap v441: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:35 smithi196 ceph-mon[81815]: pgmap v441: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:37 smithi096 ceph-mon[112443]: pgmap v442: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:37 smithi196 ceph-mon[81815]: pgmap v442: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:38.911 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:36:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:36:39.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:39 smithi096 ceph-mon[112443]: pgmap v443: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:39 smithi196 ceph-mon[81815]: pgmap v443: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:40 smithi096 ceph-mon[112443]: pgmap v444: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:40 smithi196 ceph-mon[81815]: pgmap v444: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:43.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:43 smithi096 ceph-mon[112443]: pgmap v445: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:43 smithi196 ceph-mon[81815]: pgmap v445: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:45.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:45 smithi096 ceph-mon[112443]: pgmap v446: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:45.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:45 smithi196 ceph-mon[81815]: pgmap v446: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:47.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:47 smithi096 ceph-mon[112443]: pgmap v447: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:47.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:47 smithi196 ceph-mon[81815]: pgmap v447: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:49 smithi096 ceph-mon[112443]: pgmap v448: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:49.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:49 smithi196 ceph-mon[81815]: pgmap v448: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:51 smithi096 ceph-mon[112443]: pgmap v449: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:51.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:51 smithi196 ceph-mon[81815]: pgmap v449: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:53 smithi096 ceph-mon[112443]: pgmap v450: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:36:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:53 smithi196 ceph-mon[81815]: pgmap v450: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:36:55.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:55 smithi096 ceph-mon[112443]: pgmap v451: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:55 smithi196 ceph-mon[81815]: pgmap v451: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:57.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:57 smithi096 ceph-mon[112443]: pgmap v452: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:57.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:57 smithi196 ceph-mon[81815]: pgmap v452: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:58.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.903+0000 7ff3180cf700 1 -- 172.21.15.96:0/2184511050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 msgr2=0x7ff310102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:58.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.903+0000 7ff3180cf700 1 --2- 172.21.15.96:0/2184511050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 0x7ff310102140 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009910 tx=0x7ff30c009c20 comp rx=0 tx=0).stop 2024-04-22T15:36:58.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.903+0000 7ff3180cf700 1 -- 172.21.15.96:0/2184511050 shutdown_connections 2024-04-22T15:36:58.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.903+0000 7ff3180cf700 1 --2- 172.21.15.96:0/2184511050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff310102680 0x7ff31010ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:58.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.903+0000 7ff3180cf700 1 --2- 172.21.15.96:0/2184511050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 0x7ff310102140 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:58.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.903+0000 7ff3180cf700 1 -- 172.21.15.96:0/2184511050 >> 172.21.15.96:0/2184511050 conn(0x7ff310076740 msgr2=0x7ff3100fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:58.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.903+0000 7ff3180cf700 1 -- 172.21.15.96:0/2184511050 shutdown_connections 2024-04-22T15:36:58.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.903+0000 7ff3180cf700 1 -- 172.21.15.96:0/2184511050 wait complete. 2024-04-22T15:36:58.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.904+0000 7ff3180cf700 1 Processor -- start 2024-04-22T15:36:58.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.904+0000 7ff3180cf700 1 -- start start 2024-04-22T15:36:58.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.904+0000 7ff3180cf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 0x7ff31019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:58.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.904+0000 7ff3180cf700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff310102680 0x7ff31019ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:58.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.904+0000 7ff3180cf700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31019d440 con 0x7ff310101d60 2024-04-22T15:36:58.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.904+0000 7ff3180cf700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff310196a20 con 0x7ff310102680 2024-04-22T15:36:58.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.904+0000 7ff315e6b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 0x7ff31019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.904+0000 7ff315e6b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 0x7ff31019c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56402/0 (socket says 172.21.15.96:56402) 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.904+0000 7ff315e6b700 1 -- 172.21.15.96:0/3091905101 learned_addr learned my addr 172.21.15.96:0/3091905101 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff31566a700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff310102680 0x7ff31019ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff315e6b700 1 -- 172.21.15.96:0/3091905101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff310102680 msgr2=0x7ff31019ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff315e6b700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff310102680 0x7ff31019ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff315e6b700 1 -- 172.21.15.96:0/3091905101 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c0095f0 con 0x7ff310101d60 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff315e6b700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 0x7ff31019c8e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0092e0 tx=0x7ff30c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff306ffd700 1 -- 172.21.15.96:0/3091905101 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff30c01d070 con 0x7ff310101d60 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff310196ca0 con 0x7ff310101d60 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff306ffd700 1 -- 172.21.15.96:0/3091905101 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff30c00f460 con 0x7ff310101d60 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff306ffd700 1 -- 172.21.15.96:0/3091905101 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff30c017630 con 0x7ff310101d60 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.905+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff310197190 con 0x7ff310101d60 2024-04-22T15:36:58.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.906+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3101054f0 con 0x7ff310101d60 2024-04-22T15:36:58.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.907+0000 7ff306ffd700 1 -- 172.21.15.96:0/3091905101 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff30c00f5d0 con 0x7ff310101d60 2024-04-22T15:36:58.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.907+0000 7ff306ffd700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2fc06ecd0 0x7ff2fc071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:58.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.907+0000 7ff306ffd700 1 -- 172.21.15.96:0/3091905101 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff30c0903d0 con 0x7ff310101d60 2024-04-22T15:36:58.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.907+0000 7ff31566a700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2fc06ecd0 0x7ff2fc071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:58.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.907+0000 7ff31566a700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2fc06ecd0 0x7ff2fc071190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff310197be0 tx=0x7ff300008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:58.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:58.912+0000 7ff306ffd700 1 -- 172.21.15.96:0/3091905101 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff30c05cc60 con 0x7ff310101d60 2024-04-22T15:36:59.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.130+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff310197ee0 con 0x7ff2fc06ecd0 2024-04-22T15:36:59.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.131+0000 7ff306ffd700 1 -- 172.21.15.96:0/3091905101 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff310197ee0 con 0x7ff2fc06ecd0 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.133+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2fc06ecd0 msgr2=0x7ff2fc071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.133+0000 7ff3180cf700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2fc06ecd0 0x7ff2fc071190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff310197be0 tx=0x7ff300008040 comp rx=0 tx=0).stop 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.133+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 msgr2=0x7ff31019c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.133+0000 7ff3180cf700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 0x7ff31019c8e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0092e0 tx=0x7ff30c00be20 comp rx=0 tx=0).stop 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.134+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 shutdown_connections 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.134+0000 7ff3180cf700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2fc06ecd0 0x7ff2fc071190 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.134+0000 7ff3180cf700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff310101d60 0x7ff31019c8e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.134+0000 7ff3180cf700 1 --2- 172.21.15.96:0/3091905101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff310102680 0x7ff31019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.134+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 >> 172.21.15.96:0/3091905101 conn(0x7ff310076740 msgr2=0x7ff310102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.134+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 shutdown_connections 2024-04-22T15:36:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.134+0000 7ff3180cf700 1 -- 172.21.15.96:0/3091905101 wait complete. 2024-04-22T15:36:59.147 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:36:59.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.237+0000 7fd6f2251700 1 -- 172.21.15.96:0/3086788613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec105ab0 msgr2=0x7fd6ec105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.237+0000 7fd6f2251700 1 --2- 172.21.15.96:0/3086788613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec105ab0 0x7fd6ec105e90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009910 tx=0x7fd6dc009c20 comp rx=0 tx=0).stop 2024-04-22T15:36:59.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.237+0000 7fd6f2251700 1 -- 172.21.15.96:0/3086788613 shutdown_connections 2024-04-22T15:36:59.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.237+0000 7fd6f2251700 1 --2- 172.21.15.96:0/3086788613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6ec0ffab0 0x7fd6ec0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.237+0000 7fd6f2251700 1 --2- 172.21.15.96:0/3086788613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec105ab0 0x7fd6ec105e90 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.237+0000 7fd6f2251700 1 -- 172.21.15.96:0/3086788613 >> 172.21.15.96:0/3086788613 conn(0x7fd6ec0fb5e0 msgr2=0x7fd6ec0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:59.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.237+0000 7fd6f2251700 1 -- 172.21.15.96:0/3086788613 shutdown_connections 2024-04-22T15:36:59.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.237+0000 7fd6f2251700 1 -- 172.21.15.96:0/3086788613 wait complete. 2024-04-22T15:36:59.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6f2251700 1 Processor -- start 2024-04-22T15:36:59.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6f2251700 1 -- start start 2024-04-22T15:36:59.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6f2251700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec0ffab0 0x7fd6ec19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:59.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6f2251700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6ec105ab0 0x7fd6ec19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:59.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6f2251700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec19d410 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6f2251700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec1969f0 con 0x7fd6ec105ab0 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6eb7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec0ffab0 0x7fd6ec19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6eb7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec0ffab0 0x7fd6ec19c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56422/0 (socket says 172.21.15.96:56422) 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6eb7fe700 1 -- 172.21.15.96:0/2397322254 learned_addr learned my addr 172.21.15.96:0/2397322254 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.238+0000 7fd6eaffd700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6ec105ab0 0x7fd6ec19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.239+0000 7fd6eb7fe700 1 -- 172.21.15.96:0/2397322254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6ec105ab0 msgr2=0x7fd6ec19cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.239+0000 7fd6eb7fe700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6ec105ab0 0x7fd6ec19cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.239+0000 7fd6eb7fe700 1 -- 172.21.15.96:0/2397322254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc0095f0 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.239+0000 7fd6eb7fe700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec0ffab0 0x7fd6ec19c8b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc000c00 tx=0x7fd6dc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.239+0000 7fd6e8ff9700 1 -- 172.21.15.96:0/2397322254 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6dc01d070 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.239+0000 7fd6e8ff9700 1 -- 172.21.15.96:0/2397322254 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6dc0056e0 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.239+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ec196c70 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.239+0000 7fd6e8ff9700 1 -- 172.21.15.96:0/2397322254 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6dc00fe40 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.239+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ec197160 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.240+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ec1080b0 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.240+0000 7fd6e8ff9700 1 -- 172.21.15.96:0/2397322254 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd6dc00f680 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.241+0000 7fd6e8ff9700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6d406ecd0 0x7fd6d4071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:59.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.241+0000 7fd6eaffd700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6d406ecd0 0x7fd6d4071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:59.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.241+0000 7fd6e8ff9700 1 -- 172.21.15.96:0/2397322254 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd6dc090160 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.241+0000 7fd6eaffd700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6d406ecd0 0x7fd6d4071190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec197bb0 tx=0x7fd6e000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:59.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.246+0000 7fd6e8ff9700 1 -- 172.21.15.96:0/2397322254 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd6dc05ca70 con 0x7fd6ec0ffab0 2024-04-22T15:36:59.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.468+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6ec197eb0 con 0x7fd6d406ecd0 2024-04-22T15:36:59.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.469+0000 7fd6e8ff9700 1 -- 172.21.15.96:0/2397322254 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd6ec197eb0 con 0x7fd6d406ecd0 2024-04-22T15:36:59.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6d406ecd0 msgr2=0x7fd6d4071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6d406ecd0 0x7fd6d4071190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec197bb0 tx=0x7fd6e000b410 comp rx=0 tx=0).stop 2024-04-22T15:36:59.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec0ffab0 msgr2=0x7fd6ec19c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec0ffab0 0x7fd6ec19c8b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc000c00 tx=0x7fd6dc00bd70 comp rx=0 tx=0).stop 2024-04-22T15:36:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 shutdown_connections 2024-04-22T15:36:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6d406ecd0 0x7fd6d4071190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6ec0ffab0 0x7fd6ec19c8b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 --2- 172.21.15.96:0/2397322254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6ec105ab0 0x7fd6ec19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 >> 172.21.15.96:0/2397322254 conn(0x7fd6ec0fb5e0 msgr2=0x7fd6ec109ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 shutdown_connections 2024-04-22T15:36:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.471+0000 7fd6f2251700 1 -- 172.21.15.96:0/2397322254 wait complete. 2024-04-22T15:36:59.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.575+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/3050423918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a8105ab0 msgr2=0x7fd1a8105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.575+0000 7fd1ac8a1700 1 --2- 172.21.15.96:0/3050423918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a8105ab0 0x7fd1a8105e90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd190009910 tx=0x7fd190009c20 comp rx=0 tx=0).stop 2024-04-22T15:36:59.579 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:36:59 smithi096 ceph-mon[112443]: pgmap v453: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:59.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.575+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/3050423918 shutdown_connections 2024-04-22T15:36:59.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.575+0000 7fd1ac8a1700 1 --2- 172.21.15.96:0/3050423918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1a80ffab0 0x7fd1a80fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.575+0000 7fd1ac8a1700 1 --2- 172.21.15.96:0/3050423918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a8105ab0 0x7fd1a8105e90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.575+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/3050423918 >> 172.21.15.96:0/3050423918 conn(0x7fd1a80fb5e0 msgr2=0x7fd1a80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:59.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.575+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/3050423918 shutdown_connections 2024-04-22T15:36:59.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.575+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/3050423918 wait complete. 2024-04-22T15:36:59.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.576+0000 7fd1ac8a1700 1 Processor -- start 2024-04-22T15:36:59.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.576+0000 7fd1ac8a1700 1 -- start start 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.576+0000 7fd1ac8a1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a80ffab0 0x7fd1a8198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.576+0000 7fd1ac8a1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1a8105ab0 0x7fd1a8199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.576+0000 7fd1ac8a1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a8199b00 con 0x7fd1a80ffab0 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.576+0000 7fd1ac8a1700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a819d6e0 con 0x7fd1a8105ab0 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1a659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a80ffab0 0x7fd1a8198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1a659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a80ffab0 0x7fd1a8198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56444/0 (socket says 172.21.15.96:56444) 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1a659c700 1 -- 172.21.15.96:0/793289756 learned_addr learned my addr 172.21.15.96:0/793289756 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1a5d9b700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1a8105ab0 0x7fd1a8199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1a659c700 1 -- 172.21.15.96:0/793289756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1a8105ab0 msgr2=0x7fd1a8199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1a659c700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1a8105ab0 0x7fd1a8199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1a659c700 1 -- 172.21.15.96:0/793289756 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1900095f0 con 0x7fd1a80ffab0 2024-04-22T15:36:59.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1a659c700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a80ffab0 0x7fd1a8198ee0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd19000bd70 tx=0x7fd19000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:59.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd19f7fe700 1 -- 172.21.15.96:0/793289756 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd19001d070 con 0x7fd1a80ffab0 2024-04-22T15:36:59.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd19f7fe700 1 -- 172.21.15.96:0/793289756 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd19000f460 con 0x7fd1a80ffab0 2024-04-22T15:36:59.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a819d960 con 0x7fd1a80ffab0 2024-04-22T15:36:59.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd19f7fe700 1 -- 172.21.15.96:0/793289756 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd190017630 con 0x7fd1a80ffab0 2024-04-22T15:36:59.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.577+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a819de50 con 0x7fd1a80ffab0 2024-04-22T15:36:59.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.578+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a81011f0 con 0x7fd1a80ffab0 2024-04-22T15:36:59.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.579+0000 7fd19f7fe700 1 -- 172.21.15.96:0/793289756 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd19000f5d0 con 0x7fd1a80ffab0 2024-04-22T15:36:59.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.579+0000 7fd19f7fe700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd19406ecc0 0x7fd194071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:59.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.579+0000 7fd19f7fe700 1 -- 172.21.15.96:0/793289756 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd1900903d0 con 0x7fd1a80ffab0 2024-04-22T15:36:59.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.579+0000 7fd1a5d9b700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd19406ecc0 0x7fd194071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:59.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.580+0000 7fd1a5d9b700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd19406ecc0 0x7fd194071180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd198006fd0 tx=0x7fd198008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:59.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.585+0000 7fd19f7fe700 1 -- 172.21.15.96:0/793289756 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd190059130 con 0x7fd1a80ffab0 2024-04-22T15:36:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:36:59 smithi196 ceph-mon[81815]: pgmap v453: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:36:59.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.795+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1a810cd30 con 0x7fd19406ecc0 2024-04-22T15:36:59.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.806+0000 7fd19f7fe700 1 -- 172.21.15.96:0/793289756 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd1a810cd30 con 0x7fd19406ecc0 2024-04-22T15:36:59.809 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:36:59.809 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (15m) 8m ago 26m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:36:59.809 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (27m) 8m ago 27m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:36:59.809 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (26m) 8m ago 26m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:36:59.809 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (12m) 8m ago 27m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:36:59.809 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (12m) 8m ago 26m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (15m) 8m ago 26m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (8m) 8m ago 21m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (9m) 8m ago 21m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (8m) 8m ago 21m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (21m) 8m ago 21m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (18m) 8m ago 28m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (16m) 8m ago 25m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (14m) 8m ago 28m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (13m) 8m ago 25m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (16m) 8m ago 27m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (16m) 8m ago 25m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (12m) 8m ago 25m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:36:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (11m) 8m ago 24m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:36:59.811 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (11m) 8m ago 24m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:36:59.811 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (10m) 8m ago 23m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:36:59.811 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (10m) 8m ago 23m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:36:59.811 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (9m) 8m ago 22m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:36:59.811 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (16m) 8m ago 26m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:36:59.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd19406ecc0 msgr2=0x7fd194071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd19406ecc0 0x7fd194071180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd198006fd0 tx=0x7fd198008040 comp rx=0 tx=0).stop 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a80ffab0 msgr2=0x7fd1a8198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a80ffab0 0x7fd1a8198ee0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd19000bd70 tx=0x7fd19000be50 comp rx=0 tx=0).stop 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 shutdown_connections 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd19406ecc0 0x7fd194071180 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a80ffab0 0x7fd1a8198ee0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 --2- 172.21.15.96:0/793289756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1a8105ab0 0x7fd1a8199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 >> 172.21.15.96:0/793289756 conn(0x7fd1a80fb5e0 msgr2=0x7fd1a8069230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 shutdown_connections 2024-04-22T15:36:59.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.808+0000 7fd1ac8a1700 1 -- 172.21.15.96:0/793289756 wait complete. 2024-04-22T15:36:59.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.910+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2191017838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 msgr2=0x7fb5b00fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.910+0000 7fb5b77aa700 1 --2- 172.21.15.96:0/2191017838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 0x7fb5b00fffd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009910 tx=0x7fb5a0009c20 comp rx=0 tx=0).stop 2024-04-22T15:36:59.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.910+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2191017838 shutdown_connections 2024-04-22T15:36:59.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.910+0000 7fb5b77aa700 1 --2- 172.21.15.96:0/2191017838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 0x7fb5b00fffd0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.910+0000 7fb5b77aa700 1 --2- 172.21.15.96:0/2191017838 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5b0105c00 0x7fb5b0105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.910+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2191017838 >> 172.21.15.96:0/2191017838 conn(0x7fb5b00fb680 msgr2=0x7fb5b00fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:36:59.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.910+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2191017838 shutdown_connections 2024-04-22T15:36:59.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.910+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2191017838 wait complete. 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b77aa700 1 Processor -- start 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b77aa700 1 -- start start 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b77aa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 0x7fb5b019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b77aa700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5b0105c00 0x7fb5b019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b77aa700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b0196950 con 0x7fb5b00ffb50 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b77aa700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b0196ac0 con 0x7fb5b0105c00 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b5546700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 0x7fb5b019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b5546700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 0x7fb5b019c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56462/0 (socket says 172.21.15.96:56462) 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b5546700 1 -- 172.21.15.96:0/2605547071 learned_addr learned my addr 172.21.15.96:0/2605547071 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b4d45700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5b0105c00 0x7fb5b019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b5546700 1 -- 172.21.15.96:0/2605547071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5b0105c00 msgr2=0x7fb5b019cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:36:59.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b5546700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5b0105c00 0x7fb5b019cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:36:59.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b5546700 1 -- 172.21.15.96:0/2605547071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a00095f0 con 0x7fb5b00ffb50 2024-04-22T15:36:59.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.911+0000 7fb5b4d45700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5b0105c00 0x7fb5b019cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:36:59.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.912+0000 7fb5b5546700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 0x7fb5b019c8a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00d8d0 tx=0x7fb5ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:59.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.912+0000 7fb5a67fc700 1 -- 172.21.15.96:0/2605547071 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5ac009940 con 0x7fb5b00ffb50 2024-04-22T15:36:59.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.912+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b0196da0 con 0x7fb5b00ffb50 2024-04-22T15:36:59.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.912+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b01972f0 con 0x7fb5b00ffb50 2024-04-22T15:36:59.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.912+0000 7fb5a67fc700 1 -- 172.21.15.96:0/2605547071 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5ac010460 con 0x7fb5b00ffb50 2024-04-22T15:36:59.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.912+0000 7fb5a67fc700 1 -- 172.21.15.96:0/2605547071 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5ac00f5f0 con 0x7fb5b00ffb50 2024-04-22T15:36:59.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.913+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5940052f0 con 0x7fb5b00ffb50 2024-04-22T15:36:59.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.913+0000 7fb5a67fc700 1 -- 172.21.15.96:0/2605547071 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb5ac010ad0 con 0x7fb5b00ffb50 2024-04-22T15:36:59.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.914+0000 7fb5a67fc700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb59c06ecc0 0x7fb59c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:36:59.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.914+0000 7fb5a67fc700 1 -- 172.21.15.96:0/2605547071 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb5ac08ed00 con 0x7fb5b00ffb50 2024-04-22T15:36:59.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.914+0000 7fb5b4d45700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb59c06ecc0 0x7fb59c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:36:59.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.914+0000 7fb5b4d45700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb59c06ecc0 0x7fb59c071180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0006010 tx=0x7fb5a000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:36:59.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:36:59.919+0000 7fb5a67fc700 1 -- 172.21.15.96:0/2605547071 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb5ac092030 con 0x7fb5b00ffb50 2024-04-22T15:37:00.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.204+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb594005db0 con 0x7fb5b00ffb50 2024-04-22T15:37:00.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.204+0000 7fb5a67fc700 1 -- 172.21.15.96:0/2605547071 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7fb5ac057ae0 con 0x7fb5b00ffb50 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:37:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:37:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:37:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:37:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:37:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:37:00.209 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:37:00.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb59c06ecc0 msgr2=0x7fb59c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb59c06ecc0 0x7fb59c071180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0006010 tx=0x7fb5a000b540 comp rx=0 tx=0).stop 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 msgr2=0x7fb5b019c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 0x7fb5b019c8a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00d8d0 tx=0x7fb5ac00dc90 comp rx=0 tx=0).stop 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 shutdown_connections 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb59c06ecc0 0x7fb59c071180 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5b00ffb50 0x7fb5b019c8a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 --2- 172.21.15.96:0/2605547071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5b0105c00 0x7fb5b019cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 >> 172.21.15.96:0/2605547071 conn(0x7fb5b00fb680 msgr2=0x7fb5b0069430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 shutdown_connections 2024-04-22T15:37:00.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.206+0000 7fb5b77aa700 1 -- 172.21.15.96:0/2605547071 wait complete. 2024-04-22T15:37:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.310+0000 7f6f471c7700 1 -- 172.21.15.96:0/3926698915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 msgr2=0x7f6f40105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.310+0000 7f6f471c7700 1 --2- 172.21.15.96:0/3926698915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 0x7f6f40105e80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6f34009910 tx=0x7f6f34009c20 comp rx=0 tx=0).stop 2024-04-22T15:37:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.310+0000 7f6f471c7700 1 -- 172.21.15.96:0/3926698915 shutdown_connections 2024-04-22T15:37:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.310+0000 7f6f471c7700 1 --2- 172.21.15.96:0/3926698915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f400ffaa0 0x7f6f400fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.310+0000 7f6f471c7700 1 --2- 172.21.15.96:0/3926698915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 0x7f6f40105e80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.310+0000 7f6f471c7700 1 -- 172.21.15.96:0/3926698915 >> 172.21.15.96:0/3926698915 conn(0x7f6f40076740 msgr2=0x7f6f400fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.310+0000 7f6f471c7700 1 -- 172.21.15.96:0/3926698915 shutdown_connections 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.310+0000 7f6f471c7700 1 -- 172.21.15.96:0/3926698915 wait complete. 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f471c7700 1 Processor -- start 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f471c7700 1 -- start start 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f471c7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f400ffaa0 0x7f6f4019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f471c7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 0x7f6f4019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f471c7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4019f4f0 con 0x7f6f40105aa0 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f471c7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f40198ad0 con 0x7f6f400ffaa0 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f3ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 0x7f6f4019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f3ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 0x7f6f4019eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56478/0 (socket says 172.21.15.96:56478) 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f3ffff700 1 -- 172.21.15.96:0/3836736846 learned_addr learned my addr 172.21.15.96:0/3836736846 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f44f63700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f400ffaa0 0x7f6f4019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f3ffff700 1 -- 172.21.15.96:0/3836736846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f400ffaa0 msgr2=0x7f6f4019e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f3ffff700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f400ffaa0 0x7f6f4019e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f3ffff700 1 -- 172.21.15.96:0/3836736846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f340095f0 con 0x7f6f40105aa0 2024-04-22T15:37:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f44f63700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f400ffaa0 0x7f6f4019e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:37:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.311+0000 7f6f3ffff700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 0x7f6f4019eed0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00d8d0 tx=0x7f6f2c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.312+0000 7f6f3dffb700 1 -- 172.21.15.96:0/3836736846 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f2c009940 con 0x7f6f40105aa0 2024-04-22T15:37:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.312+0000 7f6f3dffb700 1 -- 172.21.15.96:0/3836736846 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f2c010460 con 0x7f6f40105aa0 2024-04-22T15:37:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.312+0000 7f6f3dffb700 1 -- 172.21.15.96:0/3836736846 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f2c00f5f0 con 0x7f6f40105aa0 2024-04-22T15:37:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.312+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f40198db0 con 0x7f6f40105aa0 2024-04-22T15:37:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.312+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f40199300 con 0x7f6f40105aa0 2024-04-22T15:37:00.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.313+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f400fe300 con 0x7f6f40105aa0 2024-04-22T15:37:00.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.313+0000 7f6f3dffb700 1 -- 172.21.15.96:0/3836736846 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6f2c0105d0 con 0x7f6f40105aa0 2024-04-22T15:37:00.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.314+0000 7f6f3dffb700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6f3006ecc0 0x7f6f30071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:00.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.314+0000 7f6f3dffb700 1 -- 172.21.15.96:0/3836736846 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6f2c08eed0 con 0x7f6f40105aa0 2024-04-22T15:37:00.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.314+0000 7f6f44f63700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6f3006ecc0 0x7f6f30071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:00.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.314+0000 7f6f44f63700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6f3006ecc0 0x7f6f30071180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400b5c0 tx=0x7f6f3401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:00.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.319+0000 7f6f3dffb700 1 -- 172.21.15.96:0/3836736846 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6f2c05b760 con 0x7f6f40105aa0 2024-04-22T15:37:00.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.566+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6f40199f90 con 0x7f6f40105aa0 2024-04-22T15:37:00.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.566+0000 7f6f3dffb700 1 -- 172.21.15.96:0/3836736846 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f6f2c05b2f0 con 0x7f6f40105aa0 2024-04-22T15:37:00.569 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:37:00.569 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:37:00.569 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:37:00.570 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:37:00.571 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:37:00.572 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:37:00.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6f3006ecc0 msgr2=0x7f6f30071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6f3006ecc0 0x7f6f30071180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400b5c0 tx=0x7f6f3401a040 comp rx=0 tx=0).stop 2024-04-22T15:37:00.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 msgr2=0x7f6f4019eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 0x7f6f4019eed0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00d8d0 tx=0x7f6f2c00dc90 comp rx=0 tx=0).stop 2024-04-22T15:37:00.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 shutdown_connections 2024-04-22T15:37:00.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6f3006ecc0 0x7f6f30071180 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f400ffaa0 0x7f6f4019e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 --2- 172.21.15.96:0/3836736846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f40105aa0 0x7f6f4019eed0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 >> 172.21.15.96:0/3836736846 conn(0x7f6f40076740 msgr2=0x7f6f40104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:00.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 shutdown_connections 2024-04-22T15:37:00.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.568+0000 7f6f471c7700 1 -- 172.21.15.96:0/3836736846 wait complete. 2024-04-22T15:37:00.574 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:37:00.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.672+0000 7f9adac59700 1 -- 172.21.15.96:0/2202319080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 msgr2=0x7f9ad4105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.672+0000 7f9adac59700 1 --2- 172.21.15.96:0/2202319080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 0x7f9ad4105e80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000b0a0 tx=0x7f9ad000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.672+0000 7f9adac59700 1 -- 172.21.15.96:0/2202319080 shutdown_connections 2024-04-22T15:37:00.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.672+0000 7f9adac59700 1 --2- 172.21.15.96:0/2202319080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ad40ffaa0 0x7f9ad40fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.672+0000 7f9adac59700 1 --2- 172.21.15.96:0/2202319080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 0x7f9ad4105e80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.672+0000 7f9adac59700 1 -- 172.21.15.96:0/2202319080 >> 172.21.15.96:0/2202319080 conn(0x7f9ad4076740 msgr2=0x7f9ad40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:00.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.672+0000 7f9adac59700 1 -- 172.21.15.96:0/2202319080 shutdown_connections 2024-04-22T15:37:00.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.672+0000 7f9adac59700 1 -- 172.21.15.96:0/2202319080 wait complete. 2024-04-22T15:37:00.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9adac59700 1 Processor -- start 2024-04-22T15:37:00.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9adac59700 1 -- start start 2024-04-22T15:37:00.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9adac59700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ad40ffaa0 0x7f9ad40fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9adac59700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 0x7f9ad40fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9adac59700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40fc6a0 con 0x7f9ad4105aa0 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9adac59700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40fc7e0 con 0x7f9ad40ffaa0 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9acbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 0x7f9ad40fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9acbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 0x7f9ad40fc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52770/0 (socket says 172.21.15.96:52770) 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9acbfff700 1 -- 172.21.15.96:0/4198559623 learned_addr learned my addr 172.21.15.96:0/4198559623 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9ad89f5700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ad40ffaa0 0x7f9ad40fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9acbfff700 1 -- 172.21.15.96:0/4198559623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ad40ffaa0 msgr2=0x7f9ad40fbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9acbfff700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ad40ffaa0 0x7f9ad40fbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9acbfff700 1 -- 172.21.15.96:0/4198559623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad0009d00 con 0x7f9ad4105aa0 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.673+0000 7f9acbfff700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 0x7f9ad40fc160 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400d8d0 tx=0x7f9ac400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.674+0000 7f9ac9ffb700 1 -- 172.21.15.96:0/4198559623 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ac4009940 con 0x7f9ad4105aa0 2024-04-22T15:37:00.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.674+0000 7f9ac9ffb700 1 -- 172.21.15.96:0/4198559623 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ac4010460 con 0x7f9ad4105aa0 2024-04-22T15:37:00.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.674+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad41a2c90 con 0x7f9ad4105aa0 2024-04-22T15:37:00.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.674+0000 7f9ac9ffb700 1 -- 172.21.15.96:0/4198559623 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ac400f5f0 con 0x7f9ad4105aa0 2024-04-22T15:37:00.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.674+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad41a2ff0 con 0x7f9ad4105aa0 2024-04-22T15:37:00.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.675+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad41011e0 con 0x7f9ad4105aa0 2024-04-22T15:37:00.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.675+0000 7f9ac9ffb700 1 -- 172.21.15.96:0/4198559623 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9ac4009aa0 con 0x7f9ad4105aa0 2024-04-22T15:37:00.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.675+0000 7f9ac9ffb700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9abc06ecc0 0x7f9abc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:00.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.675+0000 7f9ac9ffb700 1 -- 172.21.15.96:0/4198559623 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9ac408eea0 con 0x7f9ad4105aa0 2024-04-22T15:37:00.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.676+0000 7f9ad89f5700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9abc06ecc0 0x7f9abc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:00.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.676+0000 7f9ad89f5700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9abc06ecc0 0x7f9abc071180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00062a0 tx=0x7f9ad00061f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:00.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.680+0000 7f9ac9ffb700 1 -- 172.21.15.96:0/4198559623 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9ac4057c00 con 0x7f9ad4105aa0 2024-04-22T15:37:00.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:00 smithi196 ceph-mon[81815]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:00 smithi196 ceph-mon[81815]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:00.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:00 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2605547071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:37:00.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.899+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ad410cd10 con 0x7f9abc06ecc0 2024-04-22T15:37:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:00 smithi096 ceph-mon[112443]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:00 smithi096 ceph-mon[112443]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:00 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2605547071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.901+0000 7f9ac9ffb700 1 -- 172.21.15.96:0/4198559623 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9ad410cd10 con 0x7f9abc06ecc0 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:37:00.904 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:37:00.905 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:37:00.905 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:37:00.905 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:37:00.905 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:37:00.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9abc06ecc0 msgr2=0x7f9abc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9abc06ecc0 0x7f9abc071180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00062a0 tx=0x7f9ad00061f0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 msgr2=0x7f9ad40fc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:00.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 0x7f9ad40fc160 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400d8d0 tx=0x7f9ac400dc90 comp rx=0 tx=0).stop 2024-04-22T15:37:00.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 shutdown_connections 2024-04-22T15:37:00.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9abc06ecc0 0x7f9abc071180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ad40ffaa0 0x7f9ad40fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 --2- 172.21.15.96:0/4198559623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ad4105aa0 0x7f9ad40fc160 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:00.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 >> 172.21.15.96:0/4198559623 conn(0x7f9ad4076740 msgr2=0x7f9ad40fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:00.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 shutdown_connections 2024-04-22T15:37:00.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:00.903+0000 7f9adac59700 1 -- 172.21.15.96:0/4198559623 wait complete. 2024-04-22T15:37:01.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.006+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3287777465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c540ffab0 msgr2=0x7f7c540fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:01.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.006+0000 7f7c5c39c700 1 --2- 172.21.15.96:0/3287777465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c540ffab0 0x7f7c540fff30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009910 tx=0x7f7c44009c20 comp rx=0 tx=0).stop 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.006+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3287777465 shutdown_connections 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.006+0000 7f7c5c39c700 1 --2- 172.21.15.96:0/3287777465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c540ffab0 0x7f7c540fff30 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.006+0000 7f7c5c39c700 1 --2- 172.21.15.96:0/3287777465 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c54105ab0 0x7f7c54105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.006+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3287777465 >> 172.21.15.96:0/3287777465 conn(0x7f7c540fb5e0 msgr2=0x7f7c540fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.006+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3287777465 shutdown_connections 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.006+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3287777465 wait complete. 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.007+0000 7f7c5c39c700 1 Processor -- start 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.007+0000 7f7c5c39c700 1 -- start start 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.007+0000 7f7c5c39c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c540ffab0 0x7f7c54197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.007+0000 7f7c5c39c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c54105ab0 0x7f7c54197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.007+0000 7f7c5c39c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c54198420 con 0x7f7c54105ab0 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.007+0000 7f7c5c39c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c54198560 con 0x7f7c540ffab0 2024-04-22T15:37:01.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.007+0000 7f7c59937700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c54105ab0 0x7f7c54197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.007+0000 7f7c59937700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c54105ab0 0x7f7c54197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52782/0 (socket says 172.21.15.96:52782) 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.007+0000 7f7c59937700 1 -- 172.21.15.96:0/3033996580 learned_addr learned my addr 172.21.15.96:0/3033996580 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c5a138700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c540ffab0 0x7f7c54197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c59937700 1 -- 172.21.15.96:0/3033996580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c540ffab0 msgr2=0x7f7c54197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c59937700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c540ffab0 0x7f7c54197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c59937700 1 -- 172.21.15.96:0/3033996580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c440095f0 con 0x7f7c54105ab0 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c5a138700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c540ffab0 0x7f7c54197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c59937700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c54105ab0 0x7f7c54197e50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400bd70 tx=0x7f7c4400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c4b7fe700 1 -- 172.21.15.96:0/3033996580 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c4401d070 con 0x7f7c54105ab0 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c4b7fe700 1 -- 172.21.15.96:0/3033996580 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c4400f460 con 0x7f7c54105ab0 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c4b7fe700 1 -- 172.21.15.96:0/3033996580 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c44017630 con 0x7f7c54105ab0 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c54198780 con 0x7f7c54105ab0 2024-04-22T15:37:01.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.008+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c541a0ce0 con 0x7f7c54105ab0 2024-04-22T15:37:01.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.009+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c54108040 con 0x7f7c54105ab0 2024-04-22T15:37:01.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.010+0000 7f7c4b7fe700 1 -- 172.21.15.96:0/3033996580 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7c4400f5d0 con 0x7f7c54105ab0 2024-04-22T15:37:01.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.010+0000 7f7c4b7fe700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c4006ecc0 0x7f7c40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:01.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.010+0000 7f7c4b7fe700 1 -- 172.21.15.96:0/3033996580 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7c440901c0 con 0x7f7c54105ab0 2024-04-22T15:37:01.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.010+0000 7f7c5a138700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c4006ecc0 0x7f7c40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:01.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.010+0000 7f7c5a138700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c4006ecc0 0x7f7c40071180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7c50005fd0 tx=0x7f7c50005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:01.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.015+0000 7f7c4b7fe700 1 -- 172.21.15.96:0/3033996580 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7c44058f40 con 0x7f7c54105ab0 2024-04-22T15:37:01.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.297+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7c541011f0 con 0x7f7c54105ab0 2024-04-22T15:37:01.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.297+0000 7f7c4b7fe700 1 -- 172.21.15.96:0/3033996580 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f7c4405c600 con 0x7f7c54105ab0 2024-04-22T15:37:01.301 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:37:01.301 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:37:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:37:01.301 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:37:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:37:01.301 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:37:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c4006ecc0 msgr2=0x7f7c40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c4006ecc0 0x7f7c40071180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7c50005fd0 tx=0x7f7c50005dc0 comp rx=0 tx=0).stop 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c54105ab0 msgr2=0x7f7c54197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c54105ab0 0x7f7c54197e50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400bd70 tx=0x7f7c4400be50 comp rx=0 tx=0).stop 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 shutdown_connections 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c4006ecc0 0x7f7c40071180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c540ffab0 0x7f7c54197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 --2- 172.21.15.96:0/3033996580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c54105ab0 0x7f7c54197e50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 >> 172.21.15.96:0/3033996580 conn(0x7f7c540fb5e0 msgr2=0x7f7c540fc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:01.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 shutdown_connections 2024-04-22T15:37:01.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:01.300+0000 7f7c5c39c700 1 -- 172.21.15.96:0/3033996580 wait complete. 2024-04-22T15:37:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:01 smithi196 ceph-mon[81815]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:01.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:01 smithi196 ceph-mon[81815]: pgmap v454: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:01.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:01 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3836736846' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:37:01.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:01 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3033996580' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:37:01.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:01 smithi096 ceph-mon[112443]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:01.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:01 smithi096 ceph-mon[112443]: pgmap v454: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:01.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:01 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3836736846' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:37:01.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:01 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3033996580' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:37:02.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:02 smithi196 ceph-mon[81815]: from='client.34892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:02.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:02 smithi096 ceph-mon[112443]: from='client.34892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:03 smithi196 ceph-mon[81815]: pgmap v455: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:03.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:03 smithi096 ceph-mon[112443]: pgmap v455: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:05 smithi196 ceph-mon[81815]: pgmap v456: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:05 smithi096 ceph-mon[112443]: pgmap v456: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:07 smithi096 ceph-mon[112443]: pgmap v457: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:07 smithi196 ceph-mon[81815]: pgmap v457: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:08.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:37:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:37:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:09 smithi096 ceph-mon[112443]: pgmap v458: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:09 smithi196 ceph-mon[81815]: pgmap v458: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:11.778 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:11 smithi196 ceph-mon[81815]: pgmap v459: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:11.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:11 smithi096 ceph-mon[112443]: pgmap v459: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:13 smithi096 ceph-mon[112443]: pgmap v460: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:13 smithi196 ceph-mon[81815]: pgmap v460: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:15.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:15 smithi096 ceph-mon[112443]: pgmap v461: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:15 smithi196 ceph-mon[81815]: pgmap v461: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:17.787 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:17 smithi096 ceph-mon[112443]: pgmap v462: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:17 smithi196 ceph-mon[81815]: pgmap v462: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:19 smithi096 ceph-mon[112443]: pgmap v463: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:19 smithi196 ceph-mon[81815]: pgmap v463: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:21.884 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:21 smithi196 ceph-mon[81815]: pgmap v464: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:21.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:21 smithi096 ceph-mon[112443]: pgmap v464: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:23 smithi096 ceph-mon[112443]: pgmap v465: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:37:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:23 smithi196 ceph-mon[81815]: pgmap v465: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:37:25.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:25 smithi096 ceph-mon[112443]: pgmap v466: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:25 smithi196 ceph-mon[81815]: pgmap v466: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:27.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:27 smithi096 ceph-mon[112443]: pgmap v467: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:27 smithi196 ceph-mon[81815]: pgmap v467: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:28 smithi096 ceph-mon[112443]: pgmap v468: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:28 smithi196 ceph-mon[81815]: pgmap v468: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:31.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 -- 172.21.15.96:0/325819165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 msgr2=0x7fe8a4105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:31.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 --2- 172.21.15.96:0/325819165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 0x7fe8a4105820 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe894009910 tx=0x7fe894009c20 comp rx=0 tx=0).stop 2024-04-22T15:37:31.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 -- 172.21.15.96:0/325819165 shutdown_connections 2024-04-22T15:37:31.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 --2- 172.21.15.96:0/325819165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 0x7fe8a4105820 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 --2- 172.21.15.96:0/325819165 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8a4100e20 0x7fe8a4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 -- 172.21.15.96:0/325819165 >> 172.21.15.96:0/325819165 conn(0x7fe8a40fc6d0 msgr2=0x7fe8a40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 -- 172.21.15.96:0/325819165 shutdown_connections 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 -- 172.21.15.96:0/325819165 wait complete. 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 Processor -- start 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.417+0000 7fe8ac176700 1 -- start start 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8ac176700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8a4100e20 0x7fe8a419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8ac176700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 0x7fe8a419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8ac176700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a419d4c0 con 0x7fe8a41017d0 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8ac176700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a4196aa0 con 0x7fe8a4100e20 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8a9711700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 0x7fe8a419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8a9711700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 0x7fe8a419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54124/0 (socket says 172.21.15.96:54124) 2024-04-22T15:37:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8a9711700 1 -- 172.21.15.96:0/2988301616 learned_addr learned my addr 172.21.15.96:0/2988301616 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8a9f12700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8a4100e20 0x7fe8a419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8a9711700 1 -- 172.21.15.96:0/2988301616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8a4100e20 msgr2=0x7fe8a419c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8a9711700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8a4100e20 0x7fe8a419c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8a9711700 1 -- 172.21.15.96:0/2988301616 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8940095f0 con 0x7fe8a41017d0 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe8a9711700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 0x7fe8a419cea0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe8940092e0 tx=0x7fe89400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.418+0000 7fe89affd700 1 -- 172.21.15.96:0/2988301616 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89401d070 con 0x7fe8a41017d0 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.419+0000 7fe89affd700 1 -- 172.21.15.96:0/2988301616 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe89400f460 con 0x7fe8a41017d0 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.419+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a4196d20 con 0x7fe8a41017d0 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.419+0000 7fe89affd700 1 -- 172.21.15.96:0/2988301616 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe894017630 con 0x7fe8a41017d0 2024-04-22T15:37:31.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.419+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a4197210 con 0x7fe8a41017d0 2024-04-22T15:37:31.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.420+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a4109170 con 0x7fe8a41017d0 2024-04-22T15:37:31.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.420+0000 7fe89affd700 1 -- 172.21.15.96:0/2988301616 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe89400f5d0 con 0x7fe8a41017d0 2024-04-22T15:37:31.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.420+0000 7fe89affd700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe89006ecc0 0x7fe890071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:31.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.420+0000 7fe89affd700 1 -- 172.21.15.96:0/2988301616 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe894090330 con 0x7fe8a41017d0 2024-04-22T15:37:31.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.420+0000 7fe8a9f12700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe89006ecc0 0x7fe890071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:31.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.421+0000 7fe8a9f12700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe89006ecc0 0x7fe890071180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0006fd0 tx=0x7fe8a0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:31.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.425+0000 7fe89affd700 1 -- 172.21.15.96:0/2988301616 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe89405cbc0 con 0x7fe8a41017d0 2024-04-22T15:37:31.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.647+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8a4197f60 con 0x7fe89006ecc0 2024-04-22T15:37:31.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.648+0000 7fe89affd700 1 -- 172.21.15.96:0/2988301616 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe8a4197f60 con 0x7fe89006ecc0 2024-04-22T15:37:31.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe89006ecc0 msgr2=0x7fe890071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe89006ecc0 0x7fe890071180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0006fd0 tx=0x7fe8a0008040 comp rx=0 tx=0).stop 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 msgr2=0x7fe8a419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 0x7fe8a419cea0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe8940092e0 tx=0x7fe89400be20 comp rx=0 tx=0).stop 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 shutdown_connections 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe89006ecc0 0x7fe890071180 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8a4100e20 0x7fe8a419c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 --2- 172.21.15.96:0/2988301616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8a41017d0 0x7fe8a419cea0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 >> 172.21.15.96:0/2988301616 conn(0x7fe8a40fc6d0 msgr2=0x7fe8a40fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 shutdown_connections 2024-04-22T15:37:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.650+0000 7fe8ac176700 1 -- 172.21.15.96:0/2988301616 wait complete. 2024-04-22T15:37:31.664 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:37:31.689 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:31 smithi096 ceph-mon[112443]: pgmap v469: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:31.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:31 smithi196 ceph-mon[81815]: pgmap v469: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:31.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.757+0000 7fa690e80700 1 -- 172.21.15.96:0/3776830724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c100e10 msgr2=0x7fa68c1011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:31.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.757+0000 7fa690e80700 1 --2- 172.21.15.96:0/3776830724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c100e10 0x7fa68c1011f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009910 tx=0x7fa67c009c20 comp rx=0 tx=0).stop 2024-04-22T15:37:31.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.757+0000 7fa690e80700 1 -- 172.21.15.96:0/3776830724 shutdown_connections 2024-04-22T15:37:31.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.757+0000 7fa690e80700 1 --2- 172.21.15.96:0/3776830724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa68c1017c0 0x7fa68c105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.757+0000 7fa690e80700 1 --2- 172.21.15.96:0/3776830724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c100e10 0x7fa68c1011f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.757+0000 7fa690e80700 1 -- 172.21.15.96:0/3776830724 >> 172.21.15.96:0/3776830724 conn(0x7fa68c0fc6e0 msgr2=0x7fa68c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:31.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.757+0000 7fa690e80700 1 -- 172.21.15.96:0/3776830724 shutdown_connections 2024-04-22T15:37:31.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.757+0000 7fa690e80700 1 -- 172.21.15.96:0/3776830724 wait complete. 2024-04-22T15:37:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.757+0000 7fa690e80700 1 Processor -- start 2024-04-22T15:37:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa690e80700 1 -- start start 2024-04-22T15:37:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa690e80700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa68c100e10 0x7fa68c196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa690e80700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c1017c0 0x7fa68c1970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa690e80700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c197790 con 0x7fa68c1017c0 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa690e80700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c19b3a0 con 0x7fa68c100e10 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa681d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c1017c0 0x7fa68c1970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa681d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c1017c0 0x7fa68c1970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54140/0 (socket says 172.21.15.96:54140) 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa681d9b700 1 -- 172.21.15.96:0/3806333323 learned_addr learned my addr 172.21.15.96:0/3806333323 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa68a59c700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa68c100e10 0x7fa68c196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa681d9b700 1 -- 172.21.15.96:0/3806333323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa68c100e10 msgr2=0x7fa68c196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa681d9b700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa68c100e10 0x7fa68c196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.758+0000 7fa681d9b700 1 -- 172.21.15.96:0/3806333323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67c0095f0 con 0x7fa68c1017c0 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.759+0000 7fa681d9b700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c1017c0 0x7fa68c1970b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa67400d900 tx=0x7fa67400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.759+0000 7fa683fff700 1 -- 172.21.15.96:0/3806333323 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa674009900 con 0x7fa68c1017c0 2024-04-22T15:37:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.759+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa68c19b680 con 0x7fa68c1017c0 2024-04-22T15:37:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.759+0000 7fa683fff700 1 -- 172.21.15.96:0/3806333323 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa674010460 con 0x7fa68c1017c0 2024-04-22T15:37:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.759+0000 7fa683fff700 1 -- 172.21.15.96:0/3806333323 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa67400f5f0 con 0x7fa68c1017c0 2024-04-22T15:37:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.759+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68c19bbd0 con 0x7fa68c1017c0 2024-04-22T15:37:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.760+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68c1091b0 con 0x7fa68c1017c0 2024-04-22T15:37:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.760+0000 7fa683fff700 1 -- 172.21.15.96:0/3806333323 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa6740105d0 con 0x7fa68c1017c0 2024-04-22T15:37:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.761+0000 7fa683fff700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa67806ecc0 0x7fa678071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.761+0000 7fa683fff700 1 -- 172.21.15.96:0/3806333323 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa67408ea80 con 0x7fa68c1017c0 2024-04-22T15:37:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.761+0000 7fa68a59c700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa67806ecc0 0x7fa678071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.761+0000 7fa68a59c700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa67806ecc0 0x7fa678071180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa67c000c00 tx=0x7fa67c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.766+0000 7fa683fff700 1 -- 172.21.15.96:0/3806333323 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa6740577e0 con 0x7fa68c1017c0 2024-04-22T15:37:31.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.987+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa68c197dc0 con 0x7fa67806ecc0 2024-04-22T15:37:31.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.988+0000 7fa683fff700 1 -- 172.21.15.96:0/3806333323 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa68c197dc0 con 0x7fa67806ecc0 2024-04-22T15:37:31.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa67806ecc0 msgr2=0x7fa678071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:31.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa67806ecc0 0x7fa678071180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa67c000c00 tx=0x7fa67c01a040 comp rx=0 tx=0).stop 2024-04-22T15:37:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c1017c0 msgr2=0x7fa68c1970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c1017c0 0x7fa68c1970b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa67400d900 tx=0x7fa67400dc10 comp rx=0 tx=0).stop 2024-04-22T15:37:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 shutdown_connections 2024-04-22T15:37:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa67806ecc0 0x7fa678071180 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa68c100e10 0x7fa68c196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 --2- 172.21.15.96:0/3806333323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa68c1017c0 0x7fa68c1970b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 >> 172.21.15.96:0/3806333323 conn(0x7fa68c0fc6e0 msgr2=0x7fa68c0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 shutdown_connections 2024-04-22T15:37:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:31.990+0000 7fa690e80700 1 -- 172.21.15.96:0/3806333323 wait complete. 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 -- 172.21.15.96:0/540012715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44041017d0 msgr2=0x7f4404105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 --2- 172.21.15.96:0/540012715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44041017d0 0x7f4404105820 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f43f4009910 tx=0x7f43f4009c20 comp rx=0 tx=0).stop 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 -- 172.21.15.96:0/540012715 shutdown_connections 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 --2- 172.21.15.96:0/540012715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44041017d0 0x7f4404105820 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 --2- 172.21.15.96:0/540012715 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4404100e20 0x7f4404101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 -- 172.21.15.96:0/540012715 >> 172.21.15.96:0/540012715 conn(0x7f44040fc6d0 msgr2=0x7f44040feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 -- 172.21.15.96:0/540012715 shutdown_connections 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 -- 172.21.15.96:0/540012715 wait complete. 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 Processor -- start 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.093+0000 7f440b06d700 1 -- start start 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f440b06d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404100e20 0x7f440419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:32.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f440b06d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44041017d0 0x7f440419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f440b06d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440419d380 con 0x7f4404100e20 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f440b06d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4404196960 con 0x7f44041017d0 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f4408e09700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404100e20 0x7f440419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f4408e09700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404100e20 0x7f440419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54168/0 (socket says 172.21.15.96:54168) 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f4408e09700 1 -- 172.21.15.96:0/4237753443 learned_addr learned my addr 172.21.15.96:0/4237753443 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f4408e09700 1 -- 172.21.15.96:0/4237753443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44041017d0 msgr2=0x7f440419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f4408e09700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44041017d0 0x7f440419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f4408e09700 1 -- 172.21.15.96:0/4237753443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f40095f0 con 0x7f4404100e20 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f4408e09700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404100e20 0x7f440419c820 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f440000d8d0 tx=0x7f440000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f43f9ffb700 1 -- 172.21.15.96:0/4237753443 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4400009940 con 0x7f4404100e20 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4404196c40 con 0x7f4404100e20 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f43f9ffb700 1 -- 172.21.15.96:0/4237753443 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4400010460 con 0x7f4404100e20 2024-04-22T15:37:32.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.094+0000 7f43f9ffb700 1 -- 172.21.15.96:0/4237753443 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f440000f5f0 con 0x7f4404100e20 2024-04-22T15:37:32.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.095+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4404197190 con 0x7f4404100e20 2024-04-22T15:37:32.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.096+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4404109220 con 0x7f4404100e20 2024-04-22T15:37:32.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.096+0000 7f43f9ffb700 1 -- 172.21.15.96:0/4237753443 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f44000105d0 con 0x7f4404100e20 2024-04-22T15:37:32.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.097+0000 7f43f9ffb700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f43ec06ecc0 0x7f43ec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:32.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.097+0000 7f43fbfff700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f43ec06ecc0 0x7f43ec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:32.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.097+0000 7f43f9ffb700 1 -- 172.21.15.96:0/4237753443 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f440008eed0 con 0x7f4404100e20 2024-04-22T15:37:32.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.097+0000 7f43fbfff700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f43ec06ecc0 0x7f43ec071180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f43f400b5c0 tx=0x7f43f4005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:32.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.101+0000 7f43f9ffb700 1 -- 172.21.15.96:0/4237753443 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f440005b760 con 0x7f4404100e20 2024-04-22T15:37:32.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.308+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4404197ec0 con 0x7f43ec06ecc0 2024-04-22T15:37:32.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.319+0000 7f43f9ffb700 1 -- 172.21.15.96:0/4237753443 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4404197ec0 con 0x7f43ec06ecc0 2024-04-22T15:37:32.322 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (16m) 9m ago 27m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (27m) 9m ago 27m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (26m) 9m ago 26m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (13m) 9m ago 27m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (13m) 9m ago 26m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (15m) 9m ago 27m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (9m) 9m ago 21m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (9m) 9m ago 21m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (9m) 9m ago 21m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (21m) 9m ago 21m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (18m) 9m ago 29m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (17m) 9m ago 26m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (14m) 9m ago 29m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (14m) 9m ago 26m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:37:32.323 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (17m) 9m ago 27m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:37:32.324 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (17m) 9m ago 26m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:37:32.324 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (12m) 9m ago 25m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:37:32.324 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (12m) 9m ago 25m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:37:32.324 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (11m) 9m ago 24m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:37:32.324 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (11m) 9m ago 24m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:37:32.324 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (10m) 9m ago 23m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:37:32.324 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (10m) 9m ago 23m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:37:32.324 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (16m) 9m ago 27m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f43ec06ecc0 msgr2=0x7f43ec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f43ec06ecc0 0x7f43ec071180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f43f400b5c0 tx=0x7f43f4005e20 comp rx=0 tx=0).stop 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404100e20 msgr2=0x7f440419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404100e20 0x7f440419c820 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f440000d8d0 tx=0x7f440000dc90 comp rx=0 tx=0).stop 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 shutdown_connections 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f43ec06ecc0 0x7f43ec071180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404100e20 0x7f440419c820 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 --2- 172.21.15.96:0/4237753443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44041017d0 0x7f440419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 >> 172.21.15.96:0/4237753443 conn(0x7f44040fc6d0 msgr2=0x7f44040fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 shutdown_connections 2024-04-22T15:37:32.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.322+0000 7f440b06d700 1 -- 172.21.15.96:0/4237753443 wait complete. 2024-04-22T15:37:32.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.424+0000 7f9427a10700 1 -- 172.21.15.96:0/3645967259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 msgr2=0x7f9420105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.424+0000 7f9427a10700 1 --2- 172.21.15.96:0/3645967259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 0x7f9420105840 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9410009910 tx=0x7f9410009c20 comp rx=0 tx=0).stop 2024-04-22T15:37:32.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 -- 172.21.15.96:0/3645967259 shutdown_connections 2024-04-22T15:37:32.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 --2- 172.21.15.96:0/3645967259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 0x7f9420105840 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 --2- 172.21.15.96:0/3645967259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9420100e40 0x7f9420101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 -- 172.21.15.96:0/3645967259 >> 172.21.15.96:0/3645967259 conn(0x7f94200fc710 msgr2=0x7f94200feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:32.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 -- 172.21.15.96:0/3645967259 shutdown_connections 2024-04-22T15:37:32.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 -- 172.21.15.96:0/3645967259 wait complete. 2024-04-22T15:37:32.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 Processor -- start 2024-04-22T15:37:32.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 -- start start 2024-04-22T15:37:32.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9420100e40 0x7f942019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:32.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 0x7f942019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:32.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942019d380 con 0x7f94201017f0 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.425+0000 7f9427a10700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9420196960 con 0x7f9420100e40 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f9424fab700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 0x7f942019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f9424fab700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 0x7f942019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54192/0 (socket says 172.21.15.96:54192) 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f9424fab700 1 -- 172.21.15.96:0/2014396603 learned_addr learned my addr 172.21.15.96:0/2014396603 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f94257ac700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9420100e40 0x7f942019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f9424fab700 1 -- 172.21.15.96:0/2014396603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9420100e40 msgr2=0x7f942019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f9424fab700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9420100e40 0x7f942019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f9424fab700 1 -- 172.21.15.96:0/2014396603 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94100095f0 con 0x7f94201017f0 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f94257ac700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9420100e40 0x7f942019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:37:32.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f9424fab700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 0x7f942019cd60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f941000bd70 tx=0x7f941000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:32.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f94167fc700 1 -- 172.21.15.96:0/2014396603 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941001d070 con 0x7f94201017f0 2024-04-22T15:37:32.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9420196be0 con 0x7f94201017f0 2024-04-22T15:37:32.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f94167fc700 1 -- 172.21.15.96:0/2014396603 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94100056e0 con 0x7f94201017f0 2024-04-22T15:37:32.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f94167fc700 1 -- 172.21.15.96:0/2014396603 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9410017400 con 0x7f94201017f0 2024-04-22T15:37:32.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.426+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94201970d0 con 0x7f94201017f0 2024-04-22T15:37:32.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.427+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9420109240 con 0x7f94201017f0 2024-04-22T15:37:32.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.428+0000 7f94167fc700 1 -- 172.21.15.96:0/2014396603 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f941000f460 con 0x7f94201017f0 2024-04-22T15:37:32.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.428+0000 7f94167fc700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940c06ecc0 0x7f940c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:32.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.428+0000 7f94257ac700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940c06ecc0 0x7f940c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:32.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.428+0000 7f94167fc700 1 -- 172.21.15.96:0/2014396603 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9410090290 con 0x7f94201017f0 2024-04-22T15:37:32.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.429+0000 7f94257ac700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940c06ecc0 0x7f940c071180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f941c005950 tx=0x7f941c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:32.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.433+0000 7f94167fc700 1 -- 172.21.15.96:0/2014396603 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f941005cbe0 con 0x7f94201017f0 2024-04-22T15:37:32.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.720+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9420102320 con 0x7f94201017f0 2024-04-22T15:37:32.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.721+0000 7f94167fc700 1 -- 172.21.15.96:0/2014396603 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f941005c770 con 0x7f94201017f0 2024-04-22T15:37:32.724 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:37:32.724 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:37:32.724 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:37:32.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:37:32.724 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:37:32.724 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:37:32.725 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:37:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940c06ecc0 msgr2=0x7f940c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940c06ecc0 0x7f940c071180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f941c005950 tx=0x7f941c0058e0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 msgr2=0x7f942019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 0x7f942019cd60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f941000bd70 tx=0x7f941000be50 comp rx=0 tx=0).stop 2024-04-22T15:37:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 shutdown_connections 2024-04-22T15:37:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940c06ecc0 0x7f940c071180 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9420100e40 0x7f942019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 --2- 172.21.15.96:0/2014396603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94201017f0 0x7f942019cd60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 >> 172.21.15.96:0/2014396603 conn(0x7f94200fc710 msgr2=0x7f94200fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 shutdown_connections 2024-04-22T15:37:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.723+0000 7f9427a10700 1 -- 172.21.15.96:0/2014396603 wait complete. 2024-04-22T15:37:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.826+0000 7f59f9a45700 1 -- 172.21.15.96:0/3607924675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f40ffb50 msgr2=0x7f59f40fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.826+0000 7f59f9a45700 1 --2- 172.21.15.96:0/3607924675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f40ffb50 0x7f59f40fff30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009910 tx=0x7f59e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:37:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.826+0000 7f59f9a45700 1 -- 172.21.15.96:0/3607924675 shutdown_connections 2024-04-22T15:37:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.826+0000 7f59f9a45700 1 --2- 172.21.15.96:0/3607924675 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59f4100470 0x7f59f4104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.826+0000 7f59f9a45700 1 --2- 172.21.15.96:0/3607924675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f40ffb50 0x7f59f40fff30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.826+0000 7f59f9a45700 1 -- 172.21.15.96:0/3607924675 >> 172.21.15.96:0/3607924675 conn(0x7f59f4076740 msgr2=0x7f59f40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.826+0000 7f59f9a45700 1 -- 172.21.15.96:0/3607924675 shutdown_connections 2024-04-22T15:37:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.826+0000 7f59f9a45700 1 -- 172.21.15.96:0/3607924675 wait complete. 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59f9a45700 1 Processor -- start 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59f9a45700 1 -- start start 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59f9a45700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59f40ffb50 0x7f59f419d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59f9a45700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f4100470 0x7f59f419d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59f9a45700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f419de30 con 0x7f59f4100470 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59f9a45700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f41a1b00 con 0x7f59f40ffb50 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59ea7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f4100470 0x7f59f419d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59ea7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f4100470 0x7f59f419d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54210/0 (socket says 172.21.15.96:54210) 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59ea7fc700 1 -- 172.21.15.96:0/2775937794 learned_addr learned my addr 172.21.15.96:0/2775937794 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59f2ffd700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59f40ffb50 0x7f59f419d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59ea7fc700 1 -- 172.21.15.96:0/2775937794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59f40ffb50 msgr2=0x7f59f419d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59ea7fc700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59f40ffb50 0x7f59f419d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.827+0000 7f59ea7fc700 1 -- 172.21.15.96:0/2775937794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40095f0 con 0x7f59f4100470 2024-04-22T15:37:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.828+0000 7f59ea7fc700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f4100470 0x7f59f419d750 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00b3d0 tx=0x7f59dc00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:32.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.828+0000 7f59f8a43700 1 -- 172.21.15.96:0/2775937794 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59dc011800 con 0x7f59f4100470 2024-04-22T15:37:32.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.828+0000 7f59f8a43700 1 -- 172.21.15.96:0/2775937794 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59dc011e20 con 0x7f59f4100470 2024-04-22T15:37:32.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.828+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f41a1de0 con 0x7f59f4100470 2024-04-22T15:37:32.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.828+0000 7f59f8a43700 1 -- 172.21.15.96:0/2775937794 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59dc010ea0 con 0x7f59f4100470 2024-04-22T15:37:32.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.828+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f41a2330 con 0x7f59f4100470 2024-04-22T15:37:32.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.829+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f4068ea0 con 0x7f59f4100470 2024-04-22T15:37:32.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.829+0000 7f59f8a43700 1 -- 172.21.15.96:0/2775937794 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f59dc010380 con 0x7f59f4100470 2024-04-22T15:37:32.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.829+0000 7f59f8a43700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59e006ecc0 0x7f59e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:32.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.830+0000 7f59f8a43700 1 -- 172.21.15.96:0/2775937794 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f59dc08e980 con 0x7f59f4100470 2024-04-22T15:37:32.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.830+0000 7f59f2ffd700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59e006ecc0 0x7f59e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:32.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.830+0000 7f59f2ffd700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59e006ecc0 0x7f59e0071180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59e400b5c0 tx=0x7f59e401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:32.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:32.834+0000 7f59f8a43700 1 -- 172.21.15.96:0/2775937794 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f59dc0576e0 con 0x7f59f4100470 2024-04-22T15:37:33.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.089+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f59f404ebb0 con 0x7f59f4100470 2024-04-22T15:37:33.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.090+0000 7f59f8a43700 1 -- 172.21.15.96:0/2775937794 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f59dc05ada0 con 0x7f59f4100470 2024-04-22T15:37:33.093 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:37:33.093 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:37:33.093 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:37:33.093 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:37:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:37:33.093 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:37:33.093 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:37:33.094 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:37:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:37:33.096 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:37:33.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59e006ecc0 msgr2=0x7f59e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59e006ecc0 0x7f59e0071180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59e400b5c0 tx=0x7f59e401a040 comp rx=0 tx=0).stop 2024-04-22T15:37:33.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f4100470 msgr2=0x7f59f419d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f4100470 0x7f59f419d750 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00b3d0 tx=0x7f59dc00b790 comp rx=0 tx=0).stop 2024-04-22T15:37:33.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 shutdown_connections 2024-04-22T15:37:33.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f59e006ecc0 0x7f59e0071180 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f59f40ffb50 0x7f59f419d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 --2- 172.21.15.96:0/2775937794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59f4100470 0x7f59f419d750 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 >> 172.21.15.96:0/2775937794 conn(0x7f59f4076740 msgr2=0x7f59f410e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:33.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 shutdown_connections 2024-04-22T15:37:33.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.092+0000 7f59f9a45700 1 -- 172.21.15.96:0/2775937794 wait complete. 2024-04-22T15:37:33.098 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:37:33.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.197+0000 7f8747d6a700 1 -- 172.21.15.96:0/1612425856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 msgr2=0x7f874010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.197+0000 7f8747d6a700 1 --2- 172.21.15.96:0/1612425856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 0x7f874010ab70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8734009910 tx=0x7f8734009c20 comp rx=0 tx=0).stop 2024-04-22T15:37:33.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.197+0000 7f8747d6a700 1 -- 172.21.15.96:0/1612425856 shutdown_connections 2024-04-22T15:37:33.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.197+0000 7f8747d6a700 1 --2- 172.21.15.96:0/1612425856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 0x7f874010ab70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.197+0000 7f8747d6a700 1 --2- 172.21.15.96:0/1612425856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8740101d60 0x7f8740102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.197+0000 7f8747d6a700 1 -- 172.21.15.96:0/1612425856 >> 172.21.15.96:0/1612425856 conn(0x7f8740076740 msgr2=0x7f87400fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.197+0000 7f8747d6a700 1 -- 172.21.15.96:0/1612425856 shutdown_connections 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.197+0000 7f8747d6a700 1 -- 172.21.15.96:0/1612425856 wait complete. 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8747d6a700 1 Processor -- start 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8747d6a700 1 -- start start 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8747d6a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8740101d60 0x7f874019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8747d6a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 0x7f874019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8747d6a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874019d380 con 0x7f8740102680 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8747d6a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8740196960 con 0x7f8740101d60 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8745305700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 0x7f874019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8745305700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 0x7f874019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54234/0 (socket says 172.21.15.96:54234) 2024-04-22T15:37:33.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8745305700 1 -- 172.21.15.96:0/1943151265 learned_addr learned my addr 172.21.15.96:0/1943151265 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8745b06700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8740101d60 0x7f874019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8745305700 1 -- 172.21.15.96:0/1943151265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8740101d60 msgr2=0x7f874019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8745305700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8740101d60 0x7f874019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8745305700 1 -- 172.21.15.96:0/1943151265 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87340095f0 con 0x7f8740102680 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8745b06700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8740101d60 0x7f874019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f8745305700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 0x7f874019cd60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f873400bb40 tx=0x7f873400bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.198+0000 7f873effd700 1 -- 172.21.15.96:0/1943151265 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f873401d070 con 0x7f8740102680 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.199+0000 7f873effd700 1 -- 172.21.15.96:0/1943151265 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8734003d50 con 0x7f8740102680 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.199+0000 7f873effd700 1 -- 172.21.15.96:0/1943151265 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8734017400 con 0x7f8740102680 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.199+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8740196be0 con 0x7f8740102680 2024-04-22T15:37:33.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.199+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87401970d0 con 0x7f8740102680 2024-04-22T15:37:33.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.199+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8740105480 con 0x7f8740102680 2024-04-22T15:37:33.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.200+0000 7f873effd700 1 -- 172.21.15.96:0/1943151265 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8734003890 con 0x7f8740102680 2024-04-22T15:37:33.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.200+0000 7f873effd700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f872806ecc0 0x7f8728071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:33.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.201+0000 7f8745b06700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f872806ecc0 0x7f8728071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:33.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.201+0000 7f873effd700 1 -- 172.21.15.96:0/1943151265 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8734090160 con 0x7f8740102680 2024-04-22T15:37:33.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.201+0000 7f8745b06700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f872806ecc0 0x7f8728071180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f87300060b0 tx=0x7f87300076a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:33.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.205+0000 7f873effd700 1 -- 172.21.15.96:0/1943151265 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f873405ca70 con 0x7f8740102680 2024-04-22T15:37:33.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.421+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8740197e90 con 0x7f872806ecc0 2024-04-22T15:37:33.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.422+0000 7f873effd700 1 -- 172.21.15.96:0/1943151265 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8740197e90 con 0x7f872806ecc0 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:37:33.426 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:37:33.427 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:37:33.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.424+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f872806ecc0 msgr2=0x7f8728071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.424+0000 7f8747d6a700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f872806ecc0 0x7f8728071180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f87300060b0 tx=0x7f87300076a0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.424+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 msgr2=0x7f874019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.424+0000 7f8747d6a700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 0x7f874019cd60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f873400bb40 tx=0x7f873400bc20 comp rx=0 tx=0).stop 2024-04-22T15:37:33.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.424+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 shutdown_connections 2024-04-22T15:37:33.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.424+0000 7f8747d6a700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f872806ecc0 0x7f8728071180 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.424+0000 7f8747d6a700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8740101d60 0x7f874019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.424+0000 7f8747d6a700 1 --2- 172.21.15.96:0/1943151265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8740102680 0x7f874019cd60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.424+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 >> 172.21.15.96:0/1943151265 conn(0x7f8740076740 msgr2=0x7f8740103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:33.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.425+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 shutdown_connections 2024-04-22T15:37:33.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.425+0000 7f8747d6a700 1 -- 172.21.15.96:0/1943151265 wait complete. 2024-04-22T15:37:33.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.530+0000 7f2eb46f4700 1 -- 172.21.15.96:0/1370178439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ff4a0 msgr2=0x7f2eac0ff880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.530+0000 7f2eb46f4700 1 --2- 172.21.15.96:0/1370178439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ff4a0 0x7f2eac0ff880 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009910 tx=0x7f2ea8009c20 comp rx=0 tx=0).stop 2024-04-22T15:37:33.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.531+0000 7f2eb46f4700 1 -- 172.21.15.96:0/1370178439 shutdown_connections 2024-04-22T15:37:33.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.531+0000 7f2eb46f4700 1 --2- 172.21.15.96:0/1370178439 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2eac0ffe50 0x7f2eac103ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.531+0000 7f2eb46f4700 1 --2- 172.21.15.96:0/1370178439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ff4a0 0x7f2eac0ff880 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.531+0000 7f2eb46f4700 1 -- 172.21.15.96:0/1370178439 >> 172.21.15.96:0/1370178439 conn(0x7f2eac075f30 msgr2=0x7f2eac0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:33.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.531+0000 7f2eb46f4700 1 -- 172.21.15.96:0/1370178439 shutdown_connections 2024-04-22T15:37:33.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.531+0000 7f2eb46f4700 1 -- 172.21.15.96:0/1370178439 wait complete. 2024-04-22T15:37:33.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.531+0000 7f2eb46f4700 1 Processor -- start 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.531+0000 7f2eb46f4700 1 -- start start 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.531+0000 7f2eb46f4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2eac0ff4a0 0x7f2eac19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb46f4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ffe50 0x7f2eac19cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb46f4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac19d370 con 0x7f2eac0ffe50 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb46f4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac196950 con 0x7f2eac0ff4a0 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb1c8f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ffe50 0x7f2eac19cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb1c8f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ffe50 0x7f2eac19cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54260/0 (socket says 172.21.15.96:54260) 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb1c8f700 1 -- 172.21.15.96:0/2763399510 learned_addr learned my addr 172.21.15.96:0/2763399510 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb1c8f700 1 -- 172.21.15.96:0/2763399510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2eac0ff4a0 msgr2=0x7f2eac19c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb2490700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2eac0ff4a0 0x7f2eac19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb1c8f700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2eac0ff4a0 0x7f2eac19c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb1c8f700 1 -- 172.21.15.96:0/2763399510 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea80095f0 con 0x7f2eac0ffe50 2024-04-22T15:37:33.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb2490700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2eac0ff4a0 0x7f2eac19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:37:33.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb1c8f700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ffe50 0x7f2eac19cd50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00dd70 tx=0x7f2e9c00a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:33.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2ea37fe700 1 -- 172.21.15.96:0/2763399510 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e9c00ad50 con 0x7f2eac0ffe50 2024-04-22T15:37:33.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2ea37fe700 1 -- 172.21.15.96:0/2763399510 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e9c00e910 con 0x7f2eac0ffe50 2024-04-22T15:37:33.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2ea37fe700 1 -- 172.21.15.96:0/2763399510 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e9c0169b0 con 0x7f2eac0ffe50 2024-04-22T15:37:33.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eac196c30 con 0x7f2eac0ffe50 2024-04-22T15:37:33.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.532+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eac197180 con 0x7f2eac0ffe50 2024-04-22T15:37:33.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.533+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eac109850 con 0x7f2eac0ffe50 2024-04-22T15:37:33.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.534+0000 7f2ea37fe700 1 -- 172.21.15.96:0/2763399510 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2e9c00ea80 con 0x7f2eac0ffe50 2024-04-22T15:37:33.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.534+0000 7f2ea37fe700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e9806ecd0 0x7f2e98071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:37:33.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.534+0000 7f2ea37fe700 1 -- 172.21.15.96:0/2763399510 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2e9c08ee20 con 0x7f2eac0ffe50 2024-04-22T15:37:33.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.534+0000 7f2eb2490700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e9806ecd0 0x7f2e98071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:37:33.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.535+0000 7f2eb2490700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e9806ecd0 0x7f2e98071190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009fd0 tx=0x7f2ea800b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:37:33.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.539+0000 7f2ea37fe700 1 -- 172.21.15.96:0/2763399510 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2e9c05b7e0 con 0x7f2eac0ffe50 2024-04-22T15:37:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:33 smithi096 ceph-mon[112443]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:33 smithi096 ceph-mon[112443]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:33 smithi096 ceph-mon[112443]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:33 smithi096 ceph-mon[112443]: pgmap v470: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:33 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2014396603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:37:33.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:33 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2775937794' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:37:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:33 smithi196 ceph-mon[81815]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:33.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:33 smithi196 ceph-mon[81815]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:33.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:33 smithi196 ceph-mon[81815]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:33.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:33 smithi196 ceph-mon[81815]: pgmap v470: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:33.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:33 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2014396603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:37:33.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:33 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2775937794' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:37:33.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.828+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2eac100980 con 0x7f2eac0ffe50 2024-04-22T15:37:33.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.829+0000 7f2ea37fe700 1 -- 172.21.15.96:0/2763399510 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f2e9c05b370 con 0x7f2eac0ffe50 2024-04-22T15:37:33.832 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:37:33.832 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:37:33.832 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:37:33.832 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:37:33.833 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:37:33.833 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:37:33.833 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:37:33.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e9806ecd0 msgr2=0x7f2e98071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e9806ecd0 0x7f2e98071190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009fd0 tx=0x7f2ea800b560 comp rx=0 tx=0).stop 2024-04-22T15:37:33.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ffe50 msgr2=0x7f2eac19cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:37:33.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ffe50 0x7f2eac19cd50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00dd70 tx=0x7f2e9c00a530 comp rx=0 tx=0).stop 2024-04-22T15:37:33.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 shutdown_connections 2024-04-22T15:37:33.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e9806ecd0 0x7f2e98071190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2eac0ff4a0 0x7f2eac19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 --2- 172.21.15.96:0/2763399510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eac0ffe50 0x7f2eac19cd50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:37:33.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 >> 172.21.15.96:0/2763399510 conn(0x7f2eac075f30 msgr2=0x7f2eac101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:37:33.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 shutdown_connections 2024-04-22T15:37:33.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:37:33.831+0000 7f2eb46f4700 1 -- 172.21.15.96:0/2763399510 wait complete. 2024-04-22T15:37:34.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:34 smithi096 ceph-mon[112443]: from='client.34920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:34.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:34 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2763399510' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:37:34.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:34 smithi196 ceph-mon[81815]: from='client.34920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:37:34.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:34 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2763399510' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:37:35.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:35 smithi196 ceph-mon[81815]: pgmap v471: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:35.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:35 smithi096 ceph-mon[112443]: pgmap v471: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:37.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:37 smithi196 ceph-mon[81815]: pgmap v472: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:37 smithi096 ceph-mon[112443]: pgmap v472: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:37:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:37:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:39 smithi196 ceph-mon[81815]: pgmap v473: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:39.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:39 smithi096 ceph-mon[112443]: pgmap v473: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:41 smithi196 ceph-mon[81815]: pgmap v474: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:41 smithi096 ceph-mon[112443]: pgmap v474: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:43 smithi196 ceph-mon[81815]: pgmap v475: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:43 smithi096 ceph-mon[112443]: pgmap v475: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:45.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:45 smithi196 ceph-mon[81815]: pgmap v476: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:45.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:45 smithi096 ceph-mon[112443]: pgmap v476: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:47.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:47 smithi096 ceph-mon[112443]: pgmap v477: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:47 smithi196 ceph-mon[81815]: pgmap v477: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:49.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:49 smithi096 ceph-mon[112443]: pgmap v478: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:49 smithi196 ceph-mon[81815]: pgmap v478: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:51.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:51 smithi096 ceph-mon[112443]: pgmap v479: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:51 smithi196 ceph-mon[81815]: pgmap v479: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:53 smithi096 ceph-mon[112443]: pgmap v480: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:37:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:53 smithi196 ceph-mon[81815]: pgmap v480: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:37:55.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:55 smithi096 ceph-mon[112443]: pgmap v481: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:55 smithi196 ceph-mon[81815]: pgmap v481: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:57 smithi096 ceph-mon[112443]: pgmap v482: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:58.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:57 smithi196 ceph-mon[81815]: pgmap v482: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:37:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:37:59 smithi096 ceph-mon[112443]: pgmap v483: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:37:59 smithi196 ceph-mon[81815]: pgmap v483: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:01.788 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:01 smithi096 ceph-mon[112443]: pgmap v484: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:01 smithi196 ceph-mon[81815]: pgmap v484: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:03.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.948+0000 7ff2cec78700 1 -- 172.21.15.96:0/3096884272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c80ffad0 msgr2=0x7ff2c80fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:03.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.948+0000 7ff2cec78700 1 --2- 172.21.15.96:0/3096884272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c80ffad0 0x7ff2c80fff50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009910 tx=0x7ff2b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:03.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.948+0000 7ff2cec78700 1 -- 172.21.15.96:0/3096884272 shutdown_connections 2024-04-22T15:38:03.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.948+0000 7ff2cec78700 1 --2- 172.21.15.96:0/3096884272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c80ffad0 0x7ff2c80fff50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:03.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.948+0000 7ff2cec78700 1 --2- 172.21.15.96:0/3096884272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c8105b60 0x7ff2c8105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:03.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.948+0000 7ff2cec78700 1 -- 172.21.15.96:0/3096884272 >> 172.21.15.96:0/3096884272 conn(0x7ff2c80fb5e0 msgr2=0x7ff2c80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:03.952 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:03 smithi096 ceph-mon[112443]: pgmap v485: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.948+0000 7ff2cec78700 1 -- 172.21.15.96:0/3096884272 shutdown_connections 2024-04-22T15:38:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.948+0000 7ff2cec78700 1 -- 172.21.15.96:0/3096884272 wait complete. 2024-04-22T15:38:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2cec78700 1 Processor -- start 2024-04-22T15:38:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2cec78700 1 -- start start 2024-04-22T15:38:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2cec78700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c80ffad0 0x7ff2c8197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2cec78700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c8105b60 0x7ff2c8197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2cec78700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8198460 con 0x7ff2c8105b60 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2cec78700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81985a0 con 0x7ff2c80ffad0 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2c7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c8105b60 0x7ff2c8197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2c7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c8105b60 0x7ff2c8197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48956/0 (socket says 172.21.15.96:48956) 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2c7fff700 1 -- 172.21.15.96:0/1923682382 learned_addr learned my addr 172.21.15.96:0/1923682382 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.949+0000 7ff2cca14700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c80ffad0 0x7ff2c8197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.950+0000 7ff2c7fff700 1 -- 172.21.15.96:0/1923682382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c80ffad0 msgr2=0x7ff2c8197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.950+0000 7ff2c7fff700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c80ffad0 0x7ff2c8197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.950+0000 7ff2c7fff700 1 -- 172.21.15.96:0/1923682382 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b40095f0 con 0x7ff2c8105b60 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.950+0000 7ff2c7fff700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c8105b60 0x7ff2c8197e90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400bd70 tx=0x7ff2b400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.950+0000 7ff2c5ffb700 1 -- 172.21.15.96:0/1923682382 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2b401d070 con 0x7ff2c8105b60 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.950+0000 7ff2c5ffb700 1 -- 172.21.15.96:0/1923682382 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2b400f460 con 0x7ff2c8105b60 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.950+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c81a09c0 con 0x7ff2c8105b60 2024-04-22T15:38:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.950+0000 7ff2c5ffb700 1 -- 172.21.15.96:0/1923682382 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2b4017630 con 0x7ff2c8105b60 2024-04-22T15:38:03.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.950+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c81a0d20 con 0x7ff2c8105b60 2024-04-22T15:38:03.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.951+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c81080f0 con 0x7ff2c8105b60 2024-04-22T15:38:03.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.952+0000 7ff2c5ffb700 1 -- 172.21.15.96:0/1923682382 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff2b400f5d0 con 0x7ff2c8105b60 2024-04-22T15:38:03.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.952+0000 7ff2c5ffb700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2b806ecc0 0x7ff2b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:03.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.952+0000 7ff2c5ffb700 1 -- 172.21.15.96:0/1923682382 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff2b40903f0 con 0x7ff2c8105b60 2024-04-22T15:38:03.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.952+0000 7ff2cca14700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2b806ecc0 0x7ff2b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:03.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.952+0000 7ff2cca14700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2b806ecc0 0x7ff2b8071180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc005fd0 tx=0x7ff2bc00a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:03.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:03.957+0000 7ff2c5ffb700 1 -- 172.21.15.96:0/1923682382 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff2b4059150 con 0x7ff2c8105b60 2024-04-22T15:38:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:03 smithi196 ceph-mon[81815]: pgmap v485: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:04.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.176+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2c80696b0 con 0x7ff2b806ecc0 2024-04-22T15:38:04.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.178+0000 7ff2c5ffb700 1 -- 172.21.15.96:0/1923682382 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff2c80696b0 con 0x7ff2b806ecc0 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2b806ecc0 msgr2=0x7ff2b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2b806ecc0 0x7ff2b8071180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc005fd0 tx=0x7ff2bc00a560 comp rx=0 tx=0).stop 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c8105b60 msgr2=0x7ff2c8197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c8105b60 0x7ff2c8197e90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400bd70 tx=0x7ff2b400be50 comp rx=0 tx=0).stop 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 shutdown_connections 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff2b806ecc0 0x7ff2b8071180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c80ffad0 0x7ff2c8197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 --2- 172.21.15.96:0/1923682382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2c8105b60 0x7ff2c8197e90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 >> 172.21.15.96:0/1923682382 conn(0x7ff2c80fb5e0 msgr2=0x7ff2c80fc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 shutdown_connections 2024-04-22T15:38:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.180+0000 7ff2cec78700 1 -- 172.21.15.96:0/1923682382 wait complete. 2024-04-22T15:38:04.193 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.284+0000 7ff46843a700 1 -- 172.21.15.96:0/1355279543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 msgr2=0x7ff460105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.284+0000 7ff46843a700 1 --2- 172.21.15.96:0/1355279543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 0x7ff460105e90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff450009910 tx=0x7ff450009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.284+0000 7ff46843a700 1 -- 172.21.15.96:0/1355279543 shutdown_connections 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.284+0000 7ff46843a700 1 --2- 172.21.15.96:0/1355279543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4600ffab0 0x7ff4600fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.284+0000 7ff46843a700 1 --2- 172.21.15.96:0/1355279543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 0x7ff460105e90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.284+0000 7ff46843a700 1 -- 172.21.15.96:0/1355279543 >> 172.21.15.96:0/1355279543 conn(0x7ff4600fb5e0 msgr2=0x7ff4600fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.284+0000 7ff46843a700 1 -- 172.21.15.96:0/1355279543 shutdown_connections 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.284+0000 7ff46843a700 1 -- 172.21.15.96:0/1355279543 wait complete. 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff46843a700 1 Processor -- start 2024-04-22T15:38:04.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff46843a700 1 -- start start 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff46843a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4600ffab0 0x7ff46019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff46843a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 0x7ff46019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff46843a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46019d410 con 0x7ff460105ab0 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff46843a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4601969f0 con 0x7ff4600ffab0 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff4659d5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 0x7ff46019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff4659d5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 0x7ff46019cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48966/0 (socket says 172.21.15.96:48966) 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff4659d5700 1 -- 172.21.15.96:0/2093422431 learned_addr learned my addr 172.21.15.96:0/2093422431 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff4661d6700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4600ffab0 0x7ff46019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff4659d5700 1 -- 172.21.15.96:0/2093422431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4600ffab0 msgr2=0x7ff46019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff4659d5700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4600ffab0 0x7ff46019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff4659d5700 1 -- 172.21.15.96:0/2093422431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4500095f0 con 0x7ff460105ab0 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff4661d6700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4600ffab0 0x7ff46019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:38:04.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.285+0000 7ff4659d5700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 0x7ff46019cdf0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff45c00c8a0 tx=0x7ff45c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:04.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.286+0000 7ff4577fe700 1 -- 172.21.15.96:0/2093422431 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff45c00cea0 con 0x7ff460105ab0 2024-04-22T15:38:04.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.286+0000 7ff4577fe700 1 -- 172.21.15.96:0/2093422431 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff45c004830 con 0x7ff460105ab0 2024-04-22T15:38:04.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.286+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff460196cd0 con 0x7ff460105ab0 2024-04-22T15:38:04.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.286+0000 7ff4577fe700 1 -- 172.21.15.96:0/2093422431 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff45c0055b0 con 0x7ff460105ab0 2024-04-22T15:38:04.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.286+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff460197220 con 0x7ff460105ab0 2024-04-22T15:38:04.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.287+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4601080b0 con 0x7ff460105ab0 2024-04-22T15:38:04.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.287+0000 7ff4577fe700 1 -- 172.21.15.96:0/2093422431 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff45c0049a0 con 0x7ff460105ab0 2024-04-22T15:38:04.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.288+0000 7ff4577fe700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff44c06ecc0 0x7ff44c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:04.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.288+0000 7ff4661d6700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff44c06ecc0 0x7ff44c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:04.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.288+0000 7ff4577fe700 1 -- 172.21.15.96:0/2093422431 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff45c08eaa0 con 0x7ff460105ab0 2024-04-22T15:38:04.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.288+0000 7ff4661d6700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff44c06ecc0 0x7ff44c071180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff4500059d0 tx=0x7ff45001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:04.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.292+0000 7ff4577fe700 1 -- 172.21.15.96:0/2093422431 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff45c05b3b0 con 0x7ff460105ab0 2024-04-22T15:38:04.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.510+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff460197eb0 con 0x7ff44c06ecc0 2024-04-22T15:38:04.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.512+0000 7ff4577fe700 1 -- 172.21.15.96:0/2093422431 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff460197eb0 con 0x7ff44c06ecc0 2024-04-22T15:38:04.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff44c06ecc0 msgr2=0x7ff44c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff44c06ecc0 0x7ff44c071180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff4500059d0 tx=0x7ff45001a040 comp rx=0 tx=0).stop 2024-04-22T15:38:04.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 msgr2=0x7ff46019cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 0x7ff46019cdf0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff45c00c8a0 tx=0x7ff45c00cc60 comp rx=0 tx=0).stop 2024-04-22T15:38:04.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 shutdown_connections 2024-04-22T15:38:04.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff44c06ecc0 0x7ff44c071180 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4600ffab0 0x7ff46019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 --2- 172.21.15.96:0/2093422431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff460105ab0 0x7ff46019cdf0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 >> 172.21.15.96:0/2093422431 conn(0x7ff4600fb5e0 msgr2=0x7ff460109ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:04.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 shutdown_connections 2024-04-22T15:38:04.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.514+0000 7ff46843a700 1 -- 172.21.15.96:0/2093422431 wait complete. 2024-04-22T15:38:04.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.619+0000 7f3ded072700 1 -- 172.21.15.96:0/422431480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 msgr2=0x7f3de80fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.619+0000 7f3ded072700 1 --2- 172.21.15.96:0/422431480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 0x7f3de80fff20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009910 tx=0x7f3ddc009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:04.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.619+0000 7f3ded072700 1 -- 172.21.15.96:0/422431480 shutdown_connections 2024-04-22T15:38:04.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.619+0000 7f3ded072700 1 --2- 172.21.15.96:0/422431480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 0x7f3de80fff20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.619+0000 7f3ded072700 1 --2- 172.21.15.96:0/422431480 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3de8105aa0 0x7f3de8105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.619+0000 7f3ded072700 1 -- 172.21.15.96:0/422431480 >> 172.21.15.96:0/422431480 conn(0x7f3de8076740 msgr2=0x7f3de80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:04.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.619+0000 7f3ded072700 1 -- 172.21.15.96:0/422431480 shutdown_connections 2024-04-22T15:38:04.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.619+0000 7f3ded072700 1 -- 172.21.15.96:0/422431480 wait complete. 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.619+0000 7f3ded072700 1 Processor -- start 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3ded072700 1 -- start start 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3ded072700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 0x7f3de819b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3ded072700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3de8105aa0 0x7f3de819bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3ded072700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de819c290 con 0x7f3de80ffaa0 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3ded072700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de8195800 con 0x7f3de8105aa0 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3de6d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 0x7f3de819b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3de6d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 0x7f3de819b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48982/0 (socket says 172.21.15.96:48982) 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3de6d9d700 1 -- 172.21.15.96:0/4283524458 learned_addr learned my addr 172.21.15.96:0/4283524458 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3de659c700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3de8105aa0 0x7f3de819bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3de6d9d700 1 -- 172.21.15.96:0/4283524458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3de8105aa0 msgr2=0x7f3de819bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3de6d9d700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3de8105aa0 0x7f3de819bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3de6d9d700 1 -- 172.21.15.96:0/4283524458 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc0095f0 con 0x7f3de80ffaa0 2024-04-22T15:38:04.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.620+0000 7f3de659c700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3de8105aa0 0x7f3de819bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:38:04.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.621+0000 7f3de6d9d700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 0x7f3de819b6c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800c8a0 tx=0x7f3dd800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:04.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.621+0000 7f3dcffff700 1 -- 172.21.15.96:0/4283524458 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dd800cea0 con 0x7f3de80ffaa0 2024-04-22T15:38:04.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.621+0000 7f3dcffff700 1 -- 172.21.15.96:0/4283524458 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dd8004830 con 0x7f3de80ffaa0 2024-04-22T15:38:04.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.621+0000 7f3dcffff700 1 -- 172.21.15.96:0/4283524458 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dd80055b0 con 0x7f3de80ffaa0 2024-04-22T15:38:04.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.621+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de8195a90 con 0x7f3de80ffaa0 2024-04-22T15:38:04.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.621+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de8195f90 con 0x7f3de80ffaa0 2024-04-22T15:38:04.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.622+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de81011e0 con 0x7f3de80ffaa0 2024-04-22T15:38:04.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.623+0000 7f3dcffff700 1 -- 172.21.15.96:0/4283524458 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3dd80049a0 con 0x7f3de80ffaa0 2024-04-22T15:38:04.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.623+0000 7f3dcffff700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dd006ecc0 0x7f3dd0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:04.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.624+0000 7f3dcffff700 1 -- 172.21.15.96:0/4283524458 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3dd808eba0 con 0x7f3de80ffaa0 2024-04-22T15:38:04.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.624+0000 7f3de659c700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dd006ecc0 0x7f3dd0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:04.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.624+0000 7f3de659c700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dd006ecc0 0x7f3dd0071180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc006010 tx=0x7f3ddc00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:04.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.628+0000 7f3dcffff700 1 -- 172.21.15.96:0/4283524458 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3dd805b430 con 0x7f3de80ffaa0 2024-04-22T15:38:04.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.835+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3de810cdb0 con 0x7f3dd006ecc0 2024-04-22T15:38:04.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.846+0000 7f3dcffff700 1 -- 172.21.15.96:0/4283524458 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3de810cdb0 con 0x7f3dd006ecc0 2024-04-22T15:38:04.849 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:38:04.849 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (16m) 9m ago 28m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:38:04.849 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (28m) 9m ago 28m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:38:04.849 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (27m) 9m ago 27m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (13m) 9m ago 28m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (13m) 9m ago 27m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (16m) 9m ago 27m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (9m) 9m ago 22m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (10m) 9m ago 22m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (9m) 9m ago 22m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (22m) 9m ago 22m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (19m) 9m ago 29m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (17m) 9m ago 26m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (15m) 9m ago 29m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (14m) 9m ago 26m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:38:04.850 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (17m) 9m ago 28m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:38:04.851 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (17m) 9m ago 27m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:38:04.851 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (13m) 9m ago 26m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:38:04.851 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (12m) 9m ago 25m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:38:04.851 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (12m) 9m ago 25m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:38:04.851 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (11m) 9m ago 24m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:38:04.851 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (11m) 9m ago 24m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:38:04.851 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (10m) 9m ago 23m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:38:04.851 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (17m) 9m ago 27m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:38:04.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.848+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dd006ecc0 msgr2=0x7f3dd0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.848+0000 7f3ded072700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dd006ecc0 0x7f3dd0071180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc006010 tx=0x7f3ddc00b540 comp rx=0 tx=0).stop 2024-04-22T15:38:04.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.848+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 msgr2=0x7f3de819b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.848+0000 7f3ded072700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 0x7f3de819b6c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800c8a0 tx=0x7f3dd800cc60 comp rx=0 tx=0).stop 2024-04-22T15:38:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.848+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 shutdown_connections 2024-04-22T15:38:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.848+0000 7f3ded072700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dd006ecc0 0x7f3dd0071180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.848+0000 7f3ded072700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3de80ffaa0 0x7f3de819b6c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.848+0000 7f3ded072700 1 --2- 172.21.15.96:0/4283524458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3de8105aa0 0x7f3de819bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.848+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 >> 172.21.15.96:0/4283524458 conn(0x7f3de8076740 msgr2=0x7f3de80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.849+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 shutdown_connections 2024-04-22T15:38:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.849+0000 7f3ded072700 1 -- 172.21.15.96:0/4283524458 wait complete. 2024-04-22T15:38:04.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.953+0000 7f2953db6700 1 -- 172.21.15.96:0/1071708929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c1039e0 msgr2=0x7f294c107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.953+0000 7f2953db6700 1 --2- 172.21.15.96:0/1071708929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c1039e0 0x7f294c107a30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f293c009910 tx=0x7f293c009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:04.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.953+0000 7f2953db6700 1 -- 172.21.15.96:0/1071708929 shutdown_connections 2024-04-22T15:38:04.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.953+0000 7f2953db6700 1 --2- 172.21.15.96:0/1071708929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c1039e0 0x7f294c107a30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.953+0000 7f2953db6700 1 --2- 172.21.15.96:0/1071708929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f294c103030 0x7f294c103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.953+0000 7f2953db6700 1 -- 172.21.15.96:0/1071708929 >> 172.21.15.96:0/1071708929 conn(0x7f294c0fe880 msgr2=0x7f294c100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:04.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.953+0000 7f2953db6700 1 -- 172.21.15.96:0/1071708929 shutdown_connections 2024-04-22T15:38:04.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.953+0000 7f2953db6700 1 -- 172.21.15.96:0/1071708929 wait complete. 2024-04-22T15:38:04.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2953db6700 1 Processor -- start 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2953db6700 1 -- start start 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2953db6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c103030 0x7f294c198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2953db6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f294c1039e0 0x7f294c199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2953db6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c199a20 con 0x7f294c103030 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2953db6700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c19d6a0 con 0x7f294c1039e0 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2951b52700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c103030 0x7f294c198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2951b52700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c103030 0x7f294c198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49010/0 (socket says 172.21.15.96:49010) 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2951b52700 1 -- 172.21.15.96:0/38363689 learned_addr learned my addr 172.21.15.96:0/38363689 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2951351700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f294c1039e0 0x7f294c199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2951b52700 1 -- 172.21.15.96:0/38363689 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f294c1039e0 msgr2=0x7f294c199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2951b52700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f294c1039e0 0x7f294c199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:04.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.954+0000 7f2951b52700 1 -- 172.21.15.96:0/38363689 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c0095f0 con 0x7f294c103030 2024-04-22T15:38:04.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.955+0000 7f2951b52700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c103030 0x7f294c198e00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f294400ada0 tx=0x7f294400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:04.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.955+0000 7f2942ffd700 1 -- 172.21.15.96:0/38363689 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f294400f070 con 0x7f294c103030 2024-04-22T15:38:04.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.955+0000 7f2942ffd700 1 -- 172.21.15.96:0/38363689 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2944007bf0 con 0x7f294c103030 2024-04-22T15:38:04.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.955+0000 7f2942ffd700 1 -- 172.21.15.96:0/38363689 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2944005690 con 0x7f294c103030 2024-04-22T15:38:04.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.955+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294c19d980 con 0x7f294c103030 2024-04-22T15:38:04.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.955+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294c19ded0 con 0x7f294c103030 2024-04-22T15:38:04.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.956+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294c10b430 con 0x7f294c103030 2024-04-22T15:38:04.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.956+0000 7f2942ffd700 1 -- 172.21.15.96:0/38363689 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2944005850 con 0x7f294c103030 2024-04-22T15:38:04.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.957+0000 7f2942ffd700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f293806ecd0 0x7f2938071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:04.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.957+0000 7f2951351700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f293806ecd0 0x7f2938071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:04.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.957+0000 7f2942ffd700 1 -- 172.21.15.96:0/38363689 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f294408e610 con 0x7f294c103030 2024-04-22T15:38:04.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.957+0000 7f2951351700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f293806ecd0 0x7f2938071190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f293c006010 tx=0x7f293c00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:04.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:04.962+0000 7f2942ffd700 1 -- 172.21.15.96:0/38363689 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f29440573f0 con 0x7f294c103030 2024-04-22T15:38:05.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.247+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f294c19a0c0 con 0x7f294c103030 2024-04-22T15:38:05.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.248+0000 7f2942ffd700 1 -- 172.21.15.96:0/38363689 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f294405aab0 con 0x7f294c103030 2024-04-22T15:38:05.251 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:38:05.251 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:38:05.251 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:38:05.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:38:05.251 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:38:05.252 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:38:05.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f293806ecd0 msgr2=0x7f2938071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f293806ecd0 0x7f2938071190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f293c006010 tx=0x7f293c00b540 comp rx=0 tx=0).stop 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c103030 msgr2=0x7f294c198e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c103030 0x7f294c198e00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f294400ada0 tx=0x7f294400c330 comp rx=0 tx=0).stop 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 shutdown_connections 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f293806ecd0 0x7f2938071190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c103030 0x7f294c198e00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 --2- 172.21.15.96:0/38363689 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f294c1039e0 0x7f294c199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 >> 172.21.15.96:0/38363689 conn(0x7f294c0fe880 msgr2=0x7f294c0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.250+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 shutdown_connections 2024-04-22T15:38:05.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.251+0000 7f2953db6700 1 -- 172.21.15.96:0/38363689 wait complete. 2024-04-22T15:38:05.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.353+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/436799001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c4105b60 msgr2=0x7fa3c4105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.353+0000 7fa3c9fd9700 1 --2- 172.21.15.96:0/436799001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c4105b60 0x7fa3c4105f40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8009910 tx=0x7fa3b8009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:05.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.353+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/436799001 shutdown_connections 2024-04-22T15:38:05.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.353+0000 7fa3c9fd9700 1 --2- 172.21.15.96:0/436799001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3c40ffad0 0x7fa3c40fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.353+0000 7fa3c9fd9700 1 --2- 172.21.15.96:0/436799001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c4105b60 0x7fa3c4105f40 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.353+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/436799001 >> 172.21.15.96:0/436799001 conn(0x7fa3c40fb5e0 msgr2=0x7fa3c40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:05.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.353+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/436799001 shutdown_connections 2024-04-22T15:38:05.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.353+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/436799001 wait complete. 2024-04-22T15:38:05.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c9fd9700 1 Processor -- start 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c9fd9700 1 -- start start 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c9fd9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c40ffad0 0x7fa3c419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c9fd9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3c4105b60 0x7fa3c419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c9fd9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c419d390 con 0x7fa3c40ffad0 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c9fd9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c4196970 con 0x7fa3c4105b60 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c40ffad0 0x7fa3c419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c40ffad0 0x7fa3c419c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49038/0 (socket says 172.21.15.96:49038) 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c37fe700 1 -- 172.21.15.96:0/2854049631 learned_addr learned my addr 172.21.15.96:0/2854049631 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.354+0000 7fa3c2ffd700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3c4105b60 0x7fa3c419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.355+0000 7fa3c37fe700 1 -- 172.21.15.96:0/2854049631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3c4105b60 msgr2=0x7fa3c419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.355+0000 7fa3c37fe700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3c4105b60 0x7fa3c419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.355+0000 7fa3c37fe700 1 -- 172.21.15.96:0/2854049631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b80095f0 con 0x7fa3c40ffad0 2024-04-22T15:38:05.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.355+0000 7fa3c37fe700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c40ffad0 0x7fa3c419c830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800bd70 tx=0x7fa3b800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:05.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.355+0000 7fa3c0ff9700 1 -- 172.21.15.96:0/2854049631 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3b801d070 con 0x7fa3c40ffad0 2024-04-22T15:38:05.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.355+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c4196bf0 con 0x7fa3c40ffad0 2024-04-22T15:38:05.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.355+0000 7fa3c0ff9700 1 -- 172.21.15.96:0/2854049631 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3b80056e0 con 0x7fa3c40ffad0 2024-04-22T15:38:05.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.355+0000 7fa3c0ff9700 1 -- 172.21.15.96:0/2854049631 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3b8017400 con 0x7fa3c40ffad0 2024-04-22T15:38:05.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.355+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c41970e0 con 0x7fa3c40ffad0 2024-04-22T15:38:05.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.356+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c41090e0 con 0x7fa3c40ffad0 2024-04-22T15:38:05.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.356+0000 7fa3c0ff9700 1 -- 172.21.15.96:0/2854049631 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa3b8003a60 con 0x7fa3c40ffad0 2024-04-22T15:38:05.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.357+0000 7fa3c0ff9700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa3ac06ecc0 0x7fa3ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:05.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.357+0000 7fa3c2ffd700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa3ac06ecc0 0x7fa3ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:05.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.357+0000 7fa3c0ff9700 1 -- 172.21.15.96:0/2854049631 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa3b80902f0 con 0x7fa3c40ffad0 2024-04-22T15:38:05.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.357+0000 7fa3c2ffd700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa3ac06ecc0 0x7fa3ac071180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4005950 tx=0x7fa3b40058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:05.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.362+0000 7fa3c0ff9700 1 -- 172.21.15.96:0/2854049631 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa3b805cb80 con 0x7fa3c40ffad0 2024-04-22T15:38:05.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.610+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa3c4197e30 con 0x7fa3c40ffad0 2024-04-22T15:38:05.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.610+0000 7fa3c0ff9700 1 -- 172.21.15.96:0/2854049631 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fa3b805c710 con 0x7fa3c40ffad0 2024-04-22T15:38:05.613 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:38:05.613 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:38:05.613 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:38:05.614 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:38:05.615 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:38:05.616 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:38:05.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa3ac06ecc0 msgr2=0x7fa3ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa3ac06ecc0 0x7fa3ac071180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4005950 tx=0x7fa3b40058e0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c40ffad0 msgr2=0x7fa3c419c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c40ffad0 0x7fa3c419c830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800bd70 tx=0x7fa3b800be50 comp rx=0 tx=0).stop 2024-04-22T15:38:05.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 shutdown_connections 2024-04-22T15:38:05.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa3ac06ecc0 0x7fa3ac071180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3c40ffad0 0x7fa3c419c830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 --2- 172.21.15.96:0/2854049631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3c4105b60 0x7fa3c419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 >> 172.21.15.96:0/2854049631 conn(0x7fa3c40fb5e0 msgr2=0x7fa3c4069610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:05.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 shutdown_connections 2024-04-22T15:38:05.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.612+0000 7fa3c9fd9700 1 -- 172.21.15.96:0/2854049631 wait complete. 2024-04-22T15:38:05.618 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:38:05.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.717+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/3687166973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 msgr2=0x7fcc481021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.717+0000 7fcc4f6f0700 1 --2- 172.21.15.96:0/3687166973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 0x7fcc481021f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fcc38009910 tx=0x7fcc38009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.717+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/3687166973 shutdown_connections 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.717+0000 7fcc4f6f0700 1 --2- 172.21.15.96:0/3687166973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc48102730 0x7fcc4810ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.717+0000 7fcc4f6f0700 1 --2- 172.21.15.96:0/3687166973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 0x7fcc481021f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.717+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/3687166973 >> 172.21.15.96:0/3687166973 conn(0x7fcc480fb5e0 msgr2=0x7fcc480fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.717+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/3687166973 shutdown_connections 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.717+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/3687166973 wait complete. 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4f6f0700 1 Processor -- start 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4f6f0700 1 -- start start 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4f6f0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 0x7fcc480ffa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4f6f0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc48102730 0x7fcc480fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4f6f0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc481004f0 con 0x7fcc48101e10 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4f6f0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc48100660 con 0x7fcc48102730 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4d48c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 0x7fcc480ffa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:05.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4d48c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 0x7fcc480ffa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49054/0 (socket says 172.21.15.96:49054) 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4d48c700 1 -- 172.21.15.96:0/711489531 learned_addr learned my addr 172.21.15.96:0/711489531 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.718+0000 7fcc4cc8b700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc48102730 0x7fcc480fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc4d48c700 1 -- 172.21.15.96:0/711489531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc48102730 msgr2=0x7fcc480fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc4d48c700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc48102730 0x7fcc480fffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc4d48c700 1 -- 172.21.15.96:0/711489531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc380095f0 con 0x7fcc48101e10 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc4cc8b700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc48102730 0x7fcc480fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc4d48c700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 0x7fcc480ffa70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcc3800bd70 tx=0x7fcc3800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc3e7fc700 1 -- 172.21.15.96:0/711489531 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc3801d070 con 0x7fcc48101e10 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc3e7fc700 1 -- 172.21.15.96:0/711489531 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc3800f460 con 0x7fcc48101e10 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc481008e0 con 0x7fcc48101e10 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc3e7fc700 1 -- 172.21.15.96:0/711489531 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc38017630 con 0x7fcc48101e10 2024-04-22T15:38:05.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.719+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc481a3160 con 0x7fcc48101e10 2024-04-22T15:38:05.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.720+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc48069970 con 0x7fcc48101e10 2024-04-22T15:38:05.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.720+0000 7fcc3e7fc700 1 -- 172.21.15.96:0/711489531 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc3800f5d0 con 0x7fcc48101e10 2024-04-22T15:38:05.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.721+0000 7fcc3e7fc700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc3406ecc0 0x7fcc34071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:05.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.721+0000 7fcc3e7fc700 1 -- 172.21.15.96:0/711489531 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcc38090210 con 0x7fcc48101e10 2024-04-22T15:38:05.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.721+0000 7fcc4cc8b700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc3406ecc0 0x7fcc34071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:05.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.721+0000 7fcc4cc8b700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc3406ecc0 0x7fcc34071180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcc481011a0 tx=0x7fcc44008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:05.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.726+0000 7fcc3e7fc700 1 -- 172.21.15.96:0/711489531 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcc38058f70 con 0x7fcc48101e10 2024-04-22T15:38:05.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.945+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc48101030 con 0x7fcc3406ecc0 2024-04-22T15:38:05.948 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:05 smithi096 ceph-mon[112443]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:05.948 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:05 smithi096 ceph-mon[112443]: pgmap v486: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:05.949 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:05 smithi096 ceph-mon[112443]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:05.949 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:05 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/38363689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:38:05.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.946+0000 7fcc3e7fc700 1 -- 172.21.15.96:0/711489531 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcc48101030 con 0x7fcc3406ecc0 2024-04-22T15:38:05.949 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:38:05.949 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:38:05.950 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:38:05.951 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:38:05.951 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:38:05.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc3406ecc0 msgr2=0x7fcc34071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc3406ecc0 0x7fcc34071180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcc481011a0 tx=0x7fcc44008040 comp rx=0 tx=0).stop 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 msgr2=0x7fcc480ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 0x7fcc480ffa70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcc3800bd70 tx=0x7fcc3800be50 comp rx=0 tx=0).stop 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 shutdown_connections 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcc3406ecc0 0x7fcc34071180 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc48101e10 0x7fcc480ffa70 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 --2- 172.21.15.96:0/711489531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc48102730 0x7fcc480fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 >> 172.21.15.96:0/711489531 conn(0x7fcc480fb5e0 msgr2=0x7fcc48103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 shutdown_connections 2024-04-22T15:38:05.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:05.948+0000 7fcc4f6f0700 1 -- 172.21.15.96:0/711489531 wait complete. 2024-04-22T15:38:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:05 smithi196 ceph-mon[81815]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:06.095 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:05 smithi196 ceph-mon[81815]: pgmap v486: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:06.096 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:05 smithi196 ceph-mon[81815]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:06.096 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:05 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/38363689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.052+0000 7f93c8407700 1 -- 172.21.15.96:0/1875929004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c0105aa0 msgr2=0x7f93c0105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.052+0000 7f93c8407700 1 --2- 172.21.15.96:0/1875929004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c0105aa0 0x7f93c0105e80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009910 tx=0x7f93b0009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.052+0000 7f93c8407700 1 -- 172.21.15.96:0/1875929004 shutdown_connections 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.052+0000 7f93c8407700 1 --2- 172.21.15.96:0/1875929004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93c00ffaa0 0x7f93c00fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.052+0000 7f93c8407700 1 --2- 172.21.15.96:0/1875929004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c0105aa0 0x7f93c0105e80 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.052+0000 7f93c8407700 1 -- 172.21.15.96:0/1875929004 >> 172.21.15.96:0/1875929004 conn(0x7f93c0076740 msgr2=0x7f93c00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.052+0000 7f93c8407700 1 -- 172.21.15.96:0/1875929004 shutdown_connections 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.052+0000 7f93c8407700 1 -- 172.21.15.96:0/1875929004 wait complete. 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c8407700 1 Processor -- start 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c8407700 1 -- start start 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c8407700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c00ffaa0 0x7f93c019b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:06.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c8407700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93c0105aa0 0x7f93c019bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c8407700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c019c3c0 con 0x7f93c00ffaa0 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c8407700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c01959a0 con 0x7f93c0105aa0 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c61a3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c00ffaa0 0x7f93c019b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c61a3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c00ffaa0 0x7f93c019b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49074/0 (socket says 172.21.15.96:49074) 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c61a3700 1 -- 172.21.15.96:0/2037523563 learned_addr learned my addr 172.21.15.96:0/2037523563 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c61a3700 1 -- 172.21.15.96:0/2037523563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93c0105aa0 msgr2=0x7f93c019bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c59a2700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93c0105aa0 0x7f93c019bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.053+0000 7f93c61a3700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93c0105aa0 0x7f93c019bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.054+0000 7f93c61a3700 1 -- 172.21.15.96:0/2037523563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00095f0 con 0x7f93c00ffaa0 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.054+0000 7f93c61a3700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c00ffaa0 0x7f93c019b860 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f93b000bd70 tx=0x7f93b000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.054+0000 7f93b77fe700 1 -- 172.21.15.96:0/2037523563 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93b001f070 con 0x7f93c00ffaa0 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.054+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c0195c20 con 0x7f93c00ffaa0 2024-04-22T15:38:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.054+0000 7f93b77fe700 1 -- 172.21.15.96:0/2037523563 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93b0019610 con 0x7f93c00ffaa0 2024-04-22T15:38:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.054+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c0196110 con 0x7f93c00ffaa0 2024-04-22T15:38:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.054+0000 7f93b77fe700 1 -- 172.21.15.96:0/2037523563 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93b000f690 con 0x7f93c00ffaa0 2024-04-22T15:38:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.055+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c01011e0 con 0x7f93c00ffaa0 2024-04-22T15:38:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.055+0000 7f93b77fe700 1 -- 172.21.15.96:0/2037523563 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f93b0003a60 con 0x7f93c00ffaa0 2024-04-22T15:38:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.056+0000 7f93b77fe700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f93ac06ecc0 0x7f93ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.056+0000 7f93c59a2700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f93ac06ecc0 0x7f93ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.056+0000 7f93b77fe700 1 -- 172.21.15.96:0/2037523563 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f93b0092700 con 0x7f93c00ffaa0 2024-04-22T15:38:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.056+0000 7f93c59a2700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f93ac06ecc0 0x7f93ac071180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f93bc006fd0 tx=0x7f93bc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.060+0000 7f93b77fe700 1 -- 172.21.15.96:0/2037523563 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f93b005ef90 con 0x7f93c00ffaa0 2024-04-22T15:38:06.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.346+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f93c010cd20 con 0x7f93c00ffaa0 2024-04-22T15:38:06.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.347+0000 7f93b77fe700 1 -- 172.21.15.96:0/2037523563 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f93b005eb20 con 0x7f93c00ffaa0 2024-04-22T15:38:06.350 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:38:06.350 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:38:06.350 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:38:06.350 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:38:06.350 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:38:06.350 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:38:06.350 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:38:06.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f93ac06ecc0 msgr2=0x7f93ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:06.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f93ac06ecc0 0x7f93ac071180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f93bc006fd0 tx=0x7f93bc008040 comp rx=0 tx=0).stop 2024-04-22T15:38:06.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c00ffaa0 msgr2=0x7f93c019b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:06.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c00ffaa0 0x7f93c019b860 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f93b000bd70 tx=0x7f93b000be50 comp rx=0 tx=0).stop 2024-04-22T15:38:06.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 shutdown_connections 2024-04-22T15:38:06.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f93ac06ecc0 0x7f93ac071180 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:06.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93c00ffaa0 0x7f93c019b860 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:06.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 --2- 172.21.15.96:0/2037523563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93c0105aa0 0x7f93c019bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:06.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 >> 172.21.15.96:0/2037523563 conn(0x7f93c0076740 msgr2=0x7f93c00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:06.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 shutdown_connections 2024-04-22T15:38:06.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:06.349+0000 7f93c8407700 1 -- 172.21.15.96:0/2037523563 wait complete. 2024-04-22T15:38:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:06 smithi096 ceph-mon[112443]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:06 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2854049631' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:38:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:06 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2037523563' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:38:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:06 smithi196 ceph-mon[81815]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:06 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2854049631' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:38:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:06 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2037523563' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:38:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:07 smithi196 ceph-mon[81815]: from='client.34948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:07.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:07 smithi196 ceph-mon[81815]: pgmap v487: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:07 smithi096 ceph-mon[112443]: from='client.34948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:07 smithi096 ceph-mon[112443]: pgmap v487: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:38:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:38:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:09 smithi096 ceph-mon[112443]: pgmap v488: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:09 smithi196 ceph-mon[81815]: pgmap v488: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:10.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:10 smithi096 ceph-mon[112443]: pgmap v489: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:10 smithi196 ceph-mon[81815]: pgmap v489: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:13 smithi096 ceph-mon[112443]: pgmap v490: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:13.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:13 smithi196 ceph-mon[81815]: pgmap v490: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:15.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:15 smithi196 ceph-mon[81815]: pgmap v491: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:15.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:15 smithi096 ceph-mon[112443]: pgmap v491: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:17.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:17 smithi196 ceph-mon[81815]: pgmap v492: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:17.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:17 smithi096 ceph-mon[112443]: pgmap v492: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:19 smithi196 ceph-mon[81815]: pgmap v493: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:19 smithi096 ceph-mon[112443]: pgmap v493: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:21.763 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:21 smithi196 ceph-mon[81815]: pgmap v494: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:21.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:21 smithi096 ceph-mon[112443]: pgmap v494: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:23 smithi196 ceph-mon[81815]: pgmap v495: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:23.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:38:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:23 smithi096 ceph-mon[112443]: pgmap v495: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:38:25.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:25 smithi196 ceph-mon[81815]: pgmap v496: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:25.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:25 smithi096 ceph-mon[112443]: pgmap v496: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:27.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:27 smithi096 ceph-mon[112443]: pgmap v497: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:27 smithi196 ceph-mon[81815]: pgmap v497: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:29 smithi096 ceph-mon[112443]: pgmap v498: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:29 smithi196 ceph-mon[81815]: pgmap v498: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:31.788 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:31 smithi096 ceph-mon[112443]: pgmap v499: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:32.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:31 smithi196 ceph-mon[81815]: pgmap v499: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:33.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:33 smithi096 ceph-mon[112443]: pgmap v500: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:33 smithi196 ceph-mon[81815]: pgmap v500: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:35.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:35 smithi096 ceph-mon[112443]: pgmap v501: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:35 smithi196 ceph-mon[81815]: pgmap v501: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:36.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 -- 172.21.15.96:0/3688690995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 msgr2=0x7efc5c0fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:36.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 --2- 172.21.15.96:0/3688690995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 0x7efc5c0fff50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efc48009910 tx=0x7efc48009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:36.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 -- 172.21.15.96:0/3688690995 shutdown_connections 2024-04-22T15:38:36.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 --2- 172.21.15.96:0/3688690995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 0x7efc5c0fff50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 --2- 172.21.15.96:0/3688690995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c105b60 0x7efc5c105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 -- 172.21.15.96:0/3688690995 >> 172.21.15.96:0/3688690995 conn(0x7efc5c0fb5e0 msgr2=0x7efc5c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 -- 172.21.15.96:0/3688690995 shutdown_connections 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 -- 172.21.15.96:0/3688690995 wait complete. 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 Processor -- start 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.466+0000 7efc63095700 1 -- start start 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc63095700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 0x7efc5c19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc63095700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c105b60 0x7efc5c19cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc63095700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c19d2e0 con 0x7efc5c0ffad0 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc63095700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c196950 con 0x7efc5c105b60 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc60e31700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 0x7efc5c19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc60e31700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 0x7efc5c19c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49696/0 (socket says 172.21.15.96:49696) 2024-04-22T15:38:36.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc60e31700 1 -- 172.21.15.96:0/1048136810 learned_addr learned my addr 172.21.15.96:0/1048136810 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc5bfff700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c105b60 0x7efc5c19cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc60e31700 1 -- 172.21.15.96:0/1048136810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c105b60 msgr2=0x7efc5c19cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc60e31700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c105b60 0x7efc5c19cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc60e31700 1 -- 172.21.15.96:0/1048136810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc480095f0 con 0x7efc5c0ffad0 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc5bfff700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c105b60 0x7efc5c19cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.467+0000 7efc60e31700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 0x7efc5c19c810 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c930 tx=0x7efc5000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.468+0000 7efc59ffb700 1 -- 172.21.15.96:0/1048136810 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc50007ad0 con 0x7efc5c0ffad0 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.468+0000 7efc59ffb700 1 -- 172.21.15.96:0/1048136810 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc50007c30 con 0x7efc5c0ffad0 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.468+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5c196c30 con 0x7efc5c0ffad0 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.468+0000 7efc59ffb700 1 -- 172.21.15.96:0/1048136810 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc50018610 con 0x7efc5c0ffad0 2024-04-22T15:38:36.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.468+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5c197180 con 0x7efc5c0ffad0 2024-04-22T15:38:36.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.469+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5c109100 con 0x7efc5c0ffad0 2024-04-22T15:38:36.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.469+0000 7efc59ffb700 1 -- 172.21.15.96:0/1048136810 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efc5000f450 con 0x7efc5c0ffad0 2024-04-22T15:38:36.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.470+0000 7efc59ffb700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efc4c06ecc0 0x7efc4c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:36.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.470+0000 7efc59ffb700 1 -- 172.21.15.96:0/1048136810 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7efc5008f0f0 con 0x7efc5c0ffad0 2024-04-22T15:38:36.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.470+0000 7efc5bfff700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efc4c06ecc0 0x7efc4c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:36.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.470+0000 7efc5bfff700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efc4c06ecc0 0x7efc4c071180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efc48005f50 tx=0x7efc48005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:36.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.475+0000 7efc59ffb700 1 -- 172.21.15.96:0/1048136810 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efc5005b980 con 0x7efc5c0ffad0 2024-04-22T15:38:36.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.700+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efc5c197e10 con 0x7efc4c06ecc0 2024-04-22T15:38:36.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.701+0000 7efc59ffb700 1 -- 172.21.15.96:0/1048136810 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efc5c197e10 con 0x7efc4c06ecc0 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efc4c06ecc0 msgr2=0x7efc4c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efc4c06ecc0 0x7efc4c071180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efc48005f50 tx=0x7efc48005dc0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 msgr2=0x7efc5c19c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 0x7efc5c19c810 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c930 tx=0x7efc5000ccf0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 shutdown_connections 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efc4c06ecc0 0x7efc4c071180 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc5c0ffad0 0x7efc5c19c810 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 --2- 172.21.15.96:0/1048136810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c105b60 0x7efc5c19cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 >> 172.21.15.96:0/1048136810 conn(0x7efc5c0fb5e0 msgr2=0x7efc5c069690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 shutdown_connections 2024-04-22T15:38:36.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.704+0000 7efc63095700 1 -- 172.21.15.96:0/1048136810 wait complete. 2024-04-22T15:38:36.717 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:38:36.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 -- 172.21.15.96:0/2934371084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 msgr2=0x7f36e01006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:36.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 --2- 172.21.15.96:0/2934371084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 0x7f36e01006a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009910 tx=0x7f36dc009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:36.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 -- 172.21.15.96:0/2934371084 shutdown_connections 2024-04-22T15:38:36.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 --2- 172.21.15.96:0/2934371084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 0x7f36e01006a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 --2- 172.21.15.96:0/2934371084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36e0104b30 0x7f36e0104f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 -- 172.21.15.96:0/2934371084 >> 172.21.15.96:0/2934371084 conn(0x7f36e0075f40 msgr2=0x7f36e00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:36.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 -- 172.21.15.96:0/2934371084 shutdown_connections 2024-04-22T15:38:36.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 -- 172.21.15.96:0/2934371084 wait complete. 2024-04-22T15:38:36.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 Processor -- start 2024-04-22T15:38:36.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.808+0000 7f36e75cb700 1 -- start start 2024-04-22T15:38:36.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e75cb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 0x7f36e0197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e75cb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36e0104b30 0x7f36e0197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e75cb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01983a0 con 0x7f36e0100220 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e75cb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01984e0 con 0x7f36e0104b30 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e5367700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 0x7f36e0197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e5367700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 0x7f36e0197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49704/0 (socket says 172.21.15.96:49704) 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e5367700 1 -- 172.21.15.96:0/2359639889 learned_addr learned my addr 172.21.15.96:0/2359639889 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e4b66700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36e0104b30 0x7f36e0197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e5367700 1 -- 172.21.15.96:0/2359639889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36e0104b30 msgr2=0x7f36e0197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e5367700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36e0104b30 0x7f36e0197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e5367700 1 -- 172.21.15.96:0/2359639889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc0095f0 con 0x7f36e0100220 2024-04-22T15:38:36.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e4b66700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36e0104b30 0x7f36e0197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:38:36.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e5367700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 0x7f36e0197920 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f36d400d900 tx=0x7f36d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:36.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36d27fc700 1 -- 172.21.15.96:0/2359639889 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36d4004a00 con 0x7f36e0100220 2024-04-22T15:38:36.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36d27fc700 1 -- 172.21.15.96:0/2359639889 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36d40054e0 con 0x7f36e0100220 2024-04-22T15:38:36.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e01a0990 con 0x7f36e0100220 2024-04-22T15:38:36.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.809+0000 7f36d27fc700 1 -- 172.21.15.96:0/2359639889 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36d4009d60 con 0x7f36e0100220 2024-04-22T15:38:36.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.810+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e01a0e20 con 0x7f36e0100220 2024-04-22T15:38:36.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.810+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e0108120 con 0x7f36e0100220 2024-04-22T15:38:36.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.811+0000 7f36d27fc700 1 -- 172.21.15.96:0/2359639889 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f36d4005020 con 0x7f36e0100220 2024-04-22T15:38:36.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.811+0000 7f36d27fc700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f36cc06ecc0 0x7f36cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:36.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.812+0000 7f36e4b66700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f36cc06ecc0 0x7f36cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:36.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.812+0000 7f36d27fc700 1 -- 172.21.15.96:0/2359639889 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f36d408eff0 con 0x7f36e0100220 2024-04-22T15:38:36.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.812+0000 7f36e4b66700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f36cc06ecc0 0x7f36cc071180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36dc006010 tx=0x7f36dc00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:36.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:36.816+0000 7f36d27fc700 1 -- 172.21.15.96:0/2359639889 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f36d4057dd0 con 0x7f36e0100220 2024-04-22T15:38:37.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.033+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36e00ff740 con 0x7f36cc06ecc0 2024-04-22T15:38:37.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.034+0000 7f36d27fc700 1 -- 172.21.15.96:0/2359639889 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f36e00ff740 con 0x7f36cc06ecc0 2024-04-22T15:38:37.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f36cc06ecc0 msgr2=0x7f36cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f36cc06ecc0 0x7f36cc071180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36dc006010 tx=0x7f36dc00b540 comp rx=0 tx=0).stop 2024-04-22T15:38:37.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 msgr2=0x7f36e0197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 0x7f36e0197920 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f36d400d900 tx=0x7f36d400dcc0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 shutdown_connections 2024-04-22T15:38:37.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f36cc06ecc0 0x7f36cc071180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e0100220 0x7f36e0197920 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 --2- 172.21.15.96:0/2359639889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36e0104b30 0x7f36e0197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 >> 172.21.15.96:0/2359639889 conn(0x7f36e0075f40 msgr2=0x7f36e00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:37.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 shutdown_connections 2024-04-22T15:38:37.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.036+0000 7f36e75cb700 1 -- 172.21.15.96:0/2359639889 wait complete. 2024-04-22T15:38:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.140+0000 7fb36c2be700 1 -- 172.21.15.96:0/4162268417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb364105b60 msgr2=0x7fb364105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.140+0000 7fb36c2be700 1 --2- 172.21.15.96:0/4162268417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb364105b60 0x7fb364105f40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb360009910 tx=0x7fb360009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.140+0000 7fb36c2be700 1 -- 172.21.15.96:0/4162268417 shutdown_connections 2024-04-22T15:38:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.140+0000 7fb36c2be700 1 --2- 172.21.15.96:0/4162268417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3640ffad0 0x7fb3640fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.140+0000 7fb36c2be700 1 --2- 172.21.15.96:0/4162268417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb364105b60 0x7fb364105f40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.140+0000 7fb36c2be700 1 -- 172.21.15.96:0/4162268417 >> 172.21.15.96:0/4162268417 conn(0x7fb3640fb600 msgr2=0x7fb3640fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.140+0000 7fb36c2be700 1 -- 172.21.15.96:0/4162268417 shutdown_connections 2024-04-22T15:38:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.140+0000 7fb36c2be700 1 -- 172.21.15.96:0/4162268417 wait complete. 2024-04-22T15:38:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.140+0000 7fb36c2be700 1 Processor -- start 2024-04-22T15:38:37.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36c2be700 1 -- start start 2024-04-22T15:38:37.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36c2be700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3640ffad0 0x7fb36419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:37.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36c2be700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb364105b60 0x7fb36419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:37.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36c2be700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36419d2a0 con 0x7fb3640ffad0 2024-04-22T15:38:37.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36c2be700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb364196910 con 0x7fb364105b60 2024-04-22T15:38:37.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36a05a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3640ffad0 0x7fb36419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:37.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36a05a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3640ffad0 0x7fb36419c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49728/0 (socket says 172.21.15.96:49728) 2024-04-22T15:38:37.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36a05a700 1 -- 172.21.15.96:0/2498902380 learned_addr learned my addr 172.21.15.96:0/2498902380 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:37.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb369859700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb364105b60 0x7fb36419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:37.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36a05a700 1 -- 172.21.15.96:0/2498902380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb364105b60 msgr2=0x7fb36419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36a05a700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb364105b60 0x7fb36419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36a05a700 1 -- 172.21.15.96:0/2498902380 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3600095f0 con 0x7fb3640ffad0 2024-04-22T15:38:37.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.141+0000 7fb36a05a700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3640ffad0 0x7fb36419c7d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb360006010 tx=0x7fb36000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:37.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.142+0000 7fb35b7fe700 1 -- 172.21.15.96:0/2498902380 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36001d070 con 0x7fb3640ffad0 2024-04-22T15:38:37.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.142+0000 7fb35b7fe700 1 -- 172.21.15.96:0/2498902380 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb36000f460 con 0x7fb3640ffad0 2024-04-22T15:38:37.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.142+0000 7fb35b7fe700 1 -- 172.21.15.96:0/2498902380 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3600175f0 con 0x7fb3640ffad0 2024-04-22T15:38:37.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.142+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb364196b90 con 0x7fb3640ffad0 2024-04-22T15:38:37.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.142+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb364197080 con 0x7fb3640ffad0 2024-04-22T15:38:37.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.143+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3641090e0 con 0x7fb3640ffad0 2024-04-22T15:38:37.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.143+0000 7fb35b7fe700 1 -- 172.21.15.96:0/2498902380 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb36000f5d0 con 0x7fb3640ffad0 2024-04-22T15:38:37.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.144+0000 7fb35b7fe700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb35006ecc0 0x7fb350071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:37.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.144+0000 7fb369859700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb35006ecc0 0x7fb350071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:37.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.144+0000 7fb35b7fe700 1 -- 172.21.15.96:0/2498902380 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb360090280 con 0x7fb3640ffad0 2024-04-22T15:38:37.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.144+0000 7fb369859700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb35006ecc0 0x7fb350071180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb364197ad0 tx=0x7fb35400a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:37.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.149+0000 7fb35b7fe700 1 -- 172.21.15.96:0/2498902380 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb36005cb90 con 0x7fb3640ffad0 2024-04-22T15:38:37.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.355+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb364197dd0 con 0x7fb35006ecc0 2024-04-22T15:38:37.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.366+0000 7fb35b7fe700 1 -- 172.21.15.96:0/2498902380 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb364197dd0 con 0x7fb35006ecc0 2024-04-22T15:38:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:38:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (17m) 10m ago 28m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:38:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (28m) 10m ago 28m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:38:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (27m) 10m ago 27m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:38:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (14m) 10m ago 28m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:38:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (14m) 10m ago 27m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (16m) 10m ago 28m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (10m) 10m ago 22m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (10m) 10m ago 22m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (10m) 10m ago 22m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (22m) 10m ago 22m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (19m) 10m ago 30m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (18m) 10m ago 27m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (15m) 10m ago 30m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (15m) 10m ago 27m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (18m) 10m ago 28m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (18m) 10m ago 27m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (14m) 10m ago 26m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:38:37.370 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (13m) 10m ago 26m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:38:37.371 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (12m) 10m ago 25m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:38:37.371 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (12m) 10m ago 25m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:38:37.371 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (11m) 10m ago 24m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:38:37.371 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (11m) 10m ago 24m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:38:37.371 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (17m) 10m ago 28m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:38:37.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb35006ecc0 msgr2=0x7fb350071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb35006ecc0 0x7fb350071180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb364197ad0 tx=0x7fb35400a380 comp rx=0 tx=0).stop 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3640ffad0 msgr2=0x7fb36419c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3640ffad0 0x7fb36419c7d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb360006010 tx=0x7fb36000bd70 comp rx=0 tx=0).stop 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 shutdown_connections 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb35006ecc0 0x7fb350071180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3640ffad0 0x7fb36419c7d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 --2- 172.21.15.96:0/2498902380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb364105b60 0x7fb36419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 >> 172.21.15.96:0/2498902380 conn(0x7fb3640fb600 msgr2=0x7fb364069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 shutdown_connections 2024-04-22T15:38:37.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.368+0000 7fb36c2be700 1 -- 172.21.15.96:0/2498902380 wait complete. 2024-04-22T15:38:37.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.470+0000 7f56c9e37700 1 -- 172.21.15.96:0/55591200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c41017d0 msgr2=0x7f56c4105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.470+0000 7f56c9e37700 1 --2- 172.21.15.96:0/55591200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c41017d0 0x7f56c4105820 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009910 tx=0x7f56b8009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:37.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.471+0000 7f56c9e37700 1 -- 172.21.15.96:0/55591200 shutdown_connections 2024-04-22T15:38:37.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.471+0000 7f56c9e37700 1 --2- 172.21.15.96:0/55591200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c41017d0 0x7f56c4105820 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.471+0000 7f56c9e37700 1 --2- 172.21.15.96:0/55591200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56c4100e20 0x7f56c4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.471+0000 7f56c9e37700 1 -- 172.21.15.96:0/55591200 >> 172.21.15.96:0/55591200 conn(0x7f56c40fc6d0 msgr2=0x7f56c40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.471+0000 7f56c9e37700 1 -- 172.21.15.96:0/55591200 shutdown_connections 2024-04-22T15:38:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.471+0000 7f56c9e37700 1 -- 172.21.15.96:0/55591200 wait complete. 2024-04-22T15:38:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.471+0000 7f56c9e37700 1 Processor -- start 2024-04-22T15:38:37.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.471+0000 7f56c9e37700 1 -- start start 2024-04-22T15:38:37.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c9e37700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c4100e20 0x7f56c40731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:37.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c9e37700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56c41017d0 0x7f56c4073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:37.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c9e37700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40751b0 con 0x7f56c4100e20 2024-04-22T15:38:37.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c9e37700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c4073c60 con 0x7f56c41017d0 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c4100e20 0x7f56c40731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c4100e20 0x7f56c40731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49748/0 (socket says 172.21.15.96:49748) 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c37fe700 1 -- 172.21.15.96:0/1737704947 learned_addr learned my addr 172.21.15.96:0/1737704947 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c2ffd700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56c41017d0 0x7f56c4073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c37fe700 1 -- 172.21.15.96:0/1737704947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56c41017d0 msgr2=0x7f56c4073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c37fe700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56c41017d0 0x7f56c4073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c37fe700 1 -- 172.21.15.96:0/1737704947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80095f0 con 0x7f56c4100e20 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c2ffd700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56c41017d0 0x7f56c4073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c37fe700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c4100e20 0x7f56c40731e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c930 tx=0x7f56b400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c0ff9700 1 -- 172.21.15.96:0/1737704947 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56b4007ad0 con 0x7f56c4100e20 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c0ff9700 1 -- 172.21.15.96:0/1737704947 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56b4007c30 con 0x7f56c4100e20 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.472+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c4073f10 con 0x7f56c4100e20 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.473+0000 7f56c0ff9700 1 -- 172.21.15.96:0/1737704947 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56b4018610 con 0x7f56c4100e20 2024-04-22T15:38:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.473+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c41a2f60 con 0x7f56c4100e20 2024-04-22T15:38:37.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.473+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c4109220 con 0x7f56c4100e20 2024-04-22T15:38:37.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.474+0000 7f56c0ff9700 1 -- 172.21.15.96:0/1737704947 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f56b400f450 con 0x7f56c4100e20 2024-04-22T15:38:37.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.474+0000 7f56c0ff9700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f56ac06ecd0 0x7f56ac071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:37.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.475+0000 7f56c0ff9700 1 -- 172.21.15.96:0/1737704947 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f56b408f070 con 0x7f56c4100e20 2024-04-22T15:38:37.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.475+0000 7f56c2ffd700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f56ac06ecd0 0x7f56ac071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:37.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.475+0000 7f56c2ffd700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f56ac06ecd0 0x7f56ac071190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f56b8006010 tx=0x7f56b800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:37.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.479+0000 7f56c0ff9700 1 -- 172.21.15.96:0/1737704947 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f56b4057dd0 con 0x7f56c4100e20 2024-04-22T15:38:37.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:37 smithi096 ceph-mon[112443]: pgmap v502: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:37.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.763+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f56c40746c0 con 0x7f56c4100e20 2024-04-22T15:38:37.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.763+0000 7f56c0ff9700 1 -- 172.21.15.96:0/1737704947 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f56b405b490 con 0x7f56c4100e20 2024-04-22T15:38:37.766 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:38:37.766 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:38:37.766 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:38:37.767 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:38:37.768 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:38:37.768 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:38:37.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.765+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f56ac06ecd0 msgr2=0x7f56ac071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.765+0000 7f56c9e37700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f56ac06ecd0 0x7f56ac071190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f56b8006010 tx=0x7f56b800b540 comp rx=0 tx=0).stop 2024-04-22T15:38:37.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.765+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c4100e20 msgr2=0x7f56c40731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.765+0000 7f56c9e37700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c4100e20 0x7f56c40731e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c930 tx=0x7f56b400ccf0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.765+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 shutdown_connections 2024-04-22T15:38:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.765+0000 7f56c9e37700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f56ac06ecd0 0x7f56ac071190 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.765+0000 7f56c9e37700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56c4100e20 0x7f56c40731e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.765+0000 7f56c9e37700 1 --2- 172.21.15.96:0/1737704947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56c41017d0 0x7f56c4073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.765+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 >> 172.21.15.96:0/1737704947 conn(0x7f56c40fc6d0 msgr2=0x7f56c40fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.766+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 shutdown_connections 2024-04-22T15:38:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.766+0000 7f56c9e37700 1 -- 172.21.15.96:0/1737704947 wait complete. 2024-04-22T15:38:37.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.869+0000 7fe06bbbe700 1 -- 172.21.15.96:0/273926927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0641002a0 msgr2=0x7fe06410cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.869+0000 7fe06bbbe700 1 --2- 172.21.15.96:0/273926927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0641002a0 0x7fe06410cf60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe054009910 tx=0x7fe054009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:37.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.869+0000 7fe06bbbe700 1 -- 172.21.15.96:0/273926927 shutdown_connections 2024-04-22T15:38:37.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.869+0000 7fe06bbbe700 1 --2- 172.21.15.96:0/273926927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0641002a0 0x7fe06410cf60 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.869+0000 7fe06bbbe700 1 --2- 172.21.15.96:0/273926927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0640ff8f0 0x7fe0640ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.869+0000 7fe06bbbe700 1 -- 172.21.15.96:0/273926927 >> 172.21.15.96:0/273926927 conn(0x7fe064076740 msgr2=0x7fe0640fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:37.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.869+0000 7fe06bbbe700 1 -- 172.21.15.96:0/273926927 shutdown_connections 2024-04-22T15:38:37.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.869+0000 7fe06bbbe700 1 -- 172.21.15.96:0/273926927 wait complete. 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.869+0000 7fe06bbbe700 1 Processor -- start 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06bbbe700 1 -- start start 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06bbbe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0640ff8f0 0x7fe064198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06bbbe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0641002a0 0x7fe064199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06bbbe700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0641999f0 con 0x7fe0640ff8f0 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06bbbe700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06419d670 con 0x7fe0641002a0 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06995a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0640ff8f0 0x7fe064198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06995a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0640ff8f0 0x7fe064198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49780/0 (socket says 172.21.15.96:49780) 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06995a700 1 -- 172.21.15.96:0/2910389748 learned_addr learned my addr 172.21.15.96:0/2910389748 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe069159700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0641002a0 0x7fe064199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06995a700 1 -- 172.21.15.96:0/2910389748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0641002a0 msgr2=0x7fe064199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06995a700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0641002a0 0x7fe064199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:37.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06995a700 1 -- 172.21.15.96:0/2910389748 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0540095f0 con 0x7fe0640ff8f0 2024-04-22T15:38:37.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe069159700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0641002a0 0x7fe064199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:38:37.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06995a700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0640ff8f0 0x7fe064198dd0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00ba50 tx=0x7fe05c00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:37.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe05affd700 1 -- 172.21.15.96:0/2910389748 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe05c00ca00 con 0x7fe0640ff8f0 2024-04-22T15:38:37.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06419d950 con 0x7fe0640ff8f0 2024-04-22T15:38:37.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe05affd700 1 -- 172.21.15.96:0/2910389748 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe05c00cb60 con 0x7fe0640ff8f0 2024-04-22T15:38:37.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.870+0000 7fe05affd700 1 -- 172.21.15.96:0/2910389748 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe05c0075a0 con 0x7fe0640ff8f0 2024-04-22T15:38:37.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.871+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06419dea0 con 0x7fe0640ff8f0 2024-04-22T15:38:37.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.871+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe064068ea0 con 0x7fe0640ff8f0 2024-04-22T15:38:37.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.872+0000 7fe05affd700 1 -- 172.21.15.96:0/2910389748 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe05c00ccd0 con 0x7fe0640ff8f0 2024-04-22T15:38:37.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.873+0000 7fe05affd700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe05006ecc0 0x7fe050071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:37.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.873+0000 7fe05affd700 1 -- 172.21.15.96:0/2910389748 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe05c006e40 con 0x7fe0640ff8f0 2024-04-22T15:38:37.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.873+0000 7fe069159700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe05006ecc0 0x7fe050071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:37.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.873+0000 7fe069159700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe05006ecc0 0x7fe050071180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe05400b5c0 tx=0x7fe05400b7f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:37.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:37.877+0000 7fe05affd700 1 -- 172.21.15.96:0/2910389748 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe05c057430 con 0x7fe0640ff8f0 2024-04-22T15:38:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:37 smithi196 ceph-mon[81815]: pgmap v502: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:38.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.119+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe064107860 con 0x7fe0640ff8f0 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.120+0000 7fe05affd700 1 -- 172.21.15.96:0/2910389748 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fe05c05aaf0 con 0x7fe0640ff8f0 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:38:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:38:38.125 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:38:38.126 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:38:38.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe05006ecc0 msgr2=0x7fe050071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe05006ecc0 0x7fe050071180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe05400b5c0 tx=0x7fe05400b7f0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0640ff8f0 msgr2=0x7fe064198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0640ff8f0 0x7fe064198dd0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00ba50 tx=0x7fe05c00be10 comp rx=0 tx=0).stop 2024-04-22T15:38:38.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 shutdown_connections 2024-04-22T15:38:38.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe05006ecc0 0x7fe050071180 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0640ff8f0 0x7fe064198dd0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 --2- 172.21.15.96:0/2910389748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0641002a0 0x7fe064199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 >> 172.21.15.96:0/2910389748 conn(0x7fe064076740 msgr2=0x7fe0640fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:38.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 shutdown_connections 2024-04-22T15:38:38.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.123+0000 7fe06bbbe700 1 -- 172.21.15.96:0/2910389748 wait complete. 2024-04-22T15:38:38.128 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:38:38.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.229+0000 7f40bfec2700 1 -- 172.21.15.96:0/903101516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b80ffad0 msgr2=0x7f40b80fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.229+0000 7f40bfec2700 1 --2- 172.21.15.96:0/903101516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b80ffad0 0x7f40b80fff50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009910 tx=0x7f40a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:38.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.229+0000 7f40bfec2700 1 -- 172.21.15.96:0/903101516 shutdown_connections 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.229+0000 7f40bfec2700 1 --2- 172.21.15.96:0/903101516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b80ffad0 0x7f40b80fff50 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.229+0000 7f40bfec2700 1 --2- 172.21.15.96:0/903101516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40b8105b60 0x7f40b8105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.229+0000 7f40bfec2700 1 -- 172.21.15.96:0/903101516 >> 172.21.15.96:0/903101516 conn(0x7f40b80fb600 msgr2=0x7f40b80fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.229+0000 7f40bfec2700 1 -- 172.21.15.96:0/903101516 shutdown_connections 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.229+0000 7f40bfec2700 1 -- 172.21.15.96:0/903101516 wait complete. 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bfec2700 1 Processor -- start 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bfec2700 1 -- start start 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bfec2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40b80ffad0 0x7f40b8198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bfec2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b8105b60 0x7f40b8199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bfec2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b81999e0 con 0x7f40b8105b60 2024-04-22T15:38:38.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bfec2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b819d6b0 con 0x7f40b80ffad0 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bd45d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b8105b60 0x7f40b8199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bd45d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b8105b60 0x7f40b8199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49796/0 (socket says 172.21.15.96:49796) 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bd45d700 1 -- 172.21.15.96:0/464710389 learned_addr learned my addr 172.21.15.96:0/464710389 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bdc5e700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40b80ffad0 0x7f40b8198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bd45d700 1 -- 172.21.15.96:0/464710389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40b80ffad0 msgr2=0x7f40b8198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bd45d700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40b80ffad0 0x7f40b8198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bd45d700 1 -- 172.21.15.96:0/464710389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a80095f0 con 0x7f40b8105b60 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.230+0000 7f40bdc5e700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40b80ffad0 0x7f40b8198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.231+0000 7f40bd45d700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b8105b60 0x7f40b8199300 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f40a800bd70 tx=0x7f40a800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.231+0000 7f40aeffd700 1 -- 172.21.15.96:0/464710389 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40a801d070 con 0x7f40b8105b60 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.231+0000 7f40aeffd700 1 -- 172.21.15.96:0/464710389 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40a800f460 con 0x7f40b8105b60 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.231+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b819d930 con 0x7f40b8105b60 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.231+0000 7f40aeffd700 1 -- 172.21.15.96:0/464710389 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40a8017630 con 0x7f40b8105b60 2024-04-22T15:38:38.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.231+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b819de20 con 0x7f40b8105b60 2024-04-22T15:38:38.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.232+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b81011d0 con 0x7f40b8105b60 2024-04-22T15:38:38.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.232+0000 7f40aeffd700 1 -- 172.21.15.96:0/464710389 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f40a800f5d0 con 0x7f40b8105b60 2024-04-22T15:38:38.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.233+0000 7f40aeffd700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f40a406ecc0 0x7f40a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:38.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.233+0000 7f40aeffd700 1 -- 172.21.15.96:0/464710389 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f40a80902e0 con 0x7f40b8105b60 2024-04-22T15:38:38.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.233+0000 7f40bdc5e700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f40a406ecc0 0x7f40a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:38.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.233+0000 7f40bdc5e700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f40a406ecc0 0x7f40a4071180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f40b40099d0 tx=0x7f40b4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:38.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.237+0000 7f40aeffd700 1 -- 172.21.15.96:0/464710389 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f40a80590c0 con 0x7f40b8105b60 2024-04-22T15:38:38.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.457+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f40b810ce60 con 0x7f40a406ecc0 2024-04-22T15:38:38.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.458+0000 7f40aeffd700 1 -- 172.21.15.96:0/464710389 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f40b810ce60 con 0x7f40a406ecc0 2024-04-22T15:38:38.461 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:38:38.461 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:38:38.461 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:38:38.461 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:38:38.462 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:38:38.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f40a406ecc0 msgr2=0x7f40a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f40a406ecc0 0x7f40a4071180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f40b40099d0 tx=0x7f40b4008040 comp rx=0 tx=0).stop 2024-04-22T15:38:38.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b8105b60 msgr2=0x7f40b8199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b8105b60 0x7f40b8199300 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f40a800bd70 tx=0x7f40a800be50 comp rx=0 tx=0).stop 2024-04-22T15:38:38.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 shutdown_connections 2024-04-22T15:38:38.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f40a406ecc0 0x7f40a4071180 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40b80ffad0 0x7f40b8198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 --2- 172.21.15.96:0/464710389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b8105b60 0x7f40b8199300 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 >> 172.21.15.96:0/464710389 conn(0x7f40b80fb600 msgr2=0x7f40b80fc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:38.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 shutdown_connections 2024-04-22T15:38:38.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.460+0000 7f40bfec2700 1 -- 172.21.15.96:0/464710389 wait complete. 2024-04-22T15:38:38.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.566+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3031118032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f54105b60 msgr2=0x7f8f54105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.566+0000 7f8f5b57f700 1 --2- 172.21.15.96:0/3031118032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f54105b60 0x7f8f54105f40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009910 tx=0x7f8f44009c20 comp rx=0 tx=0).stop 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.566+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3031118032 shutdown_connections 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.566+0000 7f8f5b57f700 1 --2- 172.21.15.96:0/3031118032 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f540ffad0 0x7f8f540fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.566+0000 7f8f5b57f700 1 --2- 172.21.15.96:0/3031118032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f54105b60 0x7f8f54105f40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.566+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3031118032 >> 172.21.15.96:0/3031118032 conn(0x7f8f540fb5e0 msgr2=0x7f8f540fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.566+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3031118032 shutdown_connections 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.566+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3031118032 wait complete. 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f5b57f700 1 Processor -- start 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f5b57f700 1 -- start start 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f5b57f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f540ffad0 0x7f8f5419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f5b57f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f54105b60 0x7f8f5419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f5b57f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5419d2f0 con 0x7f8f540ffad0 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f5b57f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f541969b0 con 0x7f8f54105b60 2024-04-22T15:38:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f5931b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f540ffad0 0x7f8f5419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:38.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f5931b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f540ffad0 0x7f8f5419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49818/0 (socket says 172.21.15.96:49818) 2024-04-22T15:38:38.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f5931b700 1 -- 172.21.15.96:0/3965232357 learned_addr learned my addr 172.21.15.96:0/3965232357 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:38:38.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.567+0000 7f8f58b1a700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f54105b60 0x7f8f5419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:38.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.568+0000 7f8f5931b700 1 -- 172.21.15.96:0/3965232357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f54105b60 msgr2=0x7f8f5419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.568+0000 7f8f5931b700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f54105b60 0x7f8f5419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.568+0000 7f8f5931b700 1 -- 172.21.15.96:0/3965232357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f440095f0 con 0x7f8f540ffad0 2024-04-22T15:38:38.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.568+0000 7f8f5931b700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f540ffad0 0x7f8f5419c820 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8f44006010 tx=0x7f8f4400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:38.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.568+0000 7f8f4a7fc700 1 -- 172.21.15.96:0/3965232357 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f4401d070 con 0x7f8f540ffad0 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.568+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f54196c30 con 0x7f8f540ffad0 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.568+0000 7f8f4a7fc700 1 -- 172.21.15.96:0/3965232357 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f4400f460 con 0x7f8f540ffad0 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.568+0000 7f8f4a7fc700 1 -- 172.21.15.96:0/3965232357 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f44017630 con 0x7f8f540ffad0 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.568+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f54197120 con 0x7f8f540ffad0 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.569+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f541080d0 con 0x7f8f540ffad0 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.569+0000 7f8f4a7fc700 1 -- 172.21.15.96:0/3965232357 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8f4400f5d0 con 0x7f8f540ffad0 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.570+0000 7f8f4a7fc700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8f4006ecc0 0x7f8f40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.570+0000 7f8f4a7fc700 1 -- 172.21.15.96:0/3965232357 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8f440903a0 con 0x7f8f540ffad0 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.570+0000 7f8f58b1a700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8f4006ecc0 0x7f8f40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:38:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.570+0000 7f8f58b1a700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8f4006ecc0 0x7f8f40071180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8f50005ea0 tx=0x7f8f50005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:38:38.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.575+0000 7f8f4a7fc700 1 -- 172.21.15.96:0/3965232357 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8f4405ccf0 con 0x7f8f540ffad0 2024-04-22T15:38:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:38 smithi096 ceph-mon[112443]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:38 smithi096 ceph-mon[112443]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:38 smithi096 ceph-mon[112443]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:38 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1737704947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:38:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:38 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2910389748' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:38:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:38:38.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.866+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8f541011d0 con 0x7f8f540ffad0 2024-04-22T15:38:38.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.866+0000 7f8f4a7fc700 1 -- 172.21.15.96:0/3965232357 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f8f541011d0 con 0x7f8f540ffad0 2024-04-22T15:38:38.870 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:38:38.870 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:38:38.870 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:38:38.870 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:38:38.870 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:38:38.870 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:38:38.870 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8f4006ecc0 msgr2=0x7f8f40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8f4006ecc0 0x7f8f40071180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8f50005ea0 tx=0x7f8f50005e30 comp rx=0 tx=0).stop 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f540ffad0 msgr2=0x7f8f5419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f540ffad0 0x7f8f5419c820 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8f44006010 tx=0x7f8f4400be20 comp rx=0 tx=0).stop 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 shutdown_connections 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8f4006ecc0 0x7f8f40071180 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f540ffad0 0x7f8f5419c820 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 --2- 172.21.15.96:0/3965232357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f54105b60 0x7f8f5419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 >> 172.21.15.96:0/3965232357 conn(0x7f8f540fb5e0 msgr2=0x7f8f540fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:38:38.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 shutdown_connections 2024-04-22T15:38:38.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:38:38.869+0000 7f8f5b57f700 1 -- 172.21.15.96:0/3965232357 wait complete. 2024-04-22T15:38:38.911 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:38 smithi196 ceph-mon[81815]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:38.911 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:38 smithi196 ceph-mon[81815]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:38.911 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:38 smithi196 ceph-mon[81815]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:38.911 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:38 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1737704947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:38:38.911 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:38 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2910389748' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:38:38.911 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:38:39.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:39 smithi096 ceph-mon[112443]: pgmap v503: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:39.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:39 smithi096 ceph-mon[112443]: from='client.34976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:39.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:39 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3965232357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:38:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:39 smithi196 ceph-mon[81815]: pgmap v503: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:39 smithi196 ceph-mon[81815]: from='client.34976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:38:40.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:39 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3965232357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:38:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:41 smithi096 ceph-mon[112443]: pgmap v504: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:41 smithi196 ceph-mon[81815]: pgmap v504: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:43 smithi096 ceph-mon[112443]: pgmap v505: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:43 smithi196 ceph-mon[81815]: pgmap v505: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:45.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:45 smithi096 ceph-mon[112443]: pgmap v506: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:45 smithi196 ceph-mon[81815]: pgmap v506: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:47 smithi096 ceph-mon[112443]: pgmap v507: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:47 smithi196 ceph-mon[81815]: pgmap v507: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:49.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:49 smithi096 ceph-mon[112443]: pgmap v508: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:49 smithi196 ceph-mon[81815]: pgmap v508: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:51.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:51 smithi096 ceph-mon[112443]: pgmap v509: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:51 smithi196 ceph-mon[81815]: pgmap v509: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:53.813 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:53 smithi096 ceph-mon[112443]: pgmap v510: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:53.813 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:38:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:53 smithi196 ceph-mon[81815]: pgmap v510: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:38:55.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:55 smithi096 ceph-mon[112443]: pgmap v511: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:55 smithi196 ceph-mon[81815]: pgmap v511: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:57.900 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:57 smithi196 ceph-mon[81815]: pgmap v512: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:57 smithi096 ceph-mon[112443]: pgmap v512: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:58.866 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:38:58 smithi096 ceph-mon[112443]: pgmap v513: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:38:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:38:58 smithi196 ceph-mon[81815]: pgmap v513: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:01.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:01 smithi196 ceph-mon[81815]: pgmap v514: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:01.938 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:01 smithi096 ceph-mon[112443]: pgmap v514: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:03.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:03 smithi196 ceph-mon[81815]: pgmap v515: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:03.919 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:03 smithi096 ceph-mon[112443]: pgmap v515: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:05 smithi196 ceph-mon[81815]: pgmap v516: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:05 smithi096 ceph-mon[112443]: pgmap v516: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:07 smithi096 ceph-mon[112443]: pgmap v517: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:08.007 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:07 smithi196 ceph-mon[81815]: pgmap v517: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:08.973 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:39:08.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.988+0000 7f75b1b36700 1 -- 172.21.15.96:0/606806655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 msgr2=0x7f75ac105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.988+0000 7f75b1b36700 1 --2- 172.21.15.96:0/606806655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 0x7f75ac105e80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f759c009910 tx=0x7f759c009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.988+0000 7f75b1b36700 1 -- 172.21.15.96:0/606806655 shutdown_connections 2024-04-22T15:39:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.988+0000 7f75b1b36700 1 --2- 172.21.15.96:0/606806655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ac0ffaa0 0x7f75ac0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.988+0000 7f75b1b36700 1 --2- 172.21.15.96:0/606806655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 0x7f75ac105e80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.988+0000 7f75b1b36700 1 -- 172.21.15.96:0/606806655 >> 172.21.15.96:0/606806655 conn(0x7f75ac076740 msgr2=0x7f75ac0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.988+0000 7f75b1b36700 1 -- 172.21.15.96:0/606806655 shutdown_connections 2024-04-22T15:39:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.988+0000 7f75b1b36700 1 -- 172.21.15.96:0/606806655 wait complete. 2024-04-22T15:39:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75b1b36700 1 Processor -- start 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75b1b36700 1 -- start start 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75b1b36700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ac0ffaa0 0x7f75ac1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75b1b36700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 0x7f75ac197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75b1b36700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac1984d0 con 0x7f75ac105aa0 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75b1b36700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac198610 con 0x7f75ac0ffaa0 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75aaffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 0x7f75ac197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75aaffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 0x7f75ac197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46934/0 (socket says 172.21.15.96:46934) 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75aaffd700 1 -- 172.21.15.96:0/2804098353 learned_addr learned my addr 172.21.15.96:0/2804098353 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.989+0000 7f75ab7fe700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ac0ffaa0 0x7f75ac1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.990+0000 7f75aaffd700 1 -- 172.21.15.96:0/2804098353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ac0ffaa0 msgr2=0x7f75ac1979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.990+0000 7f75aaffd700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ac0ffaa0 0x7f75ac1979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.990+0000 7f75aaffd700 1 -- 172.21.15.96:0/2804098353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c0095f0 con 0x7f75ac105aa0 2024-04-22T15:39:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.990+0000 7f75aaffd700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 0x7f75ac197f00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75a000d8d0 tx=0x7f75a000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.990+0000 7f75a8ff9700 1 -- 172.21.15.96:0/2804098353 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75a00098a0 con 0x7f75ac105aa0 2024-04-22T15:39:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.990+0000 7f75a8ff9700 1 -- 172.21.15.96:0/2804098353 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75a0010460 con 0x7f75ac105aa0 2024-04-22T15:39:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.990+0000 7f75a8ff9700 1 -- 172.21.15.96:0/2804098353 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75a000f5f0 con 0x7f75ac105aa0 2024-04-22T15:39:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.990+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac1a0a30 con 0x7f75ac105aa0 2024-04-22T15:39:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.990+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac1a0e40 con 0x7f75ac105aa0 2024-04-22T15:39:08.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.991+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ac1080a0 con 0x7f75ac105aa0 2024-04-22T15:39:08.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.992+0000 7f75a8ff9700 1 -- 172.21.15.96:0/2804098353 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f75a0009a00 con 0x7f75ac105aa0 2024-04-22T15:39:08.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.992+0000 7f75a8ff9700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f759406ecc0 0x7f7594071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:08.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.993+0000 7f75a8ff9700 1 -- 172.21.15.96:0/2804098353 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f75a008ecc0 con 0x7f75ac105aa0 2024-04-22T15:39:08.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.993+0000 7f75ab7fe700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f759406ecc0 0x7f7594071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:08.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.993+0000 7f75ab7fe700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f759406ecc0 0x7f7594071180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f759c000c00 tx=0x7f759c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:09.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:08.997+0000 7f75a8ff9700 1 -- 172.21.15.96:0/2804098353 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f75a0057aa0 con 0x7f75ac105aa0 2024-04-22T15:39:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:39:09.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.218+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75ac069650 con 0x7f759406ecc0 2024-04-22T15:39:09.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.219+0000 7f75a8ff9700 1 -- 172.21.15.96:0/2804098353 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f759c01e040 con 0x7f759406ecc0 2024-04-22T15:39:09.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f759406ecc0 msgr2=0x7f7594071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f759406ecc0 0x7f7594071180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f759c000c00 tx=0x7f759c01a040 comp rx=0 tx=0).stop 2024-04-22T15:39:09.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 msgr2=0x7f75ac197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 0x7f75ac197f00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75a000d8d0 tx=0x7f75a000dbe0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 shutdown_connections 2024-04-22T15:39:09.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f759406ecc0 0x7f7594071180 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ac0ffaa0 0x7f75ac1979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 --2- 172.21.15.96:0/2804098353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac105aa0 0x7f75ac197f00 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 >> 172.21.15.96:0/2804098353 conn(0x7f75ac076740 msgr2=0x7f75ac0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:09.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 shutdown_connections 2024-04-22T15:39:09.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.222+0000 7f75b1b36700 1 -- 172.21.15.96:0/2804098353 wait complete. 2024-04-22T15:39:09.236 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.327+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/1040657278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d941017c0 msgr2=0x7f7d94105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.327+0000 7f7d9c2ec700 1 --2- 172.21.15.96:0/1040657278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d941017c0 0x7f7d94105810 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009910 tx=0x7f7d84009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.327+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/1040657278 shutdown_connections 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.327+0000 7f7d9c2ec700 1 --2- 172.21.15.96:0/1040657278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d941017c0 0x7f7d94105810 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.327+0000 7f7d9c2ec700 1 --2- 172.21.15.96:0/1040657278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d94100e10 0x7f7d941011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.327+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/1040657278 >> 172.21.15.96:0/1040657278 conn(0x7f7d940fc6e0 msgr2=0x7f7d940feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.327+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/1040657278 shutdown_connections 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.327+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/1040657278 wait complete. 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9c2ec700 1 Processor -- start 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9c2ec700 1 -- start start 2024-04-22T15:39:09.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9c2ec700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d94100e10 0x7f7d9419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9c2ec700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d941017c0 0x7f7d9419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9c2ec700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9419d380 con 0x7f7d94100e10 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9c2ec700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d941969b0 con 0x7f7d941017c0 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9a088700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d94100e10 0x7f7d9419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9a088700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d94100e10 0x7f7d9419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46960/0 (socket says 172.21.15.96:46960) 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9a088700 1 -- 172.21.15.96:0/2275838008 learned_addr learned my addr 172.21.15.96:0/2275838008 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d99887700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d941017c0 0x7f7d9419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9a088700 1 -- 172.21.15.96:0/2275838008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d941017c0 msgr2=0x7f7d9419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9a088700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d941017c0 0x7f7d9419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9a088700 1 -- 172.21.15.96:0/2275838008 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d840095f0 con 0x7f7d94100e10 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.328+0000 7f7d9a088700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d94100e10 0x7f7d9419c820 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00d8d0 tx=0x7f7d8c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.329+0000 7f7d8b7fe700 1 -- 172.21.15.96:0/2275838008 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8c009940 con 0x7f7d94100e10 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.329+0000 7f7d8b7fe700 1 -- 172.21.15.96:0/2275838008 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d8c010460 con 0x7f7d94100e10 2024-04-22T15:39:09.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.329+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d94196c90 con 0x7f7d94100e10 2024-04-22T15:39:09.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.329+0000 7f7d8b7fe700 1 -- 172.21.15.96:0/2275838008 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8c00f5f0 con 0x7f7d94100e10 2024-04-22T15:39:09.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.329+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d941971e0 con 0x7f7d94100e10 2024-04-22T15:39:09.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.330+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d94109210 con 0x7f7d94100e10 2024-04-22T15:39:09.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.330+0000 7f7d8b7fe700 1 -- 172.21.15.96:0/2275838008 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7d8c00f820 con 0x7f7d94100e10 2024-04-22T15:39:09.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.330+0000 7f7d8b7fe700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8006ecc0 0x7f7d80071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:09.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.331+0000 7f7d99887700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8006ecc0 0x7f7d80071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:09.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.331+0000 7f7d8b7fe700 1 -- 172.21.15.96:0/2275838008 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7d8c08ee20 con 0x7f7d94100e10 2024-04-22T15:39:09.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.331+0000 7f7d99887700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8006ecc0 0x7f7d80071180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7d94197b20 tx=0x7f7d8401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:09.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.335+0000 7f7d8b7fe700 1 -- 172.21.15.96:0/2275838008 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7d8c05b7a0 con 0x7f7d94100e10 2024-04-22T15:39:09.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.555+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d94197e90 con 0x7f7d8006ecc0 2024-04-22T15:39:09.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.556+0000 7f7d8b7fe700 1 -- 172.21.15.96:0/2275838008 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7d94197e90 con 0x7f7d8006ecc0 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.558+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8006ecc0 msgr2=0x7f7d80071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.558+0000 7f7d9c2ec700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8006ecc0 0x7f7d80071180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7d94197b20 tx=0x7f7d8401a040 comp rx=0 tx=0).stop 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.558+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d94100e10 msgr2=0x7f7d9419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.558+0000 7f7d9c2ec700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d94100e10 0x7f7d9419c820 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00d8d0 tx=0x7f7d8c00dc90 comp rx=0 tx=0).stop 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.558+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 shutdown_connections 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.558+0000 7f7d9c2ec700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7d8006ecc0 0x7f7d80071180 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.558+0000 7f7d9c2ec700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d94100e10 0x7f7d9419c820 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.558+0000 7f7d9c2ec700 1 --2- 172.21.15.96:0/2275838008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d941017c0 0x7f7d9419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.558+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 >> 172.21.15.96:0/2275838008 conn(0x7f7d940fc6e0 msgr2=0x7f7d940fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:09.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.559+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 shutdown_connections 2024-04-22T15:39:09.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.559+0000 7f7d9c2ec700 1 -- 172.21.15.96:0/2275838008 wait complete. 2024-04-22T15:39:09.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.662+0000 7f44db240700 1 -- 172.21.15.96:0/2048934539 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 msgr2=0x7f44d4105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.662+0000 7f44db240700 1 --2- 172.21.15.96:0/2048934539 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 0x7f44d4105e90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009910 tx=0x7f44c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:09.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.662+0000 7f44db240700 1 -- 172.21.15.96:0/2048934539 shutdown_connections 2024-04-22T15:39:09.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.662+0000 7f44db240700 1 --2- 172.21.15.96:0/2048934539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44d40ffab0 0x7f44d40fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.662+0000 7f44db240700 1 --2- 172.21.15.96:0/2048934539 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 0x7f44d4105e90 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.662+0000 7f44db240700 1 -- 172.21.15.96:0/2048934539 >> 172.21.15.96:0/2048934539 conn(0x7f44d40fb5e0 msgr2=0x7f44d40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:09.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.662+0000 7f44db240700 1 -- 172.21.15.96:0/2048934539 shutdown_connections 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.662+0000 7f44db240700 1 -- 172.21.15.96:0/2048934539 wait complete. 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44db240700 1 Processor -- start 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44db240700 1 -- start start 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44db240700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44d40ffab0 0x7f44d40fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44db240700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 0x7f44d40fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44db240700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40fc920 con 0x7f44d4105ab0 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44db240700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40fca60 con 0x7f44d40ffab0 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44d3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 0x7f44d40fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44d3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 0x7f44d40fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46976/0 (socket says 172.21.15.96:46976) 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44d3fff700 1 -- 172.21.15.96:0/3596514757 learned_addr learned my addr 172.21.15.96:0/3596514757 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:09.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44d3fff700 1 -- 172.21.15.96:0/3596514757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44d40ffab0 msgr2=0x7f44d40fbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44d3fff700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44d40ffab0 0x7f44d40fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.663+0000 7f44d3fff700 1 -- 172.21.15.96:0/3596514757 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c80095f0 con 0x7f44d4105ab0 2024-04-22T15:39:09.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.664+0000 7f44d3fff700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 0x7f44d40fc3e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f44c000b4b0 tx=0x7f44c000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:09.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.664+0000 7f44d1ffb700 1 -- 172.21.15.96:0/3596514757 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44c0011800 con 0x7f44d4105ab0 2024-04-22T15:39:09.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.664+0000 7f44d1ffb700 1 -- 172.21.15.96:0/3596514757 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44c0011e20 con 0x7f44d4105ab0 2024-04-22T15:39:09.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.664+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d41a2dd0 con 0x7f44d4105ab0 2024-04-22T15:39:09.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.664+0000 7f44d1ffb700 1 -- 172.21.15.96:0/3596514757 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44c000f2e0 con 0x7f44d4105ab0 2024-04-22T15:39:09.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.664+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d41a3130 con 0x7f44d4105ab0 2024-04-22T15:39:09.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.665+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d41090c0 con 0x7f44d4105ab0 2024-04-22T15:39:09.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.665+0000 7f44d1ffb700 1 -- 172.21.15.96:0/3596514757 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f44c0010380 con 0x7f44d4105ab0 2024-04-22T15:39:09.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.666+0000 7f44d1ffb700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44c406ecc0 0x7f44c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:09.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.666+0000 7f44d1ffb700 1 -- 172.21.15.96:0/3596514757 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f44c008e9d0 con 0x7f44d4105ab0 2024-04-22T15:39:09.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.666+0000 7f44d8fdc700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44c406ecc0 0x7f44c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:09.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.666+0000 7f44d8fdc700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44c406ecc0 0x7f44c4071180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f44c800b5c0 tx=0x7f44c8005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:09.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.670+0000 7f44d1ffb700 1 -- 172.21.15.96:0/3596514757 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f44c0057860 con 0x7f44d4105ab0 2024-04-22T15:39:09.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.880+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f44d40fd3f0 con 0x7f44c406ecc0 2024-04-22T15:39:09.884 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:09 smithi096 ceph-mon[112443]: pgmap v518: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:09.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.891+0000 7f44d1ffb700 1 -- 172.21.15.96:0/3596514757 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f44d40fd3f0 con 0x7f44c406ecc0 2024-04-22T15:39:09.894 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:39:09.894 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (17m) 10m ago 29m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (29m) 10m ago 29m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (28m) 10m ago 28m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (15m) 10m ago 29m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (14m) 10m ago 28m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (17m) 10m ago 28m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (11m) 10m ago 23m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (11m) 10m ago 23m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (10m) 10m ago 23m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (23m) 10m ago 23m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (20m) 10m ago 30m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (19m) 10m ago 28m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (16m) 10m ago 30m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (15m) 10m ago 27m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:39:09.895 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (18m) 10m ago 29m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:39:09.896 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (18m) 10m ago 28m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:39:09.896 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (14m) 10m ago 27m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:39:09.896 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (14m) 10m ago 26m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:39:09.896 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (13m) 10m ago 26m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:39:09.896 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (12m) 10m ago 25m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:39:09.896 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (12m) 10m ago 25m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:39:09.896 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (11m) 10m ago 25m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:39:09.896 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (18m) 10m ago 28m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.893+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44c406ecc0 msgr2=0x7f44c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.893+0000 7f44db240700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44c406ecc0 0x7f44c4071180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f44c800b5c0 tx=0x7f44c8005e20 comp rx=0 tx=0).stop 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.893+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 msgr2=0x7f44d40fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.893+0000 7f44db240700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 0x7f44d40fc3e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f44c000b4b0 tx=0x7f44c000b870 comp rx=0 tx=0).stop 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.893+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 shutdown_connections 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.893+0000 7f44db240700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44c406ecc0 0x7f44c4071180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.893+0000 7f44db240700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44d40ffab0 0x7f44d40fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.893+0000 7f44db240700 1 --2- 172.21.15.96:0/3596514757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44d4105ab0 0x7f44d40fc3e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.893+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 >> 172.21.15.96:0/3596514757 conn(0x7f44d40fb5e0 msgr2=0x7f44d4104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:09.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.894+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 shutdown_connections 2024-04-22T15:39:09.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.894+0000 7f44db240700 1 -- 172.21.15.96:0/3596514757 wait complete. 2024-04-22T15:39:10.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 -- 172.21.15.96:0/3132587594 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 msgr2=0x7f149c10aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 --2- 172.21.15.96:0/3132587594 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 0x7f149c10aa80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1484009910 tx=0x7f1484009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:10.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 -- 172.21.15.96:0/3132587594 shutdown_connections 2024-04-22T15:39:10.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 --2- 172.21.15.96:0/3132587594 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 0x7f149c10aa80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 --2- 172.21.15.96:0/3132587594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f149c101c70 0x7f149c102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 -- 172.21.15.96:0/3132587594 >> 172.21.15.96:0/3132587594 conn(0x7f149c0fb440 msgr2=0x7f149c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 -- 172.21.15.96:0/3132587594 shutdown_connections 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 -- 172.21.15.96:0/3132587594 wait complete. 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 Processor -- start 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.997+0000 7f14a0da0700 1 -- start start 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f14a0da0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f149c101c70 0x7f149c19c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f14a0da0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 0x7f149c19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f14a0da0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c19d210 con 0x7f149c102590 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f14a0da0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c196780 con 0x7f149c101c70 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f1492ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 0x7f149c19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f1492ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 0x7f149c19cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47000/0 (socket says 172.21.15.96:47000) 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f1492ffd700 1 -- 172.21.15.96:0/2103023617 learned_addr learned my addr 172.21.15.96:0/2103023617 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:10.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f149b7fe700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f149c101c70 0x7f149c19c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f1492ffd700 1 -- 172.21.15.96:0/2103023617 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f149c101c70 msgr2=0x7f149c19c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f1492ffd700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f149c101c70 0x7f149c19c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f1492ffd700 1 -- 172.21.15.96:0/2103023617 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14840095f0 con 0x7f149c102590 2024-04-22T15:39:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f1492ffd700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 0x7f149c19cb80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f148400bd70 tx=0x7f148400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f14997fa700 1 -- 172.21.15.96:0/2103023617 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f148401d070 con 0x7f149c102590 2024-04-22T15:39:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f14997fa700 1 -- 172.21.15.96:0/2103023617 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14840056e0 con 0x7f149c102590 2024-04-22T15:39:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149c196a00 con 0x7f149c102590 2024-04-22T15:39:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f14997fa700 1 -- 172.21.15.96:0/2103023617 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f148400fe50 con 0x7f149c102590 2024-04-22T15:39:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.998+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149c196ef0 con 0x7f149c102590 2024-04-22T15:39:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:09.999+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f149c1081e0 con 0x7f149c102590 2024-04-22T15:39:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.000+0000 7f14997fa700 1 -- 172.21.15.96:0/2103023617 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f148400f680 con 0x7f149c102590 2024-04-22T15:39:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.000+0000 7f14997fa700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f148806ecc0 0x7f1488071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.000+0000 7f14997fa700 1 -- 172.21.15.96:0/2103023617 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f14840901c0 con 0x7f149c102590 2024-04-22T15:39:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.001+0000 7f149b7fe700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f148806ecc0 0x7f1488071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.001+0000 7f149b7fe700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f148806ecc0 0x7f1488071180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f148c005950 tx=0x7f148c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:10.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.005+0000 7f14997fa700 1 -- 172.21.15.96:0/2103023617 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f148405ca50 con 0x7f149c102590 2024-04-22T15:39:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:09 smithi196 ceph-mon[81815]: pgmap v518: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.289+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f149c069670 con 0x7f149c102590 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.290+0000 7f14997fa700 1 -- 172.21.15.96:0/2103023617 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f148405c5e0 con 0x7f149c102590 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:39:10.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:39:10.294 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:39:10.294 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:39:10.294 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:39:10.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:39:10.294 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:39:10.294 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:39:10.294 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:39:10.294 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:39:10.294 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.292+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f148806ecc0 msgr2=0x7f1488071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.292+0000 7f14a0da0700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f148806ecc0 0x7f1488071180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f148c005950 tx=0x7f148c00b410 comp rx=0 tx=0).stop 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.292+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 msgr2=0x7f149c19cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.292+0000 7f14a0da0700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 0x7f149c19cb80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f148400bd70 tx=0x7f148400bda0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.292+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 shutdown_connections 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.292+0000 7f14a0da0700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f148806ecc0 0x7f1488071180 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.292+0000 7f14a0da0700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f149c101c70 0x7f149c19c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.292+0000 7f14a0da0700 1 --2- 172.21.15.96:0/2103023617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f149c102590 0x7f149c19cb80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.292+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 >> 172.21.15.96:0/2103023617 conn(0x7f149c0fb440 msgr2=0x7f149c105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.293+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 shutdown_connections 2024-04-22T15:39:10.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.293+0000 7f14a0da0700 1 -- 172.21.15.96:0/2103023617 wait complete. 2024-04-22T15:39:10.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.396+0000 7f2b2bf69700 1 -- 172.21.15.96:0/2394224137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24104b30 msgr2=0x7f2b24104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.396+0000 7f2b2bf69700 1 --2- 172.21.15.96:0/2394224137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24104b30 0x7f2b24104f10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009910 tx=0x7f2b14009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:10.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.396+0000 7f2b2bf69700 1 -- 172.21.15.96:0/2394224137 shutdown_connections 2024-04-22T15:39:10.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.396+0000 7f2b2bf69700 1 --2- 172.21.15.96:0/2394224137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b24100220 0x7f2b241006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.396+0000 7f2b2bf69700 1 --2- 172.21.15.96:0/2394224137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24104b30 0x7f2b24104f10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.396+0000 7f2b2bf69700 1 -- 172.21.15.96:0/2394224137 >> 172.21.15.96:0/2394224137 conn(0x7f2b24075f40 msgr2=0x7f2b240fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:10.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.396+0000 7f2b2bf69700 1 -- 172.21.15.96:0/2394224137 shutdown_connections 2024-04-22T15:39:10.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.396+0000 7f2b2bf69700 1 -- 172.21.15.96:0/2394224137 wait complete. 2024-04-22T15:39:10.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b2bf69700 1 Processor -- start 2024-04-22T15:39:10.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b2bf69700 1 -- start start 2024-04-22T15:39:10.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b2bf69700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24100220 0x7f2b240fb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b2bf69700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b24104b30 0x7f2b240fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b2bf69700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240fbe00 con 0x7f2b24100220 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b2bf69700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240fbf40 con 0x7f2b24104b30 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b29d05700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24100220 0x7f2b240fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b29d05700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24100220 0x7f2b240fb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47028/0 (socket says 172.21.15.96:47028) 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b29d05700 1 -- 172.21.15.96:0/286370880 learned_addr learned my addr 172.21.15.96:0/286370880 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b29d05700 1 -- 172.21.15.96:0/286370880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b24104b30 msgr2=0x7f2b240fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b29d05700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b24104b30 0x7f2b240fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b29d05700 1 -- 172.21.15.96:0/286370880 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140095f0 con 0x7f2b24100220 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b29d05700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24100220 0x7f2b240fb380 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400bd70 tx=0x7f2b1400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b1affd700 1 -- 172.21.15.96:0/286370880 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b1401d070 con 0x7f2b24100220 2024-04-22T15:39:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.397+0000 7f2b1affd700 1 -- 172.21.15.96:0/286370880 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b1400f460 con 0x7f2b24100220 2024-04-22T15:39:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.398+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b240fc1c0 con 0x7f2b24100220 2024-04-22T15:39:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.398+0000 7f2b1affd700 1 -- 172.21.15.96:0/286370880 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b14017630 con 0x7f2b24100220 2024-04-22T15:39:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.398+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b241a2fd0 con 0x7f2b24100220 2024-04-22T15:39:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.398+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b24101440 con 0x7f2b24100220 2024-04-22T15:39:10.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.399+0000 7f2b1affd700 1 -- 172.21.15.96:0/286370880 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2b1400f5d0 con 0x7f2b24100220 2024-04-22T15:39:10.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.399+0000 7f2b1affd700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2b1006ecc0 0x7f2b10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:10.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.399+0000 7f2b1affd700 1 -- 172.21.15.96:0/286370880 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2b140903a0 con 0x7f2b24100220 2024-04-22T15:39:10.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.399+0000 7f2b29504700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2b1006ecc0 0x7f2b10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:10.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.400+0000 7f2b29504700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2b1006ecc0 0x7f2b10071180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2b240fca60 tx=0x7f2b20006d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:10.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.404+0000 7f2b1affd700 1 -- 172.21.15.96:0/286370880 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2b14059100 con 0x7f2b24100220 2024-04-22T15:39:10.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.651+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2b2410ce70 con 0x7f2b24100220 2024-04-22T15:39:10.654 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:10 smithi096 ceph-mon[112443]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:10.654 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:10 smithi096 ceph-mon[112443]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:10.654 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:10 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2103023617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:39:10.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.651+0000 7f2b1affd700 1 -- 172.21.15.96:0/286370880 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f2b1405c7c0 con 0x7f2b24100220 2024-04-22T15:39:10.655 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:39:10.655 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:39:10.655 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:39:10.655 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:39:10.655 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:39:10.655 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:39:10.655 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:39:10.655 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:39:10.656 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:39:10.657 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:39:10.658 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:39:10.658 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:39:10.658 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:39:10.658 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2b1006ecc0 msgr2=0x7f2b10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2b1006ecc0 0x7f2b10071180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2b240fca60 tx=0x7f2b20006d20 comp rx=0 tx=0).stop 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24100220 msgr2=0x7f2b240fb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24100220 0x7f2b240fb380 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400bd70 tx=0x7f2b1400be50 comp rx=0 tx=0).stop 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 shutdown_connections 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2b1006ecc0 0x7f2b10071180 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b24100220 0x7f2b240fb380 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 --2- 172.21.15.96:0/286370880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2b24104b30 0x7f2b240fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 >> 172.21.15.96:0/286370880 conn(0x7f2b24075f40 msgr2=0x7f2b240fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 shutdown_connections 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.654+0000 7f2b2bf69700 1 -- 172.21.15.96:0/286370880 wait complete. 2024-04-22T15:39:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.759+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/2415560065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64100e20 msgr2=0x7f0a64101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.759+0000 7f0a6b3a5700 1 --2- 172.21.15.96:0/2415560065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64100e20 0x7f0a64101200 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009910 tx=0x7f0a54009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.759+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/2415560065 shutdown_connections 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.759+0000 7f0a6b3a5700 1 --2- 172.21.15.96:0/2415560065 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a641017d0 0x7f0a64105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.759+0000 7f0a6b3a5700 1 --2- 172.21.15.96:0/2415560065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64100e20 0x7f0a64101200 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.759+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/2415560065 >> 172.21.15.96:0/2415560065 conn(0x7f0a640fc6d0 msgr2=0x7f0a640feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.759+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/2415560065 shutdown_connections 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.759+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/2415560065 wait complete. 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a6b3a5700 1 Processor -- start 2024-04-22T15:39:10.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a6b3a5700 1 -- start start 2024-04-22T15:39:10.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a6b3a5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a64100e20 0x7f0a6419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:10.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a6b3a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a641017d0 0x7f0a6419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a6b3a5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6419d3c0 con 0x7f0a641017d0 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a6b3a5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a641969f0 con 0x7f0a64100e20 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a68940700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a641017d0 0x7f0a6419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a68940700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a641017d0 0x7f0a6419cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41246/0 (socket says 172.21.15.96:41246) 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a68940700 1 -- 172.21.15.96:0/1470102377 learned_addr learned my addr 172.21.15.96:0/1470102377 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.760+0000 7f0a69141700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a64100e20 0x7f0a6419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.761+0000 7f0a68940700 1 -- 172.21.15.96:0/1470102377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a64100e20 msgr2=0x7f0a6419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.761+0000 7f0a68940700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a64100e20 0x7f0a6419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.761+0000 7f0a68940700 1 -- 172.21.15.96:0/1470102377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540095f0 con 0x7f0a641017d0 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.761+0000 7f0a68940700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a641017d0 0x7f0a6419cda0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000c8f0 tx=0x7f0a6000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.761+0000 7f0a5a7fc700 1 -- 172.21.15.96:0/1470102377 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a60007ad0 con 0x7f0a641017d0 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.761+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a64196cd0 con 0x7f0a641017d0 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.761+0000 7f0a5a7fc700 1 -- 172.21.15.96:0/1470102377 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a6000ce90 con 0x7f0a641017d0 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.761+0000 7f0a5a7fc700 1 -- 172.21.15.96:0/1470102377 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a60005600 con 0x7f0a641017d0 2024-04-22T15:39:10.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.761+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a64197220 con 0x7f0a641017d0 2024-04-22T15:39:10.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.762+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a641091c0 con 0x7f0a641017d0 2024-04-22T15:39:10.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.762+0000 7f0a5a7fc700 1 -- 172.21.15.96:0/1470102377 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0a60004750 con 0x7f0a641017d0 2024-04-22T15:39:10.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.763+0000 7f0a5a7fc700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a5006ecc0 0x7f0a50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:10.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.763+0000 7f0a5a7fc700 1 -- 172.21.15.96:0/1470102377 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0a6008ebb0 con 0x7f0a641017d0 2024-04-22T15:39:10.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.763+0000 7f0a69141700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a5006ecc0 0x7f0a50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:10.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.763+0000 7f0a69141700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a5006ecc0 0x7f0a50071180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b5c0 tx=0x7f0a5401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:10.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.768+0000 7f0a5a7fc700 1 -- 172.21.15.96:0/1470102377 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0a60095080 con 0x7f0a641017d0 2024-04-22T15:39:10.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.986+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a64197ed0 con 0x7f0a5006ecc0 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.988+0000 7f0a5a7fc700 1 -- 172.21.15.96:0/1470102377 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0a64197ed0 con 0x7f0a5006ecc0 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:39:10.991 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:39:10.992 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:39:10.992 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:39:10.992 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:39:10.992 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:39:10.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a5006ecc0 msgr2=0x7f0a50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a5006ecc0 0x7f0a50071180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b5c0 tx=0x7f0a5401a040 comp rx=0 tx=0).stop 2024-04-22T15:39:10.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a641017d0 msgr2=0x7f0a6419cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:10.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a641017d0 0x7f0a6419cda0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000c8f0 tx=0x7f0a6000ccb0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 shutdown_connections 2024-04-22T15:39:10.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0a5006ecc0 0x7f0a50071180 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a64100e20 0x7f0a6419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 --2- 172.21.15.96:0/1470102377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a641017d0 0x7f0a6419cda0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:10.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 >> 172.21.15.96:0/1470102377 conn(0x7f0a640fc6d0 msgr2=0x7f0a640fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:10.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 shutdown_connections 2024-04-22T15:39:10.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:10.990+0000 7f0a6b3a5700 1 -- 172.21.15.96:0/1470102377 wait complete. 2024-04-22T15:39:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:10 smithi196 ceph-mon[81815]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:11.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:10 smithi196 ceph-mon[81815]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:11.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:10 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2103023617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:39:11.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.093+0000 7fb974d3f700 1 -- 172.21.15.96:0/3808783440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 msgr2=0x7fb97010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:11.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.093+0000 7fb974d3f700 1 --2- 172.21.15.96:0/3808783440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 0x7fb97010ab70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb958009910 tx=0x7fb958009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.093+0000 7fb974d3f700 1 -- 172.21.15.96:0/3808783440 shutdown_connections 2024-04-22T15:39:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.093+0000 7fb974d3f700 1 --2- 172.21.15.96:0/3808783440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 0x7fb97010ab70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.093+0000 7fb974d3f700 1 --2- 172.21.15.96:0/3808783440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb970101d60 0x7fb970102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.093+0000 7fb974d3f700 1 -- 172.21.15.96:0/3808783440 >> 172.21.15.96:0/3808783440 conn(0x7fb970076740 msgr2=0x7fb9700fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.093+0000 7fb974d3f700 1 -- 172.21.15.96:0/3808783440 shutdown_connections 2024-04-22T15:39:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.093+0000 7fb974d3f700 1 -- 172.21.15.96:0/3808783440 wait complete. 2024-04-22T15:39:11.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb974d3f700 1 Processor -- start 2024-04-22T15:39:11.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb974d3f700 1 -- start start 2024-04-22T15:39:11.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb974d3f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb970101d60 0x7fb9700ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb974d3f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 0x7fb9700fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb974d3f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb970100440 con 0x7fb970102680 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb974d3f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb970100580 con 0x7fb970101d60 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb965d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 0x7fb9700fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb965d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 0x7fb9700fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41258/0 (socket says 172.21.15.96:41258) 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb965d9b700 1 -- 172.21.15.96:0/2282924908 learned_addr learned my addr 172.21.15.96:0/2282924908 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb96e59c700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb970101d60 0x7fb9700ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb965d9b700 1 -- 172.21.15.96:0/2282924908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb970101d60 msgr2=0x7fb9700ff9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.094+0000 7fb965d9b700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb970101d60 0x7fb9700ff9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.095+0000 7fb965d9b700 1 -- 172.21.15.96:0/2282924908 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9580095f0 con 0x7fb970102680 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.095+0000 7fb965d9b700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 0x7fb9700fff00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb95800ba70 tx=0x7fb95800baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.095+0000 7fb967fff700 1 -- 172.21.15.96:0/2282924908 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95801d070 con 0x7fb970102680 2024-04-22T15:39:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.095+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb970100800 con 0x7fb970102680 2024-04-22T15:39:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.095+0000 7fb967fff700 1 -- 172.21.15.96:0/2282924908 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb958003d10 con 0x7fb970102680 2024-04-22T15:39:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.095+0000 7fb967fff700 1 -- 172.21.15.96:0/2282924908 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95800fe10 con 0x7fb970102680 2024-04-22T15:39:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.095+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9701a2f60 con 0x7fb970102680 2024-04-22T15:39:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.096+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb970068ea0 con 0x7fb970102680 2024-04-22T15:39:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.097+0000 7fb967fff700 1 -- 172.21.15.96:0/2282924908 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb95800f680 con 0x7fb970102680 2024-04-22T15:39:11.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.097+0000 7fb967fff700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb95c06ecc0 0x7fb95c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:11.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.097+0000 7fb967fff700 1 -- 172.21.15.96:0/2282924908 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb958090100 con 0x7fb970102680 2024-04-22T15:39:11.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.097+0000 7fb96e59c700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb95c06ecc0 0x7fb95c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:11.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.097+0000 7fb96e59c700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb95c06ecc0 0x7fb95c071180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb960005950 tx=0x7fb96000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.102+0000 7fb967fff700 1 -- 172.21.15.96:0/2282924908 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb958058f20 con 0x7fb970102680 2024-04-22T15:39:11.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.385+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb97004ebb0 con 0x7fb970102680 2024-04-22T15:39:11.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.385+0000 7fb967fff700 1 -- 172.21.15.96:0/2282924908 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fb95805c5e0 con 0x7fb970102680 2024-04-22T15:39:11.388 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:39:11.388 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:39:11.388 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:39:11.388 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:39:11.389 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:39:11.389 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:39:11.389 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.387+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb95c06ecc0 msgr2=0x7fb95c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.387+0000 7fb974d3f700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb95c06ecc0 0x7fb95c071180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb960005950 tx=0x7fb96000b410 comp rx=0 tx=0).stop 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.387+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 msgr2=0x7fb9700fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.387+0000 7fb974d3f700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 0x7fb9700fff00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb95800ba70 tx=0x7fb95800baa0 comp rx=0 tx=0).stop 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.388+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 shutdown_connections 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.388+0000 7fb974d3f700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb95c06ecc0 0x7fb95c071180 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.388+0000 7fb974d3f700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb970101d60 0x7fb9700ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.388+0000 7fb974d3f700 1 --2- 172.21.15.96:0/2282924908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb970102680 0x7fb9700fff00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.388+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 >> 172.21.15.96:0/2282924908 conn(0x7fb970076740 msgr2=0x7fb970103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.388+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 shutdown_connections 2024-04-22T15:39:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:11.388+0000 7fb974d3f700 1 -- 172.21.15.96:0/2282924908 wait complete. 2024-04-22T15:39:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:11 smithi096 ceph-mon[112443]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:11 smithi096 ceph-mon[112443]: pgmap v519: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:11 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/286370880' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:39:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:11 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2282924908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:39:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:11 smithi196 ceph-mon[81815]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:12.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:11 smithi196 ceph-mon[81815]: pgmap v519: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:12.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:11 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/286370880' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:39:12.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:11 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2282924908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:39:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:12 smithi096 ceph-mon[112443]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:12 smithi196 ceph-mon[81815]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:13 smithi096 ceph-mon[112443]: pgmap v520: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:13 smithi196 ceph-mon[81815]: pgmap v520: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:15.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:15 smithi096 ceph-mon[112443]: pgmap v521: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:15 smithi196 ceph-mon[81815]: pgmap v521: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:17.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:17 smithi096 ceph-mon[112443]: pgmap v522: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:17 smithi196 ceph-mon[81815]: pgmap v522: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:19 smithi096 ceph-mon[112443]: pgmap v523: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:19 smithi196 ceph-mon[81815]: pgmap v523: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:21.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:21 smithi096 ceph-mon[112443]: pgmap v524: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:21 smithi196 ceph-mon[81815]: pgmap v524: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:23 smithi096 ceph-mon[112443]: pgmap v525: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:39:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:23 smithi196 ceph-mon[81815]: pgmap v525: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:39:25.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:25 smithi096 ceph-mon[112443]: pgmap v526: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:25 smithi196 ceph-mon[81815]: pgmap v526: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:27.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:27 smithi096 ceph-mon[112443]: pgmap v527: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:27 smithi196 ceph-mon[81815]: pgmap v527: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:29.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:29 smithi096 ceph-mon[112443]: pgmap v528: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:29 smithi196 ceph-mon[81815]: pgmap v528: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:31 smithi096 ceph-mon[112443]: pgmap v529: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:32.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:31 smithi196 ceph-mon[81815]: pgmap v529: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:33.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:33 smithi096 ceph-mon[112443]: pgmap v530: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:33 smithi196 ceph-mon[81815]: pgmap v530: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:35.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:35 smithi096 ceph-mon[112443]: pgmap v531: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:35 smithi196 ceph-mon[81815]: pgmap v531: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:37 smithi096 ceph-mon[112443]: pgmap v532: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:37 smithi196 ceph-mon[81815]: pgmap v532: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:39:38.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:39:39.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:39 smithi096 ceph-mon[112443]: pgmap v533: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:39 smithi196 ceph-mon[81815]: pgmap v533: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:40 smithi096 ceph-mon[112443]: pgmap v534: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:40 smithi196 ceph-mon[81815]: pgmap v534: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:41.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 -- 172.21.15.96:0/3375662648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 msgr2=0x7efdd4105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 --2- 172.21.15.96:0/3375662648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 0x7efdd4105f40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009910 tx=0x7efdd0009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 -- 172.21.15.96:0/3375662648 shutdown_connections 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 --2- 172.21.15.96:0/3375662648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdd40ffad0 0x7efdd40fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 --2- 172.21.15.96:0/3375662648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 0x7efdd4105f40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 -- 172.21.15.96:0/3375662648 >> 172.21.15.96:0/3375662648 conn(0x7efdd40fb5e0 msgr2=0x7efdd40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 -- 172.21.15.96:0/3375662648 shutdown_connections 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 -- 172.21.15.96:0/3375662648 wait complete. 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 Processor -- start 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.505+0000 7efddb781700 1 -- start start 2024-04-22T15:39:41.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efddb781700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdd40ffad0 0x7efdd419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:41.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efddb781700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 0x7efdd419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:41.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efddb781700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd419d450 con 0x7efdd4105b60 2024-04-22T15:39:41.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efddb781700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd4196a30 con 0x7efdd40ffad0 2024-04-22T15:39:41.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdd8d1c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 0x7efdd419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:41.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdd8d1c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 0x7efdd419ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52248/0 (socket says 172.21.15.96:52248) 2024-04-22T15:39:41.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdd8d1c700 1 -- 172.21.15.96:0/922772019 learned_addr learned my addr 172.21.15.96:0/922772019 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:41.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdd951d700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdd40ffad0 0x7efdd419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdd8d1c700 1 -- 172.21.15.96:0/922772019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdd40ffad0 msgr2=0x7efdd419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdd8d1c700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdd40ffad0 0x7efdd419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdd8d1c700 1 -- 172.21.15.96:0/922772019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd00095f0 con 0x7efdd4105b60 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdd951d700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdd40ffad0 0x7efdd419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdd8d1c700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 0x7efdd419ce30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efdc400b4b0 tx=0x7efdc400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.506+0000 7efdca7fc700 1 -- 172.21.15.96:0/922772019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdc4011800 con 0x7efdd4105b60 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.507+0000 7efdca7fc700 1 -- 172.21.15.96:0/922772019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdc4011e20 con 0x7efdd4105b60 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.507+0000 7efdca7fc700 1 -- 172.21.15.96:0/922772019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdc400f2e0 con 0x7efdd4105b60 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.507+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd4196d10 con 0x7efdd4105b60 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.507+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd4197260 con 0x7efdd4105b60 2024-04-22T15:39:41.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.508+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd41090e0 con 0x7efdd4105b60 2024-04-22T15:39:41.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.508+0000 7efdca7fc700 1 -- 172.21.15.96:0/922772019 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efdc4010380 con 0x7efdd4105b60 2024-04-22T15:39:41.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.508+0000 7efdca7fc700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efdc006ecd0 0x7efdc0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:41.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.509+0000 7efdca7fc700 1 -- 172.21.15.96:0/922772019 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7efdc408e9e0 con 0x7efdd4105b60 2024-04-22T15:39:41.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.509+0000 7efdd951d700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efdc006ecd0 0x7efdc0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:41.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.509+0000 7efdd951d700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efdc006ecd0 0x7efdc0071190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efdd000b5c0 tx=0x7efdd001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:41.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.513+0000 7efdca7fc700 1 -- 172.21.15.96:0/922772019 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efdc405b3a0 con 0x7efdd4105b60 2024-04-22T15:39:41.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.731+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efdd4197ef0 con 0x7efdc006ecd0 2024-04-22T15:39:41.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.732+0000 7efdca7fc700 1 -- 172.21.15.96:0/922772019 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efdd4197ef0 con 0x7efdc006ecd0 2024-04-22T15:39:41.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efdc006ecd0 msgr2=0x7efdc0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:41.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efdc006ecd0 0x7efdc0071190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efdd000b5c0 tx=0x7efdd001a040 comp rx=0 tx=0).stop 2024-04-22T15:39:41.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 msgr2=0x7efdd419ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:41.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 0x7efdd419ce30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efdc400b4b0 tx=0x7efdc400b870 comp rx=0 tx=0).stop 2024-04-22T15:39:41.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 shutdown_connections 2024-04-22T15:39:41.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efdc006ecd0 0x7efdc0071190 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:41.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efdd40ffad0 0x7efdd419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:41.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 --2- 172.21.15.96:0/922772019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdd4105b60 0x7efdd419ce30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:41.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 >> 172.21.15.96:0/922772019 conn(0x7efdd40fb5e0 msgr2=0x7efdd4104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:41.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 shutdown_connections 2024-04-22T15:39:41.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.734+0000 7efddb781700 1 -- 172.21.15.96:0/922772019 wait complete. 2024-04-22T15:39:41.748 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:39:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.838+0000 7f1d9b993700 1 -- 172.21.15.96:0/3859050994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 msgr2=0x7f1d94101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.838+0000 7f1d9b993700 1 --2- 172.21.15.96:0/3859050994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 0x7f1d94101200 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1d84009910 tx=0x7f1d84009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.838+0000 7f1d9b993700 1 -- 172.21.15.96:0/3859050994 shutdown_connections 2024-04-22T15:39:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.838+0000 7f1d9b993700 1 --2- 172.21.15.96:0/3859050994 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d941017d0 0x7f1d94105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.838+0000 7f1d9b993700 1 --2- 172.21.15.96:0/3859050994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 0x7f1d94101200 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.838+0000 7f1d9b993700 1 -- 172.21.15.96:0/3859050994 >> 172.21.15.96:0/3859050994 conn(0x7f1d940fc6d0 msgr2=0x7f1d940feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.838+0000 7f1d9b993700 1 -- 172.21.15.96:0/3859050994 shutdown_connections 2024-04-22T15:39:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.838+0000 7f1d9b993700 1 -- 172.21.15.96:0/3859050994 wait complete. 2024-04-22T15:39:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.839+0000 7f1d9b993700 1 Processor -- start 2024-04-22T15:39:41.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.839+0000 7f1d9b993700 1 -- start start 2024-04-22T15:39:41.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.839+0000 7f1d9b993700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 0x7f1d9419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:41.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.839+0000 7f1d9b993700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d941017d0 0x7f1d9419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:41.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.839+0000 7f1d9b993700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9419d3a0 con 0x7f1d94100e20 2024-04-22T15:39:41.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.839+0000 7f1d9b993700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d941969d0 con 0x7f1d941017d0 2024-04-22T15:39:41.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.839+0000 7f1d9972f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 0x7f1d9419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.839+0000 7f1d9972f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 0x7f1d9419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52276/0 (socket says 172.21.15.96:52276) 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.839+0000 7f1d9972f700 1 -- 172.21.15.96:0/2528066273 learned_addr learned my addr 172.21.15.96:0/2528066273 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d98f2e700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d941017d0 0x7f1d9419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d9972f700 1 -- 172.21.15.96:0/2528066273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d941017d0 msgr2=0x7f1d9419cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d9972f700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d941017d0 0x7f1d9419cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d9972f700 1 -- 172.21.15.96:0/2528066273 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d840095f0 con 0x7f1d94100e20 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d98f2e700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d941017d0 0x7f1d9419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d9972f700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 0x7f1d9419c840 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006010 tx=0x7f1d8400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d8a7fc700 1 -- 172.21.15.96:0/2528066273 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d8401d070 con 0x7f1d94100e20 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d94196c50 con 0x7f1d94100e20 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d8a7fc700 1 -- 172.21.15.96:0/2528066273 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d840056e0 con 0x7f1d94100e20 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d8a7fc700 1 -- 172.21.15.96:0/2528066273 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d84017400 con 0x7f1d94100e20 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.840+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d94197140 con 0x7f1d94100e20 2024-04-22T15:39:41.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.841+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d941091c0 con 0x7f1d94100e20 2024-04-22T15:39:41.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.842+0000 7f1d8a7fc700 1 -- 172.21.15.96:0/2528066273 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1d8400f860 con 0x7f1d94100e20 2024-04-22T15:39:41.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.842+0000 7f1d8a7fc700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8006ed10 0x7f1d800711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:41.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.842+0000 7f1d8a7fc700 1 -- 172.21.15.96:0/2528066273 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1d840902d0 con 0x7f1d94100e20 2024-04-22T15:39:41.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.842+0000 7f1d98f2e700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8006ed10 0x7f1d800711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:41.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.843+0000 7f1d98f2e700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8006ed10 0x7f1d800711d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1d90005d90 tx=0x7f1d90005d00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:41.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:41.847+0000 7f1d8a7fc700 1 -- 172.21.15.96:0/2528066273 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1d8405cb80 con 0x7f1d94100e20 2024-04-22T15:39:42.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.077+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d9410ddf0 con 0x7f1d8006ed10 2024-04-22T15:39:42.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.078+0000 7f1d8a7fc700 1 -- 172.21.15.96:0/2528066273 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1d9410ddf0 con 0x7f1d8006ed10 2024-04-22T15:39:42.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8006ed10 msgr2=0x7f1d800711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8006ed10 0x7f1d800711d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1d90005d90 tx=0x7f1d90005d00 comp rx=0 tx=0).stop 2024-04-22T15:39:42.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 msgr2=0x7f1d9419c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 0x7f1d9419c840 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006010 tx=0x7f1d8400be20 comp rx=0 tx=0).stop 2024-04-22T15:39:42.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 shutdown_connections 2024-04-22T15:39:42.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d8006ed10 0x7f1d800711d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d94100e20 0x7f1d9419c840 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 --2- 172.21.15.96:0/2528066273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d941017d0 0x7f1d9419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 >> 172.21.15.96:0/2528066273 conn(0x7f1d940fc6d0 msgr2=0x7f1d940fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:42.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 shutdown_connections 2024-04-22T15:39:42.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.080+0000 7f1d9b993700 1 -- 172.21.15.96:0/2528066273 wait complete. 2024-04-22T15:39:42.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.183+0000 7fb7258b2700 1 -- 172.21.15.96:0/553763647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073200 msgr2=0x7fb7200735e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.183+0000 7fb7258b2700 1 --2- 172.21.15.96:0/553763647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073200 0x7fb7200735e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb710009940 tx=0x7fb710009c50 comp rx=0 tx=0).stop 2024-04-22T15:39:42.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.183+0000 7fb7258b2700 1 -- 172.21.15.96:0/553763647 shutdown_connections 2024-04-22T15:39:42.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.183+0000 7fb7258b2700 1 --2- 172.21.15.96:0/553763647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb720073bb0 0x7fb72010cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.183+0000 7fb7258b2700 1 --2- 172.21.15.96:0/553763647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073200 0x7fb7200735e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.183+0000 7fb7258b2700 1 -- 172.21.15.96:0/553763647 >> 172.21.15.96:0/553763647 conn(0x7fb7200fc700 msgr2=0x7fb7200feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:42.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.183+0000 7fb7258b2700 1 -- 172.21.15.96:0/553763647 shutdown_connections 2024-04-22T15:39:42.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.183+0000 7fb7258b2700 1 -- 172.21.15.96:0/553763647 wait complete. 2024-04-22T15:39:42.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.183+0000 7fb7258b2700 1 Processor -- start 2024-04-22T15:39:42.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7258b2700 1 -- start start 2024-04-22T15:39:42.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7258b2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb720073200 0x7fb720198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7258b2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073bb0 0x7fb7201992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7258b2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7201999a0 con 0x7fb720073bb0 2024-04-22T15:39:42.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7258b2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72019d670 con 0x7fb720073200 2024-04-22T15:39:42.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7167fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073bb0 0x7fb7201992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7167fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073bb0 0x7fb7201992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52292/0 (socket says 172.21.15.96:52292) 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7167fc700 1 -- 172.21.15.96:0/2433754039 learned_addr learned my addr 172.21.15.96:0/2433754039 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb71effd700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb720073200 0x7fb720198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7167fc700 1 -- 172.21.15.96:0/2433754039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb720073200 msgr2=0x7fb720198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7167fc700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb720073200 0x7fb720198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.184+0000 7fb7167fc700 1 -- 172.21.15.96:0/2433754039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7100095f0 con 0x7fb720073bb0 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.185+0000 7fb7167fc700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073bb0 0x7fb7201992c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb70800b4b0 tx=0x7fb70800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.185+0000 7fb7248b0700 1 -- 172.21.15.96:0/2433754039 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb708011800 con 0x7fb720073bb0 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.185+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72019d950 con 0x7fb720073bb0 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.185+0000 7fb7248b0700 1 -- 172.21.15.96:0/2433754039 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb708011e20 con 0x7fb720073bb0 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.185+0000 7fb7248b0700 1 -- 172.21.15.96:0/2433754039 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb70800f2e0 con 0x7fb720073bb0 2024-04-22T15:39:42.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.185+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72019dea0 con 0x7fb720073bb0 2024-04-22T15:39:42.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.186+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb720074940 con 0x7fb720073bb0 2024-04-22T15:39:42.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.187+0000 7fb7248b0700 1 -- 172.21.15.96:0/2433754039 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb708010380 con 0x7fb720073bb0 2024-04-22T15:39:42.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.187+0000 7fb7248b0700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb70c06ecd0 0x7fb70c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.187+0000 7fb71effd700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb70c06ecd0 0x7fb70c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.187+0000 7fb7248b0700 1 -- 172.21.15.96:0/2433754039 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb70808ece0 con 0x7fb720073bb0 2024-04-22T15:39:42.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.188+0000 7fb71effd700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb70c06ecd0 0x7fb70c071190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb710006010 tx=0x7fb71000b7d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:42.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.191+0000 7fb7248b0700 1 -- 172.21.15.96:0/2433754039 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb708057b60 con 0x7fb720073bb0 2024-04-22T15:39:42.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.399+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb720107860 con 0x7fb70c06ecd0 2024-04-22T15:39:42.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.410+0000 7fb7248b0700 1 -- 172.21.15.96:0/2433754039 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb720107860 con 0x7fb70c06ecd0 2024-04-22T15:39:42.413 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:39:42.413 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (18m) 11m ago 29m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:39:42.413 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (30m) 11m ago 30m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:39:42.413 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (28m) 11m ago 28m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:39:42.413 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (15m) 11m ago 29m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:39:42.413 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (15m) 11m ago 28m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:39:42.413 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (18m) 11m ago 29m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:39:42.413 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (11m) 11m ago 23m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (11m) 11m ago 24m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (11m) 11m ago 24m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (23m) 11m ago 23m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (20m) 11m ago 31m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (19m) 11m ago 28m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (16m) 11m ago 31m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (16m) 11m ago 28m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (19m) 11m ago 29m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (19m) 11m ago 28m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (15m) 11m ago 27m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (14m) 11m ago 27m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (13m) 11m ago 26m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:39:42.414 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (13m) 11m ago 26m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:39:42.415 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (12m) 11m ago 25m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:39:42.415 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (12m) 11m ago 25m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:39:42.415 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (18m) 11m ago 29m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:39:42.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb70c06ecd0 msgr2=0x7fb70c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb70c06ecd0 0x7fb70c071190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb710006010 tx=0x7fb71000b7d0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073bb0 msgr2=0x7fb7201992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073bb0 0x7fb7201992c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb70800b4b0 tx=0x7fb70800b7c0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 shutdown_connections 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb70c06ecd0 0x7fb70c071190 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb720073200 0x7fb720198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 --2- 172.21.15.96:0/2433754039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb720073bb0 0x7fb7201992c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 >> 172.21.15.96:0/2433754039 conn(0x7fb7200fc700 msgr2=0x7fb720105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 shutdown_connections 2024-04-22T15:39:42.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.412+0000 7fb7258b2700 1 -- 172.21.15.96:0/2433754039 wait complete. 2024-04-22T15:39:42.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.515+0000 7fa1359ac700 1 -- 172.21.15.96:0/2653075775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130102680 msgr2=0x7fa13010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.515+0000 7fa1359ac700 1 --2- 172.21.15.96:0/2653075775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130102680 0x7fa13010ab70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa118009910 tx=0x7fa118009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:42.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.516+0000 7fa1359ac700 1 -- 172.21.15.96:0/2653075775 shutdown_connections 2024-04-22T15:39:42.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.516+0000 7fa1359ac700 1 --2- 172.21.15.96:0/2653075775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130102680 0x7fa13010ab70 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.516+0000 7fa1359ac700 1 --2- 172.21.15.96:0/2653075775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa130101d60 0x7fa130102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.516+0000 7fa1359ac700 1 -- 172.21.15.96:0/2653075775 >> 172.21.15.96:0/2653075775 conn(0x7fa130076740 msgr2=0x7fa1300fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:42.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.516+0000 7fa1359ac700 1 -- 172.21.15.96:0/2653075775 shutdown_connections 2024-04-22T15:39:42.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.516+0000 7fa1359ac700 1 -- 172.21.15.96:0/2653075775 wait complete. 2024-04-22T15:39:42.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.516+0000 7fa1359ac700 1 Processor -- start 2024-04-22T15:39:42.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.516+0000 7fa1359ac700 1 -- start start 2024-04-22T15:39:42.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa1359ac700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130101d60 0x7fa13019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa1359ac700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa130102680 0x7fa13019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa1359ac700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13019d380 con 0x7fa130101d60 2024-04-22T15:39:42.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa1359ac700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa130196960 con 0x7fa130102680 2024-04-22T15:39:42.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa12effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130101d60 0x7fa13019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa12effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130101d60 0x7fa13019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52314/0 (socket says 172.21.15.96:52314) 2024-04-22T15:39:42.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa12effd700 1 -- 172.21.15.96:0/950546379 learned_addr learned my addr 172.21.15.96:0/950546379 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:42.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa12effd700 1 -- 172.21.15.96:0/950546379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa130102680 msgr2=0x7fa13019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa12effd700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa130102680 0x7fa13019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa12effd700 1 -- 172.21.15.96:0/950546379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1180095f0 con 0x7fa130101d60 2024-04-22T15:39:42.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa12effd700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130101d60 0x7fa13019c820 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa12000d8d0 tx=0x7fa12000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:42.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa1349aa700 1 -- 172.21.15.96:0/950546379 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa120009940 con 0x7fa130101d60 2024-04-22T15:39:42.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa130196c40 con 0x7fa130101d60 2024-04-22T15:39:42.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa1349aa700 1 -- 172.21.15.96:0/950546379 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa120010460 con 0x7fa130101d60 2024-04-22T15:39:42.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa1349aa700 1 -- 172.21.15.96:0/950546379 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa12000f5f0 con 0x7fa130101d60 2024-04-22T15:39:42.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.517+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa130197190 con 0x7fa130101d60 2024-04-22T15:39:42.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.518+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa130105480 con 0x7fa130101d60 2024-04-22T15:39:42.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.519+0000 7fa1349aa700 1 -- 172.21.15.96:0/950546379 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa1200105d0 con 0x7fa130101d60 2024-04-22T15:39:42.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.519+0000 7fa1349aa700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa11c06ecc0 0x7fa11c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.520+0000 7fa1267fc700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa11c06ecc0 0x7fa11c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.520+0000 7fa1349aa700 1 -- 172.21.15.96:0/950546379 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa12008eed0 con 0x7fa130101d60 2024-04-22T15:39:42.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.520+0000 7fa1267fc700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa11c06ecc0 0x7fa11c071180 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa11800b5c0 tx=0x7fa118005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:42.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.524+0000 7fa1349aa700 1 -- 172.21.15.96:0/950546379 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa12005b760 con 0x7fa130101d60 2024-04-22T15:39:42.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.815+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa130068ea0 con 0x7fa130101d60 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.816+0000 7fa1349aa700 1 -- 172.21.15.96:0/950546379 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7fa12005b2f0 con 0x7fa130101d60 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa11c06ecc0 msgr2=0x7fa11c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa11c06ecc0 0x7fa11c071180 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa11800b5c0 tx=0x7fa118005960 comp rx=0 tx=0).stop 2024-04-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130101d60 msgr2=0x7fa13019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130101d60 0x7fa13019c820 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa12000d8d0 tx=0x7fa12000dc90 comp rx=0 tx=0).stop 2024-04-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 shutdown_connections 2024-04-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa11c06ecc0 0x7fa11c071180 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa130101d60 0x7fa13019c820 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 --2- 172.21.15.96:0/950546379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa130102680 0x7fa13019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 >> 172.21.15.96:0/950546379 conn(0x7fa130076740 msgr2=0x7fa130103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:42.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 shutdown_connections 2024-04-22T15:39:42.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.819+0000 7fa1359ac700 1 -- 172.21.15.96:0/950546379 wait complete. 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.921+0000 7f980bf83700 1 -- 172.21.15.96:0/2570852984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9804105b60 msgr2=0x7f9804105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.921+0000 7f980bf83700 1 --2- 172.21.15.96:0/2570852984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9804105b60 0x7f9804105f40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009910 tx=0x7f97f4009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.921+0000 7f980bf83700 1 -- 172.21.15.96:0/2570852984 shutdown_connections 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.921+0000 7f980bf83700 1 --2- 172.21.15.96:0/2570852984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98040ffad0 0x7f98040fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.921+0000 7f980bf83700 1 --2- 172.21.15.96:0/2570852984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9804105b60 0x7f9804105f40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.921+0000 7f980bf83700 1 -- 172.21.15.96:0/2570852984 >> 172.21.15.96:0/2570852984 conn(0x7f98040fb5e0 msgr2=0x7f98040fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.921+0000 7f980bf83700 1 -- 172.21.15.96:0/2570852984 shutdown_connections 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.921+0000 7f980bf83700 1 -- 172.21.15.96:0/2570852984 wait complete. 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f980bf83700 1 Processor -- start 2024-04-22T15:39:42.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f980bf83700 1 -- start start 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f980bf83700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98040ffad0 0x7f98040fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f980bf83700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9804105b60 0x7f98040fc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f980bf83700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98040fc890 con 0x7f98040ffad0 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f980bf83700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98040fc9d0 con 0x7f9804105b60 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f9809d1f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98040ffad0 0x7f98040fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f9809d1f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98040ffad0 0x7f98040fbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52326/0 (socket says 172.21.15.96:52326) 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f9809d1f700 1 -- 172.21.15.96:0/4203275697 learned_addr learned my addr 172.21.15.96:0/4203275697 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f9809d1f700 1 -- 172.21.15.96:0/4203275697 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9804105b60 msgr2=0x7f98040fc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f980951e700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9804105b60 0x7f98040fc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f9809d1f700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9804105b60 0x7f98040fc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.922+0000 7f9809d1f700 1 -- 172.21.15.96:0/4203275697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f40095f0 con 0x7f98040ffad0 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.923+0000 7f9809d1f700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98040ffad0 0x7f98040fbe10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f97f4006010 tx=0x7f97f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.923+0000 7f97faffd700 1 -- 172.21.15.96:0/4203275697 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97f401d070 con 0x7f98040ffad0 2024-04-22T15:39:42.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.923+0000 7f97faffd700 1 -- 172.21.15.96:0/4203275697 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97f40056e0 con 0x7f98040ffad0 2024-04-22T15:39:42.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.923+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98040fcc50 con 0x7f98040ffad0 2024-04-22T15:39:42.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.923+0000 7f97faffd700 1 -- 172.21.15.96:0/4203275697 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97f4017400 con 0x7f98040ffad0 2024-04-22T15:39:42.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.923+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98041a2f70 con 0x7f98040ffad0 2024-04-22T15:39:42.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.924+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98041090e0 con 0x7f98040ffad0 2024-04-22T15:39:42.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.924+0000 7f97faffd700 1 -- 172.21.15.96:0/4203275697 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f97f400f6f0 con 0x7f98040ffad0 2024-04-22T15:39:42.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.925+0000 7f97faffd700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f97f006ecc0 0x7f97f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.925+0000 7f97faffd700 1 -- 172.21.15.96:0/4203275697 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f97f4090190 con 0x7f98040ffad0 2024-04-22T15:39:42.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.925+0000 7f980951e700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f97f006ecc0 0x7f97f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.925+0000 7f980951e700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f97f006ecc0 0x7f97f0071180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9800005950 tx=0x7f98000058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:42.929+0000 7f97faffd700 1 -- 172.21.15.96:0/4203275697 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f97f4058f70 con 0x7f98040ffad0 2024-04-22T15:39:43.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.171+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f98040fd390 con 0x7f98040ffad0 2024-04-22T15:39:43.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.171+0000 7f97faffd700 1 -- 172.21.15.96:0/4203275697 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f97f405c630 con 0x7f98040ffad0 2024-04-22T15:39:43.174 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:39:43.175 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:39:43.176 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:39:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:39:43.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f97f006ecc0 msgr2=0x7f97f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f97f006ecc0 0x7f97f0071180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9800005950 tx=0x7f98000058e0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98040ffad0 msgr2=0x7f98040fbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98040ffad0 0x7f98040fbe10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f97f4006010 tx=0x7f97f400be20 comp rx=0 tx=0).stop 2024-04-22T15:39:43.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 shutdown_connections 2024-04-22T15:39:43.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f97f006ecc0 0x7f97f0071180 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98040ffad0 0x7f98040fbe10 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 --2- 172.21.15.96:0/4203275697 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9804105b60 0x7f98040fc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 >> 172.21.15.96:0/4203275697 conn(0x7f98040fb5e0 msgr2=0x7f9804069610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:43.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 shutdown_connections 2024-04-22T15:39:43.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.173+0000 7f980bf83700 1 -- 172.21.15.96:0/4203275697 wait complete. 2024-04-22T15:39:43.179 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.278+0000 7fd6cc48e700 1 -- 172.21.15.96:0/2747070377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4101df0 msgr2=0x7fd6c41021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.278+0000 7fd6cc48e700 1 --2- 172.21.15.96:0/2747070377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4101df0 0x7fd6c41021d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009910 tx=0x7fd6c0009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.278+0000 7fd6cc48e700 1 -- 172.21.15.96:0/2747070377 shutdown_connections 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.278+0000 7fd6cc48e700 1 --2- 172.21.15.96:0/2747070377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6c4102710 0x7fd6c410ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.278+0000 7fd6cc48e700 1 --2- 172.21.15.96:0/2747070377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4101df0 0x7fd6c41021d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.278+0000 7fd6cc48e700 1 -- 172.21.15.96:0/2747070377 >> 172.21.15.96:0/2747070377 conn(0x7fd6c40fb680 msgr2=0x7fd6c40fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.278+0000 7fd6cc48e700 1 -- 172.21.15.96:0/2747070377 shutdown_connections 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.278+0000 7fd6cc48e700 1 -- 172.21.15.96:0/2747070377 wait complete. 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6cc48e700 1 Processor -- start 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6cc48e700 1 -- start start 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6cc48e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6c4101df0 0x7fd6c419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6cc48e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4102710 0x7fd6c419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:43.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6cc48e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c419d3f0 con 0x7fd6c4102710 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6cc48e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c41969d0 con 0x7fd6c4101df0 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6c9a29700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4102710 0x7fd6c419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6c9a29700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4102710 0x7fd6c419cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52350/0 (socket says 172.21.15.96:52350) 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6c9a29700 1 -- 172.21.15.96:0/1609459919 learned_addr learned my addr 172.21.15.96:0/1609459919 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6c9a29700 1 -- 172.21.15.96:0/1609459919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6c4101df0 msgr2=0x7fd6c419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6c9a29700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6c4101df0 0x7fd6c419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6c9a29700 1 -- 172.21.15.96:0/1609459919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c00095f0 con 0x7fd6c4102710 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6c9a29700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4102710 0x7fd6c419cdd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400d6d0 tx=0x7fd6b400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.279+0000 7fd6bb7fe700 1 -- 172.21.15.96:0/1609459919 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6b400f860 con 0x7fd6c4102710 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.280+0000 7fd6bb7fe700 1 -- 172.21.15.96:0/1609459919 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6b400fe80 con 0x7fd6c4102710 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.280+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c4196cb0 con 0x7fd6c4102710 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.280+0000 7fd6bb7fe700 1 -- 172.21.15.96:0/1609459919 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6b4010ea0 con 0x7fd6c4102710 2024-04-22T15:39:43.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.280+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c4197200 con 0x7fd6c4102710 2024-04-22T15:39:43.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.280+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c4105510 con 0x7fd6c4102710 2024-04-22T15:39:43.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.281+0000 7fd6bb7fe700 1 -- 172.21.15.96:0/1609459919 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd6b4010460 con 0x7fd6c4102710 2024-04-22T15:39:43.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.281+0000 7fd6bb7fe700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6b006ecc0 0x7fd6b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:43.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.282+0000 7fd6bb7fe700 1 -- 172.21.15.96:0/1609459919 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd6b408e9c0 con 0x7fd6c4102710 2024-04-22T15:39:43.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.282+0000 7fd6ca22a700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6b006ecc0 0x7fd6b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:43.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.282+0000 7fd6ca22a700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6b006ecc0 0x7fd6b0071180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000b5c0 tx=0x7fd6c0005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:43.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.286+0000 7fd6bb7fe700 1 -- 172.21.15.96:0/1609459919 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd6b405b2d0 con 0x7fd6c4102710 2024-04-22T15:39:43.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.502+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6c4197e90 con 0x7fd6b006ecc0 2024-04-22T15:39:43.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.503+0000 7fd6bb7fe700 1 -- 172.21.15.96:0/1609459919 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd6c4197e90 con 0x7fd6b006ecc0 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:39:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:39:43.508 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.506+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6b006ecc0 msgr2=0x7fd6b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.506+0000 7fd6cc48e700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6b006ecc0 0x7fd6b0071180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000b5c0 tx=0x7fd6c0005e20 comp rx=0 tx=0).stop 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.506+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4102710 msgr2=0x7fd6c419cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.506+0000 7fd6cc48e700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4102710 0x7fd6c419cdd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400d6d0 tx=0x7fd6b400da90 comp rx=0 tx=0).stop 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.507+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 shutdown_connections 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.507+0000 7fd6cc48e700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd6b006ecc0 0x7fd6b0071180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.507+0000 7fd6cc48e700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6c4101df0 0x7fd6c419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.507+0000 7fd6cc48e700 1 --2- 172.21.15.96:0/1609459919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4102710 0x7fd6c419cdd0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.507+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 >> 172.21.15.96:0/1609459919 conn(0x7fd6c40fb680 msgr2=0x7fd6c41030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:43.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.507+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 shutdown_connections 2024-04-22T15:39:43.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.507+0000 7fd6cc48e700 1 -- 172.21.15.96:0/1609459919 wait complete. 2024-04-22T15:39:43.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.611+0000 7f1e5c10a700 1 -- 172.21.15.96:0/3971923515 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 msgr2=0x7f1e540fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.611+0000 7f1e5c10a700 1 --2- 172.21.15.96:0/3971923515 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 0x7f1e540fff70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009910 tx=0x7f1e50009c20 comp rx=0 tx=0).stop 2024-04-22T15:39:43.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.611+0000 7f1e5c10a700 1 -- 172.21.15.96:0/3971923515 shutdown_connections 2024-04-22T15:39:43.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.611+0000 7f1e5c10a700 1 --2- 172.21.15.96:0/3971923515 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 0x7f1e540fff70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.611+0000 7f1e5c10a700 1 --2- 172.21.15.96:0/3971923515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e54105b80 0x7f1e54105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.611+0000 7f1e5c10a700 1 -- 172.21.15.96:0/3971923515 >> 172.21.15.96:0/3971923515 conn(0x7f1e540fb5e0 msgr2=0x7f1e540fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:43.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.611+0000 7f1e5c10a700 1 -- 172.21.15.96:0/3971923515 shutdown_connections 2024-04-22T15:39:43.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.611+0000 7f1e5c10a700 1 -- 172.21.15.96:0/3971923515 wait complete. 2024-04-22T15:39:43.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e5c10a700 1 Processor -- start 2024-04-22T15:39:43.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e5c10a700 1 -- start start 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e5c10a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 0x7f1e54197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e5c10a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e54105b80 0x7f1e54197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e5c10a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e541984a0 con 0x7f1e540ffaf0 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e5c10a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e541985e0 con 0x7f1e54105b80 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e59ea6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 0x7f1e54197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e59ea6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 0x7f1e54197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52370/0 (socket says 172.21.15.96:52370) 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e59ea6700 1 -- 172.21.15.96:0/4190866014 learned_addr learned my addr 172.21.15.96:0/4190866014 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e596a5700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e54105b80 0x7f1e54197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e59ea6700 1 -- 172.21.15.96:0/4190866014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e54105b80 msgr2=0x7f1e54197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e59ea6700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e54105b80 0x7f1e54197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.612+0000 7f1e59ea6700 1 -- 172.21.15.96:0/4190866014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e500095f0 con 0x7f1e540ffaf0 2024-04-22T15:39:43.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.613+0000 7f1e59ea6700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 0x7f1e54197990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400d8d0 tx=0x7f1e4400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:43.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.613+0000 7f1e4affd700 1 -- 172.21.15.96:0/4190866014 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e440098a0 con 0x7f1e540ffaf0 2024-04-22T15:39:43.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.613+0000 7f1e4affd700 1 -- 172.21.15.96:0/4190866014 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e44010460 con 0x7f1e540ffaf0 2024-04-22T15:39:43.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.613+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e541a0a00 con 0x7f1e540ffaf0 2024-04-22T15:39:43.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.613+0000 7f1e4affd700 1 -- 172.21.15.96:0/4190866014 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e4400f5f0 con 0x7f1e540ffaf0 2024-04-22T15:39:43.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.613+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e541a0e10 con 0x7f1e540ffaf0 2024-04-22T15:39:43.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.614+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e54108110 con 0x7f1e540ffaf0 2024-04-22T15:39:43.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.614+0000 7f1e4affd700 1 -- 172.21.15.96:0/4190866014 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1e44009a00 con 0x7f1e540ffaf0 2024-04-22T15:39:43.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.615+0000 7f1e4affd700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1e4006ecc0 0x7f1e40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:43.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.615+0000 7f1e4affd700 1 -- 172.21.15.96:0/4190866014 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1e4408ed80 con 0x7f1e540ffaf0 2024-04-22T15:39:43.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.615+0000 7f1e596a5700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1e4006ecc0 0x7f1e40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:43.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.615+0000 7f1e596a5700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1e4006ecc0 0x7f1e40071180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1e540698e0 tx=0x7f1e5001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:43.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.619+0000 7f1e4affd700 1 -- 172.21.15.96:0/4190866014 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1e44057ae0 con 0x7f1e540ffaf0 2024-04-22T15:39:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:43 smithi196 ceph-mon[81815]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:43 smithi196 ceph-mon[81815]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:43 smithi196 ceph-mon[81815]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:43 smithi196 ceph-mon[81815]: pgmap v535: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:43 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/950546379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:39:43.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:43 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4203275697' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:39:43.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.903+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1e54069730 con 0x7f1e540ffaf0 2024-04-22T15:39:43.906 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:43 smithi096 ceph-mon[112443]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:43.906 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:43 smithi096 ceph-mon[112443]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:43.907 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:43 smithi096 ceph-mon[112443]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:43.907 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:43 smithi096 ceph-mon[112443]: pgmap v535: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:43.907 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:43 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/950546379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:39:43.907 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:43 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4203275697' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:39:43.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.903+0000 7f1e4affd700 1 -- 172.21.15.96:0/4190866014 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f1e4405b1a0 con 0x7f1e540ffaf0 2024-04-22T15:39:43.907 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:39:43.907 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:39:43.907 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:39:43.908 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:39:43.908 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:39:43.908 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:39:43.908 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1e4006ecc0 msgr2=0x7f1e40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1e4006ecc0 0x7f1e40071180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1e540698e0 tx=0x7f1e5001a040 comp rx=0 tx=0).stop 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 msgr2=0x7f1e54197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 0x7f1e54197990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400d8d0 tx=0x7f1e4400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 shutdown_connections 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1e4006ecc0 0x7f1e40071180 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e540ffaf0 0x7f1e54197990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 --2- 172.21.15.96:0/4190866014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e54105b80 0x7f1e54197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 >> 172.21.15.96:0/4190866014 conn(0x7f1e540fb5e0 msgr2=0x7f1e540fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 shutdown_connections 2024-04-22T15:39:43.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:39:43.906+0000 7f1e5c10a700 1 -- 172.21.15.96:0/4190866014 wait complete. 2024-04-22T15:39:44.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:44 smithi096 ceph-mon[112443]: from='client.35032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:44.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:44 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4190866014' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:39:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:44 smithi196 ceph-mon[81815]: from='client.35032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:44 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4190866014' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:39:45.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:45 smithi096 ceph-mon[112443]: pgmap v536: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:45 smithi196 ceph-mon[81815]: pgmap v536: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:47 smithi096 ceph-mon[112443]: pgmap v537: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:47 smithi196 ceph-mon[81815]: pgmap v537: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:49.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:49 smithi096 ceph-mon[112443]: pgmap v538: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:49 smithi196 ceph-mon[81815]: pgmap v538: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:51.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:51 smithi096 ceph-mon[112443]: pgmap v539: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:51 smithi196 ceph-mon[81815]: pgmap v539: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:53 smithi196 ceph-mon[81815]: pgmap v540: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:39:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:53 smithi096 ceph-mon[112443]: pgmap v540: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:39:55.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:55 smithi096 ceph-mon[112443]: pgmap v541: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:55 smithi196 ceph-mon[81815]: pgmap v541: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:57 smithi096 ceph-mon[112443]: pgmap v542: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:58.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:57 smithi196 ceph-mon[81815]: pgmap v542: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:39:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:39:59 smithi096 ceph-mon[112443]: pgmap v543: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:39:59 smithi196 ceph-mon[81815]: pgmap v543: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:00.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:39:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:40:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:00 smithi096 ceph-mon[112443]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:40:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:00 smithi196 ceph-mon[81815]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:40:01.887 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:01 smithi096 ceph-mon[112443]: pgmap v544: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:01 smithi196 ceph-mon[81815]: pgmap v544: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:03.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:03 smithi096 ceph-mon[112443]: pgmap v545: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:03 smithi196 ceph-mon[81815]: pgmap v545: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:05 smithi096 ceph-mon[112443]: pgmap v546: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:05 smithi196 ceph-mon[81815]: pgmap v546: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:07.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:07 smithi096 ceph-mon[112443]: pgmap v547: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:07 smithi196 ceph-mon[81815]: pgmap v547: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:08.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:40:08.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:40:09.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:09 smithi096 ceph-mon[112443]: pgmap v548: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:09 smithi196 ceph-mon[81815]: pgmap v548: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:11.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:11 smithi096 ceph-mon[112443]: pgmap v549: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:11 smithi196 ceph-mon[81815]: pgmap v549: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:13.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:13 smithi096 ceph-mon[112443]: pgmap v550: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:13 smithi196 ceph-mon[81815]: pgmap v550: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:14.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.023+0000 7fef923e9700 1 -- 172.21.15.96:0/2935413259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c100e20 msgr2=0x7fef8c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.023+0000 7fef923e9700 1 --2- 172.21.15.96:0/2935413259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c100e20 0x7fef8c101200 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009910 tx=0x7fef7c009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:14.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.023+0000 7fef923e9700 1 -- 172.21.15.96:0/2935413259 shutdown_connections 2024-04-22T15:40:14.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.023+0000 7fef923e9700 1 --2- 172.21.15.96:0/2935413259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef8c1017d0 0x7fef8c105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.023+0000 7fef923e9700 1 --2- 172.21.15.96:0/2935413259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c100e20 0x7fef8c101200 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.023+0000 7fef923e9700 1 -- 172.21.15.96:0/2935413259 >> 172.21.15.96:0/2935413259 conn(0x7fef8c0fc6d0 msgr2=0x7fef8c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:14.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.023+0000 7fef923e9700 1 -- 172.21.15.96:0/2935413259 shutdown_connections 2024-04-22T15:40:14.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.023+0000 7fef923e9700 1 -- 172.21.15.96:0/2935413259 wait complete. 2024-04-22T15:40:14.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.023+0000 7fef923e9700 1 Processor -- start 2024-04-22T15:40:14.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef923e9700 1 -- start start 2024-04-22T15:40:14.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef923e9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef8c100e20 0x7fef8c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef923e9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c1017d0 0x7fef8c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef923e9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c19d3a0 con 0x7fef8c1017d0 2024-04-22T15:40:14.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef923e9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c1969d0 con 0x7fef8c100e20 2024-04-22T15:40:14.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef8b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c1017d0 0x7fef8c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef8b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c1017d0 0x7fef8c19cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39432/0 (socket says 172.21.15.96:39432) 2024-04-22T15:40:14.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef8b7fe700 1 -- 172.21.15.96:0/2165066579 learned_addr learned my addr 172.21.15.96:0/2165066579 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef8bfff700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef8c100e20 0x7fef8c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef8b7fe700 1 -- 172.21.15.96:0/2165066579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef8c100e20 msgr2=0x7fef8c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef8b7fe700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef8c100e20 0x7fef8c19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.024+0000 7fef8b7fe700 1 -- 172.21.15.96:0/2165066579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c0095f0 con 0x7fef8c1017d0 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.025+0000 7fef8b7fe700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c1017d0 0x7fef8c19cd80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fef8000d900 tx=0x7fef8000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.025+0000 7fef897fa700 1 -- 172.21.15.96:0/2165066579 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef80009900 con 0x7fef8c1017d0 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.025+0000 7fef897fa700 1 -- 172.21.15.96:0/2165066579 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef80010460 con 0x7fef8c1017d0 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.025+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8c196cb0 con 0x7fef8c1017d0 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.025+0000 7fef897fa700 1 -- 172.21.15.96:0/2165066579 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef8000f5f0 con 0x7fef8c1017d0 2024-04-22T15:40:14.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.025+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8c197200 con 0x7fef8c1017d0 2024-04-22T15:40:14.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.026+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef8c1091c0 con 0x7fef8c1017d0 2024-04-22T15:40:14.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.026+0000 7fef897fa700 1 -- 172.21.15.96:0/2165066579 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fef800105d0 con 0x7fef8c1017d0 2024-04-22T15:40:14.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.027+0000 7fef897fa700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7406ecc0 0x7fef74071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.027+0000 7fef897fa700 1 -- 172.21.15.96:0/2165066579 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fef8008eae0 con 0x7fef8c1017d0 2024-04-22T15:40:14.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.027+0000 7fef8bfff700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7406ecc0 0x7fef74071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.027+0000 7fef8bfff700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7406ecc0 0x7fef74071180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fef7c000c00 tx=0x7fef7c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:14.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.032+0000 7fef897fa700 1 -- 172.21.15.96:0/2165066579 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fef8005b310 con 0x7fef8c1017d0 2024-04-22T15:40:14.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.249+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef8c197e40 con 0x7fef7406ecc0 2024-04-22T15:40:14.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.250+0000 7fef897fa700 1 -- 172.21.15.96:0/2165066579 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fef8c197e40 con 0x7fef7406ecc0 2024-04-22T15:40:14.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.252+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7406ecc0 msgr2=0x7fef74071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.252+0000 7fef923e9700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7406ecc0 0x7fef74071180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fef7c000c00 tx=0x7fef7c01a040 comp rx=0 tx=0).stop 2024-04-22T15:40:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.252+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c1017d0 msgr2=0x7fef8c19cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.252+0000 7fef923e9700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c1017d0 0x7fef8c19cd80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fef8000d900 tx=0x7fef8000dc10 comp rx=0 tx=0).stop 2024-04-22T15:40:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.252+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 shutdown_connections 2024-04-22T15:40:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.252+0000 7fef923e9700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7406ecc0 0x7fef74071180 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.252+0000 7fef923e9700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef8c100e20 0x7fef8c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.252+0000 7fef923e9700 1 --2- 172.21.15.96:0/2165066579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef8c1017d0 0x7fef8c19cd80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.252+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 >> 172.21.15.96:0/2165066579 conn(0x7fef8c0fc6d0 msgr2=0x7fef8c0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.253+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 shutdown_connections 2024-04-22T15:40:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.253+0000 7fef923e9700 1 -- 172.21.15.96:0/2165066579 wait complete. 2024-04-22T15:40:14.266 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:40:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.357+0000 7efd9935d700 1 -- 172.21.15.96:0/1645140668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 msgr2=0x7efd940fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.357+0000 7efd9935d700 1 --2- 172.21.15.96:0/1645140668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 0x7efd940fff30 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009910 tx=0x7efd7c009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.357+0000 7efd9935d700 1 -- 172.21.15.96:0/1645140668 shutdown_connections 2024-04-22T15:40:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.357+0000 7efd9935d700 1 --2- 172.21.15.96:0/1645140668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 0x7efd940fff30 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.357+0000 7efd9935d700 1 --2- 172.21.15.96:0/1645140668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd94105ab0 0x7efd94105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.357+0000 7efd9935d700 1 -- 172.21.15.96:0/1645140668 >> 172.21.15.96:0/1645140668 conn(0x7efd940fb5e0 msgr2=0x7efd940fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.357+0000 7efd9935d700 1 -- 172.21.15.96:0/1645140668 shutdown_connections 2024-04-22T15:40:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.357+0000 7efd9935d700 1 -- 172.21.15.96:0/1645140668 wait complete. 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.357+0000 7efd9935d700 1 Processor -- start 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd9935d700 1 -- start start 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd9935d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 0x7efd94101680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd9935d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd94105ab0 0x7efd94101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd9935d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94102100 con 0x7efd940ffab0 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd9935d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94102240 con 0x7efd94105ab0 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd92ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 0x7efd94101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd92ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 0x7efd94101680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39450/0 (socket says 172.21.15.96:39450) 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd92ffd700 1 -- 172.21.15.96:0/4223370752 learned_addr learned my addr 172.21.15.96:0/4223370752 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd8a7fc700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd94105ab0 0x7efd94101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd92ffd700 1 -- 172.21.15.96:0/4223370752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd94105ab0 msgr2=0x7efd94101bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd92ffd700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd94105ab0 0x7efd94101bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.358+0000 7efd92ffd700 1 -- 172.21.15.96:0/4223370752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0095f0 con 0x7efd940ffab0 2024-04-22T15:40:14.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.359+0000 7efd92ffd700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 0x7efd94101680 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efd8400d8d0 tx=0x7efd8400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:14.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.359+0000 7efd90ff9700 1 -- 172.21.15.96:0/4223370752 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd84009940 con 0x7efd940ffab0 2024-04-22T15:40:14.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.359+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd941a2cb0 con 0x7efd940ffab0 2024-04-22T15:40:14.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.359+0000 7efd90ff9700 1 -- 172.21.15.96:0/4223370752 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd84010460 con 0x7efd940ffab0 2024-04-22T15:40:14.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.359+0000 7efd90ff9700 1 -- 172.21.15.96:0/4223370752 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd8400f5f0 con 0x7efd940ffab0 2024-04-22T15:40:14.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.359+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd941a3010 con 0x7efd940ffab0 2024-04-22T15:40:14.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.360+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd941011f0 con 0x7efd940ffab0 2024-04-22T15:40:14.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.361+0000 7efd90ff9700 1 -- 172.21.15.96:0/4223370752 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efd840105d0 con 0x7efd940ffab0 2024-04-22T15:40:14.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.361+0000 7efd90ff9700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efd8006ecc0 0x7efd80071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.361+0000 7efd8a7fc700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efd8006ecc0 0x7efd80071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.361+0000 7efd90ff9700 1 -- 172.21.15.96:0/4223370752 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7efd8408eec0 con 0x7efd940ffab0 2024-04-22T15:40:14.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.362+0000 7efd8a7fc700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efd8006ecc0 0x7efd80071180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0059d0 tx=0x7efd7c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:14.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.366+0000 7efd90ff9700 1 -- 172.21.15.96:0/4223370752 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efd84057c20 con 0x7efd940ffab0 2024-04-22T15:40:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.587+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd9410ccb0 con 0x7efd8006ecc0 2024-04-22T15:40:14.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.589+0000 7efd90ff9700 1 -- 172.21.15.96:0/4223370752 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efd9410ccb0 con 0x7efd8006ecc0 2024-04-22T15:40:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efd8006ecc0 msgr2=0x7efd80071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efd8006ecc0 0x7efd80071180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0059d0 tx=0x7efd7c01a040 comp rx=0 tx=0).stop 2024-04-22T15:40:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 msgr2=0x7efd94101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 0x7efd94101680 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efd8400d8d0 tx=0x7efd8400dc90 comp rx=0 tx=0).stop 2024-04-22T15:40:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 shutdown_connections 2024-04-22T15:40:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efd8006ecc0 0x7efd80071180 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd940ffab0 0x7efd94101680 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 --2- 172.21.15.96:0/4223370752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd94105ab0 0x7efd94101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 >> 172.21.15.96:0/4223370752 conn(0x7efd940fb5e0 msgr2=0x7efd94103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:14.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 shutdown_connections 2024-04-22T15:40:14.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.591+0000 7efd9935d700 1 -- 172.21.15.96:0/4223370752 wait complete. 2024-04-22T15:40:14.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.698+0000 7f3d621b5700 1 -- 172.21.15.96:0/1043610736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c100e30 msgr2=0x7f3d5c101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.698+0000 7f3d621b5700 1 --2- 172.21.15.96:0/1043610736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c100e30 0x7f3d5c101210 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009910 tx=0x7f3d50009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:14.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.698+0000 7f3d621b5700 1 -- 172.21.15.96:0/1043610736 shutdown_connections 2024-04-22T15:40:14.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.698+0000 7f3d621b5700 1 --2- 172.21.15.96:0/1043610736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d5c1017e0 0x7f3d5c105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.698+0000 7f3d621b5700 1 --2- 172.21.15.96:0/1043610736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c100e30 0x7f3d5c101210 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.698+0000 7f3d621b5700 1 -- 172.21.15.96:0/1043610736 >> 172.21.15.96:0/1043610736 conn(0x7f3d5c0fc700 msgr2=0x7f3d5c0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:14.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.698+0000 7f3d621b5700 1 -- 172.21.15.96:0/1043610736 shutdown_connections 2024-04-22T15:40:14.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.698+0000 7f3d621b5700 1 -- 172.21.15.96:0/1043610736 wait complete. 2024-04-22T15:40:14.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d621b5700 1 Processor -- start 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d621b5700 1 -- start start 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d621b5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d5c100e30 0x7f3d5c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d621b5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c1017e0 0x7f3d5c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d621b5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c19d490 con 0x7f3d5c1017e0 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d621b5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c196a00 con 0x7f3d5c100e30 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d5affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c1017e0 0x7f3d5c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d5affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c1017e0 0x7f3d5c19ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39468/0 (socket says 172.21.15.96:39468) 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d5affd700 1 -- 172.21.15.96:0/1475033902 learned_addr learned my addr 172.21.15.96:0/1475033902 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.699+0000 7f3d5b7fe700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d5c100e30 0x7f3d5c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.700+0000 7f3d5affd700 1 -- 172.21.15.96:0/1475033902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d5c100e30 msgr2=0x7f3d5c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.700+0000 7f3d5affd700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d5c100e30 0x7f3d5c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.700+0000 7f3d5affd700 1 -- 172.21.15.96:0/1475033902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d500095f0 con 0x7f3d5c1017e0 2024-04-22T15:40:14.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.700+0000 7f3d5affd700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c1017e0 0x7f3d5c19ce00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00d900 tx=0x7f3d4c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:14.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.700+0000 7f3d58ff9700 1 -- 172.21.15.96:0/1475033902 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d4c009900 con 0x7f3d5c1017e0 2024-04-22T15:40:14.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.700+0000 7f3d58ff9700 1 -- 172.21.15.96:0/1475033902 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d4c010460 con 0x7f3d5c1017e0 2024-04-22T15:40:14.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.700+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5c196ce0 con 0x7f3d5c1017e0 2024-04-22T15:40:14.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.700+0000 7f3d58ff9700 1 -- 172.21.15.96:0/1475033902 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d4c00f5f0 con 0x7f3d5c1017e0 2024-04-22T15:40:14.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.700+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5c197230 con 0x7f3d5c1017e0 2024-04-22T15:40:14.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.701+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5c1091d0 con 0x7f3d5c1017e0 2024-04-22T15:40:14.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.702+0000 7f3d58ff9700 1 -- 172.21.15.96:0/1475033902 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3d4c0105d0 con 0x7f3d5c1017e0 2024-04-22T15:40:14.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.702+0000 7f3d58ff9700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d4406ecc0 0x7f3d44071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.702+0000 7f3d58ff9700 1 -- 172.21.15.96:0/1475033902 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3d4c08e960 con 0x7f3d5c1017e0 2024-04-22T15:40:14.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.702+0000 7f3d5b7fe700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d4406ecc0 0x7f3d44071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.703+0000 7f3d5b7fe700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d4406ecc0 0x7f3d44071180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3d50000c00 tx=0x7f3d5001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:14.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.708+0000 7f3d58ff9700 1 -- 172.21.15.96:0/1475033902 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3d4c05b270 con 0x7f3d5c1017e0 2024-04-22T15:40:14.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.917+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d5c197ec0 con 0x7f3d4406ecc0 2024-04-22T15:40:14.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.928+0000 7f3d58ff9700 1 -- 172.21.15.96:0/1475033902 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3d5c197ec0 con 0x7f3d4406ecc0 2024-04-22T15:40:14.931 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:40:14.931 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (18m) 12m ago 30m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:40:14.931 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (30m) 12m ago 30m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:40:14.931 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (29m) 11m ago 29m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:40:14.931 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (16m) 12m ago 30m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (16m) 11m ago 29m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (18m) 12m ago 29m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (12m) 12m ago 24m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (12m) 12m ago 24m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (11m) 11m ago 24m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (24m) 11m ago 24m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (21m) 12m ago 31m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (20m) 11m ago 29m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (17m) 12m ago 31m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (16m) 11m ago 29m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (19m) 12m ago 30m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (19m) 11m ago 29m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (15m) 12m ago 28m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (15m) 12m ago 27m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:40:14.932 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (14m) 12m ago 27m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:40:14.933 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (13m) 11m ago 26m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:40:14.933 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (13m) 11m ago 26m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:40:14.933 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (12m) 11m ago 26m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:40:14.933 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (19m) 12m ago 29m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:40:14.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d4406ecc0 msgr2=0x7f3d44071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d4406ecc0 0x7f3d44071180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3d50000c00 tx=0x7f3d5001a040 comp rx=0 tx=0).stop 2024-04-22T15:40:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c1017e0 msgr2=0x7f3d5c19ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c1017e0 0x7f3d5c19ce00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00d900 tx=0x7f3d4c00dc10 comp rx=0 tx=0).stop 2024-04-22T15:40:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 shutdown_connections 2024-04-22T15:40:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d4406ecc0 0x7f3d44071180 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d5c100e30 0x7f3d5c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 --2- 172.21.15.96:0/1475033902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c1017e0 0x7f3d5c19ce00 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 >> 172.21.15.96:0/1475033902 conn(0x7f3d5c0fc700 msgr2=0x7f3d5c0fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.930+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 shutdown_connections 2024-04-22T15:40:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:14.931+0000 7f3d621b5700 1 -- 172.21.15.96:0/1475033902 wait complete. 2024-04-22T15:40:15.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.032+0000 7f9719ba9700 1 -- 172.21.15.96:0/4088601483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 msgr2=0x7f9714105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.032+0000 7f9719ba9700 1 --2- 172.21.15.96:0/4088601483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 0x7f9714105e90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9704009910 tx=0x7f9704009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:15.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.032+0000 7f9719ba9700 1 -- 172.21.15.96:0/4088601483 shutdown_connections 2024-04-22T15:40:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.032+0000 7f9719ba9700 1 --2- 172.21.15.96:0/4088601483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97140ffab0 0x7f97140fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.032+0000 7f9719ba9700 1 --2- 172.21.15.96:0/4088601483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 0x7f9714105e90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.032+0000 7f9719ba9700 1 -- 172.21.15.96:0/4088601483 >> 172.21.15.96:0/4088601483 conn(0x7f97140fb5e0 msgr2=0x7f97140fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.032+0000 7f9719ba9700 1 -- 172.21.15.96:0/4088601483 shutdown_connections 2024-04-22T15:40:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.032+0000 7f9719ba9700 1 -- 172.21.15.96:0/4088601483 wait complete. 2024-04-22T15:40:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f9719ba9700 1 Processor -- start 2024-04-22T15:40:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f9719ba9700 1 -- start start 2024-04-22T15:40:15.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f9719ba9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97140ffab0 0x7f971419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f9719ba9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 0x7f971419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f9719ba9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971419d2e0 con 0x7f9714105ab0 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f9719ba9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9714196950 con 0x7f97140ffab0 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f970affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 0x7f971419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f970affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 0x7f971419cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39488/0 (socket says 172.21.15.96:39488) 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f970affd700 1 -- 172.21.15.96:0/1765321336 learned_addr learned my addr 172.21.15.96:0/1765321336 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f97137fe700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97140ffab0 0x7f971419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f970affd700 1 -- 172.21.15.96:0/1765321336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97140ffab0 msgr2=0x7f971419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f970affd700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97140ffab0 0x7f971419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.033+0000 7f970affd700 1 -- 172.21.15.96:0/1765321336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97040095f0 con 0x7f9714105ab0 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.034+0000 7f970affd700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 0x7f971419cd50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c930 tx=0x7f96fc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.034+0000 7f97117fa700 1 -- 172.21.15.96:0/1765321336 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96fc007ad0 con 0x7f9714105ab0 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.034+0000 7f97117fa700 1 -- 172.21.15.96:0/1765321336 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96fc007c30 con 0x7f9714105ab0 2024-04-22T15:40:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.034+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9714196c30 con 0x7f9714105ab0 2024-04-22T15:40:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.034+0000 7f97117fa700 1 -- 172.21.15.96:0/1765321336 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96fc018610 con 0x7f9714105ab0 2024-04-22T15:40:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.034+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9714197180 con 0x7f9714105ab0 2024-04-22T15:40:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.035+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97141011f0 con 0x7f9714105ab0 2024-04-22T15:40:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.035+0000 7f97117fa700 1 -- 172.21.15.96:0/1765321336 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f96fc00f450 con 0x7f9714105ab0 2024-04-22T15:40:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.036+0000 7f97117fa700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f970006ecc0 0x7f9700071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.036+0000 7f97117fa700 1 -- 172.21.15.96:0/1765321336 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f96fc08f0d0 con 0x7f9714105ab0 2024-04-22T15:40:15.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.036+0000 7f97137fe700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f970006ecc0 0x7f9700071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:15.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.036+0000 7f97137fe700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f970006ecc0 0x7f9700071180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9704005f50 tx=0x7f970401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:15.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.040+0000 7f97117fa700 1 -- 172.21.15.96:0/1765321336 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f96fc05b960 con 0x7f9714105ab0 2024-04-22T15:40:15.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.324+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f971410ccb0 con 0x7f9714105ab0 2024-04-22T15:40:15.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.325+0000 7f97117fa700 1 -- 172.21.15.96:0/1765321336 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f96fc05b4f0 con 0x7f9714105ab0 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:40:15.328 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:40:15.329 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:40:15.329 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:40:15.329 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:40:15.329 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f970006ecc0 msgr2=0x7f9700071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f970006ecc0 0x7f9700071180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9704005f50 tx=0x7f970401a040 comp rx=0 tx=0).stop 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 msgr2=0x7f971419cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 0x7f971419cd50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c930 tx=0x7f96fc00ccf0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 shutdown_connections 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f970006ecc0 0x7f9700071180 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97140ffab0 0x7f971419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 --2- 172.21.15.96:0/1765321336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9714105ab0 0x7f971419cd50 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 >> 172.21.15.96:0/1765321336 conn(0x7f97140fb5e0 msgr2=0x7f9714103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:15.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 shutdown_connections 2024-04-22T15:40:15.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.327+0000 7f9719ba9700 1 -- 172.21.15.96:0/1765321336 wait complete. 2024-04-22T15:40:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.430+0000 7f73613d5700 1 -- 172.21.15.96:0/1154178712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c1026b0 msgr2=0x7f735c10aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.430+0000 7f73613d5700 1 --2- 172.21.15.96:0/1154178712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c1026b0 0x7f735c10aba0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7350009910 tx=0x7f7350009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.430+0000 7f73613d5700 1 -- 172.21.15.96:0/1154178712 shutdown_connections 2024-04-22T15:40:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.430+0000 7f73613d5700 1 --2- 172.21.15.96:0/1154178712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c1026b0 0x7f735c10aba0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.430+0000 7f73613d5700 1 --2- 172.21.15.96:0/1154178712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f735c101d90 0x7f735c102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.430+0000 7f73613d5700 1 -- 172.21.15.96:0/1154178712 >> 172.21.15.96:0/1154178712 conn(0x7f735c0fb5e0 msgr2=0x7f735c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.430+0000 7f73613d5700 1 -- 172.21.15.96:0/1154178712 shutdown_connections 2024-04-22T15:40:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.430+0000 7f73613d5700 1 -- 172.21.15.96:0/1154178712 wait complete. 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.430+0000 7f73613d5700 1 Processor -- start 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f73613d5700 1 -- start start 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f73613d5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c101d90 0x7f735c0ff9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f73613d5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f735c1026b0 0x7f735c0fff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f73613d5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c100470 con 0x7f735c101d90 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f73613d5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1005b0 con 0x7f735c1026b0 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f735affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c101d90 0x7f735c0ff9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f735affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c101d90 0x7f735c0ff9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39502/0 (socket says 172.21.15.96:39502) 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f735affd700 1 -- 172.21.15.96:0/2185658456 learned_addr learned my addr 172.21.15.96:0/2185658456 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f735a7fc700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f735c1026b0 0x7f735c0fff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:15.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f735affd700 1 -- 172.21.15.96:0/2185658456 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f735c1026b0 msgr2=0x7f735c0fff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f735affd700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f735c1026b0 0x7f735c0fff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f735affd700 1 -- 172.21.15.96:0/2185658456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73500095f0 con 0x7f735c101d90 2024-04-22T15:40:15.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.431+0000 7f735affd700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c101d90 0x7f735c0ff9f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f735400d8d0 tx=0x7f735400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:15.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.432+0000 7f7343fff700 1 -- 172.21.15.96:0/2185658456 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7354009940 con 0x7f735c101d90 2024-04-22T15:40:15.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.432+0000 7f7343fff700 1 -- 172.21.15.96:0/2185658456 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7354010460 con 0x7f735c101d90 2024-04-22T15:40:15.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.432+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c1a2d30 con 0x7f735c101d90 2024-04-22T15:40:15.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.432+0000 7f7343fff700 1 -- 172.21.15.96:0/2185658456 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f735400f5f0 con 0x7f735c101d90 2024-04-22T15:40:15.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.432+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c1a3090 con 0x7f735c101d90 2024-04-22T15:40:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.433+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735c0699f0 con 0x7f735c101d90 2024-04-22T15:40:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.433+0000 7f7343fff700 1 -- 172.21.15.96:0/2185658456 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f73540105d0 con 0x7f735c101d90 2024-04-22T15:40:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.433+0000 7f7343fff700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f734406ecc0 0x7f7344071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.434+0000 7f7343fff700 1 -- 172.21.15.96:0/2185658456 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f735408eec0 con 0x7f735c101d90 2024-04-22T15:40:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.434+0000 7f735a7fc700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f734406ecc0 0x7f7344071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.434+0000 7f735a7fc700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f734406ecc0 0x7f7344071180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f735c101120 tx=0x7f735001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:15.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.438+0000 7f7343fff700 1 -- 172.21.15.96:0/2185658456 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7354057c20 con 0x7f735c101d90 2024-04-22T15:40:15.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.682+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f735c10ddf0 con 0x7f735c101d90 2024-04-22T15:40:15.685 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:15 smithi096 ceph-mon[112443]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:15.685 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:15 smithi096 ceph-mon[112443]: pgmap v551: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:15.686 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:15 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1765321336' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:40:15.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.683+0000 7f7343fff700 1 -- 172.21.15.96:0/2185658456 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f735405b2e0 con 0x7f735c101d90 2024-04-22T15:40:15.686 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:40:15.686 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:40:15.686 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:40:15.687 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:40:15.688 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:40:15.689 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:40:15.689 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:40:15.689 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:40:15.689 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:40:15.689 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:40:15.689 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:40:15.689 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:40:15.689 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:40:15.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f734406ecc0 msgr2=0x7f7344071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f734406ecc0 0x7f7344071180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f735c101120 tx=0x7f735001a040 comp rx=0 tx=0).stop 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c101d90 msgr2=0x7f735c0ff9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c101d90 0x7f735c0ff9f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f735400d8d0 tx=0x7f735400dc90 comp rx=0 tx=0).stop 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 shutdown_connections 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f734406ecc0 0x7f7344071180 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f735c101d90 0x7f735c0ff9f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 --2- 172.21.15.96:0/2185658456 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f735c1026b0 0x7f735c0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 >> 172.21.15.96:0/2185658456 conn(0x7f735c0fb5e0 msgr2=0x7f735c103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 shutdown_connections 2024-04-22T15:40:15.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.686+0000 7f73613d5700 1 -- 172.21.15.96:0/2185658456 wait complete. 2024-04-22T15:40:15.691 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:40:15.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.793+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3759657033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 msgr2=0x7f1bf41047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.793+0000 7f1bf96cd700 1 --2- 172.21.15.96:0/3759657033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 0x7f1bf41047b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009910 tx=0x7f1be4009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:15.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.793+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3759657033 shutdown_connections 2024-04-22T15:40:15.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.793+0000 7f1bf96cd700 1 --2- 172.21.15.96:0/3759657033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 0x7f1bf41047b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.793+0000 7f1bf96cd700 1 --2- 172.21.15.96:0/3759657033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bf40ffdb0 0x7f1bf4100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.793+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3759657033 >> 172.21.15.96:0/3759657033 conn(0x7f1bf40fb6a0 msgr2=0x7f1bf40fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:15.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.793+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3759657033 shutdown_connections 2024-04-22T15:40:15.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.793+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3759657033 wait complete. 2024-04-22T15:40:15.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf96cd700 1 Processor -- start 2024-04-22T15:40:15.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf96cd700 1 -- start start 2024-04-22T15:40:15.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf96cd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bf40ffdb0 0x7f1bf4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:15.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf96cd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 0x7f1bf4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:15.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf96cd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf41984d0 con 0x7f1bf4100760 2024-04-22T15:40:15.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf96cd700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf4198610 con 0x7f1bf40ffdb0 2024-04-22T15:40:15.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 0x7f1bf4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 0x7f1bf4197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39512/0 (socket says 172.21.15.96:39512) 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf27fc700 1 -- 172.21.15.96:0/3689259004 learned_addr learned my addr 172.21.15.96:0/3689259004 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.794+0000 7f1bf2ffd700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bf40ffdb0 0x7f1bf4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.795+0000 7f1bf27fc700 1 -- 172.21.15.96:0/3689259004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bf40ffdb0 msgr2=0x7f1bf4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.795+0000 7f1bf27fc700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bf40ffdb0 0x7f1bf4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.795+0000 7f1bf27fc700 1 -- 172.21.15.96:0/3689259004 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be40095f0 con 0x7f1bf4100760 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.795+0000 7f1bf27fc700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 0x7f1bf4197f90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1be400bd70 tx=0x7f1be400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.795+0000 7f1bdbfff700 1 -- 172.21.15.96:0/3689259004 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1be401d070 con 0x7f1bf4100760 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.795+0000 7f1bdbfff700 1 -- 172.21.15.96:0/3689259004 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1be40056e0 con 0x7f1bf4100760 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.795+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf4198890 con 0x7f1bf4100760 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.795+0000 7f1bdbfff700 1 -- 172.21.15.96:0/3689259004 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1be4017400 con 0x7f1bf4100760 2024-04-22T15:40:15.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.795+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf41a0df0 con 0x7f1bf4100760 2024-04-22T15:40:15.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.796+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf41081b0 con 0x7f1bf4100760 2024-04-22T15:40:15.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.796+0000 7f1bdbfff700 1 -- 172.21.15.96:0/3689259004 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1be400f460 con 0x7f1bf4100760 2024-04-22T15:40:15.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.797+0000 7f1bdbfff700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1bdc06ecc0 0x7f1bdc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:15.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.797+0000 7f1bf2ffd700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1bdc06ecc0 0x7f1bdc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:15.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.797+0000 7f1bdbfff700 1 -- 172.21.15.96:0/3689259004 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1be4090270 con 0x7f1bf4100760 2024-04-22T15:40:15.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.797+0000 7f1bf2ffd700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1bdc06ecc0 0x7f1bdc071180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1be8005950 tx=0x7f1be80058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:15.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:15.802+0000 7f1bdbfff700 1 -- 172.21.15.96:0/3689259004 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1be4059090 con 0x7f1bf4100760 2024-04-22T15:40:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:15 smithi196 ceph-mon[81815]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:15 smithi196 ceph-mon[81815]: pgmap v551: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:15 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1765321336' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:40:16.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.022+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bf4069600 con 0x7f1bdc06ecc0 2024-04-22T15:40:16.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.023+0000 7f1bdbfff700 1 -- 172.21.15.96:0/3689259004 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1bf4069600 con 0x7f1bdc06ecc0 2024-04-22T15:40:16.026 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:40:16.027 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:40:16.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.025+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1bdc06ecc0 msgr2=0x7f1bdc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.025+0000 7f1bf96cd700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1bdc06ecc0 0x7f1bdc071180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1be8005950 tx=0x7f1be80058e0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.025+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 msgr2=0x7f1bf4197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.025+0000 7f1bf96cd700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 0x7f1bf4197f90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1be400bd70 tx=0x7f1be400be50 comp rx=0 tx=0).stop 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.025+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 shutdown_connections 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.026+0000 7f1bf96cd700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1bdc06ecc0 0x7f1bdc071180 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.026+0000 7f1bf96cd700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bf40ffdb0 0x7f1bf4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.026+0000 7f1bf96cd700 1 --2- 172.21.15.96:0/3689259004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf4100760 0x7f1bf4197f90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.026+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 >> 172.21.15.96:0/3689259004 conn(0x7f1bf40fb6a0 msgr2=0x7f1bf40fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.026+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 shutdown_connections 2024-04-22T15:40:16.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.026+0000 7f1bf96cd700 1 -- 172.21.15.96:0/3689259004 wait complete. 2024-04-22T15:40:16.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.129+0000 7f27d3f12700 1 -- 172.21.15.96:0/3191146111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 msgr2=0x7f27cc073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:16.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.129+0000 7f27d3f12700 1 --2- 172.21.15.96:0/3191146111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 0x7f27cc073660 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009910 tx=0x7f27c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:16.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 -- 172.21.15.96:0/3191146111 shutdown_connections 2024-04-22T15:40:16.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 --2- 172.21.15.96:0/3191146111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27cc073ba0 0x7f27cc111690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 --2- 172.21.15.96:0/3191146111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 0x7f27cc073660 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 -- 172.21.15.96:0/3191146111 >> 172.21.15.96:0/3191146111 conn(0x7f27cc0fc6d0 msgr2=0x7f27cc0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:16.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 -- 172.21.15.96:0/3191146111 shutdown_connections 2024-04-22T15:40:16.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 -- 172.21.15.96:0/3191146111 wait complete. 2024-04-22T15:40:16.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 Processor -- start 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 -- start start 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 0x7f27cc19d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27cc073ba0 0x7f27cc19d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc19dcb0 con 0x7f27cc073280 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.130+0000 7f27d3f12700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc1a1980 con 0x7f27cc073ba0 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d1cae700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 0x7f27cc19d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d1cae700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 0x7f27cc19d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39518/0 (socket says 172.21.15.96:39518) 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d1cae700 1 -- 172.21.15.96:0/3999412615 learned_addr learned my addr 172.21.15.96:0/3999412615 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d14ad700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27cc073ba0 0x7f27cc19d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d1cae700 1 -- 172.21.15.96:0/3999412615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27cc073ba0 msgr2=0x7f27cc19d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d1cae700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27cc073ba0 0x7f27cc19d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d1cae700 1 -- 172.21.15.96:0/3999412615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c80095f0 con 0x7f27cc073280 2024-04-22T15:40:16.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d1cae700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 0x7f27cc19d090 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f27c800bd70 tx=0x7f27c800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:16.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27c2ffd700 1 -- 172.21.15.96:0/3999412615 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27c801d070 con 0x7f27cc073280 2024-04-22T15:40:16.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27c2ffd700 1 -- 172.21.15.96:0/3999412615 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27c80056e0 con 0x7f27cc073280 2024-04-22T15:40:16.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc1a1c60 con 0x7f27cc073280 2024-04-22T15:40:16.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27c2ffd700 1 -- 172.21.15.96:0/3999412615 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27c800fe40 con 0x7f27cc073280 2024-04-22T15:40:16.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.131+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc1a21b0 con 0x7f27cc073280 2024-04-22T15:40:16.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.132+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc0749c0 con 0x7f27cc073280 2024-04-22T15:40:16.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.133+0000 7f27c2ffd700 1 -- 172.21.15.96:0/3999412615 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f27c800f680 con 0x7f27cc073280 2024-04-22T15:40:16.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.133+0000 7f27c2ffd700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27b806ec60 0x7f27b8071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:16.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.133+0000 7f27c2ffd700 1 -- 172.21.15.96:0/3999412615 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f27c8090280 con 0x7f27cc073280 2024-04-22T15:40:16.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.133+0000 7f27d14ad700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27b806ec60 0x7f27b8071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:16.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.134+0000 7f27d14ad700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27b806ec60 0x7f27b8071120 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f27cc19e5f0 tx=0x7f27bc00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:16.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.138+0000 7f27c2ffd700 1 -- 172.21.15.96:0/3999412615 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f27c8058fe0 con 0x7f27cc073280 2024-04-22T15:40:16.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.416+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f27cc114360 con 0x7f27cc073280 2024-04-22T15:40:16.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.417+0000 7f27c2ffd700 1 -- 172.21.15.96:0/3999412615 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f27c805c6a0 con 0x7f27cc073280 2024-04-22T15:40:16.420 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:40:16.420 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:40:16.420 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:40:16.420 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:40:16.420 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:40:16.420 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:40:16.420 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:40:16.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27b806ec60 msgr2=0x7f27b8071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:16.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27b806ec60 0x7f27b8071120 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f27cc19e5f0 tx=0x7f27bc00b410 comp rx=0 tx=0).stop 2024-04-22T15:40:16.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 msgr2=0x7f27cc19d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:16.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 0x7f27cc19d090 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f27c800bd70 tx=0x7f27c800bda0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 shutdown_connections 2024-04-22T15:40:16.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27b806ec60 0x7f27b8071120 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27cc073280 0x7f27cc19d090 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 --2- 172.21.15.96:0/3999412615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27cc073ba0 0x7f27cc19d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 >> 172.21.15.96:0/3999412615 conn(0x7f27cc0fc6d0 msgr2=0x7f27cc100900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:16.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 shutdown_connections 2024-04-22T15:40:16.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:16.419+0000 7f27d3f12700 1 -- 172.21.15.96:0/3999412615 wait complete. 2024-04-22T15:40:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:16 smithi096 ceph-mon[112443]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:16 smithi096 ceph-mon[112443]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:16 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2185658456' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:40:16.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:16 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3999412615' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:40:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:16 smithi196 ceph-mon[81815]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:16 smithi196 ceph-mon[81815]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:17.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:16 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2185658456' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:40:17.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:16 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3999412615' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:40:17.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:17 smithi096 ceph-mon[112443]: from='client.35060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:17.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:17 smithi096 ceph-mon[112443]: pgmap v552: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:17 smithi196 ceph-mon[81815]: from='client.35060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:17 smithi196 ceph-mon[81815]: pgmap v552: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:18.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:18 smithi096 ceph-mon[112443]: pgmap v553: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:18 smithi196 ceph-mon[81815]: pgmap v553: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:21.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:21 smithi096 ceph-mon[112443]: pgmap v554: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:21 smithi196 ceph-mon[81815]: pgmap v554: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:23.807 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:23 smithi196 ceph-mon[81815]: pgmap v555: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:23.807 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:40:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:23 smithi096 ceph-mon[112443]: pgmap v555: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:40:25.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:25 smithi096 ceph-mon[112443]: pgmap v556: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:25 smithi196 ceph-mon[81815]: pgmap v556: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:27.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:27 smithi096 ceph-mon[112443]: pgmap v557: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:27 smithi196 ceph-mon[81815]: pgmap v557: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:29.831 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:29 smithi096 ceph-mon[112443]: pgmap v558: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:29 smithi196 ceph-mon[81815]: pgmap v558: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:31 smithi096 ceph-mon[112443]: pgmap v559: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:32.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:31 smithi196 ceph-mon[81815]: pgmap v559: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:33.914 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:33 smithi196 ceph-mon[81815]: pgmap v560: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:33.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:33 smithi096 ceph-mon[112443]: pgmap v560: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:35.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:35 smithi096 ceph-mon[112443]: pgmap v561: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:35 smithi196 ceph-mon[81815]: pgmap v561: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:37 smithi096 ceph-mon[112443]: pgmap v562: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:37 smithi196 ceph-mon[81815]: pgmap v562: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:38.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:40:38.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:40:39.938 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:39 smithi096 ceph-mon[112443]: pgmap v563: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:39 smithi196 ceph-mon[81815]: pgmap v563: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:41.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:41 smithi096 ceph-mon[112443]: pgmap v564: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:41 smithi196 ceph-mon[81815]: pgmap v564: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:43 smithi096 ceph-mon[112443]: pgmap v565: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:43 smithi196 ceph-mon[81815]: pgmap v565: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:45.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:45 smithi096 ceph-mon[112443]: pgmap v566: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:45 smithi196 ceph-mon[81815]: pgmap v566: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:46.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.537+0000 7f2e21024700 1 -- 172.21.15.96:0/3968267411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 msgr2=0x7f2e1c0fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:46.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.537+0000 7f2e21024700 1 --2- 172.21.15.96:0/3968267411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 0x7f2e1c0fff40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009910 tx=0x7f2e10009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:46.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.537+0000 7f2e21024700 1 -- 172.21.15.96:0/3968267411 shutdown_connections 2024-04-22T15:40:46.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.537+0000 7f2e21024700 1 --2- 172.21.15.96:0/3968267411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 0x7f2e1c0fff40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:46.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.537+0000 7f2e21024700 1 --2- 172.21.15.96:0/3968267411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c105ac0 0x7f2e1c105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:46.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.537+0000 7f2e21024700 1 -- 172.21.15.96:0/3968267411 >> 172.21.15.96:0/3968267411 conn(0x7f2e1c0fb610 msgr2=0x7f2e1c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:46.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.537+0000 7f2e21024700 1 -- 172.21.15.96:0/3968267411 shutdown_connections 2024-04-22T15:40:46.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.537+0000 7f2e21024700 1 -- 172.21.15.96:0/3968267411 wait complete. 2024-04-22T15:40:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e21024700 1 Processor -- start 2024-04-22T15:40:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e21024700 1 -- start start 2024-04-22T15:40:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e21024700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 0x7f2e1c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e21024700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c105ac0 0x7f2e1c19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e21024700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c19d390 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e21024700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c196970 con 0x7f2e1c105ac0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e1ad9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 0x7f2e1c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e1ad9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 0x7f2e1c19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39166/0 (socket says 172.21.15.96:39166) 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e1ad9d700 1 -- 172.21.15.96:0/763361903 learned_addr learned my addr 172.21.15.96:0/763361903 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.538+0000 7f2e1a59c700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c105ac0 0x7f2e1c19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.539+0000 7f2e1ad9d700 1 -- 172.21.15.96:0/763361903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c105ac0 msgr2=0x7f2e1c19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.539+0000 7f2e1ad9d700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c105ac0 0x7f2e1c19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.539+0000 7f2e1ad9d700 1 -- 172.21.15.96:0/763361903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e100095f0 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.539+0000 7f2e1ad9d700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 0x7f2e1c19c830 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00c8a0 tx=0x7f2e0c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.539+0000 7f2e03fff700 1 -- 172.21.15.96:0/763361903 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0c00cea0 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.539+0000 7f2e03fff700 1 -- 172.21.15.96:0/763361903 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e0c004830 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.539+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1c196c50 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.539+0000 7f2e03fff700 1 -- 172.21.15.96:0/763361903 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0c0055b0 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.539+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1c1971a0 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.540+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1c1090f0 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.540+0000 7f2e03fff700 1 -- 172.21.15.96:0/763361903 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2e0c0049a0 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.541+0000 7f2e03fff700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e0406ecc0 0x7f2e04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:46.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.541+0000 7f2e03fff700 1 -- 172.21.15.96:0/763361903 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2e0c08eba0 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.541+0000 7f2e1a59c700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e0406ecc0 0x7f2e04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:46.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.541+0000 7f2e1a59c700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e0406ecc0 0x7f2e04071180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2e100059d0 tx=0x7f2e1001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:46.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.546+0000 7f2e03fff700 1 -- 172.21.15.96:0/763361903 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2e0c05b430 con 0x7f2e1c0ffac0 2024-04-22T15:40:46.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.766+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e1c197ea0 con 0x7f2e0406ecc0 2024-04-22T15:40:46.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.767+0000 7f2e03fff700 1 -- 172.21.15.96:0/763361903 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2e1c197ea0 con 0x7f2e0406ecc0 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.769+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e0406ecc0 msgr2=0x7f2e04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.769+0000 7f2e21024700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e0406ecc0 0x7f2e04071180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2e100059d0 tx=0x7f2e1001a040 comp rx=0 tx=0).stop 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.769+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 msgr2=0x7f2e1c19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.769+0000 7f2e21024700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 0x7f2e1c19c830 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00c8a0 tx=0x7f2e0c00cc60 comp rx=0 tx=0).stop 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.769+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 shutdown_connections 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.769+0000 7f2e21024700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2e0406ecc0 0x7f2e04071180 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.770+0000 7f2e21024700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e1c0ffac0 0x7f2e1c19c830 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.770+0000 7f2e21024700 1 --2- 172.21.15.96:0/763361903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c105ac0 0x7f2e1c19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.770+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 >> 172.21.15.96:0/763361903 conn(0x7f2e1c0fb610 msgr2=0x7f2e1c069690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.770+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 shutdown_connections 2024-04-22T15:40:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.770+0000 7f2e21024700 1 -- 172.21.15.96:0/763361903 wait complete. 2024-04-22T15:40:46.783 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.873+0000 7f698ea36700 1 -- 172.21.15.96:0/335352672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 msgr2=0x7f69880fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.873+0000 7f698ea36700 1 --2- 172.21.15.96:0/335352672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 0x7f69880fff60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f697c009910 tx=0x7f697c009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.873+0000 7f698ea36700 1 -- 172.21.15.96:0/335352672 shutdown_connections 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.873+0000 7f698ea36700 1 --2- 172.21.15.96:0/335352672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 0x7f69880fff60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.873+0000 7f698ea36700 1 --2- 172.21.15.96:0/335352672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6988105b70 0x7f6988105f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.873+0000 7f698ea36700 1 -- 172.21.15.96:0/335352672 >> 172.21.15.96:0/335352672 conn(0x7f69880fb610 msgr2=0x7f69880fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.873+0000 7f698ea36700 1 -- 172.21.15.96:0/335352672 shutdown_connections 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.873+0000 7f698ea36700 1 -- 172.21.15.96:0/335352672 wait complete. 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.874+0000 7f698ea36700 1 Processor -- start 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.874+0000 7f698ea36700 1 -- start start 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.874+0000 7f698ea36700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 0x7f6988195790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:46.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.874+0000 7f698ea36700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6988105b70 0x7f6988195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.874+0000 7f698ea36700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698806a050 con 0x7f69880ffae0 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.874+0000 7f698ea36700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6988196210 con 0x7f6988105b70 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f6987fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 0x7f6988195790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f6987fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 0x7f6988195790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39182/0 (socket says 172.21.15.96:39182) 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f6987fff700 1 -- 172.21.15.96:0/2190261475 learned_addr learned my addr 172.21.15.96:0/2190261475 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f69877fe700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6988105b70 0x7f6988195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f6987fff700 1 -- 172.21.15.96:0/2190261475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6988105b70 msgr2=0x7f6988195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f6987fff700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6988105b70 0x7f6988195cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f6987fff700 1 -- 172.21.15.96:0/2190261475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c0095f0 con 0x7f69880ffae0 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f69877fe700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6988105b70 0x7f6988195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:40:46.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f6987fff700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 0x7f6988195790 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f697800c8a0 tx=0x7f697800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:46.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f69857fa700 1 -- 172.21.15.96:0/2190261475 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f697800cea0 con 0x7f69880ffae0 2024-04-22T15:40:46.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f69857fa700 1 -- 172.21.15.96:0/2190261475 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6978004830 con 0x7f69880ffae0 2024-04-22T15:40:46.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f69857fa700 1 -- 172.21.15.96:0/2190261475 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69780055b0 con 0x7f69880ffae0 2024-04-22T15:40:46.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6988196490 con 0x7f69880ffae0 2024-04-22T15:40:46.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.875+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69881a2fd0 con 0x7f69880ffae0 2024-04-22T15:40:46.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.876+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6988108100 con 0x7f69880ffae0 2024-04-22T15:40:46.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.877+0000 7f69857fa700 1 -- 172.21.15.96:0/2190261475 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f69780049a0 con 0x7f69880ffae0 2024-04-22T15:40:46.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.877+0000 7f69857fa700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f697006ecc0 0x7f6970071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:46.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.878+0000 7f69857fa700 1 -- 172.21.15.96:0/2190261475 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f697808eb80 con 0x7f69880ffae0 2024-04-22T15:40:46.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.878+0000 7f69877fe700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f697006ecc0 0x7f6970071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:46.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.878+0000 7f69877fe700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f697006ecc0 0x7f6970071180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f697c006010 tx=0x7f697c00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:46.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:46.882+0000 7f69857fa700 1 -- 172.21.15.96:0/2190261475 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f697805b410 con 0x7f69880ffae0 2024-04-22T15:40:47.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.100+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f698810cce0 con 0x7f697006ecc0 2024-04-22T15:40:47.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.102+0000 7f69857fa700 1 -- 172.21.15.96:0/2190261475 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f698810cce0 con 0x7f697006ecc0 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f697006ecc0 msgr2=0x7f6970071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f697006ecc0 0x7f6970071180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f697c006010 tx=0x7f697c00b540 comp rx=0 tx=0).stop 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 msgr2=0x7f6988195790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 0x7f6988195790 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f697800c8a0 tx=0x7f697800cc60 comp rx=0 tx=0).stop 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 shutdown_connections 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f697006ecc0 0x7f6970071180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69880ffae0 0x7f6988195790 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 --2- 172.21.15.96:0/2190261475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6988105b70 0x7f6988195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 >> 172.21.15.96:0/2190261475 conn(0x7f69880fb610 msgr2=0x7f69880fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 shutdown_connections 2024-04-22T15:40:47.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.104+0000 7f698ea36700 1 -- 172.21.15.96:0/2190261475 wait complete. 2024-04-22T15:40:47.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 -- 172.21.15.96:0/1328573088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 msgr2=0x7f4f9c0ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 --2- 172.21.15.96:0/1328573088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 0x7f4f9c0ff730 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009910 tx=0x7f4f90009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:47.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 -- 172.21.15.96:0/1328573088 shutdown_connections 2024-04-22T15:40:47.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 --2- 172.21.15.96:0/1328573088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f9c0ffd00 0x7f4f9c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 --2- 172.21.15.96:0/1328573088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 0x7f4f9c0ff730 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 -- 172.21.15.96:0/1328573088 >> 172.21.15.96:0/1328573088 conn(0x7f4f9c075f40 msgr2=0x7f4f9c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:47.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 -- 172.21.15.96:0/1328573088 shutdown_connections 2024-04-22T15:40:47.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 -- 172.21.15.96:0/1328573088 wait complete. 2024-04-22T15:40:47.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 Processor -- start 2024-04-22T15:40:47.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.208+0000 7f4fa30b5700 1 -- start start 2024-04-22T15:40:47.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa30b5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 0x7f4f9c196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:47.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa30b5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f9c0ffd00 0x7f4f9c197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:47.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa30b5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c1978f0 con 0x7f4f9c0ff350 2024-04-22T15:40:47.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa30b5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c19b500 con 0x7f4f9c0ffd00 2024-04-22T15:40:47.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa0e51700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 0x7f4f9c196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:47.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa0e51700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 0x7f4f9c196cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39200/0 (socket says 172.21.15.96:39200) 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa0e51700 1 -- 172.21.15.96:0/3568767813 learned_addr learned my addr 172.21.15.96:0/3568767813 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4f9bfff700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f9c0ffd00 0x7f4f9c197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa0e51700 1 -- 172.21.15.96:0/3568767813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f9c0ffd00 msgr2=0x7f4f9c197210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa0e51700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f9c0ffd00 0x7f4f9c197210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa0e51700 1 -- 172.21.15.96:0/3568767813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f900095f0 con 0x7f4f9c0ff350 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4fa0e51700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 0x7f4f9c196cd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000bda0 tx=0x7f4f9000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.209+0000 7f4f99ffb700 1 -- 172.21.15.96:0/3568767813 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f9001d070 con 0x7f4f9c0ff350 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.210+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c19b780 con 0x7f4f9c0ff350 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.210+0000 7f4f99ffb700 1 -- 172.21.15.96:0/3568767813 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f900056e0 con 0x7f4f9c0ff350 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.210+0000 7f4f99ffb700 1 -- 172.21.15.96:0/3568767813 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f90017400 con 0x7f4f9c0ff350 2024-04-22T15:40:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.210+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c19bc70 con 0x7f4f9c0ff350 2024-04-22T15:40:47.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.211+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c109210 con 0x7f4f9c0ff350 2024-04-22T15:40:47.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.211+0000 7f4f99ffb700 1 -- 172.21.15.96:0/3568767813 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4f90003a60 con 0x7f4f9c0ff350 2024-04-22T15:40:47.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.212+0000 7f4f99ffb700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f8c06ecc0 0x7f4f8c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:47.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.212+0000 7f4f99ffb700 1 -- 172.21.15.96:0/3568767813 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4f900903f0 con 0x7f4f9c0ff350 2024-04-22T15:40:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.212+0000 7f4f9bfff700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f8c06ecc0 0x7f4f8c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.212+0000 7f4f9bfff700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f8c06ecc0 0x7f4f8c071180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0fb630 tx=0x7f4f88009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:47.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.216+0000 7f4f99ffb700 1 -- 172.21.15.96:0/3568767813 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4f90059150 con 0x7f4f9c0ff350 2024-04-22T15:40:47.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.424+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f9c197f20 con 0x7f4f8c06ecc0 2024-04-22T15:40:47.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.435+0000 7f4f99ffb700 1 -- 172.21.15.96:0/3568767813 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4f9c197f20 con 0x7f4f8c06ecc0 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (19m) 12m ago 30m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (31m) 12m ago 31m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (29m) 12m ago 29m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (16m) 12m ago 31m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (16m) 12m ago 29m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (19m) 12m ago 30m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (12m) 12m ago 25m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (12m) 12m ago 25m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (12m) 12m ago 25m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:40:47.438 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (25m) 12m ago 24m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (21m) 12m ago 32m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (20m) 12m ago 29m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (17m) 12m ago 32m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (17m) 12m ago 29m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (20m) 12m ago 30m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (20m) 12m ago 29m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (16m) 12m ago 28m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (15m) 12m ago 28m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (15m) 12m ago 27m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (14m) 12m ago 27m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (13m) 12m ago 27m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (13m) 12m ago 26m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:40:47.439 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (19m) 12m ago 30m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:40:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f8c06ecc0 msgr2=0x7f4f8c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f8c06ecc0 0x7f4f8c071180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0fb630 tx=0x7f4f88009190 comp rx=0 tx=0).stop 2024-04-22T15:40:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 msgr2=0x7f4f9c196cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 0x7f4f9c196cd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000bda0 tx=0x7f4f9000bdd0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 shutdown_connections 2024-04-22T15:40:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4f8c06ecc0 0x7f4f8c071180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0ff350 0x7f4f9c196cd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 --2- 172.21.15.96:0/3568767813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4f9c0ffd00 0x7f4f9c197210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 >> 172.21.15.96:0/3568767813 conn(0x7f4f9c075f40 msgr2=0x7f4f9c104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:47.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 shutdown_connections 2024-04-22T15:40:47.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.437+0000 7f4fa30b5700 1 -- 172.21.15.96:0/3568767813 wait complete. 2024-04-22T15:40:47.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 -- 172.21.15.96:0/4007688786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358101810 msgr2=0x7fa358105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 --2- 172.21.15.96:0/4007688786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358101810 0x7fa358105860 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa348009910 tx=0x7fa348009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 -- 172.21.15.96:0/4007688786 shutdown_connections 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 --2- 172.21.15.96:0/4007688786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358101810 0x7fa358105860 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 --2- 172.21.15.96:0/4007688786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa358100e60 0x7fa358101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 -- 172.21.15.96:0/4007688786 >> 172.21.15.96:0/4007688786 conn(0x7fa3580fc710 msgr2=0x7fa3580feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 -- 172.21.15.96:0/4007688786 shutdown_connections 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 -- 172.21.15.96:0/4007688786 wait complete. 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 Processor -- start 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.539+0000 7fa35eae1700 1 -- start start 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35eae1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358100e60 0x7fa3580731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35eae1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa358101810 0x7fa358073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35eae1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa358073c60 con 0x7fa358100e60 2024-04-22T15:40:47.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35eae1700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa358073da0 con 0x7fa358101810 2024-04-22T15:40:47.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35c87d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358100e60 0x7fa3580731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:47.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35c87d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358100e60 0x7fa3580731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39222/0 (socket says 172.21.15.96:39222) 2024-04-22T15:40:47.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35c87d700 1 -- 172.21.15.96:0/3570862325 learned_addr learned my addr 172.21.15.96:0/3570862325 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:47.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35c87d700 1 -- 172.21.15.96:0/3570862325 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa358101810 msgr2=0x7fa358073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35c87d700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa358101810 0x7fa358073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35c87d700 1 -- 172.21.15.96:0/3570862325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3480095f0 con 0x7fa358100e60 2024-04-22T15:40:47.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35c87d700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358100e60 0x7fa3580731e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fa35000ba50 tx=0x7fa35000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:47.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa355ffb700 1 -- 172.21.15.96:0/3570862325 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa35000ca00 con 0x7fa358100e60 2024-04-22T15:40:47.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa355ffb700 1 -- 172.21.15.96:0/3570862325 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa35000cb60 con 0x7fa358100e60 2024-04-22T15:40:47.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa355ffb700 1 -- 172.21.15.96:0/3570862325 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3500075a0 con 0x7fa358100e60 2024-04-22T15:40:47.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.540+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3581a2ec0 con 0x7fa358100e60 2024-04-22T15:40:47.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.541+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3581a3220 con 0x7fa358100e60 2024-04-22T15:40:47.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.541+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3581091b0 con 0x7fa358100e60 2024-04-22T15:40:47.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.542+0000 7fa355ffb700 1 -- 172.21.15.96:0/3570862325 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa35000ccd0 con 0x7fa358100e60 2024-04-22T15:40:47.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.542+0000 7fa355ffb700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa34406ecc0 0x7fa344071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:47.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.542+0000 7fa355ffb700 1 -- 172.21.15.96:0/3570862325 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa350006e40 con 0x7fa358100e60 2024-04-22T15:40:47.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.542+0000 7fa357fff700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa34406ecc0 0x7fa344071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:47.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.543+0000 7fa357fff700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa34406ecc0 0x7fa344071180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa34800b5c0 tx=0x7fa348005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:47.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.547+0000 7fa355ffb700 1 -- 172.21.15.96:0/3570862325 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa350057430 con 0x7fa358100e60 2024-04-22T15:40:47.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.833+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa3580746f0 con 0x7fa358100e60 2024-04-22T15:40:47.836 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:47 smithi096 ceph-mon[112443]: pgmap v567: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.834+0000 7fa355ffb700 1 -- 172.21.15.96:0/3570862325 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7fa35005aaf0 con 0x7fa358100e60 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:40:47.837 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:40:47.838 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:40:47.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.836+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa34406ecc0 msgr2=0x7fa344071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.836+0000 7fa35eae1700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa34406ecc0 0x7fa344071180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa34800b5c0 tx=0x7fa348005e20 comp rx=0 tx=0).stop 2024-04-22T15:40:47.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.836+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358100e60 msgr2=0x7fa3580731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.836+0000 7fa35eae1700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358100e60 0x7fa3580731e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fa35000ba50 tx=0x7fa35000be10 comp rx=0 tx=0).stop 2024-04-22T15:40:47.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.836+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 shutdown_connections 2024-04-22T15:40:47.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.836+0000 7fa35eae1700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa34406ecc0 0x7fa344071180 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.836+0000 7fa35eae1700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa358100e60 0x7fa3580731e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.836+0000 7fa35eae1700 1 --2- 172.21.15.96:0/3570862325 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa358101810 0x7fa358073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.836+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 >> 172.21.15.96:0/3570862325 conn(0x7fa3580fc710 msgr2=0x7fa3580fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:47.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.837+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 shutdown_connections 2024-04-22T15:40:47.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.837+0000 7fa35eae1700 1 -- 172.21.15.96:0/3570862325 wait complete. 2024-04-22T15:40:47.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.940+0000 7fb19e9c3700 1 -- 172.21.15.96:0/1793866002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb198105ab0 msgr2=0x7fb198105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.940+0000 7fb19e9c3700 1 --2- 172.21.15.96:0/1793866002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb198105ab0 0x7fb198105e90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009910 tx=0x7fb18c009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.940+0000 7fb19e9c3700 1 -- 172.21.15.96:0/1793866002 shutdown_connections 2024-04-22T15:40:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.940+0000 7fb19e9c3700 1 --2- 172.21.15.96:0/1793866002 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1980ffab0 0x7fb1980fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.940+0000 7fb19e9c3700 1 --2- 172.21.15.96:0/1793866002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb198105ab0 0x7fb198105e90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.940+0000 7fb19e9c3700 1 -- 172.21.15.96:0/1793866002 >> 172.21.15.96:0/1793866002 conn(0x7fb1980fb5e0 msgr2=0x7fb1980fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.940+0000 7fb19e9c3700 1 -- 172.21.15.96:0/1793866002 shutdown_connections 2024-04-22T15:40:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.940+0000 7fb19e9c3700 1 -- 172.21.15.96:0/1793866002 wait complete. 2024-04-22T15:40:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb19e9c3700 1 Processor -- start 2024-04-22T15:40:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb19e9c3700 1 -- start start 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb19e9c3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1980ffab0 0x7fb19819b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb19e9c3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb198105ab0 0x7fb19819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb19e9c3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19819c330 con 0x7fb1980ffab0 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb19e9c3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb198195910 con 0x7fb198105ab0 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb197fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1980ffab0 0x7fb19819b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb197fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1980ffab0 0x7fb19819b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39242/0 (socket says 172.21.15.96:39242) 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb197fff700 1 -- 172.21.15.96:0/223912926 learned_addr learned my addr 172.21.15.96:0/223912926 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb1977fe700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb198105ab0 0x7fb19819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb197fff700 1 -- 172.21.15.96:0/223912926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb198105ab0 msgr2=0x7fb19819bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb197fff700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb198105ab0 0x7fb19819bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb197fff700 1 -- 172.21.15.96:0/223912926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c0095f0 con 0x7fb1980ffab0 2024-04-22T15:40:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.941+0000 7fb1977fe700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb198105ab0 0x7fb19819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:40:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.942+0000 7fb197fff700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1980ffab0 0x7fb19819b7d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb18c00bd70 tx=0x7fb18c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.942+0000 7fb1957fa700 1 -- 172.21.15.96:0/223912926 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb18c01d070 con 0x7fb1980ffab0 2024-04-22T15:40:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.942+0000 7fb1957fa700 1 -- 172.21.15.96:0/223912926 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb18c00f460 con 0x7fb1980ffab0 2024-04-22T15:40:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.942+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb198195b90 con 0x7fb1980ffab0 2024-04-22T15:40:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.942+0000 7fb1957fa700 1 -- 172.21.15.96:0/223912926 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb18c017630 con 0x7fb1980ffab0 2024-04-22T15:40:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.942+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb198196080 con 0x7fb1980ffab0 2024-04-22T15:40:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.943+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1981080b0 con 0x7fb1980ffab0 2024-04-22T15:40:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.943+0000 7fb1957fa700 1 -- 172.21.15.96:0/223912926 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb18c00f5d0 con 0x7fb1980ffab0 2024-04-22T15:40:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.944+0000 7fb1957fa700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb18006ecc0 0x7fb180071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.944+0000 7fb1957fa700 1 -- 172.21.15.96:0/223912926 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb18c0903d0 con 0x7fb1980ffab0 2024-04-22T15:40:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.944+0000 7fb1977fe700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb18006ecc0 0x7fb180071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:47.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.944+0000 7fb1977fe700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb18006ecc0 0x7fb180071180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb198196ad0 tx=0x7fb188008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:47.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:47.949+0000 7fb1957fa700 1 -- 172.21.15.96:0/223912926 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb18c05cc60 con 0x7fb1980ffab0 2024-04-22T15:40:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:47 smithi196 ceph-mon[81815]: pgmap v567: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:48.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.191+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb198196dd0 con 0x7fb1980ffab0 2024-04-22T15:40:48.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.191+0000 7fb1957fa700 1 -- 172.21.15.96:0/223912926 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fb18c05c7f0 con 0x7fb1980ffab0 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:40:48.195 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:40:48.196 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:40:48.197 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:40:48.198 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:40:48.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.194+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb18006ecc0 msgr2=0x7fb180071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.194+0000 7fb19e9c3700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb18006ecc0 0x7fb180071180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb198196ad0 tx=0x7fb188008040 comp rx=0 tx=0).stop 2024-04-22T15:40:48.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.194+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1980ffab0 msgr2=0x7fb19819b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.194+0000 7fb19e9c3700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1980ffab0 0x7fb19819b7d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb18c00bd70 tx=0x7fb18c00be50 comp rx=0 tx=0).stop 2024-04-22T15:40:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.194+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 shutdown_connections 2024-04-22T15:40:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.194+0000 7fb19e9c3700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb18006ecc0 0x7fb180071180 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.194+0000 7fb19e9c3700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1980ffab0 0x7fb19819b7d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.194+0000 7fb19e9c3700 1 --2- 172.21.15.96:0/223912926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb198105ab0 0x7fb19819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.194+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 >> 172.21.15.96:0/223912926 conn(0x7fb1980fb5e0 msgr2=0x7fb1980fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.195+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 shutdown_connections 2024-04-22T15:40:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.195+0000 7fb19e9c3700 1 -- 172.21.15.96:0/223912926 wait complete. 2024-04-22T15:40:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:40:48.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.300+0000 7f0c371a0700 1 -- 172.21.15.96:0/1222079106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 msgr2=0x7f0c30105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.300+0000 7f0c371a0700 1 --2- 172.21.15.96:0/1222079106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 0x7f0c30105ce0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800b0a0 tx=0x7f0c2800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.300+0000 7f0c371a0700 1 -- 172.21.15.96:0/1222079106 shutdown_connections 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.300+0000 7f0c371a0700 1 --2- 172.21.15.96:0/1222079106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c300ffa80 0x7f0c300fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.300+0000 7f0c371a0700 1 --2- 172.21.15.96:0/1222079106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 0x7f0c30105ce0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.300+0000 7f0c371a0700 1 -- 172.21.15.96:0/1222079106 >> 172.21.15.96:0/1222079106 conn(0x7f0c30075460 msgr2=0x7f0c30075870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.300+0000 7f0c371a0700 1 -- 172.21.15.96:0/1222079106 shutdown_connections 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.300+0000 7f0c371a0700 1 -- 172.21.15.96:0/1222079106 wait complete. 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c371a0700 1 Processor -- start 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c371a0700 1 -- start start 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c371a0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c300ffa80 0x7f0c3019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c371a0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 0x7f0c3019cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c371a0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3019d190 con 0x7f0c30105900 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c371a0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301967c0 con 0x7f0c300ffa80 2024-04-22T15:40:48.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c3599d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 0x7f0c3019cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c3599d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 0x7f0c3019cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39266/0 (socket says 172.21.15.96:39266) 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c3599d700 1 -- 172.21.15.96:0/650404798 learned_addr learned my addr 172.21.15.96:0/650404798 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c3619e700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c300ffa80 0x7f0c3019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c3599d700 1 -- 172.21.15.96:0/650404798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c300ffa80 msgr2=0x7f0c3019c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c3599d700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c300ffa80 0x7f0c3019c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c3599d700 1 -- 172.21.15.96:0/650404798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c28009d00 con 0x7f0c30105900 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c3599d700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 0x7f0c3019cb70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000d8d0 tx=0x7f0c2000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.301+0000 7f0c277fe700 1 -- 172.21.15.96:0/650404798 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c20009940 con 0x7f0c30105900 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.302+0000 7f0c277fe700 1 -- 172.21.15.96:0/650404798 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c20010460 con 0x7f0c30105900 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.302+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c30196aa0 con 0x7f0c30105900 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.302+0000 7f0c277fe700 1 -- 172.21.15.96:0/650404798 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c2000f5f0 con 0x7f0c30105900 2024-04-22T15:40:48.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.302+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c30196ff0 con 0x7f0c30105900 2024-04-22T15:40:48.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.302+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c30108f10 con 0x7f0c30105900 2024-04-22T15:40:48.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.303+0000 7f0c277fe700 1 -- 172.21.15.96:0/650404798 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0c200105d0 con 0x7f0c30105900 2024-04-22T15:40:48.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.303+0000 7f0c277fe700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:48.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.304+0000 7f0c277fe700 1 -- 172.21.15.96:0/650404798 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0c2008eed0 con 0x7f0c30105900 2024-04-22T15:40:48.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.304+0000 7f0c3619e700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:48.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.304+0000 7f0c3619e700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800b830 tx=0x7f0c28006000 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:48.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.308+0000 7f0c277fe700 1 -- 172.21.15.96:0/650404798 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0c2005b760 con 0x7f0c30105900 2024-04-22T15:40:48.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.527+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c3010cb00 con 0x7f0c1c06ecc0 2024-04-22T15:40:48.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.529+0000 7f0c277fe700 1 -- 172.21.15.96:0/650404798 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0c3010cb00 con 0x7f0c1c06ecc0 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:40:48.532 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:40:48.533 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:40:48.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 msgr2=0x7f0c1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800b830 tx=0x7f0c28006000 comp rx=0 tx=0).stop 2024-04-22T15:40:48.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 msgr2=0x7f0c3019cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 0x7f0c3019cb70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000d8d0 tx=0x7f0c2000dc90 comp rx=0 tx=0).stop 2024-04-22T15:40:48.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 shutdown_connections 2024-04-22T15:40:48.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c300ffa80 0x7f0c3019c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 --2- 172.21.15.96:0/650404798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c30105900 0x7f0c3019cb70 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 >> 172.21.15.96:0/650404798 conn(0x7f0c30075460 msgr2=0x7f0c30103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:48.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 shutdown_connections 2024-04-22T15:40:48.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.531+0000 7f0c371a0700 1 -- 172.21.15.96:0/650404798 wait complete. 2024-04-22T15:40:48.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.637+0000 7fcb09ffd700 1 -- 172.21.15.96:0/2493778993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb041017d0 msgr2=0x7fcb04105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.637+0000 7fcb09ffd700 1 --2- 172.21.15.96:0/2493778993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb041017d0 0x7fcb04105820 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009910 tx=0x7fcaf8009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:48.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.637+0000 7fcb09ffd700 1 -- 172.21.15.96:0/2493778993 shutdown_connections 2024-04-22T15:40:48.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.637+0000 7fcb09ffd700 1 --2- 172.21.15.96:0/2493778993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb041017d0 0x7fcb04105820 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.637+0000 7fcb09ffd700 1 --2- 172.21.15.96:0/2493778993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb04100e20 0x7fcb04101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.637+0000 7fcb09ffd700 1 -- 172.21.15.96:0/2493778993 >> 172.21.15.96:0/2493778993 conn(0x7fcb040fc6d0 msgr2=0x7fcb040feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:48.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.637+0000 7fcb09ffd700 1 -- 172.21.15.96:0/2493778993 shutdown_connections 2024-04-22T15:40:48.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.637+0000 7fcb09ffd700 1 -- 172.21.15.96:0/2493778993 wait complete. 2024-04-22T15:40:48.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.637+0000 7fcb09ffd700 1 Processor -- start 2024-04-22T15:40:48.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb09ffd700 1 -- start start 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb09ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04100e20 0x7fcb0419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb09ffd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb041017d0 0x7fcb0419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb09ffd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0419d380 con 0x7fcb04100e20 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb09ffd700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb04196960 con 0x7fcb041017d0 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb037fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04100e20 0x7fcb0419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb037fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04100e20 0x7fcb0419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39286/0 (socket says 172.21.15.96:39286) 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb037fe700 1 -- 172.21.15.96:0/332625199 learned_addr learned my addr 172.21.15.96:0/332625199 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb02ffd700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb041017d0 0x7fcb0419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb037fe700 1 -- 172.21.15.96:0/332625199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb041017d0 msgr2=0x7fcb0419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb037fe700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb041017d0 0x7fcb0419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb037fe700 1 -- 172.21.15.96:0/332625199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf80095f0 con 0x7fcb04100e20 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb037fe700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04100e20 0x7fcb0419c820 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400e920 tx=0x7fcaf400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:48.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb00ff9700 1 -- 172.21.15.96:0/332625199 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaf400c9d0 con 0x7fcb04100e20 2024-04-22T15:40:48.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb00ff9700 1 -- 172.21.15.96:0/332625199 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcaf4010430 con 0x7fcb04100e20 2024-04-22T15:40:48.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.638+0000 7fcb00ff9700 1 -- 172.21.15.96:0/332625199 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaf4018630 con 0x7fcb04100e20 2024-04-22T15:40:48.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.639+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb04196c40 con 0x7fcb04100e20 2024-04-22T15:40:48.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.639+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb04197160 con 0x7fcb04100e20 2024-04-22T15:40:48.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.640+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb04109220 con 0x7fcb04100e20 2024-04-22T15:40:48.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.640+0000 7fcb00ff9700 1 -- 172.21.15.96:0/332625199 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcaf40105a0 con 0x7fcb04100e20 2024-04-22T15:40:48.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.641+0000 7fcb00ff9700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecc0 0x7fcaec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:48.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.641+0000 7fcb02ffd700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecc0 0x7fcaec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:48.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.641+0000 7fcb00ff9700 1 -- 172.21.15.96:0/332625199 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcaf4014070 con 0x7fcb04100e20 2024-04-22T15:40:48.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.641+0000 7fcb02ffd700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecc0 0x7fcaec071180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80059d0 tx=0x7fcaf8005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:48.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.645+0000 7fcb00ff9700 1 -- 172.21.15.96:0/332625199 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcaf4054150 con 0x7fcb04100e20 2024-04-22T15:40:48.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.930+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcb04197e20 con 0x7fcb04100e20 2024-04-22T15:40:48.933 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:48 smithi096 ceph-mon[112443]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:48.934 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:48 smithi096 ceph-mon[112443]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:48.934 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:48 smithi096 ceph-mon[112443]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:48.934 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:48 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3570862325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:40:48.934 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:48 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/223912926' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:40:48.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.930+0000 7fcb00ff9700 1 -- 172.21.15.96:0/332625199 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fcaf405b9b0 con 0x7fcb04100e20 2024-04-22T15:40:48.934 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:40:48.935 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:40:48.935 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:40:48.935 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:40:48.935 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:40:48.935 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:40:48.935 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecc0 msgr2=0x7fcaec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecc0 0x7fcaec071180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80059d0 tx=0x7fcaf8005940 comp rx=0 tx=0).stop 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04100e20 msgr2=0x7fcb0419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04100e20 0x7fcb0419c820 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400e920 tx=0x7fcaf400ec30 comp rx=0 tx=0).stop 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 shutdown_connections 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecc0 0x7fcaec071180 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04100e20 0x7fcb0419c820 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 --2- 172.21.15.96:0/332625199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb041017d0 0x7fcb0419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 >> 172.21.15.96:0/332625199 conn(0x7fcb040fc6d0 msgr2=0x7fcb040fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 shutdown_connections 2024-04-22T15:40:48.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:40:48.933+0000 7fcb09ffd700 1 -- 172.21.15.96:0/332625199 wait complete. 2024-04-22T15:40:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:48 smithi196 ceph-mon[81815]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:48 smithi196 ceph-mon[81815]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:48 smithi196 ceph-mon[81815]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:48 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3570862325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:40:49.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:48 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/223912926' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:40:49.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:49 smithi096 ceph-mon[112443]: pgmap v568: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:49.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:49 smithi096 ceph-mon[112443]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:49.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:49 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/332625199' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:40:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:49 smithi196 ceph-mon[81815]: pgmap v568: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:49 smithi196 ceph-mon[81815]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:49 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/332625199' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:40:51.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:51 smithi096 ceph-mon[112443]: pgmap v569: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:51 smithi196 ceph-mon[81815]: pgmap v569: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:52 smithi096 ceph-mon[112443]: pgmap v570: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:52 smithi196 ceph-mon[81815]: pgmap v570: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:40:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:40:54.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:54 smithi096 ceph-mon[112443]: pgmap v571: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:54 smithi196 ceph-mon[81815]: pgmap v571: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:57 smithi096 ceph-mon[112443]: pgmap v572: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:58.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:57 smithi196 ceph-mon[81815]: pgmap v572: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:40:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:40:59 smithi096 ceph-mon[112443]: pgmap v573: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:40:59 smithi196 ceph-mon[81815]: pgmap v573: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:01.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:01 smithi096 ceph-mon[112443]: pgmap v574: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:01 smithi196 ceph-mon[81815]: pgmap v574: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:03.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:03 smithi096 ceph-mon[112443]: pgmap v575: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:03 smithi196 ceph-mon[81815]: pgmap v575: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:05 smithi096 ceph-mon[112443]: pgmap v576: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:05 smithi196 ceph-mon[81815]: pgmap v576: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:07 smithi096 ceph-mon[112443]: pgmap v577: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:07 smithi196 ceph-mon[81815]: pgmap v577: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:08.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:09.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:09 smithi096 ceph-mon[112443]: pgmap v578: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:09 smithi196 ceph-mon[81815]: pgmap v578: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:11.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:11 smithi096 ceph-mon[112443]: pgmap v579: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:11 smithi196 ceph-mon[81815]: pgmap v579: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:13 smithi096 ceph-mon[112443]: pgmap v580: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:13 smithi196 ceph-mon[81815]: pgmap v580: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:15.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:15 smithi096 ceph-mon[112443]: pgmap v581: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:15 smithi196 ceph-mon[81815]: pgmap v581: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:17.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:17 smithi096 ceph-mon[112443]: pgmap v582: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:17 smithi196 ceph-mon[81815]: pgmap v582: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:19.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.045+0000 7f30eb5ca700 1 -- 172.21.15.96:0/445962619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 msgr2=0x7f30e4100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.045+0000 7f30eb5ca700 1 --2- 172.21.15.96:0/445962619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 0x7f30e4100fd0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009910 tx=0x7f30e0009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:19.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.045+0000 7f30eb5ca700 1 -- 172.21.15.96:0/445962619 shutdown_connections 2024-04-22T15:41:19.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.045+0000 7f30eb5ca700 1 --2- 172.21.15.96:0/445962619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 0x7f30e4100fd0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.045+0000 7f30eb5ca700 1 --2- 172.21.15.96:0/445962619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30e4106c00 0x7f30e4106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.045+0000 7f30eb5ca700 1 -- 172.21.15.96:0/445962619 >> 172.21.15.96:0/445962619 conn(0x7f30e40fc660 msgr2=0x7f30e40fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:19.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.045+0000 7f30eb5ca700 1 -- 172.21.15.96:0/445962619 shutdown_connections 2024-04-22T15:41:19.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.045+0000 7f30eb5ca700 1 -- 172.21.15.96:0/445962619 wait complete. 2024-04-22T15:41:19.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.046+0000 7f30eb5ca700 1 Processor -- start 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.046+0000 7f30eb5ca700 1 -- start start 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.046+0000 7f30eb5ca700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 0x7f30e40731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.046+0000 7f30eb5ca700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30e4106c00 0x7f30e4073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30eb5ca700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e4075160 con 0x7f30e4100b50 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30eb5ca700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e4073c60 con 0x7f30e4106c00 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30e9366700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 0x7f30e40731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30e9366700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 0x7f30e40731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47036/0 (socket says 172.21.15.96:47036) 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30e9366700 1 -- 172.21.15.96:0/3470307706 learned_addr learned my addr 172.21.15.96:0/3470307706 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30e8b65700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30e4106c00 0x7f30e4073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30e9366700 1 -- 172.21.15.96:0/3470307706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30e4106c00 msgr2=0x7f30e4073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30e9366700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30e4106c00 0x7f30e4073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30e9366700 1 -- 172.21.15.96:0/3470307706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e00095f0 con 0x7f30e4100b50 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30e8b65700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30e4106c00 0x7f30e4073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:41:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30e9366700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 0x7f30e40731e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f30d400c930 tx=0x7f30d400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:19.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30da7fc700 1 -- 172.21.15.96:0/3470307706 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d4007ad0 con 0x7f30e4100b50 2024-04-22T15:41:19.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30da7fc700 1 -- 172.21.15.96:0/3470307706 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30d4007c30 con 0x7f30e4100b50 2024-04-22T15:41:19.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e4073f40 con 0x7f30e4100b50 2024-04-22T15:41:19.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30da7fc700 1 -- 172.21.15.96:0/3470307706 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d4018610 con 0x7f30e4100b50 2024-04-22T15:41:19.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.047+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e41a2f20 con 0x7f30e4100b50 2024-04-22T15:41:19.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.048+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e41090b0 con 0x7f30e4100b50 2024-04-22T15:41:19.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.049+0000 7f30da7fc700 1 -- 172.21.15.96:0/3470307706 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f30d400f450 con 0x7f30e4100b50 2024-04-22T15:41:19.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.049+0000 7f30da7fc700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f30d006ecc0 0x7f30d0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:19.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.049+0000 7f30da7fc700 1 -- 172.21.15.96:0/3470307706 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f30d408f0f0 con 0x7f30e4100b50 2024-04-22T15:41:19.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.049+0000 7f30e8b65700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f30d006ecc0 0x7f30d0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:19.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.050+0000 7f30e8b65700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f30d006ecc0 0x7f30d0071180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f30e00059d0 tx=0x7f30e001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:19.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.054+0000 7f30da7fc700 1 -- 172.21.15.96:0/3470307706 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f30d4057e50 con 0x7f30e4100b50 2024-04-22T15:41:19.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.273+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30e4074710 con 0x7f30d006ecc0 2024-04-22T15:41:19.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.274+0000 7f30da7fc700 1 -- 172.21.15.96:0/3470307706 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f30e4074710 con 0x7f30d006ecc0 2024-04-22T15:41:19.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f30d006ecc0 msgr2=0x7f30d0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f30d006ecc0 0x7f30d0071180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f30e00059d0 tx=0x7f30e001a040 comp rx=0 tx=0).stop 2024-04-22T15:41:19.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 msgr2=0x7f30e40731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 0x7f30e40731e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f30d400c930 tx=0x7f30d400ccf0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 shutdown_connections 2024-04-22T15:41:19.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f30d006ecc0 0x7f30d0071180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30e4100b50 0x7f30e40731e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 --2- 172.21.15.96:0/3470307706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30e4106c00 0x7f30e4073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 >> 172.21.15.96:0/3470307706 conn(0x7f30e40fc660 msgr2=0x7f30e40fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:19.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 shutdown_connections 2024-04-22T15:41:19.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.276+0000 7f30eb5ca700 1 -- 172.21.15.96:0/3470307706 wait complete. 2024-04-22T15:41:19.289 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:41:19.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.380+0000 7f4a8c701700 1 -- 172.21.15.96:0/2521087754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 msgr2=0x7f4a840ff710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.380+0000 7f4a8c701700 1 --2- 172.21.15.96:0/2521087754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 0x7f4a840ff710 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009910 tx=0x7f4a80009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:19.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.380+0000 7f4a8c701700 1 -- 172.21.15.96:0/2521087754 shutdown_connections 2024-04-22T15:41:19.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.380+0000 7f4a8c701700 1 --2- 172.21.15.96:0/2521087754 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a840ffce0 0x7f4a84105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.380+0000 7f4a8c701700 1 --2- 172.21.15.96:0/2521087754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 0x7f4a840ff710 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.380+0000 7f4a8c701700 1 -- 172.21.15.96:0/2521087754 >> 172.21.15.96:0/2521087754 conn(0x7f4a84075f30 msgr2=0x7f4a840fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:19.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.380+0000 7f4a8c701700 1 -- 172.21.15.96:0/2521087754 shutdown_connections 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.380+0000 7f4a8c701700 1 -- 172.21.15.96:0/2521087754 wait complete. 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8c701700 1 Processor -- start 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8c701700 1 -- start start 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8c701700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 0x7f4a8419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8c701700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a840ffce0 0x7f4a8419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8c701700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8419d2b0 con 0x7f4a840ff330 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8c701700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84196970 con 0x7f4a840ffce0 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8a49d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 0x7f4a8419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8a49d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 0x7f4a8419c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47052/0 (socket says 172.21.15.96:47052) 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8a49d700 1 -- 172.21.15.96:0/186980220 learned_addr learned my addr 172.21.15.96:0/186980220 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a89c9c700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a840ffce0 0x7f4a8419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8a49d700 1 -- 172.21.15.96:0/186980220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a840ffce0 msgr2=0x7f4a8419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8a49d700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a840ffce0 0x7f4a8419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.381+0000 7f4a8a49d700 1 -- 172.21.15.96:0/186980220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a800095f0 con 0x7f4a840ff330 2024-04-22T15:41:19.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.382+0000 7f4a8a49d700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 0x7f4a8419c7e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4a80006010 tx=0x7f4a8000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:19.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.382+0000 7f4a7b7fe700 1 -- 172.21.15.96:0/186980220 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a8001d070 con 0x7f4a840ff330 2024-04-22T15:41:19.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.382+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a84196bf0 con 0x7f4a840ff330 2024-04-22T15:41:19.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.382+0000 7f4a7b7fe700 1 -- 172.21.15.96:0/186980220 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a8000f460 con 0x7f4a840ff330 2024-04-22T15:41:19.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.382+0000 7f4a7b7fe700 1 -- 172.21.15.96:0/186980220 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a800175f0 con 0x7f4a840ff330 2024-04-22T15:41:19.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.382+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a841970e0 con 0x7f4a840ff330 2024-04-22T15:41:19.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.383+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a841091c0 con 0x7f4a840ff330 2024-04-22T15:41:19.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.383+0000 7f4a7b7fe700 1 -- 172.21.15.96:0/186980220 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4a8000f5d0 con 0x7f4a840ff330 2024-04-22T15:41:19.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.384+0000 7f4a7b7fe700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4a7006ecc0 0x7f4a70071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:19.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.384+0000 7f4a7b7fe700 1 -- 172.21.15.96:0/186980220 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4a80090330 con 0x7f4a840ff330 2024-04-22T15:41:19.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.384+0000 7f4a89c9c700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4a7006ecc0 0x7f4a70071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:19.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.384+0000 7f4a89c9c700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4a7006ecc0 0x7f4a70071180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4a84197ae0 tx=0x7f4a7400a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:19.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.389+0000 7f4a7b7fe700 1 -- 172.21.15.96:0/186980220 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4a8005cbc0 con 0x7f4a840ff330 2024-04-22T15:41:19.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.607+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a84197de0 con 0x7f4a7006ecc0 2024-04-22T15:41:19.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.608+0000 7f4a7b7fe700 1 -- 172.21.15.96:0/186980220 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4a84197de0 con 0x7f4a7006ecc0 2024-04-22T15:41:19.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4a7006ecc0 msgr2=0x7f4a70071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4a7006ecc0 0x7f4a70071180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4a84197ae0 tx=0x7f4a7400a380 comp rx=0 tx=0).stop 2024-04-22T15:41:19.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 msgr2=0x7f4a8419c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 0x7f4a8419c7e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4a80006010 tx=0x7f4a8000bd70 comp rx=0 tx=0).stop 2024-04-22T15:41:19.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 shutdown_connections 2024-04-22T15:41:19.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4a7006ecc0 0x7f4a70071180 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a840ff330 0x7f4a8419c7e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 --2- 172.21.15.96:0/186980220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a840ffce0 0x7f4a8419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 >> 172.21.15.96:0/186980220 conn(0x7f4a84075f30 msgr2=0x7f4a841045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:19.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 shutdown_connections 2024-04-22T15:41:19.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.610+0000 7f4a8c701700 1 -- 172.21.15.96:0/186980220 wait complete. 2024-04-22T15:41:19.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.714+0000 7ffb15d14700 1 -- 172.21.15.96:0/4027117666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 msgr2=0x7ffb10105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.714+0000 7ffb15d14700 1 --2- 172.21.15.96:0/4027117666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 0x7ffb10105f40 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000b0a0 tx=0x7ffb0000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.714+0000 7ffb15d14700 1 -- 172.21.15.96:0/4027117666 shutdown_connections 2024-04-22T15:41:19.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.714+0000 7ffb15d14700 1 --2- 172.21.15.96:0/4027117666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb100ffad0 0x7ffb100fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.714+0000 7ffb15d14700 1 --2- 172.21.15.96:0/4027117666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 0x7ffb10105f40 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.714+0000 7ffb15d14700 1 -- 172.21.15.96:0/4027117666 >> 172.21.15.96:0/4027117666 conn(0x7ffb100fb5e0 msgr2=0x7ffb100fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:19.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:19 smithi096 ceph-mon[112443]: pgmap v583: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:19.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.714+0000 7ffb15d14700 1 -- 172.21.15.96:0/4027117666 shutdown_connections 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.714+0000 7ffb15d14700 1 -- 172.21.15.96:0/4027117666 wait complete. 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.714+0000 7ffb15d14700 1 Processor -- start 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb15d14700 1 -- start start 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb15d14700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb100ffad0 0x7ffb1019c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb15d14700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 0x7ffb1019cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb15d14700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1019d530 con 0x7ffb10105b60 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb15d14700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb10196b10 con 0x7ffb100ffad0 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb0effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 0x7ffb1019cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb0effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 0x7ffb1019cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47068/0 (socket says 172.21.15.96:47068) 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb0effd700 1 -- 172.21.15.96:0/548372725 learned_addr learned my addr 172.21.15.96:0/548372725 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb0f7fe700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb100ffad0 0x7ffb1019c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb0effd700 1 -- 172.21.15.96:0/548372725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb100ffad0 msgr2=0x7ffb1019c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb0effd700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb100ffad0 0x7ffb1019c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb0effd700 1 -- 172.21.15.96:0/548372725 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb00009d00 con 0x7ffb10105b60 2024-04-22T15:41:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb0f7fe700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb100ffad0 0x7ffb1019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:41:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.715+0000 7ffb0effd700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 0x7ffb1019cf10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400b4b0 tx=0x7ffb0400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.716+0000 7ffb0cff9700 1 -- 172.21.15.96:0/548372725 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb04011800 con 0x7ffb10105b60 2024-04-22T15:41:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.716+0000 7ffb0cff9700 1 -- 172.21.15.96:0/548372725 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb04011e20 con 0x7ffb10105b60 2024-04-22T15:41:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.716+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb10196df0 con 0x7ffb10105b60 2024-04-22T15:41:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.716+0000 7ffb0cff9700 1 -- 172.21.15.96:0/548372725 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb0400f2e0 con 0x7ffb10105b60 2024-04-22T15:41:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.716+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb101972f0 con 0x7ffb10105b60 2024-04-22T15:41:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.717+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb101090e0 con 0x7ffb10105b60 2024-04-22T15:41:19.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.717+0000 7ffb0cff9700 1 -- 172.21.15.96:0/548372725 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffb0400f510 con 0x7ffb10105b60 2024-04-22T15:41:19.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.718+0000 7ffb0cff9700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffaf806ecd0 0x7ffaf8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:19.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.718+0000 7ffb0cff9700 1 -- 172.21.15.96:0/548372725 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ffb0408ea40 con 0x7ffb10105b60 2024-04-22T15:41:19.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.718+0000 7ffb0f7fe700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffaf806ecd0 0x7ffaf8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:19.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.718+0000 7ffb0f7fe700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffaf806ecd0 0x7ffaf8071190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000b830 tx=0x7ffb00006000 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:19.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.722+0000 7ffb0cff9700 1 -- 172.21.15.96:0/548372725 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffb0405b340 con 0x7ffb10105b60 2024-04-22T15:41:19.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:19 smithi196 ceph-mon[81815]: pgmap v583: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:19.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.930+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb1010cd50 con 0x7ffaf806ecd0 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.941+0000 7ffb0cff9700 1 -- 172.21.15.96:0/548372725 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ffb1010cd50 con 0x7ffaf806ecd0 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (20m) 13m ago 31m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (31m) 13m ago 31m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (30m) 12m ago 30m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (17m) 13m ago 31m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (17m) 12m ago 30m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (19m) 13m ago 31m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (13m) 13m ago 25m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:41:19.944 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (13m) 13m ago 25m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (12m) 12m ago 25m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (25m) 12m ago 25m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (22m) 13m ago 32m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (21m) 12m ago 30m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (18m) 13m ago 32m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (17m) 12m ago 30m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (20m) 13m ago 31m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (20m) 12m ago 30m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (16m) 13m ago 29m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (16m) 13m ago 28m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (15m) 13m ago 28m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (14m) 12m ago 28m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (14m) 12m ago 27m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (13m) 12m ago 27m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:41:19.945 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (20m) 13m ago 30m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:41:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffaf806ecd0 msgr2=0x7ffaf8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffaf806ecd0 0x7ffaf8071190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000b830 tx=0x7ffb00006000 comp rx=0 tx=0).stop 2024-04-22T15:41:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 msgr2=0x7ffb1019cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 0x7ffb1019cf10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400b4b0 tx=0x7ffb0400b870 comp rx=0 tx=0).stop 2024-04-22T15:41:19.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 shutdown_connections 2024-04-22T15:41:19.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ffaf806ecd0 0x7ffaf8071190 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb100ffad0 0x7ffb1019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 --2- 172.21.15.96:0/548372725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb10105b60 0x7ffb1019cf10 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:19.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 >> 172.21.15.96:0/548372725 conn(0x7ffb100fb5e0 msgr2=0x7ffb10104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:19.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 shutdown_connections 2024-04-22T15:41:19.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:19.943+0000 7ffb15d14700 1 -- 172.21.15.96:0/548372725 wait complete. 2024-04-22T15:41:20.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 -- 172.21.15.96:0/1895699190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 msgr2=0x7f5e1c105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 --2- 172.21.15.96:0/1895699190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 0x7f5e1c105820 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009910 tx=0x7f5e0c009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:20.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 -- 172.21.15.96:0/1895699190 shutdown_connections 2024-04-22T15:41:20.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 --2- 172.21.15.96:0/1895699190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 0x7f5e1c105820 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 --2- 172.21.15.96:0/1895699190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e1c100e20 0x7f5e1c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 -- 172.21.15.96:0/1895699190 >> 172.21.15.96:0/1895699190 conn(0x7f5e1c0fc6d0 msgr2=0x7f5e1c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:20.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 -- 172.21.15.96:0/1895699190 shutdown_connections 2024-04-22T15:41:20.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 -- 172.21.15.96:0/1895699190 wait complete. 2024-04-22T15:41:20.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 Processor -- start 2024-04-22T15:41:20.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.048+0000 7f5e22e5e700 1 -- start start 2024-04-22T15:41:20.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e22e5e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e1c100e20 0x7f5e1c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:20.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e22e5e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 0x7f5e1c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:20.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e22e5e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c073c60 con 0x7f5e1c1017d0 2024-04-22T15:41:20.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e22e5e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c073dd0 con 0x7f5e1c100e20 2024-04-22T15:41:20.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e13fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 0x7f5e1c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:20.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e13fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 0x7f5e1c073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47088/0 (socket says 172.21.15.96:47088) 2024-04-22T15:41:20.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e13fff700 1 -- 172.21.15.96:0/2003935033 learned_addr learned my addr 172.21.15.96:0/2003935033 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:20.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e13fff700 1 -- 172.21.15.96:0/2003935033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e1c100e20 msgr2=0x7f5e1c0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e13fff700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e1c100e20 0x7f5e1c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e13fff700 1 -- 172.21.15.96:0/2003935033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c0095f0 con 0x7f5e1c1017d0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e13fff700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 0x7f5e1c073720 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00bd70 tx=0x7f5e0c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e11ffb700 1 -- 172.21.15.96:0/2003935033 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e0c01d070 con 0x7f5e1c1017d0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e11ffb700 1 -- 172.21.15.96:0/2003935033 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e0c00f460 con 0x7f5e1c1017d0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1c074050 con 0x7f5e1c1017d0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.049+0000 7f5e11ffb700 1 -- 172.21.15.96:0/2003935033 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e0c017610 con 0x7f5e1c1017d0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.050+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1c1a30c0 con 0x7f5e1c1017d0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.051+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1c109170 con 0x7f5e1c1017d0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.051+0000 7f5e11ffb700 1 -- 172.21.15.96:0/2003935033 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5e0c003a60 con 0x7f5e1c1017d0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.051+0000 7f5e11ffb700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5e0406ecc0 0x7f5e04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.051+0000 7f5e11ffb700 1 -- 172.21.15.96:0/2003935033 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5e0c0900e0 con 0x7f5e1c1017d0 2024-04-22T15:41:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.051+0000 7f5e20bfa700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5e0406ecc0 0x7f5e04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:20.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.052+0000 7f5e20bfa700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5e0406ecc0 0x7f5e04071180 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5e180097b0 tx=0x7f5e18006d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:20.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.056+0000 7f5e11ffb700 1 -- 172.21.15.96:0/2003935033 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5e0c058ec0 con 0x7f5e1c1017d0 2024-04-22T15:41:20.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.345+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5e1c102300 con 0x7f5e1c1017d0 2024-04-22T15:41:20.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.346+0000 7f5e11ffb700 1 -- 172.21.15.96:0/2003935033 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f5e0c05c580 con 0x7f5e1c1017d0 2024-04-22T15:41:20.349 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:41:20.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:41:20.349 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:41:20.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:41:20.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:41:20.349 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:41:20.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:41:20.350 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:41:20.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5e0406ecc0 msgr2=0x7f5e04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5e0406ecc0 0x7f5e04071180 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5e180097b0 tx=0x7f5e18006d20 comp rx=0 tx=0).stop 2024-04-22T15:41:20.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 msgr2=0x7f5e1c073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 0x7f5e1c073720 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00bd70 tx=0x7f5e0c00be50 comp rx=0 tx=0).stop 2024-04-22T15:41:20.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 shutdown_connections 2024-04-22T15:41:20.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5e0406ecc0 0x7f5e04071180 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e1c100e20 0x7f5e1c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 --2- 172.21.15.96:0/2003935033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1c1017d0 0x7f5e1c073720 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 >> 172.21.15.96:0/2003935033 conn(0x7f5e1c0fc6d0 msgr2=0x7f5e1c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:20.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 shutdown_connections 2024-04-22T15:41:20.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.349+0000 7f5e22e5e700 1 -- 172.21.15.96:0/2003935033 wait complete. 2024-04-22T15:41:20.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.457+0000 7f4d649fe700 1 -- 172.21.15.96:0/3069506156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600ffa80 msgr2=0x7f4d600fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.457+0000 7f4d649fe700 1 --2- 172.21.15.96:0/3069506156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600ffa80 0x7f4d600fff00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4d54009910 tx=0x7f4d54009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:20.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.457+0000 7f4d649fe700 1 -- 172.21.15.96:0/3069506156 shutdown_connections 2024-04-22T15:41:20.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.457+0000 7f4d649fe700 1 --2- 172.21.15.96:0/3069506156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600ffa80 0x7f4d600fff00 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.457+0000 7f4d649fe700 1 --2- 172.21.15.96:0/3069506156 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d60105900 0x7f4d60105ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.457+0000 7f4d649fe700 1 -- 172.21.15.96:0/3069506156 >> 172.21.15.96:0/3069506156 conn(0x7f4d60075460 msgr2=0x7f4d60075870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:20.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.457+0000 7f4d649fe700 1 -- 172.21.15.96:0/3069506156 shutdown_connections 2024-04-22T15:41:20.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.457+0000 7f4d649fe700 1 -- 172.21.15.96:0/3069506156 wait complete. 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.458+0000 7f4d649fe700 1 Processor -- start 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.458+0000 7f4d649fe700 1 -- start start 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.458+0000 7f4d649fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d600ffa80 0x7f4d60195710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.458+0000 7f4d649fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d60105900 0x7f4d60195c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.458+0000 7f4d649fe700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60069550 con 0x7f4d60105900 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.458+0000 7f4d649fe700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60196190 con 0x7f4d600ffa80 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d5e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d60105900 0x7f4d60195c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d5e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d60105900 0x7f4d60195c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49888/0 (socket says 172.21.15.96:49888) 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d5e59c700 1 -- 172.21.15.96:0/1605695581 learned_addr learned my addr 172.21.15.96:0/1605695581 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d5ed9d700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d600ffa80 0x7f4d60195710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d5e59c700 1 -- 172.21.15.96:0/1605695581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d600ffa80 msgr2=0x7f4d60195710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d5e59c700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d600ffa80 0x7f4d60195710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d5e59c700 1 -- 172.21.15.96:0/1605695581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d540095f0 con 0x7f4d60105900 2024-04-22T15:41:20.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d5e59c700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d60105900 0x7f4d60195c50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4d54006010 tx=0x7f4d5400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d47fff700 1 -- 172.21.15.96:0/1605695581 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d5401d070 con 0x7f4d60105900 2024-04-22T15:41:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d47fff700 1 -- 172.21.15.96:0/1605695581 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d5400f460 con 0x7f4d60105900 2024-04-22T15:41:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d47fff700 1 -- 172.21.15.96:0/1605695581 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d540175f0 con 0x7f4d60105900 2024-04-22T15:41:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d60196410 con 0x7f4d60105900 2024-04-22T15:41:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.459+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d601a2f50 con 0x7f4d60105900 2024-04-22T15:41:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.460+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d60107f20 con 0x7f4d60105900 2024-04-22T15:41:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.461+0000 7f4d47fff700 1 -- 172.21.15.96:0/1605695581 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4d5400f5d0 con 0x7f4d60105900 2024-04-22T15:41:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.461+0000 7f4d47fff700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecc0 0x7f4d48071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.461+0000 7f4d5ed9d700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecc0 0x7f4d48071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.461+0000 7f4d47fff700 1 -- 172.21.15.96:0/1605695581 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4d54090390 con 0x7f4d60105900 2024-04-22T15:41:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.462+0000 7f4d5ed9d700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecc0 0x7f4d48071180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009a10 tx=0x7f4d50009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:20.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.466+0000 7f4d47fff700 1 -- 172.21.15.96:0/1605695581 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4d5405cc20 con 0x7f4d60105900 2024-04-22T15:41:20.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.713+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4d60101060 con 0x7f4d60105900 2024-04-22T15:41:20.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:20 smithi096 ceph-mon[112443]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:20.717 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:20 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2003935033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:41:20.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.714+0000 7f4d47fff700 1 -- 172.21.15.96:0/1605695581 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f4d54005740 con 0x7f4d60105900 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:41:20.718 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:41:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:41:20.720 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:41:20.721 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:41:20.721 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:41:20.721 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:41:20.721 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:41:20.721 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:41:20.721 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:41:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecc0 msgr2=0x7f4d48071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecc0 0x7f4d48071180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009a10 tx=0x7f4d50009190 comp rx=0 tx=0).stop 2024-04-22T15:41:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d60105900 msgr2=0x7f4d60195c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d60105900 0x7f4d60195c50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4d54006010 tx=0x7f4d5400bd70 comp rx=0 tx=0).stop 2024-04-22T15:41:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 shutdown_connections 2024-04-22T15:41:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecc0 0x7f4d48071180 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d600ffa80 0x7f4d60195710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 --2- 172.21.15.96:0/1605695581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d60105900 0x7f4d60195c50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 >> 172.21.15.96:0/1605695581 conn(0x7f4d60075460 msgr2=0x7f4d600ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 shutdown_connections 2024-04-22T15:41:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.717+0000 7f4d649fe700 1 -- 172.21.15.96:0/1605695581 wait complete. 2024-04-22T15:41:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:41:20.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.823+0000 7fab89548700 1 -- 172.21.15.96:0/1290872758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 msgr2=0x7fab84101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.823+0000 7fab89548700 1 --2- 172.21.15.96:0/1290872758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 0x7fab84101200 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fab78009910 tx=0x7fab78009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:20.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.823+0000 7fab89548700 1 -- 172.21.15.96:0/1290872758 shutdown_connections 2024-04-22T15:41:20.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.823+0000 7fab89548700 1 --2- 172.21.15.96:0/1290872758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab841017d0 0x7fab84105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.823+0000 7fab89548700 1 --2- 172.21.15.96:0/1290872758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 0x7fab84101200 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.823+0000 7fab89548700 1 -- 172.21.15.96:0/1290872758 >> 172.21.15.96:0/1290872758 conn(0x7fab840fc6d0 msgr2=0x7fab840feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:20.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.823+0000 7fab89548700 1 -- 172.21.15.96:0/1290872758 shutdown_connections 2024-04-22T15:41:20.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.823+0000 7fab89548700 1 -- 172.21.15.96:0/1290872758 wait complete. 2024-04-22T15:41:20.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab89548700 1 Processor -- start 2024-04-22T15:41:20.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab89548700 1 -- start start 2024-04-22T15:41:20.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab89548700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 0x7fab8419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab89548700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab841017d0 0x7fab8419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab89548700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8419d380 con 0x7fab84100e20 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab89548700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab84196970 con 0x7fab841017d0 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab82ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 0x7fab8419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab82ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 0x7fab8419c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49906/0 (socket says 172.21.15.96:49906) 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab82ffd700 1 -- 172.21.15.96:0/4052948739 learned_addr learned my addr 172.21.15.96:0/4052948739 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab82ffd700 1 -- 172.21.15.96:0/4052948739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab841017d0 msgr2=0x7fab8419cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab827fc700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab841017d0 0x7fab8419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab82ffd700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab841017d0 0x7fab8419cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.824+0000 7fab82ffd700 1 -- 172.21.15.96:0/4052948739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab780095f0 con 0x7fab84100e20 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.825+0000 7fab82ffd700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 0x7fab8419c760 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fab7800bd70 tx=0x7fab7800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.825+0000 7fab6bfff700 1 -- 172.21.15.96:0/4052948739 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab7801d070 con 0x7fab84100e20 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.825+0000 7fab6bfff700 1 -- 172.21.15.96:0/4052948739 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab7800f460 con 0x7fab84100e20 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.825+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab84196bf0 con 0x7fab84100e20 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.825+0000 7fab6bfff700 1 -- 172.21.15.96:0/4052948739 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab78017630 con 0x7fab84100e20 2024-04-22T15:41:20.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.825+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab841970e0 con 0x7fab84100e20 2024-04-22T15:41:20.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.826+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab841091c0 con 0x7fab84100e20 2024-04-22T15:41:20.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.826+0000 7fab6bfff700 1 -- 172.21.15.96:0/4052948739 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fab7800f5d0 con 0x7fab84100e20 2024-04-22T15:41:20.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.827+0000 7fab6bfff700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab6c06ecc0 0x7fab6c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:20.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.827+0000 7fab6bfff700 1 -- 172.21.15.96:0/4052948739 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fab780902e0 con 0x7fab84100e20 2024-04-22T15:41:20.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.827+0000 7fab827fc700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab6c06ecc0 0x7fab6c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:20.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.827+0000 7fab827fc700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab6c06ecc0 0x7fab6c071180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fab84105b80 tx=0x7fab74005cf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:20.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:20.831+0000 7fab6bfff700 1 -- 172.21.15.96:0/4052948739 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fab7805cbf0 con 0x7fab84100e20 2024-04-22T15:41:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:20 smithi196 ceph-mon[81815]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:20 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2003935033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:41:21.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.051+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fab84197f30 con 0x7fab6c06ecc0 2024-04-22T15:41:21.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.052+0000 7fab6bfff700 1 -- 172.21.15.96:0/4052948739 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fab84197f30 con 0x7fab6c06ecc0 2024-04-22T15:41:21.055 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:41:21.055 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:41:21.055 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:41:21.056 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:41:21.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.054+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab6c06ecc0 msgr2=0x7fab6c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:21.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.054+0000 7fab89548700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab6c06ecc0 0x7fab6c071180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fab84105b80 tx=0x7fab74005cf0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.054+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 msgr2=0x7fab8419c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:21.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.054+0000 7fab89548700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 0x7fab8419c760 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fab7800bd70 tx=0x7fab7800be50 comp rx=0 tx=0).stop 2024-04-22T15:41:21.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.054+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 shutdown_connections 2024-04-22T15:41:21.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.054+0000 7fab89548700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab6c06ecc0 0x7fab6c071180 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.054+0000 7fab89548700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab84100e20 0x7fab8419c760 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.054+0000 7fab89548700 1 --2- 172.21.15.96:0/4052948739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab841017d0 0x7fab8419cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.054+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 >> 172.21.15.96:0/4052948739 conn(0x7fab840fc6d0 msgr2=0x7fab840fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:21.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.055+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 shutdown_connections 2024-04-22T15:41:21.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.055+0000 7fab89548700 1 -- 172.21.15.96:0/4052948739 wait complete. 2024-04-22T15:41:21.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.157+0000 7fe1f257b700 1 -- 172.21.15.96:0/528398808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec0ffad0 msgr2=0x7fe1ec0fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:21.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.157+0000 7fe1f257b700 1 --2- 172.21.15.96:0/528398808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec0ffad0 0x7fe1ec0fff50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0009910 tx=0x7fe1e0009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:21.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.158+0000 7fe1f257b700 1 -- 172.21.15.96:0/528398808 shutdown_connections 2024-04-22T15:41:21.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.158+0000 7fe1f257b700 1 --2- 172.21.15.96:0/528398808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec0ffad0 0x7fe1ec0fff50 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.158+0000 7fe1f257b700 1 --2- 172.21.15.96:0/528398808 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1ec105b60 0x7fe1ec105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.158+0000 7fe1f257b700 1 -- 172.21.15.96:0/528398808 >> 172.21.15.96:0/528398808 conn(0x7fe1ec0fb600 msgr2=0x7fe1ec0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.158+0000 7fe1f257b700 1 -- 172.21.15.96:0/528398808 shutdown_connections 2024-04-22T15:41:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.158+0000 7fe1f257b700 1 -- 172.21.15.96:0/528398808 wait complete. 2024-04-22T15:41:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.158+0000 7fe1f257b700 1 Processor -- start 2024-04-22T15:41:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.158+0000 7fe1f257b700 1 -- start start 2024-04-22T15:41:21.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1f257b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1ec0ffad0 0x7fe1ec197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:21.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1f257b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec105b60 0x7fe1ec197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:21.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1f257b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec198440 con 0x7fe1ec105b60 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1f257b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec198580 con 0x7fe1ec0ffad0 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1eb7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec105b60 0x7fe1ec197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1eb7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec105b60 0x7fe1ec197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49926/0 (socket says 172.21.15.96:49926) 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1eb7fe700 1 -- 172.21.15.96:0/22029500 learned_addr learned my addr 172.21.15.96:0/22029500 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1ebfff700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1ec0ffad0 0x7fe1ec197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1eb7fe700 1 -- 172.21.15.96:0/22029500 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1ec0ffad0 msgr2=0x7fe1ec197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1eb7fe700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1ec0ffad0 0x7fe1ec197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1eb7fe700 1 -- 172.21.15.96:0/22029500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e00095f0 con 0x7fe1ec105b60 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1eb7fe700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec105b60 0x7fe1ec197e70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000bd70 tx=0x7fe1e000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1e97fa700 1 -- 172.21.15.96:0/22029500 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e001d070 con 0x7fe1ec105b60 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1e97fa700 1 -- 172.21.15.96:0/22029500 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1e00056e0 con 0x7fe1ec105b60 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1e97fa700 1 -- 172.21.15.96:0/22029500 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e000fe40 con 0x7fe1ec105b60 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ec1a09a0 con 0x7fe1ec105b60 2024-04-22T15:41:21.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.159+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec1a0d00 con 0x7fe1ec105b60 2024-04-22T15:41:21.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.160+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ec108060 con 0x7fe1ec105b60 2024-04-22T15:41:21.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.161+0000 7fe1e97fa700 1 -- 172.21.15.96:0/22029500 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe1e000f680 con 0x7fe1ec105b60 2024-04-22T15:41:21.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.161+0000 7fe1e97fa700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1d406ecc0 0x7fe1d4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:21.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.161+0000 7fe1e97fa700 1 -- 172.21.15.96:0/22029500 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe1e00901c0 con 0x7fe1ec105b60 2024-04-22T15:41:21.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.161+0000 7fe1ebfff700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1d406ecc0 0x7fe1d4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:21.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.162+0000 7fe1ebfff700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1d406ecc0 0x7fe1d4071180 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc004630 tx=0x7fe1dc00b3c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:21.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.166+0000 7fe1e97fa700 1 -- 172.21.15.96:0/22029500 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe1e0058fa0 con 0x7fe1ec105b60 2024-04-22T15:41:21.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.447+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe1ec10ccf0 con 0x7fe1ec105b60 2024-04-22T15:41:21.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.447+0000 7fe1e97fa700 1 -- 172.21.15.96:0/22029500 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fe1e005c660 con 0x7fe1ec105b60 2024-04-22T15:41:21.450 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:41:21.450 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:41:21.451 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:41:21.451 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:41:21.451 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:41:21.451 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:41:21.451 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:41:21.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.449+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1d406ecc0 msgr2=0x7fe1d4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.449+0000 7fe1f257b700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1d406ecc0 0x7fe1d4071180 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc004630 tx=0x7fe1dc00b3c0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.449+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec105b60 msgr2=0x7fe1ec197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.449+0000 7fe1f257b700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec105b60 0x7fe1ec197e70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000bd70 tx=0x7fe1e000bda0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.449+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 shutdown_connections 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.449+0000 7fe1f257b700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1d406ecc0 0x7fe1d4071180 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.449+0000 7fe1f257b700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1ec0ffad0 0x7fe1ec197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.449+0000 7fe1f257b700 1 --2- 172.21.15.96:0/22029500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1ec105b60 0x7fe1ec197e70 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.449+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 >> 172.21.15.96:0/22029500 conn(0x7fe1ec0fb600 msgr2=0x7fe1ec0fc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.450+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 shutdown_connections 2024-04-22T15:41:21.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:21.450+0000 7fe1f257b700 1 -- 172.21.15.96:0/22029500 wait complete. 2024-04-22T15:41:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:21 smithi096 ceph-mon[112443]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:21 smithi096 ceph-mon[112443]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:21 smithi096 ceph-mon[112443]: pgmap v584: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:21 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1605695581' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:41:21.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:21 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/22029500' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:41:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:21 smithi196 ceph-mon[81815]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:21 smithi196 ceph-mon[81815]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:21 smithi196 ceph-mon[81815]: pgmap v584: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:21 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1605695581' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:41:22.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:21 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/22029500' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:41:22.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:22 smithi096 ceph-mon[112443]: from='client.35116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:22 smithi196 ceph-mon[81815]: from='client.35116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:23 smithi096 ceph-mon[112443]: pgmap v585: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:23 smithi196 ceph-mon[81815]: pgmap v585: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:24 smithi196 ceph-mon[81815]: pgmap v586: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:24.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:24 smithi096 ceph-mon[112443]: pgmap v586: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:27.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:27 smithi096 ceph-mon[112443]: pgmap v587: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:27 smithi196 ceph-mon[81815]: pgmap v587: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:29.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:29 smithi196 ceph-mon[81815]: pgmap v588: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:29.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:29 smithi096 ceph-mon[112443]: pgmap v588: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:31 smithi096 ceph-mon[112443]: pgmap v589: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:32.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:31 smithi196 ceph-mon[81815]: pgmap v589: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:33.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:33 smithi096 ceph-mon[112443]: pgmap v590: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:33 smithi196 ceph-mon[81815]: pgmap v590: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:35.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:35 smithi096 ceph-mon[112443]: pgmap v591: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:35 smithi196 ceph-mon[81815]: pgmap v591: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:37.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:37 smithi096 ceph-mon[112443]: pgmap v592: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:37 smithi196 ceph-mon[81815]: pgmap v592: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:38.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:38.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:39.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:39 smithi096 ceph-mon[112443]: pgmap v593: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:39 smithi196 ceph-mon[81815]: pgmap v593: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:41 smithi096 ceph-mon[112443]: pgmap v594: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:41 smithi196 ceph-mon[81815]: pgmap v594: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:43 smithi096 ceph-mon[112443]: pgmap v595: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:43 smithi196 ceph-mon[81815]: pgmap v595: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:45.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:45 smithi096 ceph-mon[112443]: pgmap v596: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:45 smithi196 ceph-mon[81815]: pgmap v596: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:47 smithi096 ceph-mon[112443]: pgmap v597: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:47 smithi196 ceph-mon[81815]: pgmap v597: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:49.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:49 smithi096 ceph-mon[112443]: pgmap v598: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:49 smithi196 ceph-mon[81815]: pgmap v598: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:51.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.566+0000 7f8a1b611700 1 -- 172.21.15.96:0/4225784268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 msgr2=0x7f8a14101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.566+0000 7f8a1b611700 1 --2- 172.21.15.96:0/4225784268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 0x7f8a14101200 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009910 tx=0x7f8a10009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.566+0000 7f8a1b611700 1 -- 172.21.15.96:0/4225784268 shutdown_connections 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.566+0000 7f8a1b611700 1 --2- 172.21.15.96:0/4225784268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a141017d0 0x7f8a14105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.566+0000 7f8a1b611700 1 --2- 172.21.15.96:0/4225784268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 0x7f8a14101200 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.566+0000 7f8a1b611700 1 -- 172.21.15.96:0/4225784268 >> 172.21.15.96:0/4225784268 conn(0x7f8a140fc6d0 msgr2=0x7f8a140feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.566+0000 7f8a1b611700 1 -- 172.21.15.96:0/4225784268 shutdown_connections 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.566+0000 7f8a1b611700 1 -- 172.21.15.96:0/4225784268 wait complete. 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a1b611700 1 Processor -- start 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a1b611700 1 -- start start 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a1b611700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 0x7f8a140731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a1b611700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a141017d0 0x7f8a14073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a1b611700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a14073c60 con 0x7f8a14100e20 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a1b611700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a14073da0 con 0x7f8a141017d0 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a193ad700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 0x7f8a140731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:51.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a193ad700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 0x7f8a140731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59364/0 (socket says 172.21.15.96:59364) 2024-04-22T15:41:51.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a193ad700 1 -- 172.21.15.96:0/496276550 learned_addr learned my addr 172.21.15.96:0/496276550 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:51.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.567+0000 7f8a18bac700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a141017d0 0x7f8a14073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:51.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.568+0000 7f8a193ad700 1 -- 172.21.15.96:0/496276550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a141017d0 msgr2=0x7f8a14073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:51.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.568+0000 7f8a193ad700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a141017d0 0x7f8a14073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.568+0000 7f8a193ad700 1 -- 172.21.15.96:0/496276550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a100095f0 con 0x7f8a14100e20 2024-04-22T15:41:51.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.568+0000 7f8a193ad700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 0x7f8a140731e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000bd70 tx=0x7f8a1000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:51.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.568+0000 7f8a0a7fc700 1 -- 172.21.15.96:0/496276550 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a1001d070 con 0x7f8a14100e20 2024-04-22T15:41:51.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.568+0000 7f8a0a7fc700 1 -- 172.21.15.96:0/496276550 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a100056e0 con 0x7f8a14100e20 2024-04-22T15:41:51.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.568+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a14074020 con 0x7f8a14100e20 2024-04-22T15:41:51.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.568+0000 7f8a0a7fc700 1 -- 172.21.15.96:0/496276550 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a1000fe40 con 0x7f8a14100e20 2024-04-22T15:41:51.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.568+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a141a2f80 con 0x7f8a14100e20 2024-04-22T15:41:51.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.569+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a141091c0 con 0x7f8a14100e20 2024-04-22T15:41:51.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.570+0000 7f8a0a7fc700 1 -- 172.21.15.96:0/496276550 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8a1000f680 con 0x7f8a14100e20 2024-04-22T15:41:51.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.570+0000 7f8a0a7fc700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8a0006ec60 0x7f8a00071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:51.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.570+0000 7f8a0a7fc700 1 -- 172.21.15.96:0/496276550 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8a10090290 con 0x7f8a14100e20 2024-04-22T15:41:51.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.570+0000 7f8a18bac700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8a0006ec60 0x7f8a00071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:51.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.570+0000 7f8a18bac700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8a0006ec60 0x7f8a00071120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a140748c0 tx=0x7f8a0400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:51.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.575+0000 7f8a0a7fc700 1 -- 172.21.15.96:0/496276550 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8a10058ff0 con 0x7f8a14100e20 2024-04-22T15:41:51.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.798+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a1410ddf0 con 0x7f8a0006ec60 2024-04-22T15:41:51.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.799+0000 7f8a0a7fc700 1 -- 172.21.15.96:0/496276550 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8a1410ddf0 con 0x7f8a0006ec60 2024-04-22T15:41:51.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.801+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8a0006ec60 msgr2=0x7f8a00071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.801+0000 7f8a1b611700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8a0006ec60 0x7f8a00071120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a140748c0 tx=0x7f8a0400b410 comp rx=0 tx=0).stop 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.801+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 msgr2=0x7f8a140731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.801+0000 7f8a1b611700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 0x7f8a140731e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000bd70 tx=0x7f8a1000bda0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.801+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 shutdown_connections 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.801+0000 7f8a1b611700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8a0006ec60 0x7f8a00071120 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.801+0000 7f8a1b611700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a14100e20 0x7f8a140731e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.801+0000 7f8a1b611700 1 --2- 172.21.15.96:0/496276550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a141017d0 0x7f8a14073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.801+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 >> 172.21.15.96:0/496276550 conn(0x7f8a140fc6d0 msgr2=0x7f8a140fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.802+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 shutdown_connections 2024-04-22T15:41:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.802+0000 7f8a1b611700 1 -- 172.21.15.96:0/496276550 wait complete. 2024-04-22T15:41:51.815 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:41:51.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.906+0000 7efd0359e700 1 -- 172.21.15.96:0/630436233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd041058f0 msgr2=0x7efd04105cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:51.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.906+0000 7efd0359e700 1 --2- 172.21.15.96:0/630436233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd041058f0 0x7efd04105cd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00b0a0 tx=0x7efcfc00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.909 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:51 smithi096 ceph-mon[112443]: pgmap v599: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:51.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.906+0000 7efd0359e700 1 -- 172.21.15.96:0/630436233 shutdown_connections 2024-04-22T15:41:51.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.906+0000 7efd0359e700 1 --2- 172.21.15.96:0/630436233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd040ff8f0 0x7efd040ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.906+0000 7efd0359e700 1 --2- 172.21.15.96:0/630436233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd041058f0 0x7efd04105cd0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.906+0000 7efd0359e700 1 -- 172.21.15.96:0/630436233 >> 172.21.15.96:0/630436233 conn(0x7efd040fb440 msgr2=0x7efd040fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.906+0000 7efd0359e700 1 -- 172.21.15.96:0/630436233 shutdown_connections 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.906+0000 7efd0359e700 1 -- 172.21.15.96:0/630436233 wait complete. 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.907+0000 7efd0359e700 1 Processor -- start 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.907+0000 7efd0359e700 1 -- start start 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0359e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd040ff8f0 0x7efd041977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0359e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd041058f0 0x7efd04197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0359e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd04198230 con 0x7efd040ff8f0 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0359e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd04198370 con 0x7efd041058f0 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd040ff8f0 0x7efd041977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd040ff8f0 0x7efd041977b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59376/0 (socket says 172.21.15.96:59376) 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0259c700 1 -- 172.21.15.96:0/818005578 learned_addr learned my addr 172.21.15.96:0/818005578 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd01d9b700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd041058f0 0x7efd04197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:51.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0259c700 1 -- 172.21.15.96:0/818005578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd041058f0 msgr2=0x7efd04197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:51.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0259c700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd041058f0 0x7efd04197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0259c700 1 -- 172.21.15.96:0/818005578 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc009d00 con 0x7efd040ff8f0 2024-04-22T15:41:51.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efd0259c700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd040ff8f0 0x7efd041977b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7efcfc000f80 tx=0x7efcfc007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:51.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efcf37fe700 1 -- 172.21.15.96:0/818005578 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcfc00e040 con 0x7efd040ff8f0 2024-04-22T15:41:51.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efcf37fe700 1 -- 172.21.15.96:0/818005578 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efcfc0096a0 con 0x7efd040ff8f0 2024-04-22T15:41:51.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.908+0000 7efcf37fe700 1 -- 172.21.15.96:0/818005578 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcfc01d6b0 con 0x7efd040ff8f0 2024-04-22T15:41:51.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.909+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd041985f0 con 0x7efd040ff8f0 2024-04-22T15:41:51.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.909+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd041a0b00 con 0x7efd040ff8f0 2024-04-22T15:41:51.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.910+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd04107ef0 con 0x7efd040ff8f0 2024-04-22T15:41:51.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.910+0000 7efcf37fe700 1 -- 172.21.15.96:0/818005578 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efcfc003c10 con 0x7efd040ff8f0 2024-04-22T15:41:51.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.911+0000 7efcf37fe700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efcec06ecc0 0x7efcec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:51.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.911+0000 7efcf37fe700 1 -- 172.21.15.96:0/818005578 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7efcfc090090 con 0x7efd040ff8f0 2024-04-22T15:41:51.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.911+0000 7efd01d9b700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efcec06ecc0 0x7efcec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:51.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.911+0000 7efd01d9b700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efcec06ecc0 0x7efcec071180 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7efd04069610 tx=0x7efcf800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:51.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:51.916+0000 7efcf37fe700 1 -- 172.21.15.96:0/818005578 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efcfc058df0 con 0x7efd040ff8f0 2024-04-22T15:41:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:51 smithi196 ceph-mon[81815]: pgmap v599: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:52.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.135+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd0410caf0 con 0x7efcec06ecc0 2024-04-22T15:41:52.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.136+0000 7efcf37fe700 1 -- 172.21.15.96:0/818005578 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efd0410caf0 con 0x7efcec06ecc0 2024-04-22T15:41:52.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efcec06ecc0 msgr2=0x7efcec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efcec06ecc0 0x7efcec071180 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7efd04069610 tx=0x7efcf800b410 comp rx=0 tx=0).stop 2024-04-22T15:41:52.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd040ff8f0 msgr2=0x7efd041977b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd040ff8f0 0x7efd041977b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7efcfc000f80 tx=0x7efcfc007b20 comp rx=0 tx=0).stop 2024-04-22T15:41:52.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 shutdown_connections 2024-04-22T15:41:52.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7efcec06ecc0 0x7efcec071180 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd040ff8f0 0x7efd041977b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 --2- 172.21.15.96:0/818005578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd041058f0 0x7efd04197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 >> 172.21.15.96:0/818005578 conn(0x7efd040fb440 msgr2=0x7efd040fc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:52.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 shutdown_connections 2024-04-22T15:41:52.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.138+0000 7efd0359e700 1 -- 172.21.15.96:0/818005578 wait complete. 2024-04-22T15:41:52.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.242+0000 7f437a77c700 1 -- 172.21.15.96:0/444471200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43740ffab0 msgr2=0x7f43740fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.242+0000 7f437a77c700 1 --2- 172.21.15.96:0/444471200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43740ffab0 0x7f43740fff30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4368009910 tx=0x7f4368009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:52.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.242+0000 7f437a77c700 1 -- 172.21.15.96:0/444471200 shutdown_connections 2024-04-22T15:41:52.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.242+0000 7f437a77c700 1 --2- 172.21.15.96:0/444471200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43740ffab0 0x7f43740fff30 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.242+0000 7f437a77c700 1 --2- 172.21.15.96:0/444471200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4374105ab0 0x7f4374105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.242+0000 7f437a77c700 1 -- 172.21.15.96:0/444471200 >> 172.21.15.96:0/444471200 conn(0x7f43740fb5e0 msgr2=0x7f43740fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:52.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.242+0000 7f437a77c700 1 -- 172.21.15.96:0/444471200 shutdown_connections 2024-04-22T15:41:52.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.242+0000 7f437a77c700 1 -- 172.21.15.96:0/444471200 wait complete. 2024-04-22T15:41:52.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f437a77c700 1 Processor -- start 2024-04-22T15:41:52.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f437a77c700 1 -- start start 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f437a77c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43740ffab0 0x7f4374197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f437a77c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4374105ab0 0x7f4374197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f437a77c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4374198560 con 0x7f4374105ab0 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f437a77c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43741986a0 con 0x7f43740ffab0 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f43737fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4374105ab0 0x7f4374197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f43737fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4374105ab0 0x7f4374197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59390/0 (socket says 172.21.15.96:59390) 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f43737fe700 1 -- 172.21.15.96:0/3803628215 learned_addr learned my addr 172.21.15.96:0/3803628215 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f43737fe700 1 -- 172.21.15.96:0/3803628215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43740ffab0 msgr2=0x7f4374197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f4373fff700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43740ffab0 0x7f4374197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f43737fe700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43740ffab0 0x7f4374197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.243+0000 7f43737fe700 1 -- 172.21.15.96:0/3803628215 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43680095f0 con 0x7f4374105ab0 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.244+0000 7f43737fe700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4374105ab0 0x7f4374197f90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f43680092e0 tx=0x7f436800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:52.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.244+0000 7f43717fa700 1 -- 172.21.15.96:0/3803628215 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f436801d070 con 0x7f4374105ab0 2024-04-22T15:41:52.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.244+0000 7f43717fa700 1 -- 172.21.15.96:0/3803628215 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f436800f460 con 0x7f4374105ab0 2024-04-22T15:41:52.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.244+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43741a0ac0 con 0x7f4374105ab0 2024-04-22T15:41:52.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.244+0000 7f43717fa700 1 -- 172.21.15.96:0/3803628215 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4368017630 con 0x7f4374105ab0 2024-04-22T15:41:52.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.244+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43741a0e20 con 0x7f4374105ab0 2024-04-22T15:41:52.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.245+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43741080d0 con 0x7f4374105ab0 2024-04-22T15:41:52.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.245+0000 7f43717fa700 1 -- 172.21.15.96:0/3803628215 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f436800f5d0 con 0x7f4374105ab0 2024-04-22T15:41:52.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.246+0000 7f43717fa700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f435c06ecd0 0x7f435c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:52.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.246+0000 7f43717fa700 1 -- 172.21.15.96:0/3803628215 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f43680903c0 con 0x7f4374105ab0 2024-04-22T15:41:52.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.246+0000 7f4373fff700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f435c06ecd0 0x7f435c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:52.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.246+0000 7f4373fff700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f435c06ecd0 0x7f435c071190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4364006fd0 tx=0x7f4364008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:52.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.251+0000 7f43717fa700 1 -- 172.21.15.96:0/3803628215 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4368059120 con 0x7f4374105ab0 2024-04-22T15:41:52.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.458+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f437410cd30 con 0x7f435c06ecd0 2024-04-22T15:41:52.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.469+0000 7f43717fa700 1 -- 172.21.15.96:0/3803628215 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f437410cd30 con 0x7f435c06ecd0 2024-04-22T15:41:52.471 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (20m) 13m ago 31m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (32m) 13m ago 32m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (31m) 13m ago 31m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (17m) 13m ago 32m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (17m) 13m ago 30m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (20m) 13m ago 31m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (13m) 13m ago 26m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (14m) 13m ago 26m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (13m) 13m ago 26m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (26m) 13m ago 26m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (22m) 13m ago 33m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (21m) 13m ago 30m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (18m) 13m ago 33m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:41:52.472 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (18m) 13m ago 30m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:41:52.473 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (21m) 13m ago 32m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:41:52.473 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (21m) 13m ago 30m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:41:52.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (17m) 13m ago 29m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:41:52.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (16m) 13m ago 29m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:41:52.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (16m) 13m ago 28m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:41:52.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (15m) 13m ago 28m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:41:52.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (14m) 13m ago 28m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:41:52.473 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (14m) 13m ago 27m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:41:52.473 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (20m) 13m ago 31m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:41:52.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f435c06ecd0 msgr2=0x7f435c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f435c06ecd0 0x7f435c071190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4364006fd0 tx=0x7f4364008040 comp rx=0 tx=0).stop 2024-04-22T15:41:52.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4374105ab0 msgr2=0x7f4374197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4374105ab0 0x7f4374197f90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f43680092e0 tx=0x7f436800be20 comp rx=0 tx=0).stop 2024-04-22T15:41:52.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 shutdown_connections 2024-04-22T15:41:52.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f435c06ecd0 0x7f435c071190 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43740ffab0 0x7f4374197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 --2- 172.21.15.96:0/3803628215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4374105ab0 0x7f4374197f90 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 >> 172.21.15.96:0/3803628215 conn(0x7f43740fb5e0 msgr2=0x7f43740fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:52.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 shutdown_connections 2024-04-22T15:41:52.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.471+0000 7f437a77c700 1 -- 172.21.15.96:0/3803628215 wait complete. 2024-04-22T15:41:52.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.573+0000 7f47593df700 1 -- 172.21.15.96:0/183987073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4754105aa0 msgr2=0x7f4754105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.573+0000 7f47593df700 1 --2- 172.21.15.96:0/183987073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4754105aa0 0x7f4754105e80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4744009910 tx=0x7f4744009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:52.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.573+0000 7f47593df700 1 -- 172.21.15.96:0/183987073 shutdown_connections 2024-04-22T15:41:52.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.573+0000 7f47593df700 1 --2- 172.21.15.96:0/183987073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47540ffaa0 0x7f47540fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.573+0000 7f47593df700 1 --2- 172.21.15.96:0/183987073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4754105aa0 0x7f4754105e80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.573+0000 7f47593df700 1 -- 172.21.15.96:0/183987073 >> 172.21.15.96:0/183987073 conn(0x7f4754076740 msgr2=0x7f47540fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:52.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.573+0000 7f47593df700 1 -- 172.21.15.96:0/183987073 shutdown_connections 2024-04-22T15:41:52.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.573+0000 7f47593df700 1 -- 172.21.15.96:0/183987073 wait complete. 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.573+0000 7f47593df700 1 Processor -- start 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f47593df700 1 -- start start 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f47593df700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47540ffaa0 0x7f4754198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f47593df700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754105aa0 0x7f47541992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f47593df700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47541999b0 con 0x7f47540ffaa0 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f47593df700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475419d680 con 0x7f4754105aa0 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f4752ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47540ffaa0 0x7f4754198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f4752ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47540ffaa0 0x7f4754198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59418/0 (socket says 172.21.15.96:59418) 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f4752ffd700 1 -- 172.21.15.96:0/3808429547 learned_addr learned my addr 172.21.15.96:0/3808429547 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f47527fc700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754105aa0 0x7f47541992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:52.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f4752ffd700 1 -- 172.21.15.96:0/3808429547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754105aa0 msgr2=0x7f47541992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f4752ffd700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754105aa0 0x7f47541992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f4752ffd700 1 -- 172.21.15.96:0/3808429547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47440095f0 con 0x7f47540ffaa0 2024-04-22T15:41:52.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.574+0000 7f4752ffd700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47540ffaa0 0x7f4754198d90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4744006010 tx=0x7f474400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:52.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.575+0000 7f473bfff700 1 -- 172.21.15.96:0/3808429547 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f474401d070 con 0x7f47540ffaa0 2024-04-22T15:41:52.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.575+0000 7f473bfff700 1 -- 172.21.15.96:0/3808429547 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f474400f460 con 0x7f47540ffaa0 2024-04-22T15:41:52.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.575+0000 7f473bfff700 1 -- 172.21.15.96:0/3808429547 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4744017630 con 0x7f47540ffaa0 2024-04-22T15:41:52.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.575+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475419d900 con 0x7f47540ffaa0 2024-04-22T15:41:52.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.575+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475419ddf0 con 0x7f47540ffaa0 2024-04-22T15:41:52.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.576+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47541011e0 con 0x7f47540ffaa0 2024-04-22T15:41:52.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.577+0000 7f473bfff700 1 -- 172.21.15.96:0/3808429547 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f474400f5d0 con 0x7f47540ffaa0 2024-04-22T15:41:52.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.577+0000 7f473bfff700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f473c06ecd0 0x7f473c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:52.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.577+0000 7f473bfff700 1 -- 172.21.15.96:0/3808429547 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f47440902f0 con 0x7f47540ffaa0 2024-04-22T15:41:52.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.577+0000 7f47527fc700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f473c06ecd0 0x7f473c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:52.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.578+0000 7f47527fc700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f473c06ecd0 0x7f473c071190 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4748006fd0 tx=0x7f4748008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:52.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.582+0000 7f473bfff700 1 -- 172.21.15.96:0/3808429547 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4744059190 con 0x7f47540ffaa0 2024-04-22T15:41:52.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.865+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f475410cd20 con 0x7f47540ffaa0 2024-04-22T15:41:52.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.866+0000 7f473bfff700 1 -- 172.21.15.96:0/3808429547 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f474405c850 con 0x7f47540ffaa0 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:41:52.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:41:52.870 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:41:52.870 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:41:52.870 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:41:52.870 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:41:52.870 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:41:52.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f473c06ecd0 msgr2=0x7f473c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f473c06ecd0 0x7f473c071190 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4748006fd0 tx=0x7f4748008040 comp rx=0 tx=0).stop 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47540ffaa0 msgr2=0x7f4754198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47540ffaa0 0x7f4754198d90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4744006010 tx=0x7f474400be20 comp rx=0 tx=0).stop 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 shutdown_connections 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f473c06ecd0 0x7f473c071190 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47540ffaa0 0x7f4754198d90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 --2- 172.21.15.96:0/3808429547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754105aa0 0x7f47541992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 >> 172.21.15.96:0/3808429547 conn(0x7f4754076740 msgr2=0x7f4754103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 shutdown_connections 2024-04-22T15:41:52.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.868+0000 7f47593df700 1 -- 172.21.15.96:0/3808429547 wait complete. 2024-04-22T15:41:52.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.971+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2362100375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100dd0 msgr2=0x7fb2a01047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.971+0000 7fb2a55ec700 1 --2- 172.21.15.96:0/2362100375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100dd0 0x7fb2a01047a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb290009910 tx=0x7fb290009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:52.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.971+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2362100375 shutdown_connections 2024-04-22T15:41:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.971+0000 7fb2a55ec700 1 --2- 172.21.15.96:0/2362100375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100dd0 0x7fb2a01047a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.971+0000 7fb2a55ec700 1 --2- 172.21.15.96:0/2362100375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2a0100420 0x7fb2a0100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.971+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2362100375 >> 172.21.15.96:0/2362100375 conn(0x7fb2a0075f40 msgr2=0x7fb2a00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.971+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2362100375 shutdown_connections 2024-04-22T15:41:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.971+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2362100375 wait complete. 2024-04-22T15:41:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb2a55ec700 1 Processor -- start 2024-04-22T15:41:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb2a55ec700 1 -- start start 2024-04-22T15:41:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb2a55ec700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100420 0x7fb2a0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb2a55ec700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2a0100dd0 0x7fb2a0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb2a55ec700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a01985d0 con 0x7fb2a0100420 2024-04-22T15:41:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb2a55ec700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a0198710 con 0x7fb2a0100dd0 2024-04-22T15:41:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb29effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100420 0x7fb2a0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb29effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100420 0x7fb2a0197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59432/0 (socket says 172.21.15.96:59432) 2024-04-22T15:41:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb29effd700 1 -- 172.21.15.96:0/2905181571 learned_addr learned my addr 172.21.15.96:0/2905181571 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.972+0000 7fb29e7fc700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2a0100dd0 0x7fb2a0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:52.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb29effd700 1 -- 172.21.15.96:0/2905181571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2a0100dd0 msgr2=0x7fb2a0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:52.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb29effd700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2a0100dd0 0x7fb2a0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:52.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb29effd700 1 -- 172.21.15.96:0/2905181571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2900095f0 con 0x7fb2a0100420 2024-04-22T15:41:52.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb29e7fc700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2a0100dd0 0x7fb2a0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:41:52.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb29effd700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100420 0x7fb2a0197ac0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb29400b4b0 tx=0x7fb29400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:52.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb287fff700 1 -- 172.21.15.96:0/2905181571 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb294011800 con 0x7fb2a0100420 2024-04-22T15:41:52.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a01a0b30 con 0x7fb2a0100420 2024-04-22T15:41:52.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb287fff700 1 -- 172.21.15.96:0/2905181571 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb294011e20 con 0x7fb2a0100420 2024-04-22T15:41:52.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb287fff700 1 -- 172.21.15.96:0/2905181571 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb29400f2e0 con 0x7fb2a0100420 2024-04-22T15:41:52.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.973+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a01a0f40 con 0x7fb2a0100420 2024-04-22T15:41:52.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.974+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a0108110 con 0x7fb2a0100420 2024-04-22T15:41:52.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.974+0000 7fb287fff700 1 -- 172.21.15.96:0/2905181571 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb29400f510 con 0x7fb2a0100420 2024-04-22T15:41:52.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.975+0000 7fb287fff700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb28806ecc0 0x7fb288071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:52.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.975+0000 7fb287fff700 1 -- 172.21.15.96:0/2905181571 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb29408eab0 con 0x7fb2a0100420 2024-04-22T15:41:52.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.975+0000 7fb29e7fc700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb28806ecc0 0x7fb288071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:52.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.975+0000 7fb29e7fc700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb28806ecc0 0x7fb288071180 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb290006010 tx=0x7fb29001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:52.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:52.980+0000 7fb287fff700 1 -- 172.21.15.96:0/2905181571 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb294057880 con 0x7fb2a0100420 2024-04-22T15:41:53.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.224+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb2a00ff6f0 con 0x7fb2a0100420 2024-04-22T15:41:53.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.224+0000 7fb287fff700 1 -- 172.21.15.96:0/2905181571 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fb29405af40 con 0x7fb2a0100420 2024-04-22T15:41:53.226 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:41:53.226 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:41:53.226 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:41:53.227 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:41:53.228 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:41:53.229 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:41:53.229 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:41:53.229 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:41:53.229 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:41:53.229 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:41:53.229 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:41:53.229 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:41:53.229 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:41:53.229 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:41:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb28806ecc0 msgr2=0x7fb288071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb28806ecc0 0x7fb288071180 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb290006010 tx=0x7fb29001a040 comp rx=0 tx=0).stop 2024-04-22T15:41:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100420 msgr2=0x7fb2a0197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100420 0x7fb2a0197ac0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb29400b4b0 tx=0x7fb29400b870 comp rx=0 tx=0).stop 2024-04-22T15:41:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 shutdown_connections 2024-04-22T15:41:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb28806ecc0 0x7fb288071180 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2a0100420 0x7fb2a0197ac0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 --2- 172.21.15.96:0/2905181571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2a0100dd0 0x7fb2a0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 >> 172.21.15.96:0/2905181571 conn(0x7fb2a0075f40 msgr2=0x7fb2a0103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 shutdown_connections 2024-04-22T15:41:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.227+0000 7fb2a55ec700 1 -- 172.21.15.96:0/2905181571 wait complete. 2024-04-22T15:41:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:41:53.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.333+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1113177590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 msgr2=0x7f2acc105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.333+0000 7f2ad0c77700 1 --2- 172.21.15.96:0/1113177590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 0x7f2acc105e80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009910 tx=0x7f2ac0009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:53.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.333+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1113177590 shutdown_connections 2024-04-22T15:41:53.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.333+0000 7f2ad0c77700 1 --2- 172.21.15.96:0/1113177590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2acc0ffaa0 0x7f2acc0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.333+0000 7f2ad0c77700 1 --2- 172.21.15.96:0/1113177590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 0x7f2acc105e80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.333+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1113177590 >> 172.21.15.96:0/1113177590 conn(0x7f2acc076740 msgr2=0x7f2acc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:53.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.333+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1113177590 shutdown_connections 2024-04-22T15:41:53.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.333+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1113177590 wait complete. 2024-04-22T15:41:53.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2ad0c77700 1 Processor -- start 2024-04-22T15:41:53.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2ad0c77700 1 -- start start 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2ad0c77700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2acc0ffaa0 0x7f2acc1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2ad0c77700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 0x7f2acc197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2ad0c77700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc1984b0 con 0x7f2acc105aa0 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2ad0c77700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc1985f0 con 0x7f2acc0ffaa0 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2ac9d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 0x7f2acc197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2ac9d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 0x7f2acc197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59456/0 (socket says 172.21.15.96:59456) 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2ac9d9b700 1 -- 172.21.15.96:0/1614462528 learned_addr learned my addr 172.21.15.96:0/1614462528 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.334+0000 7f2aca59c700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2acc0ffaa0 0x7f2acc1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.335+0000 7f2ac9d9b700 1 -- 172.21.15.96:0/1614462528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2acc0ffaa0 msgr2=0x7f2acc1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.335+0000 7f2ac9d9b700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2acc0ffaa0 0x7f2acc1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.335+0000 7f2ac9d9b700 1 -- 172.21.15.96:0/1614462528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac00095f0 con 0x7f2acc105aa0 2024-04-22T15:41:53.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.335+0000 7f2ac9d9b700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 0x7f2acc197ee0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00d8d0 tx=0x7f2abc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:53.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.335+0000 7f2abb7fe700 1 -- 172.21.15.96:0/1614462528 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2abc0098a0 con 0x7f2acc105aa0 2024-04-22T15:41:53.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.335+0000 7f2abb7fe700 1 -- 172.21.15.96:0/1614462528 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2abc010460 con 0x7f2acc105aa0 2024-04-22T15:41:53.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.335+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc1a0a10 con 0x7f2acc105aa0 2024-04-22T15:41:53.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.335+0000 7f2abb7fe700 1 -- 172.21.15.96:0/1614462528 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2abc00f5f0 con 0x7f2acc105aa0 2024-04-22T15:41:53.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.335+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc1a0e20 con 0x7f2acc105aa0 2024-04-22T15:41:53.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.336+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc1080a0 con 0x7f2acc105aa0 2024-04-22T15:41:53.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.336+0000 7f2abb7fe700 1 -- 172.21.15.96:0/1614462528 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2abc009a00 con 0x7f2acc105aa0 2024-04-22T15:41:53.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.337+0000 7f2abb7fe700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2ab406ecc0 0x7f2ab4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:53.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.337+0000 7f2abb7fe700 1 -- 172.21.15.96:0/1614462528 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2abc08ec90 con 0x7f2acc105aa0 2024-04-22T15:41:53.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.337+0000 7f2aca59c700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2ab406ecc0 0x7f2ab4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:53.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.337+0000 7f2aca59c700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2ab406ecc0 0x7f2ab4071180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0000c00 tx=0x7f2ac001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:53.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.342+0000 7f2abb7fe700 1 -- 172.21.15.96:0/1614462528 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2abc057a70 con 0x7f2acc105aa0 2024-04-22T15:41:53.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.561+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2acc10cca0 con 0x7f2ab406ecc0 2024-04-22T15:41:53.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.563+0000 7f2abb7fe700 1 -- 172.21.15.96:0/1614462528 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2acc10cca0 con 0x7f2ab406ecc0 2024-04-22T15:41:53.565 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:41:53.565 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:41:53.565 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:41:53.565 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:41:53.565 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:41:53.566 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:41:53.566 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:41:53.566 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:41:53.566 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:41:53.566 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:41:53.566 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:41:53.566 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:41:53.566 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:41:53.566 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:41:53.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2ab406ecc0 msgr2=0x7f2ab4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2ab406ecc0 0x7f2ab4071180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0000c00 tx=0x7f2ac001a040 comp rx=0 tx=0).stop 2024-04-22T15:41:53.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 msgr2=0x7f2acc197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 0x7f2acc197ee0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00d8d0 tx=0x7f2abc00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 shutdown_connections 2024-04-22T15:41:53.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2ab406ecc0 0x7f2ab4071180 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2acc0ffaa0 0x7f2acc1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 --2- 172.21.15.96:0/1614462528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc105aa0 0x7f2acc197ee0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 >> 172.21.15.96:0/1614462528 conn(0x7f2acc076740 msgr2=0x7f2acc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:53.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 shutdown_connections 2024-04-22T15:41:53.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.565+0000 7f2ad0c77700 1 -- 172.21.15.96:0/1614462528 wait complete. 2024-04-22T15:41:53.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 -- 172.21.15.96:0/2823116769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e4100df0 msgr2=0x7f48e41011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 --2- 172.21.15.96:0/2823116769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e4100df0 0x7f48e41011d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f48d8009910 tx=0x7f48d8009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:53.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 -- 172.21.15.96:0/2823116769 shutdown_connections 2024-04-22T15:41:53.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 --2- 172.21.15.96:0/2823116769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48e41017a0 0x7f48e41057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 --2- 172.21.15.96:0/2823116769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e4100df0 0x7f48e41011d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 -- 172.21.15.96:0/2823116769 >> 172.21.15.96:0/2823116769 conn(0x7f48e40fc6a0 msgr2=0x7f48e40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:53.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 -- 172.21.15.96:0/2823116769 shutdown_connections 2024-04-22T15:41:53.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 -- 172.21.15.96:0/2823116769 wait complete. 2024-04-22T15:41:53.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 Processor -- start 2024-04-22T15:41:53.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.670+0000 7f48eae96700 1 -- start start 2024-04-22T15:41:53.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48eae96700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48e4100df0 0x7f48e419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:53.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48eae96700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e41017a0 0x7f48e419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:53.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48eae96700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e419f540 con 0x7f48e41017a0 2024-04-22T15:41:53.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48eae96700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e4198b20 con 0x7f48e4100df0 2024-04-22T15:41:53.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e41017a0 0x7f48e419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:53.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e41017a0 0x7f48e419ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59476/0 (socket says 172.21.15.96:59476) 2024-04-22T15:41:53.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e3fff700 1 -- 172.21.15.96:0/3270161574 learned_addr learned my addr 172.21.15.96:0/3270161574 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:41:53.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e8c32700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48e4100df0 0x7f48e419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:53.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e3fff700 1 -- 172.21.15.96:0/3270161574 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48e4100df0 msgr2=0x7f48e419e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e3fff700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48e4100df0 0x7f48e419e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e3fff700 1 -- 172.21.15.96:0/3270161574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d80095f0 con 0x7f48e41017a0 2024-04-22T15:41:53.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e3fff700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e41017a0 0x7f48e419ef20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f48d000d900 tx=0x7f48d000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:53.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e1ffb700 1 -- 172.21.15.96:0/3270161574 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48d0009900 con 0x7f48e41017a0 2024-04-22T15:41:53.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e1ffb700 1 -- 172.21.15.96:0/3270161574 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48d0010460 con 0x7f48e41017a0 2024-04-22T15:41:53.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e4198e00 con 0x7f48e41017a0 2024-04-22T15:41:53.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.671+0000 7f48e1ffb700 1 -- 172.21.15.96:0/3270161574 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48d000f5f0 con 0x7f48e41017a0 2024-04-22T15:41:53.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.672+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e4199350 con 0x7f48e41017a0 2024-04-22T15:41:53.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.672+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e4104f70 con 0x7f48e41017a0 2024-04-22T15:41:53.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.673+0000 7f48e1ffb700 1 -- 172.21.15.96:0/3270161574 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f48d00105d0 con 0x7f48e41017a0 2024-04-22T15:41:53.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.673+0000 7f48e1ffb700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48d406ecd0 0x7f48d4071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:53.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.673+0000 7f48e1ffb700 1 -- 172.21.15.96:0/3270161574 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f48d008eaf0 con 0x7f48e41017a0 2024-04-22T15:41:53.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.673+0000 7f48e8c32700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48d406ecd0 0x7f48d4071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:53.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.674+0000 7f48e8c32700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48d406ecd0 0x7f48d4071190 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f48d8006010 tx=0x7f48d801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:53.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.678+0000 7f48e1ffb700 1 -- 172.21.15.96:0/3270161574 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f48d005b2e0 con 0x7f48e41017a0 2024-04-22T15:41:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.961+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f48e41022d0 con 0x7f48e41017a0 2024-04-22T15:41:53.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:53 smithi096 ceph-mon[112443]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:53.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:53 smithi096 ceph-mon[112443]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:53.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:53 smithi096 ceph-mon[112443]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:53.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:53 smithi096 ceph-mon[112443]: pgmap v600: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:53.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3808429547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:41:53.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2905181571' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:41:53.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:53.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.962+0000 7f48e1ffb700 1 -- 172.21.15.96:0/3270161574 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f48d00171c0 con 0x7f48e41017a0 2024-04-22T15:41:53.966 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:41:53.966 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:41:53.966 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:41:53.966 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:41:53.966 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:41:53.966 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:41:53.966 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:41:53.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48d406ecd0 msgr2=0x7f48d4071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48d406ecd0 0x7f48d4071190 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f48d8006010 tx=0x7f48d801a040 comp rx=0 tx=0).stop 2024-04-22T15:41:53.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e41017a0 msgr2=0x7f48e419ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:53.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e41017a0 0x7f48e419ef20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f48d000d900 tx=0x7f48d000dc10 comp rx=0 tx=0).stop 2024-04-22T15:41:53.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 shutdown_connections 2024-04-22T15:41:53.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48d406ecd0 0x7f48d4071190 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48e4100df0 0x7f48e419e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 --2- 172.21.15.96:0/3270161574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48e41017a0 0x7f48e419ef20 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:53.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 >> 172.21.15.96:0/3270161574 conn(0x7f48e40fc6a0 msgr2=0x7f48e40fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:53.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 shutdown_connections 2024-04-22T15:41:53.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:41:53.964+0000 7f48eae96700 1 -- 172.21.15.96:0/3270161574 wait complete. 2024-04-22T15:41:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:53 smithi196 ceph-mon[81815]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:53 smithi196 ceph-mon[81815]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:53 smithi196 ceph-mon[81815]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:53 smithi196 ceph-mon[81815]: pgmap v600: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3808429547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:41:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2905181571' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:41:54.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:54 smithi096 ceph-mon[112443]: from='client.35144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:54.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:54 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3270161574' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:41:54.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:54 smithi096 ceph-mon[112443]: pgmap v601: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:54 smithi196 ceph-mon[81815]: from='client.35144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:41:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:54 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3270161574' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:41:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:54 smithi196 ceph-mon[81815]: pgmap v601: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:57 smithi096 ceph-mon[112443]: pgmap v602: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:58.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:57 smithi196 ceph-mon[81815]: pgmap v602: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:59.816 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:41:59 smithi196 ceph-mon[81815]: pgmap v603: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:41:59.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:41:59 smithi096 ceph-mon[112443]: pgmap v603: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:01.942 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:01 smithi096 ceph-mon[112443]: pgmap v604: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:01 smithi196 ceph-mon[81815]: pgmap v604: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:03.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:03 smithi096 ceph-mon[112443]: pgmap v605: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:03 smithi196 ceph-mon[81815]: pgmap v605: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:05.857 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:05 smithi096 ceph-mon[112443]: pgmap v606: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:05 smithi196 ceph-mon[81815]: pgmap v606: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:07 smithi096 ceph-mon[112443]: pgmap v607: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:07 smithi196 ceph-mon[81815]: pgmap v607: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:08.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:08.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:09.922 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:09 smithi196 ceph-mon[81815]: pgmap v608: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:09.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:09 smithi096 ceph-mon[112443]: pgmap v608: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:11.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:11 smithi096 ceph-mon[112443]: pgmap v609: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:11 smithi196 ceph-mon[81815]: pgmap v609: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:13.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:13 smithi096 ceph-mon[112443]: pgmap v610: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:13 smithi196 ceph-mon[81815]: pgmap v610: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:15.965 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:15 smithi096 ceph-mon[112443]: pgmap v611: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:15 smithi196 ceph-mon[81815]: pgmap v611: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:17.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:17 smithi096 ceph-mon[112443]: pgmap v612: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:17 smithi196 ceph-mon[81815]: pgmap v612: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:19 smithi096 ceph-mon[112443]: pgmap v613: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:19 smithi196 ceph-mon[81815]: pgmap v613: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:21.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:21 smithi096 ceph-mon[112443]: pgmap v614: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:22.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:21 smithi196 ceph-mon[81815]: pgmap v614: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:23 smithi096 ceph-mon[112443]: pgmap v615: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:23 smithi196 ceph-mon[81815]: pgmap v615: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:24.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.081+0000 7fddd9521700 1 -- 172.21.15.96:0/4142445723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd40ffaa0 msgr2=0x7fddd40fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.081+0000 7fddd9521700 1 --2- 172.21.15.96:0/4142445723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd40ffaa0 0x7fddd40fff20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009910 tx=0x7fddc8009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.082+0000 7fddd9521700 1 -- 172.21.15.96:0/4142445723 shutdown_connections 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.082+0000 7fddd9521700 1 --2- 172.21.15.96:0/4142445723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd40ffaa0 0x7fddd40fff20 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.082+0000 7fddd9521700 1 --2- 172.21.15.96:0/4142445723 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddd4105aa0 0x7fddd4105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.082+0000 7fddd9521700 1 -- 172.21.15.96:0/4142445723 >> 172.21.15.96:0/4142445723 conn(0x7fddd4076740 msgr2=0x7fddd40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.082+0000 7fddd9521700 1 -- 172.21.15.96:0/4142445723 shutdown_connections 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.082+0000 7fddd9521700 1 -- 172.21.15.96:0/4142445723 wait complete. 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.082+0000 7fddd9521700 1 Processor -- start 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.082+0000 7fddd9521700 1 -- start start 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd9521700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddd40ffaa0 0x7fddd4197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd9521700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd4105aa0 0x7fddd4197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd9521700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4198430 con 0x7fddd4105aa0 2024-04-22T15:42:24.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd9521700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4198570 con 0x7fddd40ffaa0 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd4105aa0 0x7fddd4197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd4105aa0 0x7fddd4197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60590/0 (socket says 172.21.15.96:60590) 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd27fc700 1 -- 172.21.15.96:0/291174095 learned_addr learned my addr 172.21.15.96:0/291174095 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd27fc700 1 -- 172.21.15.96:0/291174095 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddd40ffaa0 msgr2=0x7fddd4197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd2ffd700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddd40ffaa0 0x7fddd4197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd27fc700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddd40ffaa0 0x7fddd4197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd27fc700 1 -- 172.21.15.96:0/291174095 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc80095f0 con 0x7fddd4105aa0 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd27fc700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd4105aa0 0x7fddd4197e60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fddc8006010 tx=0x7fddc800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddbbfff700 1 -- 172.21.15.96:0/291174095 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc801d070 con 0x7fddd4105aa0 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddbbfff700 1 -- 172.21.15.96:0/291174095 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddc800f460 con 0x7fddd4105aa0 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.083+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd41a0990 con 0x7fddd4105aa0 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.084+0000 7fddbbfff700 1 -- 172.21.15.96:0/291174095 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc8017630 con 0x7fddd4105aa0 2024-04-22T15:42:24.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.084+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd41a0cf0 con 0x7fddd4105aa0 2024-04-22T15:42:24.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.085+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd41080c0 con 0x7fddd4105aa0 2024-04-22T15:42:24.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.085+0000 7fddbbfff700 1 -- 172.21.15.96:0/291174095 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fddc800f5d0 con 0x7fddd4105aa0 2024-04-22T15:42:24.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.085+0000 7fddbbfff700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fddbc06ecd0 0x7fddbc071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:24.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.086+0000 7fddbbfff700 1 -- 172.21.15.96:0/291174095 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fddc80903a0 con 0x7fddd4105aa0 2024-04-22T15:42:24.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.086+0000 7fddd2ffd700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fddbc06ecd0 0x7fddbc071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:24.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.086+0000 7fddd2ffd700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fddbc06ecd0 0x7fddbc071190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fddc4005950 tx=0x7fddc4005cf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:24.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.090+0000 7fddbbfff700 1 -- 172.21.15.96:0/291174095 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fddc80591c0 con 0x7fddd4105aa0 2024-04-22T15:42:24.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.308+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fddd410cca0 con 0x7fddbc06ecd0 2024-04-22T15:42:24.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.310+0000 7fddbbfff700 1 -- 172.21.15.96:0/291174095 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fddd410cca0 con 0x7fddbc06ecd0 2024-04-22T15:42:24.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fddbc06ecd0 msgr2=0x7fddbc071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fddbc06ecd0 0x7fddbc071190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fddc4005950 tx=0x7fddc4005cf0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd4105aa0 msgr2=0x7fddd4197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd4105aa0 0x7fddd4197e60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fddc8006010 tx=0x7fddc800be20 comp rx=0 tx=0).stop 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 shutdown_connections 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fddbc06ecd0 0x7fddbc071190 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddd40ffaa0 0x7fddd4197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 --2- 172.21.15.96:0/291174095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddd4105aa0 0x7fddd4197e60 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 >> 172.21.15.96:0/291174095 conn(0x7fddd4076740 msgr2=0x7fddd40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 shutdown_connections 2024-04-22T15:42:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.312+0000 7fddd9521700 1 -- 172.21.15.96:0/291174095 wait complete. 2024-04-22T15:42:24.325 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:42:24.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.417+0000 7fef86cf2700 1 -- 172.21.15.96:0/1841700003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef800ffab0 msgr2=0x7fef800fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.417+0000 7fef86cf2700 1 --2- 172.21.15.96:0/1841700003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef800ffab0 0x7fef800fff30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fef6c009910 tx=0x7fef6c009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:24.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.417+0000 7fef86cf2700 1 -- 172.21.15.96:0/1841700003 shutdown_connections 2024-04-22T15:42:24.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.417+0000 7fef86cf2700 1 --2- 172.21.15.96:0/1841700003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef800ffab0 0x7fef800fff30 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.417+0000 7fef86cf2700 1 --2- 172.21.15.96:0/1841700003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef80105ab0 0x7fef80105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.417+0000 7fef86cf2700 1 -- 172.21.15.96:0/1841700003 >> 172.21.15.96:0/1841700003 conn(0x7fef800fb5e0 msgr2=0x7fef800fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:24.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.417+0000 7fef86cf2700 1 -- 172.21.15.96:0/1841700003 shutdown_connections 2024-04-22T15:42:24.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.417+0000 7fef86cf2700 1 -- 172.21.15.96:0/1841700003 wait complete. 2024-04-22T15:42:24.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef86cf2700 1 Processor -- start 2024-04-22T15:42:24.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef86cf2700 1 -- start start 2024-04-22T15:42:24.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef86cf2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef800ffab0 0x7fef801958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:24.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef86cf2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef80105ab0 0x7fef80195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:24.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef86cf2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef80069d90 con 0x7fef80105ab0 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef86cf2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef80196320 con 0x7fef800ffab0 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef7ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef80105ab0 0x7fef80195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef7ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef80105ab0 0x7fef80195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60614/0 (socket says 172.21.15.96:60614) 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef7ffff700 1 -- 172.21.15.96:0/3906978911 learned_addr learned my addr 172.21.15.96:0/3906978911 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.418+0000 7fef84a8e700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef800ffab0 0x7fef801958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.419+0000 7fef7ffff700 1 -- 172.21.15.96:0/3906978911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef800ffab0 msgr2=0x7fef801958a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.419+0000 7fef7ffff700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef800ffab0 0x7fef801958a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.419+0000 7fef7ffff700 1 -- 172.21.15.96:0/3906978911 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6c0095f0 con 0x7fef80105ab0 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.419+0000 7fef7ffff700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef80105ab0 0x7fef80195de0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00bd70 tx=0x7fef6c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.419+0000 7fef7dffb700 1 -- 172.21.15.96:0/3906978911 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef6c01d070 con 0x7fef80105ab0 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.419+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef80196540 con 0x7fef80105ab0 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.419+0000 7fef7dffb700 1 -- 172.21.15.96:0/3906978911 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef6c00f460 con 0x7fef80105ab0 2024-04-22T15:42:24.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.419+0000 7fef7dffb700 1 -- 172.21.15.96:0/3906978911 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef6c017630 con 0x7fef80105ab0 2024-04-22T15:42:24.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.419+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef801a30e0 con 0x7fef80105ab0 2024-04-22T15:42:24.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.420+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef801080d0 con 0x7fef80105ab0 2024-04-22T15:42:24.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.421+0000 7fef7dffb700 1 -- 172.21.15.96:0/3906978911 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fef6c00f5d0 con 0x7fef80105ab0 2024-04-22T15:42:24.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.421+0000 7fef7dffb700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7006ecc0 0x7fef70071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:24.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.421+0000 7fef7dffb700 1 -- 172.21.15.96:0/3906978911 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fef6c0903f0 con 0x7fef80105ab0 2024-04-22T15:42:24.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.421+0000 7fef84a8e700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7006ecc0 0x7fef70071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:24.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.422+0000 7fef84a8e700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7006ecc0 0x7fef70071180 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fef74006fd0 tx=0x7fef74008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:24.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.426+0000 7fef7dffb700 1 -- 172.21.15.96:0/3906978911 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fef6c05cc80 con 0x7fef80105ab0 2024-04-22T15:42:24.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.645+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef8010cd30 con 0x7fef7006ecc0 2024-04-22T15:42:24.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.646+0000 7fef7dffb700 1 -- 172.21.15.96:0/3906978911 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fef8010cd30 con 0x7fef7006ecc0 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.648+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7006ecc0 msgr2=0x7fef70071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.648+0000 7fef86cf2700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7006ecc0 0x7fef70071180 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fef74006fd0 tx=0x7fef74008040 comp rx=0 tx=0).stop 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.648+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef80105ab0 msgr2=0x7fef80195de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.648+0000 7fef86cf2700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef80105ab0 0x7fef80195de0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00bd70 tx=0x7fef6c00be50 comp rx=0 tx=0).stop 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.649+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 shutdown_connections 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.649+0000 7fef86cf2700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fef7006ecc0 0x7fef70071180 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.649+0000 7fef86cf2700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fef800ffab0 0x7fef801958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.649+0000 7fef86cf2700 1 --2- 172.21.15.96:0/3906978911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef80105ab0 0x7fef80195de0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.649+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 >> 172.21.15.96:0/3906978911 conn(0x7fef800fb5e0 msgr2=0x7fef800fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.649+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 shutdown_connections 2024-04-22T15:42:24.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.649+0000 7fef86cf2700 1 -- 172.21.15.96:0/3906978911 wait complete. 2024-04-22T15:42:24.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:24 smithi096 ceph-mon[112443]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:24.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:24 smithi096 ceph-mon[112443]: pgmap v616: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:24.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.754+0000 7faa3279e700 1 -- 172.21.15.96:0/1675197573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 msgr2=0x7faa2c100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.754+0000 7faa3279e700 1 --2- 172.21.15.96:0/1675197573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 0x7faa2c100940 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7faa24009910 tx=0x7faa24009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:24.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.754+0000 7faa3279e700 1 -- 172.21.15.96:0/1675197573 shutdown_connections 2024-04-22T15:42:24.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.754+0000 7faa3279e700 1 --2- 172.21.15.96:0/1675197573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa2c100f10 0x7faa2c104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.754+0000 7faa3279e700 1 --2- 172.21.15.96:0/1675197573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 0x7faa2c100940 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.754+0000 7faa3279e700 1 -- 172.21.15.96:0/1675197573 >> 172.21.15.96:0/1675197573 conn(0x7faa2c075d40 msgr2=0x7faa2c076150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:24.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.754+0000 7faa3279e700 1 -- 172.21.15.96:0/1675197573 shutdown_connections 2024-04-22T15:42:24.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.754+0000 7faa3279e700 1 -- 172.21.15.96:0/1675197573 wait complete. 2024-04-22T15:42:24.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.754+0000 7faa3279e700 1 Processor -- start 2024-04-22T15:42:24.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3279e700 1 -- start start 2024-04-22T15:42:24.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3279e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 0x7faa2c1977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:24.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3279e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa2c100f10 0x7faa2c197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3279e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c1982c0 con 0x7faa2c100560 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3279e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c198400 con 0x7faa2c100f10 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3179c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 0x7faa2c1977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3179c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 0x7faa2c1977b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60636/0 (socket says 172.21.15.96:60636) 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3179c700 1 -- 172.21.15.96:0/807007285 learned_addr learned my addr 172.21.15.96:0/807007285 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa30f9b700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa2c100f10 0x7faa2c197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3179c700 1 -- 172.21.15.96:0/807007285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa2c100f10 msgr2=0x7faa2c197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3179c700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa2c100f10 0x7faa2c197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3179c700 1 -- 172.21.15.96:0/807007285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa240095f0 con 0x7faa2c100560 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa30f9b700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa2c100f10 0x7faa2c197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa3179c700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 0x7faa2c1977b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7faa24006010 tx=0x7faa2400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.755+0000 7faa2a7fc700 1 -- 172.21.15.96:0/807007285 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa2401d070 con 0x7faa2c100560 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.756+0000 7faa2a7fc700 1 -- 172.21.15.96:0/807007285 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa2400f460 con 0x7faa2c100560 2024-04-22T15:42:24.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.756+0000 7faa2a7fc700 1 -- 172.21.15.96:0/807007285 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa24017630 con 0x7faa2c100560 2024-04-22T15:42:24.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.756+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2c1a0820 con 0x7faa2c100560 2024-04-22T15:42:24.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.756+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2c1a0b80 con 0x7faa2c100560 2024-04-22T15:42:24.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.757+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2c107ee0 con 0x7faa2c100560 2024-04-22T15:42:24.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.757+0000 7faa2a7fc700 1 -- 172.21.15.96:0/807007285 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faa2400f5d0 con 0x7faa2c100560 2024-04-22T15:42:24.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.758+0000 7faa2a7fc700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa1406ecd0 0x7faa14071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:24.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.758+0000 7faa2a7fc700 1 -- 172.21.15.96:0/807007285 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7faa240903a0 con 0x7faa2c100560 2024-04-22T15:42:24.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.758+0000 7faa30f9b700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa1406ecd0 0x7faa14071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:24.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.758+0000 7faa30f9b700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa1406ecd0 0x7faa14071190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0ff9e0 tx=0x7faa20005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:24.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.762+0000 7faa2a7fc700 1 -- 172.21.15.96:0/807007285 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faa240591c0 con 0x7faa2c100560 2024-04-22T15:42:24.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.972+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faa2c10cb30 con 0x7faa1406ecd0 2024-04-22T15:42:24.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.982+0000 7faa2a7fc700 1 -- 172.21.15.96:0/807007285 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7faa2c10cb30 con 0x7faa1406ecd0 2024-04-22T15:42:24.985 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:42:24.985 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (21m) 14m ago 32m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (32m) 14m ago 32m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (31m) 13m ago 31m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (18m) 14m ago 32m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (18m) 13m ago 31m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (20m) 14m ago 32m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (14m) 14m ago 26m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (14m) 14m ago 26m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (13m) 13m ago 26m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (26m) 13m ago 26m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (23m) 14m ago 33m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (22m) 13m ago 31m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (19m) 14m ago 34m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (18m) 13m ago 31m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (22m) 14m ago 32m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:42:24.986 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (21m) 13m ago 31m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:42:24.987 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (17m) 14m ago 30m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:42:24.987 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (17m) 14m ago 30m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:42:24.987 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (16m) 14m ago 29m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:42:24.987 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (16m) 13m ago 29m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:42:24.987 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (15m) 13m ago 28m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:42:24.987 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (14m) 13m ago 28m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:42:24.987 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (21m) 14m ago 31m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa1406ecd0 msgr2=0x7faa14071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa1406ecd0 0x7faa14071190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0ff9e0 tx=0x7faa20005f50 comp rx=0 tx=0).stop 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 msgr2=0x7faa2c1977b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 0x7faa2c1977b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7faa24006010 tx=0x7faa2400be20 comp rx=0 tx=0).stop 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 shutdown_connections 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faa1406ecd0 0x7faa14071190 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa2c100560 0x7faa2c1977b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 --2- 172.21.15.96:0/807007285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa2c100f10 0x7faa2c197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 >> 172.21.15.96:0/807007285 conn(0x7faa2c075d40 msgr2=0x7faa2c0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 shutdown_connections 2024-04-22T15:42:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:24.985+0000 7faa3279e700 1 -- 172.21.15.96:0/807007285 wait complete. 2024-04-22T15:42:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:24 smithi196 ceph-mon[81815]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:24 smithi196 ceph-mon[81815]: pgmap v616: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:25.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 -- 172.21.15.96:0/1603068072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 msgr2=0x7feba8105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 --2- 172.21.15.96:0/1603068072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 0x7feba8105820 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009910 tx=0x7feb9c009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:25.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 -- 172.21.15.96:0/1603068072 shutdown_connections 2024-04-22T15:42:25.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 --2- 172.21.15.96:0/1603068072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 0x7feba8105820 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 --2- 172.21.15.96:0/1603068072 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba8100e20 0x7feba8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 -- 172.21.15.96:0/1603068072 >> 172.21.15.96:0/1603068072 conn(0x7feba80fc6d0 msgr2=0x7feba80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:25.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 -- 172.21.15.96:0/1603068072 shutdown_connections 2024-04-22T15:42:25.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 -- 172.21.15.96:0/1603068072 wait complete. 2024-04-22T15:42:25.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 Processor -- start 2024-04-22T15:42:25.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.087+0000 7febae520700 1 -- start start 2024-04-22T15:42:25.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7febae520700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba8100e20 0x7feba819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:25.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7febae520700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 0x7feba819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:25.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7febae520700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba819d380 con 0x7feba81017d0 2024-04-22T15:42:25.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7febae520700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba8196960 con 0x7feba8100e20 2024-04-22T15:42:25.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7feba77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 0x7feba819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7feba77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 0x7feba819cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60662/0 (socket says 172.21.15.96:60662) 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7feba77fe700 1 -- 172.21.15.96:0/1048138460 learned_addr learned my addr 172.21.15.96:0/1048138460 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7feba7fff700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba8100e20 0x7feba819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7feba77fe700 1 -- 172.21.15.96:0/1048138460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba8100e20 msgr2=0x7feba819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7feba77fe700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba8100e20 0x7feba819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7feba77fe700 1 -- 172.21.15.96:0/1048138460 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c0095f0 con 0x7feba81017d0 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7feba77fe700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 0x7feba819cd60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009fd0 tx=0x7feb9c00b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7feba57fa700 1 -- 172.21.15.96:0/1048138460 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb9c01d070 con 0x7feba81017d0 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.088+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba8196be0 con 0x7feba81017d0 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.089+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba81970d0 con 0x7feba81017d0 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.089+0000 7feba57fa700 1 -- 172.21.15.96:0/1048138460 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb9c0052a0 con 0x7feba81017d0 2024-04-22T15:42:25.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.089+0000 7feba57fa700 1 -- 172.21.15.96:0/1048138460 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb9c022470 con 0x7feba81017d0 2024-04-22T15:42:25.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.090+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba8109220 con 0x7feba81017d0 2024-04-22T15:42:25.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.090+0000 7feba57fa700 1 -- 172.21.15.96:0/1048138460 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feb9c00f6f0 con 0x7feba81017d0 2024-04-22T15:42:25.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.090+0000 7feba57fa700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb9006ecc0 0x7feb90071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:25.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.091+0000 7feba57fa700 1 -- 172.21.15.96:0/1048138460 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7feb9c090120 con 0x7feba81017d0 2024-04-22T15:42:25.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.091+0000 7feba7fff700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb9006ecc0 0x7feb90071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:25.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.091+0000 7feba7fff700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb9006ecc0 0x7feb90071180 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feb98009c60 tx=0x7feb98009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:25.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.098+0000 7feba57fa700 1 -- 172.21.15.96:0/1048138460 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7feb9c05c990 con 0x7feba81017d0 2024-04-22T15:42:25.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.379+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feba8197e20 con 0x7feba81017d0 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.380+0000 7feba57fa700 1 -- 172.21.15.96:0/1048138460 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7feb9c05c520 con 0x7feba81017d0 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:42:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:42:25.384 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:42:25.384 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:42:25.384 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:42:25.384 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:42:25.384 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:42:25.384 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:42:25.384 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:42:25.384 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:42:25.384 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:42:25.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb9006ecc0 msgr2=0x7feb90071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb9006ecc0 0x7feb90071180 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feb98009c60 tx=0x7feb98009260 comp rx=0 tx=0).stop 2024-04-22T15:42:25.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 msgr2=0x7feba819cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 0x7feba819cd60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009fd0 tx=0x7feb9c00b5a0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 shutdown_connections 2024-04-22T15:42:25.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb9006ecc0 0x7feb90071180 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba8100e20 0x7feba819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 --2- 172.21.15.96:0/1048138460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feba81017d0 0x7feba819cd60 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 >> 172.21.15.96:0/1048138460 conn(0x7feba80fc6d0 msgr2=0x7feba80fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:25.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 shutdown_connections 2024-04-22T15:42:25.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.382+0000 7febae520700 1 -- 172.21.15.96:0/1048138460 wait complete. 2024-04-22T15:42:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.486+0000 7f377f913700 1 -- 172.21.15.96:0/475369933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 msgr2=0x7f377810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.486+0000 7f377f913700 1 --2- 172.21.15.96:0/475369933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 0x7f377810ac10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3774009910 tx=0x7f3774009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.486+0000 7f377f913700 1 -- 172.21.15.96:0/475369933 shutdown_connections 2024-04-22T15:42:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.486+0000 7f377f913700 1 --2- 172.21.15.96:0/475369933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 0x7f377810ac10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.486+0000 7f377f913700 1 --2- 172.21.15.96:0/475369933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3778101e00 0x7f37781021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.486+0000 7f377f913700 1 -- 172.21.15.96:0/475369933 >> 172.21.15.96:0/475369933 conn(0x7f3778076740 msgr2=0x7f37780fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.486+0000 7f377f913700 1 -- 172.21.15.96:0/475369933 shutdown_connections 2024-04-22T15:42:25.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.486+0000 7f377f913700 1 -- 172.21.15.96:0/475369933 wait complete. 2024-04-22T15:42:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377f913700 1 Processor -- start 2024-04-22T15:42:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377f913700 1 -- start start 2024-04-22T15:42:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377f913700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3778101e00 0x7f377819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377f913700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 0x7f377819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377f913700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377819d480 con 0x7f3778102720 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377f913700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3778196a60 con 0x7f3778101e00 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377ceae700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 0x7f377819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377ceae700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 0x7f377819ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60674/0 (socket says 172.21.15.96:60674) 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377ceae700 1 -- 172.21.15.96:0/1734784218 learned_addr learned my addr 172.21.15.96:0/1734784218 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377d6af700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3778101e00 0x7f377819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377ceae700 1 -- 172.21.15.96:0/1734784218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3778101e00 msgr2=0x7f377819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377ceae700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3778101e00 0x7f377819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377ceae700 1 -- 172.21.15.96:0/1734784218 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37740095f0 con 0x7f3778102720 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.487+0000 7f377d6af700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3778101e00 0x7f377819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.488+0000 7f377ceae700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 0x7f377819ce60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f377400bd70 tx=0x7f377400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.488+0000 7f376e7fc700 1 -- 172.21.15.96:0/1734784218 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f377401d070 con 0x7f3778102720 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.488+0000 7f376e7fc700 1 -- 172.21.15.96:0/1734784218 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f377400f460 con 0x7f3778102720 2024-04-22T15:42:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.488+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3778196ce0 con 0x7f3778102720 2024-04-22T15:42:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.488+0000 7f376e7fc700 1 -- 172.21.15.96:0/1734784218 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3774017630 con 0x7f3778102720 2024-04-22T15:42:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.488+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37781971d0 con 0x7f3778102720 2024-04-22T15:42:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.489+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37781055f0 con 0x7f3778102720 2024-04-22T15:42:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.489+0000 7f376e7fc700 1 -- 172.21.15.96:0/1734784218 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f377400f5d0 con 0x7f3778102720 2024-04-22T15:42:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.490+0000 7f376e7fc700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f376406ecc0 0x7f3764071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.490+0000 7f376e7fc700 1 -- 172.21.15.96:0/1734784218 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3774090350 con 0x7f3778102720 2024-04-22T15:42:25.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.490+0000 7f377d6af700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f376406ecc0 0x7f3764071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:25.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.490+0000 7f377d6af700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f376406ecc0 0x7f3764071180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3768007900 tx=0x7f3768008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.495+0000 7f376e7fc700 1 -- 172.21.15.96:0/1734784218 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f377405cbe0 con 0x7f3778102720 2024-04-22T15:42:25.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:25 smithi096 ceph-mon[112443]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:25.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:25 smithi096 ceph-mon[112443]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:25.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:25 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1048138460' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:42:25.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.744+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3778068e00 con 0x7f3778102720 2024-04-22T15:42:25.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.745+0000 7f376e7fc700 1 -- 172.21.15.96:0/1734784218 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f3774005740 con 0x7f3778102720 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:42:25.748 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:42:25.749 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:42:25.750 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:42:25.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f376406ecc0 msgr2=0x7f3764071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f376406ecc0 0x7f3764071180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3768007900 tx=0x7f3768008040 comp rx=0 tx=0).stop 2024-04-22T15:42:25.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 msgr2=0x7f377819ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 0x7f377819ce60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f377400bd70 tx=0x7f377400be50 comp rx=0 tx=0).stop 2024-04-22T15:42:25.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 shutdown_connections 2024-04-22T15:42:25.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f376406ecc0 0x7f3764071180 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3778101e00 0x7f377819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 --2- 172.21.15.96:0/1734784218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3778102720 0x7f377819ce60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 >> 172.21.15.96:0/1734784218 conn(0x7f3778076740 msgr2=0x7f37781030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:25.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 shutdown_connections 2024-04-22T15:42:25.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.747+0000 7f377f913700 1 -- 172.21.15.96:0/1734784218 wait complete. 2024-04-22T15:42:25.752 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.852+0000 7fa398710700 1 -- 172.21.15.96:0/615103456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa390105ac0 msgr2=0x7fa390105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.852+0000 7fa398710700 1 --2- 172.21.15.96:0/615103456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa390105ac0 0x7fa390105ea0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa380009910 tx=0x7fa380009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.852+0000 7fa398710700 1 -- 172.21.15.96:0/615103456 shutdown_connections 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.852+0000 7fa398710700 1 --2- 172.21.15.96:0/615103456 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa3900ffac0 0x7fa3900fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.852+0000 7fa398710700 1 --2- 172.21.15.96:0/615103456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa390105ac0 0x7fa390105ea0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.852+0000 7fa398710700 1 -- 172.21.15.96:0/615103456 >> 172.21.15.96:0/615103456 conn(0x7fa3900fb5d0 msgr2=0x7fa3900fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.852+0000 7fa398710700 1 -- 172.21.15.96:0/615103456 shutdown_connections 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.852+0000 7fa398710700 1 -- 172.21.15.96:0/615103456 wait complete. 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.853+0000 7fa398710700 1 Processor -- start 2024-04-22T15:42:25.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.853+0000 7fa398710700 1 -- start start 2024-04-22T15:42:25.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.853+0000 7fa398710700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3900ffac0 0x7fa390197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:25.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.853+0000 7fa398710700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa390105ac0 0x7fa390197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:25.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.853+0000 7fa398710700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa390198500 con 0x7fa3900ffac0 2024-04-22T15:42:25.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.853+0000 7fa398710700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa390198640 con 0x7fa390105ac0 2024-04-22T15:42:25.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.853+0000 7fa3964ac700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3900ffac0 0x7fa390197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:25.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.853+0000 7fa3964ac700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3900ffac0 0x7fa390197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60698/0 (socket says 172.21.15.96:60698) 2024-04-22T15:42:25.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.853+0000 7fa3964ac700 1 -- 172.21.15.96:0/2254479233 learned_addr learned my addr 172.21.15.96:0/2254479233 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:25.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa395cab700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa390105ac0 0x7fa390197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa3964ac700 1 -- 172.21.15.96:0/2254479233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa390105ac0 msgr2=0x7fa390197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa3964ac700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa390105ac0 0x7fa390197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa3964ac700 1 -- 172.21.15.96:0/2254479233 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3800095f0 con 0x7fa3900ffac0 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa395cab700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa390105ac0 0x7fa390197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa3964ac700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3900ffac0 0x7fa390197a80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa38000bd70 tx=0x7fa38000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa3877fe700 1 -- 172.21.15.96:0/2254479233 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa38001d070 con 0x7fa3900ffac0 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa3877fe700 1 -- 172.21.15.96:0/2254479233 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa38000f460 con 0x7fa3900ffac0 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3901988c0 con 0x7fa3900ffac0 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa3877fe700 1 -- 172.21.15.96:0/2254479233 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa380017630 con 0x7fa3900ffac0 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.854+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3901a0e20 con 0x7fa3900ffac0 2024-04-22T15:42:25.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.855+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3901080c0 con 0x7fa3900ffac0 2024-04-22T15:42:25.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.856+0000 7fa3877fe700 1 -- 172.21.15.96:0/2254479233 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa38000f5d0 con 0x7fa3900ffac0 2024-04-22T15:42:25.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.856+0000 7fa3877fe700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa37c06ecc0 0x7fa37c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:25.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.856+0000 7fa3877fe700 1 -- 172.21.15.96:0/2254479233 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa380090390 con 0x7fa3900ffac0 2024-04-22T15:42:25.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.856+0000 7fa395cab700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa37c06ecc0 0x7fa37c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:25.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.856+0000 7fa395cab700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa37c06ecc0 0x7fa37c071180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa3900698d0 tx=0x7fa38c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:25.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:25.861+0000 7fa3877fe700 1 -- 172.21.15.96:0/2254479233 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa3800590f0 con 0x7fa3900ffac0 2024-04-22T15:42:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:25 smithi196 ceph-mon[81815]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:25 smithi196 ceph-mon[81815]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:25 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1048138460' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:42:26.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.081+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa390069720 con 0x7fa37c06ecc0 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.083+0000 7fa3877fe700 1 -- 172.21.15.96:0/2254479233 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa390069720 con 0x7fa37c06ecc0 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:42:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:42:26.087 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:42:26.087 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:42:26.087 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:42:26.087 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:42:26.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa37c06ecc0 msgr2=0x7fa37c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:26.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa37c06ecc0 0x7fa37c071180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa3900698d0 tx=0x7fa38c008040 comp rx=0 tx=0).stop 2024-04-22T15:42:26.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3900ffac0 msgr2=0x7fa390197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:26.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3900ffac0 0x7fa390197a80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa38000bd70 tx=0x7fa38000be50 comp rx=0 tx=0).stop 2024-04-22T15:42:26.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 shutdown_connections 2024-04-22T15:42:26.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa37c06ecc0 0x7fa37c071180 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:26.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3900ffac0 0x7fa390197a80 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:26.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 --2- 172.21.15.96:0/2254479233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa390105ac0 0x7fa390197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:26.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 >> 172.21.15.96:0/2254479233 conn(0x7fa3900fb5d0 msgr2=0x7fa3900fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:26.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 shutdown_connections 2024-04-22T15:42:26.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.085+0000 7fa398710700 1 -- 172.21.15.96:0/2254479233 wait complete. 2024-04-22T15:42:26.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.190+0000 7f76e3b8b700 1 -- 172.21.15.96:0/3311645452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ffe50 msgr2=0x7f76dc103ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:26.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.190+0000 7f76e3b8b700 1 --2- 172.21.15.96:0/3311645452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ffe50 0x7f76dc103ea0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009910 tx=0x7f76d0009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:26.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.190+0000 7f76e3b8b700 1 -- 172.21.15.96:0/3311645452 shutdown_connections 2024-04-22T15:42:26.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.190+0000 7f76e3b8b700 1 --2- 172.21.15.96:0/3311645452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ffe50 0x7f76dc103ea0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:26.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.190+0000 7f76e3b8b700 1 --2- 172.21.15.96:0/3311645452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc0ff4a0 0x7f76dc0ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:26.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.190+0000 7f76e3b8b700 1 -- 172.21.15.96:0/3311645452 >> 172.21.15.96:0/3311645452 conn(0x7f76dc075f30 msgr2=0x7f76dc0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:26.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.190+0000 7f76e3b8b700 1 -- 172.21.15.96:0/3311645452 shutdown_connections 2024-04-22T15:42:26.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.190+0000 7f76e3b8b700 1 -- 172.21.15.96:0/3311645452 wait complete. 2024-04-22T15:42:26.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e3b8b700 1 Processor -- start 2024-04-22T15:42:26.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e3b8b700 1 -- start start 2024-04-22T15:42:26.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e3b8b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ff4a0 0x7f76dc197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:26.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e3b8b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc0ffe50 0x7f76dc197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:26.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e3b8b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc198520 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e3b8b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc198660 con 0x7f76dc0ffe50 2024-04-22T15:42:26.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e1927700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ff4a0 0x7f76dc197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e1927700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ff4a0 0x7f76dc197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60724/0 (socket says 172.21.15.96:60724) 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e1927700 1 -- 172.21.15.96:0/4190667255 learned_addr learned my addr 172.21.15.96:0/4190667255 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.191+0000 7f76e1126700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc0ffe50 0x7f76dc197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76e1927700 1 -- 172.21.15.96:0/4190667255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc0ffe50 msgr2=0x7f76dc197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76e1927700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc0ffe50 0x7f76dc197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76e1927700 1 -- 172.21.15.96:0/4190667255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d00095f0 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76e1126700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc0ffe50 0x7f76dc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76e1927700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ff4a0 0x7f76dc197a10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f76d400ba50 tx=0x7f76d400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76daffd700 1 -- 172.21.15.96:0/4190667255 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76d400ca00 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76daffd700 1 -- 172.21.15.96:0/4190667255 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76d400cb60 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76dc1a0a80 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76daffd700 1 -- 172.21.15.96:0/4190667255 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76d40075a0 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.192+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76dc1a0ef0 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.193+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76dc100980 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.193+0000 7f76daffd700 1 -- 172.21.15.96:0/4190667255 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f76d400ccd0 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.194+0000 7f76daffd700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f76c406ecc0 0x7f76c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:26.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.194+0000 7f76daffd700 1 -- 172.21.15.96:0/4190667255 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f76d4006e40 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.194+0000 7f76e1126700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f76c406ecc0 0x7f76c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:26.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.194+0000 7f76e1126700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f76c406ecc0 0x7f76c4071180 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f76dc103010 tx=0x7f76d001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:26.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.199+0000 7f76daffd700 1 -- 172.21.15.96:0/4190667255 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f76d4057450 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.484+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f76dc04ebb0 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.485+0000 7f76daffd700 1 -- 172.21.15.96:0/4190667255 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f76d405ab10 con 0x7f76dc0ff4a0 2024-04-22T15:42:26.488 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:42:26.488 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:42:26.488 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:42:26.488 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:42:26.488 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:42:26.488 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:42:26.488 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:42:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f76c406ecc0 msgr2=0x7f76c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f76c406ecc0 0x7f76c4071180 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f76dc103010 tx=0x7f76d001a040 comp rx=0 tx=0).stop 2024-04-22T15:42:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ff4a0 msgr2=0x7f76dc197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ff4a0 0x7f76dc197a10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f76d400ba50 tx=0x7f76d400be10 comp rx=0 tx=0).stop 2024-04-22T15:42:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 shutdown_connections 2024-04-22T15:42:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f76c406ecc0 0x7f76c4071180 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76dc0ff4a0 0x7f76dc197a10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 --2- 172.21.15.96:0/4190667255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc0ffe50 0x7f76dc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 >> 172.21.15.96:0/4190667255 conn(0x7f76dc075f30 msgr2=0x7f76dc0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 shutdown_connections 2024-04-22T15:42:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:26.487+0000 7f76e3b8b700 1 -- 172.21.15.96:0/4190667255 wait complete. 2024-04-22T15:42:26.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:26 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1734784218' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:42:26.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:26 smithi096 ceph-mon[112443]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:26.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:26 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4190667255' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:42:26.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:26 smithi096 ceph-mon[112443]: pgmap v617: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:26 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1734784218' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:42:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:26 smithi196 ceph-mon[81815]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:26 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4190667255' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:42:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:26 smithi196 ceph-mon[81815]: pgmap v617: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:29.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:29 smithi096 ceph-mon[112443]: pgmap v618: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:29 smithi196 ceph-mon[81815]: pgmap v618: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:31.760 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:31 smithi096 ceph-mon[112443]: pgmap v619: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:32.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:31 smithi196 ceph-mon[81815]: pgmap v619: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:33.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:33 smithi096 ceph-mon[112443]: pgmap v620: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:33 smithi196 ceph-mon[81815]: pgmap v620: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:35.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:35 smithi096 ceph-mon[112443]: pgmap v621: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:35 smithi196 ceph-mon[81815]: pgmap v621: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:37.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:37 smithi096 ceph-mon[112443]: pgmap v622: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:37 smithi196 ceph-mon[81815]: pgmap v622: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:38.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:39.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:39 smithi096 ceph-mon[112443]: pgmap v623: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:39 smithi196 ceph-mon[81815]: pgmap v623: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:41.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:41 smithi096 ceph-mon[112443]: pgmap v624: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:41 smithi196 ceph-mon[81815]: pgmap v624: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:43.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:43 smithi096 ceph-mon[112443]: pgmap v625: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:43 smithi196 ceph-mon[81815]: pgmap v625: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:45.585 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:45 smithi096 ceph-mon[112443]: pgmap v626: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:45 smithi196 ceph-mon[81815]: pgmap v626: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:47 smithi096 ceph-mon[112443]: pgmap v627: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:47 smithi196 ceph-mon[81815]: pgmap v627: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:49.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:49 smithi096 ceph-mon[112443]: pgmap v628: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:49 smithi196 ceph-mon[81815]: pgmap v628: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:51.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:51 smithi096 ceph-mon[112443]: pgmap v629: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:51 smithi196 ceph-mon[81815]: pgmap v629: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:53 smithi096 ceph-mon[112443]: pgmap v630: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:53 smithi196 ceph-mon[81815]: pgmap v630: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:54 smithi096 ceph-mon[112443]: pgmap v631: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:54 smithi196 ceph-mon[81815]: pgmap v631: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:56.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.599+0000 7fe7ac9be700 1 -- 172.21.15.96:0/415208168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 msgr2=0x7fe7a8105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:56.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.599+0000 7fe7ac9be700 1 --2- 172.21.15.96:0/415208168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 0x7fe7a8105f40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe798009910 tx=0x7fe798009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:56.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.599+0000 7fe7ac9be700 1 -- 172.21.15.96:0/415208168 shutdown_connections 2024-04-22T15:42:56.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.599+0000 7fe7ac9be700 1 --2- 172.21.15.96:0/415208168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7a80ffad0 0x7fe7a80fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.599+0000 7fe7ac9be700 1 --2- 172.21.15.96:0/415208168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 0x7fe7a8105f40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.599+0000 7fe7ac9be700 1 -- 172.21.15.96:0/415208168 >> 172.21.15.96:0/415208168 conn(0x7fe7a80fb600 msgr2=0x7fe7a80fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:56.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.599+0000 7fe7ac9be700 1 -- 172.21.15.96:0/415208168 shutdown_connections 2024-04-22T15:42:56.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.599+0000 7fe7ac9be700 1 -- 172.21.15.96:0/415208168 wait complete. 2024-04-22T15:42:56.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7ac9be700 1 Processor -- start 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7ac9be700 1 -- start start 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7ac9be700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7a80ffad0 0x7fe7a8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7ac9be700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 0x7fe7a8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7ac9be700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a8198560 con 0x7fe7a8105b60 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7ac9be700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a81986a0 con 0x7fe7a80ffad0 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7a5d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 0x7fe7a8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7a5d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 0x7fe7a8197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37684/0 (socket says 172.21.15.96:37684) 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7a5d9b700 1 -- 172.21.15.96:0/1763991843 learned_addr learned my addr 172.21.15.96:0/1763991843 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7a659c700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7a80ffad0 0x7fe7a8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7a5d9b700 1 -- 172.21.15.96:0/1763991843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7a80ffad0 msgr2=0x7fe7a8197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7a5d9b700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7a80ffad0 0x7fe7a8197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7a5d9b700 1 -- 172.21.15.96:0/1763991843 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7980095f0 con 0x7fe7a8105b60 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7a659c700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7a80ffad0 0x7fe7a8197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:42:56.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.600+0000 7fe7a5d9b700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 0x7fe7a8197f90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00b4b0 tx=0x7fe79c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:56.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.601+0000 7fe7977fe700 1 -- 172.21.15.96:0/1763991843 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe79c011800 con 0x7fe7a8105b60 2024-04-22T15:42:56.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.601+0000 7fe7977fe700 1 -- 172.21.15.96:0/1763991843 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe79c011e20 con 0x7fe7a8105b60 2024-04-22T15:42:56.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.601+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a81a0ac0 con 0x7fe7a8105b60 2024-04-22T15:42:56.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.601+0000 7fe7977fe700 1 -- 172.21.15.96:0/1763991843 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe79c00f2e0 con 0x7fe7a8105b60 2024-04-22T15:42:56.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.601+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a81a0ed0 con 0x7fe7a8105b60 2024-04-22T15:42:56.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.602+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a81080d0 con 0x7fe7a8105b60 2024-04-22T15:42:56.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.603+0000 7fe7977fe700 1 -- 172.21.15.96:0/1763991843 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe79c010380 con 0x7fe7a8105b60 2024-04-22T15:42:56.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.603+0000 7fe7977fe700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe79006ecc0 0x7fe790071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:56.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.603+0000 7fe7a659c700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe79006ecc0 0x7fe790071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:56.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.603+0000 7fe7977fe700 1 -- 172.21.15.96:0/1763991843 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe79c08ea50 con 0x7fe7a8105b60 2024-04-22T15:42:56.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.604+0000 7fe7a659c700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe79006ecc0 0x7fe790071180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe798006010 tx=0x7fe79800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:56.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.608+0000 7fe7977fe700 1 -- 172.21.15.96:0/1763991843 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe79c0577b0 con 0x7fe7a8105b60 2024-04-22T15:42:56.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.827+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7a810cd50 con 0x7fe79006ecc0 2024-04-22T15:42:56.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.828+0000 7fe7977fe700 1 -- 172.21.15.96:0/1763991843 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe7a810cd50 con 0x7fe79006ecc0 2024-04-22T15:42:56.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe79006ecc0 msgr2=0x7fe790071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:56.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe79006ecc0 0x7fe790071180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe798006010 tx=0x7fe79800b540 comp rx=0 tx=0).stop 2024-04-22T15:42:56.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 msgr2=0x7fe7a8197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:56.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 0x7fe7a8197f90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00b4b0 tx=0x7fe79c00b870 comp rx=0 tx=0).stop 2024-04-22T15:42:56.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 shutdown_connections 2024-04-22T15:42:56.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe79006ecc0 0x7fe790071180 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7a80ffad0 0x7fe7a8197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 --2- 172.21.15.96:0/1763991843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7a8105b60 0x7fe7a8197f90 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 >> 172.21.15.96:0/1763991843 conn(0x7fe7a80fb600 msgr2=0x7fe7a80fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:56.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 shutdown_connections 2024-04-22T15:42:56.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.830+0000 7fe7ac9be700 1 -- 172.21.15.96:0/1763991843 wait complete. 2024-04-22T15:42:56.843 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:42:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.935+0000 7fce357fd700 1 -- 172.21.15.96:0/2523374887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce301002a0 msgr2=0x7fce3010cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.935+0000 7fce357fd700 1 --2- 172.21.15.96:0/2523374887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce301002a0 0x7fce3010cf60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fce18009910 tx=0x7fce18009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.935+0000 7fce357fd700 1 -- 172.21.15.96:0/2523374887 shutdown_connections 2024-04-22T15:42:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.935+0000 7fce357fd700 1 --2- 172.21.15.96:0/2523374887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce301002a0 0x7fce3010cf60 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.935+0000 7fce357fd700 1 --2- 172.21.15.96:0/2523374887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce300ff8f0 0x7fce300ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.935+0000 7fce357fd700 1 -- 172.21.15.96:0/2523374887 >> 172.21.15.96:0/2523374887 conn(0x7fce30076740 msgr2=0x7fce300fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.935+0000 7fce357fd700 1 -- 172.21.15.96:0/2523374887 shutdown_connections 2024-04-22T15:42:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.935+0000 7fce357fd700 1 -- 172.21.15.96:0/2523374887 wait complete. 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce357fd700 1 Processor -- start 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce357fd700 1 -- start start 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce357fd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce300ff8f0 0x7fce30198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce357fd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce301002a0 0x7fce30199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce357fd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce301998f0 con 0x7fce300ff8f0 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce357fd700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3019c560 con 0x7fce301002a0 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce2effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce300ff8f0 0x7fce30198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce2effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce300ff8f0 0x7fce30198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37694/0 (socket says 172.21.15.96:37694) 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce2effd700 1 -- 172.21.15.96:0/4088664928 learned_addr learned my addr 172.21.15.96:0/4088664928 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce267fc700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce301002a0 0x7fce30199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce2effd700 1 -- 172.21.15.96:0/4088664928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce301002a0 msgr2=0x7fce30199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce2effd700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce301002a0 0x7fce30199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.936+0000 7fce2effd700 1 -- 172.21.15.96:0/4088664928 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce180095f0 con 0x7fce300ff8f0 2024-04-22T15:42:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.937+0000 7fce2effd700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce300ff8f0 0x7fce30198d20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fce2000d900 tx=0x7fce2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.937+0000 7fce2cff9700 1 -- 172.21.15.96:0/4088664928 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce200041f0 con 0x7fce300ff8f0 2024-04-22T15:42:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.937+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3019c840 con 0x7fce300ff8f0 2024-04-22T15:42:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.937+0000 7fce2cff9700 1 -- 172.21.15.96:0/4088664928 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce20004cf0 con 0x7fce300ff8f0 2024-04-22T15:42:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.937+0000 7fce2cff9700 1 -- 172.21.15.96:0/4088664928 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce2000b750 con 0x7fce300ff8f0 2024-04-22T15:42:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.937+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3019cd90 con 0x7fce300ff8f0 2024-04-22T15:42:56.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.938+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce30068ea0 con 0x7fce300ff8f0 2024-04-22T15:42:56.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.938+0000 7fce2cff9700 1 -- 172.21.15.96:0/4088664928 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce20004350 con 0x7fce300ff8f0 2024-04-22T15:42:56.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.939+0000 7fce2cff9700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce1c06ecc0 0x7fce1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:56.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.939+0000 7fce2cff9700 1 -- 172.21.15.96:0/4088664928 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fce2008f000 con 0x7fce300ff8f0 2024-04-22T15:42:56.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.939+0000 7fce267fc700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce1c06ecc0 0x7fce1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:56.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.939+0000 7fce267fc700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce1c06ecc0 0x7fce1c071180 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fce3019a240 tx=0x7fce18005ad0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:56.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:56.943+0000 7fce2cff9700 1 -- 172.21.15.96:0/4088664928 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fce20057d60 con 0x7fce300ff8f0 2024-04-22T15:42:57.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.164+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce30107850 con 0x7fce1c06ecc0 2024-04-22T15:42:57.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.165+0000 7fce2cff9700 1 -- 172.21.15.96:0/4088664928 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fce30107850 con 0x7fce1c06ecc0 2024-04-22T15:42:57.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce1c06ecc0 msgr2=0x7fce1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce1c06ecc0 0x7fce1c071180 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fce3019a240 tx=0x7fce18005ad0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce300ff8f0 msgr2=0x7fce30198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce300ff8f0 0x7fce30198d20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fce2000d900 tx=0x7fce2000dcc0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 shutdown_connections 2024-04-22T15:42:57.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce1c06ecc0 0x7fce1c071180 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce300ff8f0 0x7fce30198d20 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 --2- 172.21.15.96:0/4088664928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce301002a0 0x7fce30199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 >> 172.21.15.96:0/4088664928 conn(0x7fce30076740 msgr2=0x7fce300fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:57.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 shutdown_connections 2024-04-22T15:42:57.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.168+0000 7fce357fd700 1 -- 172.21.15.96:0/4088664928 wait complete. 2024-04-22T15:42:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.271+0000 7f6b14d10700 1 -- 172.21.15.96:0/4001574928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ffd60 msgr2=0x7f6b101058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.271+0000 7f6b14d10700 1 --2- 172.21.15.96:0/4001574928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ffd60 0x7f6b101058a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009910 tx=0x7f6b04009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.271+0000 7f6b14d10700 1 -- 172.21.15.96:0/4001574928 shutdown_connections 2024-04-22T15:42:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.271+0000 7f6b14d10700 1 --2- 172.21.15.96:0/4001574928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ffd60 0x7f6b101058a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.271+0000 7f6b14d10700 1 --2- 172.21.15.96:0/4001574928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b100ff3b0 0x7f6b100ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.271+0000 7f6b14d10700 1 -- 172.21.15.96:0/4001574928 >> 172.21.15.96:0/4001574928 conn(0x7f6b10075f40 msgr2=0x7f6b100fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.271+0000 7f6b14d10700 1 -- 172.21.15.96:0/4001574928 shutdown_connections 2024-04-22T15:42:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.272+0000 7f6b14d10700 1 -- 172.21.15.96:0/4001574928 wait complete. 2024-04-22T15:42:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.272+0000 7f6b14d10700 1 Processor -- start 2024-04-22T15:42:57.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.272+0000 7f6b14d10700 1 -- start start 2024-04-22T15:42:57.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.272+0000 7f6b14d10700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ff3b0 0x7f6b1019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:57.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.272+0000 7f6b14d10700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b100ffd60 0x7f6b1019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:57.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.272+0000 7f6b14d10700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1019d4e0 con 0x7f6b100ff3b0 2024-04-22T15:42:57.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.272+0000 7f6b14d10700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b10196ac0 con 0x7f6b100ffd60 2024-04-22T15:42:57.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b0e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ff3b0 0x7f6b1019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:57.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b0e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ff3b0 0x7f6b1019c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37718/0 (socket says 172.21.15.96:37718) 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b0e59c700 1 -- 172.21.15.96:0/1533656180 learned_addr learned my addr 172.21.15.96:0/1533656180 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b0dd9b700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b100ffd60 0x7f6b1019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b0e59c700 1 -- 172.21.15.96:0/1533656180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b100ffd60 msgr2=0x7f6b1019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b0e59c700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b100ffd60 0x7f6b1019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b0e59c700 1 -- 172.21.15.96:0/1533656180 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b040095f0 con 0x7f6b100ff3b0 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b0e59c700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ff3b0 0x7f6b1019c980 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000b4b0 tx=0x7f6b0000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6aff7fe700 1 -- 172.21.15.96:0/1533656180 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b00011800 con 0x7f6b100ff3b0 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6aff7fe700 1 -- 172.21.15.96:0/1533656180 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b00011e20 con 0x7f6b100ff3b0 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b10196da0 con 0x7f6b100ff3b0 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6aff7fe700 1 -- 172.21.15.96:0/1533656180 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b0000f2e0 con 0x7f6b100ff3b0 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.273+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b101972c0 con 0x7f6b100ff3b0 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.274+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b10109210 con 0x7f6b100ff3b0 2024-04-22T15:42:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.275+0000 7f6aff7fe700 1 -- 172.21.15.96:0/1533656180 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6b00010380 con 0x7f6b100ff3b0 2024-04-22T15:42:57.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.275+0000 7f6aff7fe700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6af806ecd0 0x7f6af8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:57.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.275+0000 7f6b0dd9b700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6af806ecd0 0x7f6af8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:57.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.275+0000 7f6aff7fe700 1 -- 172.21.15.96:0/1533656180 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6b0008e9d0 con 0x7f6b100ff3b0 2024-04-22T15:42:57.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.276+0000 7f6b0dd9b700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6af806ecd0 0x7f6af8071190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6b04005f50 tx=0x7f6b0401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:57.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.280+0000 7f6aff7fe700 1 -- 172.21.15.96:0/1533656180 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6b0005b260 con 0x7f6b100ff3b0 2024-04-22T15:42:57.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.487+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b10197ff0 con 0x7f6af806ecd0 2024-04-22T15:42:57.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.497+0000 7f6aff7fe700 1 -- 172.21.15.96:0/1533656180 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6b10197ff0 con 0x7f6af806ecd0 2024-04-22T15:42:57.500 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:42:57.500 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (21m) 14m ago 32m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (33m) 14m ago 33m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (32m) 14m ago 32m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (18m) 14m ago 33m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (18m) 14m ago 32m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (21m) 14m ago 32m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (14m) 14m ago 27m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (15m) 14m ago 27m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (14m) 14m ago 27m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (27m) 14m ago 27m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (24m) 14m ago 34m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (22m) 14m ago 31m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (19m) 14m ago 34m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (19m) 14m ago 31m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:42:57.501 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (22m) 14m ago 33m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:42:57.502 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (22m) 14m ago 31m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:42:57.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (18m) 14m ago 31m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:42:57.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (17m) 14m ago 30m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:42:57.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (17m) 14m ago 30m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:42:57.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (16m) 14m ago 29m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:42:57.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (16m) 14m ago 29m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:42:57.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (15m) 14m ago 28m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:42:57.502 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (21m) 14m ago 32m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.499+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6af806ecd0 msgr2=0x7f6af8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6af806ecd0 0x7f6af8071190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6b04005f50 tx=0x7f6b0401a040 comp rx=0 tx=0).stop 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ff3b0 msgr2=0x7f6b1019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ff3b0 0x7f6b1019c980 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000b4b0 tx=0x7f6b0000b7c0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 shutdown_connections 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6af806ecd0 0x7f6af8071190 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b100ff3b0 0x7f6b1019c980 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 --2- 172.21.15.96:0/1533656180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b100ffd60 0x7f6b1019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 >> 172.21.15.96:0/1533656180 conn(0x7f6b10075f40 msgr2=0x7f6b10104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:57.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 shutdown_connections 2024-04-22T15:42:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.500+0000 7f6b14d10700 1 -- 172.21.15.96:0/1533656180 wait complete. 2024-04-22T15:42:57.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.603+0000 7efff4de4700 1 -- 172.21.15.96:0/65919360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff00ffbc0 msgr2=0x7efff0100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.603+0000 7efff4de4700 1 --2- 172.21.15.96:0/65919360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff00ffbc0 0x7efff0100020 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7effe0009910 tx=0x7effe0009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:57.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.603+0000 7efff4de4700 1 -- 172.21.15.96:0/65919360 shutdown_connections 2024-04-22T15:42:57.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.603+0000 7efff4de4700 1 --2- 172.21.15.96:0/65919360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff00ffbc0 0x7efff0100020 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.603+0000 7efff4de4700 1 --2- 172.21.15.96:0/65919360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efff0105a30 0x7efff0105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.603+0000 7efff4de4700 1 -- 172.21.15.96:0/65919360 >> 172.21.15.96:0/65919360 conn(0x7efff0076790 msgr2=0x7efff0076ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.603+0000 7efff4de4700 1 -- 172.21.15.96:0/65919360 shutdown_connections 2024-04-22T15:42:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.603+0000 7efff4de4700 1 -- 172.21.15.96:0/65919360 wait complete. 2024-04-22T15:42:57.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.604+0000 7efff4de4700 1 Processor -- start 2024-04-22T15:42:57.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.604+0000 7efff4de4700 1 -- start start 2024-04-22T15:42:57.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.604+0000 7efff4de4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efff00ffbc0 0x7efff019b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:57.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.604+0000 7efff4de4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff0105a30 0x7efff019bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:57.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.604+0000 7efff4de4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff019c230 con 0x7efff0105a30 2024-04-22T15:42:57.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.604+0000 7efff4de4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff01957f0 con 0x7efff00ffbc0 2024-04-22T15:42:57.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.604+0000 7effedd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff0105a30 0x7efff019bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.604+0000 7effedd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff0105a30 0x7efff019bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37738/0 (socket says 172.21.15.96:37738) 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7effedd9b700 1 -- 172.21.15.96:0/1590168150 learned_addr learned my addr 172.21.15.96:0/1590168150 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7effee59c700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efff00ffbc0 0x7efff019b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7effedd9b700 1 -- 172.21.15.96:0/1590168150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efff00ffbc0 msgr2=0x7efff019b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7effedd9b700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efff00ffbc0 0x7efff019b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7effedd9b700 1 -- 172.21.15.96:0/1590168150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe00095f0 con 0x7efff0105a30 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7effedd9b700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff0105a30 0x7efff019bba0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7effe0000c00 tx=0x7effe000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7effdf7fe700 1 -- 172.21.15.96:0/1590168150 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe001d070 con 0x7efff0105a30 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff0195a70 con 0x7efff0105a30 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7effdf7fe700 1 -- 172.21.15.96:0/1590168150 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effe00056e0 con 0x7efff0105a30 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7effdf7fe700 1 -- 172.21.15.96:0/1590168150 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe000fe40 con 0x7efff0105a30 2024-04-22T15:42:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.605+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff0195f60 con 0x7efff0105a30 2024-04-22T15:42:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.606+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff0107fc0 con 0x7efff0105a30 2024-04-22T15:42:57.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.607+0000 7effdf7fe700 1 -- 172.21.15.96:0/1590168150 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7effe000f680 con 0x7efff0105a30 2024-04-22T15:42:57.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.607+0000 7effdf7fe700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effd806ec70 0x7effd8071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:57.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.607+0000 7effdf7fe700 1 -- 172.21.15.96:0/1590168150 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7effe0090260 con 0x7efff0105a30 2024-04-22T15:42:57.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.607+0000 7effee59c700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effd806ec70 0x7effd8071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.607+0000 7effee59c700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effd806ec70 0x7effd8071130 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7effe4005950 tx=0x7effe400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:57.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.612+0000 7effdf7fe700 1 -- 172.21.15.96:0/1590168150 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7effe005caf0 con 0x7efff0105a30 2024-04-22T15:42:57.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.895+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efff0196c60 con 0x7efff0105a30 2024-04-22T15:42:57.898 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:57 smithi096 ceph-mon[112443]: pgmap v632: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.895+0000 7effdf7fe700 1 -- 172.21.15.96:0/1590168150 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7effe005c680 con 0x7efff0105a30 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:42:57.899 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:42:57.900 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:42:57.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.897+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effd806ec70 msgr2=0x7effd8071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effd806ec70 0x7effd8071130 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7effe4005950 tx=0x7effe400b410 comp rx=0 tx=0).stop 2024-04-22T15:42:57.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff0105a30 msgr2=0x7efff019bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:57.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff0105a30 0x7efff019bba0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7effe0000c00 tx=0x7effe000bd70 comp rx=0 tx=0).stop 2024-04-22T15:42:57.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 shutdown_connections 2024-04-22T15:42:57.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effd806ec70 0x7effd8071130 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efff00ffbc0 0x7efff019b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 --2- 172.21.15.96:0/1590168150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efff0105a30 0x7efff019bba0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:57.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 >> 172.21.15.96:0/1590168150 conn(0x7efff0076790 msgr2=0x7efff00ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:57.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 shutdown_connections 2024-04-22T15:42:57.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:57.898+0000 7efff4de4700 1 -- 172.21.15.96:0/1590168150 wait complete. 2024-04-22T15:42:58.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.001+0000 7fc2da4ac700 1 -- 172.21.15.96:0/2726212846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d40ffaf0 msgr2=0x7fc2d40fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.001+0000 7fc2da4ac700 1 --2- 172.21.15.96:0/2726212846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d40ffaf0 0x7fc2d40fff70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009910 tx=0x7fc2c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.001+0000 7fc2da4ac700 1 -- 172.21.15.96:0/2726212846 shutdown_connections 2024-04-22T15:42:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.001+0000 7fc2da4ac700 1 --2- 172.21.15.96:0/2726212846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d40ffaf0 0x7fc2d40fff70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.001+0000 7fc2da4ac700 1 --2- 172.21.15.96:0/2726212846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2d4105b80 0x7fc2d4105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.001+0000 7fc2da4ac700 1 -- 172.21.15.96:0/2726212846 >> 172.21.15.96:0/2726212846 conn(0x7fc2d40fb5e0 msgr2=0x7fc2d40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.001+0000 7fc2da4ac700 1 -- 172.21.15.96:0/2726212846 shutdown_connections 2024-04-22T15:42:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.001+0000 7fc2da4ac700 1 -- 172.21.15.96:0/2726212846 wait complete. 2024-04-22T15:42:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.002+0000 7fc2da4ac700 1 Processor -- start 2024-04-22T15:42:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.002+0000 7fc2da4ac700 1 -- start start 2024-04-22T15:42:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.002+0000 7fc2da4ac700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2d40ffaf0 0x7fc2d4195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.002+0000 7fc2da4ac700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d4105b80 0x7fc2d4195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.002+0000 7fc2da4ac700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d41965f0 con 0x7fc2d4105b80 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.002+0000 7fc2da4ac700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d419a340 con 0x7fc2d40ffaf0 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.002+0000 7fc2d37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d4105b80 0x7fc2d4195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d4105b80 0x7fc2d4195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37762/0 (socket says 172.21.15.96:37762) 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d37fe700 1 -- 172.21.15.96:0/3355249836 learned_addr learned my addr 172.21.15.96:0/3355249836 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d3fff700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2d40ffaf0 0x7fc2d4195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d37fe700 1 -- 172.21.15.96:0/3355249836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2d40ffaf0 msgr2=0x7fc2d4195a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d37fe700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2d40ffaf0 0x7fc2d4195a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d37fe700 1 -- 172.21.15.96:0/3355249836 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c80095f0 con 0x7fc2d4105b80 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d3fff700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2d40ffaf0 0x7fc2d4195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:42:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d37fe700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d4105b80 0x7fc2d4195f60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800bd70 tx=0x7fc2c800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:58.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d17fa700 1 -- 172.21.15.96:0/3355249836 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2c801d070 con 0x7fc2d4105b80 2024-04-22T15:42:58.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d419a5c0 con 0x7fc2d4105b80 2024-04-22T15:42:58.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d17fa700 1 -- 172.21.15.96:0/3355249836 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2c80056e0 con 0x7fc2d4105b80 2024-04-22T15:42:58.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2d17fa700 1 -- 172.21.15.96:0/3355249836 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2c8017400 con 0x7fc2d4105b80 2024-04-22T15:42:58.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.003+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d419aab0 con 0x7fc2d4105b80 2024-04-22T15:42:58.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:57 smithi196 ceph-mon[81815]: pgmap v632: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.004+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d4108110 con 0x7fc2d4105b80 2024-04-22T15:42:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.005+0000 7fc2d17fa700 1 -- 172.21.15.96:0/3355249836 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc2c8003a60 con 0x7fc2d4105b80 2024-04-22T15:42:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.005+0000 7fc2d17fa700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2bc06ecc0 0x7fc2bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.005+0000 7fc2d17fa700 1 -- 172.21.15.96:0/3355249836 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc2c80902f0 con 0x7fc2d4105b80 2024-04-22T15:42:58.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.005+0000 7fc2d3fff700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2bc06ecc0 0x7fc2bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:58.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.005+0000 7fc2d3fff700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2bc06ecc0 0x7fc2bc071180 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005950 tx=0x7fc2c40058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:58.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.010+0000 7fc2d17fa700 1 -- 172.21.15.96:0/3355249836 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc2c805cb80 con 0x7fc2d4105b80 2024-04-22T15:42:58.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.255+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc2d41011f0 con 0x7fc2d4105b80 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.256+0000 7fc2d17fa700 1 -- 172.21.15.96:0/3355249836 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7fc2c805c710 con 0x7fc2d4105b80 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:42:58.259 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:42:58.260 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:42:58.261 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:42:58.262 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:42:58.262 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:42:58.262 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:42:58.262 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:42:58.262 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2bc06ecc0 msgr2=0x7fc2bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2bc06ecc0 0x7fc2bc071180 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005950 tx=0x7fc2c40058e0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d4105b80 msgr2=0x7fc2d4195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d4105b80 0x7fc2d4195f60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800bd70 tx=0x7fc2c800be50 comp rx=0 tx=0).stop 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 shutdown_connections 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc2bc06ecc0 0x7fc2bc071180 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2d40ffaf0 0x7fc2d4195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 --2- 172.21.15.96:0/3355249836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2d4105b80 0x7fc2d4195f60 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 >> 172.21.15.96:0/3355249836 conn(0x7fc2d40fb5e0 msgr2=0x7fc2d40fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 shutdown_connections 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.258+0000 7fc2da4ac700 1 -- 172.21.15.96:0/3355249836 wait complete. 2024-04-22T15:42:58.263 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:42:58.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.363+0000 7fa8e12c5700 1 -- 172.21.15.96:0/2239137910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc100470 msgr2=0x7fa8dc104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.363+0000 7fa8e12c5700 1 --2- 172.21.15.96:0/2239137910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc100470 0x7fa8dc104940 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4009910 tx=0x7fa8c4009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.363+0000 7fa8e12c5700 1 -- 172.21.15.96:0/2239137910 shutdown_connections 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.363+0000 7fa8e12c5700 1 --2- 172.21.15.96:0/2239137910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc100470 0x7fa8dc104940 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.363+0000 7fa8e12c5700 1 --2- 172.21.15.96:0/2239137910 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8dc0ffb50 0x7fa8dc0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.363+0000 7fa8e12c5700 1 -- 172.21.15.96:0/2239137910 >> 172.21.15.96:0/2239137910 conn(0x7fa8dc076740 msgr2=0x7fa8dc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.363+0000 7fa8e12c5700 1 -- 172.21.15.96:0/2239137910 shutdown_connections 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.363+0000 7fa8e12c5700 1 -- 172.21.15.96:0/2239137910 wait complete. 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8e12c5700 1 Processor -- start 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8e12c5700 1 -- start start 2024-04-22T15:42:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8e12c5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc0ffb50 0x7fa8dc19d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8e12c5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8dc100470 0x7fa8dc19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8e12c5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc19dd30 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8e12c5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc1a1a00 con 0x7fa8dc100470 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8daffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc0ffb50 0x7fa8dc19d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8daffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc0ffb50 0x7fa8dc19d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37776/0 (socket says 172.21.15.96:37776) 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8daffd700 1 -- 172.21.15.96:0/3463901369 learned_addr learned my addr 172.21.15.96:0/3463901369 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8d3fff700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8dc100470 0x7fa8dc19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8daffd700 1 -- 172.21.15.96:0/3463901369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8dc100470 msgr2=0x7fa8dc19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8daffd700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8dc100470 0x7fa8dc19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8daffd700 1 -- 172.21.15.96:0/3463901369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c40095f0 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8d3fff700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8dc100470 0x7fa8dc19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.364+0000 7fa8daffd700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc0ffb50 0x7fa8dc19d110 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00d8d0 tx=0x7fa8cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:58.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.365+0000 7fa8d8ff9700 1 -- 172.21.15.96:0/3463901369 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8cc009940 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.365+0000 7fa8d8ff9700 1 -- 172.21.15.96:0/3463901369 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8cc010460 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.365+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8dc1a1ce0 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.365+0000 7fa8d8ff9700 1 -- 172.21.15.96:0/3463901369 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8cc00f5f0 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.365+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8dc1a2230 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.366+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8dc068ea0 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.366+0000 7fa8d8ff9700 1 -- 172.21.15.96:0/3463901369 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa8cc0105d0 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.367+0000 7fa8d8ff9700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa8c806ecc0 0x7fa8c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:58.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.367+0000 7fa8d8ff9700 1 -- 172.21.15.96:0/3463901369 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa8cc08ed90 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.367+0000 7fa8d3fff700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa8c806ecc0 0x7fa8c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:58.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.367+0000 7fa8d3fff700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa8c806ecc0 0x7fa8c8071180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4006010 tx=0x7fa8c400b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:58.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.371+0000 7fa8d8ff9700 1 -- 172.21.15.96:0/3463901369 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa8cc057b70 con 0x7fa8dc0ffb50 2024-04-22T15:42:58.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.595+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8dc114580 con 0x7fa8c806ecc0 2024-04-22T15:42:58.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.596+0000 7fa8d8ff9700 1 -- 172.21.15.96:0/3463901369 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa8dc114580 con 0x7fa8c806ecc0 2024-04-22T15:42:58.599 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:42:58.599 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:42:58.599 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:42:58.599 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:42:58.599 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:42:58.599 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:42:58.600 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:42:58.600 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:42:58.600 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:42:58.600 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:42:58.600 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:42:58.600 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:42:58.600 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:42:58.600 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:42:58.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa8c806ecc0 msgr2=0x7fa8c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa8c806ecc0 0x7fa8c8071180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4006010 tx=0x7fa8c400b540 comp rx=0 tx=0).stop 2024-04-22T15:42:58.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc0ffb50 msgr2=0x7fa8dc19d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc0ffb50 0x7fa8dc19d110 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00d8d0 tx=0x7fa8cc00dc90 comp rx=0 tx=0).stop 2024-04-22T15:42:58.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 shutdown_connections 2024-04-22T15:42:58.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa8c806ecc0 0x7fa8c8071180 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8dc0ffb50 0x7fa8dc19d110 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 --2- 172.21.15.96:0/3463901369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8dc100470 0x7fa8dc19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 >> 172.21.15.96:0/3463901369 conn(0x7fa8dc076740 msgr2=0x7fa8dc10e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:58.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 shutdown_connections 2024-04-22T15:42:58.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.598+0000 7fa8e12c5700 1 -- 172.21.15.96:0/3463901369 wait complete. 2024-04-22T15:42:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.702+0000 7f06bbd52700 1 -- 172.21.15.96:0/1855221041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b40ffaa0 msgr2=0x7f06b40fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.702+0000 7f06bbd52700 1 --2- 172.21.15.96:0/1855221041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b40ffaa0 0x7f06b40fff20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f06b0009910 tx=0x7f06b0009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.702+0000 7f06bbd52700 1 -- 172.21.15.96:0/1855221041 shutdown_connections 2024-04-22T15:42:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.702+0000 7f06bbd52700 1 --2- 172.21.15.96:0/1855221041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b40ffaa0 0x7f06b40fff20 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.702+0000 7f06bbd52700 1 --2- 172.21.15.96:0/1855221041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06b4105aa0 0x7f06b4105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.702+0000 7f06bbd52700 1 -- 172.21.15.96:0/1855221041 >> 172.21.15.96:0/1855221041 conn(0x7f06b4076740 msgr2=0x7f06b40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.702+0000 7f06bbd52700 1 -- 172.21.15.96:0/1855221041 shutdown_connections 2024-04-22T15:42:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.702+0000 7f06bbd52700 1 -- 172.21.15.96:0/1855221041 wait complete. 2024-04-22T15:42:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06bbd52700 1 Processor -- start 2024-04-22T15:42:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06bbd52700 1 -- start start 2024-04-22T15:42:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06bbd52700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06b40ffaa0 0x7f06b419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06bbd52700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4105aa0 0x7f06b419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06bbd52700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b419d2d0 con 0x7f06b4105aa0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06bbd52700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b4196940 con 0x7f06b40ffaa0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06b92ed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4105aa0 0x7f06b419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06b92ed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4105aa0 0x7f06b419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37786/0 (socket says 172.21.15.96:37786) 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06b92ed700 1 -- 172.21.15.96:0/297582594 learned_addr learned my addr 172.21.15.96:0/297582594 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06b92ed700 1 -- 172.21.15.96:0/297582594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06b40ffaa0 msgr2=0x7f06b419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06b92ed700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06b40ffaa0 0x7f06b419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06b92ed700 1 -- 172.21.15.96:0/297582594 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b00095f0 con 0x7f06b4105aa0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06b92ed700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4105aa0 0x7f06b419cd40 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f06b0006010 tx=0x7f06b000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06aaffd700 1 -- 172.21.15.96:0/297582594 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06b001d070 con 0x7f06b4105aa0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06aaffd700 1 -- 172.21.15.96:0/297582594 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06b000f460 con 0x7f06b4105aa0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06aaffd700 1 -- 172.21.15.96:0/297582594 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06b0017630 con 0x7f06b4105aa0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.703+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b4196bc0 con 0x7f06b4105aa0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.704+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b4197060 con 0x7f06b4105aa0 2024-04-22T15:42:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.705+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b41011e0 con 0x7f06b4105aa0 2024-04-22T15:42:58.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.705+0000 7f06aaffd700 1 -- 172.21.15.96:0/297582594 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f06b000f5d0 con 0x7f06b4105aa0 2024-04-22T15:42:58.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.705+0000 7f06aaffd700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f06a006ecc0 0x7f06a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:58.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.706+0000 7f06aaffd700 1 -- 172.21.15.96:0/297582594 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f06b00903a0 con 0x7f06b4105aa0 2024-04-22T15:42:58.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.706+0000 7f06b9aee700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f06a006ecc0 0x7f06a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:58.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.706+0000 7f06b9aee700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f06a006ecc0 0x7f06a0071180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f06a4005fd0 tx=0x7f06a4005d00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:58.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.710+0000 7f06aaffd700 1 -- 172.21.15.96:0/297582594 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f06b005cd10 con 0x7f06b4105aa0 2024-04-22T15:42:58.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:58 smithi096 ceph-mon[112443]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:58.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:58 smithi096 ceph-mon[112443]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:58 smithi096 ceph-mon[112443]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:58 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1590168150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:42:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:58 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3355249836' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:42:58.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.991+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f06b404ebb0 con 0x7f06b4105aa0 2024-04-22T15:42:58.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.991+0000 7f06aaffd700 1 -- 172.21.15.96:0/297582594 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f06b005c8a0 con 0x7f06b4105aa0 2024-04-22T15:42:58.994 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:42:58.994 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:42:58.994 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:42:58.994 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:42:58.995 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:42:58.995 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:42:58.995 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:42:58.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f06a006ecc0 msgr2=0x7f06a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f06a006ecc0 0x7f06a0071180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f06a4005fd0 tx=0x7f06a4005d00 comp rx=0 tx=0).stop 2024-04-22T15:42:58.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4105aa0 msgr2=0x7f06b419cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:58.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4105aa0 0x7f06b419cd40 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f06b0006010 tx=0x7f06b000be20 comp rx=0 tx=0).stop 2024-04-22T15:42:58.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 shutdown_connections 2024-04-22T15:42:58.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f06a006ecc0 0x7f06a0071180 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06b40ffaa0 0x7f06b419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 --2- 172.21.15.96:0/297582594 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4105aa0 0x7f06b419cd40 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:58.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 >> 172.21.15.96:0/297582594 conn(0x7f06b4076740 msgr2=0x7f06b4104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:58.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 shutdown_connections 2024-04-22T15:42:58.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:42:58.993+0000 7f06bbd52700 1 -- 172.21.15.96:0/297582594 wait complete. 2024-04-22T15:42:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:58 smithi196 ceph-mon[81815]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:58 smithi196 ceph-mon[81815]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:58 smithi196 ceph-mon[81815]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:58 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1590168150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:42:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:58 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3355249836' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:42:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:59 smithi096 ceph-mon[112443]: pgmap v633: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:42:59.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:42:59 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/297582594' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:43:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:59 smithi196 ceph-mon[81815]: pgmap v633: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:42:59 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/297582594' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:43:00.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:00 smithi196 ceph-mon[81815]: from='client.35200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:00.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:00 smithi096 ceph-mon[112443]: from='client.35200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:01.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:01 smithi096 ceph-mon[112443]: pgmap v634: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:01 smithi196 ceph-mon[81815]: pgmap v634: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:03.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:03 smithi096 ceph-mon[112443]: pgmap v635: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:03 smithi196 ceph-mon[81815]: pgmap v635: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:05.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:05 smithi196 ceph-mon[81815]: pgmap v636: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:05.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:05 smithi096 ceph-mon[112443]: pgmap v636: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:07 smithi096 ceph-mon[112443]: pgmap v637: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:07 smithi196 ceph-mon[81815]: pgmap v637: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:09.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:09 smithi096 ceph-mon[112443]: pgmap v638: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:09 smithi196 ceph-mon[81815]: pgmap v638: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:11.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:11 smithi096 ceph-mon[112443]: pgmap v639: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:11 smithi196 ceph-mon[81815]: pgmap v639: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:12.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:12 smithi096 ceph-mon[112443]: pgmap v640: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:12 smithi196 ceph-mon[81815]: pgmap v640: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:15.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:15 smithi096 ceph-mon[112443]: pgmap v641: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:15 smithi196 ceph-mon[81815]: pgmap v641: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:17.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:17 smithi096 ceph-mon[112443]: pgmap v642: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:17 smithi196 ceph-mon[81815]: pgmap v642: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:19 smithi096 ceph-mon[112443]: pgmap v643: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:19 smithi196 ceph-mon[81815]: pgmap v643: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:21.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:21 smithi096 ceph-mon[112443]: pgmap v644: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:21 smithi196 ceph-mon[81815]: pgmap v644: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:23 smithi096 ceph-mon[112443]: pgmap v645: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:23 smithi196 ceph-mon[81815]: pgmap v645: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:25.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:25 smithi096 ceph-mon[112443]: pgmap v646: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:25 smithi196 ceph-mon[81815]: pgmap v646: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:27.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:27 smithi096 ceph-mon[112443]: pgmap v647: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:27 smithi196 ceph-mon[81815]: pgmap v647: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:29.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.109+0000 7fd9f904b700 1 -- 172.21.15.96:0/1778246732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 msgr2=0x7fd9f41047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.109+0000 7fd9f904b700 1 --2- 172.21.15.96:0/1778246732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 0x7fd9f41047a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009910 tx=0x7fd9e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:29.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.109+0000 7fd9f904b700 1 -- 172.21.15.96:0/1778246732 shutdown_connections 2024-04-22T15:43:29.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.109+0000 7fd9f904b700 1 --2- 172.21.15.96:0/1778246732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 0x7fd9f41047a0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.109+0000 7fd9f904b700 1 --2- 172.21.15.96:0/1778246732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f4100420 0x7fd9f4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.109+0000 7fd9f904b700 1 -- 172.21.15.96:0/1778246732 >> 172.21.15.96:0/1778246732 conn(0x7fd9f4075f40 msgr2=0x7fd9f40fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:29.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.110+0000 7fd9f904b700 1 -- 172.21.15.96:0/1778246732 shutdown_connections 2024-04-22T15:43:29.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.110+0000 7fd9f904b700 1 -- 172.21.15.96:0/1778246732 wait complete. 2024-04-22T15:43:29.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.110+0000 7fd9f904b700 1 Processor -- start 2024-04-22T15:43:29.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f904b700 1 -- start start 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f904b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f4100420 0x7fd9f41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f904b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 0x7fd9f4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f904b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f4198460 con 0x7fd9f4100dd0 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f904b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f41985a0 con 0x7fd9f4100420 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 0x7fd9f4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 0x7fd9f4197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53478/0 (socket says 172.21.15.96:53478) 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f259c700 1 -- 172.21.15.96:0/1673584655 learned_addr learned my addr 172.21.15.96:0/1673584655 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f2d9d700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f4100420 0x7fd9f41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f259c700 1 -- 172.21.15.96:0/1673584655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f4100420 msgr2=0x7fd9f41979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f259c700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f4100420 0x7fd9f41979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f259c700 1 -- 172.21.15.96:0/1673584655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e80095f0 con 0x7fd9f4100dd0 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f2d9d700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f4100420 0x7fd9f41979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.111+0000 7fd9f259c700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 0x7fd9f4197f20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800bd70 tx=0x7fd9e800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.112+0000 7fd9dbfff700 1 -- 172.21.15.96:0/1673584655 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9e801d070 con 0x7fd9f4100dd0 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.112+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f4198820 con 0x7fd9f4100dd0 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.112+0000 7fd9dbfff700 1 -- 172.21.15.96:0/1673584655 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9e800f460 con 0x7fd9f4100dd0 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.112+0000 7fd9dbfff700 1 -- 172.21.15.96:0/1673584655 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9e8017610 con 0x7fd9f4100dd0 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.112+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f41a0ce0 con 0x7fd9f4100dd0 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.113+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f4108130 con 0x7fd9f4100dd0 2024-04-22T15:43:29.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.113+0000 7fd9dbfff700 1 -- 172.21.15.96:0/1673584655 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd9e800fa50 con 0x7fd9f4100dd0 2024-04-22T15:43:29.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.114+0000 7fd9dbfff700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd9dc06ecc0 0x7fd9dc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:29.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.114+0000 7fd9dbfff700 1 -- 172.21.15.96:0/1673584655 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd9e80903a0 con 0x7fd9f4100dd0 2024-04-22T15:43:29.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.114+0000 7fd9f2d9d700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd9dc06ecc0 0x7fd9dc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:29.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.114+0000 7fd9f2d9d700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd9dc06ecc0 0x7fd9dc071180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4005fd0 tx=0x7fd9e4005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:29.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.119+0000 7fd9dbfff700 1 -- 172.21.15.96:0/1673584655 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd9e8059120 con 0x7fd9f4100dd0 2024-04-22T15:43:29.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.335+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9f410cd20 con 0x7fd9dc06ecc0 2024-04-22T15:43:29.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.336+0000 7fd9dbfff700 1 -- 172.21.15.96:0/1673584655 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd9f410cd20 con 0x7fd9dc06ecc0 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.338+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd9dc06ecc0 msgr2=0x7fd9dc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.338+0000 7fd9f904b700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd9dc06ecc0 0x7fd9dc071180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4005fd0 tx=0x7fd9e4005e20 comp rx=0 tx=0).stop 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.338+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 msgr2=0x7fd9f4197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.338+0000 7fd9f904b700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 0x7fd9f4197f20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800bd70 tx=0x7fd9e800be50 comp rx=0 tx=0).stop 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.339+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 shutdown_connections 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.339+0000 7fd9f904b700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd9dc06ecc0 0x7fd9dc071180 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.339+0000 7fd9f904b700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f4100420 0x7fd9f41979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.339+0000 7fd9f904b700 1 --2- 172.21.15.96:0/1673584655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f4100dd0 0x7fd9f4197f20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.339+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 >> 172.21.15.96:0/1673584655 conn(0x7fd9f4075f40 msgr2=0x7fd9f41035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.339+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 shutdown_connections 2024-04-22T15:43:29.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.339+0000 7fd9f904b700 1 -- 172.21.15.96:0/1673584655 wait complete. 2024-04-22T15:43:29.352 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:43:29.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.448+0000 7ff92f59f700 1 -- 172.21.15.96:0/1047303611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 msgr2=0x7ff928105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.448+0000 7ff92f59f700 1 --2- 172.21.15.96:0/1047303611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 0x7ff928105820 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff918009910 tx=0x7ff918009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:29.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.448+0000 7ff92f59f700 1 -- 172.21.15.96:0/1047303611 shutdown_connections 2024-04-22T15:43:29.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.448+0000 7ff92f59f700 1 --2- 172.21.15.96:0/1047303611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 0x7ff928105820 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.448+0000 7ff92f59f700 1 --2- 172.21.15.96:0/1047303611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff928100e20 0x7ff928101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.448+0000 7ff92f59f700 1 -- 172.21.15.96:0/1047303611 >> 172.21.15.96:0/1047303611 conn(0x7ff9280fc6d0 msgr2=0x7ff9280feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.448+0000 7ff92f59f700 1 -- 172.21.15.96:0/1047303611 shutdown_connections 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.448+0000 7ff92f59f700 1 -- 172.21.15.96:0/1047303611 wait complete. 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92f59f700 1 Processor -- start 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92f59f700 1 -- start start 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92f59f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff928100e20 0x7ff92819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92f59f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 0x7ff92819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92f59f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92819d2f0 con 0x7ff9281017d0 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92f59f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff928196960 con 0x7ff928100e20 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92cb3a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 0x7ff92819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:29.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92cb3a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 0x7ff92819cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53504/0 (socket says 172.21.15.96:53504) 2024-04-22T15:43:29.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92cb3a700 1 -- 172.21.15.96:0/341150930 learned_addr learned my addr 172.21.15.96:0/341150930 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:43:29.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.449+0000 7ff92d33b700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff928100e20 0x7ff92819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:29.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.450+0000 7ff92cb3a700 1 -- 172.21.15.96:0/341150930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff928100e20 msgr2=0x7ff92819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.450+0000 7ff92cb3a700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff928100e20 0x7ff92819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.450+0000 7ff92cb3a700 1 -- 172.21.15.96:0/341150930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9180095f0 con 0x7ff9281017d0 2024-04-22T15:43:29.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.450+0000 7ff92cb3a700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 0x7ff92819cd60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff9180092e0 tx=0x7ff91800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:29.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.450+0000 7ff91e7fc700 1 -- 172.21.15.96:0/341150930 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91801d070 con 0x7ff9281017d0 2024-04-22T15:43:29.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.450+0000 7ff91e7fc700 1 -- 172.21.15.96:0/341150930 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff91800f460 con 0x7ff9281017d0 2024-04-22T15:43:29.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.450+0000 7ff91e7fc700 1 -- 172.21.15.96:0/341150930 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff918017630 con 0x7ff9281017d0 2024-04-22T15:43:29.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.451+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff928196be0 con 0x7ff9281017d0 2024-04-22T15:43:29.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.451+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff928197080 con 0x7ff9281017d0 2024-04-22T15:43:29.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.452+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff928109220 con 0x7ff9281017d0 2024-04-22T15:43:29.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.452+0000 7ff91e7fc700 1 -- 172.21.15.96:0/341150930 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff91800f5d0 con 0x7ff9281017d0 2024-04-22T15:43:29.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.453+0000 7ff91e7fc700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff91406ecc0 0x7ff914071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:29.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.453+0000 7ff92d33b700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff91406ecc0 0x7ff914071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:29.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.453+0000 7ff91e7fc700 1 -- 172.21.15.96:0/341150930 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff918090250 con 0x7ff9281017d0 2024-04-22T15:43:29.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.453+0000 7ff92d33b700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff91406ecc0 0x7ff914071180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff92400ac50 tx=0x7ff92400a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:29.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.459+0000 7ff91e7fc700 1 -- 172.21.15.96:0/341150930 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff91805cae0 con 0x7ff9281017d0 2024-04-22T15:43:29.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.708+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff928197e20 con 0x7ff91406ecc0 2024-04-22T15:43:29.712 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:29 smithi096 ceph-mon[112443]: pgmap v648: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:29.712 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:29 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:29.712 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:29 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:29.712 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:29 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:29.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.710+0000 7ff91e7fc700 1 -- 172.21.15.96:0/341150930 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff928197e20 con 0x7ff91406ecc0 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff91406ecc0 msgr2=0x7ff914071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff91406ecc0 0x7ff914071180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff92400ac50 tx=0x7ff92400a380 comp rx=0 tx=0).stop 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 msgr2=0x7ff92819cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 0x7ff92819cd60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff9180092e0 tx=0x7ff91800bd70 comp rx=0 tx=0).stop 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 shutdown_connections 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff91406ecc0 0x7ff914071180 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff928100e20 0x7ff92819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 --2- 172.21.15.96:0/341150930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9281017d0 0x7ff92819cd60 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 >> 172.21.15.96:0/341150930 conn(0x7ff9280fc6d0 msgr2=0x7ff9280fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 shutdown_connections 2024-04-22T15:43:29.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.712+0000 7ff92f59f700 1 -- 172.21.15.96:0/341150930 wait complete. 2024-04-22T15:43:29.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/3729207646 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e140ffaf0 msgr2=0x7f3e140fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 --2- 172.21.15.96:0/3729207646 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e140ffaf0 0x7f3e140fff70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009910 tx=0x7f3e08009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:29.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/3729207646 shutdown_connections 2024-04-22T15:43:29.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 --2- 172.21.15.96:0/3729207646 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e140ffaf0 0x7f3e140fff70 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 --2- 172.21.15.96:0/3729207646 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e14105b80 0x7f3e14105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/3729207646 >> 172.21.15.96:0/3729207646 conn(0x7f3e140fb5e0 msgr2=0x7f3e140fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:29.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/3729207646 shutdown_connections 2024-04-22T15:43:29.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/3729207646 wait complete. 2024-04-22T15:43:29.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 Processor -- start 2024-04-22T15:43:29.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.815+0000 7f3e1a9ea700 1 -- start start 2024-04-22T15:43:29.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e1a9ea700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e140ffaf0 0x7f3e141979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:29.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e1a9ea700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14105b80 0x7f3e14197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:29.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e1a9ea700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e14198430 con 0x7f3e14105b80 2024-04-22T15:43:29.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e1a9ea700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e14198570 con 0x7f3e140ffaf0 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e137fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14105b80 0x7f3e14197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e137fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14105b80 0x7f3e14197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53524/0 (socket says 172.21.15.96:53524) 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e137fe700 1 -- 172.21.15.96:0/1762541950 learned_addr learned my addr 172.21.15.96:0/1762541950 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e13fff700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e140ffaf0 0x7f3e141979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e137fe700 1 -- 172.21.15.96:0/1762541950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e140ffaf0 msgr2=0x7f3e141979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e137fe700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e140ffaf0 0x7f3e141979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e137fe700 1 -- 172.21.15.96:0/1762541950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e080095f0 con 0x7f3e14105b80 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e137fe700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14105b80 0x7f3e14197ef0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800bd70 tx=0x7f3e0800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.816+0000 7f3e117fa700 1 -- 172.21.15.96:0/1762541950 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e0801d070 con 0x7f3e14105b80 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.817+0000 7f3e117fa700 1 -- 172.21.15.96:0/1762541950 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e080056e0 con 0x7f3e14105b80 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.817+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e141987f0 con 0x7f3e14105b80 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.817+0000 7f3e117fa700 1 -- 172.21.15.96:0/1762541950 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e0800fe40 con 0x7f3e14105b80 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.817+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e14195970 con 0x7f3e14105b80 2024-04-22T15:43:29.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.818+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e14108110 con 0x7f3e14105b80 2024-04-22T15:43:29.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.818+0000 7f3e117fa700 1 -- 172.21.15.96:0/1762541950 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3e0800f680 con 0x7f3e14105b80 2024-04-22T15:43:29.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.819+0000 7f3e117fa700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc0730d0 0x7f3dfc075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:29.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.819+0000 7f3e117fa700 1 -- 172.21.15.96:0/1762541950 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3e080901e0 con 0x7f3e14105b80 2024-04-22T15:43:29.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.819+0000 7f3e13fff700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc0730d0 0x7f3dfc075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:29.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.819+0000 7f3e13fff700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc0730d0 0x7f3dfc075590 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3e04005950 tx=0x7f3e0400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:29.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:29.824+0000 7f3e117fa700 1 -- 172.21.15.96:0/1762541950 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3e08058fc0 con 0x7f3e14105b80 2024-04-22T15:43:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:29 smithi196 ceph-mon[81815]: pgmap v648: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:29 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:29 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:29 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:30.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.035+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e1410ccf0 con 0x7f3dfc0730d0 2024-04-22T15:43:30.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.045+0000 7f3e117fa700 1 -- 172.21.15.96:0/1762541950 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3e1410ccf0 con 0x7f3dfc0730d0 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (22m) 15m ago 33m 25.4M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (33m) 15m ago 33m 9223k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (32m) 14m ago 32m 8984k - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (19m) 15m ago 33m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (19m) 14m ago 32m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (21m) 15m ago 33m 81.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (15m) 15m ago 27m 13.9M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (15m) 15m ago 27m 13.6M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (15m) 14m ago 27m 11.3M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (27m) 14m ago 27m 11.9M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 3123b73926e5 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (24m) 15m ago 34m 511M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:43:30.049 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (23m) 14m ago 32m 452M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (20m) 15m ago 35m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (20m) 14m ago 32m 44.7M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (23m) 15m ago 33m 16.6M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (23m) 14m ago 32m 16.7M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (18m) 15m ago 31m 95.8M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (18m) 15m ago 31m 97.6M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (17m) 15m ago 30m 76.2M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (17m) 14m ago 30m 99.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (16m) 14m ago 29m 86.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (16m) 14m ago 29m 84.3M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:43:30.050 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (22m) 15m ago 33m 73.4M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:43:30.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc0730d0 msgr2=0x7f3dfc075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc0730d0 0x7f3dfc075590 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3e04005950 tx=0x7f3e0400b410 comp rx=0 tx=0).stop 2024-04-22T15:43:30.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14105b80 msgr2=0x7f3e14197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14105b80 0x7f3e14197ef0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800bd70 tx=0x7f3e0800bda0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 shutdown_connections 2024-04-22T15:43:30.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc0730d0 0x7f3dfc075590 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e140ffaf0 0x7f3e141979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 --2- 172.21.15.96:0/1762541950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14105b80 0x7f3e14197ef0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 >> 172.21.15.96:0/1762541950 conn(0x7f3e140fb5e0 msgr2=0x7f3e140fc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:30.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 shutdown_connections 2024-04-22T15:43:30.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.048+0000 7f3e1a9ea700 1 -- 172.21.15.96:0/1762541950 wait complete. 2024-04-22T15:43:30.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 -- 172.21.15.96:0/1504282759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c1002d0 msgr2=0x7f6e7c10cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 --2- 172.21.15.96:0/1504282759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c1002d0 0x7f6e7c10cf90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009910 tx=0x7f6e6c009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:30.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 -- 172.21.15.96:0/1504282759 shutdown_connections 2024-04-22T15:43:30.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 --2- 172.21.15.96:0/1504282759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c1002d0 0x7f6e7c10cf90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 --2- 172.21.15.96:0/1504282759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e7c0ff920 0x7f6e7c0ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 -- 172.21.15.96:0/1504282759 >> 172.21.15.96:0/1504282759 conn(0x7f6e7c0fb5e0 msgr2=0x7f6e7c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 -- 172.21.15.96:0/1504282759 shutdown_connections 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 -- 172.21.15.96:0/1504282759 wait complete. 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 Processor -- start 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.151+0000 7f6e8409f700 1 -- start start 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e8409f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c0ff920 0x7f6e7c19e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e8409f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e7c1002d0 0x7f6e7c19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e8409f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c19f540 con 0x7f6e7c0ff920 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e8409f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c198b20 con 0x7f6e7c1002d0 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e81e3b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c0ff920 0x7f6e7c19e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e81e3b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c0ff920 0x7f6e7c19e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53538/0 (socket says 172.21.15.96:53538) 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e81e3b700 1 -- 172.21.15.96:0/2690789685 learned_addr learned my addr 172.21.15.96:0/2690789685 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:43:30.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e8163a700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e7c1002d0 0x7f6e7c19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:30.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e81e3b700 1 -- 172.21.15.96:0/2690789685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e7c1002d0 msgr2=0x7f6e7c19ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e81e3b700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e7c1002d0 0x7f6e7c19ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e81e3b700 1 -- 172.21.15.96:0/2690789685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0095f0 con 0x7f6e7c0ff920 2024-04-22T15:43:30.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e81e3b700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c0ff920 0x7f6e7c19e9e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800e920 tx=0x7f6e7800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:30.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e72ffd700 1 -- 172.21.15.96:0/2690789685 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e7800c9d0 con 0x7f6e7c0ff920 2024-04-22T15:43:30.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c198e00 con 0x7f6e7c0ff920 2024-04-22T15:43:30.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e72ffd700 1 -- 172.21.15.96:0/2690789685 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e78010430 con 0x7f6e7c0ff920 2024-04-22T15:43:30.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e72ffd700 1 -- 172.21.15.96:0/2690789685 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e78018630 con 0x7f6e7c0ff920 2024-04-22T15:43:30.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.152+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c199350 con 0x7f6e7c0ff920 2024-04-22T15:43:30.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.153+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7c10ae10 con 0x7f6e7c0ff920 2024-04-22T15:43:30.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.154+0000 7f6e72ffd700 1 -- 172.21.15.96:0/2690789685 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6e780105a0 con 0x7f6e7c0ff920 2024-04-22T15:43:30.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.154+0000 7f6e72ffd700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e6806ecc0 0x7f6e68071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:30.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.154+0000 7f6e8163a700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e6806ecc0 0x7f6e68071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:30.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.154+0000 7f6e72ffd700 1 -- 172.21.15.96:0/2690789685 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6e78014070 con 0x7f6e7c0ff920 2024-04-22T15:43:30.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.155+0000 7f6e8163a700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e6806ecc0 0x7f6e68071180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c199ce0 tx=0x7f6e6c009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:30.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.159+0000 7f6e72ffd700 1 -- 172.21.15.96:0/2690789685 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6e78054150 con 0x7f6e7c0ff920 2024-04-22T15:43:30.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.444+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6e7c10ffa0 con 0x7f6e7c0ff920 2024-04-22T15:43:30.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.445+0000 7f6e72ffd700 1 -- 172.21.15.96:0/2690789685 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+835 (secure 0 0 0) 0x7f6e78062020 con 0x7f6e7c0ff920 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:43:30.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:43:30.449 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:43:30.449 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 18.2.2-1237-g646b8c7d (646b8c7d293d8ec31a40c69f7b3237b19c1be24d) reef (stable)": 1, 2024-04-22T15:43:30.449 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 13 2024-04-22T15:43:30.449 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:43:30.449 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:43:30.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e6806ecc0 msgr2=0x7f6e68071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e6806ecc0 0x7f6e68071180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c199ce0 tx=0x7f6e6c009f70 comp rx=0 tx=0).stop 2024-04-22T15:43:30.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c0ff920 msgr2=0x7f6e7c19e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c0ff920 0x7f6e7c19e9e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800e920 tx=0x7f6e7800ec30 comp rx=0 tx=0).stop 2024-04-22T15:43:30.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 shutdown_connections 2024-04-22T15:43:30.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e6806ecc0 0x7f6e68071180 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e7c0ff920 0x7f6e7c19e9e0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 --2- 172.21.15.96:0/2690789685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e7c1002d0 0x7f6e7c19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 >> 172.21.15.96:0/2690789685 conn(0x7f6e7c0fb5e0 msgr2=0x7f6e7c107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:30.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 shutdown_connections 2024-04-22T15:43:30.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.447+0000 7f6e8409f700 1 -- 172.21.15.96:0/2690789685 wait complete. 2024-04-22T15:43:30.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 -- 172.21.15.96:0/427754556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 msgr2=0x7f07e4105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 --2- 172.21.15.96:0/427754556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 0x7f07e4105e80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009910 tx=0x7f07d4009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:30.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 -- 172.21.15.96:0/427754556 shutdown_connections 2024-04-22T15:43:30.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 --2- 172.21.15.96:0/427754556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f07e40ffaa0 0x7f07e40fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 --2- 172.21.15.96:0/427754556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 0x7f07e4105e80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 -- 172.21.15.96:0/427754556 >> 172.21.15.96:0/427754556 conn(0x7f07e4076740 msgr2=0x7f07e40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:30.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 -- 172.21.15.96:0/427754556 shutdown_connections 2024-04-22T15:43:30.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 -- 172.21.15.96:0/427754556 wait complete. 2024-04-22T15:43:30.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 Processor -- start 2024-04-22T15:43:30.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.551+0000 7f07ea50e700 1 -- start start 2024-04-22T15:43:30.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07ea50e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f07e40ffaa0 0x7f07e41979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:30.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07ea50e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 0x7f07e4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:30.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07ea50e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e41984b0 con 0x7f07e4105aa0 2024-04-22T15:43:30.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07ea50e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e41985f0 con 0x7f07e40ffaa0 2024-04-22T15:43:30.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07e37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 0x7f07e4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:30.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07e37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 0x7f07e4197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46054/0 (socket says 172.21.15.96:46054) 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07e37fe700 1 -- 172.21.15.96:0/1182662555 learned_addr learned my addr 172.21.15.96:0/1182662555 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07e3fff700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f07e40ffaa0 0x7f07e41979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07e37fe700 1 -- 172.21.15.96:0/1182662555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f07e40ffaa0 msgr2=0x7f07e41979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07e37fe700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f07e40ffaa0 0x7f07e41979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07e37fe700 1 -- 172.21.15.96:0/1182662555 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d40095f0 con 0x7f07e4105aa0 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.552+0000 7f07e37fe700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 0x7f07e4197ee0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f07d800c8a0 tx=0x7f07d800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.553+0000 7f07e17fa700 1 -- 172.21.15.96:0/1182662555 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07d80078e0 con 0x7f07e4105aa0 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.553+0000 7f07e17fa700 1 -- 172.21.15.96:0/1182662555 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07d800f450 con 0x7f07e4105aa0 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.553+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e41a0a10 con 0x7f07e4105aa0 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.553+0000 7f07e17fa700 1 -- 172.21.15.96:0/1182662555 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07d800e5e0 con 0x7f07e4105aa0 2024-04-22T15:43:30.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.553+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e41a0e20 con 0x7f07e4105aa0 2024-04-22T15:43:30.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.554+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e41080a0 con 0x7f07e4105aa0 2024-04-22T15:43:30.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.554+0000 7f07e17fa700 1 -- 172.21.15.96:0/1182662555 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f07d800f5c0 con 0x7f07e4105aa0 2024-04-22T15:43:30.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.554+0000 7f07e17fa700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f07cc06ecc0 0x7f07cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:30.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.555+0000 7f07e17fa700 1 -- 172.21.15.96:0/1182662555 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f07d808ee90 con 0x7f07e4105aa0 2024-04-22T15:43:30.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.555+0000 7f07e3fff700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f07cc06ecc0 0x7f07cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:30.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.555+0000 7f07e3fff700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f07cc06ecc0 0x7f07cc071180 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f07d4000c00 tx=0x7f07d401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:30.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.559+0000 7f07e17fa700 1 -- 172.21.15.96:0/1182662555 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f07d8057bf0 con 0x7f07e4105aa0 2024-04-22T15:43:30.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:30 smithi096 ceph-mon[112443]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:30 smithi096 ceph-mon[112443]: pgmap v649: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:30 smithi096 ceph-mon[112443]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-22T15:43:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:30 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:30 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:30 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:30 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:30 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:30 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2690789685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:30 smithi196 ceph-mon[81815]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:30 smithi196 ceph-mon[81815]: pgmap v649: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:30 smithi196 ceph-mon[81815]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-22T15:43:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:30 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:30 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:30 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:30 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:30 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:30 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2690789685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:43:30.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.814+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f07e41011e0 con 0x7f07e4105aa0 2024-04-22T15:43:30.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.814+0000 7f07e17fa700 1 -- 172.21.15.96:0/1182662555 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1873 (secure 0 0 0) 0x7f07d805b2b0 con 0x7f07e4105aa0 2024-04-22T15:43:30.818 INFO:teuthology.orchestra.run.smithi096.stdout:e29 2024-04-22T15:43:30.818 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:43:30.818 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:43:30.818 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:43:30.818 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:43:30.818 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:43:30.819 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:43:30.819 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:43:30.819 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:43:30.819 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:43:30.819 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:43:30.819 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:43:30.819 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:43:30.820 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:43:30.820 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:43:30.820 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:43:30.820 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:43:30.820 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:43:30.820 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:43:30.821 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44225} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/2911111435,v1:172.21.15.196:6827/2911111435] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:43:30.822 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:43:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.816+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f07cc06ecc0 msgr2=0x7f07cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.816+0000 7f07ea50e700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f07cc06ecc0 0x7f07cc071180 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f07d4000c00 tx=0x7f07d401a040 comp rx=0 tx=0).stop 2024-04-22T15:43:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.816+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 msgr2=0x7f07e4197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.816+0000 7f07ea50e700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 0x7f07e4197ee0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f07d800c8a0 tx=0x7f07d800cbb0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.816+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 shutdown_connections 2024-04-22T15:43:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.816+0000 7f07ea50e700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f07cc06ecc0 0x7f07cc071180 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.816+0000 7f07ea50e700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f07e40ffaa0 0x7f07e41979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.816+0000 7f07ea50e700 1 --2- 172.21.15.96:0/1182662555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07e4105aa0 0x7f07e4197ee0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.817+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 >> 172.21.15.96:0/1182662555 conn(0x7f07e4076740 msgr2=0x7f07e40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.817+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 shutdown_connections 2024-04-22T15:43:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.817+0000 7f07ea50e700 1 -- 172.21.15.96:0/1182662555 wait complete. 2024-04-22T15:43:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 -- 172.21.15.96:0/3573063098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 msgr2=0x7fe1f4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 --2- 172.21.15.96:0/3573063098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 0x7fe1f4101200 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009910 tx=0x7fe1e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 -- 172.21.15.96:0/3573063098 shutdown_connections 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 --2- 172.21.15.96:0/3573063098 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f41017d0 0x7fe1f4105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 --2- 172.21.15.96:0/3573063098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 0x7fe1f4101200 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 -- 172.21.15.96:0/3573063098 >> 172.21.15.96:0/3573063098 conn(0x7fe1f40fc6d0 msgr2=0x7fe1f40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 -- 172.21.15.96:0/3573063098 shutdown_connections 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 -- 172.21.15.96:0/3573063098 wait complete. 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 Processor -- start 2024-04-22T15:43:30.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.920+0000 7fe1fab25700 1 -- start start 2024-04-22T15:43:30.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1fab25700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 0x7fe1f40731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:30.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1fab25700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f41017d0 0x7fe1f4073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:30.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1fab25700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f4073c60 con 0x7fe1f4100e20 2024-04-22T15:43:30.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1fab25700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f4073da0 con 0x7fe1f41017d0 2024-04-22T15:43:30.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f88c1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 0x7fe1f40731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:30.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f88c1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 0x7fe1f40731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46084/0 (socket says 172.21.15.96:46084) 2024-04-22T15:43:30.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f88c1700 1 -- 172.21.15.96:0/735794487 learned_addr learned my addr 172.21.15.96:0/735794487 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:43:30.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f3fff700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f41017d0 0x7fe1f4073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:30.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f88c1700 1 -- 172.21.15.96:0/735794487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f41017d0 msgr2=0x7fe1f4073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:30.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f88c1700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f41017d0 0x7fe1f4073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:30.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f88c1700 1 -- 172.21.15.96:0/735794487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e80095f0 con 0x7fe1f4100e20 2024-04-22T15:43:30.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f88c1700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 0x7fe1f40731e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800bd70 tx=0x7fe1e800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:30.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f1ffb700 1 -- 172.21.15.96:0/735794487 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e801d070 con 0x7fe1f4100e20 2024-04-22T15:43:30.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f1ffb700 1 -- 172.21.15.96:0/735794487 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1e80056e0 con 0x7fe1f4100e20 2024-04-22T15:43:30.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1f1ffb700 1 -- 172.21.15.96:0/735794487 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e8017400 con 0x7fe1f4100e20 2024-04-22T15:43:30.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.921+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f4074020 con 0x7fe1f4100e20 2024-04-22T15:43:30.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.922+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f41a2f80 con 0x7fe1f4100e20 2024-04-22T15:43:30.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.922+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f41091c0 con 0x7fe1f4100e20 2024-04-22T15:43:30.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.923+0000 7fe1f1ffb700 1 -- 172.21.15.96:0/735794487 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe1e8003a60 con 0x7fe1f4100e20 2024-04-22T15:43:30.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.923+0000 7fe1f1ffb700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1e406ecc0 0x7fe1e4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:30.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.923+0000 7fe1f1ffb700 1 -- 172.21.15.96:0/735794487 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe1e8090250 con 0x7fe1f4100e20 2024-04-22T15:43:30.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.923+0000 7fe1f3fff700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1e406ecc0 0x7fe1e4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:30.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.924+0000 7fe1f3fff700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1e406ecc0 0x7fe1e4071180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0005950 tx=0x7fe1e00058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:30.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:30.928+0000 7fe1f1ffb700 1 -- 172.21.15.96:0/735794487 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe1e8059030 con 0x7fe1f4100e20 2024-04-22T15:43:31.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.145+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1f410ddf0 con 0x7fe1e406ecc0 2024-04-22T15:43:31.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.147+0000 7fe1f1ffb700 1 -- 172.21.15.96:0/735794487 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe1f410ddf0 con 0x7fe1e406ecc0 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "osd" 2024-04-22T15:43:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:43:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:43:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:43:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:43:31.151 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:43:31.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1e406ecc0 msgr2=0x7fe1e4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:31.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1e406ecc0 0x7fe1e4071180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0005950 tx=0x7fe1e00058e0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 msgr2=0x7fe1f40731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:31.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 0x7fe1f40731e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800bd70 tx=0x7fe1e800be50 comp rx=0 tx=0).stop 2024-04-22T15:43:31.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 shutdown_connections 2024-04-22T15:43:31.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe1e406ecc0 0x7fe1e4071180 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f4100e20 0x7fe1f40731e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 --2- 172.21.15.96:0/735794487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f41017d0 0x7fe1f4073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 >> 172.21.15.96:0/735794487 conn(0x7fe1f40fc6d0 msgr2=0x7fe1f40fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:31.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 shutdown_connections 2024-04-22T15:43:31.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.150+0000 7fe1fab25700 1 -- 172.21.15.96:0/735794487 wait complete. 2024-04-22T15:43:31.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.255+0000 7f836f506700 1 -- 172.21.15.96:0/3514959747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83681026b0 msgr2=0x7f836810aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:31.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.255+0000 7f836f506700 1 --2- 172.21.15.96:0/3514959747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83681026b0 0x7f836810aba0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8358009910 tx=0x7f8358009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:31.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.255+0000 7f836f506700 1 -- 172.21.15.96:0/3514959747 shutdown_connections 2024-04-22T15:43:31.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.255+0000 7f836f506700 1 --2- 172.21.15.96:0/3514959747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83681026b0 0x7f836810aba0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.255+0000 7f836f506700 1 --2- 172.21.15.96:0/3514959747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8368101d90 0x7f8368102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.255+0000 7f836f506700 1 -- 172.21.15.96:0/3514959747 >> 172.21.15.96:0/3514959747 conn(0x7f83680fb600 msgr2=0x7f83680fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:31.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.255+0000 7f836f506700 1 -- 172.21.15.96:0/3514959747 shutdown_connections 2024-04-22T15:43:31.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.255+0000 7f836f506700 1 -- 172.21.15.96:0/3514959747 wait complete. 2024-04-22T15:43:31.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836f506700 1 Processor -- start 2024-04-22T15:43:31.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836f506700 1 -- start start 2024-04-22T15:43:31.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836f506700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8368101d90 0x7f836819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:31.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836f506700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83681026b0 0x7f836819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:31.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836f506700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836819d3d0 con 0x7f8368101d90 2024-04-22T15:43:31.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836f506700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83681969b0 con 0x7f83681026b0 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836d2a2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8368101d90 0x7f836819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836d2a2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8368101d90 0x7f836819c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46106/0 (socket says 172.21.15.96:46106) 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836d2a2700 1 -- 172.21.15.96:0/1815203984 learned_addr learned my addr 172.21.15.96:0/1815203984 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836caa1700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83681026b0 0x7f836819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836d2a2700 1 -- 172.21.15.96:0/1815203984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83681026b0 msgr2=0x7f836819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836d2a2700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83681026b0 0x7f836819cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.256+0000 7f836d2a2700 1 -- 172.21.15.96:0/1815203984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83580095f0 con 0x7f8368101d90 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.257+0000 7f836d2a2700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8368101d90 0x7f836819c870 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f836000d900 tx=0x7f836000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.257+0000 7f835e7fc700 1 -- 172.21.15.96:0/1815203984 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83600041f0 con 0x7f8368101d90 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.257+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8368196c90 con 0x7f8368101d90 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.257+0000 7f835e7fc700 1 -- 172.21.15.96:0/1815203984 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8360004cf0 con 0x7f8368101d90 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.257+0000 7f835e7fc700 1 -- 172.21.15.96:0/1815203984 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f836000b750 con 0x7f8368101d90 2024-04-22T15:43:31.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.257+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83681971e0 con 0x7f8368101d90 2024-04-22T15:43:31.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.258+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83681054b0 con 0x7f8368101d90 2024-04-22T15:43:31.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.258+0000 7f835e7fc700 1 -- 172.21.15.96:0/1815203984 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8360004750 con 0x7f8368101d90 2024-04-22T15:43:31.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.259+0000 7f835e7fc700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f835406ecc0 0x7f8354071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:31.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.259+0000 7f835e7fc700 1 -- 172.21.15.96:0/1815203984 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f836008edb0 con 0x7f8368101d90 2024-04-22T15:43:31.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.259+0000 7f836caa1700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f835406ecc0 0x7f8354071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:31.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.259+0000 7f836caa1700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f835406ecc0 0x7f8354071180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8368197b70 tx=0x7f835801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:31.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.263+0000 7f835e7fc700 1 -- 172.21.15.96:0/1815203984 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f836005b640 con 0x7f8368101d90 2024-04-22T15:43:31.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.548+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8368197e70 con 0x7f8368101d90 2024-04-22T15:43:31.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.549+0000 7f835e7fc700 1 -- 172.21.15.96:0/1815203984 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f836005b1d0 con 0x7f8368101d90 2024-04-22T15:43:31.552 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:43:31.552 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:43:31.552 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:43:31.552 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:43:31.552 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:43:31.553 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:43:31.554 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:43:31.554 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:43:31.554 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:43:31.554 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:43:31.554 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:43:31.554 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.551+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f835406ecc0 msgr2=0x7f8354071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.551+0000 7f836f506700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f835406ecc0 0x7f8354071180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8368197b70 tx=0x7f835801a040 comp rx=0 tx=0).stop 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.551+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8368101d90 msgr2=0x7f836819c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.551+0000 7f836f506700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8368101d90 0x7f836819c870 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f836000d900 tx=0x7f836000dc10 comp rx=0 tx=0).stop 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.552+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 shutdown_connections 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.552+0000 7f836f506700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f835406ecc0 0x7f8354071180 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.552+0000 7f836f506700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8368101d90 0x7f836819c870 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.552+0000 7f836f506700 1 --2- 172.21.15.96:0/1815203984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83681026b0 0x7f836819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.552+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 >> 172.21.15.96:0/1815203984 conn(0x7f83680fb600 msgr2=0x7f8368103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.552+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 shutdown_connections 2024-04-22T15:43:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:43:31.552+0000 7f836f506700 1 -- 172.21.15.96:0/1815203984 wait complete. 2024-04-22T15:43:31.930 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:31 smithi196 ceph-mon[81815]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:31.930 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:31 smithi196 ceph-mon[81815]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:31.930 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:31 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1182662555' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:43:31.930 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:31 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:31.930 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:31 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.pbvqlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:43:31.930 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:31 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:31.930 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:31 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1815203984' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:43:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:31 smithi096 ceph-mon[112443]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:31 smithi096 ceph-mon[112443]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:31 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1182662555' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:43:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:31 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:31 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi196.pbvqlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:43:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:31 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:31 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1815203984' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:43:32.956 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:32 smithi196 ceph-mon[81815]: from='client.35228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:32.956 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:32 smithi196 ceph-mon[81815]: Upgrade: Updating mds.cephfs.smithi196.pbvqlq 2024-04-22T15:43:32.956 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:32 smithi196 ceph-mon[81815]: Deploying daemon mds.cephfs.smithi196.pbvqlq on smithi196 2024-04-22T15:43:32.956 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:32 smithi196 ceph-mon[81815]: pgmap v650: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:32.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:32 smithi096 ceph-mon[112443]: from='client.35228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:32.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:32 smithi096 ceph-mon[112443]: Upgrade: Updating mds.cephfs.smithi196.pbvqlq 2024-04-22T15:43:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:32 smithi096 ceph-mon[112443]: Deploying daemon mds.cephfs.smithi196.pbvqlq on smithi196 2024-04-22T15:43:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:32 smithi096 ceph-mon[112443]: pgmap v650: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:34 smithi096 ceph-mon[112443]: pgmap v651: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:34 smithi096 ceph-mon[112443]: osdmap e84: 6 total, 6 up, 6 in 2024-04-22T15:43:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:34 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T15:43:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:34 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:43:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:34 smithi196 ceph-mon[81815]: pgmap v651: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:34 smithi196 ceph-mon[81815]: osdmap e84: 6 total, 6 up, 6 in 2024-04-22T15:43:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:34 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T15:43:35.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:34 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:43:35.908 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:35 smithi196 ceph-mon[81815]: pgmap v653: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:35.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:35 smithi096 ceph-mon[112443]: pgmap v653: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:38.411 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] up:boot 2024-04-22T15:43:38.411 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:43:38.411 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:43:38.411 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.411 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.411 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:38.411 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: pgmap v654: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:38.412 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.412 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.412 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.412 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] up:boot 2024-04-22T15:43:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:43:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:43:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: pgmap v654: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:38.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:38.487 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:39.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:39.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:39 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:39.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:39.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:39 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:40.713 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:40 smithi096 ceph-mon[112443]: pgmap v655: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:40 smithi196 ceph-mon[81815]: pgmap v655: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:41.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:41 smithi096 ceph-mon[112443]: pgmap v656: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:41 smithi196 ceph-mon[81815]: pgmap v656: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:43.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:43.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:44.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:43 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:44.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:43 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:44.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:43 smithi196 ceph-mon[81815]: pgmap v657: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:44.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:43 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:44.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:43 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:43:44.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:43 smithi096 ceph-mon[112443]: pgmap v657: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:46 smithi196 ceph-mon[81815]: pgmap v658: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:46.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:46 smithi096 ceph-mon[112443]: pgmap v658: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:48 smithi096 ceph-mon[112443]: pgmap v659: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:48 smithi196 ceph-mon[81815]: pgmap v659: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:50.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:50 smithi096 ceph-mon[112443]: pgmap v660: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:50.986 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:50 smithi196 ceph-mon[81815]: pgmap v660: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:52.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:52 smithi096 ceph-mon[112443]: pgmap v661: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:52 smithi196 ceph-mon[81815]: pgmap v661: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:54 smithi096 ceph-mon[112443]: pgmap v662: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:54 smithi196 ceph-mon[81815]: pgmap v662: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:56.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:56 smithi096 ceph-mon[112443]: pgmap v663: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:56 smithi196 ceph-mon[81815]: pgmap v663: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:58.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:43:58 smithi096 ceph-mon[112443]: pgmap v664: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:43:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:43:58 smithi196 ceph-mon[81815]: pgmap v664: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:00.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:00 smithi096 ceph-mon[112443]: pgmap v665: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:00 smithi196 ceph-mon[81815]: pgmap v665: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:01.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 -- 172.21.15.96:0/2157725066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 msgr2=0x7ff04410cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:01.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 --2- 172.21.15.96:0/2157725066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 0x7ff04410cf90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff034009910 tx=0x7ff034009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:01.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 -- 172.21.15.96:0/2157725066 shutdown_connections 2024-04-22T15:44:01.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 --2- 172.21.15.96:0/2157725066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 0x7ff04410cf90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:01.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 --2- 172.21.15.96:0/2157725066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0440ff920 0x7ff0440ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:01.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 -- 172.21.15.96:0/2157725066 >> 172.21.15.96:0/2157725066 conn(0x7ff0440fb5e0 msgr2=0x7ff0440fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:01.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 -- 172.21.15.96:0/2157725066 shutdown_connections 2024-04-22T15:44:01.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 -- 172.21.15.96:0/2157725066 wait complete. 2024-04-22T15:44:01.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 Processor -- start 2024-04-22T15:44:01.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.672+0000 7ff04b4e4700 1 -- start start 2024-04-22T15:44:01.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff04b4e4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0440ff920 0x7ff04419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:01.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff04b4e4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 0x7ff04419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:01.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff04b4e4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04419f530 con 0x7ff0441002d0 2024-04-22T15:44:01.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff04b4e4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff044198ba0 con 0x7ff0440ff920 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff048a7f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 0x7ff04419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff048a7f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 0x7ff04419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34580/0 (socket says 172.21.15.96:34580) 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff048a7f700 1 -- 172.21.15.96:0/412357000 learned_addr learned my addr 172.21.15.96:0/412357000 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff049280700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0440ff920 0x7ff04419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff048a7f700 1 -- 172.21.15.96:0/412357000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0440ff920 msgr2=0x7ff04419ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff048a7f700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0440ff920 0x7ff04419ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff048a7f700 1 -- 172.21.15.96:0/412357000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0340095f0 con 0x7ff0441002d0 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff049280700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0440ff920 0x7ff04419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff048a7f700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 0x7ff04419efa0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff03400b850 tx=0x7ff0340056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff03a7fc700 1 -- 172.21.15.96:0/412357000 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff03401d070 con 0x7ff0441002d0 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff03a7fc700 1 -- 172.21.15.96:0/412357000 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff034003d50 con 0x7ff0441002d0 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff044198e20 con 0x7ff0441002d0 2024-04-22T15:44:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.673+0000 7ff03a7fc700 1 -- 172.21.15.96:0/412357000 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff034021410 con 0x7ff0441002d0 2024-04-22T15:44:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.674+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff044199370 con 0x7ff0441002d0 2024-04-22T15:44:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.674+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04410ae10 con 0x7ff0441002d0 2024-04-22T15:44:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.675+0000 7ff03a7fc700 1 -- 172.21.15.96:0/412357000 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff034003890 con 0x7ff0441002d0 2024-04-22T15:44:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.675+0000 7ff03a7fc700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff03006ecc0 0x7ff030071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.676+0000 7ff03a7fc700 1 -- 172.21.15.96:0/412357000 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff034090210 con 0x7ff0441002d0 2024-04-22T15:44:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.676+0000 7ff049280700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff03006ecc0 0x7ff030071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.676+0000 7ff049280700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff03006ecc0 0x7ff030071180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff040005fd0 tx=0x7ff040005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:01.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.680+0000 7ff03a7fc700 1 -- 172.21.15.96:0/412357000 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff03405caa0 con 0x7ff0441002d0 2024-04-22T15:44:01.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.900+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff044199ff0 con 0x7ff03006ecc0 2024-04-22T15:44:01.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.902+0000 7ff03a7fc700 1 -- 172.21.15.96:0/412357000 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff044199ff0 con 0x7ff03006ecc0 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff03006ecc0 msgr2=0x7ff030071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff03006ecc0 0x7ff030071180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff040005fd0 tx=0x7ff040005e20 comp rx=0 tx=0).stop 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 msgr2=0x7ff04419efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 0x7ff04419efa0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff03400b850 tx=0x7ff0340056a0 comp rx=0 tx=0).stop 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 shutdown_connections 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff03006ecc0 0x7ff030071180 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0440ff920 0x7ff04419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 --2- 172.21.15.96:0/412357000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0441002d0 0x7ff04419efa0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 >> 172.21.15.96:0/412357000 conn(0x7ff0440fb5e0 msgr2=0x7ff044107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:01.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 shutdown_connections 2024-04-22T15:44:01.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:01.904+0000 7ff04b4e4700 1 -- 172.21.15.96:0/412357000 wait complete. 2024-04-22T15:44:01.918 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:44:02.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.013+0000 7f37993eb700 1 -- 172.21.15.96:0/1971785356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794101e10 msgr2=0x7f37941021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.013+0000 7f37993eb700 1 --2- 172.21.15.96:0/1971785356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794101e10 0x7f37941021f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3788009910 tx=0x7f3788009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:02.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.013+0000 7f37993eb700 1 -- 172.21.15.96:0/1971785356 shutdown_connections 2024-04-22T15:44:02.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.013+0000 7f37993eb700 1 --2- 172.21.15.96:0/1971785356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3794102730 0x7f379410ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.013+0000 7f37993eb700 1 --2- 172.21.15.96:0/1971785356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794101e10 0x7f37941021f0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.013+0000 7f37993eb700 1 -- 172.21.15.96:0/1971785356 >> 172.21.15.96:0/1971785356 conn(0x7f37940fb5e0 msgr2=0x7f37940fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:02.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.013+0000 7f37993eb700 1 -- 172.21.15.96:0/1971785356 shutdown_connections 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.013+0000 7f37993eb700 1 -- 172.21.15.96:0/1971785356 wait complete. 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.014+0000 7f37993eb700 1 Processor -- start 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.014+0000 7f37993eb700 1 -- start start 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.014+0000 7f37993eb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3794101e10 0x7f379419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.014+0000 7f37993eb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794102730 0x7f379419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.014+0000 7f37993eb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379419d380 con 0x7f3794102730 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.014+0000 7f37993eb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37941969b0 con 0x7f3794101e10 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.014+0000 7f37927fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794102730 0x7f379419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.014+0000 7f37927fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794102730 0x7f379419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34590/0 (socket says 172.21.15.96:34590) 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.014+0000 7f37927fc700 1 -- 172.21.15.96:0/1425446125 learned_addr learned my addr 172.21.15.96:0/1425446125 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f3792ffd700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3794101e10 0x7f379419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f37927fc700 1 -- 172.21.15.96:0/1425446125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3794101e10 msgr2=0x7f379419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f37927fc700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3794101e10 0x7f379419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f37927fc700 1 -- 172.21.15.96:0/1425446125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37880095f0 con 0x7f3794102730 2024-04-22T15:44:02.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f3792ffd700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3794101e10 0x7f379419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:44:02.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f37927fc700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794102730 0x7f379419cd60 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f378400d900 tx=0x7f378400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:02.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f377bfff700 1 -- 172.21.15.96:0/1425446125 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3784009900 con 0x7f3794102730 2024-04-22T15:44:02.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f377bfff700 1 -- 172.21.15.96:0/1425446125 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3784010460 con 0x7f3794102730 2024-04-22T15:44:02.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3794196c90 con 0x7f3794102730 2024-04-22T15:44:02.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f377bfff700 1 -- 172.21.15.96:0/1425446125 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f378400f5f0 con 0x7f3794102730 2024-04-22T15:44:02.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.015+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37941971e0 con 0x7f3794102730 2024-04-22T15:44:02.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.016+0000 7f377bfff700 1 -- 172.21.15.96:0/1425446125 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f37840105d0 con 0x7f3794102730 2024-04-22T15:44:02.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.016+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37941055a0 con 0x7f3794102730 2024-04-22T15:44:02.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.017+0000 7f377bfff700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f377c06ecc0 0x7f377c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:02.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.017+0000 7f3792ffd700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f377c06ecc0 0x7f377c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:02.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.017+0000 7f377bfff700 1 -- 172.21.15.96:0/1425446125 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f378408ec10 con 0x7f3794102730 2024-04-22T15:44:02.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.018+0000 7f3792ffd700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f377c06ecc0 0x7f377c071180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3788006010 tx=0x7f378800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:02.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.022+0000 7f377bfff700 1 -- 172.21.15.96:0/1425446125 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f378405b520 con 0x7f3794102730 2024-04-22T15:44:02.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.271+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f379410ddd0 con 0x7f377c06ecc0 2024-04-22T15:44:02.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.272+0000 7f377bfff700 1 -- 172.21.15.96:0/1425446125 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f379410ddd0 con 0x7f377c06ecc0 2024-04-22T15:44:02.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f377c06ecc0 msgr2=0x7f377c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f377c06ecc0 0x7f377c071180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3788006010 tx=0x7f378800b540 comp rx=0 tx=0).stop 2024-04-22T15:44:02.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794102730 msgr2=0x7f379419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794102730 0x7f379419cd60 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f378400d900 tx=0x7f378400dcc0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 shutdown_connections 2024-04-22T15:44:02.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f377c06ecc0 0x7f377c071180 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3794101e10 0x7f379419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 --2- 172.21.15.96:0/1425446125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3794102730 0x7f379419cd60 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 >> 172.21.15.96:0/1425446125 conn(0x7f37940fb5e0 msgr2=0x7f3794103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:02.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.274+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 shutdown_connections 2024-04-22T15:44:02.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.275+0000 7f37993eb700 1 -- 172.21.15.96:0/1425446125 wait complete. 2024-04-22T15:44:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.384+0000 7f3e1a644700 1 -- 172.21.15.96:0/2479218867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14101d60 msgr2=0x7f3e14102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.384+0000 7f3e1a644700 1 --2- 172.21.15.96:0/2479218867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14101d60 0x7f3e14102140 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009910 tx=0x7f3e08009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.384+0000 7f3e1a644700 1 -- 172.21.15.96:0/2479218867 shutdown_connections 2024-04-22T15:44:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.384+0000 7f3e1a644700 1 --2- 172.21.15.96:0/2479218867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e14102680 0x7f3e1410ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.384+0000 7f3e1a644700 1 --2- 172.21.15.96:0/2479218867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14101d60 0x7f3e14102140 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.384+0000 7f3e1a644700 1 -- 172.21.15.96:0/2479218867 >> 172.21.15.96:0/2479218867 conn(0x7f3e14076740 msgr2=0x7f3e140fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.384+0000 7f3e1a644700 1 -- 172.21.15.96:0/2479218867 shutdown_connections 2024-04-22T15:44:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.384+0000 7f3e1a644700 1 -- 172.21.15.96:0/2479218867 wait complete. 2024-04-22T15:44:02.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.385+0000 7f3e1a644700 1 Processor -- start 2024-04-22T15:44:02.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.385+0000 7f3e1a644700 1 -- start start 2024-04-22T15:44:02.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.385+0000 7f3e1a644700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e14101d60 0x7f3e1419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:02.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.385+0000 7f3e1a644700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14102680 0x7f3e1419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.385+0000 7f3e1a644700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1419d3a0 con 0x7f3e14102680 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.385+0000 7f3e1a644700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e14196980 con 0x7f3e14101d60 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.385+0000 7f3e137fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14102680 0x7f3e1419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.385+0000 7f3e137fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14102680 0x7f3e1419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34602/0 (socket says 172.21.15.96:34602) 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e137fe700 1 -- 172.21.15.96:0/1126836779 learned_addr learned my addr 172.21.15.96:0/1126836779 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e13fff700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e14101d60 0x7f3e1419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e137fe700 1 -- 172.21.15.96:0/1126836779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e14101d60 msgr2=0x7f3e1419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e137fe700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e14101d60 0x7f3e1419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e137fe700 1 -- 172.21.15.96:0/1126836779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e080095f0 con 0x7f3e14102680 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e137fe700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14102680 0x7f3e1419cd80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400e940 tx=0x7f3e0400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e117fa700 1 -- 172.21.15.96:0/1126836779 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e040096e0 con 0x7f3e14102680 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e117fa700 1 -- 172.21.15.96:0/1126836779 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e0400c7c0 con 0x7f3e14102680 2024-04-22T15:44:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e117fa700 1 -- 172.21.15.96:0/1126836779 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e04005580 con 0x7f3e14102680 2024-04-22T15:44:02.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e14196c60 con 0x7f3e14102680 2024-04-22T15:44:02.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.386+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e141971b0 con 0x7f3e14102680 2024-04-22T15:44:02.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.387+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e14105480 con 0x7f3e14102680 2024-04-22T15:44:02.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.388+0000 7f3e117fa700 1 -- 172.21.15.96:0/1126836779 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3e0400c930 con 0x7f3e14102680 2024-04-22T15:44:02.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.388+0000 7f3e117fa700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc06ecd0 0x7f3dfc071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:02.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.388+0000 7f3e117fa700 1 -- 172.21.15.96:0/1126836779 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3e04014070 con 0x7f3e14102680 2024-04-22T15:44:02.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.388+0000 7f3e13fff700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc06ecd0 0x7f3dfc071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:02.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.389+0000 7f3e13fff700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc06ecd0 0x7f3dfc071190 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800b760 tx=0x7f3e08005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:02.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.393+0000 7f3e117fa700 1 -- 172.21.15.96:0/1126836779 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3e0405b910 con 0x7f3e14102680 2024-04-22T15:44:02.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.614+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e14197ee0 con 0x7f3dfc06ecd0 2024-04-22T15:44:02.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.633+0000 7f3e117fa700 1 -- 172.21.15.96:0/1126836779 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3e14197ee0 con 0x7f3dfc06ecd0 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (22m) 19s ago 34m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (34m) 19s ago 34m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (33m) 21s ago 33m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (19m) 19s ago 34m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (19m) 21s ago 33m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (22m) 19s ago 33m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (15m) 19s ago 28m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (16m) 19s ago 28m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (15m) 21s ago 28m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:44:02.638 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (25s) 21s ago 28m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (25m) 19s ago 35m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (23m) 21s ago 32m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (21m) 19s ago 35m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (20m) 21s ago 32m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (23m) 19s ago 34m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (23m) 21s ago 32m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (19m) 19s ago 32m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (18m) 19s ago 31m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (18m) 19s ago 31m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (17m) 21s ago 30m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (17m) 21s ago 30m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (16m) 21s ago 29m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:44:02.639 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (23m) 19s ago 33m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:44:02.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc06ecd0 msgr2=0x7f3dfc071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc06ecd0 0x7f3dfc071190 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800b760 tx=0x7f3e08005a90 comp rx=0 tx=0).stop 2024-04-22T15:44:02.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14102680 msgr2=0x7f3e1419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14102680 0x7f3e1419cd80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400e940 tx=0x7f3e0400ec50 comp rx=0 tx=0).stop 2024-04-22T15:44:02.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 shutdown_connections 2024-04-22T15:44:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3dfc06ecd0 0x7f3dfc071190 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e14101d60 0x7f3e1419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 --2- 172.21.15.96:0/1126836779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14102680 0x7f3e1419cd80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 >> 172.21.15.96:0/1126836779 conn(0x7f3e14076740 msgr2=0x7f3e14103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 shutdown_connections 2024-04-22T15:44:02.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.636+0000 7f3e1a644700 1 -- 172.21.15.96:0/1126836779 wait complete. 2024-04-22T15:44:02.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.741+0000 7f954867f700 1 -- 172.21.15.96:0/718586863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 msgr2=0x7f95401011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.741+0000 7f954867f700 1 --2- 172.21.15.96:0/718586863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 0x7f95401011f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f953c009910 tx=0x7f953c009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:02.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.741+0000 7f954867f700 1 -- 172.21.15.96:0/718586863 shutdown_connections 2024-04-22T15:44:02.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.741+0000 7f954867f700 1 --2- 172.21.15.96:0/718586863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95401017c0 0x7f9540105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.741+0000 7f954867f700 1 --2- 172.21.15.96:0/718586863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 0x7f95401011f0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.741+0000 7f954867f700 1 -- 172.21.15.96:0/718586863 >> 172.21.15.96:0/718586863 conn(0x7f95400fc6e0 msgr2=0x7f95400feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:02.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.741+0000 7f954867f700 1 -- 172.21.15.96:0/718586863 shutdown_connections 2024-04-22T15:44:02.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.741+0000 7f954867f700 1 -- 172.21.15.96:0/718586863 wait complete. 2024-04-22T15:44:02.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954867f700 1 Processor -- start 2024-04-22T15:44:02.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954867f700 1 -- start start 2024-04-22T15:44:02.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954867f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 0x7f954019b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954867f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95401017c0 0x7f954019bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954867f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954019c260 con 0x7f9540100e10 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954867f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95401969a0 con 0x7f95401017c0 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954641b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 0x7f954019b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954641b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 0x7f954019b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34618/0 (socket says 172.21.15.96:34618) 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954641b700 1 -- 172.21.15.96:0/1783645503 learned_addr learned my addr 172.21.15.96:0/1783645503 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f9545c1a700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95401017c0 0x7f954019bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954641b700 1 -- 172.21.15.96:0/1783645503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95401017c0 msgr2=0x7f954019bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954641b700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95401017c0 0x7f954019bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954641b700 1 -- 172.21.15.96:0/1783645503 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c0095f0 con 0x7f9540100e10 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.742+0000 7f954641b700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 0x7f954019b640 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f953c00bd70 tx=0x7f953c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.743+0000 7f95377fe700 1 -- 172.21.15.96:0/1783645503 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f953c01d070 con 0x7f9540100e10 2024-04-22T15:44:02.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.743+0000 7f95377fe700 1 -- 172.21.15.96:0/1783645503 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f953c005700 con 0x7f9540100e10 2024-04-22T15:44:02.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.743+0000 7f95377fe700 1 -- 172.21.15.96:0/1783645503 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f953c017400 con 0x7f9540100e10 2024-04-22T15:44:02.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.743+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9540196c20 con 0x7f9540100e10 2024-04-22T15:44:02.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.743+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9540197110 con 0x7f9540100e10 2024-04-22T15:44:02.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.744+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95401091b0 con 0x7f9540100e10 2024-04-22T15:44:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.744+0000 7f95377fe700 1 -- 172.21.15.96:0/1783645503 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f953c003a60 con 0x7f9540100e10 2024-04-22T15:44:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.745+0000 7f95377fe700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f952c06ecc0 0x7f952c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.745+0000 7f95377fe700 1 -- 172.21.15.96:0/1783645503 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f953c090320 con 0x7f9540100e10 2024-04-22T15:44:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.745+0000 7f9545c1a700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f952c06ecc0 0x7f952c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.745+0000 7f9545c1a700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f952c06ecc0 0x7f952c071180 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9540107770 tx=0x7f953000a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:02.749+0000 7f95377fe700 1 -- 172.21.15.96:0/1783645503 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f953c059080 con 0x7f9540100e10 2024-04-22T15:44:02.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:02 smithi096 ceph-mon[112443]: pgmap v666: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:02 smithi196 ceph-mon[81815]: pgmap v666: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:03.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.041+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f95401022f0 con 0x7f9540100e10 2024-04-22T15:44:03.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.041+0000 7f95377fe700 1 -- 172.21.15.96:0/1783645503 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f953c05c740 con 0x7f9540100e10 2024-04-22T15:44:03.044 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:44:03.045 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:44:03.046 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:44:03.046 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:44:03.046 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:44:03.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f952c06ecc0 msgr2=0x7f952c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f952c06ecc0 0x7f952c071180 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9540107770 tx=0x7f953000a300 comp rx=0 tx=0).stop 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 msgr2=0x7f954019b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 0x7f954019b640 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f953c00bd70 tx=0x7f953c00bda0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 shutdown_connections 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f952c06ecc0 0x7f952c071180 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9540100e10 0x7f954019b640 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 --2- 172.21.15.96:0/1783645503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95401017c0 0x7f954019bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 >> 172.21.15.96:0/1783645503 conn(0x7f95400fc6e0 msgr2=0x7f95400fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 shutdown_connections 2024-04-22T15:44:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.044+0000 7f954867f700 1 -- 172.21.15.96:0/1783645503 wait complete. 2024-04-22T15:44:03.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.156+0000 7f6925353700 1 -- 172.21.15.96:0/2407470381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 msgr2=0x7f6920103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.156+0000 7f6925353700 1 --2- 172.21.15.96:0/2407470381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 0x7f6920103ed0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6914009910 tx=0x7f6914009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:03.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 -- 172.21.15.96:0/2407470381 shutdown_connections 2024-04-22T15:44:03.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 --2- 172.21.15.96:0/2407470381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 0x7f6920103ed0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 --2- 172.21.15.96:0/2407470381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69200ff630 0x7f69200ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 -- 172.21.15.96:0/2407470381 >> 172.21.15.96:0/2407470381 conn(0x7f6920074d90 msgr2=0x7f69200751a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:03.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 -- 172.21.15.96:0/2407470381 shutdown_connections 2024-04-22T15:44:03.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 -- 172.21.15.96:0/2407470381 wait complete. 2024-04-22T15:44:03.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 Processor -- start 2024-04-22T15:44:03.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 -- start start 2024-04-22T15:44:03.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69200ff630 0x7f692019b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:03.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 0x7f692019bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:03.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.157+0000 7f6925353700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692019c2c0 con 0x7f69200fffe0 2024-04-22T15:44:03.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f6925353700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6920195830 con 0x7f69200ff630 2024-04-22T15:44:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f691e7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 0x7f692019bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f691e7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 0x7f692019bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34638/0 (socket says 172.21.15.96:34638) 2024-04-22T15:44:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f691e7fc700 1 -- 172.21.15.96:0/792794773 learned_addr learned my addr 172.21.15.96:0/792794773 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f691effd700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69200ff630 0x7f692019b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f691e7fc700 1 -- 172.21.15.96:0/792794773 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69200ff630 msgr2=0x7f692019b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f691e7fc700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69200ff630 0x7f692019b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f691e7fc700 1 -- 172.21.15.96:0/792794773 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69140095f0 con 0x7f69200fffe0 2024-04-22T15:44:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f691e7fc700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 0x7f692019bc30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f69140092e0 tx=0x7f691400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:03.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f6907fff700 1 -- 172.21.15.96:0/792794773 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f691401d070 con 0x7f69200fffe0 2024-04-22T15:44:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f6907fff700 1 -- 172.21.15.96:0/792794773 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f691400f460 con 0x7f69200fffe0 2024-04-22T15:44:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6920195ab0 con 0x7f69200fffe0 2024-04-22T15:44:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f6907fff700 1 -- 172.21.15.96:0/792794773 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6914017610 con 0x7f69200fffe0 2024-04-22T15:44:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.158+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6920195fa0 con 0x7f69200fffe0 2024-04-22T15:44:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.159+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69201009b0 con 0x7f69200fffe0 2024-04-22T15:44:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.160+0000 7f6907fff700 1 -- 172.21.15.96:0/792794773 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f691400f5d0 con 0x7f69200fffe0 2024-04-22T15:44:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.160+0000 7f6907fff700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f690806ecd0 0x7f6908071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.160+0000 7f6907fff700 1 -- 172.21.15.96:0/792794773 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6914090390 con 0x7f69200fffe0 2024-04-22T15:44:03.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.160+0000 7f691effd700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f690806ecd0 0x7f6908071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:03.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.161+0000 7f691effd700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f690806ecd0 0x7f6908071190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6910005fd0 tx=0x7f691000a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:03.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.165+0000 7f6907fff700 1 -- 172.21.15.96:0/792794773 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f691405cc20 con 0x7f69200fffe0 2024-04-22T15:44:03.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.407+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f692010c470 con 0x7f69200fffe0 2024-04-22T15:44:03.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.407+0000 7f6907fff700 1 -- 172.21.15.96:0/792794773 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f691405c7b0 con 0x7f69200fffe0 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:44:03.411 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:44:03.412 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:44:03.413 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:44:03.414 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:44:03.414 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:44:03.414 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f690806ecd0 msgr2=0x7f6908071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f690806ecd0 0x7f6908071190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6910005fd0 tx=0x7f691000a560 comp rx=0 tx=0).stop 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 msgr2=0x7f692019bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 0x7f692019bc30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f69140092e0 tx=0x7f691400be20 comp rx=0 tx=0).stop 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 shutdown_connections 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f690806ecd0 0x7f6908071190 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69200ff630 0x7f692019b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 --2- 172.21.15.96:0/792794773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69200fffe0 0x7f692019bc30 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 >> 172.21.15.96:0/792794773 conn(0x7f6920074d90 msgr2=0x7f69200fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 shutdown_connections 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.410+0000 7f6925353700 1 -- 172.21.15.96:0/792794773 wait complete. 2024-04-22T15:44:03.415 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:44:03.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.516+0000 7ff98fbe3700 1 -- 172.21.15.96:0/4111495472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9881017d0 msgr2=0x7ff988105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.516+0000 7ff98fbe3700 1 --2- 172.21.15.96:0/4111495472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9881017d0 0x7ff988105820 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff984009910 tx=0x7ff984009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:03.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.516+0000 7ff98fbe3700 1 -- 172.21.15.96:0/4111495472 shutdown_connections 2024-04-22T15:44:03.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.516+0000 7ff98fbe3700 1 --2- 172.21.15.96:0/4111495472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9881017d0 0x7ff988105820 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.516+0000 7ff98fbe3700 1 --2- 172.21.15.96:0/4111495472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff988100e20 0x7ff988101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.516+0000 7ff98fbe3700 1 -- 172.21.15.96:0/4111495472 >> 172.21.15.96:0/4111495472 conn(0x7ff9880fc6d0 msgr2=0x7ff9880feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.516+0000 7ff98fbe3700 1 -- 172.21.15.96:0/4111495472 shutdown_connections 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.516+0000 7ff98fbe3700 1 -- 172.21.15.96:0/4111495472 wait complete. 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98fbe3700 1 Processor -- start 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98fbe3700 1 -- start start 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98fbe3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff988100e20 0x7ff98819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98fbe3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9881017d0 0x7ff98819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98fbe3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98819d2f0 con 0x7ff988100e20 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98fbe3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff988196960 con 0x7ff9881017d0 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98d97f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff988100e20 0x7ff98819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98d97f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff988100e20 0x7ff98819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34652/0 (socket says 172.21.15.96:34652) 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98d97f700 1 -- 172.21.15.96:0/1673185341 learned_addr learned my addr 172.21.15.96:0/1673185341 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.517+0000 7ff98d17e700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9881017d0 0x7ff98819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.518+0000 7ff98d97f700 1 -- 172.21.15.96:0/1673185341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9881017d0 msgr2=0x7ff98819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.518+0000 7ff98d97f700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9881017d0 0x7ff98819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.518+0000 7ff98d97f700 1 -- 172.21.15.96:0/1673185341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9840095f0 con 0x7ff988100e20 2024-04-22T15:44:03.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.518+0000 7ff98d97f700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff988100e20 0x7ff98819c820 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff97800c930 tx=0x7ff97800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:03.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.518+0000 7ff97effd700 1 -- 172.21.15.96:0/1673185341 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff978007ad0 con 0x7ff988100e20 2024-04-22T15:44:03.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.518+0000 7ff97effd700 1 -- 172.21.15.96:0/1673185341 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff97800ce80 con 0x7ff988100e20 2024-04-22T15:44:03.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.518+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff988196c40 con 0x7ff988100e20 2024-04-22T15:44:03.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.518+0000 7ff97effd700 1 -- 172.21.15.96:0/1673185341 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff978018630 con 0x7ff988100e20 2024-04-22T15:44:03.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.518+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff988197190 con 0x7ff988100e20 2024-04-22T15:44:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.519+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff988109220 con 0x7ff988100e20 2024-04-22T15:44:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.520+0000 7ff97effd700 1 -- 172.21.15.96:0/1673185341 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff978007c30 con 0x7ff988100e20 2024-04-22T15:44:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.520+0000 7ff97effd700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff97406ecc0 0x7ff974071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.520+0000 7ff97effd700 1 -- 172.21.15.96:0/1673185341 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff97808eef0 con 0x7ff988100e20 2024-04-22T15:44:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.520+0000 7ff98d17e700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff97406ecc0 0x7ff974071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.520+0000 7ff98d17e700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff97406ecc0 0x7ff974071180 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff988197b20 tx=0x7ff98401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:03.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.525+0000 7ff97effd700 1 -- 172.21.15.96:0/1673185341 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff97805b780 con 0x7ff988100e20 2024-04-22T15:44:03.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:03 smithi096 ceph-mon[112443]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:03 smithi096 ceph-mon[112443]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:03 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1783645503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:44:03.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:03 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/792794773' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:44:03.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.750+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff98810ddc0 con 0x7ff97406ecc0 2024-04-22T15:44:03.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.751+0000 7ff97effd700 1 -- 172.21.15.96:0/1673185341 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff98810ddc0 con 0x7ff97406ecc0 2024-04-22T15:44:03.754 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:44:03.754 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:44:03.754 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:44:03.754 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:44:03.754 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:44:03.754 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:44:03.755 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:44:03.755 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:44:03.755 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:44:03.755 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:44:03.755 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:44:03.755 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:44:03.755 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:44:03.755 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:44:03.755 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:44:03.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff97406ecc0 msgr2=0x7ff974071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff97406ecc0 0x7ff974071180 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff988197b20 tx=0x7ff98401a040 comp rx=0 tx=0).stop 2024-04-22T15:44:03.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff988100e20 msgr2=0x7ff98819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff988100e20 0x7ff98819c820 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff97800c930 tx=0x7ff97800cc40 comp rx=0 tx=0).stop 2024-04-22T15:44:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 shutdown_connections 2024-04-22T15:44:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff97406ecc0 0x7ff974071180 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff988100e20 0x7ff98819c820 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 --2- 172.21.15.96:0/1673185341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff9881017d0 0x7ff98819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 >> 172.21.15.96:0/1673185341 conn(0x7ff9880fc6d0 msgr2=0x7ff9880fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 shutdown_connections 2024-04-22T15:44:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.753+0000 7ff98fbe3700 1 -- 172.21.15.96:0/1673185341 wait complete. 2024-04-22T15:44:03.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.856+0000 7f205549f700 1 -- 172.21.15.96:0/3714696939 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 msgr2=0x7f2050105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.856+0000 7f205549f700 1 --2- 172.21.15.96:0/3714696939 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 0x7f2050105f60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2040009910 tx=0x7f2040009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.856+0000 7f205549f700 1 -- 172.21.15.96:0/3714696939 shutdown_connections 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.856+0000 7f205549f700 1 --2- 172.21.15.96:0/3714696939 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20500ffaf0 0x7f20500fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.856+0000 7f205549f700 1 --2- 172.21.15.96:0/3714696939 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 0x7f2050105f60 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.856+0000 7f205549f700 1 -- 172.21.15.96:0/3714696939 >> 172.21.15.96:0/3714696939 conn(0x7f20500fb5e0 msgr2=0x7f20500fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.856+0000 7f205549f700 1 -- 172.21.15.96:0/3714696939 shutdown_connections 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.856+0000 7f205549f700 1 -- 172.21.15.96:0/3714696939 wait complete. 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f205549f700 1 Processor -- start 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f205549f700 1 -- start start 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f205549f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20500ffaf0 0x7f20500fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f205549f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 0x7f20500fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:03.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f205549f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500fdd60 con 0x7f2050105b80 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f205549f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500fc810 con 0x7f20500ffaf0 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f204e7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 0x7f20500fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f204e7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 0x7f20500fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34664/0 (socket says 172.21.15.96:34664) 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f204e7fc700 1 -- 172.21.15.96:0/870066127 learned_addr learned my addr 172.21.15.96:0/870066127 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f204e7fc700 1 -- 172.21.15.96:0/870066127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20500ffaf0 msgr2=0x7f20500fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f204e7fc700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20500ffaf0 0x7f20500fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f204e7fc700 1 -- 172.21.15.96:0/870066127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20400095f0 con 0x7f2050105b80 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.857+0000 7f204e7fc700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 0x7f20500fc2d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f204400c930 tx=0x7f204400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.858+0000 7f2037fff700 1 -- 172.21.15.96:0/870066127 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2044007ad0 con 0x7f2050105b80 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.858+0000 7f2037fff700 1 -- 172.21.15.96:0/870066127 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2044007c30 con 0x7f2050105b80 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.858+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20500fcac0 con 0x7f2050105b80 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.858+0000 7f2037fff700 1 -- 172.21.15.96:0/870066127 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2044018610 con 0x7f2050105b80 2024-04-22T15:44:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.858+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20501a2f30 con 0x7f2050105b80 2024-04-22T15:44:03.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.859+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2050109100 con 0x7f2050105b80 2024-04-22T15:44:03.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.859+0000 7f2037fff700 1 -- 172.21.15.96:0/870066127 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f204400f450 con 0x7f2050105b80 2024-04-22T15:44:03.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.859+0000 7f2037fff700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f203806ecd0 0x7f2038071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:03.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.860+0000 7f2037fff700 1 -- 172.21.15.96:0/870066127 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f204408f0d0 con 0x7f2050105b80 2024-04-22T15:44:03.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.860+0000 7f204effd700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f203806ecd0 0x7f2038071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:03.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.860+0000 7f204effd700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f203806ecd0 0x7f2038071190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2040006010 tx=0x7f204000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:03.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:03.864+0000 7f2037fff700 1 -- 172.21.15.96:0/870066127 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2044057e30 con 0x7f2050105b80 2024-04-22T15:44:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:03 smithi196 ceph-mon[81815]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:03 smithi196 ceph-mon[81815]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:03 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1783645503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:44:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:03 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/792794773' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:44:04.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.145+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f205010ccf0 con 0x7f2050105b80 2024-04-22T15:44:04.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.146+0000 7f2037fff700 1 -- 172.21.15.96:0/870066127 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f204405b4f0 con 0x7f2050105b80 2024-04-22T15:44:04.149 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:44:04.149 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:44:04.149 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:44:04.149 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:44:04.149 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:44:04.149 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:44:04.149 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:44:04.149 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:44:04.150 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:44:04.151 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:44:04.151 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:44:04.151 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:44:04.151 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:44:04.151 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:44:04.151 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f203806ecd0 msgr2=0x7f2038071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f203806ecd0 0x7f2038071190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2040006010 tx=0x7f204000b540 comp rx=0 tx=0).stop 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 msgr2=0x7f20500fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 0x7f20500fc2d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f204400c930 tx=0x7f204400ccf0 comp rx=0 tx=0).stop 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 shutdown_connections 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f203806ecd0 0x7f2038071190 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20500ffaf0 0x7f20500fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 --2- 172.21.15.96:0/870066127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2050105b80 0x7f20500fc2d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 >> 172.21.15.96:0/870066127 conn(0x7f20500fb5e0 msgr2=0x7f2050069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 shutdown_connections 2024-04-22T15:44:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:04.148+0000 7f205549f700 1 -- 172.21.15.96:0/870066127 wait complete. 2024-04-22T15:44:04.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:04 smithi096 ceph-mon[112443]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:04 smithi096 ceph-mon[112443]: pgmap v667: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:04 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/870066127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:44:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:04 smithi196 ceph-mon[81815]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:04 smithi196 ceph-mon[81815]: pgmap v667: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:04 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/870066127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:44:05.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:05 smithi096 ceph-mon[112443]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:05 smithi196 ceph-mon[81815]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:06.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:06 smithi096 ceph-mon[112443]: pgmap v668: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:06 smithi196 ceph-mon[81815]: pgmap v668: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:08.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:08 smithi096 ceph-mon[112443]: pgmap v669: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:08 smithi196 ceph-mon[81815]: pgmap v669: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:10 smithi096 ceph-mon[112443]: pgmap v670: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:10.996 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:10 smithi196 ceph-mon[81815]: pgmap v670: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:12.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:12 smithi096 ceph-mon[112443]: pgmap v671: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:12 smithi196 ceph-mon[81815]: pgmap v671: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:14.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:14 smithi096 ceph-mon[112443]: pgmap v672: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:14 smithi196 ceph-mon[81815]: pgmap v672: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:16.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:16 smithi096 ceph-mon[112443]: pgmap v673: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:16 smithi196 ceph-mon[81815]: pgmap v673: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:18.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:18 smithi096 ceph-mon[112443]: pgmap v674: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:18 smithi196 ceph-mon[81815]: pgmap v674: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:20.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:20 smithi196 ceph-mon[81815]: pgmap v675: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:20.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:20 smithi096 ceph-mon[112443]: pgmap v675: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:22 smithi096 ceph-mon[112443]: pgmap v676: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:22 smithi196 ceph-mon[81815]: pgmap v676: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:24.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:24 smithi096 ceph-mon[112443]: pgmap v677: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:24 smithi196 ceph-mon[81815]: pgmap v677: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:26.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:26 smithi096 ceph-mon[112443]: pgmap v678: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:26.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:26 smithi196 ceph-mon[81815]: pgmap v678: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:28.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:28 smithi096 ceph-mon[112443]: pgmap v679: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:28 smithi196 ceph-mon[81815]: pgmap v679: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:30.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:30 smithi096 ceph-mon[112443]: pgmap v680: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:30 smithi196 ceph-mon[81815]: pgmap v680: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:32.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:32 smithi096 ceph-mon[112443]: pgmap v681: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:32 smithi196 ceph-mon[81815]: pgmap v681: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:34.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.267+0000 7f132c7c1700 1 -- 172.21.15.96:0/3295146439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 msgr2=0x7f13240fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.267+0000 7f132c7c1700 1 --2- 172.21.15.96:0/3295146439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 0x7f13240fff20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1314009910 tx=0x7f1314009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:34.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.267+0000 7f132c7c1700 1 -- 172.21.15.96:0/3295146439 shutdown_connections 2024-04-22T15:44:34.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.267+0000 7f132c7c1700 1 --2- 172.21.15.96:0/3295146439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 0x7f13240fff20 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.267+0000 7f132c7c1700 1 --2- 172.21.15.96:0/3295146439 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1324105aa0 0x7f1324105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.267+0000 7f132c7c1700 1 -- 172.21.15.96:0/3295146439 >> 172.21.15.96:0/3295146439 conn(0x7f1324076740 msgr2=0x7f13240fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:34.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.267+0000 7f132c7c1700 1 -- 172.21.15.96:0/3295146439 shutdown_connections 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.267+0000 7f132c7c1700 1 -- 172.21.15.96:0/3295146439 wait complete. 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132c7c1700 1 Processor -- start 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132c7c1700 1 -- start start 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132c7c1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 0x7f132419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132c7c1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1324105aa0 0x7f132419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132c7c1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132419d3b0 con 0x7f13240ffaa0 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132c7c1700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13241968d0 con 0x7f1324105aa0 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132a55d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 0x7f132419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132a55d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 0x7f132419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54390/0 (socket says 172.21.15.96:54390) 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132a55d700 1 -- 172.21.15.96:0/3646406012 learned_addr learned my addr 172.21.15.96:0/3646406012 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f1329d5c700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1324105aa0 0x7f132419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132a55d700 1 -- 172.21.15.96:0/3646406012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1324105aa0 msgr2=0x7f132419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132a55d700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1324105aa0 0x7f132419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f132a55d700 1 -- 172.21.15.96:0/3646406012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13140095f0 con 0x7f13240ffaa0 2024-04-22T15:44:34.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.268+0000 7f1329d5c700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1324105aa0 0x7f132419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:44:34.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.269+0000 7f132a55d700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 0x7f132419c820 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f132000cfd0 tx=0x7f132000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:34.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.269+0000 7f131b7fe700 1 -- 172.21.15.96:0/3646406012 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f132000ad70 con 0x7f13240ffaa0 2024-04-22T15:44:34.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.269+0000 7f131b7fe700 1 -- 172.21.15.96:0/3646406012 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13200128e0 con 0x7f13240ffaa0 2024-04-22T15:44:34.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.269+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1324196bb0 con 0x7f13240ffaa0 2024-04-22T15:44:34.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.269+0000 7f131b7fe700 1 -- 172.21.15.96:0/3646406012 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1320011880 con 0x7f13240ffaa0 2024-04-22T15:44:34.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.269+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1324197100 con 0x7f13240ffaa0 2024-04-22T15:44:34.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.270+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13241011e0 con 0x7f13240ffaa0 2024-04-22T15:44:34.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.271+0000 7f131b7fe700 1 -- 172.21.15.96:0/3646406012 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1320012420 con 0x7f13240ffaa0 2024-04-22T15:44:34.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.271+0000 7f131b7fe700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f131006ecc0 0x7f1310071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:34.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.271+0000 7f1329d5c700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f131006ecc0 0x7f1310071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:34.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.271+0000 7f131b7fe700 1 -- 172.21.15.96:0/3646406012 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f132008f020 con 0x7f13240ffaa0 2024-04-22T15:44:34.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.272+0000 7f1329d5c700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f131006ecc0 0x7f1310071180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1314005f50 tx=0x7f1314005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:34.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.276+0000 7f131b7fe700 1 -- 172.21.15.96:0/3646406012 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1320057d80 con 0x7f13240ffaa0 2024-04-22T15:44:34.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.498+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f132410cd10 con 0x7f131006ecc0 2024-04-22T15:44:34.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.499+0000 7f131b7fe700 1 -- 172.21.15.96:0/3646406012 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f132410cd10 con 0x7f131006ecc0 2024-04-22T15:44:34.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f131006ecc0 msgr2=0x7f1310071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f131006ecc0 0x7f1310071180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1314005f50 tx=0x7f1314005dc0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 msgr2=0x7f132419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 0x7f132419c820 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f132000cfd0 tx=0x7f132000a340 comp rx=0 tx=0).stop 2024-04-22T15:44:34.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 shutdown_connections 2024-04-22T15:44:34.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f131006ecc0 0x7f1310071180 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13240ffaa0 0x7f132419c820 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 --2- 172.21.15.96:0/3646406012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1324105aa0 0x7f132419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 >> 172.21.15.96:0/3646406012 conn(0x7f1324076740 msgr2=0x7f13240687a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:34.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 shutdown_connections 2024-04-22T15:44:34.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.501+0000 7f132c7c1700 1 -- 172.21.15.96:0/3646406012 wait complete. 2024-04-22T15:44:34.514 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:44:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.608+0000 7fdd115b0700 1 -- 172.21.15.96:0/2087073955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c0ffaa0 msgr2=0x7fdd0c0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.608+0000 7fdd115b0700 1 --2- 172.21.15.96:0/2087073955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c0ffaa0 0x7fdd0c0fff20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009910 tx=0x7fdcfc009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.608+0000 7fdd115b0700 1 -- 172.21.15.96:0/2087073955 shutdown_connections 2024-04-22T15:44:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.608+0000 7fdd115b0700 1 --2- 172.21.15.96:0/2087073955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c0ffaa0 0x7fdd0c0fff20 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.608+0000 7fdd115b0700 1 --2- 172.21.15.96:0/2087073955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd0c105aa0 0x7fdd0c105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.608+0000 7fdd115b0700 1 -- 172.21.15.96:0/2087073955 >> 172.21.15.96:0/2087073955 conn(0x7fdd0c076740 msgr2=0x7fdd0c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.608+0000 7fdd115b0700 1 -- 172.21.15.96:0/2087073955 shutdown_connections 2024-04-22T15:44:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.608+0000 7fdd115b0700 1 -- 172.21.15.96:0/2087073955 wait complete. 2024-04-22T15:44:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd115b0700 1 Processor -- start 2024-04-22T15:44:34.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd115b0700 1 -- start start 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd115b0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd0c0ffaa0 0x7fdd0c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd115b0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c105aa0 0x7fdd0c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd115b0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c198510 con 0x7fdd0c105aa0 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd115b0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c198650 con 0x7fdd0c0ffaa0 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd0a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c105aa0 0x7fdd0c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd0a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c105aa0 0x7fdd0c197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54400/0 (socket says 172.21.15.96:54400) 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd0a7fc700 1 -- 172.21.15.96:0/1182982856 learned_addr learned my addr 172.21.15.96:0/1182982856 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd0affd700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd0c0ffaa0 0x7fdd0c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd0a7fc700 1 -- 172.21.15.96:0/1182982856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd0c0ffaa0 msgr2=0x7fdd0c197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd0a7fc700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd0c0ffaa0 0x7fdd0c197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.609+0000 7fdd0a7fc700 1 -- 172.21.15.96:0/1182982856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc0095f0 con 0x7fdd0c105aa0 2024-04-22T15:44:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.610+0000 7fdd0a7fc700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c105aa0 0x7fdd0c197f40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00bd70 tx=0x7fdcfc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:34.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.610+0000 7fdcf3fff700 1 -- 172.21.15.96:0/1182982856 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc01d070 con 0x7fdd0c105aa0 2024-04-22T15:44:34.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.610+0000 7fdcf3fff700 1 -- 172.21.15.96:0/1182982856 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcfc00f460 con 0x7fdd0c105aa0 2024-04-22T15:44:34.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.610+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0c198870 con 0x7fdd0c105aa0 2024-04-22T15:44:34.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.610+0000 7fdcf3fff700 1 -- 172.21.15.96:0/1182982856 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc017630 con 0x7fdd0c105aa0 2024-04-22T15:44:34.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.610+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0c1a0dd0 con 0x7fdd0c105aa0 2024-04-22T15:44:34.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.611+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0c1080c0 con 0x7fdd0c105aa0 2024-04-22T15:44:34.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.611+0000 7fdcf3fff700 1 -- 172.21.15.96:0/1182982856 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdcfc00f5d0 con 0x7fdd0c105aa0 2024-04-22T15:44:34.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.612+0000 7fdcf3fff700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdcf406ecc0 0x7fdcf4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:34.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.612+0000 7fdd0affd700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdcf406ecc0 0x7fdcf4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:34.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.612+0000 7fdcf3fff700 1 -- 172.21.15.96:0/1182982856 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fdcfc090240 con 0x7fdd0c105aa0 2024-04-22T15:44:34.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.612+0000 7fdd0affd700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdcf406ecc0 0x7fdcf4071180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdd00006fd0 tx=0x7fdd00008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:34.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.617+0000 7fdcf3fff700 1 -- 172.21.15.96:0/1182982856 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdcfc058fa0 con 0x7fdd0c105aa0 2024-04-22T15:44:34.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.837+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd0c198870 con 0x7fdcf406ecc0 2024-04-22T15:44:34.840 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:34 smithi096 ceph-mon[112443]: pgmap v682: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:34.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.838+0000 7fdcf3fff700 1 -- 172.21.15.96:0/1182982856 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdd0c198870 con 0x7fdcf406ecc0 2024-04-22T15:44:34.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdcf406ecc0 msgr2=0x7fdcf4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdcf406ecc0 0x7fdcf4071180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdd00006fd0 tx=0x7fdd00008040 comp rx=0 tx=0).stop 2024-04-22T15:44:34.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c105aa0 msgr2=0x7fdd0c197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c105aa0 0x7fdd0c197f40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00bd70 tx=0x7fdcfc00be50 comp rx=0 tx=0).stop 2024-04-22T15:44:34.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 shutdown_connections 2024-04-22T15:44:34.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdcf406ecc0 0x7fdcf4071180 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd0c0ffaa0 0x7fdd0c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 --2- 172.21.15.96:0/1182982856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd0c105aa0 0x7fdd0c197f40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 >> 172.21.15.96:0/1182982856 conn(0x7fdd0c076740 msgr2=0x7fdd0c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:34.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 shutdown_connections 2024-04-22T15:44:34.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.840+0000 7fdd115b0700 1 -- 172.21.15.96:0/1182982856 wait complete. 2024-04-22T15:44:34.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.944+0000 7f8bad58c700 1 -- 172.21.15.96:0/356391656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba80ffab0 msgr2=0x7f8ba80fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.944+0000 7f8bad58c700 1 --2- 172.21.15.96:0/356391656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba80ffab0 0x7f8ba80fff30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009910 tx=0x7f8b90009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:34.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.944+0000 7f8bad58c700 1 -- 172.21.15.96:0/356391656 shutdown_connections 2024-04-22T15:44:34.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.944+0000 7f8bad58c700 1 --2- 172.21.15.96:0/356391656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba80ffab0 0x7f8ba80fff30 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.944+0000 7f8bad58c700 1 --2- 172.21.15.96:0/356391656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba8105ab0 0x7f8ba8105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.944+0000 7f8bad58c700 1 -- 172.21.15.96:0/356391656 >> 172.21.15.96:0/356391656 conn(0x7f8ba80fb5e0 msgr2=0x7f8ba80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:34.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.944+0000 7f8bad58c700 1 -- 172.21.15.96:0/356391656 shutdown_connections 2024-04-22T15:44:34.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.944+0000 7f8bad58c700 1 -- 172.21.15.96:0/356391656 wait complete. 2024-04-22T15:44:34.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8bad58c700 1 Processor -- start 2024-04-22T15:44:34.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8bad58c700 1 -- start start 2024-04-22T15:44:34.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8bad58c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba80ffab0 0x7f8ba8197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:34.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8bad58c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba8105ab0 0x7f8ba8197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:34.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8bad58c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba8198480 con 0x7f8ba8105ab0 2024-04-22T15:44:34.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8bad58c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba81985c0 con 0x7f8ba80ffab0 2024-04-22T15:44:34.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8b9ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba8105ab0 0x7f8ba8197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:34.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8b9ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba8105ab0 0x7f8ba8197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54418/0 (socket says 172.21.15.96:54418) 2024-04-22T15:44:34.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8b9ffff700 1 -- 172.21.15.96:0/2655647558 learned_addr learned my addr 172.21.15.96:0/2655647558 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8ba6ffd700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba80ffab0 0x7f8ba8197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8b9ffff700 1 -- 172.21.15.96:0/2655647558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba80ffab0 msgr2=0x7f8ba8197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8b9ffff700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba80ffab0 0x7f8ba8197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8b9ffff700 1 -- 172.21.15.96:0/2655647558 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b900095f0 con 0x7f8ba8105ab0 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8b9ffff700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba8105ab0 0x7f8ba8197eb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000bd70 tx=0x7f8b9000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.945+0000 7f8ba4ff9700 1 -- 172.21.15.96:0/2655647558 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b9001d070 con 0x7f8ba8105ab0 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.946+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba81a09e0 con 0x7f8ba8105ab0 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.946+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba81a0d40 con 0x7f8ba8105ab0 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.946+0000 7f8ba4ff9700 1 -- 172.21.15.96:0/2655647558 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b90005700 con 0x7f8ba8105ab0 2024-04-22T15:44:34.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.946+0000 7f8ba4ff9700 1 -- 172.21.15.96:0/2655647558 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b90017400 con 0x7f8ba8105ab0 2024-04-22T15:44:34.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.946+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba81080d0 con 0x7f8ba8105ab0 2024-04-22T15:44:34.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.947+0000 7f8ba4ff9700 1 -- 172.21.15.96:0/2655647558 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8b90003a60 con 0x7f8ba8105ab0 2024-04-22T15:44:34.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.947+0000 7f8ba4ff9700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b9406ecc0 0x7f8b94071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:34.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.947+0000 7f8ba4ff9700 1 -- 172.21.15.96:0/2655647558 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8b90090350 con 0x7f8ba8105ab0 2024-04-22T15:44:34.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.947+0000 7f8ba6ffd700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b9406ecc0 0x7f8b94071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:34.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.948+0000 7f8ba6ffd700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b9406ecc0 0x7f8b94071180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8b98005950 tx=0x7f8b9800a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:34.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:34.952+0000 7f8ba4ff9700 1 -- 172.21.15.96:0/2655647558 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8b900590b0 con 0x7f8ba8105ab0 2024-04-22T15:44:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:34 smithi196 ceph-mon[81815]: pgmap v682: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:35.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.159+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ba80693f0 con 0x7f8b9406ecc0 2024-04-22T15:44:35.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.170+0000 7f8ba4ff9700 1 -- 172.21.15.96:0/2655647558 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8ba80693f0 con 0x7f8b9406ecc0 2024-04-22T15:44:35.173 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:44:35.173 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (23m) 52s ago 34m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:44:35.173 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (34m) 52s ago 34m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (33m) 54s ago 33m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (20m) 52s ago 34m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (20m) 54s ago 33m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (22m) 52s ago 34m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (16m) 52s ago 28m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (16m) 52s ago 28m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (16m) 54s ago 28m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (58s) 54s ago 28m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (25m) 52s ago 36m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (24m) 54s ago 33m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (21m) 52s ago 36m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (21m) 54s ago 33m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:44:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (24m) 52s ago 34m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:44:35.175 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (24m) 54s ago 33m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:44:35.175 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (20m) 52s ago 32m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:44:35.175 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (19m) 52s ago 32m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:44:35.175 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (18m) 52s ago 31m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:44:35.175 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (18m) 54s ago 31m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:44:35.175 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (17m) 54s ago 30m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:44:35.175 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (17m) 54s ago 30m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:44:35.175 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (23m) 52s ago 34m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b9406ecc0 msgr2=0x7f8b94071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b9406ecc0 0x7f8b94071180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8b98005950 tx=0x7f8b9800a300 comp rx=0 tx=0).stop 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba8105ab0 msgr2=0x7f8ba8197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba8105ab0 0x7f8ba8197eb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000bd70 tx=0x7f8b9000bda0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 shutdown_connections 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8b9406ecc0 0x7f8b94071180 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ba80ffab0 0x7f8ba8197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 --2- 172.21.15.96:0/2655647558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba8105ab0 0x7f8ba8197eb0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 >> 172.21.15.96:0/2655647558 conn(0x7f8ba80fb5e0 msgr2=0x7f8ba80fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:35.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 shutdown_connections 2024-04-22T15:44:35.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.172+0000 7f8bad58c700 1 -- 172.21.15.96:0/2655647558 wait complete. 2024-04-22T15:44:35.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.273+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/710902078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 msgr2=0x7f9f7810abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.273+0000 7f9f7cdeb700 1 --2- 172.21.15.96:0/710902078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 0x7f9f7810abb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c009910 tx=0x7f9f6c009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.274+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/710902078 shutdown_connections 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.274+0000 7f9f7cdeb700 1 --2- 172.21.15.96:0/710902078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 0x7f9f7810abb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.274+0000 7f9f7cdeb700 1 --2- 172.21.15.96:0/710902078 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f78101da0 0x7f9f78102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.274+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/710902078 >> 172.21.15.96:0/710902078 conn(0x7f9f780fb5f0 msgr2=0x7f9f780fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.274+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/710902078 shutdown_connections 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.274+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/710902078 wait complete. 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.274+0000 7f9f7cdeb700 1 Processor -- start 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.274+0000 7f9f7cdeb700 1 -- start start 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f7cdeb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f78101da0 0x7f9f78198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f7cdeb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 0x7f9f78199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f7cdeb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f781998f0 con 0x7f9f781026c0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f7cdeb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7819c560 con 0x7f9f78101da0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f75d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 0x7f9f78199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f75d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 0x7f9f78199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54446/0 (socket says 172.21.15.96:54446) 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f75d9b700 1 -- 172.21.15.96:0/253046885 learned_addr learned my addr 172.21.15.96:0/253046885 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f7659c700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f78101da0 0x7f9f78198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f75d9b700 1 -- 172.21.15.96:0/253046885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f78101da0 msgr2=0x7f9f78198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f75d9b700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f78101da0 0x7f9f78198d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f75d9b700 1 -- 172.21.15.96:0/253046885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c0095f0 con 0x7f9f781026c0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f75d9b700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 0x7f9f78199260 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00bd70 tx=0x7f9f6c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f677fe700 1 -- 172.21.15.96:0/253046885 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f6c01d070 con 0x7f9f781026c0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7819c7e0 con 0x7f9f781026c0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.275+0000 7f9f677fe700 1 -- 172.21.15.96:0/253046885 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f6c00f460 con 0x7f9f781026c0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.276+0000 7f9f677fe700 1 -- 172.21.15.96:0/253046885 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f6c017610 con 0x7f9f781026c0 2024-04-22T15:44:35.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.276+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7819ccd0 con 0x7f9f781026c0 2024-04-22T15:44:35.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.276+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7810a6a0 con 0x7f9f781026c0 2024-04-22T15:44:35.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.277+0000 7f9f677fe700 1 -- 172.21.15.96:0/253046885 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9f6c00f5d0 con 0x7f9f781026c0 2024-04-22T15:44:35.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.277+0000 7f9f677fe700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f6006ecc0 0x7f9f60071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:35.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.277+0000 7f9f677fe700 1 -- 172.21.15.96:0/253046885 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9f6c090390 con 0x7f9f781026c0 2024-04-22T15:44:35.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.277+0000 7f9f7659c700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f6006ecc0 0x7f9f60071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:35.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.278+0000 7f9f7659c700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f6006ecc0 0x7f9f60071180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9f68005fd0 tx=0x7f9f6800a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:35.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.282+0000 7f9f677fe700 1 -- 172.21.15.96:0/253046885 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9f6c0590f0 con 0x7f9f781026c0 2024-04-22T15:44:35.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.566+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9f78199ff0 con 0x7f9f781026c0 2024-04-22T15:44:35.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.567+0000 7f9f677fe700 1 -- 172.21.15.96:0/253046885 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9f6c05c7b0 con 0x7f9f781026c0 2024-04-22T15:44:35.570 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:44:35.570 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:44:35.570 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:44:35.571 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:44:35.572 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:44:35.572 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:44:35.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f6006ecc0 msgr2=0x7f9f60071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f6006ecc0 0x7f9f60071180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9f68005fd0 tx=0x7f9f6800a560 comp rx=0 tx=0).stop 2024-04-22T15:44:35.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 msgr2=0x7f9f78199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 0x7f9f78199260 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00bd70 tx=0x7f9f6c00be50 comp rx=0 tx=0).stop 2024-04-22T15:44:35.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 shutdown_connections 2024-04-22T15:44:35.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f6006ecc0 0x7f9f60071180 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f78101da0 0x7f9f78198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 --2- 172.21.15.96:0/253046885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f781026c0 0x7f9f78199260 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 >> 172.21.15.96:0/253046885 conn(0x7f9f780fb5f0 msgr2=0x7f9f78100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:35.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 shutdown_connections 2024-04-22T15:44:35.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.569+0000 7f9f7cdeb700 1 -- 172.21.15.96:0/253046885 wait complete. 2024-04-22T15:44:35.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 -- 172.21.15.96:0/2048314719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c0ffaa0 msgr2=0x7f950c0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 --2- 172.21.15.96:0/2048314719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c0ffaa0 0x7f950c0fff20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9500009910 tx=0x7f9500009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:35.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 -- 172.21.15.96:0/2048314719 shutdown_connections 2024-04-22T15:44:35.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 --2- 172.21.15.96:0/2048314719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c0ffaa0 0x7f950c0fff20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 --2- 172.21.15.96:0/2048314719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f950c105aa0 0x7f950c105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 -- 172.21.15.96:0/2048314719 >> 172.21.15.96:0/2048314719 conn(0x7f950c076740 msgr2=0x7f950c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:35.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 -- 172.21.15.96:0/2048314719 shutdown_connections 2024-04-22T15:44:35.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 -- 172.21.15.96:0/2048314719 wait complete. 2024-04-22T15:44:35.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 Processor -- start 2024-04-22T15:44:35.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.672+0000 7f95125f0700 1 -- start start 2024-04-22T15:44:35.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f95125f0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f950c0ffaa0 0x7f950c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:35.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f95125f0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c105aa0 0x7f950c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:35.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f95125f0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c1968d0 con 0x7f950c105aa0 2024-04-22T15:44:35.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f95125f0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c196a40 con 0x7f950c0ffaa0 2024-04-22T15:44:35.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f950b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c105aa0 0x7f950c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:35.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f950b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c105aa0 0x7f950c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54476/0 (socket says 172.21.15.96:54476) 2024-04-22T15:44:35.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f950b7fe700 1 -- 172.21.15.96:0/2018651779 learned_addr learned my addr 172.21.15.96:0/2018651779 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:35.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f950b7fe700 1 -- 172.21.15.96:0/2018651779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f950c0ffaa0 msgr2=0x7f950c19c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f950b7fe700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f950c0ffaa0 0x7f950c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f950b7fe700 1 -- 172.21.15.96:0/2018651779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95000095f0 con 0x7f950c105aa0 2024-04-22T15:44:35.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f950b7fe700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c105aa0 0x7f950c19cd60 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9500006010 tx=0x7f950000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:35.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.673+0000 7f95097fa700 1 -- 172.21.15.96:0/2018651779 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f950001d070 con 0x7f950c105aa0 2024-04-22T15:44:35.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.674+0000 7f95097fa700 1 -- 172.21.15.96:0/2018651779 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f950000f460 con 0x7f950c105aa0 2024-04-22T15:44:35.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.674+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c196cc0 con 0x7f950c105aa0 2024-04-22T15:44:35.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.674+0000 7f95097fa700 1 -- 172.21.15.96:0/2018651779 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9500017630 con 0x7f950c105aa0 2024-04-22T15:44:35.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.674+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c1971b0 con 0x7f950c105aa0 2024-04-22T15:44:35.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.674+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c1011e0 con 0x7f950c105aa0 2024-04-22T15:44:35.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.675+0000 7f95097fa700 1 -- 172.21.15.96:0/2018651779 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f950000f5d0 con 0x7f950c105aa0 2024-04-22T15:44:35.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.675+0000 7f95097fa700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f94f406ecc0 0x7f94f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:35.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.676+0000 7f95097fa700 1 -- 172.21.15.96:0/2018651779 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f95000903a0 con 0x7f950c105aa0 2024-04-22T15:44:35.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.676+0000 7f950bfff700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f94f406ecc0 0x7f94f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:35.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.676+0000 7f950bfff700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f94f406ecc0 0x7f94f4071180 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00a990 tx=0x7f94fc005c80 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:35.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.680+0000 7f95097fa700 1 -- 172.21.15.96:0/2018651779 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f95000591c0 con 0x7f950c105aa0 2024-04-22T15:44:35.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.921+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f950c10cca0 con 0x7f950c105aa0 2024-04-22T15:44:35.924 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:35 smithi096 ceph-mon[112443]: from='client.35264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:35.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.921+0000 7f95097fa700 1 -- 172.21.15.96:0/2018651779 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f950005c880 con 0x7f950c105aa0 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:44:35.925 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:44:35.926 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:44:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:44:35.928 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:44:35.928 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:44:35.928 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:44:35.928 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f94f406ecc0 msgr2=0x7f94f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f94f406ecc0 0x7f94f4071180 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00a990 tx=0x7f94fc005c80 comp rx=0 tx=0).stop 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c105aa0 msgr2=0x7f950c19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c105aa0 0x7f950c19cd60 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9500006010 tx=0x7f950000be20 comp rx=0 tx=0).stop 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 shutdown_connections 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f94f406ecc0 0x7f94f4071180 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f950c0ffaa0 0x7f950c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 --2- 172.21.15.96:0/2018651779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f950c105aa0 0x7f950c19cd60 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 >> 172.21.15.96:0/2018651779 conn(0x7f950c076740 msgr2=0x7f950c0687a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 shutdown_connections 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:35.924+0000 7f95125f0700 1 -- 172.21.15.96:0/2018651779 wait complete. 2024-04-22T15:44:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:44:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:35 smithi196 ceph-mon[81815]: from='client.35264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.029+0000 7fe936f95700 1 -- 172.21.15.96:0/67384381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 msgr2=0x7fe930101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.029+0000 7fe936f95700 1 --2- 172.21.15.96:0/67384381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 0x7fe930101200 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe924009910 tx=0x7fe924009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.029+0000 7fe936f95700 1 -- 172.21.15.96:0/67384381 shutdown_connections 2024-04-22T15:44:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.029+0000 7fe936f95700 1 --2- 172.21.15.96:0/67384381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9301017d0 0x7fe930105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.029+0000 7fe936f95700 1 --2- 172.21.15.96:0/67384381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 0x7fe930101200 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.029+0000 7fe936f95700 1 -- 172.21.15.96:0/67384381 >> 172.21.15.96:0/67384381 conn(0x7fe9300fc6d0 msgr2=0x7fe9300feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.030+0000 7fe936f95700 1 -- 172.21.15.96:0/67384381 shutdown_connections 2024-04-22T15:44:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.030+0000 7fe936f95700 1 -- 172.21.15.96:0/67384381 wait complete. 2024-04-22T15:44:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.030+0000 7fe936f95700 1 Processor -- start 2024-04-22T15:44:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.030+0000 7fe936f95700 1 -- start start 2024-04-22T15:44:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe936f95700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 0x7fe93019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe936f95700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9301017d0 0x7fe93019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe936f95700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93019d4a0 con 0x7fe930100e20 2024-04-22T15:44:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe936f95700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe930196a80 con 0x7fe9301017d0 2024-04-22T15:44:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe934d31700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 0x7fe93019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe934d31700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 0x7fe93019c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54496/0 (socket says 172.21.15.96:54496) 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe934d31700 1 -- 172.21.15.96:0/345342337 learned_addr learned my addr 172.21.15.96:0/345342337 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe92ffff700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9301017d0 0x7fe93019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe934d31700 1 -- 172.21.15.96:0/345342337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9301017d0 msgr2=0x7fe93019ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe934d31700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9301017d0 0x7fe93019ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe934d31700 1 -- 172.21.15.96:0/345342337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240095f0 con 0x7fe930100e20 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe92ffff700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9301017d0 0x7fe93019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe934d31700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 0x7fe93019c940 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe92400bd70 tx=0x7fe92400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe92dffb700 1 -- 172.21.15.96:0/345342337 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe92401d070 con 0x7fe930100e20 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe930196d00 con 0x7fe930100e20 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe92dffb700 1 -- 172.21.15.96:0/345342337 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe92400f460 con 0x7fe930100e20 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.031+0000 7fe92dffb700 1 -- 172.21.15.96:0/345342337 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe924017630 con 0x7fe930100e20 2024-04-22T15:44:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.032+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9301971f0 con 0x7fe930100e20 2024-04-22T15:44:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.033+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9301091c0 con 0x7fe930100e20 2024-04-22T15:44:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.033+0000 7fe92dffb700 1 -- 172.21.15.96:0/345342337 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe924003a60 con 0x7fe930100e20 2024-04-22T15:44:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.033+0000 7fe92dffb700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe92006ecc0 0x7fe920071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.033+0000 7fe92dffb700 1 -- 172.21.15.96:0/345342337 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe924090330 con 0x7fe930100e20 2024-04-22T15:44:36.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.033+0000 7fe92ffff700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe92006ecc0 0x7fe920071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:36.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.034+0000 7fe92ffff700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe92006ecc0 0x7fe920071180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe930197c40 tx=0x7fe91c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.038+0000 7fe92dffb700 1 -- 172.21.15.96:0/345342337 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe92405cc40 con 0x7fe930100e20 2024-04-22T15:44:36.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.256+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe930197f90 con 0x7fe92006ecc0 2024-04-22T15:44:36.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.257+0000 7fe92dffb700 1 -- 172.21.15.96:0/345342337 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe930197f90 con 0x7fe92006ecc0 2024-04-22T15:44:36.260 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:44:36.260 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:44:36.260 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:44:36.261 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.259+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe92006ecc0 msgr2=0x7fe920071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.259+0000 7fe936f95700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe92006ecc0 0x7fe920071180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe930197c40 tx=0x7fe91c008040 comp rx=0 tx=0).stop 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.259+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 msgr2=0x7fe93019c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.259+0000 7fe936f95700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 0x7fe93019c940 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe92400bd70 tx=0x7fe92400be50 comp rx=0 tx=0).stop 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.259+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 shutdown_connections 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.259+0000 7fe936f95700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe92006ecc0 0x7fe920071180 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.259+0000 7fe936f95700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe930100e20 0x7fe93019c940 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.259+0000 7fe936f95700 1 --2- 172.21.15.96:0/345342337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9301017d0 0x7fe93019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.259+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 >> 172.21.15.96:0/345342337 conn(0x7fe9300fc6d0 msgr2=0x7fe9300fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.260+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 shutdown_connections 2024-04-22T15:44:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.260+0000 7fe936f95700 1 -- 172.21.15.96:0/345342337 wait complete. 2024-04-22T15:44:36.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.362+0000 7f4b56e29700 1 -- 172.21.15.96:0/1400530718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b501017c0 msgr2=0x7f4b50105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:36.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.362+0000 7f4b56e29700 1 --2- 172.21.15.96:0/1400530718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b501017c0 0x7f4b50105810 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009910 tx=0x7f4b3c009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:36.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.362+0000 7f4b56e29700 1 -- 172.21.15.96:0/1400530718 shutdown_connections 2024-04-22T15:44:36.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.362+0000 7f4b56e29700 1 --2- 172.21.15.96:0/1400530718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b501017c0 0x7f4b50105810 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.362+0000 7f4b56e29700 1 --2- 172.21.15.96:0/1400530718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b50100e10 0x7f4b501011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.362+0000 7f4b56e29700 1 -- 172.21.15.96:0/1400530718 >> 172.21.15.96:0/1400530718 conn(0x7f4b500fc6e0 msgr2=0x7f4b500feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:36.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.362+0000 7f4b56e29700 1 -- 172.21.15.96:0/1400530718 shutdown_connections 2024-04-22T15:44:36.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.362+0000 7f4b56e29700 1 -- 172.21.15.96:0/1400530718 wait complete. 2024-04-22T15:44:36.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b56e29700 1 Processor -- start 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b56e29700 1 -- start start 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b56e29700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b50100e10 0x7f4b5019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b56e29700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b501017c0 0x7f4b5019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b56e29700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5019d380 con 0x7f4b50100e10 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b56e29700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b501969b0 con 0x7f4b501017c0 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b54bc5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b50100e10 0x7f4b5019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b54bc5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b50100e10 0x7f4b5019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54502/0 (socket says 172.21.15.96:54502) 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b54bc5700 1 -- 172.21.15.96:0/1900092005 learned_addr learned my addr 172.21.15.96:0/1900092005 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b4ffff700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b501017c0 0x7f4b5019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b54bc5700 1 -- 172.21.15.96:0/1900092005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b501017c0 msgr2=0x7f4b5019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b54bc5700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b501017c0 0x7f4b5019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b54bc5700 1 -- 172.21.15.96:0/1900092005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0095f0 con 0x7f4b50100e10 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b54bc5700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b50100e10 0x7f4b5019c820 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4b4400d8d0 tx=0x7f4b4400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.363+0000 7f4b4dffb700 1 -- 172.21.15.96:0/1900092005 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b44009940 con 0x7f4b50100e10 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.364+0000 7f4b4dffb700 1 -- 172.21.15.96:0/1900092005 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b44010460 con 0x7f4b50100e10 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.364+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b50196c90 con 0x7f4b50100e10 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.364+0000 7f4b4dffb700 1 -- 172.21.15.96:0/1900092005 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b4400f5f0 con 0x7f4b50100e10 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.364+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b501971e0 con 0x7f4b50100e10 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.365+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b50109210 con 0x7f4b50100e10 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.365+0000 7f4b4dffb700 1 -- 172.21.15.96:0/1900092005 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4b440105d0 con 0x7f4b50100e10 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.366+0000 7f4b4dffb700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b4006ecc0 0x7f4b40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.366+0000 7f4b4dffb700 1 -- 172.21.15.96:0/1900092005 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4b4408edc0 con 0x7f4b50100e10 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.366+0000 7f4b4ffff700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b4006ecc0 0x7f4b40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:36.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.366+0000 7f4b4ffff700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b4006ecc0 0x7f4b40071180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4b50197b20 tx=0x7f4b3c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:36.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.370+0000 7f4b4dffb700 1 -- 172.21.15.96:0/1900092005 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4b4405b6d0 con 0x7f4b50100e10 2024-04-22T15:44:36.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.651+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4b50197e20 con 0x7f4b50100e10 2024-04-22T15:44:36.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.652+0000 7f4b4dffb700 1 -- 172.21.15.96:0/1900092005 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f4b4405b260 con 0x7f4b50100e10 2024-04-22T15:44:36.656 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:44:36.656 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:44:36.656 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:44:36.656 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:44:36.657 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:44:36.657 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:44:36.657 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:44:36.657 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:44:36.657 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:44:36.658 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:44:36.658 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:44:36.658 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:44:36.658 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:44:36.658 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:44:36.659 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:44:36.659 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:44:36.659 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:44:36.659 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:44:36.659 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:44:36.659 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:44:36.659 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:44:36.660 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:44:36.660 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:44:36.660 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:44:36.660 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:44:36.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b4006ecc0 msgr2=0x7f4b40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:36.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b4006ecc0 0x7f4b40071180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4b50197b20 tx=0x7f4b3c01a040 comp rx=0 tx=0).stop 2024-04-22T15:44:36.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b50100e10 msgr2=0x7f4b5019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:36.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b50100e10 0x7f4b5019c820 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4b4400d8d0 tx=0x7f4b4400dc90 comp rx=0 tx=0).stop 2024-04-22T15:44:36.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 shutdown_connections 2024-04-22T15:44:36.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4b4006ecc0 0x7f4b40071180 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b50100e10 0x7f4b5019c820 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 --2- 172.21.15.96:0/1900092005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b501017c0 0x7f4b5019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:36.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 >> 172.21.15.96:0/1900092005 conn(0x7f4b500fc6e0 msgr2=0x7f4b500fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:36.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 shutdown_connections 2024-04-22T15:44:36.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:44:36.654+0000 7f4b56e29700 1 -- 172.21.15.96:0/1900092005 wait complete. 2024-04-22T15:44:36.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:36 smithi196 ceph-mon[81815]: from='client.35268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:36.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:36 smithi196 ceph-mon[81815]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:36.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:36 smithi196 ceph-mon[81815]: pgmap v683: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:36.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:36 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/253046885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:44:36.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:36 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2018651779' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:44:36.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:36 smithi096 ceph-mon[112443]: from='client.35268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:36.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:36 smithi096 ceph-mon[112443]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:36 smithi096 ceph-mon[112443]: pgmap v683: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:36 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/253046885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:44:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:36 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2018651779' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:44:37.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:37 smithi096 ceph-mon[112443]: from='client.35284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:37.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:37 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1900092005' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:44:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:37 smithi196 ceph-mon[81815]: from='client.35284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:37 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1900092005' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:44:38.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:38 smithi196 ceph-mon[81815]: pgmap v684: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:38.910 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:38.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:38 smithi096 ceph-mon[112443]: pgmap v684: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:40 smithi096 ceph-mon[112443]: pgmap v685: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:40 smithi196 ceph-mon[81815]: pgmap v685: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:42 smithi096 ceph-mon[112443]: pgmap v686: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:42 smithi196 ceph-mon[81815]: pgmap v686: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:44 smithi096 ceph-mon[112443]: pgmap v687: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:44 smithi196 ceph-mon[81815]: pgmap v687: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:46 smithi096 ceph-mon[112443]: pgmap v688: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:46 smithi196 ceph-mon[81815]: pgmap v688: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:48 smithi096 ceph-mon[112443]: pgmap v689: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:48 smithi196 ceph-mon[81815]: pgmap v689: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:50 smithi096 ceph-mon[112443]: pgmap v690: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:50 smithi196 ceph-mon[81815]: pgmap v690: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:52.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:52 smithi096 ceph-mon[112443]: pgmap v691: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:52 smithi196 ceph-mon[81815]: pgmap v691: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:54 smithi096 ceph-mon[112443]: pgmap v692: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:54 smithi196 ceph-mon[81815]: pgmap v692: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:56.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:56 smithi096 ceph-mon[112443]: pgmap v693: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:56 smithi196 ceph-mon[81815]: pgmap v693: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:44:58 smithi096 ceph-mon[112443]: pgmap v694: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:44:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:44:58 smithi196 ceph-mon[81815]: pgmap v694: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:00 smithi096 ceph-mon[112443]: pgmap v695: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:00 smithi196 ceph-mon[81815]: pgmap v695: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:02.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:02 smithi096 ceph-mon[112443]: pgmap v696: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:02 smithi196 ceph-mon[81815]: pgmap v696: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:04.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:04 smithi096 ceph-mon[112443]: pgmap v697: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:04 smithi196 ceph-mon[81815]: pgmap v697: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:06.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.772+0000 7f17fda8a700 1 -- 172.21.15.96:0/3054265132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f80ffad0 msgr2=0x7f17f80fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:06.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.772+0000 7f17fda8a700 1 --2- 172.21.15.96:0/3054265132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f80ffad0 0x7f17f80fff50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009910 tx=0x7f17ec009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:06.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.772+0000 7f17fda8a700 1 -- 172.21.15.96:0/3054265132 shutdown_connections 2024-04-22T15:45:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.772+0000 7f17fda8a700 1 --2- 172.21.15.96:0/3054265132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f80ffad0 0x7f17f80fff50 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.772+0000 7f17fda8a700 1 --2- 172.21.15.96:0/3054265132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f8105b60 0x7f17f8105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.772+0000 7f17fda8a700 1 -- 172.21.15.96:0/3054265132 >> 172.21.15.96:0/3054265132 conn(0x7f17f80fb5e0 msgr2=0x7f17f80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.772+0000 7f17fda8a700 1 -- 172.21.15.96:0/3054265132 shutdown_connections 2024-04-22T15:45:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.772+0000 7f17fda8a700 1 -- 172.21.15.96:0/3054265132 wait complete. 2024-04-22T15:45:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17fda8a700 1 Processor -- start 2024-04-22T15:45:06.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17fda8a700 1 -- start start 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17fda8a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f80ffad0 0x7f17f819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17fda8a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f8105b60 0x7f17f819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17fda8a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f819d450 con 0x7f17f8105b60 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17fda8a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8196a30 con 0x7f17f80ffad0 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17f6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f8105b60 0x7f17f819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17f6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f8105b60 0x7f17f819ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57990/0 (socket says 172.21.15.96:57990) 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17f6ffd700 1 -- 172.21.15.96:0/2242468002 learned_addr learned my addr 172.21.15.96:0/2242468002 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.773+0000 7f17f77fe700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f80ffad0 0x7f17f819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.774+0000 7f17f6ffd700 1 -- 172.21.15.96:0/2242468002 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f80ffad0 msgr2=0x7f17f819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.774+0000 7f17f6ffd700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f80ffad0 0x7f17f819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.774+0000 7f17f6ffd700 1 -- 172.21.15.96:0/2242468002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec0095f0 con 0x7f17f8105b60 2024-04-22T15:45:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.774+0000 7f17f6ffd700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f8105b60 0x7f17f819ce30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0092e0 tx=0x7f17ec00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.774+0000 7f17f4ff9700 1 -- 172.21.15.96:0/2242468002 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17ec01d070 con 0x7f17f8105b60 2024-04-22T15:45:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.774+0000 7f17f4ff9700 1 -- 172.21.15.96:0/2242468002 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17ec00f460 con 0x7f17f8105b60 2024-04-22T15:45:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.774+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f8196cb0 con 0x7f17f8105b60 2024-04-22T15:45:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.774+0000 7f17f4ff9700 1 -- 172.21.15.96:0/2242468002 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17ec017630 con 0x7f17f8105b60 2024-04-22T15:45:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.774+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f81971a0 con 0x7f17f8105b60 2024-04-22T15:45:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.775+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f8109100 con 0x7f17f8105b60 2024-04-22T15:45:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.775+0000 7f17f4ff9700 1 -- 172.21.15.96:0/2242468002 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f17ec00f5d0 con 0x7f17f8105b60 2024-04-22T15:45:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.776+0000 7f17f4ff9700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17e006ecd0 0x7f17e0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.776+0000 7f17f4ff9700 1 -- 172.21.15.96:0/2242468002 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f17ec090300 con 0x7f17f8105b60 2024-04-22T15:45:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.776+0000 7f17f77fe700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17e006ecd0 0x7f17e0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.776+0000 7f17f77fe700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17e006ecd0 0x7f17e0071190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009a10 tx=0x7f17e8009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:06.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:06.781+0000 7f17f4ff9700 1 -- 172.21.15.96:0/2242468002 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f17ec05cb90 con 0x7f17f8105b60 2024-04-22T15:45:06.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:06 smithi096 ceph-mon[112443]: pgmap v698: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:07.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.001+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f17f8197ef0 con 0x7f17e006ecd0 2024-04-22T15:45:07.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.002+0000 7f17f4ff9700 1 -- 172.21.15.96:0/2242468002 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f17f8197ef0 con 0x7f17e006ecd0 2024-04-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17e006ecd0 msgr2=0x7f17e0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17e006ecd0 0x7f17e0071190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009a10 tx=0x7f17e8009190 comp rx=0 tx=0).stop 2024-04-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f8105b60 msgr2=0x7f17f819ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f8105b60 0x7f17f819ce30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0092e0 tx=0x7f17ec00bd70 comp rx=0 tx=0).stop 2024-04-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 shutdown_connections 2024-04-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17e006ecd0 0x7f17e0071190 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f80ffad0 0x7f17f819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 --2- 172.21.15.96:0/2242468002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f8105b60 0x7f17f819ce30 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 >> 172.21.15.96:0/2242468002 conn(0x7f17f80fb5e0 msgr2=0x7f17f8104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:07.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 shutdown_connections 2024-04-22T15:45:07.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.004+0000 7f17fda8a700 1 -- 172.21.15.96:0/2242468002 wait complete. 2024-04-22T15:45:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:06 smithi196 ceph-mon[81815]: pgmap v698: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:07.017 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:45:07.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.109+0000 7f9426353700 1 -- 172.21.15.96:0/1623599214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 msgr2=0x7f94201011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.109+0000 7f9426353700 1 --2- 172.21.15.96:0/1623599214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 0x7f94201011f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9414009910 tx=0x7f9414009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:07.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.109+0000 7f9426353700 1 -- 172.21.15.96:0/1623599214 shutdown_connections 2024-04-22T15:45:07.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.109+0000 7f9426353700 1 --2- 172.21.15.96:0/1623599214 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94201017c0 0x7f9420105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.109+0000 7f9426353700 1 --2- 172.21.15.96:0/1623599214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 0x7f94201011f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.109+0000 7f9426353700 1 -- 172.21.15.96:0/1623599214 >> 172.21.15.96:0/1623599214 conn(0x7f94200fc6e0 msgr2=0x7f94200feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:07.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.109+0000 7f9426353700 1 -- 172.21.15.96:0/1623599214 shutdown_connections 2024-04-22T15:45:07.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.109+0000 7f9426353700 1 -- 172.21.15.96:0/1623599214 wait complete. 2024-04-22T15:45:07.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.110+0000 7f9426353700 1 Processor -- start 2024-04-22T15:45:07.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.110+0000 7f9426353700 1 -- start start 2024-04-22T15:45:07.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.110+0000 7f9426353700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 0x7f942019b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:07.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.110+0000 7f9426353700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94201017c0 0x7f942019bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:07.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.110+0000 7f9426353700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942019c260 con 0x7f9420100e10 2024-04-22T15:45:07.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.110+0000 7f9426353700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94201969a0 con 0x7f94201017c0 2024-04-22T15:45:07.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.110+0000 7f941ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 0x7f942019b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.110+0000 7f941ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 0x7f942019b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58006/0 (socket says 172.21.15.96:58006) 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.110+0000 7f941ffff700 1 -- 172.21.15.96:0/2529689960 learned_addr learned my addr 172.21.15.96:0/2529689960 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.111+0000 7f941ffff700 1 -- 172.21.15.96:0/2529689960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94201017c0 msgr2=0x7f942019bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.111+0000 7f941ffff700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94201017c0 0x7f942019bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.111+0000 7f941ffff700 1 -- 172.21.15.96:0/2529689960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94140095f0 con 0x7f9420100e10 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.111+0000 7f941ffff700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 0x7f942019b640 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9414000c00 tx=0x7f941400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.111+0000 7f941d7fa700 1 -- 172.21.15.96:0/2529689960 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941401d070 con 0x7f9420100e10 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.111+0000 7f941d7fa700 1 -- 172.21.15.96:0/2529689960 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94140056e0 con 0x7f9420100e10 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.111+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9420196c20 con 0x7f9420100e10 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.111+0000 7f941d7fa700 1 -- 172.21.15.96:0/2529689960 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941400fe80 con 0x7f9420100e10 2024-04-22T15:45:07.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.111+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9420197110 con 0x7f9420100e10 2024-04-22T15:45:07.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.112+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94201091b0 con 0x7f9420100e10 2024-04-22T15:45:07.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.112+0000 7f941d7fa700 1 -- 172.21.15.96:0/2529689960 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f941400f6f0 con 0x7f9420100e10 2024-04-22T15:45:07.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.113+0000 7f941d7fa700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940806ecd0 0x7f9408071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:07.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.113+0000 7f941d7fa700 1 -- 172.21.15.96:0/2529689960 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f94140902c0 con 0x7f9420100e10 2024-04-22T15:45:07.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.113+0000 7f941f7fe700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940806ecd0 0x7f9408071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:07.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.113+0000 7f941f7fe700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940806ecd0 0x7f9408071190 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9410005950 tx=0x7f94100058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:07.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.118+0000 7f941d7fa700 1 -- 172.21.15.96:0/2529689960 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9414059020 con 0x7f9420100e10 2024-04-22T15:45:07.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.334+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f942010dde0 con 0x7f940806ecd0 2024-04-22T15:45:07.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.336+0000 7f941d7fa700 1 -- 172.21.15.96:0/2529689960 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f942010dde0 con 0x7f940806ecd0 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940806ecd0 msgr2=0x7f9408071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940806ecd0 0x7f9408071190 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9410005950 tx=0x7f94100058e0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 msgr2=0x7f942019b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 0x7f942019b640 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9414000c00 tx=0x7f941400be20 comp rx=0 tx=0).stop 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 shutdown_connections 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f940806ecd0 0x7f9408071190 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9420100e10 0x7f942019b640 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 --2- 172.21.15.96:0/2529689960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94201017c0 0x7f942019bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 >> 172.21.15.96:0/2529689960 conn(0x7f94200fc6e0 msgr2=0x7f94200fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 shutdown_connections 2024-04-22T15:45:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.338+0000 7f9426353700 1 -- 172.21.15.96:0/2529689960 wait complete. 2024-04-22T15:45:07.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.442+0000 7f48e05a5700 1 -- 172.21.15.96:0/23269928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d8105aa0 msgr2=0x7f48d8105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.442+0000 7f48e05a5700 1 --2- 172.21.15.96:0/23269928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d8105aa0 0x7f48d8105e80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009910 tx=0x7f48c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:07.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.442+0000 7f48e05a5700 1 -- 172.21.15.96:0/23269928 shutdown_connections 2024-04-22T15:45:07.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.442+0000 7f48e05a5700 1 --2- 172.21.15.96:0/23269928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48d80ffaa0 0x7f48d80fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.442+0000 7f48e05a5700 1 --2- 172.21.15.96:0/23269928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d8105aa0 0x7f48d8105e80 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.442+0000 7f48e05a5700 1 -- 172.21.15.96:0/23269928 >> 172.21.15.96:0/23269928 conn(0x7f48d8076740 msgr2=0x7f48d80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:07.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.442+0000 7f48e05a5700 1 -- 172.21.15.96:0/23269928 shutdown_connections 2024-04-22T15:45:07.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.442+0000 7f48e05a5700 1 -- 172.21.15.96:0/23269928 wait complete. 2024-04-22T15:45:07.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48e05a5700 1 Processor -- start 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48e05a5700 1 -- start start 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48e05a5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d80ffaa0 0x7f48d8195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48e05a5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48d8105aa0 0x7f48d8195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48e05a5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8196620 con 0x7f48d80ffaa0 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48e05a5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d819a370 con 0x7f48d8105aa0 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48de341700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d80ffaa0 0x7f48d8195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48de341700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d80ffaa0 0x7f48d8195a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58020/0 (socket says 172.21.15.96:58020) 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48de341700 1 -- 172.21.15.96:0/1954775536 learned_addr learned my addr 172.21.15.96:0/1954775536 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48ddb40700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48d8105aa0 0x7f48d8195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48de341700 1 -- 172.21.15.96:0/1954775536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48d8105aa0 msgr2=0x7f48d8195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48de341700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48d8105aa0 0x7f48d8195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48de341700 1 -- 172.21.15.96:0/1954775536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c80095f0 con 0x7f48d80ffaa0 2024-04-22T15:45:07.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48ddb40700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48d8105aa0 0x7f48d8195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:45:07.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.443+0000 7f48de341700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d80ffaa0 0x7f48d8195a50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f48c8000c00 tx=0x7f48c800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:07.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.444+0000 7f48cf7fe700 1 -- 172.21.15.96:0/1954775536 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48c801d070 con 0x7f48d80ffaa0 2024-04-22T15:45:07.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.444+0000 7f48cf7fe700 1 -- 172.21.15.96:0/1954775536 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48c80056e0 con 0x7f48d80ffaa0 2024-04-22T15:45:07.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.444+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d819a5f0 con 0x7f48d80ffaa0 2024-04-22T15:45:07.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.444+0000 7f48cf7fe700 1 -- 172.21.15.96:0/1954775536 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48c800fe80 con 0x7f48d80ffaa0 2024-04-22T15:45:07.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.444+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d819aae0 con 0x7f48d80ffaa0 2024-04-22T15:45:07.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.445+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d81011e0 con 0x7f48d80ffaa0 2024-04-22T15:45:07.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.445+0000 7f48cf7fe700 1 -- 172.21.15.96:0/1954775536 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f48c800f6f0 con 0x7f48d80ffaa0 2024-04-22T15:45:07.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.445+0000 7f48cf7fe700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48c406ecd0 0x7f48c4071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:07.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.446+0000 7f48cf7fe700 1 -- 172.21.15.96:0/1954775536 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f48c80901e0 con 0x7f48d80ffaa0 2024-04-22T15:45:07.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.446+0000 7f48ddb40700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48c406ecd0 0x7f48c4071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:07.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.446+0000 7f48ddb40700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48c406ecd0 0x7f48c4071190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f48d4005fd0 tx=0x7f48d4005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:07.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.450+0000 7f48cf7fe700 1 -- 172.21.15.96:0/1954775536 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f48c805caf0 con 0x7f48d80ffaa0 2024-04-22T15:45:07.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.662+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f48d810cca0 con 0x7f48c406ecd0 2024-04-22T15:45:07.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.674+0000 7f48cf7fe700 1 -- 172.21.15.96:0/1954775536 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f48d810cca0 con 0x7f48c406ecd0 2024-04-22T15:45:07.677 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:45:07.677 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (23m) 84s ago 35m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:45:07.677 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (35m) 84s ago 35m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:45:07.677 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (34m) 87s ago 34m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:45:07.677 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (21m) 84s ago 35m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:45:07.677 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (20m) 87s ago 34m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:45:07.677 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (23m) 84s ago 34m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (16m) 84s ago 29m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (17m) 84s ago 29m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (16m) 87s ago 29m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (90s) 87s ago 29m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (26m) 84s ago 36m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (25m) 87s ago 34m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (22m) 84s ago 36m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (21m) 87s ago 33m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (24m) 84s ago 35m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (24m) 87s ago 34m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (20m) 84s ago 33m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:45:07.678 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (19m) 84s ago 32m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:45:07.679 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (19m) 84s ago 32m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:45:07.679 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (18m) 87s ago 31m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:45:07.679 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (18m) 87s ago 31m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:45:07.679 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (17m) 87s ago 30m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:45:07.679 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (24m) 84s ago 34m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:45:07.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48c406ecd0 msgr2=0x7f48c4071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48c406ecd0 0x7f48c4071190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f48d4005fd0 tx=0x7f48d4005ee0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d80ffaa0 msgr2=0x7f48d8195a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d80ffaa0 0x7f48d8195a50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f48c8000c00 tx=0x7f48c800be20 comp rx=0 tx=0).stop 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 shutdown_connections 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f48c406ecd0 0x7f48c4071190 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48d80ffaa0 0x7f48d8195a50 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 --2- 172.21.15.96:0/1954775536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48d8105aa0 0x7f48d8195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 >> 172.21.15.96:0/1954775536 conn(0x7f48d8076740 msgr2=0x7f48d80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 shutdown_connections 2024-04-22T15:45:07.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.676+0000 7f48e05a5700 1 -- 172.21.15.96:0/1954775536 wait complete. 2024-04-22T15:45:07.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.778+0000 7f2f35db9700 1 -- 172.21.15.96:0/2718943221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 msgr2=0x7f2f300fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.778+0000 7f2f35db9700 1 --2- 172.21.15.96:0/2718943221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 0x7f2f300fff50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009910 tx=0x7f2f24009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:07.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.778+0000 7f2f35db9700 1 -- 172.21.15.96:0/2718943221 shutdown_connections 2024-04-22T15:45:07.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.778+0000 7f2f35db9700 1 --2- 172.21.15.96:0/2718943221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 0x7f2f300fff50 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.778+0000 7f2f35db9700 1 --2- 172.21.15.96:0/2718943221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f30105b60 0x7f2f30105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.778+0000 7f2f35db9700 1 -- 172.21.15.96:0/2718943221 >> 172.21.15.96:0/2718943221 conn(0x7f2f300fb5e0 msgr2=0x7f2f300fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:07.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.779+0000 7f2f35db9700 1 -- 172.21.15.96:0/2718943221 shutdown_connections 2024-04-22T15:45:07.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.779+0000 7f2f35db9700 1 -- 172.21.15.96:0/2718943221 wait complete. 2024-04-22T15:45:07.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.779+0000 7f2f35db9700 1 Processor -- start 2024-04-22T15:45:07.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.779+0000 7f2f35db9700 1 -- start start 2024-04-22T15:45:07.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.779+0000 7f2f35db9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 0x7f2f30195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:07.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.779+0000 7f2f35db9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f30105b60 0x7f2f30195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:07.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.779+0000 7f2f35db9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f30196560 con 0x7f2f300ffad0 2024-04-22T15:45:07.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.779+0000 7f2f35db9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f301966d0 con 0x7f2f30105b60 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 0x7f2f30195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 0x7f2f30195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58032/0 (socket says 172.21.15.96:58032) 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2f7fe700 1 -- 172.21.15.96:0/524993570 learned_addr learned my addr 172.21.15.96:0/524993570 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2effd700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f30105b60 0x7f2f30195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2f7fe700 1 -- 172.21.15.96:0/524993570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f30105b60 msgr2=0x7f2f30195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2f7fe700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f30105b60 0x7f2f30195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2f7fe700 1 -- 172.21.15.96:0/524993570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f240095f0 con 0x7f2f300ffad0 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2f7fe700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 0x7f2f30195a20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2f20007b50 tx=0x7f2f20007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2cff9700 1 -- 172.21.15.96:0/524993570 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f20011070 con 0x7f2f300ffad0 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2cff9700 1 -- 172.21.15.96:0/524993570 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f2000c2e0 con 0x7f2f300ffad0 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f2cff9700 1 -- 172.21.15.96:0/524993570 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f2000acc0 con 0x7f2f300ffad0 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3019a490 con 0x7f2f300ffad0 2024-04-22T15:45:07.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.780+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3019a9e0 con 0x7f2f300ffad0 2024-04-22T15:45:07.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.781+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f301011d0 con 0x7f2f300ffad0 2024-04-22T15:45:07.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.782+0000 7f2f2cff9700 1 -- 172.21.15.96:0/524993570 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2f2000e710 con 0x7f2f300ffad0 2024-04-22T15:45:07.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.782+0000 7f2f2cff9700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f1806ecc0 0x7f2f18071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:07.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.782+0000 7f2f2cff9700 1 -- 172.21.15.96:0/524993570 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2f2008ff90 con 0x7f2f300ffad0 2024-04-22T15:45:07.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.782+0000 7f2f2effd700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f1806ecc0 0x7f2f18071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:07.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.782+0000 7f2f2effd700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f1806ecc0 0x7f2f18071180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2f30196f80 tx=0x7f2f2401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:07.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:07.787+0000 7f2f2cff9700 1 -- 172.21.15.96:0/524993570 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2f2005c950 con 0x7f2f300ffad0 2024-04-22T15:45:08.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.085+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2f3004ebb0 con 0x7f2f300ffad0 2024-04-22T15:45:08.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.085+0000 7f2f2cff9700 1 -- 172.21.15.96:0/524993570 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2f2005c4e0 con 0x7f2f300ffad0 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:45:08.089 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:45:08.090 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:45:08.090 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:45:08.090 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:45:08.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f1806ecc0 msgr2=0x7f2f18071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f1806ecc0 0x7f2f18071180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2f30196f80 tx=0x7f2f2401a040 comp rx=0 tx=0).stop 2024-04-22T15:45:08.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 msgr2=0x7f2f30195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 0x7f2f30195a20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2f20007b50 tx=0x7f2f20007f10 comp rx=0 tx=0).stop 2024-04-22T15:45:08.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 shutdown_connections 2024-04-22T15:45:08.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2f1806ecc0 0x7f2f18071180 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f300ffad0 0x7f2f30195a20 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 --2- 172.21.15.96:0/524993570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f30105b60 0x7f2f30195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 >> 172.21.15.96:0/524993570 conn(0x7f2f300fb5e0 msgr2=0x7f2f300fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:08.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 shutdown_connections 2024-04-22T15:45:08.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.088+0000 7f2f35db9700 1 -- 172.21.15.96:0/524993570 wait complete. 2024-04-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.192+0000 7fc73e182700 1 -- 172.21.15.96:0/1613710544 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738102690 msgr2=0x7fc73810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.192+0000 7fc73e182700 1 --2- 172.21.15.96:0/1613710544 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738102690 0x7fc73810ab80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc728009910 tx=0x7fc728009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.192+0000 7fc73e182700 1 -- 172.21.15.96:0/1613710544 shutdown_connections 2024-04-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.192+0000 7fc73e182700 1 --2- 172.21.15.96:0/1613710544 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738102690 0x7fc73810ab80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.192+0000 7fc73e182700 1 --2- 172.21.15.96:0/1613710544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc738101ce0 0x7fc7381020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.192+0000 7fc73e182700 1 -- 172.21.15.96:0/1613710544 >> 172.21.15.96:0/1613710544 conn(0x7fc7380fb5e0 msgr2=0x7fc7380fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.192+0000 7fc73e182700 1 -- 172.21.15.96:0/1613710544 shutdown_connections 2024-04-22T15:45:08.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.192+0000 7fc73e182700 1 -- 172.21.15.96:0/1613710544 wait complete. 2024-04-22T15:45:08.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc73e182700 1 Processor -- start 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc73e182700 1 -- start start 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc73e182700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738101ce0 0x7fc73819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc73e182700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc738102690 0x7fc73819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc73e182700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73819d350 con 0x7fc738101ce0 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc73e182700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc738196930 con 0x7fc738102690 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc7377fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738101ce0 0x7fc73819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc7377fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738101ce0 0x7fc73819c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58056/0 (socket says 172.21.15.96:58056) 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc7377fe700 1 -- 172.21.15.96:0/2661602161 learned_addr learned my addr 172.21.15.96:0/2661602161 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.193+0000 7fc736ffd700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc738102690 0x7fc73819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc7377fe700 1 -- 172.21.15.96:0/2661602161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc738102690 msgr2=0x7fc73819cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc7377fe700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc738102690 0x7fc73819cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc7377fe700 1 -- 172.21.15.96:0/2661602161 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7280095f0 con 0x7fc738101ce0 2024-04-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc736ffd700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc738102690 0x7fc73819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc7377fe700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738101ce0 0x7fc73819c7f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00d8d0 tx=0x7fc72c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc734ff9700 1 -- 172.21.15.96:0/2661602161 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc72c009940 con 0x7fc738101ce0 2024-04-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc734ff9700 1 -- 172.21.15.96:0/2661602161 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc72c010460 con 0x7fc738101ce0 2024-04-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc738196c10 con 0x7fc738101ce0 2024-04-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc734ff9700 1 -- 172.21.15.96:0/2661602161 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc72c00f5f0 con 0x7fc738101ce0 2024-04-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.194+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc738197160 con 0x7fc738101ce0 2024-04-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.195+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc738105500 con 0x7fc738101ce0 2024-04-22T15:45:08.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.195+0000 7fc734ff9700 1 -- 172.21.15.96:0/2661602161 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc72c0105d0 con 0x7fc738101ce0 2024-04-22T15:45:08.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.196+0000 7fc734ff9700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc72006ecc0 0x7fc720071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:08.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.196+0000 7fc734ff9700 1 -- 172.21.15.96:0/2661602161 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc72c08edc0 con 0x7fc738101ce0 2024-04-22T15:45:08.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.196+0000 7fc736ffd700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc72006ecc0 0x7fc720071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:08.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.196+0000 7fc736ffd700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc72006ecc0 0x7fc720071180 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc738197af0 tx=0x7fc72801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:08.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.200+0000 7fc734ff9700 1 -- 172.21.15.96:0/2661602161 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc72c05b6d0 con 0x7fc738101ce0 2024-04-22T15:45:08.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.448+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc738197df0 con 0x7fc738101ce0 2024-04-22T15:45:08.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.449+0000 7fc734ff9700 1 -- 172.21.15.96:0/2661602161 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7fc72c016770 con 0x7fc738101ce0 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:45:08.452 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:45:08.453 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:45:08.454 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:45:08.455 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:45:08.455 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:45:08.455 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:45:08.455 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:45:08.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.451+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc72006ecc0 msgr2=0x7fc720071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.451+0000 7fc73e182700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc72006ecc0 0x7fc720071180 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc738197af0 tx=0x7fc72801a040 comp rx=0 tx=0).stop 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.451+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738101ce0 msgr2=0x7fc73819c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.451+0000 7fc73e182700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738101ce0 0x7fc73819c7f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00d8d0 tx=0x7fc72c00dc90 comp rx=0 tx=0).stop 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.451+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 shutdown_connections 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.451+0000 7fc73e182700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc72006ecc0 0x7fc720071180 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.451+0000 7fc73e182700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc738101ce0 0x7fc73819c7f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.451+0000 7fc73e182700 1 --2- 172.21.15.96:0/2661602161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc738102690 0x7fc73819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.451+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 >> 172.21.15.96:0/2661602161 conn(0x7fc7380fb5e0 msgr2=0x7fc738103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.452+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 shutdown_connections 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.452+0000 7fc73e182700 1 -- 172.21.15.96:0/2661602161 wait complete. 2024-04-22T15:45:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:45:08.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.557+0000 7fdb402f3700 1 -- 172.21.15.96:0/4167358380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb38105ab0 msgr2=0x7fdb38105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.557+0000 7fdb402f3700 1 --2- 172.21.15.96:0/4167358380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb38105ab0 0x7fdb38105e90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009910 tx=0x7fdb34009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.557+0000 7fdb402f3700 1 -- 172.21.15.96:0/4167358380 shutdown_connections 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.557+0000 7fdb402f3700 1 --2- 172.21.15.96:0/4167358380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb380ffab0 0x7fdb380fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.557+0000 7fdb402f3700 1 --2- 172.21.15.96:0/4167358380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb38105ab0 0x7fdb38105e90 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.557+0000 7fdb402f3700 1 -- 172.21.15.96:0/4167358380 >> 172.21.15.96:0/4167358380 conn(0x7fdb380fb5e0 msgr2=0x7fdb380fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.557+0000 7fdb402f3700 1 -- 172.21.15.96:0/4167358380 shutdown_connections 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.557+0000 7fdb402f3700 1 -- 172.21.15.96:0/4167358380 wait complete. 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb402f3700 1 Processor -- start 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb402f3700 1 -- start start 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb402f3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb380ffab0 0x7fdb38197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb402f3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb38105ab0 0x7fdb38197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:08.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb402f3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381984a0 con 0x7fdb380ffab0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb402f3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381985e0 con 0x7fdb38105ab0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb3e08f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb380ffab0 0x7fdb38197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb3e08f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb380ffab0 0x7fdb38197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58078/0 (socket says 172.21.15.96:58078) 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb3e08f700 1 -- 172.21.15.96:0/2957577059 learned_addr learned my addr 172.21.15.96:0/2957577059 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb3d88e700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb38105ab0 0x7fdb38197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb3e08f700 1 -- 172.21.15.96:0/2957577059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb38105ab0 msgr2=0x7fdb38197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb3e08f700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb38105ab0 0x7fdb38197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb3e08f700 1 -- 172.21.15.96:0/2957577059 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb340095f0 con 0x7fdb380ffab0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.558+0000 7fdb3e08f700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb380ffab0 0x7fdb38197990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400bd70 tx=0x7fdb3400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.559+0000 7fdb2f7fe700 1 -- 172.21.15.96:0/2957577059 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3401d070 con 0x7fdb380ffab0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.559+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb381a0a00 con 0x7fdb380ffab0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.559+0000 7fdb2f7fe700 1 -- 172.21.15.96:0/2957577059 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb340056e0 con 0x7fdb380ffab0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.559+0000 7fdb2f7fe700 1 -- 172.21.15.96:0/2957577059 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb34017400 con 0x7fdb380ffab0 2024-04-22T15:45:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.559+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb381a0d60 con 0x7fdb380ffab0 2024-04-22T15:45:08.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.560+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb381080b0 con 0x7fdb380ffab0 2024-04-22T15:45:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.560+0000 7fdb2f7fe700 1 -- 172.21.15.96:0/2957577059 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdb34003a60 con 0x7fdb380ffab0 2024-04-22T15:45:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.561+0000 7fdb2f7fe700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb2406ecc0 0x7fdb24071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.561+0000 7fdb2f7fe700 1 -- 172.21.15.96:0/2957577059 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fdb34090310 con 0x7fdb380ffab0 2024-04-22T15:45:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.561+0000 7fdb3d88e700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb2406ecc0 0x7fdb24071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.561+0000 7fdb3d88e700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb2406ecc0 0x7fdb24071180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdb28005950 tx=0x7fdb280058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:08.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.565+0000 7fdb2f7fe700 1 -- 172.21.15.96:0/2957577059 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdb34059070 con 0x7fdb380ffab0 2024-04-22T15:45:08.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:08 smithi096 ceph-mon[112443]: from='client.35292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:08 smithi096 ceph-mon[112443]: from='client.35296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:08 smithi096 ceph-mon[112443]: pgmap v699: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:08 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/524993570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:45:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:08 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2661602161' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:45:08.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.788+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb38069490 con 0x7fdb2406ecc0 2024-04-22T15:45:08.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.789+0000 7fdb2f7fe700 1 -- 172.21.15.96:0/2957577059 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdb38069490 con 0x7fdb2406ecc0 2024-04-22T15:45:08.792 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:45:08.792 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:45:08.792 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:45:08.792 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:45:08.792 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:45:08.792 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:45:08.792 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:45:08.793 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:45:08.793 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:45:08.793 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:45:08.793 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:45:08.793 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:45:08.793 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:45:08.793 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:45:08.793 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:45:08.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb2406ecc0 msgr2=0x7fdb24071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb2406ecc0 0x7fdb24071180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdb28005950 tx=0x7fdb280058e0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb380ffab0 msgr2=0x7fdb38197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb380ffab0 0x7fdb38197990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400bd70 tx=0x7fdb3400be50 comp rx=0 tx=0).stop 2024-04-22T15:45:08.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 shutdown_connections 2024-04-22T15:45:08.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdb2406ecc0 0x7fdb24071180 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb380ffab0 0x7fdb38197990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 --2- 172.21.15.96:0/2957577059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb38105ab0 0x7fdb38197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 >> 172.21.15.96:0/2957577059 conn(0x7fdb380fb5e0 msgr2=0x7fdb380fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:08.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 shutdown_connections 2024-04-22T15:45:08.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.791+0000 7fdb402f3700 1 -- 172.21.15.96:0/2957577059 wait complete. 2024-04-22T15:45:08.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.895+0000 7fb563c2c700 1 -- 172.21.15.96:0/3481485546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 msgr2=0x7fb55c0fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.895+0000 7fb563c2c700 1 --2- 172.21.15.96:0/3481485546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 0x7fb55c0fff30 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb558009910 tx=0x7fb558009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:08.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.895+0000 7fb563c2c700 1 -- 172.21.15.96:0/3481485546 shutdown_connections 2024-04-22T15:45:08.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.895+0000 7fb563c2c700 1 --2- 172.21.15.96:0/3481485546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 0x7fb55c0fff30 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.895+0000 7fb563c2c700 1 --2- 172.21.15.96:0/3481485546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb55c105ab0 0x7fb55c105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.895+0000 7fb563c2c700 1 -- 172.21.15.96:0/3481485546 >> 172.21.15.96:0/3481485546 conn(0x7fb55c0fb5e0 msgr2=0x7fb55c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:08.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.895+0000 7fb563c2c700 1 -- 172.21.15.96:0/3481485546 shutdown_connections 2024-04-22T15:45:08.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.895+0000 7fb563c2c700 1 -- 172.21.15.96:0/3481485546 wait complete. 2024-04-22T15:45:08.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb563c2c700 1 Processor -- start 2024-04-22T15:45:08.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb563c2c700 1 -- start start 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb563c2c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 0x7fb55c197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb563c2c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb55c105ab0 0x7fb55c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb563c2c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c198460 con 0x7fb55c0ffab0 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb563c2c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c1985a0 con 0x7fb55c105ab0 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb5619c8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 0x7fb55c197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb5619c8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 0x7fb55c197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58086/0 (socket says 172.21.15.96:58086) 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb5619c8700 1 -- 172.21.15.96:0/2373281281 learned_addr learned my addr 172.21.15.96:0/2373281281 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb5619c8700 1 -- 172.21.15.96:0/2373281281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb55c105ab0 msgr2=0x7fb55c197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb5611c7700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb55c105ab0 0x7fb55c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:08.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb5619c8700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb55c105ab0 0x7fb55c197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb5619c8700 1 -- 172.21.15.96:0/2373281281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580095f0 con 0x7fb55c0ffab0 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.896+0000 7fb5611c7700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb55c105ab0 0x7fb55c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.897+0000 7fb5619c8700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 0x7fb55c197950 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00d8d0 tx=0x7fb54c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.897+0000 7fb552ffd700 1 -- 172.21.15.96:0/2373281281 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb54c009940 con 0x7fb55c0ffab0 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.897+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55c1a09c0 con 0x7fb55c0ffab0 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.897+0000 7fb552ffd700 1 -- 172.21.15.96:0/2373281281 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb54c010460 con 0x7fb55c0ffab0 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.897+0000 7fb552ffd700 1 -- 172.21.15.96:0/2373281281 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb54c00f5f0 con 0x7fb55c0ffab0 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.897+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c1a0dd0 con 0x7fb55c0ffab0 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.898+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c1080d0 con 0x7fb55c0ffab0 2024-04-22T15:45:08.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.898+0000 7fb552ffd700 1 -- 172.21.15.96:0/2373281281 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb54c0105d0 con 0x7fb55c0ffab0 2024-04-22T15:45:08.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.899+0000 7fb552ffd700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb54806ecc0 0x7fb548071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:08.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.899+0000 7fb552ffd700 1 -- 172.21.15.96:0/2373281281 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb54c08eed0 con 0x7fb55c0ffab0 2024-04-22T15:45:08.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.899+0000 7fb5611c7700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb54806ecc0 0x7fb548071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:08.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.899+0000 7fb5611c7700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb54806ecc0 0x7fb548071180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb55800b5c0 tx=0x7fb558005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:08.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:08.903+0000 7fb552ffd700 1 -- 172.21.15.96:0/2373281281 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb54c057c30 con 0x7fb55c0ffab0 2024-04-22T15:45:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:08 smithi196 ceph-mon[81815]: from='client.35292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:08 smithi196 ceph-mon[81815]: from='client.35296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:08 smithi196 ceph-mon[81815]: pgmap v699: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:08 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/524993570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:45:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:09.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:08 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2661602161' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.183+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb55c1011f0 con 0x7fb55c0ffab0 2024-04-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.184+0000 7fb552ffd700 1 -- 172.21.15.96:0/2373281281 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fb54c05b2f0 con 0x7fb55c0ffab0 2024-04-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:45:09.188 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:45:09.189 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:45:09.189 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:45:09.189 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:45:09.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb54806ecc0 msgr2=0x7fb548071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb54806ecc0 0x7fb548071180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb55800b5c0 tx=0x7fb558005a90 comp rx=0 tx=0).stop 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 msgr2=0x7fb55c197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 0x7fb55c197950 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00d8d0 tx=0x7fb54c00dc90 comp rx=0 tx=0).stop 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 shutdown_connections 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb54806ecc0 0x7fb548071180 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb55c0ffab0 0x7fb55c197950 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 --2- 172.21.15.96:0/2373281281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb55c105ab0 0x7fb55c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 >> 172.21.15.96:0/2373281281 conn(0x7fb55c0fb5e0 msgr2=0x7fb55c0fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 shutdown_connections 2024-04-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:09.186+0000 7fb563c2c700 1 -- 172.21.15.96:0/2373281281 wait complete. 2024-04-22T15:45:09.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:09 smithi096 ceph-mon[112443]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:09.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:09 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2373281281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:45:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:09 smithi196 ceph-mon[81815]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:09 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2373281281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:45:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:10 smithi096 ceph-mon[112443]: from='client.35312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:10.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:10 smithi096 ceph-mon[112443]: pgmap v700: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:10 smithi196 ceph-mon[81815]: from='client.35312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:10 smithi196 ceph-mon[81815]: pgmap v700: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:11.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:11 smithi096 ceph-mon[112443]: pgmap v701: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:11 smithi196 ceph-mon[81815]: pgmap v701: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:14 smithi096 ceph-mon[112443]: pgmap v702: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:14 smithi196 ceph-mon[81815]: pgmap v702: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:16.896 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:16 smithi196 ceph-mon[81815]: pgmap v703: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:16.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:16 smithi096 ceph-mon[112443]: pgmap v703: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:18.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:18 smithi096 ceph-mon[112443]: pgmap v704: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:18 smithi196 ceph-mon[81815]: pgmap v704: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:20.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:20 smithi096 ceph-mon[112443]: pgmap v705: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:20 smithi196 ceph-mon[81815]: pgmap v705: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:22.962 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:22 smithi096 ceph-mon[112443]: pgmap v706: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:22 smithi196 ceph-mon[81815]: pgmap v706: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:24.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:24 smithi096 ceph-mon[112443]: pgmap v707: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:24 smithi196 ceph-mon[81815]: pgmap v707: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:26.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:26 smithi096 ceph-mon[112443]: pgmap v708: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:27.004 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:26 smithi196 ceph-mon[81815]: pgmap v708: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:28 smithi096 ceph-mon[112443]: pgmap v709: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:28 smithi196 ceph-mon[81815]: pgmap v709: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:30 smithi096 ceph-mon[112443]: pgmap v710: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:30 smithi196 ceph-mon[81815]: pgmap v710: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:32 smithi096 ceph-mon[112443]: pgmap v711: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:32 smithi196 ceph-mon[81815]: pgmap v711: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:34 smithi096 ceph-mon[112443]: pgmap v712: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:34 smithi196 ceph-mon[81815]: pgmap v712: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:36.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:36 smithi096 ceph-mon[112443]: pgmap v713: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:36 smithi196 ceph-mon[81815]: pgmap v713: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:38 smithi096 ceph-mon[112443]: pgmap v714: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:38 smithi196 ceph-mon[81815]: pgmap v714: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:39.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.299+0000 7f43800ae700 1 -- 172.21.15.96:0/643123835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 msgr2=0x7f437810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.299+0000 7f43800ae700 1 --2- 172.21.15.96:0/643123835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 0x7f437810ac50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f436c00b0a0 tx=0x7f436c00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.299+0000 7f43800ae700 1 -- 172.21.15.96:0/643123835 shutdown_connections 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.299+0000 7f43800ae700 1 --2- 172.21.15.96:0/643123835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 0x7f437810ac50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.299+0000 7f43800ae700 1 --2- 172.21.15.96:0/643123835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4378101e40 0x7f4378102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.299+0000 7f43800ae700 1 -- 172.21.15.96:0/643123835 >> 172.21.15.96:0/643123835 conn(0x7f43780fb5d0 msgr2=0x7f43780fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.299+0000 7f43800ae700 1 -- 172.21.15.96:0/643123835 shutdown_connections 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.299+0000 7f43800ae700 1 -- 172.21.15.96:0/643123835 wait complete. 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.300+0000 7f43800ae700 1 Processor -- start 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.300+0000 7f43800ae700 1 -- start start 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.300+0000 7f43800ae700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4378101e40 0x7f437819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.300+0000 7f43800ae700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 0x7f437819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.300+0000 7f43800ae700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437819d380 con 0x7f4378102760 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.300+0000 7f43800ae700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4378196960 con 0x7f4378101e40 2024-04-22T15:45:39.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.300+0000 7f437d649700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 0x7f437819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.300+0000 7f437d649700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 0x7f437819cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50348/0 (socket says 172.21.15.96:50348) 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.300+0000 7f437d649700 1 -- 172.21.15.96:0/3771051062 learned_addr learned my addr 172.21.15.96:0/3771051062 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f437d649700 1 -- 172.21.15.96:0/3771051062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4378101e40 msgr2=0x7f437819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f437de4a700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4378101e40 0x7f437819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f437d649700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4378101e40 0x7f437819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f437d649700 1 -- 172.21.15.96:0/3771051062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c009d00 con 0x7f4378102760 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f437de4a700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4378101e40 0x7f437819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f437d649700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 0x7f437819cd60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f436c015040 tx=0x7f436c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f4376ffd700 1 -- 172.21.15.96:0/3771051062 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f436c00e040 con 0x7f4378102760 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f4376ffd700 1 -- 172.21.15.96:0/3771051062 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f436c007cd0 con 0x7f4378102760 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4378196be0 con 0x7f4378102760 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f4376ffd700 1 -- 172.21.15.96:0/3771051062 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f436c01d690 con 0x7f4378102760 2024-04-22T15:45:39.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.301+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43781970d0 con 0x7f4378102760 2024-04-22T15:45:39.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.302+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43781054f0 con 0x7f4378102760 2024-04-22T15:45:39.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.303+0000 7f4376ffd700 1 -- 172.21.15.96:0/3771051062 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f436c004040 con 0x7f4378102760 2024-04-22T15:45:39.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.303+0000 7f4376ffd700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f436006ecc0 0x7f4360071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:39.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.303+0000 7f4376ffd700 1 -- 172.21.15.96:0/3771051062 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f436c090690 con 0x7f4378102760 2024-04-22T15:45:39.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.303+0000 7f437de4a700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f436006ecc0 0x7f4360071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:39.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.304+0000 7f437de4a700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f436006ecc0 0x7f4360071180 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4368005fd0 tx=0x7f4368005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:39.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.308+0000 7f4376ffd700 1 -- 172.21.15.96:0/3771051062 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f436c05cf20 con 0x7f4378102760 2024-04-22T15:45:39.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.529+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4378197e90 con 0x7f436006ecc0 2024-04-22T15:45:39.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.530+0000 7f4376ffd700 1 -- 172.21.15.96:0/3771051062 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4378197e90 con 0x7f436006ecc0 2024-04-22T15:45:39.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.532+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f436006ecc0 msgr2=0x7f4360071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.532+0000 7f43800ae700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f436006ecc0 0x7f4360071180 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4368005fd0 tx=0x7f4368005dc0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.532+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 msgr2=0x7f437819cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.532+0000 7f43800ae700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 0x7f437819cd60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f436c015040 tx=0x7f436c003ce0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.532+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 shutdown_connections 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.532+0000 7f43800ae700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f436006ecc0 0x7f4360071180 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.532+0000 7f43800ae700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4378101e40 0x7f437819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.532+0000 7f43800ae700 1 --2- 172.21.15.96:0/3771051062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4378102760 0x7f437819cd60 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.533+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 >> 172.21.15.96:0/3771051062 conn(0x7f43780fb5d0 msgr2=0x7f43781030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.533+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 shutdown_connections 2024-04-22T15:45:39.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.533+0000 7f43800ae700 1 -- 172.21.15.96:0/3771051062 wait complete. 2024-04-22T15:45:39.546 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:45:39.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.638+0000 7f3a2dc91700 1 -- 172.21.15.96:0/1014420143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100440 msgr2=0x7f3a28100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.638+0000 7f3a2dc91700 1 --2- 172.21.15.96:0/1014420143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100440 0x7f3a28100820 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009910 tx=0x7f3a18009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:39.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.638+0000 7f3a2dc91700 1 -- 172.21.15.96:0/1014420143 shutdown_connections 2024-04-22T15:45:39.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.638+0000 7f3a2dc91700 1 --2- 172.21.15.96:0/1014420143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a28100df0 0x7f3a281047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.638+0000 7f3a2dc91700 1 --2- 172.21.15.96:0/1014420143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100440 0x7f3a28100820 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.638+0000 7f3a2dc91700 1 -- 172.21.15.96:0/1014420143 >> 172.21.15.96:0/1014420143 conn(0x7f3a28075f40 msgr2=0x7f3a280fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:39.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.639+0000 7f3a2dc91700 1 -- 172.21.15.96:0/1014420143 shutdown_connections 2024-04-22T15:45:39.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.639+0000 7f3a2dc91700 1 -- 172.21.15.96:0/1014420143 wait complete. 2024-04-22T15:45:39.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.639+0000 7f3a2dc91700 1 Processor -- start 2024-04-22T15:45:39.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.639+0000 7f3a2dc91700 1 -- start start 2024-04-22T15:45:39.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a2dc91700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a28100440 0x7f3a2819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:39.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a2dc91700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100df0 0x7f3a2819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:39.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a2dc91700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2819d3d0 con 0x7f3a28100df0 2024-04-22T15:45:39.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a2dc91700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a281969b0 con 0x7f3a28100440 2024-04-22T15:45:39.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a26ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100df0 0x7f3a2819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:39.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a26ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100df0 0x7f3a2819cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50370/0 (socket says 172.21.15.96:50370) 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a26ffd700 1 -- 172.21.15.96:0/2608376746 learned_addr learned my addr 172.21.15.96:0/2608376746 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a277fe700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a28100440 0x7f3a2819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a26ffd700 1 -- 172.21.15.96:0/2608376746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a28100440 msgr2=0x7f3a2819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a26ffd700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a28100440 0x7f3a2819c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a26ffd700 1 -- 172.21.15.96:0/2608376746 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a180095f0 con 0x7f3a28100df0 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a26ffd700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100df0 0x7f3a2819cdb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00b4b0 tx=0x7f3a1c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a24ff9700 1 -- 172.21.15.96:0/2608376746 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a1c011800 con 0x7f3a28100df0 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a24ff9700 1 -- 172.21.15.96:0/2608376746 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a1c011e20 con 0x7f3a28100df0 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a28196c90 con 0x7f3a28100df0 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a24ff9700 1 -- 172.21.15.96:0/2608376746 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a1c00f2e0 con 0x7f3a28100df0 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.640+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a28197260 con 0x7f3a28100df0 2024-04-22T15:45:39.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.641+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a281090f0 con 0x7f3a28100df0 2024-04-22T15:45:39.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.642+0000 7f3a24ff9700 1 -- 172.21.15.96:0/2608376746 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3a1c010380 con 0x7f3a28100df0 2024-04-22T15:45:39.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.642+0000 7f3a24ff9700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a1006ecc0 0x7f3a10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:39.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.643+0000 7f3a277fe700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a1006ecc0 0x7f3a10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:39.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.643+0000 7f3a24ff9700 1 -- 172.21.15.96:0/2608376746 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3a1c08ea90 con 0x7f3a28100df0 2024-04-22T15:45:39.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.643+0000 7f3a277fe700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a1006ecc0 0x7f3a10071180 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800b5c0 tx=0x7f3a18005380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:39.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.647+0000 7f3a24ff9700 1 -- 172.21.15.96:0/2608376746 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3a1c05b320 con 0x7f3a28100df0 2024-04-22T15:45:39.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.870+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a28197ee0 con 0x7f3a1006ecc0 2024-04-22T15:45:39.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.871+0000 7f3a24ff9700 1 -- 172.21.15.96:0/2608376746 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3a28197ee0 con 0x7f3a1006ecc0 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a1006ecc0 msgr2=0x7f3a10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a1006ecc0 0x7f3a10071180 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800b5c0 tx=0x7f3a18005380 comp rx=0 tx=0).stop 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100df0 msgr2=0x7f3a2819cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100df0 0x7f3a2819cdb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00b4b0 tx=0x7f3a1c00b870 comp rx=0 tx=0).stop 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 shutdown_connections 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a1006ecc0 0x7f3a10071180 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a28100440 0x7f3a2819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 --2- 172.21.15.96:0/2608376746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a28100df0 0x7f3a2819cdb0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 >> 172.21.15.96:0/2608376746 conn(0x7f3a28075f40 msgr2=0x7f3a280fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 shutdown_connections 2024-04-22T15:45:39.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.873+0000 7f3a2dc91700 1 -- 172.21.15.96:0/2608376746 wait complete. 2024-04-22T15:45:39.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.977+0000 7f3e1c568700 1 -- 172.21.15.96:0/1239449358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e141017a0 msgr2=0x7f3e141057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.977+0000 7f3e1c568700 1 --2- 172.21.15.96:0/1239449358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e141017a0 0x7f3e141057f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3e04009910 tx=0x7f3e04009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:39.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 -- 172.21.15.96:0/1239449358 shutdown_connections 2024-04-22T15:45:39.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 --2- 172.21.15.96:0/1239449358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e141017a0 0x7f3e141057f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 --2- 172.21.15.96:0/1239449358 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e14100df0 0x7f3e141011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 -- 172.21.15.96:0/1239449358 >> 172.21.15.96:0/1239449358 conn(0x7f3e140fc6a0 msgr2=0x7f3e140feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:39.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 -- 172.21.15.96:0/1239449358 shutdown_connections 2024-04-22T15:45:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 -- 172.21.15.96:0/1239449358 wait complete. 2024-04-22T15:45:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 Processor -- start 2024-04-22T15:45:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 -- start start 2024-04-22T15:45:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14100df0 0x7f3e14198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.978+0000 7f3e1c568700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e141017a0 0x7f3e14199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1c568700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e141998c0 con 0x7f3e14100df0 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1c568700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1419c530 con 0x7f3e141017a0 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1a304700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14100df0 0x7f3e14198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1a304700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14100df0 0x7f3e14198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50388/0 (socket says 172.21.15.96:50388) 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1a304700 1 -- 172.21.15.96:0/1515643703 learned_addr learned my addr 172.21.15.96:0/1515643703 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e19b03700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e141017a0 0x7f3e14199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1a304700 1 -- 172.21.15.96:0/1515643703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e141017a0 msgr2=0x7f3e14199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1a304700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e141017a0 0x7f3e14199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1a304700 1 -- 172.21.15.96:0/1515643703 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e040095f0 con 0x7f3e14100df0 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e19b03700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e141017a0 0x7f3e14199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1a304700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14100df0 0x7f3e14198cf0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000d8d0 tx=0x7f3e1000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e0b7fe700 1 -- 172.21.15.96:0/1515643703 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e10009940 con 0x7f3e14100df0 2024-04-22T15:45:39.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e0b7fe700 1 -- 172.21.15.96:0/1515643703 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e10010460 con 0x7f3e14100df0 2024-04-22T15:45:39.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e0b7fe700 1 -- 172.21.15.96:0/1515643703 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e1000f5f0 con 0x7f3e14100df0 2024-04-22T15:45:39.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1419c810 con 0x7f3e14100df0 2024-04-22T15:45:39.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.979+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1419cd60 con 0x7f3e14100df0 2024-04-22T15:45:39.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.980+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e14104f70 con 0x7f3e14100df0 2024-04-22T15:45:39.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.981+0000 7f3e0b7fe700 1 -- 172.21.15.96:0/1515643703 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3e100105d0 con 0x7f3e14100df0 2024-04-22T15:45:39.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.981+0000 7f3e0b7fe700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3e0006ecc0 0x7f3e00071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:39.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.982+0000 7f3e0b7fe700 1 -- 172.21.15.96:0/1515643703 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3e1008ee50 con 0x7f3e14100df0 2024-04-22T15:45:39.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.982+0000 7f3e19b03700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3e0006ecc0 0x7f3e00071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:39.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.982+0000 7f3e19b03700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3e0006ecc0 0x7f3e00071180 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3e1419a210 tx=0x7f3e0400b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:39.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:39.986+0000 7f3e0b7fe700 1 -- 172.21.15.96:0/1515643703 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3e10057cc0 con 0x7f3e14100df0 2024-04-22T15:45:40.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.197+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e1410dd90 con 0x7f3e0006ecc0 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.208+0000 7f3e0b7fe700 1 -- 172.21.15.96:0/1515643703 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3e1410dd90 con 0x7f3e0006ecc0 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (24m) 117s ago 35m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (35m) 117s ago 35m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (34m) 119s ago 34m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (21m) 117s ago 35m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (21m) 119s ago 34m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (23m) 117s ago 35m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (17m) 117s ago 29m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (17m) 117s ago 30m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:45:40.211 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (17m) 119s ago 29m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (2m) 119s ago 29m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (26m) 117s ago 37m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (25m) 119s ago 34m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (22m) 117s ago 37m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (22m) 119s ago 34m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (25m) 117s ago 35m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (25m) 119s ago 34m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (21m) 117s ago 33m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (20m) 117s ago 33m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (19m) 117s ago 32m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (19m) 119s ago 32m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (18m) 119s ago 31m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (18m) 119s ago 31m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:45:40.212 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (24m) 117s ago 35m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:45:40.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3e0006ecc0 msgr2=0x7f3e00071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3e0006ecc0 0x7f3e00071180 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3e1419a210 tx=0x7f3e0400b540 comp rx=0 tx=0).stop 2024-04-22T15:45:40.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14100df0 msgr2=0x7f3e14198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14100df0 0x7f3e14198cf0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000d8d0 tx=0x7f3e1000dc90 comp rx=0 tx=0).stop 2024-04-22T15:45:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 shutdown_connections 2024-04-22T15:45:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3e0006ecc0 0x7f3e00071180 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e14100df0 0x7f3e14198cf0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 --2- 172.21.15.96:0/1515643703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e141017a0 0x7f3e14199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 >> 172.21.15.96:0/1515643703 conn(0x7f3e140fc6a0 msgr2=0x7f3e140fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 shutdown_connections 2024-04-22T15:45:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.210+0000 7f3e1c568700 1 -- 172.21.15.96:0/1515643703 wait complete. 2024-04-22T15:45:40.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.312+0000 7fe482af4700 1 -- 172.21.15.96:0/853681083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 msgr2=0x7fe47c102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.312+0000 7fe482af4700 1 --2- 172.21.15.96:0/853681083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 0x7fe47c102170 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe470009910 tx=0x7fe470009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:40.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.312+0000 7fe482af4700 1 -- 172.21.15.96:0/853681083 shutdown_connections 2024-04-22T15:45:40.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.312+0000 7fe482af4700 1 --2- 172.21.15.96:0/853681083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe47c1026b0 0x7fe47c10aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.312+0000 7fe482af4700 1 --2- 172.21.15.96:0/853681083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 0x7fe47c102170 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.313+0000 7fe482af4700 1 -- 172.21.15.96:0/853681083 >> 172.21.15.96:0/853681083 conn(0x7fe47c0fb5e0 msgr2=0x7fe47c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:40.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.313+0000 7fe482af4700 1 -- 172.21.15.96:0/853681083 shutdown_connections 2024-04-22T15:45:40.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.313+0000 7fe482af4700 1 -- 172.21.15.96:0/853681083 wait complete. 2024-04-22T15:45:40.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.313+0000 7fe482af4700 1 Processor -- start 2024-04-22T15:45:40.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.313+0000 7fe482af4700 1 -- start start 2024-04-22T15:45:40.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe482af4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 0x7fe47c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe482af4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe47c1026b0 0x7fe47c19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe482af4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c19d390 con 0x7fe47c101d90 2024-04-22T15:45:40.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe482af4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c196970 con 0x7fe47c1026b0 2024-04-22T15:45:40.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe480890700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 0x7fe47c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe480890700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 0x7fe47c19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50398/0 (socket says 172.21.15.96:50398) 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe480890700 1 -- 172.21.15.96:0/2659417929 learned_addr learned my addr 172.21.15.96:0/2659417929 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe47bfff700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe47c1026b0 0x7fe47c19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe480890700 1 -- 172.21.15.96:0/2659417929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe47c1026b0 msgr2=0x7fe47c19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe480890700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe47c1026b0 0x7fe47c19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe480890700 1 -- 172.21.15.96:0/2659417929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4700095f0 con 0x7fe47c101d90 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe47bfff700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe47c1026b0 0x7fe47c19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe480890700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 0x7fe47c19c830 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe470006010 tx=0x7fe47000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe479ffb700 1 -- 172.21.15.96:0/2659417929 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47001d070 con 0x7fe47c101d90 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe479ffb700 1 -- 172.21.15.96:0/2659417929 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4700056e0 con 0x7fe47c101d90 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47c196bf0 con 0x7fe47c101d90 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.314+0000 7fe479ffb700 1 -- 172.21.15.96:0/2659417929 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe470017400 con 0x7fe47c101d90 2024-04-22T15:45:40.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.315+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47c1970e0 con 0x7fe47c101d90 2024-04-22T15:45:40.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.316+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47c1054b0 con 0x7fe47c101d90 2024-04-22T15:45:40.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.316+0000 7fe479ffb700 1 -- 172.21.15.96:0/2659417929 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe470003a60 con 0x7fe47c101d90 2024-04-22T15:45:40.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.317+0000 7fe479ffb700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe46c06ecc0 0x7fe46c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.317+0000 7fe47bfff700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe46c06ecc0 0x7fe46c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:40.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.317+0000 7fe479ffb700 1 -- 172.21.15.96:0/2659417929 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe4700901b0 con 0x7fe47c101d90 2024-04-22T15:45:40.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.317+0000 7fe47bfff700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe46c06ecc0 0x7fe46c071180 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe468005d90 tx=0x7fe468005d00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:40.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.321+0000 7fe479ffb700 1 -- 172.21.15.96:0/2659417929 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe47005cac0 con 0x7fe47c101d90 2024-04-22T15:45:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.605+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe47c0699f0 con 0x7fe47c101d90 2024-04-22T15:45:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.605+0000 7fe479ffb700 1 -- 172.21.15.96:0/2659417929 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe47005c650 con 0x7fe47c101d90 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:45:40.609 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:45:40.610 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:45:40.610 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:45:40.610 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:45:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe46c06ecc0 msgr2=0x7fe46c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe46c06ecc0 0x7fe46c071180 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe468005d90 tx=0x7fe468005d00 comp rx=0 tx=0).stop 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 msgr2=0x7fe47c19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 0x7fe47c19c830 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe470006010 tx=0x7fe47000be20 comp rx=0 tx=0).stop 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 shutdown_connections 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe46c06ecc0 0x7fe46c071180 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe47c101d90 0x7fe47c19c830 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 --2- 172.21.15.96:0/2659417929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe47c1026b0 0x7fe47c19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 >> 172.21.15.96:0/2659417929 conn(0x7fe47c0fb5e0 msgr2=0x7fe47c103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 shutdown_connections 2024-04-22T15:45:40.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.608+0000 7fe482af4700 1 -- 172.21.15.96:0/2659417929 wait complete. 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.712+0000 7f03784c7700 1 -- 172.21.15.96:0/913762754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 msgr2=0x7f0370105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.712+0000 7f03784c7700 1 --2- 172.21.15.96:0/913762754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 0x7f0370105e80 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0360009910 tx=0x7f0360009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.712+0000 7f03784c7700 1 -- 172.21.15.96:0/913762754 shutdown_connections 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.712+0000 7f03784c7700 1 --2- 172.21.15.96:0/913762754 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03700ffaa0 0x7f03700fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.712+0000 7f03784c7700 1 --2- 172.21.15.96:0/913762754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 0x7f0370105e80 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.712+0000 7f03784c7700 1 -- 172.21.15.96:0/913762754 >> 172.21.15.96:0/913762754 conn(0x7f0370076740 msgr2=0x7f03700fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.712+0000 7f03784c7700 1 -- 172.21.15.96:0/913762754 shutdown_connections 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.712+0000 7f03784c7700 1 -- 172.21.15.96:0/913762754 wait complete. 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f03784c7700 1 Processor -- start 2024-04-22T15:45:40.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f03784c7700 1 -- start start 2024-04-22T15:45:40.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f03784c7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03700ffaa0 0x7f0370198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f03784c7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 0x7f03701992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f03784c7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03701999d0 con 0x7f0370105aa0 2024-04-22T15:45:40.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f03784c7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037019d6a0 con 0x7f03700ffaa0 2024-04-22T15:45:40.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f0375a62700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 0x7f03701992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:40.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f0375a62700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 0x7f03701992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35934/0 (socket says 172.21.15.96:35934) 2024-04-22T15:45:40.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f0375a62700 1 -- 172.21.15.96:0/1218732150 learned_addr learned my addr 172.21.15.96:0/1218732150 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:40.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f0375a62700 1 -- 172.21.15.96:0/1218732150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03700ffaa0 msgr2=0x7f0370198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f0376263700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03700ffaa0 0x7f0370198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:40.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f0375a62700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03700ffaa0 0x7f0370198db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f0375a62700 1 -- 172.21.15.96:0/1218732150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03600095f0 con 0x7f0370105aa0 2024-04-22T15:45:40.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.713+0000 7f0375a62700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 0x7f03701992f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f036c00b4b0 tx=0x7f036c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:40.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.714+0000 7f03677fe700 1 -- 172.21.15.96:0/1218732150 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f036c011800 con 0x7f0370105aa0 2024-04-22T15:45:40.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.714+0000 7f03677fe700 1 -- 172.21.15.96:0/1218732150 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f036c011e20 con 0x7f0370105aa0 2024-04-22T15:45:40.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.714+0000 7f03677fe700 1 -- 172.21.15.96:0/1218732150 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f036c00f2e0 con 0x7f0370105aa0 2024-04-22T15:45:40.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.714+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037019d980 con 0x7f0370105aa0 2024-04-22T15:45:40.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.714+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037019de80 con 0x7f0370105aa0 2024-04-22T15:45:40.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.715+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03701011e0 con 0x7f0370105aa0 2024-04-22T15:45:40.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.715+0000 7f03677fe700 1 -- 172.21.15.96:0/1218732150 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f036c010380 con 0x7f0370105aa0 2024-04-22T15:45:40.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.716+0000 7f03677fe700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f035c06ecc0 0x7f035c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.716+0000 7f0376263700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f035c06ecc0 0x7f035c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:40.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.716+0000 7f03677fe700 1 -- 172.21.15.96:0/1218732150 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f036c08ea50 con 0x7f0370105aa0 2024-04-22T15:45:40.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.716+0000 7f0376263700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f035c06ecc0 0x7f035c071180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f036000b5c0 tx=0x7f0360005380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:40.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.720+0000 7f03677fe700 1 -- 172.21.15.96:0/1218732150 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f036c0577b0 con 0x7f0370105aa0 2024-04-22T15:45:40.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.963+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f037010cdc0 con 0x7f0370105aa0 2024-04-22T15:45:40.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:40 smithi096 ceph-mon[112443]: pgmap v715: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:40.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:40 smithi096 ceph-mon[112443]: from='client.35320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:40.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.963+0000 7f03677fe700 1 -- 172.21.15.96:0/1218732150 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f036c05ae70 con 0x7f0370105aa0 2024-04-22T15:45:40.967 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:45:40.967 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:45:40.967 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:45:40.967 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:45:40.968 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:45:40.969 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:45:40.970 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:45:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f035c06ecc0 msgr2=0x7f035c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f035c06ecc0 0x7f035c071180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f036000b5c0 tx=0x7f0360005380 comp rx=0 tx=0).stop 2024-04-22T15:45:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 msgr2=0x7f03701992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 0x7f03701992f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f036c00b4b0 tx=0x7f036c00b870 comp rx=0 tx=0).stop 2024-04-22T15:45:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 shutdown_connections 2024-04-22T15:45:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f035c06ecc0 0x7f035c071180 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03700ffaa0 0x7f0370198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 --2- 172.21.15.96:0/1218732150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0370105aa0 0x7f03701992f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 >> 172.21.15.96:0/1218732150 conn(0x7f0370076740 msgr2=0x7f03700685e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:40.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 shutdown_connections 2024-04-22T15:45:40.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:40.966+0000 7f03784c7700 1 -- 172.21.15.96:0/1218732150 wait complete. 2024-04-22T15:45:40.972 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:45:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:40 smithi196 ceph-mon[81815]: pgmap v715: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:40 smithi196 ceph-mon[81815]: from='client.35320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:41.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.070+0000 7fe5b2294700 1 -- 172.21.15.96:0/1312446121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac1017d0 msgr2=0x7fe5ac105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.070+0000 7fe5b2294700 1 --2- 172.21.15.96:0/1312446121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac1017d0 0x7fe5ac105820 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009910 tx=0x7fe5a0009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:41.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.070+0000 7fe5b2294700 1 -- 172.21.15.96:0/1312446121 shutdown_connections 2024-04-22T15:45:41.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.070+0000 7fe5b2294700 1 --2- 172.21.15.96:0/1312446121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac1017d0 0x7fe5ac105820 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.070+0000 7fe5b2294700 1 --2- 172.21.15.96:0/1312446121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5ac100e20 0x7fe5ac101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.070+0000 7fe5b2294700 1 -- 172.21.15.96:0/1312446121 >> 172.21.15.96:0/1312446121 conn(0x7fe5ac0fc6d0 msgr2=0x7fe5ac0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:41.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.071+0000 7fe5b2294700 1 -- 172.21.15.96:0/1312446121 shutdown_connections 2024-04-22T15:45:41.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.071+0000 7fe5b2294700 1 -- 172.21.15.96:0/1312446121 wait complete. 2024-04-22T15:45:41.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.071+0000 7fe5b2294700 1 Processor -- start 2024-04-22T15:45:41.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.071+0000 7fe5b2294700 1 -- start start 2024-04-22T15:45:41.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.071+0000 7fe5b2294700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac100e20 0x7fe5ac0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.071+0000 7fe5b2294700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5ac1017d0 0x7fe5ac073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.071+0000 7fe5b2294700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac073c60 con 0x7fe5ac100e20 2024-04-22T15:45:41.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5b2294700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac073da0 con 0x7fe5ac1017d0 2024-04-22T15:45:41.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5abfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac100e20 0x7fe5ac0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:41.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5abfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac100e20 0x7fe5ac0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35956/0 (socket says 172.21.15.96:35956) 2024-04-22T15:45:41.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5abfff700 1 -- 172.21.15.96:0/1214989011 learned_addr learned my addr 172.21.15.96:0/1214989011 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5abfff700 1 -- 172.21.15.96:0/1214989011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5ac1017d0 msgr2=0x7fe5ac073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5abfff700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5ac1017d0 0x7fe5ac073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5abfff700 1 -- 172.21.15.96:0/1214989011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a00095f0 con 0x7fe5ac100e20 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5abfff700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac100e20 0x7fe5ac0731e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00b4b0 tx=0x7fe59c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5a97fa700 1 -- 172.21.15.96:0/1214989011 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe59c011800 con 0x7fe5ac100e20 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5a97fa700 1 -- 172.21.15.96:0/1214989011 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe59c011e20 con 0x7fe5ac100e20 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ac1a2e20 con 0x7fe5ac100e20 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5a97fa700 1 -- 172.21.15.96:0/1214989011 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe59c00f2e0 con 0x7fe5ac100e20 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.072+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ac1a3180 con 0x7fe5ac100e20 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.073+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ac109150 con 0x7fe5ac100e20 2024-04-22T15:45:41.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.074+0000 7fe5a97fa700 1 -- 172.21.15.96:0/1214989011 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe59c010380 con 0x7fe5ac100e20 2024-04-22T15:45:41.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.074+0000 7fe5a97fa700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe59406ecd0 0x7fe594071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.074+0000 7fe5a97fa700 1 -- 172.21.15.96:0/1214989011 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe59c08e980 con 0x7fe5ac100e20 2024-04-22T15:45:41.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.074+0000 7fe5ab7fe700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe59406ecd0 0x7fe594071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:41.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.075+0000 7fe5ab7fe700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe59406ecd0 0x7fe594071190 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0006010 tx=0x7fe5a000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:41.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.079+0000 7fe5a97fa700 1 -- 172.21.15.96:0/1214989011 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe59c057760 con 0x7fe5ac100e20 2024-04-22T15:45:41.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.294+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5ac10de40 con 0x7fe59406ecd0 2024-04-22T15:45:41.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.296+0000 7fe5a97fa700 1 -- 172.21.15.96:0/1214989011 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe5ac10de40 con 0x7fe59406ecd0 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:45:41.299 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:45:41.300 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:45:41.300 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:45:41.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe59406ecd0 msgr2=0x7fe594071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe59406ecd0 0x7fe594071190 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0006010 tx=0x7fe5a000b540 comp rx=0 tx=0).stop 2024-04-22T15:45:41.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac100e20 msgr2=0x7fe5ac0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac100e20 0x7fe5ac0731e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00b4b0 tx=0x7fe59c00b870 comp rx=0 tx=0).stop 2024-04-22T15:45:41.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 shutdown_connections 2024-04-22T15:45:41.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe59406ecd0 0x7fe594071190 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5ac100e20 0x7fe5ac0731e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 --2- 172.21.15.96:0/1214989011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5ac1017d0 0x7fe5ac073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 >> 172.21.15.96:0/1214989011 conn(0x7fe5ac0fc6d0 msgr2=0x7fe5ac0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:41.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 shutdown_connections 2024-04-22T15:45:41.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.298+0000 7fe5b2294700 1 -- 172.21.15.96:0/1214989011 wait complete. 2024-04-22T15:45:41.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.400+0000 7f8e245c4700 1 -- 172.21.15.96:0/1032519255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c105cb0 msgr2=0x7f8e1c106090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.400+0000 7f8e245c4700 1 --2- 172.21.15.96:0/1032519255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c105cb0 0x7f8e1c106090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009910 tx=0x7f8e0c009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:41.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.400+0000 7f8e245c4700 1 -- 172.21.15.96:0/1032519255 shutdown_connections 2024-04-22T15:45:41.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 --2- 172.21.15.96:0/1032519255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e1c0ff440 0x7f8e1c0ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 --2- 172.21.15.96:0/1032519255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c105cb0 0x7f8e1c106090 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 -- 172.21.15.96:0/1032519255 >> 172.21.15.96:0/1032519255 conn(0x7f8e1c074d50 msgr2=0x7f8e1c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 -- 172.21.15.96:0/1032519255 shutdown_connections 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 -- 172.21.15.96:0/1032519255 wait complete. 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 Processor -- start 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 -- start start 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c0ff440 0x7f8e1c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e1c105cb0 0x7f8e1c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c19d360 con 0x7f8e1c0ff440 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e245c4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c196990 con 0x7f8e1c105cb0 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e22360700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c0ff440 0x7f8e1c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e22360700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c0ff440 0x7f8e1c19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35972/0 (socket says 172.21.15.96:35972) 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.401+0000 7f8e22360700 1 -- 172.21.15.96:0/1320908549 learned_addr learned my addr 172.21.15.96:0/1320908549 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:45:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e21b5f700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e1c105cb0 0x7f8e1c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e22360700 1 -- 172.21.15.96:0/1320908549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e1c105cb0 msgr2=0x7f8e1c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e22360700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e1c105cb0 0x7f8e1c19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e22360700 1 -- 172.21.15.96:0/1320908549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c0095f0 con 0x7f8e1c0ff440 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e21b5f700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e1c105cb0 0x7f8e1c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e22360700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c0ff440 0x7f8e1c19c800 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00bd70 tx=0x7f8e0c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e137fe700 1 -- 172.21.15.96:0/1320908549 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e0c01d070 con 0x7f8e1c0ff440 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e137fe700 1 -- 172.21.15.96:0/1320908549 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e0c0056e0 con 0x7f8e1c0ff440 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c196c10 con 0x7f8e1c0ff440 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e137fe700 1 -- 172.21.15.96:0/1320908549 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e0c017400 con 0x7f8e1c0ff440 2024-04-22T15:45:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.402+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c197100 con 0x7f8e1c0ff440 2024-04-22T15:45:41.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.403+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1c109290 con 0x7f8e1c0ff440 2024-04-22T15:45:41.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.404+0000 7f8e137fe700 1 -- 172.21.15.96:0/1320908549 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8e0c003a60 con 0x7f8e1c0ff440 2024-04-22T15:45:41.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.404+0000 7f8e137fe700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8e0806ecc0 0x7f8e08071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.404+0000 7f8e137fe700 1 -- 172.21.15.96:0/1320908549 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8e0c090230 con 0x7f8e1c0ff440 2024-04-22T15:45:41.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.404+0000 7f8e21b5f700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8e0806ecc0 0x7f8e08071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:41.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.405+0000 7f8e21b5f700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8e0806ecc0 0x7f8e08071180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8e18005950 tx=0x7f8e180058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:41.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.409+0000 7f8e137fe700 1 -- 172.21.15.96:0/1320908549 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8e0c05cb40 con 0x7f8e1c0ff440 2024-04-22T15:45:41.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.689+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8e1c197e00 con 0x7f8e1c0ff440 2024-04-22T15:45:41.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.689+0000 7f8e137fe700 1 -- 172.21.15.96:0/1320908549 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f8e0c05c6d0 con 0x7f8e1c0ff440 2024-04-22T15:45:41.692 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:45:41.692 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:45:41.692 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:45:41.693 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:45:41.694 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:45:41.694 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:45:41.694 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:45:41.694 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:45:41.694 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:45:41.694 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:45:41.694 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:45:41.694 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8e0806ecc0 msgr2=0x7f8e08071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8e0806ecc0 0x7f8e08071180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8e18005950 tx=0x7f8e180058e0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c0ff440 msgr2=0x7f8e1c19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c0ff440 0x7f8e1c19c800 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00bd70 tx=0x7f8e0c00be50 comp rx=0 tx=0).stop 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 shutdown_connections 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8e0806ecc0 0x7f8e08071180 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e1c0ff440 0x7f8e1c19c800 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 --2- 172.21.15.96:0/1320908549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e1c105cb0 0x7f8e1c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 >> 172.21.15.96:0/1320908549 conn(0x7f8e1c074d50 msgr2=0x7f8e1c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 shutdown_connections 2024-04-22T15:45:41.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:45:41.691+0000 7f8e245c4700 1 -- 172.21.15.96:0/1320908549 wait complete. 2024-04-22T15:45:41.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:41 smithi096 ceph-mon[112443]: from='client.35324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:41 smithi096 ceph-mon[112443]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:41 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2659417929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:45:41.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:41 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1218732150' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:45:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:41 smithi196 ceph-mon[81815]: from='client.35324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:41 smithi196 ceph-mon[81815]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:41 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2659417929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:45:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:41 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1218732150' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:45:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:42 smithi096 ceph-mon[112443]: from='client.35340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:42 smithi096 ceph-mon[112443]: pgmap v716: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:42.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:42 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1320908549' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:45:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:42 smithi196 ceph-mon[81815]: from='client.35340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:45:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:42 smithi196 ceph-mon[81815]: pgmap v716: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:42 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1320908549' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:45:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:44 smithi096 ceph-mon[112443]: pgmap v717: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:44 smithi196 ceph-mon[81815]: pgmap v717: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:46 smithi096 ceph-mon[112443]: pgmap v718: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:46 smithi196 ceph-mon[81815]: pgmap v718: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:48 smithi096 ceph-mon[112443]: pgmap v719: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:48 smithi196 ceph-mon[81815]: pgmap v719: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:50.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:50 smithi096 ceph-mon[112443]: pgmap v720: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:50 smithi196 ceph-mon[81815]: pgmap v720: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:52.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:52 smithi096 ceph-mon[112443]: pgmap v721: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:52 smithi196 ceph-mon[81815]: pgmap v721: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:54 smithi096 ceph-mon[112443]: pgmap v722: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:54 smithi196 ceph-mon[81815]: pgmap v722: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:55.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:55 smithi096 ceph-mon[112443]: pgmap v723: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:55 smithi196 ceph-mon[81815]: pgmap v723: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:58.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:45:58 smithi096 ceph-mon[112443]: pgmap v724: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:45:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:45:58 smithi196 ceph-mon[81815]: pgmap v724: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:00 smithi096 ceph-mon[112443]: pgmap v725: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:00 smithi196 ceph-mon[81815]: pgmap v725: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:02.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:02 smithi196 ceph-mon[81815]: pgmap v726: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:02.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:02 smithi096 ceph-mon[112443]: pgmap v726: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:04.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:04 smithi096 ceph-mon[112443]: pgmap v727: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:04 smithi196 ceph-mon[81815]: pgmap v727: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:06.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:06 smithi096 ceph-mon[112443]: pgmap v728: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:06 smithi196 ceph-mon[81815]: pgmap v728: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:08 smithi096 ceph-mon[112443]: pgmap v729: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:08.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:08 smithi196 ceph-mon[81815]: pgmap v729: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:10.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:10 smithi096 ceph-mon[112443]: pgmap v730: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:10 smithi196 ceph-mon[81815]: pgmap v730: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/2708774365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 msgr2=0x7f1d441057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 --2- 172.21.15.96:0/2708774365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 0x7f1d441057f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009910 tx=0x7f1d30009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/2708774365 shutdown_connections 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 --2- 172.21.15.96:0/2708774365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 0x7f1d441057f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 --2- 172.21.15.96:0/2708774365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d44100df0 0x7f1d441011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/2708774365 >> 172.21.15.96:0/2708774365 conn(0x7f1d440fc6a0 msgr2=0x7f1d440feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/2708774365 shutdown_connections 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/2708774365 wait complete. 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 Processor -- start 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 -- start start 2024-04-22T15:46:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d44100df0 0x7f1d4419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 0x7f1d4419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.808+0000 7f1d4ae9e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4419d450 con 0x7f1d441017a0 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d4ae9e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d44196a30 con 0x7f1d44100df0 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d43fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 0x7f1d4419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d43fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 0x7f1d4419ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34296/0 (socket says 172.21.15.96:34296) 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d43fff700 1 -- 172.21.15.96:0/613835884 learned_addr learned my addr 172.21.15.96:0/613835884 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d48c3a700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d44100df0 0x7f1d4419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d43fff700 1 -- 172.21.15.96:0/613835884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d44100df0 msgr2=0x7f1d4419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d43fff700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d44100df0 0x7f1d4419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d43fff700 1 -- 172.21.15.96:0/613835884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d300095f0 con 0x7f1d441017a0 2024-04-22T15:46:11.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d43fff700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 0x7f1d4419ce30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1d300092e0 tx=0x7f1d3000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:11.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d41ffb700 1 -- 172.21.15.96:0/613835884 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d3001d070 con 0x7f1d441017a0 2024-04-22T15:46:11.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d41ffb700 1 -- 172.21.15.96:0/613835884 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d3000f460 con 0x7f1d441017a0 2024-04-22T15:46:11.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d44196cb0 con 0x7f1d441017a0 2024-04-22T15:46:11.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.809+0000 7f1d41ffb700 1 -- 172.21.15.96:0/613835884 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d30017610 con 0x7f1d441017a0 2024-04-22T15:46:11.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.810+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d441971a0 con 0x7f1d441017a0 2024-04-22T15:46:11.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.811+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d44109140 con 0x7f1d441017a0 2024-04-22T15:46:11.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.811+0000 7f1d41ffb700 1 -- 172.21.15.96:0/613835884 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1d3000f5d0 con 0x7f1d441017a0 2024-04-22T15:46:11.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.812+0000 7f1d41ffb700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d3406ecd0 0x7f1d34071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:11.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.812+0000 7f1d41ffb700 1 -- 172.21.15.96:0/613835884 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1d30090310 con 0x7f1d441017a0 2024-04-22T15:46:11.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.812+0000 7f1d48c3a700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d3406ecd0 0x7f1d34071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:11.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.812+0000 7f1d48c3a700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d3406ecd0 0x7f1d34071190 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009a10 tx=0x7f1d38009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:11.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:11.817+0000 7f1d41ffb700 1 -- 172.21.15.96:0/613835884 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1d3005cba0 con 0x7f1d441017a0 2024-04-22T15:46:12.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.037+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d4410de10 con 0x7f1d3406ecd0 2024-04-22T15:46:12.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.038+0000 7f1d41ffb700 1 -- 172.21.15.96:0/613835884 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1d4410de10 con 0x7f1d3406ecd0 2024-04-22T15:46:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d3406ecd0 msgr2=0x7f1d34071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d3406ecd0 0x7f1d34071190 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009a10 tx=0x7f1d38009190 comp rx=0 tx=0).stop 2024-04-22T15:46:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 msgr2=0x7f1d4419ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 0x7f1d4419ce30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1d300092e0 tx=0x7f1d3000bd70 comp rx=0 tx=0).stop 2024-04-22T15:46:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 shutdown_connections 2024-04-22T15:46:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d3406ecd0 0x7f1d34071190 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d44100df0 0x7f1d4419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 --2- 172.21.15.96:0/613835884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d441017a0 0x7f1d4419ce30 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 >> 172.21.15.96:0/613835884 conn(0x7f1d440fc6a0 msgr2=0x7f1d440fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 shutdown_connections 2024-04-22T15:46:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.040+0000 7f1d4ae9e700 1 -- 172.21.15.96:0/613835884 wait complete. 2024-04-22T15:46:12.053 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:46:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.145+0000 7f13cae16700 1 -- 172.21.15.96:0/1147001585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 msgr2=0x7f13c4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.145+0000 7f13cae16700 1 --2- 172.21.15.96:0/1147001585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 0x7f13c4101200 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009910 tx=0x7f13c0009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.145+0000 7f13cae16700 1 -- 172.21.15.96:0/1147001585 shutdown_connections 2024-04-22T15:46:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.145+0000 7f13cae16700 1 --2- 172.21.15.96:0/1147001585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13c41017d0 0x7f13c4105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.145+0000 7f13cae16700 1 --2- 172.21.15.96:0/1147001585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 0x7f13c4101200 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.145+0000 7f13cae16700 1 -- 172.21.15.96:0/1147001585 >> 172.21.15.96:0/1147001585 conn(0x7f13c40fc6d0 msgr2=0x7f13c40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.145+0000 7f13cae16700 1 -- 172.21.15.96:0/1147001585 shutdown_connections 2024-04-22T15:46:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.145+0000 7f13cae16700 1 -- 172.21.15.96:0/1147001585 wait complete. 2024-04-22T15:46:12.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13cae16700 1 Processor -- start 2024-04-22T15:46:12.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13cae16700 1 -- start start 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13cae16700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 0x7f13c419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13cae16700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13c41017d0 0x7f13c419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13cae16700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c419d3a0 con 0x7f13c4100e20 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13cae16700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c41969d0 con 0x7f13c41017d0 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13c8bb2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 0x7f13c419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13c8bb2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 0x7f13c419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34310/0 (socket says 172.21.15.96:34310) 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13c8bb2700 1 -- 172.21.15.96:0/3609765717 learned_addr learned my addr 172.21.15.96:0/3609765717 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13c8bb2700 1 -- 172.21.15.96:0/3609765717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13c41017d0 msgr2=0x7f13c419cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13bbfff700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13c41017d0 0x7f13c419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13c8bb2700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13c41017d0 0x7f13c419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.146+0000 7f13c8bb2700 1 -- 172.21.15.96:0/3609765717 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c00095f0 con 0x7f13c4100e20 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.147+0000 7f13c8bb2700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 0x7f13c419c840 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f13c000bd70 tx=0x7f13c000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.147+0000 7f13b9ffb700 1 -- 172.21.15.96:0/3609765717 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13c001d070 con 0x7f13c4100e20 2024-04-22T15:46:12.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.147+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c4196c50 con 0x7f13c4100e20 2024-04-22T15:46:12.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.147+0000 7f13b9ffb700 1 -- 172.21.15.96:0/3609765717 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13c00056e0 con 0x7f13c4100e20 2024-04-22T15:46:12.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.147+0000 7f13b9ffb700 1 -- 172.21.15.96:0/3609765717 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13c0017400 con 0x7f13c4100e20 2024-04-22T15:46:12.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.147+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c4197140 con 0x7f13c4100e20 2024-04-22T15:46:12.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.148+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c41091c0 con 0x7f13c4100e20 2024-04-22T15:46:12.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.149+0000 7f13b9ffb700 1 -- 172.21.15.96:0/3609765717 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f13c0017640 con 0x7f13c4100e20 2024-04-22T15:46:12.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.149+0000 7f13b9ffb700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f13ac06ecc0 0x7f13ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.149+0000 7f13b9ffb700 1 -- 172.21.15.96:0/3609765717 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f13c0090330 con 0x7f13c4100e20 2024-04-22T15:46:12.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.149+0000 7f13bbfff700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f13ac06ecc0 0x7f13ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.150+0000 7f13bbfff700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f13ac06ecc0 0x7f13ac071180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f13b4005950 tx=0x7f13b40058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:12.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.154+0000 7f13b9ffb700 1 -- 172.21.15.96:0/3609765717 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f13c005cc30 con 0x7f13c4100e20 2024-04-22T15:46:12.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.372+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13c4197eb0 con 0x7f13ac06ecc0 2024-04-22T15:46:12.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.374+0000 7f13b9ffb700 1 -- 172.21.15.96:0/3609765717 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f13c4197eb0 con 0x7f13ac06ecc0 2024-04-22T15:46:12.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f13ac06ecc0 msgr2=0x7f13ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f13ac06ecc0 0x7f13ac071180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f13b4005950 tx=0x7f13b40058e0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 msgr2=0x7f13c419c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 0x7f13c419c840 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f13c000bd70 tx=0x7f13c000be50 comp rx=0 tx=0).stop 2024-04-22T15:46:12.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 shutdown_connections 2024-04-22T15:46:12.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f13ac06ecc0 0x7f13ac071180 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13c4100e20 0x7f13c419c840 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 --2- 172.21.15.96:0/3609765717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13c41017d0 0x7f13c419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 >> 172.21.15.96:0/3609765717 conn(0x7f13c40fc6d0 msgr2=0x7f13c40fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:12.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 shutdown_connections 2024-04-22T15:46:12.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.376+0000 7f13cae16700 1 -- 172.21.15.96:0/3609765717 wait complete. 2024-04-22T15:46:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.480+0000 7fd22ab16700 1 -- 172.21.15.96:0/1819922975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 msgr2=0x7fd224105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.480+0000 7fd22ab16700 1 --2- 172.21.15.96:0/1819922975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 0x7fd224105e80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd218009910 tx=0x7fd218009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.480+0000 7fd22ab16700 1 -- 172.21.15.96:0/1819922975 shutdown_connections 2024-04-22T15:46:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.480+0000 7fd22ab16700 1 --2- 172.21.15.96:0/1819922975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2240ffaa0 0x7fd2240fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.480+0000 7fd22ab16700 1 --2- 172.21.15.96:0/1819922975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 0x7fd224105e80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.480+0000 7fd22ab16700 1 -- 172.21.15.96:0/1819922975 >> 172.21.15.96:0/1819922975 conn(0x7fd224076740 msgr2=0x7fd2240fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.480+0000 7fd22ab16700 1 -- 172.21.15.96:0/1819922975 shutdown_connections 2024-04-22T15:46:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.480+0000 7fd22ab16700 1 -- 172.21.15.96:0/1819922975 wait complete. 2024-04-22T15:46:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd22ab16700 1 Processor -- start 2024-04-22T15:46:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd22ab16700 1 -- start start 2024-04-22T15:46:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd22ab16700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2240ffaa0 0x7fd224197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd22ab16700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 0x7fd224197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd22ab16700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2241985b0 con 0x7fd224105aa0 2024-04-22T15:46:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd22ab16700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2241986f0 con 0x7fd2240ffaa0 2024-04-22T15:46:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd223fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 0x7fd224197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd223fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 0x7fd224197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34330/0 (socket says 172.21.15.96:34330) 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd223fff700 1 -- 172.21.15.96:0/3496429442 learned_addr learned my addr 172.21.15.96:0/3496429442 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd2288b2700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2240ffaa0 0x7fd224197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd223fff700 1 -- 172.21.15.96:0/3496429442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2240ffaa0 msgr2=0x7fd224197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd223fff700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2240ffaa0 0x7fd224197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.481+0000 7fd223fff700 1 -- 172.21.15.96:0/3496429442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180095f0 con 0x7fd224105aa0 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.482+0000 7fd223fff700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 0x7fd224197fe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd21000b4b0 tx=0x7fd21000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.482+0000 7fd221ffb700 1 -- 172.21.15.96:0/3496429442 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd210011800 con 0x7fd224105aa0 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.482+0000 7fd221ffb700 1 -- 172.21.15.96:0/3496429442 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd210011e20 con 0x7fd224105aa0 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.482+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2241a0b10 con 0x7fd224105aa0 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.482+0000 7fd221ffb700 1 -- 172.21.15.96:0/3496429442 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd21000f2e0 con 0x7fd224105aa0 2024-04-22T15:46:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.482+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2241a0f20 con 0x7fd224105aa0 2024-04-22T15:46:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.483+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2241080a0 con 0x7fd224105aa0 2024-04-22T15:46:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.483+0000 7fd221ffb700 1 -- 172.21.15.96:0/3496429442 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd210010380 con 0x7fd224105aa0 2024-04-22T15:46:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.484+0000 7fd221ffb700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd21406ecd0 0x7fd214071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.484+0000 7fd2288b2700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd21406ecd0 0x7fd214071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.484+0000 7fd221ffb700 1 -- 172.21.15.96:0/3496429442 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd21008ea50 con 0x7fd224105aa0 2024-04-22T15:46:12.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.484+0000 7fd2288b2700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd21406ecd0 0x7fd214071190 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd21800b5c0 tx=0x7fd21801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:12.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.489+0000 7fd221ffb700 1 -- 172.21.15.96:0/3496429442 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd210057790 con 0x7fd224105aa0 2024-04-22T15:46:12.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.700+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd224068b80 con 0x7fd21406ecd0 2024-04-22T15:46:12.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.711+0000 7fd221ffb700 1 -- 172.21.15.96:0/3496429442 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd224068b80 con 0x7fd21406ecd0 2024-04-22T15:46:12.714 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:46:12.714 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (24m) 2m ago 36m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:46:12.714 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (36m) 2m ago 36m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:46:12.714 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (35m) 2m ago 35m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:46:12.714 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (22m) 2m ago 36m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:46:12.714 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (21m) 2m ago 35m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:46:12.714 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (24m) 2m ago 35m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:46:12.714 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (18m) 2m ago 30m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (18m) 2m ago 30m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (17m) 2m ago 30m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (2m) 2m ago 30m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (27m) 2m ago 37m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (26m) 2m ago 35m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (23m) 2m ago 37m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (22m) 2m ago 35m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (25m) 2m ago 36m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (25m) 2m ago 35m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (21m) 2m ago 34m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (21m) 2m ago 33m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (20m) 2m ago 33m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (19m) 2m ago 32m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:46:12.715 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (19m) 2m ago 32m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:46:12.716 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (18m) 2m ago 32m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:46:12.716 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (25m) 2m ago 35m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:46:12.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd21406ecd0 msgr2=0x7fd214071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd21406ecd0 0x7fd214071190 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd21800b5c0 tx=0x7fd21801a040 comp rx=0 tx=0).stop 2024-04-22T15:46:12.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 msgr2=0x7fd224197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 0x7fd224197fe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd21000b4b0 tx=0x7fd21000b870 comp rx=0 tx=0).stop 2024-04-22T15:46:12.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 shutdown_connections 2024-04-22T15:46:12.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd21406ecd0 0x7fd214071190 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2240ffaa0 0x7fd224197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 --2- 172.21.15.96:0/3496429442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd224105aa0 0x7fd224197fe0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 >> 172.21.15.96:0/3496429442 conn(0x7fd224076740 msgr2=0x7fd2240fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:12.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 shutdown_connections 2024-04-22T15:46:12.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.713+0000 7fd22ab16700 1 -- 172.21.15.96:0/3496429442 wait complete. 2024-04-22T15:46:12.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:12 smithi096 ceph-mon[112443]: pgmap v731: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:12.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:12 smithi196 ceph-mon[81815]: pgmap v731: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.816+0000 7fbf9b751700 1 -- 172.21.15.96:0/528189197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf941017d0 msgr2=0x7fbf94105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.816+0000 7fbf9b751700 1 --2- 172.21.15.96:0/528189197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf941017d0 0x7fbf94105820 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009910 tx=0x7fbf84009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.816+0000 7fbf9b751700 1 -- 172.21.15.96:0/528189197 shutdown_connections 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.816+0000 7fbf9b751700 1 --2- 172.21.15.96:0/528189197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf941017d0 0x7fbf94105820 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.816+0000 7fbf9b751700 1 --2- 172.21.15.96:0/528189197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf94100e20 0x7fbf94101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.816+0000 7fbf9b751700 1 -- 172.21.15.96:0/528189197 >> 172.21.15.96:0/528189197 conn(0x7fbf940fc6d0 msgr2=0x7fbf940feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.816+0000 7fbf9b751700 1 -- 172.21.15.96:0/528189197 shutdown_connections 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.816+0000 7fbf9b751700 1 -- 172.21.15.96:0/528189197 wait complete. 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf9b751700 1 Processor -- start 2024-04-22T15:46:12.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf9b751700 1 -- start start 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf9b751700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf94100e20 0x7fbf940731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf9b751700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf941017d0 0x7fbf94073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf9b751700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf940751b0 con 0x7fbf94100e20 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf9b751700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94073c60 con 0x7fbf941017d0 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf994ed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf94100e20 0x7fbf940731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf994ed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf94100e20 0x7fbf940731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34348/0 (socket says 172.21.15.96:34348) 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf994ed700 1 -- 172.21.15.96:0/3915007982 learned_addr learned my addr 172.21.15.96:0/3915007982 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.817+0000 7fbf98cec700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf941017d0 0x7fbf94073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf994ed700 1 -- 172.21.15.96:0/3915007982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf941017d0 msgr2=0x7fbf94073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf994ed700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf941017d0 0x7fbf94073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf994ed700 1 -- 172.21.15.96:0/3915007982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf840095f0 con 0x7fbf94100e20 2024-04-22T15:46:12.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf98cec700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf941017d0 0x7fbf94073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:46:12.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf994ed700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf94100e20 0x7fbf940731e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00c930 tx=0x7fbf8c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:12.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf8a7fc700 1 -- 172.21.15.96:0/3915007982 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf8c007ad0 con 0x7fbf94100e20 2024-04-22T15:46:12.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf94073f10 con 0x7fbf94100e20 2024-04-22T15:46:12.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf8a7fc700 1 -- 172.21.15.96:0/3915007982 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf8c007c30 con 0x7fbf94100e20 2024-04-22T15:46:12.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf8a7fc700 1 -- 172.21.15.96:0/3915007982 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf8c018610 con 0x7fbf94100e20 2024-04-22T15:46:12.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.818+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf941a2f60 con 0x7fbf94100e20 2024-04-22T15:46:12.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.819+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf94109220 con 0x7fbf94100e20 2024-04-22T15:46:12.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.819+0000 7fbf8a7fc700 1 -- 172.21.15.96:0/3915007982 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbf8c00f450 con 0x7fbf94100e20 2024-04-22T15:46:12.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.820+0000 7fbf8a7fc700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf8006ecc0 0x7fbf80071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.820+0000 7fbf98cec700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf8006ecc0 0x7fbf80071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.820+0000 7fbf8a7fc700 1 -- 172.21.15.96:0/3915007982 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbf8c08f130 con 0x7fbf94100e20 2024-04-22T15:46:12.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.820+0000 7fbf98cec700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf8006ecc0 0x7fbf80071180 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbf94074910 tx=0x7fbf8401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:12.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:12.824+0000 7fbf8a7fc700 1 -- 172.21.15.96:0/3915007982 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbf8c057e90 con 0x7fbf94100e20 2024-04-22T15:46:13.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.106+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbf940746c0 con 0x7fbf94100e20 2024-04-22T15:46:13.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.107+0000 7fbf8a7fc700 1 -- 172.21.15.96:0/3915007982 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fbf8c05b550 con 0x7fbf94100e20 2024-04-22T15:46:13.110 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:46:13.110 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:46:13.110 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:46:13.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:46:13.110 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:46:13.111 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:46:13.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf8006ecc0 msgr2=0x7fbf80071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf8006ecc0 0x7fbf80071180 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbf94074910 tx=0x7fbf8401a040 comp rx=0 tx=0).stop 2024-04-22T15:46:13.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf94100e20 msgr2=0x7fbf940731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf94100e20 0x7fbf940731e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00c930 tx=0x7fbf8c00ccf0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 shutdown_connections 2024-04-22T15:46:13.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf8006ecc0 0x7fbf80071180 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf94100e20 0x7fbf940731e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 --2- 172.21.15.96:0/3915007982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf941017d0 0x7fbf94073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 >> 172.21.15.96:0/3915007982 conn(0x7fbf940fc6d0 msgr2=0x7fbf940fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 shutdown_connections 2024-04-22T15:46:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.109+0000 7fbf9b751700 1 -- 172.21.15.96:0/3915007982 wait complete. 2024-04-22T15:46:13.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.213+0000 7fc7875c1700 1 -- 172.21.15.96:0/2001129097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 msgr2=0x7fc780105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.213+0000 7fc7875c1700 1 --2- 172.21.15.96:0/2001129097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 0x7fc780105820 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc77c009910 tx=0x7fc77c009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.213+0000 7fc7875c1700 1 -- 172.21.15.96:0/2001129097 shutdown_connections 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.213+0000 7fc7875c1700 1 --2- 172.21.15.96:0/2001129097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 0x7fc780105820 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.213+0000 7fc7875c1700 1 --2- 172.21.15.96:0/2001129097 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc780100e20 0x7fc780101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.213+0000 7fc7875c1700 1 -- 172.21.15.96:0/2001129097 >> 172.21.15.96:0/2001129097 conn(0x7fc7800fc6d0 msgr2=0x7fc7800feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.213+0000 7fc7875c1700 1 -- 172.21.15.96:0/2001129097 shutdown_connections 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.213+0000 7fc7875c1700 1 -- 172.21.15.96:0/2001129097 wait complete. 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc7875c1700 1 Processor -- start 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc7875c1700 1 -- start start 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc7875c1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc780100e20 0x7fc78019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:13.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc7875c1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 0x7fc78019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc7875c1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78019d4c0 con 0x7fc7801017d0 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc7875c1700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc780196aa0 con 0x7fc780100e20 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc784b5c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 0x7fc78019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc784b5c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 0x7fc78019cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34368/0 (socket says 172.21.15.96:34368) 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc784b5c700 1 -- 172.21.15.96:0/1628782745 learned_addr learned my addr 172.21.15.96:0/1628782745 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc78535d700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc780100e20 0x7fc78019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc784b5c700 1 -- 172.21.15.96:0/1628782745 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc780100e20 msgr2=0x7fc78019c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc784b5c700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc780100e20 0x7fc78019c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.214+0000 7fc784b5c700 1 -- 172.21.15.96:0/1628782745 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c0095f0 con 0x7fc7801017d0 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.215+0000 7fc784b5c700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 0x7fc78019cea0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00bd70 tx=0x7fc77c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.215+0000 7fc7767fc700 1 -- 172.21.15.96:0/1628782745 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc77c01d070 con 0x7fc7801017d0 2024-04-22T15:46:13.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.215+0000 7fc7767fc700 1 -- 172.21.15.96:0/1628782745 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc77c00f460 con 0x7fc7801017d0 2024-04-22T15:46:13.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.215+0000 7fc7767fc700 1 -- 172.21.15.96:0/1628782745 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc77c017630 con 0x7fc7801017d0 2024-04-22T15:46:13.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.215+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc780196d20 con 0x7fc7801017d0 2024-04-22T15:46:13.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.215+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7801971c0 con 0x7fc7801017d0 2024-04-22T15:46:13.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.216+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc780109170 con 0x7fc7801017d0 2024-04-22T15:46:13.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.216+0000 7fc7767fc700 1 -- 172.21.15.96:0/1628782745 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc77c00f5d0 con 0x7fc7801017d0 2024-04-22T15:46:13.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.217+0000 7fc7767fc700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc76c06ecc0 0x7fc76c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:13.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.217+0000 7fc7767fc700 1 -- 172.21.15.96:0/1628782745 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc77c0902b0 con 0x7fc7801017d0 2024-04-22T15:46:13.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.217+0000 7fc78535d700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc76c06ecc0 0x7fc76c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:13.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.217+0000 7fc78535d700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc76c06ecc0 0x7fc76c071180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc770006fd0 tx=0x7fc770008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:13.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.222+0000 7fc7767fc700 1 -- 172.21.15.96:0/1628782745 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc77c05cbc0 con 0x7fc7801017d0 2024-04-22T15:46:13.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.468+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc780102300 con 0x7fc7801017d0 2024-04-22T15:46:13.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.469+0000 7fc7767fc700 1 -- 172.21.15.96:0/1628782745 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7fc77c005740 con 0x7fc7801017d0 2024-04-22T15:46:13.472 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:46:13.472 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:46:13.472 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:46:13.472 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:46:13.472 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:46:13.472 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:46:13.473 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:46:13.474 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:46:13.475 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:46:13.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc76c06ecc0 msgr2=0x7fc76c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc76c06ecc0 0x7fc76c071180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc770006fd0 tx=0x7fc770008040 comp rx=0 tx=0).stop 2024-04-22T15:46:13.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 msgr2=0x7fc78019cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 0x7fc78019cea0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00bd70 tx=0x7fc77c00be50 comp rx=0 tx=0).stop 2024-04-22T15:46:13.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 shutdown_connections 2024-04-22T15:46:13.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc76c06ecc0 0x7fc76c071180 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc780100e20 0x7fc78019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 --2- 172.21.15.96:0/1628782745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7801017d0 0x7fc78019cea0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 >> 172.21.15.96:0/1628782745 conn(0x7fc7800fc6d0 msgr2=0x7fc7800fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:13.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.471+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 shutdown_connections 2024-04-22T15:46:13.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.472+0000 7fc7875c1700 1 -- 172.21.15.96:0/1628782745 wait complete. 2024-04-22T15:46:13.477 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:46:13.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.585+0000 7fb9867eb700 1 -- 172.21.15.96:0/3595048997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 msgr2=0x7fb980102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.585+0000 7fb9867eb700 1 --2- 172.21.15.96:0/3595048997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 0x7fb980102140 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb970009910 tx=0x7fb970009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:13.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.585+0000 7fb9867eb700 1 -- 172.21.15.96:0/3595048997 shutdown_connections 2024-04-22T15:46:13.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.585+0000 7fb9867eb700 1 --2- 172.21.15.96:0/3595048997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb980102680 0x7fb98010ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.585+0000 7fb9867eb700 1 --2- 172.21.15.96:0/3595048997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 0x7fb980102140 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.585+0000 7fb9867eb700 1 -- 172.21.15.96:0/3595048997 >> 172.21.15.96:0/3595048997 conn(0x7fb980076740 msgr2=0x7fb9800fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:13.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.586+0000 7fb9867eb700 1 -- 172.21.15.96:0/3595048997 shutdown_connections 2024-04-22T15:46:13.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.586+0000 7fb9867eb700 1 -- 172.21.15.96:0/3595048997 wait complete. 2024-04-22T15:46:13.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.586+0000 7fb9867eb700 1 Processor -- start 2024-04-22T15:46:13.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.586+0000 7fb9867eb700 1 -- start start 2024-04-22T15:46:13.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb9867eb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 0x7fb9800ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:13.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb9867eb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb980102680 0x7fb9800fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:13.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb9867eb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb980100440 con 0x7fb980101d60 2024-04-22T15:46:13.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb9867eb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb980100580 con 0x7fb980102680 2024-04-22T15:46:13.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 0x7fb9800ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:13.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 0x7fb9800ff9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34380/0 (socket says 172.21.15.96:34380) 2024-04-22T15:46:13.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97ffff700 1 -- 172.21.15.96:0/14658670 learned_addr learned my addr 172.21.15.96:0/14658670 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97f7fe700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb980102680 0x7fb9800fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97ffff700 1 -- 172.21.15.96:0/14658670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb980102680 msgr2=0x7fb9800fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97ffff700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb980102680 0x7fb9800fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97ffff700 1 -- 172.21.15.96:0/14658670 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9700095f0 con 0x7fb980101d60 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97ffff700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 0x7fb9800ff9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb970005f50 tx=0x7fb97000b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97d7fa700 1 -- 172.21.15.96:0/14658670 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb97001d070 con 0x7fb980101d60 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97d7fa700 1 -- 172.21.15.96:0/14658670 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9700052a0 con 0x7fb980101d60 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb980100800 con 0x7fb980101d60 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb97d7fa700 1 -- 172.21.15.96:0/14658670 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb970017400 con 0x7fb980101d60 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.587+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9801a2fa0 con 0x7fb980101d60 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.588+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb980068ea0 con 0x7fb980101d60 2024-04-22T15:46:13.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.591+0000 7fb97d7fa700 1 -- 172.21.15.96:0/14658670 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb97000f6f0 con 0x7fb980101d60 2024-04-22T15:46:13.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.592+0000 7fb97d7fa700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb96806ecd0 0x7fb968071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:13.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.592+0000 7fb97f7fe700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb96806ecd0 0x7fb968071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:13.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.592+0000 7fb97d7fa700 1 -- 172.21.15.96:0/14658670 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb970090250 con 0x7fb980101d60 2024-04-22T15:46:13.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.592+0000 7fb97f7fe700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb96806ecd0 0x7fb968071190 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb9801010f0 tx=0x7fb974009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:13.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.597+0000 7fb97d7fa700 1 -- 172.21.15.96:0/14658670 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb9700590d0 con 0x7fb980101d60 2024-04-22T15:46:13.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:13 smithi096 ceph-mon[112443]: from='client.35348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:13.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:13 smithi096 ceph-mon[112443]: from='client.35352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:13 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3915007982' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:46:13.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:13 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1628782745' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:46:13.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.817+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb98010de00 con 0x7fb96806ecd0 2024-04-22T15:46:13.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.819+0000 7fb97d7fa700 1 -- 172.21.15.96:0/14658670 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb98010de00 con 0x7fb96806ecd0 2024-04-22T15:46:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:46:13.822 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:46:13.823 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:46:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb96806ecd0 msgr2=0x7fb968071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb96806ecd0 0x7fb968071190 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb9801010f0 tx=0x7fb974009260 comp rx=0 tx=0).stop 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 msgr2=0x7fb9800ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 0x7fb9800ff9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb970005f50 tx=0x7fb97000b5a0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 shutdown_connections 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb96806ecd0 0x7fb968071190 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb980101d60 0x7fb9800ff9c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 --2- 172.21.15.96:0/14658670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb980102680 0x7fb9800fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 >> 172.21.15.96:0/14658670 conn(0x7fb980076740 msgr2=0x7fb980103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:13.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 shutdown_connections 2024-04-22T15:46:13.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.821+0000 7fb9867eb700 1 -- 172.21.15.96:0/14658670 wait complete. 2024-04-22T15:46:13.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.925+0000 7fcf00b25700 1 -- 172.21.15.96:0/591432581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc0ffaa0 msgr2=0x7fcefc0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.925+0000 7fcf00b25700 1 --2- 172.21.15.96:0/591432581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc0ffaa0 0x7fcefc0fff20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009910 tx=0x7fcef0009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:13.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.925+0000 7fcf00b25700 1 -- 172.21.15.96:0/591432581 shutdown_connections 2024-04-22T15:46:13.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.925+0000 7fcf00b25700 1 --2- 172.21.15.96:0/591432581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc0ffaa0 0x7fcefc0fff20 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.925+0000 7fcf00b25700 1 --2- 172.21.15.96:0/591432581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcefc105aa0 0x7fcefc105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.925+0000 7fcf00b25700 1 -- 172.21.15.96:0/591432581 >> 172.21.15.96:0/591432581 conn(0x7fcefc076740 msgr2=0x7fcefc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:13.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.925+0000 7fcf00b25700 1 -- 172.21.15.96:0/591432581 shutdown_connections 2024-04-22T15:46:13.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.925+0000 7fcf00b25700 1 -- 172.21.15.96:0/591432581 wait complete. 2024-04-22T15:46:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcf00b25700 1 Processor -- start 2024-04-22T15:46:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcf00b25700 1 -- start start 2024-04-22T15:46:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcf00b25700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcefc0ffaa0 0x7fcefc19b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcf00b25700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc105aa0 0x7fcefc19bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcf00b25700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc19c290 con 0x7fcefc105aa0 2024-04-22T15:46:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcf00b25700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc195800 con 0x7fcefc0ffaa0 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcef9d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc105aa0 0x7fcefc19bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcef9d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc105aa0 0x7fcefc19bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34408/0 (socket says 172.21.15.96:34408) 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcef9d9b700 1 -- 172.21.15.96:0/884622488 learned_addr learned my addr 172.21.15.96:0/884622488 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcefa59c700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcefc0ffaa0 0x7fcefc19b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcef9d9b700 1 -- 172.21.15.96:0/884622488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcefc0ffaa0 msgr2=0x7fcefc19b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcef9d9b700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcefc0ffaa0 0x7fcefc19b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.926+0000 7fcef9d9b700 1 -- 172.21.15.96:0/884622488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef00095f0 con 0x7fcefc105aa0 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.927+0000 7fcef9d9b700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc105aa0 0x7fcefc19bc00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcef000bd70 tx=0x7fcef000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.927+0000 7fceeb7fe700 1 -- 172.21.15.96:0/884622488 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcef001d070 con 0x7fcefc105aa0 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.927+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc195a80 con 0x7fcefc105aa0 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.927+0000 7fceeb7fe700 1 -- 172.21.15.96:0/884622488 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcef00056e0 con 0x7fcefc105aa0 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.927+0000 7fceeb7fe700 1 -- 172.21.15.96:0/884622488 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcef000fe40 con 0x7fcefc105aa0 2024-04-22T15:46:13.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.927+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc195f70 con 0x7fcefc105aa0 2024-04-22T15:46:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.928+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc1080c0 con 0x7fcefc105aa0 2024-04-22T15:46:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.928+0000 7fceeb7fe700 1 -- 172.21.15.96:0/884622488 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcef000f680 con 0x7fcefc105aa0 2024-04-22T15:46:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.929+0000 7fceeb7fe700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcee406ecc0 0x7fcee4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.929+0000 7fcefa59c700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcee406ecc0 0x7fcee4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.929+0000 7fceeb7fe700 1 -- 172.21.15.96:0/884622488 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcef0090200 con 0x7fcefc105aa0 2024-04-22T15:46:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.929+0000 7fcefa59c700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcee406ecc0 0x7fcee4071180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fceec005950 tx=0x7fceec00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:13.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:13.934+0000 7fceeb7fe700 1 -- 172.21.15.96:0/884622488 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcef005cb10 con 0x7fcefc105aa0 2024-04-22T15:46:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:13 smithi196 ceph-mon[81815]: from='client.35348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:13 smithi196 ceph-mon[81815]: from='client.35352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:14.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:13 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3915007982' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:46:14.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:13 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1628782745' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:46:14.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.216+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcefc1011e0 con 0x7fcefc105aa0 2024-04-22T15:46:14.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.217+0000 7fceeb7fe700 1 -- 172.21.15.96:0/884622488 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fcef005c6a0 con 0x7fcefc105aa0 2024-04-22T15:46:14.220 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:46:14.220 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:46:14.220 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:46:14.220 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:46:14.220 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:46:14.220 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:46:14.220 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:46:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:46:14.222 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:46:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:46:14.222 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:46:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcee406ecc0 msgr2=0x7fcee4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcee406ecc0 0x7fcee4071180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fceec005950 tx=0x7fceec00b410 comp rx=0 tx=0).stop 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc105aa0 msgr2=0x7fcefc19bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc105aa0 0x7fcefc19bc00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcef000bd70 tx=0x7fcef000bda0 comp rx=0 tx=0).stop 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 shutdown_connections 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcee406ecc0 0x7fcee4071180 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcefc0ffaa0 0x7fcefc19b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 --2- 172.21.15.96:0/884622488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcefc105aa0 0x7fcefc19bc00 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 >> 172.21.15.96:0/884622488 conn(0x7fcefc076740 msgr2=0x7fcefc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 shutdown_connections 2024-04-22T15:46:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:14.219+0000 7fcf00b25700 1 -- 172.21.15.96:0/884622488 wait complete. 2024-04-22T15:46:14.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:14 smithi096 ceph-mon[112443]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:14.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:14 smithi096 ceph-mon[112443]: pgmap v732: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:14 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/884622488' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:46:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:14 smithi196 ceph-mon[81815]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:14 smithi196 ceph-mon[81815]: pgmap v732: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:14 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/884622488' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:46:15.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:15 smithi096 ceph-mon[112443]: from='client.35368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:15 smithi196 ceph-mon[81815]: from='client.35368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:16.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:16 smithi096 ceph-mon[112443]: pgmap v733: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:16 smithi196 ceph-mon[81815]: pgmap v733: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:18.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:18 smithi096 ceph-mon[112443]: pgmap v734: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:18 smithi196 ceph-mon[81815]: pgmap v734: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:20.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:20 smithi096 ceph-mon[112443]: pgmap v735: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:20 smithi196 ceph-mon[81815]: pgmap v735: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:22.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:22 smithi096 ceph-mon[112443]: pgmap v736: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:22 smithi196 ceph-mon[81815]: pgmap v736: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:24.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:24 smithi096 ceph-mon[112443]: pgmap v737: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:24 smithi196 ceph-mon[81815]: pgmap v737: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:26.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:26 smithi096 ceph-mon[112443]: pgmap v738: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:26 smithi196 ceph-mon[81815]: pgmap v738: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:28 smithi096 ceph-mon[112443]: pgmap v739: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:28 smithi196 ceph-mon[81815]: pgmap v739: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:30.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:30 smithi096 ceph-mon[112443]: pgmap v740: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:30 smithi196 ceph-mon[81815]: pgmap v740: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:32.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:32 smithi096 ceph-mon[112443]: pgmap v741: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:32 smithi196 ceph-mon[81815]: pgmap v741: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:34 smithi096 ceph-mon[112443]: pgmap v742: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:34 smithi196 ceph-mon[81815]: pgmap v742: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:35.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:35 smithi096 ceph-mon[112443]: pgmap v743: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:35 smithi196 ceph-mon[81815]: pgmap v743: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:38 smithi096 ceph-mon[112443]: pgmap v744: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:38 smithi196 ceph-mon[81815]: pgmap v744: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:40 smithi096 ceph-mon[112443]: pgmap v745: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:40 smithi196 ceph-mon[81815]: pgmap v745: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:42 smithi096 ceph-mon[112443]: pgmap v746: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:42 smithi196 ceph-mon[81815]: pgmap v746: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:44.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.336+0000 7fa6e78d3700 1 -- 172.21.15.96:0/2771551309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 msgr2=0x7fa6e0105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:44.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.336+0000 7fa6e78d3700 1 --2- 172.21.15.96:0/2771551309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 0x7fa6e0105f40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009910 tx=0x7fa6dc009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:44.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 -- 172.21.15.96:0/2771551309 shutdown_connections 2024-04-22T15:46:44.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 --2- 172.21.15.96:0/2771551309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6e00ffad0 0x7fa6e00fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 --2- 172.21.15.96:0/2771551309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 0x7fa6e0105f40 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 -- 172.21.15.96:0/2771551309 >> 172.21.15.96:0/2771551309 conn(0x7fa6e00fb600 msgr2=0x7fa6e00fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:44.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 -- 172.21.15.96:0/2771551309 shutdown_connections 2024-04-22T15:46:44.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 -- 172.21.15.96:0/2771551309 wait complete. 2024-04-22T15:46:44.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 Processor -- start 2024-04-22T15:46:44.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 -- start start 2024-04-22T15:46:44.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6e00ffad0 0x7fa6e0101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:44.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.337+0000 7fa6e78d3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 0x7fa6e0101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:44.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e78d3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e01021b0 con 0x7fa6e0105b60 2024-04-22T15:46:44.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e78d3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e01022f0 con 0x7fa6e00ffad0 2024-04-22T15:46:44.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e4e6e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 0x7fa6e0101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:44.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e4e6e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 0x7fa6e0101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36402/0 (socket says 172.21.15.96:36402) 2024-04-22T15:46:44.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e4e6e700 1 -- 172.21.15.96:0/1706368627 learned_addr learned my addr 172.21.15.96:0/1706368627 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e566f700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6e00ffad0 0x7fa6e0101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e4e6e700 1 -- 172.21.15.96:0/1706368627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6e00ffad0 msgr2=0x7fa6e0101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e4e6e700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6e00ffad0 0x7fa6e0101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e4e6e700 1 -- 172.21.15.96:0/1706368627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc0095f0 con 0x7fa6e0105b60 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e4e6e700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 0x7fa6e0101c70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000d8d0 tx=0x7fa6d000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6d67fc700 1 -- 172.21.15.96:0/1706368627 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d00098a0 con 0x7fa6e0105b60 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6d67fc700 1 -- 172.21.15.96:0/1706368627 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6d0010460 con 0x7fa6e0105b60 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e01a2ca0 con 0x7fa6e0105b60 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6d67fc700 1 -- 172.21.15.96:0/1706368627 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d000f5f0 con 0x7fa6e0105b60 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.338+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e01a3000 con 0x7fa6e0105b60 2024-04-22T15:46:44.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.339+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e01011d0 con 0x7fa6e0105b60 2024-04-22T15:46:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.340+0000 7fa6d67fc700 1 -- 172.21.15.96:0/1706368627 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa6d0009a00 con 0x7fa6e0105b60 2024-04-22T15:46:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.340+0000 7fa6d67fc700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa6cc06ecc0 0x7fa6cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.340+0000 7fa6d67fc700 1 -- 172.21.15.96:0/1706368627 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa6d008ed50 con 0x7fa6e0105b60 2024-04-22T15:46:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.340+0000 7fa6e566f700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa6cc06ecc0 0x7fa6cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:44.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.341+0000 7fa6e566f700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa6cc06ecc0 0x7fa6cc071180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000c00 tx=0x7fa6dc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:44.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.345+0000 7fa6d67fc700 1 -- 172.21.15.96:0/1706368627 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa6d0057ab0 con 0x7fa6e0105b60 2024-04-22T15:46:44.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.566+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6e010cd40 con 0x7fa6cc06ecc0 2024-04-22T15:46:44.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.567+0000 7fa6d67fc700 1 -- 172.21.15.96:0/1706368627 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa6e010cd40 con 0x7fa6cc06ecc0 2024-04-22T15:46:44.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa6cc06ecc0 msgr2=0x7fa6cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:44.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa6cc06ecc0 0x7fa6cc071180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000c00 tx=0x7fa6dc01a040 comp rx=0 tx=0).stop 2024-04-22T15:46:44.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 msgr2=0x7fa6e0101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:44.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 0x7fa6e0101c70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000d8d0 tx=0x7fa6d000dbe0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 shutdown_connections 2024-04-22T15:46:44.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa6cc06ecc0 0x7fa6cc071180 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6e00ffad0 0x7fa6e0101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 --2- 172.21.15.96:0/1706368627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6e0105b60 0x7fa6e0101c70 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 >> 172.21.15.96:0/1706368627 conn(0x7fa6e00fb600 msgr2=0x7fa6e0103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:44.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 shutdown_connections 2024-04-22T15:46:44.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.569+0000 7fa6e78d3700 1 -- 172.21.15.96:0/1706368627 wait complete. 2024-04-22T15:46:44.582 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:46:44.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.674+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2256254819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 msgr2=0x7f4acc0fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:44.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.674+0000 7f4ad2e2c700 1 --2- 172.21.15.96:0/2256254819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 0x7f4acc0fffd0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009910 tx=0x7f4ab8009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:44.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.675+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2256254819 shutdown_connections 2024-04-22T15:46:44.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.675+0000 7f4ad2e2c700 1 --2- 172.21.15.96:0/2256254819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 0x7f4acc0fffd0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.675+0000 7f4ad2e2c700 1 --2- 172.21.15.96:0/2256254819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc105c00 0x7f4acc105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.675+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2256254819 >> 172.21.15.96:0/2256254819 conn(0x7f4acc0fb680 msgr2=0x7f4acc0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:44.678 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:44 smithi096 ceph-mon[112443]: pgmap v747: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:44.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.675+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2256254819 shutdown_connections 2024-04-22T15:46:44.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.675+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2256254819 wait complete. 2024-04-22T15:46:44.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.675+0000 7f4ad2e2c700 1 Processor -- start 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.675+0000 7f4ad2e2c700 1 -- start start 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad2e2c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 0x7f4acc197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad2e2c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc105c00 0x7f4acc197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad2e2c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc198470 con 0x7f4acc0ffb50 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad2e2c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc1985e0 con 0x7f4acc105c00 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad0bc8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 0x7f4acc197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad0bc8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 0x7f4acc197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36424/0 (socket says 172.21.15.96:36424) 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad0bc8700 1 -- 172.21.15.96:0/2524053758 learned_addr learned my addr 172.21.15.96:0/2524053758 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4acbfff700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc105c00 0x7f4acc197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad0bc8700 1 -- 172.21.15.96:0/2524053758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc105c00 msgr2=0x7f4acc197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad0bc8700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc105c00 0x7f4acc197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad0bc8700 1 -- 172.21.15.96:0/2524053758 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab80095f0 con 0x7f4acc0ffb50 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4acbfff700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc105c00 0x7f4acc197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:46:44.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.676+0000 7f4ad0bc8700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 0x7f4acc197960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000d8d0 tx=0x7f4ac000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:44.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.677+0000 7f4ac9ffb700 1 -- 172.21.15.96:0/2524053758 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ac0009940 con 0x7f4acc0ffb50 2024-04-22T15:46:44.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.677+0000 7f4ac9ffb700 1 -- 172.21.15.96:0/2524053758 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ac0010460 con 0x7f4acc0ffb50 2024-04-22T15:46:44.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.677+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4acc1a09d0 con 0x7f4acc0ffb50 2024-04-22T15:46:44.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.677+0000 7f4ac9ffb700 1 -- 172.21.15.96:0/2524053758 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ac000f5f0 con 0x7f4acc0ffb50 2024-04-22T15:46:44.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.677+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4acc1a0e20 con 0x7f4acc0ffb50 2024-04-22T15:46:44.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.678+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4acc1080b0 con 0x7f4acc0ffb50 2024-04-22T15:46:44.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.678+0000 7f4ac9ffb700 1 -- 172.21.15.96:0/2524053758 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4ac00105d0 con 0x7f4acc0ffb50 2024-04-22T15:46:44.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.679+0000 7f4ac9ffb700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4abc06ecc0 0x7f4abc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:44.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.679+0000 7f4ac9ffb700 1 -- 172.21.15.96:0/2524053758 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4ac008ef70 con 0x7f4acc0ffb50 2024-04-22T15:46:44.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.679+0000 7f4acbfff700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4abc06ecc0 0x7f4abc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:44.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.679+0000 7f4acbfff700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4abc06ecc0 0x7f4abc071180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4acc069940 tx=0x7f4ab801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:44.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.683+0000 7f4ac9ffb700 1 -- 172.21.15.96:0/2524053758 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4ac0057c30 con 0x7f4acc0ffb50 2024-04-22T15:46:44.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.902+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4acc10cd40 con 0x7f4abc06ecc0 2024-04-22T15:46:44.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.904+0000 7f4ac9ffb700 1 -- 172.21.15.96:0/2524053758 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4acc10cd40 con 0x7f4abc06ecc0 2024-04-22T15:46:44.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4abc06ecc0 msgr2=0x7f4abc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:44.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4abc06ecc0 0x7f4abc071180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4acc069940 tx=0x7f4ab801a040 comp rx=0 tx=0).stop 2024-04-22T15:46:44.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 msgr2=0x7f4acc197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:44.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 0x7f4acc197960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000d8d0 tx=0x7f4ac000dc90 comp rx=0 tx=0).stop 2024-04-22T15:46:44.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 shutdown_connections 2024-04-22T15:46:44.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4abc06ecc0 0x7f4abc071180 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc0ffb50 0x7f4acc197960 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 --2- 172.21.15.96:0/2524053758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc105c00 0x7f4acc197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:44.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 >> 172.21.15.96:0/2524053758 conn(0x7f4acc0fb680 msgr2=0x7f4acc0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:44.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 shutdown_connections 2024-04-22T15:46:44.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:44.906+0000 7f4ad2e2c700 1 -- 172.21.15.96:0/2524053758 wait complete. 2024-04-22T15:46:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:44 smithi196 ceph-mon[81815]: pgmap v747: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:45.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.010+0000 7f9be3f77700 1 -- 172.21.15.96:0/1957709718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 msgr2=0x7f9bdc1047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.010+0000 7f9be3f77700 1 --2- 172.21.15.96:0/1957709718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 0x7f9bdc1047e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc009910 tx=0x7f9bcc009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.010+0000 7f9be3f77700 1 -- 172.21.15.96:0/1957709718 shutdown_connections 2024-04-22T15:46:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.010+0000 7f9be3f77700 1 --2- 172.21.15.96:0/1957709718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 0x7f9bdc1047e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.010+0000 7f9be3f77700 1 --2- 172.21.15.96:0/1957709718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bdc0ffde0 0x7f9bdc1001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be3f77700 1 -- 172.21.15.96:0/1957709718 >> 172.21.15.96:0/1957709718 conn(0x7f9bdc0fb690 msgr2=0x7f9bdc0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be3f77700 1 -- 172.21.15.96:0/1957709718 shutdown_connections 2024-04-22T15:46:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be3f77700 1 -- 172.21.15.96:0/1957709718 wait complete. 2024-04-22T15:46:45.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be3f77700 1 Processor -- start 2024-04-22T15:46:45.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be3f77700 1 -- start start 2024-04-22T15:46:45.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be3f77700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bdc0ffde0 0x7f9bdc197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be3f77700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 0x7f9bdc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be3f77700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc198500 con 0x7f9bdc100790 2024-04-22T15:46:45.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be3f77700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc198640 con 0x7f9bdc0ffde0 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.011+0000 7f9be1512700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 0x7f9bdc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be1512700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 0x7f9bdc197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36450/0 (socket says 172.21.15.96:36450) 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be1512700 1 -- 172.21.15.96:0/3554247652 learned_addr learned my addr 172.21.15.96:0/3554247652 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be1d13700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bdc0ffde0 0x7f9bdc197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be1512700 1 -- 172.21.15.96:0/3554247652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bdc0ffde0 msgr2=0x7f9bdc197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be1512700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bdc0ffde0 0x7f9bdc197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be1512700 1 -- 172.21.15.96:0/3554247652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0095f0 con 0x7f9bdc100790 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be1d13700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bdc0ffde0 0x7f9bdc197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be1512700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 0x7f9bdc197fc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00bd70 tx=0x7f9bcc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9bd2ffd700 1 -- 172.21.15.96:0/3554247652 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bcc01d070 con 0x7f9bdc100790 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bdc1988c0 con 0x7f9bdc100790 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9bd2ffd700 1 -- 172.21.15.96:0/3554247652 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bcc00f460 con 0x7f9bdc100790 2024-04-22T15:46:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9bd2ffd700 1 -- 172.21.15.96:0/3554247652 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bcc017630 con 0x7f9bdc100790 2024-04-22T15:46:45.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.012+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bdc1a0e20 con 0x7f9bdc100790 2024-04-22T15:46:45.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.013+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bdc108130 con 0x7f9bdc100790 2024-04-22T15:46:45.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.014+0000 7f9bd2ffd700 1 -- 172.21.15.96:0/3554247652 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9bcc00f5d0 con 0x7f9bdc100790 2024-04-22T15:46:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.014+0000 7f9bd2ffd700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9bc806ecc0 0x7f9bc8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.014+0000 7f9be1d13700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9bc806ecc0 0x7f9bc8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.014+0000 7f9bd2ffd700 1 -- 172.21.15.96:0/3554247652 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9bcc090360 con 0x7f9bdc100790 2024-04-22T15:46:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.014+0000 7f9be1d13700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9bc806ecc0 0x7f9bc8071180 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9bd80099d0 tx=0x7f9bd8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:45.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.019+0000 7f9bd2ffd700 1 -- 172.21.15.96:0/3554247652 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9bcc0590c0 con 0x7f9bdc100790 2024-04-22T15:46:45.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.228+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bdc0696a0 con 0x7f9bc806ecc0 2024-04-22T15:46:45.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.239+0000 7f9bd2ffd700 1 -- 172.21.15.96:0/3554247652 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9bdc0696a0 con 0x7f9bc806ecc0 2024-04-22T15:46:45.242 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:46:45.242 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (25m) 3m ago 36m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:46:45.242 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (37m) 3m ago 37m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:46:45.242 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (35m) 3m ago 35m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:46:45.242 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (22m) 3m ago 37m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:46:45.243 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (22m) 3m ago 35m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:46:45.243 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (25m) 3m ago 36m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:46:45.243 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (18m) 3m ago 31m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:46:45.243 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (18m) 3m ago 31m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:46:45.243 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (18m) 3m ago 31m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:46:45.243 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (3m) 3m ago 30m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:46:45.243 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (27m) 3m ago 38m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (26m) 3m ago 35m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (23m) 3m ago 38m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (23m) 3m ago 35m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (26m) 3m ago 36m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (26m) 3m ago 35m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (22m) 3m ago 34m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (21m) 3m ago 34m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (20m) 3m ago 33m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (20m) 3m ago 33m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (19m) 3m ago 33m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (19m) 3m ago 32m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:46:45.244 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (25m) 3m ago 36m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:46:45.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9bc806ecc0 msgr2=0x7f9bc8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9bc806ecc0 0x7f9bc8071180 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9bd80099d0 tx=0x7f9bd8008040 comp rx=0 tx=0).stop 2024-04-22T15:46:45.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 msgr2=0x7f9bdc197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 0x7f9bdc197fc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00bd70 tx=0x7f9bcc00be50 comp rx=0 tx=0).stop 2024-04-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 shutdown_connections 2024-04-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9bc806ecc0 0x7f9bc8071180 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bdc0ffde0 0x7f9bdc197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 --2- 172.21.15.96:0/3554247652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9bdc100790 0x7f9bdc197fc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 >> 172.21.15.96:0/3554247652 conn(0x7f9bdc0fb690 msgr2=0x7f9bdc0fba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 shutdown_connections 2024-04-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.241+0000 7f9be3f77700 1 -- 172.21.15.96:0/3554247652 wait complete. 2024-04-22T15:46:45.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.343+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3785210666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc105b60 msgr2=0x7f98cc105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.343+0000 7f98d2e1d700 1 --2- 172.21.15.96:0/3785210666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc105b60 0x7f98cc105f40 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f98c400b0a0 tx=0x7f98c400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.343+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3785210666 shutdown_connections 2024-04-22T15:46:45.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.343+0000 7f98d2e1d700 1 --2- 172.21.15.96:0/3785210666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98cc0ffad0 0x7f98cc0fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.343+0000 7f98d2e1d700 1 --2- 172.21.15.96:0/3785210666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc105b60 0x7f98cc105f40 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.343+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3785210666 >> 172.21.15.96:0/3785210666 conn(0x7f98cc0fb5e0 msgr2=0x7f98cc0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:45.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.343+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3785210666 shutdown_connections 2024-04-22T15:46:45.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.343+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3785210666 wait complete. 2024-04-22T15:46:45.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d2e1d700 1 Processor -- start 2024-04-22T15:46:45.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d2e1d700 1 -- start start 2024-04-22T15:46:45.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d2e1d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc0ffad0 0x7f98cc197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d2e1d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98cc105b60 0x7f98cc197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d2e1d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc198580 con 0x7f98cc0ffad0 2024-04-22T15:46:45.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d2e1d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc1986c0 con 0x7f98cc105b60 2024-04-22T15:46:45.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d0bb9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc0ffad0 0x7f98cc197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d0bb9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc0ffad0 0x7f98cc197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36454/0 (socket says 172.21.15.96:36454) 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d0bb9700 1 -- 172.21.15.96:0/3371053372 learned_addr learned my addr 172.21.15.96:0/3371053372 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98cbfff700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98cc105b60 0x7f98cc197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d0bb9700 1 -- 172.21.15.96:0/3371053372 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98cc105b60 msgr2=0x7f98cc197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d0bb9700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98cc105b60 0x7f98cc197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.344+0000 7f98d0bb9700 1 -- 172.21.15.96:0/3371053372 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c4009d00 con 0x7f98cc0ffad0 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.345+0000 7f98d0bb9700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc0ffad0 0x7f98cc197a70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f98c4000f80 tx=0x7f98c4007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.345+0000 7f98c9ffb700 1 -- 172.21.15.96:0/3371053372 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c400e040 con 0x7f98cc0ffad0 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.345+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc1a0ae0 con 0x7f98cc0ffad0 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.345+0000 7f98c9ffb700 1 -- 172.21.15.96:0/3371053372 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98c4003de0 con 0x7f98cc0ffad0 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.345+0000 7f98c9ffb700 1 -- 172.21.15.96:0/3371053372 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c401d720 con 0x7f98cc0ffad0 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.345+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc1a0e40 con 0x7f98cc0ffad0 2024-04-22T15:46:45.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.346+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc1080d0 con 0x7f98cc0ffad0 2024-04-22T15:46:45.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.346+0000 7f98c9ffb700 1 -- 172.21.15.96:0/3371053372 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f98c4004690 con 0x7f98cc0ffad0 2024-04-22T15:46:45.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.347+0000 7f98c9ffb700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f98b806ecc0 0x7f98b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.347+0000 7f98c9ffb700 1 -- 172.21.15.96:0/3371053372 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f98c4090710 con 0x7f98cc0ffad0 2024-04-22T15:46:45.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.347+0000 7f98cbfff700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f98b806ecc0 0x7f98b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.347+0000 7f98cbfff700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f98b806ecc0 0x7f98b8071180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f98bc006fd0 tx=0x7f98bc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:45.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.354+0000 7f98c9ffb700 1 -- 172.21.15.96:0/3371053372 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f98c4059470 con 0x7f98cc0ffad0 2024-04-22T15:46:45.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.639+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f98cc0696e0 con 0x7f98cc0ffad0 2024-04-22T15:46:45.643 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:45 smithi096 ceph-mon[112443]: from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:45.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.640+0000 7f98c9ffb700 1 -- 172.21.15.96:0/3371053372 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f98c405cb30 con 0x7f98cc0ffad0 2024-04-22T15:46:45.643 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:46:45.643 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:46:45.643 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:46:45.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:46:45.643 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:46:45.644 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:46:45.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f98b806ecc0 msgr2=0x7f98b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f98b806ecc0 0x7f98b8071180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f98bc006fd0 tx=0x7f98bc008040 comp rx=0 tx=0).stop 2024-04-22T15:46:45.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc0ffad0 msgr2=0x7f98cc197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc0ffad0 0x7f98cc197a70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f98c4000f80 tx=0x7f98c4007bd0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 shutdown_connections 2024-04-22T15:46:45.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f98b806ecc0 0x7f98b8071180 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98cc0ffad0 0x7f98cc197a70 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 --2- 172.21.15.96:0/3371053372 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98cc105b60 0x7f98cc197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 >> 172.21.15.96:0/3371053372 conn(0x7f98cc0fb5e0 msgr2=0x7f98cc0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:45.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 shutdown_connections 2024-04-22T15:46:45.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.642+0000 7f98d2e1d700 1 -- 172.21.15.96:0/3371053372 wait complete. 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.746+0000 7f69da48a700 1 -- 172.21.15.96:0/3442239558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 msgr2=0x7f69d40ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.746+0000 7f69da48a700 1 --2- 172.21.15.96:0/3442239558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 0x7f69d40ff790 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f69c4009910 tx=0x7f69c4009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.746+0000 7f69da48a700 1 -- 172.21.15.96:0/3442239558 shutdown_connections 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.746+0000 7f69da48a700 1 --2- 172.21.15.96:0/3442239558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69d40ffd60 0x7f69d41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.746+0000 7f69da48a700 1 --2- 172.21.15.96:0/3442239558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 0x7f69d40ff790 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.746+0000 7f69da48a700 1 -- 172.21.15.96:0/3442239558 >> 172.21.15.96:0/3442239558 conn(0x7f69d4075f40 msgr2=0x7f69d40fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.746+0000 7f69da48a700 1 -- 172.21.15.96:0/3442239558 shutdown_connections 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.746+0000 7f69da48a700 1 -- 172.21.15.96:0/3442239558 wait complete. 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.747+0000 7f69da48a700 1 Processor -- start 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.747+0000 7f69da48a700 1 -- start start 2024-04-22T15:46:45.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.747+0000 7f69da48a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 0x7f69d4100330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.747+0000 7f69da48a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69d40ffd60 0x7f69d4100870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.747+0000 7f69da48a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d4100db0 con 0x7f69d40ff3b0 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.747+0000 7f69da48a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d4100ef0 con 0x7f69d40ffd60 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 0x7f69d4100330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 0x7f69d4100330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36478/0 (socket says 172.21.15.96:36478) 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d3fff700 1 -- 172.21.15.96:0/490151837 learned_addr learned my addr 172.21.15.96:0/490151837 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d37fe700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69d40ffd60 0x7f69d4100870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d3fff700 1 -- 172.21.15.96:0/490151837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69d40ffd60 msgr2=0x7f69d4100870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d3fff700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69d40ffd60 0x7f69d4100870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d3fff700 1 -- 172.21.15.96:0/490151837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c40095f0 con 0x7f69d40ff3b0 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d3fff700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 0x7f69d4100330 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f69c400bda0 tx=0x7f69c400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d17fa700 1 -- 172.21.15.96:0/490151837 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c401d070 con 0x7f69d40ff3b0 2024-04-22T15:46:45.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d17fa700 1 -- 172.21.15.96:0/490151837 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69c400f460 con 0x7f69d40ff3b0 2024-04-22T15:46:45.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d4101170 con 0x7f69d40ff3b0 2024-04-22T15:46:45.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69d17fa700 1 -- 172.21.15.96:0/490151837 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c4017630 con 0x7f69d40ff3b0 2024-04-22T15:46:45.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.748+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d41a30c0 con 0x7f69d40ff3b0 2024-04-22T15:46:45.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.749+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d4109210 con 0x7f69d40ff3b0 2024-04-22T15:46:45.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.750+0000 7f69d17fa700 1 -- 172.21.15.96:0/490151837 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f69c400f5d0 con 0x7f69d40ff3b0 2024-04-22T15:46:45.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.750+0000 7f69d17fa700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f69bc06ecc0 0x7f69bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.750+0000 7f69d17fa700 1 -- 172.21.15.96:0/490151837 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f69c40903c0 con 0x7f69d40ff3b0 2024-04-22T15:46:45.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.750+0000 7f69d37fe700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f69bc06ecc0 0x7f69bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.751+0000 7f69d37fe700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f69bc06ecc0 0x7f69bc071180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f69c8006fd0 tx=0x7f69c8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:45.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.755+0000 7f69d17fa700 1 -- 172.21.15.96:0/490151837 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f69c4059120 con 0x7f69d40ff3b0 2024-04-22T15:46:46.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.996+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f69d404ebb0 con 0x7f69d40ff3b0 2024-04-22T15:46:46.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.997+0000 7f69d17fa700 1 -- 172.21.15.96:0/490151837 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f69c405c7e0 con 0x7f69d40ff3b0 2024-04-22T15:46:46.000 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:46:46.000 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:46:46.000 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:46:46.000 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:46:46.000 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:46:46.001 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:46:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:46:46.003 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:46:46.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f69bc06ecc0 msgr2=0x7f69bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f69bc06ecc0 0x7f69bc071180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f69c8006fd0 tx=0x7f69c8008040 comp rx=0 tx=0).stop 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 msgr2=0x7f69d4100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 0x7f69d4100330 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f69c400bda0 tx=0x7f69c400be80 comp rx=0 tx=0).stop 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 shutdown_connections 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f69bc06ecc0 0x7f69bc071180 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69d40ff3b0 0x7f69d4100330 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 --2- 172.21.15.96:0/490151837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69d40ffd60 0x7f69d4100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 >> 172.21.15.96:0/490151837 conn(0x7f69d4075f40 msgr2=0x7f69d4104730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 shutdown_connections 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:45.999+0000 7f69da48a700 1 -- 172.21.15.96:0/490151837 wait complete. 2024-04-22T15:46:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:46:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:45 smithi196 ceph-mon[81815]: from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:46.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.103+0000 7fe7ca682700 1 -- 172.21.15.96:0/2773195918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4101e50 msgr2=0x7fe7c4102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.104+0000 7fe7ca682700 1 --2- 172.21.15.96:0/2773195918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4101e50 0x7fe7c4102230 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009910 tx=0x7fe7b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:46.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.104+0000 7fe7ca682700 1 -- 172.21.15.96:0/2773195918 shutdown_connections 2024-04-22T15:46:46.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.104+0000 7fe7ca682700 1 --2- 172.21.15.96:0/2773195918 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7c4102770 0x7fe7c410ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.104+0000 7fe7ca682700 1 --2- 172.21.15.96:0/2773195918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4101e50 0x7fe7c4102230 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.104+0000 7fe7ca682700 1 -- 172.21.15.96:0/2773195918 >> 172.21.15.96:0/2773195918 conn(0x7fe7c40fb600 msgr2=0x7fe7c40fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:46.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.104+0000 7fe7ca682700 1 -- 172.21.15.96:0/2773195918 shutdown_connections 2024-04-22T15:46:46.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.104+0000 7fe7ca682700 1 -- 172.21.15.96:0/2773195918 wait complete. 2024-04-22T15:46:46.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.104+0000 7fe7ca682700 1 Processor -- start 2024-04-22T15:46:46.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.104+0000 7fe7ca682700 1 -- start start 2024-04-22T15:46:46.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7ca682700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7c4101e50 0x7fe7c419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:46.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7ca682700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4102770 0x7fe7c419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:46.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7ca682700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c419d4a0 con 0x7fe7c4102770 2024-04-22T15:46:46.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7ca682700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c4196a80 con 0x7fe7c4101e50 2024-04-22T15:46:46.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4102770 0x7fe7c419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:46.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4102770 0x7fe7c419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36504/0 (socket says 172.21.15.96:36504) 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7c37fe700 1 -- 172.21.15.96:0/2282940649 learned_addr learned my addr 172.21.15.96:0/2282940649 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7c3fff700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7c4101e50 0x7fe7c419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7c37fe700 1 -- 172.21.15.96:0/2282940649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7c4101e50 msgr2=0x7fe7c419c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7c37fe700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7c4101e50 0x7fe7c419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7c37fe700 1 -- 172.21.15.96:0/2282940649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b40095f0 con 0x7fe7c4102770 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.105+0000 7fe7c37fe700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4102770 0x7fe7c419ce80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800b480 tx=0x7fe7b800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.106+0000 7fe7c17fa700 1 -- 172.21.15.96:0/2282940649 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7b80118c0 con 0x7fe7c4102770 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.106+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c4196d60 con 0x7fe7c4102770 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.106+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c41972b0 con 0x7fe7c4102770 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.106+0000 7fe7c17fa700 1 -- 172.21.15.96:0/2282940649 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7b8010380 con 0x7fe7c4102770 2024-04-22T15:46:46.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.106+0000 7fe7c17fa700 1 -- 172.21.15.96:0/2282940649 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7b800f4f0 con 0x7fe7c4102770 2024-04-22T15:46:46.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.107+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b00052f0 con 0x7fe7c4102770 2024-04-22T15:46:46.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.107+0000 7fe7c17fa700 1 -- 172.21.15.96:0/2282940649 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe7b800f7b0 con 0x7fe7c4102770 2024-04-22T15:46:46.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.108+0000 7fe7c17fa700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe7ac06ecc0 0x7fe7ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:46.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.108+0000 7fe7c17fa700 1 -- 172.21.15.96:0/2282940649 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe7b808eac0 con 0x7fe7c4102770 2024-04-22T15:46:46.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.108+0000 7fe7c3fff700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe7ac06ecc0 0x7fe7ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:46.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.108+0000 7fe7c3fff700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe7ac06ecc0 0x7fe7ac071180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400b5c0 tx=0x7fe7b4005380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:46.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.112+0000 7fe7c17fa700 1 -- 172.21.15.96:0/2282940649 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe7b805b510 con 0x7fe7c4102770 2024-04-22T15:46:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.330+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7b0000bc0 con 0x7fe7ac06ecc0 2024-04-22T15:46:46.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.332+0000 7fe7c17fa700 1 -- 172.21.15.96:0/2282940649 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe7b0000bc0 con 0x7fe7ac06ecc0 2024-04-22T15:46:46.335 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:46:46.335 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:46:46.335 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:46:46.335 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:46:46.335 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:46:46.335 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:46:46.336 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:46:46.336 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:46:46.336 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:46:46.336 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:46:46.336 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:46:46.336 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:46:46.336 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:46:46.336 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:46:46.336 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:46:46.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.334+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe7ac06ecc0 msgr2=0x7fe7ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.334+0000 7fe7ca682700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe7ac06ecc0 0x7fe7ac071180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400b5c0 tx=0x7fe7b4005380 comp rx=0 tx=0).stop 2024-04-22T15:46:46.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.334+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4102770 msgr2=0x7fe7c419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.334+0000 7fe7ca682700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4102770 0x7fe7c419ce80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800b480 tx=0x7fe7b800b840 comp rx=0 tx=0).stop 2024-04-22T15:46:46.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.334+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 shutdown_connections 2024-04-22T15:46:46.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.334+0000 7fe7ca682700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe7ac06ecc0 0x7fe7ac071180 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.334+0000 7fe7ca682700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7c4101e50 0x7fe7c419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.334+0000 7fe7ca682700 1 --2- 172.21.15.96:0/2282940649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7c4102770 0x7fe7c419ce80 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.334+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 >> 172.21.15.96:0/2282940649 conn(0x7fe7c40fb600 msgr2=0x7fe7c4103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:46.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.335+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 shutdown_connections 2024-04-22T15:46:46.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.335+0000 7fe7ca682700 1 -- 172.21.15.96:0/2282940649 wait complete. 2024-04-22T15:46:46.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.439+0000 7f8af625c700 1 -- 172.21.15.96:0/2734444929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 msgr2=0x7f8af0105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.439+0000 7f8af625c700 1 --2- 172.21.15.96:0/2734444929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 0x7f8af0105810 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009910 tx=0x7f8ad8009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:46.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.439+0000 7f8af625c700 1 -- 172.21.15.96:0/2734444929 shutdown_connections 2024-04-22T15:46:46.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.439+0000 7f8af625c700 1 --2- 172.21.15.96:0/2734444929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 0x7f8af0105810 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.439+0000 7f8af625c700 1 --2- 172.21.15.96:0/2734444929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8af0100e10 0x7f8af01011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.439+0000 7f8af625c700 1 -- 172.21.15.96:0/2734444929 >> 172.21.15.96:0/2734444929 conn(0x7f8af00fc6e0 msgr2=0x7f8af00feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:46.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.439+0000 7f8af625c700 1 -- 172.21.15.96:0/2734444929 shutdown_connections 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.439+0000 7f8af625c700 1 -- 172.21.15.96:0/2734444929 wait complete. 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8af625c700 1 Processor -- start 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8af625c700 1 -- start start 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8af625c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8af0100e10 0x7f8af019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8af625c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 0x7f8af019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8af625c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af019d380 con 0x7f8af01017c0 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8af625c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af01969b0 con 0x7f8af0100e10 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8aeeffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 0x7f8af019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8aeeffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 0x7f8af019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36528/0 (socket says 172.21.15.96:36528) 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8aeeffd700 1 -- 172.21.15.96:0/622686969 learned_addr learned my addr 172.21.15.96:0/622686969 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.440+0000 7f8aef7fe700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8af0100e10 0x7f8af019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.441+0000 7f8aeeffd700 1 -- 172.21.15.96:0/622686969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8af0100e10 msgr2=0x7f8af019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.441+0000 7f8aeeffd700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8af0100e10 0x7f8af019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.441+0000 7f8aeeffd700 1 -- 172.21.15.96:0/622686969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad80095f0 con 0x7f8af01017c0 2024-04-22T15:46:46.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.441+0000 7f8aeeffd700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 0x7f8af019cd60 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8000c00 tx=0x7f8ad800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:46.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.441+0000 7f8aecff9700 1 -- 172.21.15.96:0/622686969 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad801d070 con 0x7f8af01017c0 2024-04-22T15:46:46.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.441+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0196c30 con 0x7f8af01017c0 2024-04-22T15:46:46.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.441+0000 7f8aecff9700 1 -- 172.21.15.96:0/622686969 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ad8005700 con 0x7f8af01017c0 2024-04-22T15:46:46.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.441+0000 7f8aecff9700 1 -- 172.21.15.96:0/622686969 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad800fda0 con 0x7f8af01017c0 2024-04-22T15:46:46.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.441+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af0197120 con 0x7f8af01017c0 2024-04-22T15:46:46.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.442+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af0109210 con 0x7f8af01017c0 2024-04-22T15:46:46.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.442+0000 7f8aecff9700 1 -- 172.21.15.96:0/622686969 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8ad800f5d0 con 0x7f8af01017c0 2024-04-22T15:46:46.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.443+0000 7f8aecff9700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8adc06ecd0 0x7f8adc071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:46.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.443+0000 7f8aecff9700 1 -- 172.21.15.96:0/622686969 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8ad8090240 con 0x7f8af01017c0 2024-04-22T15:46:46.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.443+0000 7f8aef7fe700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8adc06ecd0 0x7f8adc071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:46.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.443+0000 7f8aef7fe700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8adc06ecd0 0x7f8adc071190 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8005950 tx=0x7f8ae800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:46.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.448+0000 7f8aecff9700 1 -- 172.21.15.96:0/622686969 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8ad805cad0 con 0x7f8af01017c0 2024-04-22T15:46:46.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.734+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8af01022f0 con 0x7f8af01017c0 2024-04-22T15:46:46.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.735+0000 7f8aecff9700 1 -- 172.21.15.96:0/622686969 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f8ad805c660 con 0x7f8af01017c0 2024-04-22T15:46:46.738 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:46:46.738 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:46:46.738 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:46:46.738 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:46:46.738 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:46:46.740 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:46:46.740 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:46:46.740 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:46:46.740 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:46:46.740 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:46:46.740 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:46:46.740 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:46:46.740 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:46:46.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8adc06ecd0 msgr2=0x7f8adc071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8adc06ecd0 0x7f8adc071190 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8005950 tx=0x7f8ae800b410 comp rx=0 tx=0).stop 2024-04-22T15:46:46.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 msgr2=0x7f8af019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:46.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 0x7f8af019cd60 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8000c00 tx=0x7f8ad800bd70 comp rx=0 tx=0).stop 2024-04-22T15:46:46.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 shutdown_connections 2024-04-22T15:46:46.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8adc06ecd0 0x7f8adc071190 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8af0100e10 0x7f8af019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 --2- 172.21.15.96:0/622686969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8af01017c0 0x7f8af019cd60 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:46.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 >> 172.21.15.96:0/622686969 conn(0x7f8af00fc6e0 msgr2=0x7f8af00fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:46.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 shutdown_connections 2024-04-22T15:46:46.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:46:46.737+0000 7f8af625c700 1 -- 172.21.15.96:0/622686969 wait complete. 2024-04-22T15:46:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:46 smithi096 ceph-mon[112443]: from='client.35380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:46 smithi096 ceph-mon[112443]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:46 smithi096 ceph-mon[112443]: pgmap v748: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:46 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3371053372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:46:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:46 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/490151837' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:46:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:46 smithi196 ceph-mon[81815]: from='client.35380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:46 smithi196 ceph-mon[81815]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:46 smithi196 ceph-mon[81815]: pgmap v748: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:46 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3371053372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:46:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:46 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/490151837' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:46:47.863 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:47 smithi196 ceph-mon[81815]: from='client.35396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:47.863 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:47 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/622686969' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:46:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:47 smithi096 ceph-mon[112443]: from='client.35396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:47.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:47 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/622686969' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:46:48.879 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:48 smithi096 ceph-mon[112443]: pgmap v749: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:48 smithi196 ceph-mon[81815]: pgmap v749: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:50.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:50 smithi096 ceph-mon[112443]: pgmap v750: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:50 smithi196 ceph-mon[81815]: pgmap v750: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:52.916 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:52 smithi196 ceph-mon[81815]: pgmap v751: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:52.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:52 smithi096 ceph-mon[112443]: pgmap v751: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:53.932 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:54 smithi096 ceph-mon[112443]: pgmap v752: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:54 smithi196 ceph-mon[81815]: pgmap v752: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:56.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:56 smithi096 ceph-mon[112443]: pgmap v753: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:56 smithi196 ceph-mon[81815]: pgmap v753: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:58.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:46:58 smithi096 ceph-mon[112443]: pgmap v754: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:46:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:46:58 smithi196 ceph-mon[81815]: pgmap v754: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:00 smithi096 ceph-mon[112443]: pgmap v755: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:00 smithi196 ceph-mon[81815]: pgmap v755: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:02.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:02 smithi096 ceph-mon[112443]: pgmap v756: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:02 smithi196 ceph-mon[81815]: pgmap v756: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:04.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:04 smithi096 ceph-mon[112443]: pgmap v757: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:04 smithi196 ceph-mon[81815]: pgmap v757: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:05.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:05 smithi096 ceph-mon[112443]: pgmap v758: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:05 smithi196 ceph-mon[81815]: pgmap v758: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:08 smithi096 ceph-mon[112443]: pgmap v759: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:08 smithi196 ceph-mon[81815]: pgmap v759: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:10.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:10 smithi096 ceph-mon[112443]: pgmap v760: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:10 smithi196 ceph-mon[81815]: pgmap v760: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:12 smithi096 ceph-mon[112443]: pgmap v761: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:12 smithi196 ceph-mon[81815]: pgmap v761: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:14.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:14 smithi096 ceph-mon[112443]: pgmap v762: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:14 smithi196 ceph-mon[81815]: pgmap v762: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:16.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.853+0000 7f193915b700 1 -- 172.21.15.96:0/1706517670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19341017e0 msgr2=0x7f1934105830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:16.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.853+0000 7f193915b700 1 --2- 172.21.15.96:0/1706517670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19341017e0 0x7f1934105830 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f191c009910 tx=0x7f191c009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:16.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.853+0000 7f193915b700 1 -- 172.21.15.96:0/1706517670 shutdown_connections 2024-04-22T15:47:16.857 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:16 smithi096 ceph-mon[112443]: pgmap v763: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:16.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.853+0000 7f193915b700 1 --2- 172.21.15.96:0/1706517670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19341017e0 0x7f1934105830 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:16.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.853+0000 7f193915b700 1 --2- 172.21.15.96:0/1706517670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934100e30 0x7f1934101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.853+0000 7f193915b700 1 -- 172.21.15.96:0/1706517670 >> 172.21.15.96:0/1706517670 conn(0x7f19340fc700 msgr2=0x7f19340feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.853+0000 7f193915b700 1 -- 172.21.15.96:0/1706517670 shutdown_connections 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.853+0000 7f193915b700 1 -- 172.21.15.96:0/1706517670 wait complete. 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.854+0000 7f193915b700 1 Processor -- start 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.854+0000 7f193915b700 1 -- start start 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.854+0000 7f193915b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1934100e30 0x7f19340731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.854+0000 7f193915b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19341017e0 0x7f1934073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.854+0000 7f193915b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1934073c60 con 0x7f1934100e30 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.854+0000 7f193915b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1934073dd0 con 0x7f19341017e0 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1932d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1934100e30 0x7f19340731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1932d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1934100e30 0x7f19340731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53620/0 (socket says 172.21.15.96:53620) 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1932d9d700 1 -- 172.21.15.96:0/1154889992 learned_addr learned my addr 172.21.15.96:0/1154889992 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:16.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f192a59c700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19341017e0 0x7f1934073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:16.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1932d9d700 1 -- 172.21.15.96:0/1154889992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19341017e0 msgr2=0x7f1934073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:16.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1932d9d700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19341017e0 0x7f1934073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:16.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1932d9d700 1 -- 172.21.15.96:0/1154889992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c0095f0 con 0x7f1934100e30 2024-04-22T15:47:16.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1932d9d700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1934100e30 0x7f19340731e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f192400c8f0 tx=0x7f192400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:16.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1930d99700 1 -- 172.21.15.96:0/1154889992 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1924004a00 con 0x7f1934100e30 2024-04-22T15:47:16.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1930d99700 1 -- 172.21.15.96:0/1154889992 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19240054e0 con 0x7f1934100e30 2024-04-22T15:47:16.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19341a2d10 con 0x7f1934100e30 2024-04-22T15:47:16.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f1930d99700 1 -- 172.21.15.96:0/1154889992 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1924007cf0 con 0x7f1934100e30 2024-04-22T15:47:16.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.855+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19341a3070 con 0x7f1934100e30 2024-04-22T15:47:16.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.856+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1934109230 con 0x7f1934100e30 2024-04-22T15:47:16.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.857+0000 7f1930d99700 1 -- 172.21.15.96:0/1154889992 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1924005020 con 0x7f1934100e30 2024-04-22T15:47:16.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.857+0000 7f1930d99700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f192006ecc0 0x7f1920071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:16.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.857+0000 7f1930d99700 1 -- 172.21.15.96:0/1154889992 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1924013070 con 0x7f1934100e30 2024-04-22T15:47:16.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.857+0000 7f192a59c700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f192006ecc0 0x7f1920071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:16.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.858+0000 7f192a59c700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f192006ecc0 0x7f1920071180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1934074c20 tx=0x7f191c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:16.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:16.862+0000 7f1930d99700 1 -- 172.21.15.96:0/1154889992 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1924057a60 con 0x7f1934100e30 2024-04-22T15:47:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:16 smithi196 ceph-mon[81815]: pgmap v763: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:17.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.079+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f193410ddf0 con 0x7f192006ecc0 2024-04-22T15:47:17.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.080+0000 7f1930d99700 1 -- 172.21.15.96:0/1154889992 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f193410ddf0 con 0x7f192006ecc0 2024-04-22T15:47:17.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f192006ecc0 msgr2=0x7f1920071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f192006ecc0 0x7f1920071180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1934074c20 tx=0x7f191c01a040 comp rx=0 tx=0).stop 2024-04-22T15:47:17.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1934100e30 msgr2=0x7f19340731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1934100e30 0x7f19340731e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f192400c8f0 tx=0x7f192400cc00 comp rx=0 tx=0).stop 2024-04-22T15:47:17.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 shutdown_connections 2024-04-22T15:47:17.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f192006ecc0 0x7f1920071180 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1934100e30 0x7f19340731e0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 --2- 172.21.15.96:0/1154889992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19341017e0 0x7f1934073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 >> 172.21.15.96:0/1154889992 conn(0x7f19340fc700 msgr2=0x7f19340fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:17.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.082+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 shutdown_connections 2024-04-22T15:47:17.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.083+0000 7f193915b700 1 -- 172.21.15.96:0/1154889992 wait complete. 2024-04-22T15:47:17.095 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:47:17.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.187+0000 7fa0381e8700 1 -- 172.21.15.96:0/2485196631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 msgr2=0x7fa03010ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.187+0000 7fa0381e8700 1 --2- 172.21.15.96:0/2485196631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 0x7fa03010ac30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa020009910 tx=0x7fa020009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:17.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.187+0000 7fa0381e8700 1 -- 172.21.15.96:0/2485196631 shutdown_connections 2024-04-22T15:47:17.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.187+0000 7fa0381e8700 1 --2- 172.21.15.96:0/2485196631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 0x7fa03010ac30 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.187+0000 7fa0381e8700 1 --2- 172.21.15.96:0/2485196631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa030101e20 0x7fa030102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.187+0000 7fa0381e8700 1 -- 172.21.15.96:0/2485196631 >> 172.21.15.96:0/2485196631 conn(0x7fa0300fb610 msgr2=0x7fa0300fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:17.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.187+0000 7fa0381e8700 1 -- 172.21.15.96:0/2485196631 shutdown_connections 2024-04-22T15:47:17.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.187+0000 7fa0381e8700 1 -- 172.21.15.96:0/2485196631 wait complete. 2024-04-22T15:47:17.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa0381e8700 1 Processor -- start 2024-04-22T15:47:17.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa0381e8700 1 -- start start 2024-04-22T15:47:17.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa0381e8700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa030101e20 0x7fa03019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:17.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa0381e8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 0x7fa03019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:17.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa0381e8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03019d3a0 con 0x7fa030102740 2024-04-22T15:47:17.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa0381e8700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa030196980 con 0x7fa030101e20 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa035783700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 0x7fa03019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa035783700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 0x7fa03019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53640/0 (socket says 172.21.15.96:53640) 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa035783700 1 -- 172.21.15.96:0/1135402275 learned_addr learned my addr 172.21.15.96:0/1135402275 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa035f84700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa030101e20 0x7fa03019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa035783700 1 -- 172.21.15.96:0/1135402275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa030101e20 msgr2=0x7fa03019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa035783700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa030101e20 0x7fa03019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa035783700 1 -- 172.21.15.96:0/1135402275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0200095f0 con 0x7fa030102740 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.188+0000 7fa035f84700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa030101e20 0x7fa03019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.189+0000 7fa035783700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 0x7fa03019cd80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa020000c00 tx=0x7fa02000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.189+0000 7fa026ffd700 1 -- 172.21.15.96:0/1135402275 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa02001d070 con 0x7fa030102740 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.189+0000 7fa026ffd700 1 -- 172.21.15.96:0/1135402275 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa020005700 con 0x7fa030102740 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.189+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa030196c00 con 0x7fa030102740 2024-04-22T15:47:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.189+0000 7fa026ffd700 1 -- 172.21.15.96:0/1135402275 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa02000fe50 con 0x7fa030102740 2024-04-22T15:47:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.189+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0301970f0 con 0x7fa030102740 2024-04-22T15:47:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.190+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa030105540 con 0x7fa030102740 2024-04-22T15:47:17.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.190+0000 7fa026ffd700 1 -- 172.21.15.96:0/1135402275 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa02000f640 con 0x7fa030102740 2024-04-22T15:47:17.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.191+0000 7fa026ffd700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01c06ecd0 0x7fa01c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:17.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.191+0000 7fa035f84700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01c06ecd0 0x7fa01c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:17.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.191+0000 7fa026ffd700 1 -- 172.21.15.96:0/1135402275 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa020090330 con 0x7fa030102740 2024-04-22T15:47:17.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.191+0000 7fa035f84700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01c06ecd0 0x7fa01c071190 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa02c005950 tx=0x7fa02c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:17.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.196+0000 7fa026ffd700 1 -- 172.21.15.96:0/1135402275 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa02005cb60 con 0x7fa030102740 2024-04-22T15:47:17.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.415+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa03010dde0 con 0x7fa01c06ecd0 2024-04-22T15:47:17.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.416+0000 7fa026ffd700 1 -- 172.21.15.96:0/1135402275 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa03010dde0 con 0x7fa01c06ecd0 2024-04-22T15:47:17.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01c06ecd0 msgr2=0x7fa01c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01c06ecd0 0x7fa01c071190 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa02c005950 tx=0x7fa02c0058e0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 msgr2=0x7fa03019cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 0x7fa03019cd80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa020000c00 tx=0x7fa02000be20 comp rx=0 tx=0).stop 2024-04-22T15:47:17.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 shutdown_connections 2024-04-22T15:47:17.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa01c06ecd0 0x7fa01c071190 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa030101e20 0x7fa03019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 --2- 172.21.15.96:0/1135402275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa030102740 0x7fa03019cd80 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 >> 172.21.15.96:0/1135402275 conn(0x7fa0300fb610 msgr2=0x7fa030103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:17.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 shutdown_connections 2024-04-22T15:47:17.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.418+0000 7fa0381e8700 1 -- 172.21.15.96:0/1135402275 wait complete. 2024-04-22T15:47:17.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.521+0000 7fc18c0f7700 1 -- 172.21.15.96:0/3915830984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 msgr2=0x7fc184101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.521+0000 7fc18c0f7700 1 --2- 172.21.15.96:0/3915830984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 0x7fc184101200 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc180009910 tx=0x7fc180009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:17.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.521+0000 7fc18c0f7700 1 -- 172.21.15.96:0/3915830984 shutdown_connections 2024-04-22T15:47:17.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.521+0000 7fc18c0f7700 1 --2- 172.21.15.96:0/3915830984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1841017d0 0x7fc184105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.521+0000 7fc18c0f7700 1 --2- 172.21.15.96:0/3915830984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 0x7fc184101200 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.521+0000 7fc18c0f7700 1 -- 172.21.15.96:0/3915830984 >> 172.21.15.96:0/3915830984 conn(0x7fc1840fc6d0 msgr2=0x7fc1840feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:17.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.521+0000 7fc18c0f7700 1 -- 172.21.15.96:0/3915830984 shutdown_connections 2024-04-22T15:47:17.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.521+0000 7fc18c0f7700 1 -- 172.21.15.96:0/3915830984 wait complete. 2024-04-22T15:47:17.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.522+0000 7fc18c0f7700 1 Processor -- start 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.522+0000 7fc18c0f7700 1 -- start start 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.522+0000 7fc18c0f7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 0x7fc18419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.522+0000 7fc18c0f7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1841017d0 0x7fc18419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.522+0000 7fc18c0f7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18419d4a0 con 0x7fc184100e20 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.522+0000 7fc18c0f7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc184196a80 con 0x7fc1841017d0 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.522+0000 7fc189e93700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 0x7fc18419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.522+0000 7fc189e93700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 0x7fc18419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53662/0 (socket says 172.21.15.96:53662) 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.522+0000 7fc189e93700 1 -- 172.21.15.96:0/32116611 learned_addr learned my addr 172.21.15.96:0/32116611 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.523+0000 7fc189e93700 1 -- 172.21.15.96:0/32116611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1841017d0 msgr2=0x7fc18419ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.523+0000 7fc189e93700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1841017d0 0x7fc18419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.523+0000 7fc189e93700 1 -- 172.21.15.96:0/32116611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1800095f0 con 0x7fc184100e20 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.523+0000 7fc189e93700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 0x7fc18419c940 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc180006010 tx=0x7fc18000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:17.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.523+0000 7fc17affd700 1 -- 172.21.15.96:0/32116611 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc18001d070 con 0x7fc184100e20 2024-04-22T15:47:17.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.523+0000 7fc17affd700 1 -- 172.21.15.96:0/32116611 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc18000f460 con 0x7fc184100e20 2024-04-22T15:47:17.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.523+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc184196d00 con 0x7fc184100e20 2024-04-22T15:47:17.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.523+0000 7fc17affd700 1 -- 172.21.15.96:0/32116611 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc180017630 con 0x7fc184100e20 2024-04-22T15:47:17.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.523+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1841971f0 con 0x7fc184100e20 2024-04-22T15:47:17.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.524+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1841091c0 con 0x7fc184100e20 2024-04-22T15:47:17.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.524+0000 7fc17affd700 1 -- 172.21.15.96:0/32116611 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc18000f5d0 con 0x7fc184100e20 2024-04-22T15:47:17.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.525+0000 7fc17affd700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc17006ecc0 0x7fc170071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:17.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.525+0000 7fc17affd700 1 -- 172.21.15.96:0/32116611 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc180090340 con 0x7fc184100e20 2024-04-22T15:47:17.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.525+0000 7fc189692700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc17006ecc0 0x7fc170071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:17.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.525+0000 7fc189692700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc17006ecc0 0x7fc170071180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc184197c40 tx=0x7fc174006d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:17.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.530+0000 7fc17affd700 1 -- 172.21.15.96:0/32116611 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc18005cd10 con 0x7fc184100e20 2024-04-22T15:47:17.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.737+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc18410de60 con 0x7fc17006ecc0 2024-04-22T15:47:17.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.748+0000 7fc17affd700 1 -- 172.21.15.96:0/32116611 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc18410de60 con 0x7fc17006ecc0 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (26m) 3m ago 37m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (37m) 3m ago 37m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (36m) 3m ago 36m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (23m) 3m ago 37m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (23m) 3m ago 36m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (25m) 3m ago 36m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (19m) 3m ago 31m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (19m) 3m ago 31m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (18m) 3m ago 31m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (3m) 3m ago 31m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (28m) 3m ago 38m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (27m) 3m ago 36m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:47:17.751 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (24m) 3m ago 38m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (23m) 3m ago 36m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (26m) 3m ago 37m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (26m) 3m ago 36m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (22m) 3m ago 35m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (22m) 3m ago 34m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (21m) 3m ago 34m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (20m) 3m ago 34m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (20m) 3m ago 33m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (19m) 3m ago 33m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:47:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (26m) 3m ago 36m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:47:17.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc17006ecc0 msgr2=0x7fc170071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc17006ecc0 0x7fc170071180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc184197c40 tx=0x7fc174006d20 comp rx=0 tx=0).stop 2024-04-22T15:47:17.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 msgr2=0x7fc18419c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 0x7fc18419c940 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc180006010 tx=0x7fc18000be20 comp rx=0 tx=0).stop 2024-04-22T15:47:17.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 shutdown_connections 2024-04-22T15:47:17.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc17006ecc0 0x7fc170071180 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184100e20 0x7fc18419c940 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 --2- 172.21.15.96:0/32116611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1841017d0 0x7fc18419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 >> 172.21.15.96:0/32116611 conn(0x7fc1840fc6d0 msgr2=0x7fc1840fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:17.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 shutdown_connections 2024-04-22T15:47:17.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.750+0000 7fc18c0f7700 1 -- 172.21.15.96:0/32116611 wait complete. 2024-04-22T15:47:17.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.852+0000 7f57c4b13700 1 -- 172.21.15.96:0/1899542078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 msgr2=0x7f57c0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.852+0000 7f57c4b13700 1 --2- 172.21.15.96:0/1899542078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 0x7f57c0102140 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009910 tx=0x7f57b0009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:17.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.852+0000 7f57c4b13700 1 -- 172.21.15.96:0/1899542078 shutdown_connections 2024-04-22T15:47:17.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.852+0000 7f57c4b13700 1 --2- 172.21.15.96:0/1899542078 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c0102680 0x7f57c010ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.852+0000 7f57c4b13700 1 --2- 172.21.15.96:0/1899542078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 0x7f57c0102140 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.852+0000 7f57c4b13700 1 -- 172.21.15.96:0/1899542078 >> 172.21.15.96:0/1899542078 conn(0x7f57c0076740 msgr2=0x7f57c00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:17.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.852+0000 7f57c4b13700 1 -- 172.21.15.96:0/1899542078 shutdown_connections 2024-04-22T15:47:17.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.852+0000 7f57c4b13700 1 -- 172.21.15.96:0/1899542078 wait complete. 2024-04-22T15:47:17.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.853+0000 7f57c4b13700 1 Processor -- start 2024-04-22T15:47:17.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.853+0000 7f57c4b13700 1 -- start start 2024-04-22T15:47:17.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.853+0000 7f57c4b13700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 0x7f57c019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:17.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.853+0000 7f57c4b13700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c0102680 0x7f57c019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:17.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.853+0000 7f57c4b13700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c019d310 con 0x7f57c0101d60 2024-04-22T15:47:17.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.853+0000 7f57c4b13700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c0196980 con 0x7f57c0102680 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.853+0000 7f57be59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 0x7f57c019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.853+0000 7f57be59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 0x7f57c019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53678/0 (socket says 172.21.15.96:53678) 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.853+0000 7f57be59c700 1 -- 172.21.15.96:0/3298820733 learned_addr learned my addr 172.21.15.96:0/3298820733 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57b5d9b700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c0102680 0x7f57c019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57be59c700 1 -- 172.21.15.96:0/3298820733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c0102680 msgr2=0x7f57c019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57be59c700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c0102680 0x7f57c019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57be59c700 1 -- 172.21.15.96:0/3298820733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b00095f0 con 0x7f57c0101d60 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57b5d9b700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c0102680 0x7f57c019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57be59c700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 0x7f57c019c840 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f57b0006010 tx=0x7f57b000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57b7fff700 1 -- 172.21.15.96:0/3298820733 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57b001d070 con 0x7f57c0101d60 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c0196c00 con 0x7f57c0101d60 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57b7fff700 1 -- 172.21.15.96:0/3298820733 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57b000f460 con 0x7f57c0101d60 2024-04-22T15:47:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57b7fff700 1 -- 172.21.15.96:0/3298820733 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57b0017630 con 0x7f57c0101d60 2024-04-22T15:47:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.854+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c01970f0 con 0x7f57c0101d60 2024-04-22T15:47:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.855+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c0105480 con 0x7f57c0101d60 2024-04-22T15:47:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.856+0000 7f57b7fff700 1 -- 172.21.15.96:0/3298820733 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f57b000f5d0 con 0x7f57c0101d60 2024-04-22T15:47:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.856+0000 7f57b7fff700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.856+0000 7f57b7fff700 1 -- 172.21.15.96:0/3298820733 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f57b0090330 con 0x7f57c0101d60 2024-04-22T15:47:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.856+0000 7f57b5d9b700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.857+0000 7f57b5d9b700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f57a8005fd0 tx=0x7f57a8005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:17.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:17.861+0000 7f57b7fff700 1 -- 172.21.15.96:0/3298820733 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f57b005cc80 con 0x7f57c0101d60 2024-04-22T15:47:18.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.144+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f57c0197e40 con 0x7f57c0101d60 2024-04-22T15:47:18.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.145+0000 7f57b7fff700 1 -- 172.21.15.96:0/3298820733 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f57b005c810 con 0x7f57c0101d60 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:47:18.148 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:47:18.149 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:47:18.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:47:18.149 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:47:18.149 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:47:18.149 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:47:18.149 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 msgr2=0x7f57ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f57a8005fd0 tx=0x7f57a8005e30 comp rx=0 tx=0).stop 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 msgr2=0x7f57c019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 0x7f57c019c840 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f57b0006010 tx=0x7f57b000be20 comp rx=0 tx=0).stop 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 shutdown_connections 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c0101d60 0x7f57c019c840 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 --2- 172.21.15.96:0/3298820733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c0102680 0x7f57c019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 >> 172.21.15.96:0/3298820733 conn(0x7f57c0076740 msgr2=0x7f57c0103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 shutdown_connections 2024-04-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.147+0000 7f57c4b13700 1 -- 172.21.15.96:0/3298820733 wait complete. 2024-04-22T15:47:18.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.250+0000 7f0358f38700 1 -- 172.21.15.96:0/3807291259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 msgr2=0x7f0354105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.250+0000 7f0358f38700 1 --2- 172.21.15.96:0/3807291259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 0x7f0354105f40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f0344009910 tx=0x7f0344009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:18.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.250+0000 7f0358f38700 1 -- 172.21.15.96:0/3807291259 shutdown_connections 2024-04-22T15:47:18.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.250+0000 7f0358f38700 1 --2- 172.21.15.96:0/3807291259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540ffad0 0x7f03540fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.250+0000 7f0358f38700 1 --2- 172.21.15.96:0/3807291259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 0x7f0354105f40 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.250+0000 7f0358f38700 1 -- 172.21.15.96:0/3807291259 >> 172.21.15.96:0/3807291259 conn(0x7f03540fb600 msgr2=0x7f03540fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:18.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.250+0000 7f0358f38700 1 -- 172.21.15.96:0/3807291259 shutdown_connections 2024-04-22T15:47:18.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.250+0000 7f0358f38700 1 -- 172.21.15.96:0/3807291259 wait complete. 2024-04-22T15:47:18.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.251+0000 7f0358f38700 1 Processor -- start 2024-04-22T15:47:18.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.251+0000 7f0358f38700 1 -- start start 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.251+0000 7f0358f38700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540ffad0 0x7f0354197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.251+0000 7f0358f38700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 0x7f0354197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.251+0000 7f0358f38700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03541985d0 con 0x7f0354105b60 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.251+0000 7f0358f38700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0354198740 con 0x7f03540ffad0 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f0351d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 0x7f0354197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f0351d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 0x7f0354197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53688/0 (socket says 172.21.15.96:53688) 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f0351d9b700 1 -- 172.21.15.96:0/387003901 learned_addr learned my addr 172.21.15.96:0/387003901 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f035259c700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540ffad0 0x7f0354197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f0351d9b700 1 -- 172.21.15.96:0/387003901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540ffad0 msgr2=0x7f0354197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f0351d9b700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540ffad0 0x7f0354197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f0351d9b700 1 -- 172.21.15.96:0/387003901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03440095f0 con 0x7f0354105b60 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f0351d9b700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 0x7f0354197fb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f034800b4b0 tx=0x7f034800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f03437fe700 1 -- 172.21.15.96:0/387003901 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0348011800 con 0x7f0354105b60 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03541a0ae0 con 0x7f0354105b60 2024-04-22T15:47:18.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f03437fe700 1 -- 172.21.15.96:0/387003901 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0348011e20 con 0x7f0354105b60 2024-04-22T15:47:18.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f03437fe700 1 -- 172.21.15.96:0/387003901 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f034800f2e0 con 0x7f0354105b60 2024-04-22T15:47:18.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.252+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03541a0fd0 con 0x7f0354105b60 2024-04-22T15:47:18.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.253+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03541080d0 con 0x7f0354105b60 2024-04-22T15:47:18.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.254+0000 7f03437fe700 1 -- 172.21.15.96:0/387003901 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f034800f510 con 0x7f0354105b60 2024-04-22T15:47:18.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.254+0000 7f03437fe700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f033c06ecd0 0x7f033c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:18.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.254+0000 7f035259c700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f033c06ecd0 0x7f033c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:18.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.254+0000 7f03437fe700 1 -- 172.21.15.96:0/387003901 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f034808ea70 con 0x7f0354105b60 2024-04-22T15:47:18.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.255+0000 7f035259c700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f033c06ecd0 0x7f033c071190 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f034400b5c0 tx=0x7f034401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:18.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.260+0000 7f03437fe700 1 -- 172.21.15.96:0/387003901 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f03480577d0 con 0x7f0354105b60 2024-04-22T15:47:18.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.499+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f03541011d0 con 0x7f0354105b60 2024-04-22T15:47:18.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.500+0000 7f03437fe700 1 -- 172.21.15.96:0/387003901 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f034805ae90 con 0x7f0354105b60 2024-04-22T15:47:18.503 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:47:18.503 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:47:18.503 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:47:18.503 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:47:18.503 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:47:18.503 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:47:18.503 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:47:18.503 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:47:18.504 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:47:18.505 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:47:18.506 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:47:18.506 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:47:18.506 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:47:18.506 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:47:18.506 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f033c06ecd0 msgr2=0x7f033c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f033c06ecd0 0x7f033c071190 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f034400b5c0 tx=0x7f034401a040 comp rx=0 tx=0).stop 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 msgr2=0x7f0354197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 0x7f0354197fb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f034800b4b0 tx=0x7f034800b870 comp rx=0 tx=0).stop 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 shutdown_connections 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f033c06ecd0 0x7f033c071190 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540ffad0 0x7f0354197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 --2- 172.21.15.96:0/387003901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0354105b60 0x7f0354197fb0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 >> 172.21.15.96:0/387003901 conn(0x7f03540fb600 msgr2=0x7f03540fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 shutdown_connections 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.502+0000 7f0358f38700 1 -- 172.21.15.96:0/387003901 wait complete. 2024-04-22T15:47:18.507 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:47:18.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.606+0000 7f4776f6c700 1 -- 172.21.15.96:0/4048195304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 msgr2=0x7f47700fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.606+0000 7f4776f6c700 1 --2- 172.21.15.96:0/4048195304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 0x7f47700fff50 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f475c009910 tx=0x7f475c009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:18.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.606+0000 7f4776f6c700 1 -- 172.21.15.96:0/4048195304 shutdown_connections 2024-04-22T15:47:18.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.606+0000 7f4776f6c700 1 --2- 172.21.15.96:0/4048195304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 0x7f47700fff50 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.606+0000 7f4776f6c700 1 --2- 172.21.15.96:0/4048195304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4770105b60 0x7f4770105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.606+0000 7f4776f6c700 1 -- 172.21.15.96:0/4048195304 >> 172.21.15.96:0/4048195304 conn(0x7f47700fb5e0 msgr2=0x7f47700fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:18.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.606+0000 7f4776f6c700 1 -- 172.21.15.96:0/4048195304 shutdown_connections 2024-04-22T15:47:18.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.606+0000 7f4776f6c700 1 -- 172.21.15.96:0/4048195304 wait complete. 2024-04-22T15:47:18.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4776f6c700 1 Processor -- start 2024-04-22T15:47:18.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4776f6c700 1 -- start start 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4776f6c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 0x7f477019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4776f6c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4770105b60 0x7f477019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4776f6c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477019d370 con 0x7f47700ffad0 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4776f6c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4770196950 con 0x7f4770105b60 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4774d08700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 0x7f477019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4774d08700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 0x7f477019c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53706/0 (socket says 172.21.15.96:53706) 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4774d08700 1 -- 172.21.15.96:0/1402866572 learned_addr learned my addr 172.21.15.96:0/1402866572 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f476ffff700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4770105b60 0x7f477019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4774d08700 1 -- 172.21.15.96:0/1402866572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4770105b60 msgr2=0x7f477019cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4774d08700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4770105b60 0x7f477019cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4774d08700 1 -- 172.21.15.96:0/1402866572 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c0095f0 con 0x7f47700ffad0 2024-04-22T15:47:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.607+0000 7f4774d08700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 0x7f477019c810 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f476400d8d0 tx=0x7f476400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.608+0000 7f476dffb700 1 -- 172.21.15.96:0/1402866572 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4764009940 con 0x7f47700ffad0 2024-04-22T15:47:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.608+0000 7f476dffb700 1 -- 172.21.15.96:0/1402866572 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4764010460 con 0x7f47700ffad0 2024-04-22T15:47:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.608+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4770196c30 con 0x7f47700ffad0 2024-04-22T15:47:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.608+0000 7f476dffb700 1 -- 172.21.15.96:0/1402866572 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f476400f5f0 con 0x7f47700ffad0 2024-04-22T15:47:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.608+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4770197180 con 0x7f47700ffad0 2024-04-22T15:47:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.609+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4770109100 con 0x7f47700ffad0 2024-04-22T15:47:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.609+0000 7f476dffb700 1 -- 172.21.15.96:0/1402866572 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f47640105d0 con 0x7f47700ffad0 2024-04-22T15:47:18.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.610+0000 7f476dffb700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f476006ecc0 0x7f4760071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:18.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.610+0000 7f476dffb700 1 -- 172.21.15.96:0/1402866572 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f476408eed0 con 0x7f47700ffad0 2024-04-22T15:47:18.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.610+0000 7f476ffff700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f476006ecc0 0x7f4760071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:18.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.610+0000 7f476ffff700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f476006ecc0 0x7f4760071180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f475c00b5c0 tx=0x7f475c0052a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.614+0000 7f476dffb700 1 -- 172.21.15.96:0/1402866572 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f476405b760 con 0x7f47700ffad0 2024-04-22T15:47:18.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.837+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4770197e80 con 0x7f476006ecc0 2024-04-22T15:47:18.840 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:18 smithi096 ceph-mon[112443]: from='client.35404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:18.840 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:18 smithi096 ceph-mon[112443]: from='client.35408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:18.840 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:18 smithi096 ceph-mon[112443]: pgmap v764: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:18.840 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:18 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3298820733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:47:18.841 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:18 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/387003901' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:47:18.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.838+0000 7f476dffb700 1 -- 172.21.15.96:0/1402866572 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4770197e80 con 0x7f476006ecc0 2024-04-22T15:47:18.841 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:47:18.841 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:47:18.841 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:47:18.841 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:47:18.842 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:47:18.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f476006ecc0 msgr2=0x7f4760071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f476006ecc0 0x7f4760071180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f475c00b5c0 tx=0x7f475c0052a0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 msgr2=0x7f477019c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 0x7f477019c810 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f476400d8d0 tx=0x7f476400dc90 comp rx=0 tx=0).stop 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 shutdown_connections 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f476006ecc0 0x7f4760071180 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47700ffad0 0x7f477019c810 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 --2- 172.21.15.96:0/1402866572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4770105b60 0x7f477019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 >> 172.21.15.96:0/1402866572 conn(0x7f47700fb5e0 msgr2=0x7f4770069690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 shutdown_connections 2024-04-22T15:47:18.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.841+0000 7f4776f6c700 1 -- 172.21.15.96:0/1402866572 wait complete. 2024-04-22T15:47:18.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.944+0000 7f1d6de8c700 1 -- 172.21.15.96:0/4236053003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 msgr2=0x7f1d68105830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.944+0000 7f1d6de8c700 1 --2- 172.21.15.96:0/4236053003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 0x7f1d68105830 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00b0a0 tx=0x7f1d5c00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.944+0000 7f1d6de8c700 1 -- 172.21.15.96:0/4236053003 shutdown_connections 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.944+0000 7f1d6de8c700 1 --2- 172.21.15.96:0/4236053003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 0x7f1d68105830 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.944+0000 7f1d6de8c700 1 --2- 172.21.15.96:0/4236053003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d68100e30 0x7f1d68101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.944+0000 7f1d6de8c700 1 -- 172.21.15.96:0/4236053003 >> 172.21.15.96:0/4236053003 conn(0x7f1d680fc700 msgr2=0x7f1d680feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.944+0000 7f1d6de8c700 1 -- 172.21.15.96:0/4236053003 shutdown_connections 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.944+0000 7f1d6de8c700 1 -- 172.21.15.96:0/4236053003 wait complete. 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.944+0000 7f1d6de8c700 1 Processor -- start 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d6de8c700 1 -- start start 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d6de8c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d68100e30 0x7f1d680731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d6de8c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 0x7f1d68073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d6de8c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68073c60 con 0x7f1d681017e0 2024-04-22T15:47:18.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d6de8c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68073da0 con 0x7f1d68100e30 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d66ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 0x7f1d68073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d66ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 0x7f1d68073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53734/0 (socket says 172.21.15.96:53734) 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d66ffd700 1 -- 172.21.15.96:0/3936280021 learned_addr learned my addr 172.21.15.96:0/3936280021 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d677fe700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d68100e30 0x7f1d680731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d66ffd700 1 -- 172.21.15.96:0/3936280021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d68100e30 msgr2=0x7f1d680731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d66ffd700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d68100e30 0x7f1d680731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d66ffd700 1 -- 172.21.15.96:0/3936280021 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c009d00 con 0x7f1d681017e0 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.945+0000 7f1d66ffd700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 0x7f1d68073720 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c000f80 tx=0x7f1d5c007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.946+0000 7f1d64ff9700 1 -- 172.21.15.96:0/3936280021 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d5c00e040 con 0x7f1d681017e0 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.946+0000 7f1d64ff9700 1 -- 172.21.15.96:0/3936280021 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d5c003c10 con 0x7f1d681017e0 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.946+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d68074020 con 0x7f1d681017e0 2024-04-22T15:47:18.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.946+0000 7f1d64ff9700 1 -- 172.21.15.96:0/3936280021 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d5c01d6b0 con 0x7f1d681017e0 2024-04-22T15:47:18.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.946+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d681a2f70 con 0x7f1d681017e0 2024-04-22T15:47:18.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.947+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d68109230 con 0x7f1d681017e0 2024-04-22T15:47:18.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.947+0000 7f1d64ff9700 1 -- 172.21.15.96:0/3936280021 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1d5c004040 con 0x7f1d681017e0 2024-04-22T15:47:18.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.947+0000 7f1d64ff9700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d5006ecc0 0x7f1d50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:18.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.948+0000 7f1d64ff9700 1 -- 172.21.15.96:0/3936280021 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1d5c090520 con 0x7f1d681017e0 2024-04-22T15:47:18.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.948+0000 7f1d677fe700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d5006ecc0 0x7f1d50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:18.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.948+0000 7f1d677fe700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d5006ecc0 0x7f1d50071180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1d60005950 tx=0x7f1d6000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:18.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:18.953+0000 7f1d64ff9700 1 -- 172.21.15.96:0/3936280021 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1d5c059280 con 0x7f1d681017e0 2024-04-22T15:47:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:18 smithi196 ceph-mon[81815]: from='client.35404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:18 smithi196 ceph-mon[81815]: from='client.35408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:18 smithi196 ceph-mon[81815]: pgmap v764: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:18 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3298820733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:47:19.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:18 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/387003901' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:47:19.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.238+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1d68102310 con 0x7f1d681017e0 2024-04-22T15:47:19.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.238+0000 7f1d64ff9700 1 -- 172.21.15.96:0/3936280021 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f1d5c05c940 con 0x7f1d681017e0 2024-04-22T15:47:19.242 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:47:19.242 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:47:19.242 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:47:19.242 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:47:19.242 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:47:19.243 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:47:19.243 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:47:19.243 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:47:19.243 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:47:19.243 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:47:19.243 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:47:19.243 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:47:19.243 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:47:19.244 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:47:19.244 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:47:19.244 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:47:19.244 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:47:19.244 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:47:19.244 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:47:19.244 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:47:19.244 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:47:19.244 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:47:19.245 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:47:19.245 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:47:19.245 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:47:19.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.240+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d5006ecc0 msgr2=0x7f1d50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:19.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.240+0000 7f1d6de8c700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d5006ecc0 0x7f1d50071180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1d60005950 tx=0x7f1d6000b410 comp rx=0 tx=0).stop 2024-04-22T15:47:19.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.240+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 msgr2=0x7f1d68073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:19.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.240+0000 7f1d6de8c700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 0x7f1d68073720 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c000f80 tx=0x7f1d5c007b00 comp rx=0 tx=0).stop 2024-04-22T15:47:19.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.241+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 shutdown_connections 2024-04-22T15:47:19.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.241+0000 7f1d6de8c700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1d5006ecc0 0x7f1d50071180 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:19.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.241+0000 7f1d6de8c700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1d68100e30 0x7f1d680731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:19.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.241+0000 7f1d6de8c700 1 --2- 172.21.15.96:0/3936280021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d681017e0 0x7f1d68073720 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:19.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.241+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 >> 172.21.15.96:0/3936280021 conn(0x7f1d680fc700 msgr2=0x7f1d680fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:19.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.241+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 shutdown_connections 2024-04-22T15:47:19.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:19.241+0000 7f1d6de8c700 1 -- 172.21.15.96:0/3936280021 wait complete. 2024-04-22T15:47:19.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:19 smithi096 ceph-mon[112443]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:19.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:19 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3936280021' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:47:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:19 smithi196 ceph-mon[81815]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:19 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3936280021' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:47:20.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:20 smithi096 ceph-mon[112443]: from='client.35424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:20.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:20 smithi096 ceph-mon[112443]: pgmap v765: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:20 smithi196 ceph-mon[81815]: from='client.35424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:20 smithi196 ceph-mon[81815]: pgmap v765: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:22 smithi096 ceph-mon[112443]: pgmap v766: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:22 smithi196 ceph-mon[81815]: pgmap v766: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:23.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:24.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:24 smithi096 ceph-mon[112443]: pgmap v767: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:24 smithi196 ceph-mon[81815]: pgmap v767: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:26.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:26 smithi096 ceph-mon[112443]: pgmap v768: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:26 smithi196 ceph-mon[81815]: pgmap v768: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:28.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:28 smithi096 ceph-mon[112443]: pgmap v769: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:28 smithi196 ceph-mon[81815]: pgmap v769: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:30 smithi096 ceph-mon[112443]: pgmap v770: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:30 smithi196 ceph-mon[81815]: pgmap v770: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:32.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:32 smithi096 ceph-mon[112443]: pgmap v771: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:32 smithi196 ceph-mon[81815]: pgmap v771: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:34.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:34 smithi096 ceph-mon[112443]: pgmap v772: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:34 smithi196 ceph-mon[81815]: pgmap v772: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:36.884 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:36 smithi196 ceph-mon[81815]: pgmap v773: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:36.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:36 smithi096 ceph-mon[112443]: pgmap v773: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:38 smithi196 ceph-mon[81815]: pgmap v774: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:38.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:38 smithi096 ceph-mon[112443]: pgmap v774: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:38.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:40 smithi096 ceph-mon[112443]: pgmap v775: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:40 smithi196 ceph-mon[81815]: pgmap v775: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:42 smithi096 ceph-mon[112443]: pgmap v776: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:42 smithi196 ceph-mon[81815]: pgmap v776: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:43.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:43 smithi196 ceph-mon[81815]: pgmap v777: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:43.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:43 smithi096 ceph-mon[112443]: pgmap v777: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:46 smithi096 ceph-mon[112443]: pgmap v778: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:46 smithi196 ceph-mon[81815]: pgmap v778: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:48.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:48 smithi196 ceph-mon[81815]: pgmap v779: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:48 smithi096 ceph-mon[112443]: pgmap v779: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:49.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.357+0000 7fa746f5c700 1 -- 172.21.15.96:0/162504322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa740105ab0 msgr2=0x7fa740105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:49.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.357+0000 7fa746f5c700 1 --2- 172.21.15.96:0/162504322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa740105ab0 0x7fa740105e90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa73800b0a0 tx=0x7fa73800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.357+0000 7fa746f5c700 1 -- 172.21.15.96:0/162504322 shutdown_connections 2024-04-22T15:47:49.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.357+0000 7fa746f5c700 1 --2- 172.21.15.96:0/162504322 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7400ffab0 0x7fa7400fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.357+0000 7fa746f5c700 1 --2- 172.21.15.96:0/162504322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa740105ab0 0x7fa740105e90 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.357+0000 7fa746f5c700 1 -- 172.21.15.96:0/162504322 >> 172.21.15.96:0/162504322 conn(0x7fa7400fb5e0 msgr2=0x7fa7400fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.357+0000 7fa746f5c700 1 -- 172.21.15.96:0/162504322 shutdown_connections 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.357+0000 7fa746f5c700 1 -- 172.21.15.96:0/162504322 wait complete. 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa746f5c700 1 Processor -- start 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa746f5c700 1 -- start start 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa746f5c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400ffab0 0x7fa740198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa746f5c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa740105ab0 0x7fa740199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa746f5c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7401998d0 con 0x7fa7400ffab0 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa746f5c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74019c540 con 0x7fa740105ab0 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa744cf8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400ffab0 0x7fa740198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa744cf8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400ffab0 0x7fa740198d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49404/0 (socket says 172.21.15.96:49404) 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa744cf8700 1 -- 172.21.15.96:0/3808569787 learned_addr learned my addr 172.21.15.96:0/3808569787 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:49.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa73ffff700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa740105ab0 0x7fa740199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa744cf8700 1 -- 172.21.15.96:0/3808569787 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa740105ab0 msgr2=0x7fa740199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa744cf8700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa740105ab0 0x7fa740199240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa744cf8700 1 -- 172.21.15.96:0/3808569787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa738009d00 con 0x7fa7400ffab0 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.358+0000 7fa73ffff700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa740105ab0 0x7fa740199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.359+0000 7fa744cf8700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400ffab0 0x7fa740198d00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa738000f80 tx=0x7fa738007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.359+0000 7fa73dffb700 1 -- 172.21.15.96:0/3808569787 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73800e040 con 0x7fa7400ffab0 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.359+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74019c7c0 con 0x7fa7400ffab0 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.359+0000 7fa73dffb700 1 -- 172.21.15.96:0/3808569787 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa738003c10 con 0x7fa7400ffab0 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.359+0000 7fa73dffb700 1 -- 172.21.15.96:0/3808569787 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73801d6b0 con 0x7fa7400ffab0 2024-04-22T15:47:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.359+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74019ccb0 con 0x7fa7400ffab0 2024-04-22T15:47:49.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.360+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7401011f0 con 0x7fa7400ffab0 2024-04-22T15:47:49.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.361+0000 7fa73dffb700 1 -- 172.21.15.96:0/3808569787 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa738004040 con 0x7fa7400ffab0 2024-04-22T15:47:49.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.361+0000 7fa73dffb700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa72c06ecc0 0x7fa72c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:49.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.361+0000 7fa73ffff700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa72c06ecc0 0x7fa72c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:49.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.361+0000 7fa73dffb700 1 -- 172.21.15.96:0/3808569787 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa738090640 con 0x7fa7400ffab0 2024-04-22T15:47:49.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.362+0000 7fa73ffff700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa72c06ecc0 0x7fa72c071180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa730005fd0 tx=0x7fa730005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:49.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.366+0000 7fa73dffb700 1 -- 172.21.15.96:0/3808569787 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa7380593a0 con 0x7fa7400ffab0 2024-04-22T15:47:49.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.587+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa74010ccb0 con 0x7fa72c06ecc0 2024-04-22T15:47:49.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.588+0000 7fa73dffb700 1 -- 172.21.15.96:0/3808569787 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa74010ccb0 con 0x7fa72c06ecc0 2024-04-22T15:47:49.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa72c06ecc0 msgr2=0x7fa72c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:49.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa72c06ecc0 0x7fa72c071180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa730005fd0 tx=0x7fa730005ee0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400ffab0 msgr2=0x7fa740198d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:49.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400ffab0 0x7fa740198d00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa738000f80 tx=0x7fa738007bd0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 shutdown_connections 2024-04-22T15:47:49.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa72c06ecc0 0x7fa72c071180 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400ffab0 0x7fa740198d00 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 --2- 172.21.15.96:0/3808569787 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa740105ab0 0x7fa740199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 >> 172.21.15.96:0/3808569787 conn(0x7fa7400fb5e0 msgr2=0x7fa7400691f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:49.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 shutdown_connections 2024-04-22T15:47:49.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.590+0000 7fa746f5c700 1 -- 172.21.15.96:0/3808569787 wait complete. 2024-04-22T15:47:49.605 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:47:49.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.696+0000 7f3ff4f25700 1 -- 172.21.15.96:0/4281119060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff01017c0 msgr2=0x7f3ff0105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:49.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.696+0000 7f3ff4f25700 1 --2- 172.21.15.96:0/4281119060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff01017c0 0x7f3ff0105810 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009910 tx=0x7f3fe4009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:49.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.696+0000 7f3ff4f25700 1 -- 172.21.15.96:0/4281119060 shutdown_connections 2024-04-22T15:47:49.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.696+0000 7f3ff4f25700 1 --2- 172.21.15.96:0/4281119060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff01017c0 0x7f3ff0105810 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.696+0000 7f3ff4f25700 1 --2- 172.21.15.96:0/4281119060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ff0100e10 0x7f3ff01011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.696+0000 7f3ff4f25700 1 -- 172.21.15.96:0/4281119060 >> 172.21.15.96:0/4281119060 conn(0x7f3ff00fc6e0 msgr2=0x7f3ff00feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:49.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.696+0000 7f3ff4f25700 1 -- 172.21.15.96:0/4281119060 shutdown_connections 2024-04-22T15:47:49.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.696+0000 7f3ff4f25700 1 -- 172.21.15.96:0/4281119060 wait complete. 2024-04-22T15:47:49.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3ff4f25700 1 Processor -- start 2024-04-22T15:47:49.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3ff4f25700 1 -- start start 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3ff4f25700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff0100e10 0x7f3ff019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3ff4f25700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ff01017c0 0x7f3ff019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3ff4f25700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff019d2f0 con 0x7f3ff0100e10 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3ff4f25700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff01969b0 con 0x7f3ff01017c0 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3fee59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff0100e10 0x7f3ff019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3fee59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff0100e10 0x7f3ff019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49420/0 (socket says 172.21.15.96:49420) 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3fee59c700 1 -- 172.21.15.96:0/2952659541 learned_addr learned my addr 172.21.15.96:0/2952659541 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.697+0000 7f3fedd9b700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ff01017c0 0x7f3ff019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.698+0000 7f3fee59c700 1 -- 172.21.15.96:0/2952659541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ff01017c0 msgr2=0x7f3ff019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.698+0000 7f3fee59c700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ff01017c0 0x7f3ff019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.698+0000 7f3fee59c700 1 -- 172.21.15.96:0/2952659541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe40095f0 con 0x7f3ff0100e10 2024-04-22T15:47:49.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.698+0000 7f3fee59c700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff0100e10 0x7f3ff019c820 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c930 tx=0x7f3fe000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:49.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.698+0000 7f3fdf7fe700 1 -- 172.21.15.96:0/2952659541 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fe0007ad0 con 0x7f3ff0100e10 2024-04-22T15:47:49.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.698+0000 7f3fdf7fe700 1 -- 172.21.15.96:0/2952659541 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fe000ce80 con 0x7f3ff0100e10 2024-04-22T15:47:49.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.698+0000 7f3fdf7fe700 1 -- 172.21.15.96:0/2952659541 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fe0018630 con 0x7f3ff0100e10 2024-04-22T15:47:49.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.698+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff0196c90 con 0x7f3ff0100e10 2024-04-22T15:47:49.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.698+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff01971e0 con 0x7f3ff0100e10 2024-04-22T15:47:49.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.699+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff0109210 con 0x7f3ff0100e10 2024-04-22T15:47:49.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.700+0000 7f3fdf7fe700 1 -- 172.21.15.96:0/2952659541 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3fe0007c30 con 0x7f3ff0100e10 2024-04-22T15:47:49.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.700+0000 7f3fdf7fe700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3fd806ecc0 0x7f3fd8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:49.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.700+0000 7f3fdf7fe700 1 -- 172.21.15.96:0/2952659541 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3fe008ee20 con 0x7f3ff0100e10 2024-04-22T15:47:49.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.700+0000 7f3fedd9b700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3fd806ecc0 0x7f3fd8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:49.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.701+0000 7f3fedd9b700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3fd806ecc0 0x7f3fd8071180 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0197b20 tx=0x7f3fe401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:49.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.705+0000 7f3fdf7fe700 1 -- 172.21.15.96:0/2952659541 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3fe005b730 con 0x7f3ff0100e10 2024-04-22T15:47:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.924+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ff0197e90 con 0x7f3fd806ecc0 2024-04-22T15:47:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.925+0000 7f3fdf7fe700 1 -- 172.21.15.96:0/2952659541 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3ff0197e90 con 0x7f3fd806ecc0 2024-04-22T15:47:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3fd806ecc0 msgr2=0x7f3fd8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3fd806ecc0 0x7f3fd8071180 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0197b20 tx=0x7f3fe401a040 comp rx=0 tx=0).stop 2024-04-22T15:47:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff0100e10 msgr2=0x7f3ff019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff0100e10 0x7f3ff019c820 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c930 tx=0x7f3fe000cc40 comp rx=0 tx=0).stop 2024-04-22T15:47:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 shutdown_connections 2024-04-22T15:47:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3fd806ecc0 0x7f3fd8071180 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ff0100e10 0x7f3ff019c820 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 --2- 172.21.15.96:0/2952659541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ff01017c0 0x7f3ff019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 >> 172.21.15.96:0/2952659541 conn(0x7f3ff00fc6e0 msgr2=0x7f3ff00fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:49.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 shutdown_connections 2024-04-22T15:47:49.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:49.927+0000 7f3ff4f25700 1 -- 172.21.15.96:0/2952659541 wait complete. 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.031+0000 7f0c3999f700 1 -- 172.21.15.96:0/1789897890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c340ffaa0 msgr2=0x7f0c340fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.031+0000 7f0c3999f700 1 --2- 172.21.15.96:0/1789897890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c340ffaa0 0x7f0c340fff20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009910 tx=0x7f0c24009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.031+0000 7f0c3999f700 1 -- 172.21.15.96:0/1789897890 shutdown_connections 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.031+0000 7f0c3999f700 1 --2- 172.21.15.96:0/1789897890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c340ffaa0 0x7f0c340fff20 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.031+0000 7f0c3999f700 1 --2- 172.21.15.96:0/1789897890 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c34105aa0 0x7f0c34105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.031+0000 7f0c3999f700 1 -- 172.21.15.96:0/1789897890 >> 172.21.15.96:0/1789897890 conn(0x7f0c34076740 msgr2=0x7f0c340fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.031+0000 7f0c3999f700 1 -- 172.21.15.96:0/1789897890 shutdown_connections 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.031+0000 7f0c3999f700 1 -- 172.21.15.96:0/1789897890 wait complete. 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c3999f700 1 Processor -- start 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c3999f700 1 -- start start 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c3999f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c340ffaa0 0x7f0c3419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:50.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c3999f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34105aa0 0x7f0c3419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c3999f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3419d320 con 0x7f0c34105aa0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c3999f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c34196900 con 0x7f0c340ffaa0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c327fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34105aa0 0x7f0c3419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c327fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34105aa0 0x7f0c3419cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49430/0 (socket says 172.21.15.96:49430) 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c327fc700 1 -- 172.21.15.96:0/2344265158 learned_addr learned my addr 172.21.15.96:0/2344265158 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c32ffd700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c340ffaa0 0x7f0c3419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c327fc700 1 -- 172.21.15.96:0/2344265158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c340ffaa0 msgr2=0x7f0c3419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c327fc700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c340ffaa0 0x7f0c3419c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.032+0000 7f0c327fc700 1 -- 172.21.15.96:0/2344265158 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c240095f0 con 0x7f0c34105aa0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.033+0000 7f0c327fc700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34105aa0 0x7f0c3419cd00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400bd70 tx=0x7f0c2400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.033+0000 7f0c3899d700 1 -- 172.21.15.96:0/2344265158 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c2401d070 con 0x7f0c34105aa0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.033+0000 7f0c3899d700 1 -- 172.21.15.96:0/2344265158 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c2400f460 con 0x7f0c34105aa0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.033+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c34196b80 con 0x7f0c34105aa0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.033+0000 7f0c3899d700 1 -- 172.21.15.96:0/2344265158 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c24017630 con 0x7f0c34105aa0 2024-04-22T15:47:50.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.033+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c34197070 con 0x7f0c34105aa0 2024-04-22T15:47:50.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.034+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c341011e0 con 0x7f0c34105aa0 2024-04-22T15:47:50.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.034+0000 7f0c3899d700 1 -- 172.21.15.96:0/2344265158 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0c2400f5d0 con 0x7f0c34105aa0 2024-04-22T15:47:50.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.035+0000 7f0c3899d700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:50.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.035+0000 7f0c32ffd700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:50.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.035+0000 7f0c3899d700 1 -- 172.21.15.96:0/2344265158 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0c240902e0 con 0x7f0c34105aa0 2024-04-22T15:47:50.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.035+0000 7f0c32ffd700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0c28005950 tx=0x7f0c28005cf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:50.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.040+0000 7f0c3899d700 1 -- 172.21.15.96:0/2344265158 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0c2405cbf0 con 0x7f0c34105aa0 2024-04-22T15:47:50.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.250+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c3410cca0 con 0x7f0c1c06ecc0 2024-04-22T15:47:50.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.261+0000 7f0c3899d700 1 -- 172.21.15.96:0/2344265158 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0c3410cca0 con 0x7f0c1c06ecc0 2024-04-22T15:47:50.264 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:47:50.264 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (26m) 4m ago 37m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:47:50.264 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (38m) 4m ago 38m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:47:50.264 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (36m) 4m ago 36m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:47:50.264 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (23m) 4m ago 38m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:47:50.264 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (23m) 4m ago 36m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:47:50.264 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (26m) 4m ago 37m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (19m) 4m ago 32m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (20m) 4m ago 32m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (19m) 4m ago 32m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (4m) 4m ago 32m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (28m) 4m ago 39m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (27m) 4m ago 36m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (24m) 4m ago 39m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (24m) 4m ago 36m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (27m) 4m ago 37m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (27m) 4m ago 36m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (23m) 4m ago 35m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (22m) 4m ago 35m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:47:50.265 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (22m) 4m ago 34m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:47:50.266 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (21m) 4m ago 34m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:47:50.266 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (20m) 4m ago 34m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:47:50.266 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (20m) 4m ago 33m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:47:50.266 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (26m) 4m ago 37m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:47:50.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 msgr2=0x7f0c1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0c28005950 tx=0x7f0c28005cf0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34105aa0 msgr2=0x7f0c3419cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34105aa0 0x7f0c3419cd00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400bd70 tx=0x7f0c2400be50 comp rx=0 tx=0).stop 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 shutdown_connections 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c340ffaa0 0x7f0c3419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 --2- 172.21.15.96:0/2344265158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34105aa0 0x7f0c3419cd00 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 >> 172.21.15.96:0/2344265158 conn(0x7f0c34076740 msgr2=0x7f0c34068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 shutdown_connections 2024-04-22T15:47:50.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.263+0000 7f0c3999f700 1 -- 172.21.15.96:0/2344265158 wait complete. 2024-04-22T15:47:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.364+0000 7f50e0211700 1 -- 172.21.15.96:0/874678984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 msgr2=0x7f50d8104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.365+0000 7f50e0211700 1 --2- 172.21.15.96:0/874678984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 0x7f50d8104ed0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009910 tx=0x7f50d4009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.365+0000 7f50e0211700 1 -- 172.21.15.96:0/874678984 shutdown_connections 2024-04-22T15:47:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.365+0000 7f50e0211700 1 --2- 172.21.15.96:0/874678984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50d81001e0 0x7f50d8100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.365+0000 7f50e0211700 1 --2- 172.21.15.96:0/874678984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 0x7f50d8104ed0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.365+0000 7f50e0211700 1 -- 172.21.15.96:0/874678984 >> 172.21.15.96:0/874678984 conn(0x7f50d8075f30 msgr2=0x7f50d80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.365+0000 7f50e0211700 1 -- 172.21.15.96:0/874678984 shutdown_connections 2024-04-22T15:47:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.365+0000 7f50e0211700 1 -- 172.21.15.96:0/874678984 wait complete. 2024-04-22T15:47:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.365+0000 7f50e0211700 1 Processor -- start 2024-04-22T15:47:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.365+0000 7f50e0211700 1 -- start start 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50e0211700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50d81001e0 0x7f50d80fb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50e0211700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 0x7f50d80fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50e0211700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d80fbe00 con 0x7f50d8104af0 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50e0211700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d80fbf40 con 0x7f50d81001e0 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50dd7ac700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 0x7f50d80fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50dd7ac700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 0x7f50d80fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49454/0 (socket says 172.21.15.96:49454) 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50dd7ac700 1 -- 172.21.15.96:0/4174858112 learned_addr learned my addr 172.21.15.96:0/4174858112 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50ddfad700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50d81001e0 0x7f50d80fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50dd7ac700 1 -- 172.21.15.96:0/4174858112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50d81001e0 msgr2=0x7f50d80fb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50dd7ac700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50d81001e0 0x7f50d80fb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50dd7ac700 1 -- 172.21.15.96:0/4174858112 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d40095f0 con 0x7f50d8104af0 2024-04-22T15:47:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50dd7ac700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 0x7f50d80fb8c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f50c800c8a0 tx=0x7f50c800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:50.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50ceffd700 1 -- 172.21.15.96:0/4174858112 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50c80078e0 con 0x7f50d8104af0 2024-04-22T15:47:50.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50ceffd700 1 -- 172.21.15.96:0/4174858112 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50c800f450 con 0x7f50d8104af0 2024-04-22T15:47:50.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.366+0000 7f50ceffd700 1 -- 172.21.15.96:0/4174858112 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50c800e5e0 con 0x7f50d8104af0 2024-04-22T15:47:50.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.367+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d81a2c20 con 0x7f50d8104af0 2024-04-22T15:47:50.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.367+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d81a2f80 con 0x7f50d8104af0 2024-04-22T15:47:50.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.368+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d8101400 con 0x7f50d8104af0 2024-04-22T15:47:50.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.368+0000 7f50ceffd700 1 -- 172.21.15.96:0/4174858112 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f50c800f5c0 con 0x7f50d8104af0 2024-04-22T15:47:50.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.368+0000 7f50ceffd700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f50c406ecc0 0x7f50c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:50.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.369+0000 7f50ceffd700 1 -- 172.21.15.96:0/4174858112 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f50c808ee80 con 0x7f50d8104af0 2024-04-22T15:47:50.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.369+0000 7f50ddfad700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f50c406ecc0 0x7f50c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:50.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.369+0000 7f50ddfad700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f50c406ecc0 0x7f50c4071180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f50d4000c00 tx=0x7f50d401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:50.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.373+0000 7f50ceffd700 1 -- 172.21.15.96:0/4174858112 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f50c8057be0 con 0x7f50d8104af0 2024-04-22T15:47:50.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.658+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f50d80fc100 con 0x7f50d8104af0 2024-04-22T15:47:50.661 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:50 smithi096 ceph-mon[112443]: pgmap v780: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:50.661 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:50 smithi096 ceph-mon[112443]: from='client.35432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:50.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.658+0000 7f50ceffd700 1 -- 172.21.15.96:0/4174858112 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f50c805b2a0 con 0x7f50d8104af0 2024-04-22T15:47:50.662 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:47:50.662 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:47:50.662 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:47:50.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:47:50.662 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:47:50.662 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:47:50.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:47:50.662 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:47:50.663 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:47:50.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:47:50.663 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:47:50.663 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:47:50.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:47:50.663 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:47:50.663 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:47:50.663 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:47:50.663 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:47:50.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.660+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f50c406ecc0 msgr2=0x7f50c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.660+0000 7f50e0211700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f50c406ecc0 0x7f50c4071180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f50d4000c00 tx=0x7f50d401a040 comp rx=0 tx=0).stop 2024-04-22T15:47:50.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.660+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 msgr2=0x7f50d80fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.660+0000 7f50e0211700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 0x7f50d80fb8c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f50c800c8a0 tx=0x7f50c800cbb0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.660+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 shutdown_connections 2024-04-22T15:47:50.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.660+0000 7f50e0211700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f50c406ecc0 0x7f50c4071180 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.660+0000 7f50e0211700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50d81001e0 0x7f50d80fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.660+0000 7f50e0211700 1 --2- 172.21.15.96:0/4174858112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50d8104af0 0x7f50d80fb8c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.660+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 >> 172.21.15.96:0/4174858112 conn(0x7f50d8075f30 msgr2=0x7f50d80ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:50.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.661+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 shutdown_connections 2024-04-22T15:47:50.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.661+0000 7f50e0211700 1 -- 172.21.15.96:0/4174858112 wait complete. 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.763+0000 7f332a076700 1 -- 172.21.15.96:0/2540693094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 msgr2=0x7f33240ff750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.763+0000 7f332a076700 1 --2- 172.21.15.96:0/2540693094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 0x7f33240ff750 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3318009910 tx=0x7f3318009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.763+0000 7f332a076700 1 -- 172.21.15.96:0/2540693094 shutdown_connections 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.763+0000 7f332a076700 1 --2- 172.21.15.96:0/2540693094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33240ffd20 0x7f3324105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.763+0000 7f332a076700 1 --2- 172.21.15.96:0/2540693094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 0x7f33240ff750 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.763+0000 7f332a076700 1 -- 172.21.15.96:0/2540693094 >> 172.21.15.96:0/2540693094 conn(0x7f3324075f30 msgr2=0x7f33240fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.764+0000 7f332a076700 1 -- 172.21.15.96:0/2540693094 shutdown_connections 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.764+0000 7f332a076700 1 -- 172.21.15.96:0/2540693094 wait complete. 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.764+0000 7f332a076700 1 Processor -- start 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.764+0000 7f332a076700 1 -- start start 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f332a076700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 0x7f332419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:50.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f332a076700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33240ffd20 0x7f332419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:50.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f332a076700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332419d3c0 con 0x7f33240ff370 2024-04-22T15:47:50.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f332a076700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33241969f0 con 0x7f33240ffd20 2024-04-22T15:47:50.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f33237fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 0x7f332419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:50.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f33237fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 0x7f332419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46730/0 (socket says 172.21.15.96:46730) 2024-04-22T15:47:50.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f33237fe700 1 -- 172.21.15.96:0/766509738 learned_addr learned my addr 172.21.15.96:0/766509738 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:50.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f3322ffd700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33240ffd20 0x7f332419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f33237fe700 1 -- 172.21.15.96:0/766509738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33240ffd20 msgr2=0x7f332419cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f33237fe700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33240ffd20 0x7f332419cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f33237fe700 1 -- 172.21.15.96:0/766509738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180095f0 con 0x7f33240ff370 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f33237fe700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 0x7f332419c860 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f331800bd70 tx=0x7f331800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f3320ff9700 1 -- 172.21.15.96:0/766509738 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f331801d070 con 0x7f33240ff370 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3324196c70 con 0x7f33240ff370 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f3320ff9700 1 -- 172.21.15.96:0/766509738 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f331800f460 con 0x7f33240ff370 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f3320ff9700 1 -- 172.21.15.96:0/766509738 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3318017630 con 0x7f33240ff370 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.765+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3324197160 con 0x7f33240ff370 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.766+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33241091d0 con 0x7f33240ff370 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.767+0000 7f3320ff9700 1 -- 172.21.15.96:0/766509738 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f331800f5d0 con 0x7f33240ff370 2024-04-22T15:47:50.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.767+0000 7f3320ff9700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f330c06ecc0 0x7f330c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:50.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.767+0000 7f3320ff9700 1 -- 172.21.15.96:0/766509738 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f33180903c0 con 0x7f33240ff370 2024-04-22T15:47:50.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.767+0000 7f3322ffd700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f330c06ecc0 0x7f330c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:50.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.768+0000 7f3322ffd700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f330c06ecc0 0x7f330c071180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3314005fd0 tx=0x7f331400a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:50.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:50.772+0000 7f3320ff9700 1 -- 172.21.15.96:0/766509738 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f331805cc50 con 0x7f33240ff370 2024-04-22T15:47:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:50 smithi196 ceph-mon[81815]: pgmap v780: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:50 smithi196 ceph-mon[81815]: from='client.35432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:51.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.017+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3324197e60 con 0x7f33240ff370 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.018+0000 7f3320ff9700 1 -- 172.21.15.96:0/766509738 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f331805c7e0 con 0x7f33240ff370 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:47:51.021 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:47:51.022 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:47:51.023 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:47:51.024 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:47:51.024 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:47:51.024 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:47:51.024 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:47:51.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f330c06ecc0 msgr2=0x7f330c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f330c06ecc0 0x7f330c071180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3314005fd0 tx=0x7f331400a560 comp rx=0 tx=0).stop 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 msgr2=0x7f332419c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 0x7f332419c860 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f331800bd70 tx=0x7f331800be50 comp rx=0 tx=0).stop 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 shutdown_connections 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f330c06ecc0 0x7f330c071180 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33240ff370 0x7f332419c860 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 --2- 172.21.15.96:0/766509738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33240ffd20 0x7f332419cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 >> 172.21.15.96:0/766509738 conn(0x7f3324075f30 msgr2=0x7f33241046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 shutdown_connections 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.020+0000 7f332a076700 1 -- 172.21.15.96:0/766509738 wait complete. 2024-04-22T15:47:51.025 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:47:51.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.124+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/1770654505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc0ffaa0 msgr2=0x7f6fdc0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.124+0000 7f6fe3c2f700 1 --2- 172.21.15.96:0/1770654505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc0ffaa0 0x7f6fdc0fff20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009910 tx=0x7f6fd8009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:51.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.124+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/1770654505 shutdown_connections 2024-04-22T15:47:51.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.124+0000 7f6fe3c2f700 1 --2- 172.21.15.96:0/1770654505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc0ffaa0 0x7f6fdc0fff20 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.124+0000 7f6fe3c2f700 1 --2- 172.21.15.96:0/1770654505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fdc105aa0 0x7f6fdc105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.124+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/1770654505 >> 172.21.15.96:0/1770654505 conn(0x7f6fdc076740 msgr2=0x7f6fdc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:51.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.124+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/1770654505 shutdown_connections 2024-04-22T15:47:51.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.124+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/1770654505 wait complete. 2024-04-22T15:47:51.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe3c2f700 1 Processor -- start 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe3c2f700 1 -- start start 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe3c2f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fdc0ffaa0 0x7f6fdc0fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe3c2f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc105aa0 0x7f6fdc0fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe3c2f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc0fdbf0 con 0x7f6fdc105aa0 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe3c2f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc0fc6a0 con 0x7f6fdc0ffaa0 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe11ca700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc105aa0 0x7f6fdc0fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe11ca700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc105aa0 0x7f6fdc0fc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46742/0 (socket says 172.21.15.96:46742) 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe11ca700 1 -- 172.21.15.96:0/3129688595 learned_addr learned my addr 172.21.15.96:0/3129688595 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe11ca700 1 -- 172.21.15.96:0/3129688595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fdc0ffaa0 msgr2=0x7f6fdc0fbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe11ca700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fdc0ffaa0 0x7f6fdc0fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe11ca700 1 -- 172.21.15.96:0/3129688595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80095f0 con 0x7f6fdc105aa0 2024-04-22T15:47:51.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fe11ca700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc105aa0 0x7f6fdc0fc160 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8006010 tx=0x7f6fd800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:51.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.125+0000 7f6fd2ffd700 1 -- 172.21.15.96:0/3129688595 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd801d070 con 0x7f6fdc105aa0 2024-04-22T15:47:51.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.126+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc0fc8f0 con 0x7f6fdc105aa0 2024-04-22T15:47:51.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.126+0000 7f6fd2ffd700 1 -- 172.21.15.96:0/3129688595 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fd800f460 con 0x7f6fdc105aa0 2024-04-22T15:47:51.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.126+0000 7f6fd2ffd700 1 -- 172.21.15.96:0/3129688595 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd8017630 con 0x7f6fdc105aa0 2024-04-22T15:47:51.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.126+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc1a2e90 con 0x7f6fdc105aa0 2024-04-22T15:47:51.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.126+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fdc1011e0 con 0x7f6fdc105aa0 2024-04-22T15:47:51.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.127+0000 7f6fd2ffd700 1 -- 172.21.15.96:0/3129688595 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6fd800f5d0 con 0x7f6fdc105aa0 2024-04-22T15:47:51.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.127+0000 7f6fd2ffd700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6fc806ecc0 0x7f6fc8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:51.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.127+0000 7f6fd2ffd700 1 -- 172.21.15.96:0/3129688595 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6fd80903a0 con 0x7f6fdc105aa0 2024-04-22T15:47:51.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.128+0000 7f6fe19cb700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6fc806ecc0 0x7f6fc8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:51.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.128+0000 7f6fe19cb700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6fc806ecc0 0x7f6fc8071180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc005fd0 tx=0x7f6fcc005d00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:51.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.132+0000 7f6fd2ffd700 1 -- 172.21.15.96:0/3129688595 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6fd80591c0 con 0x7f6fdc105aa0 2024-04-22T15:47:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.350+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fdc10cdb0 con 0x7f6fc806ecc0 2024-04-22T15:47:51.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.352+0000 7f6fd2ffd700 1 -- 172.21.15.96:0/3129688595 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6fdc10cdb0 con 0x7f6fc806ecc0 2024-04-22T15:47:51.354 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:47:51.355 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:47:51.356 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:47:51.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6fc806ecc0 msgr2=0x7f6fc8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6fc806ecc0 0x7f6fc8071180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc005fd0 tx=0x7f6fcc005d00 comp rx=0 tx=0).stop 2024-04-22T15:47:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc105aa0 msgr2=0x7f6fdc0fc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc105aa0 0x7f6fdc0fc160 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8006010 tx=0x7f6fd800be20 comp rx=0 tx=0).stop 2024-04-22T15:47:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 shutdown_connections 2024-04-22T15:47:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6fc806ecc0 0x7f6fc8071180 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fdc0ffaa0 0x7f6fdc0fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 --2- 172.21.15.96:0/3129688595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fdc105aa0 0x7f6fdc0fc160 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 >> 172.21.15.96:0/3129688595 conn(0x7f6fdc076740 msgr2=0x7f6fdc104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 shutdown_connections 2024-04-22T15:47:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.354+0000 7f6fe3c2f700 1 -- 172.21.15.96:0/3129688595 wait complete. 2024-04-22T15:47:51.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.457+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2697270325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 msgr2=0x7f98a80fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.457+0000 7f98ad3e9700 1 --2- 172.21.15.96:0/2697270325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 0x7f98a80fff30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9898009910 tx=0x7f9898009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:51.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.457+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2697270325 shutdown_connections 2024-04-22T15:47:51.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.457+0000 7f98ad3e9700 1 --2- 172.21.15.96:0/2697270325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 0x7f98a80fff30 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.457+0000 7f98ad3e9700 1 --2- 172.21.15.96:0/2697270325 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98a8105ab0 0x7f98a8105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.457+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2697270325 >> 172.21.15.96:0/2697270325 conn(0x7f98a80fb5e0 msgr2=0x7f98a80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:51.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.457+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2697270325 shutdown_connections 2024-04-22T15:47:51.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.457+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2697270325 wait complete. 2024-04-22T15:47:51.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98ad3e9700 1 Processor -- start 2024-04-22T15:47:51.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98ad3e9700 1 -- start start 2024-04-22T15:47:51.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98ad3e9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 0x7f98a8197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:51.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98ad3e9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98a8105ab0 0x7f98a8197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:51.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98ad3e9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a8198480 con 0x7f98a80ffab0 2024-04-22T15:47:51.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98ad3e9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a81985c0 con 0x7f98a8105ab0 2024-04-22T15:47:51.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98a6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 0x7f98a8197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:51.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98a6ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 0x7f98a8197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46754/0 (socket says 172.21.15.96:46754) 2024-04-22T15:47:51.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98a6ffd700 1 -- 172.21.15.96:0/2881785486 learned_addr learned my addr 172.21.15.96:0/2881785486 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:47:51.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.458+0000 7f98a67fc700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98a8105ab0 0x7f98a8197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:51.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.459+0000 7f98a6ffd700 1 -- 172.21.15.96:0/2881785486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98a8105ab0 msgr2=0x7f98a8197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.459+0000 7f98a6ffd700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98a8105ab0 0x7f98a8197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.459+0000 7f98a6ffd700 1 -- 172.21.15.96:0/2881785486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98980095f0 con 0x7f98a80ffab0 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.459+0000 7f98a6ffd700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 0x7f98a8197970 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f989c00d8d0 tx=0x7f989c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.459+0000 7f988ffff700 1 -- 172.21.15.96:0/2881785486 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f989c0098a0 con 0x7f98a80ffab0 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.459+0000 7f988ffff700 1 -- 172.21.15.96:0/2881785486 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f989c010460 con 0x7f98a80ffab0 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.459+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a81a09e0 con 0x7f98a80ffab0 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.459+0000 7f988ffff700 1 -- 172.21.15.96:0/2881785486 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f989c00f5f0 con 0x7f98a80ffab0 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.459+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a81a0df0 con 0x7f98a80ffab0 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.460+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a81080d0 con 0x7f98a80ffab0 2024-04-22T15:47:51.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.460+0000 7f988ffff700 1 -- 172.21.15.96:0/2881785486 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f989c009a00 con 0x7f98a80ffab0 2024-04-22T15:47:51.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.461+0000 7f988ffff700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f989006ecc0 0x7f9890071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:51.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.461+0000 7f988ffff700 1 -- 172.21.15.96:0/2881785486 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f989c08ec80 con 0x7f98a80ffab0 2024-04-22T15:47:51.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.461+0000 7f98a67fc700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f989006ecc0 0x7f9890071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:51.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.461+0000 7f98a67fc700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f989006ecc0 0x7f9890071180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f98a8069640 tx=0x7f989801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:51.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.465+0000 7f988ffff700 1 -- 172.21.15.96:0/2881785486 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f989c057a60 con 0x7f98a80ffab0 2024-04-22T15:47:51.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:51 smithi096 ceph-mon[112443]: from='client.35436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:51 smithi096 ceph-mon[112443]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:51 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4174858112' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:47:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:51 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/766509738' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:47:51.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.753+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f98a80693f0 con 0x7f98a80ffab0 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.753+0000 7f988ffff700 1 -- 172.21.15.96:0/2881785486 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f989c05b120 con 0x7f98a80ffab0 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:47:51.756 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:47:51.757 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:47:51.758 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:47:51.758 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:47:51.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.755+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f989006ecc0 msgr2=0x7f9890071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.755+0000 7f98ad3e9700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f989006ecc0 0x7f9890071180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f98a8069640 tx=0x7f989801a040 comp rx=0 tx=0).stop 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.755+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 msgr2=0x7f98a8197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.755+0000 7f98ad3e9700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 0x7f98a8197970 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f989c00d8d0 tx=0x7f989c00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.755+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 shutdown_connections 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.755+0000 7f98ad3e9700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f989006ecc0 0x7f9890071180 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.755+0000 7f98ad3e9700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98a80ffab0 0x7f98a8197970 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.755+0000 7f98ad3e9700 1 --2- 172.21.15.96:0/2881785486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98a8105ab0 0x7f98a8197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.755+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 >> 172.21.15.96:0/2881785486 conn(0x7f98a80fb5e0 msgr2=0x7f98a80fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.756+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 shutdown_connections 2024-04-22T15:47:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:47:51.756+0000 7f98ad3e9700 1 -- 172.21.15.96:0/2881785486 wait complete. 2024-04-22T15:47:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:51 smithi196 ceph-mon[81815]: from='client.35436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:51 smithi196 ceph-mon[81815]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:51 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4174858112' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:47:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:51 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/766509738' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:47:52.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:52 smithi096 ceph-mon[112443]: from='client.35452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:52 smithi096 ceph-mon[112443]: pgmap v781: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:52 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2881785486' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:47:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:52 smithi196 ceph-mon[81815]: from='client.35452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:52 smithi196 ceph-mon[81815]: pgmap v781: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:53.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:52 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2881785486' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:47:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:54 smithi096 ceph-mon[112443]: pgmap v782: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:54 smithi196 ceph-mon[81815]: pgmap v782: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:56.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:56 smithi096 ceph-mon[112443]: pgmap v783: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:56 smithi196 ceph-mon[81815]: pgmap v783: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:58.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:47:58 smithi096 ceph-mon[112443]: pgmap v784: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:47:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:47:58 smithi196 ceph-mon[81815]: pgmap v784: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:00 smithi096 ceph-mon[112443]: pgmap v785: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:00 smithi196 ceph-mon[81815]: pgmap v785: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:02.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:02 smithi096 ceph-mon[112443]: pgmap v786: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:02 smithi196 ceph-mon[81815]: pgmap v786: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:04.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:04 smithi096 ceph-mon[112443]: pgmap v787: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:04 smithi196 ceph-mon[81815]: pgmap v787: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:06.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:06 smithi096 ceph-mon[112443]: pgmap v788: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:06 smithi196 ceph-mon[81815]: pgmap v788: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:08 smithi096 ceph-mon[112443]: pgmap v789: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:08 smithi196 ceph-mon[81815]: pgmap v789: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:10.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:10 smithi096 ceph-mon[112443]: pgmap v790: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:10 smithi196 ceph-mon[81815]: pgmap v790: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:12.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:12 smithi096 ceph-mon[112443]: pgmap v791: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:12 smithi196 ceph-mon[81815]: pgmap v791: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:14 smithi096 ceph-mon[112443]: pgmap v792: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:14 smithi196 ceph-mon[81815]: pgmap v792: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:15.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:15 smithi096 ceph-mon[112443]: pgmap v793: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:15 smithi196 ceph-mon[81815]: pgmap v793: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:18.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:18 smithi096 ceph-mon[112443]: pgmap v794: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:18 smithi196 ceph-mon[81815]: pgmap v794: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:20.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:20 smithi096 ceph-mon[112443]: pgmap v795: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:20 smithi196 ceph-mon[81815]: pgmap v795: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:21.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.872+0000 7f41c5519700 1 -- 172.21.15.96:0/2289663730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 msgr2=0x7f41c0105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:21.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.872+0000 7f41c5519700 1 --2- 172.21.15.96:0/2289663730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 0x7f41c0105ea0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009910 tx=0x7f41b0009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:21.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.872+0000 7f41c5519700 1 -- 172.21.15.96:0/2289663730 shutdown_connections 2024-04-22T15:48:21.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.872+0000 7f41c5519700 1 --2- 172.21.15.96:0/2289663730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41c00ffac0 0x7f41c00fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:21.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.872+0000 7f41c5519700 1 --2- 172.21.15.96:0/2289663730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 0x7f41c0105ea0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:21.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.872+0000 7f41c5519700 1 -- 172.21.15.96:0/2289663730 >> 172.21.15.96:0/2289663730 conn(0x7f41c00fb5d0 msgr2=0x7f41c00fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:21.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.872+0000 7f41c5519700 1 -- 172.21.15.96:0/2289663730 shutdown_connections 2024-04-22T15:48:21.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.872+0000 7f41c5519700 1 -- 172.21.15.96:0/2289663730 wait complete. 2024-04-22T15:48:21.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.873+0000 7f41c5519700 1 Processor -- start 2024-04-22T15:48:21.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.873+0000 7f41c5519700 1 -- start start 2024-04-22T15:48:21.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.873+0000 7f41c5519700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41c00ffac0 0x7f41c019b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:21.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.873+0000 7f41c5519700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 0x7f41c019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:21.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.873+0000 7f41c5519700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c019c1c0 con 0x7f41c0105ac0 2024-04-22T15:48:21.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.873+0000 7f41c5519700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0195800 con 0x7f41c00ffac0 2024-04-22T15:48:21.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41b67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 0x7f41c019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41b67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 0x7f41c019bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47046/0 (socket says 172.21.15.96:47046) 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41b67fc700 1 -- 172.21.15.96:0/4013385686 learned_addr learned my addr 172.21.15.96:0/4013385686 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41beffd700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41c00ffac0 0x7f41c019b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41b67fc700 1 -- 172.21.15.96:0/4013385686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41c00ffac0 msgr2=0x7f41c019b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41b67fc700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41c00ffac0 0x7f41c019b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41b67fc700 1 -- 172.21.15.96:0/4013385686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b00095f0 con 0x7f41c0105ac0 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41b67fc700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 0x7f41c019bb70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f41a800c930 tx=0x7f41a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41bcff9700 1 -- 172.21.15.96:0/4013385686 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41a8007ad0 con 0x7f41c0105ac0 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c0195ae0 con 0x7f41c0105ac0 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41bcff9700 1 -- 172.21.15.96:0/4013385686 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41a800ce80 con 0x7f41c0105ac0 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41bcff9700 1 -- 172.21.15.96:0/4013385686 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41a8018630 con 0x7f41c0105ac0 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.874+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c0196030 con 0x7f41c0105ac0 2024-04-22T15:48:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.875+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c01080c0 con 0x7f41c0105ac0 2024-04-22T15:48:21.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.876+0000 7f41bcff9700 1 -- 172.21.15.96:0/4013385686 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f41a8007c30 con 0x7f41c0105ac0 2024-04-22T15:48:21.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.876+0000 7f41bcff9700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f41ac06ecc0 0x7f41ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:21.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.876+0000 7f41bcff9700 1 -- 172.21.15.96:0/4013385686 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f41a808eef0 con 0x7f41c0105ac0 2024-04-22T15:48:21.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.876+0000 7f41beffd700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f41ac06ecc0 0x7f41ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:21.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.877+0000 7f41beffd700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f41ac06ecc0 0x7f41ac071180 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000c00 tx=0x7f41b001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:21.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:21.881+0000 7f41bcff9700 1 -- 172.21.15.96:0/4013385686 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f41a805b780 con 0x7f41c0105ac0 2024-04-22T15:48:22.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.103+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c010ccc0 con 0x7f41ac06ecc0 2024-04-22T15:48:22.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.104+0000 7f41bcff9700 1 -- 172.21.15.96:0/4013385686 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f41c010ccc0 con 0x7f41ac06ecc0 2024-04-22T15:48:22.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.106+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f41ac06ecc0 msgr2=0x7f41ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.106+0000 7f41c5519700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f41ac06ecc0 0x7f41ac071180 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000c00 tx=0x7f41b001a040 comp rx=0 tx=0).stop 2024-04-22T15:48:22.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.106+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 msgr2=0x7f41c019bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.106+0000 7f41c5519700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 0x7f41c019bb70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f41a800c930 tx=0x7f41a800cc40 comp rx=0 tx=0).stop 2024-04-22T15:48:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.107+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 shutdown_connections 2024-04-22T15:48:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.107+0000 7f41c5519700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f41ac06ecc0 0x7f41ac071180 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.107+0000 7f41c5519700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41c00ffac0 0x7f41c019b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.107+0000 7f41c5519700 1 --2- 172.21.15.96:0/4013385686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41c0105ac0 0x7f41c019bb70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.107+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 >> 172.21.15.96:0/4013385686 conn(0x7f41c00fb5d0 msgr2=0x7f41c00fc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.107+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 shutdown_connections 2024-04-22T15:48:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.107+0000 7f41c5519700 1 -- 172.21.15.96:0/4013385686 wait complete. 2024-04-22T15:48:22.120 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:48:22.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.211+0000 7f7e9f07d700 1 -- 172.21.15.96:0/3331162444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 msgr2=0x7f7e98101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.211+0000 7f7e9f07d700 1 --2- 172.21.15.96:0/3331162444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 0x7f7e98101210 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7e9400b0a0 tx=0x7f7e9400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.211+0000 7f7e9f07d700 1 -- 172.21.15.96:0/3331162444 shutdown_connections 2024-04-22T15:48:22.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.211+0000 7f7e9f07d700 1 --2- 172.21.15.96:0/3331162444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e981017e0 0x7f7e98105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.211+0000 7f7e9f07d700 1 --2- 172.21.15.96:0/3331162444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 0x7f7e98101210 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.211+0000 7f7e9f07d700 1 -- 172.21.15.96:0/3331162444 >> 172.21.15.96:0/3331162444 conn(0x7f7e980fc700 msgr2=0x7f7e980feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:22.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.211+0000 7f7e9f07d700 1 -- 172.21.15.96:0/3331162444 shutdown_connections 2024-04-22T15:48:22.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.211+0000 7f7e9f07d700 1 -- 172.21.15.96:0/3331162444 wait complete. 2024-04-22T15:48:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9f07d700 1 Processor -- start 2024-04-22T15:48:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9f07d700 1 -- start start 2024-04-22T15:48:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9f07d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 0x7f7e9819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9f07d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e981017e0 0x7f7e9819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9f07d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9819d400 con 0x7f7e98100e30 2024-04-22T15:48:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9f07d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e98196a00 con 0x7f7e981017e0 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9ce19700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 0x7f7e9819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9ce19700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 0x7f7e9819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47072/0 (socket says 172.21.15.96:47072) 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9ce19700 1 -- 172.21.15.96:0/1584352774 learned_addr learned my addr 172.21.15.96:0/1584352774 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9ce19700 1 -- 172.21.15.96:0/1584352774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e981017e0 msgr2=0x7f7e9819ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9ce19700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e981017e0 0x7f7e9819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.212+0000 7f7e9ce19700 1 -- 172.21.15.96:0/1584352774 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e94009d00 con 0x7f7e98100e30 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.213+0000 7f7e9ce19700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 0x7f7e9819c8c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7e94000f80 tx=0x7f7e94007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.213+0000 7f7e8dffb700 1 -- 172.21.15.96:0/1584352774 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e9400e040 con 0x7f7e98100e30 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.213+0000 7f7e8dffb700 1 -- 172.21.15.96:0/1584352774 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e94003de0 con 0x7f7e98100e30 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.213+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e98196c80 con 0x7f7e98100e30 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.213+0000 7f7e8dffb700 1 -- 172.21.15.96:0/1584352774 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e9401d720 con 0x7f7e98100e30 2024-04-22T15:48:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.213+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e981970b0 con 0x7f7e98100e30 2024-04-22T15:48:22.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.214+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e981091d0 con 0x7f7e98100e30 2024-04-22T15:48:22.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.215+0000 7f7e8dffb700 1 -- 172.21.15.96:0/1584352774 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7e94004690 con 0x7f7e98100e30 2024-04-22T15:48:22.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.215+0000 7f7e8dffb700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e8006ecc0 0x7f7e80071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:22.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.215+0000 7f7e8ffff700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e8006ecc0 0x7f7e80071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:22.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.215+0000 7f7e8dffb700 1 -- 172.21.15.96:0/1584352774 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7e94090610 con 0x7f7e98100e30 2024-04-22T15:48:22.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.216+0000 7f7e8ffff700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e8006ecc0 0x7f7e80071180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7e88005fd0 tx=0x7f7e88005d00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:22.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.220+0000 7f7e8dffb700 1 -- 172.21.15.96:0/1584352774 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7e9405cf20 con 0x7f7e98100e30 2024-04-22T15:48:22.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.439+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e98197ec0 con 0x7f7e8006ecc0 2024-04-22T15:48:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.440+0000 7f7e8dffb700 1 -- 172.21.15.96:0/1584352774 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7e98197ec0 con 0x7f7e8006ecc0 2024-04-22T15:48:22.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e8006ecc0 msgr2=0x7f7e80071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e8006ecc0 0x7f7e80071180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7e88005fd0 tx=0x7f7e88005d00 comp rx=0 tx=0).stop 2024-04-22T15:48:22.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 msgr2=0x7f7e9819c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 0x7f7e9819c8c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7e94000f80 tx=0x7f7e94007bd0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 shutdown_connections 2024-04-22T15:48:22.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e8006ecc0 0x7f7e80071180 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e98100e30 0x7f7e9819c8c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 --2- 172.21.15.96:0/1584352774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e981017e0 0x7f7e9819ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 >> 172.21.15.96:0/1584352774 conn(0x7f7e980fc700 msgr2=0x7f7e980fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:22.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 shutdown_connections 2024-04-22T15:48:22.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.442+0000 7f7e9f07d700 1 -- 172.21.15.96:0/1584352774 wait complete. 2024-04-22T15:48:22.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.546+0000 7f5371d82700 1 -- 172.21.15.96:0/934907094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c100e20 msgr2=0x7f536c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.546+0000 7f5371d82700 1 --2- 172.21.15.96:0/934907094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c100e20 0x7f536c101200 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f535c009910 tx=0x7f535c009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:22.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.546+0000 7f5371d82700 1 -- 172.21.15.96:0/934907094 shutdown_connections 2024-04-22T15:48:22.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.546+0000 7f5371d82700 1 --2- 172.21.15.96:0/934907094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f536c1017d0 0x7f536c105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.547+0000 7f5371d82700 1 --2- 172.21.15.96:0/934907094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c100e20 0x7f536c101200 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.547+0000 7f5371d82700 1 -- 172.21.15.96:0/934907094 >> 172.21.15.96:0/934907094 conn(0x7f536c0fc6d0 msgr2=0x7f536c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:22.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.547+0000 7f5371d82700 1 -- 172.21.15.96:0/934907094 shutdown_connections 2024-04-22T15:48:22.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.547+0000 7f5371d82700 1 -- 172.21.15.96:0/934907094 wait complete. 2024-04-22T15:48:22.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.547+0000 7f5371d82700 1 Processor -- start 2024-04-22T15:48:22.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.547+0000 7f5371d82700 1 -- start start 2024-04-22T15:48:22.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f5371d82700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f536c100e20 0x7f536c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:22.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f5371d82700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c1017d0 0x7f536c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:22.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f5371d82700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c19d3a0 con 0x7f536c1017d0 2024-04-22T15:48:22.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f5371d82700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1969d0 con 0x7f536c100e20 2024-04-22T15:48:22.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f536affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c1017d0 0x7f536c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:22.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f536affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c1017d0 0x7f536c19cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47078/0 (socket says 172.21.15.96:47078) 2024-04-22T15:48:22.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f536affd700 1 -- 172.21.15.96:0/919636242 learned_addr learned my addr 172.21.15.96:0/919636242 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f536affd700 1 -- 172.21.15.96:0/919636242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f536c100e20 msgr2=0x7f536c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f536affd700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f536c100e20 0x7f536c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f536affd700 1 -- 172.21.15.96:0/919636242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c0095f0 con 0x7f536c1017d0 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f536affd700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c1017d0 0x7f536c19cd80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f536000c8f0 tx=0x7f536000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f5368ff9700 1 -- 172.21.15.96:0/919636242 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5360007a50 con 0x7f536c1017d0 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f5368ff9700 1 -- 172.21.15.96:0/919636242 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5360007bb0 con 0x7f536c1017d0 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c196cb0 con 0x7f536c1017d0 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f5368ff9700 1 -- 172.21.15.96:0/919636242 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5360018630 con 0x7f536c1017d0 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.548+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c197200 con 0x7f536c1017d0 2024-04-22T15:48:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.549+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536c1091c0 con 0x7f536c1017d0 2024-04-22T15:48:22.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.550+0000 7f5368ff9700 1 -- 172.21.15.96:0/919636242 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f536000f450 con 0x7f536c1017d0 2024-04-22T15:48:22.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.550+0000 7f5368ff9700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f535406ecc0 0x7f5354071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:22.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.550+0000 7f5368ff9700 1 -- 172.21.15.96:0/919636242 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f536008edc0 con 0x7f536c1017d0 2024-04-22T15:48:22.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.550+0000 7f536b7fe700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f535406ecc0 0x7f5354071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:22.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.551+0000 7f536b7fe700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f535406ecc0 0x7f5354071180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f535c00b5c0 tx=0x7f535c005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:22.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.555+0000 7f5368ff9700 1 -- 172.21.15.96:0/919636242 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f536005b6d0 con 0x7f536c1017d0 2024-04-22T15:48:22.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.765+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f536c197ee0 con 0x7f535406ecc0 2024-04-22T15:48:22.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.776+0000 7f5368ff9700 1 -- 172.21.15.96:0/919636242 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f536c197ee0 con 0x7f535406ecc0 2024-04-22T15:48:22.779 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:48:22.779 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (27m) 4m ago 38m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:48:22.779 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (38m) 4m ago 38m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:48:22.779 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (37m) 4m ago 37m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:48:22.779 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (24m) 4m ago 38m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:48:22.779 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (24m) 4m ago 37m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (26m) 4m ago 38m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (20m) 4m ago 32m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (20m) 4m ago 32m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (19m) 4m ago 32m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (4m) 4m ago 32m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (29m) 4m ago 39m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (28m) 4m ago 37m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (25m) 4m ago 39m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (24m) 4m ago 37m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (27m) 4m ago 38m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (27m) 4m ago 37m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (23m) 4m ago 36m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (23m) 4m ago 35m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:48:22.780 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (22m) 4m ago 35m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:48:22.781 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (21m) 4m ago 35m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:48:22.781 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (21m) 4m ago 34m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:48:22.781 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (20m) 4m ago 34m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:48:22.781 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (27m) 4m ago 37m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:48:22.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f535406ecc0 msgr2=0x7f5354071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f535406ecc0 0x7f5354071180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f535c00b5c0 tx=0x7f535c005e20 comp rx=0 tx=0).stop 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c1017d0 msgr2=0x7f536c19cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c1017d0 0x7f536c19cd80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f536000c8f0 tx=0x7f536000ccb0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 shutdown_connections 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f535406ecc0 0x7f5354071180 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f536c100e20 0x7f536c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 --2- 172.21.15.96:0/919636242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f536c1017d0 0x7f536c19cd80 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 >> 172.21.15.96:0/919636242 conn(0x7f536c0fc6d0 msgr2=0x7f536c0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 shutdown_connections 2024-04-22T15:48:22.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.778+0000 7f5371d82700 1 -- 172.21.15.96:0/919636242 wait complete. 2024-04-22T15:48:22.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.879+0000 7f19dd497700 1 -- 172.21.15.96:0/1973287249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d80ffad0 msgr2=0x7f19d80fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.879+0000 7f19dd497700 1 --2- 172.21.15.96:0/1973287249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d80ffad0 0x7f19d80fff50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009910 tx=0x7f19cc009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:22.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.879+0000 7f19dd497700 1 -- 172.21.15.96:0/1973287249 shutdown_connections 2024-04-22T15:48:22.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.879+0000 7f19dd497700 1 --2- 172.21.15.96:0/1973287249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d80ffad0 0x7f19d80fff50 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.880+0000 7f19dd497700 1 --2- 172.21.15.96:0/1973287249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d8105b60 0x7f19d8105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.880+0000 7f19dd497700 1 -- 172.21.15.96:0/1973287249 >> 172.21.15.96:0/1973287249 conn(0x7f19d80fb600 msgr2=0x7f19d80fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:22.883 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:22 smithi096 ceph-mon[112443]: pgmap v796: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:22.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.880+0000 7f19dd497700 1 -- 172.21.15.96:0/1973287249 shutdown_connections 2024-04-22T15:48:22.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.880+0000 7f19dd497700 1 -- 172.21.15.96:0/1973287249 wait complete. 2024-04-22T15:48:22.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.880+0000 7f19dd497700 1 Processor -- start 2024-04-22T15:48:22.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19dd497700 1 -- start start 2024-04-22T15:48:22.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19dd497700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d80ffad0 0x7f19d8197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:22.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19dd497700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d8105b60 0x7f19d8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:22.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19dd497700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d8198520 con 0x7f19d8105b60 2024-04-22T15:48:22.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19dd497700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d8198660 con 0x7f19d80ffad0 2024-04-22T15:48:22.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19d67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d8105b60 0x7f19d8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19d67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d8105b60 0x7f19d8197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47100/0 (socket says 172.21.15.96:47100) 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19d67fc700 1 -- 172.21.15.96:0/2164540117 learned_addr learned my addr 172.21.15.96:0/2164540117 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19d6ffd700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d80ffad0 0x7f19d8197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19d67fc700 1 -- 172.21.15.96:0/2164540117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d80ffad0 msgr2=0x7f19d8197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19d67fc700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d80ffad0 0x7f19d8197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19d67fc700 1 -- 172.21.15.96:0/2164540117 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc0095f0 con 0x7f19d8105b60 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.881+0000 7f19d6ffd700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d80ffad0 0x7f19d8197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.882+0000 7f19d67fc700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d8105b60 0x7f19d8197f50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f19cc006010 tx=0x7f19cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.882+0000 7f19bffff700 1 -- 172.21.15.96:0/2164540117 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19cc01d070 con 0x7f19d8105b60 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.882+0000 7f19bffff700 1 -- 172.21.15.96:0/2164540117 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19cc00f460 con 0x7f19d8105b60 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.882+0000 7f19bffff700 1 -- 172.21.15.96:0/2164540117 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19cc017630 con 0x7f19d8105b60 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.882+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d8198880 con 0x7f19d8105b60 2024-04-22T15:48:22.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.882+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d81a0de0 con 0x7f19d8105b60 2024-04-22T15:48:22.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.883+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d81080f0 con 0x7f19d8105b60 2024-04-22T15:48:22.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.883+0000 7f19bffff700 1 -- 172.21.15.96:0/2164540117 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f19cc00f5d0 con 0x7f19d8105b60 2024-04-22T15:48:22.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.884+0000 7f19bffff700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19c006ecd0 0x7f19c0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:22.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.884+0000 7f19bffff700 1 -- 172.21.15.96:0/2164540117 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f19cc090170 con 0x7f19d8105b60 2024-04-22T15:48:22.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.884+0000 7f19d6ffd700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19c006ecd0 0x7f19c0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:22.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.884+0000 7f19d6ffd700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19c006ecd0 0x7f19c0071190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f19c80099d0 tx=0x7f19c8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:22.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:22.889+0000 7f19bffff700 1 -- 172.21.15.96:0/2164540117 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f19cc059010 con 0x7f19d8105b60 2024-04-22T15:48:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:22 smithi196 ceph-mon[81815]: pgmap v796: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:23.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.170+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f19d8198880 con 0x7f19d8105b60 2024-04-22T15:48:23.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.171+0000 7f19bffff700 1 -- 172.21.15.96:0/2164540117 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f19d8198880 con 0x7f19d8105b60 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:48:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:48:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:48:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:48:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:48:23.175 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19c006ecd0 msgr2=0x7f19c0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19c006ecd0 0x7f19c0071190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f19c80099d0 tx=0x7f19c8008040 comp rx=0 tx=0).stop 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d8105b60 msgr2=0x7f19d8197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d8105b60 0x7f19d8197f50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f19cc006010 tx=0x7f19cc00be20 comp rx=0 tx=0).stop 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 shutdown_connections 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f19c006ecd0 0x7f19c0071190 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19d80ffad0 0x7f19d8197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 --2- 172.21.15.96:0/2164540117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d8105b60 0x7f19d8197f50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 >> 172.21.15.96:0/2164540117 conn(0x7f19d80fb600 msgr2=0x7f19d80fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 shutdown_connections 2024-04-22T15:48:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.173+0000 7f19dd497700 1 -- 172.21.15.96:0/2164540117 wait complete. 2024-04-22T15:48:23.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 -- 172.21.15.96:0/3785001625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 msgr2=0x7f7ccc10aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 --2- 172.21.15.96:0/3785001625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 0x7f7ccc10aba0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009910 tx=0x7f7cc0009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:23.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 -- 172.21.15.96:0/3785001625 shutdown_connections 2024-04-22T15:48:23.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 --2- 172.21.15.96:0/3785001625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 0x7f7ccc10aba0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 --2- 172.21.15.96:0/3785001625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ccc101d90 0x7f7ccc102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 -- 172.21.15.96:0/3785001625 >> 172.21.15.96:0/3785001625 conn(0x7f7ccc0fb5e0 msgr2=0x7f7ccc0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:23.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 -- 172.21.15.96:0/3785001625 shutdown_connections 2024-04-22T15:48:23.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 -- 172.21.15.96:0/3785001625 wait complete. 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 Processor -- start 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.276+0000 7f7cd228d700 1 -- start start 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7cd228d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ccc101d90 0x7f7ccc19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7cd228d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 0x7f7ccc19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7cd228d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc19d3f0 con 0x7f7ccc1026b0 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7cd228d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1969d0 con 0x7f7ccc101d90 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7ccb7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 0x7f7ccc19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7ccb7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 0x7f7ccc19cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47110/0 (socket says 172.21.15.96:47110) 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7ccb7fe700 1 -- 172.21.15.96:0/1336702565 learned_addr learned my addr 172.21.15.96:0/1336702565 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:23.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7ccbfff700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ccc101d90 0x7f7ccc19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:23.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7ccb7fe700 1 -- 172.21.15.96:0/1336702565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ccc101d90 msgr2=0x7f7ccc19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7ccb7fe700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ccc101d90 0x7f7ccc19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7ccb7fe700 1 -- 172.21.15.96:0/1336702565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc00095f0 con 0x7f7ccc1026b0 2024-04-22T15:48:23.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7ccb7fe700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 0x7f7ccc19cdd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009fd0 tx=0x7f7cc0005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:23.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7cc97fa700 1 -- 172.21.15.96:0/1336702565 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cc001d070 con 0x7f7ccc1026b0 2024-04-22T15:48:23.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7cc97fa700 1 -- 172.21.15.96:0/1336702565 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cc00052a0 con 0x7f7ccc1026b0 2024-04-22T15:48:23.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.277+0000 7f7cc97fa700 1 -- 172.21.15.96:0/1336702565 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cc000fe20 con 0x7f7ccc1026b0 2024-04-22T15:48:23.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.278+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc196c50 con 0x7f7ccc1026b0 2024-04-22T15:48:23.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.278+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc1970a0 con 0x7f7ccc1026b0 2024-04-22T15:48:23.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.279+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc105520 con 0x7f7ccc1026b0 2024-04-22T15:48:23.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.279+0000 7f7cc97fa700 1 -- 172.21.15.96:0/1336702565 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7cc0003680 con 0x7f7ccc1026b0 2024-04-22T15:48:23.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.280+0000 7f7cc97fa700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7cb406ecc0 0x7f7cb4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:23.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.280+0000 7f7ccbfff700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7cb406ecc0 0x7f7cb4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:23.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.280+0000 7f7cc97fa700 1 -- 172.21.15.96:0/1336702565 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7cc0090110 con 0x7f7ccc1026b0 2024-04-22T15:48:23.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.280+0000 7f7ccbfff700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7cb406ecc0 0x7f7cb4071180 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009c60 tx=0x7f7cbc009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:23.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.285+0000 7f7cc97fa700 1 -- 172.21.15.96:0/1336702565 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7cc005c9a0 con 0x7f7ccc1026b0 2024-04-22T15:48:23.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.529+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7ccc069a30 con 0x7f7ccc1026b0 2024-04-22T15:48:23.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.529+0000 7f7cc97fa700 1 -- 172.21.15.96:0/1336702565 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f7cc005c530 con 0x7f7ccc1026b0 2024-04-22T15:48:23.532 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:48:23.532 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:48:23.532 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:48:23.532 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:48:23.532 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:48:23.533 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:48:23.534 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:48:23.535 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:48:23.535 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:48:23.535 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:48:23.535 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:48:23.535 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:48:23.535 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:48:23.535 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:48:23.535 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7cb406ecc0 msgr2=0x7f7cb4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7cb406ecc0 0x7f7cb4071180 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009c60 tx=0x7f7cbc009260 comp rx=0 tx=0).stop 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 msgr2=0x7f7ccc19cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 0x7f7ccc19cdd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009fd0 tx=0x7f7cc0005ea0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 shutdown_connections 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7cb406ecc0 0x7f7cb4071180 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ccc101d90 0x7f7ccc19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 --2- 172.21.15.96:0/1336702565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1026b0 0x7f7ccc19cdd0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 >> 172.21.15.96:0/1336702565 conn(0x7f7ccc0fb5e0 msgr2=0x7f7ccc103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 shutdown_connections 2024-04-22T15:48:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.532+0000 7f7cd228d700 1 -- 172.21.15.96:0/1336702565 wait complete. 2024-04-22T15:48:23.537 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:48:23.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.638+0000 7f81cf1a4700 1 -- 172.21.15.96:0/2163917015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8102700 msgr2=0x7f81c810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.638+0000 7f81cf1a4700 1 --2- 172.21.15.96:0/2163917015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8102700 0x7f81c810abf0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009940 tx=0x7f81b8009c50 comp rx=0 tx=0).stop 2024-04-22T15:48:23.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.638+0000 7f81cf1a4700 1 -- 172.21.15.96:0/2163917015 shutdown_connections 2024-04-22T15:48:23.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.638+0000 7f81cf1a4700 1 --2- 172.21.15.96:0/2163917015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8102700 0x7f81c810abf0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.638+0000 7f81cf1a4700 1 --2- 172.21.15.96:0/2163917015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81c8101de0 0x7f81c81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.638+0000 7f81cf1a4700 1 -- 172.21.15.96:0/2163917015 >> 172.21.15.96:0/2163917015 conn(0x7f81c8076740 msgr2=0x7f81c80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:23.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.638+0000 7f81cf1a4700 1 -- 172.21.15.96:0/2163917015 shutdown_connections 2024-04-22T15:48:23.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.638+0000 7f81cf1a4700 1 -- 172.21.15.96:0/2163917015 wait complete. 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81cf1a4700 1 Processor -- start 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81cf1a4700 1 -- start start 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81cf1a4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8101de0 0x7f81c819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81cf1a4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81c8102700 0x7f81c819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81cf1a4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c819d380 con 0x7f81c8101de0 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81cf1a4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8196960 con 0x7f81c8102700 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81ccf40700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8101de0 0x7f81c819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81ccf40700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8101de0 0x7f81c819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47120/0 (socket says 172.21.15.96:47120) 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81ccf40700 1 -- 172.21.15.96:0/23044663 learned_addr learned my addr 172.21.15.96:0/23044663 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81ccf40700 1 -- 172.21.15.96:0/23044663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81c8102700 msgr2=0x7f81c819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81bffff700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81c8102700 0x7f81c819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81ccf40700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81c8102700 0x7f81c819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81ccf40700 1 -- 172.21.15.96:0/23044663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b80095f0 con 0x7f81c8101de0 2024-04-22T15:48:23.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.639+0000 7f81bffff700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81c8102700 0x7f81c819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:48:23.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.640+0000 7f81ccf40700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8101de0 0x7f81c819c820 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f81c40060b0 tx=0x7f81c400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:23.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.640+0000 7f81bdffb700 1 -- 172.21.15.96:0/23044663 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81c400dda0 con 0x7f81c8101de0 2024-04-22T15:48:23.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.640+0000 7f81bdffb700 1 -- 172.21.15.96:0/23044663 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81c400f040 con 0x7f81c8101de0 2024-04-22T15:48:23.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.640+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c8196c40 con 0x7f81c8101de0 2024-04-22T15:48:23.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.640+0000 7f81bdffb700 1 -- 172.21.15.96:0/23044663 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81c4014790 con 0x7f81c8101de0 2024-04-22T15:48:23.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.640+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c8197190 con 0x7f81c8101de0 2024-04-22T15:48:23.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.641+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c8105500 con 0x7f81c8101de0 2024-04-22T15:48:23.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.641+0000 7f81bdffb700 1 -- 172.21.15.96:0/23044663 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f81c4014980 con 0x7f81c8101de0 2024-04-22T15:48:23.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.641+0000 7f81bdffb700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f81b006ecd0 0x7f81b0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:23.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.642+0000 7f81bdffb700 1 -- 172.21.15.96:0/23044663 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f81c408e950 con 0x7f81c8101de0 2024-04-22T15:48:23.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.642+0000 7f81bffff700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f81b006ecd0 0x7f81b0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:23.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.642+0000 7f81bffff700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f81b006ecd0 0x7f81b0071190 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f81b8006010 tx=0x7f81b8005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:23.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.646+0000 7f81bdffb700 1 -- 172.21.15.96:0/23044663 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f81c405b290 con 0x7f81c8101de0 2024-04-22T15:48:23.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.865+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f81c8197e90 con 0x7f81b006ecd0 2024-04-22T15:48:23.868 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:23 smithi096 ceph-mon[112443]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:23.868 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:23 smithi096 ceph-mon[112443]: from='client.35464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:23.868 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:23 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2164540117' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:48:23.868 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:23.869 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:23 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1336702565' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:48:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.866+0000 7f81bdffb700 1 -- 172.21.15.96:0/23044663 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f81c8197e90 con 0x7f81b006ecd0 2024-04-22T15:48:23.869 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:48:23.869 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:48:23.870 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:48:23.871 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:48:23.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.868+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f81b006ecd0 msgr2=0x7f81b0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.868+0000 7f81cf1a4700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f81b006ecd0 0x7f81b0071190 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f81b8006010 tx=0x7f81b8005a90 comp rx=0 tx=0).stop 2024-04-22T15:48:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.868+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8101de0 msgr2=0x7f81c819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.868+0000 7f81cf1a4700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8101de0 0x7f81c819c820 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f81c40060b0 tx=0x7f81c400d3b0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.868+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 shutdown_connections 2024-04-22T15:48:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.868+0000 7f81cf1a4700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f81b006ecd0 0x7f81b0071190 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.868+0000 7f81cf1a4700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81c8101de0 0x7f81c819c820 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.868+0000 7f81cf1a4700 1 --2- 172.21.15.96:0/23044663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81c8102700 0x7f81c819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.868+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 >> 172.21.15.96:0/23044663 conn(0x7f81c8076740 msgr2=0x7f81c81030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.869+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 shutdown_connections 2024-04-22T15:48:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.869+0000 7f81cf1a4700 1 -- 172.21.15.96:0/23044663 wait complete. 2024-04-22T15:48:23.881 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:23 smithi196 ceph-mon[81815]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:23.881 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:23 smithi196 ceph-mon[81815]: from='client.35464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:23.881 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:23 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2164540117' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:48:23.881 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:23.881 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:23 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1336702565' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:48:23.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.973+0000 7fb60f80c700 1 -- 172.21.15.96:0/4160088608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6081017e0 msgr2=0x7fb608105830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.973+0000 7fb60f80c700 1 --2- 172.21.15.96:0/4160088608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6081017e0 0x7fb608105830 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8009910 tx=0x7fb5f8009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:23.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.973+0000 7fb60f80c700 1 -- 172.21.15.96:0/4160088608 shutdown_connections 2024-04-22T15:48:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.973+0000 7fb60f80c700 1 --2- 172.21.15.96:0/4160088608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6081017e0 0x7fb608105830 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.973+0000 7fb60f80c700 1 --2- 172.21.15.96:0/4160088608 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb608100e30 0x7fb608101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.973+0000 7fb60f80c700 1 -- 172.21.15.96:0/4160088608 >> 172.21.15.96:0/4160088608 conn(0x7fb6080fc700 msgr2=0x7fb6080feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.973+0000 7fb60f80c700 1 -- 172.21.15.96:0/4160088608 shutdown_connections 2024-04-22T15:48:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.973+0000 7fb60f80c700 1 -- 172.21.15.96:0/4160088608 wait complete. 2024-04-22T15:48:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60f80c700 1 Processor -- start 2024-04-22T15:48:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60f80c700 1 -- start start 2024-04-22T15:48:23.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60f80c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb608100e30 0x7fb608198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60f80c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6081017e0 0x7fb608199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60f80c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb608199920 con 0x7fb608100e30 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60f80c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60819c540 con 0x7fb6081017e0 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60d5a8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb608100e30 0x7fb608198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60d5a8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb608100e30 0x7fb608198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47140/0 (socket says 172.21.15.96:47140) 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60d5a8700 1 -- 172.21.15.96:0/2115928512 learned_addr learned my addr 172.21.15.96:0/2115928512 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60cda7700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6081017e0 0x7fb608199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60d5a8700 1 -- 172.21.15.96:0/2115928512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6081017e0 msgr2=0x7fb608199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60d5a8700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6081017e0 0x7fb608199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60d5a8700 1 -- 172.21.15.96:0/2115928512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f80095f0 con 0x7fb608100e30 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60cda7700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6081017e0 0x7fb608199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.974+0000 7fb60d5a8700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb608100e30 0x7fb608198d50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb6000060b0 tx=0x7fb60000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.975+0000 7fb5fe7fc700 1 -- 172.21.15.96:0/2115928512 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb60000dda0 con 0x7fb608100e30 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.975+0000 7fb5fe7fc700 1 -- 172.21.15.96:0/2115928512 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb60000f040 con 0x7fb608100e30 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.975+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60819c820 con 0x7fb608100e30 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.975+0000 7fb5fe7fc700 1 -- 172.21.15.96:0/2115928512 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb600014790 con 0x7fb608100e30 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.975+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60819cd40 con 0x7fb608100e30 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.976+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb608104fb0 con 0x7fb608100e30 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.976+0000 7fb5fe7fc700 1 -- 172.21.15.96:0/2115928512 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb600014980 con 0x7fb608100e30 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.976+0000 7fb5fe7fc700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5f406ecd0 0x7fb5f4071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.977+0000 7fb5fe7fc700 1 -- 172.21.15.96:0/2115928512 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb60008e960 con 0x7fb608100e30 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.977+0000 7fb60cda7700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5f406ecd0 0x7fb5f4071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.977+0000 7fb60cda7700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5f406ecd0 0x7fb5f4071190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb60819a2c0 tx=0x7fb5f801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:23.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:23.981+0000 7fb5fe7fc700 1 -- 172.21.15.96:0/2115928512 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb600057770 con 0x7fb608100e30 2024-04-22T15:48:24.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.263+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb608199fd0 con 0x7fb608100e30 2024-04-22T15:48:24.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.263+0000 7fb5fe7fc700 1 -- 172.21.15.96:0/2115928512 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fb60005ae30 con 0x7fb608100e30 2024-04-22T15:48:24.266 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:48:24.266 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:48:24.267 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:48:24.268 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:48:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5f406ecd0 msgr2=0x7fb5f4071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5f406ecd0 0x7fb5f4071190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb60819a2c0 tx=0x7fb5f801a040 comp rx=0 tx=0).stop 2024-04-22T15:48:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb608100e30 msgr2=0x7fb608198d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb608100e30 0x7fb608198d50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb6000060b0 tx=0x7fb60000d3b0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 shutdown_connections 2024-04-22T15:48:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5f406ecd0 0x7fb5f4071190 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb608100e30 0x7fb608198d50 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 --2- 172.21.15.96:0/2115928512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6081017e0 0x7fb608199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 >> 172.21.15.96:0/2115928512 conn(0x7fb6080fc700 msgr2=0x7fb6080fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:24.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.265+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 shutdown_connections 2024-04-22T15:48:24.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:24.266+0000 7fb60f80c700 1 -- 172.21.15.96:0/2115928512 wait complete. 2024-04-22T15:48:24.891 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:24 smithi096 ceph-mon[112443]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:24.891 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:24 smithi096 ceph-mon[112443]: pgmap v797: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:24.891 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:24 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2115928512' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:48:25.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:24 smithi196 ceph-mon[81815]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:25.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:24 smithi196 ceph-mon[81815]: pgmap v797: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:25.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:24 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2115928512' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:48:25.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:25 smithi096 ceph-mon[112443]: from='client.35480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:25 smithi196 ceph-mon[81815]: from='client.35480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:26.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:26 smithi096 ceph-mon[112443]: pgmap v798: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:26 smithi196 ceph-mon[81815]: pgmap v798: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:28.935 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:28 smithi196 ceph-mon[81815]: pgmap v799: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:28.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:28 smithi096 ceph-mon[112443]: pgmap v799: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:30.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:30 smithi096 ceph-mon[112443]: pgmap v800: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:30 smithi196 ceph-mon[81815]: pgmap v800: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:32.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:32 smithi096 ceph-mon[112443]: pgmap v801: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:32 smithi196 ceph-mon[81815]: pgmap v801: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:34 smithi096 ceph-mon[112443]: pgmap v802: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:34 smithi196 ceph-mon[81815]: pgmap v802: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:36.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:36 smithi096 ceph-mon[112443]: pgmap v803: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:36 smithi196 ceph-mon[81815]: pgmap v803: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:38 smithi096 ceph-mon[112443]: pgmap v804: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:38 smithi196 ceph-mon[81815]: pgmap v804: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:40 smithi096 ceph-mon[112443]: pgmap v805: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:40 smithi196 ceph-mon[81815]: pgmap v805: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:42 smithi096 ceph-mon[112443]: pgmap v806: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:42 smithi196 ceph-mon[81815]: pgmap v806: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:44 smithi096 ceph-mon[112443]: pgmap v807: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:44 smithi196 ceph-mon[81815]: pgmap v807: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:46 smithi096 ceph-mon[112443]: pgmap v808: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:46 smithi196 ceph-mon[81815]: pgmap v808: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:47 smithi096 ceph-mon[112443]: pgmap v809: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:47 smithi196 ceph-mon[81815]: pgmap v809: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:50.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:50 smithi096 ceph-mon[112443]: pgmap v810: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:50 smithi196 ceph-mon[81815]: pgmap v810: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:52 smithi096 ceph-mon[112443]: pgmap v811: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:52 smithi196 ceph-mon[81815]: pgmap v811: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:54.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.383+0000 7fb790556700 1 -- 172.21.15.96:0/1168368525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb788100df0 msgr2=0x7fb7881011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:54.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.383+0000 7fb790556700 1 --2- 172.21.15.96:0/1168368525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb788100df0 0x7fb7881011d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb78000b0a0 tx=0x7fb78000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.384+0000 7fb790556700 1 -- 172.21.15.96:0/1168368525 shutdown_connections 2024-04-22T15:48:54.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.384+0000 7fb790556700 1 --2- 172.21.15.96:0/1168368525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7881017a0 0x7fb7881057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.384+0000 7fb790556700 1 --2- 172.21.15.96:0/1168368525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb788100df0 0x7fb7881011d0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.384+0000 7fb790556700 1 -- 172.21.15.96:0/1168368525 >> 172.21.15.96:0/1168368525 conn(0x7fb7880fc6a0 msgr2=0x7fb7880feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:54.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.384+0000 7fb790556700 1 -- 172.21.15.96:0/1168368525 shutdown_connections 2024-04-22T15:48:54.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.384+0000 7fb790556700 1 -- 172.21.15.96:0/1168368525 wait complete. 2024-04-22T15:48:54.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.384+0000 7fb790556700 1 Processor -- start 2024-04-22T15:48:54.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.384+0000 7fb790556700 1 -- start start 2024-04-22T15:48:54.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb790556700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb788100df0 0x7fb7880731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:54.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb790556700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7881017a0 0x7fb788073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:54.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb790556700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788073c60 con 0x7fb7881017a0 2024-04-22T15:48:54.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb790556700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788073da0 con 0x7fb788100df0 2024-04-22T15:48:54.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb78daf1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7881017a0 0x7fb788073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:54.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb78daf1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7881017a0 0x7fb788073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46560/0 (socket says 172.21.15.96:46560) 2024-04-22T15:48:54.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb78daf1700 1 -- 172.21.15.96:0/2141329234 learned_addr learned my addr 172.21.15.96:0/2141329234 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb78e2f2700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb788100df0 0x7fb7880731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb78daf1700 1 -- 172.21.15.96:0/2141329234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb788100df0 msgr2=0x7fb7880731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb78daf1700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb788100df0 0x7fb7880731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb78daf1700 1 -- 172.21.15.96:0/2141329234 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb780009d00 con 0x7fb7881017a0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb78daf1700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7881017a0 0x7fb788073720 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00d900 tx=0x7fb77c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb7877fe700 1 -- 172.21.15.96:0/2141329234 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77c009900 con 0x7fb7881017a0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb7877fe700 1 -- 172.21.15.96:0/2141329234 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb77c010460 con 0x7fb7881017a0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7881a2cb0 con 0x7fb7881017a0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.385+0000 7fb7877fe700 1 -- 172.21.15.96:0/2141329234 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77c00f5f0 con 0x7fb7881017a0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.386+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7881a3010 con 0x7fb7881017a0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.387+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb788109190 con 0x7fb7881017a0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.387+0000 7fb7877fe700 1 -- 172.21.15.96:0/2141329234 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb77c0105d0 con 0x7fb7881017a0 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.387+0000 7fb7877fe700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77006ecc0 0x7fb770071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:54.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.387+0000 7fb7877fe700 1 -- 172.21.15.96:0/2141329234 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb77c08e3a0 con 0x7fb7881017a0 2024-04-22T15:48:54.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.388+0000 7fb78e2f2700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77006ecc0 0x7fb770071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:54.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.388+0000 7fb78e2f2700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77006ecc0 0x7fb770071180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb780000f80 tx=0x7fb7800060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:54.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.392+0000 7fb7877fe700 1 -- 172.21.15.96:0/2141329234 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb77c057180 con 0x7fb7881017a0 2024-04-22T15:48:54.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.610+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb78810ddb0 con 0x7fb77006ecc0 2024-04-22T15:48:54.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.611+0000 7fb7877fe700 1 -- 172.21.15.96:0/2141329234 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb78810ddb0 con 0x7fb77006ecc0 2024-04-22T15:48:54.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77006ecc0 msgr2=0x7fb770071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:54.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77006ecc0 0x7fb770071180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb780000f80 tx=0x7fb7800060c0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7881017a0 msgr2=0x7fb788073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:54.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7881017a0 0x7fb788073720 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00d900 tx=0x7fb77c00dc10 comp rx=0 tx=0).stop 2024-04-22T15:48:54.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 shutdown_connections 2024-04-22T15:48:54.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb77006ecc0 0x7fb770071180 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb788100df0 0x7fb7880731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 --2- 172.21.15.96:0/2141329234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7881017a0 0x7fb788073720 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 >> 172.21.15.96:0/2141329234 conn(0x7fb7880fc6a0 msgr2=0x7fb7880fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:54.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 shutdown_connections 2024-04-22T15:48:54.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.613+0000 7fb790556700 1 -- 172.21.15.96:0/2141329234 wait complete. 2024-04-22T15:48:54.626 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:48:54.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.725+0000 7f8f07cf0700 1 -- 172.21.15.96:0/2958476603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 msgr2=0x7f8f00101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:54.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.725+0000 7f8f07cf0700 1 --2- 172.21.15.96:0/2958476603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 0x7f8f00101200 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800b0a0 tx=0x7f8ef800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.725+0000 7f8f07cf0700 1 -- 172.21.15.96:0/2958476603 shutdown_connections 2024-04-22T15:48:54.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.725+0000 7f8f07cf0700 1 --2- 172.21.15.96:0/2958476603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f001017d0 0x7f8f00105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.725+0000 7f8f07cf0700 1 --2- 172.21.15.96:0/2958476603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 0x7f8f00101200 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.725+0000 7f8f07cf0700 1 -- 172.21.15.96:0/2958476603 >> 172.21.15.96:0/2958476603 conn(0x7f8f000fc6d0 msgr2=0x7f8f000feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:54.729 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:54 smithi096 ceph-mon[112443]: pgmap v812: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:54.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.725+0000 7f8f07cf0700 1 -- 172.21.15.96:0/2958476603 shutdown_connections 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.725+0000 7f8f07cf0700 1 -- 172.21.15.96:0/2958476603 wait complete. 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f07cf0700 1 Processor -- start 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f07cf0700 1 -- start start 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f07cf0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 0x7f8f000731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f07cf0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f001017d0 0x7f8f00073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f07cf0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f00073c60 con 0x7f8f00100e20 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f07cf0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f00073da0 con 0x7f8f001017d0 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f05a8c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 0x7f8f000731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f05a8c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 0x7f8f000731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46576/0 (socket says 172.21.15.96:46576) 2024-04-22T15:48:54.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f05a8c700 1 -- 172.21.15.96:0/3755988719 learned_addr learned my addr 172.21.15.96:0/3755988719 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f05a8c700 1 -- 172.21.15.96:0/3755988719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f001017d0 msgr2=0x7f8f00073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f0528b700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f001017d0 0x7f8f00073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f05a8c700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f001017d0 0x7f8f00073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f05a8c700 1 -- 172.21.15.96:0/3755988719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef8009d00 con 0x7f8f00100e20 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.726+0000 7f8f0528b700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f001017d0 0x7f8f00073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.727+0000 7f8f05a8c700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 0x7f8f000731e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8000f80 tx=0x7f8ef8007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.727+0000 7f8ef6ffd700 1 -- 172.21.15.96:0/3755988719 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef800e040 con 0x7f8f00100e20 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.727+0000 7f8ef6ffd700 1 -- 172.21.15.96:0/3755988719 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ef8003de0 con 0x7f8f00100e20 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.727+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f00074020 con 0x7f8f00100e20 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.727+0000 7f8ef6ffd700 1 -- 172.21.15.96:0/3755988719 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef801d720 con 0x7f8f00100e20 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.727+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f001a30c0 con 0x7f8f00100e20 2024-04-22T15:48:54.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.728+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f00102300 con 0x7f8f00100e20 2024-04-22T15:48:54.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.728+0000 7f8ef6ffd700 1 -- 172.21.15.96:0/3755988719 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8ef8004690 con 0x7f8f00100e20 2024-04-22T15:48:54.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.729+0000 7f8ef6ffd700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8eec06ecc0 0x7f8eec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:54.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.729+0000 7f8f0528b700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8eec06ecc0 0x7f8eec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:54.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.729+0000 7f8ef6ffd700 1 -- 172.21.15.96:0/3755988719 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8ef80906d0 con 0x7f8f00100e20 2024-04-22T15:48:54.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.729+0000 7f8f0528b700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8eec06ecc0 0x7f8eec071180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8f00074910 tx=0x7f8ef0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:54.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.734+0000 7f8ef6ffd700 1 -- 172.21.15.96:0/3755988719 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8ef8059430 con 0x7f8f00100e20 2024-04-22T15:48:54.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.950+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f0010ded0 con 0x7f8eec06ecc0 2024-04-22T15:48:54.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.951+0000 7f8ef6ffd700 1 -- 172.21.15.96:0/3755988719 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8f0010ded0 con 0x7f8eec06ecc0 2024-04-22T15:48:54.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8eec06ecc0 msgr2=0x7f8eec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:54.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8eec06ecc0 0x7f8eec071180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8f00074910 tx=0x7f8ef0008040 comp rx=0 tx=0).stop 2024-04-22T15:48:54.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 msgr2=0x7f8f000731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:54.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 0x7f8f000731e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8000f80 tx=0x7f8ef8007bd0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 shutdown_connections 2024-04-22T15:48:54.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8eec06ecc0 0x7f8eec071180 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f00100e20 0x7f8f000731e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 --2- 172.21.15.96:0/3755988719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f001017d0 0x7f8f00073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:54.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 >> 172.21.15.96:0/3755988719 conn(0x7f8f000fc6d0 msgr2=0x7f8f000fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:54.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 shutdown_connections 2024-04-22T15:48:54.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:54.953+0000 7f8f07cf0700 1 -- 172.21.15.96:0/3755988719 wait complete. 2024-04-22T15:48:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:54 smithi196 ceph-mon[81815]: pgmap v812: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.056+0000 7fda90265700 1 -- 172.21.15.96:0/3410581434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda881017a0 msgr2=0x7fda881057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.056+0000 7fda90265700 1 --2- 172.21.15.96:0/3410581434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda881017a0 0x7fda881057f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fda78009910 tx=0x7fda78009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.056+0000 7fda90265700 1 -- 172.21.15.96:0/3410581434 shutdown_connections 2024-04-22T15:48:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.056+0000 7fda90265700 1 --2- 172.21.15.96:0/3410581434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda881017a0 0x7fda881057f0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.056+0000 7fda90265700 1 --2- 172.21.15.96:0/3410581434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda88100df0 0x7fda881011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.056+0000 7fda90265700 1 -- 172.21.15.96:0/3410581434 >> 172.21.15.96:0/3410581434 conn(0x7fda880fc6a0 msgr2=0x7fda880feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.056+0000 7fda90265700 1 -- 172.21.15.96:0/3410581434 shutdown_connections 2024-04-22T15:48:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.056+0000 7fda90265700 1 -- 172.21.15.96:0/3410581434 wait complete. 2024-04-22T15:48:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda90265700 1 Processor -- start 2024-04-22T15:48:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda90265700 1 -- start start 2024-04-22T15:48:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda90265700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda88100df0 0x7fda880731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda90265700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda881017a0 0x7fda88073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda90265700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda88073c60 con 0x7fda88100df0 2024-04-22T15:48:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda90265700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda88073da0 con 0x7fda881017a0 2024-04-22T15:48:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda8e001700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda88100df0 0x7fda880731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda8e001700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda88100df0 0x7fda880731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46596/0 (socket says 172.21.15.96:46596) 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda8e001700 1 -- 172.21.15.96:0/2668687790 learned_addr learned my addr 172.21.15.96:0/2668687790 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda8e001700 1 -- 172.21.15.96:0/2668687790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda881017a0 msgr2=0x7fda88073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda8e001700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda881017a0 0x7fda88073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda8e001700 1 -- 172.21.15.96:0/2668687790 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda780095f0 con 0x7fda88100df0 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.057+0000 7fda8e001700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda88100df0 0x7fda880731e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fda8400c8a0 tx=0x7fda8400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.058+0000 7fda7effd700 1 -- 172.21.15.96:0/2668687790 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda8400cea0 con 0x7fda88100df0 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.058+0000 7fda7effd700 1 -- 172.21.15.96:0/2668687790 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda84004830 con 0x7fda88100df0 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.058+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda881a2cd0 con 0x7fda88100df0 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.058+0000 7fda7effd700 1 -- 172.21.15.96:0/2668687790 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda840055b0 con 0x7fda88100df0 2024-04-22T15:48:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.058+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda881a3030 con 0x7fda88100df0 2024-04-22T15:48:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.059+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda881091f0 con 0x7fda88100df0 2024-04-22T15:48:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.059+0000 7fda7effd700 1 -- 172.21.15.96:0/2668687790 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fda8400f450 con 0x7fda88100df0 2024-04-22T15:48:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.059+0000 7fda7effd700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda7406ecd0 0x7fda74071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.059+0000 7fda7effd700 1 -- 172.21.15.96:0/2668687790 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fda8408eb80 con 0x7fda88100df0 2024-04-22T15:48:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.060+0000 7fda8d800700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda7406ecd0 0x7fda74071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.060+0000 7fda8d800700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda7406ecd0 0x7fda74071190 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fda78006010 tx=0x7fda7800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:55.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.064+0000 7fda7effd700 1 -- 172.21.15.96:0/2668687790 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fda84057950 con 0x7fda88100df0 2024-04-22T15:48:55.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.273+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fda8810dd90 con 0x7fda7406ecd0 2024-04-22T15:48:55.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.284+0000 7fda7effd700 1 -- 172.21.15.96:0/2668687790 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fda8810dd90 con 0x7fda7406ecd0 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (27m) 5m ago 38m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (39m) 5m ago 39m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (38m) 5m ago 38m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (24m) 5m ago 39m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (24m) 5m ago 38m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (27m) 5m ago 38m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (20m) 5m ago 33m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (21m) 5m ago 33m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (20m) 5m ago 33m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:48:55.287 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (5m) 5m ago 33m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (29m) 5m ago 40m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (28m) 5m ago 37m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (25m) 5m ago 40m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (25m) 5m ago 37m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (28m) 5m ago 39m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (28m) 5m ago 37m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (24m) 5m ago 36m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (23m) 5m ago 36m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (23m) 5m ago 36m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (22m) 5m ago 35m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (21m) 5m ago 35m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (21m) 5m ago 34m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:48:55.288 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (27m) 5m ago 38m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:48:55.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda7406ecd0 msgr2=0x7fda74071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda7406ecd0 0x7fda74071190 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fda78006010 tx=0x7fda7800b540 comp rx=0 tx=0).stop 2024-04-22T15:48:55.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda88100df0 msgr2=0x7fda880731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda88100df0 0x7fda880731e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fda8400c8a0 tx=0x7fda8400cc60 comp rx=0 tx=0).stop 2024-04-22T15:48:55.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 shutdown_connections 2024-04-22T15:48:55.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fda7406ecd0 0x7fda74071190 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda88100df0 0x7fda880731e0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 --2- 172.21.15.96:0/2668687790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fda881017a0 0x7fda88073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 >> 172.21.15.96:0/2668687790 conn(0x7fda880fc6a0 msgr2=0x7fda880fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:55.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 shutdown_connections 2024-04-22T15:48:55.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.286+0000 7fda90265700 1 -- 172.21.15.96:0/2668687790 wait complete. 2024-04-22T15:48:55.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.388+0000 7f26fdaf8700 1 -- 172.21.15.96:0/2337438328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 msgr2=0x7f26f8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.388+0000 7f26fdaf8700 1 --2- 172.21.15.96:0/2337438328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 0x7f26f8101200 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009910 tx=0x7f26ec009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:55.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.388+0000 7f26fdaf8700 1 -- 172.21.15.96:0/2337438328 shutdown_connections 2024-04-22T15:48:55.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.388+0000 7f26fdaf8700 1 --2- 172.21.15.96:0/2337438328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f81017d0 0x7f26f8105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.388+0000 7f26fdaf8700 1 --2- 172.21.15.96:0/2337438328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 0x7f26f8101200 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.388+0000 7f26fdaf8700 1 -- 172.21.15.96:0/2337438328 >> 172.21.15.96:0/2337438328 conn(0x7f26f80fc6d0 msgr2=0x7f26f80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:55.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.388+0000 7f26fdaf8700 1 -- 172.21.15.96:0/2337438328 shutdown_connections 2024-04-22T15:48:55.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.388+0000 7f26fdaf8700 1 -- 172.21.15.96:0/2337438328 wait complete. 2024-04-22T15:48:55.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.389+0000 7f26fdaf8700 1 Processor -- start 2024-04-22T15:48:55.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.389+0000 7f26fdaf8700 1 -- start start 2024-04-22T15:48:55.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26fdaf8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 0x7f26f819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:55.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26fdaf8700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f81017d0 0x7f26f819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:55.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26fdaf8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f819d4a0 con 0x7f26f8100e20 2024-04-22T15:48:55.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26fdaf8700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f8196a80 con 0x7f26f81017d0 2024-04-22T15:48:55.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26f77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 0x7f26f819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:55.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26f77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 0x7f26f819c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46610/0 (socket says 172.21.15.96:46610) 2024-04-22T15:48:55.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26f77fe700 1 -- 172.21.15.96:0/67813590 learned_addr learned my addr 172.21.15.96:0/67813590 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:55.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26f6ffd700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f81017d0 0x7f26f819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:55.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26f77fe700 1 -- 172.21.15.96:0/67813590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f81017d0 msgr2=0x7f26f819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26f77fe700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f81017d0 0x7f26f819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26f77fe700 1 -- 172.21.15.96:0/67813590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec0095f0 con 0x7f26f8100e20 2024-04-22T15:48:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26f6ffd700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f81017d0 0x7f26f819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:48:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.390+0000 7f26f77fe700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 0x7f26f819c940 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f26ec006010 tx=0x7f26ec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.391+0000 7f26f4ff9700 1 -- 172.21.15.96:0/67813590 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26ec01d070 con 0x7f26f8100e20 2024-04-22T15:48:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.391+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f8196d00 con 0x7f26f8100e20 2024-04-22T15:48:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.391+0000 7f26f4ff9700 1 -- 172.21.15.96:0/67813590 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26ec00f460 con 0x7f26f8100e20 2024-04-22T15:48:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.391+0000 7f26f4ff9700 1 -- 172.21.15.96:0/67813590 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26ec017630 con 0x7f26f8100e20 2024-04-22T15:48:55.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.391+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f81971f0 con 0x7f26f8100e20 2024-04-22T15:48:55.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.392+0000 7f26f4ff9700 1 -- 172.21.15.96:0/67813590 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f26ec0177e0 con 0x7f26f8100e20 2024-04-22T15:48:55.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.392+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f81091c0 con 0x7f26f8100e20 2024-04-22T15:48:55.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.393+0000 7f26f4ff9700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f26e006ecc0 0x7f26e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:55.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.393+0000 7f26f4ff9700 1 -- 172.21.15.96:0/67813590 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f26ec090430 con 0x7f26f8100e20 2024-04-22T15:48:55.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.393+0000 7f26f6ffd700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f26e006ecc0 0x7f26e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.393+0000 7f26f6ffd700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f26e006ecc0 0x7f26e0071180 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f26f8197c40 tx=0x7f26e8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.398+0000 7f26f4ff9700 1 -- 172.21.15.96:0/67813590 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f26ec05cd70 con 0x7f26f8100e20 2024-04-22T15:48:55.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.686+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f26f8197f40 con 0x7f26f8100e20 2024-04-22T15:48:55.689 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:55 smithi096 ceph-mon[112443]: from='client.35488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:55.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.686+0000 7f26f4ff9700 1 -- 172.21.15.96:0/67813590 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f26ec05c900 con 0x7f26f8100e20 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:48:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:48:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:48:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:48:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:48:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:48:55.691 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:48:55.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f26e006ecc0 msgr2=0x7f26e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f26e006ecc0 0x7f26e0071180 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f26f8197c40 tx=0x7f26e8008040 comp rx=0 tx=0).stop 2024-04-22T15:48:55.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 msgr2=0x7f26f819c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 0x7f26f819c940 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f26ec006010 tx=0x7f26ec00be20 comp rx=0 tx=0).stop 2024-04-22T15:48:55.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 shutdown_connections 2024-04-22T15:48:55.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f26e006ecc0 0x7f26e0071180 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f8100e20 0x7f26f819c940 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 --2- 172.21.15.96:0/67813590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f81017d0 0x7f26f819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 >> 172.21.15.96:0/67813590 conn(0x7f26f80fc6d0 msgr2=0x7f26f80fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:55.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 shutdown_connections 2024-04-22T15:48:55.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.688+0000 7f26fdaf8700 1 -- 172.21.15.96:0/67813590 wait complete. 2024-04-22T15:48:55.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.791+0000 7ff79a5f5700 1 -- 172.21.15.96:0/1460441471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794101d60 msgr2=0x7ff794102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.791+0000 7ff79a5f5700 1 --2- 172.21.15.96:0/1460441471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794101d60 0x7ff794102140 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff784009910 tx=0x7ff784009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:55.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.791+0000 7ff79a5f5700 1 -- 172.21.15.96:0/1460441471 shutdown_connections 2024-04-22T15:48:55.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.791+0000 7ff79a5f5700 1 --2- 172.21.15.96:0/1460441471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff794102680 0x7ff79410ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.791+0000 7ff79a5f5700 1 --2- 172.21.15.96:0/1460441471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794101d60 0x7ff794102140 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.791+0000 7ff79a5f5700 1 -- 172.21.15.96:0/1460441471 >> 172.21.15.96:0/1460441471 conn(0x7ff7940fb5f0 msgr2=0x7ff7940fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:55.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.792+0000 7ff79a5f5700 1 -- 172.21.15.96:0/1460441471 shutdown_connections 2024-04-22T15:48:55.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.792+0000 7ff79a5f5700 1 -- 172.21.15.96:0/1460441471 wait complete. 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.792+0000 7ff79a5f5700 1 Processor -- start 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.792+0000 7ff79a5f5700 1 -- start start 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff79a5f5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff794101d60 0x7ff79419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff79a5f5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794102680 0x7ff79419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff79a5f5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79419d480 con 0x7ff794102680 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff79a5f5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff794196a60 con 0x7ff794101d60 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7937fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794102680 0x7ff79419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7937fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794102680 0x7ff79419ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46634/0 (socket says 172.21.15.96:46634) 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7937fe700 1 -- 172.21.15.96:0/3191886459 learned_addr learned my addr 172.21.15.96:0/3191886459 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff793fff700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff794101d60 0x7ff79419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7937fe700 1 -- 172.21.15.96:0/3191886459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff794101d60 msgr2=0x7ff79419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7937fe700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff794101d60 0x7ff79419c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7937fe700 1 -- 172.21.15.96:0/3191886459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7840095f0 con 0x7ff794102680 2024-04-22T15:48:55.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7937fe700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794102680 0x7ff79419ce60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff78800b3d0 tx=0x7ff78800b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:55.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7917fa700 1 -- 172.21.15.96:0/3191886459 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff788011800 con 0x7ff794102680 2024-04-22T15:48:55.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7917fa700 1 -- 172.21.15.96:0/3191886459 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff78800be90 con 0x7ff794102680 2024-04-22T15:48:55.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff7917fa700 1 -- 172.21.15.96:0/3191886459 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff788010e50 con 0x7ff794102680 2024-04-22T15:48:55.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff794196d40 con 0x7ff794102680 2024-04-22T15:48:55.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.793+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff794197290 con 0x7ff794102680 2024-04-22T15:48:55.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.794+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7941054f0 con 0x7ff794102680 2024-04-22T15:48:55.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.795+0000 7ff7917fa700 1 -- 172.21.15.96:0/3191886459 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff78801a420 con 0x7ff794102680 2024-04-22T15:48:55.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.795+0000 7ff7917fa700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff77c06ecc0 0x7ff77c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:55.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.795+0000 7ff7917fa700 1 -- 172.21.15.96:0/3191886459 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff78808e8e0 con 0x7ff794102680 2024-04-22T15:48:55.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.795+0000 7ff793fff700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff77c06ecc0 0x7ff77c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:55.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.796+0000 7ff793fff700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff77c06ecc0 0x7ff77c071180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff784000c00 tx=0x7ff78401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:55.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:55.800+0000 7ff7917fa700 1 -- 172.21.15.96:0/3191886459 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff78805b200 con 0x7ff794102680 2024-04-22T15:48:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:55 smithi196 ceph-mon[81815]: from='client.35488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.046+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff794069a50 con 0x7ff794102680 2024-04-22T15:48:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.046+0000 7ff7917fa700 1 -- 172.21.15.96:0/3191886459 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7ff788010380 con 0x7ff794102680 2024-04-22T15:48:56.049 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:48:56.049 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:48:56.049 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:48:56.049 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:48:56.049 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:48:56.049 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:48:56.049 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:48:56.050 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:48:56.051 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:48:56.052 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:48:56.052 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:48:56.052 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:48:56.052 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:48:56.052 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:48:56.052 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:48:56.052 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:48:56.052 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:48:56.052 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:48:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff77c06ecc0 msgr2=0x7ff77c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff77c06ecc0 0x7ff77c071180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff784000c00 tx=0x7ff78401a040 comp rx=0 tx=0).stop 2024-04-22T15:48:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794102680 msgr2=0x7ff79419ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794102680 0x7ff79419ce60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff78800b3d0 tx=0x7ff78800b6e0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 shutdown_connections 2024-04-22T15:48:56.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff77c06ecc0 0x7ff77c071180 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff794101d60 0x7ff79419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 --2- 172.21.15.96:0/3191886459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff794102680 0x7ff79419ce60 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 >> 172.21.15.96:0/3191886459 conn(0x7ff7940fb5f0 msgr2=0x7ff794103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:56.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 shutdown_connections 2024-04-22T15:48:56.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.048+0000 7ff79a5f5700 1 -- 172.21.15.96:0/3191886459 wait complete. 2024-04-22T15:48:56.054 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:48:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3061892734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 msgr2=0x7fb6dc0fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 --2- 172.21.15.96:0/3061892734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 0x7fb6dc0fffd0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc009910 tx=0x7fb6cc009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3061892734 shutdown_connections 2024-04-22T15:48:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 --2- 172.21.15.96:0/3061892734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 0x7fb6dc0fffd0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 --2- 172.21.15.96:0/3061892734 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6dc105c00 0x7fb6dc105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3061892734 >> 172.21.15.96:0/3061892734 conn(0x7fb6dc0fb680 msgr2=0x7fb6dc0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3061892734 shutdown_connections 2024-04-22T15:48:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3061892734 wait complete. 2024-04-22T15:48:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 Processor -- start 2024-04-22T15:48:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.153+0000 7fb6e35d4700 1 -- start start 2024-04-22T15:48:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e35d4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 0x7fb6dc197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e35d4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6dc105c00 0x7fb6dc197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e35d4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc1984a0 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e35d4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc1985e0 con 0x7fb6dc105c00 2024-04-22T15:48:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e1370700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 0x7fb6dc197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e1370700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 0x7fb6dc197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46638/0 (socket says 172.21.15.96:46638) 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e1370700 1 -- 172.21.15.96:0/3149545350 learned_addr learned my addr 172.21.15.96:0/3149545350 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e0b6f700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6dc105c00 0x7fb6dc197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e1370700 1 -- 172.21.15.96:0/3149545350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6dc105c00 msgr2=0x7fb6dc197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e1370700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6dc105c00 0x7fb6dc197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e1370700 1 -- 172.21.15.96:0/3149545350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc0095f0 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e1370700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 0x7fb6dc197a20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800b4b0 tx=0x7fb6d800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6d27fc700 1 -- 172.21.15.96:0/3149545350 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6d8011800 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6d27fc700 1 -- 172.21.15.96:0/3149545350 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6d8011e20 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc1a0a90 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6d27fc700 1 -- 172.21.15.96:0/3149545350 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6d800f2e0 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.154+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc1a0f20 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.155+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc108030 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.156+0000 7fb6d27fc700 1 -- 172.21.15.96:0/3149545350 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb6d8010380 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.156+0000 7fb6d27fc700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb6c806ecc0 0x7fb6c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.156+0000 7fb6d27fc700 1 -- 172.21.15.96:0/3149545350 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb6d808ea50 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.156+0000 7fb6e0b6f700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb6c806ecc0 0x7fb6c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.157+0000 7fb6e0b6f700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb6c806ecc0 0x7fb6c8071180 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00b5c0 tx=0x7fb6cc005380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.161+0000 7fb6d27fc700 1 -- 172.21.15.96:0/3149545350 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb6d80577b0 con 0x7fb6dc0ffb50 2024-04-22T15:48:56.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.381+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6dc10cdc0 con 0x7fb6c806ecc0 2024-04-22T15:48:56.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.382+0000 7fb6d27fc700 1 -- 172.21.15.96:0/3149545350 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb6dc10cdc0 con 0x7fb6c806ecc0 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:48:56.385 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:48:56.386 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:48:56.386 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:48:56.386 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:48:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb6c806ecc0 msgr2=0x7fb6c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb6c806ecc0 0x7fb6c8071180 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00b5c0 tx=0x7fb6cc005380 comp rx=0 tx=0).stop 2024-04-22T15:48:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 msgr2=0x7fb6dc197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 0x7fb6dc197a20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800b4b0 tx=0x7fb6d800b870 comp rx=0 tx=0).stop 2024-04-22T15:48:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 shutdown_connections 2024-04-22T15:48:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb6c806ecc0 0x7fb6c8071180 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6dc0ffb50 0x7fb6dc197a20 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 --2- 172.21.15.96:0/3149545350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6dc105c00 0x7fb6dc197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 >> 172.21.15.96:0/3149545350 conn(0x7fb6dc0fb680 msgr2=0x7fb6dc104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:56.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 shutdown_connections 2024-04-22T15:48:56.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.384+0000 7fb6e35d4700 1 -- 172.21.15.96:0/3149545350 wait complete. 2024-04-22T15:48:56.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.488+0000 7fc17605e700 1 -- 172.21.15.96:0/938768769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 msgr2=0x7fc1701011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.488+0000 7fc17605e700 1 --2- 172.21.15.96:0/938768769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 0x7fc1701011f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc160009910 tx=0x7fc160009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:56.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.488+0000 7fc17605e700 1 -- 172.21.15.96:0/938768769 shutdown_connections 2024-04-22T15:48:56.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.488+0000 7fc17605e700 1 --2- 172.21.15.96:0/938768769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1701017c0 0x7fc170105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.488+0000 7fc17605e700 1 --2- 172.21.15.96:0/938768769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 0x7fc1701011f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.488+0000 7fc17605e700 1 -- 172.21.15.96:0/938768769 >> 172.21.15.96:0/938768769 conn(0x7fc1700fc6e0 msgr2=0x7fc1700feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:56.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.488+0000 7fc17605e700 1 -- 172.21.15.96:0/938768769 shutdown_connections 2024-04-22T15:48:56.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.488+0000 7fc17605e700 1 -- 172.21.15.96:0/938768769 wait complete. 2024-04-22T15:48:56.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc17605e700 1 Processor -- start 2024-04-22T15:48:56.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc17605e700 1 -- start start 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc17605e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 0x7fc1700731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc17605e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1701017c0 0x7fc170073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc17605e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170073c60 con 0x7fc170100e10 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc17605e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170073da0 con 0x7fc1701017c0 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc16f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 0x7fc1700731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc16f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 0x7fc1700731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46658/0 (socket says 172.21.15.96:46658) 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc16f7fe700 1 -- 172.21.15.96:0/3411074217 learned_addr learned my addr 172.21.15.96:0/3411074217 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc167fff700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1701017c0 0x7fc170073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc16f7fe700 1 -- 172.21.15.96:0/3411074217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1701017c0 msgr2=0x7fc170073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc16f7fe700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1701017c0 0x7fc170073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc16f7fe700 1 -- 172.21.15.96:0/3411074217 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1600095f0 con 0x7fc170100e10 2024-04-22T15:48:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.489+0000 7fc167fff700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1701017c0 0x7fc170073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:48:56.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.490+0000 7fc16f7fe700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 0x7fc1700731e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc16000bd70 tx=0x7fc16000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:56.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.490+0000 7fc16d7fa700 1 -- 172.21.15.96:0/3411074217 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16001d070 con 0x7fc170100e10 2024-04-22T15:48:56.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.490+0000 7fc16d7fa700 1 -- 172.21.15.96:0/3411074217 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc16000f460 con 0x7fc170100e10 2024-04-22T15:48:56.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.490+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc170074020 con 0x7fc170100e10 2024-04-22T15:48:56.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.490+0000 7fc16d7fa700 1 -- 172.21.15.96:0/3411074217 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc160017630 con 0x7fc170100e10 2024-04-22T15:48:56.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.490+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1701a3020 con 0x7fc170100e10 2024-04-22T15:48:56.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.491+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1701091b0 con 0x7fc170100e10 2024-04-22T15:48:56.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.491+0000 7fc16d7fa700 1 -- 172.21.15.96:0/3411074217 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc16000f5d0 con 0x7fc170100e10 2024-04-22T15:48:56.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.492+0000 7fc16d7fa700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc15c06ecc0 0x7fc15c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:56.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.492+0000 7fc16d7fa700 1 -- 172.21.15.96:0/3411074217 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc1600903a0 con 0x7fc170100e10 2024-04-22T15:48:56.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.492+0000 7fc167fff700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc15c06ecc0 0x7fc15c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:56.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.492+0000 7fc167fff700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc15c06ecc0 0x7fc15c071180 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc1700748c0 tx=0x7fc158008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:56.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.496+0000 7fc16d7fa700 1 -- 172.21.15.96:0/3411074217 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc160059100 con 0x7fc170100e10 2024-04-22T15:48:56.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:56 smithi096 ceph-mon[112443]: from='client.35492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:56.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:56 smithi096 ceph-mon[112443]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:56.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:56 smithi096 ceph-mon[112443]: pgmap v813: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:56.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:56 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/67813590' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:48:56.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:56 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3191886459' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:48:56.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.785+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc170074710 con 0x7fc170100e10 2024-04-22T15:48:56.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.785+0000 7fc16d7fa700 1 -- 172.21.15.96:0/3411074217 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fc16005c7c0 con 0x7fc170100e10 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:48:56.789 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:48:56.790 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:48:56.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc15c06ecc0 msgr2=0x7fc15c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc15c06ecc0 0x7fc15c071180 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc1700748c0 tx=0x7fc158008040 comp rx=0 tx=0).stop 2024-04-22T15:48:56.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 msgr2=0x7fc1700731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 0x7fc1700731e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc16000bd70 tx=0x7fc16000be50 comp rx=0 tx=0).stop 2024-04-22T15:48:56.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 shutdown_connections 2024-04-22T15:48:56.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc15c06ecc0 0x7fc15c071180 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc170100e10 0x7fc1700731e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 --2- 172.21.15.96:0/3411074217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1701017c0 0x7fc170073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 >> 172.21.15.96:0/3411074217 conn(0x7fc1700fc6e0 msgr2=0x7fc1700fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:56.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 shutdown_connections 2024-04-22T15:48:56.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:48:56.788+0000 7fc17605e700 1 -- 172.21.15.96:0/3411074217 wait complete. 2024-04-22T15:48:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:56 smithi196 ceph-mon[81815]: from='client.35492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:56 smithi196 ceph-mon[81815]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:56 smithi196 ceph-mon[81815]: pgmap v813: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:56 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/67813590' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:48:57.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:56 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3191886459' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:48:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:57 smithi096 ceph-mon[112443]: from='client.35508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:57.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:57 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3411074217' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:48:58.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:57 smithi196 ceph-mon[81815]: from='client.35508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:58.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:57 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3411074217' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:48:58.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:48:58 smithi096 ceph-mon[112443]: pgmap v814: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:48:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:48:58 smithi196 ceph-mon[81815]: pgmap v814: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:00.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:00 smithi096 ceph-mon[112443]: pgmap v815: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:00 smithi196 ceph-mon[81815]: pgmap v815: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:02.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:02 smithi096 ceph-mon[112443]: pgmap v816: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:02 smithi196 ceph-mon[81815]: pgmap v816: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:04 smithi096 ceph-mon[112443]: pgmap v817: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:04 smithi196 ceph-mon[81815]: pgmap v817: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:06.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:06 smithi096 ceph-mon[112443]: pgmap v818: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:06 smithi196 ceph-mon[81815]: pgmap v818: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:08.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:08 smithi096 ceph-mon[112443]: pgmap v819: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:08 smithi196 ceph-mon[81815]: pgmap v819: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:09.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:09 smithi096 ceph-mon[112443]: pgmap v820: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:09 smithi196 ceph-mon[81815]: pgmap v820: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:12.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:12 smithi096 ceph-mon[112443]: pgmap v821: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:12 smithi196 ceph-mon[81815]: pgmap v821: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:14.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:14 smithi196 ceph-mon[81815]: pgmap v822: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:14.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:14 smithi096 ceph-mon[112443]: pgmap v822: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:16.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:16 smithi096 ceph-mon[112443]: pgmap v823: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:16 smithi196 ceph-mon[81815]: pgmap v823: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:18.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:18 smithi096 ceph-mon[112443]: pgmap v824: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:18 smithi196 ceph-mon[81815]: pgmap v824: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:20.738 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:20 smithi096 ceph-mon[112443]: pgmap v825: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:20 smithi196 ceph-mon[81815]: pgmap v825: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:22.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:22 smithi096 ceph-mon[112443]: pgmap v826: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:22 smithi196 ceph-mon[81815]: pgmap v826: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:24.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:24 smithi096 ceph-mon[112443]: pgmap v827: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:24 smithi196 ceph-mon[81815]: pgmap v827: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:26.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.905+0000 7f87c9db0700 1 -- 172.21.15.96:0/440421004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4101e00 msgr2=0x7f87c41021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:26.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.905+0000 7f87c9db0700 1 --2- 172.21.15.96:0/440421004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4101e00 0x7f87c41021e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009910 tx=0x7f87b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:26.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.905+0000 7f87c9db0700 1 -- 172.21.15.96:0/440421004 shutdown_connections 2024-04-22T15:49:26.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.905+0000 7f87c9db0700 1 --2- 172.21.15.96:0/440421004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f87c4102720 0x7f87c410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:26.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.905+0000 7f87c9db0700 1 --2- 172.21.15.96:0/440421004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4101e00 0x7f87c41021e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:26.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.905+0000 7f87c9db0700 1 -- 172.21.15.96:0/440421004 >> 172.21.15.96:0/440421004 conn(0x7f87c4076740 msgr2=0x7f87c40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:26.909 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:26 smithi096 ceph-mon[112443]: pgmap v828: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:26.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.905+0000 7f87c9db0700 1 -- 172.21.15.96:0/440421004 shutdown_connections 2024-04-22T15:49:26.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.905+0000 7f87c9db0700 1 -- 172.21.15.96:0/440421004 wait complete. 2024-04-22T15:49:26.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.906+0000 7f87c9db0700 1 Processor -- start 2024-04-22T15:49:26.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.906+0000 7f87c9db0700 1 -- start start 2024-04-22T15:49:26.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c9db0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f87c4101e00 0x7f87c419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c9db0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4102720 0x7f87c419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c9db0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c419d360 con 0x7f87c4102720 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c9db0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c4196940 con 0x7f87c4101e00 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87baffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4102720 0x7f87c419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87baffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4102720 0x7f87c419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33874/0 (socket says 172.21.15.96:33874) 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87baffd700 1 -- 172.21.15.96:0/252693057 learned_addr learned my addr 172.21.15.96:0/252693057 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c37fe700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f87c4101e00 0x7f87c419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87baffd700 1 -- 172.21.15.96:0/252693057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f87c4101e00 msgr2=0x7f87c419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87baffd700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f87c4101e00 0x7f87c419c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87baffd700 1 -- 172.21.15.96:0/252693057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b40095f0 con 0x7f87c4102720 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87baffd700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4102720 0x7f87c419cd40 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f87ac00d900 tx=0x7f87ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c17fa700 1 -- 172.21.15.96:0/252693057 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87ac009900 con 0x7f87c4102720 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c17fa700 1 -- 172.21.15.96:0/252693057 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87ac010460 con 0x7f87c4102720 2024-04-22T15:49:26.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c4196c20 con 0x7f87c4102720 2024-04-22T15:49:26.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c17fa700 1 -- 172.21.15.96:0/252693057 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87ac00f5f0 con 0x7f87c4102720 2024-04-22T15:49:26.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.907+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c4197170 con 0x7f87c4102720 2024-04-22T15:49:26.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.909+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c4105520 con 0x7f87c4102720 2024-04-22T15:49:26.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.909+0000 7f87c17fa700 1 -- 172.21.15.96:0/252693057 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f87ac0105d0 con 0x7f87c4102720 2024-04-22T15:49:26.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.909+0000 7f87c17fa700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f87b006ecc0 0x7f87b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:26.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.910+0000 7f87c37fe700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f87b006ecc0 0x7f87b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:26.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.910+0000 7f87c17fa700 1 -- 172.21.15.96:0/252693057 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f87ac08ec50 con 0x7f87c4102720 2024-04-22T15:49:26.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.910+0000 7f87c37fe700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f87b006ecc0 0x7f87b0071180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b5c0 tx=0x7f87b40052a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:26.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:26.914+0000 7f87c17fa700 1 -- 172.21.15.96:0/252693057 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f87ac05b560 con 0x7f87c4102720 2024-04-22T15:49:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:26 smithi196 ceph-mon[81815]: pgmap v828: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:27.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.132+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87c4197e70 con 0x7f87b006ecc0 2024-04-22T15:49:27.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.134+0000 7f87c17fa700 1 -- 172.21.15.96:0/252693057 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f87c4197e70 con 0x7f87b006ecc0 2024-04-22T15:49:27.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f87b006ecc0 msgr2=0x7f87b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f87b006ecc0 0x7f87b0071180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b5c0 tx=0x7f87b40052a0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4102720 msgr2=0x7f87c419cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4102720 0x7f87c419cd40 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f87ac00d900 tx=0x7f87ac00dcc0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 shutdown_connections 2024-04-22T15:49:27.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f87b006ecc0 0x7f87b0071180 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f87c4101e00 0x7f87c419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 --2- 172.21.15.96:0/252693057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87c4102720 0x7f87c419cd40 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 >> 172.21.15.96:0/252693057 conn(0x7f87c4076740 msgr2=0x7f87c41030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:27.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 shutdown_connections 2024-04-22T15:49:27.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.136+0000 7f87c9db0700 1 -- 172.21.15.96:0/252693057 wait complete. 2024-04-22T15:49:27.149 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:49:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.240+0000 7fb893f4d700 1 -- 172.21.15.96:0/3974524326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 msgr2=0x7fb88c101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.240+0000 7fb893f4d700 1 --2- 172.21.15.96:0/3974524326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 0x7fb88c101210 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb888009910 tx=0x7fb888009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.240+0000 7fb893f4d700 1 -- 172.21.15.96:0/3974524326 shutdown_connections 2024-04-22T15:49:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.240+0000 7fb893f4d700 1 --2- 172.21.15.96:0/3974524326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb88c1017e0 0x7fb88c105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.240+0000 7fb893f4d700 1 --2- 172.21.15.96:0/3974524326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 0x7fb88c101210 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.240+0000 7fb893f4d700 1 -- 172.21.15.96:0/3974524326 >> 172.21.15.96:0/3974524326 conn(0x7fb88c0fc700 msgr2=0x7fb88c0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.240+0000 7fb893f4d700 1 -- 172.21.15.96:0/3974524326 shutdown_connections 2024-04-22T15:49:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.240+0000 7fb893f4d700 1 -- 172.21.15.96:0/3974524326 wait complete. 2024-04-22T15:49:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb893f4d700 1 Processor -- start 2024-04-22T15:49:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb893f4d700 1 -- start start 2024-04-22T15:49:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb893f4d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 0x7fb88c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb893f4d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb88c1017e0 0x7fb88c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb893f4d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c19d490 con 0x7fb88c100e30 2024-04-22T15:49:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb893f4d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c196a00 con 0x7fb88c1017e0 2024-04-22T15:49:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb891ce9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 0x7fb88c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb891ce9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 0x7fb88c19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33890/0 (socket says 172.21.15.96:33890) 2024-04-22T15:49:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb891ce9700 1 -- 172.21.15.96:0/3452227575 learned_addr learned my addr 172.21.15.96:0/3452227575 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:49:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb8914e8700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb88c1017e0 0x7fb88c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb891ce9700 1 -- 172.21.15.96:0/3452227575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb88c1017e0 msgr2=0x7fb88c19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb891ce9700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb88c1017e0 0x7fb88c19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb891ce9700 1 -- 172.21.15.96:0/3452227575 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8880095f0 con 0x7fb88c100e30 2024-04-22T15:49:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.241+0000 7fb891ce9700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 0x7fb88c19c8c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb88800bd70 tx=0x7fb88800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.242+0000 7fb882ffd700 1 -- 172.21.15.96:0/3452227575 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb88801d070 con 0x7fb88c100e30 2024-04-22T15:49:27.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.242+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88c196c80 con 0x7fb88c100e30 2024-04-22T15:49:27.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.242+0000 7fb882ffd700 1 -- 172.21.15.96:0/3452227575 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8880056e0 con 0x7fb88c100e30 2024-04-22T15:49:27.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.242+0000 7fb882ffd700 1 -- 172.21.15.96:0/3452227575 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb888017400 con 0x7fb88c100e30 2024-04-22T15:49:27.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.242+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88c197170 con 0x7fb88c100e30 2024-04-22T15:49:27.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.243+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88c1091d0 con 0x7fb88c100e30 2024-04-22T15:49:27.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.243+0000 7fb882ffd700 1 -- 172.21.15.96:0/3452227575 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb888003a60 con 0x7fb88c100e30 2024-04-22T15:49:27.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.243+0000 7fb882ffd700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb87806ecc0 0x7fb878071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:27.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.244+0000 7fb8914e8700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb87806ecc0 0x7fb878071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:27.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.244+0000 7fb882ffd700 1 -- 172.21.15.96:0/3452227575 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb8880902f0 con 0x7fb88c100e30 2024-04-22T15:49:27.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.244+0000 7fb8914e8700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb87806ecc0 0x7fb878071180 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb87c005950 tx=0x7fb87c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:27.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.248+0000 7fb882ffd700 1 -- 172.21.15.96:0/3452227575 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb88805cb80 con 0x7fb88c100e30 2024-04-22T15:49:27.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.468+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb88c197f30 con 0x7fb87806ecc0 2024-04-22T15:49:27.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.469+0000 7fb882ffd700 1 -- 172.21.15.96:0/3452227575 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb88c197f30 con 0x7fb87806ecc0 2024-04-22T15:49:27.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.471+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb87806ecc0 msgr2=0x7fb878071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.471+0000 7fb893f4d700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb87806ecc0 0x7fb878071180 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb87c005950 tx=0x7fb87c0058e0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.471+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 msgr2=0x7fb88c19c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.471+0000 7fb893f4d700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 0x7fb88c19c8c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb88800bd70 tx=0x7fb88800be50 comp rx=0 tx=0).stop 2024-04-22T15:49:27.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.471+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 shutdown_connections 2024-04-22T15:49:27.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.471+0000 7fb893f4d700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb87806ecc0 0x7fb878071180 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.471+0000 7fb893f4d700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb88c100e30 0x7fb88c19c8c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.471+0000 7fb893f4d700 1 --2- 172.21.15.96:0/3452227575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb88c1017e0 0x7fb88c19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.471+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 >> 172.21.15.96:0/3452227575 conn(0x7fb88c0fc700 msgr2=0x7fb88c0fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:27.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.472+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 shutdown_connections 2024-04-22T15:49:27.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.472+0000 7fb893f4d700 1 -- 172.21.15.96:0/3452227575 wait complete. 2024-04-22T15:49:27.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.575+0000 7f7639713700 1 -- 172.21.15.96:0/2085939031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76340ffaa0 msgr2=0x7f76340fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.575+0000 7f7639713700 1 --2- 172.21.15.96:0/2085939031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76340ffaa0 0x7f76340fff20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7628009910 tx=0x7f7628009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.575+0000 7f7639713700 1 -- 172.21.15.96:0/2085939031 shutdown_connections 2024-04-22T15:49:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.575+0000 7f7639713700 1 --2- 172.21.15.96:0/2085939031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76340ffaa0 0x7f76340fff20 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.575+0000 7f7639713700 1 --2- 172.21.15.96:0/2085939031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7634105aa0 0x7f7634105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.575+0000 7f7639713700 1 -- 172.21.15.96:0/2085939031 >> 172.21.15.96:0/2085939031 conn(0x7f7634076740 msgr2=0x7f76340fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.575+0000 7f7639713700 1 -- 172.21.15.96:0/2085939031 shutdown_connections 2024-04-22T15:49:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.575+0000 7f7639713700 1 -- 172.21.15.96:0/2085939031 wait complete. 2024-04-22T15:49:27.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.576+0000 7f7639713700 1 Processor -- start 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.576+0000 7f7639713700 1 -- start start 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.576+0000 7f7639713700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76340ffaa0 0x7f763419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.576+0000 7f7639713700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7634105aa0 0x7f763419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.576+0000 7f7639713700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7634196a50 con 0x7f7634105aa0 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.576+0000 7f7639713700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7634196bc0 con 0x7f76340ffaa0 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.576+0000 7f76327fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7634105aa0 0x7f763419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.576+0000 7f76327fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7634105aa0 0x7f763419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33908/0 (socket says 172.21.15.96:33908) 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f76327fc700 1 -- 172.21.15.96:0/89973517 learned_addr learned my addr 172.21.15.96:0/89973517 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f7632ffd700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76340ffaa0 0x7f763419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f76327fc700 1 -- 172.21.15.96:0/89973517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76340ffaa0 msgr2=0x7f763419c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f76327fc700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76340ffaa0 0x7f763419c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f76327fc700 1 -- 172.21.15.96:0/89973517 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76280095f0 con 0x7f7634105aa0 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f76327fc700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7634105aa0 0x7f763419cee0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f762800bd70 tx=0x7f762800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:27.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f761bfff700 1 -- 172.21.15.96:0/89973517 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f762801d070 con 0x7f7634105aa0 2024-04-22T15:49:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f761bfff700 1 -- 172.21.15.96:0/89973517 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7628005700 con 0x7f7634105aa0 2024-04-22T15:49:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7634196e40 con 0x7f7634105aa0 2024-04-22T15:49:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f761bfff700 1 -- 172.21.15.96:0/89973517 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7628017400 con 0x7f7634105aa0 2024-04-22T15:49:27.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.577+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7634197330 con 0x7f7634105aa0 2024-04-22T15:49:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.578+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76341011e0 con 0x7f7634105aa0 2024-04-22T15:49:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.579+0000 7f761bfff700 1 -- 172.21.15.96:0/89973517 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7628003a60 con 0x7f7634105aa0 2024-04-22T15:49:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.580+0000 7f761bfff700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f761c06ecc0 0x7f761c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.580+0000 7f761bfff700 1 -- 172.21.15.96:0/89973517 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7628090340 con 0x7f7634105aa0 2024-04-22T15:49:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.580+0000 7f7632ffd700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f761c06ecc0 0x7f761c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:27.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.581+0000 7f7632ffd700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f761c06ecc0 0x7f761c071180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7624009a10 tx=0x7f7624009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:27.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.585+0000 7f761bfff700 1 -- 172.21.15.96:0/89973517 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f76280590a0 con 0x7f7634105aa0 2024-04-22T15:49:27.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.792+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7634197fa0 con 0x7f761c06ecc0 2024-04-22T15:49:27.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.804+0000 7f761bfff700 1 -- 172.21.15.96:0/89973517 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7634197fa0 con 0x7f761c06ecc0 2024-04-22T15:49:27.806 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:49:27.806 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (28m) 5m ago 39m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:49:27.806 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (39m) 5m ago 39m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (38m) 5m ago 38m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (25m) 5m ago 39m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (25m) 5m ago 38m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (27m) 5m ago 39m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (21m) 5m ago 33m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (21m) 5m ago 33m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (20m) 5m ago 33m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (5m) 5m ago 33m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (30m) 5m ago 40m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (29m) 5m ago 38m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (26m) 5m ago 41m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (26m) 5m ago 38m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (29m) 5m ago 39m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:49:27.807 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (29m) 5m ago 38m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:49:27.808 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (24m) 5m ago 37m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:49:27.808 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (24m) 5m ago 37m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:49:27.808 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (23m) 5m ago 36m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:49:27.808 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (23m) 5m ago 36m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:49:27.808 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (22m) 5m ago 35m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:49:27.808 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (21m) 5m ago 35m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:49:27.808 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (28m) 5m ago 38m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:49:27.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f761c06ecc0 msgr2=0x7f761c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f761c06ecc0 0x7f761c071180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7624009a10 tx=0x7f7624009190 comp rx=0 tx=0).stop 2024-04-22T15:49:27.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7634105aa0 msgr2=0x7f763419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7634105aa0 0x7f763419cee0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f762800bd70 tx=0x7f762800bda0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 shutdown_connections 2024-04-22T15:49:27.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f761c06ecc0 0x7f761c071180 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76340ffaa0 0x7f763419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 --2- 172.21.15.96:0/89973517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7634105aa0 0x7f763419cee0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 >> 172.21.15.96:0/89973517 conn(0x7f7634076740 msgr2=0x7f76340fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:27.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 shutdown_connections 2024-04-22T15:49:27.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.806+0000 7f7639713700 1 -- 172.21.15.96:0/89973517 wait complete. 2024-04-22T15:49:27.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.908+0000 7f4e5377d700 1 -- 172.21.15.96:0/4136897420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c1017d0 msgr2=0x7f4e4c105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.908+0000 7f4e5377d700 1 --2- 172.21.15.96:0/4136897420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c1017d0 0x7f4e4c105820 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009910 tx=0x7f4e3c009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:27.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.908+0000 7f4e5377d700 1 -- 172.21.15.96:0/4136897420 shutdown_connections 2024-04-22T15:49:27.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.908+0000 7f4e5377d700 1 --2- 172.21.15.96:0/4136897420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c1017d0 0x7f4e4c105820 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.908+0000 7f4e5377d700 1 --2- 172.21.15.96:0/4136897420 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e4c100e20 0x7f4e4c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.908+0000 7f4e5377d700 1 -- 172.21.15.96:0/4136897420 >> 172.21.15.96:0/4136897420 conn(0x7f4e4c0fc6d0 msgr2=0x7f4e4c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:27.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.908+0000 7f4e5377d700 1 -- 172.21.15.96:0/4136897420 shutdown_connections 2024-04-22T15:49:27.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.908+0000 7f4e5377d700 1 -- 172.21.15.96:0/4136897420 wait complete. 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e5377d700 1 Processor -- start 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e5377d700 1 -- start start 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e5377d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c100e20 0x7f4e4c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e5377d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e4c1017d0 0x7f4e4c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e5377d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c19d380 con 0x7f4e4c100e20 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e5377d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c196960 con 0x7f4e4c1017d0 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e51519700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c100e20 0x7f4e4c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e51519700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c100e20 0x7f4e4c19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33914/0 (socket says 172.21.15.96:33914) 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e51519700 1 -- 172.21.15.96:0/3993541832 learned_addr learned my addr 172.21.15.96:0/3993541832 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:49:27.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e50d18700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e4c1017d0 0x7f4e4c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e51519700 1 -- 172.21.15.96:0/3993541832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e4c1017d0 msgr2=0x7f4e4c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e51519700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e4c1017d0 0x7f4e4c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.909+0000 7f4e51519700 1 -- 172.21.15.96:0/3993541832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c0095f0 con 0x7f4e4c100e20 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.910+0000 7f4e51519700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c100e20 0x7f4e4c19c820 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400c8a0 tx=0x7f4e4400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.910+0000 7f4e427fc700 1 -- 172.21.15.96:0/3993541832 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e4400cea0 con 0x7f4e4c100e20 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.910+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c196c40 con 0x7f4e4c100e20 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.910+0000 7f4e427fc700 1 -- 172.21.15.96:0/3993541832 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e44004830 con 0x7f4e4c100e20 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.910+0000 7f4e427fc700 1 -- 172.21.15.96:0/3993541832 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e440055b0 con 0x7f4e4c100e20 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.910+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4c197190 con 0x7f4e4c100e20 2024-04-22T15:49:27.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.911+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4c109220 con 0x7f4e4c100e20 2024-04-22T15:49:27.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.912+0000 7f4e427fc700 1 -- 172.21.15.96:0/3993541832 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4e440049a0 con 0x7f4e4c100e20 2024-04-22T15:49:27.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.912+0000 7f4e427fc700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e3806ecc0 0x7f4e38071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:27.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.912+0000 7f4e427fc700 1 -- 172.21.15.96:0/3993541832 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4e4408eaa0 con 0x7f4e4c100e20 2024-04-22T15:49:27.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.912+0000 7f4e50d18700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e3806ecc0 0x7f4e38071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:27.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.913+0000 7f4e50d18700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e3806ecc0 0x7f4e38071180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c197b20 tx=0x7f4e3c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:27.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:27.917+0000 7f4e427fc700 1 -- 172.21.15.96:0/3993541832 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4e4405b3b0 con 0x7f4e4c100e20 2024-04-22T15:49:28.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.202+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4e4c197e20 con 0x7f4e4c100e20 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.203+0000 7f4e427fc700 1 -- 172.21.15.96:0/3993541832 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4e4401d070 con 0x7f4e4c100e20 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:49:28.206 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:49:28.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:49:28.207 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:49:28.207 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:49:28.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:49:28.207 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:49:28.207 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:49:28.207 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:49:28.207 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e3806ecc0 msgr2=0x7f4e38071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e3806ecc0 0x7f4e38071180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c197b20 tx=0x7f4e3c01a040 comp rx=0 tx=0).stop 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c100e20 msgr2=0x7f4e4c19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c100e20 0x7f4e4c19c820 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400c8a0 tx=0x7f4e4400cc60 comp rx=0 tx=0).stop 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 shutdown_connections 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e3806ecc0 0x7f4e38071180 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e4c100e20 0x7f4e4c19c820 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 --2- 172.21.15.96:0/3993541832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e4c1017d0 0x7f4e4c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 >> 172.21.15.96:0/3993541832 conn(0x7f4e4c0fc6d0 msgr2=0x7f4e4c0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:28.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 shutdown_connections 2024-04-22T15:49:28.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.205+0000 7f4e5377d700 1 -- 172.21.15.96:0/3993541832 wait complete. 2024-04-22T15:49:28.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.308+0000 7fe530b5b700 1 -- 172.21.15.96:0/1747268641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 msgr2=0x7fe52c103ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.308+0000 7fe530b5b700 1 --2- 172.21.15.96:0/1747268641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 0x7fe52c103ea0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe514009910 tx=0x7fe514009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:28.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.308+0000 7fe530b5b700 1 -- 172.21.15.96:0/1747268641 shutdown_connections 2024-04-22T15:49:28.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.308+0000 7fe530b5b700 1 --2- 172.21.15.96:0/1747268641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 0x7fe52c103ea0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.308+0000 7fe530b5b700 1 --2- 172.21.15.96:0/1747268641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe52c0ff4a0 0x7fe52c0ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.308+0000 7fe530b5b700 1 -- 172.21.15.96:0/1747268641 >> 172.21.15.96:0/1747268641 conn(0x7fe52c075f30 msgr2=0x7fe52c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:28.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.308+0000 7fe530b5b700 1 -- 172.21.15.96:0/1747268641 shutdown_connections 2024-04-22T15:49:28.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.308+0000 7fe530b5b700 1 -- 172.21.15.96:0/1747268641 wait complete. 2024-04-22T15:49:28.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.308+0000 7fe530b5b700 1 Processor -- start 2024-04-22T15:49:28.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe530b5b700 1 -- start start 2024-04-22T15:49:28.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe530b5b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe52c0ff4a0 0x7fe52c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:28.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe530b5b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 0x7fe52c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:28.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe530b5b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c196980 con 0x7fe52c0ffe50 2024-04-22T15:49:28.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe530b5b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c196af0 con 0x7fe52c0ff4a0 2024-04-22T15:49:28.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe523fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 0x7fe52c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe523fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 0x7fe52c19ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33928/0 (socket says 172.21.15.96:33928) 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe523fff700 1 -- 172.21.15.96:0/2480012019 learned_addr learned my addr 172.21.15.96:0/2480012019 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe52a59c700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe52c0ff4a0 0x7fe52c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe523fff700 1 -- 172.21.15.96:0/2480012019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe52c0ff4a0 msgr2=0x7fe52c19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe523fff700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe52c0ff4a0 0x7fe52c19c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe523fff700 1 -- 172.21.15.96:0/2480012019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5140095f0 con 0x7fe52c0ffe50 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe523fff700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 0x7fe52c19ce10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe51400bd70 tx=0x7fe51400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe5237fe700 1 -- 172.21.15.96:0/2480012019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe51401d070 con 0x7fe52c0ffe50 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe5237fe700 1 -- 172.21.15.96:0/2480012019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5140056e0 con 0x7fe52c0ffe50 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.309+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c196d70 con 0x7fe52c0ffe50 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.310+0000 7fe5237fe700 1 -- 172.21.15.96:0/2480012019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe514017400 con 0x7fe52c0ffe50 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.310+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c197260 con 0x7fe52c0ffe50 2024-04-22T15:49:28.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.311+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c100980 con 0x7fe52c0ffe50 2024-04-22T15:49:28.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.311+0000 7fe5237fe700 1 -- 172.21.15.96:0/2480012019 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe51400f460 con 0x7fe52c0ffe50 2024-04-22T15:49:28.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.311+0000 7fe5237fe700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe51806ecc0 0x7fe518071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:28.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.312+0000 7fe5237fe700 1 -- 172.21.15.96:0/2480012019 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe514090230 con 0x7fe52c0ffe50 2024-04-22T15:49:28.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.312+0000 7fe52a59c700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe51806ecc0 0x7fe518071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:28.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.312+0000 7fe52a59c700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe51806ecc0 0x7fe518071180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe51c005950 tx=0x7fe51c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:28.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.316+0000 7fe5237fe700 1 -- 172.21.15.96:0/2480012019 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe5140590d0 con 0x7fe52c0ffe50 2024-04-22T15:49:28.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.559+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe52c197e60 con 0x7fe52c0ffe50 2024-04-22T15:49:28.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.559+0000 7fe5237fe700 1 -- 172.21.15.96:0/2480012019 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7fe51405c790 con 0x7fe52c0ffe50 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:49:28.563 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:49:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:49:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:49:28.566 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:49:28.566 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.561+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe51806ecc0 msgr2=0x7fe518071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.561+0000 7fe530b5b700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe51806ecc0 0x7fe518071180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe51c005950 tx=0x7fe51c0058e0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.561+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 msgr2=0x7fe52c19ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.561+0000 7fe530b5b700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 0x7fe52c19ce10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe51400bd70 tx=0x7fe51400be50 comp rx=0 tx=0).stop 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.562+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 shutdown_connections 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.562+0000 7fe530b5b700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe51806ecc0 0x7fe518071180 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.562+0000 7fe530b5b700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe52c0ff4a0 0x7fe52c19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.562+0000 7fe530b5b700 1 --2- 172.21.15.96:0/2480012019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe52c0ffe50 0x7fe52c19ce10 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.562+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 >> 172.21.15.96:0/2480012019 conn(0x7fe52c075f30 msgr2=0x7fe52c1097d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.562+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 shutdown_connections 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.562+0000 7fe530b5b700 1 -- 172.21.15.96:0/2480012019 wait complete. 2024-04-22T15:49:28.567 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:49:28.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 -- 172.21.15.96:0/3343550123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d480ffad0 msgr2=0x7f3d480fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 --2- 172.21.15.96:0/3343550123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d480ffad0 0x7f3d480fff50 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009910 tx=0x7f3d3c009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:28.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 -- 172.21.15.96:0/3343550123 shutdown_connections 2024-04-22T15:49:28.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 --2- 172.21.15.96:0/3343550123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d480ffad0 0x7f3d480fff50 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 --2- 172.21.15.96:0/3343550123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d48105b60 0x7f3d48105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 -- 172.21.15.96:0/3343550123 >> 172.21.15.96:0/3343550123 conn(0x7f3d480fb5e0 msgr2=0x7f3d480fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:28.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 -- 172.21.15.96:0/3343550123 shutdown_connections 2024-04-22T15:49:28.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 -- 172.21.15.96:0/3343550123 wait complete. 2024-04-22T15:49:28.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 Processor -- start 2024-04-22T15:49:28.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.667+0000 7f3d4e58d700 1 -- start start 2024-04-22T15:49:28.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d4e58d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d480ffad0 0x7f3d4819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:28.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d4e58d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d48105b60 0x7f3d4819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:28.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d4e58d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4819d3c0 con 0x7f3d48105b60 2024-04-22T15:49:28.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d4e58d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d481969a0 con 0x7f3d480ffad0 2024-04-22T15:49:28.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d48105b60 0x7f3d4819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:28.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d48105b60 0x7f3d4819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33944/0 (socket says 172.21.15.96:33944) 2024-04-22T15:49:28.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d477fe700 1 -- 172.21.15.96:0/1450454955 learned_addr learned my addr 172.21.15.96:0/1450454955 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:49:28.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d47fff700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d480ffad0 0x7f3d4819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:28.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d477fe700 1 -- 172.21.15.96:0/1450454955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d480ffad0 msgr2=0x7f3d4819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d477fe700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d480ffad0 0x7f3d4819c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d477fe700 1 -- 172.21.15.96:0/1450454955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c0095f0 con 0x7f3d48105b60 2024-04-22T15:49:28.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d477fe700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d48105b60 0x7f3d4819cda0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c006010 tx=0x7f3d3c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:28.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d457fa700 1 -- 172.21.15.96:0/1450454955 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d3c01d070 con 0x7f3d48105b60 2024-04-22T15:49:28.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d48196c20 con 0x7f3d48105b60 2024-04-22T15:49:28.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.668+0000 7f3d457fa700 1 -- 172.21.15.96:0/1450454955 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d3c0056e0 con 0x7f3d48105b60 2024-04-22T15:49:28.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.669+0000 7f3d457fa700 1 -- 172.21.15.96:0/1450454955 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d3c017400 con 0x7f3d48105b60 2024-04-22T15:49:28.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.669+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d48197110 con 0x7f3d48105b60 2024-04-22T15:49:28.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.669+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d48109100 con 0x7f3d48105b60 2024-04-22T15:49:28.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.670+0000 7f3d457fa700 1 -- 172.21.15.96:0/1450454955 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3d3c00f6f0 con 0x7f3d48105b60 2024-04-22T15:49:28.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.671+0000 7f3d457fa700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d3006ecc0 0x7f3d30071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:28.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.671+0000 7f3d47fff700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d3006ecc0 0x7f3d30071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:28.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.671+0000 7f3d457fa700 1 -- 172.21.15.96:0/1450454955 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3d3c0904f0 con 0x7f3d48105b60 2024-04-22T15:49:28.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.671+0000 7f3d47fff700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d3006ecc0 0x7f3d30071180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009af0 tx=0x7f3d380094f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:28.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.675+0000 7f3d457fa700 1 -- 172.21.15.96:0/1450454955 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3d3c05cd60 con 0x7f3d48105b60 2024-04-22T15:49:28.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.893+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d48197ed0 con 0x7f3d3006ecc0 2024-04-22T15:49:28.897 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:28 smithi096 ceph-mon[112443]: from='client.35516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:28.897 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:28 smithi096 ceph-mon[112443]: from='client.35520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:28.897 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:28 smithi096 ceph-mon[112443]: pgmap v829: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:28.897 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:28 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3993541832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:49:28.897 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:28 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2480012019' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:49:28.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.895+0000 7f3d457fa700 1 -- 172.21.15.96:0/1450454955 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3d48197ed0 con 0x7f3d3006ecc0 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:49:28.898 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:49:28.899 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:49:28.899 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:49:28.899 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d3006ecc0 msgr2=0x7f3d30071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d3006ecc0 0x7f3d30071180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009af0 tx=0x7f3d380094f0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d48105b60 msgr2=0x7f3d4819cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d48105b60 0x7f3d4819cda0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c006010 tx=0x7f3d3c00be20 comp rx=0 tx=0).stop 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 shutdown_connections 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3d3006ecc0 0x7f3d30071180 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d480ffad0 0x7f3d4819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 --2- 172.21.15.96:0/1450454955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d48105b60 0x7f3d4819cda0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 >> 172.21.15.96:0/1450454955 conn(0x7f3d480fb5e0 msgr2=0x7f3d480695f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 shutdown_connections 2024-04-22T15:49:28.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:28.897+0000 7f3d4e58d700 1 -- 172.21.15.96:0/1450454955 wait complete. 2024-04-22T15:49:29.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.000+0000 7f67565a3700 1 -- 172.21.15.96:0/1000719578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6750105aa0 msgr2=0x7f6750105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.000+0000 7f67565a3700 1 --2- 172.21.15.96:0/1000719578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6750105aa0 0x7f6750105e80 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6740009910 tx=0x7f6740009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:29.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.000+0000 7f67565a3700 1 -- 172.21.15.96:0/1000719578 shutdown_connections 2024-04-22T15:49:29.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.000+0000 7f67565a3700 1 --2- 172.21.15.96:0/1000719578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f67500ffaa0 0x7f67500fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.000+0000 7f67565a3700 1 --2- 172.21.15.96:0/1000719578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6750105aa0 0x7f6750105e80 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.000+0000 7f67565a3700 1 -- 172.21.15.96:0/1000719578 >> 172.21.15.96:0/1000719578 conn(0x7f6750076740 msgr2=0x7f67500fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:29.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.000+0000 7f67565a3700 1 -- 172.21.15.96:0/1000719578 shutdown_connections 2024-04-22T15:49:29.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.000+0000 7f67565a3700 1 -- 172.21.15.96:0/1000719578 wait complete. 2024-04-22T15:49:29.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f67565a3700 1 Processor -- start 2024-04-22T15:49:29.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f67565a3700 1 -- start start 2024-04-22T15:49:29.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f67565a3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67500ffaa0 0x7f67501958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:29.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f67565a3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6750105aa0 0x7f6750195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:29.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f67565a3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6750069550 con 0x7f67500ffaa0 2024-04-22T15:49:29.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f67565a3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6750196330 con 0x7f6750105aa0 2024-04-22T15:49:29.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f674ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67500ffaa0 0x7f67501958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:29.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f674ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67500ffaa0 0x7f67501958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33976/0 (socket says 172.21.15.96:33976) 2024-04-22T15:49:29.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f674ffff700 1 -- 172.21.15.96:0/2437162411 learned_addr learned my addr 172.21.15.96:0/2437162411 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f674f7fe700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6750105aa0 0x7f6750195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f674ffff700 1 -- 172.21.15.96:0/2437162411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6750105aa0 msgr2=0x7f6750195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f674ffff700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6750105aa0 0x7f6750195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.001+0000 7f674ffff700 1 -- 172.21.15.96:0/2437162411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67400095f0 con 0x7f67500ffaa0 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.002+0000 7f674ffff700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67500ffaa0 0x7f67501958b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6740006010 tx=0x7f674000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.002+0000 7f674d7fa700 1 -- 172.21.15.96:0/2437162411 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674001d070 con 0x7f67500ffaa0 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.002+0000 7f674d7fa700 1 -- 172.21.15.96:0/2437162411 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f674000f460 con 0x7f67500ffaa0 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.002+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67501965b0 con 0x7f67500ffaa0 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.002+0000 7f674d7fa700 1 -- 172.21.15.96:0/2437162411 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6740017630 con 0x7f67500ffaa0 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.002+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67501a30f0 con 0x7f67500ffaa0 2024-04-22T15:49:29.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.003+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67501080a0 con 0x7f67500ffaa0 2024-04-22T15:49:29.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.003+0000 7f674d7fa700 1 -- 172.21.15.96:0/2437162411 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f674000f5d0 con 0x7f67500ffaa0 2024-04-22T15:49:29.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.004+0000 7f674d7fa700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673806ecc0 0x7f6738071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:29.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.004+0000 7f674d7fa700 1 -- 172.21.15.96:0/2437162411 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6740090340 con 0x7f67500ffaa0 2024-04-22T15:49:29.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.004+0000 7f674f7fe700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673806ecc0 0x7f6738071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:29.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.004+0000 7f674f7fe700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673806ecc0 0x7f6738071180 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6744006fd0 tx=0x7f6744008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:28 smithi196 ceph-mon[81815]: from='client.35516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:28 smithi196 ceph-mon[81815]: from='client.35520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:28 smithi196 ceph-mon[81815]: pgmap v829: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:28 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3993541832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:49:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:28 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2480012019' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:49:29.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.009+0000 7f674d7fa700 1 -- 172.21.15.96:0/2437162411 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f674005cd10 con 0x7f67500ffaa0 2024-04-22T15:49:29.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.294+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6750068b60 con 0x7f67500ffaa0 2024-04-22T15:49:29.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.294+0000 7f674d7fa700 1 -- 172.21.15.96:0/2437162411 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f674005c8a0 con 0x7f67500ffaa0 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:49:29.298 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:49:29.299 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.296+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673806ecc0 msgr2=0x7f6738071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.296+0000 7f67565a3700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673806ecc0 0x7f6738071180 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6744006fd0 tx=0x7f6744008040 comp rx=0 tx=0).stop 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.296+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67500ffaa0 msgr2=0x7f67501958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.296+0000 7f67565a3700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67500ffaa0 0x7f67501958b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6740006010 tx=0x7f674000be20 comp rx=0 tx=0).stop 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.297+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 shutdown_connections 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.297+0000 7f67565a3700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f673806ecc0 0x7f6738071180 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.297+0000 7f67565a3700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67500ffaa0 0x7f67501958b0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.297+0000 7f67565a3700 1 --2- 172.21.15.96:0/2437162411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6750105aa0 0x7f6750195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.297+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 >> 172.21.15.96:0/2437162411 conn(0x7f6750076740 msgr2=0x7f67500fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:29.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.297+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 shutdown_connections 2024-04-22T15:49:29.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:29.297+0000 7f67565a3700 1 -- 172.21.15.96:0/2437162411 wait complete. 2024-04-22T15:49:29.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:29 smithi096 ceph-mon[112443]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:29.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:29 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2437162411' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:49:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:29 smithi196 ceph-mon[81815]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:29 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2437162411' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:49:30.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:30 smithi096 ceph-mon[112443]: from='client.35536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:30.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:30 smithi096 ceph-mon[112443]: pgmap v830: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:30 smithi196 ceph-mon[81815]: from='client.35536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:30 smithi196 ceph-mon[81815]: pgmap v830: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:32.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:32 smithi096 ceph-mon[112443]: pgmap v831: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:32 smithi196 ceph-mon[81815]: pgmap v831: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:33.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:33 smithi096 ceph-mon[112443]: pgmap v832: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:33 smithi196 ceph-mon[81815]: pgmap v832: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:36 smithi096 ceph-mon[112443]: pgmap v833: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:36 smithi196 ceph-mon[81815]: pgmap v833: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:38.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:38 smithi096 ceph-mon[112443]: pgmap v834: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:38 smithi196 ceph-mon[81815]: pgmap v834: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:40 smithi096 ceph-mon[112443]: pgmap v835: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:40 smithi196 ceph-mon[81815]: pgmap v835: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:42 smithi096 ceph-mon[112443]: pgmap v836: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:42 smithi196 ceph-mon[81815]: pgmap v836: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:44 smithi096 ceph-mon[112443]: pgmap v837: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:44 smithi196 ceph-mon[81815]: pgmap v837: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:46 smithi096 ceph-mon[112443]: pgmap v838: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:46 smithi196 ceph-mon[81815]: pgmap v838: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:48.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:48 smithi096 ceph-mon[112443]: pgmap v839: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:48 smithi196 ceph-mon[81815]: pgmap v839: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:50.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:50 smithi096 ceph-mon[112443]: pgmap v840: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:50 smithi196 ceph-mon[81815]: pgmap v840: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:52.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:52 smithi096 ceph-mon[112443]: pgmap v841: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:52 smithi196 ceph-mon[81815]: pgmap v841: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:53.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:54 smithi096 ceph-mon[112443]: pgmap v842: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:54 smithi196 ceph-mon[81815]: pgmap v842: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:55.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:55 smithi096 ceph-mon[112443]: pgmap v843: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:55 smithi196 ceph-mon[81815]: pgmap v843: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:49:58 smithi096 ceph-mon[112443]: pgmap v844: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:49:58 smithi196 ceph-mon[81815]: pgmap v844: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:49:59.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.414+0000 7f60d8bd4700 1 -- 172.21.15.96:0/2678635290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d41017d0 msgr2=0x7f60d4105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:59.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.414+0000 7f60d8bd4700 1 --2- 172.21.15.96:0/2678635290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d41017d0 0x7f60d4105820 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009910 tx=0x7f60c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:59.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.414+0000 7f60d8bd4700 1 -- 172.21.15.96:0/2678635290 shutdown_connections 2024-04-22T15:49:59.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.414+0000 7f60d8bd4700 1 --2- 172.21.15.96:0/2678635290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d41017d0 0x7f60d4105820 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.414+0000 7f60d8bd4700 1 --2- 172.21.15.96:0/2678635290 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60d4100e20 0x7f60d4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.414+0000 7f60d8bd4700 1 -- 172.21.15.96:0/2678635290 >> 172.21.15.96:0/2678635290 conn(0x7f60d40fc6d0 msgr2=0x7f60d40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:59.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.414+0000 7f60d8bd4700 1 -- 172.21.15.96:0/2678635290 shutdown_connections 2024-04-22T15:49:59.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.414+0000 7f60d8bd4700 1 -- 172.21.15.96:0/2678635290 wait complete. 2024-04-22T15:49:59.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d8bd4700 1 Processor -- start 2024-04-22T15:49:59.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d8bd4700 1 -- start start 2024-04-22T15:49:59.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d8bd4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d4100e20 0x7f60d419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:59.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d8bd4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60d41017d0 0x7f60d419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:59.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d8bd4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d419d380 con 0x7f60d4100e20 2024-04-22T15:49:59.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d8bd4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4196960 con 0x7f60d41017d0 2024-04-22T15:49:59.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d4100e20 0x7f60d419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d4100e20 0x7f60d419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37086/0 (socket says 172.21.15.96:37086) 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d259c700 1 -- 172.21.15.96:0/3886525218 learned_addr learned my addr 172.21.15.96:0/3886525218 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d1d9b700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60d41017d0 0x7f60d419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d259c700 1 -- 172.21.15.96:0/3886525218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60d41017d0 msgr2=0x7f60d419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d259c700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60d41017d0 0x7f60d419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d259c700 1 -- 172.21.15.96:0/3886525218 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c80095f0 con 0x7f60d4100e20 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.415+0000 7f60d259c700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d4100e20 0x7f60d419c820 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0060b0 tx=0x7f60cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.416+0000 7f60c37fe700 1 -- 172.21.15.96:0/3886525218 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60cc00dda0 con 0x7f60d4100e20 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.416+0000 7f60c37fe700 1 -- 172.21.15.96:0/3886525218 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60cc00f040 con 0x7f60d4100e20 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.416+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d4196c40 con 0x7f60d4100e20 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.416+0000 7f60c37fe700 1 -- 172.21.15.96:0/3886525218 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60cc014790 con 0x7f60d4100e20 2024-04-22T15:49:59.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.416+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d4197190 con 0x7f60d4100e20 2024-04-22T15:49:59.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.417+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d4109220 con 0x7f60d4100e20 2024-04-22T15:49:59.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.417+0000 7f60c37fe700 1 -- 172.21.15.96:0/3886525218 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f60cc0148f0 con 0x7f60d4100e20 2024-04-22T15:49:59.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.418+0000 7f60c37fe700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60bc06ecc0 0x7f60bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:59.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.418+0000 7f60c37fe700 1 -- 172.21.15.96:0/3886525218 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f60cc08e860 con 0x7f60d4100e20 2024-04-22T15:49:59.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.418+0000 7f60d1d9b700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60bc06ecc0 0x7f60bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:59.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.418+0000 7f60d1d9b700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60bc06ecc0 0x7f60bc071180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f60d4197b20 tx=0x7f60c801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:59.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.423+0000 7f60c37fe700 1 -- 172.21.15.96:0/3886525218 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f60cc05b220 con 0x7f60d4100e20 2024-04-22T15:49:59.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.638+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60d4197e90 con 0x7f60bc06ecc0 2024-04-22T15:49:59.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.639+0000 7f60c37fe700 1 -- 172.21.15.96:0/3886525218 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f60d4197e90 con 0x7f60bc06ecc0 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60bc06ecc0 msgr2=0x7f60bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60bc06ecc0 0x7f60bc071180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f60d4197b20 tx=0x7f60c801a040 comp rx=0 tx=0).stop 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d4100e20 msgr2=0x7f60d419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d4100e20 0x7f60d419c820 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0060b0 tx=0x7f60cc00d3b0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 shutdown_connections 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60bc06ecc0 0x7f60bc071180 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d4100e20 0x7f60d419c820 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 --2- 172.21.15.96:0/3886525218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60d41017d0 0x7f60d419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 >> 172.21.15.96:0/3886525218 conn(0x7f60d40fc6d0 msgr2=0x7f60d40fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 shutdown_connections 2024-04-22T15:49:59.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.642+0000 7f60d8bd4700 1 -- 172.21.15.96:0/3886525218 wait complete. 2024-04-22T15:49:59.655 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:49:59.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.746+0000 7f10b8901700 1 -- 172.21.15.96:0/2996398751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b41026d0 msgr2=0x7f10b410abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:59.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.746+0000 7f10b8901700 1 --2- 172.21.15.96:0/2996398751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b41026d0 0x7f10b410abc0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f109c009910 tx=0x7f109c009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:59.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.746+0000 7f10b8901700 1 -- 172.21.15.96:0/2996398751 shutdown_connections 2024-04-22T15:49:59.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.746+0000 7f10b8901700 1 --2- 172.21.15.96:0/2996398751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b41026d0 0x7f10b410abc0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.746+0000 7f10b8901700 1 --2- 172.21.15.96:0/2996398751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10b4101db0 0x7f10b4102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.746+0000 7f10b8901700 1 -- 172.21.15.96:0/2996398751 >> 172.21.15.96:0/2996398751 conn(0x7f10b40fb5e0 msgr2=0x7f10b40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:59.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.746+0000 7f10b8901700 1 -- 172.21.15.96:0/2996398751 shutdown_connections 2024-04-22T15:49:59.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.746+0000 7f10b8901700 1 -- 172.21.15.96:0/2996398751 wait complete. 2024-04-22T15:49:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.746+0000 7f10b8901700 1 Processor -- start 2024-04-22T15:49:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b8901700 1 -- start start 2024-04-22T15:49:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b8901700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b4101db0 0x7f10b40ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b8901700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10b41026d0 0x7f10b40fff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b8901700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b4100490 con 0x7f10b4101db0 2024-04-22T15:49:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b8901700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b41005d0 con 0x7f10b41026d0 2024-04-22T15:49:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b4101db0 0x7f10b40ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b4101db0 0x7f10b40ffa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37102/0 (socket says 172.21.15.96:37102) 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b259c700 1 -- 172.21.15.96:0/874743487 learned_addr learned my addr 172.21.15.96:0/874743487 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10a9d9b700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10b41026d0 0x7f10b40fff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b259c700 1 -- 172.21.15.96:0/874743487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10b41026d0 msgr2=0x7f10b40fff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b259c700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10b41026d0 0x7f10b40fff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b259c700 1 -- 172.21.15.96:0/874743487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109c0095f0 con 0x7f10b4101db0 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.747+0000 7f10b259c700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b4101db0 0x7f10b40ffa10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f10a400d8d0 tx=0x7f10a400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.748+0000 7f10abfff700 1 -- 172.21.15.96:0/874743487 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10a40098a0 con 0x7f10b4101db0 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.748+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b41a2d10 con 0x7f10b4101db0 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.748+0000 7f10abfff700 1 -- 172.21.15.96:0/874743487 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10a4010460 con 0x7f10b4101db0 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.748+0000 7f10abfff700 1 -- 172.21.15.96:0/874743487 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10a400f5f0 con 0x7f10b4101db0 2024-04-22T15:49:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.748+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b41a3070 con 0x7f10b4101db0 2024-04-22T15:49:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.749+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b4069990 con 0x7f10b4101db0 2024-04-22T15:49:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.749+0000 7f10abfff700 1 -- 172.21.15.96:0/874743487 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f10a4009a00 con 0x7f10b4101db0 2024-04-22T15:49:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.750+0000 7f10abfff700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10a006ecc0 0x7f10a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.750+0000 7f10a9d9b700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10a006ecc0 0x7f10a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.750+0000 7f10abfff700 1 -- 172.21.15.96:0/874743487 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f10a408ec90 con 0x7f10b4101db0 2024-04-22T15:49:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.750+0000 7f10a9d9b700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10a006ecc0 0x7f10a0071180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f10b4101140 tx=0x7f109c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:59.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.754+0000 7f10abfff700 1 -- 172.21.15.96:0/874743487 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f10a4057a70 con 0x7f10b4101db0 2024-04-22T15:49:59.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.974+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f10b410de10 con 0x7f10a006ecc0 2024-04-22T15:49:59.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.976+0000 7f10abfff700 1 -- 172.21.15.96:0/874743487 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f10b410de10 con 0x7f10a006ecc0 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10a006ecc0 msgr2=0x7f10a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10a006ecc0 0x7f10a0071180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f10b4101140 tx=0x7f109c01a040 comp rx=0 tx=0).stop 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b4101db0 msgr2=0x7f10b40ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b4101db0 0x7f10b40ffa10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f10a400d8d0 tx=0x7f10a400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 shutdown_connections 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f10a006ecc0 0x7f10a0071180 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10b4101db0 0x7f10b40ffa10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 --2- 172.21.15.96:0/874743487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10b41026d0 0x7f10b40fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 >> 172.21.15.96:0/874743487 conn(0x7f10b40fb5e0 msgr2=0x7f10b41030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 shutdown_connections 2024-04-22T15:49:59.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:49:59.978+0000 7f10b8901700 1 -- 172.21.15.96:0/874743487 wait complete. 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 -- 172.21.15.96:0/3895861488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 msgr2=0x7f6aa0105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 --2- 172.21.15.96:0/3895861488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 0x7f6aa0105fe0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009910 tx=0x7f6a90009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 -- 172.21.15.96:0/3895861488 shutdown_connections 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 --2- 172.21.15.96:0/3895861488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aa00ffb50 0x7f6aa00fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 --2- 172.21.15.96:0/3895861488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 0x7f6aa0105fe0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 -- 172.21.15.96:0/3895861488 >> 172.21.15.96:0/3895861488 conn(0x7f6aa00fb680 msgr2=0x7f6aa00fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 -- 172.21.15.96:0/3895861488 shutdown_connections 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 -- 172.21.15.96:0/3895861488 wait complete. 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 Processor -- start 2024-04-22T15:50:00.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.082+0000 7f6aa67df700 1 -- start start 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6aa67df700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aa00ffb50 0x7f6aa019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6aa67df700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 0x7f6aa019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6aa67df700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa0196910 con 0x7f6aa0105c00 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6aa67df700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa0196a80 con 0x7f6aa00ffb50 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 0x7f6aa019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 0x7f6aa019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37116/0 (socket says 172.21.15.96:37116) 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9f7fe700 1 -- 172.21.15.96:0/1230190706 learned_addr learned my addr 172.21.15.96:0/1230190706 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9ffff700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aa00ffb50 0x7f6aa019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9f7fe700 1 -- 172.21.15.96:0/1230190706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aa00ffb50 msgr2=0x7f6aa019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9f7fe700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aa00ffb50 0x7f6aa019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9f7fe700 1 -- 172.21.15.96:0/1230190706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a900095f0 con 0x7f6aa0105c00 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9f7fe700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 0x7f6aa019cda0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400c8a0 tx=0x7f6a9400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:00.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9d7fa700 1 -- 172.21.15.96:0/1230190706 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a9400cea0 con 0x7f6aa0105c00 2024-04-22T15:50:00.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa0196d60 con 0x7f6aa0105c00 2024-04-22T15:50:00.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9d7fa700 1 -- 172.21.15.96:0/1230190706 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a94004830 con 0x7f6aa0105c00 2024-04-22T15:50:00.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6a9d7fa700 1 -- 172.21.15.96:0/1230190706 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a940055b0 con 0x7f6aa0105c00 2024-04-22T15:50:00.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.083+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa01972b0 con 0x7f6aa0105c00 2024-04-22T15:50:00.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.084+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa0101200 con 0x7f6aa0105c00 2024-04-22T15:50:00.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.085+0000 7f6a9d7fa700 1 -- 172.21.15.96:0/1230190706 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6a940049a0 con 0x7f6aa0105c00 2024-04-22T15:50:00.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.085+0000 7f6a9d7fa700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a8806ecd0 0x7f6a88071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:00.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.085+0000 7f6a9d7fa700 1 -- 172.21.15.96:0/1230190706 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6a9408eb70 con 0x7f6aa0105c00 2024-04-22T15:50:00.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.085+0000 7f6a9ffff700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a8806ecd0 0x7f6a88071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:00.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.086+0000 7f6a9ffff700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a8806ecd0 0x7f6a88071190 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000b5c0 tx=0x7f6a9001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:00.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.090+0000 7f6a9d7fa700 1 -- 172.21.15.96:0/1230190706 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6a940578d0 con 0x7f6aa0105c00 2024-04-22T15:50:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.299+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6aa0197580 con 0x7f6a8806ecd0 2024-04-22T15:50:00.303 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:50:00.303 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:50:00.303 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:50:00.303 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-04-22T15:50:00.303 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:00.303 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-04-22T15:50:00.303 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:50:00.303 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : return _run_code(code, main_globals, None, 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-22T15:50:00.304 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:50:00.305 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:50:00.305 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:50:00.305 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T15:49:59.999+0000 7f5ddaaca700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-22T15:50:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.311+0000 7f6a9d7fa700 1 -- 172.21.15.96:0/1230190706 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6aa0197580 con 0x7f6a8806ecd0 2024-04-22T15:50:00.314 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (28m) 6m ago 39m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (40m) 6m ago 40m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (39m) 6m ago 39m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (25m) 6m ago 40m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (25m) 6m ago 39m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (28m) 6m ago 39m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (21m) 6m ago 34m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (22m) 6m ago 34m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (21m) 6m ago 34m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (6m) 6m ago 34m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (31m) 6m ago 41m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (29m) 6m ago 38m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:50:00.315 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (27m) 6m ago 41m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (26m) 6m ago 38m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (29m) 6m ago 40m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (29m) 6m ago 38m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (25m) 6m ago 38m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (24m) 6m ago 37m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (24m) 6m ago 37m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (23m) 6m ago 36m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (23m) 6m ago 36m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (22m) 6m ago 35m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:50:00.316 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (29m) 6m ago 39m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:50:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.313+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a8806ecd0 msgr2=0x7f6a88071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.313+0000 7f6aa67df700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a8806ecd0 0x7f6a88071190 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000b5c0 tx=0x7f6a9001a040 comp rx=0 tx=0).stop 2024-04-22T15:50:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.313+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 msgr2=0x7f6aa019cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.313+0000 7f6aa67df700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 0x7f6aa019cda0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400c8a0 tx=0x7f6a9400cc60 comp rx=0 tx=0).stop 2024-04-22T15:50:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.314+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 shutdown_connections 2024-04-22T15:50:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.314+0000 7f6aa67df700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a8806ecd0 0x7f6a88071190 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.314+0000 7f6aa67df700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6aa00ffb50 0x7f6aa019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.314+0000 7f6aa67df700 1 --2- 172.21.15.96:0/1230190706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aa0105c00 0x7f6aa019cda0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.314+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 >> 172.21.15.96:0/1230190706 conn(0x7f6aa00fb680 msgr2=0x7f6aa0069370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:00.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.314+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 shutdown_connections 2024-04-22T15:50:00.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.314+0000 7f6aa67df700 1 -- 172.21.15.96:0/1230190706 wait complete. 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.416+0000 7f60bfd39700 1 -- 172.21.15.96:0/3145542461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b81017c0 msgr2=0x7f60b8105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.416+0000 7f60bfd39700 1 --2- 172.21.15.96:0/3145542461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b81017c0 0x7f60b8105810 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009910 tx=0x7f60a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.416+0000 7f60bfd39700 1 -- 172.21.15.96:0/3145542461 shutdown_connections 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.416+0000 7f60bfd39700 1 --2- 172.21.15.96:0/3145542461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b81017c0 0x7f60b8105810 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.416+0000 7f60bfd39700 1 --2- 172.21.15.96:0/3145542461 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60b8100e10 0x7f60b81011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.416+0000 7f60bfd39700 1 -- 172.21.15.96:0/3145542461 >> 172.21.15.96:0/3145542461 conn(0x7f60b80fc6e0 msgr2=0x7f60b80feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.416+0000 7f60bfd39700 1 -- 172.21.15.96:0/3145542461 shutdown_connections 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.416+0000 7f60bfd39700 1 -- 172.21.15.96:0/3145542461 wait complete. 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bfd39700 1 Processor -- start 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bfd39700 1 -- start start 2024-04-22T15:50:00.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bfd39700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b8100e10 0x7f60b819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bfd39700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60b81017c0 0x7f60b819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bfd39700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b819d490 con 0x7f60b8100e10 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bfd39700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b8196a70 con 0x7f60b81017c0 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bdad5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b8100e10 0x7f60b819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bdad5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b8100e10 0x7f60b819c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37138/0 (socket says 172.21.15.96:37138) 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bdad5700 1 -- 172.21.15.96:0/3125118663 learned_addr learned my addr 172.21.15.96:0/3125118663 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bdad5700 1 -- 172.21.15.96:0/3125118663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60b81017c0 msgr2=0x7f60b819ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bd2d4700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60b81017c0 0x7f60b819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bdad5700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60b81017c0 0x7f60b819ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bdad5700 1 -- 172.21.15.96:0/3125118663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a80095f0 con 0x7f60b8100e10 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60bdad5700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b8100e10 0x7f60b819c930 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f60b000b4b0 tx=0x7f60b000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.417+0000 7f60aeffd700 1 -- 172.21.15.96:0/3125118663 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60b0011800 con 0x7f60b8100e10 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.418+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b8196d50 con 0x7f60b8100e10 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.418+0000 7f60aeffd700 1 -- 172.21.15.96:0/3125118663 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60b0011e20 con 0x7f60b8100e10 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.418+0000 7f60aeffd700 1 -- 172.21.15.96:0/3125118663 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60b000f2e0 con 0x7f60b8100e10 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.418+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b81972a0 con 0x7f60b8100e10 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.418+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b81022f0 con 0x7f60b8100e10 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.419+0000 7f60aeffd700 1 -- 172.21.15.96:0/3125118663 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f60b0010380 con 0x7f60b8100e10 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.419+0000 7f60aeffd700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60a406ecc0 0x7f60a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.419+0000 7f60aeffd700 1 -- 172.21.15.96:0/3125118663 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f60b008eaa0 con 0x7f60b8100e10 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.419+0000 7f60bd2d4700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60a406ecc0 0x7f60a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:00.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.420+0000 7f60bd2d4700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60a406ecc0 0x7f60a4071180 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f60a80059d0 tx=0x7f60a8005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:00.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.424+0000 7f60aeffd700 1 -- 172.21.15.96:0/3125118663 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f60b005b330 con 0x7f60b8100e10 2024-04-22T15:50:00.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.726+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f60b804ebb0 con 0x7f60b8100e10 2024-04-22T15:50:00.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.727+0000 7f60aeffd700 1 -- 172.21.15.96:0/3125118663 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f60b005aec0 con 0x7f60b8100e10 2024-04-22T15:50:00.730 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:50:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:50:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:50:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:50:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:50:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:50:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:50:00.731 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60a406ecc0 msgr2=0x7f60a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60a406ecc0 0x7f60a4071180 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f60a80059d0 tx=0x7f60a8005960 comp rx=0 tx=0).stop 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b8100e10 msgr2=0x7f60b819c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b8100e10 0x7f60b819c930 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f60b000b4b0 tx=0x7f60b000b870 comp rx=0 tx=0).stop 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 shutdown_connections 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60a406ecc0 0x7f60a4071180 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b8100e10 0x7f60b819c930 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 --2- 172.21.15.96:0/3125118663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60b81017c0 0x7f60b819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 >> 172.21.15.96:0/3125118663 conn(0x7f60b80fc6e0 msgr2=0x7f60b81047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:00.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 shutdown_connections 2024-04-22T15:50:00.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.729+0000 7f60bfd39700 1 -- 172.21.15.96:0/3125118663 wait complete. 2024-04-22T15:50:00.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.832+0000 7f55f0d7f700 1 -- 172.21.15.96:0/3470220463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 msgr2=0x7f55ec1057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.832+0000 7f55f0d7f700 1 --2- 172.21.15.96:0/3470220463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 0x7f55ec1057f0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009910 tx=0x7f55e0009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:00.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.833+0000 7f55f0d7f700 1 -- 172.21.15.96:0/3470220463 shutdown_connections 2024-04-22T15:50:00.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.833+0000 7f55f0d7f700 1 --2- 172.21.15.96:0/3470220463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 0x7f55ec1057f0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.833+0000 7f55f0d7f700 1 --2- 172.21.15.96:0/3470220463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55ec100df0 0x7f55ec1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.833+0000 7f55f0d7f700 1 -- 172.21.15.96:0/3470220463 >> 172.21.15.96:0/3470220463 conn(0x7f55ec0fc6a0 msgr2=0x7f55ec0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:00.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.833+0000 7f55f0d7f700 1 -- 172.21.15.96:0/3470220463 shutdown_connections 2024-04-22T15:50:00.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.833+0000 7f55f0d7f700 1 -- 172.21.15.96:0/3470220463 wait complete. 2024-04-22T15:50:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.833+0000 7f55f0d7f700 1 Processor -- start 2024-04-22T15:50:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.833+0000 7f55f0d7f700 1 -- start start 2024-04-22T15:50:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55f0d7f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55ec100df0 0x7f55ec19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55f0d7f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 0x7f55ec19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55f0d7f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec19d450 con 0x7f55ec1017a0 2024-04-22T15:50:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55f0d7f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec196a30 con 0x7f55ec100df0 2024-04-22T15:50:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55e9d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 0x7f55ec19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55e9d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 0x7f55ec19ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34320/0 (socket says 172.21.15.96:34320) 2024-04-22T15:50:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55e9d9b700 1 -- 172.21.15.96:0/2073714403 learned_addr learned my addr 172.21.15.96:0/2073714403 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55ea59c700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55ec100df0 0x7f55ec19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55e9d9b700 1 -- 172.21.15.96:0/2073714403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55ec100df0 msgr2=0x7f55ec19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55e9d9b700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55ec100df0 0x7f55ec19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55e9d9b700 1 -- 172.21.15.96:0/2073714403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e00095f0 con 0x7f55ec1017a0 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55e9d9b700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 0x7f55ec19ce30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f55e00092e0 tx=0x7f55e000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55db7fe700 1 -- 172.21.15.96:0/2073714403 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55e001d070 con 0x7f55ec1017a0 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55db7fe700 1 -- 172.21.15.96:0/2073714403 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55e000f460 con 0x7f55ec1017a0 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ec196cb0 con 0x7f55ec1017a0 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.834+0000 7f55db7fe700 1 -- 172.21.15.96:0/2073714403 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55e0017610 con 0x7f55ec1017a0 2024-04-22T15:50:00.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.835+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ec1971a0 con 0x7f55ec1017a0 2024-04-22T15:50:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.836+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ec109140 con 0x7f55ec1017a0 2024-04-22T15:50:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.836+0000 7f55db7fe700 1 -- 172.21.15.96:0/2073714403 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f55e0003a60 con 0x7f55ec1017a0 2024-04-22T15:50:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.836+0000 7f55db7fe700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d406ecd0 0x7f55d4071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.836+0000 7f55db7fe700 1 -- 172.21.15.96:0/2073714403 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f55e0090310 con 0x7f55ec1017a0 2024-04-22T15:50:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.836+0000 7f55ea59c700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d406ecd0 0x7f55d4071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.837+0000 7f55ea59c700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d406ecd0 0x7f55d4071190 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009a10 tx=0x7f55dc009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:00.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:00.841+0000 7f55db7fe700 1 -- 172.21.15.96:0/2073714403 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f55e005cba0 con 0x7f55ec1017a0 2024-04-22T15:50:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: pgmap v845: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: from='client.35544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:50:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:50:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:50:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: /usr/bin/podman: timeout after 895 seconds 2024-04-22T15:50:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: Traceback (most recent call last): 2024-04-22T15:50:00.903 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: return _run_code(code, main_globals, None, 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: exec(code, run_globals) 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: fs cephfs is degraded 2024-04-22T15:50:00.904 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:50:00.905 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:50:00.905 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:50:00.905 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:00 smithi096 ceph-mon[112443]: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:50:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: pgmap v845: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: from='client.35544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: /usr/bin/podman: timeout after 895 seconds 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: Traceback (most recent call last): 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: return _run_code(code, main_globals, None, 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: exec(code, run_globals) 2024-04-22T15:50:01.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: fs cephfs is degraded 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:50:01.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:00 smithi196 ceph-mon[81815]: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:50:01.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.091+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f55ec197ef0 con 0x7f55ec1017a0 2024-04-22T15:50:01.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.092+0000 7f55db7fe700 1 -- 172.21.15.96:0/2073714403 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f55e005c730 con 0x7f55ec1017a0 2024-04-22T15:50:01.095 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:50:01.095 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:50:01.095 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:50:01.095 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:50:01.095 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:50:01.095 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:50:01.095 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:50:01.095 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:50:01.096 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:50:01.097 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:50:01.098 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:50:01.098 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:50:01.098 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:50:01.098 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:50:01.098 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:50:01.098 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:50:01.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d406ecd0 msgr2=0x7f55d4071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d406ecd0 0x7f55d4071190 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009a10 tx=0x7f55dc009190 comp rx=0 tx=0).stop 2024-04-22T15:50:01.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 msgr2=0x7f55ec19ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 0x7f55ec19ce30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f55e00092e0 tx=0x7f55e000bd70 comp rx=0 tx=0).stop 2024-04-22T15:50:01.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 shutdown_connections 2024-04-22T15:50:01.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f55d406ecd0 0x7f55d4071190 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55ec100df0 0x7f55ec19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 --2- 172.21.15.96:0/2073714403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55ec1017a0 0x7f55ec19ce30 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 >> 172.21.15.96:0/2073714403 conn(0x7f55ec0fc6a0 msgr2=0x7f55ec0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:01.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 shutdown_connections 2024-04-22T15:50:01.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.094+0000 7f55f0d7f700 1 -- 172.21.15.96:0/2073714403 wait complete. 2024-04-22T15:50:01.100 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:50:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 -- 172.21.15.96:0/3314201598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 msgr2=0x7fb838111690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 --2- 172.21.15.96:0/3314201598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 0x7fb838111690 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb828009910 tx=0x7fb828009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 -- 172.21.15.96:0/3314201598 shutdown_connections 2024-04-22T15:50:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 --2- 172.21.15.96:0/3314201598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 0x7fb838111690 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 --2- 172.21.15.96:0/3314201598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb838073280 0x7fb838073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 -- 172.21.15.96:0/3314201598 >> 172.21.15.96:0/3314201598 conn(0x7fb8380fc6d0 msgr2=0x7fb8380feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 -- 172.21.15.96:0/3314201598 shutdown_connections 2024-04-22T15:50:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 -- 172.21.15.96:0/3314201598 wait complete. 2024-04-22T15:50:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 Processor -- start 2024-04-22T15:50:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.198+0000 7fb83ed44700 1 -- start start 2024-04-22T15:50:01.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb83ed44700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb838073280 0x7fb83819d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:01.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb83ed44700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 0x7fb83819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:01.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb83ed44700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83819dcd0 con 0x7fb838073ba0 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb83ed44700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8381a19a0 con 0x7fb838073280 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb82ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 0x7fb83819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb82ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 0x7fb83819d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34352/0 (socket says 172.21.15.96:34352) 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb82ffff700 1 -- 172.21.15.96:0/2167386589 learned_addr learned my addr 172.21.15.96:0/2167386589 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb83cae0700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb838073280 0x7fb83819d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb82ffff700 1 -- 172.21.15.96:0/2167386589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb838073280 msgr2=0x7fb83819d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb82ffff700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb838073280 0x7fb83819d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb82ffff700 1 -- 172.21.15.96:0/2167386589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8280095f0 con 0x7fb838073ba0 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb83cae0700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb838073280 0x7fb83819d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb82ffff700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 0x7fb83819d5f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb82800bd70 tx=0x7fb82800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb82dffb700 1 -- 172.21.15.96:0/2167386589 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb82801d070 con 0x7fb838073ba0 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8381a1c80 con 0x7fb838073ba0 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.199+0000 7fb82dffb700 1 -- 172.21.15.96:0/2167386589 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb82800f460 con 0x7fb838073ba0 2024-04-22T15:50:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.200+0000 7fb82dffb700 1 -- 172.21.15.96:0/2167386589 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb828017630 con 0x7fb838073ba0 2024-04-22T15:50:01.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.200+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8381a21d0 con 0x7fb838073ba0 2024-04-22T15:50:01.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.201+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83810ee10 con 0x7fb838073ba0 2024-04-22T15:50:01.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.201+0000 7fb82dffb700 1 -- 172.21.15.96:0/2167386589 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb82800f5d0 con 0x7fb838073ba0 2024-04-22T15:50:01.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.201+0000 7fb82dffb700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb82006ecc0 0x7fb820071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.202+0000 7fb82dffb700 1 -- 172.21.15.96:0/2167386589 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb8280903a0 con 0x7fb838073ba0 2024-04-22T15:50:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.202+0000 7fb83cae0700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb82006ecc0 0x7fb820071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.202+0000 7fb83cae0700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb82006ecc0 0x7fb820071180 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb834005950 tx=0x7fb834005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:01.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.207+0000 7fb82dffb700 1 -- 172.21.15.96:0/2167386589 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb828059100 con 0x7fb838073ba0 2024-04-22T15:50:01.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.423+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb83819e430 con 0x7fb82006ecc0 2024-04-22T15:50:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.425+0000 7fb82dffb700 1 -- 172.21.15.96:0/2167386589 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb83819e430 con 0x7fb82006ecc0 2024-04-22T15:50:01.427 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:50:01.428 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb82006ecc0 msgr2=0x7fb820071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb82006ecc0 0x7fb820071180 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb834005950 tx=0x7fb834005f50 comp rx=0 tx=0).stop 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 msgr2=0x7fb83819d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 0x7fb83819d5f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb82800bd70 tx=0x7fb82800be50 comp rx=0 tx=0).stop 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 shutdown_connections 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb82006ecc0 0x7fb820071180 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb838073280 0x7fb83819d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 --2- 172.21.15.96:0/2167386589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb838073ba0 0x7fb83819d5f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 >> 172.21.15.96:0/2167386589 conn(0x7fb8380fc6d0 msgr2=0x7fb838104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 shutdown_connections 2024-04-22T15:50:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.427+0000 7fb83ed44700 1 -- 172.21.15.96:0/2167386589 wait complete. 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.530+0000 7f5ab880b700 1 -- 172.21.15.96:0/1680391131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 msgr2=0x7f5ab4105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.530+0000 7f5ab880b700 1 --2- 172.21.15.96:0/1680391131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 0x7f5ab4105820 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009910 tx=0x7f5aa8009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.531+0000 7f5ab880b700 1 -- 172.21.15.96:0/1680391131 shutdown_connections 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.531+0000 7f5ab880b700 1 --2- 172.21.15.96:0/1680391131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 0x7f5ab4105820 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.531+0000 7f5ab880b700 1 --2- 172.21.15.96:0/1680391131 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ab4100e20 0x7f5ab4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.531+0000 7f5ab880b700 1 -- 172.21.15.96:0/1680391131 >> 172.21.15.96:0/1680391131 conn(0x7f5ab40fc6d0 msgr2=0x7f5ab40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.531+0000 7f5ab880b700 1 -- 172.21.15.96:0/1680391131 shutdown_connections 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.531+0000 7f5ab880b700 1 -- 172.21.15.96:0/1680391131 wait complete. 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.531+0000 7f5ab880b700 1 Processor -- start 2024-04-22T15:50:01.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.531+0000 7f5ab880b700 1 -- start start 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab880b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ab4100e20 0x7f5ab419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab880b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 0x7f5ab419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab880b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab419d380 con 0x7f5ab41017d0 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab880b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab4196960 con 0x7f5ab4100e20 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab1d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 0x7f5ab419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab1d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 0x7f5ab419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34376/0 (socket says 172.21.15.96:34376) 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab1d9b700 1 -- 172.21.15.96:0/2951623453 learned_addr learned my addr 172.21.15.96:0/2951623453 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab259c700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ab4100e20 0x7f5ab419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab1d9b700 1 -- 172.21.15.96:0/2951623453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ab4100e20 msgr2=0x7f5ab419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab1d9b700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ab4100e20 0x7f5ab419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab1d9b700 1 -- 172.21.15.96:0/2951623453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa80095f0 con 0x7f5ab41017d0 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab259c700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ab4100e20 0x7f5ab419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:50:01.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab1d9b700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 0x7f5ab419cd60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8000c00 tx=0x7f5aa800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5aa37fe700 1 -- 172.21.15.96:0/2951623453 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa801d070 con 0x7f5ab41017d0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5aa37fe700 1 -- 172.21.15.96:0/2951623453 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5aa8005700 con 0x7f5ab41017d0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab4196be0 con 0x7f5ab41017d0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5aa37fe700 1 -- 172.21.15.96:0/2951623453 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa800fe50 con 0x7f5ab41017d0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.532+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab41970d0 con 0x7f5ab41017d0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.533+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab4109220 con 0x7f5ab41017d0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.534+0000 7f5aa37fe700 1 -- 172.21.15.96:0/2951623453 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5aa800f640 con 0x7f5ab41017d0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.534+0000 7f5aa37fe700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5a9c06ecd0 0x7f5a9c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.534+0000 7f5aa37fe700 1 -- 172.21.15.96:0/2951623453 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5aa80902b0 con 0x7f5ab41017d0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.534+0000 7f5ab259c700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5a9c06ecd0 0x7f5a9c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:01.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.535+0000 7f5ab259c700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5a9c06ecd0 0x7f5a9c071190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400bcf0 tx=0x7f5aa400b2c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:01.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.539+0000 7f5aa37fe700 1 -- 172.21.15.96:0/2951623453 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5aa805cb40 con 0x7f5ab41017d0 2024-04-22T15:50:01.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.824+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5ab4102300 con 0x7f5ab41017d0 2024-04-22T15:50:01.827 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:01 smithi096 ceph-mon[112443]: from='client.35548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:01.827 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:01 smithi096 ceph-mon[112443]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:01.827 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:01 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3125118663' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:01.827 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:01 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2073714403' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:50:01.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.824+0000 7f5aa37fe700 1 -- 172.21.15.96:0/2951623453 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f5aa805c6d0 con 0x7f5ab41017d0 2024-04-22T15:50:01.828 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:50:01.828 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:50:01.828 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:50:01.828 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:50:01.828 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:01.828 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:50:01.828 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:50:01.828 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:50:01.829 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:50:01.830 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:50:01.830 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:50:01.830 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:50:01.830 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5a9c06ecd0 msgr2=0x7f5a9c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5a9c06ecd0 0x7f5a9c071190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400bcf0 tx=0x7f5aa400b2c0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 msgr2=0x7f5ab419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 0x7f5ab419cd60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8000c00 tx=0x7f5aa800be20 comp rx=0 tx=0).stop 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 shutdown_connections 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5a9c06ecd0 0x7f5a9c071190 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ab4100e20 0x7f5ab419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 --2- 172.21.15.96:0/2951623453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ab41017d0 0x7f5ab419cd60 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 >> 172.21.15.96:0/2951623453 conn(0x7f5ab40fc6d0 msgr2=0x7f5ab40fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 shutdown_connections 2024-04-22T15:50:01.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:01.827+0000 7f5ab880b700 1 -- 172.21.15.96:0/2951623453 wait complete. 2024-04-22T15:50:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:01 smithi196 ceph-mon[81815]: from='client.35548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:01 smithi196 ceph-mon[81815]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:01 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3125118663' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:01 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2073714403' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:50:02.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:02 smithi096 ceph-mon[112443]: from='client.35564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:02.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:02 smithi096 ceph-mon[112443]: pgmap v846: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:02.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:02 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2951623453' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:50:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:02 smithi196 ceph-mon[81815]: from='client.35564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:02 smithi196 ceph-mon[81815]: pgmap v846: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:02 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2951623453' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:50:04.955 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:04 smithi196 ceph-mon[81815]: pgmap v847: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:04 smithi096 ceph-mon[112443]: pgmap v847: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:06.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:06 smithi096 ceph-mon[112443]: pgmap v848: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:06 smithi196 ceph-mon[81815]: pgmap v848: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:08.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:08 smithi096 ceph-mon[112443]: pgmap v849: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:08 smithi196 ceph-mon[81815]: pgmap v849: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:10.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:10 smithi096 ceph-mon[112443]: pgmap v850: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:10 smithi196 ceph-mon[81815]: pgmap v850: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:12 smithi096 ceph-mon[112443]: pgmap v851: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:12 smithi196 ceph-mon[81815]: pgmap v851: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:14 smithi096 ceph-mon[112443]: pgmap v852: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:14 smithi196 ceph-mon[81815]: pgmap v852: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:16.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:16 smithi096 ceph-mon[112443]: pgmap v853: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:16 smithi196 ceph-mon[81815]: pgmap v853: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:17.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:17 smithi096 ceph-mon[112443]: pgmap v854: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:17 smithi196 ceph-mon[81815]: pgmap v854: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:20.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:20 smithi096 ceph-mon[112443]: pgmap v855: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:20 smithi196 ceph-mon[81815]: pgmap v855: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:22.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:22 smithi096 ceph-mon[112443]: pgmap v856: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:22 smithi196 ceph-mon[81815]: pgmap v856: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:24.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:24 smithi096 ceph-mon[112443]: pgmap v857: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:24 smithi196 ceph-mon[81815]: pgmap v857: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:26.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:26 smithi096 ceph-mon[112443]: pgmap v858: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:27.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:26 smithi196 ceph-mon[81815]: pgmap v858: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:28.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:28 smithi096 ceph-mon[112443]: pgmap v859: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:28 smithi196 ceph-mon[81815]: pgmap v859: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:30.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:30 smithi096 ceph-mon[112443]: pgmap v860: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:30 smithi196 ceph-mon[81815]: pgmap v860: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:31.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.944+0000 7fb5e0284700 1 -- 172.21.15.96:0/578960427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 msgr2=0x7fb5d8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:31.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.944+0000 7fb5e0284700 1 --2- 172.21.15.96:0/578960427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 0x7fb5d8101200 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000b0a0 tx=0x7fb5d000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:50:31.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.944+0000 7fb5e0284700 1 -- 172.21.15.96:0/578960427 shutdown_connections 2024-04-22T15:50:31.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.944+0000 7fb5e0284700 1 --2- 172.21.15.96:0/578960427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5d81017d0 0x7fb5d8105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:31.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.944+0000 7fb5e0284700 1 --2- 172.21.15.96:0/578960427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 0x7fb5d8101200 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.944+0000 7fb5e0284700 1 -- 172.21.15.96:0/578960427 >> 172.21.15.96:0/578960427 conn(0x7fb5d80fc6d0 msgr2=0x7fb5d80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.944+0000 7fb5e0284700 1 -- 172.21.15.96:0/578960427 shutdown_connections 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.944+0000 7fb5e0284700 1 -- 172.21.15.96:0/578960427 wait complete. 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5e0284700 1 Processor -- start 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5e0284700 1 -- start start 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5e0284700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 0x7fb5d819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5e0284700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5d81017d0 0x7fb5d819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5e0284700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d819f610 con 0x7fb5d8100e20 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5e0284700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d8198b80 con 0x7fb5d81017d0 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5de020700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 0x7fb5d819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5de020700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 0x7fb5d819ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54804/0 (socket says 172.21.15.96:54804) 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5de020700 1 -- 172.21.15.96:0/913659363 learned_addr learned my addr 172.21.15.96:0/913659363 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:31.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.945+0000 7fb5dd81f700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5d81017d0 0x7fb5d819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:31.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.946+0000 7fb5de020700 1 -- 172.21.15.96:0/913659363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5d81017d0 msgr2=0x7fb5d819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:31.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.946+0000 7fb5de020700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5d81017d0 0x7fb5d819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:31.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.946+0000 7fb5de020700 1 -- 172.21.15.96:0/913659363 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d0009d00 con 0x7fb5d8100e20 2024-04-22T15:50:31.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.946+0000 7fb5de020700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 0x7fb5d819ea40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0000f80 tx=0x7fb5d0007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:31.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.946+0000 7fb5d77fe700 1 -- 172.21.15.96:0/913659363 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5d000e040 con 0x7fb5d8100e20 2024-04-22T15:50:31.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.946+0000 7fb5d77fe700 1 -- 172.21.15.96:0/913659363 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5d00096a0 con 0x7fb5d8100e20 2024-04-22T15:50:31.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.946+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d8198e00 con 0x7fb5d8100e20 2024-04-22T15:50:31.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.946+0000 7fb5d77fe700 1 -- 172.21.15.96:0/913659363 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5d001d6b0 con 0x7fb5d8100e20 2024-04-22T15:50:31.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.946+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d81992f0 con 0x7fb5d8100e20 2024-04-22T15:50:31.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.947+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d8104fa0 con 0x7fb5d8100e20 2024-04-22T15:50:31.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.947+0000 7fb5d77fe700 1 -- 172.21.15.96:0/913659363 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb5d0003c10 con 0x7fb5d8100e20 2024-04-22T15:50:31.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.948+0000 7fb5d77fe700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5c006ecc0 0x7fb5c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:31.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.948+0000 7fb5d77fe700 1 -- 172.21.15.96:0/913659363 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb5d0090510 con 0x7fb5d8100e20 2024-04-22T15:50:31.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.948+0000 7fb5dd81f700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5c006ecc0 0x7fb5c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:31.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.948+0000 7fb5dd81f700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5c006ecc0 0x7fb5c0071180 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80fdd00 tx=0x7fb5cc00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:31.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:31.953+0000 7fb5d77fe700 1 -- 172.21.15.96:0/913659363 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb5d005cda0 con 0x7fb5d8100e20 2024-04-22T15:50:32.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.169+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5d819a040 con 0x7fb5c006ecc0 2024-04-22T15:50:32.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.171+0000 7fb5d77fe700 1 -- 172.21.15.96:0/913659363 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb5d819a040 con 0x7fb5c006ecc0 2024-04-22T15:50:32.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5c006ecc0 msgr2=0x7fb5c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5c006ecc0 0x7fb5c0071180 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80fdd00 tx=0x7fb5cc00b410 comp rx=0 tx=0).stop 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 msgr2=0x7fb5d819ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 0x7fb5d819ea40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0000f80 tx=0x7fb5d0007b20 comp rx=0 tx=0).stop 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 shutdown_connections 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb5c006ecc0 0x7fb5c0071180 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5d8100e20 0x7fb5d819ea40 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 --2- 172.21.15.96:0/913659363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5d81017d0 0x7fb5d819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 >> 172.21.15.96:0/913659363 conn(0x7fb5d80fc6d0 msgr2=0x7fb5d80fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 shutdown_connections 2024-04-22T15:50:32.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.173+0000 7fb5e0284700 1 -- 172.21.15.96:0/913659363 wait complete. 2024-04-22T15:50:32.186 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:50:32.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.277+0000 7fcdf389d700 1 -- 172.21.15.96:0/2776464150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec101df0 msgr2=0x7fcdec1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.277+0000 7fcdf389d700 1 --2- 172.21.15.96:0/2776464150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec101df0 0x7fcdec1021d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fcde8009910 tx=0x7fcde8009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.277+0000 7fcdf389d700 1 -- 172.21.15.96:0/2776464150 shutdown_connections 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.277+0000 7fcdf389d700 1 --2- 172.21.15.96:0/2776464150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdec102710 0x7fcdec10ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.277+0000 7fcdf389d700 1 --2- 172.21.15.96:0/2776464150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec101df0 0x7fcdec1021d0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.277+0000 7fcdf389d700 1 -- 172.21.15.96:0/2776464150 >> 172.21.15.96:0/2776464150 conn(0x7fcdec0fb680 msgr2=0x7fcdec0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.278+0000 7fcdf389d700 1 -- 172.21.15.96:0/2776464150 shutdown_connections 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.278+0000 7fcdf389d700 1 -- 172.21.15.96:0/2776464150 wait complete. 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.278+0000 7fcdf389d700 1 Processor -- start 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.278+0000 7fcdf389d700 1 -- start start 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.278+0000 7fcdf389d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdec101df0 0x7fcdec19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:32.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.278+0000 7fcdf389d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec102710 0x7fcdec19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.278+0000 7fcdf389d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec19d3f0 con 0x7fcdec102710 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.278+0000 7fcdf389d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec1969d0 con 0x7fcdec101df0 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf0e38700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec102710 0x7fcdec19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf0e38700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec102710 0x7fcdec19cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54826/0 (socket says 172.21.15.96:54826) 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf0e38700 1 -- 172.21.15.96:0/2244624115 learned_addr learned my addr 172.21.15.96:0/2244624115 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf1639700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdec101df0 0x7fcdec19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf0e38700 1 -- 172.21.15.96:0/2244624115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdec101df0 msgr2=0x7fcdec19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf0e38700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdec101df0 0x7fcdec19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf0e38700 1 -- 172.21.15.96:0/2244624115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80095f0 con 0x7fcdec102710 2024-04-22T15:50:32.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf1639700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdec101df0 0x7fcdec19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:50:32.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf0e38700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec102710 0x7fcdec19cdd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00d6d0 tx=0x7fcddc00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:32.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcde27fc700 1 -- 172.21.15.96:0/2244624115 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcddc00f860 con 0x7fcdec102710 2024-04-22T15:50:32.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcde27fc700 1 -- 172.21.15.96:0/2244624115 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcddc00fe80 con 0x7fcdec102710 2024-04-22T15:50:32.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec196cb0 con 0x7fcdec102710 2024-04-22T15:50:32.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcde27fc700 1 -- 172.21.15.96:0/2244624115 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcddc010ea0 con 0x7fcdec102710 2024-04-22T15:50:32.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.279+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec197200 con 0x7fcdec102710 2024-04-22T15:50:32.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.280+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec105510 con 0x7fcdec102710 2024-04-22T15:50:32.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.281+0000 7fcde27fc700 1 -- 172.21.15.96:0/2244624115 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcddc010460 con 0x7fcdec102710 2024-04-22T15:50:32.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.281+0000 7fcde27fc700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcdd806ecc0 0x7fcdd8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:32.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.281+0000 7fcdf1639700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcdd806ecc0 0x7fcdd8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:32.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.281+0000 7fcde27fc700 1 -- 172.21.15.96:0/2244624115 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcddc08e9c0 con 0x7fcdec102710 2024-04-22T15:50:32.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.282+0000 7fcdf1639700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcdd806ecc0 0x7fcdd8071180 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcde80059d0 tx=0x7fcde801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:32.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.286+0000 7fcde27fc700 1 -- 172.21.15.96:0/2244624115 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcddc05b2d0 con 0x7fcdec102710 2024-04-22T15:50:32.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.508+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdec197e90 con 0x7fcdd806ecc0 2024-04-22T15:50:32.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.509+0000 7fcde27fc700 1 -- 172.21.15.96:0/2244624115 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcdec197e90 con 0x7fcdd806ecc0 2024-04-22T15:50:32.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.511+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcdd806ecc0 msgr2=0x7fcdd8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.511+0000 7fcdf389d700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcdd806ecc0 0x7fcdd8071180 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcde80059d0 tx=0x7fcde801a040 comp rx=0 tx=0).stop 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.511+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec102710 msgr2=0x7fcdec19cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.511+0000 7fcdf389d700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec102710 0x7fcdec19cdd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00d6d0 tx=0x7fcddc00da90 comp rx=0 tx=0).stop 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.512+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 shutdown_connections 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.512+0000 7fcdf389d700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcdd806ecc0 0x7fcdd8071180 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.512+0000 7fcdf389d700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdec101df0 0x7fcdec19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.512+0000 7fcdf389d700 1 --2- 172.21.15.96:0/2244624115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdec102710 0x7fcdec19cdd0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.512+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 >> 172.21.15.96:0/2244624115 conn(0x7fcdec0fb680 msgr2=0x7fcdec1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.512+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 shutdown_connections 2024-04-22T15:50:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.512+0000 7fcdf389d700 1 -- 172.21.15.96:0/2244624115 wait complete. 2024-04-22T15:50:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.615+0000 7f38214fd700 1 -- 172.21.15.96:0/4221151273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c100e20 msgr2=0x7f381c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.615+0000 7f38214fd700 1 --2- 172.21.15.96:0/4221151273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c100e20 0x7f381c101200 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3810009910 tx=0x7f3810009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.615+0000 7f38214fd700 1 -- 172.21.15.96:0/4221151273 shutdown_connections 2024-04-22T15:50:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.615+0000 7f38214fd700 1 --2- 172.21.15.96:0/4221151273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f381c1017d0 0x7f381c105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.615+0000 7f38214fd700 1 --2- 172.21.15.96:0/4221151273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c100e20 0x7f381c101200 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.615+0000 7f38214fd700 1 -- 172.21.15.96:0/4221151273 >> 172.21.15.96:0/4221151273 conn(0x7f381c0fc6d0 msgr2=0x7f381c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.615+0000 7f38214fd700 1 -- 172.21.15.96:0/4221151273 shutdown_connections 2024-04-22T15:50:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.615+0000 7f38214fd700 1 -- 172.21.15.96:0/4221151273 wait complete. 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.616+0000 7f38214fd700 1 Processor -- start 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.616+0000 7f38214fd700 1 -- start start 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.616+0000 7f38214fd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f381c100e20 0x7f381c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.616+0000 7f38214fd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c1017d0 0x7f381c19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.616+0000 7f38214fd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c19d3b0 con 0x7f381c1017d0 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.616+0000 7f38214fd700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c196990 con 0x7f381c100e20 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.616+0000 7f381a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c1017d0 0x7f381c19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.616+0000 7f381a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c1017d0 0x7f381c19cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54836/0 (socket says 172.21.15.96:54836) 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.616+0000 7f381a7fc700 1 -- 172.21.15.96:0/3440002077 learned_addr learned my addr 172.21.15.96:0/3440002077 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f381affd700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f381c100e20 0x7f381c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f381a7fc700 1 -- 172.21.15.96:0/3440002077 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f381c100e20 msgr2=0x7f381c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f381a7fc700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f381c100e20 0x7f381c19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f381a7fc700 1 -- 172.21.15.96:0/3440002077 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38100095f0 con 0x7f381c1017d0 2024-04-22T15:50:32.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f381a7fc700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c1017d0 0x7f381c19cd90 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f380c00d900 tx=0x7f380c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:32.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f3803fff700 1 -- 172.21.15.96:0/3440002077 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f380c009900 con 0x7f381c1017d0 2024-04-22T15:50:32.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f3803fff700 1 -- 172.21.15.96:0/3440002077 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f380c010460 con 0x7f381c1017d0 2024-04-22T15:50:32.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c196c70 con 0x7f381c1017d0 2024-04-22T15:50:32.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f3803fff700 1 -- 172.21.15.96:0/3440002077 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f380c00f5f0 con 0x7f381c1017d0 2024-04-22T15:50:32.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.617+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c1971c0 con 0x7f381c1017d0 2024-04-22T15:50:32.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.618+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c1091c0 con 0x7f381c1017d0 2024-04-22T15:50:32.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.618+0000 7f3803fff700 1 -- 172.21.15.96:0/3440002077 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f380c00fa80 con 0x7f381c1017d0 2024-04-22T15:50:32.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.619+0000 7f3803fff700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f380406ecc0 0x7f3804071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:32.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.619+0000 7f3803fff700 1 -- 172.21.15.96:0/3440002077 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f380c08ed20 con 0x7f381c1017d0 2024-04-22T15:50:32.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.619+0000 7f381affd700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f380406ecc0 0x7f3804071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:32.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.619+0000 7f381affd700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f380406ecc0 0x7f3804071180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f381000b5c0 tx=0x7f381001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:32.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.624+0000 7f3803fff700 1 -- 172.21.15.96:0/3440002077 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f380c05b5b0 con 0x7f381c1017d0 2024-04-22T15:50:32.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.837+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f381c197e50 con 0x7f380406ecc0 2024-04-22T15:50:32.840 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:32 smithi096 ceph-mon[112443]: pgmap v861: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.848+0000 7f3803fff700 1 -- 172.21.15.96:0/3440002077 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f381c197e50 con 0x7f380406ecc0 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (29m) 6m ago 40m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (40m) 6m ago 40m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (39m) 6m ago 39m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (26m) 6m ago 40m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (26m) 6m ago 39m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (28m) 6m ago 40m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (22m) 6m ago 34m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:50:32.851 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (22m) 6m ago 34m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (22m) 6m ago 34m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (6m) 6m ago 34m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (31m) 6m ago 42m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (30m) 6m ago 39m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (27m) 6m ago 42m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (27m) 6m ago 39m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (30m) 6m ago 40m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (30m) 6m ago 39m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (25m) 6m ago 38m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (25m) 6m ago 38m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (24m) 6m ago 37m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:50:32.852 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (24m) 6m ago 37m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:50:32.853 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (23m) 6m ago 36m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:50:32.853 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (23m) 6m ago 36m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:50:32.853 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (29m) 6m ago 40m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:50:32.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f380406ecc0 msgr2=0x7f3804071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f380406ecc0 0x7f3804071180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f381000b5c0 tx=0x7f381001a040 comp rx=0 tx=0).stop 2024-04-22T15:50:32.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c1017d0 msgr2=0x7f381c19cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c1017d0 0x7f381c19cd90 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f380c00d900 tx=0x7f380c00dcc0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 shutdown_connections 2024-04-22T15:50:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f380406ecc0 0x7f3804071180 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f381c100e20 0x7f381c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 --2- 172.21.15.96:0/3440002077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f381c1017d0 0x7f381c19cd90 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 >> 172.21.15.96:0/3440002077 conn(0x7f381c0fc6d0 msgr2=0x7f381c0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 shutdown_connections 2024-04-22T15:50:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.850+0000 7f38214fd700 1 -- 172.21.15.96:0/3440002077 wait complete. 2024-04-22T15:50:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.952+0000 7f9f7411f700 1 -- 172.21.15.96:0/1622690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 msgr2=0x7f9f6c104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.952+0000 7f9f7411f700 1 --2- 172.21.15.96:0/1622690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 0x7f9f6c104e90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009910 tx=0x7f9f64009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.952+0000 7f9f7411f700 1 -- 172.21.15.96:0/1622690 shutdown_connections 2024-04-22T15:50:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.952+0000 7f9f7411f700 1 --2- 172.21.15.96:0/1622690 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f6c100260 0x7f9f6c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.952+0000 7f9f7411f700 1 --2- 172.21.15.96:0/1622690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 0x7f9f6c104e90 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.952+0000 7f9f7411f700 1 -- 172.21.15.96:0/1622690 >> 172.21.15.96:0/1622690 conn(0x7f9f6c075f30 msgr2=0x7f9f6c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:32.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.952+0000 7f9f7411f700 1 -- 172.21.15.96:0/1622690 shutdown_connections 2024-04-22T15:50:32.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.952+0000 7f9f7411f700 1 -- 172.21.15.96:0/1622690 wait complete. 2024-04-22T15:50:32.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f7411f700 1 Processor -- start 2024-04-22T15:50:32.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f7411f700 1 -- start start 2024-04-22T15:50:32.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f7411f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f6c100260 0x7f9f6c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:32.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f7411f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 0x7f9f6c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:32.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f7411f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1984b0 con 0x7f9f6c104ab0 2024-04-22T15:50:32.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f7411f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1985f0 con 0x7f9f6c100260 2024-04-22T15:50:32.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f716ba700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 0x7f9f6c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:32.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f716ba700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 0x7f9f6c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54862/0 (socket says 172.21.15.96:54862) 2024-04-22T15:50:32.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f716ba700 1 -- 172.21.15.96:0/810467786 learned_addr learned my addr 172.21.15.96:0/810467786 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:32.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.953+0000 7f9f71ebb700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f6c100260 0x7f9f6c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:32.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.954+0000 7f9f716ba700 1 -- 172.21.15.96:0/810467786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f6c100260 msgr2=0x7f9f6c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:32.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.954+0000 7f9f716ba700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f6c100260 0x7f9f6c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:32.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.954+0000 7f9f716ba700 1 -- 172.21.15.96:0/810467786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f640095f0 con 0x7f9f6c104ab0 2024-04-22T15:50:32.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.954+0000 7f9f716ba700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 0x7f9f6c197f70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9f600060b0 tx=0x7f9f6000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:32.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.954+0000 7f9f6affd700 1 -- 172.21.15.96:0/810467786 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f6000dda0 con 0x7f9f6c104ab0 2024-04-22T15:50:32.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.954+0000 7f9f6affd700 1 -- 172.21.15.96:0/810467786 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f6000f040 con 0x7f9f6c104ab0 2024-04-22T15:50:32.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.954+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c1a0aa0 con 0x7f9f6c104ab0 2024-04-22T15:50:32.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.954+0000 7f9f6affd700 1 -- 172.21.15.96:0/810467786 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f600146f0 con 0x7f9f6c104ab0 2024-04-22T15:50:32.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.954+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c1a0eb0 con 0x7f9f6c104ab0 2024-04-22T15:50:32.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.955+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c108060 con 0x7f9f6c104ab0 2024-04-22T15:50:32.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.956+0000 7f9f6affd700 1 -- 172.21.15.96:0/810467786 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9f60015400 con 0x7f9f6c104ab0 2024-04-22T15:50:32.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.956+0000 7f9f6affd700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f5406ecc0 0x7f9f54071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:32.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.956+0000 7f9f71ebb700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f5406ecc0 0x7f9f54071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:32.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.956+0000 7f9f6affd700 1 -- 172.21.15.96:0/810467786 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9f6008e7e0 con 0x7f9f6c104ab0 2024-04-22T15:50:32.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.956+0000 7f9f71ebb700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f5406ecc0 0x7f9f54071180 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9f64000c00 tx=0x7f9f6401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:32.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:32.961+0000 7f9f6affd700 1 -- 172.21.15.96:0/810467786 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9f60057670 con 0x7f9f6c104ab0 2024-04-22T15:50:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:32 smithi196 ceph-mon[81815]: pgmap v861: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:33.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.245+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9f6c0ff6a0 con 0x7f9f6c104ab0 2024-04-22T15:50:33.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.245+0000 7f9f6affd700 1 -- 172.21.15.96:0/810467786 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9f6005ad30 con 0x7f9f6c104ab0 2024-04-22T15:50:33.248 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:50:33.248 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:50:33.248 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:50:33.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:50:33.249 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f5406ecc0 msgr2=0x7f9f54071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f5406ecc0 0x7f9f54071180 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9f64000c00 tx=0x7f9f6401a040 comp rx=0 tx=0).stop 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 msgr2=0x7f9f6c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 0x7f9f6c197f70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9f600060b0 tx=0x7f9f6000d3b0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 shutdown_connections 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9f5406ecc0 0x7f9f54071180 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f6c100260 0x7f9f6c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 --2- 172.21.15.96:0/810467786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f6c104ab0 0x7f9f6c197f70 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 >> 172.21.15.96:0/810467786 conn(0x7f9f6c075f30 msgr2=0x7f9f6c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:33.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 shutdown_connections 2024-04-22T15:50:33.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.248+0000 7f9f7411f700 1 -- 172.21.15.96:0/810467786 wait complete. 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.350+0000 7f8de2de4700 1 -- 172.21.15.96:0/1421124862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc100e20 msgr2=0x7f8ddc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.350+0000 7f8de2de4700 1 --2- 172.21.15.96:0/1421124862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc100e20 0x7f8ddc101200 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009910 tx=0x7f8dd0009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.350+0000 7f8de2de4700 1 -- 172.21.15.96:0/1421124862 shutdown_connections 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.350+0000 7f8de2de4700 1 --2- 172.21.15.96:0/1421124862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ddc1017d0 0x7f8ddc105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.350+0000 7f8de2de4700 1 --2- 172.21.15.96:0/1421124862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc100e20 0x7f8ddc101200 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.350+0000 7f8de2de4700 1 -- 172.21.15.96:0/1421124862 >> 172.21.15.96:0/1421124862 conn(0x7f8ddc0fc6d0 msgr2=0x7f8ddc0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.350+0000 7f8de2de4700 1 -- 172.21.15.96:0/1421124862 shutdown_connections 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8de2de4700 1 -- 172.21.15.96:0/1421124862 wait complete. 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8de2de4700 1 Processor -- start 2024-04-22T15:50:33.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8de2de4700 1 -- start start 2024-04-22T15:50:33.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8de2de4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ddc100e20 0x7f8ddc19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:33.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8de2de4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc1017d0 0x7f8ddc19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:33.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8de2de4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc19d4a0 con 0x7f8ddc1017d0 2024-04-22T15:50:33.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8de2de4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc196a80 con 0x7f8ddc100e20 2024-04-22T15:50:33.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8ddbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc1017d0 0x7f8ddc19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:33.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8ddbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc1017d0 0x7f8ddc19ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54888/0 (socket says 172.21.15.96:54888) 2024-04-22T15:50:33.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8ddbfff700 1 -- 172.21.15.96:0/2070316335 learned_addr learned my addr 172.21.15.96:0/2070316335 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.351+0000 7f8de0b80700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ddc100e20 0x7f8ddc19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.352+0000 7f8ddbfff700 1 -- 172.21.15.96:0/2070316335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ddc100e20 msgr2=0x7f8ddc19c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.352+0000 7f8ddbfff700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ddc100e20 0x7f8ddc19c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.352+0000 7f8ddbfff700 1 -- 172.21.15.96:0/2070316335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00095f0 con 0x7f8ddc1017d0 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.352+0000 7f8de0b80700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ddc100e20 0x7f8ddc19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.352+0000 7f8ddbfff700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc1017d0 0x7f8ddc19ce80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800b480 tx=0x7f8dc800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.352+0000 7f8dd9ffb700 1 -- 172.21.15.96:0/2070316335 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dc80118c0 con 0x7f8ddc1017d0 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.352+0000 7f8dd9ffb700 1 -- 172.21.15.96:0/2070316335 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8dc8010380 con 0x7f8ddc1017d0 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.352+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc196d60 con 0x7f8ddc1017d0 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.353+0000 7f8dd9ffb700 1 -- 172.21.15.96:0/2070316335 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dc800f4f0 con 0x7f8ddc1017d0 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.353+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc1972b0 con 0x7f8ddc1017d0 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.353+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc1091c0 con 0x7f8ddc1017d0 2024-04-22T15:50:33.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.354+0000 7f8dd9ffb700 1 -- 172.21.15.96:0/2070316335 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8dc80104f0 con 0x7f8ddc1017d0 2024-04-22T15:50:33.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.354+0000 7f8dd9ffb700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8dcc06ecc0 0x7f8dcc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:33.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.354+0000 7f8dd9ffb700 1 -- 172.21.15.96:0/2070316335 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8dc808ea20 con 0x7f8ddc1017d0 2024-04-22T15:50:33.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.354+0000 7f8de0b80700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8dcc06ecc0 0x7f8dcc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:33.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.354+0000 7f8de0b80700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8dcc06ecc0 0x7f8dcc071180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8dd00059d0 tx=0x7f8dd001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:33.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.359+0000 7f8dd9ffb700 1 -- 172.21.15.96:0/2070316335 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8dc805b330 con 0x7f8ddc1017d0 2024-04-22T15:50:33.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.601+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8ddc102300 con 0x7f8ddc1017d0 2024-04-22T15:50:33.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.601+0000 7f8dd9ffb700 1 -- 172.21.15.96:0/2070316335 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f8dc805aec0 con 0x7f8ddc1017d0 2024-04-22T15:50:33.604 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:50:33.604 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:50:33.604 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:50:33.604 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:50:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:50:33.604 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:50:33.605 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:50:33.606 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:50:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:50:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:50:33.607 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:50:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:50:33.607 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:50:33.607 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:50:33.607 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:50:33.607 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:50:33.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8dcc06ecc0 msgr2=0x7f8dcc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8dcc06ecc0 0x7f8dcc071180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8dd00059d0 tx=0x7f8dd001a040 comp rx=0 tx=0).stop 2024-04-22T15:50:33.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc1017d0 msgr2=0x7f8ddc19ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc1017d0 0x7f8ddc19ce80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800b480 tx=0x7f8dc800b840 comp rx=0 tx=0).stop 2024-04-22T15:50:33.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 shutdown_connections 2024-04-22T15:50:33.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8dcc06ecc0 0x7f8dcc071180 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ddc100e20 0x7f8ddc19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 --2- 172.21.15.96:0/2070316335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ddc1017d0 0x7f8ddc19ce80 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 >> 172.21.15.96:0/2070316335 conn(0x7f8ddc0fc6d0 msgr2=0x7f8ddc0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:33.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 shutdown_connections 2024-04-22T15:50:33.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.604+0000 7f8de2de4700 1 -- 172.21.15.96:0/2070316335 wait complete. 2024-04-22T15:50:33.609 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:50:33.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.714+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3296310493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ff370 msgr2=0x7f37e80ff750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.714+0000 7f37ed8cf700 1 --2- 172.21.15.96:0/3296310493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ff370 0x7f37e80ff750 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009910 tx=0x7f37dc009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:33.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.714+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3296310493 shutdown_connections 2024-04-22T15:50:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.714+0000 7f37ed8cf700 1 --2- 172.21.15.96:0/3296310493 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37e80ffd20 0x7f37e8105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.714+0000 7f37ed8cf700 1 --2- 172.21.15.96:0/3296310493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ff370 0x7f37e80ff750 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.714+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3296310493 >> 172.21.15.96:0/3296310493 conn(0x7f37e8075f30 msgr2=0x7f37e80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.714+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3296310493 shutdown_connections 2024-04-22T15:50:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.714+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3296310493 wait complete. 2024-04-22T15:50:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.715+0000 7f37ed8cf700 1 Processor -- start 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.715+0000 7f37ed8cf700 1 -- start start 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.715+0000 7f37ed8cf700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37e80ff370 0x7f37e819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.715+0000 7f37ed8cf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ffd20 0x7f37e819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.715+0000 7f37ed8cf700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e819d4c0 con 0x7f37e80ffd20 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.715+0000 7f37ed8cf700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e8196aa0 con 0x7f37e80ff370 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.715+0000 7f37e67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ffd20 0x7f37e819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.715+0000 7f37e67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ffd20 0x7f37e819cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54918/0 (socket says 172.21.15.96:54918) 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.715+0000 7f37e67fc700 1 -- 172.21.15.96:0/3927525470 learned_addr learned my addr 172.21.15.96:0/3927525470 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37e6ffd700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37e80ff370 0x7f37e819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37e67fc700 1 -- 172.21.15.96:0/3927525470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37e80ff370 msgr2=0x7f37e819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37e67fc700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37e80ff370 0x7f37e819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37e67fc700 1 -- 172.21.15.96:0/3927525470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc0095f0 con 0x7f37e80ffd20 2024-04-22T15:50:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37e6ffd700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37e80ff370 0x7f37e819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:50:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37e67fc700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ffd20 0x7f37e819cea0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f37d800b4d0 tx=0x7f37d800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37ec8cd700 1 -- 172.21.15.96:0/3927525470 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37d8011800 con 0x7f37e80ffd20 2024-04-22T15:50:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37ec8cd700 1 -- 172.21.15.96:0/3927525470 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37d8011e20 con 0x7f37e80ffd20 2024-04-22T15:50:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e8196d80 con 0x7f37e80ffd20 2024-04-22T15:50:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37ec8cd700 1 -- 172.21.15.96:0/3927525470 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37d800f2e0 con 0x7f37e80ffd20 2024-04-22T15:50:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.716+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e81972d0 con 0x7f37e80ffd20 2024-04-22T15:50:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.717+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e81091f0 con 0x7f37e80ffd20 2024-04-22T15:50:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.717+0000 7f37ec8cd700 1 -- 172.21.15.96:0/3927525470 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f37d8010380 con 0x7f37e80ffd20 2024-04-22T15:50:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.718+0000 7f37ec8cd700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f37d006ecc0 0x7f37d0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.718+0000 7f37e6ffd700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f37d006ecc0 0x7f37d0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.718+0000 7f37ec8cd700 1 -- 172.21.15.96:0/3927525470 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f37d808e9f0 con 0x7f37e80ffd20 2024-04-22T15:50:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.718+0000 7f37e6ffd700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f37d006ecc0 0x7f37d0071180 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000c00 tx=0x7f37dc00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.723+0000 7f37ec8cd700 1 -- 172.21.15.96:0/3927525470 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f37d8090e60 con 0x7f37e80ffd20 2024-04-22T15:50:33.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.941+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f37e8197fd0 con 0x7f37d006ecc0 2024-04-22T15:50:33.944 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:33 smithi096 ceph-mon[112443]: from='client.35572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:33.945 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:33 smithi096 ceph-mon[112443]: from='client.35576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:33.945 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:33 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/810467786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:33.945 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:33 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2070316335' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:50:33.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.943+0000 7f37ec8cd700 1 -- 172.21.15.96:0/3927525470 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f37e8197fd0 con 0x7f37d006ecc0 2024-04-22T15:50:33.945 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:50:33.946 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:50:33.947 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:50:33.947 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f37d006ecc0 msgr2=0x7f37d0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f37d006ecc0 0x7f37d0071180 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000c00 tx=0x7f37dc00b540 comp rx=0 tx=0).stop 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ffd20 msgr2=0x7f37e819cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ffd20 0x7f37e819cea0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f37d800b4d0 tx=0x7f37d800b890 comp rx=0 tx=0).stop 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 shutdown_connections 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f37d006ecc0 0x7f37d0071180 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37e80ff370 0x7f37e819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 --2- 172.21.15.96:0/3927525470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e80ffd20 0x7f37e819cea0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 >> 172.21.15.96:0/3927525470 conn(0x7f37e8075f30 msgr2=0x7f37e8104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 shutdown_connections 2024-04-22T15:50:33.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:33.945+0000 7f37ed8cf700 1 -- 172.21.15.96:0/3927525470 wait complete. 2024-04-22T15:50:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:33 smithi196 ceph-mon[81815]: from='client.35572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:33 smithi196 ceph-mon[81815]: from='client.35576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:33 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/810467786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:34.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:33 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2070316335' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:50:34.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.048+0000 7f96357d3700 1 -- 172.21.15.96:0/2275369133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 msgr2=0x7f9630105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.048+0000 7f96357d3700 1 --2- 172.21.15.96:0/2275369133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 0x7f9630105820 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9624009910 tx=0x7f9624009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:34.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.048+0000 7f96357d3700 1 -- 172.21.15.96:0/2275369133 shutdown_connections 2024-04-22T15:50:34.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.048+0000 7f96357d3700 1 --2- 172.21.15.96:0/2275369133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 0x7f9630105820 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.048+0000 7f96357d3700 1 --2- 172.21.15.96:0/2275369133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9630100e20 0x7f9630101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.048+0000 7f96357d3700 1 -- 172.21.15.96:0/2275369133 >> 172.21.15.96:0/2275369133 conn(0x7f96300fc6d0 msgr2=0x7f96300feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:34.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.048+0000 7f96357d3700 1 -- 172.21.15.96:0/2275369133 shutdown_connections 2024-04-22T15:50:34.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.048+0000 7f96357d3700 1 -- 172.21.15.96:0/2275369133 wait complete. 2024-04-22T15:50:34.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.049+0000 7f96357d3700 1 Processor -- start 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.049+0000 7f96357d3700 1 -- start start 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.049+0000 7f96357d3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9630100e20 0x7f963019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.049+0000 7f96357d3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 0x7f963019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.049+0000 7f96357d3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963019d380 con 0x7f96301017d0 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.049+0000 7f96357d3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9630196960 con 0x7f9630100e20 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.049+0000 7f962e7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 0x7f963019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.049+0000 7f962e7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 0x7f963019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54940/0 (socket says 172.21.15.96:54940) 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.049+0000 7f962e7fc700 1 -- 172.21.15.96:0/686049038 learned_addr learned my addr 172.21.15.96:0/686049038 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f962effd700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9630100e20 0x7f963019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f962e7fc700 1 -- 172.21.15.96:0/686049038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9630100e20 msgr2=0x7f963019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f962e7fc700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9630100e20 0x7f963019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f962e7fc700 1 -- 172.21.15.96:0/686049038 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96240095f0 con 0x7f96301017d0 2024-04-22T15:50:34.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f962e7fc700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 0x7f963019cd60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f962400bd70 tx=0x7f962400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:34.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f9617fff700 1 -- 172.21.15.96:0/686049038 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f962401d070 con 0x7f96301017d0 2024-04-22T15:50:34.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9630196be0 con 0x7f96301017d0 2024-04-22T15:50:34.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f9617fff700 1 -- 172.21.15.96:0/686049038 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96240056e0 con 0x7f96301017d0 2024-04-22T15:50:34.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f9617fff700 1 -- 172.21.15.96:0/686049038 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f962400fe50 con 0x7f96301017d0 2024-04-22T15:50:34.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.050+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96301970d0 con 0x7f96301017d0 2024-04-22T15:50:34.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.051+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9630109220 con 0x7f96301017d0 2024-04-22T15:50:34.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.052+0000 7f9617fff700 1 -- 172.21.15.96:0/686049038 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f962400f680 con 0x7f96301017d0 2024-04-22T15:50:34.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.052+0000 7f9617fff700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f961806ecc0 0x7f9618071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:34.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.052+0000 7f9617fff700 1 -- 172.21.15.96:0/686049038 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f96240901c0 con 0x7f96301017d0 2024-04-22T15:50:34.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.052+0000 7f962effd700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f961806ecc0 0x7f9618071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:34.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.052+0000 7f962effd700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f961806ecc0 0x7f9618071180 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9620007be0 tx=0x7f96200073d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:34.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.057+0000 7f9617fff700 1 -- 172.21.15.96:0/686049038 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f962405ca50 con 0x7f96301017d0 2024-04-22T15:50:34.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.339+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9630102300 con 0x7f96301017d0 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.340+0000 7f9617fff700 1 -- 172.21.15.96:0/686049038 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f962405c5e0 con 0x7f96301017d0 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:50:34.343 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:50:34.344 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:50:34.345 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:50:34.345 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:50:34.345 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:50:34.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f961806ecc0 msgr2=0x7f9618071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f961806ecc0 0x7f9618071180 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9620007be0 tx=0x7f96200073d0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 msgr2=0x7f963019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 0x7f963019cd60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f962400bd70 tx=0x7f962400bda0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 shutdown_connections 2024-04-22T15:50:34.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f961806ecc0 0x7f9618071180 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9630100e20 0x7f963019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 --2- 172.21.15.96:0/686049038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96301017d0 0x7f963019cd60 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 >> 172.21.15.96:0/686049038 conn(0x7f96300fc6d0 msgr2=0x7f96300fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:34.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 shutdown_connections 2024-04-22T15:50:34.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:50:34.342+0000 7f96357d3700 1 -- 172.21.15.96:0/686049038 wait complete. 2024-04-22T15:50:34.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:34 smithi096 ceph-mon[112443]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:34 smithi096 ceph-mon[112443]: pgmap v862: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:34 smithi096 ceph-mon[112443]: from='client.35592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:34 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/686049038' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:50:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:34 smithi196 ceph-mon[81815]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:34 smithi196 ceph-mon[81815]: pgmap v862: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:34 smithi196 ceph-mon[81815]: from='client.35592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:34 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/686049038' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:50:35.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:35 smithi096 ceph-mon[112443]: pgmap v863: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:35 smithi196 ceph-mon[81815]: pgmap v863: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:38 smithi096 ceph-mon[112443]: pgmap v864: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:38 smithi196 ceph-mon[81815]: pgmap v864: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:40 smithi096 ceph-mon[112443]: pgmap v865: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:40 smithi196 ceph-mon[81815]: pgmap v865: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:42 smithi096 ceph-mon[112443]: pgmap v866: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:42 smithi196 ceph-mon[81815]: pgmap v866: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:44.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:44 smithi096 ceph-mon[112443]: pgmap v867: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:44 smithi196 ceph-mon[81815]: pgmap v867: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:46 smithi096 ceph-mon[112443]: pgmap v868: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:46 smithi196 ceph-mon[81815]: pgmap v868: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:48.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:48 smithi096 ceph-mon[112443]: pgmap v869: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:48 smithi196 ceph-mon[81815]: pgmap v869: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:50.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:50 smithi196 ceph-mon[81815]: pgmap v870: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:50.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:50 smithi096 ceph-mon[112443]: pgmap v870: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:52.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:52 smithi096 ceph-mon[112443]: pgmap v871: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:52 smithi196 ceph-mon[81815]: pgmap v871: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:54.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:54 smithi096 ceph-mon[112443]: pgmap v872: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:54 smithi196 ceph-mon[81815]: pgmap v872: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:55.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:55 smithi196 ceph-mon[81815]: pgmap v873: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:55.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:55 smithi096 ceph-mon[112443]: pgmap v873: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:58.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:50:58 smithi096 ceph-mon[112443]: pgmap v874: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:50:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:50:58 smithi196 ceph-mon[81815]: pgmap v874: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:00.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:00 smithi096 ceph-mon[112443]: pgmap v875: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:00 smithi196 ceph-mon[81815]: pgmap v875: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:02.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:02 smithi096 ceph-mon[112443]: pgmap v876: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:02 smithi196 ceph-mon[81815]: pgmap v876: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.459+0000 7f31a01b2700 1 -- 172.21.15.96:0/2577969632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 msgr2=0x7f319810ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.459+0000 7f31a01b2700 1 --2- 172.21.15.96:0/2577969632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 0x7f319810ac00 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3188009940 tx=0x7f3188009c50 comp rx=0 tx=0).stop 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.459+0000 7f31a01b2700 1 -- 172.21.15.96:0/2577969632 shutdown_connections 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.459+0000 7f31a01b2700 1 --2- 172.21.15.96:0/2577969632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 0x7f319810ac00 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.459+0000 7f31a01b2700 1 --2- 172.21.15.96:0/2577969632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3198101df0 0x7f31981021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.459+0000 7f31a01b2700 1 -- 172.21.15.96:0/2577969632 >> 172.21.15.96:0/2577969632 conn(0x7f31980fb600 msgr2=0x7f31980fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.459+0000 7f31a01b2700 1 -- 172.21.15.96:0/2577969632 shutdown_connections 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.459+0000 7f31a01b2700 1 -- 172.21.15.96:0/2577969632 wait complete. 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.459+0000 7f31a01b2700 1 Processor -- start 2024-04-22T15:51:04.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f31a01b2700 1 -- start start 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f31a01b2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3198101df0 0x7f31980ffa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f31a01b2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 0x7f31980fff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f31a01b2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31981004d0 con 0x7f3198102710 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f31a01b2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3198100610 con 0x7f3198101df0 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f319d74d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 0x7f31980fff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f319d74d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 0x7f31980fff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40950/0 (socket says 172.21.15.96:40950) 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f319d74d700 1 -- 172.21.15.96:0/4000202209 learned_addr learned my addr 172.21.15.96:0/4000202209 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f319df4e700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3198101df0 0x7f31980ffa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f319d74d700 1 -- 172.21.15.96:0/4000202209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3198101df0 msgr2=0x7f31980ffa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f319d74d700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3198101df0 0x7f31980ffa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f319d74d700 1 -- 172.21.15.96:0/4000202209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880095f0 con 0x7f3198102710 2024-04-22T15:51:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f319df4e700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3198101df0 0x7f31980ffa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:51:04.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.460+0000 7f319d74d700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 0x7f31980fff90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f31880055f0 tx=0x7f31880056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:04.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.461+0000 7f318effd700 1 -- 172.21.15.96:0/4000202209 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f318801d070 con 0x7f3198102710 2024-04-22T15:51:04.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.461+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3198100890 con 0x7f3198102710 2024-04-22T15:51:04.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.461+0000 7f318effd700 1 -- 172.21.15.96:0/4000202209 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f318800bed0 con 0x7f3198102710 2024-04-22T15:51:04.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.461+0000 7f318effd700 1 -- 172.21.15.96:0/4000202209 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3188021410 con 0x7f3198102710 2024-04-22T15:51:04.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.461+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31981a3060 con 0x7f3198102710 2024-04-22T15:51:04.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.462+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31980699d0 con 0x7f3198102710 2024-04-22T15:51:04.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.462+0000 7f318effd700 1 -- 172.21.15.96:0/4000202209 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f318800f7f0 con 0x7f3198102710 2024-04-22T15:51:04.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.463+0000 7f318effd700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f318406ecc0 0x7f3184071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:04.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.463+0000 7f318effd700 1 -- 172.21.15.96:0/4000202209 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3188090090 con 0x7f3198102710 2024-04-22T15:51:04.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.463+0000 7f319df4e700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f318406ecc0 0x7f3184071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:04.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.463+0000 7f319df4e700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f318406ecc0 0x7f3184071180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3194007900 tx=0x7f3194008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:04.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.468+0000 7f318effd700 1 -- 172.21.15.96:0/4000202209 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3188058df0 con 0x7f3198102710 2024-04-22T15:51:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.689+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f319810de70 con 0x7f318406ecc0 2024-04-22T15:51:04.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.691+0000 7f318effd700 1 -- 172.21.15.96:0/4000202209 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f319810de70 con 0x7f318406ecc0 2024-04-22T15:51:04.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f318406ecc0 msgr2=0x7f3184071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f318406ecc0 0x7f3184071180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3194007900 tx=0x7f3194008040 comp rx=0 tx=0).stop 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 msgr2=0x7f31980fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 0x7f31980fff90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f31880055f0 tx=0x7f31880056d0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 shutdown_connections 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f318406ecc0 0x7f3184071180 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3198101df0 0x7f31980ffa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 --2- 172.21.15.96:0/4000202209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3198102710 0x7f31980fff90 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 >> 172.21.15.96:0/4000202209 conn(0x7f31980fb600 msgr2=0x7f31981030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 shutdown_connections 2024-04-22T15:51:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.693+0000 7f31a01b2700 1 -- 172.21.15.96:0/4000202209 wait complete. 2024-04-22T15:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:51:04.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:04 smithi096 ceph-mon[112443]: pgmap v877: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:04.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.798+0000 7f82bcc5a700 1 -- 172.21.15.96:0/365766978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 msgr2=0x7f82b8105830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:04.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.798+0000 7f82bcc5a700 1 --2- 172.21.15.96:0/365766978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 0x7f82b8105830 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009910 tx=0x7f82a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.798+0000 7f82bcc5a700 1 -- 172.21.15.96:0/365766978 shutdown_connections 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.798+0000 7f82bcc5a700 1 --2- 172.21.15.96:0/365766978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 0x7f82b8105830 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.798+0000 7f82bcc5a700 1 --2- 172.21.15.96:0/365766978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82b8100e30 0x7f82b8101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.798+0000 7f82bcc5a700 1 -- 172.21.15.96:0/365766978 >> 172.21.15.96:0/365766978 conn(0x7f82b80fc6e0 msgr2=0x7f82b80feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.798+0000 7f82bcc5a700 1 -- 172.21.15.96:0/365766978 shutdown_connections 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.798+0000 7f82bcc5a700 1 -- 172.21.15.96:0/365766978 wait complete. 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82bcc5a700 1 Processor -- start 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82bcc5a700 1 -- start start 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82bcc5a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82b8100e30 0x7f82b819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82bcc5a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 0x7f82b819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82bcc5a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b819d350 con 0x7f82b81017e0 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82bcc5a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b8196930 con 0x7f82b8100e30 2024-04-22T15:51:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82b5d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 0x7f82b819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82b5d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 0x7f82b819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40968/0 (socket says 172.21.15.96:40968) 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82b5d9b700 1 -- 172.21.15.96:0/4180971040 learned_addr learned my addr 172.21.15.96:0/4180971040 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.799+0000 7f82b659c700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82b8100e30 0x7f82b819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.800+0000 7f82b5d9b700 1 -- 172.21.15.96:0/4180971040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82b8100e30 msgr2=0x7f82b819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.800+0000 7f82b5d9b700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82b8100e30 0x7f82b819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.800+0000 7f82b5d9b700 1 -- 172.21.15.96:0/4180971040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a80095f0 con 0x7f82b81017e0 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.800+0000 7f82b5d9b700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 0x7f82b819cd30 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f82a800bd70 tx=0x7f82a800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.800+0000 7f82a77fe700 1 -- 172.21.15.96:0/4180971040 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82a801d070 con 0x7f82b81017e0 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.800+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b8196bb0 con 0x7f82b81017e0 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.800+0000 7f82a77fe700 1 -- 172.21.15.96:0/4180971040 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82a80056e0 con 0x7f82b81017e0 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.800+0000 7f82a77fe700 1 -- 172.21.15.96:0/4180971040 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82a800fe50 con 0x7f82b81017e0 2024-04-22T15:51:04.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.800+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b81970a0 con 0x7f82b81017e0 2024-04-22T15:51:04.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.801+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b8109230 con 0x7f82b81017e0 2024-04-22T15:51:04.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.802+0000 7f82a77fe700 1 -- 172.21.15.96:0/4180971040 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f82a800f680 con 0x7f82b81017e0 2024-04-22T15:51:04.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.802+0000 7f82a77fe700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82a006ecc0 0x7f82a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:04.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.802+0000 7f82b659c700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82a006ecc0 0x7f82a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:04.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.802+0000 7f82a77fe700 1 -- 172.21.15.96:0/4180971040 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f82a80901c0 con 0x7f82b81017e0 2024-04-22T15:51:04.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.802+0000 7f82b659c700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82a006ecc0 0x7f82a0071180 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f82ac005950 tx=0x7f82ac00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:04.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:04.807+0000 7f82a77fe700 1 -- 172.21.15.96:0/4180971040 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f82a805ca50 con 0x7f82b81017e0 2024-04-22T15:51:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:04 smithi196 ceph-mon[81815]: pgmap v877: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:05.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.023+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f82b8197df0 con 0x7f82a006ecc0 2024-04-22T15:51:05.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.024+0000 7f82a77fe700 1 -- 172.21.15.96:0/4180971040 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f82b8197df0 con 0x7f82a006ecc0 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82a006ecc0 msgr2=0x7f82a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82a006ecc0 0x7f82a0071180 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f82ac005950 tx=0x7f82ac00b410 comp rx=0 tx=0).stop 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 msgr2=0x7f82b819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 0x7f82b819cd30 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f82a800bd70 tx=0x7f82a800bda0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 shutdown_connections 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82a006ecc0 0x7f82a0071180 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82b8100e30 0x7f82b819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 --2- 172.21.15.96:0/4180971040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b81017e0 0x7f82b819cd30 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 >> 172.21.15.96:0/4180971040 conn(0x7f82b80fc6e0 msgr2=0x7f82b80fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 shutdown_connections 2024-04-22T15:51:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.026+0000 7f82bcc5a700 1 -- 172.21.15.96:0/4180971040 wait complete. 2024-04-22T15:51:05.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.129+0000 7ff65890e700 1 -- 172.21.15.96:0/3336025783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6540ffad0 msgr2=0x7ff6540fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.129+0000 7ff65890e700 1 --2- 172.21.15.96:0/3336025783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6540ffad0 0x7ff6540fff50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff644009910 tx=0x7ff644009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.129+0000 7ff65890e700 1 -- 172.21.15.96:0/3336025783 shutdown_connections 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.129+0000 7ff65890e700 1 --2- 172.21.15.96:0/3336025783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6540ffad0 0x7ff6540fff50 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.129+0000 7ff65890e700 1 --2- 172.21.15.96:0/3336025783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff654105b60 0x7ff654105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.129+0000 7ff65890e700 1 -- 172.21.15.96:0/3336025783 >> 172.21.15.96:0/3336025783 conn(0x7ff6540fb600 msgr2=0x7ff6540fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.129+0000 7ff65890e700 1 -- 172.21.15.96:0/3336025783 shutdown_connections 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.129+0000 7ff65890e700 1 -- 172.21.15.96:0/3336025783 wait complete. 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff65890e700 1 Processor -- start 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff65890e700 1 -- start start 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff65890e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6540ffad0 0x7ff654197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff65890e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff654105b60 0x7ff654197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff65890e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff654198460 con 0x7ff654105b60 2024-04-22T15:51:05.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff65890e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6541985a0 con 0x7ff6540ffad0 2024-04-22T15:51:05.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff651d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff654105b60 0x7ff654197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:05.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff651d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff654105b60 0x7ff654197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40974/0 (socket says 172.21.15.96:40974) 2024-04-22T15:51:05.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff651d9b700 1 -- 172.21.15.96:0/1223658576 learned_addr learned my addr 172.21.15.96:0/1223658576 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:05.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.130+0000 7ff65259c700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6540ffad0 0x7ff654197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:05.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.131+0000 7ff651d9b700 1 -- 172.21.15.96:0/1223658576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6540ffad0 msgr2=0x7ff654197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.131+0000 7ff651d9b700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6540ffad0 0x7ff654197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.131+0000 7ff651d9b700 1 -- 172.21.15.96:0/1223658576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6440095f0 con 0x7ff654105b60 2024-04-22T15:51:05.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.131+0000 7ff651d9b700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff654105b60 0x7ff654197e90 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff64400bd70 tx=0x7ff64400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:05.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.131+0000 7ff6437fe700 1 -- 172.21.15.96:0/1223658576 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff64401d070 con 0x7ff654105b60 2024-04-22T15:51:05.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.131+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6541a09c0 con 0x7ff654105b60 2024-04-22T15:51:05.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.131+0000 7ff6437fe700 1 -- 172.21.15.96:0/1223658576 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6440056e0 con 0x7ff654105b60 2024-04-22T15:51:05.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.131+0000 7ff6437fe700 1 -- 172.21.15.96:0/1223658576 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff64400fe40 con 0x7ff654105b60 2024-04-22T15:51:05.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.131+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6541a0d20 con 0x7ff654105b60 2024-04-22T15:51:05.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.132+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6541080f0 con 0x7ff654105b60 2024-04-22T15:51:05.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.133+0000 7ff6437fe700 1 -- 172.21.15.96:0/1223658576 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff64400f680 con 0x7ff654105b60 2024-04-22T15:51:05.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.133+0000 7ff6437fe700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff63c06ec60 0x7ff63c071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:05.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.133+0000 7ff65259c700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff63c06ec60 0x7ff63c071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:05.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.133+0000 7ff6437fe700 1 -- 172.21.15.96:0/1223658576 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff644090340 con 0x7ff654105b60 2024-04-22T15:51:05.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.134+0000 7ff65259c700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff63c06ec60 0x7ff63c071120 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff648005950 tx=0x7ff64800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:05.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.138+0000 7ff6437fe700 1 -- 172.21.15.96:0/1223658576 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff644059040 con 0x7ff654105b60 2024-04-22T15:51:05.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.346+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff65410ccd0 con 0x7ff63c06ec60 2024-04-22T15:51:05.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.357+0000 7ff6437fe700 1 -- 172.21.15.96:0/1223658576 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff65410ccd0 con 0x7ff63c06ec60 2024-04-22T15:51:05.360 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:51:05.360 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (29m) 7m ago 41m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:51:05.360 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (41m) 7m ago 41m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:51:05.360 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (40m) 7m ago 40m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:51:05.360 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (26m) 7m ago 41m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:51:05.360 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (26m) 7m ago 40m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:51:05.360 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (29m) 7m ago 40m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (22m) 7m ago 35m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (23m) 7m ago 35m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (22m) 7m ago 35m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (7m) 7m ago 35m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (32m) 7m ago 42m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (30m) 7m ago 39m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (28m) 7m ago 42m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (27m) 7m ago 39m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (30m) 7m ago 41m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (30m) 7m ago 40m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (26m) 7m ago 39m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (25m) 7m ago 38m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (25m) 7m ago 38m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:51:05.361 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (24m) 7m ago 37m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:51:05.362 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (24m) 7m ago 37m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:51:05.362 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (23m) 7m ago 36m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:51:05.362 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (30m) 7m ago 40m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:51:05.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff63c06ec60 msgr2=0x7ff63c071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff63c06ec60 0x7ff63c071120 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff648005950 tx=0x7ff64800b410 comp rx=0 tx=0).stop 2024-04-22T15:51:05.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff654105b60 msgr2=0x7ff654197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff654105b60 0x7ff654197e90 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff64400bd70 tx=0x7ff64400bda0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 shutdown_connections 2024-04-22T15:51:05.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff63c06ec60 0x7ff63c071120 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6540ffad0 0x7ff654197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 --2- 172.21.15.96:0/1223658576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff654105b60 0x7ff654197e90 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 >> 172.21.15.96:0/1223658576 conn(0x7ff6540fb600 msgr2=0x7ff6540fc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:05.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 shutdown_connections 2024-04-22T15:51:05.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.359+0000 7ff65890e700 1 -- 172.21.15.96:0/1223658576 wait complete. 2024-04-22T15:51:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.461+0000 7f1990b8e700 1 -- 172.21.15.96:0/1630301922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0ff630 msgr2=0x7f198c0ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.461+0000 7f1990b8e700 1 --2- 172.21.15.96:0/1630301922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0ff630 0x7f198c0ffa10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f197c00aa20 tx=0x7f197c00ad30 comp rx=0 tx=0).stop 2024-04-22T15:51:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.461+0000 7f1990b8e700 1 -- 172.21.15.96:0/1630301922 shutdown_connections 2024-04-22T15:51:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.461+0000 7f1990b8e700 1 --2- 172.21.15.96:0/1630301922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f198c0fffe0 0x7f198c103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.461+0000 7f1990b8e700 1 --2- 172.21.15.96:0/1630301922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0ff630 0x7f198c0ffa10 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.461+0000 7f1990b8e700 1 -- 172.21.15.96:0/1630301922 >> 172.21.15.96:0/1630301922 conn(0x7f198c074db0 msgr2=0x7f198c0751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.462+0000 7f1990b8e700 1 -- 172.21.15.96:0/1630301922 shutdown_connections 2024-04-22T15:51:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.462+0000 7f1990b8e700 1 -- 172.21.15.96:0/1630301922 wait complete. 2024-04-22T15:51:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.462+0000 7f1990b8e700 1 Processor -- start 2024-04-22T15:51:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.462+0000 7f1990b8e700 1 -- start start 2024-04-22T15:51:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1990b8e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f198c0ff630 0x7f198c194700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1990b8e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0fffe0 0x7f198c194c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1990b8e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c195180 con 0x7f198c0fffe0 2024-04-22T15:51:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1990b8e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c1952c0 con 0x7f198c0ff630 2024-04-22T15:51:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1989d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0fffe0 0x7f198c194c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1989d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0fffe0 0x7f198c194c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40992/0 (socket says 172.21.15.96:40992) 2024-04-22T15:51:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1989d9b700 1 -- 172.21.15.96:0/3300212682 learned_addr learned my addr 172.21.15.96:0/3300212682 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f198a59c700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f198c0ff630 0x7f198c194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1989d9b700 1 -- 172.21.15.96:0/3300212682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f198c0ff630 msgr2=0x7f198c194700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1989d9b700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f198c0ff630 0x7f198c194700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1989d9b700 1 -- 172.21.15.96:0/3300212682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c00a700 con 0x7f198c0fffe0 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1989d9b700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0fffe0 0x7f198c194c40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f198000e8f0 tx=0x7f198000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f197b7fe700 1 -- 172.21.15.96:0/3300212682 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1980009790 con 0x7f198c0fffe0 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.463+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198c1a2cb0 con 0x7f198c0fffe0 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.464+0000 7f197b7fe700 1 -- 172.21.15.96:0/3300212682 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f198000c810 con 0x7f198c0fffe0 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.464+0000 7f197b7fe700 1 -- 172.21.15.96:0/3300212682 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1980005580 con 0x7f198c0fffe0 2024-04-22T15:51:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.464+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198c1a3010 con 0x7f198c0fffe0 2024-04-22T15:51:05.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.465+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196c0052f0 con 0x7f198c0fffe0 2024-04-22T15:51:05.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.465+0000 7f197b7fe700 1 -- 172.21.15.96:0/3300212682 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1980004a00 con 0x7f198c0fffe0 2024-04-22T15:51:05.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.466+0000 7f197b7fe700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f197406ecc0 0x7f1974071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:05.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.466+0000 7f197b7fe700 1 -- 172.21.15.96:0/3300212682 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1980014070 con 0x7f198c0fffe0 2024-04-22T15:51:05.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.466+0000 7f198a59c700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f197406ecc0 0x7f1974071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:05.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.466+0000 7f198a59c700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f197406ecc0 0x7f1974071180 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f197c000c00 tx=0x7f197c00c670 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:05.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.471+0000 7f197b7fe700 1 -- 172.21.15.96:0/3300212682 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1980057e50 con 0x7f198c0fffe0 2024-04-22T15:51:05.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.757+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f196c005db0 con 0x7f198c0fffe0 2024-04-22T15:51:05.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.757+0000 7f197b7fe700 1 -- 172.21.15.96:0/3300212682 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f198005b510 con 0x7f198c0fffe0 2024-04-22T15:51:05.761 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:51:05.761 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:51:05.761 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:51:05.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:51:05.762 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:51:05.762 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:51:05.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:51:05.762 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:51:05.762 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:51:05.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:51:05.763 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:51:05.763 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:51:05.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:51:05.763 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:51:05.763 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:51:05.763 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:51:05.763 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:51:05.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f197406ecc0 msgr2=0x7f1974071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f197406ecc0 0x7f1974071180 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f197c000c00 tx=0x7f197c00c670 comp rx=0 tx=0).stop 2024-04-22T15:51:05.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0fffe0 msgr2=0x7f198c194c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0fffe0 0x7f198c194c40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f198000e8f0 tx=0x7f198000ec00 comp rx=0 tx=0).stop 2024-04-22T15:51:05.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 shutdown_connections 2024-04-22T15:51:05.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f197406ecc0 0x7f1974071180 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f198c0ff630 0x7f198c194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 --2- 172.21.15.96:0/3300212682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f198c0fffe0 0x7f198c194c40 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 >> 172.21.15.96:0/3300212682 conn(0x7f198c074db0 msgr2=0x7f198c0fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:05.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 shutdown_connections 2024-04-22T15:51:05.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.760+0000 7f1990b8e700 1 -- 172.21.15.96:0/3300212682 wait complete. 2024-04-22T15:51:05.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.863+0000 7fa09f947700 1 -- 172.21.15.96:0/1996059173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 msgr2=0x7fa0981021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.863+0000 7fa09f947700 1 --2- 172.21.15.96:0/1996059173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 0x7fa0981021e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa08c009910 tx=0x7fa08c009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.863+0000 7fa09f947700 1 -- 172.21.15.96:0/1996059173 shutdown_connections 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.863+0000 7fa09f947700 1 --2- 172.21.15.96:0/1996059173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa098102720 0x7fa09810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.863+0000 7fa09f947700 1 --2- 172.21.15.96:0/1996059173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 0x7fa0981021e0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.863+0000 7fa09f947700 1 -- 172.21.15.96:0/1996059173 >> 172.21.15.96:0/1996059173 conn(0x7fa098076740 msgr2=0x7fa0980fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.863+0000 7fa09f947700 1 -- 172.21.15.96:0/1996059173 shutdown_connections 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.863+0000 7fa09f947700 1 -- 172.21.15.96:0/1996059173 wait complete. 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09f947700 1 Processor -- start 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09f947700 1 -- start start 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09f947700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 0x7fa09819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09f947700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa098102720 0x7fa09819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09f947700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09819d360 con 0x7fa098101e00 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09f947700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa098196940 con 0x7fa098102720 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09d6e3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 0x7fa09819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09d6e3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 0x7fa09819c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40998/0 (socket says 172.21.15.96:40998) 2024-04-22T15:51:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09d6e3700 1 -- 172.21.15.96:0/2675290708 learned_addr learned my addr 172.21.15.96:0/2675290708 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09d6e3700 1 -- 172.21.15.96:0/2675290708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa098102720 msgr2=0x7fa09819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09d6e3700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa098102720 0x7fa09819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09d6e3700 1 -- 172.21.15.96:0/2675290708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c0095f0 con 0x7fa098101e00 2024-04-22T15:51:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.864+0000 7fa09d6e3700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 0x7fa09819c800 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00bd70 tx=0x7fa08c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.865+0000 7fa08a7fc700 1 -- 172.21.15.96:0/2675290708 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa08c01d070 con 0x7fa098101e00 2024-04-22T15:51:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.865+0000 7fa08a7fc700 1 -- 172.21.15.96:0/2675290708 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa08c0056e0 con 0x7fa098101e00 2024-04-22T15:51:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.865+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa098196bc0 con 0x7fa098101e00 2024-04-22T15:51:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.865+0000 7fa08a7fc700 1 -- 172.21.15.96:0/2675290708 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa08c017400 con 0x7fa098101e00 2024-04-22T15:51:05.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.865+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0981970b0 con 0x7fa098101e00 2024-04-22T15:51:05.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.865+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa098105520 con 0x7fa098101e00 2024-04-22T15:51:05.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.866+0000 7fa08a7fc700 1 -- 172.21.15.96:0/2675290708 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa08c003a60 con 0x7fa098101e00 2024-04-22T15:51:05.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.867+0000 7fa08a7fc700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa08406ecc0 0x7fa084071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:05.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.867+0000 7fa09cee2700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa08406ecc0 0x7fa084071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:05.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.867+0000 7fa08a7fc700 1 -- 172.21.15.96:0/2675290708 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa08c0902f0 con 0x7fa098101e00 2024-04-22T15:51:05.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.867+0000 7fa09cee2700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa08406ecc0 0x7fa084071180 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa094005950 tx=0x7fa0940058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:05.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:05.871+0000 7fa08a7fc700 1 -- 172.21.15.96:0/2675290708 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa08c05cb80 con 0x7fa098101e00 2024-04-22T15:51:06.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.118+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa09810dde0 con 0x7fa098101e00 2024-04-22T15:51:06.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.119+0000 7fa08a7fc700 1 -- 172.21.15.96:0/2675290708 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7fa08c05c710 con 0x7fa098101e00 2024-04-22T15:51:06.122 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:51:06.122 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:51:06.122 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:51:06.122 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:51:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:51:06.122 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:51:06.123 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:51:06.124 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:51:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:51:06.125 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:51:06.125 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:51:06.125 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:51:06.125 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa08406ecc0 msgr2=0x7fa084071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa08406ecc0 0x7fa084071180 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa094005950 tx=0x7fa0940058e0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 msgr2=0x7fa09819c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 0x7fa09819c800 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00bd70 tx=0x7fa08c00be50 comp rx=0 tx=0).stop 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 shutdown_connections 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa08406ecc0 0x7fa084071180 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa098101e00 0x7fa09819c800 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 --2- 172.21.15.96:0/2675290708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa098102720 0x7fa09819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 >> 172.21.15.96:0/2675290708 conn(0x7fa098076740 msgr2=0x7fa0981030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 shutdown_connections 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.122+0000 7fa09f947700 1 -- 172.21.15.96:0/2675290708 wait complete. 2024-04-22T15:51:06.126 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:51:06.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.225+0000 7f6e5820b700 1 -- 172.21.15.96:0/2433575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e50105c00 msgr2=0x7f6e50105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.225+0000 7f6e5820b700 1 --2- 172.21.15.96:0/2433575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e50105c00 0x7f6e50105fe0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009910 tx=0x7f6e4c009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.226+0000 7f6e5820b700 1 -- 172.21.15.96:0/2433575 shutdown_connections 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.226+0000 7f6e5820b700 1 --2- 172.21.15.96:0/2433575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e500ffb50 0x7f6e500fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.226+0000 7f6e5820b700 1 --2- 172.21.15.96:0/2433575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e50105c00 0x7f6e50105fe0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.226+0000 7f6e5820b700 1 -- 172.21.15.96:0/2433575 >> 172.21.15.96:0/2433575 conn(0x7f6e500fb680 msgr2=0x7f6e500fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.226+0000 7f6e5820b700 1 -- 172.21.15.96:0/2433575 shutdown_connections 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.226+0000 7f6e5820b700 1 -- 172.21.15.96:0/2433575 wait complete. 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.226+0000 7f6e5820b700 1 Processor -- start 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.226+0000 7f6e5820b700 1 -- start start 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.226+0000 7f6e5820b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e500ffb50 0x7f6e5019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e5820b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e50105c00 0x7f6e5019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e5820b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5019d3e0 con 0x7f6e500ffb50 2024-04-22T15:51:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e5820b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e50196a10 con 0x7f6e50105c00 2024-04-22T15:51:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e55fa7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e500ffb50 0x7f6e5019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e55fa7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e500ffb50 0x7f6e5019c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41026/0 (socket says 172.21.15.96:41026) 2024-04-22T15:51:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e55fa7700 1 -- 172.21.15.96:0/1526324299 learned_addr learned my addr 172.21.15.96:0/1526324299 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e557a6700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e50105c00 0x7f6e5019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e55fa7700 1 -- 172.21.15.96:0/1526324299 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e50105c00 msgr2=0x7f6e5019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e55fa7700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e50105c00 0x7f6e5019cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e55fa7700 1 -- 172.21.15.96:0/1526324299 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c0095f0 con 0x7f6e500ffb50 2024-04-22T15:51:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e55fa7700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e500ffb50 0x7f6e5019c880 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c006010 tx=0x7f6e4c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e46ffd700 1 -- 172.21.15.96:0/1526324299 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e4c01d070 con 0x7f6e500ffb50 2024-04-22T15:51:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e46ffd700 1 -- 172.21.15.96:0/1526324299 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e4c00f460 con 0x7f6e500ffb50 2024-04-22T15:51:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e46ffd700 1 -- 172.21.15.96:0/1526324299 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e4c017630 con 0x7f6e500ffb50 2024-04-22T15:51:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e50196c90 con 0x7f6e500ffb50 2024-04-22T15:51:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.227+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e501971b0 con 0x7f6e500ffb50 2024-04-22T15:51:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.228+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e50101200 con 0x7f6e500ffb50 2024-04-22T15:51:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.229+0000 7f6e46ffd700 1 -- 172.21.15.96:0/1526324299 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6e4c00f5d0 con 0x7f6e500ffb50 2024-04-22T15:51:06.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.229+0000 7f6e46ffd700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3c06ecc0 0x7f6e3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.229+0000 7f6e46ffd700 1 -- 172.21.15.96:0/1526324299 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6e4c090340 con 0x7f6e500ffb50 2024-04-22T15:51:06.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.229+0000 7f6e557a6700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3c06ecc0 0x7f6e3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.230+0000 7f6e557a6700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3c06ecc0 0x7f6e3c071180 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6e40006fd0 tx=0x7f6e40008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:06.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.234+0000 7f6e46ffd700 1 -- 172.21.15.96:0/1526324299 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6e4c05cd10 con 0x7f6e500ffb50 2024-04-22T15:51:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.447+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e5010cdc0 con 0x7f6e3c06ecc0 2024-04-22T15:51:06.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.449+0000 7f6e46ffd700 1 -- 172.21.15.96:0/1526324299 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6e5010cdc0 con 0x7f6e3c06ecc0 2024-04-22T15:51:06.451 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:51:06.452 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:51:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3c06ecc0 msgr2=0x7f6e3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3c06ecc0 0x7f6e3c071180 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6e40006fd0 tx=0x7f6e40008040 comp rx=0 tx=0).stop 2024-04-22T15:51:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e500ffb50 msgr2=0x7f6e5019c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e500ffb50 0x7f6e5019c880 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c006010 tx=0x7f6e4c00be20 comp rx=0 tx=0).stop 2024-04-22T15:51:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 shutdown_connections 2024-04-22T15:51:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3c06ecc0 0x7f6e3c071180 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e500ffb50 0x7f6e5019c880 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 --2- 172.21.15.96:0/1526324299 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e50105c00 0x7f6e5019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 >> 172.21.15.96:0/1526324299 conn(0x7f6e500fb680 msgr2=0x7f6e500693e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:06.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 shutdown_connections 2024-04-22T15:51:06.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.451+0000 7f6e5820b700 1 -- 172.21.15.96:0/1526324299 wait complete. 2024-04-22T15:51:06.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 -- 172.21.15.96:0/2646382735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4544100e10 msgr2=0x7f45441011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 --2- 172.21.15.96:0/2646382735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4544100e10 0x7f45441011f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4534009910 tx=0x7f4534009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:06.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 -- 172.21.15.96:0/2646382735 shutdown_connections 2024-04-22T15:51:06.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 --2- 172.21.15.96:0/2646382735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45441017c0 0x7f4544105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 --2- 172.21.15.96:0/2646382735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4544100e10 0x7f45441011f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 -- 172.21.15.96:0/2646382735 >> 172.21.15.96:0/2646382735 conn(0x7f45440fc6e0 msgr2=0x7f45440feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:06.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 -- 172.21.15.96:0/2646382735 shutdown_connections 2024-04-22T15:51:06.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 -- 172.21.15.96:0/2646382735 wait complete. 2024-04-22T15:51:06.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 Processor -- start 2024-04-22T15:51:06.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.556+0000 7f454a4ad700 1 -- start start 2024-04-22T15:51:06.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f454a4ad700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4544100e10 0x7f4544196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f454a4ad700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45441017c0 0x7f45441970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f454a4ad700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544197790 con 0x7f45441017c0 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f454a4ad700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454419b3a0 con 0x7f4544100e10 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f45437fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45441017c0 0x7f45441970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f45437fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45441017c0 0x7f45441970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41040/0 (socket says 172.21.15.96:41040) 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f45437fe700 1 -- 172.21.15.96:0/3430425388 learned_addr learned my addr 172.21.15.96:0/3430425388 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f4543fff700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4544100e10 0x7f4544196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f45437fe700 1 -- 172.21.15.96:0/3430425388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4544100e10 msgr2=0x7f4544196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f45437fe700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4544100e10 0x7f4544196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f45437fe700 1 -- 172.21.15.96:0/3430425388 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45340095f0 con 0x7f45441017c0 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f4543fff700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4544100e10 0x7f4544196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.557+0000 7f45437fe700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45441017c0 0x7f45441970b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f453800d900 tx=0x7f453800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.558+0000 7f45417fa700 1 -- 172.21.15.96:0/3430425388 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4538009900 con 0x7f45441017c0 2024-04-22T15:51:06.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.558+0000 7f45417fa700 1 -- 172.21.15.96:0/3430425388 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4538010460 con 0x7f45441017c0 2024-04-22T15:51:06.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.558+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454419b680 con 0x7f45441017c0 2024-04-22T15:51:06.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.558+0000 7f45417fa700 1 -- 172.21.15.96:0/3430425388 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f453800f5f0 con 0x7f45441017c0 2024-04-22T15:51:06.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.558+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454419bbd0 con 0x7f45441017c0 2024-04-22T15:51:06.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.558+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45441091b0 con 0x7f45441017c0 2024-04-22T15:51:06.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.559+0000 7f45417fa700 1 -- 172.21.15.96:0/3430425388 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f45380105d0 con 0x7f45441017c0 2024-04-22T15:51:06.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.560+0000 7f45417fa700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f452c06ecc0 0x7f452c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.560+0000 7f4543fff700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f452c06ecc0 0x7f452c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.560+0000 7f45417fa700 1 -- 172.21.15.96:0/3430425388 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f453808ea20 con 0x7f45441017c0 2024-04-22T15:51:06.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.560+0000 7f4543fff700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f452c06ecc0 0x7f452c071180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4534000c00 tx=0x7f453401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:06.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.564+0000 7f45417fa700 1 -- 172.21.15.96:0/3430425388 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4538057800 con 0x7f45441017c0 2024-04-22T15:51:06.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:06 smithi096 ceph-mon[112443]: from='client.35600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:06 smithi096 ceph-mon[112443]: from='client.35604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:06 smithi096 ceph-mon[112443]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:06 smithi096 ceph-mon[112443]: pgmap v878: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:06 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3300212682' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:06.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:06 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2675290708' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:51:06.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.853+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4544197dc0 con 0x7f45441017c0 2024-04-22T15:51:06.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.853+0000 7f45417fa700 1 -- 172.21.15.96:0/3430425388 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f453805aec0 con 0x7f45441017c0 2024-04-22T15:51:06.856 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:51:06.857 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:51:06.858 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:51:06.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.855+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f452c06ecc0 msgr2=0x7f452c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.855+0000 7f454a4ad700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f452c06ecc0 0x7f452c071180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4534000c00 tx=0x7f453401a040 comp rx=0 tx=0).stop 2024-04-22T15:51:06.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.855+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45441017c0 msgr2=0x7f45441970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.855+0000 7f454a4ad700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45441017c0 0x7f45441970b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f453800d900 tx=0x7f453800dcc0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.855+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 shutdown_connections 2024-04-22T15:51:06.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.855+0000 7f454a4ad700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f452c06ecc0 0x7f452c071180 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.855+0000 7f454a4ad700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4544100e10 0x7f4544196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.855+0000 7f454a4ad700 1 --2- 172.21.15.96:0/3430425388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45441017c0 0x7f45441970b0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.855+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 >> 172.21.15.96:0/3430425388 conn(0x7f45440fc6e0 msgr2=0x7f45440fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:06.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.856+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 shutdown_connections 2024-04-22T15:51:06.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:06.856+0000 7f454a4ad700 1 -- 172.21.15.96:0/3430425388 wait complete. 2024-04-22T15:51:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:06 smithi196 ceph-mon[81815]: from='client.35600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:06 smithi196 ceph-mon[81815]: from='client.35604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:06 smithi196 ceph-mon[81815]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:06 smithi196 ceph-mon[81815]: pgmap v878: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:07.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:06 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3300212682' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:06 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2675290708' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:51:07.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:07 smithi096 ceph-mon[112443]: from='client.35620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:07 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3430425388' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:51:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:07 smithi196 ceph-mon[81815]: from='client.35620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:07 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3430425388' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:51:08.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:08 smithi096 ceph-mon[112443]: pgmap v879: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:08 smithi196 ceph-mon[81815]: pgmap v879: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:09.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:09 smithi096 ceph-mon[112443]: pgmap v880: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:09 smithi196 ceph-mon[81815]: pgmap v880: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:12 smithi096 ceph-mon[112443]: pgmap v881: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:12 smithi196 ceph-mon[81815]: pgmap v881: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:14.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:14 smithi096 ceph-mon[112443]: pgmap v882: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:14 smithi196 ceph-mon[81815]: pgmap v882: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:16.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:16 smithi096 ceph-mon[112443]: pgmap v883: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:16 smithi196 ceph-mon[81815]: pgmap v883: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:18.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:18 smithi096 ceph-mon[112443]: pgmap v884: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:18 smithi196 ceph-mon[81815]: pgmap v884: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:20.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:20 smithi096 ceph-mon[112443]: pgmap v885: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:20 smithi196 ceph-mon[81815]: pgmap v885: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:22.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:22 smithi096 ceph-mon[112443]: pgmap v886: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:22 smithi196 ceph-mon[81815]: pgmap v886: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:24.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:24 smithi096 ceph-mon[112443]: pgmap v887: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:24 smithi196 ceph-mon[81815]: pgmap v887: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:25.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:25 smithi096 ceph-mon[112443]: pgmap v888: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:25 smithi196 ceph-mon[81815]: pgmap v888: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:28.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:28 smithi096 ceph-mon[112443]: pgmap v889: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:28 smithi196 ceph-mon[81815]: pgmap v889: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:30.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:30 smithi096 ceph-mon[112443]: pgmap v890: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:30 smithi196 ceph-mon[81815]: pgmap v890: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:32.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:32 smithi096 ceph-mon[112443]: pgmap v891: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:32 smithi196 ceph-mon[81815]: pgmap v891: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:34.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:34 smithi096 ceph-mon[112443]: pgmap v892: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:34 smithi196 ceph-mon[81815]: pgmap v892: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:36.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.976+0000 7ff6482f8700 1 -- 172.21.15.96:0/4037729930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6401017d0 msgr2=0x7ff640105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:36.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.976+0000 7ff6482f8700 1 --2- 172.21.15.96:0/4037729930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6401017d0 0x7ff640105820 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009910 tx=0x7ff63c009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:36.979 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:36 smithi096 ceph-mon[112443]: pgmap v893: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:36.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.976+0000 7ff6482f8700 1 -- 172.21.15.96:0/4037729930 shutdown_connections 2024-04-22T15:51:36.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.976+0000 7ff6482f8700 1 --2- 172.21.15.96:0/4037729930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6401017d0 0x7ff640105820 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:36.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.976+0000 7ff6482f8700 1 --2- 172.21.15.96:0/4037729930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff640100e20 0x7ff640101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.976+0000 7ff6482f8700 1 -- 172.21.15.96:0/4037729930 >> 172.21.15.96:0/4037729930 conn(0x7ff6400fc6d0 msgr2=0x7ff6400feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.976+0000 7ff6482f8700 1 -- 172.21.15.96:0/4037729930 shutdown_connections 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.976+0000 7ff6482f8700 1 -- 172.21.15.96:0/4037729930 wait complete. 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.976+0000 7ff6482f8700 1 Processor -- start 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff6482f8700 1 -- start start 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff6482f8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff640100e20 0x7ff64019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff6482f8700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6401017d0 0x7ff64019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff6482f8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64019d380 con 0x7ff640100e20 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff6482f8700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff640196960 con 0x7ff6401017d0 2024-04-22T15:51:36.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff646094700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff640100e20 0x7ff64019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff646094700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff640100e20 0x7ff64019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41962/0 (socket says 172.21.15.96:41962) 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff646094700 1 -- 172.21.15.96:0/3493773560 learned_addr learned my addr 172.21.15.96:0/3493773560 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff646094700 1 -- 172.21.15.96:0/3493773560 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6401017d0 msgr2=0x7ff64019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff646094700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6401017d0 0x7ff64019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff646094700 1 -- 172.21.15.96:0/3493773560 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c0095f0 con 0x7ff640100e20 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff646094700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff640100e20 0x7ff64019c820 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff63000d8d0 tx=0x7ff63000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.977+0000 7ff6377fe700 1 -- 172.21.15.96:0/3493773560 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff630009940 con 0x7ff640100e20 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.978+0000 7ff6377fe700 1 -- 172.21.15.96:0/3493773560 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff630010460 con 0x7ff640100e20 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.978+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff640196c40 con 0x7ff640100e20 2024-04-22T15:51:36.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.978+0000 7ff6377fe700 1 -- 172.21.15.96:0/3493773560 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63000f5f0 con 0x7ff640100e20 2024-04-22T15:51:36.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.978+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff640197190 con 0x7ff640100e20 2024-04-22T15:51:36.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.979+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff640109220 con 0x7ff640100e20 2024-04-22T15:51:36.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.979+0000 7ff6377fe700 1 -- 172.21.15.96:0/3493773560 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff6300105d0 con 0x7ff640100e20 2024-04-22T15:51:36.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.979+0000 7ff6377fe700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff62c06ecd0 0x7ff62c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:36.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.980+0000 7ff6377fe700 1 -- 172.21.15.96:0/3493773560 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff63008eed0 con 0x7ff640100e20 2024-04-22T15:51:36.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.980+0000 7ff645893700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff62c06ecd0 0x7ff62c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:36.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.980+0000 7ff645893700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff62c06ecd0 0x7ff62c071190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff63c006010 tx=0x7ff63c00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:36.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:36.984+0000 7ff6377fe700 1 -- 172.21.15.96:0/3493773560 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff63005b760 con 0x7ff640100e20 2024-04-22T15:51:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:36 smithi196 ceph-mon[81815]: pgmap v893: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:37.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.204+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff640197ec0 con 0x7ff62c06ecd0 2024-04-22T15:51:37.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.205+0000 7ff6377fe700 1 -- 172.21.15.96:0/3493773560 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff640197ec0 con 0x7ff62c06ecd0 2024-04-22T15:51:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff62c06ecd0 msgr2=0x7ff62c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff62c06ecd0 0x7ff62c071190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff63c006010 tx=0x7ff63c00b540 comp rx=0 tx=0).stop 2024-04-22T15:51:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff640100e20 msgr2=0x7ff64019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff640100e20 0x7ff64019c820 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff63000d8d0 tx=0x7ff63000dc90 comp rx=0 tx=0).stop 2024-04-22T15:51:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 shutdown_connections 2024-04-22T15:51:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff62c06ecd0 0x7ff62c071190 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff640100e20 0x7ff64019c820 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 --2- 172.21.15.96:0/3493773560 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6401017d0 0x7ff64019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 >> 172.21.15.96:0/3493773560 conn(0x7ff6400fc6d0 msgr2=0x7ff6400fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:37.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 shutdown_connections 2024-04-22T15:51:37.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.207+0000 7ff6482f8700 1 -- 172.21.15.96:0/3493773560 wait complete. 2024-04-22T15:51:37.220 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:51:37.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.311+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/3696706548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a881017d0 msgr2=0x7f6a88105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.311+0000 7f6a8d9cf700 1 --2- 172.21.15.96:0/3696706548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a881017d0 0x7f6a88105820 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009910 tx=0x7f6a7c009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:37.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.311+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/3696706548 shutdown_connections 2024-04-22T15:51:37.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.311+0000 7f6a8d9cf700 1 --2- 172.21.15.96:0/3696706548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a881017d0 0x7f6a88105820 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.311+0000 7f6a8d9cf700 1 --2- 172.21.15.96:0/3696706548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a88100e20 0x7f6a88101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.311+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/3696706548 >> 172.21.15.96:0/3696706548 conn(0x7f6a880fc6d0 msgr2=0x7f6a880feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.311+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/3696706548 shutdown_connections 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.311+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/3696706548 wait complete. 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.312+0000 7f6a8d9cf700 1 Processor -- start 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.312+0000 7f6a8d9cf700 1 -- start start 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.312+0000 7f6a8d9cf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a88100e20 0x7f6a8819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.312+0000 7f6a8d9cf700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881017d0 0x7f6a8819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.312+0000 7f6a8d9cf700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8819d4c0 con 0x7f6a88100e20 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.312+0000 7f6a8d9cf700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a88196aa0 con 0x7f6a881017d0 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.312+0000 7f6a86ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a88100e20 0x7f6a8819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.312+0000 7f6a86ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a88100e20 0x7f6a8819c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41986/0 (socket says 172.21.15.96:41986) 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.312+0000 7f6a86ffd700 1 -- 172.21.15.96:0/517536952 learned_addr learned my addr 172.21.15.96:0/517536952 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:37.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a867fc700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881017d0 0x7f6a8819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a86ffd700 1 -- 172.21.15.96:0/517536952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881017d0 msgr2=0x7f6a8819cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a86ffd700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881017d0 0x7f6a8819cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a86ffd700 1 -- 172.21.15.96:0/517536952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c0095f0 con 0x7f6a88100e20 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a867fc700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881017d0 0x7f6a8819cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a86ffd700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a88100e20 0x7f6a8819c960 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800b4b0 tx=0x7f6a7800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a8c9cd700 1 -- 172.21.15.96:0/517536952 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a78011800 con 0x7f6a88100e20 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a8c9cd700 1 -- 172.21.15.96:0/517536952 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a78011e20 con 0x7f6a88100e20 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a88196d80 con 0x7f6a88100e20 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a8c9cd700 1 -- 172.21.15.96:0/517536952 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a7800f2e0 con 0x7f6a88100e20 2024-04-22T15:51:37.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.313+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a881972d0 con 0x7f6a88100e20 2024-04-22T15:51:37.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.314+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a88109150 con 0x7f6a88100e20 2024-04-22T15:51:37.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.314+0000 7f6a8c9cd700 1 -- 172.21.15.96:0/517536952 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6a78010380 con 0x7f6a88100e20 2024-04-22T15:51:37.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.315+0000 7f6a8c9cd700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a7006ecc0 0x7f6a70071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:37.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.315+0000 7f6a8c9cd700 1 -- 172.21.15.96:0/517536952 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6a7808e9e0 con 0x7f6a88100e20 2024-04-22T15:51:37.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.315+0000 7f6a867fc700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a7006ecc0 0x7f6a70071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:37.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.315+0000 7f6a867fc700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a7006ecc0 0x7f6a70071180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c005f50 tx=0x7f6a7c005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:37.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.319+0000 7f6a8c9cd700 1 -- 172.21.15.96:0/517536952 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6a7805b3a0 con 0x7f6a88100e20 2024-04-22T15:51:37.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.537+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a88197fd0 con 0x7f6a7006ecc0 2024-04-22T15:51:37.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.538+0000 7f6a8c9cd700 1 -- 172.21.15.96:0/517536952 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6a88197fd0 con 0x7f6a7006ecc0 2024-04-22T15:51:37.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.540+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a7006ecc0 msgr2=0x7f6a70071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.540+0000 7f6a8d9cf700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a7006ecc0 0x7f6a70071180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c005f50 tx=0x7f6a7c005dc0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.540+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a88100e20 msgr2=0x7f6a8819c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.540+0000 7f6a8d9cf700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a88100e20 0x7f6a8819c960 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800b4b0 tx=0x7f6a7800b870 comp rx=0 tx=0).stop 2024-04-22T15:51:37.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.540+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 shutdown_connections 2024-04-22T15:51:37.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.540+0000 7f6a8d9cf700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6a7006ecc0 0x7f6a70071180 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.540+0000 7f6a8d9cf700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a88100e20 0x7f6a8819c960 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.540+0000 7f6a8d9cf700 1 --2- 172.21.15.96:0/517536952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881017d0 0x7f6a8819cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.541+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 >> 172.21.15.96:0/517536952 conn(0x7f6a880fc6d0 msgr2=0x7f6a880fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:37.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.541+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 shutdown_connections 2024-04-22T15:51:37.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.541+0000 7f6a8d9cf700 1 -- 172.21.15.96:0/517536952 wait complete. 2024-04-22T15:51:37.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.643+0000 7fe478ff7700 1 -- 172.21.15.96:0/2585998786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4740ffaa0 msgr2=0x7fe4740fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.643+0000 7fe478ff7700 1 --2- 172.21.15.96:0/2585998786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4740ffaa0 0x7fe4740fff20 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fe464009910 tx=0x7fe464009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:37.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.643+0000 7fe478ff7700 1 -- 172.21.15.96:0/2585998786 shutdown_connections 2024-04-22T15:51:37.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.643+0000 7fe478ff7700 1 --2- 172.21.15.96:0/2585998786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4740ffaa0 0x7fe4740fff20 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.643+0000 7fe478ff7700 1 --2- 172.21.15.96:0/2585998786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe474105aa0 0x7fe474105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.643+0000 7fe478ff7700 1 -- 172.21.15.96:0/2585998786 >> 172.21.15.96:0/2585998786 conn(0x7fe474076740 msgr2=0x7fe4740fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:37.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.643+0000 7fe478ff7700 1 -- 172.21.15.96:0/2585998786 shutdown_connections 2024-04-22T15:51:37.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.643+0000 7fe478ff7700 1 -- 172.21.15.96:0/2585998786 wait complete. 2024-04-22T15:51:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe478ff7700 1 Processor -- start 2024-04-22T15:51:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe478ff7700 1 -- start start 2024-04-22T15:51:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe478ff7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4740ffaa0 0x7fe4741978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe478ff7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe474105aa0 0x7fe474197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe478ff7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4741983d0 con 0x7fe474105aa0 2024-04-22T15:51:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe478ff7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe474198510 con 0x7fe4740ffaa0 2024-04-22T15:51:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe471d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe474105aa0 0x7fe474197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe471d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe474105aa0 0x7fe474197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42004/0 (socket says 172.21.15.96:42004) 2024-04-22T15:51:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe471d9b700 1 -- 172.21.15.96:0/672205926 learned_addr learned my addr 172.21.15.96:0/672205926 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.644+0000 7fe47259c700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4740ffaa0 0x7fe4741978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe471d9b700 1 -- 172.21.15.96:0/672205926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4740ffaa0 msgr2=0x7fe4741978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe471d9b700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4740ffaa0 0x7fe4741978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe471d9b700 1 -- 172.21.15.96:0/672205926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640095f0 con 0x7fe474105aa0 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe47259c700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4740ffaa0 0x7fe4741978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe471d9b700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe474105aa0 0x7fe474197e00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fe46400bd70 tx=0x7fe46400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe4637fe700 1 -- 172.21.15.96:0/672205926 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe46401d070 con 0x7fe474105aa0 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe4637fe700 1 -- 172.21.15.96:0/672205926 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4640056e0 con 0x7fe474105aa0 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4741a0930 con 0x7fe474105aa0 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe4637fe700 1 -- 172.21.15.96:0/672205926 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe464017400 con 0x7fe474105aa0 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.645+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4741a0ce0 con 0x7fe474105aa0 2024-04-22T15:51:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.646+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4741080c0 con 0x7fe474105aa0 2024-04-22T15:51:37.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.646+0000 7fe4637fe700 1 -- 172.21.15.96:0/672205926 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe464017640 con 0x7fe474105aa0 2024-04-22T15:51:37.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.647+0000 7fe4637fe700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe45c06ecc0 0x7fe45c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:37.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.647+0000 7fe47259c700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe45c06ecc0 0x7fe45c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:37.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.647+0000 7fe4637fe700 1 -- 172.21.15.96:0/672205926 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe464090330 con 0x7fe474105aa0 2024-04-22T15:51:37.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.648+0000 7fe47259c700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe45c06ecc0 0x7fe45c071180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe468005950 tx=0x7fe4680058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:37.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.652+0000 7fe4637fe700 1 -- 172.21.15.96:0/672205926 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe464059090 con 0x7fe474105aa0 2024-04-22T15:51:37.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.863+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe474068c30 con 0x7fe45c06ecc0 2024-04-22T15:51:37.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.873+0000 7fe4637fe700 1 -- 172.21.15.96:0/672205926 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe474068c30 con 0x7fe45c06ecc0 2024-04-22T15:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (30m) 7m ago 41m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (41m) 7m ago 41m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (40m) 7m ago 40m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (27m) 7m ago 41m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (27m) 7m ago 40m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (29m) 7m ago 41m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (23m) 7m ago 35m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (23m) 7m ago 35m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (23m) 7m ago 35m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (8m) 7m ago 35m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (32m) 7m ago 43m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (31m) 7m ago 40m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (28m) 7m ago 43m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (28m) 7m ago 40m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (31m) 7m ago 41m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (31m) 7m ago 40m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (27m) 7m ago 39m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (26m) 7m ago 39m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (25m) 7m ago 38m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (25m) 7m ago 38m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (24m) 7m ago 37m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (24m) 7m ago 37m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (30m) 7m ago 41m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:51:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.875+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe45c06ecc0 msgr2=0x7fe45c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.875+0000 7fe478ff7700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe45c06ecc0 0x7fe45c071180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe468005950 tx=0x7fe4680058e0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.875+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe474105aa0 msgr2=0x7fe474197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.875+0000 7fe478ff7700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe474105aa0 0x7fe474197e00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fe46400bd70 tx=0x7fe46400be50 comp rx=0 tx=0).stop 2024-04-22T15:51:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.875+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 shutdown_connections 2024-04-22T15:51:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.875+0000 7fe478ff7700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe45c06ecc0 0x7fe45c071180 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.875+0000 7fe478ff7700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4740ffaa0 0x7fe4741978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.875+0000 7fe478ff7700 1 --2- 172.21.15.96:0/672205926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe474105aa0 0x7fe474197e00 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.875+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 >> 172.21.15.96:0/672205926 conn(0x7fe474076740 msgr2=0x7fe4740fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:37.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.876+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 shutdown_connections 2024-04-22T15:51:37.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.876+0000 7fe478ff7700 1 -- 172.21.15.96:0/672205926 wait complete. 2024-04-22T15:51:37.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.977+0000 7f108ed89700 1 -- 172.21.15.96:0/1018800200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 msgr2=0x7f1088101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.977+0000 7f108ed89700 1 --2- 172.21.15.96:0/1018800200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 0x7f1088101220 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f108000b0a0 tx=0x7f108000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.977+0000 7f108ed89700 1 -- 172.21.15.96:0/1018800200 shutdown_connections 2024-04-22T15:51:37.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.977+0000 7f108ed89700 1 --2- 172.21.15.96:0/1018800200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10881017f0 0x7f1088105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.977+0000 7f108ed89700 1 --2- 172.21.15.96:0/1018800200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 0x7f1088101220 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.977+0000 7f108ed89700 1 -- 172.21.15.96:0/1018800200 >> 172.21.15.96:0/1018800200 conn(0x7f10880fc730 msgr2=0x7f10880feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.977+0000 7f108ed89700 1 -- 172.21.15.96:0/1018800200 shutdown_connections 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.977+0000 7f108ed89700 1 -- 172.21.15.96:0/1018800200 wait complete. 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108ed89700 1 Processor -- start 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108ed89700 1 -- start start 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108ed89700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 0x7f108819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108ed89700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10881017f0 0x7f108819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108ed89700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108819d380 con 0x7f1088100e40 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108ed89700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10881969b0 con 0x7f10881017f0 2024-04-22T15:51:37.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108cb25700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 0x7f108819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:37.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108cb25700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 0x7f108819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42024/0 (socket says 172.21.15.96:42024) 2024-04-22T15:51:37.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108cb25700 1 -- 172.21.15.96:0/2114928157 learned_addr learned my addr 172.21.15.96:0/2114928157 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:37.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f1087fff700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10881017f0 0x7f108819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:37.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108cb25700 1 -- 172.21.15.96:0/2114928157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10881017f0 msgr2=0x7f108819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:37.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108cb25700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10881017f0 0x7f108819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:37.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.978+0000 7f108cb25700 1 -- 172.21.15.96:0/2114928157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1080009d00 con 0x7f1088100e40 2024-04-22T15:51:37.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.979+0000 7f108cb25700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 0x7f108819c820 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1080000f80 tx=0x7f1080007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:37.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.979+0000 7f1085ffb700 1 -- 172.21.15.96:0/2114928157 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f108000e040 con 0x7f1088100e40 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.979+0000 7f1085ffb700 1 -- 172.21.15.96:0/2114928157 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1080003de0 con 0x7f1088100e40 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.979+0000 7f1085ffb700 1 -- 172.21.15.96:0/2114928157 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f108001d720 con 0x7f1088100e40 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.979+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1088196c30 con 0x7f1088100e40 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.979+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1088197120 con 0x7f1088100e40 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.980+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10881091e0 con 0x7f1088100e40 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.981+0000 7f1085ffb700 1 -- 172.21.15.96:0/2114928157 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1080004690 con 0x7f1088100e40 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.981+0000 7f1085ffb700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f107406ecc0 0x7f1074071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.981+0000 7f1085ffb700 1 -- 172.21.15.96:0/2114928157 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1080090610 con 0x7f1088100e40 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.981+0000 7f1087fff700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f107406ecc0 0x7f1074071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:37.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.982+0000 7f1087fff700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f107406ecc0 0x7f1074071180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1078005fd0 tx=0x7f107800a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:37.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:37.986+0000 7f1085ffb700 1 -- 172.21.15.96:0/2114928157 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f108005cf20 con 0x7f1088100e40 2024-04-22T15:51:38.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.269+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1088197e20 con 0x7f1088100e40 2024-04-22T15:51:38.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.269+0000 7f1085ffb700 1 -- 172.21.15.96:0/2114928157 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f108005cab0 con 0x7f1088100e40 2024-04-22T15:51:38.273 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:51:38.273 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:51:38.273 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:51:38.273 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:51:38.273 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:51:38.274 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:51:38.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:51:38.274 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:51:38.274 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:51:38.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:51:38.275 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:51:38.275 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:51:38.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:51:38.275 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:51:38.275 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:51:38.275 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:51:38.275 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:51:38.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.271+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f107406ecc0 msgr2=0x7f1074071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.271+0000 7f108ed89700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f107406ecc0 0x7f1074071180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1078005fd0 tx=0x7f107800a560 comp rx=0 tx=0).stop 2024-04-22T15:51:38.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.271+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 msgr2=0x7f108819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.271+0000 7f108ed89700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 0x7f108819c820 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1080000f80 tx=0x7f1080007bd0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.271+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 shutdown_connections 2024-04-22T15:51:38.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.271+0000 7f108ed89700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f107406ecc0 0x7f1074071180 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.271+0000 7f108ed89700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1088100e40 0x7f108819c820 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.271+0000 7f108ed89700 1 --2- 172.21.15.96:0/2114928157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10881017f0 0x7f108819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.271+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 >> 172.21.15.96:0/2114928157 conn(0x7f10880fc730 msgr2=0x7f10880fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:38.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.272+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 shutdown_connections 2024-04-22T15:51:38.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.272+0000 7f108ed89700 1 -- 172.21.15.96:0/2114928157 wait complete. 2024-04-22T15:51:38.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.376+0000 7f6244397700 1 -- 172.21.15.96:0/1917440149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c1017e0 msgr2=0x7f623c105830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.376+0000 7f6244397700 1 --2- 172.21.15.96:0/1917440149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c1017e0 0x7f623c105830 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f622c009910 tx=0x7f622c009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:38.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.376+0000 7f6244397700 1 -- 172.21.15.96:0/1917440149 shutdown_connections 2024-04-22T15:51:38.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.376+0000 7f6244397700 1 --2- 172.21.15.96:0/1917440149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c1017e0 0x7f623c105830 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.376+0000 7f6244397700 1 --2- 172.21.15.96:0/1917440149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f623c100e30 0x7f623c101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.376+0000 7f6244397700 1 -- 172.21.15.96:0/1917440149 >> 172.21.15.96:0/1917440149 conn(0x7f623c0fc6e0 msgr2=0x7f623c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:38.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.376+0000 7f6244397700 1 -- 172.21.15.96:0/1917440149 shutdown_connections 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.376+0000 7f6244397700 1 -- 172.21.15.96:0/1917440149 wait complete. 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.376+0000 7f6244397700 1 Processor -- start 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6244397700 1 -- start start 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6244397700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c100e30 0x7f623c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6244397700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f623c1017e0 0x7f623c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6244397700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c073c60 con 0x7f623c100e30 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6244397700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c073da0 con 0x7f623c1017e0 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6242133700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c100e30 0x7f623c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6242133700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c100e30 0x7f623c0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42048/0 (socket says 172.21.15.96:42048) 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6242133700 1 -- 172.21.15.96:0/661471255 learned_addr learned my addr 172.21.15.96:0/661471255 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6241932700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f623c1017e0 0x7f623c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6242133700 1 -- 172.21.15.96:0/661471255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f623c1017e0 msgr2=0x7f623c073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6242133700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f623c1017e0 0x7f623c073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6242133700 1 -- 172.21.15.96:0/661471255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c0095f0 con 0x7f623c100e30 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f6242133700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c100e30 0x7f623c0731e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f623400ada0 tx=0x7f623400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.377+0000 7f62337fe700 1 -- 172.21.15.96:0/661471255 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f623400f070 con 0x7f623c100e30 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.378+0000 7f62337fe700 1 -- 172.21.15.96:0/661471255 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6234007bf0 con 0x7f623c100e30 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.378+0000 7f62337fe700 1 -- 172.21.15.96:0/661471255 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6234005690 con 0x7f623c100e30 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.378+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623c1a2cb0 con 0x7f623c100e30 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.378+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623c1a3010 con 0x7f623c100e30 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.379+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c109230 con 0x7f623c100e30 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.379+0000 7f62337fe700 1 -- 172.21.15.96:0/661471255 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6234005900 con 0x7f623c100e30 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.380+0000 7f62337fe700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f622806ecd0 0x7f6228071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.380+0000 7f62337fe700 1 -- 172.21.15.96:0/661471255 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f623408e880 con 0x7f623c100e30 2024-04-22T15:51:38.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.380+0000 7f6241932700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f622806ecd0 0x7f6228071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.380+0000 7f6241932700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f622806ecd0 0x7f6228071190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f623c074910 tx=0x7f622c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:38.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.384+0000 7f62337fe700 1 -- 172.21.15.96:0/661471255 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f62340575e0 con 0x7f623c100e30 2024-04-22T15:51:38.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.629+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f623c10ddd0 con 0x7f623c100e30 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.630+0000 7f62337fe700 1 -- 172.21.15.96:0/661471255 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f623408ce60 con 0x7f623c100e30 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:51:38.633 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:51:38.634 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:51:38.635 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:51:38.636 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:51:38.636 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:51:38.636 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:51:38.636 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:51:38.636 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f622806ecd0 msgr2=0x7f6228071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f622806ecd0 0x7f6228071190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f623c074910 tx=0x7f622c01a040 comp rx=0 tx=0).stop 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c100e30 msgr2=0x7f623c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c100e30 0x7f623c0731e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f623400ada0 tx=0x7f623400c330 comp rx=0 tx=0).stop 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 shutdown_connections 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f622806ecd0 0x7f6228071190 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f623c100e30 0x7f623c0731e0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 --2- 172.21.15.96:0/661471255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f623c1017e0 0x7f623c073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 >> 172.21.15.96:0/661471255 conn(0x7f623c0fc6e0 msgr2=0x7f623c0fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 shutdown_connections 2024-04-22T15:51:38.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.632+0000 7f6244397700 1 -- 172.21.15.96:0/661471255 wait complete. 2024-04-22T15:51:38.638 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:51:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.739+0000 7fd114cb6700 1 -- 172.21.15.96:0/1191544184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd110101e90 msgr2=0x7fd110102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.739+0000 7fd114cb6700 1 --2- 172.21.15.96:0/1191544184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd110101e90 0x7fd110102270 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd100009910 tx=0x7fd100009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.739+0000 7fd114cb6700 1 -- 172.21.15.96:0/1191544184 shutdown_connections 2024-04-22T15:51:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.739+0000 7fd114cb6700 1 --2- 172.21.15.96:0/1191544184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1101027b0 0x7fd11010aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.739+0000 7fd114cb6700 1 --2- 172.21.15.96:0/1191544184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd110101e90 0x7fd110102270 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.739+0000 7fd114cb6700 1 -- 172.21.15.96:0/1191544184 >> 172.21.15.96:0/1191544184 conn(0x7fd1100fb680 msgr2=0x7fd1100fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.739+0000 7fd114cb6700 1 -- 172.21.15.96:0/1191544184 shutdown_connections 2024-04-22T15:51:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.739+0000 7fd114cb6700 1 -- 172.21.15.96:0/1191544184 wait complete. 2024-04-22T15:51:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.739+0000 7fd114cb6700 1 Processor -- start 2024-04-22T15:51:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd114cb6700 1 -- start start 2024-04-22T15:51:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd114cb6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd110101e90 0x7fd11019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd114cb6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1101027b0 0x7fd11019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd114cb6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11019d3f0 con 0x7fd1101027b0 2024-04-22T15:51:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd114cb6700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1101969d0 con 0x7fd110101e90 2024-04-22T15:51:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd10dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1101027b0 0x7fd11019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd10dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1101027b0 0x7fd11019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42054/0 (socket says 172.21.15.96:42054) 2024-04-22T15:51:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd10dd9b700 1 -- 172.21.15.96:0/559398141 learned_addr learned my addr 172.21.15.96:0/559398141 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd10e59c700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd110101e90 0x7fd11019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd10dd9b700 1 -- 172.21.15.96:0/559398141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd110101e90 msgr2=0x7fd11019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd10dd9b700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd110101e90 0x7fd11019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd10dd9b700 1 -- 172.21.15.96:0/559398141 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1000095f0 con 0x7fd1101027b0 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd10dd9b700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1101027b0 0x7fd11019cdd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd10400d900 tx=0x7fd10400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd0ff7fe700 1 -- 172.21.15.96:0/559398141 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd104009900 con 0x7fd1101027b0 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd110196cb0 con 0x7fd1101027b0 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd0ff7fe700 1 -- 172.21.15.96:0/559398141 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd104010460 con 0x7fd1101027b0 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.740+0000 7fd0ff7fe700 1 -- 172.21.15.96:0/559398141 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd10400f5f0 con 0x7fd1101027b0 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.741+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd110197200 con 0x7fd1101027b0 2024-04-22T15:51:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.741+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1101055b0 con 0x7fd1101027b0 2024-04-22T15:51:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.742+0000 7fd0ff7fe700 1 -- 172.21.15.96:0/559398141 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd1040105d0 con 0x7fd1101027b0 2024-04-22T15:51:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.742+0000 7fd0ff7fe700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd0f806ecd0 0x7fd0f8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.742+0000 7fd0ff7fe700 1 -- 172.21.15.96:0/559398141 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd10408ea50 con 0x7fd1101027b0 2024-04-22T15:51:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.742+0000 7fd10e59c700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd0f806ecd0 0x7fd0f8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.743+0000 7fd10e59c700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd0f806ecd0 0x7fd0f8071190 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd100006010 tx=0x7fd10001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:38.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.747+0000 7fd0ff7fe700 1 -- 172.21.15.96:0/559398141 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd10405b2e0 con 0x7fd1101027b0 2024-04-22T15:51:38.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.964+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd110197e90 con 0x7fd0f806ecd0 2024-04-22T15:51:38.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:38 smithi096 ceph-mon[112443]: from='client.35628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:38.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:38 smithi096 ceph-mon[112443]: from='client.35632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:38.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:38 smithi096 ceph-mon[112443]: pgmap v894: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:38.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:38 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2114928157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:38.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:38 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/661471255' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:51:38.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.965+0000 7fd0ff7fe700 1 -- 172.21.15.96:0/559398141 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd110197e90 con 0x7fd0f806ecd0 2024-04-22T15:51:38.968 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:51:38.968 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:51:38.968 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:51:38.968 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:51:38.968 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:51:38.968 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:51:38.968 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd0f806ecd0 msgr2=0x7fd0f8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd0f806ecd0 0x7fd0f8071190 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd100006010 tx=0x7fd10001a040 comp rx=0 tx=0).stop 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1101027b0 msgr2=0x7fd11019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1101027b0 0x7fd11019cdd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd10400d900 tx=0x7fd10400dc10 comp rx=0 tx=0).stop 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 shutdown_connections 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd0f806ecd0 0x7fd0f8071190 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd110101e90 0x7fd11019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 --2- 172.21.15.96:0/559398141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1101027b0 0x7fd11019cdd0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 >> 172.21.15.96:0/559398141 conn(0x7fd1100fb680 msgr2=0x7fd110103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 shutdown_connections 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:38.967+0000 7fd114cb6700 1 -- 172.21.15.96:0/559398141 wait complete. 2024-04-22T15:51:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:38 smithi196 ceph-mon[81815]: from='client.35628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:39.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:38 smithi196 ceph-mon[81815]: from='client.35632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:39.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:38 smithi196 ceph-mon[81815]: pgmap v894: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:39.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:38 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2114928157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:39.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:39.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:38 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/661471255' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:51:39.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.071+0000 7f50ae4f8700 1 -- 172.21.15.96:0/4024448564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 msgr2=0x7f50a80fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.071+0000 7f50ae4f8700 1 --2- 172.21.15.96:0/4024448564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 0x7f50a80fff50 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f509c009910 tx=0x7f509c009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:39.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.071+0000 7f50ae4f8700 1 -- 172.21.15.96:0/4024448564 shutdown_connections 2024-04-22T15:51:39.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.071+0000 7f50ae4f8700 1 --2- 172.21.15.96:0/4024448564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 0x7f50a80fff50 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.071+0000 7f50ae4f8700 1 --2- 172.21.15.96:0/4024448564 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50a8105b60 0x7f50a8105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.071+0000 7f50ae4f8700 1 -- 172.21.15.96:0/4024448564 >> 172.21.15.96:0/4024448564 conn(0x7f50a80fb5e0 msgr2=0x7f50a80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:39.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.071+0000 7f50ae4f8700 1 -- 172.21.15.96:0/4024448564 shutdown_connections 2024-04-22T15:51:39.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.071+0000 7f50ae4f8700 1 -- 172.21.15.96:0/4024448564 wait complete. 2024-04-22T15:51:39.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.072+0000 7f50ae4f8700 1 Processor -- start 2024-04-22T15:51:39.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.072+0000 7f50ae4f8700 1 -- start start 2024-04-22T15:51:39.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.072+0000 7f50ae4f8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 0x7f50a8197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.072+0000 7f50ae4f8700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50a8105b60 0x7f50a8197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.072+0000 7f50ae4f8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a81984a0 con 0x7f50a80ffad0 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.072+0000 7f50ae4f8700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a81985e0 con 0x7f50a8105b60 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.072+0000 7f50a7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 0x7f50a8197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.072+0000 7f50a7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 0x7f50a8197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42070/0 (socket says 172.21.15.96:42070) 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50a7fff700 1 -- 172.21.15.96:0/3321943016 learned_addr learned my addr 172.21.15.96:0/3321943016 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50a77fe700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50a8105b60 0x7f50a8197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50a7fff700 1 -- 172.21.15.96:0/3321943016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50a8105b60 msgr2=0x7f50a8197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50a7fff700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50a8105b60 0x7f50a8197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50a7fff700 1 -- 172.21.15.96:0/3321943016 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c0095f0 con 0x7f50a80ffad0 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50a7fff700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 0x7f50a8197990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f509800d8d0 tx=0x7f509800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50a57fa700 1 -- 172.21.15.96:0/3321943016 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50980098a0 con 0x7f50a80ffad0 2024-04-22T15:51:39.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50a57fa700 1 -- 172.21.15.96:0/3321943016 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5098010460 con 0x7f50a80ffad0 2024-04-22T15:51:39.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a81a0a00 con 0x7f50a80ffad0 2024-04-22T15:51:39.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50a57fa700 1 -- 172.21.15.96:0/3321943016 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f509800f5f0 con 0x7f50a80ffad0 2024-04-22T15:51:39.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.073+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a81a0e10 con 0x7f50a80ffad0 2024-04-22T15:51:39.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.074+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a81080f0 con 0x7f50a80ffad0 2024-04-22T15:51:39.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.075+0000 7f50a57fa700 1 -- 172.21.15.96:0/3321943016 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5098009a00 con 0x7f50a80ffad0 2024-04-22T15:51:39.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.075+0000 7f50a57fa700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f509006ecc0 0x7f5090071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:39.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.075+0000 7f50a57fa700 1 -- 172.21.15.96:0/3321943016 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f509808ed80 con 0x7f50a80ffad0 2024-04-22T15:51:39.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.075+0000 7f50a77fe700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f509006ecc0 0x7f5090071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:39.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.076+0000 7f50a77fe700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f509006ecc0 0x7f5090071180 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f50a8069940 tx=0x7f509c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:39.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.080+0000 7f50a57fa700 1 -- 172.21.15.96:0/3321943016 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5098057ae0 con 0x7f50a80ffad0 2024-04-22T15:51:39.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.364+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f50a8069790 con 0x7f50a80ffad0 2024-04-22T15:51:39.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.364+0000 7f50a57fa700 1 -- 172.21.15.96:0/3321943016 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f509805b1a0 con 0x7f50a80ffad0 2024-04-22T15:51:39.367 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:51:39.368 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f509006ecc0 msgr2=0x7f5090071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f509006ecc0 0x7f5090071180 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f50a8069940 tx=0x7f509c01a040 comp rx=0 tx=0).stop 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 msgr2=0x7f50a8197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 0x7f50a8197990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f509800d8d0 tx=0x7f509800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 shutdown_connections 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f509006ecc0 0x7f5090071180 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50a80ffad0 0x7f50a8197990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 --2- 172.21.15.96:0/3321943016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50a8105b60 0x7f50a8197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 >> 172.21.15.96:0/3321943016 conn(0x7f50a80fb5e0 msgr2=0x7f50a80fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 shutdown_connections 2024-04-22T15:51:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:51:39.366+0000 7f50ae4f8700 1 -- 172.21.15.96:0/3321943016 wait complete. 2024-04-22T15:51:39.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:39 smithi096 ceph-mon[112443]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:39.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:39 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3321943016' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:51:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:39 smithi196 ceph-mon[81815]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:39 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3321943016' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:51:40.974 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:40 smithi196 ceph-mon[81815]: from='client.35648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:40.974 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:40 smithi196 ceph-mon[81815]: pgmap v895: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:40 smithi096 ceph-mon[112443]: from='client.35648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:40 smithi096 ceph-mon[112443]: pgmap v895: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:41.967 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:41 smithi096 ceph-mon[112443]: pgmap v896: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:41 smithi196 ceph-mon[81815]: pgmap v896: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:44 smithi096 ceph-mon[112443]: pgmap v897: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:44 smithi196 ceph-mon[81815]: pgmap v897: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:46 smithi096 ceph-mon[112443]: pgmap v898: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:46 smithi196 ceph-mon[81815]: pgmap v898: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:48 smithi096 ceph-mon[112443]: pgmap v899: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:48 smithi196 ceph-mon[81815]: pgmap v899: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:50.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:50 smithi096 ceph-mon[112443]: pgmap v900: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:51.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:50 smithi196 ceph-mon[81815]: pgmap v900: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:52.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:52 smithi096 ceph-mon[112443]: pgmap v901: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:52 smithi196 ceph-mon[81815]: pgmap v901: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:54 smithi096 ceph-mon[112443]: pgmap v902: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:54 smithi196 ceph-mon[81815]: pgmap v902: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:55.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:55 smithi096 ceph-mon[112443]: pgmap v903: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:55 smithi196 ceph-mon[81815]: pgmap v903: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:51:58 smithi096 ceph-mon[112443]: pgmap v904: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:51:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:51:58 smithi196 ceph-mon[81815]: pgmap v904: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:00.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:00 smithi096 ceph-mon[112443]: pgmap v905: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:00 smithi196 ceph-mon[81815]: pgmap v905: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:02.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:02 smithi096 ceph-mon[112443]: pgmap v906: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:03.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:02 smithi196 ceph-mon[81815]: pgmap v906: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:04 smithi096 ceph-mon[112443]: pgmap v907: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:04 smithi196 ceph-mon[81815]: pgmap v907: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:06.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:06 smithi096 ceph-mon[112443]: pgmap v908: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:06.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:06 smithi196 ceph-mon[81815]: pgmap v908: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:08.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:08 smithi096 ceph-mon[112443]: pgmap v909: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:08 smithi196 ceph-mon[81815]: pgmap v909: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:09.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.484+0000 7f22be9f2700 1 -- 172.21.15.96:0/3518766947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8101e00 msgr2=0x7f22b81021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:09.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.484+0000 7f22be9f2700 1 --2- 172.21.15.96:0/3518766947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8101e00 0x7f22b81021e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f22a8009910 tx=0x7f22a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:09.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.484+0000 7f22be9f2700 1 -- 172.21.15.96:0/3518766947 shutdown_connections 2024-04-22T15:52:09.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.484+0000 7f22be9f2700 1 --2- 172.21.15.96:0/3518766947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b8102720 0x7f22b810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:09.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.484+0000 7f22be9f2700 1 --2- 172.21.15.96:0/3518766947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8101e00 0x7f22b81021e0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:09.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.484+0000 7f22be9f2700 1 -- 172.21.15.96:0/3518766947 >> 172.21.15.96:0/3518766947 conn(0x7f22b8076740 msgr2=0x7f22b80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:09.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.484+0000 7f22be9f2700 1 -- 172.21.15.96:0/3518766947 shutdown_connections 2024-04-22T15:52:09.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.484+0000 7f22be9f2700 1 -- 172.21.15.96:0/3518766947 wait complete. 2024-04-22T15:52:09.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22be9f2700 1 Processor -- start 2024-04-22T15:52:09.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22be9f2700 1 -- start start 2024-04-22T15:52:09.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22be9f2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b8101e00 0x7f22b819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:09.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22be9f2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8102720 0x7f22b819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:09.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22be9f2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b819d360 con 0x7f22b8102720 2024-04-22T15:52:09.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22be9f2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8196940 con 0x7f22b8101e00 2024-04-22T15:52:09.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22b77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8102720 0x7f22b819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:09.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22b77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8102720 0x7f22b819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39680/0 (socket says 172.21.15.96:39680) 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22b77fe700 1 -- 172.21.15.96:0/3543143028 learned_addr learned my addr 172.21.15.96:0/3543143028 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22b7fff700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b8101e00 0x7f22b819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22b77fe700 1 -- 172.21.15.96:0/3543143028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b8101e00 msgr2=0x7f22b819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.485+0000 7f22b77fe700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b8101e00 0x7f22b819c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.486+0000 7f22b77fe700 1 -- 172.21.15.96:0/3543143028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80095f0 con 0x7f22b8102720 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.486+0000 7f22b77fe700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8102720 0x7f22b819cd40 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f22ac00d900 tx=0x7f22ac00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.486+0000 7f22b57fa700 1 -- 172.21.15.96:0/3543143028 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22ac009900 con 0x7f22b8102720 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.486+0000 7f22b57fa700 1 -- 172.21.15.96:0/3543143028 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22ac010460 con 0x7f22b8102720 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.486+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b8196c20 con 0x7f22b8102720 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.486+0000 7f22b57fa700 1 -- 172.21.15.96:0/3543143028 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22ac00f5f0 con 0x7f22b8102720 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.486+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b8197170 con 0x7f22b8102720 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.487+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b8105520 con 0x7f22b8102720 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.487+0000 7f22b57fa700 1 -- 172.21.15.96:0/3543143028 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f22ac0105d0 con 0x7f22b8102720 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.488+0000 7f22b57fa700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22a006ecc0 0x7f22a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:09.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.488+0000 7f22b7fff700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22a006ecc0 0x7f22a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:09.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.488+0000 7f22b57fa700 1 -- 172.21.15.96:0/3543143028 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f22ac08ea80 con 0x7f22b8102720 2024-04-22T15:52:09.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.488+0000 7f22b7fff700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22a006ecc0 0x7f22a0071180 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f22a8000c00 tx=0x7f22a801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:09.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.493+0000 7f22b57fa700 1 -- 172.21.15.96:0/3543143028 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f22ac05b310 con 0x7f22b8102720 2024-04-22T15:52:09.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.712+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f22b8197e70 con 0x7f22a006ecc0 2024-04-22T15:52:09.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.713+0000 7f22b57fa700 1 -- 172.21.15.96:0/3543143028 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f22b8197e70 con 0x7f22a006ecc0 2024-04-22T15:52:09.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.715+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22a006ecc0 msgr2=0x7f22a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:09.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.715+0000 7f22be9f2700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22a006ecc0 0x7f22a0071180 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f22a8000c00 tx=0x7f22a801a040 comp rx=0 tx=0).stop 2024-04-22T15:52:09.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.715+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8102720 msgr2=0x7f22b819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:09.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.715+0000 7f22be9f2700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8102720 0x7f22b819cd40 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f22ac00d900 tx=0x7f22ac00dc10 comp rx=0 tx=0).stop 2024-04-22T15:52:09.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.716+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 shutdown_connections 2024-04-22T15:52:09.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.716+0000 7f22be9f2700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22a006ecc0 0x7f22a0071180 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:09.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.716+0000 7f22be9f2700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b8101e00 0x7f22b819c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:09.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.716+0000 7f22be9f2700 1 --2- 172.21.15.96:0/3543143028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22b8102720 0x7f22b819cd40 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:09.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.716+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 >> 172.21.15.96:0/3543143028 conn(0x7f22b8076740 msgr2=0x7f22b81030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:09.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.716+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 shutdown_connections 2024-04-22T15:52:09.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.716+0000 7f22be9f2700 1 -- 172.21.15.96:0/3543143028 wait complete. 2024-04-22T15:52:09.729 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:52:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.820+0000 7fe40db2d700 1 -- 172.21.15.96:0/3590054281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4080ffab0 msgr2=0x7fe4080fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.820+0000 7fe40db2d700 1 --2- 172.21.15.96:0/3590054281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4080ffab0 0x7fe4080fff30 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009910 tx=0x7fe3fc009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.820+0000 7fe40db2d700 1 -- 172.21.15.96:0/3590054281 shutdown_connections 2024-04-22T15:52:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.820+0000 7fe40db2d700 1 --2- 172.21.15.96:0/3590054281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4080ffab0 0x7fe4080fff30 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.820+0000 7fe40db2d700 1 --2- 172.21.15.96:0/3590054281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe408105ab0 0x7fe408105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.820+0000 7fe40db2d700 1 -- 172.21.15.96:0/3590054281 >> 172.21.15.96:0/3590054281 conn(0x7fe4080fb5e0 msgr2=0x7fe4080fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.820+0000 7fe40db2d700 1 -- 172.21.15.96:0/3590054281 shutdown_connections 2024-04-22T15:52:09.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.820+0000 7fe40db2d700 1 -- 172.21.15.96:0/3590054281 wait complete. 2024-04-22T15:52:09.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.820+0000 7fe40db2d700 1 Processor -- start 2024-04-22T15:52:09.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe40db2d700 1 -- start start 2024-04-22T15:52:09.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe40db2d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4080ffab0 0x7fe40819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:09.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe40db2d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe408105ab0 0x7fe40819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:09.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe40db2d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40819d440 con 0x7fe408105ab0 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe40db2d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408196a20 con 0x7fe4080ffab0 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe406ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe408105ab0 0x7fe40819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe406ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe408105ab0 0x7fe40819ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39694/0 (socket says 172.21.15.96:39694) 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe406ffd700 1 -- 172.21.15.96:0/3538668174 learned_addr learned my addr 172.21.15.96:0/3538668174 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe4077fe700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4080ffab0 0x7fe40819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe406ffd700 1 -- 172.21.15.96:0/3538668174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4080ffab0 msgr2=0x7fe40819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe406ffd700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4080ffab0 0x7fe40819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe406ffd700 1 -- 172.21.15.96:0/3538668174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc0095f0 con 0x7fe408105ab0 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.821+0000 7fe4077fe700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4080ffab0 0x7fe40819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.822+0000 7fe406ffd700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe408105ab0 0x7fe40819ce20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc006010 tx=0x7fe3fc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.822+0000 7fe404ff9700 1 -- 172.21.15.96:0/3538668174 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3fc01d070 con 0x7fe408105ab0 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.822+0000 7fe404ff9700 1 -- 172.21.15.96:0/3538668174 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3fc00f460 con 0x7fe408105ab0 2024-04-22T15:52:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.822+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe408196ca0 con 0x7fe408105ab0 2024-04-22T15:52:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.822+0000 7fe404ff9700 1 -- 172.21.15.96:0/3538668174 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3fc017630 con 0x7fe408105ab0 2024-04-22T15:52:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.822+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe408197190 con 0x7fe408105ab0 2024-04-22T15:52:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.823+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4081090e0 con 0x7fe408105ab0 2024-04-22T15:52:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.823+0000 7fe404ff9700 1 -- 172.21.15.96:0/3538668174 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe3fc00f5d0 con 0x7fe408105ab0 2024-04-22T15:52:09.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.824+0000 7fe404ff9700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe3f006ecd0 0x7fe3f0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:09.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.824+0000 7fe404ff9700 1 -- 172.21.15.96:0/3538668174 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe3fc0903a0 con 0x7fe408105ab0 2024-04-22T15:52:09.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.824+0000 7fe4077fe700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe3f006ecd0 0x7fe3f0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:09.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.824+0000 7fe4077fe700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe3f006ecd0 0x7fe3f0071190 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8007900 tx=0x7fe3f8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:09.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:09.829+0000 7fe404ff9700 1 -- 172.21.15.96:0/3538668174 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe3fc05ccf0 con 0x7fe408105ab0 2024-04-22T15:52:10.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.050+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe408197f50 con 0x7fe3f006ecd0 2024-04-22T15:52:10.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.052+0000 7fe404ff9700 1 -- 172.21.15.96:0/3538668174 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe408197f50 con 0x7fe3f006ecd0 2024-04-22T15:52:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe3f006ecd0 msgr2=0x7fe3f0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe3f006ecd0 0x7fe3f0071190 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8007900 tx=0x7fe3f8008040 comp rx=0 tx=0).stop 2024-04-22T15:52:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe408105ab0 msgr2=0x7fe40819ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe408105ab0 0x7fe40819ce20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc006010 tx=0x7fe3fc00be20 comp rx=0 tx=0).stop 2024-04-22T15:52:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 shutdown_connections 2024-04-22T15:52:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe3f006ecd0 0x7fe3f0071190 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4080ffab0 0x7fe40819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 --2- 172.21.15.96:0/3538668174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe408105ab0 0x7fe40819ce20 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 >> 172.21.15.96:0/3538668174 conn(0x7fe4080fb5e0 msgr2=0x7fe408104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:10.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 shutdown_connections 2024-04-22T15:52:10.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.054+0000 7fe40db2d700 1 -- 172.21.15.96:0/3538668174 wait complete. 2024-04-22T15:52:10.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.157+0000 7f66af59e700 1 -- 172.21.15.96:0/1845001305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 msgr2=0x7f66b00fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.157+0000 7f66af59e700 1 --2- 172.21.15.96:0/1845001305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 0x7f66b00fff00 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009910 tx=0x7f66a0009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:10.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.157+0000 7f66af59e700 1 -- 172.21.15.96:0/1845001305 shutdown_connections 2024-04-22T15:52:10.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.157+0000 7f66af59e700 1 --2- 172.21.15.96:0/1845001305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 0x7f66b00fff00 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.157+0000 7f66af59e700 1 --2- 172.21.15.96:0/1845001305 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b0105900 0x7f66b0105ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.157+0000 7f66af59e700 1 -- 172.21.15.96:0/1845001305 >> 172.21.15.96:0/1845001305 conn(0x7f66b0075460 msgr2=0x7f66b0075870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:10.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.157+0000 7f66af59e700 1 -- 172.21.15.96:0/1845001305 shutdown_connections 2024-04-22T15:52:10.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.157+0000 7f66af59e700 1 -- 172.21.15.96:0/1845001305 wait complete. 2024-04-22T15:52:10.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66af59e700 1 Processor -- start 2024-04-22T15:52:10.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66af59e700 1 -- start start 2024-04-22T15:52:10.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66af59e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 0x7f66b01014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66af59e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b0105900 0x7f66b0101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66af59e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b0101f70 con 0x7f66b00ffa80 2024-04-22T15:52:10.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66af59e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b01020b0 con 0x7f66b0105900 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 0x7f66b01014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66ae59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 0x7f66b01014f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39706/0 (socket says 172.21.15.96:39706) 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66ae59c700 1 -- 172.21.15.96:0/2583687453 learned_addr learned my addr 172.21.15.96:0/2583687453 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.158+0000 7f66add9b700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b0105900 0x7f66b0101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.159+0000 7f66ae59c700 1 -- 172.21.15.96:0/2583687453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b0105900 msgr2=0x7f66b0101a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.159+0000 7f66ae59c700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b0105900 0x7f66b0101a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.159+0000 7f66ae59c700 1 -- 172.21.15.96:0/2583687453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00095f0 con 0x7f66b00ffa80 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.159+0000 7f66ae59c700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 0x7f66b01014f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f66a400c8a0 tx=0x7f66a400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.159+0000 7f669f7fe700 1 -- 172.21.15.96:0/2583687453 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a400cea0 con 0x7f66b00ffa80 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.159+0000 7f669f7fe700 1 -- 172.21.15.96:0/2583687453 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66a4004830 con 0x7f66b00ffa80 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.159+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b01a2b00 con 0x7f66b00ffa80 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.159+0000 7f669f7fe700 1 -- 172.21.15.96:0/2583687453 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a40055b0 con 0x7f66b00ffa80 2024-04-22T15:52:10.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.159+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b01a2e60 con 0x7f66b00ffa80 2024-04-22T15:52:10.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.160+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b0108f30 con 0x7f66b00ffa80 2024-04-22T15:52:10.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.161+0000 7f669f7fe700 1 -- 172.21.15.96:0/2583687453 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f66a40049a0 con 0x7f66b00ffa80 2024-04-22T15:52:10.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.161+0000 7f669f7fe700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f669806ecc0 0x7f6698071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.161+0000 7f669f7fe700 1 -- 172.21.15.96:0/2583687453 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f66a408eba0 con 0x7f66b00ffa80 2024-04-22T15:52:10.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.161+0000 7f66add9b700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f669806ecc0 0x7f6698071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.161+0000 7f66add9b700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f669806ecc0 0x7f6698071180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f66a00059d0 tx=0x7f66a001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:10.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.166+0000 7f669f7fe700 1 -- 172.21.15.96:0/2583687453 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f66a4057900 con 0x7f66b00ffa80 2024-04-22T15:52:10.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.374+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f66b010cb00 con 0x7f669806ecc0 2024-04-22T15:52:10.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.384+0000 7f669f7fe700 1 -- 172.21.15.96:0/2583687453 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f66b010cb00 con 0x7f669806ecc0 2024-04-22T15:52:10.387 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:52:10.387 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (30m) 8m ago 42m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (42m) 8m ago 42m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (41m) 8m ago 41m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (28m) 8m ago 42m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (27m) 8m ago 41m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (30m) 8m ago 41m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (24m) 8m ago 36m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (24m) 8m ago 36m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (23m) 8m ago 36m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (8m) 8m ago 36m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (33m) 8m ago 43m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (32m) 8m ago 41m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (29m) 8m ago 43m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (28m) 8m ago 40m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:52:10.388 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (31m) 8m ago 42m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:52:10.389 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (31m) 8m ago 41m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:52:10.389 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (27m) 8m ago 40m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:52:10.389 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (27m) 8m ago 39m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:52:10.389 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (26m) 8m ago 39m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:52:10.389 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (25m) 8m ago 38m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:52:10.389 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (25m) 8m ago 38m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:52:10.389 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (24m) 8m ago 38m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:52:10.389 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (31m) 8m ago 41m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:52:10.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.386+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f669806ecc0 msgr2=0x7f6698071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.386+0000 7f66af59e700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f669806ecc0 0x7f6698071180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f66a00059d0 tx=0x7f66a001a040 comp rx=0 tx=0).stop 2024-04-22T15:52:10.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.386+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 msgr2=0x7f66b01014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.386+0000 7f66af59e700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 0x7f66b01014f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f66a400c8a0 tx=0x7f66a400cc60 comp rx=0 tx=0).stop 2024-04-22T15:52:10.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.386+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 shutdown_connections 2024-04-22T15:52:10.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.386+0000 7f66af59e700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f669806ecc0 0x7f6698071180 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.386+0000 7f66af59e700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66b00ffa80 0x7f66b01014f0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.386+0000 7f66af59e700 1 --2- 172.21.15.96:0/2583687453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b0105900 0x7f66b0101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.387+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 >> 172.21.15.96:0/2583687453 conn(0x7f66b0075460 msgr2=0x7f66b0068ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:10.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.387+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 shutdown_connections 2024-04-22T15:52:10.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.387+0000 7f66af59e700 1 -- 172.21.15.96:0/2583687453 wait complete. 2024-04-22T15:52:10.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 -- 172.21.15.96:0/797070954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 msgr2=0x7f121c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 --2- 172.21.15.96:0/797070954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 0x7f121c1021c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1218009940 tx=0x7f1218009c50 comp rx=0 tx=0).stop 2024-04-22T15:52:10.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 -- 172.21.15.96:0/797070954 shutdown_connections 2024-04-22T15:52:10.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 --2- 172.21.15.96:0/797070954 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c102700 0x7f121c10abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 --2- 172.21.15.96:0/797070954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 0x7f121c1021c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 -- 172.21.15.96:0/797070954 >> 172.21.15.96:0/797070954 conn(0x7f121c076740 msgr2=0x7f121c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:10.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 -- 172.21.15.96:0/797070954 shutdown_connections 2024-04-22T15:52:10.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 -- 172.21.15.96:0/797070954 wait complete. 2024-04-22T15:52:10.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 Processor -- start 2024-04-22T15:52:10.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 -- start start 2024-04-22T15:52:10.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 0x7f121c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.490+0000 7f122453f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c102700 0x7f121c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f122453f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c19d340 con 0x7f121c101de0 2024-04-22T15:52:10.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f122453f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c196920 con 0x7f121c102700 2024-04-22T15:52:10.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12222db700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 0x7f121c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12222db700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 0x7f121c19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34488/0 (socket says 172.21.15.96:34488) 2024-04-22T15:52:10.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12222db700 1 -- 172.21.15.96:0/1070009207 learned_addr learned my addr 172.21.15.96:0/1070009207 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:10.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f1221ada700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c102700 0x7f121c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12222db700 1 -- 172.21.15.96:0/1070009207 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c102700 msgr2=0x7f121c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12222db700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c102700 0x7f121c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12222db700 1 -- 172.21.15.96:0/1070009207 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12180095f0 con 0x7f121c101de0 2024-04-22T15:52:10.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12222db700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 0x7f121c19c7e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f121800ba10 tx=0x7f121800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:10.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12137fe700 1 -- 172.21.15.96:0/1070009207 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f121801d070 con 0x7f121c101de0 2024-04-22T15:52:10.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121c196ba0 con 0x7f121c101de0 2024-04-22T15:52:10.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12137fe700 1 -- 172.21.15.96:0/1070009207 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1218003d10 con 0x7f121c101de0 2024-04-22T15:52:10.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f12137fe700 1 -- 172.21.15.96:0/1070009207 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f121800fdf0 con 0x7f121c101de0 2024-04-22T15:52:10.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.491+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121c197090 con 0x7f121c101de0 2024-04-22T15:52:10.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.492+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121c105500 con 0x7f121c101de0 2024-04-22T15:52:10.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.493+0000 7f12137fe700 1 -- 172.21.15.96:0/1070009207 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f121800f5f0 con 0x7f121c101de0 2024-04-22T15:52:10.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.493+0000 7f12137fe700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f120806ecc0 0x7f1208071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.493+0000 7f1221ada700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f120806ecc0 0x7f1208071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.493+0000 7f12137fe700 1 -- 172.21.15.96:0/1070009207 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f121808ff80 con 0x7f121c101de0 2024-04-22T15:52:10.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.494+0000 7f1221ada700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f120806ecc0 0x7f1208071180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f121c197ae0 tx=0x7f120c009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:10.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.498+0000 7f12137fe700 1 -- 172.21.15.96:0/1070009207 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f121805c810 con 0x7f121c101de0 2024-04-22T15:52:10.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.786+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f121c197de0 con 0x7f121c101de0 2024-04-22T15:52:10.790 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:10 smithi096 ceph-mon[112443]: pgmap v910: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:10.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.787+0000 7f12137fe700 1 -- 172.21.15.96:0/1070009207 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f121805c3a0 con 0x7f121c101de0 2024-04-22T15:52:10.790 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:52:10.790 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:52:10.790 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:52:10.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:52:10.791 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:52:10.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f120806ecc0 msgr2=0x7f1208071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f120806ecc0 0x7f1208071180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f121c197ae0 tx=0x7f120c009260 comp rx=0 tx=0).stop 2024-04-22T15:52:10.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 msgr2=0x7f121c19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 0x7f121c19c7e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f121800ba10 tx=0x7f121800ba40 comp rx=0 tx=0).stop 2024-04-22T15:52:10.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 shutdown_connections 2024-04-22T15:52:10.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f120806ecc0 0x7f1208071180 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121c101de0 0x7f121c19c7e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 --2- 172.21.15.96:0/1070009207 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c102700 0x7f121c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 >> 172.21.15.96:0/1070009207 conn(0x7f121c076740 msgr2=0x7f121c103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:10.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 shutdown_connections 2024-04-22T15:52:10.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.789+0000 7f122453f700 1 -- 172.21.15.96:0/1070009207 wait complete. 2024-04-22T15:52:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.892+0000 7f084c60f700 1 -- 172.21.15.96:0/3004347408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844101d60 msgr2=0x7f0844102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.892+0000 7f084c60f700 1 --2- 172.21.15.96:0/3004347408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844101d60 0x7f0844102140 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f083c00b0a0 tx=0x7f083c00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.892+0000 7f084c60f700 1 -- 172.21.15.96:0/3004347408 shutdown_connections 2024-04-22T15:52:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.892+0000 7f084c60f700 1 --2- 172.21.15.96:0/3004347408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0844102680 0x7f084410ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.892+0000 7f084c60f700 1 --2- 172.21.15.96:0/3004347408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844101d60 0x7f0844102140 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.892+0000 7f084c60f700 1 -- 172.21.15.96:0/3004347408 >> 172.21.15.96:0/3004347408 conn(0x7f0844076740 msgr2=0x7f08440fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.892+0000 7f084c60f700 1 -- 172.21.15.96:0/3004347408 shutdown_connections 2024-04-22T15:52:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.892+0000 7f084c60f700 1 -- 172.21.15.96:0/3004347408 wait complete. 2024-04-22T15:52:10.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f084c60f700 1 Processor -- start 2024-04-22T15:52:10.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f084c60f700 1 -- start start 2024-04-22T15:52:10.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f084c60f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0844101d60 0x7f084419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f084c60f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844102680 0x7f084419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f084c60f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084419d4c0 con 0x7f0844102680 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f084c60f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0844196aa0 con 0x7f0844101d60 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f0849baa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844102680 0x7f084419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f0849baa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844102680 0x7f084419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34510/0 (socket says 172.21.15.96:34510) 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f0849baa700 1 -- 172.21.15.96:0/3546783416 learned_addr learned my addr 172.21.15.96:0/3546783416 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f084a3ab700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0844101d60 0x7f084419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f0849baa700 1 -- 172.21.15.96:0/3546783416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0844101d60 msgr2=0x7f084419c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f0849baa700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0844101d60 0x7f084419c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f0849baa700 1 -- 172.21.15.96:0/3546783416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c009d00 con 0x7f0844102680 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.893+0000 7f0849baa700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844102680 0x7f084419cea0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f083400b3d0 tx=0x7f083400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.894+0000 7f083b7fe700 1 -- 172.21.15.96:0/3546783416 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0834011800 con 0x7f0844102680 2024-04-22T15:52:10.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.894+0000 7f083b7fe700 1 -- 172.21.15.96:0/3546783416 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0834011e20 con 0x7f0844102680 2024-04-22T15:52:10.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.894+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0844196d80 con 0x7f0844102680 2024-04-22T15:52:10.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.894+0000 7f083b7fe700 1 -- 172.21.15.96:0/3546783416 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0834010ea0 con 0x7f0844102680 2024-04-22T15:52:10.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.894+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08441972d0 con 0x7f0844102680 2024-04-22T15:52:10.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.895+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0844105550 con 0x7f0844102680 2024-04-22T15:52:10.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.895+0000 7f083b7fe700 1 -- 172.21.15.96:0/3546783416 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0834010380 con 0x7f0844102680 2024-04-22T15:52:10.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.895+0000 7f083b7fe700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f083006ecd0 0x7f0830071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.896+0000 7f083b7fe700 1 -- 172.21.15.96:0/3546783416 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f083408e8e0 con 0x7f0844102680 2024-04-22T15:52:10.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.896+0000 7f084a3ab700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f083006ecd0 0x7f0830071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.896+0000 7f084a3ab700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f083006ecd0 0x7f0830071190 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f083c009850 tx=0x7f083c0097c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:10.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:10.900+0000 7f083b7fe700 1 -- 172.21.15.96:0/3546783416 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f083405b1f0 con 0x7f0844102680 2024-04-22T15:52:11.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:10 smithi196 ceph-mon[81815]: pgmap v910: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:11.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.144+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0844068ea0 con 0x7f0844102680 2024-04-22T15:52:11.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.144+0000 7f083b7fe700 1 -- 172.21.15.96:0/3546783416 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f083405ad80 con 0x7f0844102680 2024-04-22T15:52:11.147 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:52:11.147 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:52:11.147 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:52:11.147 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:52:11.147 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:52:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:52:11.149 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:52:11.150 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:52:11.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f083006ecd0 msgr2=0x7f0830071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f083006ecd0 0x7f0830071190 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f083c009850 tx=0x7f083c0097c0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844102680 msgr2=0x7f084419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844102680 0x7f084419cea0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f083400b3d0 tx=0x7f083400b790 comp rx=0 tx=0).stop 2024-04-22T15:52:11.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 shutdown_connections 2024-04-22T15:52:11.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f083006ecd0 0x7f0830071190 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0844101d60 0x7f084419c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 --2- 172.21.15.96:0/3546783416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0844102680 0x7f084419cea0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 >> 172.21.15.96:0/3546783416 conn(0x7f0844076740 msgr2=0x7f0844103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 shutdown_connections 2024-04-22T15:52:11.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.146+0000 7f084c60f700 1 -- 172.21.15.96:0/3546783416 wait complete. 2024-04-22T15:52:11.152 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:52:11.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.250+0000 7fce94c75700 1 -- 172.21.15.96:0/3360049517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce900ff440 msgr2=0x7fce900ff8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.250+0000 7fce94c75700 1 --2- 172.21.15.96:0/3360049517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce900ff440 0x7fce900ff8c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fce84009910 tx=0x7fce84009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:11.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.250+0000 7fce94c75700 1 -- 172.21.15.96:0/3360049517 shutdown_connections 2024-04-22T15:52:11.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.250+0000 7fce94c75700 1 --2- 172.21.15.96:0/3360049517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce900ff440 0x7fce900ff8c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.250+0000 7fce94c75700 1 --2- 172.21.15.96:0/3360049517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce90105cb0 0x7fce90106090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.250+0000 7fce94c75700 1 -- 172.21.15.96:0/3360049517 >> 172.21.15.96:0/3360049517 conn(0x7fce90074d50 msgr2=0x7fce90075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.250+0000 7fce94c75700 1 -- 172.21.15.96:0/3360049517 shutdown_connections 2024-04-22T15:52:11.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.250+0000 7fce94c75700 1 -- 172.21.15.96:0/3360049517 wait complete. 2024-04-22T15:52:11.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce94c75700 1 Processor -- start 2024-04-22T15:52:11.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce94c75700 1 -- start start 2024-04-22T15:52:11.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce94c75700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce900ff440 0x7fce9019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce94c75700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce90105cb0 0x7fce9019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce94c75700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9019d350 con 0x7fce90105cb0 2024-04-22T15:52:11.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce94c75700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce90196930 con 0x7fce900ff440 2024-04-22T15:52:11.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce8dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce90105cb0 0x7fce9019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce8dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce90105cb0 0x7fce9019cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34528/0 (socket says 172.21.15.96:34528) 2024-04-22T15:52:11.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce8dd9b700 1 -- 172.21.15.96:0/1325501668 learned_addr learned my addr 172.21.15.96:0/1325501668 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce8e59c700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce900ff440 0x7fce9019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce8dd9b700 1 -- 172.21.15.96:0/1325501668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce900ff440 msgr2=0x7fce9019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce8dd9b700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce900ff440 0x7fce9019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce8dd9b700 1 -- 172.21.15.96:0/1325501668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840095f0 con 0x7fce90105cb0 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.251+0000 7fce8dd9b700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce90105cb0 0x7fce9019cd30 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fce8400bd70 tx=0x7fce8400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.252+0000 7fce7f7fe700 1 -- 172.21.15.96:0/1325501668 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8401d070 con 0x7fce90105cb0 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.252+0000 7fce7f7fe700 1 -- 172.21.15.96:0/1325501668 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce840056e0 con 0x7fce90105cb0 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.252+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce90196bb0 con 0x7fce90105cb0 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.252+0000 7fce7f7fe700 1 -- 172.21.15.96:0/1325501668 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8400fe40 con 0x7fce90105cb0 2024-04-22T15:52:11.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.252+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce901970a0 con 0x7fce90105cb0 2024-04-22T15:52:11.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.253+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce90109290 con 0x7fce90105cb0 2024-04-22T15:52:11.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.253+0000 7fce7f7fe700 1 -- 172.21.15.96:0/1325501668 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce8400f680 con 0x7fce90105cb0 2024-04-22T15:52:11.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.254+0000 7fce7f7fe700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce7806ec60 0x7fce78071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.254+0000 7fce8e59c700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce7806ec60 0x7fce78071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.254+0000 7fce7f7fe700 1 -- 172.21.15.96:0/1325501668 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fce840902b0 con 0x7fce90105cb0 2024-04-22T15:52:11.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.254+0000 7fce8e59c700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce7806ec60 0x7fce78071120 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fce80005950 tx=0x7fce8000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.258+0000 7fce7f7fe700 1 -- 172.21.15.96:0/1325501668 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fce8405cb40 con 0x7fce90105cb0 2024-04-22T15:52:11.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.475+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce90197e60 con 0x7fce7806ec60 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.477+0000 7fce7f7fe700 1 -- 172.21.15.96:0/1325501668 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fce90197e60 con 0x7fce7806ec60 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:52:11.481 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:52:11.481 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:52:11.481 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:52:11.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce7806ec60 msgr2=0x7fce78071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce7806ec60 0x7fce78071120 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fce80005950 tx=0x7fce8000b410 comp rx=0 tx=0).stop 2024-04-22T15:52:11.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce90105cb0 msgr2=0x7fce9019cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce90105cb0 0x7fce9019cd30 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fce8400bd70 tx=0x7fce8400bda0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 shutdown_connections 2024-04-22T15:52:11.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce7806ec60 0x7fce78071120 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce900ff440 0x7fce9019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 --2- 172.21.15.96:0/1325501668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce90105cb0 0x7fce9019cd30 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 >> 172.21.15.96:0/1325501668 conn(0x7fce90074d50 msgr2=0x7fce900fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 shutdown_connections 2024-04-22T15:52:11.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.479+0000 7fce94c75700 1 -- 172.21.15.96:0/1325501668 wait complete. 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.581+0000 7fc009e96700 1 -- 172.21.15.96:0/25748266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0041017a0 msgr2=0x7fc0041057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.581+0000 7fc009e96700 1 --2- 172.21.15.96:0/25748266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0041017a0 0x7fc0041057f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009910 tx=0x7fbfec009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.581+0000 7fc009e96700 1 -- 172.21.15.96:0/25748266 shutdown_connections 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.581+0000 7fc009e96700 1 --2- 172.21.15.96:0/25748266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0041017a0 0x7fc0041057f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.581+0000 7fc009e96700 1 --2- 172.21.15.96:0/25748266 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc004100df0 0x7fc0041011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.581+0000 7fc009e96700 1 -- 172.21.15.96:0/25748266 >> 172.21.15.96:0/25748266 conn(0x7fc0040fc6a0 msgr2=0x7fc0040feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.581+0000 7fc009e96700 1 -- 172.21.15.96:0/25748266 shutdown_connections 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.581+0000 7fc009e96700 1 -- 172.21.15.96:0/25748266 wait complete. 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.582+0000 7fc009e96700 1 Processor -- start 2024-04-22T15:52:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.582+0000 7fc009e96700 1 -- start start 2024-04-22T15:52:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.582+0000 7fc009e96700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc004100df0 0x7fc0040731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.582+0000 7fc009e96700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0041017a0 0x7fc004073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.582+0000 7fc009e96700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc004073c60 con 0x7fc004100df0 2024-04-22T15:52:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.582+0000 7fc009e96700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc004073da0 con 0x7fc0041017a0 2024-04-22T15:52:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc0037fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc004100df0 0x7fc0040731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc0037fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc004100df0 0x7fc0040731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34540/0 (socket says 172.21.15.96:34540) 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc0037fe700 1 -- 172.21.15.96:0/104940677 learned_addr learned my addr 172.21.15.96:0/104940677 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc002ffd700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0041017a0 0x7fc004073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc0037fe700 1 -- 172.21.15.96:0/104940677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0041017a0 msgr2=0x7fc004073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc0037fe700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0041017a0 0x7fc004073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc0037fe700 1 -- 172.21.15.96:0/104940677 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec0095f0 con 0x7fc004100df0 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc0037fe700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc004100df0 0x7fc0040731e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00b4b0 tx=0x7fbffc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc000ff9700 1 -- 172.21.15.96:0/104940677 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbffc011800 con 0x7fc004100df0 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc000ff9700 1 -- 172.21.15.96:0/104940677 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbffc011e20 con 0x7fc004100df0 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0041a2db0 con 0x7fc004100df0 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc000ff9700 1 -- 172.21.15.96:0/104940677 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbffc00f2e0 con 0x7fc004100df0 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.583+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0041a3110 con 0x7fc004100df0 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.584+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc004109120 con 0x7fc004100df0 2024-04-22T15:52:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.585+0000 7fc000ff9700 1 -- 172.21.15.96:0/104940677 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbffc010380 con 0x7fc004100df0 2024-04-22T15:52:11.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.585+0000 7fc000ff9700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbff006ecc0 0x7fbff0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.585+0000 7fc000ff9700 1 -- 172.21.15.96:0/104940677 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbffc08ea90 con 0x7fc004100df0 2024-04-22T15:52:11.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.585+0000 7fc002ffd700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbff006ecc0 0x7fbff0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.586+0000 7fc002ffd700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbff006ecc0 0x7fbff0071180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc004074910 tx=0x7fbfec01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.590+0000 7fc000ff9700 1 -- 172.21.15.96:0/104940677 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbffc0577f0 con 0x7fc004100df0 2024-04-22T15:52:11.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:11 smithi096 ceph-mon[112443]: from='client.35656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:11 smithi096 ceph-mon[112443]: from='client.35660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:11 smithi096 ceph-mon[112443]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:11 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1070009207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:11.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:11 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3546783416' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:52:11.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.874+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc0041022d0 con 0x7fc004100df0 2024-04-22T15:52:11.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.875+0000 7fc000ff9700 1 -- 172.21.15.96:0/104940677 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fbffc05aeb0 con 0x7fc004100df0 2024-04-22T15:52:11.878 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:52:11.878 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:52:11.878 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:52:11.878 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:52:11.878 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:52:11.878 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:52:11.878 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:52:11.878 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:52:11.879 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:52:11.880 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:52:11.880 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:52:11.880 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:52:11.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbff006ecc0 msgr2=0x7fbff0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbff006ecc0 0x7fbff0071180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc004074910 tx=0x7fbfec01a040 comp rx=0 tx=0).stop 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc004100df0 msgr2=0x7fc0040731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc004100df0 0x7fc0040731e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00b4b0 tx=0x7fbffc00b870 comp rx=0 tx=0).stop 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 shutdown_connections 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbff006ecc0 0x7fbff0071180 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc004100df0 0x7fc0040731e0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 --2- 172.21.15.96:0/104940677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0041017a0 0x7fc004073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 >> 172.21.15.96:0/104940677 conn(0x7fc0040fc6a0 msgr2=0x7fc0040fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 shutdown_connections 2024-04-22T15:52:11.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:11.877+0000 7fc009e96700 1 -- 172.21.15.96:0/104940677 wait complete. 2024-04-22T15:52:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:11 smithi196 ceph-mon[81815]: from='client.35656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:11 smithi196 ceph-mon[81815]: from='client.35660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:11 smithi196 ceph-mon[81815]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:11 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1070009207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:11 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3546783416' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:52:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:12 smithi096 ceph-mon[112443]: from='client.35676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:12 smithi096 ceph-mon[112443]: pgmap v911: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:12.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:12 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/104940677' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:52:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:12 smithi196 ceph-mon[81815]: from='client.35676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:12 smithi196 ceph-mon[81815]: pgmap v911: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:12 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/104940677' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:52:13.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:13 smithi096 ceph-mon[112443]: pgmap v912: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:14.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:13 smithi196 ceph-mon[81815]: pgmap v912: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:16.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:16 smithi096 ceph-mon[112443]: pgmap v913: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:16 smithi196 ceph-mon[81815]: pgmap v913: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:18.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:18 smithi096 ceph-mon[112443]: pgmap v914: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:18 smithi196 ceph-mon[81815]: pgmap v914: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:20.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:20 smithi096 ceph-mon[112443]: pgmap v915: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:20 smithi196 ceph-mon[81815]: pgmap v915: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:21.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:21 smithi096 ceph-mon[112443]: pgmap v916: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:21 smithi196 ceph-mon[81815]: pgmap v916: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:23.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:24.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:24 smithi096 ceph-mon[112443]: pgmap v917: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:24 smithi196 ceph-mon[81815]: pgmap v917: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:26.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:26 smithi096 ceph-mon[112443]: pgmap v918: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:26.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:26 smithi196 ceph-mon[81815]: pgmap v918: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:28.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:28 smithi096 ceph-mon[112443]: pgmap v919: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:28 smithi196 ceph-mon[81815]: pgmap v919: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:30.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:30 smithi096 ceph-mon[112443]: pgmap v920: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:30 smithi196 ceph-mon[81815]: pgmap v920: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:32.788 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:32 smithi096 ceph-mon[112443]: pgmap v921: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:33.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:32 smithi196 ceph-mon[81815]: pgmap v921: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:33.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:33 smithi096 ceph-mon[112443]: pgmap v922: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:33 smithi196 ceph-mon[81815]: pgmap v922: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:36.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:36 smithi096 ceph-mon[112443]: pgmap v923: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:37.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:36 smithi196 ceph-mon[81815]: pgmap v923: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:38 smithi096 ceph-mon[112443]: pgmap v924: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:38.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:38 smithi196 ceph-mon[81815]: pgmap v924: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:39.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:39 smithi096 ceph-mon[112443]: pgmap v925: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:39 smithi196 ceph-mon[81815]: pgmap v925: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:42.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.005+0000 7ff4f378a700 1 -- 172.21.15.96:0/2240513838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 msgr2=0x7ff4ec101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.005+0000 7ff4f378a700 1 --2- 172.21.15.96:0/2240513838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 0x7ff4ec101240 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009910 tx=0x7ff4e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:42.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.005+0000 7ff4f378a700 1 -- 172.21.15.96:0/2240513838 shutdown_connections 2024-04-22T15:52:42.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.005+0000 7ff4f378a700 1 --2- 172.21.15.96:0/2240513838 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4ec101810 0x7ff4ec105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.005+0000 7ff4f378a700 1 --2- 172.21.15.96:0/2240513838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 0x7ff4ec101240 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.005+0000 7ff4f378a700 1 -- 172.21.15.96:0/2240513838 >> 172.21.15.96:0/2240513838 conn(0x7ff4ec0fc710 msgr2=0x7ff4ec0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:42.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.005+0000 7ff4f378a700 1 -- 172.21.15.96:0/2240513838 shutdown_connections 2024-04-22T15:52:42.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.005+0000 7ff4f378a700 1 -- 172.21.15.96:0/2240513838 wait complete. 2024-04-22T15:52:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.006+0000 7ff4f378a700 1 Processor -- start 2024-04-22T15:52:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.006+0000 7ff4f378a700 1 -- start start 2024-04-22T15:52:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.006+0000 7ff4f378a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 0x7ff4ec19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.006+0000 7ff4f378a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4ec101810 0x7ff4ec19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.006+0000 7ff4f378a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec19d340 con 0x7ff4ec100e60 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.006+0000 7ff4f378a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec1969b0 con 0x7ff4ec101810 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.006+0000 7ff4f1526700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 0x7ff4ec19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.006+0000 7ff4f1526700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 0x7ff4ec19c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58730/0 (socket says 172.21.15.96:58730) 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4f1526700 1 -- 172.21.15.96:0/285104093 learned_addr learned my addr 172.21.15.96:0/285104093 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4f0d25700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4ec101810 0x7ff4ec19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4f1526700 1 -- 172.21.15.96:0/285104093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4ec101810 msgr2=0x7ff4ec19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4f1526700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4ec101810 0x7ff4ec19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4f1526700 1 -- 172.21.15.96:0/285104093 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e80095f0 con 0x7ff4ec100e60 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4f1526700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 0x7ff4ec19c870 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff4e80092e0 tx=0x7ff4e800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4e27fc700 1 -- 172.21.15.96:0/285104093 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e801d070 con 0x7ff4ec100e60 2024-04-22T15:52:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4e27fc700 1 -- 172.21.15.96:0/285104093 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4e800f460 con 0x7ff4ec100e60 2024-04-22T15:52:42.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec196c30 con 0x7ff4ec100e60 2024-04-22T15:52:42.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4e27fc700 1 -- 172.21.15.96:0/285104093 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e8017630 con 0x7ff4ec100e60 2024-04-22T15:52:42.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.007+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec197120 con 0x7ff4ec100e60 2024-04-22T15:52:42.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.008+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ec109200 con 0x7ff4ec100e60 2024-04-22T15:52:42.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.009+0000 7ff4e27fc700 1 -- 172.21.15.96:0/285104093 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff4e800f5d0 con 0x7ff4ec100e60 2024-04-22T15:52:42.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.009+0000 7ff4e27fc700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4d806ecd0 0x7ff4d8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.009+0000 7ff4e27fc700 1 -- 172.21.15.96:0/285104093 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff4e8090300 con 0x7ff4ec100e60 2024-04-22T15:52:42.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.009+0000 7ff4f0d25700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4d806ecd0 0x7ff4d8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.010+0000 7ff4f0d25700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4d806ecd0 0x7ff4d8071190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec197b70 tx=0x7ff4dc00a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.014+0000 7ff4e27fc700 1 -- 172.21.15.96:0/285104093 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff4e805cb90 con 0x7ff4ec100e60 2024-04-22T15:52:42.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.231+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4ec197e70 con 0x7ff4d806ecd0 2024-04-22T15:52:42.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.232+0000 7ff4e27fc700 1 -- 172.21.15.96:0/285104093 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff4ec197e70 con 0x7ff4d806ecd0 2024-04-22T15:52:42.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.234+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4d806ecd0 msgr2=0x7ff4d8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.234+0000 7ff4f378a700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4d806ecd0 0x7ff4d8071190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec197b70 tx=0x7ff4dc00a380 comp rx=0 tx=0).stop 2024-04-22T15:52:42.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.234+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 msgr2=0x7ff4ec19c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.234+0000 7ff4f378a700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 0x7ff4ec19c870 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff4e80092e0 tx=0x7ff4e800bd70 comp rx=0 tx=0).stop 2024-04-22T15:52:42.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.234+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 shutdown_connections 2024-04-22T15:52:42.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.234+0000 7ff4f378a700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff4d806ecd0 0x7ff4d8071190 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.234+0000 7ff4f378a700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4ec100e60 0x7ff4ec19c870 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.234+0000 7ff4f378a700 1 --2- 172.21.15.96:0/285104093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4ec101810 0x7ff4ec19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.234+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 >> 172.21.15.96:0/285104093 conn(0x7ff4ec0fc710 msgr2=0x7ff4ec0fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:42.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.235+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 shutdown_connections 2024-04-22T15:52:42.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.235+0000 7ff4f378a700 1 -- 172.21.15.96:0/285104093 wait complete. 2024-04-22T15:52:42.246 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:52:42.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.337+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1143516523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d0100e20 msgr2=0x7fc1d0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.337+0000 7fc1d7f2e700 1 --2- 172.21.15.96:0/1143516523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d0100e20 0x7fc1d0101200 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009910 tx=0x7fc1c0009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:42.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.337+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1143516523 shutdown_connections 2024-04-22T15:52:42.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.337+0000 7fc1d7f2e700 1 --2- 172.21.15.96:0/1143516523 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1d01017d0 0x7fc1d0105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.337+0000 7fc1d7f2e700 1 --2- 172.21.15.96:0/1143516523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d0100e20 0x7fc1d0101200 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.337+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1143516523 >> 172.21.15.96:0/1143516523 conn(0x7fc1d00fc6d0 msgr2=0x7fc1d00feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:42.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.337+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1143516523 shutdown_connections 2024-04-22T15:52:42.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.337+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1143516523 wait complete. 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d7f2e700 1 Processor -- start 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d7f2e700 1 -- start start 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d7f2e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1d0100e20 0x7fc1d019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d7f2e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d01017d0 0x7fc1d019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d7f2e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d019d3a0 con 0x7fc1d01017d0 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d7f2e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d01969d0 con 0x7fc1d0100e20 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d54c9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d01017d0 0x7fc1d019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d54c9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d01017d0 0x7fc1d019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58742/0 (socket says 172.21.15.96:58742) 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d54c9700 1 -- 172.21.15.96:0/1858366573 learned_addr learned my addr 172.21.15.96:0/1858366573 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.338+0000 7fc1d5cca700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1d0100e20 0x7fc1d019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.339+0000 7fc1d54c9700 1 -- 172.21.15.96:0/1858366573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1d0100e20 msgr2=0x7fc1d019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.339+0000 7fc1d54c9700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1d0100e20 0x7fc1d019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.339+0000 7fc1d54c9700 1 -- 172.21.15.96:0/1858366573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c00095f0 con 0x7fc1d01017d0 2024-04-22T15:52:42.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.339+0000 7fc1d54c9700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d01017d0 0x7fc1d019cd80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00d900 tx=0x7fc1cc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:42.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.339+0000 7fc1c6ffd700 1 -- 172.21.15.96:0/1858366573 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1cc009900 con 0x7fc1d01017d0 2024-04-22T15:52:42.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.339+0000 7fc1c6ffd700 1 -- 172.21.15.96:0/1858366573 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1cc010460 con 0x7fc1d01017d0 2024-04-22T15:52:42.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.339+0000 7fc1c6ffd700 1 -- 172.21.15.96:0/1858366573 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1cc00f5f0 con 0x7fc1d01017d0 2024-04-22T15:52:42.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.339+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d0196cb0 con 0x7fc1d01017d0 2024-04-22T15:52:42.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.340+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d01971d0 con 0x7fc1d01017d0 2024-04-22T15:52:42.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.340+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d01091c0 con 0x7fc1d01017d0 2024-04-22T15:52:42.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.341+0000 7fc1c6ffd700 1 -- 172.21.15.96:0/1858366573 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc1cc0105d0 con 0x7fc1d01017d0 2024-04-22T15:52:42.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.341+0000 7fc1c6ffd700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc1bc06eac0 0x7fc1bc070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.342+0000 7fc1c6ffd700 1 -- 172.21.15.96:0/1858366573 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc1cc08e7b0 con 0x7fc1d01017d0 2024-04-22T15:52:42.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.342+0000 7fc1d5cca700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc1bc06eac0 0x7fc1bc070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.342+0000 7fc1d5cca700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc1bc06eac0 0x7fc1bc070f80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0000c00 tx=0x7fc1c001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:42.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.346+0000 7fc1c6ffd700 1 -- 172.21.15.96:0/1858366573 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc1cc05b0c0 con 0x7fc1d01017d0 2024-04-22T15:52:42.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.569+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1d004ec50 con 0x7fc1bc06eac0 2024-04-22T15:52:42.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.571+0000 7fc1c6ffd700 1 -- 172.21.15.96:0/1858366573 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc1d004ec50 con 0x7fc1bc06eac0 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc1bc06eac0 msgr2=0x7fc1bc070f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc1bc06eac0 0x7fc1bc070f80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0000c00 tx=0x7fc1c001a040 comp rx=0 tx=0).stop 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d01017d0 msgr2=0x7fc1d019cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d01017d0 0x7fc1d019cd80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00d900 tx=0x7fc1cc00dc10 comp rx=0 tx=0).stop 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 shutdown_connections 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc1bc06eac0 0x7fc1bc070f80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1d0100e20 0x7fc1d019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 --2- 172.21.15.96:0/1858366573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1d01017d0 0x7fc1d019cd80 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 >> 172.21.15.96:0/1858366573 conn(0x7fc1d00fc6d0 msgr2=0x7fc1d00fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 shutdown_connections 2024-04-22T15:52:42.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.573+0000 7fc1d7f2e700 1 -- 172.21.15.96:0/1858366573 wait complete. 2024-04-22T15:52:42.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.684+0000 7fcf9447d700 1 -- 172.21.15.96:0/2750917876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c1010f0 msgr2=0x7fcf8c104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.684+0000 7fcf9447d700 1 --2- 172.21.15.96:0/2750917876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c1010f0 0x7fcf8c104770 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009910 tx=0x7fcf7c009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:42.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.684+0000 7fcf9447d700 1 -- 172.21.15.96:0/2750917876 shutdown_connections 2024-04-22T15:52:42.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.684+0000 7fcf9447d700 1 --2- 172.21.15.96:0/2750917876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c1010f0 0x7fcf8c104770 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.684+0000 7fcf9447d700 1 --2- 172.21.15.96:0/2750917876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf8c100740 0x7fcf8c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.684+0000 7fcf9447d700 1 -- 172.21.15.96:0/2750917876 >> 172.21.15.96:0/2750917876 conn(0x7fcf8c074d90 msgr2=0x7fcf8c0751a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:42.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.684+0000 7fcf9447d700 1 -- 172.21.15.96:0/2750917876 shutdown_connections 2024-04-22T15:52:42.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.684+0000 7fcf9447d700 1 -- 172.21.15.96:0/2750917876 wait complete. 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf9447d700 1 Processor -- start 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf9447d700 1 -- start start 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf9447d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c100740 0x7fcf8c195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf9447d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf8c1010f0 0x7fcf8c195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf9447d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c196610 con 0x7fcf8c100740 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf9447d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c19a360 con 0x7fcf8c1010f0 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf92219700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c100740 0x7fcf8c195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf92219700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c100740 0x7fcf8c195a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58758/0 (socket says 172.21.15.96:58758) 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf92219700 1 -- 172.21.15.96:0/1488155984 learned_addr learned my addr 172.21.15.96:0/1488155984 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.685+0000 7fcf91a18700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf8c1010f0 0x7fcf8c195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf92219700 1 -- 172.21.15.96:0/1488155984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf8c1010f0 msgr2=0x7fcf8c195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf92219700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf8c1010f0 0x7fcf8c195f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf92219700 1 -- 172.21.15.96:0/1488155984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c0095f0 con 0x7fcf8c100740 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf91a18700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf8c1010f0 0x7fcf8c195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf92219700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c100740 0x7fcf8c195a40 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800c8f0 tx=0x7fcf8800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf837fe700 1 -- 172.21.15.96:0/1488155984 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf88007a70 con 0x7fcf8c100740 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8c19a640 con 0x7fcf8c100740 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf837fe700 1 -- 172.21.15.96:0/1488155984 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf88004cf0 con 0x7fcf8c100740 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf837fe700 1 -- 172.21.15.96:0/1488155984 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf88005610 con 0x7fcf8c100740 2024-04-22T15:52:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.686+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8c19ab60 con 0x7fcf8c100740 2024-04-22T15:52:42.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.687+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8c108050 con 0x7fcf8c100740 2024-04-22T15:52:42.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.688+0000 7fcf837fe700 1 -- 172.21.15.96:0/1488155984 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcf88004750 con 0x7fcf8c100740 2024-04-22T15:52:42.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.688+0000 7fcf837fe700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf7806ecc0 0x7fcf78071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.688+0000 7fcf837fe700 1 -- 172.21.15.96:0/1488155984 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcf88013070 con 0x7fcf8c100740 2024-04-22T15:52:42.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.688+0000 7fcf91a18700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf7806ecc0 0x7fcf78071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.689+0000 7fcf91a18700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf7806ecc0 0x7fcf78071180 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0ff1c0 tx=0x7fcf7c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:42.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.695+0000 7fcf837fe700 1 -- 172.21.15.96:0/1488155984 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcf8805b510 con 0x7fcf8c100740 2024-04-22T15:52:42.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.903+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf8c196dc0 con 0x7fcf7806ecc0 2024-04-22T15:52:42.907 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:42 smithi096 ceph-mon[112443]: pgmap v926: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:42.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.914+0000 7fcf837fe700 1 -- 172.21.15.96:0/1488155984 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fcf8c196dc0 con 0x7fcf7806ecc0 2024-04-22T15:52:42.917 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:52:42.917 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (31m) 9m ago 42m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:52:42.917 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (43m) 9m ago 43m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:52:42.917 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (41m) 9m ago 41m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:52:42.917 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (28m) 9m ago 42m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:52:42.917 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (28m) 9m ago 41m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (31m) 9m ago 42m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (24m) 9m ago 36m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (24m) 9m ago 37m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (24m) 9m ago 37m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (9m) 9m ago 36m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (33m) 9m ago 44m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (32m) 9m ago 41m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (29m) 9m ago 44m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (29m) 9m ago 41m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (32m) 9m ago 42m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (32m) 9m ago 41m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (28m) 9m ago 40m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:52:42.918 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (27m) 9m ago 40m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:52:42.919 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (26m) 9m ago 39m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:52:42.919 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (26m) 9m ago 39m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:52:42.919 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (25m) 9m ago 39m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:52:42.919 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (25m) 9m ago 38m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:52:42.919 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (31m) 9m ago 42m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:52:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf7806ecc0 msgr2=0x7fcf78071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf7806ecc0 0x7fcf78071180 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0ff1c0 tx=0x7fcf7c01a040 comp rx=0 tx=0).stop 2024-04-22T15:52:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c100740 msgr2=0x7fcf8c195a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c100740 0x7fcf8c195a40 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800c8f0 tx=0x7fcf8800ccb0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 shutdown_connections 2024-04-22T15:52:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcf7806ecc0 0x7fcf78071180 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf8c100740 0x7fcf8c195a40 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 --2- 172.21.15.96:0/1488155984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf8c1010f0 0x7fcf8c195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 >> 172.21.15.96:0/1488155984 conn(0x7fcf8c074d90 msgr2=0x7fcf8c0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:42.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 shutdown_connections 2024-04-22T15:52:42.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:42.917+0000 7fcf9447d700 1 -- 172.21.15.96:0/1488155984 wait complete. 2024-04-22T15:52:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:42 smithi196 ceph-mon[81815]: pgmap v926: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:43.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.019+0000 7f5751546700 1 -- 172.21.15.96:0/2816010363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c100e20 msgr2=0x7f574c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.019+0000 7f5751546700 1 --2- 172.21.15.96:0/2816010363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c100e20 0x7f574c101200 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f573c009910 tx=0x7f573c009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.019+0000 7f5751546700 1 -- 172.21.15.96:0/2816010363 shutdown_connections 2024-04-22T15:52:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.019+0000 7f5751546700 1 --2- 172.21.15.96:0/2816010363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f574c1017d0 0x7f574c105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.019+0000 7f5751546700 1 --2- 172.21.15.96:0/2816010363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c100e20 0x7f574c101200 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.019+0000 7f5751546700 1 -- 172.21.15.96:0/2816010363 >> 172.21.15.96:0/2816010363 conn(0x7f574c0fc6d0 msgr2=0x7f574c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.019+0000 7f5751546700 1 -- 172.21.15.96:0/2816010363 shutdown_connections 2024-04-22T15:52:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.019+0000 7f5751546700 1 -- 172.21.15.96:0/2816010363 wait complete. 2024-04-22T15:52:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.020+0000 7f5751546700 1 Processor -- start 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.020+0000 7f5751546700 1 -- start start 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.020+0000 7f5751546700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f574c100e20 0x7f574c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.020+0000 7f5751546700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c1017d0 0x7f574c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.020+0000 7f5751546700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c19d3a0 con 0x7f574c1017d0 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.020+0000 7f5751546700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c1969d0 con 0x7f574c100e20 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f574a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c1017d0 0x7f574c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f574a7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c1017d0 0x7f574c19cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58788/0 (socket says 172.21.15.96:58788) 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f574a7fc700 1 -- 172.21.15.96:0/2277820798 learned_addr learned my addr 172.21.15.96:0/2277820798 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f574affd700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f574c100e20 0x7f574c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f574a7fc700 1 -- 172.21.15.96:0/2277820798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f574c100e20 msgr2=0x7f574c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f574a7fc700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f574c100e20 0x7f574c19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f574a7fc700 1 -- 172.21.15.96:0/2277820798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0095f0 con 0x7f574c1017d0 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f574a7fc700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c1017d0 0x7f574c19cd80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f574000d900 tx=0x7f574000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f5733fff700 1 -- 172.21.15.96:0/2277820798 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5740009900 con 0x7f574c1017d0 2024-04-22T15:52:43.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c196cb0 con 0x7f574c1017d0 2024-04-22T15:52:43.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f5733fff700 1 -- 172.21.15.96:0/2277820798 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5740010460 con 0x7f574c1017d0 2024-04-22T15:52:43.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f5733fff700 1 -- 172.21.15.96:0/2277820798 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f574000f5f0 con 0x7f574c1017d0 2024-04-22T15:52:43.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.021+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c197200 con 0x7f574c1017d0 2024-04-22T15:52:43.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.022+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c1091c0 con 0x7f574c1017d0 2024-04-22T15:52:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.023+0000 7f5733fff700 1 -- 172.21.15.96:0/2277820798 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f57400105d0 con 0x7f574c1017d0 2024-04-22T15:52:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.023+0000 7f5733fff700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573406ecc0 0x7f5734071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.023+0000 7f5733fff700 1 -- 172.21.15.96:0/2277820798 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f574008ea50 con 0x7f574c1017d0 2024-04-22T15:52:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.023+0000 7f574affd700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573406ecc0 0x7f5734071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.024+0000 7f574affd700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573406ecc0 0x7f5734071180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f573c000c00 tx=0x7f573c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.028+0000 7f5733fff700 1 -- 172.21.15.96:0/2277820798 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f574005b2e0 con 0x7f574c1017d0 2024-04-22T15:52:43.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.309+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f574c102300 con 0x7f574c1017d0 2024-04-22T15:52:43.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.309+0000 7f5733fff700 1 -- 172.21.15.96:0/2277820798 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f574005ae70 con 0x7f574c1017d0 2024-04-22T15:52:43.312 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:52:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:52:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:52:43.313 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:52:43.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.311+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573406ecc0 msgr2=0x7f5734071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.311+0000 7f5751546700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573406ecc0 0x7f5734071180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f573c000c00 tx=0x7f573c01a040 comp rx=0 tx=0).stop 2024-04-22T15:52:43.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.311+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c1017d0 msgr2=0x7f574c19cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.311+0000 7f5751546700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c1017d0 0x7f574c19cd80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f574000d900 tx=0x7f574000dc10 comp rx=0 tx=0).stop 2024-04-22T15:52:43.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.311+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 shutdown_connections 2024-04-22T15:52:43.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.311+0000 7f5751546700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573406ecc0 0x7f5734071180 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.311+0000 7f5751546700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f574c100e20 0x7f574c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.311+0000 7f5751546700 1 --2- 172.21.15.96:0/2277820798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f574c1017d0 0x7f574c19cd80 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.311+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 >> 172.21.15.96:0/2277820798 conn(0x7f574c0fc6d0 msgr2=0x7f574c0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:43.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.312+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 shutdown_connections 2024-04-22T15:52:43.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.312+0000 7f5751546700 1 -- 172.21.15.96:0/2277820798 wait complete. 2024-04-22T15:52:43.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.414+0000 7ff215154700 1 -- 172.21.15.96:0/175178176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 msgr2=0x7ff2100fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.414+0000 7ff215154700 1 --2- 172.21.15.96:0/175178176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 0x7ff2100fff20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7ff200009910 tx=0x7ff200009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:43.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.414+0000 7ff215154700 1 -- 172.21.15.96:0/175178176 shutdown_connections 2024-04-22T15:52:43.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.414+0000 7ff215154700 1 --2- 172.21.15.96:0/175178176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 0x7ff2100fff20 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.414+0000 7ff215154700 1 --2- 172.21.15.96:0/175178176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff210105aa0 0x7ff210105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.414+0000 7ff215154700 1 -- 172.21.15.96:0/175178176 >> 172.21.15.96:0/175178176 conn(0x7ff210076740 msgr2=0x7ff2100fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:43.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.414+0000 7ff215154700 1 -- 172.21.15.96:0/175178176 shutdown_connections 2024-04-22T15:52:43.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.414+0000 7ff215154700 1 -- 172.21.15.96:0/175178176 wait complete. 2024-04-22T15:52:43.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff215154700 1 Processor -- start 2024-04-22T15:52:43.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff215154700 1 -- start start 2024-04-22T15:52:43.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff215154700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 0x7ff21019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:43.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff215154700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff210105aa0 0x7ff21019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:43.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff215154700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21019d360 con 0x7ff2100ffaa0 2024-04-22T15:52:43.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff215154700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff210196940 con 0x7ff210105aa0 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff20ed9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 0x7ff21019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff20ed9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 0x7ff21019c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58808/0 (socket says 172.21.15.96:58808) 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff20ed9d700 1 -- 172.21.15.96:0/3354294331 learned_addr learned my addr 172.21.15.96:0/3354294331 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.415+0000 7ff20e59c700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff210105aa0 0x7ff21019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.416+0000 7ff20ed9d700 1 -- 172.21.15.96:0/3354294331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff210105aa0 msgr2=0x7ff21019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.416+0000 7ff20ed9d700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff210105aa0 0x7ff21019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.416+0000 7ff20ed9d700 1 -- 172.21.15.96:0/3354294331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2000095f0 con 0x7ff2100ffaa0 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.416+0000 7ff20ed9d700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 0x7ff21019c800 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff20400d900 tx=0x7ff20400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.416+0000 7ff1f7fff700 1 -- 172.21.15.96:0/3354294331 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2040041f0 con 0x7ff2100ffaa0 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.416+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff210196c20 con 0x7ff2100ffaa0 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.416+0000 7ff1f7fff700 1 -- 172.21.15.96:0/3354294331 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff204004cf0 con 0x7ff2100ffaa0 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.416+0000 7ff1f7fff700 1 -- 172.21.15.96:0/3354294331 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff20400b750 con 0x7ff2100ffaa0 2024-04-22T15:52:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.416+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff210197170 con 0x7ff2100ffaa0 2024-04-22T15:52:43.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.417+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2101011e0 con 0x7ff2100ffaa0 2024-04-22T15:52:43.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.417+0000 7ff1f7fff700 1 -- 172.21.15.96:0/3354294331 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff204004350 con 0x7ff2100ffaa0 2024-04-22T15:52:43.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.418+0000 7ff1f7fff700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff1f806ecc0 0x7ff1f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:43.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.418+0000 7ff1f7fff700 1 -- 172.21.15.96:0/3354294331 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff20408ed80 con 0x7ff2100ffaa0 2024-04-22T15:52:43.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.418+0000 7ff20e59c700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff1f806ecc0 0x7ff1f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:43.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.418+0000 7ff20e59c700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff1f806ecc0 0x7ff1f8071180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff200005b40 tx=0x7ff20001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:43.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.423+0000 7ff1f7fff700 1 -- 172.21.15.96:0/3354294331 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff20405b690 con 0x7ff2100ffaa0 2024-04-22T15:52:43.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.671+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff21010cca0 con 0x7ff2100ffaa0 2024-04-22T15:52:43.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.671+0000 7ff1f7fff700 1 -- 172.21.15.96:0/3354294331 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7ff204016770 con 0x7ff2100ffaa0 2024-04-22T15:52:43.674 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:52:43.674 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:52:43.675 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:52:43.676 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:52:43.677 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:52:43.678 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:52:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.673+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff1f806ecc0 msgr2=0x7ff1f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.673+0000 7ff215154700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff1f806ecc0 0x7ff1f8071180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff200005b40 tx=0x7ff20001a040 comp rx=0 tx=0).stop 2024-04-22T15:52:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.673+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 msgr2=0x7ff21019c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.673+0000 7ff215154700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 0x7ff21019c800 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff20400d900 tx=0x7ff20400dcc0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.673+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 shutdown_connections 2024-04-22T15:52:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.673+0000 7ff215154700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff1f806ecc0 0x7ff1f8071180 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.673+0000 7ff215154700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2100ffaa0 0x7ff21019c800 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.673+0000 7ff215154700 1 --2- 172.21.15.96:0/3354294331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff210105aa0 0x7ff21019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.673+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 >> 172.21.15.96:0/3354294331 conn(0x7ff210076740 msgr2=0x7ff210104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.674+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 shutdown_connections 2024-04-22T15:52:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.674+0000 7ff215154700 1 -- 172.21.15.96:0/3354294331 wait complete. 2024-04-22T15:52:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:52:43.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.780+0000 7f3c79e15700 1 -- 172.21.15.96:0/4236210570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c741017d0 msgr2=0x7f3c74105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.780+0000 7f3c79e15700 1 --2- 172.21.15.96:0/4236210570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c741017d0 0x7f3c74105820 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009910 tx=0x7f3c68009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:43.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.780+0000 7f3c79e15700 1 -- 172.21.15.96:0/4236210570 shutdown_connections 2024-04-22T15:52:43.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.780+0000 7f3c79e15700 1 --2- 172.21.15.96:0/4236210570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c741017d0 0x7f3c74105820 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.780+0000 7f3c79e15700 1 --2- 172.21.15.96:0/4236210570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c74100e20 0x7f3c74101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.780+0000 7f3c79e15700 1 -- 172.21.15.96:0/4236210570 >> 172.21.15.96:0/4236210570 conn(0x7f3c740fc6d0 msgr2=0x7f3c740feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:43.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.780+0000 7f3c79e15700 1 -- 172.21.15.96:0/4236210570 shutdown_connections 2024-04-22T15:52:43.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.780+0000 7f3c79e15700 1 -- 172.21.15.96:0/4236210570 wait complete. 2024-04-22T15:52:43.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.780+0000 7f3c79e15700 1 Processor -- start 2024-04-22T15:52:43.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c79e15700 1 -- start start 2024-04-22T15:52:43.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c79e15700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c74100e20 0x7f3c7419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:43.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c79e15700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c741017d0 0x7f3c7419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:43.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c79e15700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7419d4c0 con 0x7f3c74100e20 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c79e15700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c74196aa0 con 0x7f3c741017d0 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c737fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c74100e20 0x7f3c7419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c737fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c74100e20 0x7f3c7419c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58834/0 (socket says 172.21.15.96:58834) 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c737fe700 1 -- 172.21.15.96:0/1882975916 learned_addr learned my addr 172.21.15.96:0/1882975916 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c72ffd700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c741017d0 0x7f3c7419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c737fe700 1 -- 172.21.15.96:0/1882975916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c741017d0 msgr2=0x7f3c7419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c737fe700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c741017d0 0x7f3c7419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c737fe700 1 -- 172.21.15.96:0/1882975916 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c680095f0 con 0x7f3c74100e20 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c737fe700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c74100e20 0x7f3c7419c960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400b4b0 tx=0x7f3c6400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.781+0000 7f3c70ff9700 1 -- 172.21.15.96:0/1882975916 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c64011800 con 0x7f3c74100e20 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.782+0000 7f3c70ff9700 1 -- 172.21.15.96:0/1882975916 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c64011e20 con 0x7f3c74100e20 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.782+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c74196d80 con 0x7f3c74100e20 2024-04-22T15:52:43.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.782+0000 7f3c70ff9700 1 -- 172.21.15.96:0/1882975916 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c6400f2e0 con 0x7f3c74100e20 2024-04-22T15:52:43.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.782+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c741972d0 con 0x7f3c74100e20 2024-04-22T15:52:43.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.782+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c74109150 con 0x7f3c74100e20 2024-04-22T15:52:43.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.783+0000 7f3c70ff9700 1 -- 172.21.15.96:0/1882975916 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3c64010380 con 0x7f3c74100e20 2024-04-22T15:52:43.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.783+0000 7f3c70ff9700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3c5c06ecc0 0x7f3c5c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:43.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.784+0000 7f3c70ff9700 1 -- 172.21.15.96:0/1882975916 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3c6408ea50 con 0x7f3c74100e20 2024-04-22T15:52:43.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.784+0000 7f3c72ffd700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3c5c06ecc0 0x7f3c5c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:43.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.784+0000 7f3c72ffd700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3c5c06ecc0 0x7f3c5c071180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3c74197c60 tx=0x7f3c6801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:43.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:43.788+0000 7f3c70ff9700 1 -- 172.21.15.96:0/1882975916 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3c6405b2e0 con 0x7f3c74100e20 2024-04-22T15:52:43.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:43 smithi096 ceph-mon[112443]: from='client.35684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:43.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:43 smithi096 ceph-mon[112443]: from='client.35688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:43 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2277820798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:43 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3354294331' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:52:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:43 smithi196 ceph-mon[81815]: from='client.35684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:43 smithi196 ceph-mon[81815]: from='client.35688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:43 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2277820798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:43 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3354294331' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:52:44.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.010+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c74197fd0 con 0x7f3c5c06ecc0 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.011+0000 7f3c70ff9700 1 -- 172.21.15.96:0/1882975916 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3c74197fd0 con 0x7f3c5c06ecc0 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:52:44.014 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:52:44.015 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:52:44.015 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:52:44.015 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:52:44.015 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:52:44.015 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:52:44.015 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:52:44.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3c5c06ecc0 msgr2=0x7f3c5c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:44.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3c5c06ecc0 0x7f3c5c071180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3c74197c60 tx=0x7f3c6801a040 comp rx=0 tx=0).stop 2024-04-22T15:52:44.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c74100e20 msgr2=0x7f3c7419c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:44.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c74100e20 0x7f3c7419c960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400b4b0 tx=0x7f3c6400b870 comp rx=0 tx=0).stop 2024-04-22T15:52:44.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 shutdown_connections 2024-04-22T15:52:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3c5c06ecc0 0x7f3c5c071180 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c74100e20 0x7f3c7419c960 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 --2- 172.21.15.96:0/1882975916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c741017d0 0x7f3c7419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 >> 172.21.15.96:0/1882975916 conn(0x7f3c740fc6d0 msgr2=0x7f3c740fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 shutdown_connections 2024-04-22T15:52:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.013+0000 7f3c79e15700 1 -- 172.21.15.96:0/1882975916 wait complete. 2024-04-22T15:52:44.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 -- 172.21.15.96:0/4182435212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 msgr2=0x7fb0680fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:44.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 --2- 172.21.15.96:0/4182435212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 0x7fb0680fff20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb050009910 tx=0x7fb050009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:44.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 -- 172.21.15.96:0/4182435212 shutdown_connections 2024-04-22T15:52:44.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 --2- 172.21.15.96:0/4182435212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 0x7fb0680fff20 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 --2- 172.21.15.96:0/4182435212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb068105aa0 0x7fb068105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 -- 172.21.15.96:0/4182435212 >> 172.21.15.96:0/4182435212 conn(0x7fb068076740 msgr2=0x7fb0680fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 -- 172.21.15.96:0/4182435212 shutdown_connections 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 -- 172.21.15.96:0/4182435212 wait complete. 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 Processor -- start 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.117+0000 7fb06e1ee700 1 -- start start 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb06e1ee700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 0x7fb0680fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb06e1ee700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb068105aa0 0x7fb0680fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb06e1ee700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680fc6a0 con 0x7fb0680ffaa0 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb06e1ee700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680fc810 con 0x7fb068105aa0 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0677fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 0x7fb0680fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:44.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0677fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 0x7fb0680fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58842/0 (socket says 172.21.15.96:58842) 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0677fe700 1 -- 172.21.15.96:0/1886069445 learned_addr learned my addr 172.21.15.96:0/1886069445 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb05effd700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb068105aa0 0x7fb0680fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0677fe700 1 -- 172.21.15.96:0/1886069445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb068105aa0 msgr2=0x7fb0680fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0677fe700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb068105aa0 0x7fb0680fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0677fe700 1 -- 172.21.15.96:0/1886069445 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0500095f0 con 0x7fb0680ffaa0 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0677fe700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 0x7fb0680fbc20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb05800c8a0 tx=0x7fb05800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0657fa700 1 -- 172.21.15.96:0/1886069445 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0580078e0 con 0x7fb0680ffaa0 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0657fa700 1 -- 172.21.15.96:0/1886069445 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb05800f450 con 0x7fb0680ffaa0 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0681a2c70 con 0x7fb0680ffaa0 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb0657fa700 1 -- 172.21.15.96:0/1886069445 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb05800e5e0 con 0x7fb0680ffaa0 2024-04-22T15:52:44.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.118+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0681a3020 con 0x7fb0680ffaa0 2024-04-22T15:52:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.119+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0681011e0 con 0x7fb0680ffaa0 2024-04-22T15:52:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.120+0000 7fb0657fa700 1 -- 172.21.15.96:0/1886069445 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb05800f5c0 con 0x7fb0680ffaa0 2024-04-22T15:52:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.120+0000 7fb0657fa700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb05406ecc0 0x7fb054071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.121+0000 7fb05effd700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb05406ecc0 0x7fb054071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.121+0000 7fb0657fa700 1 -- 172.21.15.96:0/1886069445 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb05808ed10 con 0x7fb0680ffaa0 2024-04-22T15:52:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.121+0000 7fb05effd700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb05406ecc0 0x7fb054071180 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb0680fd350 tx=0x7fb05001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:44.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.125+0000 7fb0657fa700 1 -- 172.21.15.96:0/1886069445 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb058057af0 con 0x7fb0680ffaa0 2024-04-22T15:52:44.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.409+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb06810cca0 con 0x7fb0680ffaa0 2024-04-22T15:52:44.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.409+0000 7fb0657fa700 1 -- 172.21.15.96:0/1886069445 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fb05805b1b0 con 0x7fb0680ffaa0 2024-04-22T15:52:44.412 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:52:44.412 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:52:44.413 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:52:44.414 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:52:44.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.411+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb05406ecc0 msgr2=0x7fb054071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:44.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.411+0000 7fb06e1ee700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb05406ecc0 0x7fb054071180 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb0680fd350 tx=0x7fb05001a040 comp rx=0 tx=0).stop 2024-04-22T15:52:44.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.411+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 msgr2=0x7fb0680fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:44.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.411+0000 7fb06e1ee700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 0x7fb0680fbc20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb05800c8a0 tx=0x7fb05800cbb0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.412+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 shutdown_connections 2024-04-22T15:52:44.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.412+0000 7fb06e1ee700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb05406ecc0 0x7fb054071180 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.412+0000 7fb06e1ee700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0680ffaa0 0x7fb0680fbc20 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.412+0000 7fb06e1ee700 1 --2- 172.21.15.96:0/1886069445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb068105aa0 0x7fb0680fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:44.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.412+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 >> 172.21.15.96:0/1886069445 conn(0x7fb068076740 msgr2=0x7fb0680687a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:44.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.412+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 shutdown_connections 2024-04-22T15:52:44.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:52:44.412+0000 7fb06e1ee700 1 -- 172.21.15.96:0/1886069445 wait complete. 2024-04-22T15:52:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:44 smithi096 ceph-mon[112443]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:44.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:44 smithi096 ceph-mon[112443]: pgmap v927: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:44.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:44 smithi096 ceph-mon[112443]: from='client.35704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:44.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:44 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1886069445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:52:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:44 smithi196 ceph-mon[81815]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:44 smithi196 ceph-mon[81815]: pgmap v927: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:44 smithi196 ceph-mon[81815]: from='client.35704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:44 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1886069445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:52:45.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:45 smithi096 ceph-mon[112443]: pgmap v928: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:45 smithi196 ceph-mon[81815]: pgmap v928: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:48 smithi096 ceph-mon[112443]: pgmap v929: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:48 smithi196 ceph-mon[81815]: pgmap v929: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:49.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:49 smithi096 ceph-mon[112443]: pgmap v930: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:49 smithi196 ceph-mon[81815]: pgmap v930: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:52.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:52 smithi096 ceph-mon[112443]: pgmap v931: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:52 smithi196 ceph-mon[81815]: pgmap v931: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:54.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:54.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:54 smithi096 ceph-mon[112443]: pgmap v932: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:55.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:54 smithi196 ceph-mon[81815]: pgmap v932: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:55.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:55 smithi096 ceph-mon[112443]: pgmap v933: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:55 smithi196 ceph-mon[81815]: pgmap v933: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:58.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:52:58 smithi096 ceph-mon[112443]: pgmap v934: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:52:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:52:58 smithi196 ceph-mon[81815]: pgmap v934: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:00.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:00 smithi096 ceph-mon[112443]: pgmap v935: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:01.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:00 smithi196 ceph-mon[81815]: pgmap v935: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:01 smithi196 ceph-mon[81815]: pgmap v936: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:02.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:01 smithi096 ceph-mon[112443]: pgmap v936: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:04.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:04 smithi096 ceph-mon[112443]: pgmap v937: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:05.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:04 smithi196 ceph-mon[81815]: pgmap v937: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:05 smithi196 ceph-mon[81815]: pgmap v938: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:06.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:05 smithi096 ceph-mon[112443]: pgmap v938: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:08 smithi096 ceph-mon[112443]: pgmap v939: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:08.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:08 smithi196 ceph-mon[81815]: pgmap v939: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:10.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:09 smithi196 ceph-mon[81815]: pgmap v940: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:10.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:09 smithi096 ceph-mon[112443]: pgmap v940: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:12.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:12 smithi096 ceph-mon[112443]: pgmap v941: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:13.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:12 smithi196 ceph-mon[81815]: pgmap v941: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:14.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.529+0000 7fea18a38700 1 -- 172.21.15.96:0/250297796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea14105b60 msgr2=0x7fea14105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:14.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.529+0000 7fea18a38700 1 --2- 172.21.15.96:0/250297796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea14105b60 0x7fea14105f40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fea08009910 tx=0x7fea08009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.529+0000 7fea18a38700 1 -- 172.21.15.96:0/250297796 shutdown_connections 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.529+0000 7fea18a38700 1 --2- 172.21.15.96:0/250297796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea140ffad0 0x7fea140fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.529+0000 7fea18a38700 1 --2- 172.21.15.96:0/250297796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea14105b60 0x7fea14105f40 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.529+0000 7fea18a38700 1 -- 172.21.15.96:0/250297796 >> 172.21.15.96:0/250297796 conn(0x7fea140fb5e0 msgr2=0x7fea140fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.530+0000 7fea18a38700 1 -- 172.21.15.96:0/250297796 shutdown_connections 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.530+0000 7fea18a38700 1 -- 172.21.15.96:0/250297796 wait complete. 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.530+0000 7fea18a38700 1 Processor -- start 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.530+0000 7fea18a38700 1 -- start start 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea18a38700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea140ffad0 0x7fea141979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea18a38700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea14105b60 0x7fea14197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea18a38700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea14198430 con 0x7fea140ffad0 2024-04-22T15:53:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea18a38700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea14198570 con 0x7fea14105b60 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea1259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea140ffad0 0x7fea141979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea1259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea140ffad0 0x7fea141979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41354/0 (socket says 172.21.15.96:41354) 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea1259c700 1 -- 172.21.15.96:0/3965400232 learned_addr learned my addr 172.21.15.96:0/3965400232 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea11d9b700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea14105b60 0x7fea14197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea1259c700 1 -- 172.21.15.96:0/3965400232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea14105b60 msgr2=0x7fea14197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea1259c700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea14105b60 0x7fea14197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea1259c700 1 -- 172.21.15.96:0/3965400232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea080095f0 con 0x7fea140ffad0 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea1259c700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea140ffad0 0x7fea141979b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fea08006010 tx=0x7fea0800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.531+0000 7fea037fe700 1 -- 172.21.15.96:0/3965400232 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea0801d070 con 0x7fea140ffad0 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.532+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea141987f0 con 0x7fea140ffad0 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.532+0000 7fea037fe700 1 -- 172.21.15.96:0/3965400232 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea080056e0 con 0x7fea140ffad0 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.532+0000 7fea037fe700 1 -- 172.21.15.96:0/3965400232 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea0800fdf0 con 0x7fea140ffad0 2024-04-22T15:53:14.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.532+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea141a0d50 con 0x7fea140ffad0 2024-04-22T15:53:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.533+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea141080d0 con 0x7fea140ffad0 2024-04-22T15:53:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.533+0000 7fea037fe700 1 -- 172.21.15.96:0/3965400232 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fea08003a60 con 0x7fea140ffad0 2024-04-22T15:53:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.533+0000 7fea037fe700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9fc06ecc0 0x7fe9fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:14.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.534+0000 7fea037fe700 1 -- 172.21.15.96:0/3965400232 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fea080901e0 con 0x7fea140ffad0 2024-04-22T15:53:14.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.534+0000 7fea11d9b700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9fc06ecc0 0x7fe9fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:14.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.534+0000 7fea11d9b700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9fc06ecc0 0x7fe9fc071180 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fea140698f0 tx=0x7fea0400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:14.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.538+0000 7fea037fe700 1 -- 172.21.15.96:0/3965400232 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fea08058f40 con 0x7fea140ffad0 2024-04-22T15:53:14.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.759+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea1410ccd0 con 0x7fe9fc06ecc0 2024-04-22T15:53:14.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.760+0000 7fea037fe700 1 -- 172.21.15.96:0/3965400232 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fea1410ccd0 con 0x7fe9fc06ecc0 2024-04-22T15:53:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9fc06ecc0 msgr2=0x7fe9fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9fc06ecc0 0x7fe9fc071180 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fea140698f0 tx=0x7fea0400b410 comp rx=0 tx=0).stop 2024-04-22T15:53:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea140ffad0 msgr2=0x7fea141979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea140ffad0 0x7fea141979b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fea08006010 tx=0x7fea0800bd70 comp rx=0 tx=0).stop 2024-04-22T15:53:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 shutdown_connections 2024-04-22T15:53:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe9fc06ecc0 0x7fe9fc071180 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea140ffad0 0x7fea141979b0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 --2- 172.21.15.96:0/3965400232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea14105b60 0x7fea14197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 >> 172.21.15.96:0/3965400232 conn(0x7fea140fb5e0 msgr2=0x7fea140fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:14.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 shutdown_connections 2024-04-22T15:53:14.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.762+0000 7fea18a38700 1 -- 172.21.15.96:0/3965400232 wait complete. 2024-04-22T15:53:14.775 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:53:14.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.865+0000 7feee9d3f700 1 -- 172.21.15.96:0/52293563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 msgr2=0x7feee4105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:14.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.865+0000 7feee9d3f700 1 --2- 172.21.15.96:0/52293563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 0x7feee4105820 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7feed4009910 tx=0x7feed4009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:14.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.865+0000 7feee9d3f700 1 -- 172.21.15.96:0/52293563 shutdown_connections 2024-04-22T15:53:14.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.865+0000 7feee9d3f700 1 --2- 172.21.15.96:0/52293563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 0x7feee4105820 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.865+0000 7feee9d3f700 1 --2- 172.21.15.96:0/52293563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feee4100e20 0x7feee4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.865+0000 7feee9d3f700 1 -- 172.21.15.96:0/52293563 >> 172.21.15.96:0/52293563 conn(0x7feee40fc6d0 msgr2=0x7feee40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:14.869 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:14 smithi096 ceph-mon[112443]: pgmap v942: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.866+0000 7feee9d3f700 1 -- 172.21.15.96:0/52293563 shutdown_connections 2024-04-22T15:53:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.866+0000 7feee9d3f700 1 -- 172.21.15.96:0/52293563 wait complete. 2024-04-22T15:53:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.866+0000 7feee9d3f700 1 Processor -- start 2024-04-22T15:53:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.866+0000 7feee9d3f700 1 -- start start 2024-04-22T15:53:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.866+0000 7feee9d3f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feee4100e20 0x7feee419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.866+0000 7feee9d3f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 0x7feee419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.866+0000 7feee9d3f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee419d380 con 0x7feee41017d0 2024-04-22T15:53:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.866+0000 7feee9d3f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee4196960 con 0x7feee4100e20 2024-04-22T15:53:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 0x7feee419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 0x7feee419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41368/0 (socket says 172.21.15.96:41368) 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee2ffd700 1 -- 172.21.15.96:0/4264278340 learned_addr learned my addr 172.21.15.96:0/4264278340 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee37fe700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feee4100e20 0x7feee419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee2ffd700 1 -- 172.21.15.96:0/4264278340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feee4100e20 msgr2=0x7feee419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee2ffd700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feee4100e20 0x7feee419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee2ffd700 1 -- 172.21.15.96:0/4264278340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed40095f0 con 0x7feee41017d0 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee2ffd700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 0x7feee419cd60 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7feed400bd70 tx=0x7feed400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee0ff9700 1 -- 172.21.15.96:0/4264278340 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed401d070 con 0x7feee41017d0 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee4196be0 con 0x7feee41017d0 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee0ff9700 1 -- 172.21.15.96:0/4264278340 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feed40056e0 con 0x7feee41017d0 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee0ff9700 1 -- 172.21.15.96:0/4264278340 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed400fe50 con 0x7feee41017d0 2024-04-22T15:53:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.867+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee41970d0 con 0x7feee41017d0 2024-04-22T15:53:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.868+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee4109220 con 0x7feee41017d0 2024-04-22T15:53:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.869+0000 7feee0ff9700 1 -- 172.21.15.96:0/4264278340 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feed4017450 con 0x7feee41017d0 2024-04-22T15:53:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.869+0000 7feee0ff9700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feecc06ecc0 0x7feecc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.869+0000 7feee0ff9700 1 -- 172.21.15.96:0/4264278340 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7feed4090300 con 0x7feee41017d0 2024-04-22T15:53:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.869+0000 7feee37fe700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feecc06ecc0 0x7feecc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:14.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.870+0000 7feee37fe700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feecc06ecc0 0x7feecc071180 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feed8005950 tx=0x7feed800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:14.874+0000 7feee0ff9700 1 -- 172.21.15.96:0/4264278340 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7feed405cc00 con 0x7feee41017d0 2024-04-22T15:53:15.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:14 smithi196 ceph-mon[81815]: pgmap v942: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:15.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.093+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feee4197e20 con 0x7feecc06ecc0 2024-04-22T15:53:15.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.095+0000 7feee0ff9700 1 -- 172.21.15.96:0/4264278340 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feee4197e20 con 0x7feecc06ecc0 2024-04-22T15:53:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.096+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feecc06ecc0 msgr2=0x7feecc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.096+0000 7feee9d3f700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feecc06ecc0 0x7feecc071180 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feed8005950 tx=0x7feed800b410 comp rx=0 tx=0).stop 2024-04-22T15:53:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.097+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 msgr2=0x7feee419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.097+0000 7feee9d3f700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 0x7feee419cd60 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7feed400bd70 tx=0x7feed400bda0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.097+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 shutdown_connections 2024-04-22T15:53:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.097+0000 7feee9d3f700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feecc06ecc0 0x7feecc071180 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.097+0000 7feee9d3f700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feee4100e20 0x7feee419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.097+0000 7feee9d3f700 1 --2- 172.21.15.96:0/4264278340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee41017d0 0x7feee419cd60 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.097+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 >> 172.21.15.96:0/4264278340 conn(0x7feee40fc6d0 msgr2=0x7feee40fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.097+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 shutdown_connections 2024-04-22T15:53:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.097+0000 7feee9d3f700 1 -- 172.21.15.96:0/4264278340 wait complete. 2024-04-22T15:53:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.201+0000 7f4ff1241700 1 -- 172.21.15.96:0/1104440836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 msgr2=0x7f4fec105f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.201+0000 7f4ff1241700 1 --2- 172.21.15.96:0/1104440836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 0x7f4fec105f50 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009910 tx=0x7f4fdc009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.201+0000 7f4ff1241700 1 -- 172.21.15.96:0/1104440836 shutdown_connections 2024-04-22T15:53:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.201+0000 7f4ff1241700 1 --2- 172.21.15.96:0/1104440836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fec0ffae0 0x7f4fec0fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.201+0000 7f4ff1241700 1 --2- 172.21.15.96:0/1104440836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 0x7f4fec105f50 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.201+0000 7f4ff1241700 1 -- 172.21.15.96:0/1104440836 >> 172.21.15.96:0/1104440836 conn(0x7f4fec0fb5f0 msgr2=0x7f4fec0fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.201+0000 7f4ff1241700 1 -- 172.21.15.96:0/1104440836 shutdown_connections 2024-04-22T15:53:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.201+0000 7f4ff1241700 1 -- 172.21.15.96:0/1104440836 wait complete. 2024-04-22T15:53:15.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.202+0000 7f4ff1241700 1 Processor -- start 2024-04-22T15:53:15.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.202+0000 7f4ff1241700 1 -- start start 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.202+0000 7f4ff1241700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fec0ffae0 0x7f4fec197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.202+0000 7f4ff1241700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 0x7f4fec197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.202+0000 7f4ff1241700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec198400 con 0x7f4fec105b70 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.202+0000 7f4ff1241700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec198540 con 0x7f4fec0ffae0 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.202+0000 7f4fe3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 0x7f4fec197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.202+0000 7f4fe3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 0x7f4fec197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41390/0 (socket says 172.21.15.96:41390) 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.202+0000 7f4fe3fff700 1 -- 172.21.15.96:0/424127856 learned_addr learned my addr 172.21.15.96:0/424127856 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4fead9d700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fec0ffae0 0x7f4fec197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4fe3fff700 1 -- 172.21.15.96:0/424127856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fec0ffae0 msgr2=0x7f4fec197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4fe3fff700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fec0ffae0 0x7f4fec197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4fe3fff700 1 -- 172.21.15.96:0/424127856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc0095f0 con 0x7f4fec105b70 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4fe3fff700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 0x7f4fec197ec0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400d8d0 tx=0x7f4fd400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4fe8d99700 1 -- 172.21.15.96:0/424127856 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fd4009940 con 0x7f4fec105b70 2024-04-22T15:53:15.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4fe8d99700 1 -- 172.21.15.96:0/424127856 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fd4010460 con 0x7f4fec105b70 2024-04-22T15:53:15.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec1a09f0 con 0x7f4fec105b70 2024-04-22T15:53:15.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4fe8d99700 1 -- 172.21.15.96:0/424127856 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fd400f5f0 con 0x7f4fec105b70 2024-04-22T15:53:15.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.203+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec1a0e00 con 0x7f4fec105b70 2024-04-22T15:53:15.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.204+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fec1080e0 con 0x7f4fec105b70 2024-04-22T15:53:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.205+0000 7f4fe8d99700 1 -- 172.21.15.96:0/424127856 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4fd40105d0 con 0x7f4fec105b70 2024-04-22T15:53:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.205+0000 7f4fe8d99700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd806ecc0 0x7f4fd8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.205+0000 7f4fe8d99700 1 -- 172.21.15.96:0/424127856 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4fd408edd0 con 0x7f4fec105b70 2024-04-22T15:53:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.205+0000 7f4fead9d700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd806ecc0 0x7f4fd8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.206+0000 7f4fead9d700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd806ecc0 0x7f4fd8071180 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00b5c0 tx=0x7f4fdc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.210+0000 7f4fe8d99700 1 -- 172.21.15.96:0/424127856 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4fd4057bb0 con 0x7f4fec105b70 2024-04-22T15:53:15.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.419+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fec198700 con 0x7f4fd806ecc0 2024-04-22T15:53:15.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.430+0000 7f4fe8d99700 1 -- 172.21.15.96:0/424127856 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4fec198700 con 0x7f4fd806ecc0 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (31m) 9m ago 43m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (43m) 9m ago 43m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (42m) 9m ago 42m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (29m) 9m ago 43m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (29m) 9m ago 42m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (31m) 9m ago 42m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (25m) 9m ago 37m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (25m) 9m ago 37m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (24m) 9m ago 37m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:53:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (9m) 9m ago 37m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (34m) 9m ago 44m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (33m) 9m ago 42m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (30m) 9m ago 44m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (29m) 9m ago 42m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (32m) 9m ago 43m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (32m) 9m ago 42m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (28m) 9m ago 41m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (28m) 9m ago 40m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (27m) 9m ago 40m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (26m) 9m ago 39m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (26m) 9m ago 39m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (25m) 9m ago 39m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:53:15.436 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (32m) 9m ago 42m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:53:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd806ecc0 msgr2=0x7f4fd8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd806ecc0 0x7f4fd8071180 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00b5c0 tx=0x7f4fdc01a040 comp rx=0 tx=0).stop 2024-04-22T15:53:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 msgr2=0x7f4fec197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 0x7f4fec197ec0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400d8d0 tx=0x7f4fd400dc90 comp rx=0 tx=0).stop 2024-04-22T15:53:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 shutdown_connections 2024-04-22T15:53:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4fd806ecc0 0x7f4fd8071180 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4fec0ffae0 0x7f4fec197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 --2- 172.21.15.96:0/424127856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fec105b70 0x7f4fec197ec0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 >> 172.21.15.96:0/424127856 conn(0x7f4fec0fb5f0 msgr2=0x7f4fec0fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 shutdown_connections 2024-04-22T15:53:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.432+0000 7f4ff1241700 1 -- 172.21.15.96:0/424127856 wait complete. 2024-04-22T15:53:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.536+0000 7f4529280700 1 -- 172.21.15.96:0/2003916123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4524100d90 msgr2=0x7f4524104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.536+0000 7f4529280700 1 --2- 172.21.15.96:0/2003916123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4524100d90 0x7f4524104760 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4518009910 tx=0x7f4518009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.536+0000 7f4529280700 1 -- 172.21.15.96:0/2003916123 shutdown_connections 2024-04-22T15:53:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.536+0000 7f4529280700 1 --2- 172.21.15.96:0/2003916123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4524100d90 0x7f4524104760 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.536+0000 7f4529280700 1 --2- 172.21.15.96:0/2003916123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45241003e0 0x7f45241007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.536+0000 7f4529280700 1 -- 172.21.15.96:0/2003916123 >> 172.21.15.96:0/2003916123 conn(0x7f4524075f30 msgr2=0x7f45240fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.536+0000 7f4529280700 1 -- 172.21.15.96:0/2003916123 shutdown_connections 2024-04-22T15:53:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.536+0000 7f4529280700 1 -- 172.21.15.96:0/2003916123 wait complete. 2024-04-22T15:53:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4529280700 1 Processor -- start 2024-04-22T15:53:15.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4529280700 1 -- start start 2024-04-22T15:53:15.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4529280700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45241003e0 0x7f452419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4529280700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4524100d90 0x7f452419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4529280700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452419d380 con 0x7f45241003e0 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4529280700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524196960 con 0x7f4524100d90 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4522ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45241003e0 0x7f452419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4522ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45241003e0 0x7f452419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41416/0 (socket says 172.21.15.96:41416) 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4522ffd700 1 -- 172.21.15.96:0/3230719214 learned_addr learned my addr 172.21.15.96:0/3230719214 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f45227fc700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4524100d90 0x7f452419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4522ffd700 1 -- 172.21.15.96:0/3230719214 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4524100d90 msgr2=0x7f452419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4522ffd700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4524100d90 0x7f452419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.537+0000 7f4522ffd700 1 -- 172.21.15.96:0/3230719214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45180095f0 con 0x7f45241003e0 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.538+0000 7f4522ffd700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45241003e0 0x7f452419c820 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f451c00ef10 tx=0x7f451c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.538+0000 7f450bfff700 1 -- 172.21.15.96:0/3230719214 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f451c00ca50 con 0x7f45241003e0 2024-04-22T15:53:15.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.538+0000 7f450bfff700 1 -- 172.21.15.96:0/3230719214 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f451c00f040 con 0x7f45241003e0 2024-04-22T15:53:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.538+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4524196be0 con 0x7f45241003e0 2024-04-22T15:53:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.538+0000 7f450bfff700 1 -- 172.21.15.96:0/3230719214 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f451c0146b0 con 0x7f45241003e0 2024-04-22T15:53:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.538+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4524197130 con 0x7f45241003e0 2024-04-22T15:53:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.539+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45241013a0 con 0x7f45241003e0 2024-04-22T15:53:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.539+0000 7f450bfff700 1 -- 172.21.15.96:0/3230719214 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f451c00cbb0 con 0x7f45241003e0 2024-04-22T15:53:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.539+0000 7f450bfff700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f450c06ecc0 0x7f450c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.540+0000 7f450bfff700 1 -- 172.21.15.96:0/3230719214 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f451c08f2c0 con 0x7f45241003e0 2024-04-22T15:53:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.540+0000 7f45227fc700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f450c06ecc0 0x7f450c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.540+0000 7f45227fc700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f450c06ecc0 0x7f450c071180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4524197b20 tx=0x7f4518005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.544+0000 7f450bfff700 1 -- 172.21.15.96:0/3230719214 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f451c05bb50 con 0x7f45241003e0 2024-04-22T15:53:15.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.829+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f452410cda0 con 0x7f45241003e0 2024-04-22T15:53:15.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.829+0000 7f450bfff700 1 -- 172.21.15.96:0/3230719214 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f451c05b6e0 con 0x7f45241003e0 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:53:15.833 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:53:15.834 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:53:15.834 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:53:15.834 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f450c06ecc0 msgr2=0x7f450c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f450c06ecc0 0x7f450c071180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4524197b20 tx=0x7f4518005940 comp rx=0 tx=0).stop 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45241003e0 msgr2=0x7f452419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45241003e0 0x7f452419c820 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f451c00ef10 tx=0x7f451c00c3a0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 shutdown_connections 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f450c06ecc0 0x7f450c071180 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45241003e0 0x7f452419c820 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 --2- 172.21.15.96:0/3230719214 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4524100d90 0x7f452419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 >> 172.21.15.96:0/3230719214 conn(0x7f4524075f30 msgr2=0x7f45240fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 shutdown_connections 2024-04-22T15:53:15.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.832+0000 7f4529280700 1 -- 172.21.15.96:0/3230719214 wait complete. 2024-04-22T15:53:15.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 -- 172.21.15.96:0/3046341604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 msgr2=0x7f76b4105e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 --2- 172.21.15.96:0/3046341604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 0x7f76b4105e60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009910 tx=0x7f76a4009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:15.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 -- 172.21.15.96:0/3046341604 shutdown_connections 2024-04-22T15:53:15.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 --2- 172.21.15.96:0/3046341604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76b40ffa80 0x7f76b40fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 --2- 172.21.15.96:0/3046341604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 0x7f76b4105e60 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 -- 172.21.15.96:0/3046341604 >> 172.21.15.96:0/3046341604 conn(0x7f76b40fb5b0 msgr2=0x7f76b40fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 -- 172.21.15.96:0/3046341604 shutdown_connections 2024-04-22T15:53:15.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 -- 172.21.15.96:0/3046341604 wait complete. 2024-04-22T15:53:15.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 Processor -- start 2024-04-22T15:53:15.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.935+0000 7f76ba942700 1 -- start start 2024-04-22T15:53:15.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76ba942700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76b40ffa80 0x7f76b4197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76ba942700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 0x7f76b4197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76ba942700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b4198430 con 0x7f76b4105a80 2024-04-22T15:53:15.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76ba942700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b4198570 con 0x7f76b40ffa80 2024-04-22T15:53:15.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 0x7f76b4197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 0x7f76b4197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41440/0 (socket says 172.21.15.96:41440) 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b37fe700 1 -- 172.21.15.96:0/253607238 learned_addr learned my addr 172.21.15.96:0/253607238 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b3fff700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76b40ffa80 0x7f76b4197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b37fe700 1 -- 172.21.15.96:0/253607238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76b40ffa80 msgr2=0x7f76b4197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b37fe700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76b40ffa80 0x7f76b4197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b37fe700 1 -- 172.21.15.96:0/253607238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a40095f0 con 0x7f76b4105a80 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b37fe700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 0x7f76b4197e60 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f76a800d8d0 tx=0x7f76a800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b17fa700 1 -- 172.21.15.96:0/253607238 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76a80098a0 con 0x7f76b4105a80 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b41a0990 con 0x7f76b4105a80 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b17fa700 1 -- 172.21.15.96:0/253607238 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76a8010460 con 0x7f76b4105a80 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.936+0000 7f76b17fa700 1 -- 172.21.15.96:0/253607238 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76a800f5f0 con 0x7f76b4105a80 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.937+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b41a0da0 con 0x7f76b4105a80 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.938+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b4108080 con 0x7f76b4105a80 2024-04-22T15:53:15.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.938+0000 7f76b17fa700 1 -- 172.21.15.96:0/253607238 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f76a8009a00 con 0x7f76b4105a80 2024-04-22T15:53:15.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.938+0000 7f76b17fa700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f769c06ecc0 0x7f769c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.939+0000 7f76b17fa700 1 -- 172.21.15.96:0/253607238 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f76a808ed80 con 0x7f76b4105a80 2024-04-22T15:53:15.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.939+0000 7f76b3fff700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f769c06ecc0 0x7f769c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.939+0000 7f76b3fff700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f769c06ecc0 0x7f769c071180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f76a4000c00 tx=0x7f76a401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:15.943+0000 7f76b17fa700 1 -- 172.21.15.96:0/253607238 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f76a8057ae0 con 0x7f76b4105a80 2024-04-22T15:53:16.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.189+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f76b41011c0 con 0x7f76b4105a80 2024-04-22T15:53:16.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.190+0000 7f76b17fa700 1 -- 172.21.15.96:0/253607238 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f76a805b1a0 con 0x7f76b4105a80 2024-04-22T15:53:16.193 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:53:16.193 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:53:16.193 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:16.193 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:53:16.193 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:53:16.194 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:53:16.195 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:53:16.196 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:53:16.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f769c06ecc0 msgr2=0x7f769c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f769c06ecc0 0x7f769c071180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f76a4000c00 tx=0x7f76a401a040 comp rx=0 tx=0).stop 2024-04-22T15:53:16.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 msgr2=0x7f76b4197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 0x7f76b4197e60 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f76a800d8d0 tx=0x7f76a800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 shutdown_connections 2024-04-22T15:53:16.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f769c06ecc0 0x7f769c071180 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76b40ffa80 0x7f76b4197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 --2- 172.21.15.96:0/253607238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76b4105a80 0x7f76b4197e60 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 >> 172.21.15.96:0/253607238 conn(0x7f76b40fb5b0 msgr2=0x7f76b40fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 shutdown_connections 2024-04-22T15:53:16.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.193+0000 7f76ba942700 1 -- 172.21.15.96:0/253607238 wait complete. 2024-04-22T15:53:16.198 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.297+0000 7f7ad9016700 1 -- 172.21.15.96:0/1940421638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 msgr2=0x7f7ad4105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.297+0000 7f7ad9016700 1 --2- 172.21.15.96:0/1940421638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 0x7f7ad4105810 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009910 tx=0x7f7abc009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.297+0000 7f7ad9016700 1 -- 172.21.15.96:0/1940421638 shutdown_connections 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.297+0000 7f7ad9016700 1 --2- 172.21.15.96:0/1940421638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 0x7f7ad4105810 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.297+0000 7f7ad9016700 1 --2- 172.21.15.96:0/1940421638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ad4100e10 0x7f7ad41011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.297+0000 7f7ad9016700 1 -- 172.21.15.96:0/1940421638 >> 172.21.15.96:0/1940421638 conn(0x7f7ad40fc6e0 msgr2=0x7f7ad40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.297+0000 7f7ad9016700 1 -- 172.21.15.96:0/1940421638 shutdown_connections 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.297+0000 7f7ad9016700 1 -- 172.21.15.96:0/1940421638 wait complete. 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7ad9016700 1 Processor -- start 2024-04-22T15:53:16.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7ad9016700 1 -- start start 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7ad9016700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ad4100e10 0x7f7ad419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7ad9016700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 0x7f7ad419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7ad9016700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad419d380 con 0x7f7ad41017c0 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7ad9016700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad41969b0 con 0x7f7ad4100e10 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7acbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 0x7f7ad419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7acbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 0x7f7ad419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41460/0 (socket says 172.21.15.96:41460) 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7acbfff700 1 -- 172.21.15.96:0/996418432 learned_addr learned my addr 172.21.15.96:0/996418432 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7ad2d9d700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ad4100e10 0x7f7ad419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7acbfff700 1 -- 172.21.15.96:0/996418432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ad4100e10 msgr2=0x7f7ad419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7acbfff700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ad4100e10 0x7f7ad419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.298+0000 7f7acbfff700 1 -- 172.21.15.96:0/996418432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc0095f0 con 0x7f7ad41017c0 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.299+0000 7f7acbfff700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 0x7f7ad419cd60 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00bd70 tx=0x7f7abc00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.299+0000 7f7ad0d99700 1 -- 172.21.15.96:0/996418432 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7abc01d070 con 0x7f7ad41017c0 2024-04-22T15:53:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.299+0000 7f7ad0d99700 1 -- 172.21.15.96:0/996418432 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7abc0056e0 con 0x7f7ad41017c0 2024-04-22T15:53:16.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.299+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad4196c30 con 0x7f7ad41017c0 2024-04-22T15:53:16.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.299+0000 7f7ad0d99700 1 -- 172.21.15.96:0/996418432 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7abc00fe50 con 0x7f7ad41017c0 2024-04-22T15:53:16.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.299+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad4197120 con 0x7f7ad41017c0 2024-04-22T15:53:16.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.300+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad4109210 con 0x7f7ad41017c0 2024-04-22T15:53:16.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.300+0000 7f7ad0d99700 1 -- 172.21.15.96:0/996418432 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7abc00f680 con 0x7f7ad41017c0 2024-04-22T15:53:16.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.301+0000 7f7ad0d99700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7ac006ecc0 0x7f7ac0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.301+0000 7f7ad0d99700 1 -- 172.21.15.96:0/996418432 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7abc090260 con 0x7f7ad41017c0 2024-04-22T15:53:16.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.301+0000 7f7ad2d9d700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7ac006ecc0 0x7f7ac0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.301+0000 7f7ad2d9d700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7ac006ecc0 0x7f7ac0071180 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4005950 tx=0x7f7ac400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:16.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.305+0000 7f7ad0d99700 1 -- 172.21.15.96:0/996418432 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7abc05ca50 con 0x7f7ad41017c0 2024-04-22T15:53:16.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.525+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ad4197e20 con 0x7f7ac006ecc0 2024-04-22T15:53:16.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.527+0000 7f7ad0d99700 1 -- 172.21.15.96:0/996418432 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7ad4197e20 con 0x7f7ac006ecc0 2024-04-22T15:53:16.529 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:53:16.529 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:53:16.530 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7ac006ecc0 msgr2=0x7f7ac0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7ac006ecc0 0x7f7ac0071180 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4005950 tx=0x7f7ac400b410 comp rx=0 tx=0).stop 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 msgr2=0x7f7ad419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 0x7f7ad419cd60 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00bd70 tx=0x7f7abc00bda0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 shutdown_connections 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7ac006ecc0 0x7f7ac0071180 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ad4100e10 0x7f7ad419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 --2- 172.21.15.96:0/996418432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ad41017c0 0x7f7ad419cd60 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 >> 172.21.15.96:0/996418432 conn(0x7f7ad40fc6e0 msgr2=0x7f7ad40fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 shutdown_connections 2024-04-22T15:53:16.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.529+0000 7f7ad9016700 1 -- 172.21.15.96:0/996418432 wait complete. 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.632+0000 7f95729ba700 1 -- 172.21.15.96:0/3344091013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c105ab0 msgr2=0x7f956c105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.632+0000 7f95729ba700 1 --2- 172.21.15.96:0/3344091013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c105ab0 0x7f956c105e90 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9560009910 tx=0x7f9560009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.632+0000 7f95729ba700 1 -- 172.21.15.96:0/3344091013 shutdown_connections 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.632+0000 7f95729ba700 1 --2- 172.21.15.96:0/3344091013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f956c0ffab0 0x7f956c0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.632+0000 7f95729ba700 1 --2- 172.21.15.96:0/3344091013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c105ab0 0x7f956c105e90 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.632+0000 7f95729ba700 1 -- 172.21.15.96:0/3344091013 >> 172.21.15.96:0/3344091013 conn(0x7f956c0fb5e0 msgr2=0x7f956c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.632+0000 7f95729ba700 1 -- 172.21.15.96:0/3344091013 shutdown_connections 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.632+0000 7f95729ba700 1 -- 172.21.15.96:0/3344091013 wait complete. 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f95729ba700 1 Processor -- start 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f95729ba700 1 -- start start 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f95729ba700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ffab0 0x7f956c19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f95729ba700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f956c105ab0 0x7f956c19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f95729ba700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c19d380 con 0x7f956c0ffab0 2024-04-22T15:53:16.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f95729ba700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c1969f0 con 0x7f956c105ab0 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f956bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ffab0 0x7f956c19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f956bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ffab0 0x7f956c19c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41480/0 (socket says 172.21.15.96:41480) 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f956bfff700 1 -- 172.21.15.96:0/2277544036 learned_addr learned my addr 172.21.15.96:0/2277544036 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f956b7fe700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f956c105ab0 0x7f956c19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f956bfff700 1 -- 172.21.15.96:0/2277544036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f956c105ab0 msgr2=0x7f956c19cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f956bfff700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f956c105ab0 0x7f956c19cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f956bfff700 1 -- 172.21.15.96:0/2277544036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600095f0 con 0x7f956c0ffab0 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.633+0000 7f956b7fe700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f956c105ab0 0x7f956c19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.634+0000 7f956bfff700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ffab0 0x7f956c19c8b0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f9560006010 tx=0x7f956000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.634+0000 7f95697fa700 1 -- 172.21.15.96:0/2277544036 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f956001d070 con 0x7f956c0ffab0 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.634+0000 7f95697fa700 1 -- 172.21.15.96:0/2277544036 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f956000f460 con 0x7f956c0ffab0 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.634+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956c196c70 con 0x7f956c0ffab0 2024-04-22T15:53:16.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.634+0000 7f95697fa700 1 -- 172.21.15.96:0/2277544036 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9560017630 con 0x7f956c0ffab0 2024-04-22T15:53:16.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.634+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956c197160 con 0x7f956c0ffab0 2024-04-22T15:53:16.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.635+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c1080b0 con 0x7f956c0ffab0 2024-04-22T15:53:16.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.635+0000 7f95697fa700 1 -- 172.21.15.96:0/2277544036 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f956000f5d0 con 0x7f956c0ffab0 2024-04-22T15:53:16.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.636+0000 7f95697fa700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955406ecc0 0x7f9554071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.636+0000 7f95697fa700 1 -- 172.21.15.96:0/2277544036 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9560090330 con 0x7f956c0ffab0 2024-04-22T15:53:16.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.636+0000 7f956b7fe700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955406ecc0 0x7f9554071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.636+0000 7f956b7fe700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955406ecc0 0x7f9554071180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f955c005fd0 tx=0x7f955c005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:16.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.640+0000 7f95697fa700 1 -- 172.21.15.96:0/2277544036 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f956005cc80 con 0x7f956c0ffab0 2024-04-22T15:53:16.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.921+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f956c197eb0 con 0x7f956c0ffab0 2024-04-22T15:53:16.925 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:16 smithi096 ceph-mon[112443]: from='client.35712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.925 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:16 smithi096 ceph-mon[112443]: from='client.35716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.925 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:16 smithi096 ceph-mon[112443]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.925 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:16 smithi096 ceph-mon[112443]: pgmap v943: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:16.925 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:16 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3230719214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:16.925 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:16 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/253607238' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:53:16.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.921+0000 7f95697fa700 1 -- 172.21.15.96:0/2277544036 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f956005c810 con 0x7f956c0ffab0 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:53:16.926 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:53:16.927 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:53:16.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.923+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955406ecc0 msgr2=0x7f9554071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.923+0000 7f95729ba700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955406ecc0 0x7f9554071180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f955c005fd0 tx=0x7f955c005e20 comp rx=0 tx=0).stop 2024-04-22T15:53:16.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.924+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ffab0 msgr2=0x7f956c19c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.924+0000 7f95729ba700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ffab0 0x7f956c19c8b0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f9560006010 tx=0x7f956000be20 comp rx=0 tx=0).stop 2024-04-22T15:53:16.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.924+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 shutdown_connections 2024-04-22T15:53:16.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.924+0000 7f95729ba700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f955406ecc0 0x7f9554071180 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.924+0000 7f95729ba700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ffab0 0x7f956c19c8b0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.924+0000 7f95729ba700 1 --2- 172.21.15.96:0/2277544036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f956c105ab0 0x7f956c19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.924+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 >> 172.21.15.96:0/2277544036 conn(0x7f956c0fb5e0 msgr2=0x7f956c109ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.924+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 shutdown_connections 2024-04-22T15:53:16.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:16.924+0000 7f95729ba700 1 -- 172.21.15.96:0/2277544036 wait complete. 2024-04-22T15:53:16.985 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:16 smithi196 ceph-mon[81815]: from='client.35712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.985 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:16 smithi196 ceph-mon[81815]: from='client.35716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.986 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:16 smithi196 ceph-mon[81815]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.986 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:16 smithi196 ceph-mon[81815]: pgmap v943: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:16.986 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:16 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3230719214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:16.986 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:16 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/253607238' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:53:17.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:17 smithi096 ceph-mon[112443]: from='client.35732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:17.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:17 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2277544036' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:53:17.983 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:17 smithi096 ceph-mon[112443]: pgmap v944: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:17 smithi196 ceph-mon[81815]: from='client.35732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:17 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2277544036' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:53:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:17 smithi196 ceph-mon[81815]: pgmap v944: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:20.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:20 smithi096 ceph-mon[112443]: pgmap v945: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:20 smithi196 ceph-mon[81815]: pgmap v945: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:22.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:22 smithi096 ceph-mon[112443]: pgmap v946: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:22 smithi196 ceph-mon[81815]: pgmap v946: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:24.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:24 smithi096 ceph-mon[112443]: pgmap v947: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:24 smithi196 ceph-mon[81815]: pgmap v947: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:25.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:25 smithi096 ceph-mon[112443]: pgmap v948: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:25 smithi196 ceph-mon[81815]: pgmap v948: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:28.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:28 smithi096 ceph-mon[112443]: pgmap v949: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:29.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:28 smithi196 ceph-mon[81815]: pgmap v949: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:30.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:30 smithi096 ceph-mon[112443]: pgmap v950: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:31.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:30 smithi196 ceph-mon[81815]: pgmap v950: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:31.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:31 smithi096 ceph-mon[112443]: pgmap v951: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:32.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:31 smithi196 ceph-mon[81815]: pgmap v951: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:34.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:34 smithi096 ceph-mon[112443]: pgmap v952: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:35.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:34 smithi196 ceph-mon[81815]: pgmap v952: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:35 smithi196 ceph-mon[81815]: pgmap v953: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:36.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:35 smithi096 ceph-mon[112443]: pgmap v953: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:38.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:38 smithi096 ceph-mon[112443]: pgmap v954: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:38.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:38 smithi196 ceph-mon[81815]: pgmap v954: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:39.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:39 smithi096 ceph-mon[112443]: pgmap v955: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:40.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:39 smithi196 ceph-mon[81815]: pgmap v955: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:42 smithi096 ceph-mon[112443]: pgmap v956: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:42 smithi196 ceph-mon[81815]: pgmap v956: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:43.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:43 smithi096 ceph-mon[112443]: pgmap v957: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:43 smithi196 ceph-mon[81815]: pgmap v957: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:45 smithi196 ceph-mon[81815]: pgmap v958: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:46.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:45 smithi096 ceph-mon[112443]: pgmap v958: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.041+0000 7f9de7fa4700 1 -- 172.21.15.96:0/457991349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 msgr2=0x7f9de0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.041+0000 7f9de7fa4700 1 --2- 172.21.15.96:0/457991349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 0x7f9de0102140 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000b0a0 tx=0x7f9dd000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.041+0000 7f9de7fa4700 1 -- 172.21.15.96:0/457991349 shutdown_connections 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.041+0000 7f9de7fa4700 1 --2- 172.21.15.96:0/457991349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9de0102680 0x7f9de010ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.041+0000 7f9de7fa4700 1 --2- 172.21.15.96:0/457991349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 0x7f9de0102140 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.041+0000 7f9de7fa4700 1 -- 172.21.15.96:0/457991349 >> 172.21.15.96:0/457991349 conn(0x7f9de0076740 msgr2=0x7f9de00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.041+0000 7f9de7fa4700 1 -- 172.21.15.96:0/457991349 shutdown_connections 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.041+0000 7f9de7fa4700 1 -- 172.21.15.96:0/457991349 wait complete. 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de7fa4700 1 Processor -- start 2024-04-22T15:53:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de7fa4700 1 -- start start 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de7fa4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 0x7f9de00ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de7fa4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9de0102680 0x7f9de00fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de7fa4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0100440 con 0x7f9de0101d60 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de7fa4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0100580 con 0x7f9de0102680 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de5d40700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 0x7f9de00ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de5d40700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 0x7f9de00ff9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44092/0 (socket says 172.21.15.96:44092) 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de5d40700 1 -- 172.21.15.96:0/2674064687 learned_addr learned my addr 172.21.15.96:0/2674064687 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de5d40700 1 -- 172.21.15.96:0/2674064687 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9de0102680 msgr2=0x7f9de00fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de5d40700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9de0102680 0x7f9de00fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.042+0000 7f9de5d40700 1 -- 172.21.15.96:0/2674064687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd0009d00 con 0x7f9de0101d60 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.043+0000 7f9de5d40700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 0x7f9de00ff9c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0015040 tx=0x7f9dd0003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.043+0000 7f9ddeffd700 1 -- 172.21.15.96:0/2674064687 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd000e040 con 0x7f9de0101d60 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.043+0000 7f9ddeffd700 1 -- 172.21.15.96:0/2674064687 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dd0007cd0 con 0x7f9de0101d60 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.043+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de0100800 con 0x7f9de0101d60 2024-04-22T15:53:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.043+0000 7f9ddeffd700 1 -- 172.21.15.96:0/2674064687 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd001d6b0 con 0x7f9de0101d60 2024-04-22T15:53:47.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.043+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de01a2f80 con 0x7f9de0101d60 2024-04-22T15:53:47.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.044+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de0068ea0 con 0x7f9de0101d60 2024-04-22T15:53:47.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.044+0000 7f9ddeffd700 1 -- 172.21.15.96:0/2674064687 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9dd0004040 con 0x7f9de0101d60 2024-04-22T15:53:47.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.045+0000 7f9ddeffd700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dc806ecc0 0x7f9dc8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.045+0000 7f9ddeffd700 1 -- 172.21.15.96:0/2674064687 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9dd0090630 con 0x7f9de0101d60 2024-04-22T15:53:47.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.045+0000 7f9de553f700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dc806ecc0 0x7f9dc8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.045+0000 7f9de553f700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dc806ecc0 0x7f9dc8071180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9de01010f0 tx=0x7f9dd4005d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.050+0000 7f9ddeffd700 1 -- 172.21.15.96:0/2674064687 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9dd0059390 con 0x7f9de0101d60 2024-04-22T15:53:47.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.268+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9de010dde0 con 0x7f9dc806ecc0 2024-04-22T15:53:47.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.270+0000 7f9ddeffd700 1 -- 172.21.15.96:0/2674064687 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9de010dde0 con 0x7f9dc806ecc0 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dc806ecc0 msgr2=0x7f9dc8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dc806ecc0 0x7f9dc8071180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9de01010f0 tx=0x7f9dd4005d20 comp rx=0 tx=0).stop 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 msgr2=0x7f9de00ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 0x7f9de00ff9c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0015040 tx=0x7f9dd0003ce0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 shutdown_connections 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f9dc806ecc0 0x7f9dc8071180 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de0101d60 0x7f9de00ff9c0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 --2- 172.21.15.96:0/2674064687 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9de0102680 0x7f9de00fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 >> 172.21.15.96:0/2674064687 conn(0x7f9de0076740 msgr2=0x7f9de0103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 shutdown_connections 2024-04-22T15:53:47.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.272+0000 7f9de7fa4700 1 -- 172.21.15.96:0/2674064687 wait complete. 2024-04-22T15:53:47.285 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:53:47.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.375+0000 7f7b099f4700 1 -- 172.21.15.96:0/3088156291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b041017d0 msgr2=0x7f7b04105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.375+0000 7f7b099f4700 1 --2- 172.21.15.96:0/3088156291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b041017d0 0x7f7b04105820 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009910 tx=0x7f7af8009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:47.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.376+0000 7f7b099f4700 1 -- 172.21.15.96:0/3088156291 shutdown_connections 2024-04-22T15:53:47.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.376+0000 7f7b099f4700 1 --2- 172.21.15.96:0/3088156291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b041017d0 0x7f7b04105820 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.376+0000 7f7b099f4700 1 --2- 172.21.15.96:0/3088156291 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b04100e20 0x7f7b04101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.376+0000 7f7b099f4700 1 -- 172.21.15.96:0/3088156291 >> 172.21.15.96:0/3088156291 conn(0x7f7b040fc6d0 msgr2=0x7f7b040feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.376+0000 7f7b099f4700 1 -- 172.21.15.96:0/3088156291 shutdown_connections 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.376+0000 7f7b099f4700 1 -- 172.21.15.96:0/3088156291 wait complete. 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.376+0000 7f7b099f4700 1 Processor -- start 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.376+0000 7f7b099f4700 1 -- start start 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b099f4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b04100e20 0x7f7b040731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b099f4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b041017d0 0x7f7b04073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b099f4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b04073c60 con 0x7f7b04100e20 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b099f4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b04073da0 con 0x7f7b041017d0 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b02ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b04100e20 0x7f7b040731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b02ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b04100e20 0x7f7b040731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44098/0 (socket says 172.21.15.96:44098) 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b02ffd700 1 -- 172.21.15.96:0/1966223051 learned_addr learned my addr 172.21.15.96:0/1966223051 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:47.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b027fc700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b041017d0 0x7f7b04073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b02ffd700 1 -- 172.21.15.96:0/1966223051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b041017d0 msgr2=0x7f7b04073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b02ffd700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b041017d0 0x7f7b04073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b02ffd700 1 -- 172.21.15.96:0/1966223051 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af80095f0 con 0x7f7b04100e20 2024-04-22T15:53:47.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b02ffd700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b04100e20 0x7f7b040731e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7af400d8d0 tx=0x7f7af400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.377+0000 7f7b089f2700 1 -- 172.21.15.96:0/1966223051 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7af40098a0 con 0x7f7b04100e20 2024-04-22T15:53:47.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.378+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b041a2ce0 con 0x7f7b04100e20 2024-04-22T15:53:47.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.378+0000 7f7b089f2700 1 -- 172.21.15.96:0/1966223051 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7af4010460 con 0x7f7b04100e20 2024-04-22T15:53:47.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.378+0000 7f7b089f2700 1 -- 172.21.15.96:0/1966223051 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7af400f5f0 con 0x7f7b04100e20 2024-04-22T15:53:47.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.378+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b041a3040 con 0x7f7b04100e20 2024-04-22T15:53:47.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.379+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b04109220 con 0x7f7b04100e20 2024-04-22T15:53:47.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.379+0000 7f7b089f2700 1 -- 172.21.15.96:0/1966223051 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7af400f820 con 0x7f7b04100e20 2024-04-22T15:53:47.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.379+0000 7f7b089f2700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7aec06ecc0 0x7f7aec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.379+0000 7f7b089f2700 1 -- 172.21.15.96:0/1966223051 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7af408edd0 con 0x7f7b04100e20 2024-04-22T15:53:47.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.380+0000 7f7b027fc700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7aec06ecc0 0x7f7aec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.380+0000 7f7b027fc700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7aec06ecc0 0x7f7aec071180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7b04074910 tx=0x7f7af801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.384+0000 7f7b089f2700 1 -- 172.21.15.96:0/1966223051 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7af4057ba0 con 0x7f7b04100e20 2024-04-22T15:53:47.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.601+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b0410ddc0 con 0x7f7aec06ecc0 2024-04-22T15:53:47.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.602+0000 7f7b089f2700 1 -- 172.21.15.96:0/1966223051 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7b0410ddc0 con 0x7f7aec06ecc0 2024-04-22T15:53:47.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7aec06ecc0 msgr2=0x7f7aec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7aec06ecc0 0x7f7aec071180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7b04074910 tx=0x7f7af801a040 comp rx=0 tx=0).stop 2024-04-22T15:53:47.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b04100e20 msgr2=0x7f7b040731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b04100e20 0x7f7b040731e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7af400d8d0 tx=0x7f7af400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 shutdown_connections 2024-04-22T15:53:47.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7aec06ecc0 0x7f7aec071180 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b04100e20 0x7f7b040731e0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 --2- 172.21.15.96:0/1966223051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b041017d0 0x7f7b04073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 >> 172.21.15.96:0/1966223051 conn(0x7f7b040fc6d0 msgr2=0x7f7b040fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:47.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 shutdown_connections 2024-04-22T15:53:47.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.604+0000 7f7b099f4700 1 -- 172.21.15.96:0/1966223051 wait complete. 2024-04-22T15:53:47.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.708+0000 7f107020f700 1 -- 172.21.15.96:0/3708669776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ffd20 msgr2=0x7f1068105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.708+0000 7f107020f700 1 --2- 172.21.15.96:0/3708669776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ffd20 0x7f1068105860 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f1058009910 tx=0x7f1058009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:47.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.708+0000 7f107020f700 1 -- 172.21.15.96:0/3708669776 shutdown_connections 2024-04-22T15:53:47.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.708+0000 7f107020f700 1 --2- 172.21.15.96:0/3708669776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ffd20 0x7f1068105860 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.708+0000 7f107020f700 1 --2- 172.21.15.96:0/3708669776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10680ff370 0x7f10680ff750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.708+0000 7f107020f700 1 -- 172.21.15.96:0/3708669776 >> 172.21.15.96:0/3708669776 conn(0x7f1068075f40 msgr2=0x7f10680fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:47.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.708+0000 7f107020f700 1 -- 172.21.15.96:0/3708669776 shutdown_connections 2024-04-22T15:53:47.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.708+0000 7f107020f700 1 -- 172.21.15.96:0/3708669776 wait complete. 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f107020f700 1 Processor -- start 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f107020f700 1 -- start start 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f107020f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ff370 0x7f10681002f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f107020f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10680ffd20 0x7f1068100830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f107020f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1068100d70 con 0x7f10680ff370 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f107020f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1068100ee0 con 0x7f10680ffd20 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f106dfab700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ff370 0x7f10681002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f106dfab700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ff370 0x7f10681002f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44120/0 (socket says 172.21.15.96:44120) 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f106dfab700 1 -- 172.21.15.96:0/3597855188 learned_addr learned my addr 172.21.15.96:0/3597855188 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f106d7aa700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10680ffd20 0x7f1068100830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f106dfab700 1 -- 172.21.15.96:0/3597855188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10680ffd20 msgr2=0x7f1068100830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f106dfab700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10680ffd20 0x7f1068100830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f106dfab700 1 -- 172.21.15.96:0/3597855188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10580095f0 con 0x7f10680ff370 2024-04-22T15:53:47.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.709+0000 7f106d7aa700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10680ffd20 0x7f1068100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:53:47.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.710+0000 7f106dfab700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ff370 0x7f10681002f0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f106400b4b0 tx=0x7f106400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.710+0000 7f105effd700 1 -- 172.21.15.96:0/3597855188 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1064011800 con 0x7f10680ff370 2024-04-22T15:53:47.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.710+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10681a2e40 con 0x7f10680ff370 2024-04-22T15:53:47.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.710+0000 7f105effd700 1 -- 172.21.15.96:0/3597855188 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1064011e20 con 0x7f10680ff370 2024-04-22T15:53:47.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.710+0000 7f105effd700 1 -- 172.21.15.96:0/3597855188 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f106400f2e0 con 0x7f10680ff370 2024-04-22T15:53:47.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.710+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10681a31a0 con 0x7f10680ff370 2024-04-22T15:53:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.711+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1068109190 con 0x7f10680ff370 2024-04-22T15:53:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.711+0000 7f105effd700 1 -- 172.21.15.96:0/3597855188 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1064010380 con 0x7f10680ff370 2024-04-22T15:53:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.712+0000 7f105effd700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f105406ecd0 0x7f1054071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.712+0000 7f105effd700 1 -- 172.21.15.96:0/3597855188 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f106408eaa0 con 0x7f10680ff370 2024-04-22T15:53:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.712+0000 7f106d7aa700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f105406ecd0 0x7f1054071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.712+0000 7f106d7aa700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f105406ecd0 0x7f1054071190 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1068101d30 tx=0x7f105801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.717+0000 7f105effd700 1 -- 172.21.15.96:0/3597855188 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1064057800 con 0x7f10680ff370 2024-04-22T15:53:47.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.928+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f106810de80 con 0x7f105406ecd0 2024-04-22T15:53:47.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.938+0000 7f105effd700 1 -- 172.21.15.96:0/3597855188 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f106810de80 con 0x7f105406ecd0 2024-04-22T15:53:47.941 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:53:47.941 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (32m) 10m ago 43m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:53:47.941 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (44m) 10m ago 44m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (42m) 10m ago 42m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (29m) 10m ago 44m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (29m) 10m ago 42m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (32m) 10m ago 43m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (25m) 10m ago 38m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (25m) 10m ago 38m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (25m) 10m ago 38m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (10m) 10m ago 38m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (34m) 10m ago 45m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (33m) 10m ago 42m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (30m) 10m ago 45m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (30m) 10m ago 42m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (33m) 10m ago 43m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (33m) 10m ago 42m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:53:47.942 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (29m) 10m ago 41m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:53:47.943 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (28m) 10m ago 41m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:53:47.943 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (28m) 10m ago 40m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:53:47.943 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (27m) 10m ago 40m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:53:47.943 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (26m) 10m ago 40m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:53:47.943 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (26m) 10m ago 39m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:53:47.943 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (32m) 10m ago 43m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.940+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f105406ecd0 msgr2=0x7f1054071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.940+0000 7f107020f700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f105406ecd0 0x7f1054071190 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1068101d30 tx=0x7f105801a040 comp rx=0 tx=0).stop 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.940+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ff370 msgr2=0x7f10681002f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.940+0000 7f107020f700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ff370 0x7f10681002f0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f106400b4b0 tx=0x7f106400b870 comp rx=0 tx=0).stop 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.940+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 shutdown_connections 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.940+0000 7f107020f700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f105406ecd0 0x7f1054071190 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.940+0000 7f107020f700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10680ff370 0x7f10681002f0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.940+0000 7f107020f700 1 --2- 172.21.15.96:0/3597855188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10680ffd20 0x7f1068100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.940+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 >> 172.21.15.96:0/3597855188 conn(0x7f1068075f40 msgr2=0x7f1068104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:47.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.941+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 shutdown_connections 2024-04-22T15:53:47.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:47.941+0000 7f107020f700 1 -- 172.21.15.96:0/3597855188 wait complete. 2024-04-22T15:53:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:47 smithi096 ceph-mon[112443]: from='client.35740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:47 smithi096 ceph-mon[112443]: from='client.35744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:47 smithi096 ceph-mon[112443]: pgmap v959: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:47 smithi196 ceph-mon[81815]: from='client.35740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:47 smithi196 ceph-mon[81815]: from='client.35744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:47 smithi196 ceph-mon[81815]: pgmap v959: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.045+0000 7feb7febb700 1 -- 172.21.15.96:0/3608277746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb780ffab0 msgr2=0x7feb780fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.045+0000 7feb7febb700 1 --2- 172.21.15.96:0/3608277746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb780ffab0 0x7feb780fff30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7feb68009910 tx=0x7feb68009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.045+0000 7feb7febb700 1 -- 172.21.15.96:0/3608277746 shutdown_connections 2024-04-22T15:53:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.045+0000 7feb7febb700 1 --2- 172.21.15.96:0/3608277746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb780ffab0 0x7feb780fff30 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.045+0000 7feb7febb700 1 --2- 172.21.15.96:0/3608277746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb78105ab0 0x7feb78105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.045+0000 7feb7febb700 1 -- 172.21.15.96:0/3608277746 >> 172.21.15.96:0/3608277746 conn(0x7feb780fb5e0 msgr2=0x7feb780fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.045+0000 7feb7febb700 1 -- 172.21.15.96:0/3608277746 shutdown_connections 2024-04-22T15:53:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.045+0000 7feb7febb700 1 -- 172.21.15.96:0/3608277746 wait complete. 2024-04-22T15:53:48.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7febb700 1 Processor -- start 2024-04-22T15:53:48.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7febb700 1 -- start start 2024-04-22T15:53:48.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7febb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb780ffab0 0x7feb7810aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7febb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb78105ab0 0x7feb7810af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7febb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7810b4c0 con 0x7feb78105ab0 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7febb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7810b630 con 0x7feb780ffab0 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7d456700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb78105ab0 0x7feb7810af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7d456700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb78105ab0 0x7feb7810af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44134/0 (socket says 172.21.15.96:44134) 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7d456700 1 -- 172.21.15.96:0/3366529772 learned_addr learned my addr 172.21.15.96:0/3366529772 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7dc57700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb780ffab0 0x7feb7810aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7d456700 1 -- 172.21.15.96:0/3366529772 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb780ffab0 msgr2=0x7feb7810aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7d456700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb780ffab0 0x7feb7810aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7d456700 1 -- 172.21.15.96:0/3366529772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb680095f0 con 0x7feb78105ab0 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.046+0000 7feb7d456700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb78105ab0 0x7feb7810af80 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7feb68006010 tx=0x7feb6800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.047+0000 7feb6effd700 1 -- 172.21.15.96:0/3366529772 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb6801d070 con 0x7feb78105ab0 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.047+0000 7feb6effd700 1 -- 172.21.15.96:0/3366529772 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb6800f460 con 0x7feb78105ab0 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.047+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7810b8b0 con 0x7feb78105ab0 2024-04-22T15:53:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.047+0000 7feb6effd700 1 -- 172.21.15.96:0/3366529772 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb68017630 con 0x7feb78105ab0 2024-04-22T15:53:48.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.047+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb781a3090 con 0x7feb78105ab0 2024-04-22T15:53:48.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.048+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb781080d0 con 0x7feb78105ab0 2024-04-22T15:53:48.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.048+0000 7feb6effd700 1 -- 172.21.15.96:0/3366529772 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feb6800fab0 con 0x7feb78105ab0 2024-04-22T15:53:48.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.049+0000 7feb6effd700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb6406ecc0 0x7feb64071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.049+0000 7feb6effd700 1 -- 172.21.15.96:0/3366529772 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7feb68090200 con 0x7feb78105ab0 2024-04-22T15:53:48.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.049+0000 7feb7dc57700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb6406ecc0 0x7feb64071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.049+0000 7feb7dc57700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb6406ecc0 0x7feb64071180 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feb74006fd0 tx=0x7feb74008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.053+0000 7feb6effd700 1 -- 172.21.15.96:0/3366529772 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7feb68058f60 con 0x7feb78105ab0 2024-04-22T15:53:48.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.345+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feb7810bed0 con 0x7feb78105ab0 2024-04-22T15:53:48.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.345+0000 7feb6effd700 1 -- 172.21.15.96:0/3366529772 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7feb6805c620 con 0x7feb78105ab0 2024-04-22T15:53:48.348 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:53:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:53:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:53:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:53:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:53:48.349 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:53:48.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.347+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb6406ecc0 msgr2=0x7feb64071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.347+0000 7feb7febb700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb6406ecc0 0x7feb64071180 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feb74006fd0 tx=0x7feb74008040 comp rx=0 tx=0).stop 2024-04-22T15:53:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.347+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb78105ab0 msgr2=0x7feb7810af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.347+0000 7feb7febb700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb78105ab0 0x7feb7810af80 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7feb68006010 tx=0x7feb6800be20 comp rx=0 tx=0).stop 2024-04-22T15:53:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.347+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 shutdown_connections 2024-04-22T15:53:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.347+0000 7feb7febb700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7feb6406ecc0 0x7feb64071180 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.347+0000 7feb7febb700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb780ffab0 0x7feb7810aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.347+0000 7feb7febb700 1 --2- 172.21.15.96:0/3366529772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb78105ab0 0x7feb7810af80 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.347+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 >> 172.21.15.96:0/3366529772 conn(0x7feb780fb5e0 msgr2=0x7feb780fc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.348+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 shutdown_connections 2024-04-22T15:53:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.348+0000 7feb7febb700 1 -- 172.21.15.96:0/3366529772 wait complete. 2024-04-22T15:53:48.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.451+0000 7fdf4baf9700 1 -- 172.21.15.96:0/2265122019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 msgr2=0x7fdf44105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.451+0000 7fdf4baf9700 1 --2- 172.21.15.96:0/2265122019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 0x7fdf44105e90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009910 tx=0x7fdf40009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:48.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.451+0000 7fdf4baf9700 1 -- 172.21.15.96:0/2265122019 shutdown_connections 2024-04-22T15:53:48.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.451+0000 7fdf4baf9700 1 --2- 172.21.15.96:0/2265122019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf440ffab0 0x7fdf440fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.451+0000 7fdf4baf9700 1 --2- 172.21.15.96:0/2265122019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 0x7fdf44105e90 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.451+0000 7fdf4baf9700 1 -- 172.21.15.96:0/2265122019 >> 172.21.15.96:0/2265122019 conn(0x7fdf440fb5e0 msgr2=0x7fdf440fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.452+0000 7fdf4baf9700 1 -- 172.21.15.96:0/2265122019 shutdown_connections 2024-04-22T15:53:48.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.452+0000 7fdf4baf9700 1 -- 172.21.15.96:0/2265122019 wait complete. 2024-04-22T15:53:48.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.452+0000 7fdf4baf9700 1 Processor -- start 2024-04-22T15:53:48.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.452+0000 7fdf4baf9700 1 -- start start 2024-04-22T15:53:48.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.452+0000 7fdf4baf9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf440ffab0 0x7fdf44197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.452+0000 7fdf4baf9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 0x7fdf44197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.452+0000 7fdf4baf9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44198440 con 0x7fdf44105ab0 2024-04-22T15:53:48.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.452+0000 7fdf4baf9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44198580 con 0x7fdf440ffab0 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.452+0000 7fdf49094700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 0x7fdf44197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf49094700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 0x7fdf44197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44152/0 (socket says 172.21.15.96:44152) 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf49094700 1 -- 172.21.15.96:0/3100810155 learned_addr learned my addr 172.21.15.96:0/3100810155 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf49895700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf440ffab0 0x7fdf44197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf49094700 1 -- 172.21.15.96:0/3100810155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf440ffab0 msgr2=0x7fdf44197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf49094700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf440ffab0 0x7fdf44197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf49094700 1 -- 172.21.15.96:0/3100810155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf400095f0 con 0x7fdf44105ab0 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf49895700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf440ffab0 0x7fdf44197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf49094700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 0x7fdf44197e70 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400d8d0 tx=0x7fdf3400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf3affd700 1 -- 172.21.15.96:0/3100810155 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf34009940 con 0x7fdf44105ab0 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf3affd700 1 -- 172.21.15.96:0/3100810155 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf34010460 con 0x7fdf44105ab0 2024-04-22T15:53:48.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf441a09a0 con 0x7fdf44105ab0 2024-04-22T15:53:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf3affd700 1 -- 172.21.15.96:0/3100810155 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf3400f5f0 con 0x7fdf44105ab0 2024-04-22T15:53:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.453+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf441a0db0 con 0x7fdf44105ab0 2024-04-22T15:53:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.454+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf441080b0 con 0x7fdf44105ab0 2024-04-22T15:53:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.454+0000 7fdf3affd700 1 -- 172.21.15.96:0/3100810155 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdf340105d0 con 0x7fdf44105ab0 2024-04-22T15:53:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.455+0000 7fdf3affd700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3006ecc0 0x7fdf30071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.455+0000 7fdf3affd700 1 -- 172.21.15.96:0/3100810155 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fdf3408edc0 con 0x7fdf44105ab0 2024-04-22T15:53:48.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.455+0000 7fdf49895700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3006ecc0 0x7fdf30071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.455+0000 7fdf49895700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3006ecc0 0x7fdf30071180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdf40005f50 tx=0x7fdf40005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.460+0000 7fdf3affd700 1 -- 172.21.15.96:0/3100810155 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdf34057ba0 con 0x7fdf44105ab0 2024-04-22T15:53:48.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.708+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdf441011f0 con 0x7fdf44105ab0 2024-04-22T15:53:48.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.708+0000 7fdf3affd700 1 -- 172.21.15.96:0/3100810155 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7fdf34016770 con 0x7fdf44105ab0 2024-04-22T15:53:48.711 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:53:48.712 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:53:48.713 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:53:48.714 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:53:48.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3006ecc0 msgr2=0x7fdf30071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3006ecc0 0x7fdf30071180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdf40005f50 tx=0x7fdf40005dc0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 msgr2=0x7fdf44197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 0x7fdf44197e70 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400d8d0 tx=0x7fdf3400dc90 comp rx=0 tx=0).stop 2024-04-22T15:53:48.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 shutdown_connections 2024-04-22T15:53:48.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fdf3006ecc0 0x7fdf30071180 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf440ffab0 0x7fdf44197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 --2- 172.21.15.96:0/3100810155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf44105ab0 0x7fdf44197e70 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 >> 172.21.15.96:0/3100810155 conn(0x7fdf440fb5e0 msgr2=0x7fdf440fc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 shutdown_connections 2024-04-22T15:53:48.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.711+0000 7fdf4baf9700 1 -- 172.21.15.96:0/3100810155 wait complete. 2024-04-22T15:53:48.716 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:53:48.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.817+0000 7f68af3f3700 1 -- 172.21.15.96:0/2666880554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a81017c0 msgr2=0x7f68a8105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.817+0000 7f68af3f3700 1 --2- 172.21.15.96:0/2666880554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a81017c0 0x7f68a8105810 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6898009910 tx=0x7f6898009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:48.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.817+0000 7f68af3f3700 1 -- 172.21.15.96:0/2666880554 shutdown_connections 2024-04-22T15:53:48.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.817+0000 7f68af3f3700 1 --2- 172.21.15.96:0/2666880554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a81017c0 0x7f68a8105810 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.817+0000 7f68af3f3700 1 --2- 172.21.15.96:0/2666880554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68a8100e10 0x7f68a81011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.817+0000 7f68af3f3700 1 -- 172.21.15.96:0/2666880554 >> 172.21.15.96:0/2666880554 conn(0x7f68a80fc6e0 msgr2=0x7f68a80feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.817+0000 7f68af3f3700 1 -- 172.21.15.96:0/2666880554 shutdown_connections 2024-04-22T15:53:48.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.817+0000 7f68af3f3700 1 -- 172.21.15.96:0/2666880554 wait complete. 2024-04-22T15:53:48.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68af3f3700 1 Processor -- start 2024-04-22T15:53:48.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68af3f3700 1 -- start start 2024-04-22T15:53:48.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68af3f3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a8100e10 0x7f68a80731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68af3f3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68a81017c0 0x7f68a8073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68af3f3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8073c60 con 0x7f68a8100e10 2024-04-22T15:53:48.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68af3f3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8073da0 con 0x7f68a81017c0 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68ad18f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a8100e10 0x7f68a80731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68ad18f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a8100e10 0x7f68a80731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44158/0 (socket says 172.21.15.96:44158) 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68ad18f700 1 -- 172.21.15.96:0/2038348893 learned_addr learned my addr 172.21.15.96:0/2038348893 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68ac98e700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68a81017c0 0x7f68a8073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68ad18f700 1 -- 172.21.15.96:0/2038348893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68a81017c0 msgr2=0x7f68a8073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68ad18f700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68a81017c0 0x7f68a8073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68ad18f700 1 -- 172.21.15.96:0/2038348893 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68980095f0 con 0x7f68a8100e10 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.818+0000 7f68ad18f700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a8100e10 0x7f68a80731e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f68a400d8d0 tx=0x7f68a400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.819+0000 7f689e7fc700 1 -- 172.21.15.96:0/2038348893 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68a40098a0 con 0x7f68a8100e10 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.819+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a81a2d50 con 0x7f68a8100e10 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.819+0000 7f689e7fc700 1 -- 172.21.15.96:0/2038348893 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68a4010460 con 0x7f68a8100e10 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.819+0000 7f689e7fc700 1 -- 172.21.15.96:0/2038348893 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68a400f5f0 con 0x7f68a8100e10 2024-04-22T15:53:48.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.819+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a81a30b0 con 0x7f68a8100e10 2024-04-22T15:53:48.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.819+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a8109210 con 0x7f68a8100e10 2024-04-22T15:53:48.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.820+0000 7f689e7fc700 1 -- 172.21.15.96:0/2038348893 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f68a4009a00 con 0x7f68a8100e10 2024-04-22T15:53:48.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.820+0000 7f689e7fc700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f689406ecc0 0x7f6894071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.821+0000 7f689e7fc700 1 -- 172.21.15.96:0/2038348893 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f68a408ed60 con 0x7f68a8100e10 2024-04-22T15:53:48.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.821+0000 7f68ac98e700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f689406ecc0 0x7f6894071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.821+0000 7f68ac98e700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f689406ecc0 0x7f6894071180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f68a80748c0 tx=0x7f689801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:48.825+0000 7f689e7fc700 1 -- 172.21.15.96:0/2038348893 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f68a4057ac0 con 0x7f68a8100e10 2024-04-22T15:53:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:48 smithi096 ceph-mon[112443]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:48 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3366529772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:48 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3100810155' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:53:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:48 smithi196 ceph-mon[81815]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:48 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3366529772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:49.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:48 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3100810155' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:53:49.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.043+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f68a8073f60 con 0x7f689406ecc0 2024-04-22T15:53:49.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.045+0000 7f689e7fc700 1 -- 172.21.15.96:0/2038348893 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f68a8073f60 con 0x7f689406ecc0 2024-04-22T15:53:49.047 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:53:49.048 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:53:49.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f689406ecc0 msgr2=0x7f6894071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f689406ecc0 0x7f6894071180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f68a80748c0 tx=0x7f689801a040 comp rx=0 tx=0).stop 2024-04-22T15:53:49.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a8100e10 msgr2=0x7f68a80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a8100e10 0x7f68a80731e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f68a400d8d0 tx=0x7f68a400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 shutdown_connections 2024-04-22T15:53:49.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f689406ecc0 0x7f6894071180 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68a8100e10 0x7f68a80731e0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 --2- 172.21.15.96:0/2038348893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68a81017c0 0x7f68a8073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 >> 172.21.15.96:0/2038348893 conn(0x7f68a80fc6e0 msgr2=0x7f68a80fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:49.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 shutdown_connections 2024-04-22T15:53:49.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.047+0000 7f68af3f3700 1 -- 172.21.15.96:0/2038348893 wait complete. 2024-04-22T15:53:49.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.150+0000 7f72539ad700 1 -- 172.21.15.96:0/615475080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 msgr2=0x7f724c1020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.150+0000 7f72539ad700 1 --2- 172.21.15.96:0/615475080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 0x7f724c1020c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f724400b0a0 tx=0x7f724400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.150+0000 7f72539ad700 1 -- 172.21.15.96:0/615475080 shutdown_connections 2024-04-22T15:53:49.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.150+0000 7f72539ad700 1 --2- 172.21.15.96:0/615475080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f724c102690 0x7f724c10ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.150+0000 7f72539ad700 1 --2- 172.21.15.96:0/615475080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 0x7f724c1020c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.150+0000 7f72539ad700 1 -- 172.21.15.96:0/615475080 >> 172.21.15.96:0/615475080 conn(0x7f724c0fb5e0 msgr2=0x7f724c0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:49.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.150+0000 7f72539ad700 1 -- 172.21.15.96:0/615475080 shutdown_connections 2024-04-22T15:53:49.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.150+0000 7f72539ad700 1 -- 172.21.15.96:0/615475080 wait complete. 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f72539ad700 1 Processor -- start 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f72539ad700 1 -- start start 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f72539ad700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 0x7f724c198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f72539ad700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f724c102690 0x7f724c199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f72539ad700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c199ae0 con 0x7f724c101ce0 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f72539ad700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c19d7b0 con 0x7f724c102690 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f7251749700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 0x7f724c198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f7251749700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 0x7f724c198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44174/0 (socket says 172.21.15.96:44174) 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f7251749700 1 -- 172.21.15.96:0/1272989481 learned_addr learned my addr 172.21.15.96:0/1272989481 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f7251749700 1 -- 172.21.15.96:0/1272989481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f724c102690 msgr2=0x7f724c199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f7251749700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f724c102690 0x7f724c199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f7251749700 1 -- 172.21.15.96:0/1272989481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7244009d00 con 0x7f724c101ce0 2024-04-22T15:53:49.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.151+0000 7f7251749700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 0x7f724c198ec0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f724400b070 tx=0x7f7244007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:49.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.152+0000 7f72427fc700 1 -- 172.21.15.96:0/1272989481 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f724400e040 con 0x7f724c101ce0 2024-04-22T15:53:49.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.152+0000 7f72427fc700 1 -- 172.21.15.96:0/1272989481 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7244003de0 con 0x7f724c101ce0 2024-04-22T15:53:49.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.152+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724c19da30 con 0x7f724c101ce0 2024-04-22T15:53:49.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.152+0000 7f72427fc700 1 -- 172.21.15.96:0/1272989481 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f724401d6b0 con 0x7f724c101ce0 2024-04-22T15:53:49.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.152+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724c19df20 con 0x7f724c101ce0 2024-04-22T15:53:49.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.153+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724c10a6e0 con 0x7f724c101ce0 2024-04-22T15:53:49.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.153+0000 7f72427fc700 1 -- 172.21.15.96:0/1272989481 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7244004040 con 0x7f724c101ce0 2024-04-22T15:53:49.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.153+0000 7f72427fc700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f723806ecd0 0x7f7238071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.154+0000 7f72427fc700 1 -- 172.21.15.96:0/1272989481 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f72440906b0 con 0x7f724c101ce0 2024-04-22T15:53:49.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.154+0000 7f7250f48700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f723806ecd0 0x7f7238071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.154+0000 7f7250f48700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f723806ecd0 0x7f7238071190 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f724c19a420 tx=0x7f723c006d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:49.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.158+0000 7f72427fc700 1 -- 172.21.15.96:0/1272989481 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7244059410 con 0x7f724c101ce0 2024-04-22T15:53:49.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.442+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f724c19a1d0 con 0x7f724c101ce0 2024-04-22T15:53:49.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.442+0000 7f72427fc700 1 -- 172.21.15.96:0/1272989481 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f724405cad0 con 0x7f724c101ce0 2024-04-22T15:53:49.445 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:53:49.446 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:53:49.447 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f723806ecd0 msgr2=0x7f7238071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f723806ecd0 0x7f7238071190 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f724c19a420 tx=0x7f723c006d20 comp rx=0 tx=0).stop 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 msgr2=0x7f724c198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 0x7f724c198ec0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f724400b070 tx=0x7f7244007bd0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 shutdown_connections 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f723806ecd0 0x7f7238071190 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724c101ce0 0x7f724c198ec0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 --2- 172.21.15.96:0/1272989481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f724c102690 0x7f724c199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 >> 172.21.15.96:0/1272989481 conn(0x7f724c0fb5e0 msgr2=0x7f724c101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 shutdown_connections 2024-04-22T15:53:49.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:53:49.444+0000 7f72539ad700 1 -- 172.21.15.96:0/1272989481 wait complete. 2024-04-22T15:53:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:49 smithi196 ceph-mon[81815]: from='client.35760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:49 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1272989481' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:53:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:49 smithi196 ceph-mon[81815]: pgmap v960: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:50.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:49 smithi096 ceph-mon[112443]: from='client.35760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:50.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:49 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1272989481' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:53:50.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:49 smithi096 ceph-mon[112443]: pgmap v960: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:52.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:51 smithi196 ceph-mon[81815]: pgmap v961: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:52.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:51 smithi096 ceph-mon[112443]: pgmap v961: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:53.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:54.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:54 smithi096 ceph-mon[112443]: pgmap v962: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:54.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:54 smithi196 ceph-mon[81815]: pgmap v962: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:55 smithi196 ceph-mon[81815]: pgmap v963: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:56.114 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:55 smithi096 ceph-mon[112443]: pgmap v963: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:58.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:57 smithi196 ceph-mon[81815]: pgmap v964: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:58.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:57 smithi096 ceph-mon[112443]: pgmap v964: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:00.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:53:59 smithi196 ceph-mon[81815]: pgmap v965: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:00.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:53:59 smithi096 ceph-mon[112443]: pgmap v965: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:01 smithi196 ceph-mon[81815]: pgmap v966: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:02.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:01 smithi096 ceph-mon[112443]: pgmap v966: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:03 smithi196 ceph-mon[81815]: pgmap v967: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:04.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:03 smithi096 ceph-mon[112443]: pgmap v967: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:05 smithi196 ceph-mon[81815]: pgmap v968: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:06.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:05 smithi096 ceph-mon[112443]: pgmap v968: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:07 smithi196 ceph-mon[81815]: pgmap v969: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:08.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:07 smithi096 ceph-mon[112443]: pgmap v969: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:08.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:10.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:09 smithi096 ceph-mon[112443]: pgmap v970: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:10.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:09 smithi196 ceph-mon[81815]: pgmap v970: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:11 smithi196 ceph-mon[81815]: pgmap v971: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:12.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:11 smithi096 ceph-mon[112443]: pgmap v971: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:13.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:13 smithi096 ceph-mon[112443]: pgmap v972: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:13 smithi196 ceph-mon[81815]: pgmap v972: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:15 smithi196 ceph-mon[81815]: pgmap v973: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:16.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:15 smithi096 ceph-mon[112443]: pgmap v973: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:18.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:17 smithi196 ceph-mon[81815]: pgmap v974: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:18.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:17 smithi096 ceph-mon[112443]: pgmap v974: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:19.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.562+0000 7fa8b1518700 1 -- 172.21.15.96:0/304680491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 msgr2=0x7fa8ac10ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:19.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.562+0000 7fa8b1518700 1 --2- 172.21.15.96:0/304680491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 0x7fa8ac10ceb0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fa894009940 tx=0x7fa894009c50 comp rx=0 tx=0).stop 2024-04-22T15:54:19.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.562+0000 7fa8b1518700 1 -- 172.21.15.96:0/304680491 shutdown_connections 2024-04-22T15:54:19.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.562+0000 7fa8b1518700 1 --2- 172.21.15.96:0/304680491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 0x7fa8ac10ceb0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.562+0000 7fa8b1518700 1 --2- 172.21.15.96:0/304680491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8ac073310 0x7fa8ac0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.562+0000 7fa8b1518700 1 -- 172.21.15.96:0/304680491 >> 172.21.15.96:0/304680491 conn(0x7fa8ac0fc6e0 msgr2=0x7fa8ac0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:19.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.562+0000 7fa8b1518700 1 -- 172.21.15.96:0/304680491 shutdown_connections 2024-04-22T15:54:19.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.562+0000 7fa8b1518700 1 -- 172.21.15.96:0/304680491 wait complete. 2024-04-22T15:54:19.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.562+0000 7fa8b1518700 1 Processor -- start 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8b1518700 1 -- start start 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8b1518700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8ac073310 0x7fa8ac19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8b1518700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 0x7fa8ac19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8b1518700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac19f4d0 con 0x7fa8ac073cc0 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8b1518700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac198b00 con 0x7fa8ac073310 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8a27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 0x7fa8ac19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8a27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 0x7fa8ac19eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41636/0 (socket says 172.21.15.96:41636) 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8a27fc700 1 -- 172.21.15.96:0/4273753201 learned_addr learned my addr 172.21.15.96:0/4273753201 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8aaffd700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8ac073310 0x7fa8ac19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8a27fc700 1 -- 172.21.15.96:0/4273753201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8ac073310 msgr2=0x7fa8ac19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8a27fc700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8ac073310 0x7fa8ac19e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.563+0000 7fa8a27fc700 1 -- 172.21.15.96:0/4273753201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940095f0 con 0x7fa8ac073cc0 2024-04-22T15:54:19.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.564+0000 7fa8a27fc700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 0x7fa8ac19eeb0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fa894005620 tx=0x7fa894005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:19.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.564+0000 7fa8a8ff9700 1 -- 172.21.15.96:0/4273753201 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa89401d070 con 0x7fa8ac073cc0 2024-04-22T15:54:19.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.564+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ac198de0 con 0x7fa8ac073cc0 2024-04-22T15:54:19.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.564+0000 7fa8a8ff9700 1 -- 172.21.15.96:0/4273753201 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa89400f460 con 0x7fa8ac073cc0 2024-04-22T15:54:19.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.564+0000 7fa8a8ff9700 1 -- 172.21.15.96:0/4273753201 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa894021620 con 0x7fa8ac073cc0 2024-04-22T15:54:19.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.564+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ac199330 con 0x7fa8ac073cc0 2024-04-22T15:54:19.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.565+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ac10a650 con 0x7fa8ac073cc0 2024-04-22T15:54:19.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.565+0000 7fa8a8ff9700 1 -- 172.21.15.96:0/4273753201 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa89400fab0 con 0x7fa8ac073cc0 2024-04-22T15:54:19.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.566+0000 7fa8a8ff9700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa89806ecc0 0x7fa898071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:19.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.566+0000 7fa8a8ff9700 1 -- 172.21.15.96:0/4273753201 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa89408ffd0 con 0x7fa8ac073cc0 2024-04-22T15:54:19.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.566+0000 7fa8aaffd700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa89806ecc0 0x7fa898071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:19.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.566+0000 7fa8aaffd700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa89806ecc0 0x7fa898071180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa89c005950 tx=0x7fa89c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.570+0000 7fa8a8ff9700 1 -- 172.21.15.96:0/4273753201 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa89405c860 con 0x7fa8ac073cc0 2024-04-22T15:54:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.795+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8ac199fc0 con 0x7fa89806ecc0 2024-04-22T15:54:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.796+0000 7fa8a8ff9700 1 -- 172.21.15.96:0/4273753201 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa8ac199fc0 con 0x7fa89806ecc0 2024-04-22T15:54:19.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa89806ecc0 msgr2=0x7fa898071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:19.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa89806ecc0 0x7fa898071180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa89c005950 tx=0x7fa89c0058e0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 msgr2=0x7fa8ac19eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 0x7fa8ac19eeb0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fa894005620 tx=0x7fa894005700 comp rx=0 tx=0).stop 2024-04-22T15:54:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 shutdown_connections 2024-04-22T15:54:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa89806ecc0 0x7fa898071180 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8ac073310 0x7fa8ac19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 --2- 172.21.15.96:0/4273753201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8ac073cc0 0x7fa8ac19eeb0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 >> 172.21.15.96:0/4273753201 conn(0x7fa8ac0fc6e0 msgr2=0x7fa8ac107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 shutdown_connections 2024-04-22T15:54:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.798+0000 7fa8b1518700 1 -- 172.21.15.96:0/4273753201 wait complete. 2024-04-22T15:54:19.811 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:54:19.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.904+0000 7f81728d1700 1 -- 172.21.15.96:0/3801652324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 msgr2=0x7f816c101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:19.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.904+0000 7f81728d1700 1 --2- 172.21.15.96:0/3801652324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 0x7f816c101220 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8160009910 tx=0x7f8160009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:19.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.904+0000 7f81728d1700 1 -- 172.21.15.96:0/3801652324 shutdown_connections 2024-04-22T15:54:19.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.904+0000 7f81728d1700 1 --2- 172.21.15.96:0/3801652324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816c1017f0 0x7f816c105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.904+0000 7f81728d1700 1 --2- 172.21.15.96:0/3801652324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 0x7f816c101220 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.904+0000 7f81728d1700 1 -- 172.21.15.96:0/3801652324 >> 172.21.15.96:0/3801652324 conn(0x7f816c0fc710 msgr2=0x7f816c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:19.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.904+0000 7f81728d1700 1 -- 172.21.15.96:0/3801652324 shutdown_connections 2024-04-22T15:54:19.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.904+0000 7f81728d1700 1 -- 172.21.15.96:0/3801652324 wait complete. 2024-04-22T15:54:19.908 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:19 smithi096 ceph-mon[112443]: pgmap v975: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:19.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f81728d1700 1 Processor -- start 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f81728d1700 1 -- start start 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f81728d1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 0x7f816c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f81728d1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816c1017f0 0x7f816c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f81728d1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c19d3a0 con 0x7f816c100e40 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f81728d1700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c1969d0 con 0x7f816c1017f0 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f816bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 0x7f816c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f816bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 0x7f816c19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41656/0 (socket says 172.21.15.96:41656) 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f816bfff700 1 -- 172.21.15.96:0/3898621308 learned_addr learned my addr 172.21.15.96:0/3898621308 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f816b7fe700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816c1017f0 0x7f816c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f816bfff700 1 -- 172.21.15.96:0/3898621308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816c1017f0 msgr2=0x7f816c19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f816bfff700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816c1017f0 0x7f816c19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:19.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.905+0000 7f816bfff700 1 -- 172.21.15.96:0/3898621308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81600095f0 con 0x7f816c100e40 2024-04-22T15:54:19.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.906+0000 7f816bfff700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 0x7f816c19c840 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f816000bd70 tx=0x7f816000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:19.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.906+0000 7f81697fa700 1 -- 172.21.15.96:0/3898621308 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f816001d070 con 0x7f816c100e40 2024-04-22T15:54:19.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.906+0000 7f81697fa700 1 -- 172.21.15.96:0/3898621308 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81600056e0 con 0x7f816c100e40 2024-04-22T15:54:19.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.906+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c196c50 con 0x7f816c100e40 2024-04-22T15:54:19.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.906+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c197140 con 0x7f816c100e40 2024-04-22T15:54:19.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.906+0000 7f81697fa700 1 -- 172.21.15.96:0/3898621308 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f816000fe40 con 0x7f816c100e40 2024-04-22T15:54:19.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.907+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816c1091e0 con 0x7f816c100e40 2024-04-22T15:54:19.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.907+0000 7f81697fa700 1 -- 172.21.15.96:0/3898621308 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f816000f680 con 0x7f816c100e40 2024-04-22T15:54:19.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.908+0000 7f81697fa700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f815406ec60 0x7f8154071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:19.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.908+0000 7f81697fa700 1 -- 172.21.15.96:0/3898621308 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f81600902e0 con 0x7f816c100e40 2024-04-22T15:54:19.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.908+0000 7f816b7fe700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f815406ec60 0x7f8154071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:19.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.908+0000 7f816b7fe700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f815406ec60 0x7f8154071120 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f816c197b40 tx=0x7f815c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:19.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:19.912+0000 7f81697fa700 1 -- 172.21.15.96:0/3898621308 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f816005cb70 con 0x7f816c100e40 2024-04-22T15:54:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:19 smithi196 ceph-mon[81815]: pgmap v975: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:20.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.127+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f816c197e40 con 0x7f815406ec60 2024-04-22T15:54:20.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.128+0000 7f81697fa700 1 -- 172.21.15.96:0/3898621308 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f816c197e40 con 0x7f815406ec60 2024-04-22T15:54:20.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.130+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f815406ec60 msgr2=0x7f8154071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.130+0000 7f81728d1700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f815406ec60 0x7f8154071120 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f816c197b40 tx=0x7f815c00b410 comp rx=0 tx=0).stop 2024-04-22T15:54:20.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.130+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 msgr2=0x7f816c19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.130+0000 7f81728d1700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 0x7f816c19c840 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f816000bd70 tx=0x7f816000bda0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.130+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 shutdown_connections 2024-04-22T15:54:20.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.130+0000 7f81728d1700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f815406ec60 0x7f8154071120 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.130+0000 7f81728d1700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c100e40 0x7f816c19c840 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.130+0000 7f81728d1700 1 --2- 172.21.15.96:0/3898621308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816c1017f0 0x7f816c19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.130+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 >> 172.21.15.96:0/3898621308 conn(0x7f816c0fc710 msgr2=0x7f816c0fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.131+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 shutdown_connections 2024-04-22T15:54:20.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.131+0000 7f81728d1700 1 -- 172.21.15.96:0/3898621308 wait complete. 2024-04-22T15:54:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.235+0000 7f64227fc700 1 -- 172.21.15.96:0/4231413407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 msgr2=0x7f641c105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.235+0000 7f64227fc700 1 --2- 172.21.15.96:0/4231413407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 0x7f641c105e80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f640c009910 tx=0x7f640c009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.235+0000 7f64227fc700 1 -- 172.21.15.96:0/4231413407 shutdown_connections 2024-04-22T15:54:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.235+0000 7f64227fc700 1 --2- 172.21.15.96:0/4231413407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f641c0ffaa0 0x7f641c0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.235+0000 7f64227fc700 1 --2- 172.21.15.96:0/4231413407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 0x7f641c105e80 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.235+0000 7f64227fc700 1 -- 172.21.15.96:0/4231413407 >> 172.21.15.96:0/4231413407 conn(0x7f641c076740 msgr2=0x7f641c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.235+0000 7f64227fc700 1 -- 172.21.15.96:0/4231413407 shutdown_connections 2024-04-22T15:54:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.235+0000 7f64227fc700 1 -- 172.21.15.96:0/4231413407 wait complete. 2024-04-22T15:54:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.235+0000 7f64227fc700 1 Processor -- start 2024-04-22T15:54:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f64227fc700 1 -- start start 2024-04-22T15:54:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f64227fc700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f641c0ffaa0 0x7f641c195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f64227fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 0x7f641c195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f64227fc700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c196620 con 0x7f641c105aa0 2024-04-22T15:54:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f64227fc700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c19a370 con 0x7f641c0ffaa0 2024-04-22T15:54:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f641b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 0x7f641c195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f641b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 0x7f641c195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41684/0 (socket says 172.21.15.96:41684) 2024-04-22T15:54:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f641b7fe700 1 -- 172.21.15.96:0/1437427634 learned_addr learned my addr 172.21.15.96:0/1437427634 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f641bfff700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f641c0ffaa0 0x7f641c195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f641b7fe700 1 -- 172.21.15.96:0/1437427634 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f641c0ffaa0 msgr2=0x7f641c195a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f641b7fe700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f641c0ffaa0 0x7f641c195a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f641b7fe700 1 -- 172.21.15.96:0/1437427634 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c0095f0 con 0x7f641c105aa0 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.236+0000 7f641b7fe700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 0x7f641c195f90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f641000e920 tx=0x7f641000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.237+0000 7f64197fa700 1 -- 172.21.15.96:0/1437427634 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641000c9d0 con 0x7f641c105aa0 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.237+0000 7f64197fa700 1 -- 172.21.15.96:0/1437427634 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6410010430 con 0x7f641c105aa0 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.237+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641c19a650 con 0x7f641c105aa0 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.237+0000 7f64197fa700 1 -- 172.21.15.96:0/1437427634 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6410018630 con 0x7f641c105aa0 2024-04-22T15:54:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.237+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641c19aa60 con 0x7f641c105aa0 2024-04-22T15:54:20.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.238+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641c1011e0 con 0x7f641c105aa0 2024-04-22T15:54:20.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.238+0000 7f64197fa700 1 -- 172.21.15.96:0/1437427634 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f64100105a0 con 0x7f641c105aa0 2024-04-22T15:54:20.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.239+0000 7f64197fa700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f640406ecd0 0x7f6404071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.239+0000 7f64197fa700 1 -- 172.21.15.96:0/1437427634 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6410014070 con 0x7f641c105aa0 2024-04-22T15:54:20.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.239+0000 7f641bfff700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f640406ecd0 0x7f6404071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.239+0000 7f641bfff700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f640406ecd0 0x7f6404071190 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f640c00b760 tx=0x7f640c005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.243+0000 7f64197fa700 1 -- 172.21.15.96:0/1437427634 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6410054150 con 0x7f641c105aa0 2024-04-22T15:54:20.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.451+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f641c10cd80 con 0x7f640406ecd0 2024-04-22T15:54:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.462+0000 7f64197fa700 1 -- 172.21.15.96:0/1437427634 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f641c10cd80 con 0x7f640406ecd0 2024-04-22T15:54:20.465 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:54:20.465 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (33m) 10m ago 44m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:54:20.465 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (44m) 10m ago 44m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:54:20.465 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (43m) 10m ago 43m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:54:20.465 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (30m) 10m ago 44m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:54:20.465 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (30m) 10m ago 43m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (32m) 10m ago 44m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (26m) 10m ago 38m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (26m) 10m ago 38m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (25m) 10m ago 38m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (10m) 10m ago 38m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (35m) 10m ago 45m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (34m) 10m ago 43m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (31m) 10m ago 45m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (30m) 10m ago 43m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (33m) 10m ago 44m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (33m) 10m ago 43m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (29m) 10m ago 42m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (29m) 10m ago 41m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:54:20.466 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (28m) 10m ago 41m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:54:20.467 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (27m) 10m ago 41m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:54:20.467 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (27m) 10m ago 40m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:54:20.467 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (26m) 10m ago 40m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:54:20.467 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (33m) 10m ago 43m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.464+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f640406ecd0 msgr2=0x7f6404071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.464+0000 7f64227fc700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f640406ecd0 0x7f6404071190 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f640c00b760 tx=0x7f640c005a90 comp rx=0 tx=0).stop 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.464+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 msgr2=0x7f641c195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.464+0000 7f64227fc700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 0x7f641c195f90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f641000e920 tx=0x7f641000ec30 comp rx=0 tx=0).stop 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.464+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 shutdown_connections 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.464+0000 7f64227fc700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f640406ecd0 0x7f6404071190 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.464+0000 7f64227fc700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f641c0ffaa0 0x7f641c195a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.464+0000 7f64227fc700 1 --2- 172.21.15.96:0/1437427634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f641c105aa0 0x7f641c195f90 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.464+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 >> 172.21.15.96:0/1437427634 conn(0x7f641c076740 msgr2=0x7f641c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.465+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 shutdown_connections 2024-04-22T15:54:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.465+0000 7f64227fc700 1 -- 172.21.15.96:0/1437427634 wait complete. 2024-04-22T15:54:20.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.566+0000 7f230a220700 1 -- 172.21.15.96:0/1658703406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 msgr2=0x7f230410aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.566+0000 7f230a220700 1 --2- 172.21.15.96:0/1658703406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 0x7f230410aba0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009910 tx=0x7f22f4009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:20.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.566+0000 7f230a220700 1 -- 172.21.15.96:0/1658703406 shutdown_connections 2024-04-22T15:54:20.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.566+0000 7f230a220700 1 --2- 172.21.15.96:0/1658703406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 0x7f230410aba0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.566+0000 7f230a220700 1 --2- 172.21.15.96:0/1658703406 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2304101d90 0x7f2304102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.566+0000 7f230a220700 1 -- 172.21.15.96:0/1658703406 >> 172.21.15.96:0/1658703406 conn(0x7f23040fb5e0 msgr2=0x7f23040fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.566+0000 7f230a220700 1 -- 172.21.15.96:0/1658703406 shutdown_connections 2024-04-22T15:54:20.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.567+0000 7f230a220700 1 -- 172.21.15.96:0/1658703406 wait complete. 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.567+0000 7f230a220700 1 Processor -- start 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f230a220700 1 -- start start 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f230a220700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2304101d90 0x7f230419ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f230a220700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 0x7f230419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f230a220700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2304196ab0 con 0x7f23041026b0 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f230a220700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2304196c20 con 0x7f2304101d90 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f2302ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 0x7f230419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f2302ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 0x7f230419cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36410/0 (socket says 172.21.15.96:36410) 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f2302ffd700 1 -- 172.21.15.96:0/536469432 learned_addr learned my addr 172.21.15.96:0/536469432 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f23037fe700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2304101d90 0x7f230419ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f2302ffd700 1 -- 172.21.15.96:0/536469432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2304101d90 msgr2=0x7f230419ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f2302ffd700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2304101d90 0x7f230419ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.568+0000 7f2302ffd700 1 -- 172.21.15.96:0/536469432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f40095f0 con 0x7f23041026b0 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.569+0000 7f2302ffd700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 0x7f230419cf40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f22f400bd70 tx=0x7f22f400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.569+0000 7f2300ff9700 1 -- 172.21.15.96:0/536469432 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22f401d070 con 0x7f23041026b0 2024-04-22T15:54:20.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.569+0000 7f2300ff9700 1 -- 172.21.15.96:0/536469432 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22f40056c0 con 0x7f23041026b0 2024-04-22T15:54:20.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.569+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2304196ea0 con 0x7f23041026b0 2024-04-22T15:54:20.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.569+0000 7f2300ff9700 1 -- 172.21.15.96:0/536469432 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22f4017400 con 0x7f23041026b0 2024-04-22T15:54:20.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.569+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2304197390 con 0x7f23041026b0 2024-04-22T15:54:20.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.570+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23040699f0 con 0x7f23041026b0 2024-04-22T15:54:20.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.571+0000 7f2300ff9700 1 -- 172.21.15.96:0/536469432 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f22f4003a60 con 0x7f23041026b0 2024-04-22T15:54:20.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.571+0000 7f2300ff9700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22ec06ecc0 0x7f22ec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.571+0000 7f2300ff9700 1 -- 172.21.15.96:0/536469432 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f22f4090270 con 0x7f23041026b0 2024-04-22T15:54:20.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.572+0000 7f23037fe700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22ec06ecc0 0x7f22ec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.572+0000 7f23037fe700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22ec06ecc0 0x7f22ec071180 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009a10 tx=0x7f22f8009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.576+0000 7f2300ff9700 1 -- 172.21.15.96:0/536469432 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f22f4058fd0 con 0x7f23041026b0 2024-04-22T15:54:20.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.861+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2304105360 con 0x7f23041026b0 2024-04-22T15:54:20.865 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:20 smithi096 ceph-mon[112443]: from='client.35768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:20.865 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:20 smithi096 ceph-mon[112443]: from='client.35772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:20.865 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:20 smithi096 ceph-mon[112443]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:20.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.862+0000 7f2300ff9700 1 -- 172.21.15.96:0/536469432 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f22f407be60 con 0x7f23041026b0 2024-04-22T15:54:20.865 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:54:20.865 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:54:20.865 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:54:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:54:20.867 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:54:20.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22ec06ecc0 msgr2=0x7f22ec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22ec06ecc0 0x7f22ec071180 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009a10 tx=0x7f22f8009190 comp rx=0 tx=0).stop 2024-04-22T15:54:20.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 msgr2=0x7f230419cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 0x7f230419cf40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f22f400bd70 tx=0x7f22f400bda0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 shutdown_connections 2024-04-22T15:54:20.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f22ec06ecc0 0x7f22ec071180 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2304101d90 0x7f230419ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 --2- 172.21.15.96:0/536469432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23041026b0 0x7f230419cf40 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 >> 172.21.15.96:0/536469432 conn(0x7f23040fb5e0 msgr2=0x7f23041030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 shutdown_connections 2024-04-22T15:54:20.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.864+0000 7f230a220700 1 -- 172.21.15.96:0/536469432 wait complete. 2024-04-22T15:54:20.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.968+0000 7f82f2b40700 1 -- 172.21.15.96:0/974088963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 msgr2=0x7f82ec105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.968+0000 7f82f2b40700 1 --2- 172.21.15.96:0/974088963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 0x7f82ec105e90 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f82e400b0a0 tx=0x7f82e400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.968+0000 7f82f2b40700 1 -- 172.21.15.96:0/974088963 shutdown_connections 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.968+0000 7f82f2b40700 1 --2- 172.21.15.96:0/974088963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82ec0ffab0 0x7f82ec0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.968+0000 7f82f2b40700 1 --2- 172.21.15.96:0/974088963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 0x7f82ec105e90 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.968+0000 7f82f2b40700 1 -- 172.21.15.96:0/974088963 >> 172.21.15.96:0/974088963 conn(0x7f82ec0fb5e0 msgr2=0x7f82ec0fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.968+0000 7f82f2b40700 1 -- 172.21.15.96:0/974088963 shutdown_connections 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.968+0000 7f82f2b40700 1 -- 172.21.15.96:0/974088963 wait complete. 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.968+0000 7f82f2b40700 1 Processor -- start 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82f2b40700 1 -- start start 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82f2b40700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82ec0ffab0 0x7f82ec19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82f2b40700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 0x7f82ec19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82f2b40700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec19d410 con 0x7f82ec105ab0 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82f2b40700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec1969f0 con 0x7f82ec0ffab0 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82ebfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 0x7f82ec19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82ebfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 0x7f82ec19cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36424/0 (socket says 172.21.15.96:36424) 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82ebfff700 1 -- 172.21.15.96:0/2626437270 learned_addr learned my addr 172.21.15.96:0/2626437270 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82f08dc700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82ec0ffab0 0x7f82ec19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82ebfff700 1 -- 172.21.15.96:0/2626437270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82ec0ffab0 msgr2=0x7f82ec19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82ebfff700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82ec0ffab0 0x7f82ec19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82ebfff700 1 -- 172.21.15.96:0/2626437270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e4009d00 con 0x7f82ec105ab0 2024-04-22T15:54:20.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82ebfff700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 0x7f82ec19cdf0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00e920 tx=0x7f82dc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82e9ffb700 1 -- 172.21.15.96:0/2626437270 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82dc00c9d0 con 0x7f82ec105ab0 2024-04-22T15:54:20.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.969+0000 7f82e9ffb700 1 -- 172.21.15.96:0/2626437270 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82dc010430 con 0x7f82ec105ab0 2024-04-22T15:54:20.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.970+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec196cd0 con 0x7f82ec105ab0 2024-04-22T15:54:20.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.970+0000 7f82e9ffb700 1 -- 172.21.15.96:0/2626437270 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82dc018630 con 0x7f82ec105ab0 2024-04-22T15:54:20.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.970+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec197220 con 0x7f82ec105ab0 2024-04-22T15:54:20.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.971+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec1080b0 con 0x7f82ec105ab0 2024-04-22T15:54:20.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.971+0000 7f82e9ffb700 1 -- 172.21.15.96:0/2626437270 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f82dc0105a0 con 0x7f82ec105ab0 2024-04-22T15:54:20.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.971+0000 7f82e9ffb700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82d806ecc0 0x7f82d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.972+0000 7f82e9ffb700 1 -- 172.21.15.96:0/2626437270 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f82dc014070 con 0x7f82ec105ab0 2024-04-22T15:54:20.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.972+0000 7f82f08dc700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82d806ecc0 0x7f82d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.972+0000 7f82f08dc700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82d806ecc0 0x7f82d8071180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000f80 tx=0x7f82e40060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:20.976+0000 7f82e9ffb700 1 -- 172.21.15.96:0/2626437270 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f82dc054150 con 0x7f82ec105ab0 2024-04-22T15:54:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:20 smithi196 ceph-mon[81815]: from='client.35768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:20 smithi196 ceph-mon[81815]: from='client.35772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:20 smithi196 ceph-mon[81815]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:21.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.219+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f82ec197eb0 con 0x7f82ec105ab0 2024-04-22T15:54:21.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.220+0000 7f82e9ffb700 1 -- 172.21.15.96:0/2626437270 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f82dc05b9b0 con 0x7f82ec105ab0 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:54:21.223 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:54:21.224 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:54:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:54:21.226 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:54:21.226 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:54:21.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82d806ecc0 msgr2=0x7f82d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82d806ecc0 0x7f82d8071180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000f80 tx=0x7f82e40060c0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 msgr2=0x7f82ec19cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 0x7f82ec19cdf0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00e920 tx=0x7f82dc00ec30 comp rx=0 tx=0).stop 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 shutdown_connections 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f82d806ecc0 0x7f82d8071180 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82ec0ffab0 0x7f82ec19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 --2- 172.21.15.96:0/2626437270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82ec105ab0 0x7f82ec19cdf0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 >> 172.21.15.96:0/2626437270 conn(0x7f82ec0fb5e0 msgr2=0x7f82ec109ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 shutdown_connections 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.222+0000 7f82f2b40700 1 -- 172.21.15.96:0/2626437270 wait complete. 2024-04-22T15:54:21.227 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:54:21.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 -- 172.21.15.96:0/3662405137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c100d90 msgr2=0x7ff03c104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 --2- 172.21.15.96:0/3662405137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c100d90 0x7ff03c104760 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff030009910 tx=0x7ff030009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 -- 172.21.15.96:0/3662405137 shutdown_connections 2024-04-22T15:54:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 --2- 172.21.15.96:0/3662405137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c100d90 0x7ff03c104760 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 --2- 172.21.15.96:0/3662405137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff03c1003e0 0x7ff03c1007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 -- 172.21.15.96:0/3662405137 >> 172.21.15.96:0/3662405137 conn(0x7ff03c075f30 msgr2=0x7ff03c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 -- 172.21.15.96:0/3662405137 shutdown_connections 2024-04-22T15:54:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 -- 172.21.15.96:0/3662405137 wait complete. 2024-04-22T15:54:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 Processor -- start 2024-04-22T15:54:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.327+0000 7ff042e11700 1 -- start start 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff042e11700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c1003e0 0x7ff03c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff042e11700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff03c100d90 0x7ff03c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff042e11700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c19d380 con 0x7ff03c1003e0 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff042e11700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c1969b0 con 0x7ff03c100d90 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff040bad700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c1003e0 0x7ff03c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff040bad700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c1003e0 0x7ff03c19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36440/0 (socket says 172.21.15.96:36440) 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff040bad700 1 -- 172.21.15.96:0/1136466649 learned_addr learned my addr 172.21.15.96:0/1136466649 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff03bfff700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff03c100d90 0x7ff03c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff040bad700 1 -- 172.21.15.96:0/1136466649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff03c100d90 msgr2=0x7ff03c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff040bad700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff03c100d90 0x7ff03c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff040bad700 1 -- 172.21.15.96:0/1136466649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0300095f0 con 0x7ff03c1003e0 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff040bad700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c1003e0 0x7ff03c19c820 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff0340060b0 tx=0x7ff03400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff039ffb700 1 -- 172.21.15.96:0/1136466649 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff03400dda0 con 0x7ff03c1003e0 2024-04-22T15:54:21.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff039ffb700 1 -- 172.21.15.96:0/1136466649 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff03400f040 con 0x7ff03c1003e0 2024-04-22T15:54:21.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03c196c90 con 0x7ff03c1003e0 2024-04-22T15:54:21.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.328+0000 7ff039ffb700 1 -- 172.21.15.96:0/1136466649 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff034014790 con 0x7ff03c1003e0 2024-04-22T15:54:21.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.329+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03c1971b0 con 0x7ff03c1003e0 2024-04-22T15:54:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.330+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff03c1013a0 con 0x7ff03c1003e0 2024-04-22T15:54:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.330+0000 7ff039ffb700 1 -- 172.21.15.96:0/1136466649 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff0340148f0 con 0x7ff03c1003e0 2024-04-22T15:54:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.330+0000 7ff039ffb700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff02406ecc0 0x7ff024071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.330+0000 7ff03bfff700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff02406ecc0 0x7ff024071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.331+0000 7ff039ffb700 1 -- 172.21.15.96:0/1136466649 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff03408e880 con 0x7ff03c1003e0 2024-04-22T15:54:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.331+0000 7ff03bfff700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff02406ecc0 0x7ff024071180 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff030000c00 tx=0x7ff03001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:21.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.335+0000 7ff039ffb700 1 -- 172.21.15.96:0/1136466649 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff03405b240 con 0x7ff03c1003e0 2024-04-22T15:54:21.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.556+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff03c10cd50 con 0x7ff02406ecc0 2024-04-22T15:54:21.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.557+0000 7ff039ffb700 1 -- 172.21.15.96:0/1136466649 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff03c10cd50 con 0x7ff02406ecc0 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:54:21.560 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:54:21.561 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:54:21.561 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:54:21.561 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:54:21.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff02406ecc0 msgr2=0x7ff024071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff02406ecc0 0x7ff024071180 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff030000c00 tx=0x7ff03001a040 comp rx=0 tx=0).stop 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c1003e0 msgr2=0x7ff03c19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c1003e0 0x7ff03c19c820 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff0340060b0 tx=0x7ff03400d3b0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 shutdown_connections 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff02406ecc0 0x7ff024071180 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff03c1003e0 0x7ff03c19c820 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 --2- 172.21.15.96:0/1136466649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff03c100d90 0x7ff03c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 >> 172.21.15.96:0/1136466649 conn(0x7ff03c075f30 msgr2=0x7ff03c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.559+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 shutdown_connections 2024-04-22T15:54:21.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.560+0000 7ff042e11700 1 -- 172.21.15.96:0/1136466649 wait complete. 2024-04-22T15:54:21.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.662+0000 7fa590cc2700 1 -- 172.21.15.96:0/3392870868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c0ffb50 msgr2=0x7fa58c0fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.662+0000 7fa590cc2700 1 --2- 172.21.15.96:0/3392870868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c0ffb50 0x7fa58c0fffd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009910 tx=0x7fa57c009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:21.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.662+0000 7fa590cc2700 1 -- 172.21.15.96:0/3392870868 shutdown_connections 2024-04-22T15:54:21.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.662+0000 7fa590cc2700 1 --2- 172.21.15.96:0/3392870868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c0ffb50 0x7fa58c0fffd0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.662+0000 7fa590cc2700 1 --2- 172.21.15.96:0/3392870868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa58c105c00 0x7fa58c105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.662+0000 7fa590cc2700 1 -- 172.21.15.96:0/3392870868 >> 172.21.15.96:0/3392870868 conn(0x7fa58c0fb680 msgr2=0x7fa58c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.662+0000 7fa590cc2700 1 -- 172.21.15.96:0/3392870868 shutdown_connections 2024-04-22T15:54:21.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.662+0000 7fa590cc2700 1 -- 172.21.15.96:0/3392870868 wait complete. 2024-04-22T15:54:21.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa590cc2700 1 Processor -- start 2024-04-22T15:54:21.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa590cc2700 1 -- start start 2024-04-22T15:54:21.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa590cc2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa58c0ffb50 0x7fa58c19b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa590cc2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c105c00 0x7fa58c19bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa590cc2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c19c2c0 con 0x7fa58c105c00 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa590cc2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c195840 con 0x7fa58c0ffb50 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa589d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c105c00 0x7fa58c19bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa589d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c105c00 0x7fa58c19bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36456/0 (socket says 172.21.15.96:36456) 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa589d9b700 1 -- 172.21.15.96:0/674348922 learned_addr learned my addr 172.21.15.96:0/674348922 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa58a59c700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa58c0ffb50 0x7fa58c19b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa589d9b700 1 -- 172.21.15.96:0/674348922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa58c0ffb50 msgr2=0x7fa58c19b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa589d9b700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa58c0ffb50 0x7fa58c19b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.663+0000 7fa589d9b700 1 -- 172.21.15.96:0/674348922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57c0095f0 con 0x7fa58c105c00 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.664+0000 7fa589d9b700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c105c00 0x7fa58c19bbe0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00bd70 tx=0x7fa57c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.664+0000 7fa57b7fe700 1 -- 172.21.15.96:0/674348922 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa57c01d070 con 0x7fa58c105c00 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.664+0000 7fa57b7fe700 1 -- 172.21.15.96:0/674348922 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa57c0056e0 con 0x7fa58c105c00 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.664+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58c195ac0 con 0x7fa58c105c00 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.664+0000 7fa57b7fe700 1 -- 172.21.15.96:0/674348922 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa57c017400 con 0x7fa58c105c00 2024-04-22T15:54:21.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.664+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58c195fb0 con 0x7fa58c105c00 2024-04-22T15:54:21.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.665+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58c1080b0 con 0x7fa58c105c00 2024-04-22T15:54:21.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.665+0000 7fa57b7fe700 1 -- 172.21.15.96:0/674348922 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa57c003a60 con 0x7fa58c105c00 2024-04-22T15:54:21.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.666+0000 7fa57b7fe700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa57406ecc0 0x7fa574071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.666+0000 7fa57b7fe700 1 -- 172.21.15.96:0/674348922 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa57c090200 con 0x7fa58c105c00 2024-04-22T15:54:21.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.666+0000 7fa58a59c700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa57406ecc0 0x7fa574071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.666+0000 7fa58a59c700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa57406ecc0 0x7fa574071180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa580005950 tx=0x7fa5800058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:21.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.670+0000 7fa57b7fe700 1 -- 172.21.15.96:0/674348922 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa57c05cb10 con 0x7fa58c105c00 2024-04-22T15:54:21.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.956+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa58c101200 con 0x7fa58c105c00 2024-04-22T15:54:21.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:21 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/536469432' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:21.959 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:21 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2626437270' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:54:21.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:21 smithi096 ceph-mon[112443]: from='client.35788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:21.960 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:21 smithi096 ceph-mon[112443]: pgmap v976: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:21.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.956+0000 7fa57b7fe700 1 -- 172.21.15.96:0/674348922 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fa57c05c6a0 con 0x7fa58c105c00 2024-04-22T15:54:21.960 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:54:21.960 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:54:21.960 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:54:21.960 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:54:21.961 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:54:21.962 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:54:21.962 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:54:21.962 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:54:21.962 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:54:21.962 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:54:21.962 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:54:21.962 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:54:21.962 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:54:21.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa57406ecc0 msgr2=0x7fa574071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa57406ecc0 0x7fa574071180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa580005950 tx=0x7fa5800058e0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c105c00 msgr2=0x7fa58c19bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c105c00 0x7fa58c19bbe0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00bd70 tx=0x7fa57c00be50 comp rx=0 tx=0).stop 2024-04-22T15:54:21.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 shutdown_connections 2024-04-22T15:54:21.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa57406ecc0 0x7fa574071180 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa58c0ffb50 0x7fa58c19b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 --2- 172.21.15.96:0/674348922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa58c105c00 0x7fa58c19bbe0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 >> 172.21.15.96:0/674348922 conn(0x7fa58c0fb680 msgr2=0x7fa58c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 shutdown_connections 2024-04-22T15:54:21.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:21.959+0000 7fa590cc2700 1 -- 172.21.15.96:0/674348922 wait complete. 2024-04-22T15:54:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:21 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/536469432' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:21 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2626437270' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:54:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:21 smithi196 ceph-mon[81815]: from='client.35788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:21 smithi196 ceph-mon[81815]: pgmap v976: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:23.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:22 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/674348922' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:54:23.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:22 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/674348922' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:54:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:23.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:23 smithi096 ceph-mon[112443]: pgmap v977: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:23 smithi196 ceph-mon[81815]: pgmap v977: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:26.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:25 smithi196 ceph-mon[81815]: pgmap v978: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:26.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:25 smithi096 ceph-mon[112443]: pgmap v978: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:27 smithi196 ceph-mon[81815]: pgmap v979: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:28.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:27 smithi096 ceph-mon[112443]: pgmap v979: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:29 smithi196 ceph-mon[81815]: pgmap v980: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:30.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:29 smithi096 ceph-mon[112443]: pgmap v980: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:32.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:31 smithi196 ceph-mon[81815]: pgmap v981: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:32.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:31 smithi096 ceph-mon[112443]: pgmap v981: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:33 smithi196 ceph-mon[81815]: pgmap v982: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:34.037 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:33 smithi096 ceph-mon[112443]: pgmap v982: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:35 smithi196 ceph-mon[81815]: pgmap v983: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:36.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:35 smithi096 ceph-mon[112443]: pgmap v983: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:38.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:37 smithi196 ceph-mon[81815]: pgmap v984: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:38.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:37 smithi096 ceph-mon[112443]: pgmap v984: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:39.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:39.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:40.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:39 smithi096 ceph-mon[112443]: pgmap v985: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:40.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:39 smithi196 ceph-mon[81815]: pgmap v985: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:42.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:41 smithi196 ceph-mon[81815]: pgmap v986: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:42.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:41 smithi096 ceph-mon[112443]: pgmap v986: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:44.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:43 smithi096 ceph-mon[112443]: pgmap v987: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:44.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:43 smithi196 ceph-mon[81815]: pgmap v987: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:46.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:45 smithi096 ceph-mon[112443]: pgmap v988: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:45 smithi196 ceph-mon[81815]: pgmap v988: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:48.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:47 smithi096 ceph-mon[112443]: pgmap v989: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:48.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:47 smithi196 ceph-mon[81815]: pgmap v989: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:50.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:49 smithi096 ceph-mon[112443]: pgmap v990: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:50.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:49 smithi196 ceph-mon[81815]: pgmap v990: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:52.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.076+0000 7f8edd67e700 1 -- 172.21.15.96:0/3704544077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed8100e20 msgr2=0x7f8ed8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.076+0000 7f8edd67e700 1 --2- 172.21.15.96:0/3704544077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed8100e20 0x7f8ed8101200 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009910 tx=0x7f8ec8009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:52.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.076+0000 7f8edd67e700 1 -- 172.21.15.96:0/3704544077 shutdown_connections 2024-04-22T15:54:52.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.076+0000 7f8edd67e700 1 --2- 172.21.15.96:0/3704544077 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ed81017d0 0x7f8ed8105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.076+0000 7f8edd67e700 1 --2- 172.21.15.96:0/3704544077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed8100e20 0x7f8ed8101200 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.076+0000 7f8edd67e700 1 -- 172.21.15.96:0/3704544077 >> 172.21.15.96:0/3704544077 conn(0x7f8ed80fc6d0 msgr2=0x7f8ed80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:52.080 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:51 smithi096 ceph-mon[112443]: pgmap v991: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:52.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.077+0000 7f8edd67e700 1 -- 172.21.15.96:0/3704544077 shutdown_connections 2024-04-22T15:54:52.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.077+0000 7f8edd67e700 1 -- 172.21.15.96:0/3704544077 wait complete. 2024-04-22T15:54:52.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.077+0000 7f8edd67e700 1 Processor -- start 2024-04-22T15:54:52.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.077+0000 7f8edd67e700 1 -- start start 2024-04-22T15:54:52.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8edd67e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ed8100e20 0x7f8ed80731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8edd67e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed81017d0 0x7f8ed8073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8edd67e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8073c60 con 0x7f8ed81017d0 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8edd67e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8073da0 con 0x7f8ed8100e20 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ed67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed81017d0 0x7f8ed8073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ed67fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed81017d0 0x7f8ed8073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35154/0 (socket says 172.21.15.96:35154) 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ed67fc700 1 -- 172.21.15.96:0/3479935019 learned_addr learned my addr 172.21.15.96:0/3479935019 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ed6ffd700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ed8100e20 0x7f8ed80731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ed67fc700 1 -- 172.21.15.96:0/3479935019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ed8100e20 msgr2=0x7f8ed80731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ed67fc700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ed8100e20 0x7f8ed80731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ed67fc700 1 -- 172.21.15.96:0/3479935019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec80095f0 con 0x7f8ed81017d0 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ed67fc700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed81017d0 0x7f8ed8073720 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00b480 tx=0x7f8ecc00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ebffff700 1 -- 172.21.15.96:0/3479935019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ecc011800 con 0x7f8ed81017d0 2024-04-22T15:54:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed81a2de0 con 0x7f8ed81017d0 2024-04-22T15:54:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ebffff700 1 -- 172.21.15.96:0/3479935019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ecc011e20 con 0x7f8ed81017d0 2024-04-22T15:54:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8ebffff700 1 -- 172.21.15.96:0/3479935019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ecc010ea0 con 0x7f8ed81017d0 2024-04-22T15:54:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.078+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed81a3140 con 0x7f8ed81017d0 2024-04-22T15:54:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.079+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed8102300 con 0x7f8ed81017d0 2024-04-22T15:54:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.080+0000 7f8ebffff700 1 -- 172.21.15.96:0/3479935019 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8ecc010380 con 0x7f8ed81017d0 2024-04-22T15:54:52.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.081+0000 7f8ebffff700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ec006ecc0 0x7f8ec0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.081+0000 7f8ed6ffd700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ec006ecc0 0x7f8ec0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.081+0000 7f8ebffff700 1 -- 172.21.15.96:0/3479935019 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8ecc08e980 con 0x7f8ed81017d0 2024-04-22T15:54:52.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.081+0000 7f8ed6ffd700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ec006ecc0 0x7f8ec0071180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8000c00 tx=0x7f8ec801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:52.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.086+0000 7f8ebffff700 1 -- 172.21.15.96:0/3479935019 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8ecc0576e0 con 0x7f8ed81017d0 2024-04-22T15:54:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:51 smithi196 ceph-mon[81815]: pgmap v991: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:52.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.301+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ed810ded0 con 0x7f8ec006ecc0 2024-04-22T15:54:52.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.303+0000 7f8ebffff700 1 -- 172.21.15.96:0/3479935019 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8ed810ded0 con 0x7f8ec006ecc0 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ec006ecc0 msgr2=0x7f8ec0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ec006ecc0 0x7f8ec0071180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8000c00 tx=0x7f8ec801a040 comp rx=0 tx=0).stop 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed81017d0 msgr2=0x7f8ed8073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed81017d0 0x7f8ed8073720 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00b480 tx=0x7f8ecc00b790 comp rx=0 tx=0).stop 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 shutdown_connections 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f8ec006ecc0 0x7f8ec0071180 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ed8100e20 0x7f8ed80731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 --2- 172.21.15.96:0/3479935019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ed81017d0 0x7f8ed8073720 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 >> 172.21.15.96:0/3479935019 conn(0x7f8ed80fc6d0 msgr2=0x7f8ed8104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 shutdown_connections 2024-04-22T15:54:52.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.305+0000 7f8edd67e700 1 -- 172.21.15.96:0/3479935019 wait complete. 2024-04-22T15:54:52.318 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:54:52.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.409+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/2862449756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18100e40 msgr2=0x7fbe18101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.409+0000 7fbe1cc3e700 1 --2- 172.21.15.96:0/2862449756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18100e40 0x7fbe18101220 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009910 tx=0x7fbe0c009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:52.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.409+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/2862449756 shutdown_connections 2024-04-22T15:54:52.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.409+0000 7fbe1cc3e700 1 --2- 172.21.15.96:0/2862449756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe181017f0 0x7fbe18105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.409+0000 7fbe1cc3e700 1 --2- 172.21.15.96:0/2862449756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18100e40 0x7fbe18101220 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.409+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/2862449756 >> 172.21.15.96:0/2862449756 conn(0x7fbe180fc710 msgr2=0x7fbe180feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:52.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.409+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/2862449756 shutdown_connections 2024-04-22T15:54:52.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.409+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/2862449756 wait complete. 2024-04-22T15:54:52.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.410+0000 7fbe1cc3e700 1 Processor -- start 2024-04-22T15:54:52.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.410+0000 7fbe1cc3e700 1 -- start start 2024-04-22T15:54:52.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.410+0000 7fbe1cc3e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe18100e40 0x7fbe1819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.410+0000 7fbe1cc3e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe181017f0 0x7fbe1819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.410+0000 7fbe1cc3e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1819d4a0 con 0x7fbe181017f0 2024-04-22T15:54:52.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.410+0000 7fbe1cc3e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe18196a80 con 0x7fbe18100e40 2024-04-22T15:54:52.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe15d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe181017f0 0x7fbe1819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe15d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe181017f0 0x7fbe1819ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35182/0 (socket says 172.21.15.96:35182) 2024-04-22T15:54:52.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe15d9b700 1 -- 172.21.15.96:0/3304235248 learned_addr learned my addr 172.21.15.96:0/3304235248 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe1659c700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe18100e40 0x7fbe1819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe15d9b700 1 -- 172.21.15.96:0/3304235248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe18100e40 msgr2=0x7fbe1819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe15d9b700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe18100e40 0x7fbe1819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe15d9b700 1 -- 172.21.15.96:0/3304235248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c0095f0 con 0x7fbe181017f0 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe15d9b700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe181017f0 0x7fbe1819ce80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800b480 tx=0x7fbe0800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe077fe700 1 -- 172.21.15.96:0/3304235248 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe080118c0 con 0x7fbe181017f0 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe18196d60 con 0x7fbe181017f0 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe077fe700 1 -- 172.21.15.96:0/3304235248 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe08010380 con 0x7fbe181017f0 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe077fe700 1 -- 172.21.15.96:0/3304235248 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe0800f4f0 con 0x7fbe181017f0 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.411+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe181972b0 con 0x7fbe181017f0 2024-04-22T15:54:52.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.412+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe181091e0 con 0x7fbe181017f0 2024-04-22T15:54:52.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.413+0000 7fbe077fe700 1 -- 172.21.15.96:0/3304235248 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbe080104f0 con 0x7fbe181017f0 2024-04-22T15:54:52.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.414+0000 7fbe077fe700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe0006ecd0 0x7fbe00071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.414+0000 7fbe1659c700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe0006ecd0 0x7fbe00071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.414+0000 7fbe077fe700 1 -- 172.21.15.96:0/3304235248 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbe0808ea20 con 0x7fbe181017f0 2024-04-22T15:54:52.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.414+0000 7fbe1659c700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe0006ecd0 0x7fbe00071190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b5c0 tx=0x7fbe0c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:52.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.418+0000 7fbe077fe700 1 -- 172.21.15.96:0/3304235248 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbe0805b330 con 0x7fbe181017f0 2024-04-22T15:54:52.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.636+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe1810de80 con 0x7fbe0006ecd0 2024-04-22T15:54:52.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.637+0000 7fbe077fe700 1 -- 172.21.15.96:0/3304235248 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbe1810de80 con 0x7fbe0006ecd0 2024-04-22T15:54:52.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe0006ecd0 msgr2=0x7fbe00071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe0006ecd0 0x7fbe00071190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b5c0 tx=0x7fbe0c01a040 comp rx=0 tx=0).stop 2024-04-22T15:54:52.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe181017f0 msgr2=0x7fbe1819ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe181017f0 0x7fbe1819ce80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800b480 tx=0x7fbe0800b840 comp rx=0 tx=0).stop 2024-04-22T15:54:52.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 shutdown_connections 2024-04-22T15:54:52.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe0006ecd0 0x7fbe00071190 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe18100e40 0x7fbe1819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 --2- 172.21.15.96:0/3304235248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe181017f0 0x7fbe1819ce80 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 >> 172.21.15.96:0/3304235248 conn(0x7fbe180fc710 msgr2=0x7fbe180fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:52.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 shutdown_connections 2024-04-22T15:54:52.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.639+0000 7fbe1cc3e700 1 -- 172.21.15.96:0/3304235248 wait complete. 2024-04-22T15:54:52.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.742+0000 7fb2c74d2700 1 -- 172.21.15.96:0/942694717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c0105b70 msgr2=0x7fb2c0105f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.742+0000 7fb2c74d2700 1 --2- 172.21.15.96:0/942694717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c0105b70 0x7fb2c0105f50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009910 tx=0x7fb2bc009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:52.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.742+0000 7fb2c74d2700 1 -- 172.21.15.96:0/942694717 shutdown_connections 2024-04-22T15:54:52.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.742+0000 7fb2c74d2700 1 --2- 172.21.15.96:0/942694717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2c00ffae0 0x7fb2c00fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.742+0000 7fb2c74d2700 1 --2- 172.21.15.96:0/942694717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c0105b70 0x7fb2c0105f50 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.742+0000 7fb2c74d2700 1 -- 172.21.15.96:0/942694717 >> 172.21.15.96:0/942694717 conn(0x7fb2c00fb5f0 msgr2=0x7fb2c00fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:52.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.742+0000 7fb2c74d2700 1 -- 172.21.15.96:0/942694717 shutdown_connections 2024-04-22T15:54:52.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c74d2700 1 -- 172.21.15.96:0/942694717 wait complete. 2024-04-22T15:54:52.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c74d2700 1 Processor -- start 2024-04-22T15:54:52.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c74d2700 1 -- start start 2024-04-22T15:54:52.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c74d2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c00ffae0 0x7fb2c00fbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c74d2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2c0105b70 0x7fb2c00fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c74d2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c00fde30 con 0x7fb2c00ffae0 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c74d2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c00fc8e0 con 0x7fb2c0105b70 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c526e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c00ffae0 0x7fb2c00fbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c526e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c00ffae0 0x7fb2c00fbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35198/0 (socket says 172.21.15.96:35198) 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.743+0000 7fb2c526e700 1 -- 172.21.15.96:0/2800815904 learned_addr learned my addr 172.21.15.96:0/2800815904 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2c4a6d700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2c0105b70 0x7fb2c00fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2c526e700 1 -- 172.21.15.96:0/2800815904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2c0105b70 msgr2=0x7fb2c00fc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2c526e700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2c0105b70 0x7fb2c00fc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2c526e700 1 -- 172.21.15.96:0/2800815904 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc0095f0 con 0x7fb2c00ffae0 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2c526e700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c00ffae0 0x7fb2c00fbe60 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc006010 tx=0x7fb2bc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2b67fc700 1 -- 172.21.15.96:0/2800815904 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2bc01d070 con 0x7fb2c00ffae0 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2b67fc700 1 -- 172.21.15.96:0/2800815904 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2bc00f460 con 0x7fb2c00ffae0 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c00fcb30 con 0x7fb2c00ffae0 2024-04-22T15:54:52.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2b67fc700 1 -- 172.21.15.96:0/2800815904 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2bc017630 con 0x7fb2c00ffae0 2024-04-22T15:54:52.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.744+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c01a2e90 con 0x7fb2c00ffae0 2024-04-22T15:54:52.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.745+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c01090f0 con 0x7fb2c00ffae0 2024-04-22T15:54:52.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.746+0000 7fb2b67fc700 1 -- 172.21.15.96:0/2800815904 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb2bc00f5d0 con 0x7fb2c00ffae0 2024-04-22T15:54:52.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.746+0000 7fb2b67fc700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb2ac06ecc0 0x7fb2ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.746+0000 7fb2b67fc700 1 -- 172.21.15.96:0/2800815904 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb2bc0903a0 con 0x7fb2c00ffae0 2024-04-22T15:54:52.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.746+0000 7fb2c4a6d700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb2ac06ecc0 0x7fb2ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.746+0000 7fb2c4a6d700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb2ac06ecc0 0x7fb2ac071180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0005ea0 tx=0x7fb2b0005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:52.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.751+0000 7fb2b67fc700 1 -- 172.21.15.96:0/2800815904 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb2bc0591c0 con 0x7fb2c00ffae0 2024-04-22T15:54:52.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.961+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2c010cce0 con 0x7fb2ac06ecc0 2024-04-22T15:54:52.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.972+0000 7fb2b67fc700 1 -- 172.21.15.96:0/2800815904 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb2c010cce0 con 0x7fb2ac06ecc0 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (33m) 11m ago 44m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (45m) 11m ago 45m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (44m) 11m ago 44m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (30m) 11m ago 45m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (30m) 11m ago 43m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (33m) 11m ago 44m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (26m) 11m ago 39m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (27m) 11m ago 39m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (26m) 11m ago 39m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (11m) 11m ago 39m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (35m) 11m ago 46m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (34m) 11m ago 43m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (31m) 11m ago 46m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:54:52.975 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (31m) 11m ago 43m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:54:52.976 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (34m) 11m ago 45m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:54:52.976 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (34m) 11m ago 43m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:54:52.976 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (30m) 11m ago 42m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:54:52.976 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (29m) 11m ago 42m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:54:52.976 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (29m) 11m ago 42m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:54:52.976 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (28m) 11m ago 41m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:54:52.976 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (27m) 11m ago 41m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:54:52.976 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (27m) 11m ago 40m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:54:52.976 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (33m) 11m ago 44m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb2ac06ecc0 msgr2=0x7fb2ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb2ac06ecc0 0x7fb2ac071180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0005ea0 tx=0x7fb2b0005e30 comp rx=0 tx=0).stop 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c00ffae0 msgr2=0x7fb2c00fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c00ffae0 0x7fb2c00fbe60 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc006010 tx=0x7fb2bc00be20 comp rx=0 tx=0).stop 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 shutdown_connections 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb2ac06ecc0 0x7fb2ac071180 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2c00ffae0 0x7fb2c00fbe60 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 --2- 172.21.15.96:0/2800815904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2c0105b70 0x7fb2c00fc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 >> 172.21.15.96:0/2800815904 conn(0x7fb2c00fb5f0 msgr2=0x7fb2c0104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 shutdown_connections 2024-04-22T15:54:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:52.974+0000 7fb2c74d2700 1 -- 172.21.15.96:0/2800815904 wait complete. 2024-04-22T15:54:53.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.076+0000 7eff302af700 1 -- 172.21.15.96:0/2509516562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28102680 msgr2=0x7eff2810ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.076+0000 7eff302af700 1 --2- 172.21.15.96:0/2509516562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28102680 0x7eff2810ab70 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7eff18009910 tx=0x7eff18009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:53.079 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:52 smithi096 ceph-mon[112443]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:53.079 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:52 smithi096 ceph-mon[112443]: from='client.35800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.076+0000 7eff302af700 1 -- 172.21.15.96:0/2509516562 shutdown_connections 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.076+0000 7eff302af700 1 --2- 172.21.15.96:0/2509516562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28102680 0x7eff2810ab70 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.076+0000 7eff302af700 1 --2- 172.21.15.96:0/2509516562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff28101d60 0x7eff28102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.076+0000 7eff302af700 1 -- 172.21.15.96:0/2509516562 >> 172.21.15.96:0/2509516562 conn(0x7eff28076740 msgr2=0x7eff280fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.076+0000 7eff302af700 1 -- 172.21.15.96:0/2509516562 shutdown_connections 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.076+0000 7eff302af700 1 -- 172.21.15.96:0/2509516562 wait complete. 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff302af700 1 Processor -- start 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff302af700 1 -- start start 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff302af700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28101d60 0x7eff2819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff302af700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff28102680 0x7eff2819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff302af700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2819d2f0 con 0x7eff28101d60 2024-04-22T15:54:53.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff302af700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff28196960 con 0x7eff28102680 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff2e04b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28101d60 0x7eff2819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff2e04b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28101d60 0x7eff2819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35218/0 (socket says 172.21.15.96:35218) 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff2e04b700 1 -- 172.21.15.96:0/3925017833 learned_addr learned my addr 172.21.15.96:0/3925017833 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff2d84a700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff28102680 0x7eff2819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff2e04b700 1 -- 172.21.15.96:0/3925017833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff28102680 msgr2=0x7eff2819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff2e04b700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff28102680 0x7eff2819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff2e04b700 1 -- 172.21.15.96:0/3925017833 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff180095f0 con 0x7eff28101d60 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff2d84a700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff28102680 0x7eff2819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.077+0000 7eff2e04b700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28101d60 0x7eff2819c820 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7eff2400c930 tx=0x7eff2400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.078+0000 7eff1f7fe700 1 -- 172.21.15.96:0/3925017833 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff24007ad0 con 0x7eff28101d60 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.078+0000 7eff1f7fe700 1 -- 172.21.15.96:0/3925017833 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff24007c30 con 0x7eff28101d60 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.078+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff28196c40 con 0x7eff28101d60 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.078+0000 7eff1f7fe700 1 -- 172.21.15.96:0/3925017833 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff24018610 con 0x7eff28101d60 2024-04-22T15:54:53.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.078+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff28197190 con 0x7eff28101d60 2024-04-22T15:54:53.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.079+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff28105480 con 0x7eff28101d60 2024-04-22T15:54:53.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.079+0000 7eff1f7fe700 1 -- 172.21.15.96:0/3925017833 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7eff2400f450 con 0x7eff28101d60 2024-04-22T15:54:53.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.079+0000 7eff1f7fe700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7eff1406ecc0 0x7eff14071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.080+0000 7eff1f7fe700 1 -- 172.21.15.96:0/3925017833 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7eff2408f0d0 con 0x7eff28101d60 2024-04-22T15:54:53.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.080+0000 7eff2d84a700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7eff1406ecc0 0x7eff14071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.080+0000 7eff2d84a700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7eff1406ecc0 0x7eff14071180 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7eff28197b20 tx=0x7eff1801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.084+0000 7eff1f7fe700 1 -- 172.21.15.96:0/3925017833 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7eff2405b960 con 0x7eff28101d60 2024-04-22T15:54:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:52 smithi196 ceph-mon[81815]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:52 smithi196 ceph-mon[81815]: from='client.35800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:53.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.370+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7eff28197e20 con 0x7eff28101d60 2024-04-22T15:54:53.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.371+0000 7eff1f7fe700 1 -- 172.21.15.96:0/3925017833 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7eff2405b4f0 con 0x7eff28101d60 2024-04-22T15:54:53.374 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:54:53.374 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:54:53.374 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:54:53.374 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:54:53.374 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:54:53.374 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:54:53.374 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:54:53.375 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:54:53.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7eff1406ecc0 msgr2=0x7eff14071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7eff1406ecc0 0x7eff14071180 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7eff28197b20 tx=0x7eff1801a040 comp rx=0 tx=0).stop 2024-04-22T15:54:53.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28101d60 msgr2=0x7eff2819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28101d60 0x7eff2819c820 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7eff2400c930 tx=0x7eff2400ccf0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 shutdown_connections 2024-04-22T15:54:53.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7eff1406ecc0 0x7eff14071180 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff28101d60 0x7eff2819c820 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 --2- 172.21.15.96:0/3925017833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff28102680 0x7eff2819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 >> 172.21.15.96:0/3925017833 conn(0x7eff28076740 msgr2=0x7eff28103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 shutdown_connections 2024-04-22T15:54:53.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.373+0000 7eff302af700 1 -- 172.21.15.96:0/3925017833 wait complete. 2024-04-22T15:54:53.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.475+0000 7fce64f3d700 1 -- 172.21.15.96:0/3763441782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce60105aa0 msgr2=0x7fce60105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.475+0000 7fce64f3d700 1 --2- 172.21.15.96:0/3763441782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce60105aa0 0x7fce60105e80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fce50009910 tx=0x7fce50009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:53.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.476+0000 7fce64f3d700 1 -- 172.21.15.96:0/3763441782 shutdown_connections 2024-04-22T15:54:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.476+0000 7fce64f3d700 1 --2- 172.21.15.96:0/3763441782 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce600ffaa0 0x7fce600fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.476+0000 7fce64f3d700 1 --2- 172.21.15.96:0/3763441782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce60105aa0 0x7fce60105e80 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.476+0000 7fce64f3d700 1 -- 172.21.15.96:0/3763441782 >> 172.21.15.96:0/3763441782 conn(0x7fce60076740 msgr2=0x7fce600fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.476+0000 7fce64f3d700 1 -- 172.21.15.96:0/3763441782 shutdown_connections 2024-04-22T15:54:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.476+0000 7fce64f3d700 1 -- 172.21.15.96:0/3763441782 wait complete. 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.476+0000 7fce64f3d700 1 Processor -- start 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.476+0000 7fce64f3d700 1 -- start start 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce64f3d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce600ffaa0 0x7fce601979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce64f3d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce60105aa0 0x7fce60197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce64f3d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce601984d0 con 0x7fce600ffaa0 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce64f3d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce60198610 con 0x7fce60105aa0 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce5e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce600ffaa0 0x7fce601979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce5e59c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce600ffaa0 0x7fce601979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35232/0 (socket says 172.21.15.96:35232) 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce5e59c700 1 -- 172.21.15.96:0/2063995379 learned_addr learned my addr 172.21.15.96:0/2063995379 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce5dd9b700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce60105aa0 0x7fce60197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce5e59c700 1 -- 172.21.15.96:0/2063995379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce60105aa0 msgr2=0x7fce60197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce5e59c700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce60105aa0 0x7fce60197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce5e59c700 1 -- 172.21.15.96:0/2063995379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce500095f0 con 0x7fce600ffaa0 2024-04-22T15:54:53.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce5e59c700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce600ffaa0 0x7fce601979c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fce5000bd70 tx=0x7fce5000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce4f7fe700 1 -- 172.21.15.96:0/2063995379 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce5001d070 con 0x7fce600ffaa0 2024-04-22T15:54:53.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce4f7fe700 1 -- 172.21.15.96:0/2063995379 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce500056e0 con 0x7fce600ffaa0 2024-04-22T15:54:53.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce4f7fe700 1 -- 172.21.15.96:0/2063995379 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce50017400 con 0x7fce600ffaa0 2024-04-22T15:54:53.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.477+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce601a0a30 con 0x7fce600ffaa0 2024-04-22T15:54:53.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.478+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce601a0d90 con 0x7fce600ffaa0 2024-04-22T15:54:53.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.479+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce601080a0 con 0x7fce600ffaa0 2024-04-22T15:54:53.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.479+0000 7fce4f7fe700 1 -- 172.21.15.96:0/2063995379 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce50003a60 con 0x7fce600ffaa0 2024-04-22T15:54:53.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.479+0000 7fce4f7fe700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce4806ecc0 0x7fce48071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.479+0000 7fce5dd9b700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce4806ecc0 0x7fce48071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.479+0000 7fce4f7fe700 1 -- 172.21.15.96:0/2063995379 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fce500902f0 con 0x7fce600ffaa0 2024-04-22T15:54:53.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.480+0000 7fce5dd9b700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce4806ecc0 0x7fce48071180 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fce54005950 tx=0x7fce540058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.484+0000 7fce4f7fe700 1 -- 172.21.15.96:0/2063995379 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fce50059050 con 0x7fce600ffaa0 2024-04-22T15:54:53.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.731+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fce60068b60 con 0x7fce600ffaa0 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.732+0000 7fce4f7fe700 1 -- 172.21.15.96:0/2063995379 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7fce5005c710 con 0x7fce600ffaa0 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:54:53.735 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:54:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:54:53.737 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:54:53.738 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:54:53.738 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:54:53.738 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:54:53.738 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:54:53.738 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce4806ecc0 msgr2=0x7fce48071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce4806ecc0 0x7fce48071180 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fce54005950 tx=0x7fce540058e0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce600ffaa0 msgr2=0x7fce601979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce600ffaa0 0x7fce601979c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fce5000bd70 tx=0x7fce5000be50 comp rx=0 tx=0).stop 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 shutdown_connections 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce4806ecc0 0x7fce48071180 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce600ffaa0 0x7fce601979c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 --2- 172.21.15.96:0/2063995379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce60105aa0 0x7fce60197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 >> 172.21.15.96:0/2063995379 conn(0x7fce60076740 msgr2=0x7fce600fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 shutdown_connections 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.734+0000 7fce64f3d700 1 -- 172.21.15.96:0/2063995379 wait complete. 2024-04-22T15:54:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:54:53.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.839+0000 7f7f782a1700 1 -- 172.21.15.96:0/1215208693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70101d60 msgr2=0x7f7f70102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.839+0000 7f7f782a1700 1 --2- 172.21.15.96:0/1215208693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70101d60 0x7f7f70102140 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400b0a0 tx=0x7f7f6400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.839+0000 7f7f782a1700 1 -- 172.21.15.96:0/1215208693 shutdown_connections 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.839+0000 7f7f782a1700 1 --2- 172.21.15.96:0/1215208693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f70102680 0x7f7f7010ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.839+0000 7f7f782a1700 1 --2- 172.21.15.96:0/1215208693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70101d60 0x7f7f70102140 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.839+0000 7f7f782a1700 1 -- 172.21.15.96:0/1215208693 >> 172.21.15.96:0/1215208693 conn(0x7f7f70076740 msgr2=0x7f7f700fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.839+0000 7f7f782a1700 1 -- 172.21.15.96:0/1215208693 shutdown_connections 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.839+0000 7f7f782a1700 1 -- 172.21.15.96:0/1215208693 wait complete. 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f782a1700 1 Processor -- start 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f782a1700 1 -- start start 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f782a1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f70101d60 0x7f7f7019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f782a1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70102680 0x7f7f7019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f782a1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7019d3a0 con 0x7f7f70102680 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f782a1700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f70196980 con 0x7f7f70101d60 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f7583c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70102680 0x7f7f7019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f7583c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70102680 0x7f7f7019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35244/0 (socket says 172.21.15.96:35244) 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f7583c700 1 -- 172.21.15.96:0/3692024682 learned_addr learned my addr 172.21.15.96:0/3692024682 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f7603d700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f70101d60 0x7f7f7019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f7583c700 1 -- 172.21.15.96:0/3692024682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f70101d60 msgr2=0x7f7f7019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f7583c700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f70101d60 0x7f7f7019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f7583c700 1 -- 172.21.15.96:0/3692024682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f64009d00 con 0x7f7f70102680 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.840+0000 7f7f7583c700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70102680 0x7f7f7019cd80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00c8f0 tx=0x7f7f5c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.841+0000 7f7f6f7fe700 1 -- 172.21.15.96:0/3692024682 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5c00e850 con 0x7f7f70102680 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.841+0000 7f7f6f7fe700 1 -- 172.21.15.96:0/3692024682 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f5c00ee70 con 0x7f7f70102680 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.841+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f70196c60 con 0x7f7f70102680 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.841+0000 7f7f6f7fe700 1 -- 172.21.15.96:0/3692024682 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5c018630 con 0x7f7f70102680 2024-04-22T15:54:53.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.841+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f701971b0 con 0x7f7f70102680 2024-04-22T15:54:53.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.842+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f70105480 con 0x7f7f70102680 2024-04-22T15:54:53.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.842+0000 7f7f6f7fe700 1 -- 172.21.15.96:0/3692024682 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7f5c00f450 con 0x7f7f70102680 2024-04-22T15:54:53.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.843+0000 7f7f6f7fe700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f6006ecc0 0x7f7f60071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.843+0000 7f7f6f7fe700 1 -- 172.21.15.96:0/3692024682 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7f5c08ec60 con 0x7f7f70102680 2024-04-22T15:54:53.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.843+0000 7f7f7603d700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f6006ecc0 0x7f7f60071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.843+0000 7f7f7603d700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f6006ecc0 0x7f7f60071180 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7f64000f80 tx=0x7f7f640060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:53.848+0000 7f7f6f7fe700 1 -- 172.21.15.96:0/3692024682 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7f5c05b570 con 0x7f7f70102680 2024-04-22T15:54:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:53 smithi096 ceph-mon[112443]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3925017833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:53 smithi096 ceph-mon[112443]: pgmap v992: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:53.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2063995379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:54:54.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.069+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f70197eb0 con 0x7f7f6006ecc0 2024-04-22T15:54:54.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.070+0000 7f7f6f7fe700 1 -- 172.21.15.96:0/3692024682 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7f70197eb0 con 0x7f7f6006ecc0 2024-04-22T15:54:54.073 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:54:54.073 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:54:54.074 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:54:54.075 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:54:54.075 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:54:54.075 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:54:54.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.072+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f6006ecc0 msgr2=0x7f7f60071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.072+0000 7f7f782a1700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f6006ecc0 0x7f7f60071180 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7f64000f80 tx=0x7f7f640060c0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.073+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70102680 msgr2=0x7f7f7019cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.073+0000 7f7f782a1700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70102680 0x7f7f7019cd80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00c8f0 tx=0x7f7f5c00cc00 comp rx=0 tx=0).stop 2024-04-22T15:54:54.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.073+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 shutdown_connections 2024-04-22T15:54:54.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.073+0000 7f7f782a1700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f6006ecc0 0x7f7f60071180 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.073+0000 7f7f782a1700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f70101d60 0x7f7f7019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.073+0000 7f7f782a1700 1 --2- 172.21.15.96:0/3692024682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70102680 0x7f7f7019cd80 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.073+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 >> 172.21.15.96:0/3692024682 conn(0x7f7f70076740 msgr2=0x7f7f70103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:54.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.073+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 shutdown_connections 2024-04-22T15:54:54.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.073+0000 7f7f782a1700 1 -- 172.21.15.96:0/3692024682 wait complete. 2024-04-22T15:54:54.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/513306973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc102680 msgr2=0x7fb7cc10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 --2- 172.21.15.96:0/513306973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc102680 0x7fb7cc10ab70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009910 tx=0x7fb7b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/513306973 shutdown_connections 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 --2- 172.21.15.96:0/513306973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc102680 0x7fb7cc10ab70 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 --2- 172.21.15.96:0/513306973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7cc101d60 0x7fb7cc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/513306973 >> 172.21.15.96:0/513306973 conn(0x7fb7cc076740 msgr2=0x7fb7cc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/513306973 shutdown_connections 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/513306973 wait complete. 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 Processor -- start 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.176+0000 7fb7d1ab3700 1 -- start start 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7d1ab3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc101d60 0x7fb7cc19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7d1ab3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7cc102680 0x7fb7cc19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7d1ab3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc19d380 con 0x7fb7cc101d60 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7d1ab3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc196960 con 0x7fb7cc102680 2024-04-22T15:54:54.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7cb7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc101d60 0x7fb7cc19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7cb7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc101d60 0x7fb7cc19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35262/0 (socket says 172.21.15.96:35262) 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7cb7fe700 1 -- 172.21.15.96:0/4119247995 learned_addr learned my addr 172.21.15.96:0/4119247995 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7cb7fe700 1 -- 172.21.15.96:0/4119247995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7cc102680 msgr2=0x7fb7cc19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7cb7fe700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7cc102680 0x7fb7cc19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7cb7fe700 1 -- 172.21.15.96:0/4119247995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b40095f0 con 0x7fb7cc101d60 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7cb7fe700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc101d60 0x7fb7cc19c820 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00d900 tx=0x7fb7bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7c97fa700 1 -- 172.21.15.96:0/4119247995 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7bc0041f0 con 0x7fb7cc101d60 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.177+0000 7fb7c97fa700 1 -- 172.21.15.96:0/4119247995 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7bc004cf0 con 0x7fb7cc101d60 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.178+0000 7fb7c97fa700 1 -- 172.21.15.96:0/4119247995 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7bc00b750 con 0x7fb7cc101d60 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.178+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc196c40 con 0x7fb7cc101d60 2024-04-22T15:54:54.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.178+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc197140 con 0x7fb7cc101d60 2024-04-22T15:54:54.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.179+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7cc105480 con 0x7fb7cc101d60 2024-04-22T15:54:54.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.179+0000 7fb7c97fa700 1 -- 172.21.15.96:0/4119247995 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb7bc004350 con 0x7fb7cc101d60 2024-04-22T15:54:54.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.180+0000 7fb7c97fa700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb7b806ecc0 0x7fb7b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:54.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.180+0000 7fb7c97fa700 1 -- 172.21.15.96:0/4119247995 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb7bc08f000 con 0x7fb7cc101d60 2024-04-22T15:54:54.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.180+0000 7fb7c2ffd700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb7b806ecc0 0x7fb7b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:54.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.180+0000 7fb7c2ffd700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb7b806ecc0 0x7fb7b8071180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400b5c0 tx=0x7fb7b4005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:54.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.184+0000 7fb7c97fa700 1 -- 172.21.15.96:0/4119247995 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb7bc05b890 con 0x7fb7cc101d60 2024-04-22T15:54:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:53 smithi196 ceph-mon[81815]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3925017833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:53 smithi196 ceph-mon[81815]: pgmap v992: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2063995379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:54:54.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.470+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb7cc068ea0 con 0x7fb7cc101d60 2024-04-22T15:54:54.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.470+0000 7fb7c97fa700 1 -- 172.21.15.96:0/4119247995 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fb7bc05b420 con 0x7fb7cc101d60 2024-04-22T15:54:54.473 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:54:54.473 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:54:54.473 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:54:54.473 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:54:54.473 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:54:54.473 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:54:54.474 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:54:54.475 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:54:54.475 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:54:54.475 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:54:54.475 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:54:54.475 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:54:54.475 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:54:54.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb7b806ecc0 msgr2=0x7fb7b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb7b806ecc0 0x7fb7b8071180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400b5c0 tx=0x7fb7b4005e20 comp rx=0 tx=0).stop 2024-04-22T15:54:54.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc101d60 msgr2=0x7fb7cc19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc101d60 0x7fb7cc19c820 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00d900 tx=0x7fb7bc00dcc0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 shutdown_connections 2024-04-22T15:54:54.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb7b806ecc0 0x7fb7b8071180 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc101d60 0x7fb7cc19c820 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 --2- 172.21.15.96:0/4119247995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7cc102680 0x7fb7cc19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 >> 172.21.15.96:0/4119247995 conn(0x7fb7cc076740 msgr2=0x7fb7cc103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:54.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 shutdown_connections 2024-04-22T15:54:54.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:54:54.473+0000 7fb7d1ab3700 1 -- 172.21.15.96:0/4119247995 wait complete. 2024-04-22T15:54:55.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:54 smithi096 ceph-mon[112443]: from='client.35816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:55.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:54 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4119247995' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:54:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:54 smithi196 ceph-mon[81815]: from='client.35816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:54 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4119247995' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:54:56.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:55 smithi096 ceph-mon[112443]: pgmap v993: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:55 smithi196 ceph-mon[81815]: pgmap v993: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:58.049 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:57 smithi196 ceph-mon[81815]: pgmap v994: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:54:58.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:57 smithi096 ceph-mon[112443]: pgmap v994: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:00.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:54:59 smithi096 ceph-mon[112443]: pgmap v995: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:00.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:54:59 smithi196 ceph-mon[81815]: pgmap v995: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:02.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:01 smithi096 ceph-mon[112443]: pgmap v996: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:02.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:01 smithi196 ceph-mon[81815]: pgmap v996: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:04.037 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:03 smithi096 ceph-mon[112443]: pgmap v997: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:04.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:03 smithi196 ceph-mon[81815]: pgmap v997: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:06.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:05 smithi096 ceph-mon[112443]: pgmap v998: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:06.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:05 smithi196 ceph-mon[81815]: pgmap v998: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:08.156 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:07 smithi196 ceph-mon[81815]: pgmap v999: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:08.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:07 smithi096 ceph-mon[112443]: pgmap v999: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:09.158 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:10.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:09 smithi096 ceph-mon[112443]: pgmap v1000: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:10.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:09 smithi196 ceph-mon[81815]: pgmap v1000: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:12.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:11 smithi196 ceph-mon[81815]: pgmap v1001: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:12.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:11 smithi096 ceph-mon[112443]: pgmap v1001: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:14.212 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:13 smithi096 ceph-mon[112443]: pgmap v1002: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:14.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:13 smithi196 ceph-mon[81815]: pgmap v1002: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:16.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:15 smithi096 ceph-mon[112443]: pgmap v1003: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:16.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:15 smithi196 ceph-mon[81815]: pgmap v1003: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:18.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:17 smithi096 ceph-mon[112443]: pgmap v1004: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:18.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:17 smithi196 ceph-mon[81815]: pgmap v1004: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:20.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:19 smithi096 ceph-mon[112443]: pgmap v1005: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:20.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:19 smithi196 ceph-mon[81815]: pgmap v1005: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:22.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:21 smithi096 ceph-mon[112443]: pgmap v1006: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:21 smithi196 ceph-mon[81815]: pgmap v1006: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:23.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:24.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.586+0000 7f74e5db5700 1 -- 172.21.15.96:0/2030873828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 msgr2=0x7f74e0105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:24.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.586+0000 7f74e5db5700 1 --2- 172.21.15.96:0/2030873828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 0x7f74e0105fe0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009910 tx=0x7f74d0009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:24.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.587+0000 7f74e5db5700 1 -- 172.21.15.96:0/2030873828 shutdown_connections 2024-04-22T15:55:24.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.587+0000 7f74e5db5700 1 --2- 172.21.15.96:0/2030873828 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74e00ffb50 0x7f74e00fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:24.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.587+0000 7f74e5db5700 1 --2- 172.21.15.96:0/2030873828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 0x7f74e0105fe0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:24.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.587+0000 7f74e5db5700 1 -- 172.21.15.96:0/2030873828 >> 172.21.15.96:0/2030873828 conn(0x7f74e00fb680 msgr2=0x7f74e00fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:24.590 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:24 smithi096 ceph-mon[112443]: pgmap v1007: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:24.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.587+0000 7f74e5db5700 1 -- 172.21.15.96:0/2030873828 shutdown_connections 2024-04-22T15:55:24.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.587+0000 7f74e5db5700 1 -- 172.21.15.96:0/2030873828 wait complete. 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.587+0000 7f74e5db5700 1 Processor -- start 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.587+0000 7f74e5db5700 1 -- start start 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74e5db5700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74e00ffb50 0x7f74e0197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74e5db5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 0x7f74e0197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74e5db5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e0198430 con 0x7f74e0105c00 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74e5db5700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e0198570 con 0x7f74e00ffb50 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74deffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 0x7f74e0197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74deffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 0x7f74e0197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40102/0 (socket says 172.21.15.96:40102) 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74deffd700 1 -- 172.21.15.96:0/1834301587 learned_addr learned my addr 172.21.15.96:0/1834301587 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74df7fe700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74e00ffb50 0x7f74e0197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74deffd700 1 -- 172.21.15.96:0/1834301587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74e00ffb50 msgr2=0x7f74e0197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74deffd700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74e00ffb50 0x7f74e0197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74deffd700 1 -- 172.21.15.96:0/1834301587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d00095f0 con 0x7f74e0105c00 2024-04-22T15:55:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74df7fe700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74e00ffb50 0x7f74e0197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:55:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74deffd700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 0x7f74e0197e60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f74d400d8d0 tx=0x7f74d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.588+0000 7f74dcff9700 1 -- 172.21.15.96:0/1834301587 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74d4009940 con 0x7f74e0105c00 2024-04-22T15:55:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.589+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e01a0990 con 0x7f74e0105c00 2024-04-22T15:55:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.589+0000 7f74dcff9700 1 -- 172.21.15.96:0/1834301587 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74d4010460 con 0x7f74e0105c00 2024-04-22T15:55:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.589+0000 7f74dcff9700 1 -- 172.21.15.96:0/1834301587 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74d400f5f0 con 0x7f74e0105c00 2024-04-22T15:55:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.589+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e01a0da0 con 0x7f74e0105c00 2024-04-22T15:55:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.590+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e0108030 con 0x7f74e0105c00 2024-04-22T15:55:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.590+0000 7f74dcff9700 1 -- 172.21.15.96:0/1834301587 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f74d40105d0 con 0x7f74e0105c00 2024-04-22T15:55:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.591+0000 7f74dcff9700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74c806ecc0 0x7f74c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.591+0000 7f74dcff9700 1 -- 172.21.15.96:0/1834301587 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f74d408eed0 con 0x7f74e0105c00 2024-04-22T15:55:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.591+0000 7f74df7fe700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74c806ecc0 0x7f74c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.591+0000 7f74df7fe700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74c806ecc0 0x7f74c8071180 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f74d0006010 tx=0x7f74d000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.596+0000 7f74dcff9700 1 -- 172.21.15.96:0/1834301587 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f74d4057c30 con 0x7f74e0105c00 2024-04-22T15:55:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:24 smithi196 ceph-mon[81815]: pgmap v1007: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:24.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.815+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f74e010cd40 con 0x7f74c806ecc0 2024-04-22T15:55:24.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.816+0000 7f74dcff9700 1 -- 172.21.15.96:0/1834301587 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f74e010cd40 con 0x7f74c806ecc0 2024-04-22T15:55:24.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74c806ecc0 msgr2=0x7f74c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74c806ecc0 0x7f74c8071180 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f74d0006010 tx=0x7f74d000b540 comp rx=0 tx=0).stop 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 msgr2=0x7f74e0197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 0x7f74e0197e60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f74d400d8d0 tx=0x7f74d400dc90 comp rx=0 tx=0).stop 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 shutdown_connections 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f74c806ecc0 0x7f74c8071180 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74e00ffb50 0x7f74e0197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 --2- 172.21.15.96:0/1834301587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e0105c00 0x7f74e0197e60 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 >> 172.21.15.96:0/1834301587 conn(0x7f74e00fb680 msgr2=0x7f74e00fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.818+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 shutdown_connections 2024-04-22T15:55:24.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.819+0000 7f74e5db5700 1 -- 172.21.15.96:0/1834301587 wait complete. 2024-04-22T15:55:24.831 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:55:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.923+0000 7f740d336700 1 -- 172.21.15.96:0/2466730380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 msgr2=0x7f7408101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.923+0000 7f740d336700 1 --2- 172.21.15.96:0/2466730380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 0x7f7408101200 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009910 tx=0x7f73f8009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.923+0000 7f740d336700 1 -- 172.21.15.96:0/2466730380 shutdown_connections 2024-04-22T15:55:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.923+0000 7f740d336700 1 --2- 172.21.15.96:0/2466730380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74081017d0 0x7f7408105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.923+0000 7f740d336700 1 --2- 172.21.15.96:0/2466730380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 0x7f7408101200 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.923+0000 7f740d336700 1 -- 172.21.15.96:0/2466730380 >> 172.21.15.96:0/2466730380 conn(0x7f74080fc6d0 msgr2=0x7f74080feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.924+0000 7f740d336700 1 -- 172.21.15.96:0/2466730380 shutdown_connections 2024-04-22T15:55:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.924+0000 7f740d336700 1 -- 172.21.15.96:0/2466730380 wait complete. 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.924+0000 7f740d336700 1 Processor -- start 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.924+0000 7f740d336700 1 -- start start 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.924+0000 7f740d336700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 0x7f740819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.924+0000 7f740d336700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74081017d0 0x7f740819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.924+0000 7f740d336700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740819d4a0 con 0x7f7408100e20 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.924+0000 7f740d336700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7408196a80 con 0x7f74081017d0 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f7406ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 0x7f740819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f7406ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 0x7f740819c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40114/0 (socket says 172.21.15.96:40114) 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f7406ffd700 1 -- 172.21.15.96:0/2432735599 learned_addr learned my addr 172.21.15.96:0/2432735599 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f74067fc700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74081017d0 0x7f740819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f7406ffd700 1 -- 172.21.15.96:0/2432735599 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74081017d0 msgr2=0x7f740819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f7406ffd700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74081017d0 0x7f740819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f7406ffd700 1 -- 172.21.15.96:0/2432735599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f80095f0 con 0x7f7408100e20 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f74067fc700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74081017d0 0x7f740819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f7406ffd700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 0x7f740819c940 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f73f8006010 tx=0x7f73f800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f73effff700 1 -- 172.21.15.96:0/2432735599 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73f801d070 con 0x7f7408100e20 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f73effff700 1 -- 172.21.15.96:0/2432735599 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f73f800f460 con 0x7f7408100e20 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7408196d00 con 0x7f7408100e20 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74081971f0 con 0x7f7408100e20 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.925+0000 7f73effff700 1 -- 172.21.15.96:0/2432735599 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73f8017630 con 0x7f7408100e20 2024-04-22T15:55:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.926+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f40052f0 con 0x7f7408100e20 2024-04-22T15:55:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.927+0000 7f73effff700 1 -- 172.21.15.96:0/2432735599 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f73f800fab0 con 0x7f7408100e20 2024-04-22T15:55:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.927+0000 7f73effff700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f73f006ecc0 0x7f73f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.927+0000 7f73effff700 1 -- 172.21.15.96:0/2432735599 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f73f80902e0 con 0x7f7408100e20 2024-04-22T15:55:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.927+0000 7f74067fc700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f73f006ecc0 0x7f73f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.928+0000 7f74067fc700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f73f006ecc0 0x7f73f0071180 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7408197c40 tx=0x7f73fc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:24.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:24.932+0000 7f73effff700 1 -- 172.21.15.96:0/2432735599 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f73f805cc30 con 0x7f7408100e20 2024-04-22T15:55:25.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.149+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f73f4000bc0 con 0x7f73f006ecc0 2024-04-22T15:55:25.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.150+0000 7f73effff700 1 -- 172.21.15.96:0/2432735599 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f73f4000bc0 con 0x7f73f006ecc0 2024-04-22T15:55:25.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f73f006ecc0 msgr2=0x7f73f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f73f006ecc0 0x7f73f0071180 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7408197c40 tx=0x7f73fc008040 comp rx=0 tx=0).stop 2024-04-22T15:55:25.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 msgr2=0x7f740819c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 0x7f740819c940 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f73f8006010 tx=0x7f73f800be20 comp rx=0 tx=0).stop 2024-04-22T15:55:25.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 shutdown_connections 2024-04-22T15:55:25.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f73f006ecc0 0x7f73f0071180 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7408100e20 0x7f740819c940 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 --2- 172.21.15.96:0/2432735599 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74081017d0 0x7f740819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 >> 172.21.15.96:0/2432735599 conn(0x7f74080fc6d0 msgr2=0x7f74080fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:25.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 shutdown_connections 2024-04-22T15:55:25.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.152+0000 7f740d336700 1 -- 172.21.15.96:0/2432735599 wait complete. 2024-04-22T15:55:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 -- 172.21.15.96:0/4270602848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71340ffab0 msgr2=0x7f71340fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 --2- 172.21.15.96:0/4270602848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71340ffab0 0x7f71340fff30 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f7128009910 tx=0x7f7128009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 -- 172.21.15.96:0/4270602848 shutdown_connections 2024-04-22T15:55:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 --2- 172.21.15.96:0/4270602848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71340ffab0 0x7f71340fff30 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 --2- 172.21.15.96:0/4270602848 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7134105ab0 0x7f7134105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 -- 172.21.15.96:0/4270602848 >> 172.21.15.96:0/4270602848 conn(0x7f71340fb5e0 msgr2=0x7f71340fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:25.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 -- 172.21.15.96:0/4270602848 shutdown_connections 2024-04-22T15:55:25.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 -- 172.21.15.96:0/4270602848 wait complete. 2024-04-22T15:55:25.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 Processor -- start 2024-04-22T15:55:25.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.256+0000 7f7139f6e700 1 -- start start 2024-04-22T15:55:25.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7139f6e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71340ffab0 0x7f713419b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7139f6e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7134105ab0 0x7f713419bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7139f6e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713419c290 con 0x7f7134105ab0 2024-04-22T15:55:25.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7139f6e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7134195850 con 0x7f71340ffab0 2024-04-22T15:55:25.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7132ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7134105ab0 0x7f713419bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7132ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7134105ab0 0x7f713419bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40132/0 (socket says 172.21.15.96:40132) 2024-04-22T15:55:25.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7132ffd700 1 -- 172.21.15.96:0/1424319706 learned_addr learned my addr 172.21.15.96:0/1424319706 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f71337fe700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71340ffab0 0x7f713419b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7132ffd700 1 -- 172.21.15.96:0/1424319706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71340ffab0 msgr2=0x7f713419b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7132ffd700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71340ffab0 0x7f713419b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7132ffd700 1 -- 172.21.15.96:0/1424319706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71280095f0 con 0x7f7134105ab0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7132ffd700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7134105ab0 0x7f713419bc00 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7128006010 tx=0x7f712800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.257+0000 7f7130ff9700 1 -- 172.21.15.96:0/1424319706 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f712801d070 con 0x7f7134105ab0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.258+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7134195ad0 con 0x7f7134105ab0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.258+0000 7f7130ff9700 1 -- 172.21.15.96:0/1424319706 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7128005700 con 0x7f7134105ab0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.258+0000 7f7130ff9700 1 -- 172.21.15.96:0/1424319706 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7128017400 con 0x7f7134105ab0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.258+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7134195fc0 con 0x7f7134105ab0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.258+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71341080d0 con 0x7f7134105ab0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.259+0000 7f7130ff9700 1 -- 172.21.15.96:0/1424319706 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f712800f460 con 0x7f7134105ab0 2024-04-22T15:55:25.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.259+0000 7f7130ff9700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f711c06ecc0 0x7f711c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.259+0000 7f7130ff9700 1 -- 172.21.15.96:0/1424319706 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f71280902f0 con 0x7f7134105ab0 2024-04-22T15:55:25.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.259+0000 7f71337fe700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f711c06ecc0 0x7f711c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.260+0000 7f71337fe700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f711c06ecc0 0x7f711c071180 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7124005950 tx=0x7f712400a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.264+0000 7f7130ff9700 1 -- 172.21.15.96:0/1424319706 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f712805cb80 con 0x7f7134105ab0 2024-04-22T15:55:25.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.471+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f713410ccb0 con 0x7f711c06ecc0 2024-04-22T15:55:25.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.481+0000 7f7130ff9700 1 -- 172.21.15.96:0/1424319706 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f713410ccb0 con 0x7f711c06ecc0 2024-04-22T15:55:25.484 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (34m) 11m ago 45m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (45m) 11m ago 45m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (44m) 11m ago 44m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (31m) 11m ago 45m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (31m) 11m ago 44m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (33m) 11m ago 45m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (27m) 11m ago 39m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (27m) 11m ago 39m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (26m) 11m ago 39m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (11m) 11m ago 39m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (36m) 11m ago 46m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (35m) 11m ago 44m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (32m) 11m ago 47m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:55:25.485 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (32m) 11m ago 44m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:55:25.486 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (35m) 11m ago 45m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:55:25.486 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (34m) 11m ago 44m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:55:25.486 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (30m) 11m ago 43m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:55:25.486 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (30m) 11m ago 43m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:55:25.486 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (29m) 11m ago 42m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:55:25.486 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (29m) 11m ago 42m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:55:25.486 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (28m) 11m ago 41m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:55:25.486 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (27m) 11m ago 41m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:55:25.486 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (34m) 11m ago 44m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:55:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.483+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f711c06ecc0 msgr2=0x7f711c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.483+0000 7f7139f6e700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f711c06ecc0 0x7f711c071180 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7124005950 tx=0x7f712400a300 comp rx=0 tx=0).stop 2024-04-22T15:55:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.483+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7134105ab0 msgr2=0x7f713419bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.483+0000 7f7139f6e700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7134105ab0 0x7f713419bc00 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7128006010 tx=0x7f712800bd70 comp rx=0 tx=0).stop 2024-04-22T15:55:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.483+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 shutdown_connections 2024-04-22T15:55:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.484+0000 7f7139f6e700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f711c06ecc0 0x7f711c071180 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.484+0000 7f7139f6e700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71340ffab0 0x7f713419b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.484+0000 7f7139f6e700 1 --2- 172.21.15.96:0/1424319706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7134105ab0 0x7f713419bc00 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.484+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 >> 172.21.15.96:0/1424319706 conn(0x7f71340fb5e0 msgr2=0x7f71340fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.484+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 shutdown_connections 2024-04-22T15:55:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.484+0000 7f7139f6e700 1 -- 172.21.15.96:0/1424319706 wait complete. 2024-04-22T15:55:25.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.586+0000 7f479a17b700 1 -- 172.21.15.96:0/1115671863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 msgr2=0x7f479410ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.586+0000 7f479a17b700 1 --2- 172.21.15.96:0/1115671863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 0x7f479410ab80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f4788009910 tx=0x7f4788009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:25.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.586+0000 7f479a17b700 1 -- 172.21.15.96:0/1115671863 shutdown_connections 2024-04-22T15:55:25.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.586+0000 7f479a17b700 1 --2- 172.21.15.96:0/1115671863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 0x7f479410ab80 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.586+0000 7f479a17b700 1 --2- 172.21.15.96:0/1115671863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4794101ce0 0x7f47941020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.586+0000 7f479a17b700 1 -- 172.21.15.96:0/1115671863 >> 172.21.15.96:0/1115671863 conn(0x7f47940fb5e0 msgr2=0x7f47940fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:25.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.586+0000 7f479a17b700 1 -- 172.21.15.96:0/1115671863 shutdown_connections 2024-04-22T15:55:25.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.586+0000 7f479a17b700 1 -- 172.21.15.96:0/1115671863 wait complete. 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f479a17b700 1 Processor -- start 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f479a17b700 1 -- start start 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f479a17b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4794101ce0 0x7f479419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f479a17b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 0x7f479419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f479a17b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479419d3b0 con 0x7f4794102690 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f479a17b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4794196990 con 0x7f4794101ce0 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f4792ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 0x7f479419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f4792ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 0x7f479419cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40150/0 (socket says 172.21.15.96:40150) 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f4792ffd700 1 -- 172.21.15.96:0/4235731200 learned_addr learned my addr 172.21.15.96:0/4235731200 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f47937fe700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4794101ce0 0x7f479419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f4792ffd700 1 -- 172.21.15.96:0/4235731200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4794101ce0 msgr2=0x7f479419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.587+0000 7f4792ffd700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4794101ce0 0x7f479419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.588+0000 7f4792ffd700 1 -- 172.21.15.96:0/4235731200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880095f0 con 0x7f4794102690 2024-04-22T15:55:25.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.588+0000 7f4792ffd700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 0x7f479419cd90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f478800bd70 tx=0x7f478800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.588+0000 7f4790ff9700 1 -- 172.21.15.96:0/4235731200 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f478801d070 con 0x7f4794102690 2024-04-22T15:55:25.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.588+0000 7f4790ff9700 1 -- 172.21.15.96:0/4235731200 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47880056e0 con 0x7f4794102690 2024-04-22T15:55:25.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.588+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4794196c10 con 0x7f4794102690 2024-04-22T15:55:25.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.588+0000 7f4790ff9700 1 -- 172.21.15.96:0/4235731200 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f478800fe50 con 0x7f4794102690 2024-04-22T15:55:25.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.588+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4794197100 con 0x7f4794102690 2024-04-22T15:55:25.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.589+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4794105500 con 0x7f4794102690 2024-04-22T15:55:25.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.589+0000 7f4790ff9700 1 -- 172.21.15.96:0/4235731200 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f478800f680 con 0x7f4794102690 2024-04-22T15:55:25.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.590+0000 7f4790ff9700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f477c06ecc0 0x7f477c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.590+0000 7f4790ff9700 1 -- 172.21.15.96:0/4235731200 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4788090130 con 0x7f4794102690 2024-04-22T15:55:25.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.590+0000 7f47937fe700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f477c06ecc0 0x7f477c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.591+0000 7f47937fe700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f477c06ecc0 0x7f477c071180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4784004630 tx=0x7f478400b3c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.595+0000 7f4790ff9700 1 -- 172.21.15.96:0/4235731200 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f478805ca40 con 0x7f4794102690 2024-04-22T15:55:25.765 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:25 smithi096 ceph-mon[112443]: from='client.35824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:25.767 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:25 smithi096 ceph-mon[112443]: from='client.35828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:25.767 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:25 smithi096 ceph-mon[112443]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:25.767 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:25 smithi096 ceph-mon[112443]: pgmap v1008: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:25.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.878+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4794069730 con 0x7f4794102690 2024-04-22T15:55:25.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.878+0000 7f4790ff9700 1 -- 172.21.15.96:0/4235731200 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f478805c5d0 con 0x7f4794102690 2024-04-22T15:55:25.881 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:55:25.881 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:55:25.881 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:55:25.882 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:55:25.883 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:55:25.883 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:55:25.883 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:55:25.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.880+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f477c06ecc0 msgr2=0x7f477c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.880+0000 7f479a17b700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f477c06ecc0 0x7f477c071180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4784004630 tx=0x7f478400b3c0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.880+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 msgr2=0x7f479419cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.880+0000 7f479a17b700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 0x7f479419cd90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f478800bd70 tx=0x7f478800bda0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.880+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 shutdown_connections 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.880+0000 7f479a17b700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f477c06ecc0 0x7f477c071180 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.880+0000 7f479a17b700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4794101ce0 0x7f479419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.880+0000 7f479a17b700 1 --2- 172.21.15.96:0/4235731200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4794102690 0x7f479419cd90 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.880+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 >> 172.21.15.96:0/4235731200 conn(0x7f47940fb5e0 msgr2=0x7f4794103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.881+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 shutdown_connections 2024-04-22T15:55:25.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.881+0000 7f479a17b700 1 -- 172.21.15.96:0/4235731200 wait complete. 2024-04-22T15:55:25.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.983+0000 7f42dd1ee700 1 -- 172.21.15.96:0/4061463507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 msgr2=0x7f42d8105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.983+0000 7f42dd1ee700 1 --2- 172.21.15.96:0/4061463507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 0x7f42d8105e80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009910 tx=0x7f42d0009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:25.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.983+0000 7f42dd1ee700 1 -- 172.21.15.96:0/4061463507 shutdown_connections 2024-04-22T15:55:25.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.983+0000 7f42dd1ee700 1 --2- 172.21.15.96:0/4061463507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42d80ffaa0 0x7f42d80fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.983+0000 7f42dd1ee700 1 --2- 172.21.15.96:0/4061463507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 0x7f42d8105e80 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.983+0000 7f42dd1ee700 1 -- 172.21.15.96:0/4061463507 >> 172.21.15.96:0/4061463507 conn(0x7f42d8076740 msgr2=0x7f42d80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:25.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.983+0000 7f42dd1ee700 1 -- 172.21.15.96:0/4061463507 shutdown_connections 2024-04-22T15:55:25.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.983+0000 7f42dd1ee700 1 -- 172.21.15.96:0/4061463507 wait complete. 2024-04-22T15:55:25.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42dd1ee700 1 Processor -- start 2024-04-22T15:55:25.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42dd1ee700 1 -- start start 2024-04-22T15:55:25.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42dd1ee700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42d80ffaa0 0x7f42d81979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42dd1ee700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 0x7f42d8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42dd1ee700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d81984d0 con 0x7f42d8105aa0 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42dd1ee700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d8198610 con 0x7f42d80ffaa0 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42d659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 0x7f42d8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42d659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 0x7f42d8197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40176/0 (socket says 172.21.15.96:40176) 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42d659c700 1 -- 172.21.15.96:0/3651622930 learned_addr learned my addr 172.21.15.96:0/3651622930 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42d659c700 1 -- 172.21.15.96:0/3651622930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42d80ffaa0 msgr2=0x7f42d81979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42d659c700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42d80ffaa0 0x7f42d81979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.984+0000 7f42d659c700 1 -- 172.21.15.96:0/3651622930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d00095f0 con 0x7f42d8105aa0 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.985+0000 7f42d659c700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 0x7f42d8197f00 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00d8d0 tx=0x7f42cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.985+0000 7f42bffff700 1 -- 172.21.15.96:0/3651622930 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42cc009940 con 0x7f42d8105aa0 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.985+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d81a0a30 con 0x7f42d8105aa0 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.985+0000 7f42bffff700 1 -- 172.21.15.96:0/3651622930 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42cc010460 con 0x7f42d8105aa0 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.985+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d81a0e40 con 0x7f42d8105aa0 2024-04-22T15:55:25.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.985+0000 7f42bffff700 1 -- 172.21.15.96:0/3651622930 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42cc00f5f0 con 0x7f42d8105aa0 2024-04-22T15:55:25.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.986+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d81080a0 con 0x7f42d8105aa0 2024-04-22T15:55:25.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.986+0000 7f42bffff700 1 -- 172.21.15.96:0/3651622930 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f42cc0105d0 con 0x7f42d8105aa0 2024-04-22T15:55:25.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.987+0000 7f42bffff700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f42c006ecc0 0x7f42c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.987+0000 7f42bffff700 1 -- 172.21.15.96:0/3651622930 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f42cc08eec0 con 0x7f42d8105aa0 2024-04-22T15:55:25.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.987+0000 7f42d6d9d700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f42c006ecc0 0x7f42c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.987+0000 7f42d6d9d700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f42c006ecc0 0x7f42c0071180 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f42d000b5c0 tx=0x7f42d0005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:25.991+0000 7f42bffff700 1 -- 172.21.15.96:0/3651622930 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f42cc057c20 con 0x7f42d8105aa0 2024-04-22T15:55:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.235+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f42d8068b60 con 0x7f42d8105aa0 2024-04-22T15:55:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.236+0000 7f42bffff700 1 -- 172.21.15.96:0/3651622930 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f42cc05b2e0 con 0x7f42d8105aa0 2024-04-22T15:55:26.239 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:55:26.239 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:55:26.239 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:55:26.239 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:55:26.239 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:55:26.239 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:55:26.239 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:55:26.239 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:55:26.239 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:55:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:55:26.241 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:55:26.242 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:55:26.242 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:55:26.242 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:55:26.242 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:55:26.242 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:55:26.242 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:55:26.242 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:55:26.242 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:55:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f42c006ecc0 msgr2=0x7f42c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f42c006ecc0 0x7f42c0071180 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f42d000b5c0 tx=0x7f42d0005e20 comp rx=0 tx=0).stop 2024-04-22T15:55:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 msgr2=0x7f42d8197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 0x7f42d8197f00 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00d8d0 tx=0x7f42cc00dc90 comp rx=0 tx=0).stop 2024-04-22T15:55:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 shutdown_connections 2024-04-22T15:55:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f42c006ecc0 0x7f42c0071180 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42d80ffaa0 0x7f42d81979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 --2- 172.21.15.96:0/3651622930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d8105aa0 0x7f42d8197f00 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 >> 172.21.15.96:0/3651622930 conn(0x7f42d8076740 msgr2=0x7f42d80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 shutdown_connections 2024-04-22T15:55:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.238+0000 7f42dd1ee700 1 -- 172.21.15.96:0/3651622930 wait complete. 2024-04-22T15:55:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:55:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:25 smithi196 ceph-mon[81815]: from='client.35824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:25 smithi196 ceph-mon[81815]: from='client.35828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:25 smithi196 ceph-mon[81815]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:25 smithi196 ceph-mon[81815]: pgmap v1008: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.341+0000 7f912e51a700 1 -- 172.21.15.96:0/1627258933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 msgr2=0x7f91280fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.341+0000 7f912e51a700 1 --2- 172.21.15.96:0/1627258933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 0x7f91280fff50 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f911c009910 tx=0x7f911c009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.341+0000 7f912e51a700 1 -- 172.21.15.96:0/1627258933 shutdown_connections 2024-04-22T15:55:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.341+0000 7f912e51a700 1 --2- 172.21.15.96:0/1627258933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 0x7f91280fff50 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.341+0000 7f912e51a700 1 --2- 172.21.15.96:0/1627258933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9128105b60 0x7f9128105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.341+0000 7f912e51a700 1 -- 172.21.15.96:0/1627258933 >> 172.21.15.96:0/1627258933 conn(0x7f91280fb5e0 msgr2=0x7f91280fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.341+0000 7f912e51a700 1 -- 172.21.15.96:0/1627258933 shutdown_connections 2024-04-22T15:55:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.341+0000 7f912e51a700 1 -- 172.21.15.96:0/1627258933 wait complete. 2024-04-22T15:55:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.342+0000 7f912e51a700 1 Processor -- start 2024-04-22T15:55:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.342+0000 7f912e51a700 1 -- start start 2024-04-22T15:55:26.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.342+0000 7f912e51a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 0x7f912819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.342+0000 7f912e51a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9128105b60 0x7f912819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.342+0000 7f912e51a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912819d370 con 0x7f91280ffad0 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.342+0000 7f912e51a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128196950 con 0x7f9128105b60 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f9127fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 0x7f912819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f9127fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 0x7f912819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40194/0 (socket says 172.21.15.96:40194) 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f9127fff700 1 -- 172.21.15.96:0/1210672021 learned_addr learned my addr 172.21.15.96:0/1210672021 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f91277fe700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9128105b60 0x7f912819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f9127fff700 1 -- 172.21.15.96:0/1210672021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9128105b60 msgr2=0x7f912819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f9127fff700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9128105b60 0x7f912819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f9127fff700 1 -- 172.21.15.96:0/1210672021 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c0095f0 con 0x7f91280ffad0 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f9127fff700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 0x7f912819c810 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f911800d900 tx=0x7f911800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f91257fa700 1 -- 172.21.15.96:0/1210672021 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91180041f0 con 0x7f91280ffad0 2024-04-22T15:55:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9128196c30 con 0x7f91280ffad0 2024-04-22T15:55:26.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f91257fa700 1 -- 172.21.15.96:0/1210672021 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9118004cf0 con 0x7f91280ffad0 2024-04-22T15:55:26.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f91257fa700 1 -- 172.21.15.96:0/1210672021 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f911800b750 con 0x7f91280ffad0 2024-04-22T15:55:26.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.343+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9128197180 con 0x7f91280ffad0 2024-04-22T15:55:26.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.344+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9128109100 con 0x7f91280ffad0 2024-04-22T15:55:26.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.345+0000 7f91257fa700 1 -- 172.21.15.96:0/1210672021 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9118004350 con 0x7f91280ffad0 2024-04-22T15:55:26.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.345+0000 7f91257fa700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f911006ecc0 0x7f9110071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.345+0000 7f91257fa700 1 -- 172.21.15.96:0/1210672021 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f911808f000 con 0x7f91280ffad0 2024-04-22T15:55:26.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.345+0000 7f91277fe700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f911006ecc0 0x7f9110071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.345+0000 7f91277fe700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f911006ecc0 0x7f9110071180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f911c005b40 tx=0x7f911c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:26.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.350+0000 7f91257fa700 1 -- 172.21.15.96:0/1210672021 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f911805b890 con 0x7f91280ffad0 2024-04-22T15:55:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.570+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9128197e80 con 0x7f911006ecc0 2024-04-22T15:55:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.571+0000 7f91257fa700 1 -- 172.21.15.96:0/1210672021 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9128197e80 con 0x7f911006ecc0 2024-04-22T15:55:26.574 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:55:26.574 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:55:26.574 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:55:26.574 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:55:26.574 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:55:26.574 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:55:26.574 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:55:26.574 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:55:26.575 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:55:26.575 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:55:26.575 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:55:26.575 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:55:26.575 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:55:26.575 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:55:26.575 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:55:26.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.573+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f911006ecc0 msgr2=0x7f9110071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.573+0000 7f912e51a700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f911006ecc0 0x7f9110071180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f911c005b40 tx=0x7f911c01a040 comp rx=0 tx=0).stop 2024-04-22T15:55:26.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.573+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 msgr2=0x7f912819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.573+0000 7f912e51a700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 0x7f912819c810 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f911800d900 tx=0x7f911800dcc0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.573+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 shutdown_connections 2024-04-22T15:55:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.574+0000 7f912e51a700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f911006ecc0 0x7f9110071180 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.574+0000 7f912e51a700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91280ffad0 0x7f912819c810 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.574+0000 7f912e51a700 1 --2- 172.21.15.96:0/1210672021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9128105b60 0x7f912819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.574+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 >> 172.21.15.96:0/1210672021 conn(0x7f91280fb5e0 msgr2=0x7f9128069690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.574+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 shutdown_connections 2024-04-22T15:55:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.574+0000 7f912e51a700 1 -- 172.21.15.96:0/1210672021 wait complete. 2024-04-22T15:55:26.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.677+0000 7f47b8581700 1 -- 172.21.15.96:0/3915858722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b01017a0 msgr2=0x7f47b01057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.677+0000 7f47b8581700 1 --2- 172.21.15.96:0/3915858722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b01017a0 0x7f47b01057f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009910 tx=0x7f47a0009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:26.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.677+0000 7f47b8581700 1 -- 172.21.15.96:0/3915858722 shutdown_connections 2024-04-22T15:55:26.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.677+0000 7f47b8581700 1 --2- 172.21.15.96:0/3915858722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b01017a0 0x7f47b01057f0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.677+0000 7f47b8581700 1 --2- 172.21.15.96:0/3915858722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47b0100df0 0x7f47b01011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.677+0000 7f47b8581700 1 -- 172.21.15.96:0/3915858722 >> 172.21.15.96:0/3915858722 conn(0x7f47b00fc6a0 msgr2=0x7f47b00feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.677+0000 7f47b8581700 1 -- 172.21.15.96:0/3915858722 shutdown_connections 2024-04-22T15:55:26.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b8581700 1 -- 172.21.15.96:0/3915858722 wait complete. 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b8581700 1 Processor -- start 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b8581700 1 -- start start 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b8581700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b0100df0 0x7f47b00731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b8581700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47b01017a0 0x7f47b0073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b8581700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0073c60 con 0x7f47b0100df0 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b8581700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0073da0 con 0x7f47b01017a0 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b631d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b0100df0 0x7f47b00731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b631d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b0100df0 0x7f47b00731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40206/0 (socket says 172.21.15.96:40206) 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.678+0000 7f47b631d700 1 -- 172.21.15.96:0/38233367 learned_addr learned my addr 172.21.15.96:0/38233367 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47b5b1c700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47b01017a0 0x7f47b0073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47b631d700 1 -- 172.21.15.96:0/38233367 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47b01017a0 msgr2=0x7f47b0073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47b631d700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47b01017a0 0x7f47b0073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47b631d700 1 -- 172.21.15.96:0/38233367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a00095f0 con 0x7f47b0100df0 2024-04-22T15:55:26.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47b631d700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b0100df0 0x7f47b00731e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00b4b0 tx=0x7f47ac00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:26.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47a77fe700 1 -- 172.21.15.96:0/38233367 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47ac011800 con 0x7f47b0100df0 2024-04-22T15:55:26.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47a77fe700 1 -- 172.21.15.96:0/38233367 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47ac011e20 con 0x7f47b0100df0 2024-04-22T15:55:26.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b01a2db0 con 0x7f47b0100df0 2024-04-22T15:55:26.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47a77fe700 1 -- 172.21.15.96:0/38233367 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47ac00f2e0 con 0x7f47b0100df0 2024-04-22T15:55:26.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.679+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b01a3110 con 0x7f47b0100df0 2024-04-22T15:55:26.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.680+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b0109120 con 0x7f47b0100df0 2024-04-22T15:55:26.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.681+0000 7f47a77fe700 1 -- 172.21.15.96:0/38233367 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f47ac010380 con 0x7f47b0100df0 2024-04-22T15:55:26.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.681+0000 7f47a77fe700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f479c06ecd0 0x7f479c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.681+0000 7f47a77fe700 1 -- 172.21.15.96:0/38233367 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f47ac08eaa0 con 0x7f47b0100df0 2024-04-22T15:55:26.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.681+0000 7f47b5b1c700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f479c06ecd0 0x7f479c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.681+0000 7f47b5b1c700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f479c06ecd0 0x7f479c071190 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f47b0074910 tx=0x7f47a001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:26.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.686+0000 7f47a77fe700 1 -- 172.21.15.96:0/38233367 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f47ac057800 con 0x7f47b0100df0 2024-04-22T15:55:26.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.966+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f47b01022d0 con 0x7f47b0100df0 2024-04-22T15:55:26.969 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:26 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4235731200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:26.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:26 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3651622930' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:55:26.970 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:26 smithi096 ceph-mon[112443]: from='client.35844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:26.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.967+0000 7f47a77fe700 1 -- 172.21.15.96:0/38233367 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f47ac05aec0 con 0x7f47b0100df0 2024-04-22T15:55:26.970 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:55:26.970 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:55:26.970 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:55:26.971 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:55:26.972 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:55:26.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f479c06ecd0 msgr2=0x7f479c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f479c06ecd0 0x7f479c071190 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f47b0074910 tx=0x7f47a001a040 comp rx=0 tx=0).stop 2024-04-22T15:55:26.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b0100df0 msgr2=0x7f47b00731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b0100df0 0x7f47b00731e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00b4b0 tx=0x7f47ac00b870 comp rx=0 tx=0).stop 2024-04-22T15:55:26.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 shutdown_connections 2024-04-22T15:55:26.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f479c06ecd0 0x7f479c071190 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47b0100df0 0x7f47b00731e0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 --2- 172.21.15.96:0/38233367 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47b01017a0 0x7f47b0073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 >> 172.21.15.96:0/38233367 conn(0x7f47b00fc6a0 msgr2=0x7f47b00fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 shutdown_connections 2024-04-22T15:55:26.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:26.970+0000 7f47b8581700 1 -- 172.21.15.96:0/38233367 wait complete. 2024-04-22T15:55:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:26 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4235731200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:26 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3651622930' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:55:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:26 smithi196 ceph-mon[81815]: from='client.35844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:27 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/38233367' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:55:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:27 smithi096 ceph-mon[112443]: pgmap v1009: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:28.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:27 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/38233367' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:55:28.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:27 smithi196 ceph-mon[81815]: pgmap v1009: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:30.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:29 smithi096 ceph-mon[112443]: pgmap v1010: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:30.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:29 smithi196 ceph-mon[81815]: pgmap v1010: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:32.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:31 smithi096 ceph-mon[112443]: pgmap v1011: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:32.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:31 smithi196 ceph-mon[81815]: pgmap v1011: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:34.038 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:33 smithi096 ceph-mon[112443]: pgmap v1012: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:34.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:33 smithi196 ceph-mon[81815]: pgmap v1012: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:36.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:35 smithi096 ceph-mon[112443]: pgmap v1013: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:36.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:35 smithi196 ceph-mon[81815]: pgmap v1013: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:38.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:37 smithi096 ceph-mon[112443]: pgmap v1014: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:38.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:37 smithi196 ceph-mon[81815]: pgmap v1014: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:39.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:39.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:40.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:39 smithi096 ceph-mon[112443]: pgmap v1015: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:40.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:39 smithi196 ceph-mon[81815]: pgmap v1015: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:42.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:41 smithi196 ceph-mon[81815]: pgmap v1016: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:42.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:41 smithi096 ceph-mon[112443]: pgmap v1016: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:43 smithi196 ceph-mon[81815]: pgmap v1017: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:44.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:43 smithi096 ceph-mon[112443]: pgmap v1017: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:46.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:45 smithi096 ceph-mon[112443]: pgmap v1018: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:45 smithi196 ceph-mon[81815]: pgmap v1018: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:48.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:47 smithi096 ceph-mon[112443]: pgmap v1019: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:48.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:47 smithi196 ceph-mon[81815]: pgmap v1019: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:49.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:49 smithi096 ceph-mon[112443]: pgmap v1020: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:50.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:49 smithi196 ceph-mon[81815]: pgmap v1020: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:52.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:51 smithi096 ceph-mon[112443]: pgmap v1021: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:51 smithi196 ceph-mon[81815]: pgmap v1021: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:53.640 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:53.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:54.642 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:54 smithi096 ceph-mon[112443]: pgmap v1022: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:54.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:54 smithi196 ceph-mon[81815]: pgmap v1022: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:56.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:55 smithi096 ceph-mon[112443]: pgmap v1023: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:55 smithi196 ceph-mon[81815]: pgmap v1023: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.087+0000 7fa0e9488700 1 -- 172.21.15.96:0/2079852272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 msgr2=0x7fa0e41011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.087+0000 7fa0e9488700 1 --2- 172.21.15.96:0/2079852272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 0x7fa0e41011d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009910 tx=0x7fa0d4009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.087+0000 7fa0e9488700 1 -- 172.21.15.96:0/2079852272 shutdown_connections 2024-04-22T15:55:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.088+0000 7fa0e9488700 1 --2- 172.21.15.96:0/2079852272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e41017a0 0x7fa0e41057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.088+0000 7fa0e9488700 1 --2- 172.21.15.96:0/2079852272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 0x7fa0e41011d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.088+0000 7fa0e9488700 1 -- 172.21.15.96:0/2079852272 >> 172.21.15.96:0/2079852272 conn(0x7fa0e40fc6a0 msgr2=0x7fa0e40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.088+0000 7fa0e9488700 1 -- 172.21.15.96:0/2079852272 shutdown_connections 2024-04-22T15:55:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.088+0000 7fa0e9488700 1 -- 172.21.15.96:0/2079852272 wait complete. 2024-04-22T15:55:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.088+0000 7fa0e9488700 1 Processor -- start 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.088+0000 7fa0e9488700 1 -- start start 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e9488700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 0x7fa0e419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e9488700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e41017a0 0x7fa0e419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e9488700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e419d2c0 con 0x7fa0e4100df0 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e9488700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4196930 con 0x7fa0e41017a0 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 0x7fa0e419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e2ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 0x7fa0e419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60110/0 (socket says 172.21.15.96:60110) 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e2ffd700 1 -- 172.21.15.96:0/4256871227 learned_addr learned my addr 172.21.15.96:0/4256871227 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e27fc700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e41017a0 0x7fa0e419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:57.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e2ffd700 1 -- 172.21.15.96:0/4256871227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e41017a0 msgr2=0x7fa0e419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e2ffd700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e41017a0 0x7fa0e419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e2ffd700 1 -- 172.21.15.96:0/4256871227 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d40095f0 con 0x7fa0e4100df0 2024-04-22T15:55:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e2ffd700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 0x7fa0e419c7f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4006010 tx=0x7fa0d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0cbfff700 1 -- 172.21.15.96:0/4256871227 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d401d070 con 0x7fa0e4100df0 2024-04-22T15:55:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0cbfff700 1 -- 172.21.15.96:0/4256871227 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0d400f460 con 0x7fa0e4100df0 2024-04-22T15:55:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e4196bb0 con 0x7fa0e4100df0 2024-04-22T15:55:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0cbfff700 1 -- 172.21.15.96:0/4256871227 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d4017630 con 0x7fa0e4100df0 2024-04-22T15:55:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.089+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e41970a0 con 0x7fa0e4100df0 2024-04-22T15:55:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.091+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e4109190 con 0x7fa0e4100df0 2024-04-22T15:55:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.091+0000 7fa0cbfff700 1 -- 172.21.15.96:0/4256871227 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa0d40178a0 con 0x7fa0e4100df0 2024-04-22T15:55:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.091+0000 7fa0cbfff700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa0cc06ecc0 0x7fa0cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.091+0000 7fa0cbfff700 1 -- 172.21.15.96:0/4256871227 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa0d4090310 con 0x7fa0e4100df0 2024-04-22T15:55:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.091+0000 7fa0e27fc700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa0cc06ecc0 0x7fa0cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.092+0000 7fa0e27fc700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa0cc06ecc0 0x7fa0cc071180 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005ea0 tx=0x7fa0d8005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:57.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.096+0000 7fa0cbfff700 1 -- 172.21.15.96:0/4256871227 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa0d405cce0 con 0x7fa0e4100df0 2024-04-22T15:55:57.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.314+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0e4197df0 con 0x7fa0cc06ecc0 2024-04-22T15:55:57.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.315+0000 7fa0cbfff700 1 -- 172.21.15.96:0/4256871227 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa0e4197df0 con 0x7fa0cc06ecc0 2024-04-22T15:55:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.317+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa0cc06ecc0 msgr2=0x7fa0cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.317+0000 7fa0e9488700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa0cc06ecc0 0x7fa0cc071180 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005ea0 tx=0x7fa0d8005e30 comp rx=0 tx=0).stop 2024-04-22T15:55:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.317+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 msgr2=0x7fa0e419c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.317+0000 7fa0e9488700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 0x7fa0e419c7f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4006010 tx=0x7fa0d400be20 comp rx=0 tx=0).stop 2024-04-22T15:55:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.317+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 shutdown_connections 2024-04-22T15:55:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.317+0000 7fa0e9488700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa0cc06ecc0 0x7fa0cc071180 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.317+0000 7fa0e9488700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0e4100df0 0x7fa0e419c7f0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.317+0000 7fa0e9488700 1 --2- 172.21.15.96:0/4256871227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e41017a0 0x7fa0e419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.317+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 >> 172.21.15.96:0/4256871227 conn(0x7fa0e40fc6a0 msgr2=0x7fa0e40fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.318+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 shutdown_connections 2024-04-22T15:55:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.318+0000 7fa0e9488700 1 -- 172.21.15.96:0/4256871227 wait complete. 2024-04-22T15:55:57.331 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:55:57.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 -- 172.21.15.96:0/3992858800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 msgr2=0x7f445c104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 --2- 172.21.15.96:0/3992858800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 0x7f445c104960 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f444c009910 tx=0x7f444c009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:57.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 -- 172.21.15.96:0/3992858800 shutdown_connections 2024-04-22T15:55:57.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 --2- 172.21.15.96:0/3992858800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 0x7f445c104960 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 --2- 172.21.15.96:0/3992858800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f445c0ffb70 0x7f445c0fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 -- 172.21.15.96:0/3992858800 >> 172.21.15.96:0/3992858800 conn(0x7f445c0fb5d0 msgr2=0x7f445c0fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:57.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 -- 172.21.15.96:0/3992858800 shutdown_connections 2024-04-22T15:55:57.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 -- 172.21.15.96:0/3992858800 wait complete. 2024-04-22T15:55:57.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 Processor -- start 2024-04-22T15:55:57.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.422+0000 7f44633e6700 1 -- start start 2024-04-22T15:55:57.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f44633e6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f445c0ffb70 0x7f445c19d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:57.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f44633e6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 0x7f445c19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:57.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f44633e6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c19dd70 con 0x7f445c100490 2024-04-22T15:55:57.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f44633e6700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c1a1a40 con 0x7f445c0ffb70 2024-04-22T15:55:57.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f4460981700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 0x7f445c19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:57.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f4460981700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 0x7f445c19d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60122/0 (socket says 172.21.15.96:60122) 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f4460981700 1 -- 172.21.15.96:0/2127579057 learned_addr learned my addr 172.21.15.96:0/2127579057 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f4461182700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f445c0ffb70 0x7f445c19d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f4460981700 1 -- 172.21.15.96:0/2127579057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f445c0ffb70 msgr2=0x7f445c19d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f4460981700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f445c0ffb70 0x7f445c19d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f4460981700 1 -- 172.21.15.96:0/2127579057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c0095f0 con 0x7f445c100490 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f4460981700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 0x7f445c19d690 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f444c00ba70 tx=0x7f444c00baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f44527fc700 1 -- 172.21.15.96:0/2127579057 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f444c01d070 con 0x7f445c100490 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f44527fc700 1 -- 172.21.15.96:0/2127579057 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f444c003d10 con 0x7f445c100490 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f44527fc700 1 -- 172.21.15.96:0/2127579057 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f444c00fe10 con 0x7f445c100490 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.423+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445c1a1cc0 con 0x7f445c100490 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.424+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445c1a21b0 con 0x7f445c100490 2024-04-22T15:55:57.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.424+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445c069a30 con 0x7f445c100490 2024-04-22T15:55:57.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.425+0000 7f44527fc700 1 -- 172.21.15.96:0/2127579057 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f444c00f680 con 0x7f445c100490 2024-04-22T15:55:57.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.425+0000 7f44527fc700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f444806ecc0 0x7f4448071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:57.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.425+0000 7f44527fc700 1 -- 172.21.15.96:0/2127579057 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f444c090060 con 0x7f445c100490 2024-04-22T15:55:57.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.425+0000 7f4461182700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f444806ecc0 0x7f4448071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:57.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.426+0000 7f4461182700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f444806ecc0 0x7f4448071180 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4458005950 tx=0x7f445800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:57.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.430+0000 7f44527fc700 1 -- 172.21.15.96:0/2127579057 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f444c058f00 con 0x7f445c100490 2024-04-22T15:55:57.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.649+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f445c114450 con 0x7f444806ecc0 2024-04-22T15:55:57.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.650+0000 7f44527fc700 1 -- 172.21.15.96:0/2127579057 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f445c114450 con 0x7f444806ecc0 2024-04-22T15:55:57.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f444806ecc0 msgr2=0x7f4448071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f444806ecc0 0x7f4448071180 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4458005950 tx=0x7f445800b410 comp rx=0 tx=0).stop 2024-04-22T15:55:57.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 msgr2=0x7f445c19d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 0x7f445c19d690 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f444c00ba70 tx=0x7f444c00baa0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 shutdown_connections 2024-04-22T15:55:57.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f444806ecc0 0x7f4448071180 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f445c0ffb70 0x7f445c19d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 --2- 172.21.15.96:0/2127579057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f445c100490 0x7f445c19d690 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 >> 172.21.15.96:0/2127579057 conn(0x7f445c0fb5d0 msgr2=0x7f445c10e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 shutdown_connections 2024-04-22T15:55:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.652+0000 7f44633e6700 1 -- 172.21.15.96:0/2127579057 wait complete. 2024-04-22T15:55:57.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.760+0000 7fbc02882700 1 -- 172.21.15.96:0/2375855984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 msgr2=0x7fbbfc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.760+0000 7fbc02882700 1 --2- 172.21.15.96:0/2375855984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 0x7fbbfc101200 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009910 tx=0x7fbbec009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:57.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.760+0000 7fbc02882700 1 -- 172.21.15.96:0/2375855984 shutdown_connections 2024-04-22T15:55:57.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.760+0000 7fbc02882700 1 --2- 172.21.15.96:0/2375855984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbfc1017d0 0x7fbbfc105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.760+0000 7fbc02882700 1 --2- 172.21.15.96:0/2375855984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 0x7fbbfc101200 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.760+0000 7fbc02882700 1 -- 172.21.15.96:0/2375855984 >> 172.21.15.96:0/2375855984 conn(0x7fbbfc0fc6d0 msgr2=0x7fbbfc0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:57.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.761+0000 7fbc02882700 1 -- 172.21.15.96:0/2375855984 shutdown_connections 2024-04-22T15:55:57.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.761+0000 7fbc02882700 1 -- 172.21.15.96:0/2375855984 wait complete. 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.761+0000 7fbc02882700 1 Processor -- start 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.761+0000 7fbc02882700 1 -- start start 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.761+0000 7fbc02882700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 0x7fbbfc0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbc02882700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbfc1017d0 0x7fbbfc073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbc02882700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc073c60 con 0x7fbbfc100e20 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbc02882700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc073da0 con 0x7fbbfc1017d0 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbfbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 0x7fbbfc0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbfbfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 0x7fbbfc0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60138/0 (socket says 172.21.15.96:60138) 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbfbfff700 1 -- 172.21.15.96:0/3748831669 learned_addr learned my addr 172.21.15.96:0/3748831669 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbfbfff700 1 -- 172.21.15.96:0/3748831669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbfc1017d0 msgr2=0x7fbbfc073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbfb7fe700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbfc1017d0 0x7fbbfc073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:57.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbfbfff700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbfc1017d0 0x7fbbfc073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbfbfff700 1 -- 172.21.15.96:0/3748831669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec0095f0 con 0x7fbbfc100e20 2024-04-22T15:55:57.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbfbfff700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 0x7fbbfc0731e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00bd70 tx=0x7fbbec00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:57.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbf97fa700 1 -- 172.21.15.96:0/3748831669 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbec01d070 con 0x7fbbfc100e20 2024-04-22T15:55:57.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbf97fa700 1 -- 172.21.15.96:0/3748831669 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbec0056e0 con 0x7fbbfc100e20 2024-04-22T15:55:57.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbfc074020 con 0x7fbbfc100e20 2024-04-22T15:55:57.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbbf97fa700 1 -- 172.21.15.96:0/3748831669 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbec017400 con 0x7fbbfc100e20 2024-04-22T15:55:57.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.762+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbfc1a2f80 con 0x7fbbfc100e20 2024-04-22T15:55:57.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.763+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbfc1091c0 con 0x7fbbfc100e20 2024-04-22T15:55:57.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.764+0000 7fbbf97fa700 1 -- 172.21.15.96:0/3748831669 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbbec003a60 con 0x7fbbfc100e20 2024-04-22T15:55:57.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.764+0000 7fbbf97fa700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbe406ecc0 0x7fbbe4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:57.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.764+0000 7fbbf97fa700 1 -- 172.21.15.96:0/3748831669 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbbec0902e0 con 0x7fbbfc100e20 2024-04-22T15:55:57.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.764+0000 7fbbfb7fe700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbe406ecc0 0x7fbbe4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:57.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.765+0000 7fbbfb7fe700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbe406ecc0 0x7fbbe4071180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0005950 tx=0x7fbbf00058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:57.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.769+0000 7fbbf97fa700 1 -- 172.21.15.96:0/3748831669 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbbec059040 con 0x7fbbfc100e20 2024-04-22T15:55:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.979+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbfc10ddf0 con 0x7fbbe406ecc0 2024-04-22T15:55:57.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.990+0000 7fbbf97fa700 1 -- 172.21.15.96:0/3748831669 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbbfc10ddf0 con 0x7fbbe406ecc0 2024-04-22T15:55:57.992 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (34m) 12m ago 45m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (46m) 12m ago 46m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (45m) 12m ago 45m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (31m) 12m ago 46m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (31m) 12m ago 45m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (34m) 12m ago 45m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (27m) 12m ago 40m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (28m) 12m ago 40m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (27m) 12m ago 40m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (12m) 12m ago 40m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (37m) 12m ago 47m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:55:57.993 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (35m) 12m ago 44m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (32m) 12m ago 47m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (32m) 12m ago 44m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (35m) 12m ago 46m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (35m) 12m ago 44m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (31m) 12m ago 44m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (30m) 12m ago 43m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (30m) 12m ago 43m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (29m) 12m ago 42m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (29m) 12m ago 42m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (28m) 12m ago 41m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:55:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (34m) 12m ago 45m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:55:57.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbe406ecc0 msgr2=0x7fbbe4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbe406ecc0 0x7fbbe4071180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0005950 tx=0x7fbbf00058e0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 msgr2=0x7fbbfc0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 0x7fbbfc0731e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00bd70 tx=0x7fbbec00be50 comp rx=0 tx=0).stop 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 shutdown_connections 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbbe406ecc0 0x7fbbe4071180 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbfc100e20 0x7fbbfc0731e0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 --2- 172.21.15.96:0/3748831669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbfc1017d0 0x7fbbfc073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 >> 172.21.15.96:0/3748831669 conn(0x7fbbfc0fc6d0 msgr2=0x7fbbfc0fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 shutdown_connections 2024-04-22T15:55:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:57.992+0000 7fbc02882700 1 -- 172.21.15.96:0/3748831669 wait complete. 2024-04-22T15:55:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.094+0000 7f0edb8b0700 1 -- 172.21.15.96:0/2995262030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed4105ab0 msgr2=0x7f0ed4105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.094+0000 7f0edb8b0700 1 --2- 172.21.15.96:0/2995262030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed4105ab0 0x7f0ed4105e90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00b0a0 tx=0x7f0ecc00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.094+0000 7f0edb8b0700 1 -- 172.21.15.96:0/2995262030 shutdown_connections 2024-04-22T15:55:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.094+0000 7f0edb8b0700 1 --2- 172.21.15.96:0/2995262030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed40ffab0 0x7f0ed40fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.094+0000 7f0edb8b0700 1 --2- 172.21.15.96:0/2995262030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed4105ab0 0x7f0ed4105e90 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.094+0000 7f0edb8b0700 1 -- 172.21.15.96:0/2995262030 >> 172.21.15.96:0/2995262030 conn(0x7f0ed40fb5e0 msgr2=0x7f0ed40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.098 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:57 smithi096 ceph-mon[112443]: from='client.35852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:58.098 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:57 smithi096 ceph-mon[112443]: from='client.35856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:58.098 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:57 smithi096 ceph-mon[112443]: pgmap v1024: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.094+0000 7f0edb8b0700 1 -- 172.21.15.96:0/2995262030 shutdown_connections 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.094+0000 7f0edb8b0700 1 -- 172.21.15.96:0/2995262030 wait complete. 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0edb8b0700 1 Processor -- start 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0edb8b0700 1 -- start start 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0edb8b0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed40ffab0 0x7f0ed40fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0edb8b0700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed4105ab0 0x7f0ed40fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0edb8b0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40fc920 con 0x7f0ed40ffab0 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0edb8b0700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40fca60 con 0x7f0ed4105ab0 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0ed964c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed40ffab0 0x7f0ed40fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0ed964c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed40ffab0 0x7f0ed40fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60162/0 (socket says 172.21.15.96:60162) 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0ed964c700 1 -- 172.21.15.96:0/315710435 learned_addr learned my addr 172.21.15.96:0/315710435 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0ed8e4b700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed4105ab0 0x7f0ed40fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0ed964c700 1 -- 172.21.15.96:0/315710435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed4105ab0 msgr2=0x7f0ed40fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0ed964c700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed4105ab0 0x7f0ed40fc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0ed964c700 1 -- 172.21.15.96:0/315710435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc009d00 con 0x7f0ed40ffab0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.095+0000 7f0ed964c700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed40ffab0 0x7f0ed40fbea0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc000f80 tx=0x7f0ecc007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.096+0000 7f0ed27fc700 1 -- 172.21.15.96:0/315710435 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ecc00e040 con 0x7f0ed40ffab0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.096+0000 7f0ed27fc700 1 -- 172.21.15.96:0/315710435 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ecc003de0 con 0x7f0ed40ffab0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.096+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed40fcce0 con 0x7f0ed40ffab0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.096+0000 7f0ed27fc700 1 -- 172.21.15.96:0/315710435 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ecc01d720 con 0x7f0ed40ffab0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.096+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed41a3050 con 0x7f0ed40ffab0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.097+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed41090c0 con 0x7f0ed40ffab0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.097+0000 7f0ed27fc700 1 -- 172.21.15.96:0/315710435 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0ecc004690 con 0x7f0ed40ffab0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.098+0000 7f0ed27fc700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0ebc06ecc0 0x7f0ebc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.098+0000 7f0ed27fc700 1 -- 172.21.15.96:0/315710435 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0ecc0906b0 con 0x7f0ed40ffab0 2024-04-22T15:55:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.098+0000 7f0ed8e4b700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0ebc06ecc0 0x7f0ebc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.098+0000 7f0ed8e4b700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0ebc06ecc0 0x7f0ebc071180 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006fd0 tx=0x7f0ec8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.104+0000 7f0ed27fc700 1 -- 172.21.15.96:0/315710435 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0ecc0594c0 con 0x7f0ed40ffab0 2024-04-22T15:55:58.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:57 smithi196 ceph-mon[81815]: from='client.35852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:58.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:57 smithi196 ceph-mon[81815]: from='client.35856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:58.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:57 smithi196 ceph-mon[81815]: pgmap v1024: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:55:58.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.387+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0ed40fd380 con 0x7f0ed40ffab0 2024-04-22T15:55:58.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.387+0000 7f0ed27fc700 1 -- 172.21.15.96:0/315710435 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f0ecc05cb80 con 0x7f0ed40ffab0 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:55:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:55:58.392 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:55:58.392 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:55:58.392 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:55:58.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0ebc06ecc0 msgr2=0x7f0ebc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0ebc06ecc0 0x7f0ebc071180 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006fd0 tx=0x7f0ec8008040 comp rx=0 tx=0).stop 2024-04-22T15:55:58.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed40ffab0 msgr2=0x7f0ed40fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed40ffab0 0x7f0ed40fbea0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc000f80 tx=0x7f0ecc007bd0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 shutdown_connections 2024-04-22T15:55:58.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0ebc06ecc0 0x7f0ebc071180 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ed40ffab0 0x7f0ed40fbea0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 --2- 172.21.15.96:0/315710435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed4105ab0 0x7f0ed40fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 >> 172.21.15.96:0/315710435 conn(0x7f0ed40fb5e0 msgr2=0x7f0ed4104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 shutdown_connections 2024-04-22T15:55:58.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.390+0000 7f0edb8b0700 1 -- 172.21.15.96:0/315710435 wait complete. 2024-04-22T15:55:58.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.493+0000 7f1a4308f700 1 -- 172.21.15.96:0/591691971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 msgr2=0x7f1a3c105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.493+0000 7f1a4308f700 1 --2- 172.21.15.96:0/591691971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 0x7f1a3c105e80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009910 tx=0x7f1a30009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:58.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.493+0000 7f1a4308f700 1 -- 172.21.15.96:0/591691971 shutdown_connections 2024-04-22T15:55:58.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.493+0000 7f1a4308f700 1 --2- 172.21.15.96:0/591691971 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c0ffaa0 0x7f1a3c0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.493+0000 7f1a4308f700 1 --2- 172.21.15.96:0/591691971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 0x7f1a3c105e80 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.493+0000 7f1a4308f700 1 -- 172.21.15.96:0/591691971 >> 172.21.15.96:0/591691971 conn(0x7f1a3c076740 msgr2=0x7f1a3c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.493+0000 7f1a4308f700 1 -- 172.21.15.96:0/591691971 shutdown_connections 2024-04-22T15:55:58.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.493+0000 7f1a4308f700 1 -- 172.21.15.96:0/591691971 wait complete. 2024-04-22T15:55:58.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a4308f700 1 Processor -- start 2024-04-22T15:55:58.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a4308f700 1 -- start start 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a4308f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c0ffaa0 0x7f1a3c197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a4308f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 0x7f1a3c197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a4308f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c198470 con 0x7f1a3c105aa0 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a4308f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c1985b0 con 0x7f1a3c0ffaa0 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a3bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 0x7f1a3c197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a3bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 0x7f1a3c197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60186/0 (socket says 172.21.15.96:60186) 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a3bfff700 1 -- 172.21.15.96:0/1261048274 learned_addr learned my addr 172.21.15.96:0/1261048274 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a40e2b700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c0ffaa0 0x7f1a3c197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a3bfff700 1 -- 172.21.15.96:0/1261048274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c0ffaa0 msgr2=0x7f1a3c197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.494+0000 7f1a3bfff700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c0ffaa0 0x7f1a3c197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.495+0000 7f1a3bfff700 1 -- 172.21.15.96:0/1261048274 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a300095f0 con 0x7f1a3c105aa0 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.495+0000 7f1a3bfff700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 0x7f1a3c197ea0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800d8d0 tx=0x7f1a2800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.495+0000 7f1a39ffb700 1 -- 172.21.15.96:0/1261048274 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a28009940 con 0x7f1a3c105aa0 2024-04-22T15:55:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.495+0000 7f1a39ffb700 1 -- 172.21.15.96:0/1261048274 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a28010460 con 0x7f1a3c105aa0 2024-04-22T15:55:58.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.495+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3c1a09d0 con 0x7f1a3c105aa0 2024-04-22T15:55:58.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.495+0000 7f1a39ffb700 1 -- 172.21.15.96:0/1261048274 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a2800f5f0 con 0x7f1a3c105aa0 2024-04-22T15:55:58.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.495+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3c1a0e40 con 0x7f1a3c105aa0 2024-04-22T15:55:58.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.496+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3c1080a0 con 0x7f1a3c105aa0 2024-04-22T15:55:58.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.497+0000 7f1a39ffb700 1 -- 172.21.15.96:0/1261048274 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1a280105d0 con 0x7f1a3c105aa0 2024-04-22T15:55:58.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.497+0000 7f1a39ffb700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1a2c06ecc0 0x7f1a2c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.497+0000 7f1a39ffb700 1 -- 172.21.15.96:0/1261048274 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1a2808edc0 con 0x7f1a3c105aa0 2024-04-22T15:55:58.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.497+0000 7f1a40e2b700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1a2c06ecc0 0x7f1a2c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.497+0000 7f1a40e2b700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1a2c06ecc0 0x7f1a2c071180 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000b5c0 tx=0x7f1a3001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.502+0000 7f1a39ffb700 1 -- 172.21.15.96:0/1261048274 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1a28057ba0 con 0x7f1a3c105aa0 2024-04-22T15:55:58.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.745+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1a3c1011e0 con 0x7f1a3c105aa0 2024-04-22T15:55:58.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.745+0000 7f1a39ffb700 1 -- 172.21.15.96:0/1261048274 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f1a28016770 con 0x7f1a3c105aa0 2024-04-22T15:55:58.748 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:55:58.749 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:55:58.750 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:55:58.751 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:55:58.752 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:55:58.752 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:55:58.752 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:55:58.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.747+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1a2c06ecc0 msgr2=0x7f1a2c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.747+0000 7f1a4308f700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1a2c06ecc0 0x7f1a2c071180 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000b5c0 tx=0x7f1a3001a040 comp rx=0 tx=0).stop 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.747+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 msgr2=0x7f1a3c197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.747+0000 7f1a4308f700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 0x7f1a3c197ea0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800d8d0 tx=0x7f1a2800dc90 comp rx=0 tx=0).stop 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.748+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 shutdown_connections 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.748+0000 7f1a4308f700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1a2c06ecc0 0x7f1a2c071180 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.748+0000 7f1a4308f700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c0ffaa0 0x7f1a3c197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.748+0000 7f1a4308f700 1 --2- 172.21.15.96:0/1261048274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c105aa0 0x7f1a3c197ea0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.748+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 >> 172.21.15.96:0/1261048274 conn(0x7f1a3c076740 msgr2=0x7f1a3c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.748+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 shutdown_connections 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.748+0000 7f1a4308f700 1 -- 172.21.15.96:0/1261048274 wait complete. 2024-04-22T15:55:58.756 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:55:58.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.853+0000 7fc337b0c700 1 -- 172.21.15.96:0/1677500933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330101df0 msgr2=0x7fc3301021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.853+0000 7fc337b0c700 1 --2- 172.21.15.96:0/1677500933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330101df0 0x7fc3301021d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc320009940 tx=0x7fc320009c50 comp rx=0 tx=0).stop 2024-04-22T15:55:58.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.853+0000 7fc337b0c700 1 -- 172.21.15.96:0/1677500933 shutdown_connections 2024-04-22T15:55:58.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.853+0000 7fc337b0c700 1 --2- 172.21.15.96:0/1677500933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc330102710 0x7fc33010ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.853+0000 7fc337b0c700 1 --2- 172.21.15.96:0/1677500933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330101df0 0x7fc3301021d0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.853+0000 7fc337b0c700 1 -- 172.21.15.96:0/1677500933 >> 172.21.15.96:0/1677500933 conn(0x7fc3300fb600 msgr2=0x7fc3300fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.853+0000 7fc337b0c700 1 -- 172.21.15.96:0/1677500933 shutdown_connections 2024-04-22T15:55:58.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.853+0000 7fc337b0c700 1 -- 172.21.15.96:0/1677500933 wait complete. 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc337b0c700 1 Processor -- start 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc337b0c700 1 -- start start 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc337b0c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc330101df0 0x7fc3300ffa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc337b0c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330102710 0x7fc3300fff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc337b0c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3301004d0 con 0x7fc330102710 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc337b0c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc330100610 con 0x7fc330101df0 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc3350a7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330102710 0x7fc3300fff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc3350a7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330102710 0x7fc3300fff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60204/0 (socket says 172.21.15.96:60204) 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc3350a7700 1 -- 172.21.15.96:0/3115404360 learned_addr learned my addr 172.21.15.96:0/3115404360 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.854+0000 7fc3358a8700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc330101df0 0x7fc3300ffa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.855+0000 7fc3350a7700 1 -- 172.21.15.96:0/3115404360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc330101df0 msgr2=0x7fc3300ffa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.855+0000 7fc3350a7700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc330101df0 0x7fc3300ffa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.855+0000 7fc3350a7700 1 -- 172.21.15.96:0/3115404360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3200095f0 con 0x7fc330102710 2024-04-22T15:55:58.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.855+0000 7fc3350a7700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330102710 0x7fc3300fff90 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00b480 tx=0x7fc32c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.855+0000 7fc326ffd700 1 -- 172.21.15.96:0/3115404360 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc32c011800 con 0x7fc330102710 2024-04-22T15:55:58.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.855+0000 7fc326ffd700 1 -- 172.21.15.96:0/3115404360 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc32c011e20 con 0x7fc330102710 2024-04-22T15:55:58.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.855+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3301a2d90 con 0x7fc330102710 2024-04-22T15:55:58.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.855+0000 7fc326ffd700 1 -- 172.21.15.96:0/3115404360 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc32c010ea0 con 0x7fc330102710 2024-04-22T15:55:58.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.855+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3301a30f0 con 0x7fc330102710 2024-04-22T15:55:58.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.856+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3300699d0 con 0x7fc330102710 2024-04-22T15:55:58.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.856+0000 7fc326ffd700 1 -- 172.21.15.96:0/3115404360 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc32c010380 con 0x7fc330102710 2024-04-22T15:55:58.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.857+0000 7fc326ffd700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc31c06ecc0 0x7fc31c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.857+0000 7fc3358a8700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc31c06ecc0 0x7fc31c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.857+0000 7fc326ffd700 1 -- 172.21.15.96:0/3115404360 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc32c08e8e0 con 0x7fc330102710 2024-04-22T15:55:58.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.857+0000 7fc3358a8700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc31c06ecc0 0x7fc31c071180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc320006010 tx=0x7fc32000b790 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:58.862+0000 7fc326ffd700 1 -- 172.21.15.96:0/3115404360 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc32c0576c0 con 0x7fc330102710 2024-04-22T15:55:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:58 smithi196 ceph-mon[81815]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:59.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:58 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/315710435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:59.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:58 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1261048274' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:55:59.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.079+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3301055b0 con 0x7fc31c06ecc0 2024-04-22T15:55:59.082 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:58 smithi096 ceph-mon[112443]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:59.083 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:58 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/315710435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:59.083 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:58 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1261048274' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:55:59.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.080+0000 7fc326ffd700 1 -- 172.21.15.96:0/3115404360 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc32001e040 con 0x7fc31c06ecc0 2024-04-22T15:55:59.083 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:55:59.083 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:55:59.083 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:55:59.083 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:55:59.084 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.082+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc31c06ecc0 msgr2=0x7fc31c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.082+0000 7fc337b0c700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc31c06ecc0 0x7fc31c071180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc320006010 tx=0x7fc32000b790 comp rx=0 tx=0).stop 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.082+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330102710 msgr2=0x7fc3300fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.082+0000 7fc337b0c700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330102710 0x7fc3300fff90 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00b480 tx=0x7fc32c00b790 comp rx=0 tx=0).stop 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.083+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 shutdown_connections 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.083+0000 7fc337b0c700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc31c06ecc0 0x7fc31c071180 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.083+0000 7fc337b0c700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc330101df0 0x7fc3300ffa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.083+0000 7fc337b0c700 1 --2- 172.21.15.96:0/3115404360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc330102710 0x7fc3300fff90 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.083+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 >> 172.21.15.96:0/3115404360 conn(0x7fc3300fb600 msgr2=0x7fc330103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.083+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 shutdown_connections 2024-04-22T15:55:59.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.083+0000 7fc337b0c700 1 -- 172.21.15.96:0/3115404360 wait complete. 2024-04-22T15:55:59.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.185+0000 7f4accc11700 1 -- 172.21.15.96:0/233285181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 msgr2=0x7f4ac8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.185+0000 7f4accc11700 1 --2- 172.21.15.96:0/233285181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 0x7f4ac8101200 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009910 tx=0x7f4abc009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:59.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.185+0000 7f4accc11700 1 -- 172.21.15.96:0/233285181 shutdown_connections 2024-04-22T15:55:59.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.185+0000 7f4accc11700 1 --2- 172.21.15.96:0/233285181 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ac81017d0 0x7f4ac8105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.185+0000 7f4accc11700 1 --2- 172.21.15.96:0/233285181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 0x7f4ac8101200 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.185+0000 7f4accc11700 1 -- 172.21.15.96:0/233285181 >> 172.21.15.96:0/233285181 conn(0x7f4ac80fc6d0 msgr2=0x7f4ac80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:59.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.185+0000 7f4accc11700 1 -- 172.21.15.96:0/233285181 shutdown_connections 2024-04-22T15:55:59.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.185+0000 7f4accc11700 1 -- 172.21.15.96:0/233285181 wait complete. 2024-04-22T15:55:59.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.186+0000 7f4accc11700 1 Processor -- start 2024-04-22T15:55:59.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.186+0000 7f4accc11700 1 -- start start 2024-04-22T15:55:59.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.186+0000 7f4accc11700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 0x7f4ac80731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:59.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.186+0000 7f4accc11700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ac81017d0 0x7f4ac8073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:59.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.186+0000 7f4accc11700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac8073c60 con 0x7f4ac8100e20 2024-04-22T15:55:59.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.186+0000 7f4accc11700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac8073da0 con 0x7f4ac81017d0 2024-04-22T15:55:59.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.186+0000 7f4ac659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 0x7f4ac80731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:59.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.186+0000 7f4ac659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 0x7f4ac80731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60216/0 (socket says 172.21.15.96:60216) 2024-04-22T15:55:59.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.186+0000 7f4ac659c700 1 -- 172.21.15.96:0/1759217993 learned_addr learned my addr 172.21.15.96:0/1759217993 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:55:59.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4ac5d9b700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ac81017d0 0x7f4ac8073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:59.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4ac659c700 1 -- 172.21.15.96:0/1759217993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ac81017d0 msgr2=0x7f4ac8073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4ac659c700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ac81017d0 0x7f4ac8073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4ac659c700 1 -- 172.21.15.96:0/1759217993 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc0095f0 con 0x7f4ac8100e20 2024-04-22T15:55:59.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4ac659c700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 0x7f4ac80731e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00bd70 tx=0x7f4abc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:59.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4ab77fe700 1 -- 172.21.15.96:0/1759217993 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4abc01d070 con 0x7f4ac8100e20 2024-04-22T15:55:59.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4ab77fe700 1 -- 172.21.15.96:0/1759217993 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4abc00f460 con 0x7f4ac8100e20 2024-04-22T15:55:59.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac8074020 con 0x7f4ac8100e20 2024-04-22T15:55:59.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4ab77fe700 1 -- 172.21.15.96:0/1759217993 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4abc017630 con 0x7f4ac8100e20 2024-04-22T15:55:59.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.187+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac81a3080 con 0x7f4ac8100e20 2024-04-22T15:55:59.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.188+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac81091c0 con 0x7f4ac8100e20 2024-04-22T15:55:59.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.188+0000 7f4ab77fe700 1 -- 172.21.15.96:0/1759217993 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4abc00f5d0 con 0x7f4ac8100e20 2024-04-22T15:55:59.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.189+0000 7f4ab77fe700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4ab006ecc0 0x7f4ab0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:59.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.189+0000 7f4ab77fe700 1 -- 172.21.15.96:0/1759217993 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4abc0903c0 con 0x7f4ac8100e20 2024-04-22T15:55:59.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.189+0000 7f4ac5d9b700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4ab006ecc0 0x7f4ab0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:59.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.189+0000 7f4ac5d9b700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4ab006ecc0 0x7f4ab0071180 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8006fd0 tx=0x7f4ab8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:59.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.194+0000 7f4ab77fe700 1 -- 172.21.15.96:0/1759217993 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4abc059120 con 0x7f4ac8100e20 2024-04-22T15:55:59.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.473+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4ac8102300 con 0x7f4ac8100e20 2024-04-22T15:55:59.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.474+0000 7f4ab77fe700 1 -- 172.21.15.96:0/1759217993 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f4abc05c7e0 con 0x7f4ac8100e20 2024-04-22T15:55:59.477 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:55:59.477 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:55:59.477 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:55:59.477 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:55:59.477 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:55:59.477 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:55:59.478 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:55:59.479 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:55:59.479 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:55:59.479 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:55:59.479 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:55:59.479 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:55:59.479 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:55:59.479 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:55:59.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4ab006ecc0 msgr2=0x7f4ab0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4ab006ecc0 0x7f4ab0071180 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8006fd0 tx=0x7f4ab8008040 comp rx=0 tx=0).stop 2024-04-22T15:55:59.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 msgr2=0x7f4ac80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 0x7f4ac80731e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00bd70 tx=0x7f4abc00be50 comp rx=0 tx=0).stop 2024-04-22T15:55:59.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 shutdown_connections 2024-04-22T15:55:59.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4ab006ecc0 0x7f4ab0071180 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ac8100e20 0x7f4ac80731e0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 --2- 172.21.15.96:0/1759217993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ac81017d0 0x7f4ac8073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 >> 172.21.15.96:0/1759217993 conn(0x7f4ac80fc6d0 msgr2=0x7f4ac80fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:59.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 shutdown_connections 2024-04-22T15:55:59.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:55:59.477+0000 7f4accc11700 1 -- 172.21.15.96:0/1759217993 wait complete. 2024-04-22T15:56:00.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:59 smithi096 ceph-mon[112443]: from='client.35872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:00.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:59 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1759217993' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:56:00.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:55:59 smithi096 ceph-mon[112443]: pgmap v1025: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:00.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:59 smithi196 ceph-mon[81815]: from='client.35872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:00.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:59 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1759217993' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:56:00.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:55:59 smithi196 ceph-mon[81815]: pgmap v1025: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:02.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:01 smithi096 ceph-mon[112443]: pgmap v1026: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:02.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:01 smithi196 ceph-mon[81815]: pgmap v1026: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:04.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:03 smithi196 ceph-mon[81815]: pgmap v1027: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:04.038 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:03 smithi096 ceph-mon[112443]: pgmap v1027: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:06.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:05 smithi096 ceph-mon[112443]: pgmap v1028: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:06.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:05 smithi196 ceph-mon[81815]: pgmap v1028: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:08.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:07 smithi096 ceph-mon[112443]: pgmap v1029: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:07 smithi196 ceph-mon[81815]: pgmap v1029: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:09.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:10.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:09 smithi096 ceph-mon[112443]: pgmap v1030: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:10.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:09 smithi196 ceph-mon[81815]: pgmap v1030: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:12.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:11 smithi196 ceph-mon[81815]: pgmap v1031: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:12.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:11 smithi096 ceph-mon[112443]: pgmap v1031: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:14.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:13 smithi096 ceph-mon[112443]: pgmap v1032: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:14.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:13 smithi196 ceph-mon[81815]: pgmap v1032: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:16.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:15 smithi096 ceph-mon[112443]: pgmap v1033: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:16.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:15 smithi196 ceph-mon[81815]: pgmap v1033: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:18.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:17 smithi096 ceph-mon[112443]: pgmap v1034: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:18.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:17 smithi196 ceph-mon[81815]: pgmap v1034: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:20.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:19 smithi096 ceph-mon[112443]: pgmap v1035: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:20.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:19 smithi196 ceph-mon[81815]: pgmap v1035: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:22.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:21 smithi096 ceph-mon[112443]: pgmap v1036: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:21 smithi196 ceph-mon[81815]: pgmap v1036: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:23.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:24.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:24 smithi096 ceph-mon[112443]: pgmap v1037: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:24 smithi196 ceph-mon[81815]: pgmap v1037: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:26.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:26 smithi096 ceph-mon[112443]: pgmap v1038: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:26.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:26 smithi196 ceph-mon[81815]: pgmap v1038: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:28.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:27 smithi096 ceph-mon[112443]: pgmap v1039: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:28.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:27 smithi196 ceph-mon[81815]: pgmap v1039: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:29.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.593+0000 7f7534ceb700 1 -- 172.21.15.96:0/1797547729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7530100e40 msgr2=0x7f7530101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:29.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.593+0000 7f7534ceb700 1 --2- 172.21.15.96:0/1797547729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7530100e40 0x7f7530101220 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f7520009910 tx=0x7f7520009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:29.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.593+0000 7f7534ceb700 1 -- 172.21.15.96:0/1797547729 shutdown_connections 2024-04-22T15:56:29.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.593+0000 7f7534ceb700 1 --2- 172.21.15.96:0/1797547729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75301017f0 0x7f7530105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:29.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.593+0000 7f7534ceb700 1 --2- 172.21.15.96:0/1797547729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7530100e40 0x7f7530101220 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:29.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.593+0000 7f7534ceb700 1 -- 172.21.15.96:0/1797547729 >> 172.21.15.96:0/1797547729 conn(0x7f75300fc730 msgr2=0x7f75300feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:29.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.593+0000 7f7534ceb700 1 -- 172.21.15.96:0/1797547729 shutdown_connections 2024-04-22T15:56:29.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.593+0000 7f7534ceb700 1 -- 172.21.15.96:0/1797547729 wait complete. 2024-04-22T15:56:29.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f7534ceb700 1 Processor -- start 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f7534ceb700 1 -- start start 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f7534ceb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7530100e40 0x7f75300731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f7534ceb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75301017f0 0x7f7530073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f7534ceb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530075160 con 0x7f75301017f0 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f7534ceb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530073c60 con 0x7f7530100e40 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f752dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75301017f0 0x7f7530073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f752dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75301017f0 0x7f7530073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49346/0 (socket says 172.21.15.96:49346) 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f752dd9b700 1 -- 172.21.15.96:0/1284013644 learned_addr learned my addr 172.21.15.96:0/1284013644 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f752e59c700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7530100e40 0x7f75300731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f752dd9b700 1 -- 172.21.15.96:0/1284013644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7530100e40 msgr2=0x7f75300731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f752dd9b700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7530100e40 0x7f75300731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f752dd9b700 1 -- 172.21.15.96:0/1284013644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75200095f0 con 0x7f75301017f0 2024-04-22T15:56:29.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.594+0000 7f752e59c700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7530100e40 0x7f75300731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:56:29.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.595+0000 7f752dd9b700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75301017f0 0x7f7530073720 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f752400c960 tx=0x7f752400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:29.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.595+0000 7f751f7fe700 1 -- 172.21.15.96:0/1284013644 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7524007a70 con 0x7f75301017f0 2024-04-22T15:56:29.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.595+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7530073f40 con 0x7f75301017f0 2024-04-22T15:56:29.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.595+0000 7f751f7fe700 1 -- 172.21.15.96:0/1284013644 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7524007bd0 con 0x7f75301017f0 2024-04-22T15:56:29.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.595+0000 7f751f7fe700 1 -- 172.21.15.96:0/1284013644 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75240186c0 con 0x7f75301017f0 2024-04-22T15:56:29.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.595+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75301a2fb0 con 0x7f75301017f0 2024-04-22T15:56:29.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.596+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75301091e0 con 0x7f75301017f0 2024-04-22T15:56:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.596+0000 7f751f7fe700 1 -- 172.21.15.96:0/1284013644 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f752400faa0 con 0x7f75301017f0 2024-04-22T15:56:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.597+0000 7f751f7fe700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751806ecc0 0x7f7518071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.597+0000 7f752e59c700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751806ecc0 0x7f7518071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.597+0000 7f751f7fe700 1 -- 172.21.15.96:0/1284013644 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f752408eed0 con 0x7f75301017f0 2024-04-22T15:56:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.597+0000 7f752e59c700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751806ecc0 0x7f7518071180 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7520005f50 tx=0x7f7520005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:29.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.602+0000 7f751f7fe700 1 -- 172.21.15.96:0/1284013644 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f752405b760 con 0x7f75301017f0 2024-04-22T15:56:29.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.821+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75300746e0 con 0x7f751806ecc0 2024-04-22T15:56:30.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.822+0000 7f751f7fe700 1 -- 172.21.15.96:0/1284013644 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f75300746e0 con 0x7f751806ecc0 2024-04-22T15:56:30.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751806ecc0 msgr2=0x7f7518071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751806ecc0 0x7f7518071180 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7520005f50 tx=0x7f7520005dc0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75301017f0 msgr2=0x7f7530073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75301017f0 0x7f7530073720 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f752400c960 tx=0x7f752400cd20 comp rx=0 tx=0).stop 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 shutdown_connections 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751806ecc0 0x7f7518071180 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7530100e40 0x7f75300731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 --2- 172.21.15.96:0/1284013644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75301017f0 0x7f7530073720 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 >> 172.21.15.96:0/1284013644 conn(0x7f75300fc730 msgr2=0x7f75300fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 shutdown_connections 2024-04-22T15:56:30.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.824+0000 7f7534ceb700 1 -- 172.21.15.96:0/1284013644 wait complete. 2024-04-22T15:56:30.169 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:56:30.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.928+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/771058912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 msgr2=0x7fe0d810ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.928+0000 7fe0dcfc6700 1 --2- 172.21.15.96:0/771058912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 0x7fe0d810ac60 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0009910 tx=0x7fe0c0009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:30.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.928+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/771058912 shutdown_connections 2024-04-22T15:56:30.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.928+0000 7fe0dcfc6700 1 --2- 172.21.15.96:0/771058912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 0x7fe0d810ac60 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.928+0000 7fe0dcfc6700 1 --2- 172.21.15.96:0/771058912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0d8101e50 0x7fe0d8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.928+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/771058912 >> 172.21.15.96:0/771058912 conn(0x7fe0d80fb640 msgr2=0x7fe0d80fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:30.170 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:29 smithi096 ceph-mon[112443]: pgmap v1040: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:30.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.928+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/771058912 shutdown_connections 2024-04-22T15:56:30.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.928+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/771058912 wait complete. 2024-04-22T15:56:30.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0dcfc6700 1 Processor -- start 2024-04-22T15:56:30.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0dcfc6700 1 -- start start 2024-04-22T15:56:30.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0dcfc6700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0d8101e50 0x7fe0d819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:30.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0dcfc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 0x7fe0d819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:30.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0dcfc6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d819d3d0 con 0x7fe0d8102770 2024-04-22T15:56:30.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0dcfc6700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d81969b0 con 0x7fe0d8101e50 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0cffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 0x7fe0d819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0cffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 0x7fe0d819cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49354/0 (socket says 172.21.15.96:49354) 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0cffff700 1 -- 172.21.15.96:0/1739982708 learned_addr learned my addr 172.21.15.96:0/1739982708 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0d659c700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0d8101e50 0x7fe0d819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0cffff700 1 -- 172.21.15.96:0/1739982708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0d8101e50 msgr2=0x7fe0d819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0cffff700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0d8101e50 0x7fe0d819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.929+0000 7fe0cffff700 1 -- 172.21.15.96:0/1739982708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c00095f0 con 0x7fe0d8102770 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.930+0000 7fe0cffff700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 0x7fe0d819cdb0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000bd70 tx=0x7fe0c000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.930+0000 7fe0cf7fe700 1 -- 172.21.15.96:0/1739982708 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0c001d070 con 0x7fe0d8102770 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.930+0000 7fe0cf7fe700 1 -- 172.21.15.96:0/1739982708 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0c00056e0 con 0x7fe0d8102770 2024-04-22T15:56:30.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.930+0000 7fe0cf7fe700 1 -- 172.21.15.96:0/1739982708 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0c000fe50 con 0x7fe0d8102770 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.930+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d8196c30 con 0x7fe0d8102770 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.930+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d8197120 con 0x7fe0d8102770 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.931+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d8105570 con 0x7fe0d8102770 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.932+0000 7fe0cf7fe700 1 -- 172.21.15.96:0/1739982708 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe0c000f680 con 0x7fe0d8102770 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.932+0000 7fe0cf7fe700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe0c406ecc0 0x7fe0c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.932+0000 7fe0cf7fe700 1 -- 172.21.15.96:0/1739982708 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe0c00901a0 con 0x7fe0d8102770 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.932+0000 7fe0d659c700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe0c406ecc0 0x7fe0c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.933+0000 7fe0d659c700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe0c406ecc0 0x7fe0c4071180 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8005950 tx=0x7fe0c800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:29.937+0000 7fe0cf7fe700 1 -- 172.21.15.96:0/1739982708 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe0c005ca30 con 0x7fe0d8102770 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.166+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0d8197ee0 con 0x7fe0c406ecc0 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.167+0000 7fe0cf7fe700 1 -- 172.21.15.96:0/1739982708 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe0d8197ee0 con 0x7fe0c406ecc0 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe0c406ecc0 msgr2=0x7fe0c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe0c406ecc0 0x7fe0c4071180 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8005950 tx=0x7fe0c800b410 comp rx=0 tx=0).stop 2024-04-22T15:56:30.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 msgr2=0x7fe0d819cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 0x7fe0d819cdb0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000bd70 tx=0x7fe0c000bda0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 shutdown_connections 2024-04-22T15:56:30.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fe0c406ecc0 0x7fe0c4071180 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0d8101e50 0x7fe0d819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 --2- 172.21.15.96:0/1739982708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0d8102770 0x7fe0d819cdb0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 >> 172.21.15.96:0/1739982708 conn(0x7fe0d80fb640 msgr2=0x7fe0d8103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:30.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 shutdown_connections 2024-04-22T15:56:30.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.169+0000 7fe0dcfc6700 1 -- 172.21.15.96:0/1739982708 wait complete. 2024-04-22T15:56:30.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:29 smithi196 ceph-mon[81815]: pgmap v1040: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:30.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.273+0000 7f2ed77fe700 1 -- 172.21.15.96:0/1856985057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed00ffaa0 msgr2=0x7f2ed00fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.273+0000 7f2ed77fe700 1 --2- 172.21.15.96:0/1856985057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed00ffaa0 0x7f2ed00fff20 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4009910 tx=0x7f2ec4009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:30.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.273+0000 7f2ed77fe700 1 -- 172.21.15.96:0/1856985057 shutdown_connections 2024-04-22T15:56:30.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.273+0000 7f2ed77fe700 1 --2- 172.21.15.96:0/1856985057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed00ffaa0 0x7f2ed00fff20 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.273+0000 7f2ed77fe700 1 --2- 172.21.15.96:0/1856985057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ed0105aa0 0x7f2ed0105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.273+0000 7f2ed77fe700 1 -- 172.21.15.96:0/1856985057 >> 172.21.15.96:0/1856985057 conn(0x7f2ed0076740 msgr2=0x7f2ed00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:30.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.273+0000 7f2ed77fe700 1 -- 172.21.15.96:0/1856985057 shutdown_connections 2024-04-22T15:56:30.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.273+0000 7f2ed77fe700 1 -- 172.21.15.96:0/1856985057 wait complete. 2024-04-22T15:56:30.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed77fe700 1 Processor -- start 2024-04-22T15:56:30.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed77fe700 1 -- start start 2024-04-22T15:56:30.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ed00ffaa0 0x7f2ed0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:30.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed0105aa0 0x7f2ed0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed77fe700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed0198510 con 0x7f2ed0105aa0 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed77fe700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed0198650 con 0x7f2ed00ffaa0 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed4d99700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed0105aa0 0x7f2ed0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed4d99700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed0105aa0 0x7f2ed0197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49360/0 (socket says 172.21.15.96:49360) 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed4d99700 1 -- 172.21.15.96:0/490724522 learned_addr learned my addr 172.21.15.96:0/490724522 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed559a700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ed00ffaa0 0x7f2ed0197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed4d99700 1 -- 172.21.15.96:0/490724522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ed00ffaa0 msgr2=0x7f2ed0197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed4d99700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ed00ffaa0 0x7f2ed0197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed4d99700 1 -- 172.21.15.96:0/490724522 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec40095f0 con 0x7f2ed0105aa0 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed559a700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ed00ffaa0 0x7f2ed0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.274+0000 7f2ed4d99700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed0105aa0 0x7f2ed0197f40 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006010 tx=0x7f2ec400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.275+0000 7f2ece7fc700 1 -- 172.21.15.96:0/490724522 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ec401d070 con 0x7f2ed0105aa0 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.275+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed0198870 con 0x7f2ed0105aa0 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.275+0000 7f2ece7fc700 1 -- 172.21.15.96:0/490724522 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ec400f460 con 0x7f2ed0105aa0 2024-04-22T15:56:30.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.275+0000 7f2ece7fc700 1 -- 172.21.15.96:0/490724522 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ec4017630 con 0x7f2ed0105aa0 2024-04-22T15:56:30.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.275+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed01a0dd0 con 0x7f2ed0105aa0 2024-04-22T15:56:30.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.276+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed01080c0 con 0x7f2ed0105aa0 2024-04-22T15:56:30.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.276+0000 7f2ece7fc700 1 -- 172.21.15.96:0/490724522 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2ec400f5d0 con 0x7f2ed0105aa0 2024-04-22T15:56:30.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.276+0000 7f2ece7fc700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2eb806ecc0 0x7f2eb8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:30.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.277+0000 7f2ed559a700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2eb806ecc0 0x7f2eb8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:30.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.277+0000 7f2ece7fc700 1 -- 172.21.15.96:0/490724522 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2ec4090190 con 0x7f2ed0105aa0 2024-04-22T15:56:30.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.277+0000 7f2ed559a700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2eb806ecc0 0x7f2eb8071180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8007eb0 tx=0x7f2ec80073c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:30.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.281+0000 7f2ece7fc700 1 -- 172.21.15.96:0/490724522 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2ec4058fb0 con 0x7f2ed0105aa0 2024-04-22T15:56:30.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.486+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ed0198870 con 0x7f2eb806ecc0 2024-04-22T15:56:30.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.497+0000 7f2ece7fc700 1 -- 172.21.15.96:0/490724522 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2ed0198870 con 0x7f2eb806ecc0 2024-04-22T15:56:30.500 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:56:30.500 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (35m) 12m ago 46m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:56:30.500 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (46m) 12m ago 46m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (45m) 12m ago 45m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (32m) 12m ago 46m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (32m) 12m ago 45m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (34m) 12m ago 46m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (28m) 12m ago 40m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (28m) 12m ago 40m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (28m) 12m ago 40m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (12m) 12m ago 40m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (37m) 12m ago 47m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (36m) 12m ago 45m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (33m) 12m ago 48m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (33m) 12m ago 45m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (36m) 12m ago 46m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:56:30.501 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (36m) 12m ago 45m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:56:30.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (31m) 12m ago 44m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:56:30.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (31m) 12m ago 44m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:56:30.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (30m) 12m ago 43m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:56:30.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (30m) 12m ago 43m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:56:30.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (29m) 12m ago 42m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:56:30.502 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (29m) 12m ago 42m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:56:30.502 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (35m) 12m ago 46m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2eb806ecc0 msgr2=0x7f2eb8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2eb806ecc0 0x7f2eb8071180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8007eb0 tx=0x7f2ec80073c0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed0105aa0 msgr2=0x7f2ed0197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed0105aa0 0x7f2ed0197f40 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006010 tx=0x7f2ec400be20 comp rx=0 tx=0).stop 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 shutdown_connections 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2eb806ecc0 0x7f2eb8071180 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ed00ffaa0 0x7f2ed0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 --2- 172.21.15.96:0/490724522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ed0105aa0 0x7f2ed0197f40 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 >> 172.21.15.96:0/490724522 conn(0x7f2ed0076740 msgr2=0x7f2ed00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 shutdown_connections 2024-04-22T15:56:30.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.499+0000 7f2ed77fe700 1 -- 172.21.15.96:0/490724522 wait complete. 2024-04-22T15:56:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.602+0000 7f06495da700 1 -- 172.21.15.96:0/38910144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 msgr2=0x7f064410ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.602+0000 7f06495da700 1 --2- 172.21.15.96:0/38910144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 0x7f064410ac20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f062c009910 tx=0x7f062c009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.602+0000 7f06495da700 1 -- 172.21.15.96:0/38910144 shutdown_connections 2024-04-22T15:56:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.602+0000 7f06495da700 1 --2- 172.21.15.96:0/38910144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 0x7f064410ac20 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.602+0000 7f06495da700 1 --2- 172.21.15.96:0/38910144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0644101e10 0x7f06441021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.602+0000 7f06495da700 1 -- 172.21.15.96:0/38910144 >> 172.21.15.96:0/38910144 conn(0x7f06440fb6a0 msgr2=0x7f06440fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.602+0000 7f06495da700 1 -- 172.21.15.96:0/38910144 shutdown_connections 2024-04-22T15:56:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.602+0000 7f06495da700 1 -- 172.21.15.96:0/38910144 wait complete. 2024-04-22T15:56:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f06495da700 1 Processor -- start 2024-04-22T15:56:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f06495da700 1 -- start start 2024-04-22T15:56:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f06495da700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0644101e10 0x7f064419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f06495da700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 0x7f064419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f06495da700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064419d360 con 0x7f0644102730 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f06495da700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06441969d0 con 0x7f0644101e10 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f063bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 0x7f064419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f063bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 0x7f064419cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51012/0 (socket says 172.21.15.96:51012) 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f063bfff700 1 -- 172.21.15.96:0/248568076 learned_addr learned my addr 172.21.15.96:0/248568076 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f0642ffd700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0644101e10 0x7f064419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f063bfff700 1 -- 172.21.15.96:0/248568076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0644101e10 msgr2=0x7f064419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f063bfff700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0644101e10 0x7f064419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.603+0000 7f063bfff700 1 -- 172.21.15.96:0/248568076 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062c0095f0 con 0x7f0644102730 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.604+0000 7f063bfff700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 0x7f064419cdd0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f062c0092e0 tx=0x7f062c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.604+0000 7f0640ff9700 1 -- 172.21.15.96:0/248568076 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f062c01d070 con 0x7f0644102730 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.604+0000 7f0640ff9700 1 -- 172.21.15.96:0/248568076 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f062c003d50 con 0x7f0644102730 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.604+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0644196c50 con 0x7f0644102730 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.604+0000 7f0640ff9700 1 -- 172.21.15.96:0/248568076 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f062c017400 con 0x7f0644102730 2024-04-22T15:56:30.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.604+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0644197140 con 0x7f0644102730 2024-04-22T15:56:30.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.605+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0644105530 con 0x7f0644102730 2024-04-22T15:56:30.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.605+0000 7f0640ff9700 1 -- 172.21.15.96:0/248568076 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f062c003890 con 0x7f0644102730 2024-04-22T15:56:30.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.606+0000 7f0640ff9700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f063006ecc0 0x7f0630071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:30.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.606+0000 7f0642ffd700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f063006ecc0 0x7f0630071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:30.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.606+0000 7f0640ff9700 1 -- 172.21.15.96:0/248568076 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f062c090250 con 0x7f0644102730 2024-04-22T15:56:30.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.606+0000 7f0642ffd700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f063006ecc0 0x7f0630071180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0634005ea0 tx=0x7f0634005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.611+0000 7f0640ff9700 1 -- 172.21.15.96:0/248568076 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f062c05cae0 con 0x7f0644102730 2024-04-22T15:56:30.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.895+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0644197e90 con 0x7f0644102730 2024-04-22T15:56:30.898 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:30 smithi096 ceph-mon[112443]: from='client.35880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:30.898 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:30 smithi096 ceph-mon[112443]: from='client.35884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:30.898 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:30 smithi096 ceph-mon[112443]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:30.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.895+0000 7f0640ff9700 1 -- 172.21.15.96:0/248568076 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f062c05c670 con 0x7f0644102730 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:56:30.899 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:56:30.900 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:56:30.900 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:56:30.900 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:56:30.900 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:56:30.900 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:56:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.897+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f063006ecc0 msgr2=0x7f0630071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.897+0000 7f06495da700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f063006ecc0 0x7f0630071180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0634005ea0 tx=0x7f0634005e30 comp rx=0 tx=0).stop 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.897+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 msgr2=0x7f064419cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.897+0000 7f06495da700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 0x7f064419cdd0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f062c0092e0 tx=0x7f062c00be20 comp rx=0 tx=0).stop 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.897+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 shutdown_connections 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.897+0000 7f06495da700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f063006ecc0 0x7f0630071180 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.897+0000 7f06495da700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0644101e10 0x7f064419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.897+0000 7f06495da700 1 --2- 172.21.15.96:0/248568076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644102730 0x7f064419cdd0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.897+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 >> 172.21.15.96:0/248568076 conn(0x7f06440fb6a0 msgr2=0x7f06441030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.898+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 shutdown_connections 2024-04-22T15:56:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:30.898+0000 7f06495da700 1 -- 172.21.15.96:0/248568076 wait complete. 2024-04-22T15:56:31.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.001+0000 7f5ed6619700 1 -- 172.21.15.96:0/82726846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 msgr2=0x7f5ed01021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.001+0000 7f5ed6619700 1 --2- 172.21.15.96:0/82726846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 0x7f5ed01021a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009940 tx=0x7f5ec4009c50 comp rx=0 tx=0).stop 2024-04-22T15:56:31.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.001+0000 7f5ed6619700 1 -- 172.21.15.96:0/82726846 shutdown_connections 2024-04-22T15:56:31.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.001+0000 7f5ed6619700 1 --2- 172.21.15.96:0/82726846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ed01026e0 0x7f5ed010abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.001+0000 7f5ed6619700 1 --2- 172.21.15.96:0/82726846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 0x7f5ed01021a0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.001+0000 7f5ed6619700 1 -- 172.21.15.96:0/82726846 >> 172.21.15.96:0/82726846 conn(0x7f5ed0076740 msgr2=0x7f5ed00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.001+0000 7f5ed6619700 1 -- 172.21.15.96:0/82726846 shutdown_connections 2024-04-22T15:56:31.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.001+0000 7f5ed6619700 1 -- 172.21.15.96:0/82726846 wait complete. 2024-04-22T15:56:31.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.001+0000 7f5ed6619700 1 Processor -- start 2024-04-22T15:56:31.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ed6619700 1 -- start start 2024-04-22T15:56:31.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ed6619700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 0x7f5ed019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ed6619700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ed01026e0 0x7f5ed019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ed6619700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed019d4e0 con 0x7f5ed0101dc0 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ed6619700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0196ac0 con 0x7f5ed01026e0 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ecffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 0x7f5ed019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ecffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 0x7f5ed019c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51024/0 (socket says 172.21.15.96:51024) 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ecffff700 1 -- 172.21.15.96:0/2846620577 learned_addr learned my addr 172.21.15.96:0/2846620577 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ecffff700 1 -- 172.21.15.96:0/2846620577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ed01026e0 msgr2=0x7f5ed019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ecf7fe700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ed01026e0 0x7f5ed019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ecffff700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ed01026e0 0x7f5ed019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ecffff700 1 -- 172.21.15.96:0/2846620577 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec40095f0 con 0x7f5ed0101dc0 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ecf7fe700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ed01026e0 0x7f5ed019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.002+0000 7f5ecffff700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 0x7f5ed019c980 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4005620 tx=0x7f5ec4005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.003+0000 7f5ecd7fa700 1 -- 172.21.15.96:0/2846620577 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ec401d070 con 0x7f5ed0101dc0 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.003+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed0196d40 con 0x7f5ed0101dc0 2024-04-22T15:56:31.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.003+0000 7f5ecd7fa700 1 -- 172.21.15.96:0/2846620577 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ec400bed0 con 0x7f5ed0101dc0 2024-04-22T15:56:31.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.003+0000 7f5ecd7fa700 1 -- 172.21.15.96:0/2846620577 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ec4021410 con 0x7f5ed0101dc0 2024-04-22T15:56:31.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.003+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed0197230 con 0x7f5ed0101dc0 2024-04-22T15:56:31.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.004+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed010a8b0 con 0x7f5ed0101dc0 2024-04-22T15:56:31.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.004+0000 7f5ecd7fa700 1 -- 172.21.15.96:0/2846620577 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5ec400f7f0 con 0x7f5ed0101dc0 2024-04-22T15:56:31.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.005+0000 7f5ecd7fa700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5eb806ecc0 0x7f5eb8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.005+0000 7f5ecd7fa700 1 -- 172.21.15.96:0/2846620577 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5ec4090120 con 0x7f5ed0101dc0 2024-04-22T15:56:31.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.005+0000 7f5ecf7fe700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5eb806ecc0 0x7f5eb8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.005+0000 7f5ecf7fe700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5eb806ecc0 0x7f5eb8071180 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0197c80 tx=0x7f5ec0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.009+0000 7f5ecd7fa700 1 -- 172.21.15.96:0/2846620577 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5ec405c9b0 con 0x7f5ed0101dc0 2024-04-22T15:56:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:30 smithi196 ceph-mon[81815]: from='client.35880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:30 smithi196 ceph-mon[81815]: from='client.35884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:30 smithi196 ceph-mon[81815]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:31.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.261+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5ed0197f80 con 0x7f5ed0101dc0 2024-04-22T15:56:31.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.261+0000 7f5ecd7fa700 1 -- 172.21.15.96:0/2846620577 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f5ec405c540 con 0x7f5ed0101dc0 2024-04-22T15:56:31.264 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:56:31.264 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:56:31.264 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:56:31.264 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:56:31.264 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:56:31.265 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:56:31.266 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:56:31.267 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:56:31.267 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:56:31.267 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:56:31.267 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:56:31.267 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:56:31.267 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:56:31.267 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:56:31.267 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:56:31.267 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.263+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5eb806ecc0 msgr2=0x7f5eb8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.263+0000 7f5ed6619700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5eb806ecc0 0x7f5eb8071180 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0197c80 tx=0x7f5ec0008040 comp rx=0 tx=0).stop 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.263+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 msgr2=0x7f5ed019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.263+0000 7f5ed6619700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 0x7f5ed019c980 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4005620 tx=0x7f5ec4005700 comp rx=0 tx=0).stop 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.263+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 shutdown_connections 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.264+0000 7f5ed6619700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5eb806ecc0 0x7f5eb8071180 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.264+0000 7f5ed6619700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ed0101dc0 0x7f5ed019c980 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.264+0000 7f5ed6619700 1 --2- 172.21.15.96:0/2846620577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ed01026e0 0x7f5ed019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.264+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 >> 172.21.15.96:0/2846620577 conn(0x7f5ed0076740 msgr2=0x7f5ed0103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.264+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 shutdown_connections 2024-04-22T15:56:31.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.264+0000 7f5ed6619700 1 -- 172.21.15.96:0/2846620577 wait complete. 2024-04-22T15:56:31.269 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:56:31.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.367+0000 7f44cb89f700 1 -- 172.21.15.96:0/4090182030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c4105b80 msgr2=0x7f44c4105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.367+0000 7f44cb89f700 1 --2- 172.21.15.96:0/4090182030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c4105b80 0x7f44c4105f60 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009910 tx=0x7f44b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:31.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.367+0000 7f44cb89f700 1 -- 172.21.15.96:0/4090182030 shutdown_connections 2024-04-22T15:56:31.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.367+0000 7f44cb89f700 1 --2- 172.21.15.96:0/4090182030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44c40ffaf0 0x7f44c40fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.367+0000 7f44cb89f700 1 --2- 172.21.15.96:0/4090182030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c4105b80 0x7f44c4105f60 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.367+0000 7f44cb89f700 1 -- 172.21.15.96:0/4090182030 >> 172.21.15.96:0/4090182030 conn(0x7f44c40fb640 msgr2=0x7f44c40fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.367+0000 7f44cb89f700 1 -- 172.21.15.96:0/4090182030 shutdown_connections 2024-04-22T15:56:31.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.367+0000 7f44cb89f700 1 -- 172.21.15.96:0/4090182030 wait complete. 2024-04-22T15:56:31.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44cb89f700 1 Processor -- start 2024-04-22T15:56:31.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44cb89f700 1 -- start start 2024-04-22T15:56:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44cb89f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c40ffaf0 0x7f44c4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44cb89f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44c4105b80 0x7f44c4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44cb89f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c41985a0 con 0x7f44c40ffaf0 2024-04-22T15:56:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44cb89f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c41986e0 con 0x7f44c4105b80 2024-04-22T15:56:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44c963b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c40ffaf0 0x7f44c4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44c963b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c40ffaf0 0x7f44c4197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51050/0 (socket says 172.21.15.96:51050) 2024-04-22T15:56:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44c963b700 1 -- 172.21.15.96:0/2797798644 learned_addr learned my addr 172.21.15.96:0/2797798644 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:56:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.368+0000 7f44c8e3a700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44c4105b80 0x7f44c4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.369+0000 7f44c963b700 1 -- 172.21.15.96:0/2797798644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44c4105b80 msgr2=0x7f44c4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.369+0000 7f44c963b700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44c4105b80 0x7f44c4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.369+0000 7f44c963b700 1 -- 172.21.15.96:0/2797798644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b40095f0 con 0x7f44c40ffaf0 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.369+0000 7f44c963b700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c40ffaf0 0x7f44c4197a90 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f44b4006010 tx=0x7f44b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.369+0000 7f44ba7fc700 1 -- 172.21.15.96:0/2797798644 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44b401d070 con 0x7f44c40ffaf0 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.369+0000 7f44ba7fc700 1 -- 172.21.15.96:0/2797798644 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44b400f460 con 0x7f44c40ffaf0 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.369+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c41a0b00 con 0x7f44c40ffaf0 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.369+0000 7f44ba7fc700 1 -- 172.21.15.96:0/2797798644 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44b4017630 con 0x7f44c40ffaf0 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.369+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c41a0e60 con 0x7f44c40ffaf0 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.370+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c41080f0 con 0x7f44c40ffaf0 2024-04-22T15:56:31.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.370+0000 7f44ba7fc700 1 -- 172.21.15.96:0/2797798644 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f44b400f5d0 con 0x7f44c40ffaf0 2024-04-22T15:56:31.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.371+0000 7f44ba7fc700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44b006ecc0 0x7f44b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.371+0000 7f44ba7fc700 1 -- 172.21.15.96:0/2797798644 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f44b4090340 con 0x7f44c40ffaf0 2024-04-22T15:56:31.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.371+0000 7f44c8e3a700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44b006ecc0 0x7f44b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.371+0000 7f44c8e3a700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44b006ecc0 0x7f44b0071180 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f44c0006fd0 tx=0x7f44c0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.375+0000 7f44ba7fc700 1 -- 172.21.15.96:0/2797798644 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f44b40591e0 con 0x7f44c40ffaf0 2024-04-22T15:56:31.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.592+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44c4069770 con 0x7f44b006ecc0 2024-04-22T15:56:31.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.594+0000 7f44ba7fc700 1 -- 172.21.15.96:0/2797798644 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f44c4069770 con 0x7f44b006ecc0 2024-04-22T15:56:31.596 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:56:31.597 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44b006ecc0 msgr2=0x7f44b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44b006ecc0 0x7f44b0071180 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f44c0006fd0 tx=0x7f44c0008040 comp rx=0 tx=0).stop 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c40ffaf0 msgr2=0x7f44c4197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c40ffaf0 0x7f44c4197a90 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f44b4006010 tx=0x7f44b400be20 comp rx=0 tx=0).stop 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 shutdown_connections 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f44b006ecc0 0x7f44b0071180 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44c40ffaf0 0x7f44c4197a90 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 --2- 172.21.15.96:0/2797798644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44c4105b80 0x7f44c4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 >> 172.21.15.96:0/2797798644 conn(0x7f44c40fb640 msgr2=0x7f44c40fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 shutdown_connections 2024-04-22T15:56:31.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.596+0000 7f44cb89f700 1 -- 172.21.15.96:0/2797798644 wait complete. 2024-04-22T15:56:31.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 -- 172.21.15.96:0/392715376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 msgr2=0x7fee74105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 --2- 172.21.15.96:0/392715376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 0x7fee74105820 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fee64009910 tx=0x7fee64009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:31.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 -- 172.21.15.96:0/392715376 shutdown_connections 2024-04-22T15:56:31.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 --2- 172.21.15.96:0/392715376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 0x7fee74105820 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 --2- 172.21.15.96:0/392715376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee74100e20 0x7fee74101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 -- 172.21.15.96:0/392715376 >> 172.21.15.96:0/392715376 conn(0x7fee740fc6d0 msgr2=0x7fee740feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 -- 172.21.15.96:0/392715376 shutdown_connections 2024-04-22T15:56:31.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 -- 172.21.15.96:0/392715376 wait complete. 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 Processor -- start 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.699+0000 7fee7ac74700 1 -- start start 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee7ac74700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee74100e20 0x7fee740731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee7ac74700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 0x7fee74073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee7ac74700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee740751b0 con 0x7fee741017d0 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee7ac74700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee74073c60 con 0x7fee74100e20 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 0x7fee74073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 0x7fee74073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51076/0 (socket says 172.21.15.96:51076) 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee6bfff700 1 -- 172.21.15.96:0/1507990886 learned_addr learned my addr 172.21.15.96:0/1507990886 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:56:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee78a10700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee74100e20 0x7fee740731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee6bfff700 1 -- 172.21.15.96:0/1507990886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee74100e20 msgr2=0x7fee740731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee6bfff700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee74100e20 0x7fee740731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee6bfff700 1 -- 172.21.15.96:0/1507990886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee640095f0 con 0x7fee741017d0 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee78a10700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee74100e20 0x7fee740731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee6bfff700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 0x7fee74073720 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fee64005b40 tx=0x7fee6400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee69ffb700 1 -- 172.21.15.96:0/1507990886 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee6401d070 con 0x7fee741017d0 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee69ffb700 1 -- 172.21.15.96:0/1507990886 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee6400f460 con 0x7fee741017d0 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee74073eb0 con 0x7fee741017d0 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.700+0000 7fee69ffb700 1 -- 172.21.15.96:0/1507990886 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee64017630 con 0x7fee741017d0 2024-04-22T15:56:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.701+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee741a2eb0 con 0x7fee741017d0 2024-04-22T15:56:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.702+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee74109220 con 0x7fee741017d0 2024-04-22T15:56:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.702+0000 7fee69ffb700 1 -- 172.21.15.96:0/1507990886 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fee64003a60 con 0x7fee741017d0 2024-04-22T15:56:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.702+0000 7fee69ffb700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fee5c06ecd0 0x7fee5c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.702+0000 7fee69ffb700 1 -- 172.21.15.96:0/1507990886 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fee640902e0 con 0x7fee741017d0 2024-04-22T15:56:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.702+0000 7fee78a10700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fee5c06ecd0 0x7fee5c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.703+0000 7fee78a10700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fee5c06ecd0 0x7fee5c071190 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fee7000d340 tx=0x7fee70009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.708+0000 7fee69ffb700 1 -- 172.21.15.96:0/1507990886 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fee640590c0 con 0x7fee741017d0 2024-04-22T15:56:31.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:31 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/248568076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:31.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:31 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2846620577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:56:31.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:31 smithi096 ceph-mon[112443]: from='client.35900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:31.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:31 smithi096 ceph-mon[112443]: pgmap v1041: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:31.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.991+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fee74102300 con 0x7fee741017d0 2024-04-22T15:56:31.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.991+0000 7fee69ffb700 1 -- 172.21.15.96:0/1507990886 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fee6405c780 con 0x7fee741017d0 2024-04-22T15:56:31.994 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:56:31.994 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:56:31.994 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:56:31.994 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:56:31.994 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:56:31.994 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:56:31.994 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:56:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:56:31.996 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:56:31.996 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:56:31.996 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:56:31.996 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:56:31.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.993+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fee5c06ecd0 msgr2=0x7fee5c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.993+0000 7fee7ac74700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fee5c06ecd0 0x7fee5c071190 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fee7000d340 tx=0x7fee70009040 comp rx=0 tx=0).stop 2024-04-22T15:56:31.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.993+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 msgr2=0x7fee74073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.993+0000 7fee7ac74700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 0x7fee74073720 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fee64005b40 tx=0x7fee6400be20 comp rx=0 tx=0).stop 2024-04-22T15:56:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.993+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 shutdown_connections 2024-04-22T15:56:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.993+0000 7fee7ac74700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fee5c06ecd0 0x7fee5c071190 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.993+0000 7fee7ac74700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee74100e20 0x7fee740731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.993+0000 7fee7ac74700 1 --2- 172.21.15.96:0/1507990886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee741017d0 0x7fee74073720 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.993+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 >> 172.21.15.96:0/1507990886 conn(0x7fee740fc6d0 msgr2=0x7fee740fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.994+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 shutdown_connections 2024-04-22T15:56:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:56:31.994+0000 7fee7ac74700 1 -- 172.21.15.96:0/1507990886 wait complete. 2024-04-22T15:56:32.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:31 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/248568076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:32.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:31 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2846620577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:56:32.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:31 smithi196 ceph-mon[81815]: from='client.35900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:32.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:31 smithi196 ceph-mon[81815]: pgmap v1041: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:33.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:32 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1507990886' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:56:33.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:32 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1507990886' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:56:34.070 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:33 smithi196 ceph-mon[81815]: pgmap v1042: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:34.137 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:33 smithi096 ceph-mon[112443]: pgmap v1042: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:36.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:35 smithi096 ceph-mon[112443]: pgmap v1043: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:36.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:35 smithi196 ceph-mon[81815]: pgmap v1043: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:38.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:37 smithi096 ceph-mon[112443]: pgmap v1044: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:38.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:37 smithi196 ceph-mon[81815]: pgmap v1044: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:39.123 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:39.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:40.122 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:39 smithi096 ceph-mon[112443]: pgmap v1045: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:40.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:39 smithi196 ceph-mon[81815]: pgmap v1045: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:42.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:41 smithi196 ceph-mon[81815]: pgmap v1046: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:42.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:41 smithi096 ceph-mon[112443]: pgmap v1046: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:44.177 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:43 smithi196 ceph-mon[81815]: pgmap v1047: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:44.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:43 smithi096 ceph-mon[112443]: pgmap v1047: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:46.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:45 smithi096 ceph-mon[112443]: pgmap v1048: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:45 smithi196 ceph-mon[81815]: pgmap v1048: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:48.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:47 smithi096 ceph-mon[112443]: pgmap v1049: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:48.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:47 smithi196 ceph-mon[81815]: pgmap v1049: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:50.229 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:49 smithi096 ceph-mon[112443]: pgmap v1050: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:50.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:49 smithi196 ceph-mon[81815]: pgmap v1050: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:52.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:51 smithi096 ceph-mon[112443]: pgmap v1051: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:51 smithi196 ceph-mon[81815]: pgmap v1051: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:53.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:54.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:54 smithi096 ceph-mon[112443]: pgmap v1052: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:54.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:54 smithi196 ceph-mon[81815]: pgmap v1052: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:56.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:55 smithi096 ceph-mon[112443]: pgmap v1053: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:55 smithi196 ceph-mon[81815]: pgmap v1053: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:58.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:57 smithi096 ceph-mon[112443]: pgmap v1054: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:56:58.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:57 smithi196 ceph-mon[81815]: pgmap v1054: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:00.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:56:59 smithi096 ceph-mon[112443]: pgmap v1055: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:00.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:56:59 smithi196 ceph-mon[81815]: pgmap v1055: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:02.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.110+0000 7f18b712a700 1 -- 172.21.15.96:0/839782288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b0100d90 msgr2=0x7f18b0104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.110+0000 7f18b712a700 1 --2- 172.21.15.96:0/839782288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b0100d90 0x7f18b0104760 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f189c009910 tx=0x7f189c009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.110+0000 7f18b712a700 1 -- 172.21.15.96:0/839782288 shutdown_connections 2024-04-22T15:57:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.110+0000 7f18b712a700 1 --2- 172.21.15.96:0/839782288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b0100d90 0x7f18b0104760 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.110+0000 7f18b712a700 1 --2- 172.21.15.96:0/839782288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18b01003e0 0x7f18b01007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.110+0000 7f18b712a700 1 -- 172.21.15.96:0/839782288 >> 172.21.15.96:0/839782288 conn(0x7f18b0075f30 msgr2=0x7f18b00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.110+0000 7f18b712a700 1 -- 172.21.15.96:0/839782288 shutdown_connections 2024-04-22T15:57:02.115 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:01 smithi096 ceph-mon[112443]: pgmap v1056: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.111+0000 7f18b712a700 1 -- 172.21.15.96:0/839782288 wait complete. 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.111+0000 7f18b712a700 1 Processor -- start 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.111+0000 7f18b712a700 1 -- start start 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.111+0000 7f18b712a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b01003e0 0x7f18b0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.111+0000 7f18b712a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18b0100d90 0x7f18b0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.111+0000 7f18b712a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b01985b0 con 0x7f18b01003e0 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.111+0000 7f18b712a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b01986f0 con 0x7f18b0100d90 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18b4ec6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b01003e0 0x7f18b0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18b4ec6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b01003e0 0x7f18b0197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49084/0 (socket says 172.21.15.96:49084) 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18b4ec6700 1 -- 172.21.15.96:0/1593151478 learned_addr learned my addr 172.21.15.96:0/1593151478 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18affff700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18b0100d90 0x7f18b0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18b4ec6700 1 -- 172.21.15.96:0/1593151478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18b0100d90 msgr2=0x7f18b0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18b4ec6700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18b0100d90 0x7f18b0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18b4ec6700 1 -- 172.21.15.96:0/1593151478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189c0095f0 con 0x7f18b01003e0 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18affff700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18b0100d90 0x7f18b0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18b4ec6700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b01003e0 0x7f18b0197aa0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f18a400b4b0 tx=0x7f18a400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18adffb700 1 -- 172.21.15.96:0/1593151478 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a4011800 con 0x7f18b01003e0 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b01a0b10 con 0x7f18b01003e0 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18adffb700 1 -- 172.21.15.96:0/1593151478 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18a4011e20 con 0x7f18b01003e0 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.112+0000 7f18adffb700 1 -- 172.21.15.96:0/1593151478 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a400f2e0 con 0x7f18b01003e0 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.113+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b01a0f20 con 0x7f18b01003e0 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.114+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b01080d0 con 0x7f18b01003e0 2024-04-22T15:57:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.114+0000 7f18adffb700 1 -- 172.21.15.96:0/1593151478 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f18a4010380 con 0x7f18b01003e0 2024-04-22T15:57:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.114+0000 7f18adffb700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f18a006ecc0 0x7f18a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.114+0000 7f18adffb700 1 -- 172.21.15.96:0/1593151478 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f18a408ea80 con 0x7f18b01003e0 2024-04-22T15:57:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.114+0000 7f18affff700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f18a006ecc0 0x7f18a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.115+0000 7f18affff700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f18a006ecc0 0x7f18a0071180 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f189c000c00 tx=0x7f189c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:02.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.119+0000 7f18adffb700 1 -- 172.21.15.96:0/1593151478 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f18a40577e0 con 0x7f18b01003e0 2024-04-22T15:57:02.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:01 smithi196 ceph-mon[81815]: pgmap v1056: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:02.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.334+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f18b010cd60 con 0x7f18a006ecc0 2024-04-22T15:57:02.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.335+0000 7f18adffb700 1 -- 172.21.15.96:0/1593151478 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f18b010cd60 con 0x7f18a006ecc0 2024-04-22T15:57:02.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.337+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f18a006ecc0 msgr2=0x7f18a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.337+0000 7f18b712a700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f18a006ecc0 0x7f18a0071180 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f189c000c00 tx=0x7f189c01a040 comp rx=0 tx=0).stop 2024-04-22T15:57:02.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.337+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b01003e0 msgr2=0x7f18b0197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.337+0000 7f18b712a700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b01003e0 0x7f18b0197aa0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f18a400b4b0 tx=0x7f18a400b870 comp rx=0 tx=0).stop 2024-04-22T15:57:02.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.337+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 shutdown_connections 2024-04-22T15:57:02.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.337+0000 7f18b712a700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f18a006ecc0 0x7f18a0071180 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.337+0000 7f18b712a700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18b01003e0 0x7f18b0197aa0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.337+0000 7f18b712a700 1 --2- 172.21.15.96:0/1593151478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18b0100d90 0x7f18b0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.337+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 >> 172.21.15.96:0/1593151478 conn(0x7f18b0075f30 msgr2=0x7f18b0103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:02.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.338+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 shutdown_connections 2024-04-22T15:57:02.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.338+0000 7f18b712a700 1 -- 172.21.15.96:0/1593151478 wait complete. 2024-04-22T15:57:02.350 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:57:02.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.442+0000 7f96dee5e700 1 -- 172.21.15.96:0/502805379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d81017d0 msgr2=0x7f96d8105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.442+0000 7f96dee5e700 1 --2- 172.21.15.96:0/502805379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d81017d0 0x7f96d8105820 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009910 tx=0x7f96c4009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:02.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.443+0000 7f96dee5e700 1 -- 172.21.15.96:0/502805379 shutdown_connections 2024-04-22T15:57:02.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.443+0000 7f96dee5e700 1 --2- 172.21.15.96:0/502805379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d81017d0 0x7f96d8105820 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.443+0000 7f96dee5e700 1 --2- 172.21.15.96:0/502805379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96d8100e20 0x7f96d8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.443+0000 7f96dee5e700 1 -- 172.21.15.96:0/502805379 >> 172.21.15.96:0/502805379 conn(0x7f96d80fc6d0 msgr2=0x7f96d80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.443+0000 7f96dee5e700 1 -- 172.21.15.96:0/502805379 shutdown_connections 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.443+0000 7f96dee5e700 1 -- 172.21.15.96:0/502805379 wait complete. 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.443+0000 7f96dee5e700 1 Processor -- start 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.443+0000 7f96dee5e700 1 -- start start 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dee5e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d8100e20 0x7f96d819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dee5e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96d81017d0 0x7f96d819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dee5e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d819d370 con 0x7f96d8100e20 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dee5e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d81969a0 con 0x7f96d81017d0 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dcbfa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d8100e20 0x7f96d819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dcbfa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d8100e20 0x7f96d819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49094/0 (socket says 172.21.15.96:49094) 2024-04-22T15:57:02.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dcbfa700 1 -- 172.21.15.96:0/1861994730 learned_addr learned my addr 172.21.15.96:0/1861994730 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96d7fff700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96d81017d0 0x7f96d819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dcbfa700 1 -- 172.21.15.96:0/1861994730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96d81017d0 msgr2=0x7f96d819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dcbfa700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96d81017d0 0x7f96d819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dcbfa700 1 -- 172.21.15.96:0/1861994730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c40095f0 con 0x7f96d8100e20 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dcbfa700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d8100e20 0x7f96d819c810 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f96cc00d900 tx=0x7f96cc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96d5ffb700 1 -- 172.21.15.96:0/1861994730 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96cc0041f0 con 0x7f96d8100e20 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96d5ffb700 1 -- 172.21.15.96:0/1861994730 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96cc004cf0 con 0x7f96d8100e20 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d8196c80 con 0x7f96d8100e20 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.444+0000 7f96d5ffb700 1 -- 172.21.15.96:0/1861994730 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96cc00b750 con 0x7f96d8100e20 2024-04-22T15:57:02.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.445+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d81971a0 con 0x7f96d8100e20 2024-04-22T15:57:02.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.446+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d8109220 con 0x7f96d8100e20 2024-04-22T15:57:02.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.446+0000 7f96d5ffb700 1 -- 172.21.15.96:0/1861994730 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f96cc004750 con 0x7f96d8100e20 2024-04-22T15:57:02.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.446+0000 7f96d5ffb700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96c806ecd0 0x7f96c8071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:02.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.447+0000 7f96d5ffb700 1 -- 172.21.15.96:0/1861994730 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f96cc08edb0 con 0x7f96d8100e20 2024-04-22T15:57:02.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.447+0000 7f96d7fff700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96c806ecd0 0x7f96c8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:02.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.447+0000 7f96d7fff700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96c806ecd0 0x7f96c8071190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009fd0 tx=0x7f96c401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:02.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.451+0000 7f96d5ffb700 1 -- 172.21.15.96:0/1861994730 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f96cc05b640 con 0x7f96d8100e20 2024-04-22T15:57:02.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.670+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f96d8197e80 con 0x7f96c806ecd0 2024-04-22T15:57:02.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.671+0000 7f96d5ffb700 1 -- 172.21.15.96:0/1861994730 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f96d8197e80 con 0x7f96c806ecd0 2024-04-22T15:57:02.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.673+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96c806ecd0 msgr2=0x7f96c8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.673+0000 7f96dee5e700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96c806ecd0 0x7f96c8071190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009fd0 tx=0x7f96c401a040 comp rx=0 tx=0).stop 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.673+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d8100e20 msgr2=0x7f96d819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.673+0000 7f96dee5e700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d8100e20 0x7f96d819c810 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f96cc00d900 tx=0x7f96cc00dc10 comp rx=0 tx=0).stop 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.673+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 shutdown_connections 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.673+0000 7f96dee5e700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f96c806ecd0 0x7f96c8071190 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.673+0000 7f96dee5e700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96d8100e20 0x7f96d819c810 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.674+0000 7f96dee5e700 1 --2- 172.21.15.96:0/1861994730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96d81017d0 0x7f96d819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.674+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 >> 172.21.15.96:0/1861994730 conn(0x7f96d80fc6d0 msgr2=0x7f96d80fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.674+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 shutdown_connections 2024-04-22T15:57:02.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.674+0000 7f96dee5e700 1 -- 172.21.15.96:0/1861994730 wait complete. 2024-04-22T15:57:02.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.778+0000 7f71a1f88700 1 -- 172.21.15.96:0/4150460450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c102710 msgr2=0x7f719c10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.778+0000 7f71a1f88700 1 --2- 172.21.15.96:0/4150460450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c102710 0x7f719c10ac00 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f7190009910 tx=0x7f7190009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:02.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.778+0000 7f71a1f88700 1 -- 172.21.15.96:0/4150460450 shutdown_connections 2024-04-22T15:57:02.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.778+0000 7f71a1f88700 1 --2- 172.21.15.96:0/4150460450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c102710 0x7f719c10ac00 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.778+0000 7f71a1f88700 1 --2- 172.21.15.96:0/4150460450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f719c101df0 0x7f719c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.778+0000 7f71a1f88700 1 -- 172.21.15.96:0/4150460450 >> 172.21.15.96:0/4150460450 conn(0x7f719c0fb680 msgr2=0x7f719c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:02.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.778+0000 7f71a1f88700 1 -- 172.21.15.96:0/4150460450 shutdown_connections 2024-04-22T15:57:02.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.778+0000 7f71a1f88700 1 -- 172.21.15.96:0/4150460450 wait complete. 2024-04-22T15:57:02.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f71a1f88700 1 Processor -- start 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f71a1f88700 1 -- start start 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f71a1f88700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c101df0 0x7f719c0ffa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f71a1f88700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f719c102710 0x7f719c0fff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f71a1f88700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c101a20 con 0x7f719c101df0 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f71a1f88700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c1004d0 con 0x7f719c102710 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f719b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c101df0 0x7f719c0ffa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f719b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c101df0 0x7f719c0ffa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49120/0 (socket says 172.21.15.96:49120) 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f719b7fe700 1 -- 172.21.15.96:0/1809296924 learned_addr learned my addr 172.21.15.96:0/1809296924 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.779+0000 7f719affd700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f719c102710 0x7f719c0fff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.780+0000 7f719b7fe700 1 -- 172.21.15.96:0/1809296924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f719c102710 msgr2=0x7f719c0fff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.780+0000 7f719b7fe700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f719c102710 0x7f719c0fff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:02.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.780+0000 7f719b7fe700 1 -- 172.21.15.96:0/1809296924 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71900095f0 con 0x7f719c101df0 2024-04-22T15:57:02.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.780+0000 7f719b7fe700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c101df0 0x7f719c0ffa50 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c930 tx=0x7f718c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:02.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.780+0000 7f7198ff9700 1 -- 172.21.15.96:0/1809296924 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f718c007ad0 con 0x7f719c101df0 2024-04-22T15:57:02.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.780+0000 7f7198ff9700 1 -- 172.21.15.96:0/1809296924 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f718c00ce80 con 0x7f719c101df0 2024-04-22T15:57:02.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.780+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719c100780 con 0x7f719c101df0 2024-04-22T15:57:02.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.780+0000 7f7198ff9700 1 -- 172.21.15.96:0/1809296924 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f718c018630 con 0x7f719c101df0 2024-04-22T15:57:02.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.780+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c1a3010 con 0x7f719c101df0 2024-04-22T15:57:02.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.781+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719c069a10 con 0x7f719c101df0 2024-04-22T15:57:02.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.782+0000 7f7198ff9700 1 -- 172.21.15.96:0/1809296924 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f718c007c30 con 0x7f719c101df0 2024-04-22T15:57:02.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.782+0000 7f7198ff9700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f718406ecd0 0x7f7184071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:02.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.782+0000 7f7198ff9700 1 -- 172.21.15.96:0/1809296924 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f718c08ed80 con 0x7f719c101df0 2024-04-22T15:57:02.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.782+0000 7f719affd700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f718406ecd0 0x7f7184071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:02.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.783+0000 7f719affd700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f718406ecd0 0x7f7184071190 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f7190009fd0 tx=0x7f719001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.787+0000 7f7198ff9700 1 -- 172.21.15.96:0/1809296924 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f718c057b60 con 0x7f719c101df0 2024-04-22T15:57:02.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:02.995+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f719c105540 con 0x7f718406ecd0 2024-04-22T15:57:03.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.006+0000 7f7198ff9700 1 -- 172.21.15.96:0/1809296924 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f719001e040 con 0x7f718406ecd0 2024-04-22T15:57:03.008 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (35m) 13m ago 47m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (47m) 13m ago 47m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (46m) 13m ago 46m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (32m) 13m ago 47m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (32m) 13m ago 46m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (35m) 13m ago 46m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (28m) 13m ago 41m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (29m) 13m ago 41m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (28m) 13m ago 41m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (13m) 13m ago 41m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (38m) 13m ago 48m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (36m) 13m ago 45m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (34m) 13m ago 48m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (33m) 13m ago 45m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:57:03.009 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (36m) 13m ago 47m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:57:03.010 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (36m) 13m ago 45m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:57:03.010 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (32m) 13m ago 45m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:57:03.010 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (31m) 13m ago 44m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:57:03.010 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (31m) 13m ago 44m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:57:03.010 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (30m) 13m ago 43m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:57:03.010 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (30m) 13m ago 43m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:57:03.010 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (29m) 13m ago 42m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:57:03.010 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (36m) 13m ago 46m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f718406ecd0 msgr2=0x7f7184071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f718406ecd0 0x7f7184071190 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f7190009fd0 tx=0x7f719001a040 comp rx=0 tx=0).stop 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c101df0 msgr2=0x7f719c0ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c101df0 0x7f719c0ffa50 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c930 tx=0x7f718c00cc40 comp rx=0 tx=0).stop 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 shutdown_connections 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f718406ecd0 0x7f7184071190 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c101df0 0x7f719c0ffa50 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 --2- 172.21.15.96:0/1809296924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f719c102710 0x7f719c0fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 >> 172.21.15.96:0/1809296924 conn(0x7f719c0fb680 msgr2=0x7f719c1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 shutdown_connections 2024-04-22T15:57:03.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.008+0000 7f71a1f88700 1 -- 172.21.15.96:0/1809296924 wait complete. 2024-04-22T15:57:03.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.111+0000 7f6745bab700 1 -- 172.21.15.96:0/389411072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 msgr2=0x7f6740105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.111+0000 7f6745bab700 1 --2- 172.21.15.96:0/389411072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 0x7f6740105820 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6734009910 tx=0x7f6734009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:03.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.111+0000 7f6745bab700 1 -- 172.21.15.96:0/389411072 shutdown_connections 2024-04-22T15:57:03.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.111+0000 7f6745bab700 1 --2- 172.21.15.96:0/389411072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 0x7f6740105820 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.111+0000 7f6745bab700 1 --2- 172.21.15.96:0/389411072 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6740100e20 0x7f6740101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.111+0000 7f6745bab700 1 -- 172.21.15.96:0/389411072 >> 172.21.15.96:0/389411072 conn(0x7f67400fc6d0 msgr2=0x7f67400feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:03.114 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:02 smithi096 ceph-mon[112443]: from='client.35908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:03.114 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:02 smithi096 ceph-mon[112443]: from='client.35912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:03.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.111+0000 7f6745bab700 1 -- 172.21.15.96:0/389411072 shutdown_connections 2024-04-22T15:57:03.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.111+0000 7f6745bab700 1 -- 172.21.15.96:0/389411072 wait complete. 2024-04-22T15:57:03.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.111+0000 7f6745bab700 1 Processor -- start 2024-04-22T15:57:03.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f6745bab700 1 -- start start 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f6745bab700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6740100e20 0x7f674019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f6745bab700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 0x7f674019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f6745bab700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674019d380 con 0x7f67401017d0 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f6745bab700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6740196960 con 0x7f6740100e20 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f673effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 0x7f674019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f673effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 0x7f674019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49150/0 (socket says 172.21.15.96:49150) 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f673effd700 1 -- 172.21.15.96:0/4228008885 learned_addr learned my addr 172.21.15.96:0/4228008885 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f673f7fe700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6740100e20 0x7f674019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f673effd700 1 -- 172.21.15.96:0/4228008885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6740100e20 msgr2=0x7f674019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f673effd700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6740100e20 0x7f674019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f673effd700 1 -- 172.21.15.96:0/4228008885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340095f0 con 0x7f67401017d0 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.112+0000 7f673effd700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 0x7f674019cd60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6734009fd0 tx=0x7f673400b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.113+0000 7f673cff9700 1 -- 172.21.15.96:0/4228008885 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673401d070 con 0x7f67401017d0 2024-04-22T15:57:03.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.113+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6740196be0 con 0x7f67401017d0 2024-04-22T15:57:03.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.113+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67401970d0 con 0x7f67401017d0 2024-04-22T15:57:03.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.113+0000 7f673cff9700 1 -- 172.21.15.96:0/4228008885 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67340052a0 con 0x7f67401017d0 2024-04-22T15:57:03.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.113+0000 7f673cff9700 1 -- 172.21.15.96:0/4228008885 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6734022470 con 0x7f67401017d0 2024-04-22T15:57:03.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.114+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6740109220 con 0x7f67401017d0 2024-04-22T15:57:03.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.115+0000 7f673cff9700 1 -- 172.21.15.96:0/4228008885 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f673400f770 con 0x7f67401017d0 2024-04-22T15:57:03.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.115+0000 7f673cff9700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f672806ecc0 0x7f6728071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.115+0000 7f673cff9700 1 -- 172.21.15.96:0/4228008885 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6734090120 con 0x7f67401017d0 2024-04-22T15:57:03.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.115+0000 7f673f7fe700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f672806ecc0 0x7f6728071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.116+0000 7f673f7fe700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f672806ecc0 0x7f6728071180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6730009bf0 tx=0x7f6730009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:03.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.122+0000 7f673cff9700 1 -- 172.21.15.96:0/4228008885 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f673405ca30 con 0x7f67401017d0 2024-04-22T15:57:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:02 smithi196 ceph-mon[81815]: from='client.35908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:02 smithi196 ceph-mon[81815]: from='client.35912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:03.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.401+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6740197e20 con 0x7f67401017d0 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.402+0000 7f673cff9700 1 -- 172.21.15.96:0/4228008885 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f673405c5c0 con 0x7f67401017d0 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:57:03.405 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:57:03.406 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:57:03.406 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:57:03.406 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:57:03.406 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:57:03.406 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:57:03.406 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:57:03.406 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:57:03.406 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f672806ecc0 msgr2=0x7f6728071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f672806ecc0 0x7f6728071180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6730009bf0 tx=0x7f6730009260 comp rx=0 tx=0).stop 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 msgr2=0x7f674019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 0x7f674019cd60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6734009fd0 tx=0x7f673400b650 comp rx=0 tx=0).stop 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 shutdown_connections 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f672806ecc0 0x7f6728071180 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6740100e20 0x7f674019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 --2- 172.21.15.96:0/4228008885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67401017d0 0x7f674019cd60 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 >> 172.21.15.96:0/4228008885 conn(0x7f67400fc6d0 msgr2=0x7f67400fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:03.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 shutdown_connections 2024-04-22T15:57:03.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.404+0000 7f6745bab700 1 -- 172.21.15.96:0/4228008885 wait complete. 2024-04-22T15:57:03.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.507+0000 7f3ad4ded700 1 -- 172.21.15.96:0/2136561813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 msgr2=0x7f3ad010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.507+0000 7f3ad4ded700 1 --2- 172.21.15.96:0/2136561813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 0x7f3ad010ab80 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009910 tx=0x7f3ac4009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:03.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.508+0000 7f3ad4ded700 1 -- 172.21.15.96:0/2136561813 shutdown_connections 2024-04-22T15:57:03.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.508+0000 7f3ad4ded700 1 --2- 172.21.15.96:0/2136561813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 0x7f3ad010ab80 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.508+0000 7f3ad4ded700 1 --2- 172.21.15.96:0/2136561813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ad0101ce0 0x7f3ad01020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.508+0000 7f3ad4ded700 1 -- 172.21.15.96:0/2136561813 >> 172.21.15.96:0/2136561813 conn(0x7f3ad00fb5e0 msgr2=0x7f3ad00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:03.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.508+0000 7f3ad4ded700 1 -- 172.21.15.96:0/2136561813 shutdown_connections 2024-04-22T15:57:03.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.508+0000 7f3ad4ded700 1 -- 172.21.15.96:0/2136561813 wait complete. 2024-04-22T15:57:03.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.508+0000 7f3ad4ded700 1 Processor -- start 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.508+0000 7f3ad4ded700 1 -- start start 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3ad4ded700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ad0101ce0 0x7f3ad019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3ad4ded700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 0x7f3ad019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3ad4ded700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad019d4b0 con 0x7f3ad0102690 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3ad4ded700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad0196a90 con 0x7f3ad0101ce0 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3acdd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 0x7f3ad019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3acdd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 0x7f3ad019ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49178/0 (socket says 172.21.15.96:49178) 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3acdd9b700 1 -- 172.21.15.96:0/1123122830 learned_addr learned my addr 172.21.15.96:0/1123122830 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3ace59c700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ad0101ce0 0x7f3ad019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3acdd9b700 1 -- 172.21.15.96:0/1123122830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ad0101ce0 msgr2=0x7f3ad019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3acdd9b700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ad0101ce0 0x7f3ad019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3acdd9b700 1 -- 172.21.15.96:0/1123122830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac40095f0 con 0x7f3ad0102690 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3acdd9b700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 0x7f3ad019ce90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400bd70 tx=0x7f3ac400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:03.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3abf7fe700 1 -- 172.21.15.96:0/1123122830 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ac401d070 con 0x7f3ad0102690 2024-04-22T15:57:03.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3abf7fe700 1 -- 172.21.15.96:0/1123122830 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ac400f460 con 0x7f3ad0102690 2024-04-22T15:57:03.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad0196d10 con 0x7f3ad0102690 2024-04-22T15:57:03.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3abf7fe700 1 -- 172.21.15.96:0/1123122830 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ac4017630 con 0x7f3ad0102690 2024-04-22T15:57:03.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.509+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad0197200 con 0x7f3ad0102690 2024-04-22T15:57:03.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.510+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad0105570 con 0x7f3ad0102690 2024-04-22T15:57:03.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.511+0000 7f3abf7fe700 1 -- 172.21.15.96:0/1123122830 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3ac400f5d0 con 0x7f3ad0102690 2024-04-22T15:57:03.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.511+0000 7f3abf7fe700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ab806ecc0 0x7f3ab8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.512+0000 7f3abf7fe700 1 -- 172.21.15.96:0/1123122830 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3ac40902b0 con 0x7f3ad0102690 2024-04-22T15:57:03.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.512+0000 7f3ace59c700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ab806ecc0 0x7f3ab8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.512+0000 7f3ace59c700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ab806ecc0 0x7f3ab8071180 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0006fd0 tx=0x7f3ac0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:03.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.516+0000 7f3abf7fe700 1 -- 172.21.15.96:0/1123122830 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3ac405cbc0 con 0x7f3ad0102690 2024-04-22T15:57:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.766+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3ad0069730 con 0x7f3ad0102690 2024-04-22T15:57:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.767+0000 7f3abf7fe700 1 -- 172.21.15.96:0/1123122830 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f3ac4005740 con 0x7f3ad0102690 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:57:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:57:03.771 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:57:03.772 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:57:03.773 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:57:03.773 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:57:03.773 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:57:03.773 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:57:03.773 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ab806ecc0 msgr2=0x7f3ab8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ab806ecc0 0x7f3ab8071180 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0006fd0 tx=0x7f3ac0008040 comp rx=0 tx=0).stop 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 msgr2=0x7f3ad019ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 0x7f3ad019ce90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400bd70 tx=0x7f3ac400be50 comp rx=0 tx=0).stop 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 shutdown_connections 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3ab806ecc0 0x7f3ab8071180 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ad0101ce0 0x7f3ad019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 --2- 172.21.15.96:0/1123122830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad0102690 0x7f3ad019ce90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 >> 172.21.15.96:0/1123122830 conn(0x7f3ad00fb5e0 msgr2=0x7f3ad0103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 shutdown_connections 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.769+0000 7f3ad4ded700 1 -- 172.21.15.96:0/1123122830 wait complete. 2024-04-22T15:57:03.774 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:57:03.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.872+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2833531604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 msgr2=0x7fb4c4105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.872+0000 7fb4cbfd7700 1 --2- 172.21.15.96:0/2833531604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 0x7fb4c4105820 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009910 tx=0x7fb4b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:03.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2833531604 shutdown_connections 2024-04-22T15:57:03.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 --2- 172.21.15.96:0/2833531604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 0x7fb4c4105820 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 --2- 172.21.15.96:0/2833531604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4c4100e20 0x7fb4c4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2833531604 >> 172.21.15.96:0/2833531604 conn(0x7fb4c40fc6d0 msgr2=0x7fb4c40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:03.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2833531604 shutdown_connections 2024-04-22T15:57:03.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2833531604 wait complete. 2024-04-22T15:57:03.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 Processor -- start 2024-04-22T15:57:03.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 -- start start 2024-04-22T15:57:03.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4c4100e20 0x7fb4c419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 0x7fb4c419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c419d380 con 0x7fb4c41017d0 2024-04-22T15:57:03.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.873+0000 7fb4cbfd7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c4196960 con 0x7fb4c4100e20 2024-04-22T15:57:03.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4c9572700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 0x7fb4c419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4c9572700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 0x7fb4c419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49196/0 (socket says 172.21.15.96:49196) 2024-04-22T15:57:03.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4c9572700 1 -- 172.21.15.96:0/2793859155 learned_addr learned my addr 172.21.15.96:0/2793859155 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4c9d73700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4c4100e20 0x7fb4c419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4c9572700 1 -- 172.21.15.96:0/2793859155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4c4100e20 msgr2=0x7fb4c419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4c9572700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4c4100e20 0x7fb4c419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4c9572700 1 -- 172.21.15.96:0/2793859155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b40095f0 con 0x7fb4c41017d0 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4c9d73700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4c4100e20 0x7fb4c419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4c9572700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 0x7fb4c419cd60 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400bd70 tx=0x7fb4b400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4baffd700 1 -- 172.21.15.96:0/2793859155 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b401d070 con 0x7fb4c41017d0 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4baffd700 1 -- 172.21.15.96:0/2793859155 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4b40056e0 con 0x7fb4c41017d0 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c4196be0 con 0x7fb4c41017d0 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4baffd700 1 -- 172.21.15.96:0/2793859155 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b4017400 con 0x7fb4c41017d0 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.874+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c41970d0 con 0x7fb4c41017d0 2024-04-22T15:57:03.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.875+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c4109220 con 0x7fb4c41017d0 2024-04-22T15:57:03.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.876+0000 7fb4baffd700 1 -- 172.21.15.96:0/2793859155 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb4b400f460 con 0x7fb4c41017d0 2024-04-22T15:57:03.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.876+0000 7fb4baffd700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4b006ecc0 0x7fb4b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.876+0000 7fb4c9d73700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4b006ecc0 0x7fb4b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.876+0000 7fb4baffd700 1 -- 172.21.15.96:0/2793859155 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb4b4090230 con 0x7fb4c41017d0 2024-04-22T15:57:03.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.877+0000 7fb4c9d73700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4b006ecc0 0x7fb4b0071180 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0005950 tx=0x7fb4c00058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:03.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:03.881+0000 7fb4baffd700 1 -- 172.21.15.96:0/2793859155 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb4b405cc00 con 0x7fb4c41017d0 2024-04-22T15:57:04.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.099+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4c4197e90 con 0x7fb4b006ecc0 2024-04-22T15:57:04.102 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:03 smithi096 ceph-mon[112443]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:04.102 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:03 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4228008885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:04.102 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:03 smithi096 ceph-mon[112443]: pgmap v1057: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:04.102 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:03 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1123122830' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.100+0000 7fb4baffd700 1 -- 172.21.15.96:0/2793859155 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb4c4197e90 con 0x7fb4b006ecc0 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:57:04.103 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:57:04.104 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:57:04.104 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:57:04.104 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:57:04.104 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4b006ecc0 msgr2=0x7fb4b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4b006ecc0 0x7fb4b0071180 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0005950 tx=0x7fb4c00058e0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 msgr2=0x7fb4c419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 0x7fb4c419cd60 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400bd70 tx=0x7fb4b400be50 comp rx=0 tx=0).stop 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 shutdown_connections 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb4b006ecc0 0x7fb4b0071180 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4c4100e20 0x7fb4c419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 --2- 172.21.15.96:0/2793859155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4c41017d0 0x7fb4c419cd60 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 >> 172.21.15.96:0/2793859155 conn(0x7fb4c40fc6d0 msgr2=0x7fb4c40fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 shutdown_connections 2024-04-22T15:57:04.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.103+0000 7fb4cbfd7700 1 -- 172.21.15.96:0/2793859155 wait complete. 2024-04-22T15:57:04.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.206+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2467421369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 msgr2=0x7f0cfc0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.206+0000 7f0d02dcb700 1 --2- 172.21.15.96:0/2467421369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 0x7f0cfc0fff20 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009910 tx=0x7f0ce8009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:04.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.206+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2467421369 shutdown_connections 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.206+0000 7f0d02dcb700 1 --2- 172.21.15.96:0/2467421369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 0x7f0cfc0fff20 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.206+0000 7f0d02dcb700 1 --2- 172.21.15.96:0/2467421369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cfc105aa0 0x7f0cfc105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.206+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2467421369 >> 172.21.15.96:0/2467421369 conn(0x7f0cfc0fb5f0 msgr2=0x7f0cfc0fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2467421369 shutdown_connections 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2467421369 wait complete. 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d02dcb700 1 Processor -- start 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d02dcb700 1 -- start start 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d02dcb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 0x7f0cfc19b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d02dcb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cfc105aa0 0x7f0cfc19bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d02dcb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc19c2b0 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d02dcb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc195820 con 0x7f0cfc105aa0 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d00b67700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 0x7f0cfc19b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d00b67700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 0x7f0cfc19b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49222/0 (socket says 172.21.15.96:49222) 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.207+0000 7f0d00b67700 1 -- 172.21.15.96:0/2057507848 learned_addr learned my addr 172.21.15.96:0/2057507848 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0cfbfff700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cfc105aa0 0x7f0cfc19bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0d00b67700 1 -- 172.21.15.96:0/2057507848 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cfc105aa0 msgr2=0x7f0cfc19bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0d00b67700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cfc105aa0 0x7f0cfc19bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0d00b67700 1 -- 172.21.15.96:0/2057507848 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80095f0 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0cfbfff700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cfc105aa0 0x7f0cfc19bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0d00b67700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 0x7f0cfc19b6e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000c8a0 tx=0x7f0cf000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0cf9ffb700 1 -- 172.21.15.96:0/2057507848 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cf000cea0 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc195aa0 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0cf9ffb700 1 -- 172.21.15.96:0/2057507848 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cf0004830 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0cf9ffb700 1 -- 172.21.15.96:0/2057507848 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cf00055b0 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.208+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc195ff0 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.209+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc1011e0 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.210+0000 7f0cf9ffb700 1 -- 172.21.15.96:0/2057507848 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0cf00049a0 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.210+0000 7f0cf9ffb700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cec06ecc0 0x7f0cec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.210+0000 7f0cf9ffb700 1 -- 172.21.15.96:0/2057507848 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0cf008eb70 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.210+0000 7f0cfbfff700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cec06ecc0 0x7f0cec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.210+0000 7f0cfbfff700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cec06ecc0 0x7f0cec071180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8006010 tx=0x7f0ce800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:04.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.215+0000 7f0cf9ffb700 1 -- 172.21.15.96:0/2057507848 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0cf005b400 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:03 smithi196 ceph-mon[81815]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:04.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:03 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4228008885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:04.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:03 smithi196 ceph-mon[81815]: pgmap v1057: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:04.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:03 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1123122830' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:57:04.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.499+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0cfc04ebb0 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.500+0000 7f0cf9ffb700 1 -- 172.21.15.96:0/2057507848 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f0cf0004c50 con 0x7f0cfc0ffaa0 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:57:04.503 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:57:04.504 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:57:04.505 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:57:04.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cec06ecc0 msgr2=0x7f0cec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cec06ecc0 0x7f0cec071180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8006010 tx=0x7f0ce800b540 comp rx=0 tx=0).stop 2024-04-22T15:57:04.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 msgr2=0x7f0cfc19b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 0x7f0cfc19b6e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000c8a0 tx=0x7f0cf000cc60 comp rx=0 tx=0).stop 2024-04-22T15:57:04.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 shutdown_connections 2024-04-22T15:57:04.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0cec06ecc0 0x7f0cec071180 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cfc0ffaa0 0x7f0cfc19b6e0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 --2- 172.21.15.96:0/2057507848 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cfc105aa0 0x7f0cfc19bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 >> 172.21.15.96:0/2057507848 conn(0x7f0cfc0fb5f0 msgr2=0x7f0cfc0fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:04.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 shutdown_connections 2024-04-22T15:57:04.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:04.502+0000 7f0d02dcb700 1 -- 172.21.15.96:0/2057507848 wait complete. 2024-04-22T15:57:05.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:04 smithi096 ceph-mon[112443]: from='client.35928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:05.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:04 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2057507848' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:57:05.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:04 smithi196 ceph-mon[81815]: from='client.35928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:05.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:04 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2057507848' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:57:06.116 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:05 smithi096 ceph-mon[112443]: pgmap v1058: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:06.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:05 smithi196 ceph-mon[81815]: pgmap v1058: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:08.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:07 smithi096 ceph-mon[112443]: pgmap v1059: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:07 smithi196 ceph-mon[81815]: pgmap v1059: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:09.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:10.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:09 smithi096 ceph-mon[112443]: pgmap v1060: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:10.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:09 smithi196 ceph-mon[81815]: pgmap v1060: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:12.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:11 smithi196 ceph-mon[81815]: pgmap v1061: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:12.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:11 smithi096 ceph-mon[112443]: pgmap v1061: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:14.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:13 smithi096 ceph-mon[112443]: pgmap v1062: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:14.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:13 smithi196 ceph-mon[81815]: pgmap v1062: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:16.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:15 smithi096 ceph-mon[112443]: pgmap v1063: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:16.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:15 smithi196 ceph-mon[81815]: pgmap v1063: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:18.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:17 smithi096 ceph-mon[112443]: pgmap v1064: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:18.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:17 smithi196 ceph-mon[81815]: pgmap v1064: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:19 smithi196 ceph-mon[81815]: pgmap v1065: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:20.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:19 smithi096 ceph-mon[112443]: pgmap v1065: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:22.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:21 smithi096 ceph-mon[112443]: pgmap v1066: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:21 smithi196 ceph-mon[81815]: pgmap v1066: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:23.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:24.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:24 smithi096 ceph-mon[112443]: pgmap v1067: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:24 smithi196 ceph-mon[81815]: pgmap v1067: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:25.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:25 smithi096 ceph-mon[112443]: pgmap v1068: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:25 smithi196 ceph-mon[81815]: pgmap v1068: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:28.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:27 smithi096 ceph-mon[112443]: pgmap v1069: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:28.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:27 smithi196 ceph-mon[81815]: pgmap v1069: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:29 smithi196 ceph-mon[81815]: pgmap v1070: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:30.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:29 smithi096 ceph-mon[112443]: pgmap v1070: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:32.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:31 smithi096 ceph-mon[112443]: pgmap v1071: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:32.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:31 smithi196 ceph-mon[81815]: pgmap v1071: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:34.038 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:33 smithi096 ceph-mon[112443]: pgmap v1072: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:34.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:33 smithi196 ceph-mon[81815]: pgmap v1072: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:34.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.619+0000 7fc344960700 1 -- 172.21.15.96:0/1544873329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340101d60 msgr2=0x7fc340102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:34.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.619+0000 7fc344960700 1 --2- 172.21.15.96:0/1544873329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340101d60 0x7fc340102140 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc33800b0a0 tx=0x7fc33800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.619+0000 7fc344960700 1 -- 172.21.15.96:0/1544873329 shutdown_connections 2024-04-22T15:57:34.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.619+0000 7fc344960700 1 --2- 172.21.15.96:0/1544873329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc340102680 0x7fc34010ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.619+0000 7fc344960700 1 --2- 172.21.15.96:0/1544873329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340101d60 0x7fc340102140 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.619+0000 7fc344960700 1 -- 172.21.15.96:0/1544873329 >> 172.21.15.96:0/1544873329 conn(0x7fc340076740 msgr2=0x7fc3400fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:34.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.619+0000 7fc344960700 1 -- 172.21.15.96:0/1544873329 shutdown_connections 2024-04-22T15:57:34.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.619+0000 7fc344960700 1 -- 172.21.15.96:0/1544873329 wait complete. 2024-04-22T15:57:34.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc344960700 1 Processor -- start 2024-04-22T15:57:34.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc344960700 1 -- start start 2024-04-22T15:57:34.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc344960700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc340101d60 0x7fc340198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:34.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc344960700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340102680 0x7fc3401993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc344960700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc340199ac0 con 0x7fc340102680 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc344960700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34019d790 con 0x7fc340101d60 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc33dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340102680 0x7fc3401993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc33dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340102680 0x7fc3401993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35334/0 (socket says 172.21.15.96:35334) 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc33dd9b700 1 -- 172.21.15.96:0/2543329067 learned_addr learned my addr 172.21.15.96:0/2543329067 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.620+0000 7fc33e59c700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc340101d60 0x7fc340198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.621+0000 7fc33dd9b700 1 -- 172.21.15.96:0/2543329067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc340101d60 msgr2=0x7fc340198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.621+0000 7fc33dd9b700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc340101d60 0x7fc340198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.621+0000 7fc33dd9b700 1 -- 172.21.15.96:0/2543329067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc338009d00 con 0x7fc340102680 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.621+0000 7fc33dd9b700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340102680 0x7fc3401993e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc32800b3d0 tx=0x7fc32800b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.621+0000 7fc3377fe700 1 -- 172.21.15.96:0/2543329067 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc328011800 con 0x7fc340102680 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.621+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34019da70 con 0x7fc340102680 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.621+0000 7fc3377fe700 1 -- 172.21.15.96:0/2543329067 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc32800be90 con 0x7fc340102680 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.621+0000 7fc3377fe700 1 -- 172.21.15.96:0/2543329067 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc328010e50 con 0x7fc340102680 2024-04-22T15:57:34.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.621+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34019dfc0 con 0x7fc340102680 2024-04-22T15:57:34.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.622+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34010a6d0 con 0x7fc340102680 2024-04-22T15:57:34.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.623+0000 7fc3377fe700 1 -- 172.21.15.96:0/2543329067 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc32801a400 con 0x7fc340102680 2024-04-22T15:57:34.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.623+0000 7fc3377fe700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32c06ec70 0x7fc32c071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:34.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.623+0000 7fc3377fe700 1 -- 172.21.15.96:0/2543329067 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc32808e910 con 0x7fc340102680 2024-04-22T15:57:34.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.623+0000 7fc33e59c700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32c06ec70 0x7fc32c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:34.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.624+0000 7fc33e59c700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32c06ec70 0x7fc32c071130 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc338000f80 tx=0x7fc3380097e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:34.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.628+0000 7fc3377fe700 1 -- 172.21.15.96:0/2543329067 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc328057670 con 0x7fc340102680 2024-04-22T15:57:34.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.846+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc34010de60 con 0x7fc32c06ec70 2024-04-22T15:57:34.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.847+0000 7fc3377fe700 1 -- 172.21.15.96:0/2543329067 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc34010de60 con 0x7fc32c06ec70 2024-04-22T15:57:34.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32c06ec70 msgr2=0x7fc32c071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:34.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32c06ec70 0x7fc32c071130 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc338000f80 tx=0x7fc3380097e0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340102680 msgr2=0x7fc3401993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:34.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340102680 0x7fc3401993e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc32800b3d0 tx=0x7fc32800b6e0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 shutdown_connections 2024-04-22T15:57:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32c06ec70 0x7fc32c071130 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc340101d60 0x7fc340198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 --2- 172.21.15.96:0/2543329067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc340102680 0x7fc3401993e0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 >> 172.21.15.96:0/2543329067 conn(0x7fc340076740 msgr2=0x7fc340101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 shutdown_connections 2024-04-22T15:57:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.849+0000 7fc344960700 1 -- 172.21.15.96:0/2543329067 wait complete. 2024-04-22T15:57:34.862 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:57:34.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.955+0000 7f047fde2700 1 -- 172.21.15.96:0/371625004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04780ffac0 msgr2=0x7f04780fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:34.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.955+0000 7f047fde2700 1 --2- 172.21.15.96:0/371625004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04780ffac0 0x7f04780fff40 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f0468009910 tx=0x7f0468009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:34.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.955+0000 7f047fde2700 1 -- 172.21.15.96:0/371625004 shutdown_connections 2024-04-22T15:57:34.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.955+0000 7f047fde2700 1 --2- 172.21.15.96:0/371625004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04780ffac0 0x7f04780fff40 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.955+0000 7f047fde2700 1 --2- 172.21.15.96:0/371625004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0478105ac0 0x7f0478105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.955+0000 7f047fde2700 1 -- 172.21.15.96:0/371625004 >> 172.21.15.96:0/371625004 conn(0x7f04780fb610 msgr2=0x7f04780fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:34.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.955+0000 7f047fde2700 1 -- 172.21.15.96:0/371625004 shutdown_connections 2024-04-22T15:57:34.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.955+0000 7f047fde2700 1 -- 172.21.15.96:0/371625004 wait complete. 2024-04-22T15:57:34.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.955+0000 7f047fde2700 1 Processor -- start 2024-04-22T15:57:34.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047fde2700 1 -- start start 2024-04-22T15:57:34.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047fde2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04780ffac0 0x7f04781979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:34.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047fde2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0478105ac0 0x7f0478197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:34.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047fde2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0478198440 con 0x7f0478105ac0 2024-04-22T15:57:34.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047fde2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0478198580 con 0x7f04780ffac0 2024-04-22T15:57:34.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047d37d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0478105ac0 0x7f0478197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:34.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047d37d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0478105ac0 0x7f0478197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35362/0 (socket says 172.21.15.96:35362) 2024-04-22T15:57:34.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047d37d700 1 -- 172.21.15.96:0/2849919740 learned_addr learned my addr 172.21.15.96:0/2849919740 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047db7e700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04780ffac0 0x7f04781979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047d37d700 1 -- 172.21.15.96:0/2849919740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04780ffac0 msgr2=0x7f04781979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047d37d700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04780ffac0 0x7f04781979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047d37d700 1 -- 172.21.15.96:0/2849919740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680095f0 con 0x7f0478105ac0 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.956+0000 7f047d37d700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0478105ac0 0x7f0478197f00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f046800bd70 tx=0x7f046800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.957+0000 7f046effd700 1 -- 172.21.15.96:0/2849919740 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f046801d070 con 0x7f0478105ac0 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.957+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0478198800 con 0x7f0478105ac0 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.957+0000 7f046effd700 1 -- 172.21.15.96:0/2849919740 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f046800f460 con 0x7f0478105ac0 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.957+0000 7f046effd700 1 -- 172.21.15.96:0/2849919740 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0468005dc0 con 0x7f0478105ac0 2024-04-22T15:57:34.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.957+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04781a0d60 con 0x7f0478105ac0 2024-04-22T15:57:34.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.958+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04781080e0 con 0x7f0478105ac0 2024-04-22T15:57:34.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.958+0000 7f046effd700 1 -- 172.21.15.96:0/2849919740 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0468003a60 con 0x7f0478105ac0 2024-04-22T15:57:34.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.959+0000 7f046effd700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f046406ead0 0x7f0464070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:34.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.959+0000 7f046effd700 1 -- 172.21.15.96:0/2849919740 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f04680903a0 con 0x7f0478105ac0 2024-04-22T15:57:34.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.959+0000 7f047db7e700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f046406ead0 0x7f0464070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:34.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.959+0000 7f047db7e700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f046406ead0 0x7f0464070f90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0474005fd0 tx=0x7f047400a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:34.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:34.965+0000 7f046effd700 1 -- 172.21.15.96:0/2849919740 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f046805ccb0 con 0x7f0478105ac0 2024-04-22T15:57:35.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.184+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f04780696e0 con 0x7f046406ead0 2024-04-22T15:57:35.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.185+0000 7f046effd700 1 -- 172.21.15.96:0/2849919740 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f04780696e0 con 0x7f046406ead0 2024-04-22T15:57:35.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f046406ead0 msgr2=0x7f0464070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f046406ead0 0x7f0464070f90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0474005fd0 tx=0x7f047400a560 comp rx=0 tx=0).stop 2024-04-22T15:57:35.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0478105ac0 msgr2=0x7f0478197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0478105ac0 0x7f0478197f00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f046800bd70 tx=0x7f046800be50 comp rx=0 tx=0).stop 2024-04-22T15:57:35.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 shutdown_connections 2024-04-22T15:57:35.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f046406ead0 0x7f0464070f90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04780ffac0 0x7f04781979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 --2- 172.21.15.96:0/2849919740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0478105ac0 0x7f0478197f00 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 >> 172.21.15.96:0/2849919740 conn(0x7f04780fb610 msgr2=0x7f04780fc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:35.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 shutdown_connections 2024-04-22T15:57:35.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.187+0000 7f047fde2700 1 -- 172.21.15.96:0/2849919740 wait complete. 2024-04-22T15:57:35.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.292+0000 7f4711c28700 1 -- 172.21.15.96:0/1225499579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c0ffb50 msgr2=0x7f470c0fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.292+0000 7f4711c28700 1 --2- 172.21.15.96:0/1225499579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c0ffb50 0x7f470c0fffd0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f4700009910 tx=0x7f4700009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:35.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.292+0000 7f4711c28700 1 -- 172.21.15.96:0/1225499579 shutdown_connections 2024-04-22T15:57:35.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.292+0000 7f4711c28700 1 --2- 172.21.15.96:0/1225499579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c0ffb50 0x7f470c0fffd0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.292+0000 7f4711c28700 1 --2- 172.21.15.96:0/1225499579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f470c105c00 0x7f470c105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.292+0000 7f4711c28700 1 -- 172.21.15.96:0/1225499579 >> 172.21.15.96:0/1225499579 conn(0x7f470c0fb680 msgr2=0x7f470c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:35.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.292+0000 7f4711c28700 1 -- 172.21.15.96:0/1225499579 shutdown_connections 2024-04-22T15:57:35.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.292+0000 7f4711c28700 1 -- 172.21.15.96:0/1225499579 wait complete. 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f4711c28700 1 Processor -- start 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f4711c28700 1 -- start start 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f4711c28700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f470c0ffb50 0x7f470c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f4711c28700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c105c00 0x7f470c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f4711c28700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c19d440 con 0x7f470c105c00 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f4711c28700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c196a20 con 0x7f470c0ffb50 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f470affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c105c00 0x7f470c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f470affd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c105c00 0x7f470c19ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35386/0 (socket says 172.21.15.96:35386) 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f470affd700 1 -- 172.21.15.96:0/1945819350 learned_addr learned my addr 172.21.15.96:0/1945819350 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.293+0000 7f470b7fe700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f470c0ffb50 0x7f470c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:35.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.294+0000 7f470affd700 1 -- 172.21.15.96:0/1945819350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f470c0ffb50 msgr2=0x7f470c19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.294+0000 7f470affd700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f470c0ffb50 0x7f470c19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.294+0000 7f470affd700 1 -- 172.21.15.96:0/1945819350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47000095f0 con 0x7f470c105c00 2024-04-22T15:57:35.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.294+0000 7f470affd700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c105c00 0x7f470c19ce20 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f470000bd70 tx=0x7f470000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:35.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.294+0000 7f4708ff9700 1 -- 172.21.15.96:0/1945819350 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f470001d070 con 0x7f470c105c00 2024-04-22T15:57:35.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.294+0000 7f4708ff9700 1 -- 172.21.15.96:0/1945819350 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4700005700 con 0x7f470c105c00 2024-04-22T15:57:35.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.294+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470c196ca0 con 0x7f470c105c00 2024-04-22T15:57:35.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.294+0000 7f4708ff9700 1 -- 172.21.15.96:0/1945819350 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4700017400 con 0x7f470c105c00 2024-04-22T15:57:35.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.294+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470c197210 con 0x7f470c105c00 2024-04-22T15:57:35.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.295+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470c101200 con 0x7f470c105c00 2024-04-22T15:57:35.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.295+0000 7f4708ff9700 1 -- 172.21.15.96:0/1945819350 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4700003a60 con 0x7f470c105c00 2024-04-22T15:57:35.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.296+0000 7f4708ff9700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f46f406ecc0 0x7f46f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:35.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.296+0000 7f4708ff9700 1 -- 172.21.15.96:0/1945819350 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4700090250 con 0x7f470c105c00 2024-04-22T15:57:35.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.296+0000 7f470b7fe700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f46f406ecc0 0x7f46f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:35.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.296+0000 7f470b7fe700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f46f406ecc0 0x7f46f4071180 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009a10 tx=0x7f46fc009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:35.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.300+0000 7f4708ff9700 1 -- 172.21.15.96:0/1945819350 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f470005cb60 con 0x7f470c105c00 2024-04-22T15:57:35.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.510+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f470c10cdc0 con 0x7f46f406ecc0 2024-04-22T15:57:35.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.521+0000 7f4708ff9700 1 -- 172.21.15.96:0/1945819350 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f470c10cdc0 con 0x7f46f406ecc0 2024-04-22T15:57:35.523 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:57:35.523 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (36m) 13m ago 47m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (47m) 13m ago 47m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (46m) 13m ago 46m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (33m) 13m ago 47m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (33m) 13m ago 46m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (35m) 13m ago 47m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (29m) 13m ago 41m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (29m) 13m ago 41m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (29m) 13m ago 41m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (13m) 13m ago 41m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (38m) 13m ago 49m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (37m) 13m ago 46m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (34m) 13m ago 49m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (34m) 13m ago 46m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (37m) 13m ago 47m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:57:35.524 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (37m) 13m ago 46m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:57:35.525 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (33m) 13m ago 45m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:57:35.525 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (32m) 13m ago 45m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:57:35.525 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (31m) 13m ago 44m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:57:35.525 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (31m) 13m ago 44m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:57:35.525 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (30m) 13m ago 43m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:57:35.525 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (30m) 13m ago 43m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:57:35.525 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (36m) 13m ago 47m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f46f406ecc0 msgr2=0x7f46f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f46f406ecc0 0x7f46f4071180 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009a10 tx=0x7f46fc009190 comp rx=0 tx=0).stop 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c105c00 msgr2=0x7f470c19ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c105c00 0x7f470c19ce20 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f470000bd70 tx=0x7f470000bda0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 shutdown_connections 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f46f406ecc0 0x7f46f4071180 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f470c0ffb50 0x7f470c19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 --2- 172.21.15.96:0/1945819350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c105c00 0x7f470c19ce20 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 >> 172.21.15.96:0/1945819350 conn(0x7f470c0fb680 msgr2=0x7f470c069430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 shutdown_connections 2024-04-22T15:57:35.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.523+0000 7f4711c28700 1 -- 172.21.15.96:0/1945819350 wait complete. 2024-04-22T15:57:35.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.624+0000 7f39bac73700 1 -- 172.21.15.96:0/3000776861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 msgr2=0x7f39b4105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.624+0000 7f39bac73700 1 --2- 172.21.15.96:0/3000776861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 0x7f39b4105f40 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009910 tx=0x7f39a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:35.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.624+0000 7f39bac73700 1 -- 172.21.15.96:0/3000776861 shutdown_connections 2024-04-22T15:57:35.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.624+0000 7f39bac73700 1 --2- 172.21.15.96:0/3000776861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b40ffad0 0x7f39b40fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.624+0000 7f39bac73700 1 --2- 172.21.15.96:0/3000776861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 0x7f39b4105f40 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.624+0000 7f39bac73700 1 -- 172.21.15.96:0/3000776861 >> 172.21.15.96:0/3000776861 conn(0x7f39b40fb600 msgr2=0x7f39b40fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:35.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.624+0000 7f39bac73700 1 -- 172.21.15.96:0/3000776861 shutdown_connections 2024-04-22T15:57:35.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.624+0000 7f39bac73700 1 -- 172.21.15.96:0/3000776861 wait complete. 2024-04-22T15:57:35.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39bac73700 1 Processor -- start 2024-04-22T15:57:35.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39bac73700 1 -- start start 2024-04-22T15:57:35.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39bac73700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b40ffad0 0x7f39b40fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:35.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39bac73700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 0x7f39b40fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:35.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39bac73700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b40fc810 con 0x7f39b4105b60 2024-04-22T15:57:35.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39bac73700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b40fc950 con 0x7f39b40ffad0 2024-04-22T15:57:35.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39b3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 0x7f39b40fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:35.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39b3fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 0x7f39b40fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35410/0 (socket says 172.21.15.96:35410) 2024-04-22T15:57:35.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39b3fff700 1 -- 172.21.15.96:0/135705986 learned_addr learned my addr 172.21.15.96:0/135705986 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.625+0000 7f39b8a0f700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b40ffad0 0x7f39b40fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.626+0000 7f39b3fff700 1 -- 172.21.15.96:0/135705986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b40ffad0 msgr2=0x7f39b40fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.626+0000 7f39b3fff700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b40ffad0 0x7f39b40fbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.626+0000 7f39b3fff700 1 -- 172.21.15.96:0/135705986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a80095f0 con 0x7f39b4105b60 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.626+0000 7f39b3fff700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 0x7f39b40fc2d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f39a000d8d0 tx=0x7f39a000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.626+0000 7f39b1ffb700 1 -- 172.21.15.96:0/135705986 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a00098a0 con 0x7f39b4105b60 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.626+0000 7f39b1ffb700 1 -- 172.21.15.96:0/135705986 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39a0010460 con 0x7f39b4105b60 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.626+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b41a2c90 con 0x7f39b4105b60 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.626+0000 7f39b1ffb700 1 -- 172.21.15.96:0/135705986 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a000f5f0 con 0x7f39b4105b60 2024-04-22T15:57:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.626+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b41a2ff0 con 0x7f39b4105b60 2024-04-22T15:57:35.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.628+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b41090e0 con 0x7f39b4105b60 2024-04-22T15:57:35.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.628+0000 7f39b1ffb700 1 -- 172.21.15.96:0/135705986 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f39a0009a00 con 0x7f39b4105b60 2024-04-22T15:57:35.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.628+0000 7f39b1ffb700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39a406ecd0 0x7f39a4071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:35.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.628+0000 7f39b1ffb700 1 -- 172.21.15.96:0/135705986 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f39a008ed80 con 0x7f39b4105b60 2024-04-22T15:57:35.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.629+0000 7f39b8a0f700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39a406ecd0 0x7f39a4071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:35.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.629+0000 7f39b8a0f700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39a406ecd0 0x7f39a4071190 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f39a8006010 tx=0x7f39a801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:35.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.633+0000 7f39b1ffb700 1 -- 172.21.15.96:0/135705986 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f39a0057ae0 con 0x7f39b4105b60 2024-04-22T15:57:35.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.923+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f39b40fd270 con 0x7f39b4105b60 2024-04-22T15:57:35.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:35 smithi096 ceph-mon[112443]: from='client.35936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:35.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:35 smithi096 ceph-mon[112443]: from='client.35940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:35.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:35 smithi096 ceph-mon[112443]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:35.926 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:35 smithi096 ceph-mon[112443]: pgmap v1073: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:35.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.923+0000 7f39b1ffb700 1 -- 172.21.15.96:0/135705986 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f39a005b1a0 con 0x7f39b4105b60 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:57:35.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:57:35.928 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:57:35.928 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:57:35.928 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:57:35.928 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:57:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39a406ecd0 msgr2=0x7f39a4071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39a406ecd0 0x7f39a4071190 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f39a8006010 tx=0x7f39a801a040 comp rx=0 tx=0).stop 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 msgr2=0x7f39b40fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 0x7f39b40fc2d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f39a000d8d0 tx=0x7f39a000dbe0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 shutdown_connections 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f39a406ecd0 0x7f39a4071190 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f39b40ffad0 0x7f39b40fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 --2- 172.21.15.96:0/135705986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39b4105b60 0x7f39b40fc2d0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 >> 172.21.15.96:0/135705986 conn(0x7f39b40fb600 msgr2=0x7f39b4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 shutdown_connections 2024-04-22T15:57:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:35.926+0000 7f39bac73700 1 -- 172.21.15.96:0/135705986 wait complete. 2024-04-22T15:57:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.029+0000 7f19652b9700 1 -- 172.21.15.96:0/4103751051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1960105aa0 msgr2=0x7f1960105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.029+0000 7f19652b9700 1 --2- 172.21.15.96:0/4103751051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1960105aa0 0x7f1960105e80 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f1950009910 tx=0x7f1950009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.029+0000 7f19652b9700 1 -- 172.21.15.96:0/4103751051 shutdown_connections 2024-04-22T15:57:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.029+0000 7f19652b9700 1 --2- 172.21.15.96:0/4103751051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19600ffaa0 0x7f19600fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.029+0000 7f19652b9700 1 --2- 172.21.15.96:0/4103751051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1960105aa0 0x7f1960105e80 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.029+0000 7f19652b9700 1 -- 172.21.15.96:0/4103751051 >> 172.21.15.96:0/4103751051 conn(0x7f1960076740 msgr2=0x7f19600fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.029+0000 7f19652b9700 1 -- 172.21.15.96:0/4103751051 shutdown_connections 2024-04-22T15:57:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.029+0000 7f19652b9700 1 -- 172.21.15.96:0/4103751051 wait complete. 2024-04-22T15:57:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f19652b9700 1 Processor -- start 2024-04-22T15:57:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f19652b9700 1 -- start start 2024-04-22T15:57:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f19652b9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19600ffaa0 0x7f19600fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f19652b9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1960105aa0 0x7f19600fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f19652b9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19600fc6a0 con 0x7f19600ffaa0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f19652b9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19600fc7e0 con 0x7f1960105aa0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f195effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19600ffaa0 0x7f19600fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f195effd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19600ffaa0 0x7f19600fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35422/0 (socket says 172.21.15.96:35422) 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f195effd700 1 -- 172.21.15.96:0/2215767974 learned_addr learned my addr 172.21.15.96:0/2215767974 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f195e7fc700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1960105aa0 0x7f19600fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f195effd700 1 -- 172.21.15.96:0/2215767974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1960105aa0 msgr2=0x7f19600fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f195effd700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1960105aa0 0x7f19600fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f195effd700 1 -- 172.21.15.96:0/2215767974 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500095f0 con 0x7f19600ffaa0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.030+0000 7f195e7fc700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1960105aa0 0x7f19600fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.031+0000 7f195effd700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19600ffaa0 0x7f19600fbc20 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1950006010 tx=0x7f195000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.031+0000 7f1947fff700 1 -- 172.21.15.96:0/2215767974 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f195001d070 con 0x7f19600ffaa0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.031+0000 7f1947fff700 1 -- 172.21.15.96:0/2215767974 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f195000f460 con 0x7f19600ffaa0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.031+0000 7f1947fff700 1 -- 172.21.15.96:0/2215767974 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1950017630 con 0x7f19600ffaa0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.031+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19600fca60 con 0x7f19600ffaa0 2024-04-22T15:57:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.031+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19601a2fd0 con 0x7f19600ffaa0 2024-04-22T15:57:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.032+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19601011e0 con 0x7f19600ffaa0 2024-04-22T15:57:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.032+0000 7f1947fff700 1 -- 172.21.15.96:0/2215767974 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f195000fab0 con 0x7f19600ffaa0 2024-04-22T15:57:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.032+0000 7f1947fff700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f194806ecc0 0x7f1948071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.033+0000 7f1947fff700 1 -- 172.21.15.96:0/2215767974 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1950090280 con 0x7f19600ffaa0 2024-04-22T15:57:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.033+0000 7f195e7fc700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f194806ecc0 0x7f1948071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.033+0000 7f195e7fc700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f194806ecc0 0x7f1948071180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f19600fd350 tx=0x7f1954008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.037+0000 7f1947fff700 1 -- 172.21.15.96:0/2215767974 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1950059060 con 0x7f19600ffaa0 2024-04-22T15:57:36.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:35 smithi196 ceph-mon[81815]: from='client.35936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:36.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:35 smithi196 ceph-mon[81815]: from='client.35940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:36.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:35 smithi196 ceph-mon[81815]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:36.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:35 smithi196 ceph-mon[81815]: pgmap v1073: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:36.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.280+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f196010cde0 con 0x7f19600ffaa0 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.281+0000 7f1947fff700 1 -- 172.21.15.96:0/2215767974 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f195005c720 con 0x7f19600ffaa0 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:57:36.284 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:57:36.285 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:57:36.286 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:57:36.287 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:57:36.287 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:57:36.287 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:57:36.287 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:57:36.287 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:57:36.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f194806ecc0 msgr2=0x7f1948071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f194806ecc0 0x7f1948071180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f19600fd350 tx=0x7f1954008040 comp rx=0 tx=0).stop 2024-04-22T15:57:36.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19600ffaa0 msgr2=0x7f19600fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19600ffaa0 0x7f19600fbc20 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1950006010 tx=0x7f195000be20 comp rx=0 tx=0).stop 2024-04-22T15:57:36.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 shutdown_connections 2024-04-22T15:57:36.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f194806ecc0 0x7f1948071180 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19600ffaa0 0x7f19600fbc20 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 --2- 172.21.15.96:0/2215767974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1960105aa0 0x7f19600fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 >> 172.21.15.96:0/2215767974 conn(0x7f1960076740 msgr2=0x7f19600fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:36.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 shutdown_connections 2024-04-22T15:57:36.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.284+0000 7f19652b9700 1 -- 172.21.15.96:0/2215767974 wait complete. 2024-04-22T15:57:36.289 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:57:36.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.388+0000 7fbdfe759700 1 -- 172.21.15.96:0/3133221043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf8101db0 msgr2=0x7fbdf8102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.388+0000 7fbdfe759700 1 --2- 172.21.15.96:0/3133221043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf8101db0 0x7fbdf8102190 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009910 tx=0x7fbde8009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:36.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.388+0000 7fbdfe759700 1 -- 172.21.15.96:0/3133221043 shutdown_connections 2024-04-22T15:57:36.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.388+0000 7fbdfe759700 1 --2- 172.21.15.96:0/3133221043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdf81026d0 0x7fbdf810abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.388+0000 7fbdfe759700 1 --2- 172.21.15.96:0/3133221043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf8101db0 0x7fbdf8102190 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.388+0000 7fbdfe759700 1 -- 172.21.15.96:0/3133221043 >> 172.21.15.96:0/3133221043 conn(0x7fbdf80fb5e0 msgr2=0x7fbdf80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:36.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.388+0000 7fbdfe759700 1 -- 172.21.15.96:0/3133221043 shutdown_connections 2024-04-22T15:57:36.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.388+0000 7fbdfe759700 1 -- 172.21.15.96:0/3133221043 wait complete. 2024-04-22T15:57:36.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdfe759700 1 Processor -- start 2024-04-22T15:57:36.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdfe759700 1 -- start start 2024-04-22T15:57:36.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdfe759700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdf8101db0 0x7fbdf819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdfe759700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf81026d0 0x7fbdf819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdfe759700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf81969d0 con 0x7fbdf81026d0 2024-04-22T15:57:36.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdfe759700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf8196b40 con 0x7fbdf8101db0 2024-04-22T15:57:36.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdf77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf81026d0 0x7fbdf819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdf77fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf81026d0 0x7fbdf819ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35434/0 (socket says 172.21.15.96:35434) 2024-04-22T15:57:36.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdf77fe700 1 -- 172.21.15.96:0/939431963 learned_addr learned my addr 172.21.15.96:0/939431963 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:36.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.389+0000 7fbdf7fff700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdf8101db0 0x7fbdf819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.390+0000 7fbdf77fe700 1 -- 172.21.15.96:0/939431963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdf8101db0 msgr2=0x7fbdf819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.390+0000 7fbdf77fe700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdf8101db0 0x7fbdf819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.390+0000 7fbdf77fe700 1 -- 172.21.15.96:0/939431963 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde80095f0 con 0x7fbdf81026d0 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.390+0000 7fbdf77fe700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf81026d0 0x7fbdf819ce60 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fbdec00e8f0 tx=0x7fbdec00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.390+0000 7fbdf57fa700 1 -- 172.21.15.96:0/939431963 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdec009790 con 0x7fbdf81026d0 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.390+0000 7fbdf57fa700 1 -- 172.21.15.96:0/939431963 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdec00c810 con 0x7fbdf81026d0 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.390+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf8196e20 con 0x7fbdf81026d0 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.390+0000 7fbdf57fa700 1 -- 172.21.15.96:0/939431963 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdec003d20 con 0x7fbdf81026d0 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.390+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf8197370 con 0x7fbdf81026d0 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.391+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf80699d0 con 0x7fbdf81026d0 2024-04-22T15:57:36.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.391+0000 7fbdf57fa700 1 -- 172.21.15.96:0/939431963 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbdec0041f0 con 0x7fbdf81026d0 2024-04-22T15:57:36.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.392+0000 7fbdf57fa700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbde006ecc0 0x7fbde0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.392+0000 7fbdf57fa700 1 -- 172.21.15.96:0/939431963 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbdec014070 con 0x7fbdf81026d0 2024-04-22T15:57:36.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.392+0000 7fbdf7fff700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbde006ecc0 0x7fbde0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.392+0000 7fbdf7fff700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbde006ecc0 0x7fbde0071180 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbde80059d0 tx=0x7fbde8005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.397+0000 7fbdf57fa700 1 -- 172.21.15.96:0/939431963 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbdec057f20 con 0x7fbdf81026d0 2024-04-22T15:57:36.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.614+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdf810ddf0 con 0x7fbde006ecc0 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.615+0000 7fbdf57fa700 1 -- 172.21.15.96:0/939431963 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbdf810ddf0 con 0x7fbde006ecc0 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:57:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:57:36.619 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:57:36.619 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:57:36.619 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:57:36.619 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:57:36.619 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:57:36.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbde006ecc0 msgr2=0x7fbde0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbde006ecc0 0x7fbde0071180 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbde80059d0 tx=0x7fbde8005940 comp rx=0 tx=0).stop 2024-04-22T15:57:36.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf81026d0 msgr2=0x7fbdf819ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf81026d0 0x7fbdf819ce60 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fbdec00e8f0 tx=0x7fbdec00ec00 comp rx=0 tx=0).stop 2024-04-22T15:57:36.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 shutdown_connections 2024-04-22T15:57:36.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbde006ecc0 0x7fbde0071180 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdf8101db0 0x7fbdf819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 --2- 172.21.15.96:0/939431963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf81026d0 0x7fbdf819ce60 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 >> 172.21.15.96:0/939431963 conn(0x7fbdf80fb5e0 msgr2=0x7fbdf8103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:36.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 shutdown_connections 2024-04-22T15:57:36.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.617+0000 7fbdfe759700 1 -- 172.21.15.96:0/939431963 wait complete. 2024-04-22T15:57:36.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 -- 172.21.15.96:0/1523050733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 msgr2=0x7fd1ec105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 --2- 172.21.15.96:0/1523050733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 0x7fd1ec105e80 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00b0a0 tx=0x7fd1dc00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 -- 172.21.15.96:0/1523050733 shutdown_connections 2024-04-22T15:57:36.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 --2- 172.21.15.96:0/1523050733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1ec0ffaa0 0x7fd1ec0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 --2- 172.21.15.96:0/1523050733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 0x7fd1ec105e80 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 -- 172.21.15.96:0/1523050733 >> 172.21.15.96:0/1523050733 conn(0x7fd1ec076740 msgr2=0x7fd1ec0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 -- 172.21.15.96:0/1523050733 shutdown_connections 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 -- 172.21.15.96:0/1523050733 wait complete. 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 Processor -- start 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.729+0000 7fd1f18fe700 1 -- start start 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1f18fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1ec0ffaa0 0x7fd1ec0fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1f18fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 0x7fd1ec0fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1f18fe700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec0fc6a0 con 0x7fd1ec105aa0 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1f18fe700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec0fc810 con 0x7fd1ec0ffaa0 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1e27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 0x7fd1ec0fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1e27fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 0x7fd1ec0fc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35450/0 (socket says 172.21.15.96:35450) 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1e27fc700 1 -- 172.21.15.96:0/2316367799 learned_addr learned my addr 172.21.15.96:0/2316367799 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1eaffd700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1ec0ffaa0 0x7fd1ec0fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1e27fc700 1 -- 172.21.15.96:0/2316367799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1ec0ffaa0 msgr2=0x7fd1ec0fbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1e27fc700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1ec0ffaa0 0x7fd1ec0fbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.730+0000 7fd1e27fc700 1 -- 172.21.15.96:0/2316367799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc009d00 con 0x7fd1ec105aa0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.731+0000 7fd1e27fc700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 0x7fd1ec0fc160 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400d8d0 tx=0x7fd1d400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.731+0000 7fd1f08fc700 1 -- 172.21.15.96:0/2316367799 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d40098a0 con 0x7fd1ec105aa0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.731+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ec1a2d60 con 0x7fd1ec105aa0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.731+0000 7fd1f08fc700 1 -- 172.21.15.96:0/2316367799 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1d4010460 con 0x7fd1ec105aa0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.731+0000 7fd1f08fc700 1 -- 172.21.15.96:0/2316367799 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d400f5f0 con 0x7fd1ec105aa0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.731+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec1a30c0 con 0x7fd1ec105aa0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.732+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ec1011e0 con 0x7fd1ec105aa0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.732+0000 7fd1f08fc700 1 -- 172.21.15.96:0/2316367799 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd1d4009a00 con 0x7fd1ec105aa0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.732+0000 7fd1f08fc700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd1d806ecc0 0x7fd1d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.733+0000 7fd1f08fc700 1 -- 172.21.15.96:0/2316367799 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd1d408ed60 con 0x7fd1ec105aa0 2024-04-22T15:57:36.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.733+0000 7fd1eaffd700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd1d806ecc0 0x7fd1d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.733+0000 7fd1eaffd700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd1d806ecc0 0x7fd1d8071180 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc000f80 tx=0x7fd1dc0060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:36.737+0000 7fd1f08fc700 1 -- 172.21.15.96:0/2316367799 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd1d4057ac0 con 0x7fd1ec105aa0 2024-04-22T15:57:36.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:36 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/135705986' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:36.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:36 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2215767974' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:57:36.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:36 smithi096 ceph-mon[112443]: from='client.35956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:37.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.025+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd1ec0fd0b0 con 0x7fd1ec105aa0 2024-04-22T15:57:37.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.026+0000 7fd1f08fc700 1 -- 172.21.15.96:0/2316367799 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fd1d405b180 con 0x7fd1ec105aa0 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:57:37.029 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:57:37.030 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:57:37.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd1d806ecc0 msgr2=0x7fd1d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd1d806ecc0 0x7fd1d8071180 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc000f80 tx=0x7fd1dc0060c0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 msgr2=0x7fd1ec0fc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 0x7fd1ec0fc160 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400d8d0 tx=0x7fd1d400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 shutdown_connections 2024-04-22T15:57:37.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd1d806ecc0 0x7fd1d8071180 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1ec0ffaa0 0x7fd1ec0fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 --2- 172.21.15.96:0/2316367799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1ec105aa0 0x7fd1ec0fc160 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 >> 172.21.15.96:0/2316367799 conn(0x7fd1ec076740 msgr2=0x7fd1ec068860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:37.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 shutdown_connections 2024-04-22T15:57:37.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:57:37.028+0000 7fd1f18fe700 1 -- 172.21.15.96:0/2316367799 wait complete. 2024-04-22T15:57:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:36 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/135705986' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:36 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2215767974' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:57:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:36 smithi196 ceph-mon[81815]: from='client.35956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:38.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:37 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2316367799' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:57:38.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:37 smithi096 ceph-mon[112443]: pgmap v1074: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:38.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:37 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2316367799' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:57:38.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:37 smithi196 ceph-mon[81815]: pgmap v1074: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:39.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:39.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:40.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:39 smithi096 ceph-mon[112443]: pgmap v1075: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:40.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:39 smithi196 ceph-mon[81815]: pgmap v1075: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:42.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:41 smithi196 ceph-mon[81815]: pgmap v1076: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:42.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:41 smithi096 ceph-mon[112443]: pgmap v1076: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:44.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:43 smithi096 ceph-mon[112443]: pgmap v1077: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:44.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:43 smithi196 ceph-mon[81815]: pgmap v1077: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:46.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:45 smithi096 ceph-mon[112443]: pgmap v1078: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:45 smithi196 ceph-mon[81815]: pgmap v1078: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:48.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:47 smithi096 ceph-mon[112443]: pgmap v1079: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:48.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:47 smithi196 ceph-mon[81815]: pgmap v1079: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:50.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:49 smithi096 ceph-mon[112443]: pgmap v1080: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:50.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:49 smithi196 ceph-mon[81815]: pgmap v1080: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:52.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:51 smithi096 ceph-mon[112443]: pgmap v1081: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:51 smithi196 ceph-mon[81815]: pgmap v1081: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:54.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:54 smithi096 ceph-mon[112443]: pgmap v1082: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:54.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:54 smithi196 ceph-mon[81815]: pgmap v1082: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:56.108 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:55 smithi096 ceph-mon[112443]: pgmap v1083: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:55 smithi196 ceph-mon[81815]: pgmap v1083: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:58.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:57 smithi096 ceph-mon[112443]: pgmap v1084: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:57:58.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:57 smithi196 ceph-mon[81815]: pgmap v1084: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:00.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:57:59 smithi096 ceph-mon[112443]: pgmap v1085: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:00.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:57:59 smithi196 ceph-mon[81815]: pgmap v1085: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:02.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:01 smithi096 ceph-mon[112443]: pgmap v1086: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:02.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:01 smithi196 ceph-mon[81815]: pgmap v1086: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:04.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:03 smithi096 ceph-mon[112443]: pgmap v1087: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:04.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:03 smithi196 ceph-mon[81815]: pgmap v1087: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:06.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:05 smithi096 ceph-mon[112443]: pgmap v1088: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:06.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:05 smithi196 ceph-mon[81815]: pgmap v1088: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:07.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.145+0000 7f57ca91b700 1 -- 172.21.15.96:0/910388718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 msgr2=0x7f57c4105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.145+0000 7f57ca91b700 1 --2- 172.21.15.96:0/910388718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 0x7f57c4105e90 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009910 tx=0x7f57b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:07.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.145+0000 7f57ca91b700 1 -- 172.21.15.96:0/910388718 shutdown_connections 2024-04-22T15:58:07.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.145+0000 7f57ca91b700 1 --2- 172.21.15.96:0/910388718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c40ffab0 0x7f57c40fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.145+0000 7f57ca91b700 1 --2- 172.21.15.96:0/910388718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 0x7f57c4105e90 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.145+0000 7f57ca91b700 1 -- 172.21.15.96:0/910388718 >> 172.21.15.96:0/910388718 conn(0x7f57c40fb5e0 msgr2=0x7f57c40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:07.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.145+0000 7f57ca91b700 1 -- 172.21.15.96:0/910388718 shutdown_connections 2024-04-22T15:58:07.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.145+0000 7f57ca91b700 1 -- 172.21.15.96:0/910388718 wait complete. 2024-04-22T15:58:07.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.146+0000 7f57ca91b700 1 Processor -- start 2024-04-22T15:58:07.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.146+0000 7f57ca91b700 1 -- start start 2024-04-22T15:58:07.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.146+0000 7f57ca91b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c40ffab0 0x7f57c4197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:07.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.146+0000 7f57ca91b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 0x7f57c4197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:07.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.146+0000 7f57ca91b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4198480 con 0x7f57c4105ab0 2024-04-22T15:58:07.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.146+0000 7f57ca91b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c41985c0 con 0x7f57c40ffab0 2024-04-22T15:58:07.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.146+0000 7f57c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 0x7f57c4197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:07.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.146+0000 7f57c37fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 0x7f57c4197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36100/0 (socket says 172.21.15.96:36100) 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.146+0000 7f57c37fe700 1 -- 172.21.15.96:0/1785649678 learned_addr learned my addr 172.21.15.96:0/1785649678 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57c3fff700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c40ffab0 0x7f57c4197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57c37fe700 1 -- 172.21.15.96:0/1785649678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c40ffab0 msgr2=0x7f57c4197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57c37fe700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c40ffab0 0x7f57c4197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57c37fe700 1 -- 172.21.15.96:0/1785649678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40095f0 con 0x7f57c4105ab0 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57c3fff700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c40ffab0 0x7f57c4197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57c37fe700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 0x7f57c4197eb0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f57b800c8a0 tx=0x7f57b800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57c17fa700 1 -- 172.21.15.96:0/1785649678 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57b800cea0 con 0x7f57c4105ab0 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57c17fa700 1 -- 172.21.15.96:0/1785649678 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57b8004830 con 0x7f57c4105ab0 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c41a09e0 con 0x7f57c4105ab0 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57c17fa700 1 -- 172.21.15.96:0/1785649678 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57b80055b0 con 0x7f57c4105ab0 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.147+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c41a0df0 con 0x7f57c4105ab0 2024-04-22T15:58:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.148+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c41080b0 con 0x7f57c4105ab0 2024-04-22T15:58:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.149+0000 7f57c17fa700 1 -- 172.21.15.96:0/1785649678 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f57b80049a0 con 0x7f57c4105ab0 2024-04-22T15:58:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.149+0000 7f57c17fa700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.149+0000 7f57c3fff700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.149+0000 7f57c17fa700 1 -- 172.21.15.96:0/1785649678 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f57b808ec40 con 0x7f57c4105ab0 2024-04-22T15:58:07.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.150+0000 7f57c3fff700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f57b40059d0 tx=0x7f57b401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:07.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.154+0000 7f57c17fa700 1 -- 172.21.15.96:0/1785649678 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f57b8057900 con 0x7f57c4105ab0 2024-04-22T15:58:07.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.371+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f57c410ccb0 con 0x7f57ac06ecc0 2024-04-22T15:58:07.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.373+0000 7f57c17fa700 1 -- 172.21.15.96:0/1785649678 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f57c410ccb0 con 0x7f57ac06ecc0 2024-04-22T15:58:07.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 msgr2=0x7f57ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f57b40059d0 tx=0x7f57b401a040 comp rx=0 tx=0).stop 2024-04-22T15:58:07.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 msgr2=0x7f57c4197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 0x7f57c4197eb0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f57b800c8a0 tx=0x7f57b800cc60 comp rx=0 tx=0).stop 2024-04-22T15:58:07.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 shutdown_connections 2024-04-22T15:58:07.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f57ac06ecc0 0x7f57ac071180 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57c40ffab0 0x7f57c4197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 --2- 172.21.15.96:0/1785649678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57c4105ab0 0x7f57c4197eb0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 >> 172.21.15.96:0/1785649678 conn(0x7f57c40fb5e0 msgr2=0x7f57c40fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:07.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 shutdown_connections 2024-04-22T15:58:07.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.375+0000 7f57ca91b700 1 -- 172.21.15.96:0/1785649678 wait complete. 2024-04-22T15:58:07.388 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:58:07.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 -- 172.21.15.96:0/1408003098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 msgr2=0x7f669c0fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 --2- 172.21.15.96:0/1408003098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 0x7f669c0fff50 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f6690009910 tx=0x7f6690009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:07.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 -- 172.21.15.96:0/1408003098 shutdown_connections 2024-04-22T15:58:07.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 --2- 172.21.15.96:0/1408003098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 0x7f669c0fff50 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 --2- 172.21.15.96:0/1408003098 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f669c105b60 0x7f669c105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 -- 172.21.15.96:0/1408003098 >> 172.21.15.96:0/1408003098 conn(0x7f669c0fb600 msgr2=0x7f669c0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:07.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 -- 172.21.15.96:0/1408003098 shutdown_connections 2024-04-22T15:58:07.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 -- 172.21.15.96:0/1408003098 wait complete. 2024-04-22T15:58:07.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 Processor -- start 2024-04-22T15:58:07.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.479+0000 7f66a235a700 1 -- start start 2024-04-22T15:58:07.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f66a235a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 0x7f669c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:07.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f66a235a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f669c105b60 0x7f669c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:07.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f66a235a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c19d3a0 con 0x7f669c0ffad0 2024-04-22T15:58:07.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f66a235a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c196980 con 0x7f669c105b60 2024-04-22T15:58:07.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f669bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 0x7f669c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:07.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f669bfff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 0x7f669c19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36110/0 (socket says 172.21.15.96:36110) 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f669bfff700 1 -- 172.21.15.96:0/2013095058 learned_addr learned my addr 172.21.15.96:0/2013095058 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f669b7fe700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f669c105b60 0x7f669c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f669bfff700 1 -- 172.21.15.96:0/2013095058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f669c105b60 msgr2=0x7f669c19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f669bfff700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f669c105b60 0x7f669c19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f669bfff700 1 -- 172.21.15.96:0/2013095058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66900095f0 con 0x7f669c0ffad0 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f669b7fe700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f669c105b60 0x7f669c19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f669bfff700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 0x7f669c19c840 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f668c00d8d0 tx=0x7f668c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f66997fa700 1 -- 172.21.15.96:0/2013095058 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f668c009940 con 0x7f669c0ffad0 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669c196c60 con 0x7f669c0ffad0 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f66997fa700 1 -- 172.21.15.96:0/2013095058 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f668c010460 con 0x7f669c0ffad0 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f66997fa700 1 -- 172.21.15.96:0/2013095058 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f668c00f5f0 con 0x7f669c0ffad0 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.480+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669c1971b0 con 0x7f669c0ffad0 2024-04-22T15:58:07.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.481+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669c109100 con 0x7f669c0ffad0 2024-04-22T15:58:07.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.482+0000 7f66997fa700 1 -- 172.21.15.96:0/2013095058 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f668c0105d0 con 0x7f669c0ffad0 2024-04-22T15:58:07.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.482+0000 7f66997fa700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f668406ecd0 0x7f6684071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:07.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.482+0000 7f66997fa700 1 -- 172.21.15.96:0/2013095058 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f668c08eed0 con 0x7f669c0ffad0 2024-04-22T15:58:07.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.482+0000 7f669b7fe700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f668406ecd0 0x7f6684071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:07.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.483+0000 7f669b7fe700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f668406ecd0 0x7f6684071190 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f669c197b40 tx=0x7f669000b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:07.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.487+0000 7f66997fa700 1 -- 172.21.15.96:0/2013095058 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f668c05b760 con 0x7f669c0ffad0 2024-04-22T15:58:07.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.703+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f669c197eb0 con 0x7f668406ecd0 2024-04-22T15:58:07.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.704+0000 7f66997fa700 1 -- 172.21.15.96:0/2013095058 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f669c197eb0 con 0x7f668406ecd0 2024-04-22T15:58:07.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f668406ecd0 msgr2=0x7f6684071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f668406ecd0 0x7f6684071190 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f669c197b40 tx=0x7f669000b560 comp rx=0 tx=0).stop 2024-04-22T15:58:07.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 msgr2=0x7f669c19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 0x7f669c19c840 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f668c00d8d0 tx=0x7f668c00dc90 comp rx=0 tx=0).stop 2024-04-22T15:58:07.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 shutdown_connections 2024-04-22T15:58:07.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f668406ecd0 0x7f6684071190 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f669c0ffad0 0x7f669c19c840 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 --2- 172.21.15.96:0/2013095058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f669c105b60 0x7f669c19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 >> 172.21.15.96:0/2013095058 conn(0x7f669c0fb600 msgr2=0x7f669c069670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:07.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 shutdown_connections 2024-04-22T15:58:07.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.706+0000 7f66a235a700 1 -- 172.21.15.96:0/2013095058 wait complete. 2024-04-22T15:58:07.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.812+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/4113537897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 msgr2=0x7fbf48101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.812+0000 7fbf4fbe7700 1 --2- 172.21.15.96:0/4113537897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 0x7fbf48101200 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009910 tx=0x7fbf44009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:07.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.812+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/4113537897 shutdown_connections 2024-04-22T15:58:07.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.812+0000 7fbf4fbe7700 1 --2- 172.21.15.96:0/4113537897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf481017d0 0x7fbf48105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.812+0000 7fbf4fbe7700 1 --2- 172.21.15.96:0/4113537897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 0x7fbf48101200 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.812+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/4113537897 >> 172.21.15.96:0/4113537897 conn(0x7fbf480fc6d0 msgr2=0x7fbf480feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:07.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.812+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/4113537897 shutdown_connections 2024-04-22T15:58:07.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.812+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/4113537897 wait complete. 2024-04-22T15:58:07.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4fbe7700 1 Processor -- start 2024-04-22T15:58:07.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4fbe7700 1 -- start start 2024-04-22T15:58:07.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4fbe7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 0x7fbf4819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:07.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4fbe7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf481017d0 0x7fbf4819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:07.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4fbe7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4819d3a0 con 0x7fbf48100e20 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4fbe7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf481969d0 con 0x7fbf481017d0 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4d983700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 0x7fbf4819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4d983700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 0x7fbf4819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36130/0 (socket says 172.21.15.96:36130) 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4d983700 1 -- 172.21.15.96:0/422483673 learned_addr learned my addr 172.21.15.96:0/422483673 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4d983700 1 -- 172.21.15.96:0/422483673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf481017d0 msgr2=0x7fbf4819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4d182700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf481017d0 0x7fbf4819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4d983700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf481017d0 0x7fbf4819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4d983700 1 -- 172.21.15.96:0/422483673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf440095f0 con 0x7fbf48100e20 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.813+0000 7fbf4d182700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf481017d0 0x7fbf4819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.814+0000 7fbf4d983700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 0x7fbf4819c840 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400bd70 tx=0x7fbf4400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.814+0000 7fbf3effd700 1 -- 172.21.15.96:0/422483673 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf4401d070 con 0x7fbf48100e20 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.814+0000 7fbf3effd700 1 -- 172.21.15.96:0/422483673 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf4400f460 con 0x7fbf48100e20 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.814+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf48196c50 con 0x7fbf48100e20 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.814+0000 7fbf3effd700 1 -- 172.21.15.96:0/422483673 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf44017630 con 0x7fbf48100e20 2024-04-22T15:58:07.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.814+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf48197140 con 0x7fbf48100e20 2024-04-22T15:58:07.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.815+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf481091c0 con 0x7fbf48100e20 2024-04-22T15:58:07.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.815+0000 7fbf3effd700 1 -- 172.21.15.96:0/422483673 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbf4400f5d0 con 0x7fbf48100e20 2024-04-22T15:58:07.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.816+0000 7fbf3effd700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf3406ecc0 0x7fbf34071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:07.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.816+0000 7fbf3effd700 1 -- 172.21.15.96:0/422483673 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbf440903c0 con 0x7fbf48100e20 2024-04-22T15:58:07.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.816+0000 7fbf4d182700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf3406ecc0 0x7fbf34071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:07.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.816+0000 7fbf4d182700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf3406ecc0 0x7fbf34071180 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbf38005fd0 tx=0x7fbf38005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:07.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:07.820+0000 7fbf3effd700 1 -- 172.21.15.96:0/422483673 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbf4405cc50 con 0x7fbf48100e20 2024-04-22T15:58:08.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.027+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf48197eb0 con 0x7fbf3406ecc0 2024-04-22T15:58:08.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.038+0000 7fbf3effd700 1 -- 172.21.15.96:0/422483673 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbf48197eb0 con 0x7fbf3406ecc0 2024-04-22T15:58:08.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:07 smithi096 ceph-mon[112443]: from='client.35964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:08.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:07 smithi096 ceph-mon[112443]: from='client.35968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:08.041 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:07 smithi096 ceph-mon[112443]: pgmap v1089: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:08.041 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:58:08.041 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (36m) 14m ago 48m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:58:08.041 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (48m) 14m ago 48m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:58:08.041 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (47m) 14m ago 47m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:58:08.041 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (34m) 14m ago 48m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:58:08.041 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (33m) 14m ago 47m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:58:08.041 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (36m) 14m ago 47m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:58:08.041 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (29m) 14m ago 42m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (30m) 14m ago 42m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (29m) 14m ago 42m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (14m) 14m ago 42m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (39m) 14m ago 49m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (38m) 14m ago 47m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (35m) 14m ago 49m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (34m) 14m ago 46m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (37m) 14m ago 48m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (37m) 14m ago 47m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (33m) 14m ago 46m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (32m) 14m ago 45m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (32m) 14m ago 45m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (31m) 14m ago 44m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:58:08.042 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (31m) 14m ago 44m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:58:08.043 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (30m) 14m ago 43m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:58:08.043 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (37m) 14m ago 47m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:58:08.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf3406ecc0 msgr2=0x7fbf34071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf3406ecc0 0x7fbf34071180 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbf38005fd0 tx=0x7fbf38005dc0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 msgr2=0x7fbf4819c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 0x7fbf4819c840 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400bd70 tx=0x7fbf4400be50 comp rx=0 tx=0).stop 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 shutdown_connections 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbf3406ecc0 0x7fbf34071180 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf48100e20 0x7fbf4819c840 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 --2- 172.21.15.96:0/422483673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf481017d0 0x7fbf4819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 >> 172.21.15.96:0/422483673 conn(0x7fbf480fc6d0 msgr2=0x7fbf480fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 shutdown_connections 2024-04-22T15:58:08.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.040+0000 7fbf4fbe7700 1 -- 172.21.15.96:0/422483673 wait complete. 2024-04-22T15:58:08.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.142+0000 7f3cc78fa700 1 -- 172.21.15.96:0/1562790799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 msgr2=0x7f3cc01007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.142+0000 7f3cc78fa700 1 --2- 172.21.15.96:0/1562790799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 0x7f3cc01007c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009910 tx=0x7f3cb0009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:08.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.142+0000 7f3cc78fa700 1 -- 172.21.15.96:0/1562790799 shutdown_connections 2024-04-22T15:58:08.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.142+0000 7f3cc78fa700 1 --2- 172.21.15.96:0/1562790799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3cc0100d90 0x7f3cc0104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.142+0000 7f3cc78fa700 1 --2- 172.21.15.96:0/1562790799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 0x7f3cc01007c0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.142+0000 7f3cc78fa700 1 -- 172.21.15.96:0/1562790799 >> 172.21.15.96:0/1562790799 conn(0x7f3cc0075f30 msgr2=0x7f3cc00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:08.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.142+0000 7f3cc78fa700 1 -- 172.21.15.96:0/1562790799 shutdown_connections 2024-04-22T15:58:08.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.142+0000 7f3cc78fa700 1 -- 172.21.15.96:0/1562790799 wait complete. 2024-04-22T15:58:08.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc78fa700 1 Processor -- start 2024-04-22T15:58:08.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc78fa700 1 -- start start 2024-04-22T15:58:08.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc78fa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 0x7f3cc019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc78fa700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3cc0100d90 0x7f3cc019ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc78fa700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc019d460 con 0x7f3cc01003e0 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc78fa700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc01969d0 con 0x7f3cc0100d90 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc5696700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 0x7f3cc019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc5696700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 0x7f3cc019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36158/0 (socket says 172.21.15.96:36158) 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc5696700 1 -- 172.21.15.96:0/4107258238 learned_addr learned my addr 172.21.15.96:0/4107258238 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc4e95700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3cc0100d90 0x7f3cc019ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc5696700 1 -- 172.21.15.96:0/4107258238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3cc0100d90 msgr2=0x7f3cc019ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc5696700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3cc0100d90 0x7f3cc019ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.143+0000 7f3cc5696700 1 -- 172.21.15.96:0/4107258238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb00095f0 con 0x7f3cc01003e0 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.144+0000 7f3cc5696700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 0x7f3cc019c8d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0006010 tx=0x7f3cb000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.144+0000 7f3cb67fc700 1 -- 172.21.15.96:0/4107258238 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cb001d070 con 0x7f3cc01003e0 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.144+0000 7f3cb67fc700 1 -- 172.21.15.96:0/4107258238 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cb000f460 con 0x7f3cc01003e0 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.144+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc0196c50 con 0x7f3cc01003e0 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.144+0000 7f3cb67fc700 1 -- 172.21.15.96:0/4107258238 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cb0017630 con 0x7f3cc01003e0 2024-04-22T15:58:08.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.144+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc0197140 con 0x7f3cc01003e0 2024-04-22T15:58:08.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.145+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc01013a0 con 0x7f3cc01003e0 2024-04-22T15:58:08.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.145+0000 7f3cb67fc700 1 -- 172.21.15.96:0/4107258238 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3cb000f5d0 con 0x7f3cc01003e0 2024-04-22T15:58:08.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.145+0000 7f3cb67fc700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3cac06ecc0 0x7f3cac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.146+0000 7f3cb67fc700 1 -- 172.21.15.96:0/4107258238 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3cb0090340 con 0x7f3cc01003e0 2024-04-22T15:58:08.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.146+0000 7f3cc4e95700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3cac06ecc0 0x7f3cac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.146+0000 7f3cc4e95700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3cac06ecc0 0x7f3cac071180 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3cc01980f0 tx=0x7f3cbc005d50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:08.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.150+0000 7f3cb67fc700 1 -- 172.21.15.96:0/4107258238 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3cb00591e0 con 0x7f3cc01003e0 2024-04-22T15:58:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:07 smithi196 ceph-mon[81815]: from='client.35964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:07 smithi196 ceph-mon[81815]: from='client.35968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:07 smithi196 ceph-mon[81815]: pgmap v1089: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:08.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.440+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3cc010cd20 con 0x7f3cc01003e0 2024-04-22T15:58:08.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.440+0000 7f3cb67fc700 1 -- 172.21.15.96:0/4107258238 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f3cb005c8a0 con 0x7f3cc01003e0 2024-04-22T15:58:08.443 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:58:08.443 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:58:08.443 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:58:08.443 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:58:08.444 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3cac06ecc0 msgr2=0x7f3cac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3cac06ecc0 0x7f3cac071180 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3cc01980f0 tx=0x7f3cbc005d50 comp rx=0 tx=0).stop 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 msgr2=0x7f3cc019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 0x7f3cc019c8d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0006010 tx=0x7f3cb000be20 comp rx=0 tx=0).stop 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 shutdown_connections 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3cac06ecc0 0x7f3cac071180 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc01003e0 0x7f3cc019c8d0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 --2- 172.21.15.96:0/4107258238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3cc0100d90 0x7f3cc019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 >> 172.21.15.96:0/4107258238 conn(0x7f3cc0075f30 msgr2=0x7f3cc00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 shutdown_connections 2024-04-22T15:58:08.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.443+0000 7f3cc78fa700 1 -- 172.21.15.96:0/4107258238 wait complete. 2024-04-22T15:58:08.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.546+0000 7f80788cb700 1 -- 172.21.15.96:0/4195520123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074102640 msgr2=0x7f807410ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.546+0000 7f80788cb700 1 --2- 172.21.15.96:0/4195520123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074102640 0x7f807410ab30 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8068009910 tx=0x7f8068009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:08.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.547+0000 7f80788cb700 1 -- 172.21.15.96:0/4195520123 shutdown_connections 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.547+0000 7f80788cb700 1 --2- 172.21.15.96:0/4195520123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074102640 0x7f807410ab30 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.547+0000 7f80788cb700 1 --2- 172.21.15.96:0/4195520123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8074101d20 0x7f8074102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.547+0000 7f80788cb700 1 -- 172.21.15.96:0/4195520123 >> 172.21.15.96:0/4195520123 conn(0x7f80740fb590 msgr2=0x7f80740fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.547+0000 7f80788cb700 1 -- 172.21.15.96:0/4195520123 shutdown_connections 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.547+0000 7f80788cb700 1 -- 172.21.15.96:0/4195520123 wait complete. 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.547+0000 7f80788cb700 1 Processor -- start 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.547+0000 7f80788cb700 1 -- start start 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f80788cb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074101d20 0x7f807419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f80788cb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8074102640 0x7f807419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f80788cb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807419d320 con 0x7f8074101d20 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f80788cb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8074196900 con 0x7f8074102640 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f807259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074101d20 0x7f807419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f807259c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074101d20 0x7f807419c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36186/0 (socket says 172.21.15.96:36186) 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f807259c700 1 -- 172.21.15.96:0/580503193 learned_addr learned my addr 172.21.15.96:0/580503193 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f8071d9b700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8074102640 0x7f807419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f807259c700 1 -- 172.21.15.96:0/580503193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8074102640 msgr2=0x7f807419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f807259c700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8074102640 0x7f807419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f807259c700 1 -- 172.21.15.96:0/580503193 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80680095f0 con 0x7f8074101d20 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f807259c700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074101d20 0x7f807419c7c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f806400c8a0 tx=0x7f806400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f80637fe700 1 -- 172.21.15.96:0/580503193 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f806400cea0 con 0x7f8074101d20 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8074196be0 con 0x7f8074101d20 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.548+0000 7f80637fe700 1 -- 172.21.15.96:0/580503193 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8064004830 con 0x7f8074101d20 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.549+0000 7f80637fe700 1 -- 172.21.15.96:0/580503193 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80640055b0 con 0x7f8074101d20 2024-04-22T15:58:08.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.549+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8074197130 con 0x7f8074101d20 2024-04-22T15:58:08.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.549+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8074105440 con 0x7f8074101d20 2024-04-22T15:58:08.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.550+0000 7f80637fe700 1 -- 172.21.15.96:0/580503193 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f80640049a0 con 0x7f8074101d20 2024-04-22T15:58:08.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.550+0000 7f80637fe700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f805c06ecd0 0x7f805c071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.550+0000 7f8071d9b700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f805c06ecd0 0x7f805c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.550+0000 7f80637fe700 1 -- 172.21.15.96:0/580503193 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f806408eb70 con 0x7f8074101d20 2024-04-22T15:58:08.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.551+0000 7f8071d9b700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f805c06ecd0 0x7f805c071190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8074197ac0 tx=0x7f806801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:08.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.555+0000 7f80637fe700 1 -- 172.21.15.96:0/580503193 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f806405b400 con 0x7f8074101d20 2024-04-22T15:58:08.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.802+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8074197dc0 con 0x7f8074101d20 2024-04-22T15:58:08.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.803+0000 7f80637fe700 1 -- 172.21.15.96:0/580503193 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f806401d770 con 0x7f8074101d20 2024-04-22T15:58:08.806 INFO:teuthology.orchestra.run.smithi096.stdout:e31 2024-04-22T15:58:08.806 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:58:08.806 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:58:08.806 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:58:08.806 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:58:08.806 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:58:08.806 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:58:08.806 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 22 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:27:39.581378+0000 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:58:08.807 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:up {} 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:failed 0,1 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:58:08.808 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:58:08.809 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:58:08.809 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:58:08.809 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:58:08.809 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:58:08.809 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:08.809 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:08.809 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:08.809 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:08.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f805c06ecd0 msgr2=0x7f805c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f805c06ecd0 0x7f805c071190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8074197ac0 tx=0x7f806801a040 comp rx=0 tx=0).stop 2024-04-22T15:58:08.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074101d20 msgr2=0x7f807419c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074101d20 0x7f807419c7c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f806400c8a0 tx=0x7f806400cc60 comp rx=0 tx=0).stop 2024-04-22T15:58:08.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 shutdown_connections 2024-04-22T15:58:08.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f805c06ecd0 0x7f805c071190 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8074101d20 0x7f807419c7c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 --2- 172.21.15.96:0/580503193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8074102640 0x7f807419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 >> 172.21.15.96:0/580503193 conn(0x7f80740fb590 msgr2=0x7f8074103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:08.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 shutdown_connections 2024-04-22T15:58:08.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.806+0000 7f80788cb700 1 -- 172.21.15.96:0/580503193 wait complete. 2024-04-22T15:58:08.811 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T15:58:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.910+0000 7f99cc134700 1 -- 172.21.15.96:0/3736850731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 msgr2=0x7f99c4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.910+0000 7f99cc134700 1 --2- 172.21.15.96:0/3736850731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 0x7f99c4101200 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009910 tx=0x7f99c0009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.910+0000 7f99cc134700 1 -- 172.21.15.96:0/3736850731 shutdown_connections 2024-04-22T15:58:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.910+0000 7f99cc134700 1 --2- 172.21.15.96:0/3736850731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99c41017d0 0x7f99c4105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.910+0000 7f99cc134700 1 --2- 172.21.15.96:0/3736850731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 0x7f99c4101200 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.910+0000 7f99cc134700 1 -- 172.21.15.96:0/3736850731 >> 172.21.15.96:0/3736850731 conn(0x7f99c40fc6d0 msgr2=0x7f99c40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:08.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.910+0000 7f99cc134700 1 -- 172.21.15.96:0/3736850731 shutdown_connections 2024-04-22T15:58:08.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.910+0000 7f99cc134700 1 -- 172.21.15.96:0/3736850731 wait complete. 2024-04-22T15:58:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99cc134700 1 Processor -- start 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99cc134700 1 -- start start 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99cc134700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 0x7f99c40731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99cc134700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99c41017d0 0x7f99c4073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99cc134700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c4073c60 con 0x7f99c4100e20 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99cc134700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c4073da0 con 0x7f99c41017d0 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99c9ed0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 0x7f99c40731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99c9ed0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 0x7f99c40731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36202/0 (socket says 172.21.15.96:36202) 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99c9ed0700 1 -- 172.21.15.96:0/2845812001 learned_addr learned my addr 172.21.15.96:0/2845812001 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.911+0000 7f99c96cf700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99c41017d0 0x7f99c4073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.912+0000 7f99c9ed0700 1 -- 172.21.15.96:0/2845812001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99c41017d0 msgr2=0x7f99c4073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.912+0000 7f99c9ed0700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99c41017d0 0x7f99c4073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.912+0000 7f99c9ed0700 1 -- 172.21.15.96:0/2845812001 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c00095f0 con 0x7f99c4100e20 2024-04-22T15:58:08.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.912+0000 7f99c9ed0700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 0x7f99c40731e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f99c0006010 tx=0x7f99c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:08.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.912+0000 7f99baffd700 1 -- 172.21.15.96:0/2845812001 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99c001d070 con 0x7f99c4100e20 2024-04-22T15:58:08.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.912+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c4074020 con 0x7f99c4100e20 2024-04-22T15:58:08.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.912+0000 7f99baffd700 1 -- 172.21.15.96:0/2845812001 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99c000f460 con 0x7f99c4100e20 2024-04-22T15:58:08.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.912+0000 7f99baffd700 1 -- 172.21.15.96:0/2845812001 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99c0017630 con 0x7f99c4100e20 2024-04-22T15:58:08.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.912+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c41a3080 con 0x7f99c4100e20 2024-04-22T15:58:08.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.913+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c41091c0 con 0x7f99c4100e20 2024-04-22T15:58:08.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.914+0000 7f99baffd700 1 -- 172.21.15.96:0/2845812001 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f99c000f5d0 con 0x7f99c4100e20 2024-04-22T15:58:08.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.914+0000 7f99baffd700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f99b006ecd0 0x7f99b0071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.914+0000 7f99baffd700 1 -- 172.21.15.96:0/2845812001 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f99c00903d0 con 0x7f99c4100e20 2024-04-22T15:58:08.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.914+0000 7f99c96cf700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f99b006ecd0 0x7f99b0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.914+0000 7f99c96cf700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f99b006ecd0 0x7f99b0071190 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f99b4006fd0 tx=0x7f99b4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:08.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:08.919+0000 7f99baffd700 1 -- 172.21.15.96:0/2845812001 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f99c00591f0 con 0x7f99c4100e20 2024-04-22T15:58:09.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.140+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99c410de60 con 0x7f99b006ecd0 2024-04-22T15:58:09.143 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:08 smithi096 ceph-mon[112443]: from='client.35972 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:09.143 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:09.143 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:08 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4107258238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.141+0000 7f99baffd700 1 -- 172.21.15.96:0/2845812001 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f99c410de60 con 0x7f99b006ecd0 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:58:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:58:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:58:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:58:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:58:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:58:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:58:09.145 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:58:09.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.144+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f99b006ecd0 msgr2=0x7f99b0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.144+0000 7f99cc134700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f99b006ecd0 0x7f99b0071190 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f99b4006fd0 tx=0x7f99b4008040 comp rx=0 tx=0).stop 2024-04-22T15:58:09.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.144+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 msgr2=0x7f99c40731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.144+0000 7f99cc134700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 0x7f99c40731e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f99c0006010 tx=0x7f99c000be20 comp rx=0 tx=0).stop 2024-04-22T15:58:09.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.144+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 shutdown_connections 2024-04-22T15:58:09.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.144+0000 7f99cc134700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f99b006ecd0 0x7f99b0071190 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.144+0000 7f99cc134700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99c4100e20 0x7f99c40731e0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.144+0000 7f99cc134700 1 --2- 172.21.15.96:0/2845812001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99c41017d0 0x7f99c4073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.144+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 >> 172.21.15.96:0/2845812001 conn(0x7f99c40fc6d0 msgr2=0x7f99c40fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:09.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.145+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 shutdown_connections 2024-04-22T15:58:09.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.145+0000 7f99cc134700 1 -- 172.21.15.96:0/2845812001 wait complete. 2024-04-22T15:58:09.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.249+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/1933017103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad01017d0 msgr2=0x7f1ad0105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.249+0000 7f1ad6fdc700 1 --2- 172.21.15.96:0/1933017103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad01017d0 0x7f1ad0105820 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009910 tx=0x7f1abc009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.249+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/1933017103 shutdown_connections 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.249+0000 7f1ad6fdc700 1 --2- 172.21.15.96:0/1933017103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad01017d0 0x7f1ad0105820 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.249+0000 7f1ad6fdc700 1 --2- 172.21.15.96:0/1933017103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ad0100e20 0x7f1ad0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.249+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/1933017103 >> 172.21.15.96:0/1933017103 conn(0x7f1ad00fc6d0 msgr2=0x7f1ad00feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.249+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/1933017103 shutdown_connections 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.249+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/1933017103 wait complete. 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad6fdc700 1 Processor -- start 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad6fdc700 1 -- start start 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad6fdc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad0100e20 0x7f1ad019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad6fdc700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ad01017d0 0x7f1ad019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:09.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad6fdc700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad019d4c0 con 0x7f1ad0100e20 2024-04-22T15:58:09.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad6fdc700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad0196aa0 con 0x7f1ad01017d0 2024-04-22T15:58:09.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad4d78700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad0100e20 0x7f1ad019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:09.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad4d78700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad0100e20 0x7f1ad019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36218/0 (socket says 172.21.15.96:36218) 2024-04-22T15:58:09.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad4d78700 1 -- 172.21.15.96:0/831490204 learned_addr learned my addr 172.21.15.96:0/831490204 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:09.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad4d78700 1 -- 172.21.15.96:0/831490204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ad01017d0 msgr2=0x7f1ad019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad4d78700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ad01017d0 0x7f1ad019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad4d78700 1 -- 172.21.15.96:0/831490204 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc0095f0 con 0x7f1ad0100e20 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.250+0000 7f1ad4d78700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad0100e20 0x7f1ad019c960 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400b4b0 tx=0x7f1ac400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.251+0000 7f1acdffb700 1 -- 172.21.15.96:0/831490204 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ac4011800 con 0x7f1ad0100e20 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.251+0000 7f1acdffb700 1 -- 172.21.15.96:0/831490204 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ac4011e20 con 0x7f1ad0100e20 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.251+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad0196d80 con 0x7f1ad0100e20 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.251+0000 7f1acdffb700 1 -- 172.21.15.96:0/831490204 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ac400f2e0 con 0x7f1ad0100e20 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.251+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad01972d0 con 0x7f1ad0100e20 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.252+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad0109150 con 0x7f1ad0100e20 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.252+0000 7f1acdffb700 1 -- 172.21.15.96:0/831490204 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1ac4010380 con 0x7f1ad0100e20 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.253+0000 7f1acdffb700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1ac006ecc0 0x7f1ac0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.253+0000 7f1acdffb700 1 -- 172.21.15.96:0/831490204 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1ac408ea80 con 0x7f1ad0100e20 2024-04-22T15:58:09.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.253+0000 7f1acffff700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1ac006ecc0 0x7f1ac0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:09.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.253+0000 7f1acffff700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1ac006ecc0 0x7f1ac0071180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00b5c0 tx=0x7f1abc005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:08 smithi196 ceph-mon[81815]: from='client.35972 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:08 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4107258238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:09.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.257+0000 7f1acdffb700 1 -- 172.21.15.96:0/831490204 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1ac405b310 con 0x7f1ad0100e20 2024-04-22T15:58:09.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.539+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1ad0197f60 con 0x7f1ad0100e20 2024-04-22T15:58:09.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.540+0000 7f1acdffb700 1 -- 172.21.15.96:0/831490204 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f1ac405aea0 con 0x7f1ad0100e20 2024-04-22T15:58:09.542 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:58:09.543 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:58:09.544 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:58:09.544 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:58:09.544 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvvrarxcx:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:58:09.544 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:58:09.544 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:58:09.544 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:58:09.544 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:58:09.544 INFO:teuthology.orchestra.run.smithi096.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T15:58:09.544 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1ac006ecc0 msgr2=0x7f1ac0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1ac006ecc0 0x7f1ac0071180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00b5c0 tx=0x7f1abc005960 comp rx=0 tx=0).stop 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad0100e20 msgr2=0x7f1ad019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad0100e20 0x7f1ad019c960 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400b4b0 tx=0x7f1ac400b870 comp rx=0 tx=0).stop 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 shutdown_connections 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f1ac006ecc0 0x7f1ac0071180 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad0100e20 0x7f1ad019c960 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 --2- 172.21.15.96:0/831490204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ad01017d0 0x7f1ad019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 >> 172.21.15.96:0/831490204 conn(0x7f1ad00fc6d0 msgr2=0x7f1ad00fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 shutdown_connections 2024-04-22T15:58:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:09.542+0000 7f1ad6fdc700 1 -- 172.21.15.96:0/831490204 wait complete. 2024-04-22T15:58:10.093 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:09 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/580503193' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:58:10.093 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:09 smithi196 ceph-mon[81815]: from='client.35984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:10.093 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:09 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/831490204' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:58:10.093 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:09 smithi196 ceph-mon[81815]: pgmap v1090: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:10.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:09 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/580503193' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:58:10.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:09 smithi096 ceph-mon[112443]: from='client.35984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:10.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:09 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/831490204' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:58:10.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:09 smithi096 ceph-mon[112443]: pgmap v1090: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:12.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:11 smithi096 ceph-mon[112443]: pgmap v1091: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:12.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:11 smithi196 ceph-mon[81815]: pgmap v1091: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:14.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:13 smithi096 ceph-mon[112443]: pgmap v1092: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:14.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:13 smithi196 ceph-mon[81815]: pgmap v1092: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:16.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:15 smithi096 ceph-mon[112443]: pgmap v1093: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:16.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:15 smithi196 ceph-mon[81815]: pgmap v1093: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:18.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:17 smithi096 ceph-mon[112443]: pgmap v1094: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:18.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:17 smithi196 ceph-mon[81815]: pgmap v1094: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:20.200 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:19 smithi196 ceph-mon[81815]: pgmap v1095: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:20.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:19 smithi096 ceph-mon[112443]: pgmap v1095: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:22.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:21 smithi096 ceph-mon[112443]: pgmap v1096: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:22.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:21 smithi196 ceph-mon[81815]: pgmap v1096: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:23.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:23.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:24.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:24 smithi096 ceph-mon[112443]: pgmap v1097: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:24.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:24 smithi196 ceph-mon[81815]: pgmap v1097: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:26.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:25 smithi096 ceph-mon[112443]: pgmap v1098: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:26.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:25 smithi196 ceph-mon[81815]: pgmap v1098: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:28.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:27 smithi096 ceph-mon[112443]: pgmap v1099: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:28.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:27 smithi196 ceph-mon[81815]: pgmap v1099: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:30.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:29 smithi096 ceph-mon[112443]: pgmap v1100: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:30.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:29 smithi196 ceph-mon[81815]: pgmap v1100: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:32.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:31 smithi096 ceph-mon[112443]: pgmap v1101: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:32.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:31 smithi196 ceph-mon[81815]: pgmap v1101: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:34.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:33 smithi096 ceph-mon[112443]: pgmap v1102: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:34.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:33 smithi196 ceph-mon[81815]: pgmap v1102: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:36.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:35 smithi096 ceph-mon[112443]: pgmap v1103: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:36.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:35 smithi196 ceph-mon[81815]: pgmap v1103: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:38.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:37 smithi096 ceph-mon[112443]: pgmap v1104: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:38.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:37 smithi196 ceph-mon[81815]: pgmap v1104: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:39.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:39.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:39.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:39.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: pgmap v1105: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: pgmap v1106: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all mds 2024-04-22T15:58:39.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:39.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:58:39.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi096.drkkdi"}]': finished 2024-04-22T15:58:39.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:58:39.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi096.dymqlv"}]': finished 2024-04-22T15:58:39.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:58:39.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi196.hwpcvg"}]': finished 2024-04-22T15:58:39.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:58:39.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi196.pbvqlq"}]': finished 2024-04-22T15:58:39.238 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: Upgrade: Setting filesystem cephfs Joinable 2024-04-22T15:58:39.239 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-22T15:58:39.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: pgmap v1105: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: pgmap v1106: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.260 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:39.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all mds 2024-04-22T15:58:39.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:39.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi096.drkkdi"}]: dispatch 2024-04-22T15:58:39.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi096.drkkdi"}]': finished 2024-04-22T15:58:39.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi096.dymqlv"}]: dispatch 2024-04-22T15:58:39.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi096.dymqlv"}]': finished 2024-04-22T15:58:39.261 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi196.hwpcvg"}]: dispatch 2024-04-22T15:58:39.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi196.hwpcvg"}]': finished 2024-04-22T15:58:39.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi196.pbvqlq"}]: dispatch 2024-04-22T15:58:39.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi196.pbvqlq"}]': finished 2024-04-22T15:58:39.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: Upgrade: Setting filesystem cephfs Joinable 2024-04-22T15:58:39.262 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-22T15:58:39.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.659+0000 7fb9b776c700 1 -- 172.21.15.96:0/4090500193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 msgr2=0x7fb9b0105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:39.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.659+0000 7fb9b776c700 1 --2- 172.21.15.96:0/4090500193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 0x7fb9b0105e90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009910 tx=0x7fb9a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:39.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.659+0000 7fb9b776c700 1 -- 172.21.15.96:0/4090500193 shutdown_connections 2024-04-22T15:58:39.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.659+0000 7fb9b776c700 1 --2- 172.21.15.96:0/4090500193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9b00ffab0 0x7fb9b00fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:39.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.659+0000 7fb9b776c700 1 --2- 172.21.15.96:0/4090500193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 0x7fb9b0105e90 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:39.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.659+0000 7fb9b776c700 1 -- 172.21.15.96:0/4090500193 >> 172.21.15.96:0/4090500193 conn(0x7fb9b00fb5e0 msgr2=0x7fb9b00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:39.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.659+0000 7fb9b776c700 1 -- 172.21.15.96:0/4090500193 shutdown_connections 2024-04-22T15:58:39.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.659+0000 7fb9b776c700 1 -- 172.21.15.96:0/4090500193 wait complete. 2024-04-22T15:58:39.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.660+0000 7fb9b776c700 1 Processor -- start 2024-04-22T15:58:39.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.660+0000 7fb9b776c700 1 -- start start 2024-04-22T15:58:39.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.660+0000 7fb9b776c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9b00ffab0 0x7fb9b0197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:39.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.660+0000 7fb9b776c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 0x7fb9b0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:39.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.660+0000 7fb9b776c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b01984a0 con 0x7fb9b0105ab0 2024-04-22T15:58:39.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.660+0000 7fb9b776c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b01985e0 con 0x7fb9b00ffab0 2024-04-22T15:58:39.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.660+0000 7fb9b4d07700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 0x7fb9b0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:39.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.660+0000 7fb9b4d07700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 0x7fb9b0197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37936/0 (socket says 172.21.15.96:37936) 2024-04-22T15:58:39.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.660+0000 7fb9b4d07700 1 -- 172.21.15.96:0/2632316788 learned_addr learned my addr 172.21.15.96:0/2632316788 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:39.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9b5508700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9b00ffab0 0x7fb9b0197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:39.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9b4d07700 1 -- 172.21.15.96:0/2632316788 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9b00ffab0 msgr2=0x7fb9b0197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:39.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9b4d07700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9b00ffab0 0x7fb9b0197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:39.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9b4d07700 1 -- 172.21.15.96:0/2632316788 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a80095f0 con 0x7fb9b0105ab0 2024-04-22T15:58:39.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9b4d07700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 0x7fb9b0197ed0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400ada0 tx=0x7fb9a400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:39.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9ae7fc700 1 -- 172.21.15.96:0/2632316788 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9a400f070 con 0x7fb9b0105ab0 2024-04-22T15:58:39.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b01a0a00 con 0x7fb9b0105ab0 2024-04-22T15:58:39.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9ae7fc700 1 -- 172.21.15.96:0/2632316788 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9a400ae90 con 0x7fb9b0105ab0 2024-04-22T15:58:39.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9ae7fc700 1 -- 172.21.15.96:0/2632316788 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9a4005690 con 0x7fb9b0105ab0 2024-04-22T15:58:39.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.661+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b01a0e10 con 0x7fb9b0105ab0 2024-04-22T15:58:39.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.662+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b01080b0 con 0x7fb9b0105ab0 2024-04-22T15:58:39.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.663+0000 7fb9ae7fc700 1 -- 172.21.15.96:0/2632316788 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb9a4005900 con 0x7fb9b0105ab0 2024-04-22T15:58:39.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.663+0000 7fb9ae7fc700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb99806ecc0 0x7fb998071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:39.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.664+0000 7fb9b5508700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb99806ecc0 0x7fb998071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:39.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.664+0000 7fb9ae7fc700 1 -- 172.21.15.96:0/2632316788 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb9a408e830 con 0x7fb9b0105ab0 2024-04-22T15:58:39.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.664+0000 7fb9b5508700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb99806ecc0 0x7fb998071180 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8000c00 tx=0x7fb9a801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:39.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.668+0000 7fb9ae7fc700 1 -- 172.21.15.96:0/2632316788 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb9a4057590 con 0x7fb9b0105ab0 2024-04-22T15:58:39.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.908+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9b010ccb0 con 0x7fb99806ecc0 2024-04-22T15:58:39.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.909+0000 7fb9ae7fc700 1 -- 172.21.15.96:0/2632316788 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb9b010ccb0 con 0x7fb99806ecc0 2024-04-22T15:58:39.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.911+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb99806ecc0 msgr2=0x7fb998071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:39.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.911+0000 7fb9b776c700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb99806ecc0 0x7fb998071180 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8000c00 tx=0x7fb9a801a040 comp rx=0 tx=0).stop 2024-04-22T15:58:39.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.911+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 msgr2=0x7fb9b0197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:39.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.911+0000 7fb9b776c700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 0x7fb9b0197ed0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400ada0 tx=0x7fb9a400c330 comp rx=0 tx=0).stop 2024-04-22T15:58:39.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.911+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 shutdown_connections 2024-04-22T15:58:39.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.911+0000 7fb9b776c700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb99806ecc0 0x7fb998071180 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:39.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.911+0000 7fb9b776c700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9b00ffab0 0x7fb9b0197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:39.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.911+0000 7fb9b776c700 1 --2- 172.21.15.96:0/2632316788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9b0105ab0 0x7fb9b0197ed0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:39.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.911+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 >> 172.21.15.96:0/2632316788 conn(0x7fb9b00fb5e0 msgr2=0x7fb9b00fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:39.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.912+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 shutdown_connections 2024-04-22T15:58:39.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:39.912+0000 7fb9b776c700 1 -- 172.21.15.96:0/2632316788 wait complete. 2024-04-22T15:58:39.925 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T15:58:40.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.020+0000 7f0c3a70c700 1 -- 172.21.15.96:0/80451770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 msgr2=0x7f0c341011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.020+0000 7f0c3a70c700 1 --2- 172.21.15.96:0/80451770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 0x7f0c341011f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009910 tx=0x7f0c24009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:40.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.020+0000 7f0c3a70c700 1 -- 172.21.15.96:0/80451770 shutdown_connections 2024-04-22T15:58:40.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.020+0000 7f0c3a70c700 1 --2- 172.21.15.96:0/80451770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c341017c0 0x7f0c34105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.020+0000 7f0c3a70c700 1 --2- 172.21.15.96:0/80451770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 0x7f0c341011f0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.020+0000 7f0c3a70c700 1 -- 172.21.15.96:0/80451770 >> 172.21.15.96:0/80451770 conn(0x7f0c340fc6e0 msgr2=0x7f0c340feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:40.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.020+0000 7f0c3a70c700 1 -- 172.21.15.96:0/80451770 shutdown_connections 2024-04-22T15:58:40.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.020+0000 7f0c3a70c700 1 -- 172.21.15.96:0/80451770 wait complete. 2024-04-22T15:58:40.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c3a70c700 1 Processor -- start 2024-04-22T15:58:40.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c3a70c700 1 -- start start 2024-04-22T15:58:40.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c3a70c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 0x7f0c34073200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:40.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c3a70c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c341017c0 0x7f0c34073740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:40.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c3a70c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340751e0 con 0x7f0c34100e10 2024-04-22T15:58:40.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c3a70c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c34073c80 con 0x7f0c341017c0 2024-04-22T15:58:40.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c33fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 0x7f0c34073200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:40.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c33fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 0x7f0c34073200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37948/0 (socket says 172.21.15.96:37948) 2024-04-22T15:58:40.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c33fff700 1 -- 172.21.15.96:0/691157191 learned_addr learned my addr 172.21.15.96:0/691157191 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:40.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c337fe700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c341017c0 0x7f0c34073740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:40.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c33fff700 1 -- 172.21.15.96:0/691157191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c341017c0 msgr2=0x7f0c34073740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c33fff700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c341017c0 0x7f0c34073740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c33fff700 1 -- 172.21.15.96:0/691157191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c240095f0 con 0x7f0c34100e10 2024-04-22T15:58:40.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.021+0000 7f0c337fe700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c341017c0 0x7f0c34073740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:58:40.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.022+0000 7f0c33fff700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 0x7f0c34073200 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f0c24006010 tx=0x7f0c2400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:40.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.022+0000 7f0c317fa700 1 -- 172.21.15.96:0/691157191 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c2401d070 con 0x7f0c34100e10 2024-04-22T15:58:40.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.022+0000 7f0c317fa700 1 -- 172.21.15.96:0/691157191 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c2400f460 con 0x7f0c34100e10 2024-04-22T15:58:40.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.022+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c34073f00 con 0x7f0c34100e10 2024-04-22T15:58:40.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.022+0000 7f0c317fa700 1 -- 172.21.15.96:0/691157191 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c24017630 con 0x7f0c34100e10 2024-04-22T15:58:40.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.022+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c341a2f40 con 0x7f0c34100e10 2024-04-22T15:58:40.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.023+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c341091b0 con 0x7f0c34100e10 2024-04-22T15:58:40.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.024+0000 7f0c317fa700 1 -- 172.21.15.96:0/691157191 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0c2400f5d0 con 0x7f0c34100e10 2024-04-22T15:58:40.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.024+0000 7f0c317fa700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:40.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.025+0000 7f0c317fa700 1 -- 172.21.15.96:0/691157191 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0c24090500 con 0x7f0c34100e10 2024-04-22T15:58:40.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.025+0000 7f0c337fe700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:40.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.025+0000 7f0c337fe700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0c34074920 tx=0x7f0c28005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:40.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.032+0000 7f0c317fa700 1 -- 172.21.15.96:0/691157191 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0c24059440 con 0x7f0c34100e10 2024-04-22T15:58:40.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.272+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c3410dde0 con 0x7f0c1c06ecc0 2024-04-22T15:58:40.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.274+0000 7f0c317fa700 1 -- 172.21.15.96:0/691157191 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0c3410dde0 con 0x7f0c1c06ecc0 2024-04-22T15:58:40.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 msgr2=0x7f0c1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0c34074920 tx=0x7f0c28005f50 comp rx=0 tx=0).stop 2024-04-22T15:58:40.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 msgr2=0x7f0c34073200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 0x7f0c34073200 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f0c24006010 tx=0x7f0c2400be20 comp rx=0 tx=0).stop 2024-04-22T15:58:40.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 shutdown_connections 2024-04-22T15:58:40.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0c1c06ecc0 0x7f0c1c071180 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c34100e10 0x7f0c34073200 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 --2- 172.21.15.96:0/691157191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c341017c0 0x7f0c34073740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 >> 172.21.15.96:0/691157191 conn(0x7f0c340fc6e0 msgr2=0x7f0c340fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:40.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 shutdown_connections 2024-04-22T15:58:40.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.276+0000 7f0c3a70c700 1 -- 172.21.15.96:0/691157191 wait complete. 2024-04-22T15:58:40.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.390+0000 7f487a9da700 1 -- 172.21.15.96:0/19110674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 msgr2=0x7f4874103f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.390+0000 7f487a9da700 1 --2- 172.21.15.96:0/19110674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 0x7f4874103f20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f486800aa20 tx=0x7f486800ad30 comp rx=0 tx=0).stop 2024-04-22T15:58:40.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.390+0000 7f487a9da700 1 -- 172.21.15.96:0/19110674 shutdown_connections 2024-04-22T15:58:40.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.390+0000 7f487a9da700 1 --2- 172.21.15.96:0/19110674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 0x7f4874103f20 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.390+0000 7f487a9da700 1 --2- 172.21.15.96:0/19110674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48740ff520 0x7f48740ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.390+0000 7f487a9da700 1 -- 172.21.15.96:0/19110674 >> 172.21.15.96:0/19110674 conn(0x7f4874075f00 msgr2=0x7f48740fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:40.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.390+0000 7f487a9da700 1 -- 172.21.15.96:0/19110674 shutdown_connections 2024-04-22T15:58:40.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.390+0000 7f487a9da700 1 -- 172.21.15.96:0/19110674 wait complete. 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f487a9da700 1 Processor -- start 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f487a9da700 1 -- start start 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f487a9da700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48740ff520 0x7f487419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f487a9da700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 0x7f487419cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f487a9da700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487419d400 con 0x7f48740ffed0 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f487a9da700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48741969e0 con 0x7f48740ff520 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f48737fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 0x7f487419cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f48737fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 0x7f487419cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37960/0 (socket says 172.21.15.96:37960) 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f48737fe700 1 -- 172.21.15.96:0/3068928137 learned_addr learned my addr 172.21.15.96:0/3068928137 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:40.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.391+0000 7f4873fff700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48740ff520 0x7f487419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.392+0000 7f48737fe700 1 -- 172.21.15.96:0/3068928137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48740ff520 msgr2=0x7f487419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.392+0000 7f48737fe700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48740ff520 0x7f487419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.392+0000 7f48737fe700 1 -- 172.21.15.96:0/3068928137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486800a700 con 0x7f48740ffed0 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.392+0000 7f4873fff700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48740ff520 0x7f487419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.392+0000 7f48737fe700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 0x7f487419cde0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f486800d010 tx=0x7f486800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.392+0000 7f48717fa700 1 -- 172.21.15.96:0/3068928137 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4868004af0 con 0x7f48740ffed0 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.392+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4874196c60 con 0x7f48740ffed0 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.392+0000 7f48717fa700 1 -- 172.21.15.96:0/3068928137 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4868004c50 con 0x7f48740ffed0 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.393+0000 7f48717fa700 1 -- 172.21.15.96:0/3068928137 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4868016880 con 0x7f48740ffed0 2024-04-22T15:58:40.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.393+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4874197150 con 0x7f48740ffed0 2024-04-22T15:58:40.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.393+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4874109930 con 0x7f48740ffed0 2024-04-22T15:58:40.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.394+0000 7f48717fa700 1 -- 172.21.15.96:0/3068928137 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f486800ec10 con 0x7f48740ffed0 2024-04-22T15:58:40.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.394+0000 7f48717fa700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f485c06ecc0 0x7f485c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:40.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.394+0000 7f48717fa700 1 -- 172.21.15.96:0/3068928137 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4868012070 con 0x7f48740ffed0 2024-04-22T15:58:40.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.394+0000 7f4873fff700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f485c06ecc0 0x7f485c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:40.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.395+0000 7f4873fff700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f485c06ecc0 0x7f485c071180 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4864005950 tx=0x7f48640058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:40.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.399+0000 7f48717fa700 1 -- 172.21.15.96:0/3068928137 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f486805d3d0 con 0x7f48740ffed0 2024-04-22T15:58:40.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.615+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4874197ea0 con 0x7f485c06ecc0 2024-04-22T15:58:40.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.626+0000 7f48717fa700 1 -- 172.21.15.96:0/3068928137 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4874197ea0 con 0x7f485c06ecc0 2024-04-22T15:58:40.629 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (37m) 14m ago 48m 24.8M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (49m) 14m ago 48m 10.2M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2d1852e471fb 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (47m) 14m ago 47m 9.94M - 18.2.2-1237-g646b8c7d 9900ed1a6b81 2eb4efbc8f49 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (34m) 14m ago 48m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (34m) 14m ago 47m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (36m) 14m ago 48m 82.4M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (30m) 14m ago 42m 21.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (30m) 14m ago 43m 20.4M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (30m) 14m ago 43m 20.1M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:58:40.630 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (15m) 14m ago 42m 11.7M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (39m) 14m ago 50m 514M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (38m) 14m ago 47m 459M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (35m) 14m ago 50m 72.1M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (35m) 14m ago 47m 52.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (38m) 14m ago 48m 17.2M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (38m) 14m ago 47m 16.5M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (34m) 14m ago 46m 99.0M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (33m) 14m ago 46m 101M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (32m) 14m ago 45m 80.1M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (32m) 14m ago 45m 103M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (31m) 14m ago 44m 88.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (31m) 14m ago 44m 88.4M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:58:40.631 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (37m) 14m ago 48m 90.9M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:58:40.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f485c06ecc0 msgr2=0x7f485c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f485c06ecc0 0x7f485c071180 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4864005950 tx=0x7f48640058e0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 msgr2=0x7f487419cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 0x7f487419cde0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f486800d010 tx=0x7f486800cba0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 shutdown_connections 2024-04-22T15:58:40.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f485c06ecc0 0x7f485c071180 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48740ff520 0x7f487419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 --2- 172.21.15.96:0/3068928137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48740ffed0 0x7f487419cde0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 >> 172.21.15.96:0/3068928137 conn(0x7f4874075f00 msgr2=0x7f48740fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:40.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 shutdown_connections 2024-04-22T15:58:40.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.629+0000 7f487a9da700 1 -- 172.21.15.96:0/3068928137 wait complete. 2024-04-22T15:58:40.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.755+0000 7f23c8136700 1 -- 172.21.15.96:0/2364459987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c0104b70 msgr2=0x7f23c0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.755+0000 7f23c8136700 1 --2- 172.21.15.96:0/2364459987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c0104b70 0x7f23c0104f50 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009910 tx=0x7f23bc009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:40.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.755+0000 7f23c8136700 1 -- 172.21.15.96:0/2364459987 shutdown_connections 2024-04-22T15:58:40.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.755+0000 7f23c8136700 1 --2- 172.21.15.96:0/2364459987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23c01002a0 0x7f23c0100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.755+0000 7f23c8136700 1 --2- 172.21.15.96:0/2364459987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c0104b70 0x7f23c0104f50 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.755+0000 7f23c8136700 1 -- 172.21.15.96:0/2364459987 >> 172.21.15.96:0/2364459987 conn(0x7f23c0075f00 msgr2=0x7f23c00fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:40.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.755+0000 7f23c8136700 1 -- 172.21.15.96:0/2364459987 shutdown_connections 2024-04-22T15:58:40.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.755+0000 7f23c8136700 1 -- 172.21.15.96:0/2364459987 wait complete. 2024-04-22T15:58:40.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.756+0000 7f23c8136700 1 Processor -- start 2024-04-22T15:58:40.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.756+0000 7f23c8136700 1 -- start start 2024-04-22T15:58:40.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.756+0000 7f23c8136700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c01002a0 0x7f23c019b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:40.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.756+0000 7f23c8136700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23c0104b70 0x7f23c019bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.756+0000 7f23c8136700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c019c290 con 0x7f23c01002a0 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.756+0000 7f23c8136700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0195800 con 0x7f23c0104b70 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.756+0000 7f23c5ed2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c01002a0 0x7f23c019b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.756+0000 7f23c5ed2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c01002a0 0x7f23c019b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34228/0 (socket says 172.21.15.96:34228) 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.756+0000 7f23c5ed2700 1 -- 172.21.15.96:0/1681884352 learned_addr learned my addr 172.21.15.96:0/1681884352 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23c56d1700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23c0104b70 0x7f23c019bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23c5ed2700 1 -- 172.21.15.96:0/1681884352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23c0104b70 msgr2=0x7f23c019bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23c5ed2700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23c0104b70 0x7f23c019bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23c5ed2700 1 -- 172.21.15.96:0/1681884352 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0095f0 con 0x7f23c01002a0 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23c5ed2700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c01002a0 0x7f23c019b6c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0092e0 tx=0x7f23bc005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23b6ffd700 1 -- 172.21.15.96:0/1681884352 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc005e20 con 0x7f23c01002a0 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23b6ffd700 1 -- 172.21.15.96:0/1681884352 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23bc00bea0 con 0x7f23c01002a0 2024-04-22T15:58:40.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23b6ffd700 1 -- 172.21.15.96:0/1681884352 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc00f460 con 0x7f23c01002a0 2024-04-22T15:58:40.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c0195a80 con 0x7f23c01002a0 2024-04-22T15:58:40.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.757+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c0195f70 con 0x7f23c01002a0 2024-04-22T15:58:40.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.758+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c004ebb0 con 0x7f23c01002a0 2024-04-22T15:58:40.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.759+0000 7f23b6ffd700 1 -- 172.21.15.96:0/1681884352 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f23bc020810 con 0x7f23c01002a0 2024-04-22T15:58:40.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.760+0000 7f23b6ffd700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f23ac06ed10 0x7f23ac0711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:40.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.760+0000 7f23b6ffd700 1 -- 172.21.15.96:0/1681884352 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f23bc08ff00 con 0x7f23c01002a0 2024-04-22T15:58:40.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.760+0000 7f23c56d1700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f23ac06ed10 0x7f23ac0711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:40.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.761+0000 7f23c56d1700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f23ac06ed10 0x7f23ac0711d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f23b0005950 tx=0x7f23b00058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:40.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:40.765+0000 7f23b6ffd700 1 -- 172.21.15.96:0/1681884352 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f23bc05c770 con 0x7f23c01002a0 2024-04-22T15:58:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-22T15:58:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:40 smithi096 ceph-mon[112443]: Standby daemon mds.cephfs.smithi096.dymqlv assigned to filesystem cephfs as rank 0 2024-04-22T15:58:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:40 smithi096 ceph-mon[112443]: Standby daemon mds.cephfs.smithi096.drkkdi assigned to filesystem cephfs as rank 1 2024-04-22T15:58:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:40 smithi096 ceph-mon[112443]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:58:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:40 smithi096 ceph-mon[112443]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:58:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:40 smithi096 ceph-mon[112443]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-22T15:58:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:40 smithi096 ceph-mon[112443]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:replay,1=cephfs.smithi096.drkkdi=up:replay} 2 up:standby 2024-04-22T15:58:40.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:40 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-22T15:58:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-22T15:58:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:40 smithi196 ceph-mon[81815]: Standby daemon mds.cephfs.smithi096.dymqlv assigned to filesystem cephfs as rank 0 2024-04-22T15:58:41.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:40 smithi196 ceph-mon[81815]: Standby daemon mds.cephfs.smithi096.drkkdi assigned to filesystem cephfs as rank 1 2024-04-22T15:58:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:40 smithi196 ceph-mon[81815]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:58:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:40 smithi196 ceph-mon[81815]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T15:58:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:40 smithi196 ceph-mon[81815]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-22T15:58:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:40 smithi196 ceph-mon[81815]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:replay,1=cephfs.smithi096.drkkdi=up:replay} 2 up:standby 2024-04-22T15:58:41.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:40 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-22T15:58:41.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.066+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f23c010cd40 con 0x7f23c01002a0 2024-04-22T15:58:41.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.066+0000 7f23b6ffd700 1 -- 172.21.15.96:0/1681884352 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f23bc05c300 con 0x7f23c01002a0 2024-04-22T15:58:41.069 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:58:41.069 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:58:41.069 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:58:41.070 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:58:41.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.068+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f23ac06ed10 msgr2=0x7f23ac0711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.068+0000 7f23c8136700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f23ac06ed10 0x7f23ac0711d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f23b0005950 tx=0x7f23b00058e0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.068+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c01002a0 msgr2=0x7f23c019b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.068+0000 7f23c8136700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c01002a0 0x7f23c019b6c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0092e0 tx=0x7f23bc005790 comp rx=0 tx=0).stop 2024-04-22T15:58:41.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.069+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 shutdown_connections 2024-04-22T15:58:41.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.069+0000 7f23c8136700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f23ac06ed10 0x7f23ac0711d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.069+0000 7f23c8136700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23c01002a0 0x7f23c019b6c0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.069+0000 7f23c8136700 1 --2- 172.21.15.96:0/1681884352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23c0104b70 0x7f23c019bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.069+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 >> 172.21.15.96:0/1681884352 conn(0x7f23c0075f00 msgr2=0x7f23c00fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:41.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.069+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 shutdown_connections 2024-04-22T15:58:41.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.069+0000 7f23c8136700 1 -- 172.21.15.96:0/1681884352 wait complete. 2024-04-22T15:58:41.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.203+0000 7fcb02f46700 1 -- 172.21.15.96:0/1741318042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 msgr2=0x7fcafc0ff900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.203+0000 7fcb02f46700 1 --2- 172.21.15.96:0/1741318042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 0x7fcafc0ff900 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0009910 tx=0x7fcaf0009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:41.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.203+0000 7fcb02f46700 1 -- 172.21.15.96:0/1741318042 shutdown_connections 2024-04-22T15:58:41.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.203+0000 7fcb02f46700 1 --2- 172.21.15.96:0/1741318042 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc0ffed0 0x7fcafc103f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.203+0000 7fcb02f46700 1 --2- 172.21.15.96:0/1741318042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 0x7fcafc0ff900 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.203+0000 7fcb02f46700 1 -- 172.21.15.96:0/1741318042 >> 172.21.15.96:0/1741318042 conn(0x7fcafc075f00 msgr2=0x7fcafc0fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:41.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.203+0000 7fcb02f46700 1 -- 172.21.15.96:0/1741318042 shutdown_connections 2024-04-22T15:58:41.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.203+0000 7fcb02f46700 1 -- 172.21.15.96:0/1741318042 wait complete. 2024-04-22T15:58:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcb02f46700 1 Processor -- start 2024-04-22T15:58:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcb02f46700 1 -- start start 2024-04-22T15:58:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcb02f46700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 0x7fcafc198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcb02f46700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc0ffed0 0x7fcafc1992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcb02f46700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc199980 con 0x7fcafc0ff520 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcb02f46700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc19d650 con 0x7fcafc0ffed0 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcb00ce2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 0x7fcafc198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcafbfff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc0ffed0 0x7fcafc1992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcb00ce2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 0x7fcafc198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34252/0 (socket says 172.21.15.96:34252) 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.204+0000 7fcb00ce2700 1 -- 172.21.15.96:0/3941393929 learned_addr learned my addr 172.21.15.96:0/3941393929 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.205+0000 7fcb00ce2700 1 -- 172.21.15.96:0/3941393929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc0ffed0 msgr2=0x7fcafc1992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.205+0000 7fcb00ce2700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc0ffed0 0x7fcafc1992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.205+0000 7fcb00ce2700 1 -- 172.21.15.96:0/3941393929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf00095f0 con 0x7fcafc0ff520 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.205+0000 7fcb00ce2700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 0x7fcafc198d60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0005f50 tx=0x7fcaf000b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.205+0000 7fcaf9ffb700 1 -- 172.21.15.96:0/3941393929 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaf001d070 con 0x7fcafc0ff520 2024-04-22T15:58:41.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.205+0000 7fcaf9ffb700 1 -- 172.21.15.96:0/3941393929 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcaf00052a0 con 0x7fcafc0ff520 2024-04-22T15:58:41.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.205+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc19d8d0 con 0x7fcafc0ff520 2024-04-22T15:58:41.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.205+0000 7fcaf9ffb700 1 -- 172.21.15.96:0/3941393929 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaf0017400 con 0x7fcafc0ff520 2024-04-22T15:58:41.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.206+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcafc19ddc0 con 0x7fcafc0ff520 2024-04-22T15:58:41.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.207+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcafc100a00 con 0x7fcafc0ff520 2024-04-22T15:58:41.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.207+0000 7fcaf9ffb700 1 -- 172.21.15.96:0/3941393929 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcaf0017560 con 0x7fcafc0ff520 2024-04-22T15:58:41.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.207+0000 7fcaf9ffb700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecd0 0x7fcaec071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:41.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.208+0000 7fcaf9ffb700 1 -- 172.21.15.96:0/3941393929 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcaf00902f0 con 0x7fcafc0ff520 2024-04-22T15:58:41.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.208+0000 7fcafbfff700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecd0 0x7fcaec071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:41.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.208+0000 7fcafbfff700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecd0 0x7fcaec071190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcafc19a2c0 tx=0x7fcae8009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.214+0000 7fcaf9ffb700 1 -- 172.21.15.96:0/3941393929 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcaf00590c0 con 0x7fcafc0ff520 2024-04-22T15:58:41.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.493+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcafc10c530 con 0x7fcafc0ff520 2024-04-22T15:58:41.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.494+0000 7fcaf9ffb700 1 -- 172.21.15.96:0/3941393929 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1959 (secure 0 0 0) 0x7fcaf005c780 con 0x7fcafc0ff520 2024-04-22T15:58:41.497 INFO:teuthology.orchestra.run.smithi096.stdout:e34 2024-04-22T15:58:41.497 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:58:41.497 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:58:41.497 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T15:58:41.497 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:58:41.497 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:58:41.497 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 34 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:58:39.714905+0000 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T15:58:41.498 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=34416,1=34420} 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:58:41.499 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:34416} state up:replay seq 1 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:41.500 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{1:34420} state up:resolve seq 459 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:41.500 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:58:41.500 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:58:41.500 INFO:teuthology.orchestra.run.smithi096.stdout:Standby daemons: 2024-04-22T15:58:41.500 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T15:58:41.500 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:41.500 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{-1:44757} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecd0 msgr2=0x7fcaec071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecd0 0x7fcaec071190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcafc19a2c0 tx=0x7fcae8009260 comp rx=0 tx=0).stop 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 msgr2=0x7fcafc198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 0x7fcafc198d60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0005f50 tx=0x7fcaf000b5a0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 shutdown_connections 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fcaec06ecd0 0x7fcaec071190 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcafc0ff520 0x7fcafc198d60 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 --2- 172.21.15.96:0/3941393929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc0ffed0 0x7fcafc1992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 >> 172.21.15.96:0/3941393929 conn(0x7fcafc075f00 msgr2=0x7fcafc102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 shutdown_connections 2024-04-22T15:58:41.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.497+0000 7fcb02f46700 1 -- 172.21.15.96:0/3941393929 wait complete. 2024-04-22T15:58:41.502 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 34 2024-04-22T15:58:41.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.623+0000 7f27c801a700 1 -- 172.21.15.96:0/1533443396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0101df0 msgr2=0x7f27c01021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.623+0000 7f27c801a700 1 --2- 172.21.15.96:0/1533443396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0101df0 0x7f27c01021d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f27bc009910 tx=0x7f27bc009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:41.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.623+0000 7f27c801a700 1 -- 172.21.15.96:0/1533443396 shutdown_connections 2024-04-22T15:58:41.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.623+0000 7f27c801a700 1 --2- 172.21.15.96:0/1533443396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27c0102710 0x7f27c010ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.623+0000 7f27c801a700 1 --2- 172.21.15.96:0/1533443396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0101df0 0x7f27c01021d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.623+0000 7f27c801a700 1 -- 172.21.15.96:0/1533443396 >> 172.21.15.96:0/1533443396 conn(0x7f27c00fb5e0 msgr2=0x7f27c00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.623+0000 7f27c801a700 1 -- 172.21.15.96:0/1533443396 shutdown_connections 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.623+0000 7f27c801a700 1 -- 172.21.15.96:0/1533443396 wait complete. 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c801a700 1 Processor -- start 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c801a700 1 -- start start 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c801a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27c0101df0 0x7f27c019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c801a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0102710 0x7f27c019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c801a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c019d370 con 0x7f27c0102710 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c801a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c0196950 con 0x7f27c0101df0 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c55b5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0102710 0x7f27c019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c55b5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0102710 0x7f27c019cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34278/0 (socket says 172.21.15.96:34278) 2024-04-22T15:58:41.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c55b5700 1 -- 172.21.15.96:0/2223955463 learned_addr learned my addr 172.21.15.96:0/2223955463 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.624+0000 7f27c5db6700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27c0101df0 0x7f27c019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.625+0000 7f27c55b5700 1 -- 172.21.15.96:0/2223955463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27c0101df0 msgr2=0x7f27c019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.625+0000 7f27c55b5700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27c0101df0 0x7f27c019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.625+0000 7f27c55b5700 1 -- 172.21.15.96:0/2223955463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc0095f0 con 0x7f27c0102710 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.625+0000 7f27c55b5700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0102710 0x7f27c019cd50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f27b000d900 tx=0x7f27b000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.625+0000 7f27b6ffd700 1 -- 172.21.15.96:0/2223955463 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27b0009900 con 0x7f27c0102710 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.625+0000 7f27b6ffd700 1 -- 172.21.15.96:0/2223955463 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27b0010460 con 0x7f27c0102710 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.625+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c0196c30 con 0x7f27c0102710 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.625+0000 7f27b6ffd700 1 -- 172.21.15.96:0/2223955463 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27b000f5f0 con 0x7f27c0102710 2024-04-22T15:58:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.625+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c0197180 con 0x7f27c0102710 2024-04-22T15:58:41.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.627+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c0105510 con 0x7f27c0102710 2024-04-22T15:58:41.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.627+0000 7f27b6ffd700 1 -- 172.21.15.96:0/2223955463 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f27b00105d0 con 0x7f27c0102710 2024-04-22T15:58:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.627+0000 7f27b6ffd700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27ac06ecc0 0x7f27ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.627+0000 7f27b6ffd700 1 -- 172.21.15.96:0/2223955463 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f27b008e960 con 0x7f27c0102710 2024-04-22T15:58:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.627+0000 7f27c5db6700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27ac06ecc0 0x7f27ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.628+0000 7f27c5db6700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27ac06ecc0 0x7f27ac071180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f27bc000c00 tx=0x7f27bc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:41.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.636+0000 7f27b6ffd700 1 -- 172.21.15.96:0/2223955463 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f27b005b270 con 0x7f27c0102710 2024-04-22T15:58:41.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.935+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27c0197e80 con 0x7f27ac06ecc0 2024-04-22T15:58:41.938 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='client.35992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:41.938 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='client.35996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:41.938 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: pgmap v1107: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 1 op/s 2024-04-22T15:58:41.938 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:41.938 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-22T15:58:41.938 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:resolve 2024-04-22T15:58:41.938 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:replay,1=cephfs.smithi096.drkkdi=up:resolve} 2 up:standby 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all rgw 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1681884352' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.939 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:41 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3941393929' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.936+0000 7f27b6ffd700 1 -- 172.21.15.96:0/2223955463 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f27c0197e80 con 0x7f27ac06ecc0 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": true, 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [ 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr", 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "osd", 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "crash", 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "mon", 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "mds" 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-04-22T15:58:41.940 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T15:58:41.941 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Currently upgrading ceph-exporter daemons", 2024-04-22T15:58:41.941 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:58:41.941 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:58:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27ac06ecc0 msgr2=0x7f27ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27ac06ecc0 0x7f27ac071180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f27bc000c00 tx=0x7f27bc01a040 comp rx=0 tx=0).stop 2024-04-22T15:58:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0102710 msgr2=0x7f27c019cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0102710 0x7f27c019cd50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f27b000d900 tx=0x7f27b000dc10 comp rx=0 tx=0).stop 2024-04-22T15:58:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 shutdown_connections 2024-04-22T15:58:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f27ac06ecc0 0x7f27ac071180 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27c0101df0 0x7f27c019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 --2- 172.21.15.96:0/2223955463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27c0102710 0x7f27c019cd50 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 >> 172.21.15.96:0/2223955463 conn(0x7f27c00fb5e0 msgr2=0x7f27c01030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:41.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 shutdown_connections 2024-04-22T15:58:41.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:41.940+0000 7f27c801a700 1 -- 172.21.15.96:0/2223955463 wait complete. 2024-04-22T15:58:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='client.35992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='client.35996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: pgmap v1107: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 1 op/s 2024-04-22T15:58:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:resolve 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:replay,1=cephfs.smithi096.drkkdi=up:resolve} 2 up:standby 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all rgw 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1681884352' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:42.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:41 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3941393929' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:58:42.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.046+0000 7f7f47db2700 1 -- 172.21.15.96:0/3937009558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f40100e10 msgr2=0x7f7f401011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.046+0000 7f7f47db2700 1 --2- 172.21.15.96:0/3937009558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f40100e10 0x7f7f401011f0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009910 tx=0x7f7f3c009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.047+0000 7f7f47db2700 1 -- 172.21.15.96:0/3937009558 shutdown_connections 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.047+0000 7f7f47db2700 1 --2- 172.21.15.96:0/3937009558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f401017c0 0x7f7f40105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.047+0000 7f7f47db2700 1 --2- 172.21.15.96:0/3937009558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f40100e10 0x7f7f401011f0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.047+0000 7f7f47db2700 1 -- 172.21.15.96:0/3937009558 >> 172.21.15.96:0/3937009558 conn(0x7f7f400fc6e0 msgr2=0x7f7f400feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.047+0000 7f7f47db2700 1 -- 172.21.15.96:0/3937009558 shutdown_connections 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.047+0000 7f7f47db2700 1 -- 172.21.15.96:0/3937009558 wait complete. 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.047+0000 7f7f47db2700 1 Processor -- start 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.047+0000 7f7f47db2700 1 -- start start 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f47db2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f40100e10 0x7f7f4019b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f47db2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f401017c0 0x7f7f4019bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f47db2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4019c260 con 0x7f7f401017c0 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f47db2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f401969a0 con 0x7f7f40100e10 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f4534d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f401017c0 0x7f7f4019bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f4534d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f401017c0 0x7f7f4019bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34296/0 (socket says 172.21.15.96:34296) 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f4534d700 1 -- 172.21.15.96:0/624272194 learned_addr learned my addr 172.21.15.96:0/624272194 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f45b4e700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f40100e10 0x7f7f4019b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f4534d700 1 -- 172.21.15.96:0/624272194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f40100e10 msgr2=0x7f7f4019b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f4534d700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f40100e10 0x7f7f4019b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f4534d700 1 -- 172.21.15.96:0/624272194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c0095f0 con 0x7f7f401017c0 2024-04-22T15:58:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.048+0000 7f7f4534d700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f401017c0 0x7f7f4019bb80 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000c8f0 tx=0x7f7f3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.049+0000 7f7f36ffd700 1 -- 172.21.15.96:0/624272194 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f30007a50 con 0x7f7f401017c0 2024-04-22T15:58:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.049+0000 7f7f36ffd700 1 -- 172.21.15.96:0/624272194 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f30007bb0 con 0x7f7f401017c0 2024-04-22T15:58:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.049+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f40196c80 con 0x7f7f401017c0 2024-04-22T15:58:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.049+0000 7f7f36ffd700 1 -- 172.21.15.96:0/624272194 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f30018630 con 0x7f7f401017c0 2024-04-22T15:58:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.049+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f40197180 con 0x7f7f401017c0 2024-04-22T15:58:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.050+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f401091b0 con 0x7f7f401017c0 2024-04-22T15:58:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.051+0000 7f7f36ffd700 1 -- 172.21.15.96:0/624272194 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7f30007d20 con 0x7f7f401017c0 2024-04-22T15:58:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.051+0000 7f7f36ffd700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f2c06ecc0 0x7f7f2c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.051+0000 7f7f36ffd700 1 -- 172.21.15.96:0/624272194 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7f3008eea0 con 0x7f7f401017c0 2024-04-22T15:58:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.051+0000 7f7f45b4e700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f2c06ecc0 0x7f7f2c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.052+0000 7f7f45b4e700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f2c06ecc0 0x7f7f2c071180 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c005b40 tx=0x7f7f3c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:42.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.056+0000 7f7f36ffd700 1 -- 172.21.15.96:0/624272194 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7f30057c00 con 0x7f7f401017c0 2024-04-22T15:58:42.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.355+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7f401022f0 con 0x7f7f401017c0 2024-04-22T15:58:42.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.355+0000 7f7f36ffd700 1 -- 172.21.15.96:0/624272194 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4367 (secure 0 0 0) 0x7f7f3005b2c0 con 0x7f7f401017c0 2024-04-22T15:58:42.358 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout: host smithi196 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi196/config 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7ddabvda:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout:Traceback (most recent call last): 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout: return _run_code(code, main_globals, None, 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout: exec(code, run_globals) 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5642, in 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 5630, in main 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 408, in _infer_config 2024-04-22T15:58:42.359 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 323, in _infer_fsid 2024-04-22T15:58:42.360 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 436, in _infer_image 2024-04-22T15:58:42.360 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 310, in _validate_fsid 2024-04-22T15:58:42.360 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/__main__.py", line 3380, in command_ceph_volume 2024-04-22T15:58:42.360 INFO:teuthology.orchestra.run.smithi096.stdout: File "/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/cephadm.6073b07ab069519e3286caa9b3ba4d63a482527cb6f564b205dd8e6cfafb5da0/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T15:58:42.360 INFO:teuthology.orchestra.run.smithi096.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb -e NODE_NAME=smithi196 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7ddabvda:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb inventory --format=json-pretty --filter-for-batch 2024-04-22T15:58:42.360 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:58:42.360 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs is degraded 2024-04-22T15:58:42.360 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:58:42.360 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:58:42.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.357+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f2c06ecc0 msgr2=0x7f7f2c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.357+0000 7f7f47db2700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f2c06ecc0 0x7f7f2c071180 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c005b40 tx=0x7f7f3c01a040 comp rx=0 tx=0).stop 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.357+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f401017c0 msgr2=0x7f7f4019bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.357+0000 7f7f47db2700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f401017c0 0x7f7f4019bb80 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000c8f0 tx=0x7f7f3000ccb0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.358+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 shutdown_connections 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.358+0000 7f7f47db2700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7f2c06ecc0 0x7f7f2c071180 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.358+0000 7f7f47db2700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f40100e10 0x7f7f4019b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.358+0000 7f7f47db2700 1 --2- 172.21.15.96:0/624272194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f401017c0 0x7f7f4019bb80 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.358+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 >> 172.21.15.96:0/624272194 conn(0x7f7f400fc6e0 msgr2=0x7f7f400fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.358+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 shutdown_connections 2024-04-22T15:58:42.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:58:42.358+0000 7f7f47db2700 1 -- 172.21.15.96:0/624272194 wait complete. 2024-04-22T15:58:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:42 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/624272194' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:58:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:58:42.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:42 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:42 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/624272194' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:58:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi096", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:58:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:42 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:43 smithi096 ceph-mon[112443]: from='client.36012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:43 smithi096 ceph-mon[112443]: pgmap v1108: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.0 MiB/s rd, 3 op/s 2024-04-22T15:58:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:43 smithi096 ceph-mon[112443]: Upgrade: Updating ceph-exporter.smithi096 (1/2) 2024-04-22T15:58:43.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:43 smithi096 ceph-mon[112443]: Deploying daemon ceph-exporter.smithi096 on smithi096 2024-04-22T15:58:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:43 smithi196 ceph-mon[81815]: from='client.36012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:43 smithi196 ceph-mon[81815]: pgmap v1108: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.0 MiB/s rd, 3 op/s 2024-04-22T15:58:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:43 smithi196 ceph-mon[81815]: Upgrade: Updating ceph-exporter.smithi096 (1/2) 2024-04-22T15:58:44.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:43 smithi196 ceph-mon[81815]: Deploying daemon ceph-exporter.smithi096 on smithi096 2024-04-22T15:58:45.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:45 smithi096 ceph-mon[112443]: pgmap v1109: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 7.0 MiB/s rd, 3 op/s 2024-04-22T15:58:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:45 smithi196 ceph-mon[81815]: pgmap v1109: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 7.0 MiB/s rd, 3 op/s 2024-04-22T15:58:47.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:47 smithi096 ceph-mon[112443]: pgmap v1110: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8 op/s 2024-04-22T15:58:47.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:47 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:47.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:47 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:47 smithi196 ceph-mon[81815]: pgmap v1110: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8 op/s 2024-04-22T15:58:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:47 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:48.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:47 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:49.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:49 smithi096 ceph-mon[112443]: Upgrade: Updating ceph-exporter.smithi196 (2/2) 2024-04-22T15:58:49.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:49 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:49 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:58:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:49 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:49 smithi096 ceph-mon[112443]: Deploying daemon ceph-exporter.smithi196 on smithi196 2024-04-22T15:58:49.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:49 smithi096 ceph-mon[112443]: pgmap v1111: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 7 op/s 2024-04-22T15:58:49.746 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:49 smithi196 ceph-mon[81815]: Upgrade: Updating ceph-exporter.smithi196 (2/2) 2024-04-22T15:58:49.746 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:49 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:49.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:49 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi196", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:58:49.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:49 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:49.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:49 smithi196 ceph-mon[81815]: Deploying daemon ceph-exporter.smithi196 on smithi196 2024-04-22T15:58:49.747 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:49 smithi196 ceph-mon[81815]: pgmap v1111: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 7 op/s 2024-04-22T15:58:50.603 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:50 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:resolve 2024-04-22T15:58:50.604 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:50 smithi196 ceph-mon[81815]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:resolve,1=cephfs.smithi096.drkkdi=up:resolve} 2 up:standby 2024-04-22T15:58:50.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:50 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:resolve 2024-04-22T15:58:50.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:50 smithi096 ceph-mon[112443]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:resolve,1=cephfs.smithi096.drkkdi=up:resolve} 2 up:standby 2024-04-22T15:58:51.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:51 smithi196 ceph-mon[81815]: reconnect by client.24309 192.168.0.1:0/310702855 after 0.000999978 2024-04-22T15:58:51.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:51 smithi196 ceph-mon[81815]: reconnect by client.24309 192.168.0.1:0/310702855 after 0.000999978 2024-04-22T15:58:51.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:51 smithi196 ceph-mon[81815]: reconnect by client.14544 192.168.0.1:0/4034229763 after 0.000999978 2024-04-22T15:58:51.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:51 smithi196 ceph-mon[81815]: reconnect by client.14544 192.168.0.1:0/4034229763 after 0.000999978 2024-04-22T15:58:51.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:51 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:reconnect 2024-04-22T15:58:51.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:51 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:reconnect 2024-04-22T15:58:51.510 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:51 smithi196 ceph-mon[81815]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:reconnect,1=cephfs.smithi096.drkkdi=up:reconnect} 2 up:standby 2024-04-22T15:58:51.511 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:51 smithi196 ceph-mon[81815]: pgmap v1112: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-04-22T15:58:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:51 smithi096 ceph-mon[112443]: reconnect by client.24309 192.168.0.1:0/310702855 after 0.000999978 2024-04-22T15:58:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:51 smithi096 ceph-mon[112443]: reconnect by client.24309 192.168.0.1:0/310702855 after 0.000999978 2024-04-22T15:58:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:51 smithi096 ceph-mon[112443]: reconnect by client.14544 192.168.0.1:0/4034229763 after 0.000999978 2024-04-22T15:58:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:51 smithi096 ceph-mon[112443]: reconnect by client.14544 192.168.0.1:0/4034229763 after 0.000999978 2024-04-22T15:58:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:51 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:reconnect 2024-04-22T15:58:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:51 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:reconnect 2024-04-22T15:58:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:51 smithi096 ceph-mon[112443]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:reconnect,1=cephfs.smithi096.drkkdi=up:reconnect} 2 up:standby 2024-04-22T15:58:51.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:51 smithi096 ceph-mon[112443]: pgmap v1112: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-04-22T15:58:52.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:52 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:rejoin 2024-04-22T15:58:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:52 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:rejoin 2024-04-22T15:58:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:52 smithi096 ceph-mon[112443]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:rejoin,1=cephfs.smithi096.drkkdi=up:rejoin} 2 up:standby 2024-04-22T15:58:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:52 smithi096 ceph-mon[112443]: daemon mds.cephfs.smithi096.drkkdi is now active in filesystem cephfs as rank 1 2024-04-22T15:58:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:52 smithi096 ceph-mon[112443]: daemon mds.cephfs.smithi096.dymqlv is now active in filesystem cephfs as rank 0 2024-04-22T15:58:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:52 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:52 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:52.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:52 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:52.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:52 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:rejoin 2024-04-22T15:58:52.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:52 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:rejoin 2024-04-22T15:58:52.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:52 smithi196 ceph-mon[81815]: fsmap cephfs:2/2 {0=cephfs.smithi096.dymqlv=up:rejoin,1=cephfs.smithi096.drkkdi=up:rejoin} 2 up:standby 2024-04-22T15:58:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:52 smithi196 ceph-mon[81815]: daemon mds.cephfs.smithi096.drkkdi is now active in filesystem cephfs as rank 1 2024-04-22T15:58:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:52 smithi196 ceph-mon[81815]: daemon mds.cephfs.smithi096.dymqlv is now active in filesystem cephfs as rank 0 2024-04-22T15:58:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:52 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:52 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:52.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:52 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:58:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:active 2024-04-22T15:58:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:active 2024-04-22T15:58:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi096.drkkdi=up:active} 2 up:standby 2024-04-22T15:58:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi096.drkkdi=up:active} 2 up:standby-replay 2024-04-22T15:58:53.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: pgmap v1113: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 6 op/s 2024-04-22T15:58:53.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:53.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:53.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:53.737 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:58:53.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] up:active 2024-04-22T15:58:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: mds.? [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] up:active 2024-04-22T15:58:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi096.drkkdi=up:active} 2 up:standby 2024-04-22T15:58:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi096.drkkdi=up:active} 2 up:standby-replay 2024-04-22T15:58:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: pgmap v1113: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 6 op/s 2024-04-22T15:58:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:53.760 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:54.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:54.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:54 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:54.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:54.963 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:54 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:55.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:55 smithi096 ceph-mon[112443]: pgmap v1114: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 5 op/s 2024-04-22T15:58:56.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:55 smithi196 ceph-mon[81815]: pgmap v1114: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 5 op/s 2024-04-22T15:58:57.483 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:57.484 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:57 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:57.484 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:57 smithi096 ceph-mon[112443]: pgmap v1115: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 5.1 KiB/s wr, 18 op/s 2024-04-22T15:58:57.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:57.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:57 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:57.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:57 smithi196 ceph-mon[81815]: pgmap v1115: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 5.1 KiB/s wr, 18 op/s 2024-04-22T15:58:58.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:58.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:58.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:58.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:58 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:58.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:58.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:58.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:58 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:59.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:59 smithi096 ceph-mon[112443]: pgmap v1116: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 5.2 KiB/s wr, 17 op/s 2024-04-22T15:58:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:59 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:59.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:58:59 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:59 smithi196 ceph-mon[81815]: pgmap v1116: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 5.2 KiB/s wr, 17 op/s 2024-04-22T15:58:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:59 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:58:59.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:58:59 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:01.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:01 smithi096 ceph-mon[112443]: pgmap v1117: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 5.2 KiB/s wr, 18 op/s 2024-04-22T15:59:02.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:01 smithi196 ceph-mon[81815]: pgmap v1117: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 5.2 KiB/s wr, 18 op/s 2024-04-22T15:59:03.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:03 smithi096 ceph-mon[112443]: pgmap v1118: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-04-22T15:59:03.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:03.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:03 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:03 smithi196 ceph-mon[81815]: pgmap v1118: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-04-22T15:59:04.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:04.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:03 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: pgmap v1119: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-04-22T15:59:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:05.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: pgmap v1120: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 6.1 KiB/s wr, 22 op/s 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:05.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:05 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-22T15:59:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: pgmap v1119: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-04-22T15:59:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:06.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: pgmap v1120: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 6.1 KiB/s wr, 22 op/s 2024-04-22T15:59:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-22T15:59:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:05 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-22T15:59:06.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: Upgrade: Setting filesystem cephfs Joinable 2024-04-22T15:59:06.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-22T15:59:06.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi096.drkkdi=up:active} 2 up:standby-replay 2024-04-22T15:59:06.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all ceph-exporter 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi096"}]: dispatch 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi096"}]': finished 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi196"}]: dispatch 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi196"}]': finished 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.987 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.988 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-22T15:59:06.989 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T15:59:06.990 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T15:59:06.991 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:06.992 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:06.992 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:06.992 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:06.992 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:06.992 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:06.992 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:06.992 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: Upgrade: Setting filesystem cephfs Joinable 2024-04-22T15:59:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-22T15:59:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: fsmap cephfs:2 {0=cephfs.smithi096.dymqlv=up:active,1=cephfs.smithi096.drkkdi=up:active} 2 up:standby-replay 2024-04-22T15:59:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all ceph-exporter 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi096"}]: dispatch 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi096"}]': finished 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi196"}]: dispatch 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi196"}]': finished 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:07.011 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T15:59:07.012 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:07.013 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:07.014 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:07.015 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:07 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all iscsi 2024-04-22T15:59:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:07 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all nfs 2024-04-22T15:59:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:07 smithi096 ceph-mon[112443]: Upgrade: Setting container_image for all nvmeof 2024-04-22T15:59:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:07 smithi096 ceph-mon[112443]: Upgrade: Finalizing container_image settings 2024-04-22T15:59:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:07 smithi096 ceph-mon[112443]: Upgrade: Complete! 2024-04-22T15:59:07.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:07 smithi096 ceph-mon[112443]: pgmap v1121: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 100 B/s wr, 8 op/s 2024-04-22T15:59:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:07 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all iscsi 2024-04-22T15:59:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:07 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all nfs 2024-04-22T15:59:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:07 smithi196 ceph-mon[81815]: Upgrade: Setting container_image for all nvmeof 2024-04-22T15:59:08.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:07 smithi196 ceph-mon[81815]: Upgrade: Finalizing container_image settings 2024-04-22T15:59:08.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:07 smithi196 ceph-mon[81815]: Upgrade: Complete! 2024-04-22T15:59:08.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:07 smithi196 ceph-mon[81815]: pgmap v1121: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 100 B/s wr, 8 op/s 2024-04-22T15:59:09.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:09 smithi096 ceph-mon[112443]: pgmap v1122: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 5 op/s 2024-04-22T15:59:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T15:59:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:09 smithi196 ceph-mon[81815]: pgmap v1122: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 5 op/s 2024-04-22T15:59:12.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:11 smithi196 ceph-mon[81815]: pgmap v1123: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.1 MiB/s rd, 4 op/s 2024-04-22T15:59:12.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:11 smithi096 ceph-mon[112443]: pgmap v1123: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.1 MiB/s rd, 4 op/s 2024-04-22T15:59:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.478+0000 7fb8b0646700 1 -- 172.21.15.96:0/3395174566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8102700 msgr2=0x7fb8a810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.478+0000 7fb8b0646700 1 --2- 172.21.15.96:0/3395174566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8102700 0x7fb8a810abf0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fb898009940 tx=0x7fb898009c50 comp rx=0 tx=0).stop 2024-04-22T15:59:12.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.478+0000 7fb8b0646700 1 -- 172.21.15.96:0/3395174566 shutdown_connections 2024-04-22T15:59:12.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.478+0000 7fb8b0646700 1 --2- 172.21.15.96:0/3395174566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8102700 0x7fb8a810abf0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:12.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.478+0000 7fb8b0646700 1 --2- 172.21.15.96:0/3395174566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8a8101de0 0x7fb8a81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:12.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.478+0000 7fb8b0646700 1 -- 172.21.15.96:0/3395174566 >> 172.21.15.96:0/3395174566 conn(0x7fb8a80fb610 msgr2=0x7fb8a80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:12.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.478+0000 7fb8b0646700 1 -- 172.21.15.96:0/3395174566 shutdown_connections 2024-04-22T15:59:12.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.478+0000 7fb8b0646700 1 -- 172.21.15.96:0/3395174566 wait complete. 2024-04-22T15:59:12.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8b0646700 1 Processor -- start 2024-04-22T15:59:12.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8b0646700 1 -- start start 2024-04-22T15:59:12.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8b0646700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8101de0 0x7fb8a819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:12.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8b0646700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8a8102700 0x7fb8a819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8b0646700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a819d380 con 0x7fb8a8101de0 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8b0646700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a8196960 con 0x7fb8a8102700 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8ae3e2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8101de0 0x7fb8a819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8ae3e2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8101de0 0x7fb8a819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39406/0 (socket says 172.21.15.96:39406) 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8ae3e2700 1 -- 172.21.15.96:0/3401605806 learned_addr learned my addr 172.21.15.96:0/3401605806 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8adbe1700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8a8102700 0x7fb8a819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8ae3e2700 1 -- 172.21.15.96:0/3401605806 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8a8102700 msgr2=0x7fb8a819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.479+0000 7fb8ae3e2700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8a8102700 0x7fb8a819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.480+0000 7fb8ae3e2700 1 -- 172.21.15.96:0/3401605806 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980095f0 con 0x7fb8a8101de0 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.480+0000 7fb8ae3e2700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8101de0 0x7fb8a819c820 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400d900 tx=0x7fb8a400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.480+0000 7fb89f7fe700 1 -- 172.21.15.96:0/3401605806 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8a40041f0 con 0x7fb8a8101de0 2024-04-22T15:59:12.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.480+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a8196c40 con 0x7fb8a8101de0 2024-04-22T15:59:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.480+0000 7fb89f7fe700 1 -- 172.21.15.96:0/3401605806 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8a4004cf0 con 0x7fb8a8101de0 2024-04-22T15:59:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.480+0000 7fb89f7fe700 1 -- 172.21.15.96:0/3401605806 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8a400b750 con 0x7fb8a8101de0 2024-04-22T15:59:12.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.480+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a8197160 con 0x7fb8a8101de0 2024-04-22T15:59:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.481+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a8105500 con 0x7fb8a8101de0 2024-04-22T15:59:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.481+0000 7fb89f7fe700 1 -- 172.21.15.96:0/3401605806 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb8a4004750 con 0x7fb8a8101de0 2024-04-22T15:59:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.482+0000 7fb89f7fe700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb89406ecc0 0x7fb894071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.482+0000 7fb89f7fe700 1 -- 172.21.15.96:0/3401605806 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb8a408edb0 con 0x7fb8a8101de0 2024-04-22T15:59:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.482+0000 7fb8adbe1700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb89406ecc0 0x7fb894071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.482+0000 7fb8adbe1700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb89406ecc0 0x7fb894071180 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb898009910 tx=0x7fb898005ab0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:12.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.487+0000 7fb89f7fe700 1 -- 172.21.15.96:0/3401605806 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb8a405b640 con 0x7fb8a8101de0 2024-04-22T15:59:12.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.709+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8a8197e90 con 0x7fb89406ecc0 2024-04-22T15:59:12.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.710+0000 7fb89f7fe700 1 -- 172.21.15.96:0/3401605806 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fb8a8197e90 con 0x7fb89406ecc0 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb89406ecc0 msgr2=0x7fb894071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb89406ecc0 0x7fb894071180 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb898009910 tx=0x7fb898005ab0 comp rx=0 tx=0).stop 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8101de0 msgr2=0x7fb8a819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8101de0 0x7fb8a819c820 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400d900 tx=0x7fb8a400dc10 comp rx=0 tx=0).stop 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 shutdown_connections 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fb89406ecc0 0x7fb894071180 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8a8101de0 0x7fb8a819c820 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 --2- 172.21.15.96:0/3401605806 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8a8102700 0x7fb8a819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 >> 172.21.15.96:0/3401605806 conn(0x7fb8a80fb610 msgr2=0x7fb8a81030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:12.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 shutdown_connections 2024-04-22T15:59:12.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:12.712+0000 7fb8b0646700 1 -- 172.21.15.96:0/3401605806 wait complete. 2024-04-22T15:59:13.380 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch ps' 2024-04-22T15:59:13.631 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:59:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:13 smithi196 ceph-mon[81815]: pgmap v1124: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 410 KiB/s rd, 3 op/s 2024-04-22T15:59:14.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:13 smithi196 ceph-mon[81815]: from='client.36022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:14.059 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:13 smithi096 ceph-mon[112443]: pgmap v1124: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 410 KiB/s rd, 3 op/s 2024-04-22T15:59:14.059 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:13 smithi096 ceph-mon[112443]: from='client.36022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:15.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.285+0000 7f61672ea700 1 -- 172.21.15.96:0/1506104655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 msgr2=0x7f6160105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.285+0000 7f61672ea700 1 --2- 172.21.15.96:0/1506104655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 0x7f6160105820 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f615c009910 tx=0x7f615c009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:15.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.285+0000 7f61672ea700 1 -- 172.21.15.96:0/1506104655 shutdown_connections 2024-04-22T15:59:15.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 --2- 172.21.15.96:0/1506104655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 0x7f6160105820 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 --2- 172.21.15.96:0/1506104655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6160100e20 0x7f6160101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 -- 172.21.15.96:0/1506104655 >> 172.21.15.96:0/1506104655 conn(0x7f61600fc6d0 msgr2=0x7f61600feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 -- 172.21.15.96:0/1506104655 shutdown_connections 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 -- 172.21.15.96:0/1506104655 wait complete. 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 Processor -- start 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 -- start start 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6160100e20 0x7f616019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 0x7f616019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616019d380 con 0x7f61601017d0 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.286+0000 7f61672ea700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6160196960 con 0x7f6160100e20 2024-04-22T15:59:15.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f6164885700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 0x7f616019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f6164885700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 0x7f616019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39422/0 (socket says 172.21.15.96:39422) 2024-04-22T15:59:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f6164885700 1 -- 172.21.15.96:0/3890117477 learned_addr learned my addr 172.21.15.96:0/3890117477 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:59:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f6165086700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6160100e20 0x7f616019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f6164885700 1 -- 172.21.15.96:0/3890117477 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6160100e20 msgr2=0x7f616019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f6164885700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6160100e20 0x7f616019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f6164885700 1 -- 172.21.15.96:0/3890117477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c0095f0 con 0x7f61601017d0 2024-04-22T15:59:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f6164885700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 0x7f616019cd60 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f615c00bd70 tx=0x7f615c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f61567fc700 1 -- 172.21.15.96:0/3890117477 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f615c01d070 con 0x7f61601017d0 2024-04-22T15:59:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f61567fc700 1 -- 172.21.15.96:0/3890117477 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f615c0056e0 con 0x7f61601017d0 2024-04-22T15:59:15.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.287+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6160196be0 con 0x7f61601017d0 2024-04-22T15:59:15.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.288+0000 7f61567fc700 1 -- 172.21.15.96:0/3890117477 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f615c00fe80 con 0x7f61601017d0 2024-04-22T15:59:15.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.288+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61601970d0 con 0x7f61601017d0 2024-04-22T15:59:15.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.289+0000 7f61567fc700 1 -- 172.21.15.96:0/3890117477 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f615c00f460 con 0x7f61601017d0 2024-04-22T15:59:15.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.289+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6160109220 con 0x7f61601017d0 2024-04-22T15:59:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.289+0000 7f61567fc700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f614c06ead0 0x7f614c070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.289+0000 7f61567fc700 1 -- 172.21.15.96:0/3890117477 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f615c090280 con 0x7f61601017d0 2024-04-22T15:59:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.289+0000 7f6165086700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f614c06ead0 0x7f614c070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.290+0000 7f6165086700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f614c06ead0 0x7f614c070f90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6150005950 tx=0x7f61500058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.294+0000 7f61567fc700 1 -- 172.21.15.96:0/3890117477 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f615c05cb90 con 0x7f61601017d0 2024-04-22T15:59:15.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.502+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f616004ec20 con 0x7f614c06ead0 2024-04-22T15:59:15.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.512+0000 7f61567fc700 1 -- 172.21.15.96:0/3890117477 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f616004ec20 con 0x7f614c06ead0 2024-04-22T15:59:15.515 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:59:15.515 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (37m) 17s ago 49m 24.2M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (29s) 17s ago 49m 7801k - 19.0.0-3223-g5883d494 3d983c26e278 348dbb003759 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (23s) 19s ago 48m 6903k - 19.0.0-3223-g5883d494 3d983c26e278 4d699b8512db 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (35m) 17s ago 49m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (35m) 19s ago 48m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (37m) 17s ago 48m 82.1M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (31m) 17s ago 43m 25.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (31m) 17s ago 43m 179M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (30m) 19s ago 43m 173M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (15m) 19s ago 43m 20.3M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (40m) 17s ago 50m 519M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (39m) 19s ago 48m 460M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (36m) 17s ago 50m 87.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (35m) 19s ago 48m 59.9M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T15:59:15.516 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (38m) 17s ago 49m 16.7M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T15:59:15.517 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (38m) 19s ago 48m 17.1M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T15:59:15.517 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (34m) 17s ago 47m 106M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T15:59:15.517 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (34m) 17s ago 46m 116M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T15:59:15.517 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (33m) 17s ago 46m 88.3M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T15:59:15.517 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (32m) 19s ago 45m 114M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T15:59:15.517 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (32m) 19s ago 45m 93.7M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T15:59:15.517 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (31m) 19s ago 45m 95.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T15:59:15.517 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (38m) 17s ago 48m 94.5M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T15:59:15.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.514+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f614c06ead0 msgr2=0x7f614c070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.514+0000 7f61672ea700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f614c06ead0 0x7f614c070f90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6150005950 tx=0x7f61500058e0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.514+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 msgr2=0x7f616019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.514+0000 7f61672ea700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 0x7f616019cd60 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f615c00bd70 tx=0x7f615c00be50 comp rx=0 tx=0).stop 2024-04-22T15:59:15.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.515+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 shutdown_connections 2024-04-22T15:59:15.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.515+0000 7f61672ea700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f614c06ead0 0x7f614c070f90 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.515+0000 7f61672ea700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6160100e20 0x7f616019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.515+0000 7f61672ea700 1 --2- 172.21.15.96:0/3890117477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61601017d0 0x7f616019cd60 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.515+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 >> 172.21.15.96:0/3890117477 conn(0x7f61600fc6d0 msgr2=0x7f61600fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:15.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.515+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 shutdown_connections 2024-04-22T15:59:15.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:15.515+0000 7f61672ea700 1 -- 172.21.15.96:0/3890117477 wait complete. 2024-04-22T15:59:16.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:15 smithi196 ceph-mon[81815]: pgmap v1125: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 410 KiB/s rd, 4 op/s 2024-04-22T15:59:16.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:15 smithi096 ceph-mon[112443]: pgmap v1125: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 410 KiB/s rd, 4 op/s 2024-04-22T15:59:16.150 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch upgrade status' 2024-04-22T15:59:16.355 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:59:17.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:16 smithi196 ceph-mon[81815]: from='client.36026 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:17.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:16 smithi196 ceph-mon[81815]: pgmap v1126: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.6 KiB/s rd, 4 op/s 2024-04-22T15:59:17.090 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:16 smithi096 ceph-mon[112443]: from='client.36026 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:17.091 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:16 smithi096 ceph-mon[112443]: pgmap v1126: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.6 KiB/s rd, 4 op/s 2024-04-22T15:59:17.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.948+0000 7f121325b700 1 -- 172.21.15.96:0/2319796986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 msgr2=0x7f120c1011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:17.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.948+0000 7f121325b700 1 --2- 172.21.15.96:0/2319796986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 0x7f120c1011f0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f1200009910 tx=0x7f1200009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:17.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.948+0000 7f121325b700 1 -- 172.21.15.96:0/2319796986 shutdown_connections 2024-04-22T15:59:17.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.948+0000 7f121325b700 1 --2- 172.21.15.96:0/2319796986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f120c1017c0 0x7f120c105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.948+0000 7f121325b700 1 --2- 172.21.15.96:0/2319796986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 0x7f120c1011f0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.948+0000 7f121325b700 1 -- 172.21.15.96:0/2319796986 >> 172.21.15.96:0/2319796986 conn(0x7f120c0fc6e0 msgr2=0x7f120c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.948+0000 7f121325b700 1 -- 172.21.15.96:0/2319796986 shutdown_connections 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.948+0000 7f121325b700 1 -- 172.21.15.96:0/2319796986 wait complete. 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f121325b700 1 Processor -- start 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f121325b700 1 -- start start 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f121325b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 0x7f120c198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f121325b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f120c1017c0 0x7f120c199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f121325b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c199a10 con 0x7f120c100e10 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f121325b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c19d6e0 con 0x7f120c1017c0 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f1210ff7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 0x7f120c198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f1210ff7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 0x7f120c198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39432/0 (socket says 172.21.15.96:39432) 2024-04-22T15:59:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f1210ff7700 1 -- 172.21.15.96:0/3710544626 learned_addr learned my addr 172.21.15.96:0/3710544626 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f120bfff700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f120c1017c0 0x7f120c199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f1210ff7700 1 -- 172.21.15.96:0/3710544626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f120c1017c0 msgr2=0x7f120c199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f1210ff7700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f120c1017c0 0x7f120c199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.949+0000 7f1210ff7700 1 -- 172.21.15.96:0/3710544626 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12000095f0 con 0x7f120c100e10 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.950+0000 7f120bfff700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f120c1017c0 0x7f120c199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.950+0000 7f1210ff7700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 0x7f120c198df0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1200006010 tx=0x7f120000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.950+0000 7f1209ffb700 1 -- 172.21.15.96:0/3710544626 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f120001d070 con 0x7f120c100e10 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.950+0000 7f1209ffb700 1 -- 172.21.15.96:0/3710544626 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f120000f460 con 0x7f120c100e10 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.950+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120c19d960 con 0x7f120c100e10 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.950+0000 7f1209ffb700 1 -- 172.21.15.96:0/3710544626 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1200017630 con 0x7f120c100e10 2024-04-22T15:59:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.950+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120c19de50 con 0x7f120c100e10 2024-04-22T15:59:17.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.951+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120c1022f0 con 0x7f120c100e10 2024-04-22T15:59:17.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.951+0000 7f1209ffb700 1 -- 172.21.15.96:0/3710544626 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f12000178a0 con 0x7f120c100e10 2024-04-22T15:59:17.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.952+0000 7f1209ffb700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f11fc06ecc0 0x7f11fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:17.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.952+0000 7f1209ffb700 1 -- 172.21.15.96:0/3710544626 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1200090350 con 0x7f120c100e10 2024-04-22T15:59:17.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.952+0000 7f120bfff700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f11fc06ecc0 0x7f11fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:17.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.952+0000 7f120bfff700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f11fc06ecc0 0x7f11fc071180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f120c19a350 tx=0x7f11f8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:17.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:17.957+0000 7f1209ffb700 1 -- 172.21.15.96:0/3710544626 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1200059170 con 0x7f120c100e10 2024-04-22T15:59:18.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.176+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f120c10de50 con 0x7f11fc06ecc0 2024-04-22T15:59:18.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.177+0000 7f1209ffb700 1 -- 172.21.15.96:0/3710544626 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f120c10de50 con 0x7f11fc06ecc0 2024-04-22T15:59:18.180 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:59:18.180 INFO:teuthology.orchestra.run.smithi096.stdout: "target_image": null, 2024-04-22T15:59:18.181 INFO:teuthology.orchestra.run.smithi096.stdout: "in_progress": false, 2024-04-22T15:59:18.181 INFO:teuthology.orchestra.run.smithi096.stdout: "which": "", 2024-04-22T15:59:18.181 INFO:teuthology.orchestra.run.smithi096.stdout: "services_complete": [], 2024-04-22T15:59:18.181 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": null, 2024-04-22T15:59:18.181 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "", 2024-04-22T15:59:18.181 INFO:teuthology.orchestra.run.smithi096.stdout: "is_paused": false 2024-04-22T15:59:18.181 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f11fc06ecc0 msgr2=0x7f11fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f11fc06ecc0 0x7f11fc071180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f120c19a350 tx=0x7f11f8008040 comp rx=0 tx=0).stop 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 msgr2=0x7f120c198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 0x7f120c198df0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1200006010 tx=0x7f120000be20 comp rx=0 tx=0).stop 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 shutdown_connections 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f11fc06ecc0 0x7f11fc071180 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f120c100e10 0x7f120c198df0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 --2- 172.21.15.96:0/3710544626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f120c1017c0 0x7f120c199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 >> 172.21.15.96:0/3710544626 conn(0x7f120c0fc6e0 msgr2=0x7f120c074400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:18.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 shutdown_connections 2024-04-22T15:59:18.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:18.180+0000 7f121325b700 1 -- 172.21.15.96:0/3710544626 wait complete. 2024-04-22T15:59:18.850 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph health detail' 2024-04-22T15:59:19.068 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:59:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:19 smithi196 ceph-mon[81815]: from='client.36030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:20.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:19 smithi196 ceph-mon[81815]: pgmap v1127: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:20.193 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:19 smithi096 ceph-mon[112443]: from='client.36030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:20.193 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:19 smithi096 ceph-mon[112443]: pgmap v1127: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:20.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.716+0000 7f4a16e77700 1 -- 172.21.15.96:0/1475497956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 msgr2=0x7f4a100fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.716+0000 7f4a16e77700 1 --2- 172.21.15.96:0/1475497956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 0x7f4a100fff20 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009910 tx=0x7f4a00009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.716+0000 7f4a16e77700 1 -- 172.21.15.96:0/1475497956 shutdown_connections 2024-04-22T15:59:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.716+0000 7f4a16e77700 1 --2- 172.21.15.96:0/1475497956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 0x7f4a100fff20 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.716+0000 7f4a16e77700 1 --2- 172.21.15.96:0/1475497956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a10105aa0 0x7f4a10105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.716+0000 7f4a16e77700 1 -- 172.21.15.96:0/1475497956 >> 172.21.15.96:0/1475497956 conn(0x7f4a10076740 msgr2=0x7f4a100fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.716+0000 7f4a16e77700 1 -- 172.21.15.96:0/1475497956 shutdown_connections 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.716+0000 7f4a16e77700 1 -- 172.21.15.96:0/1475497956 wait complete. 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a16e77700 1 Processor -- start 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a16e77700 1 -- start start 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a16e77700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 0x7f4a1019b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a16e77700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a10105aa0 0x7f4a1019bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a16e77700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1019c200 con 0x7f4a100ffaa0 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a16e77700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a10195800 con 0x7f4a10105aa0 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a14c13700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 0x7f4a1019b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a14c13700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 0x7f4a1019b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50648/0 (socket says 172.21.15.96:50648) 2024-04-22T15:59:20.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a14c13700 1 -- 172.21.15.96:0/2635221648 learned_addr learned my addr 172.21.15.96:0/2635221648 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a0ffff700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a10105aa0 0x7f4a1019bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a14c13700 1 -- 172.21.15.96:0/2635221648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a10105aa0 msgr2=0x7f4a1019bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a14c13700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a10105aa0 0x7f4a1019bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a14c13700 1 -- 172.21.15.96:0/2635221648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a000095f0 con 0x7f4a100ffaa0 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.717+0000 7f4a0ffff700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a10105aa0 0x7f4a1019bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.718+0000 7f4a14c13700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 0x7f4a1019b6c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800cfd0 tx=0x7f4a0800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.718+0000 7f4a0dffb700 1 -- 172.21.15.96:0/2635221648 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a0800ad70 con 0x7f4a100ffaa0 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.718+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a10195ae0 con 0x7f4a100ffaa0 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.718+0000 7f4a0dffb700 1 -- 172.21.15.96:0/2635221648 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a080128e0 con 0x7f4a100ffaa0 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.718+0000 7f4a0dffb700 1 -- 172.21.15.96:0/2635221648 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a08011880 con 0x7f4a100ffaa0 2024-04-22T15:59:20.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.718+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a10196030 con 0x7f4a100ffaa0 2024-04-22T15:59:20.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.719+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a101080c0 con 0x7f4a100ffaa0 2024-04-22T15:59:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.719+0000 7f4a0dffb700 1 -- 172.21.15.96:0/2635221648 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4a08012420 con 0x7f4a100ffaa0 2024-04-22T15:59:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.720+0000 7f4a0dffb700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f49fc06ecc0 0x7f49fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.720+0000 7f4a0dffb700 1 -- 172.21.15.96:0/2635221648 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4a0808f010 con 0x7f4a100ffaa0 2024-04-22T15:59:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.720+0000 7f4a0ffff700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f49fc06ecc0 0x7f49fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.720+0000 7f4a0ffff700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f49fc06ecc0 0x7f49fc071180 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4a10196c20 tx=0x7f4a0001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:20.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:20.725+0000 7f4a0dffb700 1 -- 172.21.15.96:0/2635221648 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4a0805b8a0 con 0x7f4a100ffaa0 2024-04-22T15:59:21.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.007+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4a101011e0 con 0x7f4a100ffaa0 2024-04-22T15:59:21.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.008+0000 7f4a0dffb700 1 -- 172.21.15.96:0/2635221648 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f4a0805b430 con 0x7f4a100ffaa0 2024-04-22T15:59:21.011 INFO:teuthology.orchestra.run.smithi096.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:59:21.011 INFO:teuthology.orchestra.run.smithi096.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:59:21.011 INFO:teuthology.orchestra.run.smithi096.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:59:21.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f49fc06ecc0 msgr2=0x7f49fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:21.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f49fc06ecc0 0x7f49fc071180 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4a10196c20 tx=0x7f4a0001a040 comp rx=0 tx=0).stop 2024-04-22T15:59:21.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 msgr2=0x7f4a1019b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:21.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 0x7f4a1019b6c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800cfd0 tx=0x7f4a0800a340 comp rx=0 tx=0).stop 2024-04-22T15:59:21.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 shutdown_connections 2024-04-22T15:59:21.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f49fc06ecc0 0x7f49fc071180 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:21.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a100ffaa0 0x7f4a1019b6c0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:21.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 --2- 172.21.15.96:0/2635221648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a10105aa0 0x7f4a1019bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:21.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 >> 172.21.15.96:0/2635221648 conn(0x7f4a10076740 msgr2=0x7f4a100fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:21.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 shutdown_connections 2024-04-22T15:59:21.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:21.010+0000 7f4a16e77700 1 -- 172.21.15.96:0/2635221648 wait complete. 2024-04-22T15:59:21.668 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions' 2024-04-22T15:59:21.894 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:59:21.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:21 smithi096 ceph-mon[112443]: pgmap v1128: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:59:21.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:21 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2635221648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:59:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:21 smithi196 ceph-mon[81815]: pgmap v1128: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:59:22.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:21 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2635221648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:59:23.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.568+0000 7fa4857db700 1 -- 172.21.15.96:0/3359296642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 msgr2=0x7fa48010aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:23.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.568+0000 7fa4857db700 1 --2- 172.21.15.96:0/3359296642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 0x7fa48010aba0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa468009910 tx=0x7fa468009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:23.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.568+0000 7fa4857db700 1 -- 172.21.15.96:0/3359296642 shutdown_connections 2024-04-22T15:59:23.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.568+0000 7fa4857db700 1 --2- 172.21.15.96:0/3359296642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 0x7fa48010aba0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:23.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.568+0000 7fa4857db700 1 --2- 172.21.15.96:0/3359296642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa480101d90 0x7fa480102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:23.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.568+0000 7fa4857db700 1 -- 172.21.15.96:0/3359296642 >> 172.21.15.96:0/3359296642 conn(0x7fa4800fb5e0 msgr2=0x7fa4800fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:23.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.568+0000 7fa4857db700 1 -- 172.21.15.96:0/3359296642 shutdown_connections 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.568+0000 7fa4857db700 1 -- 172.21.15.96:0/3359296642 wait complete. 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.568+0000 7fa4857db700 1 Processor -- start 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4857db700 1 -- start start 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4857db700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa480101d90 0x7fa48019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4857db700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 0x7fa48019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4857db700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa480196ab0 con 0x7fa4801026b0 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4857db700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa480196c20 con 0x7fa480101d90 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4767fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 0x7fa48019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4767fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 0x7fa48019cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50660/0 (socket says 172.21.15.96:50660) 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4767fc700 1 -- 172.21.15.96:0/462663862 learned_addr learned my addr 172.21.15.96:0/462663862 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa47effd700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa480101d90 0x7fa48019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4767fc700 1 -- 172.21.15.96:0/462663862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa480101d90 msgr2=0x7fa48019ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4767fc700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa480101d90 0x7fa48019ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.569+0000 7fa4767fc700 1 -- 172.21.15.96:0/462663862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680095f0 con 0x7fa4801026b0 2024-04-22T15:59:23.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.570+0000 7fa4767fc700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 0x7fa48019cf40 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa46800bd70 tx=0x7fa46800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.570+0000 7fa47cff9700 1 -- 172.21.15.96:0/462663862 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa46801d070 con 0x7fa4801026b0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.570+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa480196ea0 con 0x7fa4801026b0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.570+0000 7fa47cff9700 1 -- 172.21.15.96:0/462663862 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4680056c0 con 0x7fa4801026b0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.570+0000 7fa47cff9700 1 -- 172.21.15.96:0/462663862 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa468017400 con 0x7fa4801026b0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.570+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa480197390 con 0x7fa4801026b0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.571+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4800699f0 con 0x7fa4801026b0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.571+0000 7fa47cff9700 1 -- 172.21.15.96:0/462663862 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa468003a60 con 0x7fa4801026b0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.572+0000 7fa47cff9700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa46c06ecc0 0x7fa46c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.572+0000 7fa47cff9700 1 -- 172.21.15.96:0/462663862 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa468090270 con 0x7fa4801026b0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.572+0000 7fa47effd700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa46c06ecc0 0x7fa46c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:23.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.572+0000 7fa47effd700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa46c06ecc0 0x7fa46c071180 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa470009a10 tx=0x7fa470009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:23.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.576+0000 7fa47cff9700 1 -- 172.21.15.96:0/462663862 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa468058fd0 con 0x7fa4801026b0 2024-04-22T15:59:23.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.863+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa480105360 con 0x7fa4801026b0 2024-04-22T15:59:23.866 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:23 smithi096 ceph-mon[112443]: pgmap v1129: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:23.866 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:23.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.863+0000 7fa47cff9700 1 -- 172.21.15.96:0/462663862 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fa46807be60 con 0x7fa4801026b0 2024-04-22T15:59:23.866 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T15:59:23.866 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:59:23.867 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T15:59:23.868 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T15:59:23.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa46c06ecc0 msgr2=0x7fa46c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa46c06ecc0 0x7fa46c071180 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa470009a10 tx=0x7fa470009190 comp rx=0 tx=0).stop 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 msgr2=0x7fa48019cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 0x7fa48019cf40 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa46800bd70 tx=0x7fa46800bda0 comp rx=0 tx=0).stop 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 shutdown_connections 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fa46c06ecc0 0x7fa46c071180 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa480101d90 0x7fa48019ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 --2- 172.21.15.96:0/462663862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4801026b0 0x7fa48019cf40 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 >> 172.21.15.96:0/462663862 conn(0x7fa4800fb5e0 msgr2=0x7fa4801030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 shutdown_connections 2024-04-22T15:59:23.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T15:59:23.866+0000 7fa4857db700 1 -- 172.21.15.96:0/462663862 wait complete. 2024-04-22T15:59:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:23 smithi196 ceph-mon[81815]: pgmap v1129: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:24.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:24.520 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-22T15:59:24.726 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:59:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:24 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/462663862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:25.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:24 smithi196 ceph-mon[81815]: pgmap v1130: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:59:25.069 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:24 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/462663862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:25.069 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:24 smithi096 ceph-mon[112443]: pgmap v1130: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:59:26.577 INFO:teuthology.orchestra.run.smithi096.stdout:wait for servicemap items w/ changing names to refresh 2024-04-22T15:59:27.229 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'sleep 60' 2024-04-22T15:59:27.434 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T15:59:27.893 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:27 smithi096 ceph-mon[112443]: pgmap v1131: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:59:28.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:27 smithi196 ceph-mon[81815]: pgmap v1131: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:59:29.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:29 smithi096 ceph-mon[112443]: pgmap v1132: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:30.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:29 smithi196 ceph-mon[81815]: pgmap v1132: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:31.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:30 smithi096 ceph-mon[112443]: pgmap v1133: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:59:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:30 smithi196 ceph-mon[81815]: pgmap v1133: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:59:34.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:33 smithi196 ceph-mon[81815]: pgmap v1134: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:34.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:33 smithi096 ceph-mon[112443]: pgmap v1134: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:36.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:35 smithi196 ceph-mon[81815]: pgmap v1135: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:59:36.038 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:35 smithi096 ceph-mon[112443]: pgmap v1135: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:59:37.054 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:36 smithi096 ceph-mon[112443]: pgmap v1136: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:59:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:36 smithi196 ceph-mon[81815]: pgmap v1136: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:59:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:39.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:39 smithi096 ceph-mon[112443]: pgmap v1137: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:39 smithi196 ceph-mon[81815]: pgmap v1137: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:42.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:41 smithi196 ceph-mon[81815]: pgmap v1138: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:59:42.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:41 smithi096 ceph-mon[112443]: pgmap v1138: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:59:43.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:42 smithi196 ceph-mon[81815]: pgmap v1139: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:43.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:42 smithi096 ceph-mon[112443]: pgmap v1139: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:46.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:45 smithi196 ceph-mon[81815]: pgmap v1140: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:59:46.109 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:45 smithi096 ceph-mon[112443]: pgmap v1140: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:59:47.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:46 smithi096 ceph-mon[112443]: pgmap v1141: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:59:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:46 smithi196 ceph-mon[81815]: pgmap v1141: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:59:50.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:49 smithi196 ceph-mon[81815]: pgmap v1142: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:50.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:49 smithi096 ceph-mon[112443]: pgmap v1142: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:51.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:50 smithi096 ceph-mon[112443]: pgmap v1143: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:59:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:50 smithi196 ceph-mon[81815]: pgmap v1143: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T15:59:53.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:52 smithi196 ceph-mon[81815]: pgmap v1144: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:53.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:52 smithi096 ceph-mon[112443]: pgmap v1144: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:54.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:55.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:54 smithi096 ceph-mon[112443]: pgmap v1145: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:59:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:54 smithi196 ceph-mon[81815]: pgmap v1145: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T15:59:57.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:56 smithi096 ceph-mon[112443]: pgmap v1146: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:59:57.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:56 smithi196 ceph-mon[81815]: pgmap v1146: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T15:59:59.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 15:59:58 smithi096 ceph-mon[112443]: pgmap v1147: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T15:59:59.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 15:59:58 smithi196 ceph-mon[81815]: pgmap v1147: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:00.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:00 smithi096 ceph-mon[112443]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T16:00:00.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:00 smithi096 ceph-mon[112443]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T16:00:00.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:00 smithi096 ceph-mon[112443]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T16:00:00.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:00 smithi196 ceph-mon[81815]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T16:00:00.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:00 smithi196 ceph-mon[81815]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T16:00:00.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:00 smithi196 ceph-mon[81815]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T16:00:01.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:01 smithi096 ceph-mon[112443]: pgmap v1148: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:01.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:01 smithi196 ceph-mon[81815]: pgmap v1148: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:03.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:02 smithi096 ceph-mon[112443]: pgmap v1149: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:02 smithi196 ceph-mon[81815]: pgmap v1149: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:05.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:04 smithi096 ceph-mon[112443]: pgmap v1150: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:05.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:04 smithi196 ceph-mon[81815]: pgmap v1150: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:06.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:06 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:00:06.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:06 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:00:07.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:07 smithi096 ceph-mon[112443]: pgmap v1151: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:07.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:07 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:00:07.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:07 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:00:07.486 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:07 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:00:07.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:07 smithi196 ceph-mon[81815]: pgmap v1151: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:07.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:07 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:00:07.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:07 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:00:07.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:07 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:00:08.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:08.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:09.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:09 smithi096 ceph-mon[112443]: pgmap v1152: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:09 smithi196 ceph-mon[81815]: pgmap v1152: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:11.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:10 smithi096 ceph-mon[112443]: pgmap v1153: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:11.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:10 smithi196 ceph-mon[81815]: pgmap v1153: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:12.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:12 smithi096 ceph-mon[112443]: pgmap v1154: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:13.061 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:12 smithi196 ceph-mon[81815]: pgmap v1154: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:15.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:14 smithi096 ceph-mon[112443]: pgmap v1155: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:15.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:14 smithi196 ceph-mon[81815]: pgmap v1155: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:17.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:16 smithi096 ceph-mon[112443]: pgmap v1156: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:17.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:16 smithi196 ceph-mon[81815]: pgmap v1156: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:19.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:18 smithi096 ceph-mon[112443]: pgmap v1157: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:19.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:18 smithi196 ceph-mon[81815]: pgmap v1157: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:21.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:20 smithi096 ceph-mon[112443]: pgmap v1158: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:21.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:20 smithi196 ceph-mon[81815]: pgmap v1158: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:22.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:22 smithi096 ceph-mon[112443]: pgmap v1159: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:23.087 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:22 smithi196 ceph-mon[81815]: pgmap v1159: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:24.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:25.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:24 smithi096 ceph-mon[112443]: pgmap v1160: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:25.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:24 smithi196 ceph-mon[81815]: pgmap v1160: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:27.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:26 smithi096 ceph-mon[112443]: pgmap v1161: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:26 smithi196 ceph-mon[81815]: pgmap v1161: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:29.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:28 smithi096 ceph-mon[112443]: pgmap v1162: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:28 smithi196 ceph-mon[81815]: pgmap v1162: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:29.611 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch ps' 2024-04-22T16:00:29.826 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T16:00:30.940 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:30 smithi096 ceph-mon[112443]: pgmap v1163: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:30 smithi196 ceph-mon[81815]: pgmap v1163: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 -- 172.21.15.96:0/2827288590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 msgr2=0x7fbe500fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 --2- 172.21.15.96:0/2827288590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 0x7fbe500fff20 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009910 tx=0x7fbe44009c20 comp rx=0 tx=0).stop 2024-04-22T16:00:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 -- 172.21.15.96:0/2827288590 shutdown_connections 2024-04-22T16:00:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 --2- 172.21.15.96:0/2827288590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 0x7fbe500fff20 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 --2- 172.21.15.96:0/2827288590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe50105aa0 0x7fbe50105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 -- 172.21.15.96:0/2827288590 >> 172.21.15.96:0/2827288590 conn(0x7fbe50076740 msgr2=0x7fbe500fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 -- 172.21.15.96:0/2827288590 shutdown_connections 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 -- 172.21.15.96:0/2827288590 wait complete. 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 Processor -- start 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 -- start start 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 0x7fbe500fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe50105aa0 0x7fbe500fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe500fc6a0 con 0x7fbe500ffaa0 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.561+0000 7fbe57062700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe500fc810 con 0x7fbe50105aa0 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe54dfe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 0x7fbe500fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe54dfe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 0x7fbe500fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35254/0 (socket says 172.21.15.96:35254) 2024-04-22T16:00:31.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe54dfe700 1 -- 172.21.15.96:0/1018080246 learned_addr learned my addr 172.21.15.96:0/1018080246 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe4ffff700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe50105aa0 0x7fbe500fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe54dfe700 1 -- 172.21.15.96:0/1018080246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe50105aa0 msgr2=0x7fbe500fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe54dfe700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe50105aa0 0x7fbe500fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe54dfe700 1 -- 172.21.15.96:0/1018080246 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe440095f0 con 0x7fbe500ffaa0 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe54dfe700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 0x7fbe500fbc20 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fbe480060b0 tx=0x7fbe4800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe4dffb700 1 -- 172.21.15.96:0/1018080246 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe4800dda0 con 0x7fbe500ffaa0 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe4dffb700 1 -- 172.21.15.96:0/1018080246 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe4800f040 con 0x7fbe500ffaa0 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe501a2c90 con 0x7fbe500ffaa0 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe4dffb700 1 -- 172.21.15.96:0/1018080246 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe480146f0 con 0x7fbe500ffaa0 2024-04-22T16:00:31.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.562+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe501a3040 con 0x7fbe500ffaa0 2024-04-22T16:00:31.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.563+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe501011e0 con 0x7fbe500ffaa0 2024-04-22T16:00:31.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.564+0000 7fbe4dffb700 1 -- 172.21.15.96:0/1018080246 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbe48015400 con 0x7fbe500ffaa0 2024-04-22T16:00:31.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.564+0000 7fbe4dffb700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3806ecc0 0x7fbe38071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:31.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.564+0000 7fbe4dffb700 1 -- 172.21.15.96:0/1018080246 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbe4808e890 con 0x7fbe500ffaa0 2024-04-22T16:00:31.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.564+0000 7fbe4ffff700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3806ecc0 0x7fbe38071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:31.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.565+0000 7fbe4ffff700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3806ecc0 0x7fbe38071180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbe500fd350 tx=0x7fbe4401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:31.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.569+0000 7fbe4dffb700 1 -- 172.21.15.96:0/1018080246 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbe480576a0 con 0x7fbe500ffaa0 2024-04-22T16:00:31.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.777+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 --> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe500fc9d0 con 0x7fbe3806ecc0 2024-04-22T16:00:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.788+0000 7fbe4dffb700 1 -- 172.21.15.96:0/1018080246 <== mgr.34104 v2:172.21.15.96:6800/4190215134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fbe500fc9d0 con 0x7fbe3806ecc0 2024-04-22T16:00:31.791 INFO:teuthology.orchestra.run.smithi096.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T16:00:31.791 INFO:teuthology.orchestra.run.smithi096.stdout:alertmanager.smithi096 smithi096 *:9093,9094 running (39m) 93s ago 50m 24.2M - 0.25.0 c8568f914cd2 9c4c7830470b 2024-04-22T16:00:31.791 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi096 smithi096 running (105s) 93s ago 50m 7801k - 19.0.0-3223-g5883d494 3d983c26e278 348dbb003759 2024-04-22T16:00:31.791 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-exporter.smithi196 smithi196 running (100s) 95s ago 49m 6903k - 19.0.0-3223-g5883d494 3d983c26e278 4d699b8512db 2024-04-22T16:00:31.791 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi096 smithi096 running (36m) 93s ago 50m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 b16d21ccd592 2024-04-22T16:00:31.791 INFO:teuthology.orchestra.run.smithi096.stdout:crash.smithi196 smithi196 running (36m) 95s ago 49m 6660k - 19.0.0-3223-g5883d494 3d983c26e278 9e6bbae73d97 2024-04-22T16:00:31.791 INFO:teuthology.orchestra.run.smithi096.stdout:grafana.smithi096 smithi096 *:3000 running (38m) 93s ago 50m 82.1M - 9.4.12 2bacad6d85d8 f6370da5f1cb 2024-04-22T16:00:31.791 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.drkkdi smithi096 running (32m) 93s ago 44m 25.6M - 19.0.0-3223-g5883d494 3d983c26e278 8db59caa3d83 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi096.dymqlv smithi096 running (32m) 93s ago 44m 179M - 19.0.0-3223-g5883d494 3d983c26e278 ad1b8da92b2e 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.hwpcvg smithi196 running (32m) 95s ago 44m 173M - 19.0.0-3223-g5883d494 3d983c26e278 bfbde47c971b 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:mds.cephfs.smithi196.pbvqlq smithi196 running (16m) 95s ago 44m 20.3M - 19.0.0-3223-g5883d494 3d983c26e278 ffbe60c6e26f 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi096.hhkvum smithi096 *:8443,9283,8765 running (41m) 93s ago 51m 519M - 19.0.0-3223-g5883d494 3d983c26e278 86e0cf5d0fc1 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:mgr.smithi196.gdaegc smithi196 *:8443,9283,8765 running (40m) 95s ago 49m 460M - 19.0.0-3223-g5883d494 3d983c26e278 4eccbc8ac76d 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi096 smithi096 running (37m) 93s ago 52m 87.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 cc6f8ad67833 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:mon.smithi196 smithi196 running (37m) 95s ago 49m 59.9M 2048M 19.0.0-3223-g5883d494 3d983c26e278 351725aca606 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi096 smithi096 *:9100 running (40m) 93s ago 50m 16.7M - 1.5.0 0da6a335fe13 f7a3e7649163 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:node-exporter.smithi196 smithi196 *:9100 running (40m) 95s ago 49m 17.1M - 1.5.0 0da6a335fe13 90fccb1992af 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:osd.0 smithi096 running (35m) 93s ago 48m 106M 1216M 19.0.0-3223-g5883d494 3d983c26e278 bbf14d18e921 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:osd.1 smithi096 running (35m) 93s ago 48m 116M 1216M 19.0.0-3223-g5883d494 3d983c26e278 8ca99058d1fb 2024-04-22T16:00:31.792 INFO:teuthology.orchestra.run.smithi096.stdout:osd.2 smithi096 running (34m) 93s ago 47m 88.3M 1216M 19.0.0-3223-g5883d494 3d983c26e278 afcdcb7020b9 2024-04-22T16:00:31.793 INFO:teuthology.orchestra.run.smithi096.stdout:osd.3 smithi196 running (34m) 95s ago 47m 114M 2240M 19.0.0-3223-g5883d494 3d983c26e278 984101ebadb9 2024-04-22T16:00:31.793 INFO:teuthology.orchestra.run.smithi096.stdout:osd.4 smithi196 running (33m) 95s ago 46m 93.7M 2240M 19.0.0-3223-g5883d494 3d983c26e278 ad74967cb5b7 2024-04-22T16:00:31.793 INFO:teuthology.orchestra.run.smithi096.stdout:osd.5 smithi196 running (33m) 95s ago 46m 95.0M 2240M 19.0.0-3223-g5883d494 3d983c26e278 1628335216db 2024-04-22T16:00:31.793 INFO:teuthology.orchestra.run.smithi096.stdout:prometheus.smithi096 smithi096 *:9095 running (39m) 93s ago 50m 94.5M - 2.43.0 a07b618ecd1d 5984482423c6 2024-04-22T16:00:31.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3806ecc0 msgr2=0x7fbe38071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:31.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3806ecc0 0x7fbe38071180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbe500fd350 tx=0x7fbe4401a040 comp rx=0 tx=0).stop 2024-04-22T16:00:31.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 msgr2=0x7fbe500fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:31.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 0x7fbe500fbc20 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fbe480060b0 tx=0x7fbe4800d3b0 comp rx=0 tx=0).stop 2024-04-22T16:00:31.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 shutdown_connections 2024-04-22T16:00:31.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbe3806ecc0 0x7fbe38071180 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:31.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe500ffaa0 0x7fbe500fbc20 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:31.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 --2- 172.21.15.96:0/1018080246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe50105aa0 0x7fbe500fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:31.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 >> 172.21.15.96:0/1018080246 conn(0x7fbe50076740 msgr2=0x7fbe500fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:31.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 shutdown_connections 2024-04-22T16:00:31.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:31.790+0000 7fbe57062700 1 -- 172.21.15.96:0/1018080246 wait complete. 2024-04-22T16:00:32.504 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions' 2024-04-22T16:00:32.725 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T16:00:33.042 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:32 smithi096 ceph-mon[112443]: from='client.36042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T16:00:33.042 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:32 smithi096 ceph-mon[112443]: pgmap v1164: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:33.201 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:32 smithi196 ceph-mon[81815]: from='client.36042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T16:00:33.201 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:32 smithi196 ceph-mon[81815]: pgmap v1164: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:34.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.416+0000 7effe653e700 1 -- 172.21.15.96:0/609328725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 msgr2=0x7effe010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:34.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.416+0000 7effe653e700 1 --2- 172.21.15.96:0/609328725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 0x7effe010ab70 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7effd4009910 tx=0x7effd4009c20 comp rx=0 tx=0).stop 2024-04-22T16:00:34.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.416+0000 7effe653e700 1 -- 172.21.15.96:0/609328725 shutdown_connections 2024-04-22T16:00:34.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.416+0000 7effe653e700 1 --2- 172.21.15.96:0/609328725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 0x7effe010ab70 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:34.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.416+0000 7effe653e700 1 --2- 172.21.15.96:0/609328725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effe0101d60 0x7effe0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:34.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.416+0000 7effe653e700 1 -- 172.21.15.96:0/609328725 >> 172.21.15.96:0/609328725 conn(0x7effe0076740 msgr2=0x7effe00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:34.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.416+0000 7effe653e700 1 -- 172.21.15.96:0/609328725 shutdown_connections 2024-04-22T16:00:34.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.416+0000 7effe653e700 1 -- 172.21.15.96:0/609328725 wait complete. 2024-04-22T16:00:34.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.416+0000 7effe653e700 1 Processor -- start 2024-04-22T16:00:34.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effe653e700 1 -- start start 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effe653e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effe0101d60 0x7effe019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effe653e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 0x7effe019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effe653e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe019d3c0 con 0x7effe0102680 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effe653e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe01969a0 con 0x7effe0101d60 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effdf7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 0x7effe019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effdf7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 0x7effe019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35274/0 (socket says 172.21.15.96:35274) 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effdf7fe700 1 -- 172.21.15.96:0/2715826482 learned_addr learned my addr 172.21.15.96:0/2715826482 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effdffff700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effe0101d60 0x7effe019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effdf7fe700 1 -- 172.21.15.96:0/2715826482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effe0101d60 msgr2=0x7effe019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:34.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effdf7fe700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effe0101d60 0x7effe019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effdf7fe700 1 -- 172.21.15.96:0/2715826482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd40095f0 con 0x7effe0102680 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effdffff700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effe0101d60 0x7effe019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.417+0000 7effdf7fe700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 0x7effe019cda0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7effd40092e0 tx=0x7effd400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.418+0000 7effdd7fa700 1 -- 172.21.15.96:0/2715826482 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effd401d070 con 0x7effe0102680 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.418+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe0196c20 con 0x7effe0102680 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.418+0000 7effdd7fa700 1 -- 172.21.15.96:0/2715826482 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effd4003d50 con 0x7effe0102680 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.418+0000 7effdd7fa700 1 -- 172.21.15.96:0/2715826482 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effd4017400 con 0x7effe0102680 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.418+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe0197110 con 0x7effe0102680 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.419+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe0105480 con 0x7effe0102680 2024-04-22T16:00:34.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.419+0000 7effdd7fa700 1 -- 172.21.15.96:0/2715826482 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7effd4003890 con 0x7effe0102680 2024-04-22T16:00:34.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.419+0000 7effdd7fa700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effc806ecc0 0x7effc8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:34.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.420+0000 7effdd7fa700 1 -- 172.21.15.96:0/2715826482 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7effd4090270 con 0x7effe0102680 2024-04-22T16:00:34.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.420+0000 7effdffff700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effc806ecc0 0x7effc8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:34.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.420+0000 7effdffff700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effc806ecc0 0x7effc8071180 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7effd0005fd0 tx=0x7effd0005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:34.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.424+0000 7effdd7fa700 1 -- 172.21.15.96:0/2715826482 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7effd405cb00 con 0x7effe0102680 2024-04-22T16:00:34.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.707+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7effe0068ea0 con 0x7effe0102680 2024-04-22T16:00:34.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.708+0000 7effdd7fa700 1 -- 172.21.15.96:0/2715826482 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7effd407be60 con 0x7effe0102680 2024-04-22T16:00:34.711 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T16:00:34.711 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-04-22T16:00:34.712 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T16:00:34.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T16:00:34.712 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-04-22T16:00:34.712 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T16:00:34.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T16:00:34.712 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": { 2024-04-22T16:00:34.712 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T16:00:34.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T16:00:34.712 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T16:00:34.713 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T16:00:34.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T16:00:34.713 INFO:teuthology.orchestra.run.smithi096.stdout: "overall": { 2024-04-22T16:00:34.713 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T16:00:34.713 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T16:00:34.713 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T16:00:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.710+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effc806ecc0 msgr2=0x7effc8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.710+0000 7effe653e700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effc806ecc0 0x7effc8071180 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7effd0005fd0 tx=0x7effd0005dc0 comp rx=0 tx=0).stop 2024-04-22T16:00:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.710+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 msgr2=0x7effe019cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.710+0000 7effe653e700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 0x7effe019cda0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7effd40092e0 tx=0x7effd400be20 comp rx=0 tx=0).stop 2024-04-22T16:00:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.710+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 shutdown_connections 2024-04-22T16:00:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.710+0000 7effe653e700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7effc806ecc0 0x7effc8071180 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.710+0000 7effe653e700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effe0101d60 0x7effe019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.710+0000 7effe653e700 1 --2- 172.21.15.96:0/2715826482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effe0102680 0x7effe019cda0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.710+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 >> 172.21.15.96:0/2715826482 conn(0x7effe0076740 msgr2=0x7effe0103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.711+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 shutdown_connections 2024-04-22T16:00:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:34.711+0000 7effe653e700 1 -- 172.21.15.96:0/2715826482 wait complete. 2024-04-22T16:00:34.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:34 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2715826482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:00:34.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:34 smithi096 ceph-mon[112443]: pgmap v1165: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:35.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:34 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2715826482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:00:35.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:34 smithi196 ceph-mon[81815]: pgmap v1165: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:35.293 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-22T16:00:35.499 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T16:00:36.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:36 smithi096 ceph-mon[112443]: pgmap v1166: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.227+0000 7f752d1aa700 1 -- 172.21.15.96:0/2283690095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 msgr2=0x7f7528105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.227+0000 7f752d1aa700 1 --2- 172.21.15.96:0/2283690095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 0x7f7528105f40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f752000b0a0 tx=0x7f752000b3b0 comp rx=0 tx=0).stop 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.227+0000 7f752d1aa700 1 -- 172.21.15.96:0/2283690095 shutdown_connections 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.227+0000 7f752d1aa700 1 --2- 172.21.15.96:0/2283690095 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75280ffad0 0x7f75280fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.227+0000 7f752d1aa700 1 --2- 172.21.15.96:0/2283690095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 0x7f7528105f40 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.227+0000 7f752d1aa700 1 -- 172.21.15.96:0/2283690095 >> 172.21.15.96:0/2283690095 conn(0x7f75280fb5e0 msgr2=0x7f75280fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.227+0000 7f752d1aa700 1 -- 172.21.15.96:0/2283690095 shutdown_connections 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.227+0000 7f752d1aa700 1 -- 172.21.15.96:0/2283690095 wait complete. 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.228+0000 7f752d1aa700 1 Processor -- start 2024-04-22T16:00:37.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.228+0000 7f752d1aa700 1 -- start start 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.228+0000 7f752d1aa700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75280ffad0 0x7f752819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.228+0000 7f752d1aa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 0x7f752819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.228+0000 7f752d1aa700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752819d450 con 0x7f7528105b60 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.228+0000 7f752d1aa700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7528196a30 con 0x7f75280ffad0 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f752659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 0x7f752819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f752659c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 0x7f752819ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35292/0 (socket says 172.21.15.96:35292) 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f752659c700 1 -- 172.21.15.96:0/3681694279 learned_addr learned my addr 172.21.15.96:0/3681694279 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f7526d9d700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75280ffad0 0x7f752819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f752659c700 1 -- 172.21.15.96:0/3681694279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75280ffad0 msgr2=0x7f752819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f752659c700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75280ffad0 0x7f752819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f752659c700 1 -- 172.21.15.96:0/3681694279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7520009d00 con 0x7f7528105b60 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f752659c700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 0x7f752819ce30 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f751000b4b0 tx=0x7f751000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f751ffff700 1 -- 172.21.15.96:0/3681694279 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7510011800 con 0x7f7528105b60 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7528196d10 con 0x7f7528105b60 2024-04-22T16:00:37.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f751ffff700 1 -- 172.21.15.96:0/3681694279 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7510011e20 con 0x7f7528105b60 2024-04-22T16:00:37.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f751ffff700 1 -- 172.21.15.96:0/3681694279 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f751000f2e0 con 0x7f7528105b60 2024-04-22T16:00:37.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.229+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7528197260 con 0x7f7528105b60 2024-04-22T16:00:37.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.230+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75281090e0 con 0x7f7528105b60 2024-04-22T16:00:37.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.231+0000 7f751ffff700 1 -- 172.21.15.96:0/3681694279 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7510010380 con 0x7f7528105b60 2024-04-22T16:00:37.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.231+0000 7f751ffff700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751406ecc0 0x7f7514071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:37.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.231+0000 7f751ffff700 1 -- 172.21.15.96:0/3681694279 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f751008e9d0 con 0x7f7528105b60 2024-04-22T16:00:37.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.231+0000 7f7526d9d700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751406ecc0 0x7f7514071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:37.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.232+0000 7f7526d9d700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751406ecc0 0x7f7514071180 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7520000f80 tx=0x7f75200060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:37.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.236+0000 7f751ffff700 1 -- 172.21.15.96:0/3681694279 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f751005b260 con 0x7f7528105b60 2024-04-22T16:00:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:36 smithi196 ceph-mon[81815]: pgmap v1166: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:37.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.519+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7528197f40 con 0x7f7528105b60 2024-04-22T16:00:37.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.520+0000 7f751ffff700 1 -- 172.21.15.96:0/3681694279 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f751005adf0 con 0x7f7528105b60 2024-04-22T16:00:37.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751406ecc0 msgr2=0x7f7514071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:37.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751406ecc0 0x7f7514071180 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7520000f80 tx=0x7f75200060c0 comp rx=0 tx=0).stop 2024-04-22T16:00:37.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 msgr2=0x7f752819ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:37.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 0x7f752819ce30 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f751000b4b0 tx=0x7f751000b7c0 comp rx=0 tx=0).stop 2024-04-22T16:00:37.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 shutdown_connections 2024-04-22T16:00:37.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f751406ecc0 0x7f7514071180 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:37.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75280ffad0 0x7f752819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:37.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 --2- 172.21.15.96:0/3681694279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7528105b60 0x7f752819ce30 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:37.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 >> 172.21.15.96:0/3681694279 conn(0x7f75280fb5e0 msgr2=0x7f7528104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:37.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 shutdown_connections 2024-04-22T16:00:37.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:37.522+0000 7f752d1aa700 1 -- 172.21.15.96:0/3681694279 wait complete. 2024-04-22T16:00:37.535 INFO:teuthology.orchestra.run.smithi096.stdout:true 2024-04-22T16:00:38.165 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:37 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3681694279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:00:38.186 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-22T16:00:38.257 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:37 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3681694279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:00:38.392 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T16:00:39.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:39.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:38 smithi096 ceph-mon[112443]: pgmap v1167: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:39.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:39.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:38 smithi196 ceph-mon[81815]: pgmap v1167: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.025+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/1074208596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a58101d90 msgr2=0x7f3a58102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.025+0000 7f3a5f3aa700 1 --2- 172.21.15.96:0/1074208596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a58101d90 0x7f3a58102170 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400b0a0 tx=0x7f3a5400b3b0 comp rx=0 tx=0).stop 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.025+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/1074208596 shutdown_connections 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.025+0000 7f3a5f3aa700 1 --2- 172.21.15.96:0/1074208596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a581026b0 0x7f3a5810aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.025+0000 7f3a5f3aa700 1 --2- 172.21.15.96:0/1074208596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a58101d90 0x7f3a58102170 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.025+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/1074208596 >> 172.21.15.96:0/1074208596 conn(0x7f3a580fb5e0 msgr2=0x7f3a580fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.025+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/1074208596 shutdown_connections 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.025+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/1074208596 wait complete. 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5f3aa700 1 Processor -- start 2024-04-22T16:00:40.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5f3aa700 1 -- start start 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5f3aa700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a58101d90 0x7f3a5819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5f3aa700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a581026b0 0x7f3a5819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5f3aa700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5819d3f0 con 0x7f3a581026b0 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5f3aa700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a581969d0 con 0x7f3a58101d90 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5c945700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a581026b0 0x7f3a5819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5c945700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a581026b0 0x7f3a5819cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35312/0 (socket says 172.21.15.96:35312) 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5c945700 1 -- 172.21.15.96:0/2211458262 learned_addr learned my addr 172.21.15.96:0/2211458262 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5d146700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a58101d90 0x7f3a5819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5c945700 1 -- 172.21.15.96:0/2211458262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a58101d90 msgr2=0x7f3a5819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5c945700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a58101d90 0x7f3a5819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5c945700 1 -- 172.21.15.96:0/2211458262 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a54009d00 con 0x7f3a581026b0 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a5c945700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a581026b0 0x7f3a5819cdd0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800d6d0 tx=0x7f3a4800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a4e7fc700 1 -- 172.21.15.96:0/2211458262 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a4800f860 con 0x7f3a581026b0 2024-04-22T16:00:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.026+0000 7f3a4e7fc700 1 -- 172.21.15.96:0/2211458262 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a4800fe80 con 0x7f3a581026b0 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.027+0000 7f3a4e7fc700 1 -- 172.21.15.96:0/2211458262 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a48010ea0 con 0x7f3a581026b0 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.027+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a58196cb0 con 0x7f3a581026b0 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.027+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a58197200 con 0x7f3a581026b0 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.028+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a581054b0 con 0x7f3a581026b0 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.028+0000 7f3a4e7fc700 1 -- 172.21.15.96:0/2211458262 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3a48010460 con 0x7f3a581026b0 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.028+0000 7f3a4e7fc700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a4406ecd0 0x7f3a44071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.028+0000 7f3a4e7fc700 1 -- 172.21.15.96:0/2211458262 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3a4808ead0 con 0x7f3a581026b0 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.028+0000 7f3a5d146700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a4406ecd0 0x7f3a44071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.029+0000 7f3a5d146700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a4406ecd0 0x7f3a44071190 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3a540060b0 tx=0x7f3a54006000 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:40.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.033+0000 7f3a4e7fc700 1 -- 172.21.15.96:0/2211458262 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3a4805b300 con 0x7f3a581026b0 2024-04-22T16:00:40.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.319+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3a58197e90 con 0x7f3a581026b0 2024-04-22T16:00:40.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.320+0000 7f3a4e7fc700 1 -- 172.21.15.96:0/2211458262 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f3a4805ae90 con 0x7f3a581026b0 2024-04-22T16:00:40.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a4406ecd0 msgr2=0x7f3a44071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:40.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a4406ecd0 0x7f3a44071190 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3a540060b0 tx=0x7f3a54006000 comp rx=0 tx=0).stop 2024-04-22T16:00:40.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a581026b0 msgr2=0x7f3a5819cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:40.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a581026b0 0x7f3a5819cdd0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800d6d0 tx=0x7f3a4800da90 comp rx=0 tx=0).stop 2024-04-22T16:00:40.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 shutdown_connections 2024-04-22T16:00:40.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a4406ecd0 0x7f3a44071190 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:40.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a58101d90 0x7f3a5819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:40.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 --2- 172.21.15.96:0/2211458262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a581026b0 0x7f3a5819cdd0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:40.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 >> 172.21.15.96:0/2211458262 conn(0x7f3a580fb5e0 msgr2=0x7f3a58103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:40.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 shutdown_connections 2024-04-22T16:00:40.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:40.322+0000 7f3a5f3aa700 1 -- 172.21.15.96:0/2211458262 wait complete. 2024-04-22T16:00:40.335 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)" 2024-04-22T16:00:40.730 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:40 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2211458262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:00:40.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:40 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2211458262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:00:41.055 DEBUG:teuthology.parallel:result is None 2024-04-22T16:00:41.055 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T16:00:41.066 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T16:00:41.067 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d5e86090-00b9-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T16:00:41.273 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/mon.smithi096/config 2024-04-22T16:00:41.700 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:41 smithi096 ceph-mon[112443]: pgmap v1168: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:41.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:41 smithi196 ceph-mon[81815]: pgmap v1168: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.923+0000 7fce703d7700 1 -- 172.21.15.96:0/3706707026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 msgr2=0x7fce681047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.923+0000 7fce703d7700 1 --2- 172.21.15.96:0/3706707026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 0x7fce681047a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fce64009910 tx=0x7fce64009c20 comp rx=0 tx=0).stop 2024-04-22T16:00:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.923+0000 7fce703d7700 1 -- 172.21.15.96:0/3706707026 shutdown_connections 2024-04-22T16:00:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.923+0000 7fce703d7700 1 --2- 172.21.15.96:0/3706707026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 0x7fce681047a0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:42.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.923+0000 7fce703d7700 1 --2- 172.21.15.96:0/3706707026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce68100420 0x7fce68100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:42.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.923+0000 7fce703d7700 1 -- 172.21.15.96:0/3706707026 >> 172.21.15.96:0/3706707026 conn(0x7fce68075f40 msgr2=0x7fce680fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:42.928 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:42 smithi096 ceph-mon[112443]: pgmap v1169: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:42.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.924+0000 7fce703d7700 1 -- 172.21.15.96:0/3706707026 shutdown_connections 2024-04-22T16:00:42.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.924+0000 7fce703d7700 1 -- 172.21.15.96:0/3706707026 wait complete. 2024-04-22T16:00:42.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.924+0000 7fce703d7700 1 Processor -- start 2024-04-22T16:00:42.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce703d7700 1 -- start start 2024-04-22T16:00:42.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce703d7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce68100420 0x7fce68197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:42.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce703d7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 0x7fce68197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:42.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce703d7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce681985b0 con 0x7fce68100dd0 2024-04-22T16:00:42.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce703d7700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce681986f0 con 0x7fce68100420 2024-04-22T16:00:42.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce6d972700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 0x7fce68197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:42.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce6d972700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 0x7fce68197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54584/0 (socket says 172.21.15.96:54584) 2024-04-22T16:00:42.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce6d972700 1 -- 172.21.15.96:0/707045203 learned_addr learned my addr 172.21.15.96:0/707045203 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:42.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce6d972700 1 -- 172.21.15.96:0/707045203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce68100420 msgr2=0x7fce68197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:42.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce6e173700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce68100420 0x7fce68197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce6d972700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce68100420 0x7fce68197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce6d972700 1 -- 172.21.15.96:0/707045203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce640095f0 con 0x7fce68100dd0 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce6e173700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce68100420 0x7fce68197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce6d972700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 0x7fce68197fe0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fce64015000 tx=0x7fce6400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.925+0000 7fce5f7fe700 1 -- 172.21.15.96:0/707045203 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce6401d070 con 0x7fce68100dd0 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.926+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce68198910 con 0x7fce68100dd0 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.926+0000 7fce5f7fe700 1 -- 172.21.15.96:0/707045203 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce6400f460 con 0x7fce68100dd0 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.926+0000 7fce5f7fe700 1 -- 172.21.15.96:0/707045203 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce64017630 con 0x7fce68100dd0 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.926+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce681a0e70 con 0x7fce68100dd0 2024-04-22T16:00:42.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.927+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce68108130 con 0x7fce68100dd0 2024-04-22T16:00:42.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.927+0000 7fce5f7fe700 1 -- 172.21.15.96:0/707045203 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce6400f5d0 con 0x7fce68100dd0 2024-04-22T16:00:42.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.928+0000 7fce5f7fe700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce5406ecc0 0x7fce54071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:42.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.928+0000 7fce5f7fe700 1 -- 172.21.15.96:0/707045203 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fce64090160 con 0x7fce68100dd0 2024-04-22T16:00:42.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.928+0000 7fce6e173700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce5406ecc0 0x7fce54071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:42.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.928+0000 7fce6e173700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce5406ecc0 0x7fce54071180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fce58007900 tx=0x7fce58008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:42.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:42.932+0000 7fce5f7fe700 1 -- 172.21.15.96:0/707045203 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fce64058ec0 con 0x7fce68100dd0 2024-04-22T16:00:43.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.178+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fce681013e0 con 0x7fce68100dd0 2024-04-22T16:00:43.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.178+0000 7fce5f7fe700 1 -- 172.21.15.96:0/707045203 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 40 v40) v1 ==== 76+0+1968 (secure 0 0 0) 0x7fce64005740 con 0x7fce68100dd0 2024-04-22T16:00:43.181 INFO:teuthology.orchestra.run.smithi096.stdout:e40 2024-04-22T16:00:43.182 INFO:teuthology.orchestra.run.smithi096.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T16:00:43.182 INFO:teuthology.orchestra.run.smithi096.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T16:00:43.182 INFO:teuthology.orchestra.run.smithi096.stdout:legacy client fscid: 1 2024-04-22T16:00:43.182 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:43.182 INFO:teuthology.orchestra.run.smithi096.stdout:Filesystem 'cephfs' (1) 2024-04-22T16:00:43.182 INFO:teuthology.orchestra.run.smithi096.stdout:fs_name cephfs 2024-04-22T16:00:43.182 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 40 2024-04-22T16:00:43.182 INFO:teuthology.orchestra.run.smithi096.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T16:00:43.182 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-04-22T15:15:33.756786+0000 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:modified 2024-04-22T15:59:04.839573+0000 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:tableserver 0 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:root 0 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:session_timeout 60 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:session_autoclose 300 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:max_file_size 1099511627776 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:max_xattr_size 65536 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:required_client_features {} 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure 0 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:last_failure_osd_epoch 80 2024-04-22T16:00:43.183 INFO:teuthology.orchestra.run.smithi096.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T16:00:43.184 INFO:teuthology.orchestra.run.smithi096.stdout:max_mds 2 2024-04-22T16:00:43.184 INFO:teuthology.orchestra.run.smithi096.stdout:in 0,1 2024-04-22T16:00:43.184 INFO:teuthology.orchestra.run.smithi096.stdout:up {0=34416,1=34420} 2024-04-22T16:00:43.184 INFO:teuthology.orchestra.run.smithi096.stdout:failed 2024-04-22T16:00:43.184 INFO:teuthology.orchestra.run.smithi096.stdout:damaged 2024-04-22T16:00:43.184 INFO:teuthology.orchestra.run.smithi096.stdout:stopped 2024-04-22T16:00:43.184 INFO:teuthology.orchestra.run.smithi096.stdout:data_pools [3] 2024-04-22T16:00:43.184 INFO:teuthology.orchestra.run.smithi096.stdout:metadata_pool 2 2024-04-22T16:00:43.184 INFO:teuthology.orchestra.run.smithi096.stdout:inline_data enabled 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout:balancer 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout:bal_rank_mask -1 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout:standby_count_wanted 1 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout:qdb_cluster leader: 34416 members: 34416,34420 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.dymqlv{0:34416} state up:active seq 470 join_fscid=1 addr [v2:172.21.15.96:6826/1647958878,v1:172.21.15.96:6827/1647958878] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.hwpcvg{0:44279} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.196:6824/4268726265,v1:172.21.15.196:6825/4268726265] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi096.drkkdi{1:34420} state up:active seq 464 join_fscid=1 addr [v2:172.21.15.96:6828/370524327,v1:172.21.15.96:6829/370524327] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.cephfs.smithi196.pbvqlq{1:44757} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.196:6826/4276735029,v1:172.21.15.196:6827/4276735029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:43.185 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:43.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce5406ecc0 msgr2=0x7fce54071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:43.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce5406ecc0 0x7fce54071180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fce58007900 tx=0x7fce58008040 comp rx=0 tx=0).stop 2024-04-22T16:00:43.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 msgr2=0x7fce68197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:43.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 0x7fce68197fe0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fce64015000 tx=0x7fce6400be20 comp rx=0 tx=0).stop 2024-04-22T16:00:43.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 shutdown_connections 2024-04-22T16:00:43.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fce5406ecc0 0x7fce54071180 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:43.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce68100420 0x7fce68197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:43.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 --2- 172.21.15.96:0/707045203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce68100dd0 0x7fce68197fe0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:43.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 >> 172.21.15.96:0/707045203 conn(0x7fce68075f40 msgr2=0x7fce68103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:43.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 shutdown_connections 2024-04-22T16:00:43.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.181+0000 7fce703d7700 1 -- 172.21.15.96:0/707045203 wait complete. 2024-04-22T16:00:43.187 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 40 2024-04-22T16:00:43.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:42 smithi196 ceph-mon[81815]: pgmap v1169: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:43.819 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-22T16:00:43.832 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T16:00:43.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.930+0000 7f5d27b4e640 1 -- 172.21.15.96:0/714901496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d20100c70 msgr2=0x7f5d20104cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:43.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.930+0000 7f5d27b4e640 1 --2- 172.21.15.96:0/714901496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d20100c70 0x7f5d20104cb0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009880 tx=0x7f5d1002bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:43.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.930+0000 7f5d27b4e640 1 -- 172.21.15.96:0/714901496 shutdown_connections 2024-04-22T16:00:43.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.930+0000 7f5d27b4e640 1 --2- 172.21.15.96:0/714901496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d20100c70 0x7f5d20104cb0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:43.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.930+0000 7f5d27b4e640 1 --2- 172.21.15.96:0/714901496 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d201002c0 0x7f5d201006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:43.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.930+0000 7f5d27b4e640 1 -- 172.21.15.96:0/714901496 >> 172.21.15.96:0/714901496 conn(0x7f5d200fbb40 msgr2=0x7f5d200fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:43.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.930+0000 7f5d27b4e640 1 -- 172.21.15.96:0/714901496 shutdown_connections 2024-04-22T16:00:43.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.930+0000 7f5d27b4e640 1 -- 172.21.15.96:0/714901496 wait complete. 2024-04-22T16:00:43.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d27b4e640 1 Processor -- start 2024-04-22T16:00:43.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d27b4e640 1 -- start start 2024-04-22T16:00:43.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d27b4e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d201002c0 0x7f5d200725a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:43.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d27b4e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d20072ae0 0x7f5d20072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d27b4e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d201a1f40 con 0x7f5d201002c0 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d27b4e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d201a2080 con 0x7f5d20072ae0 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d258ea640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d201002c0 0x7f5d200725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d258ea640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d201002c0 0x7f5d200725a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54614/0 (socket says 172.21.15.96:54614) 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d258ea640 1 -- 172.21.15.96:0/3627442031 learned_addr learned my addr 172.21.15.96:0/3627442031 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d258ea640 1 -- 172.21.15.96:0/3627442031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d20072ae0 msgr2=0x7f5d20072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d258ea640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d20072ae0 0x7f5d20072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d258ea640 1 -- 172.21.15.96:0/3627442031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10009560 con 0x7f5d201002c0 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d258ea640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d201002c0 0x7f5d200725a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00b450 tx=0x7f5d1c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:43.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d16ffd640 1 -- 172.21.15.96:0/3627442031 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d1c011860 con 0x7f5d201002c0 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d16ffd640 1 -- 172.21.15.96:0/3627442031 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d1c011e80 con 0x7f5d201002c0 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d16ffd640 1 -- 172.21.15.96:0/3627442031 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d1c00f570 con 0x7f5d201002c0 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.931+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d201a21c0 con 0x7f5d201002c0 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.932+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d201a2720 con 0x7f5d201002c0 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.932+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2004e040 con 0x7f5d201002c0 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.933+0000 7f5d16ffd640 1 -- 172.21.15.96:0/3627442031 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5d1c0103e0 con 0x7f5d201002c0 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.933+0000 7f5d16ffd640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5d0c06ecf0 0x7f5d0c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.933+0000 7f5d16ffd640 1 -- 172.21.15.96:0/3627442031 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f5d1c08eda0 con 0x7f5d201002c0 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.933+0000 7f5d250e9640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5d0c06ecf0 0x7f5d0c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:43.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.934+0000 7f5d250e9640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5d0c06ecf0 0x7f5d0c0711b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5d10036040 tx=0x7f5d10000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:43.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:43.938+0000 7f5d16ffd640 1 -- 172.21.15.96:0/3627442031 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5d1c057b00 con 0x7f5d201002c0 2024-04-22T16:00:44.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.117+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d20100d10 con 0x7f5d201002c0 2024-04-22T16:00:44.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.119+0000 7f5d16ffd640 1 -- 172.21.15.96:0/3627442031 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16771 (secure 0 0 0) 0x7f5d1c05b1c0 con 0x7f5d201002c0 2024-04-22T16:00:44.122 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:44.122 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":84,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","created":"2024-04-22T15:08:23.492994+0000","modified":"2024-04-22T15:43:33.206096+0000","last_up_change":"2024-04-22T15:27:37.407115+0000","last_in_change":"2024-04-22T15:13:56.089739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:13:01.924433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T15:15:32.350570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T15:15:32.982528+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2f1b7772-06b4-467f-b14c-698a0de7357f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6803","nonce":312948544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6805","nonce":312948544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6809","nonce":312948544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6807","nonce":312948544}]},"public_addr":"172.21.15.96:6803/312948544","cluster_addr":"172.21.15.96:6805/312948544","heartbeat_back_addr":"172.21.15.96:6809/312948544","heartbeat_front_addr":"172.21.15.96:6807/312948544","state":["exists","up"]},{"osd":1,"uuid":"59260a36-a2ed-42f9-8f3a-209aae4c4238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6811","nonce":3607865374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6813","nonce":3607865374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6817","nonce":3607865374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6815","nonce":3607865374}]},"public_addr":"172.21.15.96:6811/3607865374","cluster_addr":"172.21.15.96:6813/3607865374","heartbeat_back_addr":"172.21.15.96:6817/3607865374","heartbeat_front_addr":"172.21.15.96:6815/3607865374","state":["exists","up"]},{"osd":2,"uuid":"897e21a9-3009-41c6-acf1-e44414fc69c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6819","nonce":2071585004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6821","nonce":2071585004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6825","nonce":2071585004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6823","nonce":2071585004}]},"public_addr":"172.21.15.96:6819/2071585004","cluster_addr":"172.21.15.96:6821/2071585004","heartbeat_back_addr":"172.21.15.96:6825/2071585004","heartbeat_front_addr":"172.21.15.96:6823/2071585004","state":["exists","up"]},{"osd":3,"uuid":"201c3153-a284-471e-894d-2273001cc29e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6801","nonce":2743487260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6803","nonce":2743487260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6807","nonce":2743487260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6805","nonce":2743487260}]},"public_addr":"172.21.15.196:6801/2743487260","cluster_addr":"172.21.15.196:6803/2743487260","heartbeat_back_addr":"172.21.15.196:6807/2743487260","heartbeat_front_addr":"172.21.15.196:6805/2743487260","state":["exists","up"]},{"osd":4,"uuid":"780ed007-14ef-4c92-a5e9-6c57afcad448","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6809","nonce":2228895659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6811","nonce":2228895659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6815","nonce":2228895659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6813","nonce":2228895659}]},"public_addr":"172.21.15.196:6809/2228895659","cluster_addr":"172.21.15.196:6811/2228895659","heartbeat_back_addr":"172.21.15.196:6815/2228895659","heartbeat_front_addr":"172.21.15.196:6813/2228895659","state":["exists","up"]},{"osd":5,"uuid":"16384d8d-bef4-4f65-9079-e2b4518241da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6817","nonce":288820489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6819","nonce":288820489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6823","nonce":288820489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6821","nonce":288820489}]},"public_addr":"172.21.15.196:6817/288820489","cluster_addr":"172.21.15.196:6819/288820489","heartbeat_back_addr":"172.21.15.196:6823/288820489","heartbeat_front_addr":"172.21.15.196:6821/288820489","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:00.654872+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:28.691804+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:57.737056+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:21.679902+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:47.489372+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:14:13.600900+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6829/3534778299":"2024-04-23T15:27:39.581162+0000","172.21.15.96:6828/3534778299":"2024-04-23T15:27:39.581162+0000","172.21.15.196:0/465874235":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/1920801848":"2024-04-23T15:17:52.351792+0000","172.21.15.96:6801/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6800/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/838799683":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6800/2441203700":"2024-04-23T15:17:52.351792+0000","172.21.15.196:6825/3248990491":"2024-04-23T15:27:39.581179+0000","172.21.15.96:6800/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1319578885":"2024-04-23T15:09:23.920294+0000","172.21.15.196:0/427333217":"2024-04-23T15:19:21.709732+0000","172.21.15.196:6828/2093434004":"2024-04-23T15:19:21.709732+0000","172.21.15.96:6800/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:0/328396090":"2024-04-23T15:23:08.192054+0000","172.21.15.96:0/168995622":"2024-04-23T15:17:52.351792+0000","172.21.15.196:0/4000780168":"2024-04-23T15:19:21.709732+0000","172.21.15.196:6824/3248990491":"2024-04-23T15:27:39.581179+0000","172.21.15.196:0/4127822214":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/340578852":"2024-04-23T15:17:52.351792+0000","172.21.15.96:0/1237353220":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3486391579":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/4224647460":"2024-04-23T15:08:51.463883+0000","172.21.15.96:6801/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3667378212":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1860450164":"2024-04-23T15:17:52.351792+0000","172.21.15.196:0/1567934302":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/3321679733":"2024-04-23T15:09:23.920294+0000","172.21.15.196:0/391452056":"2024-04-23T15:19:21.709732+0000","172.21.15.196:6829/2093434004":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/3921802975":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/2918293154":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6826/887933858":"2024-04-23T15:15:49.819302+0000","172.21.15.96:6827/887933858":"2024-04-23T15:15:49.819302+0000","172.21.15.96:0/3360662441":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6801/2441203700":"2024-04-23T15:17:52.351792+0000","172.21.15.96:0/1794679103":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3926078840":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6800/3666812526":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6801/3666812526":"2024-04-23T15:23:08.192054+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T16:00:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5d0c06ecf0 msgr2=0x7f5d0c0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5d0c06ecf0 0x7f5d0c0711b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5d10036040 tx=0x7f5d10000bc0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d201002c0 msgr2=0x7f5d200725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d201002c0 0x7f5d200725a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00b450 tx=0x7f5d1c00b910 comp rx=0 tx=0).stop 2024-04-22T16:00:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 shutdown_connections 2024-04-22T16:00:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f5d0c06ecf0 0x7f5d0c0711b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d20072ae0 0x7f5d20072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 --2- 172.21.15.96:0/3627442031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d201002c0 0x7f5d200725a0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 >> 172.21.15.96:0/3627442031 conn(0x7f5d200fbb40 msgr2=0x7f5d200fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:44.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 shutdown_connections 2024-04-22T16:00:44.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.121+0000 7f5d27b4e640 1 -- 172.21.15.96:0/3627442031 wait complete. 2024-04-22T16:00:44.133 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:43 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/707045203' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T16:00:44.135 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:13:01.924433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T15:15:32.350570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T15:15:32.982528+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T16:00:44.135 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.234+0000 7fc3429f6640 1 -- 172.21.15.96:0/1824655871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 msgr2=0x7fc33c0ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.234+0000 7fc3429f6640 1 --2- 172.21.15.96:0/1824655871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 0x7fc33c0ff640 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009880 tx=0x7fc32c02bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.234+0000 7fc3429f6640 1 -- 172.21.15.96:0/1824655871 shutdown_connections 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.234+0000 7fc3429f6640 1 --2- 172.21.15.96:0/1824655871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc33c0ffc10 0x7fc33c103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.234+0000 7fc3429f6640 1 --2- 172.21.15.96:0/1824655871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 0x7fc33c0ff640 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.234+0000 7fc3429f6640 1 -- 172.21.15.96:0/1824655871 >> 172.21.15.96:0/1824655871 conn(0x7fc33c074a50 msgr2=0x7fc33c074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.234+0000 7fc3429f6640 1 -- 172.21.15.96:0/1824655871 shutdown_connections 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.234+0000 7fc3429f6640 1 -- 172.21.15.96:0/1824655871 wait complete. 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc3429f6640 1 Processor -- start 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc3429f6640 1 -- start start 2024-04-22T16:00:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc3429f6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 0x7fc33c194cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc3429f6640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc33c1951f0 0x7fc33c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc3429f6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c1957f0 con 0x7fc33c0ff260 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc3429f6640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c195930 con 0x7fc33c1951f0 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc33bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 0x7fc33c194cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc33bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 0x7fc33c194cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54632/0 (socket says 172.21.15.96:54632) 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc33bfff640 1 -- 172.21.15.96:0/3749422190 learned_addr learned my addr 172.21.15.96:0/3749422190 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc33bfff640 1 -- 172.21.15.96:0/3749422190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc33c1951f0 msgr2=0x7fc33c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc33bfff640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc33c1951f0 0x7fc33c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc33bfff640 1 -- 172.21.15.96:0/3749422190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32c009560 con 0x7fc33c0ff260 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.235+0000 7fc33bfff640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 0x7fc33c194cb0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009230 tx=0x7fc32c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.236+0000 7fc3397fa640 1 -- 172.21.15.96:0/3749422190 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc32c0035f0 con 0x7fc33c0ff260 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.236+0000 7fc3397fa640 1 -- 172.21.15.96:0/3749422190 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc32c003c10 con 0x7fc33c0ff260 2024-04-22T16:00:44.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.236+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33c198b40 con 0x7fc33c0ff260 2024-04-22T16:00:44.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.236+0000 7fc3397fa640 1 -- 172.21.15.96:0/3749422190 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc32c005670 con 0x7fc33c0ff260 2024-04-22T16:00:44.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.236+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33c1990a0 con 0x7fc33c0ff260 2024-04-22T16:00:44.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.237+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33c04e040 con 0x7fc33c0ff260 2024-04-22T16:00:44.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.237+0000 7fc3397fa640 1 -- 172.21.15.96:0/3749422190 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc32c004dd0 con 0x7fc33c0ff260 2024-04-22T16:00:44.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.237+0000 7fc3397fa640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32406ebe0 0x7fc3240710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:44.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.237+0000 7fc3397fa640 1 -- 172.21.15.96:0/3749422190 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc32c0b12c0 con 0x7fc33c0ff260 2024-04-22T16:00:44.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.237+0000 7fc33b7fe640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32406ebe0 0x7fc3240710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:44.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.238+0000 7fc33b7fe640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32406ebe0 0x7fc3240710a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc330005fd0 tx=0x7fc330005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:44.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.242+0000 7fc3397fa640 1 -- 172.21.15.96:0/3749422190 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc32c07a020 con 0x7fc33c0ff260 2024-04-22T16:00:44.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:43 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/707045203' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T16:00:44.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.419+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc33c195fc0 con 0x7fc33c0ff260 2024-04-22T16:00:44.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.419+0000 7fc3397fa640 1 -- 172.21.15.96:0/3749422190 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc32c07d6e0 con 0x7fc33c0ff260 2024-04-22T16:00:44.422 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-04-22T16:00:44.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32406ebe0 msgr2=0x7fc3240710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32406ebe0 0x7fc3240710a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc330005fd0 tx=0x7fc330005850 comp rx=0 tx=0).stop 2024-04-22T16:00:44.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 msgr2=0x7fc33c194cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 0x7fc33c194cb0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009230 tx=0x7fc32c004040 comp rx=0 tx=0).stop 2024-04-22T16:00:44.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 shutdown_connections 2024-04-22T16:00:44.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc32406ebe0 0x7fc3240710a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc33c1951f0 0x7fc33c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 --2- 172.21.15.96:0/3749422190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0ff260 0x7fc33c194cb0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 >> 172.21.15.96:0/3749422190 conn(0x7fc33c074a50 msgr2=0x7fc33c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:44.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.421+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 shutdown_connections 2024-04-22T16:00:44.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.422+0000 7fc3429f6640 1 -- 172.21.15.96:0/3749422190 wait complete. 2024-04-22T16:00:44.435 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.533+0000 7fec99ac7640 1 -- 172.21.15.96:0/2275068676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101b20 msgr2=0x7fec9410a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.533+0000 7fec99ac7640 1 --2- 172.21.15.96:0/2275068676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101b20 0x7fec9410a000 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fec88009880 tx=0x7fec8802bd70 comp rx=0 tx=0).stop 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.533+0000 7fec99ac7640 1 -- 172.21.15.96:0/2275068676 shutdown_connections 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.533+0000 7fec99ac7640 1 --2- 172.21.15.96:0/2275068676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101b20 0x7fec9410a000 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.533+0000 7fec99ac7640 1 --2- 172.21.15.96:0/2275068676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec94101200 0x7fec941015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.533+0000 7fec99ac7640 1 -- 172.21.15.96:0/2275068676 >> 172.21.15.96:0/2275068676 conn(0x7fec940faa40 msgr2=0x7fec940fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.533+0000 7fec99ac7640 1 -- 172.21.15.96:0/2275068676 shutdown_connections 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.533+0000 7fec99ac7640 1 -- 172.21.15.96:0/2275068676 wait complete. 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec99ac7640 1 Processor -- start 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec99ac7640 1 -- start start 2024-04-22T16:00:44.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec99ac7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101200 0x7fec94195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec99ac7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec94101b20 0x7fec94196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec99ac7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec94196900 con 0x7fec94101200 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec99ac7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9419a6d0 con 0x7fec94101b20 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec937fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101200 0x7fec94195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec937fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101200 0x7fec94195d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54640/0 (socket says 172.21.15.96:54640) 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec937fe640 1 -- 172.21.15.96:0/786694453 learned_addr learned my addr 172.21.15.96:0/786694453 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.534+0000 7fec92ffd640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec94101b20 0x7fec94196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.535+0000 7fec937fe640 1 -- 172.21.15.96:0/786694453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec94101b20 msgr2=0x7fec94196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.535+0000 7fec937fe640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec94101b20 0x7fec94196270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.535+0000 7fec937fe640 1 -- 172.21.15.96:0/786694453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec88009560 con 0x7fec94101200 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.535+0000 7fec937fe640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101200 0x7fec94195d30 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fec8400c810 tx=0x7fec8400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.535+0000 7fec90ff9640 1 -- 172.21.15.96:0/786694453 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec840041f0 con 0x7fec94101200 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.535+0000 7fec90ff9640 1 -- 172.21.15.96:0/786694453 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec84007e70 con 0x7fec94101200 2024-04-22T16:00:44.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.535+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9419a9b0 con 0x7fec94101200 2024-04-22T16:00:44.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.535+0000 7fec90ff9640 1 -- 172.21.15.96:0/786694453 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec840049d0 con 0x7fec94101200 2024-04-22T16:00:44.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.535+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9419aee0 con 0x7fec94101200 2024-04-22T16:00:44.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.536+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9404e040 con 0x7fec94101200 2024-04-22T16:00:44.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.536+0000 7fec90ff9640 1 -- 172.21.15.96:0/786694453 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fec840079b0 con 0x7fec94101200 2024-04-22T16:00:44.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.537+0000 7fec90ff9640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fec7c06ec30 0x7fec7c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:44.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.537+0000 7fec90ff9640 1 -- 172.21.15.96:0/786694453 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fec8408ec50 con 0x7fec94101200 2024-04-22T16:00:44.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.537+0000 7fec92ffd640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fec7c06ec30 0x7fec7c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:44.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.537+0000 7fec92ffd640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fec7c06ec30 0x7fec7c0710f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fec88006010 tx=0x7fec88005b50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:44.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.542+0000 7fec90ff9640 1 -- 172.21.15.96:0/786694453 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fec84057a60 con 0x7fec94101200 2024-04-22T16:00:44.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.719+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fec94107670 con 0x7fec94101200 2024-04-22T16:00:44.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.719+0000 7fec90ff9640 1 -- 172.21.15.96:0/786694453 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fec8401d020 con 0x7fec94101200 2024-04-22T16:00:44.723 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.721+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fec7c06ec30 msgr2=0x7fec7c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.721+0000 7fec99ac7640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fec7c06ec30 0x7fec7c0710f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fec88006010 tx=0x7fec88005b50 comp rx=0 tx=0).stop 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.721+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101200 msgr2=0x7fec94195d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.721+0000 7fec99ac7640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101200 0x7fec94195d30 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fec8400c810 tx=0x7fec8400ccd0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.721+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 shutdown_connections 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.721+0000 7fec99ac7640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fec7c06ec30 0x7fec7c0710f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.722+0000 7fec99ac7640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec94101b20 0x7fec94196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.722+0000 7fec99ac7640 1 --2- 172.21.15.96:0/786694453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec94101200 0x7fec94195d30 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.722+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 >> 172.21.15.96:0/786694453 conn(0x7fec940faa40 msgr2=0x7fec94104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.722+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 shutdown_connections 2024-04-22T16:00:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.722+0000 7fec99ac7640 1 -- 172.21.15.96:0/786694453 wait complete. 2024-04-22T16:00:44.735 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T16:00:44.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.833+0000 7f7518b0c640 1 -- 172.21.15.96:0/4080090349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75140ffb50 msgr2=0x7f7514103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.833+0000 7f7518b0c640 1 --2- 172.21.15.96:0/4080090349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75140ffb50 0x7f7514103b90 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009880 tx=0x7f74fc02bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.833+0000 7f7518b0c640 1 -- 172.21.15.96:0/4080090349 shutdown_connections 2024-04-22T16:00:44.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.833+0000 7f7518b0c640 1 --2- 172.21.15.96:0/4080090349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75140ffb50 0x7f7514103b90 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.833+0000 7f7518b0c640 1 --2- 172.21.15.96:0/4080090349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75140ff1a0 0x7f75140ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.833+0000 7f7518b0c640 1 -- 172.21.15.96:0/4080090349 >> 172.21.15.96:0/4080090349 conn(0x7f75140faa40 msgr2=0x7f75140fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:44.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.833+0000 7f7518b0c640 1 -- 172.21.15.96:0/4080090349 shutdown_connections 2024-04-22T16:00:44.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.833+0000 7f7518b0c640 1 -- 172.21.15.96:0/4080090349 wait complete. 2024-04-22T16:00:44.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f7518b0c640 1 Processor -- start 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f7518b0c640 1 -- start start 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f7518b0c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75140ff1a0 0x7f7514195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f7518b0c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75141962a0 0x7f751419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f7518b0c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75141968a0 con 0x7f75141962a0 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f7518b0c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7514196a10 con 0x7f75140ff1a0 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f750bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75141962a0 0x7f751419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f750bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75141962a0 0x7f751419a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54664/0 (socket says 172.21.15.96:54664) 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f750bfff640 1 -- 172.21.15.96:0/3021317794 learned_addr learned my addr 172.21.15.96:0/3021317794 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f750bfff640 1 -- 172.21.15.96:0/3021317794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75140ff1a0 msgr2=0x7f7514195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f750bfff640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75140ff1a0 0x7f7514195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.834+0000 7f750bfff640 1 -- 172.21.15.96:0/3021317794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc009560 con 0x7f75141962a0 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.835+0000 7f750bfff640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75141962a0 0x7f751419a700 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f74fc036040 tx=0x7f74fc02c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:44.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.835+0000 7f750b7fe640 1 -- 172.21.15.96:0/3021317794 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74fc0035f0 con 0x7f75141962a0 2024-04-22T16:00:44.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.835+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751419aca0 con 0x7f75141962a0 2024-04-22T16:00:44.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.835+0000 7f750b7fe640 1 -- 172.21.15.96:0/3021317794 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74fc003c10 con 0x7f75141962a0 2024-04-22T16:00:44.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.835+0000 7f750b7fe640 1 -- 172.21.15.96:0/3021317794 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74fc034440 con 0x7f75141962a0 2024-04-22T16:00:44.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.835+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751419b200 con 0x7f75141962a0 2024-04-22T16:00:44.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.836+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751404e040 con 0x7f75141962a0 2024-04-22T16:00:44.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.836+0000 7f750b7fe640 1 -- 172.21.15.96:0/3021317794 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f74fc003750 con 0x7f75141962a0 2024-04-22T16:00:44.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.837+0000 7f750b7fe640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f750006ec30 0x7f75000710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:44.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.837+0000 7f750b7fe640 1 -- 172.21.15.96:0/3021317794 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f74fc0b1270 con 0x7f75141962a0 2024-04-22T16:00:44.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.837+0000 7f751259c640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f750006ec30 0x7f75000710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:44.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.837+0000 7f751259c640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f750006ec30 0x7f75000710f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7504005fd0 tx=0x7f7504005c70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:44.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:44.842+0000 7f750b7fe640 1 -- 172.21.15.96:0/3021317794 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f74fc079fd0 con 0x7f75141962a0 2024-04-22T16:00:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:44 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3627442031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:00:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:44 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3749422190' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:00:44.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:44 smithi096 ceph-mon[112443]: pgmap v1170: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:44.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:44 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/786694453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:00:45.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.021+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f75140ffb50 con 0x7f75141962a0 2024-04-22T16:00:45.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.021+0000 7f750b7fe640 1 -- 172.21.15.96:0/3021317794 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f74fc09ce60 con 0x7f75141962a0 2024-04-22T16:00:45.024 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-04-22T16:00:45.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.023+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f750006ec30 msgr2=0x7f75000710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.023+0000 7f7518b0c640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f750006ec30 0x7f75000710f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7504005fd0 tx=0x7f7504005c70 comp rx=0 tx=0).stop 2024-04-22T16:00:45.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.023+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75141962a0 msgr2=0x7f751419a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.023+0000 7f7518b0c640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75141962a0 0x7f751419a700 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f74fc036040 tx=0x7f74fc02c850 comp rx=0 tx=0).stop 2024-04-22T16:00:45.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.024+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 shutdown_connections 2024-04-22T16:00:45.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.024+0000 7f7518b0c640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f750006ec30 0x7f75000710f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.024+0000 7f7518b0c640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75141962a0 0x7f751419a700 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.024+0000 7f7518b0c640 1 --2- 172.21.15.96:0/3021317794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75140ff1a0 0x7f7514195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.024+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 >> 172.21.15.96:0/3021317794 conn(0x7f75140faa40 msgr2=0x7f75140fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:45.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.024+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 shutdown_connections 2024-04-22T16:00:45.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.024+0000 7f7518b0c640 1 -- 172.21.15.96:0/3021317794 wait complete. 2024-04-22T16:00:45.038 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T16:00:45.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.133+0000 7f4d6528f640 1 -- 172.21.15.96:0/2322368367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 msgr2=0x7f4d600ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.133+0000 7f4d6528f640 1 --2- 172.21.15.96:0/2322368367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 0x7f4d600ff440 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4d54009880 tx=0x7f4d5402bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.133+0000 7f4d6528f640 1 -- 172.21.15.96:0/2322368367 shutdown_connections 2024-04-22T16:00:45.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.133+0000 7f4d6528f640 1 --2- 172.21.15.96:0/2322368367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 0x7f4d600ff440 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.133+0000 7f4d6528f640 1 --2- 172.21.15.96:0/2322368367 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d60103f20 0x7f4d60104300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.133+0000 7f4d6528f640 1 -- 172.21.15.96:0/2322368367 >> 172.21.15.96:0/2322368367 conn(0x7f4d60075b40 msgr2=0x7f4d60075f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.133+0000 7f4d6528f640 1 -- 172.21.15.96:0/2322368367 shutdown_connections 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.133+0000 7f4d6528f640 1 -- 172.21.15.96:0/2322368367 wait complete. 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.133+0000 7f4d6528f640 1 Processor -- start 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d6528f640 1 -- start start 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d6528f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 0x7f4d60197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d6528f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d60103f20 0x7f4d60198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d6528f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60198ab0 con 0x7f4d600fefe0 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d6528f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6019c880 con 0x7f4d60103f20 2024-04-22T16:00:45.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d5effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 0x7f4d60197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d5effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 0x7f4d60197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54686/0 (socket says 172.21.15.96:54686) 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d5effd640 1 -- 172.21.15.96:0/3610477103 learned_addr learned my addr 172.21.15.96:0/3610477103 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d5e7fc640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d60103f20 0x7f4d60198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d5effd640 1 -- 172.21.15.96:0/3610477103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d60103f20 msgr2=0x7f4d60198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d5effd640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d60103f20 0x7f4d60198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d5effd640 1 -- 172.21.15.96:0/3610477103 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d54009560 con 0x7f4d600fefe0 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d5e7fc640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d60103f20 0x7f4d60198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.134+0000 7f4d5effd640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 0x7f4d60197ee0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800d8d0 tx=0x7f4d5800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.135+0000 7f4d47fff640 1 -- 172.21.15.96:0/3610477103 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d5800bea0 con 0x7f4d600fefe0 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.135+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6019cb60 con 0x7f4d600fefe0 2024-04-22T16:00:45.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.135+0000 7f4d47fff640 1 -- 172.21.15.96:0/3610477103 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d58009c00 con 0x7f4d600fefe0 2024-04-22T16:00:45.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.135+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6019d0c0 con 0x7f4d600fefe0 2024-04-22T16:00:45.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.135+0000 7f4d47fff640 1 -- 172.21.15.96:0/3610477103 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d5800f710 con 0x7f4d600fefe0 2024-04-22T16:00:45.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.135+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6004e040 con 0x7f4d600fefe0 2024-04-22T16:00:45.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.136+0000 7f4d47fff640 1 -- 172.21.15.96:0/3610477103 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4d5800f990 con 0x7f4d600fefe0 2024-04-22T16:00:45.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.136+0000 7f4d47fff640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecf0 0x7f4d480711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:45.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.136+0000 7f4d47fff640 1 -- 172.21.15.96:0/3610477103 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4d5808ec10 con 0x7f4d600fefe0 2024-04-22T16:00:45.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.136+0000 7f4d5e7fc640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecf0 0x7f4d480711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:45.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.137+0000 7f4d5e7fc640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecf0 0x7f4d480711b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4d601994f0 tx=0x7f4d5402cb40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:45.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.141+0000 7f4d47fff640 1 -- 172.21.15.96:0/3610477103 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4d58057970 con 0x7f4d600fefe0 2024-04-22T16:00:45.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:44 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3627442031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:00:45.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:44 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3749422190' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:00:45.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:44 smithi196 ceph-mon[81815]: pgmap v1170: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:45.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:44 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/786694453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:00:45.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.357+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d60199230 con 0x7f4d600fefe0 2024-04-22T16:00:45.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.358+0000 7f4d47fff640 1 -- 172.21.15.96:0/3610477103 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5251 (secure 0 0 0) 0x7f4d60199230 con 0x7f4d600fefe0 2024-04-22T16:00:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:45.361 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:59:04.839573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":0,"incarnation":33,"state":"up:active","state_seq":470,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":1,"incarnation":33,"state":"up:active","state_seq":464,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44279":{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44757":{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34416,"qdb_cluster":[34416,34420]},"id":1}]} 2024-04-22T16:00:45.361 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 40 2024-04-22T16:00:45.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecf0 msgr2=0x7f4d480711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecf0 0x7f4d480711b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4d601994f0 tx=0x7f4d5402cb40 comp rx=0 tx=0).stop 2024-04-22T16:00:45.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 msgr2=0x7f4d60197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 0x7f4d60197ee0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800d8d0 tx=0x7f4d5800dd90 comp rx=0 tx=0).stop 2024-04-22T16:00:45.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 shutdown_connections 2024-04-22T16:00:45.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4d4806ecf0 0x7f4d480711b0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d60103f20 0x7f4d60198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 --2- 172.21.15.96:0/3610477103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d600fefe0 0x7f4d60197ee0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 >> 172.21.15.96:0/3610477103 conn(0x7f4d60075b40 msgr2=0x7f4d60067370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:45.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 shutdown_connections 2024-04-22T16:00:45.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.360+0000 7f4d6528f640 1 -- 172.21.15.96:0/3610477103 wait complete. 2024-04-22T16:00:45.374 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 20, 'max_mds': 2, 'flags': 50} 2024-04-22T16:00:45.374 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-22T16:00:45.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.470+0000 7f7c4210c640 1 -- 172.21.15.96:0/1251734938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c0fe730 msgr2=0x7f7c3c0feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.470+0000 7f7c4210c640 1 --2- 172.21.15.96:0/1251734938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c0fe730 0x7f7c3c0feb70 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009880 tx=0x7f7c2402bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.470+0000 7f7c4210c640 1 -- 172.21.15.96:0/1251734938 shutdown_connections 2024-04-22T16:00:45.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.470+0000 7f7c4210c640 1 --2- 172.21.15.96:0/1251734938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c0fe730 0x7f7c3c0feb70 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.470+0000 7f7c4210c640 1 --2- 172.21.15.96:0/1251734938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c3c106070 0x7f7c3c106450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.470+0000 7f7c4210c640 1 -- 172.21.15.96:0/1251734938 >> 172.21.15.96:0/1251734938 conn(0x7f7c3c074250 msgr2=0x7f7c3c074660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.470+0000 7f7c4210c640 1 -- 172.21.15.96:0/1251734938 shutdown_connections 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.470+0000 7f7c4210c640 1 -- 172.21.15.96:0/1251734938 wait complete. 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c4210c640 1 Processor -- start 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c4210c640 1 -- start start 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c4210c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c3c0fe730 0x7f7c3c19bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c4210c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c106070 0x7f7c3c19c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c4210c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c195b30 con 0x7f7c3c106070 2024-04-22T16:00:45.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c4210c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c195c70 con 0x7f7c3c0fe730 2024-04-22T16:00:45.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c3affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c106070 0x7f7c3c19c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:45.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c3affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c106070 0x7f7c3c19c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54700/0 (socket says 172.21.15.96:54700) 2024-04-22T16:00:45.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c3affd640 1 -- 172.21.15.96:0/1233121019 learned_addr learned my addr 172.21.15.96:0/1233121019 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:45.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.471+0000 7f7c3b7fe640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c3c0fe730 0x7f7c3c19bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:45.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.472+0000 7f7c3affd640 1 -- 172.21.15.96:0/1233121019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c3c0fe730 msgr2=0x7f7c3c19bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.472+0000 7f7c3affd640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c3c0fe730 0x7f7c3c19bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.472+0000 7f7c3affd640 1 -- 172.21.15.96:0/1233121019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c24009560 con 0x7f7c3c106070 2024-04-22T16:00:45.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.472+0000 7f7c3affd640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c106070 0x7f7c3c19c090 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009230 tx=0x7f7c24004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:45.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.472+0000 7f7c38ff9640 1 -- 172.21.15.96:0/1233121019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c240035f0 con 0x7f7c3c106070 2024-04-22T16:00:45.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.472+0000 7f7c38ff9640 1 -- 172.21.15.96:0/1233121019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c24003c10 con 0x7f7c3c106070 2024-04-22T16:00:45.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.472+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c195ef0 con 0x7f7c3c106070 2024-04-22T16:00:45.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.472+0000 7f7c38ff9640 1 -- 172.21.15.96:0/1233121019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c24005670 con 0x7f7c3c106070 2024-04-22T16:00:45.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.472+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c1963f0 con 0x7f7c3c106070 2024-04-22T16:00:45.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.473+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c04e040 con 0x7f7c3c106070 2024-04-22T16:00:45.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.473+0000 7f7c38ff9640 1 -- 172.21.15.96:0/1233121019 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7c24004dd0 con 0x7f7c3c106070 2024-04-22T16:00:45.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.474+0000 7f7c38ff9640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c2806ebe0 0x7f7c280710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:45.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.474+0000 7f7c38ff9640 1 -- 172.21.15.96:0/1233121019 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7c240b1210 con 0x7f7c3c106070 2024-04-22T16:00:45.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.474+0000 7f7c3b7fe640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c2806ebe0 0x7f7c280710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:45.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.474+0000 7f7c3b7fe640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c2806ebe0 0x7f7c280710a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c005fd0 tx=0x7f7c2c00b390 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:45.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.479+0000 7f7c38ff9640 1 -- 172.21.15.96:0/1233121019 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7c2407a020 con 0x7f7c3c106070 2024-04-22T16:00:45.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.695+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f7c3c10d2f0 con 0x7f7c3c106070 2024-04-22T16:00:45.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.695+0000 7f7c38ff9640 1 -- 172.21.15.96:0/1233121019 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v40) v1 ==== 107+0+3334 (secure 0 0 0) 0x7f7c24041d20 con 0x7f7c3c106070 2024-04-22T16:00:45.699 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:45.699 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.419933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14494,"mds_1":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14494":{"gid":14494,"name":"cephfs.smithi096.drkkdi","rank":0,"incarnation":15,"state":"up:active","state_seq":6,"addr":"172.21.15.96:6829/3534778299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3534778299},{"type":"v1","addr":"172.21.15.96:6829","nonce":3534778299}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24257":{"gid":24257,"name":"cephfs.smithi196.hwpcvg","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6825/3248990491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":3248990491},{"type":"v1","addr":"172.21.15.196:6825","nonce":3248990491}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14494,"qdb_cluster":[14494,24257]},"id":1}]} 2024-04-22T16:00:45.699 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 21 2024-04-22T16:00:45.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.697+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c2806ebe0 msgr2=0x7f7c280710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.697+0000 7f7c4210c640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c2806ebe0 0x7f7c280710a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c005fd0 tx=0x7f7c2c00b390 comp rx=0 tx=0).stop 2024-04-22T16:00:45.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.697+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c106070 msgr2=0x7f7c3c19c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.697+0000 7f7c4210c640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c106070 0x7f7c3c19c090 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009230 tx=0x7f7c24004040 comp rx=0 tx=0).stop 2024-04-22T16:00:45.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.697+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 shutdown_connections 2024-04-22T16:00:45.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.697+0000 7f7c4210c640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7c2806ebe0 0x7f7c280710a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.697+0000 7f7c4210c640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c3c106070 0x7f7c3c19c090 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.697+0000 7f7c4210c640 1 --2- 172.21.15.96:0/1233121019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c3c0fe730 0x7f7c3c19bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.697+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 >> 172.21.15.96:0/1233121019 conn(0x7f7c3c074250 msgr2=0x7f7c3c0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:45.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.698+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 shutdown_connections 2024-04-22T16:00:45.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.698+0000 7f7c4210c640 1 -- 172.21.15.96:0/1233121019 wait complete. 2024-04-22T16:00:45.712 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 21 2024-04-22T16:00:45.712 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-22T16:00:45.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.808+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2000289171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c240ff260 msgr2=0x7f2c240ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.808+0000 7f2c2c07d640 1 --2- 172.21.15.96:0/2000289171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c240ff260 0x7f2c240ff640 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009880 tx=0x7f2c2002bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.808+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2000289171 shutdown_connections 2024-04-22T16:00:45.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.808+0000 7f2c2c07d640 1 --2- 172.21.15.96:0/2000289171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c240ffc10 0x7f2c24103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.808+0000 7f2c2c07d640 1 --2- 172.21.15.96:0/2000289171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c240ff260 0x7f2c240ff640 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.808+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2000289171 >> 172.21.15.96:0/2000289171 conn(0x7f2c24074a50 msgr2=0x7f2c24074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.808+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2000289171 shutdown_connections 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.808+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2000289171 wait complete. 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c2c07d640 1 Processor -- start 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c2c07d640 1 -- start start 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c2c07d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c240ff260 0x7f2c2419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c2c07d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c2419afa0 0x7f2c24194ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c2c07d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2419b570 con 0x7f2c2419afa0 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c2c07d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c24195010 con 0x7f2c240ff260 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c29618640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c2419afa0 0x7f2c24194ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c29618640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c2419afa0 0x7f2c24194ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54720/0 (socket says 172.21.15.96:54720) 2024-04-22T16:00:45.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c29618640 1 -- 172.21.15.96:0/2859322483 learned_addr learned my addr 172.21.15.96:0/2859322483 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c29e19640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c240ff260 0x7f2c2419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c29618640 1 -- 172.21.15.96:0/2859322483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c240ff260 msgr2=0x7f2c2419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c29618640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c240ff260 0x7f2c2419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.809+0000 7f2c29618640 1 -- 172.21.15.96:0/2859322483 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c20009560 con 0x7f2c2419afa0 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.810+0000 7f2c29618640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c2419afa0 0x7f2c24194ad0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400d6f0 tx=0x7f2c1400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.810+0000 7f2c1affd640 1 -- 172.21.15.96:0/2859322483 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c14009a30 con 0x7f2c2419afa0 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.810+0000 7f2c1affd640 1 -- 172.21.15.96:0/2859322483 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c14009b90 con 0x7f2c2419afa0 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.810+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c241952f0 con 0x7f2c2419afa0 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.810+0000 7f2c1affd640 1 -- 172.21.15.96:0/2859322483 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1400f680 con 0x7f2c2419afa0 2024-04-22T16:00:45.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.810+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c24195850 con 0x7f2c2419afa0 2024-04-22T16:00:45.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.811+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2404e040 con 0x7f2c2419afa0 2024-04-22T16:00:45.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.811+0000 7f2c1affd640 1 -- 172.21.15.96:0/2859322483 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2c14009d00 con 0x7f2c2419afa0 2024-04-22T16:00:45.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.811+0000 7f2c1affd640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2c1006ec30 0x7f2c100710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:45.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.812+0000 7f2c1affd640 1 -- 172.21.15.96:0/2859322483 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2c1408ed10 con 0x7f2c2419afa0 2024-04-22T16:00:45.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.812+0000 7f2c29e19640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2c1006ec30 0x7f2c100710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:45.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.812+0000 7f2c29e19640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2c1006ec30 0x7f2c100710f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2c20006010 tx=0x7f2c20005dd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:45.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:45.816+0000 7f2c1affd640 1 -- 172.21.15.96:0/2859322483 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2c14057a70 con 0x7f2c2419afa0 2024-04-22T16:00:45.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:45 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3021317794' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:00:45.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:45 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3610477103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T16:00:45.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:45 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1233121019' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-22T16:00:46.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.034+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f2c240ffc10 con 0x7f2c2419afa0 2024-04-22T16:00:46.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.034+0000 7f2c1affd640 1 -- 172.21.15.96:0/2859322483 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v40) v1 ==== 107+0+3298 (secure 0 0 0) 0x7f2c14017d00 con 0x7f2c2419afa0 2024-04-22T16:00:46.037 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:46.037 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34376,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1411150341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1411150341},{"type":"v1","addr":"172.21.15.96:6827","nonce":1411150341}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":44225,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/2911111435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2911111435},{"type":"v1","addr":"172.21.15.196:6827","nonce":2911111435}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:46.037 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 22 2024-04-22T16:00:46.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.036+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2c1006ec30 msgr2=0x7f2c100710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.036+0000 7f2c2c07d640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2c1006ec30 0x7f2c100710f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2c20006010 tx=0x7f2c20005dd0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.036+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c2419afa0 msgr2=0x7f2c24194ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.036+0000 7f2c2c07d640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c2419afa0 0x7f2c24194ad0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400d6f0 tx=0x7f2c1400dbb0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.036+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 shutdown_connections 2024-04-22T16:00:46.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.036+0000 7f2c2c07d640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f2c1006ec30 0x7f2c100710f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.036+0000 7f2c2c07d640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c2419afa0 0x7f2c24194ad0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.036+0000 7f2c2c07d640 1 --2- 172.21.15.96:0/2859322483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c240ff260 0x7f2c2419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.036+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 >> 172.21.15.96:0/2859322483 conn(0x7f2c24074a50 msgr2=0x7f2c240fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:46.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.037+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 shutdown_connections 2024-04-22T16:00:46.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.037+0000 7f2c2c07d640 1 -- 172.21.15.96:0/2859322483 wait complete. 2024-04-22T16:00:46.050 DEBUG:tasks.fs:max_mds not reduced in epoch 22 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:46.050 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 22 2024-04-22T16:00:46.050 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.146+0000 7f6817d3c640 1 -- 172.21.15.96:0/3146095199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68100ff260 msgr2=0x7f68100ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.146+0000 7f6817d3c640 1 --2- 172.21.15.96:0/3146095199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68100ff260 0x7f68100ff640 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f680c009880 tx=0x7f680c02bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.146+0000 7f6817d3c640 1 -- 172.21.15.96:0/3146095199 shutdown_connections 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.146+0000 7f6817d3c640 1 --2- 172.21.15.96:0/3146095199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68100ffc10 0x7f6810103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.146+0000 7f6817d3c640 1 --2- 172.21.15.96:0/3146095199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68100ff260 0x7f68100ff640 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.146+0000 7f6817d3c640 1 -- 172.21.15.96:0/3146095199 >> 172.21.15.96:0/3146095199 conn(0x7f6810074a50 msgr2=0x7f6810074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.146+0000 7f6817d3c640 1 -- 172.21.15.96:0/3146095199 shutdown_connections 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.146+0000 7f6817d3c640 1 -- 172.21.15.96:0/3146095199 wait complete. 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f6817d3c640 1 Processor -- start 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f6817d3c640 1 -- start start 2024-04-22T16:00:46.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f6817d3c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68100ff260 0x7f6810196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f6817d3c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68101972a0 0x7f6810197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f6817d3c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681019fdd0 con 0x7f68101972a0 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f6817d3c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681019ff10 con 0x7f68100ff260 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f68152d7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68101972a0 0x7f6810197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f68152d7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68101972a0 0x7f6810197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54744/0 (socket says 172.21.15.96:54744) 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f68152d7640 1 -- 172.21.15.96:0/2592882336 learned_addr learned my addr 172.21.15.96:0/2592882336 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.147+0000 7f6815ad8640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68100ff260 0x7f6810196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.148+0000 7f68152d7640 1 -- 172.21.15.96:0/2592882336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68100ff260 msgr2=0x7f6810196d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.148+0000 7f68152d7640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68100ff260 0x7f6810196d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.148+0000 7f68152d7640 1 -- 172.21.15.96:0/2592882336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c009560 con 0x7f68101972a0 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.148+0000 7f68152d7640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68101972a0 0x7f6810197700 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f680000b400 tx=0x7f680000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.148+0000 7f6806ffd640 1 -- 172.21.15.96:0/2592882336 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6800011800 con 0x7f68101972a0 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.148+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68101a0170 con 0x7f68101972a0 2024-04-22T16:00:46.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.148+0000 7f6806ffd640 1 -- 172.21.15.96:0/2592882336 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6800011e20 con 0x7f68101972a0 2024-04-22T16:00:46.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.148+0000 7f6806ffd640 1 -- 172.21.15.96:0/2592882336 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f680000f2e0 con 0x7f68101972a0 2024-04-22T16:00:46.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.148+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68101a0680 con 0x7f68101972a0 2024-04-22T16:00:46.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.149+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681004e040 con 0x7f68101972a0 2024-04-22T16:00:46.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.149+0000 7f6806ffd640 1 -- 172.21.15.96:0/2592882336 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6800010380 con 0x7f68101972a0 2024-04-22T16:00:46.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.150+0000 7f6806ffd640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f67fc06ec30 0x7f67fc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:46.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.150+0000 7f6806ffd640 1 -- 172.21.15.96:0/2592882336 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f680008eb40 con 0x7f68101972a0 2024-04-22T16:00:46.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.150+0000 7f6815ad8640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f67fc06ec30 0x7f67fc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:46.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.150+0000 7f6815ad8640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f67fc06ec30 0x7f67fc0710f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f680c006010 tx=0x7f680c000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:46.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.155+0000 7f6806ffd640 1 -- 172.21.15.96:0/2592882336 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f68000578a0 con 0x7f68101972a0 2024-04-22T16:00:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:45 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3021317794' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:00:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:45 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3610477103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T16:00:46.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:45 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1233121019' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-22T16:00:46.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.369+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f6810107480 con 0x7f68101972a0 2024-04-22T16:00:46.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.369+0000 7f6806ffd640 1 -- 172.21.15.96:0/2592882336 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v40) v1 ==== 107+0+4859 (secure 0 0 0) 0x7f6800057070 con 0x7f68101972a0 2024-04-22T16:00:46.372 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:46.372 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34376,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1411150341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1411150341},{"type":"v1","addr":"172.21.15.96:6827","nonce":1411150341}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":34384,"name":"cephfs.smithi096.drkkdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6829/1619485870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1619485870},{"type":"v1","addr":"172.21.15.96:6829","nonce":1619485870}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44225,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/2911111435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2911111435},{"type":"v1","addr":"172.21.15.196:6827","nonce":2911111435}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":44233,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4222304137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4222304137},{"type":"v1","addr":"172.21.15.196:6825","nonce":4222304137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:46.372 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 23 2024-04-22T16:00:46.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f67fc06ec30 msgr2=0x7f67fc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f67fc06ec30 0x7f67fc0710f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f680c006010 tx=0x7f680c000bc0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68101972a0 msgr2=0x7f6810197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68101972a0 0x7f6810197700 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f680000b400 tx=0x7f680000b8c0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 shutdown_connections 2024-04-22T16:00:46.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f67fc06ec30 0x7f67fc0710f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68101972a0 0x7f6810197700 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 --2- 172.21.15.96:0/2592882336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68100ff260 0x7f6810196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 >> 172.21.15.96:0/2592882336 conn(0x7f6810074a50 msgr2=0x7f68100fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:46.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 shutdown_connections 2024-04-22T16:00:46.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.371+0000 7f6817d3c640 1 -- 172.21.15.96:0/2592882336 wait complete. 2024-04-22T16:00:46.386 DEBUG:tasks.fs:max_mds not reduced in epoch 23 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:46.386 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 23 2024-04-22T16:00:46.386 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-22T16:00:46.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.482+0000 7fab4cc0b640 1 -- 172.21.15.96:0/3733945996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab480ff1a0 msgr2=0x7fab480ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.482+0000 7fab4cc0b640 1 --2- 172.21.15.96:0/3733945996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab480ff1a0 0x7fab480ff580 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fab38009880 tx=0x7fab3802bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.482+0000 7fab4cc0b640 1 -- 172.21.15.96:0/3733945996 shutdown_connections 2024-04-22T16:00:46.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.482+0000 7fab4cc0b640 1 --2- 172.21.15.96:0/3733945996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab480ffb50 0x7fab48103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.482+0000 7fab4cc0b640 1 --2- 172.21.15.96:0/3733945996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab480ff1a0 0x7fab480ff580 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.482+0000 7fab4cc0b640 1 -- 172.21.15.96:0/3733945996 >> 172.21.15.96:0/3733945996 conn(0x7fab480faa40 msgr2=0x7fab480fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:46.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.482+0000 7fab4cc0b640 1 -- 172.21.15.96:0/3733945996 shutdown_connections 2024-04-22T16:00:46.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.482+0000 7fab4cc0b640 1 -- 172.21.15.96:0/3733945996 wait complete. 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab4cc0b640 1 Processor -- start 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab4cc0b640 1 -- start start 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab4cc0b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab480ff1a0 0x7fab481980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab4cc0b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab481985f0 0x7fab4819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab4cc0b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48198bf0 con 0x7fab481985f0 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab4cc0b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48198d60 con 0x7fab480ff1a0 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab45d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab481985f0 0x7fab4819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab45d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab481985f0 0x7fab4819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54762/0 (socket says 172.21.15.96:54762) 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab45d9b640 1 -- 172.21.15.96:0/213928637 learned_addr learned my addr 172.21.15.96:0/213928637 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab45d9b640 1 -- 172.21.15.96:0/213928637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab480ff1a0 msgr2=0x7fab481980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab4659c640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab480ff1a0 0x7fab481980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab45d9b640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab480ff1a0 0x7fab481980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab45d9b640 1 -- 172.21.15.96:0/213928637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab38009560 con 0x7fab481985f0 2024-04-22T16:00:46.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.483+0000 7fab4659c640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab480ff1a0 0x7fab481980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:46.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.484+0000 7fab45d9b640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab481985f0 0x7fab4819ca50 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00b400 tx=0x7fab3c00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:46.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.484+0000 7fab377fe640 1 -- 172.21.15.96:0/213928637 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab3c011800 con 0x7fab481985f0 2024-04-22T16:00:46.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.484+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4819d050 con 0x7fab481985f0 2024-04-22T16:00:46.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.484+0000 7fab377fe640 1 -- 172.21.15.96:0/213928637 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab3c011e20 con 0x7fab481985f0 2024-04-22T16:00:46.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.484+0000 7fab377fe640 1 -- 172.21.15.96:0/213928637 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab3c00f2e0 con 0x7fab481985f0 2024-04-22T16:00:46.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.484+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4819d5b0 con 0x7fab481985f0 2024-04-22T16:00:46.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.485+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4804e040 con 0x7fab481985f0 2024-04-22T16:00:46.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.485+0000 7fab377fe640 1 -- 172.21.15.96:0/213928637 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fab3c010380 con 0x7fab481985f0 2024-04-22T16:00:46.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.485+0000 7fab377fe640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab3006ec30 0x7fab300710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:46.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.485+0000 7fab377fe640 1 -- 172.21.15.96:0/213928637 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fab3c08eba0 con 0x7fab481985f0 2024-04-22T16:00:46.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.486+0000 7fab4659c640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab3006ec30 0x7fab300710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:46.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.486+0000 7fab4659c640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab3006ec30 0x7fab300710f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fab38036040 tx=0x7fab3802cf90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:46.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.490+0000 7fab377fe640 1 -- 172.21.15.96:0/213928637 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fab3c057900 con 0x7fab481985f0 2024-04-22T16:00:46.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.704+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7fab480ffb50 con 0x7fab481985f0 2024-04-22T16:00:46.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.705+0000 7fab377fe640 1 -- 172.21.15.96:0/213928637 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v40) v1 ==== 107+0+4080 (secure 0 0 0) 0x7fab3c05afc0 con 0x7fab481985f0 2024-04-22T16:00:46.708 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:46.708 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi096.drkkdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6829/1619485870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1619485870},{"type":"v1","addr":"172.21.15.96:6829","nonce":1619485870}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44225,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/2911111435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2911111435},{"type":"v1","addr":"172.21.15.196:6827","nonce":2911111435}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":44233,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4222304137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4222304137},{"type":"v1","addr":"172.21.15.196:6825","nonce":4222304137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:46.708 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 24 2024-04-22T16:00:46.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab3006ec30 msgr2=0x7fab300710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab3006ec30 0x7fab300710f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fab38036040 tx=0x7fab3802cf90 comp rx=0 tx=0).stop 2024-04-22T16:00:46.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab481985f0 msgr2=0x7fab4819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab481985f0 0x7fab4819ca50 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00b400 tx=0x7fab3c00b8c0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 shutdown_connections 2024-04-22T16:00:46.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fab3006ec30 0x7fab300710f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab481985f0 0x7fab4819ca50 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 --2- 172.21.15.96:0/213928637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab480ff1a0 0x7fab481980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 >> 172.21.15.96:0/213928637 conn(0x7fab480faa40 msgr2=0x7fab48068c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:46.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 shutdown_connections 2024-04-22T16:00:46.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.707+0000 7fab4cc0b640 1 -- 172.21.15.96:0/213928637 wait complete. 2024-04-22T16:00:46.721 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:46.721 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 24 2024-04-22T16:00:46.721 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-22T16:00:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.817+0000 7f3a949ae640 1 -- 172.21.15.96:0/2174952561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a900ff260 msgr2=0x7f3a900ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.817+0000 7f3a949ae640 1 --2- 172.21.15.96:0/2174952561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a900ff260 0x7f3a900ff640 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009880 tx=0x7f3a8002bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.817+0000 7f3a949ae640 1 -- 172.21.15.96:0/2174952561 shutdown_connections 2024-04-22T16:00:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.817+0000 7f3a949ae640 1 --2- 172.21.15.96:0/2174952561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a900ffc10 0x7f3a90103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.817+0000 7f3a949ae640 1 --2- 172.21.15.96:0/2174952561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a900ff260 0x7f3a900ff640 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.817+0000 7f3a949ae640 1 -- 172.21.15.96:0/2174952561 >> 172.21.15.96:0/2174952561 conn(0x7f3a90074a50 msgr2=0x7f3a90074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.817+0000 7f3a949ae640 1 -- 172.21.15.96:0/2174952561 shutdown_connections 2024-04-22T16:00:46.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.817+0000 7f3a949ae640 1 -- 172.21.15.96:0/2174952561 wait complete. 2024-04-22T16:00:46.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a949ae640 1 Processor -- start 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a949ae640 1 -- start start 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a949ae640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a900ff260 0x7f3a90195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a949ae640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a901962f0 0x7f3a9019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a949ae640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a901968f0 con 0x7f3a901962f0 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a949ae640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90196a60 con 0x7f3a900ff260 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a8dd9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a901962f0 0x7f3a9019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a8dd9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a901962f0 0x7f3a9019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54776/0 (socket says 172.21.15.96:54776) 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a8dd9b640 1 -- 172.21.15.96:0/493682951 learned_addr learned my addr 172.21.15.96:0/493682951 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a8e59c640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a900ff260 0x7f3a90195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a8dd9b640 1 -- 172.21.15.96:0/493682951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a900ff260 msgr2=0x7f3a90195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a8dd9b640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a900ff260 0x7f3a90195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.818+0000 7f3a8dd9b640 1 -- 172.21.15.96:0/493682951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a80009560 con 0x7f3a901962f0 2024-04-22T16:00:46.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.819+0000 7f3a8dd9b640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a901962f0 0x7f3a9019a750 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400d6f0 tx=0x7f3a8400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:46.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.819+0000 7f3a7f7fe640 1 -- 172.21.15.96:0/493682951 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a840051f0 con 0x7f3a901962f0 2024-04-22T16:00:46.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.819+0000 7f3a7f7fe640 1 -- 172.21.15.96:0/493682951 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a84009ce0 con 0x7f3a901962f0 2024-04-22T16:00:46.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.819+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9019ad50 con 0x7f3a901962f0 2024-04-22T16:00:46.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.819+0000 7f3a7f7fe640 1 -- 172.21.15.96:0/493682951 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a84004e80 con 0x7f3a901962f0 2024-04-22T16:00:46.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.819+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9019b2b0 con 0x7f3a901962f0 2024-04-22T16:00:46.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.820+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9019b580 con 0x7f3a901962f0 2024-04-22T16:00:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.820+0000 7f3a7f7fe640 1 -- 172.21.15.96:0/493682951 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3a840046c0 con 0x7f3a901962f0 2024-04-22T16:00:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.821+0000 7f3a7f7fe640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a7806ec30 0x7f3a780710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.821+0000 7f3a7f7fe640 1 -- 172.21.15.96:0/493682951 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3a8408e800 con 0x7f3a901962f0 2024-04-22T16:00:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.821+0000 7f3a8e59c640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a7806ec30 0x7f3a780710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.821+0000 7f3a8e59c640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a7806ec30 0x7f3a780710f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006010 tx=0x7f3a80000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:46.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:46.827+0000 7f3a7f7fe640 1 -- 172.21.15.96:0/493682951 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3a840575e0 con 0x7f3a901962f0 2024-04-22T16:00:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:46 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2859322483' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-22T16:00:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:46 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2592882336' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-22T16:00:46.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:46 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/213928637' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-22T16:00:46.986 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:46 smithi096 ceph-mon[112443]: pgmap v1171: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:47.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.064+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f3a9004e040 con 0x7f3a901962f0 2024-04-22T16:00:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.065+0000 7f3a7f7fe640 1 -- 172.21.15.96:0/493682951 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v40) v1 ==== 107+0+4927 (secure 0 0 0) 0x7f3a84017090 con 0x7f3a901962f0 2024-04-22T16:00:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:47.068 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi096.drkkdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6829/1619485870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1619485870},{"type":"v1","addr":"172.21.15.96:6829","nonce":1619485870}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44225,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/2911111435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2911111435},{"type":"v1","addr":"172.21.15.196:6827","nonce":2911111435}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":44233,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4222304137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4222304137},{"type":"v1","addr":"172.21.15.196:6825","nonce":4222304137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:47.068 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 25 2024-04-22T16:00:47.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a7806ec30 msgr2=0x7f3a780710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a7806ec30 0x7f3a780710f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006010 tx=0x7f3a80000bc0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a901962f0 msgr2=0x7f3a9019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a901962f0 0x7f3a9019a750 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400d6f0 tx=0x7f3a8400dbb0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 shutdown_connections 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f3a7806ec30 0x7f3a780710f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a901962f0 0x7f3a9019a750 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 --2- 172.21.15.96:0/493682951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a900ff260 0x7f3a90195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 >> 172.21.15.96:0/493682951 conn(0x7f3a90074a50 msgr2=0x7f3a901033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 shutdown_connections 2024-04-22T16:00:47.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.067+0000 7f3a949ae640 1 -- 172.21.15.96:0/493682951 wait complete. 2024-04-22T16:00:47.082 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:47.082 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-04-22T16:00:47.083 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-22T16:00:47.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 -- 172.21.15.96:0/3896597607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2940ff1c0 msgr2=0x7ff2940ff5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 --2- 172.21.15.96:0/3896597607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2940ff1c0 0x7ff2940ff5a0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7ff284009880 tx=0x7ff28402bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 -- 172.21.15.96:0/3896597607 shutdown_connections 2024-04-22T16:00:47.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 --2- 172.21.15.96:0/3896597607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2940ffb70 0x7ff294103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 --2- 172.21.15.96:0/3896597607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2940ff1c0 0x7ff2940ff5a0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 -- 172.21.15.96:0/3896597607 >> 172.21.15.96:0/3896597607 conn(0x7ff2940faa20 msgr2=0x7ff2940fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:47.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 -- 172.21.15.96:0/3896597607 shutdown_connections 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 -- 172.21.15.96:0/3896597607 wait complete. 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 Processor -- start 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.179+0000 7ff299eaf640 1 -- start start 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff299eaf640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2940ff1c0 0x7ff29419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff299eaf640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29419afa0 0x7ff294194ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff299eaf640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29419b570 con 0x7ff29419afa0 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff299eaf640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff294195010 con 0x7ff2940ff1c0 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff28affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29419afa0 0x7ff294194ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff28affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29419afa0 0x7ff294194ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54800/0 (socket says 172.21.15.96:54800) 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff28affd640 1 -- 172.21.15.96:0/503298438 learned_addr learned my addr 172.21.15.96:0/503298438 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:47.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff2937fe640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2940ff1c0 0x7ff29419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:47.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff28affd640 1 -- 172.21.15.96:0/503298438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2940ff1c0 msgr2=0x7ff29419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff28affd640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2940ff1c0 0x7ff29419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff28affd640 1 -- 172.21.15.96:0/503298438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284009560 con 0x7ff29419afa0 2024-04-22T16:00:47.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff28affd640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29419afa0 0x7ff294194ad0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00d6c0 tx=0x7ff27c00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:47.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.180+0000 7ff2917fa640 1 -- 172.21.15.96:0/503298438 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27c0051f0 con 0x7ff29419afa0 2024-04-22T16:00:47.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.181+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2941952f0 con 0x7ff29419afa0 2024-04-22T16:00:47.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.181+0000 7ff2917fa640 1 -- 172.21.15.96:0/503298438 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff27c009ce0 con 0x7ff29419afa0 2024-04-22T16:00:47.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.181+0000 7ff2917fa640 1 -- 172.21.15.96:0/503298438 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27c010460 con 0x7ff29419afa0 2024-04-22T16:00:47.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.181+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff294195850 con 0x7ff29419afa0 2024-04-22T16:00:47.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.182+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29404e040 con 0x7ff29419afa0 2024-04-22T16:00:47.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.182+0000 7ff2917fa640 1 -- 172.21.15.96:0/503298438 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff27c005350 con 0x7ff29419afa0 2024-04-22T16:00:47.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.182+0000 7ff2917fa640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff28006ec30 0x7ff2800710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:47.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.183+0000 7ff2937fe640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff28006ec30 0x7ff2800710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:47.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.183+0000 7ff2917fa640 1 -- 172.21.15.96:0/503298438 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff27c08e880 con 0x7ff29419afa0 2024-04-22T16:00:47.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.183+0000 7ff2937fe640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff28006ec30 0x7ff2800710f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff284006010 tx=0x7ff284000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:47.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.187+0000 7ff2917fa640 1 -- 172.21.15.96:0/503298438 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff27c057660 con 0x7ff29419afa0 2024-04-22T16:00:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:46 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2859322483' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-22T16:00:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:46 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2592882336' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-22T16:00:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:46 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/213928637' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-22T16:00:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:46 smithi196 ceph-mon[81815]: pgmap v1171: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:47.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.401+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7ff2940ffb70 con 0x7ff29419afa0 2024-04-22T16:00:47.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.401+0000 7ff2917fa640 1 -- 172.21.15.96:0/503298438 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v40) v1 ==== 107+0+4148 (secure 0 0 0) 0x7ff27c08ce60 con 0x7ff29419afa0 2024-04-22T16:00:47.404 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:47.404 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44225,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/2911111435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2911111435},{"type":"v1","addr":"172.21.15.196:6827","nonce":2911111435}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":44233,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4222304137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4222304137},{"type":"v1","addr":"172.21.15.196:6825","nonce":4222304137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:47.405 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 26 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff28006ec30 msgr2=0x7ff2800710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff28006ec30 0x7ff2800710f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff284006010 tx=0x7ff284000bc0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29419afa0 msgr2=0x7ff294194ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29419afa0 0x7ff294194ad0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00d6c0 tx=0x7ff27c00db80 comp rx=0 tx=0).stop 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 shutdown_connections 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff28006ec30 0x7ff2800710f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29419afa0 0x7ff294194ad0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 --2- 172.21.15.96:0/503298438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2940ff1c0 0x7ff29419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 >> 172.21.15.96:0/503298438 conn(0x7ff2940faa20 msgr2=0x7ff2940fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 shutdown_connections 2024-04-22T16:00:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.403+0000 7ff299eaf640 1 -- 172.21.15.96:0/503298438 wait complete. 2024-04-22T16:00:47.417 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:47.417 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-04-22T16:00:47.418 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-22T16:00:47.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.513+0000 7f6c0e56e640 1 -- 172.21.15.96:0/889168790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 msgr2=0x7f6c080679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.513+0000 7f6c0e56e640 1 --2- 172.21.15.96:0/889168790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 0x7f6c080679a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8009880 tx=0x7f6bf802bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.513+0000 7f6c0e56e640 1 -- 172.21.15.96:0/889168790 shutdown_connections 2024-04-22T16:00:47.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.513+0000 7f6c0e56e640 1 --2- 172.21.15.96:0/889168790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c08067f70 0x7f6c08104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.513+0000 7f6c0e56e640 1 --2- 172.21.15.96:0/889168790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 0x7f6c080679a0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.513+0000 7f6c0e56e640 1 -- 172.21.15.96:0/889168790 >> 172.21.15.96:0/889168790 conn(0x7f6c08074a50 msgr2=0x7f6c08074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:47.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.514+0000 7f6c0e56e640 1 -- 172.21.15.96:0/889168790 shutdown_connections 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.514+0000 7f6c0e56e640 1 -- 172.21.15.96:0/889168790 wait complete. 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.514+0000 7f6c0e56e640 1 Processor -- start 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.514+0000 7f6c0e56e640 1 -- start start 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.514+0000 7f6c0e56e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 0x7f6c08195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.514+0000 7f6c0e56e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c081962f0 0x7f6c0819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.514+0000 7f6c0e56e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c081968f0 con 0x7f6c080675c0 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.514+0000 7f6c0e56e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c08196a60 con 0x7f6c081962f0 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.514+0000 7f6c07fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 0x7f6c08195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c07fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 0x7f6c08195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54822/0 (socket says 172.21.15.96:54822) 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c07fff640 1 -- 172.21.15.96:0/1593560206 learned_addr learned my addr 172.21.15.96:0/1593560206 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c077fe640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c081962f0 0x7f6c0819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:47.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c07fff640 1 -- 172.21.15.96:0/1593560206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c081962f0 msgr2=0x7f6c0819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c07fff640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c081962f0 0x7f6c0819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c07fff640 1 -- 172.21.15.96:0/1593560206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf8009560 con 0x7f6c080675c0 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c077fe640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c081962f0 0x7f6c0819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c07fff640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 0x7f6c08195db0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8036040 tx=0x7f6bf802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c057fa640 1 -- 172.21.15.96:0/1593560206 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bf8004520 con 0x7f6c080675c0 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c057fa640 1 -- 172.21.15.96:0/1593560206 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bf8003ab0 con 0x7f6c080675c0 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0819acf0 con 0x7f6c080675c0 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c057fa640 1 -- 172.21.15.96:0/1593560206 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bf8034440 con 0x7f6c080675c0 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.515+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0819b250 con 0x7f6c080675c0 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.516+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0804e040 con 0x7f6c080675c0 2024-04-22T16:00:47.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.517+0000 7f6c057fa640 1 -- 172.21.15.96:0/1593560206 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6bf8003c20 con 0x7f6c080675c0 2024-04-22T16:00:47.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.517+0000 7f6c057fa640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bf006ec30 0x7f6bf00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:47.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.517+0000 7f6c057fa640 1 -- 172.21.15.96:0/1593560206 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6bf80b1260 con 0x7f6c080675c0 2024-04-22T16:00:47.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.517+0000 7f6c077fe640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bf006ec30 0x7f6bf00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:47.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.517+0000 7f6c077fe640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bf006ec30 0x7f6bf00710f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6c08072e40 tx=0x7f6bfc005e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:47.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.522+0000 7f6c057fa640 1 -- 172.21.15.96:0/1593560206 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6bf8079fc0 con 0x7f6c080675c0 2024-04-22T16:00:47.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.736+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f6c0819b4c0 con 0x7f6c080675c0 2024-04-22T16:00:47.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.737+0000 7f6c057fa640 1 -- 172.21.15.96:0/1593560206 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v40) v1 ==== 107+0+4992 (secure 0 0 0) 0x7f6bf807d680 con 0x7f6c080675c0 2024-04-22T16:00:47.740 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:47.740 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44225,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/2911111435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2911111435},{"type":"v1","addr":"172.21.15.196:6827","nonce":2911111435}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":44233,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4222304137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4222304137},{"type":"v1","addr":"172.21.15.196:6825","nonce":4222304137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:47.740 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 27 2024-04-22T16:00:47.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bf006ec30 msgr2=0x7f6bf00710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bf006ec30 0x7f6bf00710f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6c08072e40 tx=0x7f6bfc005e50 comp rx=0 tx=0).stop 2024-04-22T16:00:47.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 msgr2=0x7f6c08195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 0x7f6c08195db0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8036040 tx=0x7f6bf802c850 comp rx=0 tx=0).stop 2024-04-22T16:00:47.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 shutdown_connections 2024-04-22T16:00:47.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6bf006ec30 0x7f6bf00710f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c081962f0 0x7f6c0819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 --2- 172.21.15.96:0/1593560206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c080675c0 0x7f6c08195db0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 >> 172.21.15.96:0/1593560206 conn(0x7f6c08074a50 msgr2=0x7f6c080fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:47.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 shutdown_connections 2024-04-22T16:00:47.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.739+0000 7f6c0e56e640 1 -- 172.21.15.96:0/1593560206 wait complete. 2024-04-22T16:00:47.753 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:47.753 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-04-22T16:00:47.753 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-22T16:00:47.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.850+0000 7f0f93bae640 1 -- 172.21.15.96:0/3776049418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c0ff260 msgr2=0x7f0f8c0ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.850+0000 7f0f93bae640 1 --2- 172.21.15.96:0/3776049418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c0ff260 0x7f0f8c0ff640 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009880 tx=0x7f0f8802bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.850+0000 7f0f93bae640 1 -- 172.21.15.96:0/3776049418 shutdown_connections 2024-04-22T16:00:47.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.850+0000 7f0f93bae640 1 --2- 172.21.15.96:0/3776049418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f8c0ffc10 0x7f0f8c103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.850+0000 7f0f93bae640 1 --2- 172.21.15.96:0/3776049418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c0ff260 0x7f0f8c0ff640 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.850+0000 7f0f93bae640 1 -- 172.21.15.96:0/3776049418 >> 172.21.15.96:0/3776049418 conn(0x7f0f8c074a50 msgr2=0x7f0f8c074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:47.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.850+0000 7f0f93bae640 1 -- 172.21.15.96:0/3776049418 shutdown_connections 2024-04-22T16:00:47.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.850+0000 7f0f93bae640 1 -- 172.21.15.96:0/3776049418 wait complete. 2024-04-22T16:00:47.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f93bae640 1 Processor -- start 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f93bae640 1 -- start start 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f93bae640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f8c0ff260 0x7f0f8c196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f93bae640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c197130 0x7f0f8c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f93bae640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c19fc60 con 0x7f0f8c197130 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f93bae640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c19fda0 con 0x7f0f8c0ff260 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f91149640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c197130 0x7f0f8c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f91149640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c197130 0x7f0f8c197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54848/0 (socket says 172.21.15.96:54848) 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f91149640 1 -- 172.21.15.96:0/3914906604 learned_addr learned my addr 172.21.15.96:0/3914906604 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f9194a640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f8c0ff260 0x7f0f8c196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f91149640 1 -- 172.21.15.96:0/3914906604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f8c0ff260 msgr2=0x7f0f8c196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f91149640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f8c0ff260 0x7f0f8c196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:47.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f91149640 1 -- 172.21.15.96:0/3914906604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f88009560 con 0x7f0f8c197130 2024-04-22T16:00:47.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.851+0000 7f0f9194a640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f8c0ff260 0x7f0f8c196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:00:47.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.852+0000 7f0f91149640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c197130 0x7f0f8c197590 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00d8d0 tx=0x7f0f7c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:47.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.852+0000 7f0f82ffd640 1 -- 172.21.15.96:0/3914906604 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f7c00bea0 con 0x7f0f8c197130 2024-04-22T16:00:47.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.852+0000 7f0f82ffd640 1 -- 172.21.15.96:0/3914906604 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f7c009c00 con 0x7f0f8c197130 2024-04-22T16:00:47.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.852+0000 7f0f82ffd640 1 -- 172.21.15.96:0/3914906604 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f7c010460 con 0x7f0f8c197130 2024-04-22T16:00:47.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.852+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c1a0000 con 0x7f0f8c197130 2024-04-22T16:00:47.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.852+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c1a04e0 con 0x7f0f8c197130 2024-04-22T16:00:47.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.853+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8c04e040 con 0x7f0f8c197130 2024-04-22T16:00:47.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.853+0000 7f0f82ffd640 1 -- 172.21.15.96:0/3914906604 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0f7c003f10 con 0x7f0f8c197130 2024-04-22T16:00:47.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.854+0000 7f0f82ffd640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0f7806ec30 0x7f0f780710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:47.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.854+0000 7f0f82ffd640 1 -- 172.21.15.96:0/3914906604 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0f7c08ec90 con 0x7f0f8c197130 2024-04-22T16:00:47.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.854+0000 7f0f9194a640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0f7806ec30 0x7f0f780710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:47.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.854+0000 7f0f9194a640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0f7806ec30 0x7f0f780710f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0f88000c00 tx=0x7f0f8802cb40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:47.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:47.858+0000 7f0f82ffd640 1 -- 172.21.15.96:0/3914906604 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0f7c0579f0 con 0x7f0f8c197130 2024-04-22T16:00:48.010 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:47 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/493682951' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-22T16:00:48.010 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:47 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/503298438' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-22T16:00:48.010 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:47 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1593560206' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-22T16:00:48.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.081+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f0f8c0ffcb0 con 0x7f0f8c197130 2024-04-22T16:00:48.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.082+0000 7f0f82ffd640 1 -- 172.21.15.96:0/3914906604 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v40) v1 ==== 107+0+4210 (secure 0 0 0) 0x7f0f7c05b0b0 con 0x7f0f8c197130 2024-04-22T16:00:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:48.085 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44225,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/2911111435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2911111435},{"type":"v1","addr":"172.21.15.196:6827","nonce":2911111435}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:48.086 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 28 2024-04-22T16:00:48.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0f7806ec30 msgr2=0x7f0f780710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0f7806ec30 0x7f0f780710f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0f88000c00 tx=0x7f0f8802cb40 comp rx=0 tx=0).stop 2024-04-22T16:00:48.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c197130 msgr2=0x7f0f8c197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c197130 0x7f0f8c197590 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00d8d0 tx=0x7f0f7c00dd90 comp rx=0 tx=0).stop 2024-04-22T16:00:48.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 shutdown_connections 2024-04-22T16:00:48.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0f7806ec30 0x7f0f780710f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f8c197130 0x7f0f8c197590 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 --2- 172.21.15.96:0/3914906604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f8c0ff260 0x7f0f8c196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 >> 172.21.15.96:0/3914906604 conn(0x7f0f8c074a50 msgr2=0x7f0f8c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:48.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 shutdown_connections 2024-04-22T16:00:48.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.084+0000 7f0f93bae640 1 -- 172.21.15.96:0/3914906604 wait complete. 2024-04-22T16:00:48.098 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:48.098 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-04-22T16:00:48.099 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-22T16:00:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.194+0000 7fd693677640 1 -- 172.21.15.96:0/811643730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0feff0 msgr2=0x7fd68c0ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.194+0000 7fd693677640 1 --2- 172.21.15.96:0/811643730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0feff0 0x7fd68c0ff3d0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fd684009c20 tx=0x7fd68402c1b0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.194+0000 7fd693677640 1 -- 172.21.15.96:0/811643730 shutdown_connections 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.194+0000 7fd693677640 1 --2- 172.21.15.96:0/811643730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd68c0ff910 0x7fd68c103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.194+0000 7fd693677640 1 --2- 172.21.15.96:0/811643730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0feff0 0x7fd68c0ff3d0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.194+0000 7fd693677640 1 -- 172.21.15.96:0/811643730 >> 172.21.15.96:0/811643730 conn(0x7fd68c0faa60 msgr2=0x7fd68c0fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.194+0000 7fd693677640 1 -- 172.21.15.96:0/811643730 shutdown_connections 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.194+0000 7fd693677640 1 -- 172.21.15.96:0/811643730 wait complete. 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd693677640 1 Processor -- start 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd693677640 1 -- start start 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd693677640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd68c0feff0 0x7fd68c19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:48.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd693677640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0ff910 0x7fd68c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:48.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd693677640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c19cf00 con 0x7fd68c0ff910 2024-04-22T16:00:48.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd693677640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c1a0c80 con 0x7fd68c0feff0 2024-04-22T16:00:48.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd690c12640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0ff910 0x7fd68c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:48.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd690c12640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0ff910 0x7fd68c19c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54874/0 (socket says 172.21.15.96:54874) 2024-04-22T16:00:48.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd690c12640 1 -- 172.21.15.96:0/1198672405 learned_addr learned my addr 172.21.15.96:0/1198672405 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:48.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd691413640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd68c0feff0 0x7fd68c19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:48.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd690c12640 1 -- 172.21.15.96:0/1198672405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd68c0feff0 msgr2=0x7fd68c19c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd690c12640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd68c0feff0 0x7fd68c19c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd690c12640 1 -- 172.21.15.96:0/1198672405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd684009900 con 0x7fd68c0ff910 2024-04-22T16:00:48.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.195+0000 7fd691413640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd68c0feff0 0x7fd68c19c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:00:48.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.196+0000 7fd690c12640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0ff910 0x7fd68c19c870 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00d6f0 tx=0x7fd67c00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:48.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.196+0000 7fd6827fc640 1 -- 172.21.15.96:0/1198672405 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd67c009a30 con 0x7fd68c0ff910 2024-04-22T16:00:48.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.196+0000 7fd6827fc640 1 -- 172.21.15.96:0/1198672405 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd67c0054e0 con 0x7fd68c0ff910 2024-04-22T16:00:48.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.196+0000 7fd6827fc640 1 -- 172.21.15.96:0/1198672405 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd67c010460 con 0x7fd68c0ff910 2024-04-22T16:00:48.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.196+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c1a0f60 con 0x7fd68c0ff910 2024-04-22T16:00:48.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.196+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c1a14c0 con 0x7fd68c0ff910 2024-04-22T16:00:48.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.197+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68c04e040 con 0x7fd68c0ff910 2024-04-22T16:00:48.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.197+0000 7fd6827fc640 1 -- 172.21.15.96:0/1198672405 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd67c009b90 con 0x7fd68c0ff910 2024-04-22T16:00:48.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.198+0000 7fd6827fc640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd67806ec30 0x7fd6780710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:48.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.198+0000 7fd6827fc640 1 -- 172.21.15.96:0/1198672405 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd67c08ea60 con 0x7fd68c0ff910 2024-04-22T16:00:48.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.198+0000 7fd691413640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd67806ec30 0x7fd6780710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:48.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.198+0000 7fd691413640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd67806ec30 0x7fd6780710f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd684005e30 tx=0x7fd684005da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:48.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.203+0000 7fd6827fc640 1 -- 172.21.15.96:0/1198672405 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd67c0577c0 con 0x7fd68c0ff910 2024-04-22T16:00:48.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:47 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/493682951' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-22T16:00:48.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:47 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/503298438' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-22T16:00:48.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:47 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1593560206' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-22T16:00:48.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.417+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7fd68c1014a0 con 0x7fd68c0ff910 2024-04-22T16:00:48.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.418+0000 7fd6827fc640 1 -- 172.21.15.96:0/1198672405 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v40) v1 ==== 107+0+5060 (secure 0 0 0) 0x7fd67c017090 con 0x7fd68c0ff910 2024-04-22T16:00:48.421 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:48.421 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44225,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/2911111435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2911111435},{"type":"v1","addr":"172.21.15.196:6827","nonce":2911111435}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:48.421 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 29 2024-04-22T16:00:48.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.419+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd67806ec30 msgr2=0x7fd6780710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.419+0000 7fd693677640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd67806ec30 0x7fd6780710f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd684005e30 tx=0x7fd684005da0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.419+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0ff910 msgr2=0x7fd68c19c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.419+0000 7fd693677640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0ff910 0x7fd68c19c870 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00d6f0 tx=0x7fd67c00dbb0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.419+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 shutdown_connections 2024-04-22T16:00:48.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.419+0000 7fd693677640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fd67806ec30 0x7fd6780710f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.419+0000 7fd693677640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd68c0ff910 0x7fd68c19c870 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.419+0000 7fd693677640 1 --2- 172.21.15.96:0/1198672405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd68c0feff0 0x7fd68c19c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.420+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 >> 172.21.15.96:0/1198672405 conn(0x7fd68c0faa60 msgr2=0x7fd68c110450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:48.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.420+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 shutdown_connections 2024-04-22T16:00:48.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.420+0000 7fd693677640 1 -- 172.21.15.96:0/1198672405 wait complete. 2024-04-22T16:00:48.434 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:48.434 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-04-22T16:00:48.434 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-22T16:00:48.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.530+0000 7ff3661d7640 1 -- 172.21.15.96:0/3577390517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 msgr2=0x7ff360100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.530+0000 7ff3661d7640 1 --2- 172.21.15.96:0/3577390517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 0x7ff360100680 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7ff350009880 tx=0x7ff35002bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 -- 172.21.15.96:0/3577390517 shutdown_connections 2024-04-22T16:00:48.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 --2- 172.21.15.96:0/3577390517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff360100c50 0x7ff360104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 --2- 172.21.15.96:0/3577390517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 0x7ff360100680 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 -- 172.21.15.96:0/3577390517 >> 172.21.15.96:0/3577390517 conn(0x7ff3600fbb20 msgr2=0x7ff3600fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:48.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 -- 172.21.15.96:0/3577390517 shutdown_connections 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 -- 172.21.15.96:0/3577390517 wait complete. 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 Processor -- start 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 -- start start 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 0x7ff360195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff360196340 0x7ff36019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff360196940 con 0x7ff3601002a0 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff3661d7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff360196ab0 con 0x7ff360196340 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff35f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 0x7ff360195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff35f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 0x7ff360195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54898/0 (socket says 172.21.15.96:54898) 2024-04-22T16:00:48.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff35f7fe640 1 -- 172.21.15.96:0/4087913261 learned_addr learned my addr 172.21.15.96:0/4087913261 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.531+0000 7ff357fff640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff360196340 0x7ff36019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.532+0000 7ff35f7fe640 1 -- 172.21.15.96:0/4087913261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff360196340 msgr2=0x7ff36019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.532+0000 7ff35f7fe640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff360196340 0x7ff36019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.532+0000 7ff35f7fe640 1 -- 172.21.15.96:0/4087913261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff350009560 con 0x7ff3601002a0 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.532+0000 7ff35f7fe640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 0x7ff360195e00 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7ff350009430 tx=0x7ff350005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.532+0000 7ff35d7fa640 1 -- 172.21.15.96:0/4087913261 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3500044a0 con 0x7ff3601002a0 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.532+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36019ad40 con 0x7ff3601002a0 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.532+0000 7ff35d7fa640 1 -- 172.21.15.96:0/4087913261 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff350003ab0 con 0x7ff3601002a0 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.532+0000 7ff35d7fa640 1 -- 172.21.15.96:0/4087913261 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3500055f0 con 0x7ff3601002a0 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.532+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36019b2a0 con 0x7ff3601002a0 2024-04-22T16:00:48.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.533+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36004e040 con 0x7ff3601002a0 2024-04-22T16:00:48.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.533+0000 7ff35d7fa640 1 -- 172.21.15.96:0/4087913261 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff350004e60 con 0x7ff3601002a0 2024-04-22T16:00:48.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.534+0000 7ff35d7fa640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff34c06ec30 0x7ff34c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:48.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.534+0000 7ff35d7fa640 1 -- 172.21.15.96:0/4087913261 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff3500b1230 con 0x7ff3601002a0 2024-04-22T16:00:48.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.534+0000 7ff357fff640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff34c06ec30 0x7ff34c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:48.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.534+0000 7ff357fff640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff34c06ec30 0x7ff34c0710f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff3600696e0 tx=0x7ff348005f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:48.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.539+0000 7ff35d7fa640 1 -- 172.21.15.96:0/4087913261 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff35007a040 con 0x7ff3601002a0 2024-04-22T16:00:48.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.758+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7ff36019b5b0 con 0x7ff3601002a0 2024-04-22T16:00:48.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.759+0000 7ff35d7fa640 1 -- 172.21.15.96:0/4087913261 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v40) v1 ==== 107+0+4278 (secure 0 0 0) 0x7ff35007d700 con 0x7ff3601002a0 2024-04-22T16:00:48.762 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:48.762 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:48.763 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 30 2024-04-22T16:00:48.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff34c06ec30 msgr2=0x7ff34c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff34c06ec30 0x7ff34c0710f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff3600696e0 tx=0x7ff348005f90 comp rx=0 tx=0).stop 2024-04-22T16:00:48.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 msgr2=0x7ff360195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 0x7ff360195e00 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7ff350009430 tx=0x7ff350005e50 comp rx=0 tx=0).stop 2024-04-22T16:00:48.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 shutdown_connections 2024-04-22T16:00:48.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7ff34c06ec30 0x7ff34c0710f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff360196340 0x7ff36019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 --2- 172.21.15.96:0/4087913261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3601002a0 0x7ff360195e00 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 >> 172.21.15.96:0/4087913261 conn(0x7ff3600fbb20 msgr2=0x7ff3600fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:48.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 shutdown_connections 2024-04-22T16:00:48.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.761+0000 7ff3661d7640 1 -- 172.21.15.96:0/4087913261 wait complete. 2024-04-22T16:00:48.775 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:48.775 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-04-22T16:00:48.776 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-22T16:00:48.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.875+0000 7fbb01dd7640 1 -- 172.21.15.96:0/3347542687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc0ff1a0 msgr2=0x7fbafc0ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.875+0000 7fbb01dd7640 1 --2- 172.21.15.96:0/3347542687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc0ff1a0 0x7fbafc0ff580 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009880 tx=0x7fbaec02bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.875+0000 7fbb01dd7640 1 -- 172.21.15.96:0/3347542687 shutdown_connections 2024-04-22T16:00:48.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.875+0000 7fbb01dd7640 1 --2- 172.21.15.96:0/3347542687 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbafc0ffb50 0x7fbafc103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.875+0000 7fbb01dd7640 1 --2- 172.21.15.96:0/3347542687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc0ff1a0 0x7fbafc0ff580 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.875+0000 7fbb01dd7640 1 -- 172.21.15.96:0/3347542687 >> 172.21.15.96:0/3347542687 conn(0x7fbafc0faa20 msgr2=0x7fbafc0fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:48.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.875+0000 7fbb01dd7640 1 -- 172.21.15.96:0/3347542687 shutdown_connections 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.875+0000 7fbb01dd7640 1 -- 172.21.15.96:0/3347542687 wait complete. 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbb01dd7640 1 Processor -- start 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbb01dd7640 1 -- start start 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbb01dd7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbafc0ff1a0 0x7fbafc196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbb01dd7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc197150 0x7fbafc1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbb01dd7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc19fc80 con 0x7fbafc197150 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbb01dd7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc19fdc0 con 0x7fbafc0ff1a0 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbafaffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc197150 0x7fbafc1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbafaffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc197150 0x7fbafc1975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54914/0 (socket says 172.21.15.96:54914) 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbafaffd640 1 -- 172.21.15.96:0/2035866216 learned_addr learned my addr 172.21.15.96:0/2035866216 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:48.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbafb7fe640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbafc0ff1a0 0x7fbafc196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbafaffd640 1 -- 172.21.15.96:0/2035866216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbafc0ff1a0 msgr2=0x7fbafc196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbafaffd640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbafc0ff1a0 0x7fbafc196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.876+0000 7fbafaffd640 1 -- 172.21.15.96:0/2035866216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec009560 con 0x7fbafc197150 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.877+0000 7fbafaffd640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc197150 0x7fbafc1975b0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000c920 tx=0x7fbaf000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.877+0000 7fbaf8ff9640 1 -- 172.21.15.96:0/2035866216 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbaf0007b80 con 0x7fbafc197150 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.877+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc1a0020 con 0x7fbafc197150 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.877+0000 7fbaf8ff9640 1 -- 172.21.15.96:0/2035866216 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbaf0007ce0 con 0x7fbafc197150 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.877+0000 7fbaf8ff9640 1 -- 172.21.15.96:0/2035866216 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbaf00187d0 con 0x7fbafc197150 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.877+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc1a0500 con 0x7fbafc197150 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.878+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbafc04e040 con 0x7fbafc197150 2024-04-22T16:00:48.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.878+0000 7fbaf8ff9640 1 -- 172.21.15.96:0/2035866216 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbaf000f450 con 0x7fbafc197150 2024-04-22T16:00:48.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.878+0000 7fbaf8ff9640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbae406ec30 0x7fbae40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:48.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.879+0000 7fbaf8ff9640 1 -- 172.21.15.96:0/2035866216 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fbaf008f020 con 0x7fbafc197150 2024-04-22T16:00:48.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.879+0000 7fbafb7fe640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbae406ec30 0x7fbae40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:48.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.879+0000 7fbafb7fe640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbae406ec30 0x7fbae40710f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbaec006010 tx=0x7fbaec000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:48.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:48.883+0000 7fbaf8ff9640 1 -- 172.21.15.96:0/2035866216 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbaf0057d80 con 0x7fbafc197150 2024-04-22T16:00:49.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.100+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7fbafc0ffbf0 con 0x7fbafc197150 2024-04-22T16:00:49.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.100+0000 7fbaf8ff9640 1 -- 172.21.15.96:0/2035866216 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v40) v1 ==== 107+0+5128 (secure 0 0 0) 0x7fbaf005b440 con 0x7fbafc197150 2024-04-22T16:00:49.103 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:49.104 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:27:39.581378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:49.104 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 31 2024-04-22T16:00:49.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbae406ec30 msgr2=0x7fbae40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbae406ec30 0x7fbae40710f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbaec006010 tx=0x7fbaec000bc0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc197150 msgr2=0x7fbafc1975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc197150 0x7fbafc1975b0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000c920 tx=0x7fbaf000cde0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 shutdown_connections 2024-04-22T16:00:49.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fbae406ec30 0x7fbae40710f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbafc197150 0x7fbafc1975b0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 --2- 172.21.15.96:0/2035866216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbafc0ff1a0 0x7fbafc196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 >> 172.21.15.96:0/2035866216 conn(0x7fbafc0faa20 msgr2=0x7fbafc0fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:49.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 shutdown_connections 2024-04-22T16:00:49.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.102+0000 7fbb01dd7640 1 -- 172.21.15.96:0/2035866216 wait complete. 2024-04-22T16:00:49.115 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:48 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3914906604' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-22T16:00:49.115 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:48 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1198672405' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-22T16:00:49.116 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:48 smithi096 ceph-mon[112443]: pgmap v1172: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:49.116 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:48 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/4087913261' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-22T16:00:49.117 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-22T16:00:49.117 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-04-22T16:00:49.117 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-22T16:00:49.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.213+0000 7f17ea03f640 1 -- 172.21.15.96:0/1906180836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e40ffae0 msgr2=0x7f17e4103b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.213+0000 7f17ea03f640 1 --2- 172.21.15.96:0/1906180836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e40ffae0 0x7f17e4103b20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009880 tx=0x7f17cc02bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.213+0000 7f17ea03f640 1 -- 172.21.15.96:0/1906180836 shutdown_connections 2024-04-22T16:00:49.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.213+0000 7f17ea03f640 1 --2- 172.21.15.96:0/1906180836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e40ffae0 0x7f17e4103b20 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.213+0000 7f17ea03f640 1 --2- 172.21.15.96:0/1906180836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17e40ff130 0x7f17e40ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.213+0000 7f17ea03f640 1 -- 172.21.15.96:0/1906180836 >> 172.21.15.96:0/1906180836 conn(0x7f17e40fa9b0 msgr2=0x7f17e40fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.213+0000 7f17ea03f640 1 -- 172.21.15.96:0/1906180836 shutdown_connections 2024-04-22T16:00:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.213+0000 7f17ea03f640 1 -- 172.21.15.96:0/1906180836 wait complete. 2024-04-22T16:00:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.213+0000 7f17ea03f640 1 Processor -- start 2024-04-22T16:00:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17ea03f640 1 -- start start 2024-04-22T16:00:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17ea03f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17e40ff130 0x7f17e4194920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17ea03f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e4194e60 0x7f17e41952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17ea03f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40692d0 con 0x7f17e4194e60 2024-04-22T16:00:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17ea03f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e41a1f90 con 0x7f17e40ff130 2024-04-22T16:00:49.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e2ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e4194e60 0x7f17e41952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:49.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e2ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e4194e60 0x7f17e41952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54942/0 (socket says 172.21.15.96:54942) 2024-04-22T16:00:49.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e2ffd640 1 -- 172.21.15.96:0/3035893261 learned_addr learned my addr 172.21.15.96:0/3035893261 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:49.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e37fe640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17e40ff130 0x7f17e4194920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:49.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e2ffd640 1 -- 172.21.15.96:0/3035893261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17e40ff130 msgr2=0x7f17e4194920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e2ffd640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17e40ff130 0x7f17e4194920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e2ffd640 1 -- 172.21.15.96:0/3035893261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc009560 con 0x7f17e4194e60 2024-04-22T16:00:49.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e37fe640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17e40ff130 0x7f17e4194920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:00:49.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e2ffd640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e4194e60 0x7f17e41952e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009230 tx=0x7f17cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:49.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e0ff9640 1 -- 172.21.15.96:0/3035893261 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17cc0037c0 con 0x7f17e4194e60 2024-04-22T16:00:49.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e41a20d0 con 0x7f17e4194e60 2024-04-22T16:00:49.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e0ff9640 1 -- 172.21.15.96:0/3035893261 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17cc003de0 con 0x7f17e4194e60 2024-04-22T16:00:49.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.214+0000 7f17e0ff9640 1 -- 172.21.15.96:0/3035893261 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17cc034440 con 0x7f17e4194e60 2024-04-22T16:00:49.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.215+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e41a25d0 con 0x7f17e4194e60 2024-04-22T16:00:49.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.216+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e404e040 con 0x7f17e4194e60 2024-04-22T16:00:49.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.216+0000 7f17e0ff9640 1 -- 172.21.15.96:0/3035893261 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f17cc004fb0 con 0x7f17e4194e60 2024-04-22T16:00:49.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.216+0000 7f17e0ff9640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17d006ec30 0x7f17d00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:49.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.216+0000 7f17e0ff9640 1 -- 172.21.15.96:0/3035893261 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f17cc0b13a0 con 0x7f17e4194e60 2024-04-22T16:00:49.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.216+0000 7f17e37fe640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17d006ec30 0x7f17d00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:49.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.217+0000 7f17e37fe640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17d006ec30 0x7f17d00710f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f17d400abe0 tx=0x7f17d400a5c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:49.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.221+0000 7f17e0ff9640 1 -- 172.21.15.96:0/3035893261 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f17cc07a100 con 0x7f17e4194e60 2024-04-22T16:00:49.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:48 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3914906604' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-22T16:00:49.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:48 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1198672405' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-22T16:00:49.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:48 smithi196 ceph-mon[81815]: pgmap v1172: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:49.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:48 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/4087913261' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-22T16:00:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.441+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f17e4107430 con 0x7f17e4194e60 2024-04-22T16:00:49.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.441+0000 7f17e0ff9640 1 -- 172.21.15.96:0/3035893261 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v40) v1 ==== 107+0+5127 (secure 0 0 0) 0x7f17cc07d7c0 con 0x7f17e4194e60 2024-04-22T16:00:49.445 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:49.445 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:58:38.704435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:49.445 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 32 2024-04-22T16:00:49.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17d006ec30 msgr2=0x7f17d00710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17d006ec30 0x7f17d00710f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f17d400abe0 tx=0x7f17d400a5c0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e4194e60 msgr2=0x7f17e41952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e4194e60 0x7f17e41952e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009230 tx=0x7f17cc004270 comp rx=0 tx=0).stop 2024-04-22T16:00:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 shutdown_connections 2024-04-22T16:00:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f17d006ec30 0x7f17d00710f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17e4194e60 0x7f17e41952e0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 --2- 172.21.15.96:0/3035893261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17e40ff130 0x7f17e4194920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 >> 172.21.15.96:0/3035893261 conn(0x7f17e40fa9b0 msgr2=0x7f17e40fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 shutdown_connections 2024-04-22T16:00:49.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.443+0000 7f17ea03f640 1 -- 172.21.15.96:0/3035893261 wait complete. 2024-04-22T16:00:49.457 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-04-22T16:00:49.458 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-22T16:00:49.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.554+0000 7f919d039640 1 -- 172.21.15.96:0/560878921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91980ffb30 msgr2=0x7f9198103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.554+0000 7f919d039640 1 --2- 172.21.15.96:0/560878921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91980ffb30 0x7f9198103b70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f918c009c20 tx=0x7f918c02c1b0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.554+0000 7f919d039640 1 -- 172.21.15.96:0/560878921 shutdown_connections 2024-04-22T16:00:49.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.554+0000 7f919d039640 1 --2- 172.21.15.96:0/560878921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91980ffb30 0x7f9198103b70 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.554+0000 7f919d039640 1 --2- 172.21.15.96:0/560878921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91980ff180 0x7f91980ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.554+0000 7f919d039640 1 -- 172.21.15.96:0/560878921 >> 172.21.15.96:0/560878921 conn(0x7f91980faa40 msgr2=0x7f91980fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:49.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.554+0000 7f919d039640 1 -- 172.21.15.96:0/560878921 shutdown_connections 2024-04-22T16:00:49.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.554+0000 7f919d039640 1 -- 172.21.15.96:0/560878921 wait complete. 2024-04-22T16:00:49.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.554+0000 7f919d039640 1 Processor -- start 2024-04-22T16:00:49.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919d039640 1 -- start start 2024-04-22T16:00:49.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919d039640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91980ff180 0x7f919819aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:49.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919d039640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f919819af70 0x7f9198194aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:49.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919d039640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919819b540 con 0x7f919819af70 2024-04-22T16:00:49.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919d039640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9198194fe0 con 0x7f91980ff180 2024-04-22T16:00:49.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f919819af70 0x7f9198194aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:49.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f919819af70 0x7f9198194aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54966/0 (socket says 172.21.15.96:54966) 2024-04-22T16:00:49.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919659c640 1 -- 172.21.15.96:0/555241279 learned_addr learned my addr 172.21.15.96:0/555241279 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:49.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f9196d9d640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91980ff180 0x7f919819aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:49.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919659c640 1 -- 172.21.15.96:0/555241279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91980ff180 msgr2=0x7f919819aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919659c640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91980ff180 0x7f919819aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919659c640 1 -- 172.21.15.96:0/555241279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c009900 con 0x7f919819af70 2024-04-22T16:00:49.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f9196d9d640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91980ff180 0x7f919819aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:49.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.555+0000 7f919659c640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f919819af70 0x7f9198194aa0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f918c009430 tx=0x7f918c032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:49.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.556+0000 7f917ffff640 1 -- 172.21.15.96:0/555241279 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918c032ed0 con 0x7f919819af70 2024-04-22T16:00:49.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.556+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9198195260 con 0x7f919819af70 2024-04-22T16:00:49.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.556+0000 7f917ffff640 1 -- 172.21.15.96:0/555241279 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f918c003e30 con 0x7f919819af70 2024-04-22T16:00:49.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.556+0000 7f917ffff640 1 -- 172.21.15.96:0/555241279 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918c03f440 con 0x7f919819af70 2024-04-22T16:00:49.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.556+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9198195760 con 0x7f919819af70 2024-04-22T16:00:49.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.557+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919804e040 con 0x7f919819af70 2024-04-22T16:00:49.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.557+0000 7f917ffff640 1 -- 172.21.15.96:0/555241279 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f918c03b040 con 0x7f919819af70 2024-04-22T16:00:49.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.558+0000 7f917ffff640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f918006ec30 0x7f91800710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:49.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.558+0000 7f917ffff640 1 -- 172.21.15.96:0/555241279 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f918c0b1b90 con 0x7f919819af70 2024-04-22T16:00:49.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.558+0000 7f9196d9d640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f918006ec30 0x7f91800710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:49.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.558+0000 7f9196d9d640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f918006ec30 0x7f91800710f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9190009b40 tx=0x7f9190005e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:49.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.562+0000 7f917ffff640 1 -- 172.21.15.96:0/555241279 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f918c07a9b0 con 0x7f919819af70 2024-04-22T16:00:49.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.781+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f9198107480 con 0x7f919819af70 2024-04-22T16:00:49.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.782+0000 7f917ffff640 1 -- 172.21.15.96:0/555241279 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v40) v1 ==== 107+0+5218 (secure 0 0 0) 0x7f918c07e070 con 0x7f919819af70 2024-04-22T16:00:49.785 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:49.785 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:58:39.705176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":0,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":1,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:49.786 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 33 2024-04-22T16:00:49.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f918006ec30 msgr2=0x7f91800710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f918006ec30 0x7f91800710f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9190009b40 tx=0x7f9190005e50 comp rx=0 tx=0).stop 2024-04-22T16:00:49.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f919819af70 msgr2=0x7f9198194aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f919819af70 0x7f9198194aa0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f918c009430 tx=0x7f918c032b60 comp rx=0 tx=0).stop 2024-04-22T16:00:49.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 shutdown_connections 2024-04-22T16:00:49.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f918006ec30 0x7f91800710f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f919819af70 0x7f9198194aa0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 --2- 172.21.15.96:0/555241279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91980ff180 0x7f919819aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 >> 172.21.15.96:0/555241279 conn(0x7f91980faa40 msgr2=0x7f91980fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:49.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 shutdown_connections 2024-04-22T16:00:49.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.784+0000 7f919d039640 1 -- 172.21.15.96:0/555241279 wait complete. 2024-04-22T16:00:49.798 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-04-22T16:00:49.798 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-22T16:00:49.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.895+0000 7f25ec13c640 1 -- 172.21.15.96:0/2786053085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 msgr2=0x7f25e4103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.895+0000 7f25ec13c640 1 --2- 172.21.15.96:0/2786053085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 0x7f25e4103db0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009880 tx=0x7f25d402bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.895+0000 7f25ec13c640 1 -- 172.21.15.96:0/2786053085 shutdown_connections 2024-04-22T16:00:49.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.895+0000 7f25ec13c640 1 --2- 172.21.15.96:0/2786053085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 0x7f25e4103db0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.895+0000 7f25ec13c640 1 --2- 172.21.15.96:0/2786053085 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25e40fefd0 0x7f25e40ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.895+0000 7f25ec13c640 1 -- 172.21.15.96:0/2786053085 >> 172.21.15.96:0/2786053085 conn(0x7f25e40faa00 msgr2=0x7f25e40fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:49.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.895+0000 7f25ec13c640 1 -- 172.21.15.96:0/2786053085 shutdown_connections 2024-04-22T16:00:49.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.895+0000 7f25ec13c640 1 -- 172.21.15.96:0/2786053085 wait complete. 2024-04-22T16:00:49.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25ec13c640 1 Processor -- start 2024-04-22T16:00:49.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25ec13c640 1 -- start start 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25ec13c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25e40fefd0 0x7f25e419c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25ec13c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 0x7f25e419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25ec13c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e419cf60 con 0x7f25e40ff8f0 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25ec13c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e41a0ce0 con 0x7f25e40fefd0 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25e96d7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 0x7f25e419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25e96d7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 0x7f25e419c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54982/0 (socket says 172.21.15.96:54982) 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25e96d7640 1 -- 172.21.15.96:0/1405858853 learned_addr learned my addr 172.21.15.96:0/1405858853 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25e9ed8640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25e40fefd0 0x7f25e419c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25e96d7640 1 -- 172.21.15.96:0/1405858853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25e40fefd0 msgr2=0x7f25e419c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:49.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25e96d7640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25e40fefd0 0x7f25e419c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25e96d7640 1 -- 172.21.15.96:0/1405858853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d4009560 con 0x7f25e40ff8f0 2024-04-22T16:00:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.896+0000 7f25e96d7640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 0x7f25e419c8d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009230 tx=0x7f25d402c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.897+0000 7f25daffd640 1 -- 172.21.15.96:0/1405858853 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25d40035f0 con 0x7f25e40ff8f0 2024-04-22T16:00:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.897+0000 7f25daffd640 1 -- 172.21.15.96:0/1405858853 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25d4003c10 con 0x7f25e40ff8f0 2024-04-22T16:00:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.897+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e41a0f60 con 0x7f25e40ff8f0 2024-04-22T16:00:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.897+0000 7f25daffd640 1 -- 172.21.15.96:0/1405858853 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25d4005690 con 0x7f25e40ff8f0 2024-04-22T16:00:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.897+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e41a1460 con 0x7f25e40ff8f0 2024-04-22T16:00:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.898+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e404e040 con 0x7f25e40ff8f0 2024-04-22T16:00:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.898+0000 7f25daffd640 1 -- 172.21.15.96:0/1405858853 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f25d4004ed0 con 0x7f25e40ff8f0 2024-04-22T16:00:49.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.898+0000 7f25daffd640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25d006ec30 0x7f25d00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:49.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.899+0000 7f25daffd640 1 -- 172.21.15.96:0/1405858853 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f25d40b1260 con 0x7f25e40ff8f0 2024-04-22T16:00:49.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.899+0000 7f25e9ed8640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25d006ec30 0x7f25d00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:49.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.899+0000 7f25e9ed8640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25d006ec30 0x7f25d00710f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f25e0005fd0 tx=0x7f25e000b590 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:49.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:49.903+0000 7f25daffd640 1 -- 172.21.15.96:0/1405858853 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f25d407a040 con 0x7f25e40ff8f0 2024-04-22T16:00:50.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.122+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f25e41138b0 con 0x7f25e40ff8f0 2024-04-22T16:00:50.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.123+0000 7f25daffd640 1 -- 172.21.15.96:0/1405858853 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v40) v1 ==== 107+0+5220 (secure 0 0 0) 0x7f25d407d700 con 0x7f25e40ff8f0 2024-04-22T16:00:50.126 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:50.126 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:58:39.714905+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":0,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":1,"incarnation":33,"state":"up:resolve","state_seq":459,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 34 2024-04-22T16:00:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25d006ec30 msgr2=0x7f25d00710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25d006ec30 0x7f25d00710f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f25e0005fd0 tx=0x7f25e000b590 comp rx=0 tx=0).stop 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 msgr2=0x7f25e419c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 0x7f25e419c8d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009230 tx=0x7f25d402c850 comp rx=0 tx=0).stop 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 shutdown_connections 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f25d006ec30 0x7f25d00710f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25e40ff8f0 0x7f25e419c8d0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 --2- 172.21.15.96:0/1405858853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25e40fefd0 0x7f25e419c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 >> 172.21.15.96:0/1405858853 conn(0x7f25e40faa00 msgr2=0x7f25e4110430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 shutdown_connections 2024-04-22T16:00:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.125+0000 7f25ec13c640 1 -- 172.21.15.96:0/1405858853 wait complete. 2024-04-22T16:00:50.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:49 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2035866216' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-22T16:00:50.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:49 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3035893261' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-22T16:00:50.138 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:49 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/555241279' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-22T16:00:50.140 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-22T16:00:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.236+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/860391535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 msgr2=0x7f6e4410a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.236+0000 7f6e4ba1c640 1 --2- 172.21.15.96:0/860391535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 0x7f6e4410a010 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009880 tx=0x7f6e3402bd70 comp rx=0 tx=0).stop 2024-04-22T16:00:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.236+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/860391535 shutdown_connections 2024-04-22T16:00:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.236+0000 7f6e4ba1c640 1 --2- 172.21.15.96:0/860391535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 0x7f6e4410a010 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.236+0000 7f6e4ba1c640 1 --2- 172.21.15.96:0/860391535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e44101210 0x7f6e441015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.236+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/860391535 >> 172.21.15.96:0/860391535 conn(0x7f6e440faa30 msgr2=0x7f6e440fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/860391535 shutdown_connections 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/860391535 wait complete. 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e4ba1c640 1 Processor -- start 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e4ba1c640 1 -- start start 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e4ba1c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e44101210 0x7f6e44195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e4ba1c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 0x7f6e441962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e4ba1c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e44196940 con 0x7f6e44101b30 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e4ba1c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4419a710 con 0x7f6e44101210 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e48fb7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 0x7f6e441962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e48fb7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 0x7f6e441962b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54996/0 (socket says 172.21.15.96:54996) 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e48fb7640 1 -- 172.21.15.96:0/3138549470 learned_addr learned my addr 172.21.15.96:0/3138549470 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.237+0000 7f6e497b8640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e44101210 0x7f6e44195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.238+0000 7f6e48fb7640 1 -- 172.21.15.96:0/3138549470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e44101210 msgr2=0x7f6e44195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.238+0000 7f6e48fb7640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e44101210 0x7f6e44195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.238+0000 7f6e48fb7640 1 -- 172.21.15.96:0/3138549470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e34009560 con 0x7f6e44101b30 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.238+0000 7f6e48fb7640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 0x7f6e441962b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009230 tx=0x7f6e3402c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.238+0000 7f6e3a7fc640 1 -- 172.21.15.96:0/3138549470 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e340044a0 con 0x7f6e44101b30 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.238+0000 7f6e3a7fc640 1 -- 172.21.15.96:0/3138549470 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e34003ab0 con 0x7f6e44101b30 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.238+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4419a990 con 0x7f6e44101b30 2024-04-22T16:00:50.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.238+0000 7f6e3a7fc640 1 -- 172.21.15.96:0/3138549470 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e34005600 con 0x7f6e44101b30 2024-04-22T16:00:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.238+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4419aef0 con 0x7f6e44101b30 2024-04-22T16:00:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.239+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4404e040 con 0x7f6e44101b30 2024-04-22T16:00:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.239+0000 7f6e3a7fc640 1 -- 172.21.15.96:0/3138549470 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6e34004dd0 con 0x7f6e44101b30 2024-04-22T16:00:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.240+0000 7f6e3a7fc640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3006ec30 0x7f6e300710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.240+0000 7f6e497b8640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3006ec30 0x7f6e300710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.240+0000 7f6e3a7fc640 1 -- 172.21.15.96:0/3138549470 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6e340b18f0 con 0x7f6e44101b30 2024-04-22T16:00:50.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.240+0000 7f6e497b8640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3006ec30 0x7f6e300710f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6e40005fd0 tx=0x7f6e4000b390 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:50.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.245+0000 7f6e3a7fc640 1 -- 172.21.15.96:0/3138549470 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6e3407a650 con 0x7f6e44101b30 2024-04-22T16:00:50.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:49 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2035866216' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-22T16:00:50.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:49 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3035893261' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-22T16:00:50.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:49 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/555241279' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-22T16:00:50.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.458+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f6e44107680 con 0x7f6e44101b30 2024-04-22T16:00:50.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.459+0000 7f6e3a7fc640 1 -- 172.21.15.96:0/3138549470 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v40) v1 ==== 107+0+5223 (secure 0 0 0) 0x7f6e3407dd10 con 0x7f6e44101b30 2024-04-22T16:00:50.462 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:50.462 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:58:49.182242+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":0,"incarnation":33,"state":"up:resolve","state_seq":467,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":1,"incarnation":33,"state":"up:resolve","state_seq":459,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:50.463 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 35 2024-04-22T16:00:50.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3006ec30 msgr2=0x7f6e300710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3006ec30 0x7f6e300710f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6e40005fd0 tx=0x7f6e4000b390 comp rx=0 tx=0).stop 2024-04-22T16:00:50.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 msgr2=0x7f6e441962b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 0x7f6e441962b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009230 tx=0x7f6e3402c7a0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 shutdown_connections 2024-04-22T16:00:50.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f6e3006ec30 0x7f6e300710f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e44101b30 0x7f6e441962b0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 --2- 172.21.15.96:0/3138549470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e44101210 0x7f6e44195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 >> 172.21.15.96:0/3138549470 conn(0x7f6e440faa30 msgr2=0x7f6e44104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:50.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 shutdown_connections 2024-04-22T16:00:50.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.461+0000 7f6e4ba1c640 1 -- 172.21.15.96:0/3138549470 wait complete. 2024-04-22T16:00:50.476 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-04-22T16:00:50.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 -- 172.21.15.96:0/217710869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 msgr2=0x7f7e200ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 --2- 172.21.15.96:0/217710869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 0x7f7e200ff640 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009880 tx=0x7f7e1002bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 -- 172.21.15.96:0/217710869 shutdown_connections 2024-04-22T16:00:50.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 --2- 172.21.15.96:0/217710869 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e200ffc10 0x7f7e20103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 --2- 172.21.15.96:0/217710869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 0x7f7e200ff640 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 -- 172.21.15.96:0/217710869 >> 172.21.15.96:0/217710869 conn(0x7f7e20074a50 msgr2=0x7f7e20074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 -- 172.21.15.96:0/217710869 shutdown_connections 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 -- 172.21.15.96:0/217710869 wait complete. 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 Processor -- start 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.573+0000 7f7e26096640 1 -- start start 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e26096640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 0x7f7e20198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e26096640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e20198560 0x7f7e2019c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e26096640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e20198b60 con 0x7f7e200ff260 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e26096640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e20198cd0 con 0x7f7e20198560 2024-04-22T16:00:50.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 0x7f7e20198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:50.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 0x7f7e20198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44854/0 (socket says 172.21.15.96:44854) 2024-04-22T16:00:50.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1f7fe640 1 -- 172.21.15.96:0/3875982562 learned_addr learned my addr 172.21.15.96:0/3875982562 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:50.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1f7fe640 1 -- 172.21.15.96:0/3875982562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e20198560 msgr2=0x7f7e2019c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1f7fe640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e20198560 0x7f7e2019c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1f7fe640 1 -- 172.21.15.96:0/3875982562 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10009560 con 0x7f7e200ff260 2024-04-22T16:00:50.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1f7fe640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 0x7f7e20198020 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009430 tx=0x7f7e10004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:50.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1d7fa640 1 -- 172.21.15.96:0/3875982562 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e100035f0 con 0x7f7e200ff260 2024-04-22T16:00:50.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2019cf60 con 0x7f7e200ff260 2024-04-22T16:00:50.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1d7fa640 1 -- 172.21.15.96:0/3875982562 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e10003c10 con 0x7f7e200ff260 2024-04-22T16:00:50.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e1d7fa640 1 -- 172.21.15.96:0/3875982562 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e10005670 con 0x7f7e200ff260 2024-04-22T16:00:50.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.574+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2019d4c0 con 0x7f7e200ff260 2024-04-22T16:00:50.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.575+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2004e040 con 0x7f7e200ff260 2024-04-22T16:00:50.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.576+0000 7f7e1d7fa640 1 -- 172.21.15.96:0/3875982562 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7e10004dd0 con 0x7f7e200ff260 2024-04-22T16:00:50.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.576+0000 7f7e1d7fa640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e0c06ebe0 0x7f7e0c0710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:50.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.576+0000 7f7e1d7fa640 1 -- 172.21.15.96:0/3875982562 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f7e100b12d0 con 0x7f7e200ff260 2024-04-22T16:00:50.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.576+0000 7f7e17fff640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e0c06ebe0 0x7f7e0c0710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:50.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.577+0000 7f7e17fff640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e0c06ebe0 0x7f7e0c0710a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7e08005fd0 tx=0x7f7e08005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:50.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.581+0000 7f7e1d7fa640 1 -- 172.21.15.96:0/3875982562 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7e1007a030 con 0x7f7e200ff260 2024-04-22T16:00:50.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.798+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7f7e200ffc10 con 0x7f7e200ff260 2024-04-22T16:00:50.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.798+0000 7f7e1d7fa640 1 -- 172.21.15.96:0/3875982562 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v40) v1 ==== 107+0+5227 (secure 0 0 0) 0x7f7e1007d6f0 con 0x7f7e200ff260 2024-04-22T16:00:50.802 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:50.802 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:58:49.514414+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":0,"incarnation":33,"state":"up:reconnect","state_seq":468,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":1,"incarnation":33,"state":"up:reconnect","state_seq":462,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:50.802 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 36 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.800+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e0c06ebe0 msgr2=0x7f7e0c0710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.800+0000 7f7e26096640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e0c06ebe0 0x7f7e0c0710a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7e08005fd0 tx=0x7f7e08005850 comp rx=0 tx=0).stop 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.800+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 msgr2=0x7f7e20198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.800+0000 7f7e26096640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 0x7f7e20198020 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009430 tx=0x7f7e10004040 comp rx=0 tx=0).stop 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.800+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 shutdown_connections 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.800+0000 7f7e26096640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f7e0c06ebe0 0x7f7e0c0710a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.800+0000 7f7e26096640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e20198560 0x7f7e2019c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.800+0000 7f7e26096640 1 --2- 172.21.15.96:0/3875982562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e200ff260 0x7f7e20198020 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.801+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 >> 172.21.15.96:0/3875982562 conn(0x7f7e20074a50 msgr2=0x7f7e20103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.801+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 shutdown_connections 2024-04-22T16:00:50.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.801+0000 7f7e26096640 1 -- 172.21.15.96:0/3875982562 wait complete. 2024-04-22T16:00:50.815 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-04-22T16:00:50.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.910+0000 7f57576af640 1 -- 172.21.15.96:0/1610697449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57500675c0 msgr2=0x7f57500679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.910+0000 7f57576af640 1 --2- 172.21.15.96:0/1610697449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57500675c0 0x7f57500679a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f574c009c20 tx=0x7f574c02c1b0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.910+0000 7f57576af640 1 -- 172.21.15.96:0/1610697449 shutdown_connections 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.910+0000 7f57576af640 1 --2- 172.21.15.96:0/1610697449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5750067f70 0x7f5750104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.910+0000 7f57576af640 1 --2- 172.21.15.96:0/1610697449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57500675c0 0x7f57500679a0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.910+0000 7f57576af640 1 -- 172.21.15.96:0/1610697449 >> 172.21.15.96:0/1610697449 conn(0x7f5750074a50 msgr2=0x7f5750074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.910+0000 7f57576af640 1 -- 172.21.15.96:0/1610697449 shutdown_connections 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f57576af640 1 -- 172.21.15.96:0/1610697449 wait complete. 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f57576af640 1 Processor -- start 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f57576af640 1 -- start start 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f57576af640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57500675c0 0x7f5750195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:50.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f57576af640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57501962f0 0x7f575019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f57576af640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57501968f0 con 0x7f57501962f0 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f57576af640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5750196a60 con 0x7f57500675c0 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f5754c4a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57501962f0 0x7f575019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f5754c4a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57501962f0 0x7f575019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44868/0 (socket says 172.21.15.96:44868) 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.911+0000 7f5754c4a640 1 -- 172.21.15.96:0/2956507678 learned_addr learned my addr 172.21.15.96:0/2956507678 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f575544b640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57500675c0 0x7f5750195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f5754c4a640 1 -- 172.21.15.96:0/2956507678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57500675c0 msgr2=0x7f5750195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f5754c4a640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57500675c0 0x7f5750195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f5754c4a640 1 -- 172.21.15.96:0/2956507678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c009900 con 0x7f57501962f0 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f5754c4a640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57501962f0 0x7f575019a750 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f574000d6f0 tx=0x7f574000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f57467fc640 1 -- 172.21.15.96:0/2956507678 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57400051f0 con 0x7f57501962f0 2024-04-22T16:00:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f57467fc640 1 -- 172.21.15.96:0/2956507678 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5740009ce0 con 0x7f57501962f0 2024-04-22T16:00:50.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575019ad50 con 0x7f57501962f0 2024-04-22T16:00:50.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f57467fc640 1 -- 172.21.15.96:0/2956507678 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5740004e80 con 0x7f57501962f0 2024-04-22T16:00:50.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.912+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575019b2b0 con 0x7f57501962f0 2024-04-22T16:00:50.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.913+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575019b580 con 0x7f57501962f0 2024-04-22T16:00:50.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.913+0000 7f57467fc640 1 -- 172.21.15.96:0/2956507678 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f57400046c0 con 0x7f57501962f0 2024-04-22T16:00:50.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.914+0000 7f57467fc640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573c06ebe0 0x7f573c0710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:50.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.914+0000 7f57467fc640 1 -- 172.21.15.96:0/2956507678 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f574008e970 con 0x7f57501962f0 2024-04-22T16:00:50.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.914+0000 7f575544b640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573c06ebe0 0x7f573c0710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:50.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.914+0000 7f575544b640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573c06ebe0 0x7f573c0710a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f574c004af0 tx=0x7f574c004a40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:50.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:50.918+0000 7f57467fc640 1 -- 172.21.15.96:0/2956507678 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f57400576d0 con 0x7f57501962f0 2024-04-22T16:00:51.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.133+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f575004e040 con 0x7f57501962f0 2024-04-22T16:00:51.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.134+0000 7f57467fc640 1 -- 172.21.15.96:0/2956507678 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v40) v1 ==== 107+0+5221 (secure 0 0 0) 0x7f574005ad90 con 0x7f57501962f0 2024-04-22T16:00:51.137 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:51.137 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:58:50.476994+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":0,"incarnation":33,"state":"up:rejoin","state_seq":469,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":1,"incarnation":33,"state":"up:rejoin","state_seq":463,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:00:51.137 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 37 2024-04-22T16:00:51.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573c06ebe0 msgr2=0x7f573c0710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573c06ebe0 0x7f573c0710a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f574c004af0 tx=0x7f574c004a40 comp rx=0 tx=0).stop 2024-04-22T16:00:51.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57501962f0 msgr2=0x7f575019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57501962f0 0x7f575019a750 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f574000d6f0 tx=0x7f574000dbb0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 shutdown_connections 2024-04-22T16:00:51.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f573c06ebe0 0x7f573c0710a0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57501962f0 0x7f575019a750 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 --2- 172.21.15.96:0/2956507678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57500675c0 0x7f5750195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 >> 172.21.15.96:0/2956507678 conn(0x7f5750074a50 msgr2=0x7f57500fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:51.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 shutdown_connections 2024-04-22T16:00:51.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.136+0000 7f57576af640 1 -- 172.21.15.96:0/2956507678 wait complete. 2024-04-22T16:00:51.148 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:50 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1405858853' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-22T16:00:51.149 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:50 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3138549470' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-22T16:00:51.149 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:50 smithi096 ceph-mon[112443]: pgmap v1173: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:51.149 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:50 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3875982562' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-22T16:00:51.150 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-04-22T16:00:51.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.246+0000 7f0de50b6640 1 -- 172.21.15.96:0/1499421638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 msgr2=0x7f0de00ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.246+0000 7f0de50b6640 1 --2- 172.21.15.96:0/1499421638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 0x7f0de00ff640 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009880 tx=0x7f0dd402bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.246+0000 7f0de50b6640 1 -- 172.21.15.96:0/1499421638 shutdown_connections 2024-04-22T16:00:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.246+0000 7f0de50b6640 1 --2- 172.21.15.96:0/1499421638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0de00ffc10 0x7f0de0103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.246+0000 7f0de50b6640 1 --2- 172.21.15.96:0/1499421638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 0x7f0de00ff640 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.246+0000 7f0de50b6640 1 -- 172.21.15.96:0/1499421638 >> 172.21.15.96:0/1499421638 conn(0x7f0de0074a50 msgr2=0x7f0de0074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.246+0000 7f0de50b6640 1 -- 172.21.15.96:0/1499421638 shutdown_connections 2024-04-22T16:00:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.246+0000 7f0de50b6640 1 -- 172.21.15.96:0/1499421638 wait complete. 2024-04-22T16:00:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.247+0000 7f0de50b6640 1 Processor -- start 2024-04-22T16:00:51.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.247+0000 7f0de50b6640 1 -- start start 2024-04-22T16:00:51.318 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:50 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1405858853' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-22T16:00:51.318 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:50 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3138549470' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-22T16:00:51.318 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:50 smithi196 ceph-mon[81815]: pgmap v1173: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:00:51.318 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:50 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3875982562' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-22T16:00:51.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0de50b6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 0x7f0de0194e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:51.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0de50b6640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0de0195360 0x7f0de01997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0de50b6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de0195960 con 0x7f0de00ff260 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0de50b6640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de0195ad0 con 0x7f0de0195360 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dded9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 0x7f0de0194e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dded9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 0x7f0de0194e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44900/0 (socket says 172.21.15.96:44900) 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dded9d640 1 -- 172.21.15.96:0/2381919856 learned_addr learned my addr 172.21.15.96:0/2381919856 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dded9d640 1 -- 172.21.15.96:0/2381919856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0de0195360 msgr2=0x7f0de01997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dde59c640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0de0195360 0x7f0de01997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dded9d640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0de0195360 0x7f0de01997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dded9d640 1 -- 172.21.15.96:0/2381919856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4009560 con 0x7f0de00ff260 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dde59c640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0de0195360 0x7f0de01997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dded9d640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 0x7f0de0194e20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009430 tx=0x7f0dd4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dc7fff640 1 -- 172.21.15.96:0/2381919856 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dd40035f0 con 0x7f0de00ff260 2024-04-22T16:00:51.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de0199d60 con 0x7f0de00ff260 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dc7fff640 1 -- 172.21.15.96:0/2381919856 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dd4003c10 con 0x7f0de00ff260 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0dc7fff640 1 -- 172.21.15.96:0/2381919856 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dd4034440 con 0x7f0de00ff260 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.248+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de019a2c0 con 0x7f0de00ff260 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.249+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de004e040 con 0x7f0de00ff260 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.250+0000 7f0dc7fff640 1 -- 172.21.15.96:0/2381919856 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0dd4004fd0 con 0x7f0de00ff260 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.250+0000 7f0dc7fff640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dc806ec80 0x7f0dc8071140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.250+0000 7f0dc7fff640 1 -- 172.21.15.96:0/2381919856 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0dd40b13b0 con 0x7f0de00ff260 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.250+0000 7f0dde59c640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dc806ec80 0x7f0dc8071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.251+0000 7f0dde59c640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dc806ec80 0x7f0dc8071140 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0006fd0 tx=0x7f0dd0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:51.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.255+0000 7f0dc7fff640 1 -- 172.21.15.96:0/2381919856 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0dd407a070 con 0x7f0de00ff260 2024-04-22T16:00:51.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.471+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7f0de019a4e0 con 0x7f0de00ff260 2024-04-22T16:00:51.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.472+0000 7f0dc7fff640 1 -- 172.21.15.96:0/2381919856 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v40) v1 ==== 107+0+5236 (secure 0 0 0) 0x7f0dd407d730 con 0x7f0de00ff260 2024-04-22T16:00:51.474 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:51.475 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:58:52.480033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":0,"incarnation":33,"state":"up:active","state_seq":470,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":1,"incarnation":33,"state":"up:active","state_seq":464,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34416,"qdb_cluster":[34416,34420]},"id":1}]} 2024-04-22T16:00:51.475 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 38 2024-04-22T16:00:51.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dc806ec80 msgr2=0x7f0dc8071140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dc806ec80 0x7f0dc8071140 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0006fd0 tx=0x7f0dd0008040 comp rx=0 tx=0).stop 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 msgr2=0x7f0de0194e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 0x7f0de0194e20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009430 tx=0x7f0dd4004040 comp rx=0 tx=0).stop 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 shutdown_connections 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0dc806ec80 0x7f0dc8071140 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0de0195360 0x7f0de01997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 --2- 172.21.15.96:0/2381919856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de00ff260 0x7f0de0194e20 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 >> 172.21.15.96:0/2381919856 conn(0x7f0de0074a50 msgr2=0x7f0de00fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 shutdown_connections 2024-04-22T16:00:51.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.474+0000 7f0de50b6640 1 -- 172.21.15.96:0/2381919856 wait complete. 2024-04-22T16:00:51.488 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.583+0000 7f60fe228640 1 -- 172.21.15.96:0/879841507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f8100c50 msgr2=0x7f60f8104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.583+0000 7f60fe228640 1 --2- 172.21.15.96:0/879841507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f8100c50 0x7f60f8104c90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f60ec009880 tx=0x7f60ec02bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.583+0000 7f60fe228640 1 -- 172.21.15.96:0/879841507 shutdown_connections 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.583+0000 7f60fe228640 1 --2- 172.21.15.96:0/879841507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f8100c50 0x7f60f8104c90 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.583+0000 7f60fe228640 1 --2- 172.21.15.96:0/879841507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60f81002a0 0x7f60f8100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.583+0000 7f60fe228640 1 -- 172.21.15.96:0/879841507 >> 172.21.15.96:0/879841507 conn(0x7f60f80fbb20 msgr2=0x7f60f80fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.583+0000 7f60fe228640 1 -- 172.21.15.96:0/879841507 shutdown_connections 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.583+0000 7f60fe228640 1 -- 172.21.15.96:0/879841507 wait complete. 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60fe228640 1 Processor -- start 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60fe228640 1 -- start start 2024-04-22T16:00:51.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60fe228640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f81002a0 0x7f60f8195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60fe228640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60f81962c0 0x7f60f819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60fe228640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f81968c0 con 0x7f60f81002a0 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60fe228640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f8196a30 con 0x7f60f81962c0 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60f77fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f81002a0 0x7f60f8195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60f77fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f81002a0 0x7f60f8195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44916/0 (socket says 172.21.15.96:44916) 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60f77fe640 1 -- 172.21.15.96:0/2965918796 learned_addr learned my addr 172.21.15.96:0/2965918796 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60f6ffd640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60f81962c0 0x7f60f819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60f77fe640 1 -- 172.21.15.96:0/2965918796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60f81962c0 msgr2=0x7f60f819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60f77fe640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60f81962c0 0x7f60f819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60f77fe640 1 -- 172.21.15.96:0/2965918796 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec009560 con 0x7f60f81002a0 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60f6ffd640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60f81962c0 0x7f60f819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.584+0000 7f60f77fe640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f81002a0 0x7f60f8195d80 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f60e800d720 tx=0x7f60e800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:51.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.585+0000 7f60f4ff9640 1 -- 172.21.15.96:0/2965918796 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60e80051f0 con 0x7f60f81002a0 2024-04-22T16:00:51.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.585+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f819ad20 con 0x7f60f81002a0 2024-04-22T16:00:51.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.585+0000 7f60f4ff9640 1 -- 172.21.15.96:0/2965918796 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60e8009ce0 con 0x7f60f81002a0 2024-04-22T16:00:51.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.585+0000 7f60f4ff9640 1 -- 172.21.15.96:0/2965918796 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60e8010460 con 0x7f60f81002a0 2024-04-22T16:00:51.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.585+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f819b280 con 0x7f60f81002a0 2024-04-22T16:00:51.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.586+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f804e040 con 0x7f60f81002a0 2024-04-22T16:00:51.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.586+0000 7f60f4ff9640 1 -- 172.21.15.96:0/2965918796 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f60e8005350 con 0x7f60f81002a0 2024-04-22T16:00:51.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.587+0000 7f60f4ff9640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60e006ec30 0x7f60e00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:51.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.587+0000 7f60f4ff9640 1 -- 172.21.15.96:0/2965918796 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f60e808ea40 con 0x7f60f81002a0 2024-04-22T16:00:51.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.587+0000 7f60f6ffd640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60e006ec30 0x7f60e00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:51.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.587+0000 7f60f6ffd640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60e006ec30 0x7f60e00710f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f60ec005ec0 tx=0x7f60ec005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:51.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.591+0000 7f60f4ff9640 1 -- 172.21.15.96:0/2965918796 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f60e80577a0 con 0x7f60f81002a0 2024-04-22T16:00:51.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.819+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7f60f81970d0 con 0x7f60f81002a0 2024-04-22T16:00:51.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.819+0000 7f60f4ff9640 1 -- 172.21.15.96:0/2965918796 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v40) v1 ==== 107+0+5251 (secure 0 0 0) 0x7f60e805ae60 con 0x7f60f81002a0 2024-04-22T16:00:51.823 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:51.823 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:15:33.756786+0000","modified":"2024-04-22T15:58:52.489325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi096.dymqlv","rank":0,"incarnation":33,"state":"up:active","state_seq":470,"addr":"172.21.15.96:6827/1647958878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1647958878},{"type":"v1","addr":"172.21.15.96:6827","nonce":1647958878}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi096.drkkdi","rank":1,"incarnation":33,"state":"up:active","state_seq":464,"addr":"172.21.15.96:6829/370524327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":370524327},{"type":"v1","addr":"172.21.15.96:6829","nonce":370524327}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44279":{"gid":44279,"name":"cephfs.smithi196.hwpcvg","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6825/4268726265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4268726265},{"type":"v1","addr":"172.21.15.196:6825","nonce":4268726265}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44757":{"gid":44757,"name":"cephfs.smithi196.pbvqlq","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6827/4276735029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4276735029},{"type":"v1","addr":"172.21.15.196:6827","nonce":4276735029}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34416,"qdb_cluster":[34420,34416]},"id":1}]} 2024-04-22T16:00:51.823 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 39 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60e006ec30 msgr2=0x7f60e00710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60e006ec30 0x7f60e00710f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f60ec005ec0 tx=0x7f60ec005fd0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f81002a0 msgr2=0x7f60f8195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f81002a0 0x7f60f8195d80 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f60e800d720 tx=0x7f60e800dbe0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 shutdown_connections 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f60e006ec30 0x7f60e00710f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60f81962c0 0x7f60f819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 --2- 172.21.15.96:0/2965918796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60f81002a0 0x7f60f8195d80 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 >> 172.21.15.96:0/2965918796 conn(0x7f60f80fbb20 msgr2=0x7f60f80fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 shutdown_connections 2024-04-22T16:00:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:51.822+0000 7f60fe228640 1 -- 172.21.15.96:0/2965918796 wait complete. 2024-04-22T16:00:51.836 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-22T16:00:51.853 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-22T16:00:51.853 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T16:00:51.853 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T16:00:51.870 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T16:00:51.870 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T16:00:51.926 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T16:00:52.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 -- 172.21.15.96:0/761520819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82400ffb30 msgr2=0x7f8240103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 --2- 172.21.15.96:0/761520819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82400ffb30 0x7f8240103b70 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f8234009880 tx=0x7f823402bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 -- 172.21.15.96:0/761520819 shutdown_connections 2024-04-22T16:00:52.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 --2- 172.21.15.96:0/761520819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82400ffb30 0x7f8240103b70 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 --2- 172.21.15.96:0/761520819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82400ff180 0x7f82400ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 -- 172.21.15.96:0/761520819 >> 172.21.15.96:0/761520819 conn(0x7f82400faa00 msgr2=0x7f82400fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:52.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 -- 172.21.15.96:0/761520819 shutdown_connections 2024-04-22T16:00:52.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 -- 172.21.15.96:0/761520819 wait complete. 2024-04-22T16:00:52.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 Processor -- start 2024-04-22T16:00:52.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.062+0000 7f82468ce640 1 -- start start 2024-04-22T16:00:52.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f82468ce640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82400ff180 0x7f824019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f82468ce640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f824019aeb0 0x7f82401949e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f82468ce640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824019b480 con 0x7f824019aeb0 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f82468ce640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8240194f20 con 0x7f82400ff180 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f823f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f824019aeb0 0x7f82401949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f823f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f824019aeb0 0x7f82401949e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44940/0 (socket says 172.21.15.96:44940) 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f823f7fe640 1 -- 172.21.15.96:0/2764906541 learned_addr learned my addr 172.21.15.96:0/2764906541 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f823f7fe640 1 -- 172.21.15.96:0/2764906541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82400ff180 msgr2=0x7f824019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f823ffff640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82400ff180 0x7f824019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f823f7fe640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82400ff180 0x7f824019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f823f7fe640 1 -- 172.21.15.96:0/2764906541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8234009560 con 0x7f824019aeb0 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f823f7fe640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f824019aeb0 0x7f82401949e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f8234009230 tx=0x7f823402c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.063+0000 7f823d7fa640 1 -- 172.21.15.96:0/2764906541 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82340035f0 con 0x7f824019aeb0 2024-04-22T16:00:52.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.064+0000 7f823d7fa640 1 -- 172.21.15.96:0/2764906541 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8234003c10 con 0x7f824019aeb0 2024-04-22T16:00:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.064+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82401951a0 con 0x7f824019aeb0 2024-04-22T16:00:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.064+0000 7f823d7fa640 1 -- 172.21.15.96:0/2764906541 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8234005690 con 0x7f824019aeb0 2024-04-22T16:00:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.064+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82401956a0 con 0x7f824019aeb0 2024-04-22T16:00:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.064+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824004e040 con 0x7f824019aeb0 2024-04-22T16:00:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.065+0000 7f823d7fa640 1 -- 172.21.15.96:0/2764906541 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8234004ed0 con 0x7f824019aeb0 2024-04-22T16:00:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.065+0000 7f823d7fa640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f822806ecf0 0x7f82280711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.065+0000 7f823d7fa640 1 -- 172.21.15.96:0/2764906541 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f82340b1340 con 0x7f824019aeb0 2024-04-22T16:00:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.065+0000 7f823ffff640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f822806ecf0 0x7f82280711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.066+0000 7f823ffff640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f822806ecf0 0x7f82280711b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8230005fd0 tx=0x7f8230005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:52.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.070+0000 7f823d7fa640 1 -- 172.21.15.96:0/2764906541 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f823407a040 con 0x7f824019aeb0 2024-04-22T16:00:52.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:51 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2956507678' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-22T16:00:52.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:51 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2381919856' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-22T16:00:52.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:51 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2965918796' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-22T16:00:52.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.248+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8240195ef0 con 0x7f824019aeb0 2024-04-22T16:00:52.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.249+0000 7f823d7fa640 1 -- 172.21.15.96:0/2764906541 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16771 (secure 0 0 0) 0x7f823407d700 con 0x7f824019aeb0 2024-04-22T16:00:52.252 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:52.252 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":84,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","created":"2024-04-22T15:08:23.492994+0000","modified":"2024-04-22T15:43:33.206096+0000","last_up_change":"2024-04-22T15:27:37.407115+0000","last_in_change":"2024-04-22T15:13:56.089739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:13:01.924433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T15:15:32.350570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T15:15:32.982528+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2f1b7772-06b4-467f-b14c-698a0de7357f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6803","nonce":312948544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6805","nonce":312948544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6809","nonce":312948544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6807","nonce":312948544}]},"public_addr":"172.21.15.96:6803/312948544","cluster_addr":"172.21.15.96:6805/312948544","heartbeat_back_addr":"172.21.15.96:6809/312948544","heartbeat_front_addr":"172.21.15.96:6807/312948544","state":["exists","up"]},{"osd":1,"uuid":"59260a36-a2ed-42f9-8f3a-209aae4c4238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6811","nonce":3607865374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6813","nonce":3607865374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6817","nonce":3607865374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6815","nonce":3607865374}]},"public_addr":"172.21.15.96:6811/3607865374","cluster_addr":"172.21.15.96:6813/3607865374","heartbeat_back_addr":"172.21.15.96:6817/3607865374","heartbeat_front_addr":"172.21.15.96:6815/3607865374","state":["exists","up"]},{"osd":2,"uuid":"897e21a9-3009-41c6-acf1-e44414fc69c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6819","nonce":2071585004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6821","nonce":2071585004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6825","nonce":2071585004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6823","nonce":2071585004}]},"public_addr":"172.21.15.96:6819/2071585004","cluster_addr":"172.21.15.96:6821/2071585004","heartbeat_back_addr":"172.21.15.96:6825/2071585004","heartbeat_front_addr":"172.21.15.96:6823/2071585004","state":["exists","up"]},{"osd":3,"uuid":"201c3153-a284-471e-894d-2273001cc29e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6801","nonce":2743487260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6803","nonce":2743487260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6807","nonce":2743487260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6805","nonce":2743487260}]},"public_addr":"172.21.15.196:6801/2743487260","cluster_addr":"172.21.15.196:6803/2743487260","heartbeat_back_addr":"172.21.15.196:6807/2743487260","heartbeat_front_addr":"172.21.15.196:6805/2743487260","state":["exists","up"]},{"osd":4,"uuid":"780ed007-14ef-4c92-a5e9-6c57afcad448","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6809","nonce":2228895659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6811","nonce":2228895659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6815","nonce":2228895659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6813","nonce":2228895659}]},"public_addr":"172.21.15.196:6809/2228895659","cluster_addr":"172.21.15.196:6811/2228895659","heartbeat_back_addr":"172.21.15.196:6815/2228895659","heartbeat_front_addr":"172.21.15.196:6813/2228895659","state":["exists","up"]},{"osd":5,"uuid":"16384d8d-bef4-4f65-9079-e2b4518241da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6817","nonce":288820489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6819","nonce":288820489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6823","nonce":288820489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6821","nonce":288820489}]},"public_addr":"172.21.15.196:6817/288820489","cluster_addr":"172.21.15.196:6819/288820489","heartbeat_back_addr":"172.21.15.196:6823/288820489","heartbeat_front_addr":"172.21.15.196:6821/288820489","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:00.654872+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:28.691804+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:57.737056+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:21.679902+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:47.489372+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:14:13.600900+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6829/3534778299":"2024-04-23T15:27:39.581162+0000","172.21.15.96:6828/3534778299":"2024-04-23T15:27:39.581162+0000","172.21.15.196:0/465874235":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/1920801848":"2024-04-23T15:17:52.351792+0000","172.21.15.96:6801/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6800/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/838799683":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6800/2441203700":"2024-04-23T15:17:52.351792+0000","172.21.15.196:6825/3248990491":"2024-04-23T15:27:39.581179+0000","172.21.15.96:6800/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1319578885":"2024-04-23T15:09:23.920294+0000","172.21.15.196:0/427333217":"2024-04-23T15:19:21.709732+0000","172.21.15.196:6828/2093434004":"2024-04-23T15:19:21.709732+0000","172.21.15.96:6800/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:0/328396090":"2024-04-23T15:23:08.192054+0000","172.21.15.96:0/168995622":"2024-04-23T15:17:52.351792+0000","172.21.15.196:0/4000780168":"2024-04-23T15:19:21.709732+0000","172.21.15.196:6824/3248990491":"2024-04-23T15:27:39.581179+0000","172.21.15.196:0/4127822214":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/340578852":"2024-04-23T15:17:52.351792+0000","172.21.15.96:0/1237353220":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3486391579":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/4224647460":"2024-04-23T15:08:51.463883+0000","172.21.15.96:6801/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3667378212":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1860450164":"2024-04-23T15:17:52.351792+0000","172.21.15.196:0/1567934302":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/3321679733":"2024-04-23T15:09:23.920294+0000","172.21.15.196:0/391452056":"2024-04-23T15:19:21.709732+0000","172.21.15.196:6829/2093434004":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/3921802975":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/2918293154":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6826/887933858":"2024-04-23T15:15:49.819302+0000","172.21.15.96:6827/887933858":"2024-04-23T15:15:49.819302+0000","172.21.15.96:0/3360662441":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6801/2441203700":"2024-04-23T15:17:52.351792+0000","172.21.15.96:0/1794679103":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3926078840":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6800/3666812526":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6801/3666812526":"2024-04-23T15:23:08.192054+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f822806ecf0 msgr2=0x7f82280711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f822806ecf0 0x7f82280711b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8230005fd0 tx=0x7f8230005850 comp rx=0 tx=0).stop 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f824019aeb0 msgr2=0x7f82401949e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f824019aeb0 0x7f82401949e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f8234009230 tx=0x7f823402c850 comp rx=0 tx=0).stop 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 shutdown_connections 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f822806ecf0 0x7f82280711b0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f824019aeb0 0x7f82401949e0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 --2- 172.21.15.96:0/2764906541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82400ff180 0x7f824019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 >> 172.21.15.96:0/2764906541 conn(0x7f82400faa00 msgr2=0x7f82400fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 shutdown_connections 2024-04-22T16:00:52.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.252+0000 7f82468ce640 1 -- 172.21.15.96:0/2764906541 wait complete. 2024-04-22T16:00:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:51 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2956507678' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-22T16:00:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:51 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2381919856' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-22T16:00:52.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:51 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2965918796' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-22T16:00:52.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:13:01.924433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T15:15:32.350570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T15:15:32.982528+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T16:00:52.266 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T16:00:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.363+0000 7faf86a88640 1 -- 172.21.15.96:0/2130960833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf80100c90 msgr2=0x7faf80104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.363+0000 7faf86a88640 1 --2- 172.21.15.96:0/2130960833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf80100c90 0x7faf80104cd0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7faf70009c20 tx=0x7faf7002c1b0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 -- 172.21.15.96:0/2130960833 shutdown_connections 2024-04-22T16:00:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 --2- 172.21.15.96:0/2130960833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf80100c90 0x7faf80104cd0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 --2- 172.21.15.96:0/2130960833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf801002e0 0x7faf801006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 -- 172.21.15.96:0/2130960833 >> 172.21.15.96:0/2130960833 conn(0x7faf800fbb80 msgr2=0x7faf800fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 -- 172.21.15.96:0/2130960833 shutdown_connections 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 -- 172.21.15.96:0/2130960833 wait complete. 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 Processor -- start 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 -- start start 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf801002e0 0x7faf80195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf801962e0 0x7faf8019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf801968e0 con 0x7faf801002e0 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf86a88640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf80196a50 con 0x7faf801962e0 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf84824640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf801002e0 0x7faf80195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf84824640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf801002e0 0x7faf80195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44956/0 (socket says 172.21.15.96:44956) 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.364+0000 7faf84824640 1 -- 172.21.15.96:0/3156546630 learned_addr learned my addr 172.21.15.96:0/3156546630 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf7ffff640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf801962e0 0x7faf8019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf84824640 1 -- 172.21.15.96:0/3156546630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf801962e0 msgr2=0x7faf8019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf84824640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf801962e0 0x7faf8019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf84824640 1 -- 172.21.15.96:0/3156546630 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf70009900 con 0x7faf801002e0 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf7ffff640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf801962e0 0x7faf8019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf84824640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf801002e0 0x7faf80195da0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7faf7800d900 tx=0x7faf7800ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf7dffb640 1 -- 172.21.15.96:0/3156546630 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf78009c00 con 0x7faf801002e0 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8019ad40 con 0x7faf801002e0 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf7dffb640 1 -- 172.21.15.96:0/3156546630 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf78009d60 con 0x7faf801002e0 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf7dffb640 1 -- 172.21.15.96:0/3156546630 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf780049d0 con 0x7faf801002e0 2024-04-22T16:00:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.365+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8019b2a0 con 0x7faf801002e0 2024-04-22T16:00:52.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.366+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8019b580 con 0x7faf801002e0 2024-04-22T16:00:52.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.366+0000 7faf7dffb640 1 -- 172.21.15.96:0/3156546630 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faf78004020 con 0x7faf801002e0 2024-04-22T16:00:52.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.367+0000 7faf7dffb640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faf6c06ec30 0x7faf6c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.367+0000 7faf7dffb640 1 -- 172.21.15.96:0/3156546630 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7faf7808ec20 con 0x7faf801002e0 2024-04-22T16:00:52.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.367+0000 7faf7ffff640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faf6c06ec30 0x7faf6c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.367+0000 7faf7ffff640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faf6c06ec30 0x7faf6c0710f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7faf70009430 tx=0x7faf70009360 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:52.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.371+0000 7faf7dffb640 1 -- 172.21.15.96:0/3156546630 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faf78057980 con 0x7faf801002e0 2024-04-22T16:00:52.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.548+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faf801085e0 con 0x7faf801002e0 2024-04-22T16:00:52.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.548+0000 7faf7dffb640 1 -- 172.21.15.96:0/3156546630 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7faf7805b040 con 0x7faf801002e0 2024-04-22T16:00:52.551 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-04-22T16:00:52.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faf6c06ec30 msgr2=0x7faf6c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faf6c06ec30 0x7faf6c0710f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7faf70009430 tx=0x7faf70009360 comp rx=0 tx=0).stop 2024-04-22T16:00:52.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf801002e0 msgr2=0x7faf80195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf801002e0 0x7faf80195da0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7faf7800d900 tx=0x7faf7800ddc0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 shutdown_connections 2024-04-22T16:00:52.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7faf6c06ec30 0x7faf6c0710f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf801962e0 0x7faf8019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 --2- 172.21.15.96:0/3156546630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf801002e0 0x7faf80195da0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 >> 172.21.15.96:0/3156546630 conn(0x7faf800fbb80 msgr2=0x7faf800fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:52.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 shutdown_connections 2024-04-22T16:00:52.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.550+0000 7faf86a88640 1 -- 172.21.15.96:0/3156546630 wait complete. 2024-04-22T16:00:52.564 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T16:00:52.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.661+0000 7f0ffafa2640 1 -- 172.21.15.96:0/2458195344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 msgr2=0x7f0ff40ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.661+0000 7f0ffafa2640 1 --2- 172.21.15.96:0/2458195344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 0x7f0ff40ff580 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009880 tx=0x7f0fe802bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.661+0000 7f0ffafa2640 1 -- 172.21.15.96:0/2458195344 shutdown_connections 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.661+0000 7f0ffafa2640 1 --2- 172.21.15.96:0/2458195344 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ff40ffb50 0x7f0ff4103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.661+0000 7f0ffafa2640 1 --2- 172.21.15.96:0/2458195344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 0x7f0ff40ff580 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.661+0000 7f0ffafa2640 1 -- 172.21.15.96:0/2458195344 >> 172.21.15.96:0/2458195344 conn(0x7f0ff40faa20 msgr2=0x7f0ff40fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.661+0000 7f0ffafa2640 1 -- 172.21.15.96:0/2458195344 shutdown_connections 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.661+0000 7f0ffafa2640 1 -- 172.21.15.96:0/2458195344 wait complete. 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ffafa2640 1 Processor -- start 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ffafa2640 1 -- start start 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ffafa2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 0x7f0ff4194ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ffafa2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ff4195220 0x7f0ff41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ffafa2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4195790 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ffafa2640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff41958d0 con 0x7f0ff4195220 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff8d3e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 0x7f0ff4194ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff8d3e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 0x7f0ff4194ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44972/0 (socket says 172.21.15.96:44972) 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff8d3e640 1 -- 172.21.15.96:0/1261368996 learned_addr learned my addr 172.21.15.96:0/1261368996 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff8d3e640 1 -- 172.21.15.96:0/1261368996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ff4195220 msgr2=0x7f0ff41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff8d3e640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ff4195220 0x7f0ff41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff8d3e640 1 -- 172.21.15.96:0/1261368996 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8009560 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff8d3e640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 0x7f0ff4194ce0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009230 tx=0x7f0fe8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff1ffb640 1 -- 172.21.15.96:0/1261368996 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe80035f0 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff1ffb640 1 -- 172.21.15.96:0/1261368996 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fe8003c10 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff4198b10 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.662+0000 7f0ff1ffb640 1 -- 172.21.15.96:0/1261368996 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe803b440 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.663+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff4199070 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.664+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff404e040 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.664+0000 7f0ff1ffb640 1 -- 172.21.15.96:0/1261368996 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0fe8004fd0 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.664+0000 7f0ff1ffb640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0fe406ec30 0x7f0fe40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.665+0000 7f0ff1ffb640 1 -- 172.21.15.96:0/1261368996 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0fe80b3110 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.665+0000 7f0ff3fff640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0fe406ec30 0x7f0fe40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.665+0000 7f0ff3fff640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0fe406ec30 0x7f0fe40710f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40697f0 tx=0x7f0fe0005c70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:52.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.669+0000 7f0ff1ffb640 1 -- 172.21.15.96:0/1261368996 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0fe807be70 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.848+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f0ff40008d0 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.849+0000 7f0ff1ffb640 1 -- 172.21.15.96:0/1261368996 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0fe807f530 con 0x7f0ff40ff1a0 2024-04-22T16:00:52.852 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0fe406ec30 msgr2=0x7f0fe40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0fe406ec30 0x7f0fe40710f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40697f0 tx=0x7f0fe0005c70 comp rx=0 tx=0).stop 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 msgr2=0x7f0ff4194ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 0x7f0ff4194ce0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009230 tx=0x7f0fe8004040 comp rx=0 tx=0).stop 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 shutdown_connections 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f0fe406ec30 0x7f0fe40710f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ff4195220 0x7f0ff41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 --2- 172.21.15.96:0/1261368996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ff40ff1a0 0x7f0ff4194ce0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 >> 172.21.15.96:0/1261368996 conn(0x7f0ff40faa20 msgr2=0x7f0ff40fc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 shutdown_connections 2024-04-22T16:00:52.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.851+0000 7f0ffafa2640 1 -- 172.21.15.96:0/1261368996 wait complete. 2024-04-22T16:00:52.864 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.963+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/421162111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f41002a0 msgr2=0x7fc8f4100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.963+0000 7fc8fa4e5640 1 --2- 172.21.15.96:0/421162111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f41002a0 0x7fc8f4100680 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009880 tx=0x7fc8e802bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.963+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/421162111 shutdown_connections 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.963+0000 7fc8fa4e5640 1 --2- 172.21.15.96:0/421162111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f4100c50 0x7fc8f4104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.963+0000 7fc8fa4e5640 1 --2- 172.21.15.96:0/421162111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f41002a0 0x7fc8f4100680 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.963+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/421162111 >> 172.21.15.96:0/421162111 conn(0x7fc8f40fbb20 msgr2=0x7fc8f40fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.963+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/421162111 shutdown_connections 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.963+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/421162111 wait complete. 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8fa4e5640 1 Processor -- start 2024-04-22T16:00:52.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8fa4e5640 1 -- start start 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8fa4e5640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f41002a0 0x7fc8f40725a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8fa4e5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f4072ae0 0x7fc8f4072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8fa4e5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f41a2040 con 0x7fc8f4072ae0 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8fa4e5640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f41a2180 con 0x7fc8f41002a0 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8f37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f4072ae0 0x7fc8f4072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8f37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f4072ae0 0x7fc8f4072f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44986/0 (socket says 172.21.15.96:44986) 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8f37fe640 1 -- 172.21.15.96:0/3866477171 learned_addr learned my addr 172.21.15.96:0/3866477171 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8f3fff640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f41002a0 0x7fc8f40725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8f37fe640 1 -- 172.21.15.96:0/3866477171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f41002a0 msgr2=0x7fc8f40725a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8f37fe640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f41002a0 0x7fc8f40725a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8f37fe640 1 -- 172.21.15.96:0/3866477171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e8009560 con 0x7fc8f4072ae0 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.964+0000 7fc8f3fff640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f41002a0 0x7fc8f40725a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:52.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.965+0000 7fc8f37fe640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f4072ae0 0x7fc8f4072f40 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b400 tx=0x7fc8e400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:52.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.965+0000 7fc8f17fa640 1 -- 172.21.15.96:0/3866477171 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e4011800 con 0x7fc8f4072ae0 2024-04-22T16:00:52.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.965+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f41a22c0 con 0x7fc8f4072ae0 2024-04-22T16:00:52.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.965+0000 7fc8f17fa640 1 -- 172.21.15.96:0/3866477171 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8e4011e20 con 0x7fc8f4072ae0 2024-04-22T16:00:52.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.965+0000 7fc8f17fa640 1 -- 172.21.15.96:0/3866477171 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e400f2e0 con 0x7fc8f4072ae0 2024-04-22T16:00:52.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.965+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f41a2820 con 0x7fc8f4072ae0 2024-04-22T16:00:52.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.966+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f404e040 con 0x7fc8f4072ae0 2024-04-22T16:00:52.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.966+0000 7fc8f17fa640 1 -- 172.21.15.96:0/3866477171 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc8e4011960 con 0x7fc8f4072ae0 2024-04-22T16:00:52.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.966+0000 7fc8f17fa640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8dc06ec30 0x7fc8dc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:52.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.967+0000 7fc8f17fa640 1 -- 172.21.15.96:0/3866477171 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc8e408eb60 con 0x7fc8f4072ae0 2024-04-22T16:00:52.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.967+0000 7fc8f3fff640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8dc06ec30 0x7fc8dc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:52.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.967+0000 7fc8f3fff640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8dc06ec30 0x7fc8dc0710f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8005ec0 tx=0x7fc8e8005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:52.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:52.971+0000 7fc8f17fa640 1 -- 172.21.15.96:0/3866477171 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc8e40578c0 con 0x7fc8f4072ae0 2024-04-22T16:00:53.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.157+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fc8f41085a0 con 0x7fc8f4072ae0 2024-04-22T16:00:53.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.158+0000 7fc8f17fa640 1 -- 172.21.15.96:0/3866477171 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc8e405af80 con 0x7fc8f4072ae0 2024-04-22T16:00:53.161 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8dc06ec30 msgr2=0x7fc8dc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8dc06ec30 0x7fc8dc0710f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8005ec0 tx=0x7fc8e8005fd0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f4072ae0 msgr2=0x7fc8f4072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f4072ae0 0x7fc8f4072f40 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b400 tx=0x7fc8e400b8c0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 shutdown_connections 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fc8dc06ec30 0x7fc8dc0710f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8f4072ae0 0x7fc8f4072f40 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 --2- 172.21.15.96:0/3866477171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8f41002a0 0x7fc8f40725a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 >> 172.21.15.96:0/3866477171 conn(0x7fc8f40fbb20 msgr2=0x7fc8f40fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:53.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 shutdown_connections 2024-04-22T16:00:53.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.160+0000 7fc8fa4e5640 1 -- 172.21.15.96:0/3866477171 wait complete. 2024-04-22T16:00:53.172 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:52 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2764906541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:00:53.173 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:52 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3156546630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:00:53.173 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:52 smithi096 ceph-mon[112443]: pgmap v1174: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:53.174 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T16:00:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:52 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2764906541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:00:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:52 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3156546630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:00:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:52 smithi196 ceph-mon[81815]: pgmap v1174: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:53.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.269+0000 7fabfd2ab640 1 -- 172.21.15.96:0/1784177826 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 msgr2=0x7fabf8109fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.269+0000 7fabfd2ab640 1 --2- 172.21.15.96:0/1784177826 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 0x7fabf8109fd0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fabec009880 tx=0x7fabec02bd70 comp rx=0 tx=0).stop 2024-04-22T16:00:53.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.269+0000 7fabfd2ab640 1 -- 172.21.15.96:0/1784177826 shutdown_connections 2024-04-22T16:00:53.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.269+0000 7fabfd2ab640 1 --2- 172.21.15.96:0/1784177826 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 0x7fabf8109fd0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.269+0000 7fabfd2ab640 1 --2- 172.21.15.96:0/1784177826 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf81011d0 0x7fabf81015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.269+0000 7fabfd2ab640 1 -- 172.21.15.96:0/1784177826 >> 172.21.15.96:0/1784177826 conn(0x7fabf80fa9d0 msgr2=0x7fabf80fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.269+0000 7fabfd2ab640 1 -- 172.21.15.96:0/1784177826 shutdown_connections 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.269+0000 7fabfd2ab640 1 -- 172.21.15.96:0/1784177826 wait complete. 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.270+0000 7fabfd2ab640 1 Processor -- start 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.270+0000 7fabfd2ab640 1 -- start start 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.270+0000 7fabfd2ab640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf81011d0 0x7fabf819ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.270+0000 7fabfd2ab640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 0x7fabf819bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.270+0000 7fabfd2ab640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8195aa0 con 0x7fabf8101af0 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.270+0000 7fabfd2ab640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8195c10 con 0x7fabf81011d0 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.270+0000 7fabf67fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 0x7fabf819bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.270+0000 7fabf67fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 0x7fabf819bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45008/0 (socket says 172.21.15.96:45008) 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.270+0000 7fabf67fc640 1 -- 172.21.15.96:0/3246803530 learned_addr learned my addr 172.21.15.96:0/3246803530 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabf6ffd640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf81011d0 0x7fabf819ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabf67fc640 1 -- 172.21.15.96:0/3246803530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf81011d0 msgr2=0x7fabf819ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabf67fc640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf81011d0 0x7fabf819ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabf67fc640 1 -- 172.21.15.96:0/3246803530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec009560 con 0x7fabf8101af0 2024-04-22T16:00:53.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabf67fc640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 0x7fabf819bf70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fabec009230 tx=0x7fabec02c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:53.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabdffff640 1 -- 172.21.15.96:0/3246803530 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabec0044a0 con 0x7fabf8101af0 2024-04-22T16:00:53.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabdffff640 1 -- 172.21.15.96:0/3246803530 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabec003ab0 con 0x7fabf8101af0 2024-04-22T16:00:53.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf8195e90 con 0x7fabf8101af0 2024-04-22T16:00:53.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabdffff640 1 -- 172.21.15.96:0/3246803530 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabec005600 con 0x7fabf8101af0 2024-04-22T16:00:53.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.271+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf81963f0 con 0x7fabf8101af0 2024-04-22T16:00:53.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.272+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf804e040 con 0x7fabf8101af0 2024-04-22T16:00:53.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.272+0000 7fabdffff640 1 -- 172.21.15.96:0/3246803530 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fabec004dd0 con 0x7fabf8101af0 2024-04-22T16:00:53.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.273+0000 7fabdffff640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabe006ecf0 0x7fabe00711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:53.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.273+0000 7fabf6ffd640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabe006ecf0 0x7fabe00711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:53.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.273+0000 7fabdffff640 1 -- 172.21.15.96:0/3246803530 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fabec0b19e0 con 0x7fabf8101af0 2024-04-22T16:00:53.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.273+0000 7fabf6ffd640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabe006ecf0 0x7fabe00711b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fabe8005fd0 tx=0x7fabe8005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:53.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.278+0000 7fabdffff640 1 -- 172.21.15.96:0/3246803530 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fabec07a740 con 0x7fabf8101af0 2024-04-22T16:00:53.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.472+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fabf8107640 con 0x7fabf8101af0 2024-04-22T16:00:53.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.472+0000 7fabdffff640 1 -- 172.21.15.96:0/3246803530 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 41 entries v84) v1 ==== 81+0+2431 (secure 0 0 0) 0x7fabec07de00 con 0x7fabf8101af0 2024-04-22T16:00:53.475 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6829/3534778299 2024-04-23T15:27:39.581162+0000 2024-04-22T16:00:53.475 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6828/3534778299 2024-04-23T15:27:39.581162+0000 2024-04-22T16:00:53.475 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/465874235 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:53.475 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1920801848 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:53.475 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/976363826 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:53.475 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/976363826 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/838799683 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/3275369520 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/2441203700 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:6825/3248990491 2024-04-23T15:27:39.581179+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/2684869170 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1319578885 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/427333217 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:6828/2093434004 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/3275369520 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/328396090 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/168995622 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:53.476 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/4000780168 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:6824/3248990491 2024-04-23T15:27:39.581179+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/4127822214 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/340578852 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1237353220 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3486391579 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/4224647460 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/2684869170 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3667378212 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1860450164 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/1567934302 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3321679733 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/391452056 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:6829/2093434004 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:53.477 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3921802975 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:53.478 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/2918293154 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:53.478 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6826/887933858 2024-04-23T15:15:49.819302+0000 2024-04-22T16:00:53.478 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6827/887933858 2024-04-23T15:15:49.819302+0000 2024-04-22T16:00:53.478 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3360662441 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:53.478 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/2441203700 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:53.478 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1794679103 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:53.478 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3926078840 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:53.478 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/3666812526 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:53.478 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/3666812526 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:listed 41 entries 2024-04-22T16:00:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabe006ecf0 msgr2=0x7fabe00711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabe006ecf0 0x7fabe00711b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fabe8005fd0 tx=0x7fabe8005850 comp rx=0 tx=0).stop 2024-04-22T16:00:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 msgr2=0x7fabf819bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 0x7fabf819bf70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fabec009230 tx=0x7fabec02c7a0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 shutdown_connections 2024-04-22T16:00:53.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7fabe006ecf0 0x7fabe00711b0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabf8101af0 0x7fabf819bf70 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 --2- 172.21.15.96:0/3246803530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabf81011d0 0x7fabf819ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 >> 172.21.15.96:0/3246803530 conn(0x7fabf80fa9d0 msgr2=0x7fabf8104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:53.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 shutdown_connections 2024-04-22T16:00:53.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.474+0000 7fabfd2ab640 1 -- 172.21.15.96:0/3246803530 wait complete. 2024-04-22T16:00:53.488 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T16:00:53.488 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T16:00:53.505 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T16:00:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.641+0000 7f544967d640 1 -- 172.21.15.96:0/2742301369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ffb50 msgr2=0x7f5444103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.641+0000 7f544967d640 1 --2- 172.21.15.96:0/2742301369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ffb50 0x7f5444103b90 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f5438009880 tx=0x7f543802bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.641+0000 7f544967d640 1 -- 172.21.15.96:0/2742301369 shutdown_connections 2024-04-22T16:00:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.641+0000 7f544967d640 1 --2- 172.21.15.96:0/2742301369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ffb50 0x7f5444103b90 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.641+0000 7f544967d640 1 --2- 172.21.15.96:0/2742301369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54440ff1a0 0x7f54440ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.641+0000 7f544967d640 1 -- 172.21.15.96:0/2742301369 >> 172.21.15.96:0/2742301369 conn(0x7f54440faa00 msgr2=0x7f54440fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.641+0000 7f544967d640 1 -- 172.21.15.96:0/2742301369 shutdown_connections 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.641+0000 7f544967d640 1 -- 172.21.15.96:0/2742301369 wait complete. 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.641+0000 7f544967d640 1 Processor -- start 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f544967d640 1 -- start start 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f544967d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ff1a0 0x7f5444196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f544967d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5444197060 0x7f54441974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f544967d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544419fb90 con 0x7f54440ff1a0 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f544967d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544419fcd0 con 0x7f5444197060 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f5442ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ff1a0 0x7f5444196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f5442ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ff1a0 0x7f5444196b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45024/0 (socket says 172.21.15.96:45024) 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f5442ffd640 1 -- 172.21.15.96:0/118108053 learned_addr learned my addr 172.21.15.96:0/118108053 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f54427fc640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5444197060 0x7f54441974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f5442ffd640 1 -- 172.21.15.96:0/118108053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5444197060 msgr2=0x7f54441974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f5442ffd640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5444197060 0x7f54441974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f5442ffd640 1 -- 172.21.15.96:0/118108053 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438009560 con 0x7f54440ff1a0 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f54427fc640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5444197060 0x7f54441974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f5442ffd640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ff1a0 0x7f5444196b20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f543400d720 tx=0x7f543400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f542bfff640 1 -- 172.21.15.96:0/118108053 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54340051f0 con 0x7f54440ff1a0 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544419ff30 con 0x7f54440ff1a0 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f542bfff640 1 -- 172.21.15.96:0/118108053 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5434009ce0 con 0x7f54440ff1a0 2024-04-22T16:00:53.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.642+0000 7f542bfff640 1 -- 172.21.15.96:0/118108053 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5434010460 con 0x7f54440ff1a0 2024-04-22T16:00:53.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.643+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54441a03b0 con 0x7f54440ff1a0 2024-04-22T16:00:53.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.644+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544404e040 con 0x7f54440ff1a0 2024-04-22T16:00:53.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.644+0000 7f542bfff640 1 -- 172.21.15.96:0/118108053 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5434005350 con 0x7f54440ff1a0 2024-04-22T16:00:53.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.644+0000 7f542bfff640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f542c06ecf0 0x7f542c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:53.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.644+0000 7f542bfff640 1 -- 172.21.15.96:0/118108053 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f543408ea20 con 0x7f54440ff1a0 2024-04-22T16:00:53.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.644+0000 7f54427fc640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f542c06ecf0 0x7f542c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:53.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.645+0000 7f54427fc640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f542c06ecf0 0x7f542c0711b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5438006010 tx=0x7f543802cb40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:53.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.649+0000 7f542bfff640 1 -- 172.21.15.96:0/118108053 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5434057780 con 0x7f54440ff1a0 2024-04-22T16:00:53.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.830+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f544410c1e0 con 0x7f54440ff1a0 2024-04-22T16:00:53.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.831+0000 7f542bfff640 1 -- 172.21.15.96:0/118108053 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16771 (secure 0 0 0) 0x7f543405ae40 con 0x7f54440ff1a0 2024-04-22T16:00:53.834 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T16:00:53.834 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":84,"fsid":"d5e86090-00b9-11ef-bc93-c7b262605968","created":"2024-04-22T15:08:23.492994+0000","modified":"2024-04-22T15:43:33.206096+0000","last_up_change":"2024-04-22T15:27:37.407115+0000","last_in_change":"2024-04-22T15:13:56.089739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:13:01.924433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T15:15:32.350570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T15:15:32.982528+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2f1b7772-06b4-467f-b14c-698a0de7357f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6803","nonce":312948544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6805","nonce":312948544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6809","nonce":312948544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":312948544},{"type":"v1","addr":"172.21.15.96:6807","nonce":312948544}]},"public_addr":"172.21.15.96:6803/312948544","cluster_addr":"172.21.15.96:6805/312948544","heartbeat_back_addr":"172.21.15.96:6809/312948544","heartbeat_front_addr":"172.21.15.96:6807/312948544","state":["exists","up"]},{"osd":1,"uuid":"59260a36-a2ed-42f9-8f3a-209aae4c4238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6811","nonce":3607865374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6813","nonce":3607865374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6817","nonce":3607865374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3607865374},{"type":"v1","addr":"172.21.15.96:6815","nonce":3607865374}]},"public_addr":"172.21.15.96:6811/3607865374","cluster_addr":"172.21.15.96:6813/3607865374","heartbeat_back_addr":"172.21.15.96:6817/3607865374","heartbeat_front_addr":"172.21.15.96:6815/3607865374","state":["exists","up"]},{"osd":2,"uuid":"897e21a9-3009-41c6-acf1-e44414fc69c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6819","nonce":2071585004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6821","nonce":2071585004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6825","nonce":2071585004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2071585004},{"type":"v1","addr":"172.21.15.96:6823","nonce":2071585004}]},"public_addr":"172.21.15.96:6819/2071585004","cluster_addr":"172.21.15.96:6821/2071585004","heartbeat_back_addr":"172.21.15.96:6825/2071585004","heartbeat_front_addr":"172.21.15.96:6823/2071585004","state":["exists","up"]},{"osd":3,"uuid":"201c3153-a284-471e-894d-2273001cc29e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6801","nonce":2743487260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6803","nonce":2743487260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6807","nonce":2743487260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":2743487260},{"type":"v1","addr":"172.21.15.196:6805","nonce":2743487260}]},"public_addr":"172.21.15.196:6801/2743487260","cluster_addr":"172.21.15.196:6803/2743487260","heartbeat_back_addr":"172.21.15.196:6807/2743487260","heartbeat_front_addr":"172.21.15.196:6805/2743487260","state":["exists","up"]},{"osd":4,"uuid":"780ed007-14ef-4c92-a5e9-6c57afcad448","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6809","nonce":2228895659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6811","nonce":2228895659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6815","nonce":2228895659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2228895659},{"type":"v1","addr":"172.21.15.196:6813","nonce":2228895659}]},"public_addr":"172.21.15.196:6809/2228895659","cluster_addr":"172.21.15.196:6811/2228895659","heartbeat_back_addr":"172.21.15.196:6815/2228895659","heartbeat_front_addr":"172.21.15.196:6813/2228895659","state":["exists","up"]},{"osd":5,"uuid":"16384d8d-bef4-4f65-9079-e2b4518241da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6817","nonce":288820489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6819","nonce":288820489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6823","nonce":288820489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":288820489},{"type":"v1","addr":"172.21.15.196:6821","nonce":288820489}]},"public_addr":"172.21.15.196:6817/288820489","cluster_addr":"172.21.15.196:6819/288820489","heartbeat_back_addr":"172.21.15.196:6823/288820489","heartbeat_front_addr":"172.21.15.196:6821/288820489","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:00.654872+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:28.691804+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:12:57.737056+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:21.679902+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:13:47.489372+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:14:13.600900+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6829/3534778299":"2024-04-23T15:27:39.581162+0000","172.21.15.96:6828/3534778299":"2024-04-23T15:27:39.581162+0000","172.21.15.196:0/465874235":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/1920801848":"2024-04-23T15:17:52.351792+0000","172.21.15.96:6801/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:6800/976363826":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/838799683":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6801/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:6800/2441203700":"2024-04-23T15:17:52.351792+0000","172.21.15.196:6825/3248990491":"2024-04-23T15:27:39.581179+0000","172.21.15.96:6800/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1319578885":"2024-04-23T15:09:23.920294+0000","172.21.15.196:0/427333217":"2024-04-23T15:19:21.709732+0000","172.21.15.196:6828/2093434004":"2024-04-23T15:19:21.709732+0000","172.21.15.96:6800/3275369520":"2024-04-23T15:09:23.920294+0000","172.21.15.96:0/328396090":"2024-04-23T15:23:08.192054+0000","172.21.15.96:0/168995622":"2024-04-23T15:17:52.351792+0000","172.21.15.196:0/4000780168":"2024-04-23T15:19:21.709732+0000","172.21.15.196:6824/3248990491":"2024-04-23T15:27:39.581179+0000","172.21.15.196:0/4127822214":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/340578852":"2024-04-23T15:17:52.351792+0000","172.21.15.96:0/1237353220":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3486391579":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/4224647460":"2024-04-23T15:08:51.463883+0000","172.21.15.96:6801/2684869170":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/3667378212":"2024-04-23T15:08:51.463883+0000","172.21.15.96:0/1860450164":"2024-04-23T15:17:52.351792+0000","172.21.15.196:0/1567934302":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/3321679733":"2024-04-23T15:09:23.920294+0000","172.21.15.196:0/391452056":"2024-04-23T15:19:21.709732+0000","172.21.15.196:6829/2093434004":"2024-04-23T15:19:21.709732+0000","172.21.15.96:0/3921802975":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/2918293154":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6826/887933858":"2024-04-23T15:15:49.819302+0000","172.21.15.96:6827/887933858":"2024-04-23T15:15:49.819302+0000","172.21.15.96:0/3360662441":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6801/2441203700":"2024-04-23T15:17:52.351792+0000","172.21.15.96:0/1794679103":"2024-04-23T15:10:37.605301+0000","172.21.15.96:0/3926078840":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6800/3666812526":"2024-04-23T15:23:08.192054+0000","172.21.15.96:6801/3666812526":"2024-04-23T15:23:08.192054+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T16:00:53.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f542c06ecf0 msgr2=0x7f542c0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f542c06ecf0 0x7f542c0711b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5438006010 tx=0x7f543802cb40 comp rx=0 tx=0).stop 2024-04-22T16:00:53.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ff1a0 msgr2=0x7f5444196b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ff1a0 0x7f5444196b20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f543400d720 tx=0x7f543400dbe0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 shutdown_connections 2024-04-22T16:00:53.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f542c06ecf0 0x7f542c0711b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5444197060 0x7f54441974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 --2- 172.21.15.96:0/118108053 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54440ff1a0 0x7f5444196b20 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 >> 172.21.15.96:0/118108053 conn(0x7f54440faa00 msgr2=0x7f54440fc000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:53.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 shutdown_connections 2024-04-22T16:00:53.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.833+0000 7f544967d640 1 -- 172.21.15.96:0/118108053 wait complete. 2024-04-22T16:00:53.847 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:13:01.924433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T15:15:32.350570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T15:15:32.982528+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T16:00:53.848 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T16:00:53.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.944+0000 7f50775c3640 1 -- 172.21.15.96:0/2926879919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50700675c0 msgr2=0x7f50700679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.944+0000 7f50775c3640 1 --2- 172.21.15.96:0/2926879919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50700675c0 0x7f50700679a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f506c009c20 tx=0x7f506c02c1b0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.944+0000 7f50775c3640 1 -- 172.21.15.96:0/2926879919 shutdown_connections 2024-04-22T16:00:53.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.944+0000 7f50775c3640 1 --2- 172.21.15.96:0/2926879919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5070067f70 0x7f5070104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.944+0000 7f50775c3640 1 --2- 172.21.15.96:0/2926879919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50700675c0 0x7f50700679a0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.944+0000 7f50775c3640 1 -- 172.21.15.96:0/2926879919 >> 172.21.15.96:0/2926879919 conn(0x7f5070074a50 msgr2=0x7f5070074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f50775c3640 1 -- 172.21.15.96:0/2926879919 shutdown_connections 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f50775c3640 1 -- 172.21.15.96:0/2926879919 wait complete. 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f50775c3640 1 Processor -- start 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f50775c3640 1 -- start start 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f50775c3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50700675c0 0x7f5070195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f50775c3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50701962d0 0x7f507019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f50775c3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50701968d0 con 0x7f50701962d0 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f50775c3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5070196a40 con 0x7f50700675c0 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f5074b5e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50701962d0 0x7f507019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f5074b5e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50701962d0 0x7f507019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45048/0 (socket says 172.21.15.96:45048) 2024-04-22T16:00:53.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.945+0000 7f5074b5e640 1 -- 172.21.15.96:0/380802922 learned_addr learned my addr 172.21.15.96:0/380802922 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f507535f640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50700675c0 0x7f5070195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f5074b5e640 1 -- 172.21.15.96:0/380802922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50700675c0 msgr2=0x7f5070195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f5074b5e640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50700675c0 0x7f5070195d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f5074b5e640 1 -- 172.21.15.96:0/380802922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c009900 con 0x7f50701962d0 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f507535f640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50700675c0 0x7f5070195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f5074b5e640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50701962d0 0x7f507019a730 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f506000c920 tx=0x7f506000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f50667fc640 1 -- 172.21.15.96:0/380802922 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5060007b80 con 0x7f50701962d0 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507019ad30 con 0x7f50701962d0 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f50667fc640 1 -- 172.21.15.96:0/380802922 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50600054e0 con 0x7f50701962d0 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f50667fc640 1 -- 172.21.15.96:0/380802922 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f506000f450 con 0x7f50701962d0 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.946+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507019b290 con 0x7f50701962d0 2024-04-22T16:00:53.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.947+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507019b570 con 0x7f50701962d0 2024-04-22T16:00:53.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.947+0000 7f50667fc640 1 -- 172.21.15.96:0/380802922 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f50600046c0 con 0x7f50701962d0 2024-04-22T16:00:53.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.948+0000 7f50667fc640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f505c06ec30 0x7f505c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:53.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.948+0000 7f50667fc640 1 -- 172.21.15.96:0/380802922 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f506008ed10 con 0x7f50701962d0 2024-04-22T16:00:53.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.948+0000 7f507535f640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f505c06ec30 0x7f505c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:53.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.948+0000 7f507535f640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f505c06ec30 0x7f505c0710f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f506c005e30 tx=0x7f506c005da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:53.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:53.953+0000 7f50667fc640 1 -- 172.21.15.96:0/380802922 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5060057a70 con 0x7f50701962d0 2024-04-22T16:00:54.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.130+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f50701970e0 con 0x7f50701962d0 2024-04-22T16:00:54.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.130+0000 7f50667fc640 1 -- 172.21.15.96:0/380802922 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f506001d020 con 0x7f50701962d0 2024-04-22T16:00:54.133 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-04-22T16:00:54.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.132+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f505c06ec30 msgr2=0x7f505c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.132+0000 7f50775c3640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f505c06ec30 0x7f505c0710f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f506c005e30 tx=0x7f506c005da0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.132+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50701962d0 msgr2=0x7f507019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.132+0000 7f50775c3640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50701962d0 0x7f507019a730 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f506000c920 tx=0x7f506000cde0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.132+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 shutdown_connections 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.132+0000 7f50775c3640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f505c06ec30 0x7f505c0710f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.132+0000 7f50775c3640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50701962d0 0x7f507019a730 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.132+0000 7f50775c3640 1 --2- 172.21.15.96:0/380802922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50700675c0 0x7f5070195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.132+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 >> 172.21.15.96:0/380802922 conn(0x7f5070074a50 msgr2=0x7f50700fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.133+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 shutdown_connections 2024-04-22T16:00:54.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.133+0000 7f50775c3640 1 -- 172.21.15.96:0/380802922 wait complete. 2024-04-22T16:00:54.145 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1261368996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:00:54.145 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3866477171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:00:54.146 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:54.146 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/3246803530' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T16:00:54.146 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:53 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/118108053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:00:54.147 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T16:00:54.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.245+0000 7f62c37e9640 1 -- 172.21.15.96:0/2273431155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 msgr2=0x7f62bc0fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.245+0000 7f62c37e9640 1 --2- 172.21.15.96:0/2273431155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 0x7f62bc0fed70 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009880 tx=0x7f62b802bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 -- 172.21.15.96:0/2273431155 shutdown_connections 2024-04-22T16:00:54.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 --2- 172.21.15.96:0/2273431155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62bc0ff340 0x7f62bc104c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 --2- 172.21.15.96:0/2273431155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 0x7f62bc0fed70 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 -- 172.21.15.96:0/2273431155 >> 172.21.15.96:0/2273431155 conn(0x7f62bc075250 msgr2=0x7f62bc075660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:54.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 -- 172.21.15.96:0/2273431155 shutdown_connections 2024-04-22T16:00:54.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 -- 172.21.15.96:0/2273431155 wait complete. 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 Processor -- start 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 -- start start 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 0x7f62bc0ff900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62bc0ffe40 0x7f62bc1002a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc1a1f80 con 0x7f62bc0fe990 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c37e9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc1a20c0 con 0x7f62bc0ffe40 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c1585640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 0x7f62bc0ff900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c1585640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 0x7f62bc0ff900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45062/0 (socket says 172.21.15.96:45062) 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.246+0000 7f62c1585640 1 -- 172.21.15.96:0/156260081 learned_addr learned my addr 172.21.15.96:0/156260081 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62c0d84640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62bc0ffe40 0x7f62bc1002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62c1585640 1 -- 172.21.15.96:0/156260081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62bc0ffe40 msgr2=0x7f62bc1002a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62c1585640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62bc0ffe40 0x7f62bc1002a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62c1585640 1 -- 172.21.15.96:0/156260081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b8009560 con 0x7f62bc0fe990 2024-04-22T16:00:54.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62c1585640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 0x7f62bc0ff900 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009430 tx=0x7f62b8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:54.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62b27fc640 1 -- 172.21.15.96:0/156260081 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b80035f0 con 0x7f62bc0fe990 2024-04-22T16:00:54.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62bc1a2200 con 0x7f62bc0fe990 2024-04-22T16:00:54.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62b27fc640 1 -- 172.21.15.96:0/156260081 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62b8003c10 con 0x7f62bc0fe990 2024-04-22T16:00:54.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62b27fc640 1 -- 172.21.15.96:0/156260081 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b8005670 con 0x7f62bc0fe990 2024-04-22T16:00:54.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.247+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62bc1a26b0 con 0x7f62bc0fe990 2024-04-22T16:00:54.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.248+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62bc04e040 con 0x7f62bc0fe990 2024-04-22T16:00:54.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.249+0000 7f62b27fc640 1 -- 172.21.15.96:0/156260081 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f62b8004dd0 con 0x7f62bc0fe990 2024-04-22T16:00:54.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.249+0000 7f62b27fc640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62a806ebe0 0x7f62a80710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:54.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.249+0000 7f62b27fc640 1 -- 172.21.15.96:0/156260081 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f62b80b12a0 con 0x7f62bc0fe990 2024-04-22T16:00:54.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.249+0000 7f62c0d84640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62a806ebe0 0x7f62a80710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:54.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.249+0000 7f62c0d84640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62a806ebe0 0x7f62a80710a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f62ac005fd0 tx=0x7f62ac005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:54.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.254+0000 7f62b27fc640 1 -- 172.21.15.96:0/156260081 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f62b807a000 con 0x7f62bc0fe990 2024-04-22T16:00:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1261368996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:00:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3866477171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:00:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/3246803530' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T16:00:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:53 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/118108053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:00:54.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.429+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f62bc1085c0 con 0x7f62bc0fe990 2024-04-22T16:00:54.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.430+0000 7f62b27fc640 1 -- 172.21.15.96:0/156260081 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f62b807d6c0 con 0x7f62bc0fe990 2024-04-22T16:00:54.433 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-04-22T16:00:54.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.431+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62a806ebe0 msgr2=0x7f62a80710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.431+0000 7f62c37e9640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62a806ebe0 0x7f62a80710a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f62ac005fd0 tx=0x7f62ac005850 comp rx=0 tx=0).stop 2024-04-22T16:00:54.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.431+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 msgr2=0x7f62bc0ff900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.431+0000 7f62c37e9640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 0x7f62bc0ff900 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009430 tx=0x7f62b8004040 comp rx=0 tx=0).stop 2024-04-22T16:00:54.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.432+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 shutdown_connections 2024-04-22T16:00:54.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.432+0000 7f62c37e9640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f62a806ebe0 0x7f62a80710a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.432+0000 7f62c37e9640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62bc0ffe40 0x7f62bc1002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.432+0000 7f62c37e9640 1 --2- 172.21.15.96:0/156260081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc0fe990 0x7f62bc0ff900 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.432+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 >> 172.21.15.96:0/156260081 conn(0x7f62bc075250 msgr2=0x7f62bc0fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:54.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.432+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 shutdown_connections 2024-04-22T16:00:54.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.432+0000 7f62c37e9640 1 -- 172.21.15.96:0/156260081 wait complete. 2024-04-22T16:00:54.446 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T16:00:54.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.543+0000 7f4e87dae640 1 -- 172.21.15.96:0/2092762184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067350 msgr2=0x7f4e80067730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.543+0000 7f4e87dae640 1 --2- 172.21.15.96:0/2092762184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067350 0x7f4e80067730 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c009880 tx=0x7f4e7c02bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 -- 172.21.15.96:0/2092762184 shutdown_connections 2024-04-22T16:00:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 --2- 172.21.15.96:0/2092762184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e80067c70 0x7f4e8010c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 --2- 172.21.15.96:0/2092762184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067350 0x7f4e80067730 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 -- 172.21.15.96:0/2092762184 >> 172.21.15.96:0/2092762184 conn(0x7f4e80074a50 msgr2=0x7f4e80074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 -- 172.21.15.96:0/2092762184 shutdown_connections 2024-04-22T16:00:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 -- 172.21.15.96:0/2092762184 wait complete. 2024-04-22T16:00:54.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 Processor -- start 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 -- start start 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e80067350 0x7f4e80197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067c70 0x7f4e801984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e80198b80 con 0x7f4e80067c70 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e87dae640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8019c950 con 0x7f4e80067350 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e85349640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067c70 0x7f4e801984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e85349640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067c70 0x7f4e801984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45072/0 (socket says 172.21.15.96:45072) 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.544+0000 7f4e85349640 1 -- 172.21.15.96:0/2259336289 learned_addr learned my addr 172.21.15.96:0/2259336289 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e85b4a640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e80067350 0x7f4e80197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:54.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e85349640 1 -- 172.21.15.96:0/2259336289 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e80067350 msgr2=0x7f4e80197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e85349640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e80067350 0x7f4e80197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e85349640 1 -- 172.21.15.96:0/2259336289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c009560 con 0x7f4e80067c70 2024-04-22T16:00:54.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e85b4a640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e80067350 0x7f4e80197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:54.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e85349640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067c70 0x7f4e801984f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c8f0 tx=0x7f4e7000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:54.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e76ffd640 1 -- 172.21.15.96:0/2259336289 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e70007b80 con 0x7f4e80067c70 2024-04-22T16:00:54.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e76ffd640 1 -- 172.21.15.96:0/2259336289 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e700054e0 con 0x7f4e80067c70 2024-04-22T16:00:54.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e76ffd640 1 -- 172.21.15.96:0/2259336289 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e7000f450 con 0x7f4e80067c70 2024-04-22T16:00:54.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8019cc30 con 0x7f4e80067c70 2024-04-22T16:00:54.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.545+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8019d190 con 0x7f4e80067c70 2024-04-22T16:00:54.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.546+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8004e040 con 0x7f4e80067c70 2024-04-22T16:00:54.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.546+0000 7f4e76ffd640 1 -- 172.21.15.96:0/2259336289 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4e70004f60 con 0x7f4e80067c70 2024-04-22T16:00:54.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.547+0000 7f4e76ffd640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e6c06ec30 0x7f4e6c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:54.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.547+0000 7f4e76ffd640 1 -- 172.21.15.96:0/2259336289 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f4e7008ed10 con 0x7f4e80067c70 2024-04-22T16:00:54.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.547+0000 7f4e85b4a640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e6c06ec30 0x7f4e6c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:54.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.547+0000 7f4e85b4a640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e6c06ec30 0x7f4e6c0710f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c005ec0 tx=0x7f4e7c005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:54.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.552+0000 7f4e76ffd640 1 -- 172.21.15.96:0/2259336289 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4e70057a70 con 0x7f4e80067c70 2024-04-22T16:00:54.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.731+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f4e8010f4d0 con 0x7f4e80067c70 2024-04-22T16:00:54.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.732+0000 7f4e76ffd640 1 -- 172.21.15.96:0/2259336289 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f4e7001d020 con 0x7f4e80067c70 2024-04-22T16:00:54.735 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-04-22T16:00:54.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e6c06ec30 msgr2=0x7f4e6c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e6c06ec30 0x7f4e6c0710f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c005ec0 tx=0x7f4e7c005fd0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067c70 msgr2=0x7f4e801984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067c70 0x7f4e801984f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c8f0 tx=0x7f4e7000cdb0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 shutdown_connections 2024-04-22T16:00:54.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f4e6c06ec30 0x7f4e6c0710f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e80067c70 0x7f4e801984f0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 --2- 172.21.15.96:0/2259336289 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e80067350 0x7f4e80197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 >> 172.21.15.96:0/2259336289 conn(0x7f4e80074a50 msgr2=0x7f4e800fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:54.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 shutdown_connections 2024-04-22T16:00:54.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.734+0000 7f4e87dae640 1 -- 172.21.15.96:0/2259336289 wait complete. 2024-04-22T16:00:54.748 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T16:00:54.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.844+0000 7f45ece2f640 1 -- 172.21.15.96:0/2360426908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e8100c60 msgr2=0x7f45e8104ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.844+0000 7f45ece2f640 1 --2- 172.21.15.96:0/2360426908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e8100c60 0x7f45e8104ca0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009880 tx=0x7f45d002bce0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.844+0000 7f45ece2f640 1 -- 172.21.15.96:0/2360426908 shutdown_connections 2024-04-22T16:00:54.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.844+0000 7f45ece2f640 1 --2- 172.21.15.96:0/2360426908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e8100c60 0x7f45e8104ca0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.844+0000 7f45ece2f640 1 --2- 172.21.15.96:0/2360426908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e81002b0 0x7f45e8100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.844+0000 7f45ece2f640 1 -- 172.21.15.96:0/2360426908 >> 172.21.15.96:0/2360426908 conn(0x7f45e80fbaf0 msgr2=0x7f45e80fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.844+0000 7f45ece2f640 1 -- 172.21.15.96:0/2360426908 shutdown_connections 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.844+0000 7f45ece2f640 1 -- 172.21.15.96:0/2360426908 wait complete. 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.845+0000 7f45ece2f640 1 Processor -- start 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.845+0000 7f45ece2f640 1 -- start start 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.845+0000 7f45ece2f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e81002b0 0x7f45e8195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.845+0000 7f45ece2f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e81962c0 0x7f45e819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.845+0000 7f45ece2f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e81968c0 con 0x7f45e81962c0 2024-04-22T16:00:54.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.845+0000 7f45ece2f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e8196a30 con 0x7f45e81002b0 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45e5d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e81962c0 0x7f45e819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45e5d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e81962c0 0x7f45e819a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45082/0 (socket says 172.21.15.96:45082) 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45e5d9b640 1 -- 172.21.15.96:0/1821221416 learned_addr learned my addr 172.21.15.96:0/1821221416 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45e5d9b640 1 -- 172.21.15.96:0/1821221416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e81002b0 msgr2=0x7f45e8195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45e659c640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e81002b0 0x7f45e8195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45e5d9b640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e81002b0 0x7f45e8195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45e5d9b640 1 -- 172.21.15.96:0/1821221416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d0009560 con 0x7f45e81962c0 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45e659c640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e81002b0 0x7f45e8195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45e5d9b640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e81962c0 0x7f45e819a720 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f45d0036040 tx=0x7f45d0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45df7fe640 1 -- 172.21.15.96:0/1821221416 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45d00037c0 con 0x7f45e81962c0 2024-04-22T16:00:54.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45df7fe640 1 -- 172.21.15.96:0/1821221416 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45d0003de0 con 0x7f45e81962c0 2024-04-22T16:00:54.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e819acc0 con 0x7f45e81962c0 2024-04-22T16:00:54.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45df7fe640 1 -- 172.21.15.96:0/1821221416 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45d0034440 con 0x7f45e81962c0 2024-04-22T16:00:54.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.846+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e819b220 con 0x7f45e81962c0 2024-04-22T16:00:54.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.847+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e804e040 con 0x7f45e81962c0 2024-04-22T16:00:54.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.848+0000 7f45df7fe640 1 -- 172.21.15.96:0/1821221416 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f45d0004fb0 con 0x7f45e81962c0 2024-04-22T16:00:54.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.848+0000 7f45df7fe640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f45d406ecf0 0x7f45d40711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:00:54.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.848+0000 7f45df7fe640 1 -- 172.21.15.96:0/1821221416 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f45d00b13a0 con 0x7f45e81962c0 2024-04-22T16:00:54.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.848+0000 7f45e659c640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f45d406ecf0 0x7f45d40711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:00:54.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.848+0000 7f45e659c640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f45d406ecf0 0x7f45d40711b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f45e0006fd0 tx=0x7f45e0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:00:54.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:54.853+0000 7f45df7fe640 1 -- 172.21.15.96:0/1821221416 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f45d007a100 con 0x7f45e81962c0 2024-04-22T16:00:55.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.047+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f45e81970d0 con 0x7f45e81962c0 2024-04-22T16:00:55.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.047+0000 7f45df7fe640 1 -- 172.21.15.96:0/1821221416 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 41 entries v84) v1 ==== 81+0+2431 (secure 0 0 0) 0x7f45d007d7c0 con 0x7f45e81962c0 2024-04-22T16:00:55.050 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6829/3534778299 2024-04-23T15:27:39.581162+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6828/3534778299 2024-04-23T15:27:39.581162+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/465874235 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1920801848 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/976363826 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/976363826 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/838799683 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/3275369520 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/2441203700 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:6825/3248990491 2024-04-23T15:27:39.581179+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/2684869170 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1319578885 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:55.051 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/427333217 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:6828/2093434004 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/3275369520 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/328396090 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/168995622 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/4000780168 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:6824/3248990491 2024-04-23T15:27:39.581179+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/4127822214 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/340578852 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1237353220 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3486391579 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/4224647460 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/2684869170 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:55.052 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3667378212 2024-04-23T15:08:51.463883+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1860450164 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/1567934302 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3321679733 2024-04-23T15:09:23.920294+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:0/391452056 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.196:6829/2093434004 2024-04-23T15:19:21.709732+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3921802975 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/2918293154 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6826/887933858 2024-04-23T15:15:49.819302+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6827/887933858 2024-04-23T15:15:49.819302+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3360662441 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/2441203700 2024-04-23T15:17:52.351792+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/1794679103 2024-04-23T15:10:37.605301+0000 2024-04-22T16:00:55.053 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:0/3926078840 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:55.054 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6800/3666812526 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:55.054 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96:6801/3666812526 2024-04-23T15:23:08.192054+0000 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:listed 41 entries 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f45d406ecf0 msgr2=0x7f45d40711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f45d406ecf0 0x7f45d40711b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f45e0006fd0 tx=0x7f45e0008040 comp rx=0 tx=0).stop 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e81962c0 msgr2=0x7f45e819a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e81962c0 0x7f45e819a720 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f45d0036040 tx=0x7f45d0004270 comp rx=0 tx=0).stop 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 shutdown_connections 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:6800/4190215134,v1:172.21.15.96:6801/4190215134] conn(0x7f45d406ecf0 0x7f45d40711b0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e81962c0 0x7f45e819a720 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 --2- 172.21.15.96:0/1821221416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e81002b0 0x7f45e8195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 >> 172.21.15.96:0/1821221416 conn(0x7f45e80fbaf0 msgr2=0x7f45e80fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:00:55.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 shutdown_connections 2024-04-22T16:00:55.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T16:00:55.049+0000 7f45ece2f640 1 -- 172.21.15.96:0/1821221416 wait complete. 2024-04-22T16:00:55.062 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:54 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/380802922' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:00:55.062 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:54 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/156260081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:00:55.062 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:54 smithi096 ceph-mon[112443]: pgmap v1175: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:55.062 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:54 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/2259336289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:00:55.063 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-04-22T16:00:55.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T16:00:55.063 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-22T16:00:55.087 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T16:00:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:54 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/380802922' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:00:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:54 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/156260081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:00:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:54 smithi196 ceph-mon[81815]: pgmap v1175: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:00:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:54 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/2259336289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:00:56.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:55 smithi096 ceph-mon[112443]: from='client.? 172.21.15.96:0/1821221416' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T16:00:56.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:55 smithi196 ceph-mon[81815]: from='client.? 172.21.15.96:0/1821221416' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T16:00:57.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:56 smithi096 ceph-mon[112443]: pgmap v1176: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:57.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:56 smithi196 ceph-mon[81815]: pgmap v1176: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:00:59.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:00:58 smithi096 ceph-mon[112443]: pgmap v1177: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:00:59.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:00:58 smithi196 ceph-mon[81815]: pgmap v1177: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:01.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:00 smithi096 ceph-mon[112443]: pgmap v1178: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:01.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:00 smithi196 ceph-mon[81815]: pgmap v1178: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:03.185 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:02 smithi096 ceph-mon[112443]: pgmap v1179: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:02 smithi196 ceph-mon[81815]: pgmap v1179: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:05.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:04 smithi096 ceph-mon[112443]: pgmap v1180: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:05.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:04 smithi196 ceph-mon[81815]: pgmap v1180: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:07.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:06 smithi096 ceph-mon[112443]: pgmap v1181: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:07.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:06 smithi196 ceph-mon[81815]: pgmap v1181: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:08.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:07 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:01:08.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:07 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:01:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:07 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:01:08.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:07 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:01:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:07 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:01:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:07 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:01:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:07 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:01:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:07 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:01:09.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:09.010 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:08 smithi196 ceph-mon[81815]: pgmap v1182: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:09.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:09.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:08 smithi096 ceph-mon[112443]: pgmap v1182: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:11.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:10 smithi096 ceph-mon[112443]: pgmap v1183: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:11.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:10 smithi196 ceph-mon[81815]: pgmap v1183: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:13.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:12 smithi096 ceph-mon[112443]: pgmap v1184: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:13.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:12 smithi196 ceph-mon[81815]: pgmap v1184: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:15.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:14 smithi096 ceph-mon[112443]: pgmap v1185: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:15.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:14 smithi196 ceph-mon[81815]: pgmap v1185: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:17.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:16 smithi096 ceph-mon[112443]: pgmap v1186: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:17.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:16 smithi196 ceph-mon[81815]: pgmap v1186: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:19.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:18 smithi196 ceph-mon[81815]: pgmap v1187: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:19.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:18 smithi096 ceph-mon[112443]: pgmap v1187: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:21.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:20 smithi096 ceph-mon[112443]: pgmap v1188: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:21.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:20 smithi196 ceph-mon[81815]: pgmap v1188: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:23.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:22 smithi096 ceph-mon[112443]: pgmap v1189: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:23.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:22 smithi196 ceph-mon[81815]: pgmap v1189: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:24.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:25.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:24 smithi096 ceph-mon[112443]: pgmap v1190: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:25.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:24 smithi196 ceph-mon[81815]: pgmap v1190: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:27.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:26 smithi096 ceph-mon[112443]: pgmap v1191: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:26 smithi196 ceph-mon[81815]: pgmap v1191: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:29.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:28 smithi096 ceph-mon[112443]: pgmap v1192: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:28 smithi196 ceph-mon[81815]: pgmap v1192: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:31.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:30 smithi096 ceph-mon[112443]: pgmap v1193: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:30 smithi196 ceph-mon[81815]: pgmap v1193: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:33.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:32 smithi096 ceph-mon[112443]: pgmap v1194: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:33.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:32 smithi196 ceph-mon[81815]: pgmap v1194: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:35.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:34 smithi096 ceph-mon[112443]: pgmap v1195: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:35.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:34 smithi196 ceph-mon[81815]: pgmap v1195: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:37.154 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:36 smithi096 ceph-mon[112443]: pgmap v1196: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:36 smithi196 ceph-mon[81815]: pgmap v1196: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:38.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:39.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:39 smithi096 ceph-mon[112443]: pgmap v1197: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:39 smithi196 ceph-mon[81815]: pgmap v1197: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:41.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:40 smithi096 ceph-mon[112443]: pgmap v1198: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:41.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:40 smithi196 ceph-mon[81815]: pgmap v1198: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:43.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:42 smithi096 ceph-mon[112443]: pgmap v1199: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:43.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:42 smithi196 ceph-mon[81815]: pgmap v1199: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:45.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:44 smithi096 ceph-mon[112443]: pgmap v1200: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:45.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:44 smithi196 ceph-mon[81815]: pgmap v1200: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:47.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:46 smithi096 ceph-mon[112443]: pgmap v1201: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:46 smithi196 ceph-mon[81815]: pgmap v1201: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:48.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:48 smithi096 ceph-mon[112443]: pgmap v1202: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:49.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:48 smithi196 ceph-mon[81815]: pgmap v1202: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:51.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:50 smithi096 ceph-mon[112443]: pgmap v1203: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:50 smithi196 ceph-mon[81815]: pgmap v1203: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:01:53.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:52 smithi096 ceph-mon[112443]: pgmap v1204: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:52 smithi196 ceph-mon[81815]: pgmap v1204: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:54.108 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:54.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:55.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:54 smithi096 ceph-mon[112443]: pgmap v1205: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:54 smithi196 ceph-mon[81815]: pgmap v1205: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:01:57.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:56 smithi096 ceph-mon[112443]: pgmap v1206: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:57.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:56 smithi196 ceph-mon[81815]: pgmap v1206: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:01:59.165 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:01:58 smithi196 ceph-mon[81815]: pgmap v1207: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:01:59.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:01:58 smithi096 ceph-mon[112443]: pgmap v1207: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:01.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:00 smithi096 ceph-mon[112443]: pgmap v1208: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:01.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:00 smithi196 ceph-mon[81815]: pgmap v1208: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:03.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:02 smithi096 ceph-mon[112443]: pgmap v1209: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:02 smithi196 ceph-mon[81815]: pgmap v1209: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:05.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:04 smithi096 ceph-mon[112443]: pgmap v1210: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:05.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:04 smithi196 ceph-mon[81815]: pgmap v1210: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:07.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:06 smithi096 ceph-mon[112443]: pgmap v1211: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:07.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:06 smithi196 ceph-mon[81815]: pgmap v1211: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:08.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:07 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:02:08.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:07 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:02:09.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:02:09.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:02:09.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:02:09.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:09.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:08 smithi096 ceph-mon[112443]: pgmap v1212: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:02:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:02:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:02:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:09.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:08 smithi196 ceph-mon[81815]: pgmap v1212: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:11.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:10 smithi096 ceph-mon[112443]: pgmap v1213: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:11.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:10 smithi196 ceph-mon[81815]: pgmap v1213: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:13.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:12 smithi096 ceph-mon[112443]: pgmap v1214: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:13.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:12 smithi196 ceph-mon[81815]: pgmap v1214: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:15.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:14 smithi096 ceph-mon[112443]: pgmap v1215: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:15.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:14 smithi196 ceph-mon[81815]: pgmap v1215: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:17.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:16 smithi096 ceph-mon[112443]: pgmap v1216: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:17.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:16 smithi196 ceph-mon[81815]: pgmap v1216: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:19.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:18 smithi096 ceph-mon[112443]: pgmap v1217: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:19.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:18 smithi196 ceph-mon[81815]: pgmap v1217: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:21.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:21 smithi096 ceph-mon[112443]: pgmap v1218: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:21.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:21 smithi196 ceph-mon[81815]: pgmap v1218: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:23.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:22 smithi096 ceph-mon[112443]: pgmap v1219: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:23.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:22 smithi196 ceph-mon[81815]: pgmap v1219: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:24.099 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:25.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:24 smithi096 ceph-mon[112443]: pgmap v1220: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:25.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:24 smithi196 ceph-mon[81815]: pgmap v1220: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:27.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:26 smithi096 ceph-mon[112443]: pgmap v1221: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:26 smithi196 ceph-mon[81815]: pgmap v1221: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:29.156 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:28 smithi096 ceph-mon[112443]: pgmap v1222: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:28 smithi196 ceph-mon[81815]: pgmap v1222: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:31.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:30 smithi096 ceph-mon[112443]: pgmap v1223: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:30 smithi196 ceph-mon[81815]: pgmap v1223: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:33.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:32 smithi096 ceph-mon[112443]: pgmap v1224: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:33.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:32 smithi196 ceph-mon[81815]: pgmap v1224: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:35.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:34 smithi096 ceph-mon[112443]: pgmap v1225: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:35.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:34 smithi196 ceph-mon[81815]: pgmap v1225: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:37.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:36 smithi096 ceph-mon[112443]: pgmap v1226: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:37.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:36 smithi196 ceph-mon[81815]: pgmap v1226: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:38.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:39.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:39 smithi096 ceph-mon[112443]: pgmap v1227: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:39 smithi196 ceph-mon[81815]: pgmap v1227: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:41.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:40 smithi096 ceph-mon[112443]: pgmap v1228: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:41.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:40 smithi196 ceph-mon[81815]: pgmap v1228: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:43.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:42 smithi096 ceph-mon[112443]: pgmap v1229: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:43.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:42 smithi196 ceph-mon[81815]: pgmap v1229: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:45.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:44 smithi196 ceph-mon[81815]: pgmap v1230: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:45.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:44 smithi096 ceph-mon[112443]: pgmap v1230: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:47.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:46 smithi096 ceph-mon[112443]: pgmap v1231: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:46 smithi196 ceph-mon[81815]: pgmap v1231: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:49.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:48 smithi096 ceph-mon[112443]: pgmap v1232: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:49.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:48 smithi196 ceph-mon[81815]: pgmap v1232: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:51.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:50 smithi096 ceph-mon[112443]: pgmap v1233: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:51.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:50 smithi196 ceph-mon[81815]: pgmap v1233: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:02:53.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:52 smithi096 ceph-mon[112443]: pgmap v1234: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:53.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:52 smithi196 ceph-mon[81815]: pgmap v1234: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:54.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:54.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:55.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:54 smithi096 ceph-mon[112443]: pgmap v1235: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:54 smithi196 ceph-mon[81815]: pgmap v1235: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:02:57.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:56 smithi096 ceph-mon[112443]: pgmap v1236: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:57.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:56 smithi196 ceph-mon[81815]: pgmap v1236: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:02:59.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:02:58 smithi096 ceph-mon[112443]: pgmap v1237: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:02:59.307 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:02:58 smithi196 ceph-mon[81815]: pgmap v1237: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:01.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:00 smithi096 ceph-mon[112443]: pgmap v1238: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:01.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:00 smithi196 ceph-mon[81815]: pgmap v1238: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:03.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:02 smithi096 ceph-mon[112443]: pgmap v1239: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:03.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:02 smithi196 ceph-mon[81815]: pgmap v1239: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:05.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:04 smithi096 ceph-mon[112443]: pgmap v1240: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:05.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:04 smithi196 ceph-mon[81815]: pgmap v1240: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:07.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:06 smithi096 ceph-mon[112443]: pgmap v1241: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:07.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:06 smithi196 ceph-mon[81815]: pgmap v1241: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:08.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:08.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:09.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:03:09.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:09 smithi096 ceph-mon[112443]: pgmap v1242: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:09.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:03:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:03:09.736 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:03:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:03:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:09 smithi196 ceph-mon[81815]: pgmap v1242: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:03:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:03:09.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:03:11.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:10 smithi096 ceph-mon[112443]: pgmap v1243: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:11.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:10 smithi196 ceph-mon[81815]: pgmap v1243: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:13.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:12 smithi096 ceph-mon[112443]: pgmap v1244: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:13.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:12 smithi196 ceph-mon[81815]: pgmap v1244: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:14.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:14 smithi096 ceph-mon[112443]: pgmap v1245: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:15.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:14 smithi196 ceph-mon[81815]: pgmap v1245: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:17.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:16 smithi096 ceph-mon[112443]: pgmap v1246: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:17.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:16 smithi196 ceph-mon[81815]: pgmap v1246: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:19.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:18 smithi096 ceph-mon[112443]: pgmap v1247: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:19.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:18 smithi196 ceph-mon[81815]: pgmap v1247: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:21.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:20 smithi096 ceph-mon[112443]: pgmap v1248: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:21.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:20 smithi196 ceph-mon[81815]: pgmap v1248: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:23.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:22 smithi096 ceph-mon[112443]: pgmap v1249: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:23.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:22 smithi196 ceph-mon[81815]: pgmap v1249: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:24.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:25.127 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:24 smithi196 ceph-mon[81815]: pgmap v1250: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:25.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:24 smithi096 ceph-mon[112443]: pgmap v1250: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:27.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:26 smithi096 ceph-mon[112443]: pgmap v1251: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:27.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:26 smithi196 ceph-mon[81815]: pgmap v1251: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:29.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:28 smithi096 ceph-mon[112443]: pgmap v1252: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:29.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:28 smithi196 ceph-mon[81815]: pgmap v1252: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:31.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:30 smithi096 ceph-mon[112443]: pgmap v1253: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:31.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:30 smithi196 ceph-mon[81815]: pgmap v1253: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:33.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:32 smithi096 ceph-mon[112443]: pgmap v1254: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:33.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:32 smithi196 ceph-mon[81815]: pgmap v1254: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:35.237 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:34 smithi096 ceph-mon[112443]: pgmap v1255: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:35.239 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:34 smithi196 ceph-mon[81815]: pgmap v1255: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:37.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:36 smithi096 ceph-mon[112443]: pgmap v1256: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:37.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:36 smithi196 ceph-mon[81815]: pgmap v1256: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:38.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:38.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:39.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:39 smithi096 ceph-mon[112443]: pgmap v1257: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:39.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:39 smithi196 ceph-mon[81815]: pgmap v1257: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:41.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:40 smithi096 ceph-mon[112443]: pgmap v1258: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:41.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:40 smithi196 ceph-mon[81815]: pgmap v1258: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:43.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:42 smithi096 ceph-mon[112443]: pgmap v1259: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:43.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:42 smithi196 ceph-mon[81815]: pgmap v1259: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:45.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:44 smithi096 ceph-mon[112443]: pgmap v1260: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:45.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:44 smithi196 ceph-mon[81815]: pgmap v1260: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:47.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:46 smithi096 ceph-mon[112443]: pgmap v1261: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:47.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:46 smithi196 ceph-mon[81815]: pgmap v1261: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:49.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:48 smithi096 ceph-mon[112443]: pgmap v1262: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:49.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:48 smithi196 ceph-mon[81815]: pgmap v1262: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:51.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:50 smithi096 ceph-mon[112443]: pgmap v1263: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:51.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:50 smithi196 ceph-mon[81815]: pgmap v1263: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:03:53.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:52 smithi096 ceph-mon[112443]: pgmap v1264: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:53.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:52 smithi196 ceph-mon[81815]: pgmap v1264: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:54.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:54.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:55.134 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:54 smithi096 ceph-mon[112443]: pgmap v1265: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:54 smithi196 ceph-mon[81815]: pgmap v1265: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:03:57.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:56 smithi096 ceph-mon[112443]: pgmap v1266: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:57.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:56 smithi196 ceph-mon[81815]: pgmap v1266: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:03:59.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:03:58 smithi096 ceph-mon[112443]: pgmap v1267: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:03:59.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:03:58 smithi196 ceph-mon[81815]: pgmap v1267: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:01.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:00 smithi096 ceph-mon[112443]: pgmap v1268: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:01.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:00 smithi196 ceph-mon[81815]: pgmap v1268: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:03.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:02 smithi096 ceph-mon[112443]: pgmap v1269: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:03.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:02 smithi196 ceph-mon[81815]: pgmap v1269: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:05.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:04 smithi096 ceph-mon[112443]: pgmap v1270: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:05.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:04 smithi196 ceph-mon[81815]: pgmap v1270: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:07.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:06 smithi096 ceph-mon[112443]: pgmap v1271: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:07.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:06 smithi196 ceph-mon[81815]: pgmap v1271: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:08.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:08.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:09.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:09 smithi096 ceph-mon[112443]: pgmap v1272: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:09.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:09 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:04:09.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:09 smithi196 ceph-mon[81815]: pgmap v1272: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:09.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:09 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:04:10.633 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:10 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:04:10.633 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:10 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:04:10.633 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:10 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:04:10.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:10 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:04:10.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:10 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:04:10.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:10 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:04:11.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:11 smithi096 ceph-mon[112443]: pgmap v1273: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:11.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:11 smithi196 ceph-mon[81815]: pgmap v1273: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:13.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:12 smithi096 ceph-mon[112443]: pgmap v1274: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:13.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:12 smithi196 ceph-mon[81815]: pgmap v1274: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:15.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:14 smithi096 ceph-mon[112443]: pgmap v1275: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:15.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:14 smithi196 ceph-mon[81815]: pgmap v1275: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:17.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:16 smithi096 ceph-mon[112443]: pgmap v1276: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:17.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:16 smithi196 ceph-mon[81815]: pgmap v1276: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:19.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:18 smithi096 ceph-mon[112443]: pgmap v1277: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:19.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:18 smithi196 ceph-mon[81815]: pgmap v1277: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:21.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:20 smithi196 ceph-mon[81815]: pgmap v1278: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:21.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:20 smithi096 ceph-mon[112443]: pgmap v1278: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:23.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:22 smithi096 ceph-mon[112443]: pgmap v1279: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:23.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:22 smithi196 ceph-mon[81815]: pgmap v1279: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:24.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:24.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:25.236 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:24 smithi096 ceph-mon[112443]: pgmap v1280: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:25.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:24 smithi196 ceph-mon[81815]: pgmap v1280: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:27.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:26 smithi096 ceph-mon[112443]: pgmap v1281: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:27.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:26 smithi196 ceph-mon[81815]: pgmap v1281: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:29.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:28 smithi096 ceph-mon[112443]: pgmap v1282: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:29.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:28 smithi196 ceph-mon[81815]: pgmap v1282: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:31.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:30 smithi096 ceph-mon[112443]: pgmap v1283: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:31.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:30 smithi196 ceph-mon[81815]: pgmap v1283: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:33.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:32 smithi096 ceph-mon[112443]: pgmap v1284: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:33.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:32 smithi196 ceph-mon[81815]: pgmap v1284: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:35.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:34 smithi096 ceph-mon[112443]: pgmap v1285: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:35.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:34 smithi196 ceph-mon[81815]: pgmap v1285: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:37.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:36 smithi096 ceph-mon[112443]: pgmap v1286: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:37.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:36 smithi196 ceph-mon[81815]: pgmap v1286: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:38.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:38.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:39.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:39 smithi096 ceph-mon[112443]: pgmap v1287: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:39.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:39 smithi196 ceph-mon[81815]: pgmap v1287: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:40.985 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:40 smithi096 ceph-mon[112443]: pgmap v1288: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:41.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:40 smithi196 ceph-mon[81815]: pgmap v1288: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:43.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:42 smithi096 ceph-mon[112443]: pgmap v1289: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:43.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:42 smithi196 ceph-mon[81815]: pgmap v1289: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:45.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:44 smithi096 ceph-mon[112443]: pgmap v1290: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:45.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:44 smithi196 ceph-mon[81815]: pgmap v1290: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:47.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:46 smithi096 ceph-mon[112443]: pgmap v1291: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:47.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:46 smithi196 ceph-mon[81815]: pgmap v1291: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:49.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:48 smithi096 ceph-mon[112443]: pgmap v1292: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:49.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:48 smithi196 ceph-mon[81815]: pgmap v1292: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:51.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:50 smithi096 ceph-mon[112443]: pgmap v1293: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:51.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:50 smithi196 ceph-mon[81815]: pgmap v1293: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:04:53.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:52 smithi096 ceph-mon[112443]: pgmap v1294: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:53.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:52 smithi196 ceph-mon[81815]: pgmap v1294: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:54.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:54.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:55.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:54 smithi096 ceph-mon[112443]: pgmap v1295: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:55.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:54 smithi196 ceph-mon[81815]: pgmap v1295: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:04:57.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:56 smithi096 ceph-mon[112443]: pgmap v1296: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:57.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:56 smithi196 ceph-mon[81815]: pgmap v1296: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:04:59.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:04:58 smithi096 ceph-mon[112443]: pgmap v1297: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:04:59.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:04:58 smithi196 ceph-mon[81815]: pgmap v1297: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:01.193 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:00 smithi196 ceph-mon[81815]: pgmap v1298: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:01.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:00 smithi096 ceph-mon[112443]: pgmap v1298: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:03.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:02 smithi096 ceph-mon[112443]: pgmap v1299: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:03.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:02 smithi196 ceph-mon[81815]: pgmap v1299: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:05.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:04 smithi096 ceph-mon[112443]: pgmap v1300: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:05.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:04 smithi196 ceph-mon[81815]: pgmap v1300: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:07.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:06 smithi096 ceph-mon[112443]: pgmap v1301: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:07.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:06 smithi196 ceph-mon[81815]: pgmap v1301: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:08.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:08 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:08.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:08 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:09.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:09 smithi096 ceph-mon[112443]: pgmap v1302: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:09.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:09 smithi196 ceph-mon[81815]: pgmap v1302: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:10.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:10 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:05:10.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:10 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:05:11.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:11 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:05:11.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:11 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:05:11.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:11 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:05:11.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:11 smithi096 ceph-mon[112443]: pgmap v1303: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:11 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:05:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:11 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:05:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:11 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' 2024-04-22T16:05:11.759 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:11 smithi196 ceph-mon[81815]: pgmap v1303: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:13.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:12 smithi096 ceph-mon[112443]: pgmap v1304: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:13.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:12 smithi196 ceph-mon[81815]: pgmap v1304: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:15.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:14 smithi096 ceph-mon[112443]: pgmap v1305: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:15.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:14 smithi196 ceph-mon[81815]: pgmap v1305: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:17.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:16 smithi096 ceph-mon[112443]: pgmap v1306: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:17.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:16 smithi196 ceph-mon[81815]: pgmap v1306: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:19.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:18 smithi096 ceph-mon[112443]: pgmap v1307: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:19.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:18 smithi196 ceph-mon[81815]: pgmap v1307: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:21.106 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:20 smithi096 ceph-mon[112443]: pgmap v1308: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:21.259 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:20 smithi196 ceph-mon[81815]: pgmap v1308: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:23.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:22 smithi096 ceph-mon[112443]: pgmap v1309: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:23.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:22 smithi196 ceph-mon[81815]: pgmap v1309: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:24.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:23 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:24.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:23 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:25.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:24 smithi096 ceph-mon[112443]: pgmap v1310: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:25.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:24 smithi196 ceph-mon[81815]: pgmap v1310: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:27.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:26 smithi096 ceph-mon[112443]: pgmap v1311: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:27.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:26 smithi196 ceph-mon[81815]: pgmap v1311: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:29.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:28 smithi096 ceph-mon[112443]: pgmap v1312: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:29.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:28 smithi196 ceph-mon[81815]: pgmap v1312: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:31.222 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:30 smithi096 ceph-mon[112443]: pgmap v1313: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:31.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:30 smithi196 ceph-mon[81815]: pgmap v1313: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:33.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:32 smithi096 ceph-mon[112443]: pgmap v1314: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:33.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:32 smithi196 ceph-mon[81815]: pgmap v1314: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:35.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:34 smithi096 ceph-mon[112443]: pgmap v1315: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:35.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:34 smithi196 ceph-mon[81815]: pgmap v1315: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:37.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:36 smithi096 ceph-mon[112443]: pgmap v1316: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:37.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:36 smithi196 ceph-mon[81815]: pgmap v1316: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:38.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:38 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:38.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:38 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:39.735 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:39 smithi096 ceph-mon[112443]: pgmap v1317: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:39.758 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:39 smithi196 ceph-mon[81815]: pgmap v1317: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:41.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:40 smithi096 ceph-mon[112443]: pgmap v1318: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:41.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:40 smithi196 ceph-mon[81815]: pgmap v1318: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:43.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:42 smithi096 ceph-mon[112443]: pgmap v1319: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:43.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:42 smithi196 ceph-mon[81815]: pgmap v1319: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:45.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:44 smithi096 ceph-mon[112443]: pgmap v1320: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:45.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:44 smithi196 ceph-mon[81815]: pgmap v1320: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-22T16:05:47.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:46 smithi196 ceph-mon[81815]: pgmap v1321: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:47.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:46 smithi096 ceph-mon[112443]: pgmap v1321: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T16:05:49.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:48 smithi096 ceph-mon[112443]: pgmap v1322: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:49.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:48 smithi196 ceph-mon[81815]: pgmap v1322: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:51.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:50 smithi096 ceph-mon[112443]: pgmap v1323: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:51.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:50 smithi196 ceph-mon[81815]: pgmap v1323: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T16:05:53.235 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:52 smithi096 ceph-mon[112443]: pgmap v1324: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:53.258 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:52 smithi196 ceph-mon[81815]: pgmap v1324: 65 pgs: 65 active+clean; 260 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T16:05:54.160 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-22T16:05:54.161 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-22T16:05:54.163 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T16:05:54.176 INFO:tasks.cephadm:Teardown begin 2024-04-22T16:05:54.176 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephadm.py", line 2246, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-22T16:05:54.178 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T16:05:54.205 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:53 smithi096 ceph-mon[112443]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:54.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T16:05:54.237 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:53 smithi196 ceph-mon[81815]: from='mgr.34104 172.21.15.96:0/3463990898' entity='mgr.smithi096.hhkvum' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:54.243 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T16:05:54.243 DEBUG:teuthology.orchestra.run.smithi096:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T16:05:54.268 DEBUG:teuthology.orchestra.run.smithi196:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T16:05:54.300 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T16:05:54.300 INFO:tasks.cephadm.mon.smithi096:Stopping mon.smithi096... 2024-04-22T16:05:54.301 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096 2024-04-22T16:05:54.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:54 smithi096 systemd[1]: Stopping Ceph mon.smithi096 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T16:05:54.893 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:54 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T16:05:54.626+0000 7f5dddad0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi096 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:05:54.894 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:54 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096[112439]: 2024-04-22T16:05:54.626+0000 7f5dddad0700 -1 mon.smithi096@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T16:05:55.180 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:54 smithi096 podman[168330]: 2024-04-22 16:05:54.892350309 +0000 UTC m=+0.392667779 container died cc6f8ad67833cb73f09be75789545f37f7b11913b0dfd53503ede9b300676d3d (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0) 2024-04-22T16:05:55.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:55 smithi096 podman[168330]: 2024-04-22 16:05:55.335767567 +0000 UTC m=+0.836085091 container cleanup cc6f8ad67833cb73f09be75789545f37f7b11913b0dfd53503ede9b300676d3d (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T16:05:55.485 INFO:journalctl@ceph.mon.smithi096.smithi096.stdout:Apr 22 16:05:55 smithi096 bash[168330]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi096 2024-04-22T16:05:55.861 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi096.service' 2024-04-22T16:05:55.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:05:55.929 INFO:tasks.cephadm.mon.smithi096:Stopped mon.smithi096 2024-04-22T16:05:55.930 INFO:tasks.cephadm.mon.smithi196:Stopping mon.smithi196... 2024-04-22T16:05:55.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi196 2024-04-22T16:05:56.248 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:55 smithi196 systemd[1]: Stopping Ceph mon.smithi196 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T16:05:56.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196[81810]: 2024-04-22T16:05:56.245+0000 7fe9564e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi196 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:05:56.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:56 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196[81810]: 2024-04-22T16:05:56.245+0000 7fe9564e7700 -1 mon.smithi196@1(peon) e3 *** Got Signal Terminated *** 2024-04-22T16:05:56.509 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:56 smithi196 podman[119289]: 2024-04-22 16:05:56.470403458 +0000 UTC m=+0.343609387 container died 351725aca606a722c5bdb0f92830f73e97222074e413700c75812b55a9bd3878 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-22T16:05:57.008 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:56 smithi196 podman[119289]: 2024-04-22 16:05:56.921171742 +0000 UTC m=+0.794377713 container cleanup 351725aca606a722c5bdb0f92830f73e97222074e413700c75812b55a9bd3878 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-22T16:05:57.009 INFO:journalctl@ceph.mon.smithi196.smithi196.stdout:Apr 22 16:05:56 smithi196 bash[119289]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-mon-smithi196 2024-04-22T16:05:57.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@mon.smithi196.service' 2024-04-22T16:05:57.451 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:05:57.451 INFO:tasks.cephadm.mon.smithi196:Stopped mon.smithi196 2024-04-22T16:05:57.451 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T16:05:57.451 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.0 2024-04-22T16:05:57.735 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:05:57 smithi096 systemd[1]: Stopping Ceph osd.0 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T16:05:58.235 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:05:57 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[124347]: 2024-04-22T16:05:57.790+0000 7f68bd0f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:05:58.235 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:05:57 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[124347]: 2024-04-22T16:05:57.790+0000 7f68bd0f8700 -1 osd.0 84 *** Got signal Terminated *** 2024-04-22T16:05:58.235 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:05:57 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0[124347]: 2024-04-22T16:05:57.790+0000 7f68bd0f8700 -1 osd.0 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:06:03.485 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:03 smithi096 podman[168465]: 2024-04-22 16:06:03.224129576 +0000 UTC m=+5.577159201 container died bbf14d18e92119bbc1195752427c96ba91a029ea97c1527dc20efaffdf9f35ef (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T16:06:03.877 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:03 smithi096 podman[168465]: 2024-04-22 16:06:03.632326658 +0000 UTC m=+5.985356282 container cleanup bbf14d18e92119bbc1195752427c96ba91a029ea97c1527dc20efaffdf9f35ef (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T16:06:03.877 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:03 smithi096 bash[168465]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0 2024-04-22T16:06:04.211 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:03 smithi096 podman[168483]: 2024-04-22 16:06:03.875637336 +0000 UTC m=+0.643731940 container remove bbf14d18e92119bbc1195752427c96ba91a029ea97c1527dc20efaffdf9f35ef (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-22T16:06:04.472 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:04 smithi096 podman[168575]: 2024-04-22 16:06:04.113575752 +0000 UTC m=+0.018970641 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:06:04.472 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:04 smithi096 podman[168575]: 2024-04-22 16:06:04.218719433 +0000 UTC m=+0.124114322 container create a0b5530d54fd0bb1fc370d99d3cd11b2bb1fabd0345b51efc8b378a82510d432 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-22T16:06:04.473 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:04 smithi096 podman[168575]: 2024-04-22 16:06:04.471339355 +0000 UTC m=+0.376734244 container init a0b5530d54fd0bb1fc370d99d3cd11b2bb1fabd0345b51efc8b378a82510d432 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416) 2024-04-22T16:06:04.735 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:04 smithi096 podman[168575]: 2024-04-22 16:06:04.47488469 +0000 UTC m=+0.380279588 container start a0b5530d54fd0bb1fc370d99d3cd11b2bb1fabd0345b51efc8b378a82510d432 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T16:06:04.735 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:04 smithi096 podman[168575]: 2024-04-22 16:06:04.554642462 +0000 UTC m=+0.460037362 container attach a0b5530d54fd0bb1fc370d99d3cd11b2bb1fabd0345b51efc8b378a82510d432 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-22T16:06:05.135 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:04 smithi096 podman[168575]: 2024-04-22 16:06:04.770664801 +0000 UTC m=+0.676059693 container died a0b5530d54fd0bb1fc370d99d3cd11b2bb1fabd0345b51efc8b378a82510d432 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T16:06:05.384 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T16:06:05.385 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:05 smithi096 podman[168600]: 2024-04-22 16:06:05.358398905 +0000 UTC m=+0.581468349 container remove a0b5530d54fd0bb1fc370d99d3cd11b2bb1fabd0345b51efc8b378a82510d432 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-22T16:06:05.385 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:05 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-22T16:06:05.385 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:05 smithi096 systemd[1]: Stopped Ceph osd.0 for d5e86090-00b9-11ef-bc93-c7b262605968. 2024-04-22T16:06:05.385 INFO:journalctl@ceph.osd.0.smithi096.stdout:Apr 22 16:06:05 smithi096 systemd[1]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.0.service: Consumed 20.709s CPU time. 2024-04-22T16:06:05.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:05.425 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T16:06:05.425 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T16:06:05.425 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.1 2024-04-22T16:06:05.648 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:05 smithi096 systemd[1]: Stopping Ceph osd.1 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T16:06:05.985 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:05 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[129791]: 2024-04-22T16:06:05.777+0000 7f7feefc7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:05.985 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:05 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[129791]: 2024-04-22T16:06:05.777+0000 7f7feefc7700 -1 osd.1 84 *** Got signal Terminated *** 2024-04-22T16:06:05.985 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:05 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1[129791]: 2024-04-22T16:06:05.777+0000 7f7feefc7700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:06:11.415 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:11 smithi096 podman[168680]: 2024-04-22 16:06:11.14828646 +0000 UTC m=+5.498527551 container died 8ca99058d1fb8c3bb7648d662b6dbc6e55faabe92bcf32d3454fc58fa7b011f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-04-22T16:06:11.735 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:11 smithi096 podman[168680]: 2024-04-22 16:06:11.519392785 +0000 UTC m=+5.869633865 container cleanup 8ca99058d1fb8c3bb7648d662b6dbc6e55faabe92bcf32d3454fc58fa7b011f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True) 2024-04-22T16:06:11.735 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:11 smithi096 bash[168680]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1 2024-04-22T16:06:12.131 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:11 smithi096 podman[168708]: 2024-04-22 16:06:11.82110166 +0000 UTC m=+0.673691835 container remove 8ca99058d1fb8c3bb7648d662b6dbc6e55faabe92bcf32d3454fc58fa7b011f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-22T16:06:12.451 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:12 smithi096 podman[168801]: 2024-04-22 16:06:12.033497956 +0000 UTC m=+0.018241990 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:06:12.451 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:12 smithi096 podman[168801]: 2024-04-22 16:06:12.13881398 +0000 UTC m=+0.123557999 container create f891066a881ee4e766e7452727317925d28f4e373139fda67a07ffbc5c05ad5a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T16:06:12.451 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:12 smithi096 podman[168801]: 2024-04-22 16:06:12.366566246 +0000 UTC m=+0.351310263 container init f891066a881ee4e766e7452727317925d28f4e373139fda67a07ffbc5c05ad5a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T16:06:12.451 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:12 smithi096 podman[168801]: 2024-04-22 16:06:12.369762852 +0000 UTC m=+0.354506877 container start f891066a881ee4e766e7452727317925d28f4e373139fda67a07ffbc5c05ad5a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-22T16:06:12.451 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:12 smithi096 podman[168801]: 2024-04-22 16:06:12.449656757 +0000 UTC m=+0.434400778 container attach f891066a881ee4e766e7452727317925d28f4e373139fda67a07ffbc5c05ad5a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T16:06:12.734 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 16:06:12 smithi096 podman[168801]: 2024-04-22 16:06:12.659131102 +0000 UTC m=+0.643875121 container died f891066a881ee4e766e7452727317925d28f4e373139fda67a07ffbc5c05ad5a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T16:06:13.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T16:06:13.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:13.377 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T16:06:13.377 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T16:06:13.377 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.2 2024-04-22T16:06:13.725 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:13 smithi096 systemd[1]: Stopping Ceph osd.2 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T16:06:13.985 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:13 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[135637]: 2024-04-22T16:06:13.722+0000 7f4f79ee3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:13.985 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:13 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[135637]: 2024-04-22T16:06:13.722+0000 7f4f79ee3700 -1 osd.2 84 *** Got signal Terminated *** 2024-04-22T16:06:13.985 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:13 smithi096 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2[135637]: 2024-04-22T16:06:13.722+0000 7f4f79ee3700 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:06:19.442 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:19 smithi096 podman[168906]: 2024-04-22 16:06:19.054085981 +0000 UTC m=+5.455592848 container died afcdcb7020b9ebcd860d256700bc844dab36b5874f90645a2526b8944e362901 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T16:06:19.735 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:19 smithi096 podman[168906]: 2024-04-22 16:06:19.539928662 +0000 UTC m=+5.941435517 container cleanup afcdcb7020b9ebcd860d256700bc844dab36b5874f90645a2526b8944e362901 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T16:06:19.735 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:19 smithi096 bash[168906]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2 2024-04-22T16:06:20.119 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:19 smithi096 podman[168926]: 2024-04-22 16:06:19.824980885 +0000 UTC m=+0.761810207 container remove afcdcb7020b9ebcd860d256700bc844dab36b5874f90645a2526b8944e362901 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T16:06:20.379 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:20 smithi096 podman[169018]: 2024-04-22 16:06:20.117708779 +0000 UTC m=+0.110248937 container create 96d683ee6fda99fe963dfb6d690edb2d4eecd151663ae634de16dec8f979b647 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T16:06:20.380 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:20 smithi096 podman[169018]: 2024-04-22 16:06:20.026069869 +0000 UTC m=+0.018610000 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:06:20.380 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:20 smithi096 podman[169018]: 2024-04-22 16:06:20.303601913 +0000 UTC m=+0.296142029 container init 96d683ee6fda99fe963dfb6d690edb2d4eecd151663ae634de16dec8f979b647 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T16:06:20.380 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:20 smithi096 podman[169018]: 2024-04-22 16:06:20.30659323 +0000 UTC m=+0.299133348 container start 96d683ee6fda99fe963dfb6d690edb2d4eecd151663ae634de16dec8f979b647 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-22T16:06:20.380 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:20 smithi096 podman[169018]: 2024-04-22 16:06:20.37826278 +0000 UTC m=+0.370802898 container attach 96d683ee6fda99fe963dfb6d690edb2d4eecd151663ae634de16dec8f979b647 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T16:06:20.735 INFO:journalctl@ceph.osd.2.smithi096.stdout:Apr 22 16:06:20 smithi096 podman[169018]: 2024-04-22 16:06:20.60232409 +0000 UTC m=+0.594864207 container died 96d683ee6fda99fe963dfb6d690edb2d4eecd151663ae634de16dec8f979b647 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T16:06:21.161 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T16:06:21.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:21.201 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T16:06:21.201 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T16:06:21.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.3 2024-04-22T16:06:21.508 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:21 smithi196 systemd[1]: Stopping Ceph osd.3 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T16:06:22.008 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:21 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[91389]: 2024-04-22T16:06:21.539+0000 7f1f5a7b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:22.008 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:21 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[91389]: 2024-04-22T16:06:21.539+0000 7f1f5a7b0700 -1 osd.3 84 *** Got signal Terminated *** 2024-04-22T16:06:22.008 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:21 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[91389]: 2024-04-22T16:06:21.539+0000 7f1f5a7b0700 -1 osd.3 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:06:24.759 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:24 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:24.338+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:25.259 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:24 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:24.779+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:25.759 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:25 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:25.372+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:26.259 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:25 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3[91389]: 2024-04-22T16:06:25.832+0000 7f1f52d9c700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:02.720974+0000 front 2024-04-22T16:06:02.720999+0000 (oldest deadline 2024-04-22T16:06:25.620774+0000) 2024-04-22T16:06:26.259 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:25 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:25.811+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:26.759 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:26 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:26.324+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:27.108 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:26 smithi196 podman[119426]: 2024-04-22 16:06:26.947984361 +0000 UTC m=+5.552861429 container died 984101ebadb9324e0344bd08b4e2290f32f16d9e998abbaefdf663ba2e0e1dd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T16:06:27.108 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:26 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:26.839+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:27.508 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:27 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:27.343+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:27.509 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:27 smithi196 podman[119426]: 2024-04-22 16:06:27.306633073 +0000 UTC m=+5.911510133 container cleanup 984101ebadb9324e0344bd08b4e2290f32f16d9e998abbaefdf663ba2e0e1dd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T16:06:27.509 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:27 smithi196 bash[119426]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3 2024-04-22T16:06:27.942 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:27 smithi196 podman[119445]: 2024-04-22 16:06:27.624724121 +0000 UTC m=+0.679445722 container remove 984101ebadb9324e0344bd08b4e2290f32f16d9e998abbaefdf663ba2e0e1dd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T16:06:27.942 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:27 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:27.813+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:28.221 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:27 smithi196 podman[119536]: 2024-04-22 16:06:27.843871665 +0000 UTC m=+0.018618845 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:06:28.221 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:27 smithi196 podman[119536]: 2024-04-22 16:06:27.942139483 +0000 UTC m=+0.116886653 container create 88580f3ca7989ac4853cebbfe1a0d63a4c306cf20b4debdec21e9b8c0583f48f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-22T16:06:28.221 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:28 smithi196 podman[119536]: 2024-04-22 16:06:28.161522703 +0000 UTC m=+0.336269887 container init 88580f3ca7989ac4853cebbfe1a0d63a4c306cf20b4debdec21e9b8c0583f48f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T16:06:28.221 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:28 smithi196 podman[119536]: 2024-04-22 16:06:28.165287749 +0000 UTC m=+0.340034929 container start 88580f3ca7989ac4853cebbfe1a0d63a4c306cf20b4debdec21e9b8c0583f48f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T16:06:28.221 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:28 smithi196 podman[119536]: 2024-04-22 16:06:28.219446863 +0000 UTC m=+0.394194042 container attach 88580f3ca7989ac4853cebbfe1a0d63a4c306cf20b4debdec21e9b8c0583f48f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T16:06:28.509 INFO:journalctl@ceph.osd.3.smithi196.stdout:Apr 22 16:06:28 smithi196 podman[119536]: 2024-04-22 16:06:28.453338205 +0000 UTC m=+0.628085380 container died 88580f3ca7989ac4853cebbfe1a0d63a4c306cf20b4debdec21e9b8c0583f48f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416) 2024-04-22T16:06:28.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:28 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:28.295+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:29.049 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:28 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:28.850+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:29.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T16:06:29.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:29.111 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T16:06:29.111 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T16:06:29.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.4 2024-04-22T16:06:29.315 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:29 smithi196 systemd[1]: Stopping Ceph osd.4 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T16:06:29.758 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:29.313+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:29.759 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:29.451+0000 7f75108f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:29.759 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:29.451+0000 7f75108f8700 -1 osd.4 84 *** Got signal Terminated *** 2024-04-22T16:06:29.759 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:29.451+0000 7f75108f8700 -1 osd.4 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:06:30.258 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:29 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:29.818+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:30.758 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:30 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:30.265+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:31.233 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:30 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:30.860+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:31.508 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:31 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:31.232+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:32.148 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:31 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:31.850+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:32.508 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:32 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:32.194+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:33.245 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:32 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:32.838+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:33.245 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:32 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:32.838+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:07.834475+0000 front 2024-04-22T16:06:07.834408+0000 (oldest deadline 2024-04-22T16:06:31.934390+0000) 2024-04-22T16:06:33.508 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:33 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:33.243+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:33.508 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:33 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:33.243+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:34.245 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:33 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:33.815+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:05:59.232556+0000 front 2024-04-22T16:05:59.232575+0000 (oldest deadline 2024-04-22T16:06:24.532466+0000) 2024-04-22T16:06:34.245 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:33 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4[96596]: 2024-04-22T16:06:33.815+0000 7f750ceec700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:07.834475+0000 front 2024-04-22T16:06:07.834408+0000 (oldest deadline 2024-04-22T16:06:31.934390+0000) 2024-04-22T16:06:34.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:34 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:34.243+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:34.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:34 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:34.243+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:35.245 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:34 smithi196 podman[119640]: 2024-04-22 16:06:34.973261863 +0000 UTC m=+5.634875413 container died ad74967cb5b79696a1fee88446eecba3bef328257e437dd6ad7f581f2425568e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T16:06:35.508 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:35 smithi196 podman[119640]: 2024-04-22 16:06:35.334277217 +0000 UTC m=+5.995890800 container cleanup ad74967cb5b79696a1fee88446eecba3bef328257e437dd6ad7f581f2425568e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T16:06:35.508 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:35 smithi196 bash[119640]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4 2024-04-22T16:06:35.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:35 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:35.255+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:35.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:35 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:35.255+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:35.962 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:35 smithi196 podman[119658]: 2024-04-22 16:06:35.626949748 +0000 UTC m=+0.649508917 container remove ad74967cb5b79696a1fee88446eecba3bef328257e437dd6ad7f581f2425568e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T16:06:36.248 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:35 smithi196 podman[119746]: 2024-04-22 16:06:35.961067558 +0000 UTC m=+0.114257765 container create 3094a7fcf2c664a4f2aef1534a19927fb9884855e09b5f293fe543eefddf9230 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T16:06:36.248 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:35 smithi196 podman[119746]: 2024-04-22 16:06:35.865173927 +0000 UTC m=+0.018364138 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:06:36.248 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:36 smithi196 podman[119746]: 2024-04-22 16:06:36.197191442 +0000 UTC m=+0.350381648 container init 3094a7fcf2c664a4f2aef1534a19927fb9884855e09b5f293fe543eefddf9230 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-04-22T16:06:36.248 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:36 smithi196 podman[119746]: 2024-04-22 16:06:36.200424521 +0000 UTC m=+0.353614728 container start 3094a7fcf2c664a4f2aef1534a19927fb9884855e09b5f293fe543eefddf9230 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T16:06:36.500 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:36 smithi196 podman[119746]: 2024-04-22 16:06:36.297211107 +0000 UTC m=+0.450401324 container attach 3094a7fcf2c664a4f2aef1534a19927fb9884855e09b5f293fe543eefddf9230 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T16:06:36.500 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:36 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:36.246+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:36.500 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:36 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:36.246+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:36.758 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:36 smithi196 podman[119746]: 2024-04-22 16:06:36.499920515 +0000 UTC m=+0.653110731 container died 3094a7fcf2c664a4f2aef1534a19927fb9884855e09b5f293fe543eefddf9230 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T16:06:37.058 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 22 16:06:37 smithi196 podman[119772]: 2024-04-22 16:06:37.034122914 +0000 UTC m=+0.527108996 container remove 3094a7fcf2c664a4f2aef1534a19927fb9884855e09b5f293fe543eefddf9230 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T16:06:37.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T16:06:37.101 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:37.101 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T16:06:37.101 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T16:06:37.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.5 2024-04-22T16:06:37.422 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:37 smithi196 systemd[1]: Stopping Ceph osd.5 for d5e86090-00b9-11ef-bc93-c7b262605968... 2024-04-22T16:06:37.422 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:37 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:37.247+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:37.422 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:37 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:37.247+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:37.422 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:37 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:37.420+0000 7f09ac790700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:37.422 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:37 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:37.420+0000 7f09ac790700 -1 osd.5 84 *** Got signal Terminated *** 2024-04-22T16:06:37.758 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:37 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:37.420+0000 7f09ac790700 -1 osd.5 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:06:38.759 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:38 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:38.260+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:38.759 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:38 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:38.260+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:39.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:39 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:39.310+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:39.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:39 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:39.310+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:40.759 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:40 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:40.293+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:40.759 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:40 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:40.293+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:41.508 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:41 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:41.244+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:41.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:41 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:41.244+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:42.508 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:42 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:42.227+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6806 osd.0 since back 2024-04-22T16:06:00.745348+0000 front 2024-04-22T16:06:00.745339+0000 (oldest deadline 2024-04-22T16:06:24.245173+0000) 2024-04-22T16:06:42.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:42 smithi196 ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5[101786]: 2024-04-22T16:06:42.227+0000 7f09a4d7c700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.96:6814 osd.1 since back 2024-04-22T16:06:06.446285+0000 front 2024-04-22T16:06:06.446293+0000 (oldest deadline 2024-04-22T16:06:32.346108+0000) 2024-04-22T16:06:43.005 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:42 smithi196 podman[119853]: 2024-04-22 16:06:42.742077217 +0000 UTC m=+5.416760934 container died 1628335216db531c67d036812dd6b1f2867c168b2d7ebd24c6fc3d9e5b819476 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T16:06:43.259 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:43 smithi196 podman[119853]: 2024-04-22 16:06:43.10324325 +0000 UTC m=+5.777926977 container cleanup 1628335216db531c67d036812dd6b1f2867c168b2d7ebd24c6fc3d9e5b819476 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T16:06:43.259 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:43 smithi196 bash[119853]: ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5 2024-04-22T16:06:43.722 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:43 smithi196 podman[119870]: 2024-04-22 16:06:43.396336988 +0000 UTC m=+0.646482246 container remove 1628335216db531c67d036812dd6b1f2867c168b2d7ebd24c6fc3d9e5b819476 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-22T16:06:44.009 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:43 smithi196 podman[119963]: 2024-04-22 16:06:43.623436568 +0000 UTC m=+0.018243760 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:06:44.009 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:43 smithi196 podman[119963]: 2024-04-22 16:06:43.730472981 +0000 UTC m=+0.125280172 container create 80c79050c43c7a1900f8aec707e998d9f87999b448b18cf96760d565889b1422 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T16:06:44.009 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:43 smithi196 podman[119963]: 2024-04-22 16:06:43.949338121 +0000 UTC m=+0.344145314 container init 80c79050c43c7a1900f8aec707e998d9f87999b448b18cf96760d565889b1422 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T16:06:44.009 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:43 smithi196 podman[119963]: 2024-04-22 16:06:43.952742048 +0000 UTC m=+0.347549250 container start 80c79050c43c7a1900f8aec707e998d9f87999b448b18cf96760d565889b1422 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-22T16:06:44.508 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:44 smithi196 podman[119963]: 2024-04-22 16:06:44.034261952 +0000 UTC m=+0.429069144 container attach 80c79050c43c7a1900f8aec707e998d9f87999b448b18cf96760d565889b1422 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T16:06:44.509 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 22 16:06:44 smithi196 podman[119963]: 2024-04-22 16:06:44.258827854 +0000 UTC m=+0.653635038 container died 80c79050c43c7a1900f8aec707e998d9f87999b448b18cf96760d565889b1422 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-d5e86090-00b9-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T16:06:44.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d5e86090-00b9-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T16:06:44.876 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:44.877 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T16:06:44.877 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d5e86090-00b9-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T16:06:45.034 INFO:teuthology.orchestra.run.smithi096.stdout:Deleting cluster with fsid: d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T16:06:49.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[75874]: fuse finished with error 0 and tester_r 0 2024-04-22T16:07:10.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d5e86090-00b9-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T16:07:10.424 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T16:07:26.686 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T16:07:26.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T16:07:26.745 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T16:07:26.747 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344/remote/smithi096/crash 2024-04-22T16:07:26.748 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash -- . 2024-04-22T16:07:26.789 INFO:teuthology.orchestra.run.smithi096.stderr:tar: /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T16:07:26.789 INFO:teuthology.orchestra.run.smithi096.stderr:tar: Error is not recoverable: exiting now 2024-04-22T16:07:26.791 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344/remote/smithi196/crash 2024-04-22T16:07:26.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash -- . 2024-04-22T16:07:26.815 INFO:teuthology.orchestra.run.smithi196.stderr:tar: /var/lib/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T16:07:26.815 INFO:teuthology.orchestra.run.smithi196.stderr:tar: Error is not recoverable: exiting now 2024-04-22T16:07:26.816 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T16:07:26.816 DEBUG:teuthology.orchestra.run.smithi096:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-22T16:07:26.868 INFO:teuthology.orchestra.run.smithi096.stdout:2024-04-22T15:30:00.000170+0000 mon.smithi096 (mon.0) 686 : cluster [ERR] fs cephfs is degraded 2024-04-22T16:07:26.868 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-22T16:07:26.868 INFO:tasks.cephadm:Compressing logs... 2024-04-22T16:07:26.869 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T16:07:26.911 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T16:07:26.935 INFO:teuthology.orchestra.run.smithi096.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-22T16:07:26.935 INFO:teuthology.orchestra.run.smithi096.stderr:: No such file or directory 2024-04-22T16:07:26.935 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.log 2024-04-22T16:07:26.935 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T16:07:26.935 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi096.log 2024-04-22T16:07:26.935 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T16:07:26.945 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mon.smithi096.log 2024-04-22T16:07:26.945 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi096.log: /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T16:07:26.945 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mon.smithi096.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi096.drkkdi.log 2024-04-22T16:07:26.946 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T16:07:26.946 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi096.drkkdi.log: 89.5% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T16:07:26.946 INFO:teuthology.orchestra.run.smithi096.stderr: 94.5% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi096.log.gzgzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mgr.smithi096.hhkvum.log 2024-04-22T16:07:26.946 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-04-22T16:07:26.946 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T16:07:26.946 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mgr.smithi096.hhkvum.log: 91.4% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T16:07:26.948 INFO:teuthology.orchestra.run.smithi196.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/ceph-client.1.log 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.log 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.1.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi196.log 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi196.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mgr.smithi196.gdaegc.log 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mon.smithi196.log 2024-04-22T16:07:26.949 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mgr.smithi196.gdaegc.log: 89.6% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T16:07:26.950 INFO:teuthology.orchestra.run.smithi196.stderr: 94.5% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi196.log.gz 2024-04-22T16:07:26.950 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T16:07:26.950 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mon.smithi196.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi196.hwpcvg.log 2024-04-22T16:07:26.950 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.cephadm.log: 91.5% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T16:07:26.950 INFO:teuthology.orchestra.run.smithi196.stderr: 85.8% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T16:07:26.950 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi196.pbvqlq.log 2024-04-22T16:07:26.950 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi196.hwpcvg.log: gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T16:07:26.958 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi096.dymqlv.log 2024-04-22T16:07:26.958 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.cephadm.log: 85.8% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T16:07:26.959 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi196.pbvqlq.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T16:07:26.967 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T16:07:26.977 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T16:07:26.981 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi096.dymqlv.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T16:07:27.004 INFO:teuthology.orchestra.run.smithi196.stderr: 91.1% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T16:07:27.022 INFO:teuthology.orchestra.run.smithi196.stderr: 90.1% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mgr.smithi196.gdaegc.log.gz 2024-04-22T16:07:27.032 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 91.2% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T16:07:27.038 INFO:teuthology.orchestra.run.smithi196.stderr: 94.6% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi196.pbvqlq.log.gz 2024-04-22T16:07:27.056 INFO:teuthology.orchestra.run.smithi096.stderr: 90.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T16:07:27.527 INFO:teuthology.orchestra.run.smithi196.stderr: 92.5% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mon.smithi196.log.gz 2024-04-22T16:07:27.639 INFO:teuthology.orchestra.run.smithi096.stderr: 89.6% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mgr.smithi096.hhkvum.log.gz 2024-04-22T16:07:29.320 INFO:teuthology.orchestra.run.smithi096.stderr: 89.9% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mon.smithi096.log.gz 2024-04-22T16:07:30.793 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T16:07:31.737 INFO:teuthology.orchestra.run.smithi096.stderr: 93.7% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T16:07:32.178 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T16:07:32.593 INFO:teuthology.orchestra.run.smithi096.stderr: 93.7% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T16:07:33.424 INFO:teuthology.orchestra.run.smithi196.stderr: 94.8% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi196.hwpcvg.log.gz 2024-04-22T16:07:33.776 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T16:07:34.110 INFO:teuthology.orchestra.run.smithi096.stderr: 93.8% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T16:07:41.772 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-22T16:07:41.773 INFO:teuthology.orchestra.run.smithi196.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-04-22T16:07:41.774 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-04-22T16:07:41.774 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m14.848s 2024-04-22T16:07:41.774 INFO:teuthology.orchestra.run.smithi196.stderr:user 0m36.867s 2024-04-22T16:07:41.774 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m1.130s 2024-04-22T16:07:41.797 INFO:teuthology.orchestra.run.smithi096.stderr: 94.8% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi096.dymqlv.log.gz 2024-04-22T16:07:41.938 INFO:teuthology.orchestra.run.smithi096.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-22T16:08:53.364 INFO:teuthology.orchestra.run.smithi096.stderr: 92.6% -- replaced with /var/log/ceph/d5e86090-00b9-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi096.drkkdi.log.gz 2024-04-22T16:08:53.366 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-04-22T16:08:53.366 INFO:teuthology.orchestra.run.smithi096.stderr:real 1m26.441s 2024-04-22T16:08:53.366 INFO:teuthology.orchestra.run.smithi096.stderr:user 2m12.652s 2024-04-22T16:08:53.366 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m4.116s 2024-04-22T16:08:53.367 INFO:tasks.cephadm:Archiving logs... 2024-04-22T16:08:53.367 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344/remote/smithi096/log 2024-04-22T16:08:53.369 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T16:09:12.187 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344/remote/smithi196/log 2024-04-22T16:09:12.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T16:09:17.259 INFO:tasks.cephadm:Removing cluster... 2024-04-22T16:09:17.259 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d5e86090-00b9-11ef-bc93-c7b262605968 --force 2024-04-22T16:09:17.412 INFO:teuthology.orchestra.run.smithi096.stdout:Deleting cluster with fsid: d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T16:09:17.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d5e86090-00b9-11ef-bc93-c7b262605968 --force 2024-04-22T16:09:17.973 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: d5e86090-00b9-11ef-bc93-c7b262605968 2024-04-22T16:09:18.262 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T16:09:18.263 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T16:09:18.279 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T16:09:18.294 INFO:tasks.cephadm:Teardown complete 2024-04-22T16:09:18.294 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T16:09:18.305 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-22T16:09:18.305 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T16:09:18.306 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T16:09:18.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T16:09:18.407 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T16:09:18.418 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T16:09:18.418 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T16:09:18.421 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T16:09:18.432 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-04-22T16:09:18.434 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-04-22T16:09:18.436 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T16:09:18.436 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-04-22T16:09:18.436 INFO:teuthology.orchestra.run.smithi096.stdout:^- hv01.front.sepia.ceph.com 2 6 377 29 -2088us[-2088us] +/- 574ms 2024-04-22T16:09:18.436 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv02.front.sepia.ceph.com 3 7 377 91 +259us[ +263us] +/- 53ms 2024-04-22T16:09:18.436 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 -394us[ -394us] +/- 81ms 2024-04-22T16:09:18.437 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T16:09:18.439 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T16:09:18.439 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-04-22T16:09:18.439 INFO:teuthology.orchestra.run.smithi196.stdout:^- hv01.front.sepia.ceph.com 2 6 377 27 -2042us[-2064us] +/- 574ms 2024-04-22T16:09:18.439 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 +235us[ +212us] +/- 54ms 2024-04-22T16:09:18.439 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -355us[ -377us] +/- 81ms 2024-04-22T16:09:18.439 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T16:09:18.439 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T16:09:18.449 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T16:09:18.450 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T16:09:18.459 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T16:09:18.556 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T16:09:18.627 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T16:09:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:18.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T16:09:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:18.683 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T16:09:18.694 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T16:09:18.704 INFO:teuthology.task.internal:Duration was 4156.645268 seconds 2024-04-22T16:09:18.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T16:09:18.715 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T16:09:18.715 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T16:09:18.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T16:09:18.751 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T16:09:18.759 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T16:09:19.050 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T16:09:19.050 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T16:09:19.050 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T16:09:19.086 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-04-22T16:09:19.086 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T16:09:19.111 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T16:09:19.111 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T16:09:19.129 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T16:09:19.176 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T16:09:19.176 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T16:09:19.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T16:09:20.395 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T16:09:20.407 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T16:09:20.407 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T16:09:20.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T16:09:20.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T16:09:20.474 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T16:09:20.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T16:09:20.496 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-04-22T16:09:20.521 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-04-22T16:09:20.548 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T16:09:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:20.591 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T16:09:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:20.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T16:09:20.624 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T16:09:20.625 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344/remote/smithi096 2024-04-22T16:09:20.625 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T16:09:20.693 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344/remote/smithi196 2024-04-22T16:09:20.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T16:09:20.731 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T16:09:20.731 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T16:09:20.734 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T16:09:20.780 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T16:09:20.793 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T16:09:20.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T16:09:20.805 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T16:09:20.805 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T16:09:20.807 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T16:09:20.819 INFO:teuthology.orchestra.run.smithi096.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 22 16:09 /home/ubuntu/cephtest 2024-04-22T16:09:20.819 INFO:teuthology.orchestra.run.smithi096.stdout: 550851 4 d--------- 2 ubuntu ubuntu 4096 Apr 22 15:16 /home/ubuntu/cephtest/mnt.0 2024-04-22T16:09:20.820 INFO:teuthology.orchestra.run.smithi096.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-22T16:09:20.820 INFO:teuthology.orchestra.run.smithi096.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-22T16:09:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:20.822 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi096 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-22T16:09:20.823 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T16:09:20.836 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T16:09:20.849 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-22T16:09:20.850 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T16:09:20.876 DEBUG:teuthology.nuke:shortname: smithi096 2024-04-22T16:09:20.876 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T16:09:20.900 DEBUG:teuthology.nuke:shortname: smithi196 2024-04-22T16:09:20.900 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T16:09:20.903 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 14:52:18.526873', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHlvGdYD94z82Et7ziTp7r2vpxLWc3ovL8uK8FtNWRt4atHctIMAyMUHfU0wfScGwApJxIoo8QJHVaKlGn1J+wo='} 2024-04-22T16:09:20.926 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668344', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 14:52:18.525569', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvkyQLaIR3B6CoD65dCiMwm4qB5viW/gh1YcPYtVK1Odr0uwF4gZnu2zK6kH7+4cNp9FhbFGY0bDsTapUACCz8='} 2024-04-22T16:09:20.951 INFO:teuthology.orchestra.console.smithi096:Power off 2024-04-22T16:09:20.951 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T16:09:20.973 DEBUG:teuthology.orchestra.console.smithi096:power off output: Chassis Power Control: Down/Off 2024-04-22T16:09:20.973 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:09:20.990 INFO:teuthology.orchestra.console.smithi196:Power off 2024-04-22T16:09:20.990 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T16:09:20.999 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-04-22T16:09:21.012 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-04-22T16:09:21.012 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:09:21.033 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-04-22T16:09:25.001 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:09:25.034 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:09:25.113 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-04-22T16:09:25.145 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-04-22T16:09:29.118 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:09:29.145 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:09:29.227 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is off 2024-04-22T16:09:29.254 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-04-22T16:09:29.329 INFO:teuthology.orchestra.console.smithi096:Power off completed 2024-04-22T16:09:29.355 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-04-22T16:09:29.460 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4156.645267724991 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_adking@teuthology status: fail success: false 2024-04-22T16:09:29.460 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T16:09:29.584 INFO:teuthology.run:FAIL